The COVID pandemic from early 2020 led to the requirement for many employees to work from home (WFH). IT Departments scrambled to enable this capability, many making use of Microsoft (MS) Teams that was already bundled with their Microsoft 365 licences.
The rapid enabling and uptake (rather than an actual ‘implementation’) of MS Teams was more often than not achieved without much consideration for recordkeeping requirements or an overall plan for using Microsoft 365.
MS Teams became popular quickly, increasing from around 30 million active users daily in early 2020 to around 250 million by mid 2021 (Source: ZDNet quoting Microsoft latest results). End-users could chat with each other and with external people (and on their phones too!), have video meetings, create new teams with channels and private channels, share and collaborate on content via the ‘Files’ tab in Teams, create and manage tasks, and more. They also continued to use email.
Anecdotal evidence suggests that the capture of records to on-premise electronic document and records management system (EDRMS) declined from early 2020. One reason suggested for this was that it was too hard to save some cloud records such as Teams chats or content from the Files tab to an on-premise system. Alternative approaches for managing records with Microsoft 365 began to evolve.
This post discusses four approaches to managing records in Microsoft 365, summarised in the diagram below.
Which approach have you taken? Answer my (anonymous) short survey here (Microsoft Forms).
Approach 1 – EDRMS + key Microsoft 365 applications to create and capture
This model has two elements:
- Retaining an existing centralised recordkeeping system (the EDRMS) for the storage of records.
- Using email, Teams, SharePoint or OneDrive to create or capture records to be copied to the EDRMS, and leaving other content (in theory non-records) ‘in place’.
The main positive aspect of this model is that records are (in theory) captured and managed in the EDRMS with all the traditional recordkeeping options. Some leading EDRMS vendors now offer solutions that integrate with Microsoft 365 and make it easier to capture records from Microsoft 365. But the model is still based on a centralised recordkeeping system and the requirement for end-users to copy content identified as records.
The main negative aspects of this model include the following points:
- End-users still have to identify and copy records to the EDRMS.
- Not all records created or captured in Microsoft 365 can be copied to the EDRMS.
- Additional products or add-ons may be required to enable the copying.
- The record is copied to the EDRMS, not moved, so remains in place with no controls.
- Records that remain stored in Microsoft 365 applications may not be subject to the same degree of recordkeeping controls available in the EDRMS. Unless they acquire a third-party product (see next approach) to overcome this problem (which is unlikely for cost reasons), organisations must use the out of the box recordkeeping capability in Microsoft 365. This capability may not meet all requirements for keeping records if not properly configured.
- There is a real risk that some records that remain in Microsoft 365 may be lost, especially if settings allow content to be deleted and there is no retention policy or backup.
- EDRM system admins and records managers will need to learn a lot more about Microsoft 365.
- The unified logs in Microsoft 365 only retain the details for 3 months (E3) or 12 months (E5) – although SharePoint’s versioning history can provide a lot of ‘modified’ event metadata for the life of the document (up to the the maximum number of versions allowed). (Update: Microsoft 365 customers can retain the audit log for up to 10 years with an add-on license. Many export audit data to a SEIM such as Azure Sentinel where they can retain the log for as long as they want.)
On a positive note, however, Microsoft 365 includes a wide range of search, audit, monitoring and reporting tools, as well as security and protection controls, that improve the ability for records managers to find, manage and protect records (or potential records) in Exchange mailboxes, MS Teams chats and posts, SharePoint sites and OneDrive accounts AND put that content on a legal hold. So, as long as those options are enabled, the risk of losing records is reduced.
Approach 2 – Third-party application + Microsoft 365 applications for creation, capture and storage
A number of Microsoft partners have developed applications to manage records in Microsoft 365. Several have been available for a decade or more, originally designed to manage records primarily in on-premise SharePoint environments.
Most of these third-party applications were developed to comply with the same recordkeeping standards used by EDRMS vendors. These applications are generally either:
- Replacements for EDRM systems (often requiring migration from the EDRMS).
- New implementations where there was no EDRMS beforehand.
It is not common to see both an EDRMS and one of these third-party products being used together, because of licensing cost reasons.
The main positive aspect of using a third-party dedicated application is that records created or captured in Microsoft 365 can be stay there and be managed according to recordkeeping requirements. Some of these applications are invisible to end-users, making them even more attractive.
The main potential negative aspect of using a third-party application, which is the same for any other vendor product, is that it creates a dependency on the vendor to maintain the product. Microsoft 365 continues to evolve and any third-party application must keep up with these changes. Two questions might be asked:
- Will this dependency become a ‘tech debt’ liability in the future, if a ‘better’ option comes along?
- How hard will it be to transfer to a different vendor in the future? Generally speaking this is less likely if the vendor is an established Microsoft partner, but the question should still be asked. For example, many organisations decided to use the Google suite of products but have now decided to use Microsoft 365.
Organisations seeking to implement third-party applications to manage records in Microsoft 365 should have a very detailed understanding of the underlying Microsoft 365 environment beforehand and the impact the third-party application might have on this environment. Some of the considerations might include:
- The requirement to provide the third-party vendor with admin (including global admin) access to the Microsoft 365 tenant. Is this a security concern?
- The location of records – in some cases, third-party vendors may use, move or back up content to one of their Microsoft 365 tenants. Is this a security concern? How can you monitor activity on your content if it’s not in your tenant?
- The use of the central Term Store or Content Types to support the application. Will this create a dependency or make it harder for people to work, for example by requiring end-users to select Content Types or add metadata.
- Changes to SharePoint settings and architecture, including the addition of hidden columns. Will these changes be consistent with your own architecture model?
- How and where event metadata (audit logs) will be captured and managed.
- How retention outcomes will be managed.
Approach 3 – One or more Microsoft 365 applications are the default ‘recordkeeping systems’ (no EDRMS or other application)
This approach focuses on the applications where most records are likely to be created or captured in Microsoft 365 – Exchange mailboxes, MS Teams, SharePoint, and OneDrive for Business – and therefore considers other content created and/or stored in other Microsoft 365 applications (e.g., Yammer, Forms, Planner/Tasks, etc) as being non-records.
There are several variations on this model including the following:
- Outlook and Teams are the primary ‘recordkeeping systems’ as they are the two applications that are most used. Teams has been positioned as the primary interface for both SharePoint and OneDrive (via the ‘Files’ tab). The ability to also access both SharePoint and OneDrive from File Explorer via the sync option makes it even less likely that SharePoint or OneDrive will be accessed by end-users.
- All four applications are the recordkeeping systems, using the various controls and settings available in the various admin portals, as well as the Compliance admin portal for retention policies.
- SharePoint is the primary recordkeeping system, configured to mimic EDRMS capability. In this case, end-users would be expected to copy emails from Outlook or records from OneDrive, similar to the way they would have to do this for an EDRMS. Various controls and settings, such as ‘back end’ retention policies, might be applied to the other main applications to ensure that any records in those systems (such as Teams chats or emails) are not destroyed before a given period.
The main positive aspects of this approach are (a) simplicity and (b) cost savings, mostly by not having to purchase an EDRMS or third-party application.
However, these potential positives should not compromise the requirement for both IT and records management to have a very good understanding of, detailed approach to, and governance for, managing records in Microsoft 365. In other words, simply saying that one or more of these four applications is the recordkeeping system is not sufficient; additional work is required to ensure that records stored in them are managed appropriately.
There are several potential negative aspects of this model:
- With the exception of SharePoint, none of the other three systems can be configured to manage records based on standards used for EDRM systems. Given that SharePoint has been positioned behind the Teams user interface, and SharePoint document libraries can be synced via Teams to File Explorer, any recordkeeping functionality configured in SharePoint should in theory be accessible or useable via Teams and possibly also File Explorer, but this is mostly not the case. So, SharePoint on its own, accessed via the browser only, is not really an option. Additionally, without effective controls, the Files (SharePoint) element of Teams has the potential to become the future equivalent of legacy network file shares full of redundant, outdated and trivial content.
- If only one or two systems are considered to be the only recordkeeping systems, there is a risk that records may not be saved and/or could be lost, especially if end-users can delete records and there is no back up option.
- Managing records in this way requires both access to and a very good understanding of the applications designated to be the recordkeeping systems by both IT and records managers.
- Retention policies (either the base level information governance or more expensive records management) may not be adequate, in terms of both application and coverage, and retention outcome management.
- Exporting the records to another system or transferring them to another organisation, could become a complex task.
- Accessing audit logs over a long period (see first approach, last dot point, above).
Approach 4 – All of Microsoft 365 is the recordkeeping system
This approach is similar to the previous one except that it takes a broader approach and requires a degree of ‘letting go’ of the standards used by EDRMS systems (and third-party products). It is also the Microsoft default.
The approach assumes that records may be created or captured anywhere in Microsoft 365, saved to Microsoft 365 via archive connectors, or accessed (subject to access controls) via search connectors. Records are managed ‘in place’, meaning wherever they are created or captured, using a range of tools already available in Microsoft 365. Additional ‘in place’ controls allow certain items to be declared as records.
The approach requires both a very good technical understanding of the Microsoft 365 environment and effective governance by IT and records managers. If internal skills are lacking, it may also require a third-party organisation to implement the system – but based on what recordkeeping model? A reliance on a third-party to implement the recordkeeping elements has several risks (see below).
The main positives of this approach include the following:
- Records that are created or captured in the Microsoft 365 environment remain there. There is no requirement to copy them to a separate system.
- Some records, such as emails, can be copied to SharePoint if required.
- The combination of Teams and SharePoint sites allows for multiple models to manage records – for example, high value records could be managed in a dedicated SharePoint site with multiple dedicated libraries and additional controls (metadata, retention, permissions etc), whereas low level records could be managed in the single ‘Documents’ library presented as the Files tab in a Team, or via File Explorer.
- All the content (records and non-records) stored across Exchange, Teams, SharePoint/One drive can be searched (subject to roles and permissions). This allows records managers (and others such as Legal) to identify if records may be hidden in personal mailboxes or Teams chat or OneDrive accounts.
- Minimum retention periods can be applied to all the content (not just records), ensuring that records that may be hidden in Teams chats, OneDrive accounts, or personal mailboxes, will be retained for minimum periods. This option also helps to reduce the volume of redundant, outdated and trivial content that may build up over time otherwise.
- Retention labels can be applied, including automatically (and using machine learning), to records in mailboxes, SharePoint sites and OneDrive accounts (but not Teams chats or posts, yet).
The main negatives of this model are the same as those listed for the previous model with more focus on the need for both IT and records managers to have a very detailed understanding of and establish effective governance for the entire environment where records may be created or captured, not just the main four applications. This requires some effort to achieve and should not be understated. It is not uncommon to see IT staff with Global Admin managing the entire Microsoft 365 environment using default settings and/or records managers will little technical knowledge or appropriate access struggling to understand how the environment works and drawing on experience with EDRM systems.
Some organisations may engage third-party implementation specialists to configure and set up the environment. Organisations that decide to go down this path should ensure they have the details of this configuration and can support it in the longer run, or the environment (or parts of it) could end up becoming difficult to manage or support over time.
Approach 5 – A potential future model
Microsoft 365 includes a wide range of settings, options and capabilities that have a significant impact on the way records can and will be managed across Microsoft 365 in the future.
Microsoft 365 will continue to evolve over time, including in ways that will support how records are managed. But it is important to keep in mind that Microsoft 365, or its component applications, is not and will never be an EDRMS based on standards such as DOD 5015.2. Microsoft 365 is too complex, and the volume and type of content stored in it too large, for any part of it to be considered the ‘records management’ system.
A new approach is required for the identification and management of records. This approach may draw on existing recordkeeping standards and concepts but is likely to rely more heavily on new and evolving ways to work with information, including records.
Some of these ways have been around for a decade or so in the form of graph-based machine learning (ML), process automation, artificial intelligence (AI). Examples include Google, Facebook, LinkedIn, Netflix, Amazon, eBay and so on. These examples have one thing in common – they all take advantage of the various ‘signals’ and ‘digital exhaust’ voluntarily offered by their users to identify and present things that match your interests – jobs, friends, things to purchase, movies. Post something on Facebook or (perhaps) talk about a particular subject near your phone, and related ads will appear.
So, what is different about Microsoft 365? End-users are related to each other thanks to Active Directory, they connect and communicate with others via email or Teams, they share content, they attend meetings. All of these (and a lot more) signals feed into the underlying Graph and allow connections to be made and suggestions.
There is nothing stopping organisations setting up dedicated SharePoint sites with multiple well-named libraries to manage certain records and leaving other content and records to the world of Teams Files. But all of this information can be related based on context, including who created it, what team that person was in, who they connect with, what access do they have and so on.
Perhaps by 2035, the primary approach to records management will be relying on all the digital connections and signals, machine learning, the Graph and AI to identify all related records in context, not just the ones neatly placed in a SharePoint document library. Records may be automatically identified as important and needing stronger controls based on this context – who created, sent or received it, whether it relates to a subject that is trending (or was in the past).
Instead of just a simple pre-defined aggregation of records (which will still be a valid way to aggregate records), future aggregations will include a wider range of content, created automatically, likely presented in the form of ‘cards’.
Viva Topics is an interesting pre-cursor to this possible future model.
The following text is from the Microsoft page ‘Alexandria in Microsoft Viva Topics: from big data to big knowledge‘:
Looking further ahead, Alexandria’s ability to extract information automatically gives us the opportunity to customize the knowledge discovery process. By automatically retrieving the set of types and properties being talked about in an organization’s documents, Alexandria can create a knowledge base with a bespoke schema exactly tailored to the needs of each organization and using the familiar language and terminology that people in the organization are used to. Read more about the proposed schema-based design in our research paper.
We are only beginning to dream of the experiences that an automatically created and updated knowledge base can enable, but it is already clear that it could transform the future of how we work. The era of big knowledge is coming sooner than you might think.
Whatever the new approach is, managing records in Microsoft 365 will require new skills on the part of information and records managers.
Records about the world 
Reblogged this on M365 IM and commented:
Good overview of different approaches for managing records with Microsoft 365.