Microsoft 365 – ‘Records Management’ solution

Microsoft announced the General Availability of its so-called ‘records management’ solution for Microsoft 365 on 30 April 2020. The announcement included a screenshot of the ‘Overview’ tab of the ‘records management’ section of the Microsoft 365 Compliance portal which contains a range of other options including the very similar looking ‘information governance’ section.

Screenshot of the ‘overview’ tab of the records management section

The announcement noted that organisations would be able to use the records management solution to:

• Classify, retain, review, dispose, and manage content without compromising productivity or data security.
• Leverage machine learning capabilities to identify and classify regulatory, legal, and business critical records at scale.
• Demonstrate compliance with regulations through defensible audit trails and proof of destruction.

It also noted that the solution would be limited to eligible Microsoft 365 E5 customers.

This post:

• Examines what the records management solution in Microsoft 365 can do and its limitations.
• Highlights the need for records managers to be more actively involved (a) at the very least in the governance of and provision of advice relating to the management of records in Microsoft 365, and (b) preferably in actually having a specific role to play in managing records in that environment.

Before going into more detail, it is important to understand the differences between an E3 and an E5 licence, and how this relates to managing records in Microsoft 365.

Licensing

Microsoft have published extensive information in its service description for the Security and Compliance elements of Microsoft 365. It is a good idea to discuss these options with a Microsoft licensing partner.

At a general level, the difference between the ‘information governance’ (E3/E5) and ‘records management’ (E5 only) options are as follows:

• Info Governance – ‘Broad policies to catch everything and keep it simple’. According to the service description (link above) this provides for ‘a single organization-wide or location-wide retention policy and/or manual retention labeling’.
• Records Management (E5) – ‘Deeper dives into classifying content, enforcing particular process and workflows’. According to the service description, this means ‘automatically applying retention labels or policies, starting the retention period of a retention label based on a custom event, triggering a manual disposition review at the end of the label’s retention period, importing third-party data through native data connectors, discovering labeled content and monitoring labeling activity.’ and ‘automatically applying retention labels based on trainable classifiers’.

Microsoft have not stated that the ‘information governance’ section is being deprecated but it is clear that ‘records management’ is an advanced version of ‘information governance’.

What it’s not

As a starting point, the new ‘records management’ solution is not a standalone recordkeeping system or application within Microsoft 365, along the lines of an electronic document and records management system (EDRMS).

It is not a place where records are stored. It is one of several options (in addition to the lesser ‘information governance’ section) within the Compliance portal as shown in the screenshot below of the left hand navigation for that portal:

The solution does not make reference to, and is not based on, recordkeeping standards such as ISO 15489 or ISO 16175 Part 2.

It appears to assume that records managers will have a role to play, at the very least in providing advice about the management of records, being part of the governance for Microsoft 365, or having a specific role.

What is ‘records management’ in Microsoft 365

In summary, the ‘records management’ solution (which is still separate from the ‘information governance’ set of options – see below) is a set of advanced options in Microsoft 365 that allow organisations to:

• group (‘classify’) records for retention purposes based on several options including search, auto-classification and pre-defined events;
• allow a review of certain records due for disposal;
• destroy records; and
• retain a record of what was destroyed.

As described, the solution appears to be designed to manage records with minimal manual intervention. To quote from this Microsoft article ‘Automate event-driven retention‘:

‘The explosion of content in organizations and how it can become ROT (redundant, obsolete, trivial) is serious business. To continue to meet legal, business, and regulatory compliance challenges, organizations must be able to keep and protect important information and quickly find what’s relevant. Retaining only important, pertinent information is key to an organization’s success.’

These options are described below.

Responsibilities for records management in Microsoft 365

As noted above, access to the ‘records management’ section of the Compliance portal requires having one of several roles in Microsoft 365:

• Global Admin – usually restricted to a very small (~3) group of individuals in IT, as it gives access to all parts of (and content in) the Microsoft 365 tenant.
• Compliance admin. Gives access to all parts of the Compliance portal including the audit logs and content search for all of the tenant, the Information Governance (E3) and Records Management (E5) sections, and a lot more.
• Customised admin. Gives access to some parts of the Compliance portal.

There is no dedicated ‘Records Management’ role that gives access just to the ‘Records Management’ section of the Compliance portal.

An indication of Microsoft’s thinking about who should have access to what part of Microsoft 365 is contained in this article on automating event-driven retention that defines specific roles in relation to records management:

• (Global/Compliance) Admin – Creates Retention Event types, Retention labels and Record repositories in SharePoint [my emphasis as it is not clear what this is intended to mean in relation to the Admin roles].
• Records Manager – Provides Retention Policies and Retention Schedules guidance and compliance details [presumably to the Admin].
• System Admin (business) – Sets up and manages external systems to work with Microsoft 365.
• Information Worker – Manages the lifecycle of their business process (HR, Finance, IT, and so on).

What does the records management solution actually do?

According to Microsoft, the records management solution allows organisations to:

• Classify, retain, review, dispose, and manage content without compromising productivity or data security.
• Leverage machine learning capabilities to identify and classify regulatory, legal, and business critical records at scale.
• Demonstrate compliance with regulations through defensible audit trails and proof of destruction.

Many of these options are (still) also available in the ‘information governance’ section available with an E3 licence. Where they are only available in the ‘records management’ section, ‘(E5 licence only)’ is indicated.

Let’s look at each of these options, which are also described in this article: ‘Records Management in Microsoft 365‘ and other links.

Classify (via a retention label and policy)

The article ‘Overview of Retention Labels‘ provides an insight into how Microsoft sees the classification of records. It states that ‘Auto-apply’ retention labels are powerful because:

• You don’t need to train your users on all of your classifications.
• You don’t need to rely on users to classify all content correctly.
• Users no longer need to know about data governance policies – they can focus on their work.

In Microsoft 365, ‘classify’ has the meaning of identifying and grouping related or ‘like’ records primarily for the purposes of managing retention. It does not mean applying recordkeeping classification labels to records across Microsoft 365; the only thing that is applied is a retention label that can map to the selected classification.

The Microsoft article ‘Records management in Microsoft 365‘ describes several ways that information can be ‘classified’ when linked with retention labels.

When a new retention label is created a decision must be made about:

• Retention period (e.g., 7 years)
• Retention trigger (date created, date modified, date label applied, an event)
• Disposal action (do nothing, review before disposal, just destroy)

After the label is created, it may then be auto-applied to content across Microsoft 365 or to specific locations (such as SharePoint only). The three auto-apply options are:

• Content that contains sensitive information.
• Content that contains specific words or phrases, or properties.
• Content that matches a built-in or trainable classifier (E5 licence only).

The following describes the three options:

• Sensitive information. This option uses the same set of policies used for Data Loss Prevention (DLP) including custom policies. For example, the policy ‘financial information’ that identifies credit card numbers. In practice in the Australian context, these built-in policies are not very accurate. (See this article for more information on DLP Policies).
• Specific words or phrases or properties. These match queries built via the Keyword Query Language (KQL) or other keyword searches – the same that would be used for Content Searches across Microsoft 365.
• This option also includes the metadata properties (including metadata from the Managed Metadata Service) contained in SharePoint Content Types added to a document library. These metadata elements are crawled properties that can be used to find and automatically apply a retention label. This option allows organisations to include their business classification scheme terms in the Managed Metadata Service (MMS) and use terms from this MMS as metadata options in custom Content Types that are applied to document libraries. A retention policy can then be automatically applied to content that has been classified in this way. In practice, this is a complex model to develop and manage overtime and should only be used for specific types of records.(See this Microsoft article for a very detailed description of how this is achieved: ‘Manage the lifecycle of SharePoint documents with retention labels‘). 
• Trainable classifiers. This refers to both (a) the built-in options of: Offensive Language, Resumes, SourceCode, Targeted Harassment, Profanity, and Threat and (b) additional trainable classifiers (E5 licence only). Both built-in classifiers and trainable classifiers are available as a condition when retention labels are auto-applied to content. (See further discussion below and these Microsoft articles for an Introduction to Trainable Classifiers and directions to ‘Create a trainable classifier‘.)

Note: The ‘auto-apply’ option may take up to 7 days to take effect.

Review

When a new retention label is created, one of three disposal actions must be selected for what will happen at the end of the retention period:

• Do nothing.
• Review disposition (currently E3 and E5). This option requires an email address to notify the person who will do the review. This person must have access to the Dispositions area of the Compliance portal to do this.
• Just delete the content without a review.

If the retention label includes the second option to review the disposition, the nominated reviewer (with the required access role) will receive a notification. They can also navigate at any time to the Dispositions tab (of the Information Governance and Records Management sections of the Compliance portal).

The dispositions section displays individual records that are due for disposal based on the retention label settings, with an option to view by ‘Documents’ or ‘Emails’.

Several points to note:

• Records are not displayed in their original containers (e.g., mailboxes or document libraries etc) but they can be grouped this way via the search option.
• The metadata that may have been assigned to records in a SharePoint library is not displayed and cannot be exported from the Dispositions area. The only way to export the metadata is to access the document library on the SharePoint site – and this would assume the review has the permissions to see everything on the site (Site Collection Admin).
• Only those records covered by the label in the document library will be displayed. This is another reason to review the original library.

If anything, the Dispositions area is useful to provide a heads-up for records managers to review the actual library content in SharePoint. The records manager may then export the metadata of the records to be destroyed.

Note – Any records that are subject to a label with the ‘just delete’ option selected will not appear in the Dispositions section. They will simply be deleted (via a 90 day period in the Recycle Bin).

Leverage machine learning capabilities to identify and classify records (E5 licence only)

This option has been described above in the section relating to the auto-application of a retention label based on the terms defined in a trainable classifier. The diagram below is from the Microsoft article ‘Create a trainable classifier‘:

The question arises whether it would be possible to develop a set of classification terms that (a) map to the records class descriptions contained in a records retention schedule and (b) can accurately identify content that matches those descriptions across the Microsoft 365 ecosystem. This would certainly be an ideal goal.

Demonstrate compliance with regulations through defensible audit trails and proof of destruction

The records management section includes a tab named ‘Dispositions’ as shown in the screenshot below (from the article ‘Disposition of content‘). This is currently the same for both E3 and E5 licences, but some functionality may be restricted to E5.

As noted in this article, ‘Items that are shown in the Disposed Items tab for record labels are kept for up to 7 years after the item was disposed, with a limit of one million items per record for that period.’ (It is not clear yet if this is for E5 only).

The data about records destroyed for each label can be exported.

The image below is from the same article and shows the limited amount of content provided for each item that is destroyed. It does not include any metadata from the original location and does not destroy the original document library. It is up to the organisation as to whether this simple form of disposition review will be suitable or if more details are required.

For most records of corporate value, the disposition review process is too limited in terms of the record it retains of what was destroyed. However, it does provide a heads-up for records managers (provided they can access it).

Most records that are of low value should never require a disposition review, however many organisations may be loathe to automatically delete content – even low-level content – that may be required beyond the minimum retention period.

What about Information Governance?

The alternative to the options provided in the records management section are the labels and retention policies in the ‘information governance’ section of the Compliance portal. These options have been described in a separate post but in summary allow organisations to use one or both of the following options:

• Label-based retention policies. These policies are based on labels that can include a disposition review. Once published, they must be manually applied to Microsoft 365 workloads but are most likely to be used on SharePoint document libraries.
• Retention policies. These policies are simple ‘background’ retention policies that can be applied to the main Microsoft 365 workloads (Exchange, SharePoint, MS Teams, OneDrive) They delete content automatically but have no disposition review and no record is kept beyond the period of the audit logs (90 days).

These policies can be combined on individual SharePoint sites for maximum effect.

It is assumed, but cannot be confirmed, that these options will continue to exist for some time.

Conclusion

The records management solution is not a solution to manage records across Microsoft 365. It has a specific purpose that would be more accurately described as ‘advanced information governance’.

The solution offers organisations with (more expensive) E5 licences a way to automatically identify (classify) and manage certain types of records through to the end of their retention period. It is designed to address the high volumes of both low-level digital content (‘ROT’ in particular) and specific high-value records in organisations.

For organisations that have E3 licences, the alternative options offered from the Information Governance section of the Compliance may be sufficient for as long as these options remain available.