Classifying records in Microsoft 365

The classification of records is fundamental recordkeeping activity. It is defined in the international standard ISO 15489-1:2016 (Information and Documentation – Records Management) as the ‘systematic identification and/or arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules‘. (Terms and Definitions, 3.4)

The purpose of classification is defined by State Records NSW as follows: ‘In records management, records are classified according to the business functions and activities which generate the records. This functional approach to classification means that classification can be used for a range of records management purposes, including appraisal and disposal, determining handling, storage and security requirements, and setting user permissions, as well as providing a basis for titling and indexing‘. (Records Classification, accessed 13 January 2021.)

The ever-increasing volume of digital records, the many different ways to create them, and the multitude of record types that are created and storage locations, have made it more difficult to accurately and consistently manually classify records, including through the creation of pre-defined ‘containers’ or aggregations based on classification terms. Despite this, the requirement to link the classification of records with their retention and and disposal remains.

For over three decades, Microsoft’s applications and technology platforms have been used to create, capture, store and manage records. Some of these records (in the earlier period) were printed and placed on paper files, or stored (from around 2000) in dedicated electronic document and records management (EDRM) systems.

But the volume and type of digital content, including with new types of records (e.g., chat messages) and storage locations, continues to grow. In response, Microsoft invested heavily in addressing the need to classify records ‘at scale’.

This post looks at various ways to classify records, for retention and disposition purposes, in Microsoft 365.

The old-school, manual method – metadata

Most of the records in Microsoft 365 will be created, captured or stored in one of the four primary workloads: Exchange mailboxes, SharePoint sites/libraries, MS Teams chats (a ‘compliance copy’ of which is stored in Exchange mailboxes), and OneDrive for Business libraries. Some records may also exist in Yammer or other web page content (e.g., intranet).

Most SharePoint sites as well as Teams (that have a SharePoint site) will be created according to some form of business need to create, capture, store and share records; that is, the site or team purpose may be based on a business function or activity. This way of grouping records may in some ways be used as a way to classify records – by SharePoint site (e.g., function) or document library (e.g., activity).

Records may be stored in multiple document libraries, or within a folder structure of a single library.

A number of methods (some of which rely on others) can be used to add classification (and other) metadata to records stored in SharePoint document libraries:

1 – Creating the classification taxonomy in the Managed Metadata Service (MMS)/Term Store via the SharePoint Admin portal – Content Services – Term store, and then applying these terms in content types that are then deployed in SharePoint sites.

An example of Business Classification Terms in the MMS

2 – Creating global content types from the SharePoint Admin portal, in the Content Services – Content type gallery area (see ‘Finance Document’ example below) and then deploying these in specific SharePoint sites where site columns that contain classification terms will be added.

3 – Creating site columns that contain classification terms, including from the MMS, and adding these to global or site content types or document libraries where they can be applied to records.

In this example, the site column ‘BCS Function’ maps to the MMS BCS terms

4 – Creating site content types and adding site columns (including MMS-based columns), then adding these content types to document libraries.

In this example, the MMS-based column now appears in the library columns

But, most of the above is somewhat complicated and cumbersome and would normally only be used for and manually applied to specific types of records.

The simplest way to apply BCS/File Plan terms at the document (or document set) level is to (a) store records to the same BCS function or activity in the same library, (b) create site or library columns with default values and add these to the library. This way means that the default terms are applied automatically as soon as a new record is uploaded, including when shared/inherited from the site columns added to a document set that ‘contains’ a document content type.

Example metadata columns shared from the document set content type

However, keep in mind that SharePoint is just one of the workloads where records are stored.

Records in the form of emails, chats and ‘personal’ content (as well as Yammer messages and web pages) are created in and stored across the other workloads. Some attempt may be made to copy these other records (especially emails) in SharePoint sites but it starts to get complicated or impossible to do so with things like Teams chat messages.

In most cases (and according to Microsoft’s own recommendations), it is better to leave the records where they were created or captured (‘in place’), and apply centralised compliance controls (classification, retention labels and policies) to this content.

Leaving the records in place in this way does not exclude the ability to create SharePoint sites and document libraries in those sites that map to classification terms, and/or use the site column approach described above but these are more likely to be exceptions.

In fact, some form of logical structure is almost certain anyway as most end-users will probably want to access and manage information in their own specific work context (the Team/SharePoint site).

Trainable classifiers

Since not all records are stored in SharePoint and the ever-increasing volume of digital content stored across the Microsoft 365 platform, Microsoft needed to find a way to classify records ‘at scale’.

The solution was to use machine learning (ML) via trainable classifiers accessed in the ‘Data Classification’ section of the Microsoft 365 Compliance portal. This capability is only available to E5 licences.

The trainable classifiers solution was released to General Availability on 12 January 2021 (‘Announcing GA of machine learning trainable classifiers for your compliance needs‘, accessed 13 January 2021).

See the Microsoft web page ‘Learn about trainable classifiers‘ to learn more about this option. To quote from that page:

‘This classification method is particularly well suited to content that isn’t easily identified by either the manual or automated pattern matching methods. This method of classification is more about training a classifier to identify an item based on what the item is, not by elements that are in the item (pattern matching).’

Organisations (including E3 licence holders) may make use of five pre-defined trainable classifiers (Resumes, Source Code, Targeted Harassment, Profanity or Threat. A sixth classifier ‘Offensive language’, is to be deprecated). Custom classifiers require an E5 licence.

Custom classifiers require ‘significantly more work’ than the pre-existing classifiers and the process is quite involved (see the process flow diagram in the ‘Learn about’ page link above) but in summary it involves the following steps:

• Creating the custom classifier.
• Creating a set of manually selected example records (50 to 500) in a dedicated SharePoint Online site as the ‘seed’. This would include a range of emails in the seed examples.
• Testing the classifier with the seeded documents.
• Re-training with additional content – both positive and negative matches.

Once the classifier is published, it can be used to identify and classify related content across SharePoint Online, Exchange, and OneDrive (but not Teams).

The page ‘Default crawled file name extensions and parsed file types‘ provides details of all the record types that can be classified in this way. Note it is not clear if trainable classifiers can crawl the compliance copy of Teams chat messages stored in hidden folders in Exchange mailboxes.

Label-based retention policies can then be automatically applied to content that has been identified through the trainable classifier.

However, note that the classifier does not ‘group’, aggregate or ‘present’ (list) the records for review (except broadly via the Content Explorer); however, the label applied to the records can be searched via the ‘Content Search’ option in the Compliance portal. This is a much better option than not having any idea how many records of a particular classification may exist in Exchange mailboxes, OneDrive accounts, or general SharePoint sites. It requires some degree of ‘letting go’ of the ability to view and browse content classified this way, and trusting the system.

The main limit with trainable classifiers is that it requires an E5 or E5 compliance licence.

The other limitation is the management of the disposition of records that have been identified with trainable classifiers and had a label-based retention policy applied. There are significant shortcomings with the current ‘Disposition Review’ process, specifically the lack of adequate metadata to review records due for disposal or the details of what has been destroyed.

SharePoint Syntex

Another (but limited) option might be to use SharePoint Syntex (see ‘Introduction to Microsoft SharePoint Syntex‘ for an overview), although its range is limited to SharePoint and – it seems – only records that have a relatively consistent structure and format.

SharePoint Syntex evolved out of Project Cortex’s ability to extract and capture metadata from records. It can also be used through its ‘Document Understanding Model‘ (DUM) to provide a way to classify records stored in SharePoint Online (only). It makes use of a ‘seeding’ model that is similar to trainable classifiers (and may be based on the same underlying AI engine).

Broadly speaking, the DUM works on the basis of loading a small ‘seed’ set of (relatively consistently formated) example files into a dedicated Content Center (or Centers). This is very similar to the process of using trainable classifiers, except that the latter does not require a ‘content center’ SharePoint site to be created.

• The example files are ‘trained’ by being ‘classified’ through the document understanding process based on a set of ‘explanation types‘ that are used to help find the relevant content. The three explanation types are: (a) phrase list (a list of words, phrases, numbers, or other characters used in the document or information that you are extracting); (b) pattern list (patterns of numbers, letters, or other characters); and (c) proximity (describes how close other explanations are to each other).
• The document understanding model (DUM) produced through the explanation types is associated (and deployed) with a new or existing content type. 
• Once applied to a SharePoint site library, the DUM/content type provides the basis for identifying and tagging (with metadata) other similar records in the location (e.g., the library) where the DUM has been deployed. 
• If the documents have consistent content such as invoices, certain data from those documents can be extracted as metadata. 

Retention labels may be applied to records classified using SharePoint Syntex, as described on this page ‘Apply a retention label to a document understanding model‘.

Summing up – which one should be used?

The answer to this question will depend on your compliance requirements.

Smaller organisations may be able to set up SharePoint sites and document libraries with site columns/metadata that maps to their business classification scheme or file plan, and copy emails to those libraries. There may be little need to use AI-based classification methods.

In large and more complex organisations (with E5 licences), especially those with a lot of content stored across Exchange mailboxes and SharePoint sites (including Teams-based sites) there will most certainly be a need for some form of AI-based classification in addition to classification-mapped SharePoint sites (and Teams).

Organisations with E3 licences might use the manual methods described above for specific types of records, and consider acquiring additional E5 Compliance licences to make use of trainable classifiers or SharePoint Syntex for other records.

Managing the retention of records in Microsoft 365 with an E3 licence

Microsoft released its ‘Records Management’ solution for Microsoft 365 during 2020. The solution is only accessible to organisations with an E5 licence (or an E5 Security and Compliance licence).

Some of the retention-related options previously available to E3 licences, such as disposition review, are now only available with an E5 licence. However, for cost and other reasons, many organisations have decided to stay with E3 and asked if it is still possible to manage the retention of records.

According its licencing guidance ‘Microsoft 365 licence guidance for security and compliance‘, E3/A3 (and E5/A5) licences provide the following ‘information governance’ capability:

• A basic organization-wide or location-wide Exchange mailbox retention policy and/or to manually apply a non-record retention labeling to mailbox data.
• A basic SharePoint or OneDrive retention policy and/or to manually apply a non-record retention label to files in SharePoint or OneDrive.
• A Teams retention policy.

This post describes how the retention of records could be managed with an E3 licence and recommends that organisations intending to deploy Microsoft’s retention policies (whether E3 or E5) develop a plan for their deployment based on a detailed understanding of the following:

• What records exist and where they are stored across the main Microsoft 365 workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.
• What retention is required, including for EXO mailboxes and personal/ODfB accounts for which retention was previously ‘managed’ through backups, and MS Teams chats.
• What type of retention policy will apply to what workloads.
• How will disposition be managed (including of original storage locations, not just the records), and what evidence of destruction is required?
• How will any limitations and shortcomings be addressed to minimise legal risk?

Where are the records?

A good starting point with retention planning is to establish where the records that will be subject to retention policies are stored.

Most records are created and stored in one of the primary four workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.

While SharePoint sites (including Teams-based sites and Teams channels) may be used to logically ‘group’ records (e.g., in a team site, a document library, or a channel), this is not the case for EXO mailboxes, ODfB accounts, MS Teams 1:1 or private channel chats (a compliance copy of these chats is stored in the personal EXO mailboxes of participants in the chat).

EXO mailboxes and ODfB accounts usually contain a range of records on different subjects, with different retention requirements. Personal chat messages could be about any subject.

For most of the past three decades, the requirement to keep specific emails or other records meant that end-users had to copy those records to another system, including an EDRMS or even SharePoint, or print and place them on a file. The ability to find old records depended more often than note on the ability to recover them from back-up tapes, a process ironically often referred to as ‘archiving’.

Action: Organisations should establish a list of where records are stored in Microsoft 365 and how retention (including via backups) is currently managed.

What retention is required?

The next step in the process is to establish how long these records need to be kept.

Most of the time this will be based on an organisational records retention policy or schedule. These policies or schedules describe groups of records and how long they must be retained – for example, financial records generally must be retained for seven years. Specific types of financial records may require shorter or longer retention.

This model generally works well when records have been stored in logical groups (e.g., a whole SPO site/Team, or document library), but are more difficult to apply for individual records stored with other records in personal EXO mailboxes, ODfB accounts, SPO sites, or MS Teams chats. If records with longer retention requirements in these locations are not stored elsewhere, or cannot be specifically identified, all the records may have to be kept for the longest retention period.

The start of retention is usually based on a trigger action. Typically these are based on (a) date created, (b) date modified, or (c) date of last action. However, they may also be based on less specific events, for example: (a) 7 years after a contract has expired, (b) 25 years after an employee leaves, (c) when a child turns 21. These less specific events need special attention.

Action: Organisations should ‘map’ retention requirements to the locations where the records are stored. This process will likely involve a discussion regarding the replacement (or supplementation) of email backups with retention policies, and may end up looking something like the following:

Workload	Retention
EXO mailboxes of senior managers	25 years
EXO mailboxes of all other employees	7 years
MS Teams 1:1 chats of senior managers	25 years
MS Teams 1:1 chats of all other employees	7 years
MS Teams private channel chats	As per retention policies
ODfB accounts of senior managers	10 years
ODfB accounts of all other employees	7 years
SPO sites that are not subject to more specific policies	7 years
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies
SPO sites with specific retention requirements, per site or library	As per retention policies

What are the retention options with an E3 licence?

The three main options available with an E3 licence – labels, label policies, and retention policies – are all set from the Compliance admin portal under the ‘Information Governance’ section.

The table below summarises the options:

Type of policy	Can be used for
‘Implicit’, ‘safety net’ retention policies. These policies: (a) work in the back end and cannot be changed by an end-user; (b) create a preservation hold library in SPO sites and ODfB accounts, and hold deleted emails in a hidden EXO mailbox folder; (c) provide an alternative to back-ups, although it should be kept in mind that all content that is retained in this way contributes to the overall storage quota; (d) do not retain record of what was destroyed.	EXO, SPO, MS Teams, ODfB
‘Explicit’ label-based retention policies. Labels must be published to the required workloads before they become visible. Records cannot be deleted once a label has been applied, but end-users can change or remove the label and then delete the record. Label-based policies do not retain a record of what was destroyed.	EXO, SPO, ODfB
‘Explicit’ label-based retention policies that are auto-applied based on three limited options – see below. Records cannot be deleted once a label has been applied. These policies do not retain a record of what was destroyed.	EXO, SPO, ODfB

The three auto-apply options are as follows:

• Apply label to content that contains sensitive information. Unlikely to be used as retention is never based on sensitivity.
• Apply label to content that contains specific words or phrases, or properties. Possibly useful. Recommend that organisations do a Content Search first to see what records may exist.
• Apply label to content that matches a trainable classifier. E3 licences provide six out of the box, limited classification options including ‘profanity’. These are unlikely to be of any value.

Mapping the E3 options

Given the options available, the following is a suggested mapping of records, retention and policy options:

Workload	Retention	Retention policy
EXO mailboxes of senior managers	25 years	Retention policy, plus possibly labels auto-applied to specific records
EXO mailboxes of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
MS Teams 1:1 chats of senior managers	25 years	Retention policy
MS Teams 1:1 chats of all other employees	7 years	Retention policy
MS Teams private channel chats	As per retention policies	Retention policy
ODfB accounts of senior managers	10 years	Retention policy, plus possibly labels auto-applied to specific records
ODfB accounts of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
SPO sites that are not subject to more specific policies	7 years	Retention policy
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies	Retention policy
SPO sites with specific retention requirements, per site or library	As per retention policies	Retention policy (safety net) plus label-based retention policy for specific records

Policy limits

Organisations planning to deploy retention policies should be aware of the limits on custom policies, as described on this Microsoft page, ‘Create and configure retention policies‘. There are no limits on policies that apply to an entire workload (e.g., all EXO mailboxes).

• 1,000 mailboxes
• 1,000 Microsoft 365 groups
• 1,000 users for Teams private chats
• 100 sites (OneDrive or SharePoint)

The page above notes also that ‘There is also a maximum number of policies that are supported for a tenant: 10,000. However, for Exchange Online, the maximum number is 1,800. The maximum number includes retention policies, retention label policies, and auto-apply retention policies.’

How will disposition be managed

Microsoft 365 retention policies retain records for a specified period and usually then delete the records automatically. No record is retained of what was deleted. Even with an E5 licence, only limited metadata is retained and only on those records subject to disposition review.

Organisations deploying retention policies with an E3 licence need to understand the potential risks associated with being unable to provide evidence of what was destroyed.

There are at least two ways to approach this point, but in all cases the approach and options must be legally defensible:

• Ensure that organisational policies clearly indicate what records will be destroyed without any record being kept. For example, with certain exceptions, most emails, Teams chats, SPO sites and the content of ODfB accounts will only be kept for 7 years and then destroyed.
• Establish a process to ensure that a record is kept of specific records that have been destroyed. For example, the metadata of records, due for disposal and stored in document libraries in specific SPO sites, will be captured (and stored separately) before the records are destroyed, and then the document library will be deleted. This is a labour-intensive process and may only be used on some sites.

Addressing E3 licence limitations and shortcomings

The limitations of E3 licence retention policies (and also E5 licences from an evidentiary point of view) should not put organisations off using the out of the box options or cause them to acquire third-party products.

The retention options available in Microsoft 365 now provide functionality that assists recordkeeping compliance, for example the ability to apply ‘back end, safety net’ retention policies to EXO mailboxes, MS Teams chats, ODfB accounts and whole SPO sites. Coupled with the ability to locate all records, including ‘deleted’ records via Content Search, this should be a boon to records managers and also to IT (saving the latter from having to recover records from backup tapes).

On the other hand, deploying the options can be complex and requires good planning and governance documentation. The lack of good governance can lead to the inadvertent loss of records as KPMG found in August 2020 (‘IT blunder permanently erases 145,000 users’ personal chats in KPMG’s MS Teams deployment‘).

Once retention policies have been deployed, the next most complex task may be the disposition approval and destruction process for those records for which a formal disposition review and approval process is required, including the requirement to establish a list of records to be destroyed.

As noted above, even the E5 licence options for disposition review and ‘proof of disposition’ are inadequate in terms of the metadata that is presented and retained. Until Microsoft provides the ability to record the metadata of every record that is due for disposal and/or destroyed, based on ANY type retention policy this process will continue to be a manual task for records managers.

Organisations may consider acquiring a limited number of E5 Security and Compliance licences to gain access to the E5 ‘Records Management’ capability to gain access to other options including File Plan, auto-classification options, and Disposition Review. However, aside from these options (and the ability to auto-apply labels more broadly) these options may not add much more capability to the existing options already available with an E3 licence.

Summarising the options

Based on the above details, organisations with E3 licences might do the following:

• Create several ‘safety net’ retention policies for EXO mailboxes, ODfB accounts, MS Teams chats and general SPO sites. EXO/ODfB retention policies may be (a) based on current back-up periods and (b) be split into ‘senior’ and all other employees. They may also create policies for Microsoft 365 Groups which will cover the Group’s EXO mailbox and SPO site. Some of these policies may map to existing retention classes in the records retention schedules, but others such as the EXO, ODfB and MS Teams policies may need to be added.
• Create label-based retention policies for content stored in specific SPO sites where more granular retention policies are required per library. However, keep in mind these can be changed or removed by end-users with add/edit access. Generally it is not recommended to apply label-based policies to EXO or ODfB unless end-users will apply these accurately and consistently.
• Auto-apply some labels to specific content stored in SPO, EXO and ODfB (having identified that these records exist first via a Content Search). These labels should have a retention period that is longer than the safety net policy or any other label-based policy.
• Establish a ‘manual’ disposition review process for records that are subject to label-based retention policies.

References:

Microsoft 365 licensing guidance for security and compliance

All the ways SharePoint sites can be created

SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.

While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.

Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.

So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.

Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.

The following is a list of the main ways a person can create a SharePoint site. Have I missed any?

1. Via a PowerShell script

As described in the Microsoft docs web page ‘Create SharePoint Online sites and add users with PowerShell‘. The script is based on a csv file (‘sitecollections.csv) and looks something like the following (see the link for more details):

Import-Csv C:\users\MyAlias\desktop\SiteCollections.csv | ForEach-Object {New-SPOSite -Owner $_.Owner -StorageQuota $_.StorageQuota -Url $_.Url -NoWait -ResourceQuota $_.ResourceQuota -Template $_.Template -TimeZoneID $_.TimeZoneID -Title $_.Name}

This option also allows the administrator to provision new SharePoint sites.

2. Via the SharePoint Admin portal (+ Create)

This option allows the creation of three main types of sites: modern team sites (Team site),
communication sites, and non-Microsoft 365 Group-linked sites (Other options).

3. By creating a Microsoft 365 Group

Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.

Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.

The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.

Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.

4. By an end-user creating a new Team in MS Teams

Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.

Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an
Exchange Online mailbox that is not visible in Outlook.

5. By creating a Private Channel in MS Teams

If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.

For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.

A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.

6. By the Teams Admin creating a new Team

The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.

As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.

7. From the end-user SharePoint portal (+ Create site)

If not disabled, end users can create a new SharePoint site by clicking on ‘+ Create site’ from the SharePoint portal – https://tenantname.sharepoint.com/_layouts/15/sharepoint.aspx

This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.

It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.

8. From OneDrive for Business as a ‘shared library’

This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.

9. When a new Plan is created in Planner

If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.

10. When a new Yammer community is created

End users with access to Yammer can click on ‘Create a Community’ from Yammer.

To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’

Why have Microsoft allowed this?

It’s a smarter way to manage access.

Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.

The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.

But, some of the decisions don’t seem to make sense.

• Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
• Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
• Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
• And why does a new Plan create a SharePoint site? For what purpose?

Perhaps there is a reason for it. It’s just not clear.

Using Microsoft 365 connectors to support records management

Microsoft 365 includes a range of connectors, in three categories, that can be used to support the management of records created by other applications. The three categories are:

• Search connectors, that find content created by and/or stored in a range of internal and external applications, including social media.
• Archive connectors, that import and archive content created by third-party applications.
• API connectors, that support business processes such as capturing email attachments.

This post how these connectors can assist with the management of records.

The recordkeeping dilemma

Finding, capturing and managing records across an ever increasing volume of digital content and content types has been one of the biggest challenges for recordkeeping since the early 2000s.

The primary method of managing digital records for most of the past 20 years has been to require digital records (mostly emails and other digital content created on file shares) to be saved to or stored in an electronic document and records management system (EDRMS). The EDRMS was established as ‘the’ recordkeeping system for the organisation.

EDRM systems were also used to manage paper records which, over the past 20 years, have mostly contained the printed version of born-digital records that remain stored in the systems where they were created or captured.

There were two fundamental flaws in the EDRMS model. The first was an expectation that end-users would be willing to save digital records to the EDRMS. The second was that the original digital record remained in place where it was created or captured, usually ignored but often the source of rich pickings for eDiscovery.

The introduction of web-based email and document storage systems, smart phones, social media and personal messaging applications from around 2005 (in addition to already existing text messaging/SMS messages) further challenged the concept of a centralised recordkeeping system; in many cases, the only option to save these records was to print and scan, screenshot and save the image, or save to PDF, none of which were particularly effective in capturing the full set of records.

The hasty introduction from early 2020 of ‘work from home’ applications such as Zoom and Microsoft Teams has been a further blow to these methods.

In place records management

To the chagrin of records managers around the world, Microsoft never made it easy to save an email from Outlook to another system. Emails stubbornly remained stored in Exchange mailboxes with no sign of integration with file shares.

And for good reason – they have a different purpose and architecture to support that purpose. It would be similar to asking when it would be possible to create and send an email in Word.

The introduction of Office 365 (later Microsoft 365) from the mid 2010s changed the paradigm from a centralised model – where records were all copied to a central location and the originals left where they were created or captured, to a de-centralised or ‘in place’ model – where records are mostly left where they were created or captured.

The decentralised model does not exclude the ability to store copies of some records (e.g., emails) in other applications (e.g., SharePoint document libraries), but these are exceptions to the general rule.

It also does not exclude the ability to import or migrate content from third-party applications where necessary for recordkeeping purposes.

Microsoft 365 connectors

Microsoft 365 includes a wide range of options to connect with both internal and external systems. Many of these connectors simplify business processes and support integration models.

Connectors may also be used to support recordkeeping requirements, in three broad categories.

The three connectors

Archive connectors

Archive connectors allow organisations to import and archive data from third-party systems such as social media, instant messaging and document collaboration* platforms. Most of this data will be stored in Exchange mailboxes, where it can be subject to retention policies, eDiscovery and legal holds.

(*This option is still limited via connectors, but also see below under Search).

The social media and instant messaging data that can be archived in this way currently includes Facebook (business pages), LinkedIn company page data, Twitter, Webex Teams, Webpages, WhatsApp, Workplace from Facebook, Zoom Meetings. For the full listing, and a detailed description of what is required to connect each service, see this Microsoft description ‘Archive third-party data‘.

An important thing to keep in mind is that the data will be archived to an Exchange mailbox; this will require an account to be created for the purpose. Any data archived ot the mailbox will contribute to the overall storage quotas.

Search connectors

Search connectors (also known as Microsoft Graph connectors) index third-party data that then appears in Microsoft search results, including via Bing (the ‘Work’ tab), from http://www.office.com, and via SharePoint Online.

Most ECM/EDRM systems are listed, which means that organisations that continue to use those systems can allow end-users to find content from a single search point, only surfacing content that users are permitted to see.

The following is an example of what a Bing search looks like in the ‘Work’ tab (when enabled).

Example Bing search showing the Work tab

Note: as at 17 November 2020, Microsoft’s page ‘Overview of Microsoft Graph connectors‘ (which includes a very helpful architecture diagram) states that these are ‘currently in preview status available for tenants in Targeted release.’

There are two main types of search connector:

• Microsoft built: Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL, Enterprise websites, MediaWiki, Microsoft SQL, and ServiceNow.
• Partner built. Includes the following on-premise and online document management/ECM/EDRM connectors – Alfresco, Alfresco Content Services, Box, Confluence, Documentum, Facebook Workplace, File Share (on prem), File System (on prem), Google Drive, IBM Connections, Lotus Notes, iManage, MicroFocus Content Manager (HPE Records Manager, HP TRIM), Objective, OneDrive, Open Text, Oracle, SharePoint (on prem), Slack, Twitter, Xerox DocuShare, Yammer

See the ‘Microsft Graph connectors gallery‘ web page for the full set of current connectors.

A consideration when deploying search connectors is the quality of the data that will be surfaced via searches. Duplicate content is likely to be a problem in identifying the single – or most recent – source of truth of any particular digital record, especially when the organisation has required records to be copied from one system (mailbox/file share) to another (EDRMS).

API Connectors

API connectors provide a way for Microsoft 365 to access and use content, including in third-party applications. To quote from the Microsoft ‘Connectors‘ web page:

‘A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.’

To see the complete list and for more information about each connector, see the Microsoft web page ‘Connector reference overview‘.

Each connector provides two things:

• Actions. These are changes initiated by an end-user.
• Triggers. There are two types of triggers: Polling and Push. Triggers may notify the app when a specific event occurs, resulting in an action. See the above web page for more details.

API connectors can support records management requirements in different ways (such as triggering an action when a specific event occurs) but they should not be confused with archiving or search connectors.

Summing up

The connectors available in Microsoft 365 support the model of keeping records in place where they were first created or captured. They enable the ability to archive data from third-party cloud applications, search for data in those (and on-premise) applications, and triggers actions based on events.

The use of connectors should be part of an overall strategic plan for managing records across the organisation. This may include a business decision to continue using an ECM/EDRMS in addition to the content created and captured in Microsoft 365. Ideally, however, the content in the ECM/EDRMS should not be a copy of what already exists in Microsoft 365.

The enduring problem of emails as records

Ever since emails first appeared as a way to communicate more than 30 years ago they have been a problem for records management, for two main reasons.

• Emails (and attachments) are created and captured in a separate (email) system, and are stored in mailboxes that are inaccessible to records managers (a bit like ‘personal’ drives).
• The only way to manage them in the context of other records was/is to print and file or copy them to a separate recordkeeping system, leaving the originals in place.

Thirty-plus years of email has left a trail of mostly inaccessible digital debris. An unknown volume of records remains locked away in ‘personal’ and archived mailboxes. Often, the only way to find these records is via legal eDiscovery, but even that can be limited in terms of how back you can go.

Options for the preservation of legacy emails

The Council on Information and Library Resources (CLIR) published a detailed report in August 2018 titled ‘The Future of Email Archives: A Report from the Task Force on Technical Approaches to Email Archives‘.

The report noted (from page 58) three common approaches to the preservation of legacy emails:

• Bit-Level Preservation
• Migration (to MBOX, EML or even XML)
• Emulation

In a follow up article, the Australian IDM magazine published an article in March 2020 by one of the CLIR report authors (Chris Prom). The article, titled ‘The Future of Past Email is PDF‘, suggested that PDF may be (or become) a more suitable long-term solution for preservation of legacy emails.

Preservation is one thing, what about access

There is little point in preserving important records if they cannot be accessed. The two must go together. In fact, preservation without the ability access a record is not a long different from destruction through negligence.

Assuming emails can be migrated to a long-term and accessible format, what then?

No-one (except possible well-funded archival institutions perhaps) is seriously likely to attempt to move or copy individual legacy emails to pre-defined and pre-existing containers or aggregations of other records. This would be like printing individual emails and storing them in the same paper file or box that other records on the same subject are stored.

Access to legacy emails in an digitally accessible, metadata-rich format like PDF provides a range of potential opportunities to ‘harvest’ and make use of the content, including through machine learning and artificial intelligence.

These options have been available for close to twenty years in the eDiscovery world, but to support specific legal requirements.

Search, discovery and retention/disposal tools available in the Microsoft 365 Compliance portal, along with the underlying Graph and AI tools (including SharePoint Syntex) provide the potential to manage legacy content, including emails.

The starting point is migrating all those old legacy emails to an accessible format.

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search.

But what happens if there is no Exchange Online mailbox? It seems the chats become inaccessible which could be an issue from a recordkeeping and compliance point of view.

This post explains what happens, and why it may not be a good idea (from a compliance and recordkeeping point of view) not to disable the Exchange Online mailbox option as part of licence provisioning.

Licences and Exchange Online mailboxes

When an end-user is allocated a licence for Microsoft 365, a decision (sometimes incorporated into a script) is made about which of the purchased licences – and apps in those licences – will be assigned to that person.

E1, E3 and E5 licences include ‘Exchange Online’ as an option under ‘Apps’. This option is checked by default (along with many of the other options), but it can be disabled (as shown below).

If the checkbox option is disabled as part of the licence assigning process (not after), the end-user won’t have an Exchange mailbox and so won’t see the Outlook option when they log on to office.com portal. (Note – If they have an on-premise mailbox, that will continue to exist, nothing changes).

Having an Exchange Online mailbox is important if end-users are using MS Teams, because the ‘compliance’ copy of 1:1 chat messages in MS Teams are stored in a hidden folder (/Conversation History/Team Chat) in the Exchange Online mailbox of every participant in the chat. If the mailbox doesn’t exist, those copies aren’t made and so aren’t accessible and may be deleted.

If end-users chat with other end-users who don’t have an Exchange mailbox as shown in the example below, the same thing happen – no compliance copy is kept. The chat remains inaccessible (unless the Global Admins take over the account).

The exchange above, between Roger Bond and Charles, includes some specific key words. As we will see below, these chats cannot be found via a Content Search.

(On a related note, if the ability to create private channels is enabled and they create a private channel and chat there, the chats are also not saved because a compliance copy of private channel chats are stored in the mailboxes of the individual participants.)

Searching for chats when no mailbox exists

As we can see above, the word ‘mosquito’ was contained in the chat messages between Roger and Charles.

Content Searches are carried out via the Compliance portal and are more or less the same as eDiscovery searches in that they are created as cases.

From the Content Search option, a new search is created by clicking on ‘+New Search’, as shown below. The word ‘mosquito’ has been added as a keyword.

We then need to determine where the search will look. In this case the search will look through all the options shown below, including all mailboxes and Teams messages.

When the search was run, the results area shows the words ‘No results found’.

Clicking on ‘Status details’ in the search results, the following information is displayed – ‘0 items’ found. The ‘5 unindexed items’ is unrelated to this search and simply indicates that there are 5 unindexed items.

Double-checking the results

To confirm the results were accurate, another search was conducted where the end-user originally did not have a mailbox, and then was assigned one.

If the end-user didn’t have a mailbox but the other recipient/s of the message did, the Content Search found one copy of the chat message in the mailbox of the other participants. Only one item was found.

When the Exchange Online option was enabled for the end-user who previously did not have a mailbox (so they were now assigned a mailbox), a copy of the chat was found in the mailbox of both participants, as shown in the details below (‘2 items’).

Summary and implications

In summary:

• If end users chat in the 1:1 area of MS Teams and don’t have an Exchange Online mailbox, no compliance copy of the chat will be saved, and so it will not be found via Content Search.
• If any of the participants in the 1:1 chat have an Exchange Online mailbox, the chat will appear in the mailboxes of those participants.
• If all participants in the 1:1 chat have an Exchange Online mailbox, the chat will be found in the mailbox of all participants.

Further to the above:

• If end users can delete chats (via Teams policies) and don’t have a mailbox, no copy of the chat will exist.
• If end-users with a mailbox can delete Teams chats, but a retention policy has been applied to the chats, the chats will be retained as per the retention policy (in a hidden folder).

And finally, if you allow private channels, end-users can create private channels in the Organisation Team. The chats in these private channels are usually stored in the personal mailboxes of participants (not the Group mailbox) – so these chats will also be inaccessible and cannot be found via Content Search.

The implications for the above are that, if you need to ensure that personal chat messages can be accessed (from Content Search), then the participants in the chat must have an Exchange Online mailbox.

Further, if you allow deletion of chats but need to be able to recover them for compliance purposes, a retention policy should be applied to Teams 1:1 chat.

A modern way to manage the retention of digital records

In his April 2007 article titled ‘Useful Void: The Art of Forgetting in the Age of Ubiquitous Computing’ (Harvard University RWP07-022), Viktor Mayer-Schönberger noted that the default human behaviour for millenia was to forget. Only information that needed to be kept would be retained. He noted that the digital world had changed the default to remembering, and that the concept of forgetting needed to be re-introduced through the active deletion of digital content that does not need to be retained.

The harsh reality is that there is now so much digital information in the world, including digital content created and captured by individual organisations, that active deletion of content that does not need to be retained, seems an almost impossible task.

This post explores issues with the traditional model of records retention in the digital world, and why newer options such as the records retention capability of Microsoft 365 is a more effective way to manage the retention and disposal of records, and all other digital content.

The traditional retention model

The traditional model of managing the retention and disposal/disposition of records was based on the ability to apply a retention policy to a group or aggregation of information identified as records. For the most part, those paper records were the only copy that existed (with some allowance for working and carbon copies).

The model worked reasonably well for paper records, but started to falter when paper records became the printed versions of born-digital records, and where the original digital versions remained where they were created or captured – on network files shares, in email systems, and on backups. Although, technically, the official record was on a file, a digital version was likely to remain on network file shares or in an email mailbox after the paper version was destroyed at the end of the retention period, and remain overlooked.

How many of us have had to wade through the content of old network file shares to examine the content, determine its value, and perhaps see if it can even still be accessed? Or do the same with old backup tapes?

The volume of unmanaged digital content, not subject to any retention policy, only continued to increase. This situation continued to worsen when electronic document and records management (EDRM) systems were introduced from the late 1990s. End-users had to copy records to the EDRMS, thereby creating yet another digital copy, in addition to the born-digital originals stored in mailboxes or file shares.

Even if the record in the EDRMS were destroyed, there was a good chance the original ‘uncontrolled’ version of the digital record – along with an unknown volume of digital records that probably should have been consigned to the EDRMS but weren’t – remained in email mailboxs, on file shares, or on a backup tape somewhere.

eDiscovery was born.

The emergence of new forms of digital records, including instant messages, social media, and smart-phone based chat and other apps from the early 2000s only added to the volume of digital content, much of which was stored in third-party cloud-based and mobile-device accessible applications completely out of the reach and ability of the organisation trying to manage records.

• 

Modern retention management

A modern approach to retention management should be based on the following principles:

• Information, not just records, should only be kept for as long as it is required.
• It is no longer possible to accurately and/or consistently identify and capture all records in a single recordkeeping system.
• Duplication of digital content can be reduced by creating and capturing records in place, promoting ‘working out loud’, co-authoring and sharing (no more attachments and private copies).

None of the above points excludes the ability to manage certain types of records at a more granular level where this is required. But these records, or the location in which they are created or captured, should not be regarded as the only form of record.

Ideally, these records should be created (or captured) directly in the system where they are to be managed – not copied to it.

Change management is necessary

Some of these new ways of working are likely to come up against deeply ingrained behaviours, many of which go back several decades and have contributed to a reluctance to ‘forget’ and destroy old digital content, including:

• hiding/hoarding content in personal drives (and personal cloud-based systems or on USB drives);
• communicating by email, the content is which is inaccessible to anyone else;
• attaching documents to emails;
• printing and filing born-digital content; and
• sometimes, scanning/digitising the printed copies of born-digital records and saving them back to a digital system.

What about destruction?

Records managers in organisations moving away from the authorised destruction of digital content identified as records, to the destruction of all digital content (including identified records) need to consider what is required to achieve this outcome, and the implications for existing process and practices (including those described above).

• Some activities will remain unchanged. For example, the need to review certain types of records before they are destroyed (aka ‘disposition review’), to seek approval for that destruction, and to keep a record of what was destroyed.
• Some activities are new and can replace other existing actions and activities. For example, the application of retention policies to mailboxes can remove the requirement to backup those mailboxes.
• Some of activities or outcomes may be challenging. For example, the automatic destruction without review of digital content that is not the subject of more granular retention requirements, such as emails out of mailboxes, documents in personal working drives. This content will simply disappear after the retention period expires.

How Microsoft 365 can support modern retention management

Microsoft recognised some time ago that it was becoming increasingly difficult to manage the volumes and types of digital content that was being created every day by organisations.

Exisiting and newly released functionality in the Compliance portal of Microsoft 365 includes the ability to create and apply both label-based retention policies to specific types of records, including automatically based on machine learning capabilities, and broader ‘workload’ specific (e.g., mailboxes, SharePoint sites, OneDrive accounts, MS Teams chats) retention policies. This capability helps organisations to focus retention requirements on the records that need to be retained, while destroying digital content that is no longer relevant and can be forgotten.

Instead of directing end-users to identify records and copy them from one system to another (thereby creating two versions), Microsoft 365 allows end-users to create and capture records in place, providing a single source of truth that can be shared (rather than attached), be the subject of co-authoring, and protected from unauthorised changes (and even downloads).

Limitations with Microsoft 365

It is important to keep in mind that there are some limitations with the current (October 2020) retention capability in Microsoft 365.

• Retention and disposal is based on individual digital objects, not aggregations. There are limited ways to group individual records by the original aggregations in which they may have been stored (e.g., document libraries in SharePoint).
• Only the (minimal) details of records that were subject to a disposition review are recorded in the ‘disposed items’ listing, and this is only kept for a year (but can be exported). No record is kept of any other destroyed record, except in audit logs (for a limited period).
• The metadata details of records subject to a disposition review that were destroyed is minimal – the document type and name, date destroyed, destroyed by whom.
• When records are destroyed from SharePoint document libraries or lists, the library or list remains with no record kept of what was previously stored there. It is not possible to leave a ‘stub’ for a destroyed record.

Summing up

The primary outcome from introducing modern ways to manage retention will be that all digital content, not just content that has been identified as records or copied to a recordkeeping system, will be subject to some form of retention and disposal management.

In other words, a change from exception-based retention (where all the other digital content is overlooked), to a more holistic method of retention with both granular controls on certain types of records where this is required, and broader retention capability allowing us to forget the content that is no longer relevant – the ‘redundant, trivial and outdated’ (ROT) content often scattered across network file shares.

The Microsoft 365 experience – Teams, Exchange, Outlook, Edge: Where did SharePoint Go?

At the 2020 Microsoft Ignite conference, Jeff Teper presented a diagram titled ‘Microsoft 365’. The diagram showed only four icons: Teams, Outlook, Office and Edge.

The implication of this diagram was that, for most end-users, Teams is now (or will become) their primary portal into Microsoft 365. As stated by Jeff Teper, SharePoint is a foundation platform, the out of sight content engine. Edge’s ability to serve up search results from Microsoft 365 further reduces the need to go to SharePoint.

So, what are the implications for managing records?

SharePoint as a recordkeeping system

For a long time, records have been created, captured and stored in recordkeeping systems.

In the paper world, the recordkeeping system consisted of paper records stored in files and boxes and detailed in registers. With the introduction of computers in the 1980s, registers were transferred to databases, making it a bit easier to find records. In the late 1990s, recordkeeping databases were linked with (separate) file stores and became electronic document and records management (EDRM) systems that continued to manage paper records (the so-called ‘hybrid’ systems).

For almost a decade (since SharePoint 2010 was introduced), SharePoint has contended with files shares and EDRM systems as an alternative recordkeeping system, providing almost all the same core functionality.

The ability to create a record in a single location, then share and co-author it from that location, has completely removed the requirement to copy a record to a separate recordkeeping system.

And then came Teams

Someone at Microsoft had incredible foresight to see the potential for a new user interface that would replace products like Lync and Skype for chat and conferencing, and would also provide access to files stored in SharePoint.

SharePoint has been a core part of the Microsoft productivity offerings for a very long time and people have built careers around developing functionality on the SharePoint platform to appeal to end-users, the intranet being the most common case in point, with customised team sites close behind.

The arrival of Microsoft 365 Groups and then Teams in 2017 was perhaps not widely noticed. One could argue that end by the beginning of 2020, it was still largely unnoticed.

And then came a pandemic and working from home. Teams – which may have been largely ignored or overlooked until then – was already ready to take its place next to Outlook, Office and Edge as a primary end-user interface.

New Teams were created, sometimes with abandon (and were sometimes just as quickly abandoned).

Both 1:1 (or 1:many) chats and channel chats took off. Files were created and shared via OneDrive for Business (‘Files’ in the 1:1 chat area), or via the back-end SharePoint sites (‘Files’ in the channel chat area).

There was (and maybe still is) a belief that files were being saved to Teams but not SharePoint. ‘We are storing everything in Teams’ was not an uncommon expression, sometimes followed by ‘but we’re not using SharePoint or OneDrive’.

The year 2020 saw a huge increase in the volume of records stored in SharePoint sites linked with Teams, as well as a completely new set of records – chats (‘compliance’ copies of which are stored in Exchange mailboxes).

The diagram below provides an overview of the relationship between Teams, Microsoft 365 Groups, Exchange mailboxes, SharePoint and OneDrive for Business.

What about SharePoint?

As the diagram above shows, SharePoint has not disappeared. Many organisations will continue to use, and ask end-users to access, SharePoint sites directly to store and manage records.

But accessing SharePoint from SharePoint may become less necessary over time. At Ignite 2020, the ability to pin a ‘home site’ (such as an intranet) to Teams was demonstrated. Even the intranet may end up in Teams.

As Jeff Teper said, SharePoint is a foundation platform, one that does not get in the way of collaboration and productivity but powers it.

Implications for records managers

Records managers, who were likely already on a steep learning curve regarding SharePoint, need to continue to improve their knowledge of the SharePoint platform. On a positive note, SharePoint Online is a much easier application to learn and manage, compared with its earlier on-premise predecessors.

In organisations that have been using SharePoint for a while and/or have allowed the free-creation of Teams in MS Teams, there will some requirement for retrospective analysis, review, and cleaning up.

In all organisations, there will be a requirement to establish some form of governance and oversight of records (files and chats) that have been created, including for the purpose of retention and disposal/disposition.

Retrospective implications

Where MS Teams has been implemented with little thought given to naming conventions, SharePoint site provisioning, or access controls, records managers should been given access to and review the list of all SharePoint sites that have been created, including from MS Teams. This will provide an initial idea of the volume of content and activity on each site, and what action needs to be taken on things like inactive Teams.

Ideally, records managers should be added to the Site Collection Administrators (SCA) group of every SharePoint site, including MS Teams-based sites. This action will give records managers access to the content on every site and to help advise on the management of records in those sites (including Team-based sites).

• The best way to do this is to add records managers to a Security Group and then add that Group to the SCA group of every site. This access could be deferred for sites that contain very sensitive information, although typically records managers would have access to all records, including if they had an EDRMS. And, access is always recorded in audit logs or the local site ‘viewers’ (where enabled) and ‘last modified by’ information.

Access to the chat content of Teams (including 1:1 chats) will not normally be required; some understanding of the content could be inferred from the name of the Team or the SharePoint content. If necessary, Global Admins or a Compliance Admin can run a Content Search across Teams to find chat content, and/or export that content by an individual person or subject.

Records managers will also need to advise on the appropriate retention policy or policies that need to be created and then applied to:

• The chat content in 1:1 chats.
• The chat content in the various Teams.
• SharePoint sites linked with Teams.
• Exchange mailboxes.
• OneDrive for Business accounts. An additional consideration is how long the content of inactive ODfB acccounts should be retained via the ‘Storage’ policy (default is 30 days then permanent deletion).
• SharePoint sites not linked with MS Teams. This includes whole sites as well as library-based retention policies.
• Office 365 Groups (mailbox/SharePoint site). If linked with a Team, a second retention policy is required for the Team chat content retention (second dot point above). For example, one policy ‘GroupABC’ and a second policy ‘GroupABCTeamChat’.

As many of the above retention policies replace the need for backups, records managers need to discuss the options with their IT colleagues.

Forward looking implications

Ideally, there should be some form of governance around the creation of new Teams in MS Teams. These governance arrangements might include:

• The necessary access for records managers. For example, Site Collection Administrator on every site, and/or a customised Compliance Admin role to create and access retention policies.
• Controls around the creation of new Teams, including naming conventions. If not controlled, what processes will ensure that records are properly managed.
• Retention implications. For example, can the new site and/or the channel chat content be covered by another retention policy – e.g., ‘All Teams with assessed low-level working content should be kept for 5 years’.
• Simple best practice guidance for all new users, including on how to share and co-author.
• Retention policies for all Microsoft 365 content, not just SharePoint.
• Reviews of the content of OneDrive for Business accounts of departed end-users, especially for people in senior or decision making positions. It is relatively common practice for end-users to delete (and download) this content before they leave their jobs.
• Monitoring and oversight of content, including access to reporting dashboards.

So, is Microsoft 365 just Teams, Outlook and Office (in Edge)?

Perhaps, yes.

For many, or not most information based end-users, MS Teams is likely to become the primary interface to Microsoft 365 collaboration team spaces including SharePoint and OneDrive. Just like Outlook, Teams will probably be left open all day.

In theory, the volume of low-value emails, and emails with attachments, should reduce over time.

The developing role of records managers

In this new world, the role of records managers will change from being the curators of records copied to and stored in a separate ‘records and document management’ system, to being records compliance analysts or perhaps, corporate knowledge and information managers and content analysts.

They will learn what the Graph can do, and help to guide AI tools including machine learning and machine teaching, Project Cortex and SharePoint Syntex. They will be responsible for monitoring content across the Microsoft 365 platform, creating and applying retention policies and managing the outcome of those policies, working more interactively with the Graph, and with a range of data.

In organisations that have a requirement to transfer records to archival institutions, the new knowledge and information managers will have a key role in ensuring that this data is suitable for transfer.

They might even have oversight of old paper records gathering dust until they can be destroyed.

Recordkeeping roles and permissions in Microsoft 365

(Updated 3 September 2020 with reference to customised admin roles)

Microsoft 365 is a cloud-based collaboration and content system that includes a wide range of functionality to create, capture and manage records, primarily in SharePoint Online but also in OneDrive for Business, Exchange Online and in MS Teams. 

This post outlines the roles and permissions required by records managers to manage records in Microsoft 365.

Whether all the roles and permissions will be granted may depend on a number of factors including technical competence, security and risk. Where they are not granted, records managers will need to ensure that the relevant IT resources can and will set up and manage the recordkeeping functionality as required.

Azure AD/Microsoft 365 Admin Center roles

There are around 50 roles that can be assigned to individuals in the Microsoft 365 admin center or the Azure Admin portal (which includes 11 more roles).

These roles may be grouped as follows:

• Admin. For example, Global Admin and the Admins for Exchange Online, MS Teams, and SharePoint Online/OneDrive for Business.
• Security and Compliance. For example, Security Admin, Compliance Admin, Compliance Data Admin
• Identity management. For example, Authentication Admin, Guest Inviter, Licence Admin, Password Admin, User Admin
• Device management. For example, InTune Admin, Printer Admin
• Reader. For example, Global Reader, Message Center Reader, Reports Reader, Security Reader

There is no specific ‘records manager’ role in Microsoft 365. The closest in terms of functionality is the Compliance admin role that includes several several sub-roles including ‘RecordManagement’, ‘Disposition Management’ and ‘Retention Management’. Alternatively, a custom role may be created with those (and a couple of other) sub-roles, thereby restricted access to only the sub-roles that are specific to or required by records managers.

In addition to the role and sub-roles required to access the Compliance portal and carry out records management activities, records managers should also be assigned the Global Reader and Reports Reader roles so they can access and view the various dashboards on the Microsoft 365 admin center:

Example dashboard

Compliance admin portal roles and sub-roles

The Compliance admin portal (https://compliance.microsoft.com) includes the following sections that are all relevant for records managers:

• Reports (dashboard)
• Audit logs. These cover the entire Microsoft 365 environment, kept for only 3 months (E3) or 12 months (E5).
• Content Search (effectively eDiscovery)
• Information Governance (where retention labels and retention policies are created and managed)
• Records Management (which is essentially an extended set of IG functionality, including auto-application of labels, available to E5 licence holders, and disposition management)

Access to the Compliance admin portal is restricted to the Global Admins and Compliance Admin and Compliance Data Admin roles. These two roles include various sub-roles (including sub-roles that are not relevant to records management) that are described in considerable detail in this Microsoft page ‘Permissions in the Security & Compliance center‘.

The sub-roles that are most relevant to records managers are:

• RecordManagement (required to manage and dispose record content)
• Retention Management (required to create retention labels)
• Audit Logs
• View-Only Audit Logs (audit logs cannot be modified)
• Disposition Management (required to manage disposition)
• Compliance Search (required to conduct a global ‘case’ search of anything anywhere in the Microsoft 365 platform, including ‘personal’ mailboxes and 1:1 Teams chats)
• Case Management
• Hold

It is recommended that records managers – or select individuals with higher compliance responsibilities, be assigned either to one of the two Compliance Admin roles, or a custom role group with just the sub-roles listed above. This will enable records managers to access the Compliance portal to create, apply and manage records retention policies. They will also have access to the audit logs and content search options. 

Note: The ‘Audit logs’ sub-role is actually assigned via a role group in the Exchange Online admin portal under the Permissions section. The three key roles in this section that contain these sub-roles are ‘Organisation Management’, ‘Compliance Management’ and ‘Records Management’. As the first two contain a very long list of sub-roles, it is recommended that the records manager/s be added to the ‘Records Management’ role group that includes the ‘Audit logs’ and ‘Retention Management’ sub-roles.

SharePoint Admin roles

From an admin point of view, there are essentially three SharePoint admin roles:

• SharePoint administrator. This person has access to the SharePoint admin portal, manages the settings, creates and provisions new sites, and monitors the environment. They are usually also responsible for troubleshooting issues and may have some responsibility for development (including scripts) and customisations or integrations. Subject to the size and complexity of the environment, a records manager with good technical skills, including being an EDRM system admin, may be able to take on the role of SharePoint admin with some training. In most cases, however, this is likely to remain a specialised IT role.
• Site Collection Administrator. This role sits between the SharePoint Admin and the Site Owner role and provides ‘back-end’ access to the SharePoint site. Generally speaking, the SharePoint Admin will always be a Site Collection Administrator, ideally added via an AD Security Group. If records managers are added to this AD Security Group, and that Group is added to the Site Collection Admin section of every SharePoint, they will have the ability to access every site (with all access and actions recorded in the audit logs). This access can be revoked on individual sites if necessary. 
• SharePoint Site Owner. The person assigned to this role will usually be someone working in the business area or group responsible for day to day management of the site. Records managers should not be Site Owners as this suggests that the records managers have day to day responsibility for managing the site (creating libraries for example).

Other factors to consider

Any content stored in OneDrive for Business accounts, Exchange mailboxes and MS Teams will remain accessible via a Content Search as long as it exists. If no retention policy has been applied to these workloads and the end-users deletes that content, there is no way to retrieve the deleted content after minimum periods (90 days for ODfB, 14 days for Exchange mailbox content).

The OneDrive portal includes a Storage section that determines how long the content will be retained after the account becomes inactive. This is separate from any retention policy that may be applied to the accounts via the Compliance portal. Records managers should understand these two elements (retention and storage period).

The Exchange Online admin portal includes a number of legacy recordkeeping elements, in particular the Messaging Records Management (MRM) policies in the compliance/retention policies section. Records managers do not need to be assigned the role of Exchange Online admin but need to engage with the admins regarding the application of Microsoft 365 retention policies. While it is possible to apply label-based retention policies to Exchange mailboxes, including advanced auto-application with E5 licences, in practice it may be much simpler to apply a few broad non-label retention policies to mailboxes.

Screenshot of the MRM policy area

The MS Teams admin portal does not include any recordkeeping settings or elements. However, the records manager should discuss and determine suitable retention requirements for both 1:1 chats and channel chats with the relevant admin. These are created and added via the Compliance admin portal. It is not possible to apply a label-based retention policy to Teams chats, accordingly there is (currently) no disposition review record of what was destroyed.

Conclusion

Records managers need an appropriate level of access to the Microsoft 365 ecosystem to manage records that have been created, captured and stored across the system. The following is recommended:

• Global reader and Reports reader. These two roles provide read-only access to dashboards in the Microsoft 365 Admin portal, allowing records managers to review volumes and activities in the various workloads. 

• Compliance admin or a customised role group. The role group allows the creation and management of records retention policies and dispositions. It also provides access to audit logs and global content searches. 

• SharePoint admin (optional). This role would be suitable for a records manager with the required level of technical competence to manage SharePoint. 

• SharePoint Site Collection Admin (via a Security Group). This role allows records managers to access every site where the Security Group has been added to the Site Collection Admin group. 

 

 

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

• Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
• Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.