How records managers can manage retention labels and policies in Microsoft 365

Microsoft 365 includes the capability, subject to licencing, to apply retention labels and non-label retention policies to content stored in Exchange mailboxes, SharePoint Online sites, Microsoft 365 Groups and OneDrive for Business accounts.

Non-label retention policies may also be applied to Teams chats, Teams channel (including private channel) posts, and Yammer messages. For Teams chats and posts, these policies apply to the ‘compliance copy’ of the messages stored in a hidden folder in Exchange mailboxes.

Both retention labels and retention policies work against individual items stored in these location, not the containers (e.g., Exchange mailbox, SharePoint document libraries, OneDrive account, Team chats and channels. This means that individual items stored in those aggregations will disappear (when destroyed) leaving the container still in place.

Creating and applying retention labels and policies is a relatively straightforward matter. For records and information managers, knowing where the labels and policies have been applied is more complex, as this post explains.

Creating retention labels and policies

Both retention labels and retention policies are created in and applied from the Microsoft 365 Compliance admin portal.

The Information Governance section is used to create both: (a) retention labels that can be published as label policies, or auto-applied (E5 licence required), and (b) retention policies.

The Records Management section is used to create retention labels, based on a file plan that can be published as label policies or auto-applied (E5 licence required).

The two other tabs in the records management section are (a) Events and (b) Dispositions. The latter area is where basic details or records due for disposition, and records that have been destroyed, are displayed.

Publishing retention labels

Retention labels do not do anything until they are either:

• Published individually or in groups as label policies to all or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
• Auto-applied (with an E5 licence) including: (a) via keyword searches, (b) based on information sensitivity, (c), based on basic classifiers, (d) as part of trainable classifiers or (e) SharePoint Syntex (linked with a Content Type).

Labels cannot be published to or applied to Teams chats or channel posts, or to Yammer content.

Applying retention labels

Retention policies may be published (applied) to either:

• All or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
• All or none of: Teams chats, Teams channel posts, Teams private channel posts, Skype for Business chats, Yammer community posts and chat messages.

Viewing labels, label policies, and retention labels

Once they have been created, the basic details of labels, label policies and retention policies can be seen in the Information Governance and Records Management sections.

Information Governance > Labels

Labels listing

The ‘Labels‘ tab displays all labels that have been created (including via the Records Management – File Plan section, see below), with the following columns (bold indicates what is visible in the default view for this listing – see also below for the list of labels in the Records Management section). The listing (and individual items when clicked) does not include any information to indicate if the label has been published to what label policy, or auto-applied.

• Name
• Status (active, inactive)
• Based on (trigger – when created, last modified, label applied, event, none)
• Is record (if ‘records declaration’ is checked)
• Retention duration (days, months, years or none)
• Disposition type (auto delete, (disposition) review required, no action)
• Reference ID (File Plan detail)
• Function/department (File Plan detail)
• Category (File Plan detail)
• Subcategory (File Plan detail)
• Authority type (File Plan detail)
• Provision/citation (File Plan detail)
• Created date
• Created by
• Last modified
• Last modified by

Information Governance > Label policies

Label policies listing

The ‘Label policies‘ tab displays all label policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, e.g., Exchange mailboxes, SharePoint sites, not specific locations), the labels that have been included, and if a preservation lock has been enabled.

• Name
• Type (Publish, other options may also be listed)
• Created by
• Last modified by
• Last modified date

Information Governance > Retention policies

Retention policies listing

The ‘Retention policies‘ tab displays all retention policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, not specifically), retention period and if a preservation lock has been enabled.

• Name
• Created by
• Last modified

Records management section

Records Management > File Plan

File plan details (cut off at right)

The ‘File Plan‘ tab displays all labels (including those created via the Information Governance – Labels section above) with all the columns shown above for the Information Governance labels. Again, there is no information to indicate if the label has been auto-applied or what label policy it has been published in.

Records Management > Label policies

Records management label policies

The ‘Label policies‘ tab displays all label policies that have been created. It contains the same content as the Information Governance ‘Label policies’ details above.

How do you know where retention labels and retention policies have been applied?

As noted above, both retention labels (published as label policies or auto-applied, but not to Teams or Yammer) and retention policies are applied to individual items stored in containers (mailboxes, sites/document libraries, OneDrive accounts, Team chats, Teams channels) in the various workloads.

This model, and the ability for end-users with edit rights to change or remove labels on content stored in SharePoint that is not declared as a record or change a label on emails, can make it very difficult to know (for records managers in particular) what has been applied where.

Microsoft 365 does not provide any out of the box option to view where retention policies have been applied (to individual records). There are currently only two ways to know where retention labels have been applied:

• Post facto – via Content Explorer in the Data Classification section (requires E5 licence), for labels only.
• As part of retention planning.

Both of these options are described below.

Data Classification > Content Explorer

For those with E5 licences, the ‘Content Explorer’ area in the Data Classification section of the Compliance admin portal can provide details of where labels (only) have been (or are currently) applied after the fact.

According to the Microsoft guidance Get started with content explorer, (which includes helpful screenshots) ‘Content explorer shows a current snapshot of the items that have a sensitivity label, a retention label or have been classified as a sensitive information type in your organization.’

It also notes that ‘Access to content explorer is highly restricted because it lets you read the contents of scanned files’, via the second of the two roles required to access this capability:

• Content Explorer List viewer: Membership in this role group allows you to see each item and its location in list view.
• Content Explorer Content viewer: Membership in this role group allows you to view the contents of each item in the list.

So, Content Explorer is helpful, but only to know where retention labels have been applied after the fact. And it requires an E5 licence (or E5 Compliance licence add on).

Documenting retention via retention planning

Planning for retention in Microsoft 365 is an essential requirement for records and information managers. It should always be based on regulatory, archival or business requirements for keeping records.

One of the challenges for the retention model in Microsoft 365 is the application of retention labels and/or retention policies to Exchange Online mailboxes, OneDrive accounts, Teams chats and Teams channel posts (including private channel posts). It is a challenge because the traditional recordkeeping model involved copying content identified as records from the create/capture application to a central recordkeeping system where the records would be subject to retention, whereas in Microsoft 365 the reality is that not all records may (or can) be captured.

Retention and disposal authorities/schedules typically do not include coverage for entire mailboxes or OneDrive accounts, let alone Teams content. In response to this, the United States National Archives and Records Administration (NARA) developed the ‘Capstone‘ (PDF) approach for email, whereby ‘final disposition is determined by the role or position of the account user, rather than the content of each individual email.’ Something similar to this model, also for OneDrive accounts and Teams chats and channel posts (both of which have compliance copies stored in Exchange mailboxes), may also be required.

Tackle retention policies first

Retention policies are sometimes referred to as ‘back-end’ or ‘safety net’ policies because they work in the background and help to capture deleted content that needs to be retained for minimum periods of time after which it can be deleted unless a longer retention (e.g., label-based) applies. Retention policies cannot normally be mapped to retention classes because they target entire workloads not specific aggregations in those workloads (e.g., they target all content in SharePoint sites, not the content in the libraries of those sites).

Retention policies do not include any form of disposal review. The content is either destroyed or ‘nothing happens’ (allowing the content to be destroyed after the retention period expires).

Retention policies can be applied to the following parts of Microsoft 365:

• All or selected groups of Exchange mailboxes
• All or selected groups of SharePoint sites
• All or selected groups of OneDrive accounts
• All or selected groups of Microsoft 365 Groups
• All (or none of) Teams chats
• All (or none of) Teams channel posts
• All (or none of) Teams private channel posts
• All (or none of) Skype for Business chats
• All (or none of) Yammer community posts
• All (or none of) Yammer chats

(Keep in mind that Teams chats and Teams private channel posts are stored in the personal Exchange mailboxes of participants, while Teams channel posts are stored in the Exchange mailboxes of the Microsoft 365 Group linked with the Team, BUT Teams retention policies can only apply to all or none.)

Once a decision is made to create retention policies, this detail should be documented separately from Microsoft 365 – in a spreadsheet or SharePoint list perhaps. The following is a basic example of what might be included in that documentation.

• Senior executive EXO mailboxes – 15 years / date modified / do nothing
• All other EXO mailboxes – 7 years / date modified / destroy
• All SharePoint sites – 7 years / date modified / destroy
• M365 Groups – 7 years / date modified / destroy
• All ODfB accounts – 7 years / date modified / destroy (see also note below)
• All MS Teams chats – 7 years / date created / destroy
• MS Teams channel posts – 7 years / date created / destroy
• MS Teams private channel posts – 7 years / date created / destroy
• Yammer user messages – 1 year / date created / destroy
• Yammer community messages – 2 years / date created / destroy

Keep in mind that retention is based on either date created or date modified, so the content stored in these locations will begin to disappear when the retention period expires. This will leave empty mailboxes, empty SharePoint document libraries, OneDrive accounts and empty Teams. Part of the retention planning needs to include a process for destroying those containers once the content inside them has been destroyed – and documenting that destruction also.

For OneDrive accounts, consideration might be given instead to actively reviewing the accounts of departed employees to removing any content considered valuable to another location (e.g., SharePoint) or account, rather than simply leaving it hidden from view for the period of the retention. This would allow much shorter retention after the employee leaves.

Documenting retention labels

Retention labels will typically be based on individual classes in a records retention schedule or disposal authority (DA). Accordingly, it will be useful to have a listing of all classes in a spreadsheet or SharePoint list with the following details as a starting point:

• Retention schedule class ID
• Retention schedule class name
• Retention schedule class description (or abbreviated version).

Retention labels can then be created based on those classes. Every label includes the following details (see also screenshot below):

• A name that should be obvious to end users who may end up having to choose it or know from a glance what it means. For example ‘5.1.1 Financial records – 7 years’.
• A description – same as the original retention class description, or an abbreviated version.
• A retention period. For example, 7 years.
• A trigger. The options are: ‘When items were created’, ‘When items were modified’, or ‘When label was applied’. (Note that triggers may also be based on events, but these should be used sparingly and only after the full implication of using them is understood.)
• An action at the end of the retention. For example: ‘Delete items automatically’, ‘Trigger a disposition review’ (E5 only), ‘Do nothing’. ‘Do nothing’ is a good option for permanent records as it will stop deletion and tag a record making it easier to find later. There are three other options instead of retain: ‘Retain forever’, ‘Only delete items when they reach a certain age’, and ‘Don’t retain or delete items’.
• (Note: When labels are created from the Records Management section, two additional options appear: (a) Mark items as a record and (b) Mark items as a regulatory record. The implications of using either option needs to be understood).

Options when a new label is created. Note that two additional items appear in the Records Management labels.

As each retention label is created the following details should be recorded next to the classes in new columns in the spreadsheet or list:

• Retention label name (may be a combination of the ID and name, and retention)
• Retention period. For example, 7 years.
• Retention trigger. For example, ‘When created’, ‘When modified’, ‘When label applied’.
• End of retention action. For example, ‘Destroy’, ‘Review’, ‘Do nothing’
• Declare as a record: Yes/No
• Regulatory record: Yes/No

As labels are published, individually or in groups, this information should be added to columns in the spreadsheet or list:

• Retention policy. That is, what is the name of the policy where this policy has been published.
• Policy location. That is, the location/s were the policy has been applied.
• Date applied?
• Date for destruction. (Date applied plus retention period)

If the label is being used via trainable classifiers, SharePoint Syntex, or otherwise auto-applied, that information should also be added to the spreadsheet or list.

• Trainable classifier? Yes/No
• SharePoint Syntex? Yes/No
• Auto applied? Yes/No
• Auto applied location.

End product – for all retention labels and policies

The end of product of is either a spreadsheet with two workbooks, or two SharePoint lists (one for the retention policies, one for the labels), that describe what retention labels and retention policies have been created and where they have been applied across Microsoft 365.

Records and/or information managers will need to proactively monitor the Microsoft 365 environment and provide advice to IT about the destruction of content and records, as well as the containers in which that content was stored.

There may be other ways to achieve this outcome, but records and information managers need to understand what capability is available (or not) out of the box to make informed decisions.

Featured image source: https://www.pexels.com/photo/architecture-black-and-white-challenge-chance-277593/

In place vs in place management of records in Microsoft 365

The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.

This post explains the difference between the two ‘in place’ options. In brief:

• The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
• The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.

The Microsoft in-place model

SharePoint

The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.

Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:

• The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
• Mandatory record metadata.
• Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
• Unique computer-generated record identifiers for each record, regardless of where that record is stored.
• The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
• The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
• Not permitting modification of certain metadata fields.
• The capability to support multiple renditions of a record.
• The capability to increment versions of records when filing.
  Linking the record metadata to the record so that it can be accessed for display, export.
• Enforcement of data integrity, referential integrity, and relational integrity.

Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):

Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.

The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’

The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’

Exchange

A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.

One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.

Archive mailboxes, including ‘auto-expanding archives’, also exist in Exchange Online. In the Exchange Online archiving service description, it is noted that:

Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.

The new ‘in place’ model

A newer form of in-place records management has appeared with Microsoft 365.

Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.

The newer model of in place records management is based on the assumptions that:

• It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
• Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
• The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.

Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.

The relationship between Microsoft’s ‘in place’ focussed records management, and managing everything (including records) in place.

In place co-existence

Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.

Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021

A decision tree for managing records in SharePoint and Teams

One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.

Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.

Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:

• Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
• The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.

Decision tree

With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.

The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:

The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.

Note: the box outline colours represent the four different types of site that are likely to be created.

How will the content be accessed?

It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.

Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.

They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.

If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.

It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).

What type of SharePoint site will be required?

Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.

As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.

Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.

Creation method

The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.

Security Controls

Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.

Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.

Retention

Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.

As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.

Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.

Provisioning

Once the site has been created, it should be provisioned with three key elements shown below.

• Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
• Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
• The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.

The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.

Classifying records in Microsoft 365

The classification of records is fundamental recordkeeping activity. It is defined in the international standard ISO 15489-1:2016 (Information and Documentation – Records Management) as the ‘systematic identification and/or arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules‘. (Terms and Definitions, 3.4)

The purpose of classification is defined by State Records NSW as follows: ‘In records management, records are classified according to the business functions and activities which generate the records. This functional approach to classification means that classification can be used for a range of records management purposes, including appraisal and disposal, determining handling, storage and security requirements, and setting user permissions, as well as providing a basis for titling and indexing‘. (Records Classification, accessed 13 January 2021.)

The ever-increasing volume of digital records, the many different ways to create them, and the multitude of record types that are created and storage locations, have made it more difficult to accurately and consistently manually classify records, including through the creation of pre-defined ‘containers’ or aggregations based on classification terms. Despite this, the requirement to link the classification of records with their retention and and disposal remains.

For over three decades, Microsoft’s applications and technology platforms have been used to create, capture, store and manage records. Some of these records (in the earlier period) were printed and placed on paper files, or stored (from around 2000) in dedicated electronic document and records management (EDRM) systems.

But the volume and type of digital content, including with new types of records (e.g., chat messages) and storage locations, continues to grow. In response, Microsoft invested heavily in addressing the need to classify records ‘at scale’.

This post looks at various ways to classify records, for retention and disposition purposes, in Microsoft 365.

The old-school, manual method – metadata

Most of the records in Microsoft 365 will be created, captured or stored in one of the four primary workloads: Exchange mailboxes, SharePoint sites/libraries, MS Teams chats (a ‘compliance copy’ of which is stored in Exchange mailboxes), and OneDrive for Business libraries. Some records may also exist in Yammer or other web page content (e.g., intranet).

Most SharePoint sites as well as Teams (that have a SharePoint site) will be created according to some form of business need to create, capture, store and share records; that is, the site or team purpose may be based on a business function or activity. This way of grouping records may in some ways be used as a way to classify records – by SharePoint site (e.g., function) or document library (e.g., activity).

Records may be stored in multiple document libraries, or within a folder structure of a single library.

A number of methods (some of which rely on others) can be used to add classification (and other) metadata to records stored in SharePoint document libraries:

1 – Creating the classification taxonomy in the Managed Metadata Service (MMS)/Term Store via the SharePoint Admin portal – Content Services – Term store, and then applying these terms in content types that are then deployed in SharePoint sites.

An example of Business Classification Terms in the MMS

2 – Creating global content types from the SharePoint Admin portal, in the Content Services – Content type gallery area (see ‘Finance Document’ example below) and then deploying these in specific SharePoint sites where site columns that contain classification terms will be added.

3 – Creating site columns that contain classification terms, including from the MMS, and adding these to global or site content types or document libraries where they can be applied to records.

In this example, the site column ‘BCS Function’ maps to the MMS BCS terms

4 – Creating site content types and adding site columns (including MMS-based columns), then adding these content types to document libraries.

In this example, the MMS-based column now appears in the library columns

But, most of the above is somewhat complicated and cumbersome and would normally only be used for and manually applied to specific types of records.

The simplest way to apply BCS/File Plan terms at the document (or document set) level is to (a) store records to the same BCS function or activity in the same library, (b) create site or library columns with default values and add these to the library. This way means that the default terms are applied automatically as soon as a new record is uploaded, including when shared/inherited from the site columns added to a document set that ‘contains’ a document content type.

Example metadata columns shared from the document set content type

However, keep in mind that SharePoint is just one of the workloads where records are stored.

Records in the form of emails, chats and ‘personal’ content (as well as Yammer messages and web pages) are created in and stored across the other workloads. Some attempt may be made to copy these other records (especially emails) in SharePoint sites but it starts to get complicated or impossible to do so with things like Teams chat messages.

In most cases (and according to Microsoft’s own recommendations), it is better to leave the records where they were created or captured (‘in place’), and apply centralised compliance controls (classification, retention labels and policies) to this content.

Leaving the records in place in this way does not exclude the ability to create SharePoint sites and document libraries in those sites that map to classification terms, and/or use the site column approach described above but these are more likely to be exceptions.

In fact, some form of logical structure is almost certain anyway as most end-users will probably want to access and manage information in their own specific work context (the Team/SharePoint site).

Trainable classifiers

Since not all records are stored in SharePoint and the ever-increasing volume of digital content stored across the Microsoft 365 platform, Microsoft needed to find a way to classify records ‘at scale’.

The solution was to use machine learning (ML) via trainable classifiers accessed in the ‘Data Classification’ section of the Microsoft 365 Compliance portal. This capability is only available to E5 licences.

The trainable classifiers solution was released to General Availability on 12 January 2021 (‘Announcing GA of machine learning trainable classifiers for your compliance needs‘, accessed 13 January 2021).

See the Microsoft web page ‘Learn about trainable classifiers‘ to learn more about this option. To quote from that page:

‘This classification method is particularly well suited to content that isn’t easily identified by either the manual or automated pattern matching methods. This method of classification is more about training a classifier to identify an item based on what the item is, not by elements that are in the item (pattern matching).’

Organisations (including E3 licence holders) may make use of five pre-defined trainable classifiers (Resumes, Source Code, Targeted Harassment, Profanity or Threat. A sixth classifier ‘Offensive language’, is to be deprecated). Custom classifiers require an E5 licence.

Custom classifiers require ‘significantly more work’ than the pre-existing classifiers and the process is quite involved (see the process flow diagram in the ‘Learn about’ page link above) but in summary it involves the following steps:

• Creating the custom classifier.
• Creating a set of manually selected example records (50 to 500) in a dedicated SharePoint Online site as the ‘seed’. This would include a range of emails in the seed examples.
• Testing the classifier with the seeded documents.
• Re-training with additional content – both positive and negative matches.

Once the classifier is published, it can be used to identify and classify related content across SharePoint Online, Exchange, and OneDrive (but not Teams).

The page ‘Default crawled file name extensions and parsed file types‘ provides details of all the record types that can be classified in this way. Note it is not clear if trainable classifiers can crawl the compliance copy of Teams chat messages stored in hidden folders in Exchange mailboxes.

Label-based retention policies can then be automatically applied to content that has been identified through the trainable classifier.

However, note that the classifier does not ‘group’, aggregate or ‘present’ (list) the records for review (except broadly via the Content Explorer); however, the label applied to the records can be searched via the ‘Content Search’ option in the Compliance portal. This is a much better option than not having any idea how many records of a particular classification may exist in Exchange mailboxes, OneDrive accounts, or general SharePoint sites. It requires some degree of ‘letting go’ of the ability to view and browse content classified this way, and trusting the system.

The main limit with trainable classifiers is that it requires an E5 or E5 compliance licence.

The other limitation is the management of the disposition of records that have been identified with trainable classifiers and had a label-based retention policy applied. There are significant shortcomings with the current ‘Disposition Review’ process, specifically the lack of adequate metadata to review records due for disposal or the details of what has been destroyed.

SharePoint Syntex

Another (but limited) option might be to use SharePoint Syntex (see ‘Introduction to Microsoft SharePoint Syntex‘ for an overview), although its range is limited to SharePoint and – it seems – only records that have a relatively consistent structure and format.

SharePoint Syntex evolved out of Project Cortex’s ability to extract and capture metadata from records. It can also be used through its ‘Document Understanding Model‘ (DUM) to provide a way to classify records stored in SharePoint Online (only). It makes use of a ‘seeding’ model that is similar to trainable classifiers (and may be based on the same underlying AI engine).

Broadly speaking, the DUM works on the basis of loading a small ‘seed’ set of (relatively consistently formated) example files into a dedicated Content Center (or Centers). This is very similar to the process of using trainable classifiers, except that the latter does not require a ‘content center’ SharePoint site to be created.

• The example files are ‘trained’ by being ‘classified’ through the document understanding process based on a set of ‘explanation types‘ that are used to help find the relevant content. The three explanation types are: (a) phrase list (a list of words, phrases, numbers, or other characters used in the document or information that you are extracting); (b) pattern list (patterns of numbers, letters, or other characters); and (c) proximity (describes how close other explanations are to each other).
• The document understanding model (DUM) produced through the explanation types is associated (and deployed) with a new or existing content type. 
• Once applied to a SharePoint site library, the DUM/content type provides the basis for identifying and tagging (with metadata) other similar records in the location (e.g., the library) where the DUM has been deployed. 
• If the documents have consistent content such as invoices, certain data from those documents can be extracted as metadata. 

Retention labels may be applied to records classified using SharePoint Syntex, as described on this page ‘Apply a retention label to a document understanding model‘.

Summing up – which one should be used?

The answer to this question will depend on your compliance requirements.

Smaller organisations may be able to set up SharePoint sites and document libraries with site columns/metadata that maps to their business classification scheme or file plan, and copy emails to those libraries. There may be little need to use AI-based classification methods.

In large and more complex organisations (with E5 licences), especially those with a lot of content stored across Exchange mailboxes and SharePoint sites (including Teams-based sites) there will most certainly be a need for some form of AI-based classification in addition to classification-mapped SharePoint sites (and Teams).

Organisations with E3 licences might use the manual methods described above for specific types of records, and consider acquiring additional E5 Compliance licences to make use of trainable classifiers or SharePoint Syntex for other records.

All the ways SharePoint sites can be created

SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.

While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.

Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.

So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.

Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.

The following is a list of the main ways a person can create a SharePoint site. Have I missed any?

1. Via a PowerShell script

As described in the Microsoft docs web page ‘Create SharePoint Online sites and add users with PowerShell‘. The script is based on a csv file (‘sitecollections.csv) and looks something like the following (see the link for more details):

Import-Csv C:\users\MyAlias\desktop\SiteCollections.csv | ForEach-Object {New-SPOSite -Owner $_.Owner -StorageQuota $_.StorageQuota -Url $_.Url -NoWait -ResourceQuota $_.ResourceQuota -Template $_.Template -TimeZoneID $_.TimeZoneID -Title $_.Name}

This option also allows the administrator to provision new SharePoint sites.

2. Via the SharePoint Admin portal (+ Create)

This option allows the creation of three main types of sites: modern team sites (Team site),
communication sites, and non-Microsoft 365 Group-linked sites (Other options).

3. By creating a Microsoft 365 Group

Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.

Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.

The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.

Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.

4. By an end-user creating a new Team in MS Teams

Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.

Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an
Exchange Online mailbox that is not visible in Outlook.

5. By creating a Private Channel in MS Teams

If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.

For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.

A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.

6. By the Teams Admin creating a new Team

The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.

As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.

7. From the end-user SharePoint portal (+ Create site)

If not disabled, end users can create a new SharePoint site by clicking on ‘+ Create site’ from the SharePoint portal – https://tenantname.sharepoint.com/_layouts/15/sharepoint.aspx

This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.

It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.

8. From OneDrive for Business as a ‘shared library’

This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.

9. When a new Plan is created in Planner

If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.

10. When a new Yammer community is created

End users with access to Yammer can click on ‘Create a Community’ from Yammer.

To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’

Why have Microsoft allowed this?

It’s a smarter way to manage access.

Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.

The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.

But, some of the decisions don’t seem to make sense.

• Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
• Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
• Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
• And why does a new Plan create a SharePoint site? For what purpose?

Perhaps there is a reason for it. It’s just not clear.

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search.

But what happens if there is no Exchange Online mailbox? It seems the chats become inaccessible which could be an issue from a recordkeeping and compliance point of view.

This post explains what happens, and why it may not be a good idea (from a compliance and recordkeeping point of view) not to disable the Exchange Online mailbox option as part of licence provisioning.

Licences and Exchange Online mailboxes

When an end-user is allocated a licence for Microsoft 365, a decision (sometimes incorporated into a script) is made about which of the purchased licences – and apps in those licences – will be assigned to that person.

E1, E3 and E5 licences include ‘Exchange Online’ as an option under ‘Apps’. This option is checked by default (along with many of the other options), but it can be disabled (as shown below).

If the checkbox option is disabled as part of the licence assigning process (not after), the end-user won’t have an Exchange mailbox and so won’t see the Outlook option when they log on to office.com portal. (Note – If they have an on-premise mailbox, that will continue to exist, nothing changes).

Having an Exchange Online mailbox is important if end-users are using MS Teams, because the ‘compliance’ copy of 1:1 chat messages in MS Teams are stored in a hidden folder (/Conversation History/Team Chat) in the Exchange Online mailbox of every participant in the chat. If the mailbox doesn’t exist, those copies aren’t made and so aren’t accessible and may be deleted.

If end-users chat with other end-users who don’t have an Exchange mailbox as shown in the example below, the same thing happen – no compliance copy is kept. The chat remains inaccessible (unless the Global Admins take over the account).

The exchange above, between Roger Bond and Charles, includes some specific key words. As we will see below, these chats cannot be found via a Content Search.

(On a related note, if the ability to create private channels is enabled and they create a private channel and chat there, the chats are also not saved because a compliance copy of private channel chats are stored in the mailboxes of the individual participants.)

Searching for chats when no mailbox exists

As we can see above, the word ‘mosquito’ was contained in the chat messages between Roger and Charles.

Content Searches are carried out via the Compliance portal and are more or less the same as eDiscovery searches in that they are created as cases.

From the Content Search option, a new search is created by clicking on ‘+New Search’, as shown below. The word ‘mosquito’ has been added as a keyword.

We then need to determine where the search will look. In this case the search will look through all the options shown below, including all mailboxes and Teams messages.

When the search was run, the results area shows the words ‘No results found’.

Clicking on ‘Status details’ in the search results, the following information is displayed – ‘0 items’ found. The ‘5 unindexed items’ is unrelated to this search and simply indicates that there are 5 unindexed items.

Double-checking the results

To confirm the results were accurate, another search was conducted where the end-user originally did not have a mailbox, and then was assigned one.

If the end-user didn’t have a mailbox but the other recipient/s of the message did, the Content Search found one copy of the chat message in the mailbox of the other participants. Only one item was found.

When the Exchange Online option was enabled for the end-user who previously did not have a mailbox (so they were now assigned a mailbox), a copy of the chat was found in the mailbox of both participants, as shown in the details below (‘2 items’).

Summary and implications

In summary:

• If end users chat in the 1:1 area of MS Teams and don’t have an Exchange Online mailbox, no compliance copy of the chat will be saved, and so it will not be found via Content Search.
• If any of the participants in the 1:1 chat have an Exchange Online mailbox, the chat will appear in the mailboxes of those participants.
• If all participants in the 1:1 chat have an Exchange Online mailbox, the chat will be found in the mailbox of all participants.

Further to the above:

• If end users can delete chats (via Teams policies) and don’t have a mailbox, no copy of the chat will exist.
• If end-users with a mailbox can delete Teams chats, but a retention policy has been applied to the chats, the chats will be retained as per the retention policy (in a hidden folder).

And finally, if you allow private channels, end-users can create private channels in the Organisation Team. The chats in these private channels are usually stored in the personal mailboxes of participants (not the Group mailbox) – so these chats will also be inaccessible and cannot be found via Content Search.

The implications for the above are that, if you need to ensure that personal chat messages can be accessed (from Content Search), then the participants in the chat must have an Exchange Online mailbox.

Further, if you allow deletion of chats but need to be able to recover them for compliance purposes, a retention policy should be applied to Teams 1:1 chat.

Recordkeeping roles and permissions in Microsoft 365

(Updated 3 September 2020 with reference to customised admin roles)

Microsoft 365 is a cloud-based collaboration and content system that includes a wide range of functionality to create, capture and manage records, primarily in SharePoint Online but also in OneDrive for Business, Exchange Online and in MS Teams. 

This post outlines the roles and permissions required by records managers to manage records in Microsoft 365.

Whether all the roles and permissions will be granted may depend on a number of factors including technical competence, security and risk. Where they are not granted, records managers will need to ensure that the relevant IT resources can and will set up and manage the recordkeeping functionality as required.

Azure AD/Microsoft 365 Admin Center roles

There are around 50 roles that can be assigned to individuals in the Microsoft 365 admin center or the Azure Admin portal (which includes 11 more roles).

These roles may be grouped as follows:

• Admin. For example, Global Admin and the Admins for Exchange Online, MS Teams, and SharePoint Online/OneDrive for Business.
• Security and Compliance. For example, Security Admin, Compliance Admin, Compliance Data Admin
• Identity management. For example, Authentication Admin, Guest Inviter, Licence Admin, Password Admin, User Admin
• Device management. For example, InTune Admin, Printer Admin
• Reader. For example, Global Reader, Message Center Reader, Reports Reader, Security Reader

There is no specific ‘records manager’ role in Microsoft 365. The closest in terms of functionality is the Compliance admin role that includes several several sub-roles including ‘RecordManagement’, ‘Disposition Management’ and ‘Retention Management’. Alternatively, a custom role may be created with those (and a couple of other) sub-roles, thereby restricted access to only the sub-roles that are specific to or required by records managers.

In addition to the role and sub-roles required to access the Compliance portal and carry out records management activities, records managers should also be assigned the Global Reader and Reports Reader roles so they can access and view the various dashboards on the Microsoft 365 admin center:

Example dashboard

Compliance admin portal roles and sub-roles

The Compliance admin portal (https://compliance.microsoft.com) includes the following sections that are all relevant for records managers:

• Reports (dashboard)
• Audit logs. These cover the entire Microsoft 365 environment, kept for only 3 months (E3) or 12 months (E5).
• Content Search (effectively eDiscovery)
• Information Governance (where retention labels and retention policies are created and managed)
• Records Management (which is essentially an extended set of IG functionality, including auto-application of labels, available to E5 licence holders, and disposition management)

Access to the Compliance admin portal is restricted to the Global Admins and Compliance Admin and Compliance Data Admin roles. These two roles include various sub-roles (including sub-roles that are not relevant to records management) that are described in considerable detail in this Microsoft page ‘Permissions in the Security & Compliance center‘.

The sub-roles that are most relevant to records managers are:

• RecordManagement (required to manage and dispose record content)
• Retention Management (required to create retention labels)
• Audit Logs
• View-Only Audit Logs (audit logs cannot be modified)
• Disposition Management (required to manage disposition)
• Compliance Search (required to conduct a global ‘case’ search of anything anywhere in the Microsoft 365 platform, including ‘personal’ mailboxes and 1:1 Teams chats)
• Case Management
• Hold

It is recommended that records managers – or select individuals with higher compliance responsibilities, be assigned either to one of the two Compliance Admin roles, or a custom role group with just the sub-roles listed above. This will enable records managers to access the Compliance portal to create, apply and manage records retention policies. They will also have access to the audit logs and content search options. 

Note: The ‘Audit logs’ sub-role is actually assigned via a role group in the Exchange Online admin portal under the Permissions section. The three key roles in this section that contain these sub-roles are ‘Organisation Management’, ‘Compliance Management’ and ‘Records Management’. As the first two contain a very long list of sub-roles, it is recommended that the records manager/s be added to the ‘Records Management’ role group that includes the ‘Audit logs’ and ‘Retention Management’ sub-roles.

SharePoint Admin roles

From an admin point of view, there are essentially three SharePoint admin roles:

• SharePoint administrator. This person has access to the SharePoint admin portal, manages the settings, creates and provisions new sites, and monitors the environment. They are usually also responsible for troubleshooting issues and may have some responsibility for development (including scripts) and customisations or integrations. Subject to the size and complexity of the environment, a records manager with good technical skills, including being an EDRM system admin, may be able to take on the role of SharePoint admin with some training. In most cases, however, this is likely to remain a specialised IT role.
• Site Collection Administrator. This role sits between the SharePoint Admin and the Site Owner role and provides ‘back-end’ access to the SharePoint site. Generally speaking, the SharePoint Admin will always be a Site Collection Administrator, ideally added via an AD Security Group. If records managers are added to this AD Security Group, and that Group is added to the Site Collection Admin section of every SharePoint, they will have the ability to access every site (with all access and actions recorded in the audit logs). This access can be revoked on individual sites if necessary. 
• SharePoint Site Owner. The person assigned to this role will usually be someone working in the business area or group responsible for day to day management of the site. Records managers should not be Site Owners as this suggests that the records managers have day to day responsibility for managing the site (creating libraries for example).

Other factors to consider

Any content stored in OneDrive for Business accounts, Exchange mailboxes and MS Teams will remain accessible via a Content Search as long as it exists. If no retention policy has been applied to these workloads and the end-users deletes that content, there is no way to retrieve the deleted content after minimum periods (90 days for ODfB, 14 days for Exchange mailbox content).

The OneDrive portal includes a Storage section that determines how long the content will be retained after the account becomes inactive. This is separate from any retention policy that may be applied to the accounts via the Compliance portal. Records managers should understand these two elements (retention and storage period).

The Exchange Online admin portal includes a number of legacy recordkeeping elements, in particular the Messaging Records Management (MRM) policies in the compliance/retention policies section. Records managers do not need to be assigned the role of Exchange Online admin but need to engage with the admins regarding the application of Microsoft 365 retention policies. While it is possible to apply label-based retention policies to Exchange mailboxes, including advanced auto-application with E5 licences, in practice it may be much simpler to apply a few broad non-label retention policies to mailboxes.

Screenshot of the MRM policy area

The MS Teams admin portal does not include any recordkeeping settings or elements. However, the records manager should discuss and determine suitable retention requirements for both 1:1 chats and channel chats with the relevant admin. These are created and added via the Compliance admin portal. It is not possible to apply a label-based retention policy to Teams chats, accordingly there is (currently) no disposition review record of what was destroyed.

Conclusion

Records managers need an appropriate level of access to the Microsoft 365 ecosystem to manage records that have been created, captured and stored across the system. The following is recommended:

• Global reader and Reports reader. These two roles provide read-only access to dashboards in the Microsoft 365 Admin portal, allowing records managers to review volumes and activities in the various workloads. 

• Compliance admin or a customised role group. The role group allows the creation and management of records retention policies and dispositions. It also provides access to audit logs and global content searches. 

• SharePoint admin (optional). This role would be suitable for a records manager with the required level of technical competence to manage SharePoint. 

• SharePoint Site Collection Admin (via a Security Group). This role allows records managers to access every site where the Security Group has been added to the Site Collection Admin group. 

 

 

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

• Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
• Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.

A Tale of Two Retention Policies in Microsoft 365

There are two ways to create retention policies in Microsoft 365 – (a) by creating a label and publishing it as retention policy, or (b) creating a retention policy without a label. Both are created from the Information Governance section of the Compliance portal in Microsoft 365.

This post describes how the policies are created and applied, and the outcome for each.

Retention policy based on a label

Retention labels are created from the Labels tab of the Information Governance section in the Compliance portal.

Ideally, the name of the label would be based on a disposal class in a records retention schedule or records disposal authority. For the purpose of this post, the new label was named ‘Test One – Retain five years then delete‘.

Adding the details of the retention provided a quick visual clue to the retention details.

The label was configured to have a retention period of 5 years after the date modified after which the content could be deleted, with no disposition review. No record would be kept of what was deleted.

Alternatively, it would be possible to select the Disposition Review option. This option sends an alert to the account indicated so the records could be reviewed before disposal. However, keep in mind that if the trigger is left as ‘Date created’ or ‘Date modified’, records will be ready for disposition review based on those dates. It might be better to choose ‘Date applied’ instead so that a group of records could be reviewed at the same time – for example, in a SharePoint library.

Retention labels have no life unless they are published. This is achieved from the ‘Publish labels’ option in the same tab.

One or more labels can be published in a retention policy. This could be useful if the records disposal authority contained multiple labels for a single function.

As part of the publishing process, a decision has to be made to apply the policy to somewhere in Microsoft 365. In this case, the decision was made to apply it to a single SharePoint site because that site contained records that mapped to the retention label.

Finally, the new policy was given a name. In this case the policy had only one label so the policy had the same name as the label.

If the organisation’s records disposal authority/retention schedule had multiple retention classes mapped to a business function, the policy could be named after the function and include all the relevant classes created as labels.

All of these labels would appear in the drop down list from the Library Settings on every site where the policy was applied.

But even when it was published in this way and applied to a specific SharePoint site, the label does not do anything. The policy has to be applied to a document library in that site – but even then it wouldn’t do anything until the label was selected from a drop down list.

The screenshot below shows how the specific label is selected on the library and whether it should be applied to all existing items.

The label can now be seen in the ‘Retention Label’ column (when made visible in the view settings).

Anyone who uses the library to add and edit content (site members) will notice an obvious change if they try to delete anything, including from their synced document libraries in File Explorer.

They can’t.

There is, however, a workaround that end-users can use to delete documents with a label: (a) check the box next to the document, (b) open the information panel and (c) remove the label on individual documents.

Once the label is removed it no longer appears against the record.

The document can now be deleted. It will move to the Recycle Bin, from where it could be restored for up to 93 days if a mistake was made.

The lessons to be learned from this are that:

• It might be useful to make the library read only after the label is applied.
• The label should only be applied when the library became inactive, to allow people to get on with their work while it is still active.

If anyone was concerned about content being deleted from an active library, they could set an alert on the library.

In the meantime, for all the other records that retained their label, when the retention period ended the records in the library would be transferred to the Recycle Bin where they would sit for 93 days before complete and permanent deletion.

If no-one restored a document from the Recycle Bin, the content would be deleted, permanently, forever, without any record. The records managers might need to make a copy of the metadata for the documents to be deleted before they are deleted.

Alternatively, if they had selected the option for a ‘disposition review’ when the label was created, this would alert the records managers (and others) of the need to review the contents of the library.

Once everything had been deleted, all that will be left will be an empty library with no record of what used to be there.

Retention policy without a label

Non-label retention policies are created from the ‘Retention’ tab of the Information Governance portal.

For the purpose of this post, this label will be named ‘Label Two – Retain 5 years then destroy’.

This policy has the same retention period, trigger, and action as Label One – 5 years after date modified – and then the content could be deleted without any record of this being kept. [Note – yes the screenshot below shows 7 years].

As the policy was being created it was applied directly to a SharePoint site.

The policy could start working immediately.

Nobody, apart from the person who created and applied it knew it had been applied. It didn’t appear anywhere on the SharePoint site where it had been applied.

But, as part of the application process, it created a Preservation Hold (PH) library visible only to Admins, including the Site Collection Admins but not the Site Owners.

(Note – if end-users are allowed to create SharePoint sites, or Teams in MS Teams that also creates a SharePoint site, they are made Site Collection Admins by default and can accordingly see the Preservation Hold Library).

Anything that was deleted in advance of the retention policy expiry would be moved to the Preservation Hold library.

The PH library had a specific purpose – to ensure that everything on the site was retained until the end of the retention period.

It was not possible to restore any items from the Preservation Hold library. It was also not possible to ‘clear’ that library. If an administrator tried, they would be advised that ‘something went wrong’.

The only way to delete from the Preservation Hold library would be to remove the policy.

However, if a record was deleted it would be placed in the Recycle Bin for up to 93 days, from where it could be restored. If a record was restored in this way (from the Recycle Bin), the copy in the Preservation Hold library would remain.

For most people this type of policy was fine. They could (appear to) delete records, unaware that nothing was actually deleted if it were subject to a retention policy.

The same thing happened if a similar type of retention policy was applied to OneDrive accounts. End-users could ‘delete’, but nothing was actually deleted as long as the retention policy was active.

At the end of the retention period, the content that was subject to this policy was transferred to the Recycle Bin where it remained for 93 days. It would then be deleted, permanently, forever, without any record.

All that was left would be an empty library with no record of its previous content. The only clue would be in the ID or the ID part of the Document ID (if enabled). If anything new was added to the library, the next number in sequence would indicate how many records had previously been saved to that library.

Lessons to be learned

There are several lessons to be learned.

The first is that you need a plan to create and apply retention labels or retention policies to content across Microsoft 365. The configuration options and implications of each option need to be understood before they are applied.

You need to understand the difference between (a) a retention label published as a policy and applied to a SharePoint site and then a library on that site, and (b) a retention policy applied to an entire SharePoint site.

Retention labels can be mapped to retention schedule/disposal authority classes and then applied to specific libraries on SharePoint sites where the policy is applied. Accordingly, it may be useful to create document libraries that map to those classes, and only store content in those libraries that is covered by a single retention class.

If single SharePoint document libraries are used to store content that maps to different retention classes, then it will become very complicated to accurately map retention labels to content.

In many cases, it may be possible to apply a single non-label retention policy to a single SharePoint site or sites. For example, all project sites might have the same 7 year retention applied to all the content on the site as there may be no real need to apply policies to individual libraries.

Whatever you do, have a plan to act, document your settings, and keep it up to date.

And understand what happens at the end of the retention period. There is no back up.

Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

There are two types of chat message in MS Chat:

• One-to-one/one-to-many ‘chat’ (top icon above).
• Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

• Knowing if chat messages on any given subject exist. 
• Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

Searches can be configured to find content in any or all of the following locations:

• Users, Groups, Teams
  • Exchange email
  • Office 365 group email
  • Skype for Business
  • Teams messages [the copy in the mailbox]
  • To-Do
  • Sway
  • Forms
• SharePoint
  • SharePoint sites
  • OneDrive accounts
  • Office 365 group sites
  • Teams sites
• Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

• As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
• There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

Classification

How records managers can manage retention labels and policies in Microsoft 365

Microsoft 365 includes the capability, subject to licencing, to apply retention labels and non-label retention policies to content stored in Exchange mailboxes, SharePoint Online sites, Microsoft 365 Groups and OneDrive for Business accounts. Non-label retention policies may also be applied to Teams chats, Teams channel (including private channel) posts, and Yammer messages. For Teams chats […]

October 18, 2021

Information Management

Crossing the gangplank – How Microsoft 365 supports the capture of tacit knowledge

This post is dedicated to the memory of my former colleague and good friend Walter de Ruyter who passed away on 4 October 2021. The text is based on Walter’s many presentations and our close collaboration over almost 10 years. Henri Fayol (1841 – 1925) was a French mining engineer who developed a general theory of […]

October 5, 2021

Electronic records

A basic retention model for Microsoft Teams

In my previous post about managing inactive Teams, the third option listed was to apply retention policies to those Teams. It included the graphic below. This post provides more details of a basic retention model that can be applied to both active and inactive Teams. Key takeaways Key takeaways from this post for records and […]

September 29, 2021