The Microsoft 365 experience – Teams, Exchange, Outlook, Edge: Where did SharePoint Go?

At the 2020 Microsoft Ignite conference, Jeff Teper presented a diagram titled ‘Microsoft 365’. The diagram showed only four icons: Teams, Outlook, Office and Edge.

The implication of this diagram was that, for most end-users, Teams is now (or will become) their primary portal into Microsoft 365. As stated by Jeff Teper, SharePoint is a foundation platform, the out of sight content engine. Edge’s ability to serve up search results from Microsoft 365 further reduces the need to go to SharePoint.

So, what are the implications for managing records?

SharePoint as a recordkeeping system

For a long time, records have been created, captured and stored in recordkeeping systems.

In the paper world, the recordkeeping system consisted of paper records stored in files and boxes and detailed in registers. With the introduction of computers in the 1980s, registers were transferred to databases, making it a bit easier to find records. In the late 1990s, recordkeeping databases were linked with (separate) file stores and became electronic document and records management (EDRM) systems that continued to manage paper records (the so-called ‘hybrid’ systems).

For almost a decade (since SharePoint 2010 was introduced), SharePoint has contended with files shares and EDRM systems as an alternative recordkeeping system, providing almost all the same core functionality.

The ability to create a record in a single location, then share and co-author it from that location, has completely removed the requirement to copy a record to a separate recordkeeping system.

And then came Teams

Someone at Microsoft had incredible foresight to see the potential for a new user interface that would replace products like Lync and Skype for chat and conferencing, and would also provide access to files stored in SharePoint.

SharePoint has been a core part of the Microsoft productivity offerings for a very long time and people have built careers around developing functionality on the SharePoint platform to appeal to end-users, the intranet being the most common case in point, with customised team sites close behind.

The arrival of Microsoft 365 Groups and then Teams in 2017 was perhaps not widely noticed. One could argue that end by the beginning of 2020, it was still largely unnoticed.

And then came a pandemic and working from home. Teams – which may have been largely ignored or overlooked until then – was already ready to take its place next to Outlook, Office and Edge as a primary end-user interface.

New Teams were created, sometimes with abandon (and were sometimes just as quickly abandoned).

Both 1:1 (or 1:many) chats and channel chats took off. Files were created and shared via OneDrive for Business (‘Files’ in the 1:1 chat area), or via the back-end SharePoint sites (‘Files’ in the channel chat area).

There was (and maybe still is) a belief that files were being saved to Teams but not SharePoint. ‘We are storing everything in Teams’ was not an uncommon expression, sometimes followed by ‘but we’re not using SharePoint or OneDrive’.

The year 2020 saw a huge increase in the volume of records stored in SharePoint sites linked with Teams, as well as a completely new set of records – chats (‘compliance’ copies of which are stored in Exchange mailboxes).

The diagram below provides an overview of the relationship between Teams, Microsoft 365 Groups, Exchange mailboxes, SharePoint and OneDrive for Business.

What about SharePoint?

As the diagram above shows, SharePoint has not disappeared. Many organisations will continue to use, and ask end-users to access, SharePoint sites directly to store and manage records.

But accessing SharePoint from SharePoint may become less necessary over time. At Ignite 2020, the ability to pin a ‘home site’ (such as an intranet) to Teams was demonstrated. Even the intranet may end up in Teams.

As Jeff Teper said, SharePoint is a foundation platform, one that does not get in the way of collaboration and productivity but powers it.

Implications for records managers

Records managers, who were likely already on a steep learning curve regarding SharePoint, need to continue to improve their knowledge of the SharePoint platform. On a positive note, SharePoint Online is a much easier application to learn and manage, compared with its earlier on-premise predecessors.

In organisations that have been using SharePoint for a while and/or have allowed the free-creation of Teams in MS Teams, there will some requirement for retrospective analysis, review, and cleaning up.

In all organisations, there will be a requirement to establish some form of governance and oversight of records (files and chats) that have been created, including for the purpose of retention and disposal/disposition.

Retrospective implications

Where MS Teams has been implemented with little thought given to naming conventions, SharePoint site provisioning, or access controls, records managers should been given access to and review the list of all SharePoint sites that have been created, including from MS Teams. This will provide an initial idea of the volume of content and activity on each site, and what action needs to be taken on things like inactive Teams.

Ideally, records managers should be added to the Site Collection Administrators (SCA) group of every SharePoint site, including MS Teams-based sites. This action will give records managers access to the content on every site and to help advise on the management of records in those sites (including Team-based sites).

• The best way to do this is to add records managers to a Security Group and then add that Group to the SCA group of every site. This access could be deferred for sites that contain very sensitive information, although typically records managers would have access to all records, including if they had an EDRMS. And, access is always recorded in audit logs or the local site ‘viewers’ (where enabled) and ‘last modified by’ information.

Access to the chat content of Teams (including 1:1 chats) will not normally be required; some understanding of the content could be inferred from the name of the Team or the SharePoint content. If necessary, Global Admins or a Compliance Admin can run a Content Search across Teams to find chat content, and/or export that content by an individual person or subject.

Records managers will also need to advise on the appropriate retention policy or policies that need to be created and then applied to:

• The chat content in 1:1 chats.
• The chat content in the various Teams.
• SharePoint sites linked with Teams.
• Exchange mailboxes.
• OneDrive for Business accounts. An additional consideration is how long the content of inactive ODfB acccounts should be retained via the ‘Storage’ policy (default is 30 days then permanent deletion).
• SharePoint sites not linked with MS Teams. This includes whole sites as well as library-based retention policies.
• Office 365 Groups (mailbox/SharePoint site). If linked with a Team, a second retention policy is required for the Team chat content retention (second dot point above). For example, one policy ‘GroupABC’ and a second policy ‘GroupABCTeamChat’.

As many of the above retention policies replace the need for backups, records managers need to discuss the options with their IT colleagues.

Forward looking implications

Ideally, there should be some form of governance around the creation of new Teams in MS Teams. These governance arrangements might include:

• The necessary access for records managers. For example, Site Collection Administrator on every site, and/or a customised Compliance Admin role to create and access retention policies.
• Controls around the creation of new Teams, including naming conventions. If not controlled, what processes will ensure that records are properly managed.
• Retention implications. For example, can the new site and/or the channel chat content be covered by another retention policy – e.g., ‘All Teams with assessed low-level working content should be kept for 5 years’.
• Simple best practice guidance for all new users, including on how to share and co-author.
• Retention policies for all Microsoft 365 content, not just SharePoint.
• Reviews of the content of OneDrive for Business accounts of departed end-users, especially for people in senior or decision making positions. It is relatively common practice for end-users to delete (and download) this content before they leave their jobs.
• Monitoring and oversight of content, including access to reporting dashboards.

So, is Microsoft 365 just Teams, Outlook and Office (in Edge)?

Perhaps, yes.

For many, or not most information based end-users, MS Teams is likely to become the primary interface to Microsoft 365 collaboration team spaces including SharePoint and OneDrive. Just like Outlook, Teams will probably be left open all day.

In theory, the volume of low-value emails, and emails with attachments, should reduce over time.

The developing role of records managers

In this new world, the role of records managers will change from being the curators of records copied to and stored in a separate ‘records and document management’ system, to being records compliance analysts or perhaps, corporate knowledge and information managers and content analysts.

They will learn what the Graph can do, and help to guide AI tools including machine learning and machine teaching, Project Cortex and SharePoint Syntex. They will be responsible for monitoring content across the Microsoft 365 platform, creating and applying retention policies and managing the outcome of those policies, working more interactively with the Graph, and with a range of data.

In organisations that have a requirement to transfer records to archival institutions, the new knowledge and information managers will have a key role in ensuring that this data is suitable for transfer.

They might even have oversight of old paper records gathering dust until they can be destroyed.

Auto-populating Microsoft Word templates – a no-workflow option

In August 2019 I published a post titled ‘Auto-populating document templates via a form in SharePoint‘. The suggested model was based on using a SharePoint 2010 workflow in SharePoint Designer.

In July 2020, Microsoft announced that it would turn off SharePoint 2010 workflows for newly created tenants from 1 August 2020, and would remove the ability to run or create these workflows from existing tenants from 1 November 2020. It recommended that organisations use Power Automate instead. (Source: Microsoft Ending Workflows for SharePoint 2010 Online Next Month).

This post details an alternative option, with no workflows required – but with the same set up – that may be sufficient in most cases. At this sage (September 2020), there does not appear to be an out of the box option with Power Automate that will create a document (from a content type) when a new document set is created.

Overview

The alternative option requires the following elements:

• The document set feature enabled.
• New site columns for each unique metadata required in the output documents.
• New document set and document content types, both with the same metadata.
• One or more Word document templates.
• A SPO site with a document library. It is recommended that a dedicated library (e.g., ‘Client Agreements’) is created for this purpose.

Note – the person who does this must be a Site Collection Administrator.

The main difference with the earlier model is that there is no workflow to create the document/s. Instead, the end user creates and saves (without modification except a name) the document into the library/document set where the custom metadata is stored. The method below then auto-populates those new documents with the required metadata.

This option does not use a separate list as the metadata details can be extracted from the document sets.

Enabling the document set feature

The document set feature must be enabled via Site Settings > Site Collection Administration > Site collection features. If you don’t see this option, you don’t have the required level of permission.

It may be a good idea to activate the Document ID Service at the same time as this can be a useful additional piece of metadata in the final documents.

New site columns

From the same Site Settings area, go to the Web Designer Galleries section to add the required new site columns. Directly below that option is where you create new site content types.

The Site Columns section has a very large number of default columns that come with every SPO site. Check to make sure the required columns don’t already exist, or are non-recommended column names (like ‘Library’).

Assuming the required columns do not exist, select the ‘Create’ option to create a new custom column. A couple of points to keep in mind here:

• Give the column a name that is obvious (easy to find).
• Don’t put spaces between words but capitalise each word (e.g., ‘ClientName’, ‘ClientAddress’). From experience, this space can cause the metadata element to wrap to a second line and cause problems.
• Keep in mind that the ‘Single line of text’ is free text. If you want controlled choices, use the Choice option instead.
• Date and Time may require some tweaking to get the correct format in the output Word document. It may be easier to use ‘Single line of text’ if some dates may vary or aren’t exact.
• Don’t use the ‘Yes/No’ option. Instead, use a Choice column with those options instead.
• Person or Group is only for internal names recorded in Active Directory (AD). If you need to insert any other name, use the single line of text.

The additional column settings that may be useful are:

• Require that this column contain information. If changed to ‘Yes’, this metadata field will be mandatory.
• Enforce unique values. Generally not used unless there is a requirement to enforce unique values (e.g., unique client IDs).
• Maximum number of characters. This can be useful to restrict the number of characters. For example, ‘Postcode’ will only have four characters. Don’t use the number field for postcodes.
• Default text. This option is useful if there is a requirement to have a default value, especially from a choice list, or if the value will usually always be the same (although that could also be simply added to the text of the document template.
• Column formatting. Don’t use this if the content is to be copied to a document template.
• Column validation. This may be useful to ensure that only certain characters are entered. For example, postcodes will never have a letter, only numbers.

Repeat this process until all the required metadata columns exist.

New document set content type

Create a document set content type so the output Word documents (and any other digital content) can be stored in the same ‘folder’.

To create a new document set content type, return to the Site Settings and, under Web Designer Galleries, select ‘Site content types’.

Give the new content type a name that is obvious for its purpose (e.g., ‘ClientAgreementsFolder’).

Importantly, change ‘Select parent content type from’ > Document Set Content Types, and ‘Parent Content Type’ > Document Set.

Open the content type that has been created. Under the ‘Columns’ section, add the columns ‘from existing site columns’. The example below shows a series of site columns that were added. Don’t remove any of the other columns.

In the above example. the ‘Title’ column is hidden (default setting). If you click this column, you will see the option under the ‘Change Content Type Column’ to make it required, optional or hidden. Generally, these settings should not be changed.

New Document Content Type

Now, create a new document content type. Give the content type a name that reflects its purpose. Ensure that ‘Select parent content type from’ is set to ‘Document Content Types’ and ‘Parent Content Type’ is set to ‘Document’ as shown below.

The next step is to add the two content types to a document library.

Adding the content types to a document library

All the steps below can be carried out by a Site Owner.

Open the library where the content types are to be added and go to Library Settings (via the Cog/Gear option). In the Library Settings, under ‘Advanced Settings’, click the option to ‘Allow management of content types’ (change from No to Yes).

If you don’t want other folders to be created in this library, change ‘Make ‘New Folder’ command available’ from Yes to No.

A new section will now appear in the Library Settings – ‘Content Types’.

To add the two content types that were created, click on ‘Add from existing site content types’, select each content type from the list that is displayed, and click OK.

The two content types should now appear in the list of Content Types and the custom site columns should appear in the ‘Columns’ list.

Click on the document set content type (e.g., ‘ClientAgreementsFolder’). The custom columns should be visible. Now, we need to share these columns with the custom document. Click on ‘Document Set Settings’.

• From the list of ‘Available Site Content Types’, chose the custom document content type that was created above. Ignore the ‘Document’ content type.
• In the ‘Shared Columns’ section, check the box against all the custom columns.
• Click OK.

The steps above will mean that all the custom columns in the document set will be shared with the the document content type. When you return to the Library Settings, you will see under the ‘Columns’ section that the columns are now shared with the two custom content types as well as the default Document content type (which can be ignored).

Add the Word template to the document content type

Now, add the Word template to the document content type. Note that this step is NOT done at the Site Content Types level, but at the local library level.

Ensure the Word document template is in the latest version of Word (e.g., docx) and has all the text required. It can be updated at any time (see below) but it is important to ensure that the document is as complete as possible, and the sections where the metadata is to be placed is obvious. One way to do this is to put text such as CLIENT NAME in the body of the document. Avoid using highlights as this can be troublesome (but not impossible) to remove.

From the Library Settings, Content Types section, click on the custom document content type (e.g., ‘ClientAgeements’). Click on Advanced settings.

Click ‘upload a new document template’ and add the document template. The name of the template should be obvious. It is possible to go back and edit the template at any time after it has been uploaded. This is usually much easier if a change is required, compared with uploading a new template and having to re-link all the metadata.

Connect the metadata to the document content

From the Document Template section above, click on ‘Edit Template’. This should open the document in the installed Word application. These steps cannot be carried out using Word Online.

Place the cursor in the body of the text where the metadata is to be placed. Click on the ‘Insert’ tab in Word, then click on ‘Quick Parts’ in the text section.

Click on ‘Document Property’ and the custom columns will appear.

When you select each property, the word document will display it as shown below. Continue this process until all the required metadata is added to the document:

As noted earlier, it is usually much easier to edit an existing template ‘in place’ instead of uploading a new version because the steps involved in adding the metadata to the template must be repeated if a new template is uploaded.

Auto-populating Word documents

Now that all the elements are in place, metadata is added automatically to the Word template as follows.

First, open the document library. Click on ‘+ New’ and choose the document set content type (e.g., Client Folder).

Fill in the metadata elements required, then click ‘Save’.

Now, open the document set/folder, and click on the document content type (e.g., ‘ClientAgreement’). This will open the Word template.

Don’t change the document, just click File – Save As. This should display the list of SharePoint sites on the right. Give the document a name, then choose the relevant site (e.g., ‘Client Agreements’), then the relevant library and document set/folder and press save. This is the most complicated part of the process once it is set up.

The template document, with all the metadata elements, should now appear in the document library under the document set/folder. As it is a Word document, it is possible to ask someone to sign on glass, then ‘save as’ a PDF.

If the document requires a sensitivity label and these are enabled, this can be added directly from the Word document menu when it is created.

Other relevant documents (e.g., emails) could be added to the same folder, including drag and drop from Outlook or other locations when the library is synced to File Explorer.

Note that the metadata elements that were added to the document in this way form part of properties of the document. These properties, which will remain with the document if it downloaded or attached to an email, can be viewed from the ‘File – Info’ section of the Word document.

Summary

The disabling of SharePoint 2010 workflows in SharePoint Designer has removed the ability to auto-create a document in SharePoint with the required metadata. However, it is not particularly difficult to set up a site, with two custom content types, one of which contains a document template, to achieve the same outcome without any workflow at all. The only complicating factor is the requirement to save the document template back to the library – without making any changes at all.

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

• Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
• Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.

Microsoft 365 – Options for collating, grouping and aggregating records

The international standard for records management, ISO 15489-1:2016 (‘Information and documentation – Records management – Part 1: Concepts and Principles’), defines records as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’.

Among other things, the standard notes that records systems may exist in a variety of forms, not necessary as or in a single or dedicated application. It also underlines the importance of appraisal; that is, the recurrent analysis of business context, business activity, processes and risk for the purpose of determining what records to make and keep and how to manage them over time – especially given the complexity of contemporary recordkeeping.

In terms of risks, the standard states that risk management is required to develop strategies for managing records and the management of records as a risk management strategy in itself.

Unlike traditional electronic document and records management (EDRM) systems that are used to store copies of records created and stored in other applications (‘exception management’), the Microsoft 365 environment is a single system in which records are a sub-set of the entire content (‘exception identification’).

This post discusses how records can be collated, grouped and aggregated in Microsoft 365 to meet requirements for management records. It emphases the point made in the international standard that the risk to records should be understood and minimised.

Records and context

Records are usually created or captured in some form of context – for example a business activity or project. This in turn provides the basis for collating, grouping or aggregating those records according to that context – commonly, a ‘subject’ or ‘topic’.

Records may be a subset of a broader subject (or series). They may be relevant or relate to more than one context or subject.

Digital records that may have no obvious context when they are first created or capture (for example a casual email about an ‘unusual virus outbreak’ in November 2019) may form part of a specific context only when their value is recognised (‘global pandemic’).

Grouping digital records

Grouping records in the digital world has up until now usually involved copying a digital record, created or captured in one system (such as email or a network file share), to a digital ‘file’ in another system such as an electronic document and records management (EDRM) system. The digital ‘file’ in those systems is a virtual representation; the records are actually stored in a file share, linked by metadata in the form of a file number.

The grouping of digital records as exceptions had (and continues to have) several flaws:

• It assumed that all types of digital records could be stored in a digital ‘file’ from where they could be faithfully and reliably rendered (and not just stored as zipped versions of exported content from the originating system).
• It relied on the willingness of end-users (often after training) and/or a technical third-party system, to copy a record to the system. This ‘exception management’ meant that some records were not copied to the EDRMS.
• It was a ‘point in time’ capture. The original digital record remained in the system where it was created or captured, and might also be attached to emails and from there saved to multiple other locations.
• There was no way of knowing if all the records in the file were all the records relating to the subject.

Where are the records created or captured in Microsoft 365

Most business records in Microsoft 365 will be created or captured in Outlook/Exchange mailboxes, SharePoint site libraries or MS Teams (which stores chat in Exchange mailboxes and documents in SharePoint or OneDrive). (For the purpose of this post, OneDrive is seen as a personal working space that should not be used to store business records.)

Regardless of whether they are created or captured in Exchange or SharePoint (including via Teams), all of the content – records and non records – created or captured in Microsoft 365 is stored in the Azure substrate. This effectively means that records in Microsoft 365 are a sub-set of all the other content stored in the Azure substrate.

Consequently, the management of records in Microsoft 365 involves exception identification. That is, identifying records and ensuring they are managed appropriately as much as possible where they are captured or created – and placing other controls over all the other content as necessary.

Everything created and stored in Microsoft 365 – including all the very rich metadata associated with every digital record – is subject to the Graph. The Graph identifies relationships and ‘signals’ not only between digital content but between people (agents) and business activities.

The Graph powers Delve and Discovery and the soon-to-be-released Project Cortex, presenting information (they have access to) to end-users that can sometimes be unsettling for people used to working in relative privacy. See below for further discussion about Project Cortex.

Additionally, as all the content in Microsoft 365 is stored in the Azure back-end, most of it can be searched and (where necessary) exported through the Content Search option in the Compliance portal, a capability that supports eDiscovery. This capability means that even when records are not ‘manually’ identified as records, there is a better chance they will be found.

How are records aggregated in Microsoft 365

There are three main ways that records are, or can be, aggregated in Microsoft 365: Exchange mailboxes, SharePoint site libraries, and Microsoft Groups that have a mailbox and a SharePoint site and can be linked to (or created from) a Team in MS Teams.

Exchange/Outlook

Exchange aggregates email records by:

• Personal mailboxes, accessible only the ‘owner’ (end-user).
• Shared mailboxes, accessible to those who have access.
• Microsoft 365 Group mailboxes, accessible to the members of the Group (including anyone added to the Group).

Although a mailbox is a form of aggregation, there is no way to relate or link emails stored there with other related records stored in SharePoint unless they are copied to a SharePoint document library, as can be seen in the example below. This is recommended if an organisation wants to keep emails together with other records.

A SharePoint library with an email and a document

Emails copied to a SharePoint document library are a ‘point in time’ copy; there may be additional replies to the email, forming a thread that isn’t captured.

The alternatives to copying emails to SharePoint are:

• Leave all emails in mailboxes and use Content Search to find and export them to SharePoint as a PST.
• Creating a Microsoft 365 Group with an associated mailbox and SharePoint site, so that the records are retained in the context of the Group.

In any case, all mailboxes should be subject to a minimum retention period to ensure that any email that might be a record is preserved for that period. Certain mailboxes (for example, senior or key staff members) may be kept for longer periods and then exported for permanent storage.

SharePoint

SharePoint document libraries are logical aggregations for the storage of records, including emails copied from Exchange mailboxes.

A SharePoint library configured to manage records

Ideally, individual libraries that are used for the storage of records should map to a business activity and/or records retention class; this mapping should be reflected in the library name.

NOTE: Individual document libraries should not be used to store records relating to multiple subjects or mapping to more than one retention class or policy.

Document libraries may be assigned as much metadata as required, and content stored in them can be defined through the use of metadata and/or content types.

Almost every type of digital file, with a (newly announced) 100GB single file limit, can be stored in SharePoint Online, as noted in the article ‘Types of files that cannot be added to a list or library‘ (restrictions only apply to on-premise versions).

Microsoft 365 Groups (including Teams in MS Teams)

Microsoft 365 Groups provide a way to group and manage records, including MS Teams channel chats, in the context of the Group.

The relationship between Teams and Groups, Exchange and SharePoint

Every Group includes a mailbox (visible in Outlook) and a SharePoint site, and can be linked to new Team in MS Teams. Teams channel chats are stored in a hidden folder in the Group mailbox. Any documents and records are stored in the ‘Files’ tab of the channel, which surfaces the default ‘Documents’ library in the connected SharePoint site.

If the creation of Teams is allowed from the MS Teams application, every new Team creates a Microsoft Group (with the same name) and a SharePoint site (with the same name), however the mailbox (with the hidden folder for channel chats) is not visible from Outlook.

(The exception here are private channels; if these are allowed: (a) the chat content is stored in the Exchange mailbox of the each participant, and (b) a new SharePoint site is created for the ‘Files’.

The relationship between the content created by the Group is most obviously visible from the ‘Activity’ web part of the SharePoint site of the Group as can be seen in the screenshot below. This shows (right to left), an original incoming email from Outlook in the Group’s mailbox, the copy saved to the SharePoint document library, and the Word document reply. The specific context of the record (= the ‘file’) – ‘Correspondence 2020’ – is defined by the document library.

What about records in 1:1 Teams chat

As with OneDrive, Teams 1:1 chat should not be used to create or capture records, but may be used as a ‘working’ space.

However, ‘should’ and ‘reality’ can be different things. There are two ways to address this:

• Explictly, through communication to end-users. Make it clear that Teams 1:1 chat and OneDrive are NOT to be used to create or capture records. Applying short-term retention policies to this content may assist with reducing (or increasing) this risk.
• Implicitly, through monitoring and retention policies. Apply longer-term retention policies to the content and use Content Search/eDiscovery to look for content that may be records. Additionally, review the content of the OneDrive of departed staff and ensure that any records are kept.

Implications for managing records

The implications for collating, grouping and aggregating records in Microsoft 365 are as follows.

• SharePoint document libraries will continue to be the primary aggregation for managing corporate records, including emails copied from Outlook.
• Organisations should establish an architecture model for SharePoint sites that are used to manage records. The model may include a mix of the following: (a) sites mapped to business functions with libraries mapped to business activities and retention classes, (b) entire sites used to create and capture records relating to a single activity, where the entire site is mapped to a retention class, and (c) MS Groups (and Teams) with an associated SharePoint site, where the Group (mailbox/SharePoint site) is subject to a single retention class (and the Team channel chat also).
• More effort, in terms of site/library set up, metadata, access controls, retention and end-of-retention process is likely to be required for the management of high-level, high-risk and permanent records.
• Personal mailboxes in Exchange will continue to exist as a form of aggregation, and consideration should be given to having different retention policies for different ‘types’ of mailbox, to ensure that any email that could be a record is not deleted too quickly.

Addendum – Other options that collate, group and aggregate content in Microsoft 365

Content Search/eDiscovery

As noted earlier, all of the content created or captured in Microsoft 365 is stored in the backend Azure substrate. Consequently, it is possible to search across all or part of that content to find related information and, where required, export it to a different location.

The global Content Search is accessed from the Compliance portal and access requires elevated privileges – Global Admin or Compliance Admin.

Searches are created as cases and are based on keywords, conditions (such as ‘Sender’ for emails), and locations – all or specific. When a new content search is created or run, the Global Admins are alerted, providing a form of oversight in addition to audit logs.

While content searches find content is related to the search parameters, and legal holds can then be applied to that content, they do not create any form of aggregation in a recordkeeping sense.

The Graph, Delve, Discovery

Microsoft describe the Graph as being ‘the gateway to data and intelligence in Microsoft 365 [that can be used via the Microsoft Graph API] to access the tremendous amount of data in Microsoft 365, Windows 10, and Enterprise Mobility + Security’ and ‘… build apps that support scenarios spanning across productivity, collaboration, education, people and workplace intelligence, and much more. (Source ‘Overview of Microsoft Graph‘)

The Graph is commonly represented in diagrams similar to the one below.

Most end-users will encounter the Graph through either Delve or the Discover option in both the office.com portal and their OneDrive for Business accounts.

It is not uncommon for end-users to express surprise at the content (that they have access to) that is presented. Commonly this will show documents that a colleague is working on, or connections between people. Disabling Delve does not fix permissions; if a person has access to a document that appears in Delve, they will be able to search for it and find it that way.

Over time, the Graph can also provide other information based on the relationships or ‘signals’ it finds between all the different content in Microsoft 365.

While the Graph can present groups of records that have some relationship to the end-user, it does not aggregate those records or maintain a single consistent view. However, the Graph powers the new Project Cortex that does do something similar.

Project Cortex

Project Cortex was announced by Microsoft in April 2019. To quote the announcement, Project Cortex:

• Uses advanced AI to deliver insights and expertise in the apps you use every day, to harness collective knowledge and to empower people and teams to learn, upskill and innovate faster.
• Uses AI to reason over content across teams and systems, recognizing content types, extracting important information, and automatically organizing content into shared topics like projects, products, processes and customers. Cortex then creates a knowledge network based on relationships among topics, content, and people.

From a recordkeeping aggregation point of view, a core functionality of Project Cortex is its ability to create ‘topic cards’ based on the rich metadata that makes up all the content in Microsoft 365. Again to quote the announcement:

• Project Cortex securely collects content that is created and shared every day in Microsoft 365—including files, conversations, recorded meetings and video—and it categorizes the content based on its type, and tags it with extracted metadata.
• AI then applies advanced topic mining logic—whether its content contained in Microsoft 365 or connected from external systems—to identify topics and relate content to those topics.
• Topics can reflect any knowledge that’s important, including customers, products, projects, policies and procedures. Technically, AI is creating knowledge entities, a new object class, in the Microsoft Graph. The relationships between those topics—those knowledge entities—and the experiences that connect this knowledge with people creates your knowledge network.

Topic cards – or ‘knowledge entities’ – are a form of AI-generated aggregation.

However, topic cards will only present information that an end-user has access to and so the nirvana of presenting emails or Teams 1:1 chats in these cards as a form of aggregation for recordkeeping purposes is not likely to be realised through Project Cortex.

A Tale of Two Retention Policies in Microsoft 365

There are two ways to create retention policies in Microsoft 365 – (a) by creating a label and publishing it as retention policy, or (b) creating a retention policy without a label. Both are created from the Information Governance section of the Compliance portal in Microsoft 365.

This post describes how the policies are created and applied, and the outcome for each.

Retention policy based on a label

Retention labels are created from the Labels tab of the Information Governance section in the Compliance portal.

Ideally, the name of the label would be based on a disposal class in a records retention schedule or records disposal authority. For the purpose of this post, the new label was named ‘Test One – Retain five years then delete‘.

Adding the details of the retention provided a quick visual clue to the retention details.

The label was configured to have a retention period of 5 years after the date modified after which the content could be deleted, with no disposition review. No record would be kept of what was deleted.

Alternatively, it would be possible to select the Disposition Review option. This option sends an alert to the account indicated so the records could be reviewed before disposal. However, keep in mind that if the trigger is left as ‘Date created’ or ‘Date modified’, records will be ready for disposition review based on those dates. It might be better to choose ‘Date applied’ instead so that a group of records could be reviewed at the same time – for example, in a SharePoint library.

Retention labels have no life unless they are published. This is achieved from the ‘Publish labels’ option in the same tab.

One or more labels can be published in a retention policy. This could be useful if the records disposal authority contained multiple labels for a single function.

As part of the publishing process, a decision has to be made to apply the policy to somewhere in Microsoft 365. In this case, the decision was made to apply it to a single SharePoint site because that site contained records that mapped to the retention label.

Finally, the new policy was given a name. In this case the policy had only one label so the policy had the same name as the label.

If the organisation’s records disposal authority/retention schedule had multiple retention classes mapped to a business function, the policy could be named after the function and include all the relevant classes created as labels.

All of these labels would appear in the drop down list from the Library Settings on every site where the policy was applied.

But even when it was published in this way and applied to a specific SharePoint site, the label does not do anything. The policy has to be applied to a document library in that site – but even then it wouldn’t do anything until the label was selected from a drop down list.

The screenshot below shows how the specific label is selected on the library and whether it should be applied to all existing items.

The label can now be seen in the ‘Retention Label’ column (when made visible in the view settings).

Anyone who uses the library to add and edit content (site members) will notice an obvious change if they try to delete anything, including from their synced document libraries in File Explorer.

They can’t.

There is, however, a workaround that end-users can use to delete documents with a label: (a) check the box next to the document, (b) open the information panel and (c) remove the label on individual documents.

Once the label is removed it no longer appears against the record.

The document can now be deleted. It will move to the Recycle Bin, from where it could be restored for up to 93 days if a mistake was made.

The lessons to be learned from this are that:

• It might be useful to make the library read only after the label is applied.
• The label should only be applied when the library became inactive, to allow people to get on with their work while it is still active.

If anyone was concerned about content being deleted from an active library, they could set an alert on the library.

In the meantime, for all the other records that retained their label, when the retention period ended the records in the library would be transferred to the Recycle Bin where they would sit for 93 days before complete and permanent deletion.

If no-one restored a document from the Recycle Bin, the content would be deleted, permanently, forever, without any record. The records managers might need to make a copy of the metadata for the documents to be deleted before they are deleted.

Alternatively, if they had selected the option for a ‘disposition review’ when the label was created, this would alert the records managers (and others) of the need to review the contents of the library.

Once everything had been deleted, all that will be left will be an empty library with no record of what used to be there.

Retention policy without a label

Non-label retention policies are created from the ‘Retention’ tab of the Information Governance portal.

For the purpose of this post, this label will be named ‘Label Two – Retain 5 years then destroy’.

This policy has the same retention period, trigger, and action as Label One – 5 years after date modified – and then the content could be deleted without any record of this being kept. [Note – yes the screenshot below shows 7 years].

As the policy was being created it was applied directly to a SharePoint site.

The policy could start working immediately.

Nobody, apart from the person who created and applied it knew it had been applied. It didn’t appear anywhere on the SharePoint site where it had been applied.

But, as part of the application process, it created a Preservation Hold (PH) library visible only to Admins, including the Site Collection Admins but not the Site Owners.

(Note – if end-users are allowed to create SharePoint sites, or Teams in MS Teams that also creates a SharePoint site, they are made Site Collection Admins by default and can accordingly see the Preservation Hold Library).

Anything that was deleted in advance of the retention policy expiry would be moved to the Preservation Hold library.

The PH library had a specific purpose – to ensure that everything on the site was retained until the end of the retention period.

It was not possible to restore any items from the Preservation Hold library. It was also not possible to ‘clear’ that library. If an administrator tried, they would be advised that ‘something went wrong’.

The only way to delete from the Preservation Hold library would be to remove the policy.

However, if a record was deleted it would be placed in the Recycle Bin for up to 93 days, from where it could be restored. If a record was restored in this way (from the Recycle Bin), the copy in the Preservation Hold library would remain.

For most people this type of policy was fine. They could (appear to) delete records, unaware that nothing was actually deleted if it were subject to a retention policy.

The same thing happened if a similar type of retention policy was applied to OneDrive accounts. End-users could ‘delete’, but nothing was actually deleted as long as the retention policy was active.

At the end of the retention period, the content that was subject to this policy was transferred to the Recycle Bin where it remained for 93 days. It would then be deleted, permanently, forever, without any record.

All that was left would be an empty library with no record of its previous content. The only clue would be in the ID or the ID part of the Document ID (if enabled). If anything new was added to the library, the next number in sequence would indicate how many records had previously been saved to that library.

Lessons to be learned

There are several lessons to be learned.

The first is that you need a plan to create and apply retention labels or retention policies to content across Microsoft 365. The configuration options and implications of each option need to be understood before they are applied.

You need to understand the difference between (a) a retention label published as a policy and applied to a SharePoint site and then a library on that site, and (b) a retention policy applied to an entire SharePoint site.

Retention labels can be mapped to retention schedule/disposal authority classes and then applied to specific libraries on SharePoint sites where the policy is applied. Accordingly, it may be useful to create document libraries that map to those classes, and only store content in those libraries that is covered by a single retention class.

If single SharePoint document libraries are used to store content that maps to different retention classes, then it will become very complicated to accurately map retention labels to content.

In many cases, it may be possible to apply a single non-label retention policy to a single SharePoint site or sites. For example, all project sites might have the same 7 year retention applied to all the content on the site as there may be no real need to apply policies to individual libraries.

Whatever you do, have a plan to act, document your settings, and keep it up to date.

And understand what happens at the end of the retention period. There is no back up.

What to use when – metadata, content types, folders or document sets

Most end-users will be familiar with using folders to ‘categorise’ content in SharePoint document libraries, but not everyone will be familiar with other options including document sets or the use of metadata or content types to achieve similar ‘grouping’ outcomes.

Any (and all of) these options can be used to categorise content in a document library. But, aside from the default folder option, some thought should be given to the actual purpose and the end-user experience for the other three options.

This post provides some suggestions for what to use when.

In summary:

• Use folders most of the time, but try to limit them (through end-user communication) to no more than two levels. Any more and it may be better to create a new library.
• Use document sets where there is a (genuine) need to group documents with additional metadata and security controls. In some cases, metadata may be a better option.
• Use content types (including document sets) sparingly and for specific business requirements, not just to categorise document types in a single library when a metadata choice, naming conventions, or separate libraries may provide a better outcome.
• Use metadata columns when it makes sense to display content grouped and filtered by that metadata.

Document libraries are logical aggregations for records

As a starting point, keep in mind that SharePoint document libraries are logical aggregations or ‘containers’ for records.

When a SharePoint site is mapped to a business function, the libraries in that site can be mapped to or named after the activities that produce records.

It is generally not a good idea, from a recordkeeping point of view, to mix records created by different business activities (which is not uncommon in the default ‘Documents’ SharePoint library, especially the site attached to a Team in MS Teams).

Access controls and records retention are much harder to manage in single libraries that contain records created from multiple different business activities.

As the library is the logical container, any option use to categorize or break up content in a library – whether it be metadata, content types, folders or documents sets – is similar to a divider in a lever-arch folder. Ideally, the entire content of the document library should contain records about the same subject.

Metadata

Content in a document library can be categorized using metadata.

SharePoint sites have more or less unlimited metadata that can be applied to content in document libraries. All document libraries include the default ‘Name’ and ‘Title’ metadata fields as well as system generated metadata.

SharePoint also includes the Managed Metadata Service (MMS) that allows for the creation of a centralised set of hierarchical metadata terms, based on taxonomies and thesauri, accessible to all sites.

Microsoft announced in April 2020 that it was modernizing the SharePoint Managed Metadata Service (MMS) and that the updates ‘will be essential to delivering new premium value to Project Cortex later this year’ (2020).

To add metadata to a document library, it must be either (a) added via a content type that contains one or more site columns, or (b) created as a ‘local’ library column. Both of these options can link to the MMS.

Once the metadata has been added, documents may be grouped in a view by the metadata. In the example below, they are grouped by the choice metadata column ‘Document Type’.

The benefits with using metadata include:

• Ability to draw on the global terms defined in the MMS.
• Ability to group and filter content, especially useful in libraries with a lot of content. Note that certain columns, including those that use MMS terms, cannot be grouped.
• Supports multiple pre-defined views that can replace search (especially useful since ‘faceted’ searching was deprecated).
• Support for recordkeeping requirements.
• Support for search.

The negatives with using metadata relate to usability and syncing.

• End-users generally don’t like to add more than two items of metadata when they upload a document.
• End-users generally don’t ‘get’ categorization or grouping by metadata (but generally they don’t seem to mind pre-defined views).
• Metadata columns are not visible in libraries that are synced to File Explorer.
• If a metadata column is mandatory, the synced library will become read only.
• Global metadata defined in the MMS may not suit all parts of the business. We found that even a simple set of terms to define ‘Document Type’ (e.g., ‘Meetings’, ‘Agenda’) varied across the organisation.

Suggested use case:

• Use metadata columns to categorise records (and the MMS) only when it meets a specific business need, not by default. For example, a document library for organisational policies that lists each policy (or group of policies and procedures) via the ‘Policy Name’ metadata field. The metadata can be used to describe the policies and related documents, categorize them in multiple ways. End users are unlikely to need to sync this library. This method also assists with cross-referencing related policies including by using links.

Content Types

Content types are a fundamental building block in SharePoint, and are used to define content.

SharePoint has long included the concept of a ‘Content Type Hub’ where, in theory, all Content Types are centralised. This will be replaced in the near future with the new ‘Content Types Gallery’ option accessible from the SharePoint Admin portal.

All content types derive from the top level ‘System’ content type that is used to define the ‘Item’ content type which in turn defines the ‘Document’ content type.

Every SharePoint document library includes (a) the default ‘Document’ content type with two metadata columns ‘Title’ and ‘Name’ and (b) the default ‘Folder’ content type. They also include the ‘Document Set’ content type when this is enabled as a site feature.

New content types are created from the Site Settings area of every SharePoint site but must be added to a library by first allowing content types (from Library Settings – Advanced) and then adding them from ‘Existing Site Content Types’. This process includes document sets which are discussed below.

Given their direct relationship with metadata, there may be a tendency to want to create a lot of content types whenever additional metadata is required and to apply multiple custom content types to a single document library to define a type of content when other options, such as a metadata choice or even folders, may work better.

In the screenshot below, the library column shows the Content Type for the particular object. This column can be grouped in a view, providing a way to show different content types in the library.

The benefits with using content types are as follows:

• Help to define content through the use of metadata.
• May include a template document which can use the metadata applied to the item. For example, automatically adding the metadata in the body of a Word document as a content control.

The negatives with using content types are as follows:

• There can be too many content types.
• End users don’t like having to choose a content type, especially if the only apparent purpose is to define a document type (e.g., ‘Meetings’ vs ‘Agenda’). In this case, a simple metadata column would be sufficient OR use document naming conventions instead.
• End users don’t like adding metadata after having to choose a content type.
• If any column in the content type is mandatory, it will cause the synced library to become read-only.

Suggested use case:

• A document library that is used to support a specific business purpose where it is necessary to define one or more documents using additional metadata, and a pre-defined document template is required. For example, a document library used to store client agreements based on a template. For each client agreement, there is a requirement to capture certain metadata (client name, address, phone, etc) which will then be auto-populated into the template agreement document.

Document Sets

Document sets are a like a super folder in that they behave like normal folders but have a lot more functionality.

As noted above, document sets must be enabled as a feature before they become visible in the list of available content types. Once enabled, and the option to ‘Allow management of content types’ is enabled in the Library Settings (under ‘Advanced’), they can be added to a document library from the list of available content types.

Document sets can be very useful where there is a need for (a) a unique identifier for a folder inside a library, (b) additional metadata to help define a group of documents and (c) access controls that may be different from the library. (Note – this last option is also possible with folders, see below).

While document sets may sound appealing to business areas that want to ‘hard categorize’ content in a document library (for example, every document set contains a policy, and related procedures, forms and so on), this can actually result in making it harder for end-users to find information. The reverse of this is that they can be very useful if there is a genuine requirement to restrict access to information.

Document sets are visible in the screenshot above regarding content types. When added to a document library, they are visible from the ‘New’ menu as shown below:

The positives of document sets include the following:

• They can have a unique ID (based on the Document ID being enabled in the library); the Document ID is the same used for the documents.
• Metadata can be added automatically to the body of template documents (via content controls) stored in document content types linked with the document set.
• They can be used to restrict access.

The negatives of document sets include the following:

• Can make it harder to find information, especially if there are more than around 30 in a library.
• Searches may retrieve only one document. It will also not retrieve the document set. There may be other related documents.
• Can be more difficult to replace with other options because of the relationship between document sets and linked document content types.
• In synced file libraries, document sets appear as folders. For this reason it is recommended that any library using document sets includes at least one mandatory column to make the synced version read-only. Ideally, end-users should not need to sync these libraries.

Suggested use cases:

• Document sets are a good way to group client-related or employee documents in a single library, especially when there is a need to restrict access between the document sets. In this sense, document sets work like ‘sub-files’ in a library. For example, Manager ‘Jim’ may need to see five document sets, while Manager ‘Mary’ may need to see eight others. Neither will be able to see the others files when they access the library.
• In smaller organisations, a SharePoint site might include a library called ‘Projects’, with one document set per project. Larger organisations might have a library per project, even larger organisations might have a single SharePoint site (perhaps based on a Microsoft 365 Group) per project.

Folders

Pretty much everyone knows what folders are and how they are used. Every SharePoint document library includes the option to create folders. When synced to File Explorer, end users can create folders there too.

It is possible to prevent the creation of folders in SharePoint via the Library Settings – Advanced – Folders option but this option should really only be enforced in exceptional circumstances.

It is also possible to view all the content in a library without folders by editing the view (or creating a new ‘No folders’ view) and changing the option in the ‘Folders’ section from ‘Display items in folders’ to ‘Show all items without folders’. It is not uncommon to find, when the no-folder view is displayed, to see fewer documents than folders. It is sometimes a good way for admins to demonstrate the benefit of using metadata columns instead.

The positives of using folders include the following:

• Everyone knows how to use them.
• They can be created from File Explorer.

The negatives of using folders including the following:

• Too many folders and levels.
• Folders that repeat other previous words in the URL path, including the name of the site and the library.
• One word folders – for example: Meetings – 2020 – June – 8.
• Folders with unique permissions.

Suggested use case:

• Folders are best for simple document libraries. It is a good idea to let end users know about the 400 character limit and suggest they might consider creating new libraries if the folders are clearly about entirely different subjects (unless this is very low level). For example, don’t mix records about meetings with records about policies or staff rosters all in the same library.
• Note: Try to avoid using unique permissions on folders. If unique permissions are really required, create a separate library.

Using all four (or a mix of the four) options

It is possible to use all four options in the same library:

• Folders
• Additional metadata colums, which may be used to group content in views without folders
• Added content types, including document sets with added metadata.

But it’s probably not a good idea to do this.

Keep it simple

If you are trying to categorise records in a way that makes some sense, it is not a bad idea to keep it simple and only deploy the option or options that best meets the business needs.

• Folders are best for most end users interacting with a library including via File Explorer. Limit the use of extra metadata – for example ‘Document Type’, keeping in mind that added metadata is not (yet) visible in libraries synced to File Explorer.
• Use metadata to create read-only views of records or other content stored in a library (for example, policies), and a small group of people is responsible for that metadata. Using metadata to group content in this way can be better than using folders or document sets. Keep in mind that not all columns (including MMS columns) can be grouped in views but they can be filtered.
• Use custom content types including document sets in situations where there is a need to add specific metadata to records (and use it in templates), or group them in ‘sub-files’ in a document library. Folders can be used in document sets to categorise content further.

Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

There are two types of chat message in MS Chat:

• One-to-one/one-to-many ‘chat’ (top icon above).
• Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

• Knowing if chat messages on any given subject exist. 
• Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

Searches can be configured to find content in any or all of the following locations:

• Users, Groups, Teams
  • Exchange email
  • Office 365 group email
  • Skype for Business
  • Teams messages [the copy in the mailbox]
  • To-Do
  • Sway
  • Forms
• SharePoint
  • SharePoint sites
  • OneDrive accounts
  • Office 365 group sites
  • Teams sites
• Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

• As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
• There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

Compliance

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search. But what happens if there is no Exchange Online mailbox? It seems the chats become […]

October 22, 2020

Electronic records

A modern way to manage the retention of digital records

In his April 2007 article titled ‘Useful Void: The Art of Forgetting in the Age of Ubiquitous Computing’ (Harvard University RWP07-022), Viktor Mayer-Schönberger noted that the default human behaviour for millenia was to forget. Only information that needed to be kept would be retained. He noted that the digital world had changed the default to […]

October 14, 2020

Electronic records

The Microsoft 365 experience – Teams, Exchange, Outlook, Edge: Where did SharePoint Go?

At the 2020 Microsoft Ignite conference, Jeff Teper presented a diagram titled ‘Microsoft 365’. The diagram showed only four icons: Teams, Outlook, Office and Edge. The implication of this diagram was that, for most end-users, Teams is now (or will become) their primary portal into Microsoft 365. As stated by Jeff Teper, SharePoint is a […]

September 28, 2020

Applying multiple retention policies to a SharePoint Online site

Many organisations have complex records retention requirements that are described in records retention schedules, disposal authorities or records authorities. For example:

• There may be different ‘levels’ of retention depending on the ‘state’ of a record. The final versions of certain records may have a longer retention requirement than the working versions.
• For each business function there may be multiple types of records, each with their own retention requirement or ‘class’.
• In some disposal authorities based on business functions, activities that produce records (for example ‘Meetings’) may appear in multiple functions with the same retention requirement.

This post describes multiple and different types of Microsoft 365 retention policies created with an E3 licence in the Information Governance section of the Compliance admin portal can be applied to a single SharePoint site.

Example retention schedules/disposal authorities

Most records retention schedules or disposal authorities list types (or ‘classes’) of records that are created or captured by the organisation, including through the completion of various activities or transactions, and define how long these records must be kept or retained by the organisation (or transferred to an archival institution).

These record types or classes are usually grouped, by business subject or function.

The following extract, from a private sector company records retention schedule, shows records grouped by subject type (‘Company records’).

In the example below, from the Victorian (Australia) government, records are grouped by function (‘Enquiries and Complaints’).

The diagram below presents a simple view of the examples above. For every subject type or business function, there may be one or more records description (based on the activity or transaction that creates or captures the record) with a corresponding retention period.

How does SharePoint manage records?

SharePoint Online team sites (including the sites linked with Microsoft 365 Groups and MS Teams) may be created to manage the records for a particular business area or function, or for a specific business activity.

Whether a single or multiple document libraries are used, SharePoint sites may contain a mix of record content. It may not always be possible to apply a single retention policy to the site.

Use case

For the purpose of this post, we will assume that the organisation has a business function named ‘Client Services’ – a generic name for a business unit that delivers client services.

The Client Services area has several SharePoint sites. One of these sites is named ‘Client Services’.

The ‘Client Services’ site, which has been active for several years, has multiple libraries for the activities it performs, including ‘Meetings’, ‘Procedures’, ‘Working papers’, ‘Rosters’, ‘Marketing’ and so on. Most of these libraries are created annually and consequently the year is added to the library name to help group content more efficiently – for example, ‘Meetings 2018’, ‘Meetings 2019’.

The organisation’s records retention authority has multiple classes for the Client Services function, including:

• Marketing – Retain for five years
• Meetings – Retain for seven years
• Procedures – Retain for seven years.
• Rosters – Retain for ten years

There is no class for general ‘working papers’ that may be created in support of the above activities, but the organisation would like to ensure that all content not otherwise covered by one of the ‘explicit’ retention policies above is retained by an ‘implicit’ or background policy.

Creating the Office 365 retention policy

Based on its requirements, the organisation will require two different options.

• A single retention policy with a minimum three year retention for content (including ‘working papers’) not covered by any other longer retention period. This will be created as an ‘implicit’ or background policy and applied to the site. Any content that is deleted by the end users will be moved to the invisible (to end users) Preservation Hold library. Records covered by this policy will be automatically deleted – via the Recycle Bin – at the end of the retention period.
• Multiple retention labels published in a single retention policy, that is applied on this site or other sites that can be mapped to the same function. This means that, when applied to a document library, every one of the labels will appear in the drop down menu in the library settings to apply a label. Depending on how the label has been configured, the records may be automatically deleted or subject to a disposition review.

Label-based retention policies – retention settings

Each retention label that is created will include a name and description, and then the label retention settings.

• How long it is to be kept (e.g., 7 years).
• What happens at the end of that period (delete automatically, disposition review, nothing).
• Trigger for disposal – date created, date modified, date labeled. The ‘Date labelled’ option is preferred as it will not prevent day-to-day actions on the library or make the synced version read-only.

This process is repeated for each label. Each label can include the ‘File Plan’ settings, for example any reference numbers, the Function and Activity, and so on.

Here are two of the labels that have been created:

Publishing the labels

After each label has been created, they can then be published together in a single (‘Client Services) retention policy that is applied to the site (Client Services).

The published policy now appears in the list of label-based retention policies. It also appears under the ‘Retention’ tab of the Information Governance section, along with all other published label-based policies and policies that are not based on policies.

Non-label retention policy

The ‘implicit’ or background policy is created directly as a retention policy, without the need for a label. This policy, named ‘Temporary records’, has a three-year retention. It is applied directly to the site (or multiple sites).

Applying the label-based policies to the site

The Client Services site has several libraries as shown below.

We want to apply the label-based policies to the libraries named ‘Meetings 2020’, ‘Rosters 2020’ and ‘Marketing’. The general ‘Documents’ library will be covered by the implicit retention policy for ‘Temporary records’.

To apply the label-based policies to the library, click on the library and navigate to Library Settings where the option to ‘Apply label to items in this list or library’ is found.

A drop down list shows all available label-based policies. As the ‘Client Services’ policy was only applied to this site, only those labels appear. Only one option can be selected for each library.

It is usually a good idea to check the box (hidden behind the list of policies in the screenshot below) to ensure that anything already stored in the library will be covered by the policy.

The Meetings 2020 library has now been assigned the Client Services Meetings – 7 years policy. As soon as this label as been applied:

• It will no longer be possible to delete any content.
• If the library has been synced to File Explorer, the library in File Explorer will become read only.

The only way to to remove this restriction is to remove the policy. Accordingly, it may be better to apply the label only when the library has become inactive.

Note – The Temporary records implicit policy will continue to operate in the background and will apply to any content in any library or list not covered by an explicit policy. Anything deleted will be moved to the Preservation Hold Library accessible only by the Site Collection Admins or higher.

The final model can be visualised as follows:

 

The longest retention option will always take precedence. So, if an explicit label-based policy has a retention period of 2 years, and the background implicit retention policy has a retention of 5 years, the content will be kept for 5 years.

Note also that only the content of the libraries or lists is deleted at the end of the retention period. The library or list – and the site – remain.

Conclusion

As described in this post, it is possible to create multiple retention policies and apply them only to a single SharePoint site.

This allow organisations to create targeted groups of retention policies which is likely to be useful in organisations with detailed or function/activity based retention schedules.

Planning is required to ensure that there is appropriate and effective retention coverage for all the content created and captured in all SharePoint sites.

Planning for records retention in Office 365

Office 365 is sometimes referred to as an ‘ecosystem’. In theory this means that records could be stored anywhere across that ecosystem.

Unlike the ‘old’ on-premise world of standalone servers for each Microsoft application (Exchange, SharePoint, Skype) – and where specific retention policies could apply (including the Exchange Messaging Records Management MRM policy), the various elements that make up Office 365 are interconnected.

The most obvious example of this interconnectivity is Microsoft Teams which stores chat content in Exchange and provides access to content stored in both SharePoint (primarily the SharePoint site of the linked Office 365 Group) and OneDrive, and has links to other elements such as Planner.

Records continue to be created and kept in the various applications but retention policies are set centrally and can apply to any or all of the content across the ecosystem.

Managing records in Office 365, and applying retention rules to those records, requires an understanding of at least the key parts of the ecosystem – Exchange, Teams, SharePoint and OneDrive and how they interrelate, and from there establishing a plan for the implementation of retention.

What types of records are created in Office 365?

Records are defined as ‘evidence of business activity’ and are often associated with some form of metadata.

Evidence of business activity is an overarching term that can include:

• Emails
• Calendars
• Documents and notebooks (in the sense of text on a page)
• Plans, including both project plans and architectural plans and diagrams
• Images/photographs and video
• Chat and/or messages
• Conversations (audio and/or video based)
• Social media posts

All digital records contain some form of metadata, usually displayed as ‘Properties’.

Where are the records stored in Office 365?

Most records created organisations using Office 365 are likely to be created or stored in the following parts of the ecosystem:

• Exchange/Outlook – for emails and calendars.
• SharePoint and OneDrive – for documents and notebooks (in the sense of text on a page), plans, images/photographs and video.
• Stream – for audio and video recordings.
• MS Teams – for chat and/or messages, conversations (audio and/or video based). Note that 1:1 chats are stored in a hidden folder of the Exchange mailbox of the end-user/s participating in the chat, while Teams channel chat is stored in a hidden folder of the linked Office 365 Group mailbox.
• Yammer – for (internal) social media posts.

It is also possible to import and archive certain external content such as Twitter tweets and Facebook content in Office 365.

The diagram below provides a overview of the main Office 365 applications and locations where records are created or stored. Under SharePoint, the term ‘Sites’ refers to all types of SharePoint sites, including those associated with Office 365 Groups. Libraries are shown separately because of the potential to apply a retention policy to a library – see below.

Note also that this diagram does not include network file shares (NFS) as the assumption is made that (a) NFS content will be migrated to SharePoint and the NFS made read only, and (b) all new content that would previously have been stored on the NFS is instead saved either to OneDrive for Business (for ‘personal’ or working documents) or SharePoint only.

Creating a plan to manage records retention across Office 365

In previous posts I have recommended that organisations implementing Office 365 have the following:

• A basic architecture design model for SharePoint sites, including SharePoint sites linked with Office 365 Groups (and Teams in MS Teams).
• A plan for creating and applying retention policies across the ecosystem.

Because SharePoint is the most likely location for records to be stored (aside from Exchange mailboxes and OneDrive accounts), there should be at least one retention policy for every SharePoint site (or group of sites), as well as policies for specific document libraries if the retention for the content in those libraries may be different from the retention on the overall site.

For example, a ‘Management’ site may contain a range of general content as well as specific content that needs to be retained for longer. 

• The site can be covered by a single implicit retention policy of (say) 7 years. This policy will delete content in the background, based on date created or data modified. 
• The document library where specific types of records with longer or different retention requirements are stored may have one or more explicit label-based policies applied to those libraries. This content will be retained while the rest of the site content is deleted via the first policy.

Structure of a retention plan for records in Office 365

A basic plan for creating and applying retention policies might look something like the following:

• User mailboxes – one ‘general’ (implicit) retention policy for all mailboxes (say, 7 years after creation) and another more specific retention policy for specific mailboxes that require longer retention.
• SharePoint sites – multiple (implicit) retention policies targeting one or more sites.
• SharePoint libraries – multiple (explicit) label-based retention policies that are applied manually. These policies will usually a retention policy that is longer than any implicit retention policy as any implicit site policy will prevent the deletion of content before it reaches the end of that retention period.
• Office 365 Groups (includes the associated mailbox and SharePoint site) – one ‘general’ (implicit) retention policy. See also below.
• Teams channel chat – one ‘general’ (implicit) retention policy. Note that this content is stored in a special folder of the Office 365 Group mailbox.
• 1:1 chat – one ‘general’ (implicit) retention policy. This content is stored in a special folder of the participant mailboxes.
• OneDrive documents – one ‘general’ (implicit) retention policy for all ODfB accounts, plus the configuration of retention after the account is inactive.

At a high level, the retention policy plan might look something like the following – ‘implicit’ policies are shown in yellow, SharePoint document libraries may be subject to ‘explicit’, label-based policies. The ‘+7 years’ for OneDrive relates to inactive accounts, a setting set in the OneDrive Admin portal.

Regarding Microsoft Office 365 Groups, Microsoft notes the following on this page about managing retention in Office 365:

To retain content for a Microsoft 365 group, you need to use the Microsoft 365 groups location. Even though an Microsoft 365 group has an Exchange mailbox, a retention policy that includes the entire Exchange location won’t include content in Microsoft 365 group mailboxes. A retention policy applied to an Microsoft 365 group includes both the group mailbox and site. A retention policy applied to an Microsoft 365 group protects the resources created by an Microsoft 365 group, which would include Microsoft Teams.

The actual plan should contain more detail and included as part of other recordkeeping documentation (perhaps stored on a ‘Records Management’ SharePoint site). The plan should include details about (a) where the policies have been applied and (b) the expected outcomes or actions for the policies, including automatic deletion or disposition review (for document libraries).

Keep in mind that, unless the organisation decides to acquire this option, there is no default backup for content in Office 365 – once a record had been deleted, it is gone forever and there may be no record of this beyond 90 days.

What happens when you create a Team in MS Teams

On 27 March 2020 I asked, via Twitter, whether organisations that rolled out MS Teams will wonder in the future who created all the random (and randomly-named) SharePoint sites.

The reason for this question was because many organisations, scrambling to establish ways for staff to work from home, decided to make use of MS Teams in their (often newly implemented) Office 365 suite of apps.

I have seen multiple organisations since late 2019 ask ‘who created all those SharePoint sites?’ when they reviewed the list. The current COVID-19 work-from-home situation will only make this situation ‘worse’ and, without effective oversight or controls, result in the creation of multiple uncontrolled SharePoint sites.

Unlike other products like Zoom, Whatsapp, Facetime and Skype, however, MS Teams is not a standalone product, but a core element in the Microsoft Office 365 ecosystem.

The key point is this – every Team in MS Teams has a linked SharePoint site (and an Exchange mailbox, where all the chat content is stored). You can’t disable these options.

What happens if you create a Team in MS Teams?

The good thing about the one-to-one chat element of MS Teams is that it’s relatively intuitive and easy to use, including on the mobile app. You only need to tell users it’s like Skype or Whatsapp, but for internal user only, and most pick it up quickly.

The Teams part of MS Teams is not quite as intuitive, but early adopters generally understand the basic concepts – that a Team has members, and you can have multiple chat channels for each Team.

Once end-users understand how a Team works (and this can take some time because one-to-one chat can include multiple people), they might notice this option at the bottom left of the app:

Creating a new team sounds like a great idea, so end-users may try:

My guess is that end-users are more likely to want to ‘build a team from scratch’ as shown below, because the second option doesn’t really make sense.

There is a good chance they will want the Team to be ‘Private’, although may not fully understand what this means. A Public Team sounds like a Yammer Group (or Community).

So far, so good, the end-user can give the Team any name they like:

At the bottom of the naming screen is the option to ‘Create’. The end-user is then invited to add members to their new Team. This seems a fairly obvious step, and they can add whoever they want. New members are by default ‘Members’ but they can be changed to ‘Owners’ if necessary. There is no control over this process.

The new team now appears on the left-hand menu of MS Teams:

The new team opens at the default ‘General’ channel.

On the main part of the Team, the following options are offered:

• Along the top, ‘Posts’, ‘Files’, ‘Wiki’ and a + to add more applications. (Hint – the ‘Files’ option points to the SharePoint site that has been created behind the scenes).
• Across the middle, three options to ‘Add more people’, ‘Create more channels’ and ‘Open the FAQ’
• At the bottom, the option to ‘Start a new conversation’ with various other options including the ‘Meet now’ video option.

The end-user can now get on with chatting, sharing files, and adding apps to do other things.

But what else has happened?

As noted above, the ‘Files’ tab in the General channel gives a clue to the existence of the connected SharePoint site. End-users may not care terribly much about this, for them it provides the option to create, upload, share and collaborate on files.

A new Office 365 Group is created

But before we get to the SharePoint site, it’s important to understand the one-to-one relationship between a Team in MS Teams and an Office 365 Group. If you do not know what an Office 365 Group is, please read this Microsoft guidance on Office 365 Groups.

In very simple terms:

• Every new Team in MS Teams creates a new Office 365 Group.
• The Owner of the Office 365 Group is the Owner of the team; the members of the Group are the Members of the team, as added by the person who created the Team.

The new Office 365 Group appears in the list of Groups in the Office 365 Admin portal, as shown below. Access to this part of the Admin portal is normally restricted to Global Admins (who would normally be responsible for creating other types of AD Groups, such as Security Groups and Distribution Lists.

A new Exchange mailbox has been created

Note that the process has also created an Exchange mailbox with a Group email address. The new Exchange mailbox will now appear in the Outlook client of everyone in the Team – something they are unlikely to notice.

As noted above, all the chat messages in the Team are stored in a hidden folder in the Exchange mailbox for the Team.

A new SharePoint site has been created

If we go across to the SharePoint Admin portal, which is normally restricted to Global Admins and SharePoint Admins, we can see that a new SharePoint site has been created, and is owned by the ‘Group owners’.

The SharePoint Admin has had no involvement in the creation, naming, or structure of this new site. And, just to add another factor, the SharePoint Admin cannot access the site – see below.

The Team owner may not realise it, but they now have a SharePoint site. The new site’s ‘Documents’ library appears in the ‘Files’ tab as shown below.

And, just to add a confusing element, the site includes the invitation (at the bottom left) to create a new Team!

As noted above the SharePoint Admin can ‘see’ that this site exists in the list of sites but cannot actually access it. The Global Admin, on the other hand, can access it.

So the person responsible for managing SharePoint across the organisation cannot access the SharePoint site, which is not a good thing from an information governance point of view.

The reason they cannot access the site is because they were not added to the Site Collection Admin Group when the site was created. And, just to make it a bit more confusing, the ‘Users and Permissions’ section of Site Settings, where the ‘Site collection administrators’ section is found (see screenshot below), does not appear in Office 365 Group-based SharePoint sites.

So, how does the SharePoint Admin get access to this site to configure and manage it? There are two ways:

• The Global Admin can go to /_layouts/15/mngsiteadmin.aspx (after the site name URL) and add them (or a Security Group with them in it) there.
• The SharePoint Admin can click on the site details in the SharePoint admin portal and add him/herself as an Owner. This puts them in the Site Collection Admin section along with the Group Owner.

Summary

This post began with a simple question – if organisations allow end-users to create Teams to work from home, how will they manage all the SharePoint sites that are created through the process described above?

There is no one answer to this question but it’s worth understanding exactly what happens – and what else is created (including Planner) – when a Team is created. Organisations seem to go one of two ways:

• Let end users create Teams and deal with the consequences later, including attempts at auto-classification and retention policy application across the various elements of the new Office 365 Group – mailbox, SharePoint site, Team chat. This is the Microsoft default and the preference of many organisations that are don’t have compliance issues or can accept the risks of uncontrolled information stores.
• Control the creation of Teams, but make any controlled process as easy as possible for end-users to keep them working quickly, and manage the content in mailboxes, SharePoint and Teams proactively. While not the preferred option, it will help with the management of corporate information down the track.