Teams chats and channel posts are a bit of a problem to manage as records because it is not easy to capture them – either individually or as a thread. The most common solution is to screenshot them but this is hardly ideal.
End-users may be able to delete or edit sent chats and posts. What happens to these? How can you know if a sent message was deleted or edited? Can you see the version history for edited messages?
In addition to the global settings, Team Owners may also modify the same settings for a Team only if the above options are enabled:
Editing sent chat messages
Here is an example of a sent chat message. We can see from the little ‘eye’ icon on the right of the sent message that the recipient has seen it.
The image below is the same message that has now been edited. The time stamp doesn’t change and no version history is available to the sender or recipient (but see below). The person who received it may not necessarily notice that the text was edited (who goes back to read messages?).
Editing sent channel posts
The following is an example channel post. Only the author of a channel post can edit it.
If the message is edited, this is indicated as shown but the time stamp stays the same. Again, no version history.
How to see the version history for chats and posts
A ‘compliance copy’ of all Teams chats and channel posts is stored in Exchange Online mailboxes until they are deleted. If there is no retention policy set for these chats and posts, deleted messages are (permanently) deleted relatively quickly. If there is a retention policy, they remain in the mailbox until the end of the retention period and are then subject to whatever action is set on the policy (e.g., destroy, do nothing).
The only way to check the previous versions of a chat or post is to create a Content Search case in the Compliance admin center. However, access to Content Search is restricted because it allows the person with access to search across all emails, Teams chats and posts, OneDrive and SharePoint.
Consequently, content searches for Teams chats/posts are likely to be only carried out for specific reasons – for example, threatening or harassing messages that were then edited.
The image below shows the results of a content search for myself as the author. The search returned two results for the edited channel post – one for the original and one for the edited message. Every time the message is edited, another copy will be retained. Note that the visible timestamp hasn’t changed.
When the messages are exported (as a .eml file), the only difference between the two is the ‘CorrelationVector‘, ‘a format and protocol standard for tracing and correlation of events through a distributed system based on a light weight vector clock’.
Original message correlation vector: “phE37FG2rU289DE7MmZndA.18.104.22.1684298459.1.0”}
Everything else (apart from the edited text) is identical.
What happens when messages are deleted?
Microsoft’s guidance on Teams messaging policies dated 25 November 2021 appears to show no difference between deleting chat and deleting sent messages.
If deleting is allowed, what happens to these deleted chats or messages depends on whether a retention policy has been applied. Either way, if the chat or post has been deleted, it will be indicated as follows:
When a chat or message is deleted:
If there is no retention policy, the messages are completely deleted within a day or a few days and cannot be found after that via a content search.
If a retention policy has been applied to chats and posts and it hasn’t expired, the messages can be found via a content search.
If a retention policy that applied to these messages has expired, then the messages will be deleted within a few days unless the disposal action is ‘do nothing’ or ‘keep forever’.
Implications for records
Organisations need to decide if Teams chats/posts may be records and, if they are, how they are to be captured especially given that a ‘compliance’ copy is stored in Exchange mailboxes until they are deleted.
At least four approaches are possible:
Allow end-users to delete Teams chats/posts whenever they like, with no retention applied. Chats or posts that are not deleted will simply remain until the Team is disposed of. This approach has potential implications if chats/posts contain harassing or offensive content.
Set a relatively short-term retention period for both (e.g., 30 days or 3 months), on the basis that chats/posts are not official records – but end-users are encouraged to capture any that could be records, probably as screenshots stored in SharePoint (or somewhere else). This approach provides the option to recover a message within a reasonable period of time.
For Teams 1:1 chats (that are stored in the personal mailboxes of participants), set the same retention period as the Exchange mailbox.
For Teams channel posts (that are stored in the mailbox of the associated Microsoft 365 Group), set the same retention period as the SharePoint content.
Managing Teams chats and posts as records is not a simple process. Organisations should consider the implications of allowing end-users to delete or edit sent chats or posts and set retention periods accordingly.
Addendum – there are a number of third-party products that can capture Teams chats and posts. However, one of the biggest problems with any type of chat is that a related thread may contain several messages. It may actually be easier to focus on applying retention to the chats and posts stored in the mailboxes.)
Folders provide a simple way to group content on network file shares. But, while they may look the same in SharePoint (or in a synced library in Windows File Explorer), and are the default way to group content in Teams-linked SharePoint sites, they are not always the best way to aggregate let alone manage digital records, especially high value records or records that require unique access controls and permanent retention.
This post explains why.
What is an aggregation?
According to the international standard ISO 16175:2020, Part 1, ‘Functional requirements and associated guidance for any applications that manage digital records’, aggregations:
Are accumulations of related digital records.
May reflect relationships such as shared characteristics or attributes, or the existence of sequential relationships between records.
Can collectively constitute a narrative of events in which the records may have a sequential relationship with each other, determined through the metadata associated with the records.
May be formal, structured relationships or may exist as less formalised (but) tightly bound metadata relationships that establish links between related records.
(Section 7.5.3, paraphrased)
On the face of it, the definition doesn’t seem to prevent the use of folders to aggregate records. But the standard becomes more specific. It must be possible to:
Enable the capture and maintenance of metadata for both records and the aggregation. (R1.2.1)
Associate records at individual object and/or aggregation level to their business context. (R1.3.1)
Allocate an appropriate retention and disposition period for every record and record aggregation. (2.1.1)
Support the migration or export of aggregations. (R2.2.1)
Ensure that the content of records, and their metadata, can be fixed or protected from unauthorised access, alteration or destruction. (R3.1.1, R3.2.1, R4.2.1 – 2)
There are several other highly desirable requirements, but the mandatory requirements above should be sufficient to make the point that folders aren’t the greatest way to manage records.
All metadata in a SharePoint document library is set at the library level – either by adding a content type that includes site columns, a site column, or a ‘local’ library column.
You cannot set unique metadata on folders, although you can set (via Library Settings – Column Default Value Settings) a default value for individual folders that gets inherited by the items in the folder.
All items in a library, regardless of how they are grouped by folder, have the same metadata elements.
This may not be an issue for smaller organisations, but in larger organisations there may be a need to apply unique metadata to records. This requirement is harder to achieve using folders.
Folders in a SharePoint site exist in document libraries. The site/library URL path shows where they are situated:
The ability to identify the business context for the records will depend to some degree on the sitename/libraryname combination and how much content is stored on the site. Smaller organisations may find it useful to have a single Team for, say, Corporate Services, and create channels that map to separate business functions with no requirement for different metadata, access controls or retention – e.g., Financial Management, Property, Legal etc.
Larger organisations may find this model becomes very complicated once other factors and requirements for different metadata (set at the library level), access controls and retention are taken into account.
Two types of retention can be applied (from the Compliance admin center) to content stored in SharePoint:
Back-end retention policies that create a Preservation Hold library on SharePoint sites. These policies work in the background and do not include any form of disposition review.
Retention labels published as label policies to SharePoint sites or auto-applied. Some labels may include disposition reviews, the ability to declare records, and lock records entirely with the regulatory option (E5 licence only).
Once they are published to a SharePoint site, retention labels may be applied at the library, folder or individual item level. The image below shows how to apply them to a folder.
When applied at the folder level, all the items in the folder will inherit the label. However, if the records are not locked when the label is applied, end-users with edit rights can remove the label.
If records are grouped in folders in a document library, someone (the records manager, the site owner, the end-users?) will need to ensure the correct label has been published to each folder. Applying labels – and the rights ones – to folders becomes more complex when the library contains a mix of retention periods, and even more so if there are permanent and temporary records.
If labels include the option for a disposition review, someone will need to manage that process. Permanent records may need to be transferred to archives; this is a much harder process compared with storing all related records with permanent retention in the same document library.
Again, this may not be such a problem for small organisations, but in larger organisations, the application of specific retention labels to multiple folders in a library will start to become very complex process that will be difficult to manage over time.
Moving records within the same library in SharePoint or even to a different site is generally straightforward, although the ‘Move’ option should only be used to move a small number of items at a time. Third-party tools should be used to move larger volumes of records.
As noted above, every item in a SharePoint library shares the same metadata columns. Migrating or exporting the content of a folder will generally also require the migration of the metadata as well. Records with permanent value that have to be transferred to an archive will typically have more metadata compared with temporary records. How will this metadata be extracted?
It is not unusual, especially in Teams-based SharePoint sites, to see separate access controls being applied to (channel-linked) folders or even sub-folders in a document library. This involves breaking the existing permission inheritance and adding unique users.
While this may seem like a good idea in the short-term, the creation of unique access controls via folders is a poor way to restrict access to records.
Records that have unique access control permissions would be much better stored in a separate document library, or even a separate SharePoint site.
Every SharePoint library has a default ‘All Documents’ list view and the ability to create multiple alternative views. The content in the library can be grouped, sorted or filtered as required.
And, unlike folders in File Explorer, folders in these views can be made to ‘disappear’ in the browser view (and also via Teams) through the ‘Folders’ setting shown above. The documents still retain their original URL folder path (which can be seen when you click on ‘Copy Link’ – in the example below, the document is stored in the ‘July’ folder).
If the organisation has created a folder-based structure to group and manage records, what will happen if someone disables the ability to see folders? All they (and everyone else) will see will be the documents (that they have permission to access).
The ability to make folders disappear highlights why folders are a poor way to aggregate content. When the folders vanish, how do you group or manage records?
A better approach – keep it simple
Aggregating records in folders may seem like a straightforward thing to do, especially via Teams-linked channels, but if the records stored in the same library have competing metadata, retention, and access control requirements, it will become a nightmare to manage them over time.
A much better model is to use document libraries not folders as the logical ‘containers’ for records, or separate sites with multiple document libraries, with metadata as required, specific retention and inherited access controls.
Teams-based SharePoint sites may be suitable for storing some records, but only if all the records (a) relate to the same business context, (b) share the same metadata, (c) have the same access controls and (d) the same retention requirements.
The bottom line – Try to avoid over-complicating the management of records early on. It will only make it harder to manage records longer-term.
A recent (September 2021) Verge post titled ‘File Not Found‘ by Monica Chin (@mcsquared96) noted how pre-defined storage locations for digital content have become mostly redundant for the younger generation that grew up using Google to find content – ‘the concept of file folders and directories, essential to previous generations’ understanding of computers, is gibberish to many modern students.’
The post resonated for two specific reasons:
The way people create and access content in the modern digital environment, especially how they do this out of the office on mobile devices.
The enduring ‘requirement’ in many organisations to create pre-defined aggregations to store records, with an expectation that end-users will voluntary and consistently store content in those aggregations.
In my opinion, there is a fundamental clash between these two things in the modern digital world.
It reminds me of the transition to Web 2.0 from around 15 years ago. The ‘semantic web’ changed the way we interacted with the internet.
According to the University of Melbourne in a blog post in 2008, ‘Web 2.0 is the term used to describe a variety of web sites and applications that allow anyone to create and share online information or material they have created. A key element of the technology is that it allows people to create, share, collaborate and communicate. Web 2.0 differs from other types of websites as it does not require any web design or publishing skills to participate, making it easy for people to create and publish or communicate their work to the world.’
Fast forward ten years, and end-users can do just that, creating and publishing as much web content as they like through a host of apps, none of which requires a pre-defined folder structure to manage the content.
Is this the future for the modern digital office?
Aggregate or collate?
Expecting end-users to capture all digital content about the same subject in pre-defined aggregations has been a challenge since computers first appeared in the workplace. The fact that email systems have always been physically separate from other digital content stored on file shares didn’t help.
Organisations attempted to resolve this dilemma, with varying degrees of success, through ‘controlled’ folder structures on file shares, or acquiring EDM/ERM systems.
The success of these models was (and remains) often linked to the degree of compliance required by regulators, or the penalties for non-compliance. That is, the best document and records management systems were usually (and continue to be) in organisations that had a high compliance requirement. Everything else was about managing risk (and reputation).
The reality, for most organisations, is that not all relevant digital content is (or can be) captured in pre-defined aggregations. Some of the key ‘missing’ content may include emails, text and chat messages, content created in ‘personal’ spaces, and content not created using corporate systems. This is why FOI and discovery requests inevitably involve asking end-users to locate and produce relevant content.
So, it is just a waste of time to try to aggregate (some) digital records?
I think there is still a comfortable place for the creation and management of content in pre-defined aggregations, where it makes sense to do so and/or there is a specific compliance requirement to do this AND it works.
But, with some exceptions, pre-defined aggregations have a key shortcoming – they may not capture all digital content. Yahoo attempted to categorise the internet from 1994 (as ‘Jerry and David’s Guide to the World Wide Web’), when the reality was that most people preferred the simple Google search bar – without worrying about how the information may be have been aggregated or collated.
The modern workplace is heading in a similar direction. The illusion that all related digital content, of whatever type or format, can be grouped in pre-defined aggregations makes no sense. There is simply too much digital content and a lot of it cannot even be copied to or saved to a single aggregation point. Phone text messages have always been a case in point here.
What makes more sense, like Web 2.0, is to use technology to automatically find the connections between disparate digital objects and collate them in a way that makes sense in the context we want to see it. We already see this model in many online platforms that draw on our digital exhaust (and possibly our voices) to work out our interests or potential interests.
The digital office
The 2 November 2021 announcement of a range of updates to the Microsoft office.com portal, ‘Creating a hub for your content with office.com‘, is consistent with the need to address changing patterns of work and the shift away from pre-defined aggregations as the primary method of storing and finding all relevant content. Instead, the focus is on ease of creation, quick access to content that is relevant to the use based on their digital activities, search, and recommended actions.
As noted already, this new approach does not remove the requirement to create pre-defined aggregations; it simply needs to be understood that these may only contain some of the content or records. The rest may remain stored in email accounts or other systems, in many cases out of reach (thanks to access controls) to the people who may want to see it.
However, behind the scenes, the Microsoft Graph makes use of this information (‘signals’) to draw inferences and make recommendations including for content that the end-user can access. For example, Fred and Mary may chat in Teams or exchange emails about a given subject, but Fred is unaware that Mary is working on a document that he has access to. The Graph works out that this may be of interest to Fred and recommends it to him.
Microsoft 365 provides the capability to recommend content and, with products like Viva Topics, has the ability to automatically group relevant content beyond the scope of pre-defined aggregations.
It may not be able, or ever need, to completely replace pre-defined aggregations, but it is heading in the direction to make that scenario increasingly more likely.
Microsoft 365 provides two primary locations for end-users to create and capture document type content – OneDrive for Business (‘my files’) and SharePoint Online (‘our files’). Both of these can be accessed from the Chats (ODfB) and Team channels files (SPO) tabs in MS Teams.
Both OneDrive for Business and SharePoint (including via the Files tab in Teams) include the option to Sync content to Windows File Explorer or iOS Finder, allowing end-users to access both from a familiar interface.
In 2021, Microsoft added the option to ‘Add shortcut to OneDrive‘ from a SharePoint library or folder via the browser but not (yet) from the Files tab in Teams. As the name suggests, this creates a shortcut in OneDrive for Business to a SharePoint library or folder. While syncing only syncs to File Explorer or Finder, a short cut to OneDrive also makes the library or folder accessible from the browser version of OneDrive, the OneDrive mobile device app, and File Explorer/Finder.
This post explains the difference between Sync and Add shortcut to OneDrive and the potential impacts both of these options may have on managing records. In summary, both options are useful and likely to be popular but a range of records management related capability in SharePoint is neither visible nor accessible in File Explorer.
The Sync option, in both OneDrive For Business and SharePoint Online, allows content in those locations to be synced to and then accessed directly from File Explorer (Windows) or Finder (iOS). The option replaces earlier methods to map SharePoint document libraries to drives. The option is useful and popular because it means that end-users can continue to work in a familiar space.
On both Windows and iOS machines, syncing from both SharePoint and OneDrive is managed by the same OneDrive sync app. As the screenshot below shows, on Windows machines, the sync client adds separate sections in File Explorer for the content synced from SharePoint (‘andrewwarland’ is the tenant name) and OneDrive for Business (‘OneDrive’ – tenant name) as can be seen in the example below (a personal OneDrive section can also be seen).
Positives of syncing
Creating, accessing and managing ‘personal’ or working content in the synced OneDrive area of File Explorer makes a lot of sense and usually replaces ‘home’ drives.
Negatives of syncing
Creating, accessing and managing content synced from SharePoint via File Explorer minimises a range of records management functionality, including the visibility and editability of columns that can otherwise be seen in both the Teams Files tab and in SharePoint via a browser. For example:
A synced library does not display the unique Document ID, added columns, or retention and sensitivity information.
Content can be added even when a column is mandatory (an error message displays in the SharePoint site, but not in File Explorer).
For these reasons, syncing may be best suited for lower-level ‘working’ content and should probably disabled if this functionality needs to be visible or accessible.
In addition to the above, creating new folder in File Explorer at the top level of a synced Teams document library does not create a new channel in Teams.
Add shortcut to OneDrive
As the name indicates, the option to ‘Add shortcut to OneDrive’ allows an end-user to add a shortcut from a SharePoint document library (or a folder in that library) to their OneDrive for Business unless the library (or a folder in it) have already been synced, in which case an error message will appear.
The option is not yet available via the Files tab in a Teams channel.
Shortcuts to OneDrive can be accessed:
Via the browser version of OneDrive
In File Explorer/Finder, under the OneDrive – (Tenant name) section (along with any other synced OneDrive content).
Via the OneDrive app on a mobile device.
When a shortcut is added to OneDrive, a folder with the same name with a ‘link’ icon will appear in all three of the OneDrive options above. In the OneDrive – tenant name section of File Explorer, both library and folder shortcuts display as a folder (with the site name if there is a conflict) with a ‘link’ icon, as can be seen in the example below.
When shortcuts are added in this way, File Explorer displays the same (cloud/accessed) icons as for other synced content. The screenshot below shows the content in the ‘Documents’ library of the ‘Modern Site’ SharePoint site.
The cloud icon indicates that the content has not been accessed or downloaded (or fully downloaded for the content of folders).
The green circle/tick icon indicates that the content has been accessed and downloaded to the local machine.
Positives of adding shortcuts
The ability to add shortcuts to SharePoint content in OneDrive is likely to be useful and popular as, even though this option is more or less identical with syncing (in terms of the outcome), it means that SharePoint content can be accessed from a familiar location (and also on a mobile app).
Negatives of adding shortcuts
The main negative of being able to add shortcuts to SharePoint content in OneDrive, in addition to the issues with syncing already noted, is the potential for confusion.
There is no obvious indicator in SharePoint to say if a shortcut has already been added to OneDrive; only when the end-users tries to share it again will an error message appear.
End users who add shortcuts to SharePoint document library folders may not be aware of other folders in the same library. There is no visible hierarchical navigation to indicate that there may be other folders, or the position of a folder among others.
Despite having a ‘link’ indicator in the folder icon, end-users may treat the content in the shortcut folder as being in their ‘personal’ OneDrive space.
Additionally, that if a library or folder in a library has been added as a shortcut to OneDrive and is then deleted from SharePoint or the Teams/Files tab, the shortcut will still exist in the person’s OneDrive until they remove it. A shortcut folder can be removed at any time by an end-user also by using the remove option. This implies that end-users will proactively manage shortcuts in their OneDrive.
Creating, accessing and managing content in File Explorer/Finder is a common and popular way to work. However, if corporate records are being stored and managed in SharePoint, it is a good idea to ensure that end-users are aware of the various ‘levels’ of functionality:
SharePoint document libraries. These are a logical ‘container’ for storing records and provide the full set of document and records management options for managing records, including a wide range of system and added metadata, column information including added metadata, retention and sensitivity labels, check out/in, the ability to create multiple views, versioning, etc.
Teams Files tab. This area of Teams provides a slightly cut-down version of the document library experience and is focused on the default Documents library that comes with every Team; each new (non-private) channel creates a folder in that library. It is not (yet) possible to add a shortcut from the Files tab, and versioning is not visible.
File Explorer/Finder provides basic information only about the content stored in the document library although with Windows 10/11 it includes the ability to navigate to the SharePoint site and see the version history.
While both syncing and adding shortcuts provide a very similar outcome in File Explorer, good communication is important to ensure this functionality is used appropriately.
One of the difficulties for many records managers new to Microsoft 365 is that they may not know or have access to everything they need to know to manage records in that environment.
This post provides an overview of the accounts, licences and roles that may be needed to manage records in Microsoft 365.
As will be seen, a good understanding of both (a) the options provided with different licence types, and (b) the roles and what access or functionality they provide, is essential. The simplest options may be summarised as follows:
Technically skilled or qualified records and information managers, including those who previously administered EDRM systems, could become SharePoint admins (an assigned role), ideally with a cloud-only account in addition to their normal user account.
Records and information managers with broader compliance-type responsibilities, including the requirement to access detailed audit logs, search across all content, establish and run eDiscovery cases, create and apply DLP policies, create and apply records retention labels and policies, and create and apply information protection labels, should be assigned the role of Compliance Admin or Compliance Data Admin. They should also be assigned the Global Reader role to be able to access reporting and details of all settings in the various admin portals. If there is sensitivity about this access, they could instead be assigned the Reports Reader role which provides access to (and the ability to download) usage reports in the Microsoft 365 admin portal.
Records and information managers who need to do most of the above excluding access to audit logs and some other features could be assigned the roles of Records Administrator and Reports Reader.
Accounts and licences
To access Microsoft 365 anyone will need an active Active Directory (AD) account and, except for Global Admins, a Microsoft 365 licence.
Microsoft 365 accounts may either (a) be synced from on-premise AD to Azure Active Directory (AAD), or (b) be cloud-based only (for admin accounts) as described below.
Global Administrators (GAs) have access to all parts of Microsoft 365. It is unlikely that records or information managers will be GAs. In organisations that outsource their IT, the IT provider will usually be the GAs, further limiting internal access and knowledge.
Microsoft recommend (‘Protect your Microsoft 365 Global Administrator accounts‘) that (a) there should be no more than four Global Administrator (GA) accounts, and (b) that GA accounts should always be created in the cloud and not be privileged accounts synced from AD. This is not always the case.
GAs do not need a Microsoft 365 licence.
In the same link above, Microsoft notes that ‘you should consider whether additional accounts with wide-ranging permissions to access the data in your subscription, such as eDiscovery administrator or security or compliance administrator accounts, should be protected in the same way’ – that is, created as cloud-only accounts.
These administrator accounts include the following:
Exchange Online administrator
MS Teams administrator
Note that certain ‘admin’ roles may be created to provide access to parts of the Compliance admin portal. These roles, which may not necessarily require a cloud-only account, are described under the ‘Roles’ section below.
Cloud-only account names should include a user ID to make it possible to identify the actual person assigned to the account who used it, including in audit logs. For example:
Suitably qualified or skilled records and information managers, especially EDRMS admins, could be assigned either a SharePoint admin or Compliance admin account.
To be clear, this means that records and information managers should ideally have two separate accounts: (a) an admin account with a specific role (see below), and (b) their normal account.
Note that being assigned to an account doesn’t provide any access, yet. The account holder still needs a licence and a role.
Some organisations do not follow Microsoft’s advice and don’t create separate admin accounts for SharePoint/OneDrive, Exchange, Teams or Compliance/Security admins. Instead, they assign roles to normal end-user accounts (see below). This may be partially to reduce licencing requirements.
With the exception of the GA account, all other accounts must be assigned a licence to access the relevant parts of Microsoft 365. For example, a SharePoint admin needs a licence that includes access to SharePoint.
The type of licence that is assigned will affect what options the account can access. From a records and information management point of view:
E5, or the E5 Compliance add-on (for specific users), provides the highest level of recordkeeping functionality including all the options in the Compliance admin portal including Data Classification, Audit, Content Search, Advanced eDiscovery, Data Loss Prevention, Information Governance, Information Protection, and Records Management (including auto-application of labels based on trainable classifiers or SharePoint Syntex, and Dispositions).
E3 provides a basic level of functionality – mostly the ‘Information Governance’ part (retention labels, label policies and retention policies) of the Compliance admin portal (access to which requires a role – see below) and some other parts such as Audit (logs), Content Search, Data Loss Prevention, Information Protection, and eDiscovery.
So, to summarise this section, records and information managers will need the following at a minimum to be able to manage records in Microsoft 365:
An admin account, or their normal account assigned a specific role (see below)
A licence (E5 or E3)
A licenced account will not provide access to anything more than the basic options available to all users, until the account has been assigned a role.
There are multiple roles in Microsoft 365 for a wide range of tasks.
Roles and custom roles (and ‘role groups’ that contain sub-roles) may be assigned to accounts via:
The Azure AD admin portal (~ 77 pre-defined roles most of which are repeated in the two dot points below)
The Microsoft 365 admin portal via ‘Role assignment’ (shows everyone who has been assigned that role) or applied directly on accounts (~ 64 Azure AD roles and ~ 13 Exchange admin roles)
The Compliance admin portal under ‘Permissions’ (9 Azure AD roles and 49 ‘Compliance centre’ roles). Same roles as the Security admin portal.
The Security admin portal under ‘Permissions and roles’ (9 Azure AD roles and 49 ‘Email and collaboration’ roles). Same roles as the Compliance admin portal.
Microsoft note that ‘admin roles give users permission to view data and complete tasks’. Therefore, they recommend to ‘give users only the access they need by assigning the least-permissive role’.
Most roles can be assigned to user accounts from either the Azure admin portal or the Microsoft 365 admin portal. Only the most common roles are listed below, most of the other roles can be viewed from the section ‘Show all by category’ below this listing. When a role is assigned to an account, the ‘Admin’ icon appears in the list of apps available to that account in office.com.
Organisations will need to decide what roles will be assigned to what accounts. The licence assigned to that account may affect what options the account can access through the role they are assigned to.
What roles do records and information managers need?
The most obvious roles for records and information managers accounts that are set in either the Azure or Microsoft 365 admin portals, are:
The following roles that relate to the management of records that are set in the Microsoft 365 admin portal, Azure admin portal, or the Compliance admin portals are as follows:
Compliance data admin
Records Administrator (only set in the Compliance admin portal)
Records Management (only set in the Compliance admin portal)
Knowledge Administrator (only set in the Azure admin portal)
Knowledge Manager (only set in the Azure admin portal)
Details of what these roles can do or have access to are described below.
In most cases, the simplest option will be to grant the records or information manager account (preferably a cloud-only account) the ‘full’ Compliance admin role or alternatively the Compliance Data admin role (with slightly reduced capability).
While additional roles, sub-roles, or even custom roles may be assigned or created for specific purposes, these may only be required in more complex organisations where those roles can be assigned to someone with that responsibility. For example, a ‘Privacy Manager’ person/account could be assigned the Privacy Management role
Records or information managers who were EDRMS administrators might be assigned the SharePoint admin role, including in support of more technical SharePoint admins.
Accounts assigned to the SharePoint admin role can create new sites and manage the SharePoint environment, including managing the Managed Metadata Term Store. They can guide site owners how to use their sites, including:
Creating and applying site columns and content types to libraries and lists.
Creating and configuring new document libraries and lists
Editing page content
SharePoint admins may apply, or help site owners and members apply, retention labels and recover content from the Recycle Bins and Preservation Hold library (if a retention policy has been applied to the site).
This role provides read-only access to all of the options in all the Microsoft 365 admin portals. Accounts assigned to this role can also download reports. It is a useful role to have if the records or information manager needs to view settings and reports.
This role provides read-only access to various sections of the Microsoft 365 admin portal only including: Users, Teams & Groups, Roles, Billing, Settings, Reports (Usage reports), Health (but not the Message Centre). Accounts assigned to this role can also download reports.
Compliance admin/Compliance Data admin
Accounts assigned to the Compliance admin role group have full access to the Compliance admin portal, including the following areas:
Data classification (E5). Access to trainable classifiers, content explorer, activity explorer.
Audit logs (3 months for E3, 12 months for E5)
Content Search (search across all Exchange mailboxes (including Teams compliance chats and posts) and SharePoint/OneDrive)
Data Loss Prevention (create policies based on a range of criteria)
eDiscovery (search for content, apply legal holds)
Information Governance (with E3 licences, basic retention labels, label policies, and retention policies)
Information Protection (information sensitivity labels)
Records Management (with E5 licences, more advanced retention label options, label policies, dispositions)
The Compliance admin role group includes all the following sub-roles. Items marked with an asterix are not included in the Compliance Data Admin role group and will mostly not be required. (That is, most records and information manager accounts could be assigned the Compliance Data Admin role).
Case Management *
Data Classification Feedback Provider *
Data Classification Feedback Reviewer *
Data Investigation Management *
Disposition Management (manage disposition)
DLP Compliance Management
IB Compliance Management
Information Protection Analyst
Information Protection Investigator
Information Protection Policy Admin
Information Protection Reader
RecordManagement (‘manage and dispose record content’)
Retention Management (‘create retention labels’)
(‘Sensitivity Label Administrator’ is included in the Compliance Data admin role)
View-Only Audit Logs
View-Only Case *
View-Only Device Management
View-Only DLP Compliance Management
View-Only IB Compliance Management
View-Only Manage Alerts
View-Only Record Management
View-Only Retention Management
Records Administrator/Records Management/Knowledge Administrator
There are three specific role groups that provide more restricted records management-related capability.
The Records Administrator role group includes access to most of the day-to-day functionality that will be required, via the following sub-roles.
Audit Logs (but note that the account assigned this role must also be added to the Records Administrator role in the EXO admin portal to access the logs)
View-Only Audit Logs (comment as above)
Journaling (in EXO admin)
Messaging Tracking (in EXO admin)
Transport Rules (in EXO admin)
The Records Management role group include the following sub-roles:
This role provides access to most of the records management related options in the Compliance admin portal except the Audit logs.
The Knowledge Administrator role group provides access to a number of options in the Compliance admin portal has a single sub-role.
The Knowledge Manager role provides access to a number of options in the Compliance admin portal.
Summarising the options
In summary, records and information managers will need the following at a minimum to manage records in Microsoft 365:
An active AD or AAD account
A Microsoft 365 licence, ideally E5 or E5 Compliance add-on. E3 may be suitable but provides fewer options.
One or more roles:
SharePoint Admin, if they will be managing SharePoint.
Compliance admin. This is suitable for most advanced records management related activities including accessing audit logs, creating, applying and managing retention labels and policies, DLP, information protection and more). Alternatively, one of the reduced Compliance roles.
Global Reader (or Reports Reader) to view usage reports/dashboards and settings.
Whatever role is required, records and information managers need to work closely with IT to determine the most appropriate role based on business needs.
Microsoft 365 includes the capability, subject to licencing, to apply retention labels and non-label retention policies to content stored in Exchange mailboxes, SharePoint Online sites, Microsoft 365 Groups and OneDrive for Business accounts.
Non-label retention policies may also be applied to Teams chats, Teams channel (including private channel) posts, and Yammer messages. For Teams chats and posts, these policies apply to the ‘compliance copy’ of the messages stored in a hidden folder in Exchange mailboxes.
Both retention labels and retention policies work against individual items stored in these location, not the containers (e.g., Exchange mailbox, SharePoint document libraries, OneDrive account, Team chats and channels. This means that individual items stored in those aggregations will disappear (when destroyed) leaving the container still in place.
Creating and applying retention labels and policies is a relatively straightforward matter. For records and information managers, knowing where the labels and policies have been applied is more complex, as this post explains.
Creating retention labels and policies
Both retention labels and retention policies are created in and applied from the Microsoft 365 Compliance admin portal.
The Information Governance section is used to create both: (a) retention labels that can be published as label policies, or auto-applied (E5 licence required), and (b) retention policies.
The Records Management section is used to create retention labels, based on a file plan that can be published as label policies or auto-applied (E5 licence required).
The two other tabs in the records management section are (a) Events and (b) Dispositions. The latter area is where basic details or records due for disposition, and records that have been destroyed, are displayed.
Publishing retention labels
Retention labels do not do anything until they are either:
Published individually or in groups as label policies to all or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
Auto-applied (with an E5 licence) including: (a) via keyword searches, (b) based on information sensitivity, (c), based on basic classifiers, (d) as part of trainable classifiers or (e) SharePoint Syntex (linked with a Content Type).
Labels cannot be published to or applied to Teams chats or channel posts, or to Yammer content.
Applying retention labels
Retention policies may be published (applied) to either:
All or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
All or none of: Teams chats, Teams channel posts, Teams private channel posts, Skype for Business chats, Yammer community posts and chat messages.
Viewing labels, label policies, and retention labels
Once they have been created, the basic details of labels, label policies and retention policies can be seen in the Information Governance and Records Management sections.
Information Governance > Labels
The ‘Labels‘ tab displays all labels that have been created (including via the Records Management – File Plan section, see below), with the following columns (bold indicates what is visible in the default view for this listing – see also below for the list of labels in the Records Management section). The listing (and individual items when clicked) does not include any information to indicate if the label has been published to what label policy, or auto-applied.
Status (active, inactive)
Based on (trigger – when created, last modified, label applied, event, none)
Is record (if ‘records declaration’ is checked)
Retention duration (days, months, years or none)
Disposition type (auto delete, (disposition) review required, no action)
Reference ID (File Plan detail)
Function/department (File Plan detail)
Category (File Plan detail)
Subcategory (File Plan detail)
Authority type (File Plan detail)
Provision/citation (File Plan detail)
Last modified by
Information Governance > Label policies
The ‘Label policies‘ tab displays all label policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, e.g., Exchange mailboxes, SharePoint sites, not specific locations), the labels that have been included, and if a preservation lock has been enabled.
Type (Publish, other options may also be listed)
Last modified by
Last modified date
Information Governance > Retention policies
The ‘Retention policies‘ tab displays all retention policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, not specifically), retention period and if a preservation lock has been enabled.
Records management section
Records Management > File Plan
The ‘File Plan‘ tab displays all labels (including those created via the Information Governance – Labels section above) with all the columns shown above for the Information Governance labels. Again, there is no information to indicate if the label has been auto-applied or what label policy it has been published in.
Records Management > Label policies
The ‘Label policies‘ tab displays all label policies that have been created. It contains the same content as the Information Governance ‘Label policies’ details above.
How do you know where retention labels and retention policies have been applied?
As noted above, both retention labels (published as label policies or auto-applied, but not to Teams or Yammer) and retention policies are applied to individual items stored in containers (mailboxes, sites/document libraries, OneDrive accounts, Team chats, Teams channels) in the various workloads.
This model, and the ability for end-users with edit rights to change or remove labels on content stored in SharePoint that is not declared as a record or change a label on emails, can make it very difficult to know (for records managers in particular) what has been applied where.
Microsoft 365 does not provide any out of the box option to view where retention policies have been applied (to individual records). There are currently only two ways to know where retention labels have been applied:
Post facto – via Content Explorer in the Data Classification section (requires E5 licence), for labels only.
As part of retention planning.
Both of these options are described below.
Data Classification > Content Explorer
For those with E5 licences, the ‘Content Explorer’ area in the Data Classification section of the Compliance admin portal can provide details of where labels (only) have been (or are currently) applied after the fact.
According to the Microsoft guidance Get started with content explorer, (which includes helpful screenshots) ‘Content explorer shows a current snapshot of the items that have a sensitivity label, a retention label or have been classified as a sensitive information type in your organization.’
It also notes that ‘Access to content explorer is highly restricted because it lets you read the contents of scanned files’, via the second of the two roles required to access this capability:
Content Explorer List viewer: Membership in this role group allows you to see each item and its location in list view.
Content Explorer Content viewer: Membership in this role group allows you to view the contents of each item in the list.
So, Content Explorer is helpful, but only to know where retention labels have been applied after the fact. And it requires an E5 licence (or E5 Compliance licence add on).
Documenting retention via retention planning
Planning for retention in Microsoft 365 is an essential requirement for records and information managers. It should always be based on regulatory, archival or business requirements for keeping records.
One of the challenges for the retention model in Microsoft 365 is the application of retention labels and/or retention policies to Exchange Online mailboxes, OneDrive accounts, Teams chats and Teams channel posts (including private channel posts). It is a challenge because the traditional recordkeeping model involved copying content identified as records from the create/capture application to a central recordkeeping system where the records would be subject to retention, whereas in Microsoft 365 the reality is that not all records may (or can) be captured.
Retention and disposal authorities/schedules typically do not include coverage for entire mailboxes or OneDrive accounts, let alone Teams content. In response to this, the United States National Archives and Records Administration (NARA) developed the ‘Capstone‘ (PDF) approach for email, whereby ‘final disposition is determined by the role or position of the account user, rather than the content of each individual email.’ Something similar to this model, also for OneDrive accounts and Teams chats and channel posts (both of which have compliance copies stored in Exchange mailboxes), may also be required.
Tackle retention policies first
Retention policies are sometimes referred to as ‘back-end’ or ‘safety net’ policies because they work in the background and help to capture deleted content that needs to be retained for minimum periods of time after which it can be deleted unless a longer retention (e.g., label-based) applies. Retention policies cannot normally be mapped to retention classes because they target entire workloads not specific aggregations in those workloads (e.g., they target all content in SharePoint sites, not the content in the libraries of those sites).
Retention policies do not include any form of disposal review. The content is either destroyed or ‘nothing happens’ (allowing the content to be destroyed after the retention period expires).
Retention policies can be applied to the following parts of Microsoft 365:
All or selected groups of Exchange mailboxes
All or selected groups of SharePoint sites
All or selected groups of OneDrive accounts
All or selected groups of Microsoft 365 Groups
All (or none of) Teams chats
All (or none of) Teams channel posts
All (or none of) Teams private channel posts
All (or none of) Skype for Business chats
All (or none of) Yammer community posts
All (or none of) Yammer chats
(Keep in mind that Teams chats and Teams private channel posts are stored in the personal Exchange mailboxes of participants, while Teams channel posts are stored in the Exchange mailboxes of the Microsoft 365 Group linked with the Team, BUT Teams retention policies can only apply to all or none.)
Once a decision is made to create retention policies, this detail should be documented separately from Microsoft 365 – in a spreadsheet or SharePoint list perhaps. The following is a basic example of what might be included in that documentation.
Senior executive EXO mailboxes – 15 years / date modified / do nothing
All other EXO mailboxes – 7 years / date modified / destroy
All SharePoint sites – 7 years / date modified / destroy
M365 Groups – 7 years / date modified / destroy
All ODfB accounts – 7 years / date modified / destroy (see also note below)
All MS Teams chats – 7 years / date created / destroy
MS Teams channel posts – 7 years / date created / destroy
MS Teams private channel posts – 7 years / date created / destroy
Yammer user messages – 1 year / date created / destroy
Yammer community messages – 2 years / date created / destroy
Keep in mind that retention is based on either date created or date modified, so the content stored in these locations will begin to disappear when the retention period expires. This will leave empty mailboxes, empty SharePoint document libraries, OneDrive accounts and empty Teams. Part of the retention planning needs to include a process for destroying those containers once the content inside them has been destroyed – and documenting that destruction also.
For OneDrive accounts, consideration might be given instead to actively reviewing the accounts of departed employees to removing any content considered valuable to another location (e.g., SharePoint) or account, rather than simply leaving it hidden from view for the period of the retention. This would allow much shorter retention after the employee leaves.
Documenting retention labels
Retention labels will typically be based on individual classes in a records retention schedule or disposal authority (DA). Accordingly, it will be useful to have a listing of all classes in a spreadsheet or SharePoint list with the following details as a starting point:
Retention schedule class ID
Retention schedule class name
Retention schedule class description (or abbreviated version).
Retention labels can then be created based on those classes. Every label includes the following details (see also screenshot below):
A name that should be obvious to end users who may end up having to choose it or know from a glance what it means. For example ‘5.1.1 Financial records – 7 years’.
A description – same as the original retention class description, or an abbreviated version.
A retention period. For example, 7 years.
A trigger. The options are: ‘When items were created’, ‘When items were modified’, or ‘When label was applied’. (Note that triggers may also be based on events, but these should be used sparingly and only after the full implication of using them is understood.)
An action at the end of the retention. For example: ‘Delete items automatically’, ‘Trigger a disposition review’ (E5 only), ‘Do nothing’. ‘Do nothing’ is a good option for permanent records as it will stop deletion and tag a record making it easier to find later. There are three other options instead of retain: ‘Retain forever’, ‘Only delete items when they reach a certain age’, and ‘Don’t retain or delete items’.
(Note: When labels are created from the Records Management section, two additional options appear: (a) Mark items as a record and (b) Mark items as a regulatory record. The implications of using either option needs to be understood).
As each retention label is created the following details should be recorded next to the classes in new columns in the spreadsheet or list:
Retention label name (may be a combination of the ID and name, and retention)
Retention period. For example, 7 years.
Retention trigger. For example, ‘When created’, ‘When modified’, ‘When label applied’.
End of retention action. For example, ‘Destroy’, ‘Review’, ‘Do nothing’
Declare as a record: Yes/No
Regulatory record: Yes/No
As labels are published, individually or in groups, this information should be added to columns in the spreadsheet or list:
Retention policy. That is, what is the name of the policy where this policy has been published.
Policy location. That is, the location/s were the policy has been applied.
Date for destruction. (Date applied plus retention period)
If the label is being used via trainable classifiers, SharePoint Syntex, or otherwise auto-applied, that information should also be added to the spreadsheet or list.
Trainable classifier? Yes/No
SharePoint Syntex? Yes/No
Auto applied? Yes/No
Auto applied location.
End product – for all retention labels and policies
The end of product of is either a spreadsheet with two workbooks, or two SharePoint lists (one for the retention policies, one for the labels), that describe what retention labels and retention policies have been created and where they have been applied across Microsoft 365.
Records and/or information managers will need to proactively monitor the Microsoft 365 environment and provide advice to IT about the destruction of content and records, as well as the containers in which that content was stored.
There may be other ways to achieve this outcome, but records and information managers need to understand what capability is available (or not) out of the box to make informed decisions.
In my previous post about managing inactive Teams, the third option listed was to apply retention policies to those Teams. It included the graphic below.
This post provides more details of a basic retention model that can be applied to both active and inactive Teams.
Key takeaways from this post for records and information managers:
Every Team has a ‘Posts’ (group chat messages) and ‘Files’ (documents etc) tab, and usually also starts with a Wiki tab (which can be removed). Other tabs may be added via the + option.
A Team in Microsoft Teams is not a single container or aggregation for the capture and storage of records. Almost all the records in a Team are stored in a hidden folder in Exchange Online (EXO) mailboxes (posts) or SharePoint Online (SPO) (files). Some records (conversations) may also be created and captured in the EXO mailbox of the associated Microsoft 365 (M365) Group.
It is not possible to apply a single retention policy to a Team; at least two separate policies will be required – one policy for the Team channel posts of EVERY team, and one or more policies for the content captured in SPO sites (files) or groups of sites.
Some records, created in and accessible from Teams, may be stored in other M365 applications (e.g., Tasks, Forms, WhiteBoard, etc) or third-party applications. It is not possible to apply any Microsoft 365 retention policy to records created by or captured in these applications.
Records and information managers should have access to the details (not necessarily the content) of every M365 Group, Team, and SPO site in order to establish a plan for the creation and application of retention policies to Teams. At a minimum, they should be assigned the Global Reader role (for details of M365 Groups and SPO sites) and the Compliance admin role (for retention policies).
It is relatively easy to overcomplicate the retention model for Teams, for example by applying separate retention labels to different folders and sub-folders in each channel ‘files’ tab.
Try to keep the model simple for as long as possible.
Core components of a Team
The main components of every Team are shown in the diagram below. If private channels are not allowed in the organisation, ignore the top two left and right elements.
As shown in the diagram above:
Every Team is directly linked with an M365 Group. Every M365 Group has an Exchange Online (EXO) mailbox and a SharePoint Online (SPO) site.
The Team, M365 Group, SPO site, and mailbox address (teamname@) all share the same name. The original name (which should be brief, <20 characters if possible) and the display name may be different.
The Owners and Members of the Team are the Owners and Members of the M365 Group and those Groups are added to the SPO site Owners and Members permission groups respectively.
A ‘compliance copy’ of every post in a normal channel is copied from the Azure-based Teams chat service (which is always inaccessible) to a hidden folder of the EXO mailbox of the M365 Group linked with the Team.
Where private channels are allowed, a ‘compliance copy’ of every post in a private channel is copied to a hidden folder of the ‘personal’ EXO mailboxes of all participants in the private channel.
Any content created or captured in the ‘Files’ tab of the Team channels is stored in the SPO site of the M365 Group linked with the Team. If any lists are created, they are either stored on the same SPO site or are linked from another site.
Where private channels are allowed, a separate SPO site is created (using the name of the ‘parent’ site followed by a hyphen then the private channel name, e.g., parentsitename-privatechannelnamesite). Any content created or captured in the ‘Files’ tab is stored in that SPO site.
So, a Team is a combination of at least four elements: the Teams user-interface (and back-end database), an M365 Group, a SPO site, and an EXO mailbox. The mailbox is used for three main purposes:
Email-based ‘conversations’ (when used).
Storage of Teams posts.
This is why it is not possible to apply a single retention policy to a Team.
The basic retention model
The basic retention model for Teams assumes the following:
If the organisation’s retention schedule/disposal authority does not include coverage for Team posts (chat messages) and also general Team chats, there is a legally defensible policy that defines how long Team channel (including private channel) posts (and chats) will be retained. Note: This policy will define a single retention period for ALL posts and and a separate policy for ALL chats.
Records and information managers know the details of every M365 Group, Team (including number of private channels) and SPO site (including last activity and number of files).
One or more retention policies will be created for SPO sites.
One or more retention policies may be created for M365 Groups.
Unless it is done ‘manually’, there will be no review process before the content is destroyed at the end of the retention period.
No label-based retention policies will be applied (at this point). They may be added later as required (see below).
Unless the option to auto-expiry M365 Groups is used, there will be a manual process to delete inactive and empty M365 Groups or Teams; deleting either will also delete the linked SPO site.
Creating retention policies
Retention policies are created in the Information Governance section of the M365 Compliance admin portal under ‘Retention policies’.
Generally speaking, organisations should not create many of these policies as they should ideally target entire workloads (all SPO sites, all EXO mailboxes, etc) or in some cases major groupings (e.g., EXO mailboxes of senior executives, all other mailboxes).
And remember, these policies do NOT destroy the container (Team, SPO site, EXO mailbox), only the content in those containers.
Every new retention policy has three parts.
The name of the retention policy should be easily recognisable, for example ‘Teams channel posts 7 years’ (all encompassing, for all channel posts, see next dot point), or ‘General SPO site retention 7 years’. The name section also includes a description that should always be used to link the policy to details in a retention schedule/disposal authority or corporate policy.
The ‘location’ element is where the complexity arises as it is not possible to create a single retention policy for all the elements in a Team. Selecting either ‘Teams channel messages’ or ‘Teams private channel messages’ will disable all other options. It is not possible to select ‘SharePoint sites’ or ‘Microsoft 365 Groups’ AND any of the Teams options in the same policy.
Because of this limitation, at least two separate retention policies will be required for a basic retention model, with an additional one for private channels (if required):
A retention policy for either all or selected SharePoint sites, including private channel sites. The simplest model is to create a single retention policy for all SharePoint sites. This creates a preservation hold library on every site, retaining all deleted content for the minimum period required. Alternatively, and especially if there is a way to ‘group’ SPO sites (e.g., all project team sites), create retention policies for those groups and add in the site names. Always keep in mind that a retention policy applied to the SPO site has no connection with or impact on the channel posts.
A retention policy for all Teams channel messages. Note that this cannot include or exclude any Teams – it’s all or none. Depending on the retention selected for channel posts (next point), this could mean that channel posts are destroyed before (or after) the Team’s SPO content.
A retention policy for all Teams private channel posts. Similar to the previous point, this is an ‘all or none’ policy.
If the Team is also making use of the M365 Group’s ‘conversations’ in Outlook, consideration may also be given to creating a retention policy for M365 Groups (or included/excluded Groups). This policy will cover (a) Group ‘conversations’ and (b) the SharePoint site linked with the Group/Team. It will NOT cover the Team channel posts that may be stored in the M365 Group EXO mailbox. Note: It is possible to select just the M365 Group mailbox OR the M365 Group’s SPO site in this policy via a PowerShell script.
Retention options are shown in the screenshot below. These options are the same for every retention policy.
Retention policies either automatically delete content after a minimum period or do nothing (includes the ‘retain items forever’ option). There is no disposition review. This means that the content in the SPO site and Team channel (including any ‘deleted’ content, which is not actually deleted, just hidden) simply disappears when the retention period expires.
Organisations may of course have different requirements or decide to apply retention differently. Each of these will still be some variation on the above model.
In most cases, there should be at least one retention policy in place for each of the different elements that make up a Team – the M365 Group, the SPO site, the channel posts, the private channel posts. Whether those policies have the same retention period will be up the organisation to determine, but in all cases, the details should be documented somewhere as currently this information is not easily available.
It is not possible to apply retention labels to Teams channel or private channel posts (or chats). There is only one option, and that is a single retention policy for each of these.
Retention labels may be applied to the content stored in the Teams linked SPO site, and these may be applied instead of using retention policies. This may be an effective model when combined with auto-expiry of M365 Groups as this (auto-expiry) will not occur if the content is subject to an active retention policy or retention label.
However, applying labels to the content stored in each Team channel ‘files’ tab has the potential to be a very complicated model that will become almost impossible to monitor or manage in time.
Each channel ‘files’ tab maps to a folder with the same name in the Documents library of the linked SPO site. As each Team channel may have been created for the records of a different subject with a different retention requirement, this means that each folder (or potentially even sub-folders) in the library may have a different label.
As retention labels (and policies) apply to individual items in the library (but not the folder), this means that individual items, stored in folders, that are subject to disposition review will come up for review in the future.
The application of multiple retention labels to folders within the single Document library of the SPO site is already complicated; having to review some of the individual items as part of a disposition review in the future is just adding to the complexity.
My view is that Teams should, as far as possible, ‘contain’ records relating to the same subject with the same single retention period that can be applied to the entire SPO site. Applying individual labels to folders or sub-folders within a single document library is a complex model both to apply and manage into the future.
What do to with empty Teams?
As noted already, retention policies (and labels) do not delete the SPO site, Team or M365 Group, only the content stored in them. Each of these ‘containers’ remain after the content has been destroyed within them.
Accordingly, it is advisable for records and information managers to (a) have access to the details of every SPO site, Team and M365 Group and (b) work closely with IT to determine when these containers can be deleted (and document that activity). Otherwise, the M365 environment will be left with the hollow shells of sites, Teams and Groups.
The following Microsoft links provide further details on this subject.
The rapid and often uncontrolled rollout of Microsoft (MS) Teams as part of Microsoft 365 (M365) deployments from early 2020 has become a headache for many records and information managers. In many organisations, inactive Teams – some with no owners and inaccessible to records managers – litter the M365 landscape.
The introduction of private channels in 2020 added a new layer of complexity for the management of inactive Teams.
This post examines three ways to manage inactive Teams, especially those that may contain records.
Auto-expiration (and deletion) of M365 Groups.
Applying (separate) retention policies to the elements that make up each Team.
It assumes that records and information managers will or should:
Take a leading role or be involved in decisions with IT departments around the creation of new Teams and the management of inactive Teams and their associated SPO sites.
Have access to the details of all active and inactive M365 Groups, Teams (including private channels), and SharePoint sites, including through role assignment (e.g., Global Reader, Compliance admin).
Know how and where Teams stores content in different applications.
Be directly involved in decisions about the creation and application of retention policies to Teams content, and disposition actions when those policies expire.
Where appropriate, be made the owners of inactive Teams (and M365 Groups) to allow them to review the content of that Team.
Option 1 – Auto-expiry of M365 Groups
Every Team in MS Teams is directly connected with an M365 Group; a Team uses the M365 Group’s EXO mailbox and SPO site for the storage of content. Therefore, if the M365 Group is destroyed, so will the Team and all its content.
Microsoft 365 includes the ability to automatically ‘expire’ and then delete all or selected M365 Groups after a given period of inactivity.
The Group’s expiration option is set in the Azure Active Directory (AAD) admin portal under Groups > Settings > General. This option includes renewal notifications (which will appear in Teams) and the ability to select specific M365 Groups (the default is None).
Pros of auto-expiry
Automatically expiring and then deleting M365 Groups can be a simple way to clean up inactive Groups and the linked Teams, based on the last activity of the Group or in the Team (SPO site, EXO email-based ‘conversations’, or channel posts). This may be particularly effective for general Teams that have been hardly used and/or known not to contain records.
Auto-expiry may be a useful option in conjunction with retention policies; M365 Groups and linked Teams subject to both will be retained beyond the expiry date if they are subject to retention policies.
If the expiry notification is missed or overlooked and the Team is soft-deleted, M365 Groups (and their associated Team content) can be restored for up to 30 days. The SPO site will be recoverable for 93 days. But, beyond 30 days the deleted M365 Group and all the content associated with it (including Teams) is irrecoverable (93 days for the SPO site).
Cons of auto-expiry
Auto-expiry is effectively auto-deletion without review. This option may work best for organisations with a relatively low number of Groups and/or where there is low concern or risk of deleting records prematurely. Organisations that are concerned about the deletion of records without review should be cautious of this approach.
Note that even if auto-expiry is set, this will not destroy any M365 Group or Team that is still subject to a retention policy – see below.
Any Team in MS Teams can be archived either by the MS Teams admin (via the admin portal), or by a Team Owner via the gear icon at the bottom left of the MS Teams application, next to ‘Join or Create a Team’. Clicking the gear icon opens a list of Teams; at the far right, the three-dot menu includes the options (including ‘Archive Team’) listed below.
The process of archiving a Team includes the option to make the linked SharePoint site read only, and makes the Team’s channels read only.
If the SPO site is not also made read only, the members of the Team can continue to upload and edit content via the Team’s channels or via the SPO site directly (and also via File Explorer for synced libraries).
Teams that have been archived appear in a separate ‘Archived’ section, from where they can be ‘restored’ (un-archived, made editable again) provided they are not subject to an auto-expiry policy or retention policies.
Pros of archiving Teams
Archiving Teams (and making the linked SPO site read only) may be a useful way to prevent any further changes to those Teams, but it does not do more than that. Additional options, including either auto-expiry (for low-risk Teams) or retention policies (for Teams with records) should be considered to ensure that inactive archived Teams are destroyed when this is allowed.
Archiving Teams may also be a useful way to ‘tag’ Teams that cease to be active, making them more easily identifiable for retention or disposal.
Cons of archiving Teams
Archiving Teams is not an effective or safe way to ensure that any records contained in the Team remain unchanged for as long as the Team still exists. It simply makes the Team’s channels read-only, and may also make the SharePoint site read only, if that option is selected.
If an archived Team is subject to an auto-expiry policy, it will be destroyed (with prior notification after a specified period. A better option for Teams used to create or capture records would be to apply retention policies to the Team.
This is probably the most complex area of M365 for records and information managers to understand given the multiple elements that make up MS Teams. Careful planning is necessary before any retention policy is applied, based on a thorough understanding of the structure of Teams and where the content is stored.
As a starting point, it is important to understand that:
A single retention policy cannot be applied to all the content of a Team and its associated M365 Group (private channel chats, channel posts, SPO files, Outlook ‘conversations’). Multiple retention policies will be required.
It is NOT possible to apply retention labels to either Teams public or private channel posts. These can only be covered by retention policies. Retention labels could be applied to content stored in the SPO site.
The model for applying retention to Teams (not the 1:1 chats area) may include up to four separate retention policies (and also retention labels):
One or more retention policies for the Team (non private) channel posts. These policies will apply to the compliance copies of those posts stored in a hidden folder of the linked M365 Group’s EXO mailbox.
One or more retention policies for the Team’s private channel posts if they exist. These policies will apply to the compliance copies of those posts stored in a hidden folder in the EXO mailbox of all members of the private channel.
One or more retention policies for the Team’s files stored in the SPO site. Additional retention labels may also be applied (see below).
If the mailbox is used for Group conversations, one or more retention policies for the M365 Group, which includes coverage for both the emails and the files.
So, each Team could potentially be subject to up to four separate retention policies.
In addition to the above, retention labels may be applied either ‘manually’ or automatically (including via trainable classifiers or SharePoint syntex) to content stored in the SPO site (the channel files – each channel is a folder in the default Documents library). These labels will likely have retention periods that are longer than the retention policy and may include disposition review.
A even more complex model is to apply multiple retention labels to the channel-linked folders (and sub-folders) in the SPO site’s Documents library. This model is fraught with complexity in terms of future disposition review and would be the equivalent of applying retention policies to different folders and subfolders in a network file share.
Pros of applying retention policies (and labels)
Retention policies ensure that content is not destroyed for the period set in the retention policy.
Retention policies are better than auto-expiry because they capture any content that is ‘deleted’ by end-users for the life of the policy. They are better than ‘archiving’ Teams as they set a minimum retention period, protect the content from destruction during that time (‘in place holds’), then destroy the content.
Retention policies could also be used in conjunction with the other two options as necessary. For example, there may be some Teams that contain no records and could simply be deleted via the auto-expiry option. If they contain records, a retention policy will retain the content for as long as required.
Cons of applying retention policies
The main negative of applying retention policies is the complexity of the model, and knowing what has been applied and where. This is especially true if there are many Teams. Consultation and coordinated planning between RM/IM and IT, and documentation of the model, are all essential.
Unfortunately, the Microsoft 365 Compliance admin portal does not provide a single view of what policies have applied where. Unless a third-party application is used, the only way to achieve this is by recording the details of the policies in – say – a spreadsheet or a SharePoint list.
Retention policies do not include the option for disposition review, so records and information managers might need to consider the requirement to find a way to document the disposition (deletion) process and retain a record of what was destroyed.
By actively monitoring Teams, records and information managers should know when the content in Teams is due for destruction, allowing time to extract metadata (where possible) and other information.
All of the above underlines why records and information managers need to know what Teams exist, where the records are stored, and be proactively involved in decisions about what happens to inactive Teams.
As long as retention policies have been correctly applied to the various parts of the Team, that content will be retained for minimum periods. End-users may think they are deleting content, but it remains stored and accessible via a Content Search.
Feature Image Credit: David Yu (image 2081166, via Pexels)
As organisations increasingly move content from network file shares to SharePoint, many have found that folders in SharePoint (and via the Files tab in Teams channels) are a bit less friendly.
For example, there is no way to ‘expand’ the folder view, so it is not possible to see the folder hierarchy (unless the library is synced to File Explorer). And although it is possible to add more or less unlimited metadata to content stored in document libraries, this may be a pointless exercise given that the metadata is not yet visible in File Explorer and folder-based structures are more familiar.
Fortunately, SharePoint not only has the ability to use both folders and metadata, but also to see all the content in a document library without the folders AND using that metadata to group, filter and sort the content regardless of how the folders were structured. This capability can be (surprisingly) useful for records managers.
This post explains how it works, and why this option doesn’t work in the File Explorer view of a synced library.
A typical folder-based library
The screenshot below shows three folders (out of many more) in a fairly typical document library for a SharePoint site.
SharePoint (via a browser) does not show a hierarchical view of folders, so the end-user must click on every folder to find out what they contain. (And yes, you can enable the ‘Item child count’ option but that only tells you how many items are directly under the folder, there is no expanded view).
The ‘Files’ tab in Teams also presents a the same view of the Team’s linked SharePoint site, except that each channel maps to a folder in the default ‘Documents’ library, so that folder (content, if any) is displayed.
SharePoint allows for almost unlimited metadata on document libraries and lists. (Whether you need to add unlimited metadata is a separate question).
Metadata (known as ‘site columns’ or ‘library columns’) may be created at the site or library level, and then added to any library on the site via the Library Settings > Columns > ‘Create column’ OR ‘Add from existing site columns’.
The screenshot below shows an example of a site or library column named ‘Document Type’ to be added to the ‘Meetings’ library. This column has three choice options because there are typically three types of document in this particular library: Agendas, Drafts, and Minutes. You can of course have as many choices as you like but consider the end-user experience. No-one wants to scroll through a hundred choices.
The column above has three choices (selected from a drop down menu), is not mandatory, does not enforce unique values, does not allow end-users to add their own options, and has a default value set. This means that every new document added to the library will be assigned the ‘Agenda’ choice.
Default values may also be useful to automatically add recordkeeping terms such as ‘Function’ and/or ‘Activity’, without the end-user having to do anything.
Once added to the library (via the Library Settings > ‘Create column’ or ‘Add from existing site column’), the metadata column can be displayed in the view, and the metadata options may be added to the documents via ‘Edit in Grid View’ or by adding or changing the metadata on individual documents via the information properties pane on the right.
(Note that there is no option to add or change additional metadata on individual items via the Teams ‘Files’ tab interface.)
But, adding or changing metadata by navigating through the folder structure would be onerous and a waste of time when there is a much better way of doing this – by making the folders disappear.
How to make folders disappear
Every SharePoint document library (including those in sites connected to Teams) has the ability to hide the folders in a document library. This – and the ability to add metadata and more – is achieved via the library view options.
The screenshot below shows the location of default ‘All documents’ view of a library, accessed from the top right of the document library in either SharePoint or via the Files tab in Teams.
Generally the ‘All Documents’ view should be left ‘as is’ and a new view created.
To do this, select ‘Create new view’ or ‘Save view as’ to create the view required. Give the new view a name that reflects what it is, for example ‘NoFolders-Minutes’ (and don’t use a long name or the text may not be visible).
Every view has a range of options, including the following.
Choose columns to be displayed
Select which (system or added) metadata columns are to be displayed in the view.
Select how the content will be sorted. For example, sort by ‘Created’ date.
Select how the content will be filtered by the metadata columns. For example, only show content that meets a specific criteria such as Document Type = ‘Minutes’.
Select how the content will be grouped by the metadata columns. This groups the content by a particular site column. For example, Group by ‘Created by’ or other options. Grouping (and sub-grouping, two levels are allowed) content in this way is often a much better approach than using folders.
The screenshot below shows an example of two-level grouping of metadata.
A bit further down is the ‘magic’ button to make folders vanish – ‘show all items without folders’. This options also hides document sets.
What this looks like
In the document library that was originally folder-based, we have created a new view without any folders. This now allows us to see all the documents in the library without folders. We can see from the view below that some of the items have been mistagged (Minutes with ‘Agenda’ for example). These can now be easily fixed via the ‘Edit in Grid View’ option.
(Note – if the library has more than 5,000 items without folders, the view won’t work and additional filtering may be required; when this happens, press the back space and fix the view, otherwise you may be locked out).
The view can be further modified. In the example below, the view has now been grouped by ‘Created by’ and also filtered to only anything created after 1 August 2021.
Views created in SharePoint are also visible via the Teams Files tab, and can be created in that tab. The screenshot below is of a Teams-based SharePoint library that had multiple folders under the ‘General’ folder and one document (‘Cover Letter’) under General. We can now see that there were in fact no documents in the other folders.
Benefits of views without folders
The primary benefit of creating views without folders is that (subject to the 5,000 item view limit) you can easily see what content (if anything) is hidden in the folder structure of a document library. The metadata that is displayed can be easily exported to a csv or Excel file.
It also makes it easier to apply and/or change metadata; this metadata can in turn be used to sort, filter and group the content in a library.
Note that, if there are more than 5,000 items in the library, additional views can be created to reduce the volume that is displayed at any one time.
As every view has a URL, they can also be added as links in the left hand or page navigation or emailed, making them a kind of pre-canned search.
In the example below, there is only one library ‘Policy Library’; both the ‘Aged Care Policies’ and ‘Due for Review’ are views based on metadata assigned to documents. Additional views (not visible) can group policies based on business areas or relevancy, or various other options. This means that policies (in this case) are not restricted to a single ‘folder’, but can be viewed in multiple ways depending on their context.
Are there any drawbacks doing this?
Aside from the 5,000 view item limit, the other things to be aware of are (a) what happens when the library is synced, and (b) potential confusion.
Libraries (including Teams-based libraries, via the ‘Files’ tab) synced to File Explorer do not currently display any additional metadata added to the SharePoint library, although this may be coming soon. Even if this metadata is displayed, the views are not available. So hiding folders and grouping/filtering may only be useful via the Teams client or browser.
In terms of the potential for confusion, folders on network file shares are very good at hiding duplicates or versions of the same document. When folders are removed from SharePoint, the same document may appear multiple times without any details of the original folder structure or path, resulting in confusion or uncertainty. What if someone always accessed the document via a particular path, then discovers it is missing? How do they know it is hiding in another folder path? If you are going to turn off folders, make sure that end-users about this and how to continue to access their documents via the (preferably unchanged) ‘All documents’ view.
Can we get rid of folders, yet?
Folders have been around for a long time and are a very hard habit to break. They are unlikely to go away any time soon.
However, metadata-based views provide a useful way to ‘see through’ folders and can also be used to apply additional metadata that, in turn, can be used – in conjunction with folders – to group, sort and filter the content hidden away in the folders. Perhaps, in some cases, metadata-based grouping could replace folders, but this will take time and may not be always useful – for example in synced document libraries.
Either way, a combination of folders and metadata can be useful to support recordkeeping activities.
Humans have natural instinct for grouping, classification and categorisation of things. It helps us find what we are looking for and gives us a sense of satisfaction, whether it be household items, computer storage, or much broader social and population groupings.
Humans have created and kept records ever since we developed a way to record them, on stones, clay shards, papyrus, bamboo sheets, velum, paper and various other means. Multiple records were aggregated in ways that made sense to the people who created or kept them and wanted to find them again.
The introduction of computers at work from the late 1980s/early 1990s began the decline of traditional ways of aggregating records about a particular subject together in a physical ‘file’, although that practice has persisted to the present day because it was and still is easier to refer to. Lawyers (or more often the legal clerks) still attend digital courtrooms armed with printed copies of (usually digital) evidence and other materials for this reason.
The ‘problem’ of digital aggregations
While physical files provided the ability to store anything (printable) about a given subject in the one location, digital ‘files’ (or aggregations) suffered from the fact that emails and other content are created or stored in completely different locations.
The only way to keep emails together with other content about the same subject was for end-users to copy them to a network file share folder location or a digital recordkeeping system. In almost every case, the original email remained in the mailbox where it might still have an active life. Some email mailboxes became a primary (or alternative) storage location for both emails and attachments (as did some desktops!).
Keeping all digital records about a given subject in a single aggregation was never an easy task. It was never possible to be sure that everything was captured because it relied on end-users.
The email mailbox – SharePoint conundrum
In the same way that organisations decided to store copies of emails in network file shares or EDRM system, it was easy to see SharePoint as the replacement for both.
But Microsoft have never made it easy to ‘natively’ copy an email from Outlook to SharePoint. There isn’t even a download option for emails. Emails can be dragged and dropped to synced document libraries, and various third-party products exist, but the process usually relies on end-users (a) to copy the emails and (b) to copy them consistently. Neither of these can be guaranteed.
And, of course, the records created and captured in Microsoft 365 is not just in Outlook mailboxes and SharePoint. A number of other apps create content that could records (for example Yammer conversations, Teams chats, calendar entries, Planner tasks, even Whiteboard diagrams). Few of these records can be saved to SharePoint.
So, are digital aggregations impossible?
There is nothing stopping organisations doing whatever they can or want to group related records together. In Microsoft 365, the most logical way to do this is in SharePoint document libraries (the ‘Files’ tab in Teams channels). An entire SharePoint site (the ‘Files’ tab in MS Teams channels) provides a form of meta-grouping; that is, multiple document libraries grouped by the SharePoint site/Team.
But if we stand back for a moment, to look at the (Microsoft 365) forest, what we see is not just individual trees (SharePoint sites, Exchange mailboxes and so on). Just as in a forest the roots of all the trees connect via mycorrhiza networks, sometimes known as ‘wood wide webs’, something similar happens in Microsoft 365 (and many other online systems, including Facebook).
The equivalent of networks in these systems are the ‘graphs’.
Like other graphs, the Microsoft Graph draws on all the rich data created and stored by end-users, in this case across the Microsoft 365 ecosystem – our corporate relationships, who we connect with and how, what we are communicating or writing, what we like, the way we use our time and so on. The graph learns what is popular or trending and makes suggestions (while respecting permissions) as to what we might want to see or know about.
Project Alexandria and Viva
According to a post in the Microsoft Research blog published in April 2021 and titled ‘Alexandria in Microsoft Viva Topics: from big data to big knowledge‘, Project Alexandria is ‘a research project within Microsoft Research Cambridge dedicated to discovering entities, or topics of information, and their associated properties from unstructured documents’.
The blog post also noted that ‘Alexandria technology plays a central role in the recently announced Microsoft Viva Topics, an AI product that automatically organizes large amounts of content and expertise, making it easier for people to find information and act on it’.
The outcomes sound similar to traditional ‘manually’ created aggregations, although they don’t replace them. In fact, the more that content is manually curated, the more likely that Viva Topics can accurately connect them and other related content that might otherwise be missed.
While Viva Topics might appear to primarily focussed on supporting knowledge management outcomes and is currently limited to content stored in SharePoint, the technology has potential implications for records management. In particular, the age-old issue of how to find all information about a given subject (or know that a pre-defined aggregation contains all relevant information).
Viva Topic cards
As noted already, there is nothing stopping organisations from creating aggregations in ways that make sense to them and their end-users. SharePoint document libraries are the most logical form of aggregation that also happen to allow complex metadata, versioning and other features typically associated with EDRM systems. SharePoint document libraries are just one of several ways that content may be aggregated; Exchange mailboxes are another.
But, in most organisations, potentially relevant information AND records is frequently hidden from view in personal mailboxes and OneDrive accounts, in Teams chats, and in other applications (e.g., Planner). Viva Topics has the potential to leverage this information.
While Topics are still limited to SharePoint content and people, there is potential to extend this model even further by including details about emails, chat messages or other content across the Microsoft 365 ecosystem – even if that information cannot be seen. For example:
Suggested people (perhaps grouped by AD manager or business area)
Suggested files and pages (you can see)
Authors of (n number of) emails that are related to the topic with an indication of volume over given periods (e.g., ‘251 emails in the past 6 months’) or a graphic representing this activity
Names of Teams that contain (n number of) chat messages related to the topic.
Participants in Teams 1:1 chats that contain (n number of) messages related to the topic.
Volume and date range of other related content (e.g., Tasks, Whiteboards, Forms, Yammer conversations).
Could Topic cards be the new aggregations?
Topic cards have the potential to resolve the age-old dilemma of digital aggregations, but they are unlikely to replace pre-defined ways to aggregate records including by copying emails to SharePoint document libraries. Those older methods will continue to exist for a long time.
But more importantly, they have the potential to draw out or highlight content that would otherwise be hidden from view – even if that content remains inaccessible.
When configured, Viva Topics already appear in search results, enhancing search outcomes.
It is only a matter of time before the probabilistic programming techniques of Project Alexandria, with expert human curation, begins to provide the type of high precision knowledge base construction for all relevant content about a given subject, first described by Microsoft researchers in May 2019.
Perhaps they may even support or link with retention and disposal processes, highlighting records due for disposal within a given period or even preventing their premature disposal.