Posted in Compliance, Electronic records, Exchange Online, Information Management, Microsoft Teams, Records management, Retention and disposal, Security

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search.

But what happens if there is no Exchange Online mailbox? It seems the chats become inaccessible which could be an issue from a recordkeeping and compliance point of view.

This post explains what happens, and why it may not be a good idea (from a compliance and recordkeeping point of view) not to disable the Exchange Online mailbox option as part of licence provisioning.

Licences and Exchange Online mailboxes

When an end-user is allocated a licence for Microsoft 365, a decision (sometimes incorporated into a script) is made about which of the purchased licences – and apps in those licences – will be assigned to that person.

E1, E3 and E5 licences include ‘Exchange Online’ as an option under ‘Apps’. This option is checked by default (along with many of the other options), but it can be disabled (as shown below).

If the checkbox option is disabled as part of the licence assigning process (not after), the end-user won’t have an Exchange mailbox and so won’t see the Outlook option when they log on to office.com portal. (Note – If they have an on-premise mailbox, that will continue to exist, nothing changes).

Having an Exchange Online mailbox is important if end-users are using MS Teams, because the ‘compliance’ copy of 1:1 chat messages in MS Teams are stored in a hidden folder (/Conversation History/Team Chat) in the Exchange Online mailbox of every participant in the chat. If the mailbox doesn’t exist, those copies aren’t made and so aren’t accessible and may be deleted.

If end-users chat with other end-users who don’t have an Exchange mailbox as shown in the example below, the same thing happen – no compliance copy is kept. The chat remains inaccessible (unless the Global Admins take over the account).

The exchange above, between Roger Bond and Charles, includes some specific key words. As we will see below, these chats cannot be found via a Content Search.

(On a related note, if the ability to create private channels is enabled and they create a private channel and chat there, the chats are also not saved because a compliance copy of private channel chats are stored in the mailboxes of the individual participants.)

Searching for chats when no mailbox exists

As we can see above, the word ‘mosquito’ was contained in the chat messages between Roger and Charles.

Content Searches are carried out via the Compliance portal and are more or less the same as eDiscovery searches in that they are created as cases.

From the Content Search option, a new search is created by clicking on ‘+New Search’, as shown below. The word ‘mosquito’ has been added as a keyword.

We then need to determine where the search will look. In this case the search will look through all the options shown below, including all mailboxes and Teams messages.

When the search was run, the results area shows the words ‘No results found’.

Clicking on ‘Status details’ in the search results, the following information is displayed – ‘0 items’ found. The ‘5 unindexed items’ is unrelated to this search and simply indicates that there are 5 unindexed items.

Double-checking the results

To confirm the results were accurate, another search was conducted where the end-user originally did not have a mailbox, and then was assigned one.

If the end-user didn’t have a mailbox but the other recipient/s of the message did, the Content Search found one copy of the chat message in the mailbox of the other participants. Only one item was found.

When the Exchange Online option was enabled for the end-user who previously did not have a mailbox (so they were now assigned a mailbox), a copy of the chat was found in the mailbox of both participants, as shown in the details below (‘2 items’).

Summary and implications

In summary:

  • If end users chat in the 1:1 area of MS Teams and don’t have an Exchange Online mailbox, no compliance copy of the chat will be saved, and so it will not be found via Content Search.
  • If any of the participants in the 1:1 chat have an Exchange Online mailbox, the chat will appear in the mailboxes of those participants.
  • If all participants in the 1:1 chat have an Exchange Online mailbox, the chat will be found in the mailbox of all participants.

Further to the above:

  • If end users can delete chats (via Teams policies) and don’t have a mailbox, no copy of the chat will exist.
  • If end-users with a mailbox can delete Teams chats, but a retention policy has been applied to the chats, the chats will be retained as per the retention policy (in a hidden folder).

And finally, if you allow private channels, end-users can create private channels in the Organisation Team. The chats in these private channels are usually stored in the personal mailboxes of participants (not the Group mailbox) – so these chats will also be inaccessible and cannot be found via Content Search.

The implications for the above are that, if you need to ensure that personal chat messages can be accessed (from Content Search), then the participants in the chat must have an Exchange Online mailbox.

Further, if you allow deletion of chats but need to be able to recover them for compliance purposes, a retention policy should be applied to Teams 1:1 chat.

Posted in Electronic records, Information Management, Microsoft 365, Records management, Retention and disposal

A modern way to manage the retention of digital records

In his April 2007 article titled ‘Useful Void: The Art of Forgetting in the Age of Ubiquitous Computing’ (Harvard University RWP07-022), Viktor Mayer-Schönberger noted that the default human behaviour for millenia was to forget. Only information that needed to be kept would be retained. He noted that the digital world had changed the default to remembering, and that the concept of forgetting needed to be re-introduced through the active deletion of digital content that does not need to be retained.

The harsh reality is that there is now so much digital information in the world, including digital content created and captured by individual organisations, that active deletion of content that does not need to be retained, seems an almost impossible task.

This post explores issues with the traditional model of records retention in the digital world, and why newer options such as the records retention capability of Microsoft 365 is a more effective way to manage the retention and disposal of records, and all other digital content.

The traditional retention model

The traditional model of managing the retention and disposal/disposition of records was based on the ability to apply a retention policy to a group or aggregation of information identified as records. For the most part, those paper records were the only copy that existed (with some allowance for working and carbon copies).

The model worked reasonably well for paper records, but started to falter when paper records became the printed versions of born-digital records, and where the original digital versions remained where they were created or captured – on network files shares, in email systems, and on backups. Although, technically, the official record was on a file, a digital version was likely to remain on network file shares or in an email mailbox after the paper version was destroyed at the end of the retention period, and remain overlooked.

How many of us have had to wade through the content of old network file shares to examine the content, determine its value, and perhaps see if it can even still be accessed? Or do the same with old backup tapes?

The volume of unmanaged digital content, not subject to any retention policy, only continued to increase. This situation continued to worsen when electronic document and records management (EDRM) systems were introduced from the late 1990s. End-users had to copy records to the EDRMS, thereby creating yet another digital copy, in addition to the born-digital originals stored in mailboxes or file shares.

Even if the record in the EDRMS were destroyed, there was a good chance the original ‘uncontrolled’ version of the digital record – along with an unknown volume of digital records that probably should have been consigned to the EDRMS but weren’t – remained in email mailboxs, on file shares, or on a backup tape somewhere.

eDiscovery was born.

The emergence of new forms of digital records, including instant messages, social media, and smart-phone based chat and other apps from the early 2000s only added to the volume of digital content, much of which was stored in third-party cloud-based and mobile-device accessible applications completely out of the reach and ability of the organisation trying to manage records.

Modern retention management

A modern approach to retention management should be based on the following principles:

  • Information, not just records, should only be kept for as long as it is required.
  • It is no longer possible to accurately and/or consistently identify and capture all records in a single recordkeeping system.
  • Duplication of digital content can be reduced by creating and capturing records in place, promoting ‘working out loud’, co-authoring and sharing (no more attachments and private copies).

None of the above points excludes the ability to manage certain types of records at a more granular level where this is required. But these records, or the location in which they are created or captured, should not be regarded as the only form of record.

Ideally, these records should be created (or captured) directly in the system where they are to be managed – not copied to it.

Change management is necessary

Some of these new ways of working are likely to come up against deeply ingrained behaviours, many of which go back several decades and have contributed to a reluctance to ‘forget’ and destroy old digital content, including:

  • hiding/hoarding content in personal drives (and personal cloud-based systems or on USB drives);
  • communicating by email, the content is which is inaccessible to anyone else;
  • attaching documents to emails;
  • printing and filing born-digital content; and
  • sometimes, scanning/digitising the printed copies of born-digital records and saving them back to a digital system.

What about destruction?

Records managers in organisations moving away from the authorised destruction of digital content identified as records, to the destruction of all digital content (including identified records) need to consider what is required to achieve this outcome, and the implications for existing process and practices (including those described above).

  • Some activities will remain unchanged. For example, the need to review certain types of records before they are destroyed (aka ‘disposition review’), to seek approval for that destruction, and to keep a record of what was destroyed.
  • Some activities are new and can replace other existing actions and activities. For example, the application of retention policies to mailboxes can remove the requirement to backup those mailboxes.
  • Some of activities or outcomes may be challenging. For example, the automatic destruction without review of digital content that is not the subject of more granular retention requirements, such as emails out of mailboxes, documents in personal working drives. This content will simply disappear after the retention period expires.

How Microsoft 365 can support modern retention management

Microsoft recognised some time ago that it was becoming increasingly difficult to manage the volumes and types of digital content that was being created every day by organisations.

Exisiting and newly released functionality in the Compliance portal of Microsoft 365 includes the ability to create and apply both label-based retention policies to specific types of records, including automatically based on machine learning capabilities, and broader ‘workload’ specific (e.g., mailboxes, SharePoint sites, OneDrive accounts, MS Teams chats) retention policies. This capability helps organisations to focus retention requirements on the records that need to be retained, while destroying digital content that is no longer relevant and can be forgotten.

Instead of directing end-users to identify records and copy them from one system to another (thereby creating two versions), Microsoft 365 allows end-users to create and capture records in place, providing a single source of truth that can be shared (rather than attached), be the subject of co-authoring, and protected from unauthorised changes (and even downloads).

Limitations with Microsoft 365

It is important to keep in mind that there are some limitations with the current (October 2020) retention capability in Microsoft 365.

  • Retention and disposal is based on individual digital objects, not aggregations. There are limited ways to group individual records by the original aggregations in which they may have been stored (e.g., document libraries in SharePoint).
  • Only the (minimal) details of records that were subject to a disposition review are recorded in the ‘disposed items’ listing, and this is only kept for a year (but can be exported). No record is kept of any other destroyed record, except in audit logs (for a limited period).
  • The metadata details of records subject to a disposition review that were destroyed is minimal – the document type and name, date destroyed, destroyed by whom.
  • When records are destroyed from SharePoint document libraries or lists, the library or list remains with no record kept of what was previously stored there. It is not possible to leave a ‘stub’ for a destroyed record.

Summing up

The primary outcome from introducing modern ways to manage retention will be that all digital content, not just content that has been identified as records or copied to a recordkeeping system, will be subject to some form of retention and disposal management.

In other words, a change from exception-based retention (where all the other digital content is overlooked), to a more holistic method of retention with both granular controls on certain types of records where this is required, and broader retention capability allowing us to forget the content that is no longer relevant – the ‘redundant, trivial and outdated’ (ROT) content often scattered across network file shares.

Posted in Electronic records, Information Management, Microsoft 365, Microsoft Teams, Records management, Retention and disposal, SharePoint Online

The Microsoft 365 experience – Teams, Exchange, Outlook, Edge: Where did SharePoint Go?

At the 2020 Microsoft Ignite conference, Jeff Teper presented a diagram titled ‘Microsoft 365’. The diagram showed only four icons: Teams, Outlook, Office and Edge.

The implication of this diagram was that, for most end-users, Teams is now (or will become) their primary portal into Microsoft 365. As stated by Jeff Teper, SharePoint is a foundation platform, the out of sight content engine. Edge’s ability to serve up search results from Microsoft 365 further reduces the need to go to SharePoint.

So, what are the implications for managing records?

SharePoint as a recordkeeping system

For a long time, records have been created, captured and stored in recordkeeping systems.

In the paper world, the recordkeeping system consisted of paper records stored in files and boxes and detailed in registers. With the introduction of computers in the 1980s, registers were transferred to databases, making it a bit easier to find records. In the late 1990s, recordkeeping databases were linked with (separate) file stores and became electronic document and records management (EDRM) systems that continued to manage paper records (the so-called ‘hybrid’ systems).

For almost a decade (since SharePoint 2010 was introduced), SharePoint has contended with files shares and EDRM systems as an alternative recordkeeping system, providing almost all the same core functionality.

The ability to create a record in a single location, then share and co-author it from that location, has completely removed the requirement to copy a record to a separate recordkeeping system.

And then came Teams

Someone at Microsoft had incredible foresight to see the potential for a new user interface that would replace products like Lync and Skype for chat and conferencing, and would also provide access to files stored in SharePoint.

SharePoint has been a core part of the Microsoft productivity offerings for a very long time and people have built careers around developing functionality on the SharePoint platform to appeal to end-users, the intranet being the most common case in point, with customised team sites close behind.

The arrival of Microsoft 365 Groups and then Teams in 2017 was perhaps not widely noticed. One could argue that end by the beginning of 2020, it was still largely unnoticed.

And then came a pandemic and working from home. Teams – which may have been largely ignored or overlooked until then – was already ready to take its place next to Outlook, Office and Edge as a primary end-user interface.

New Teams were created, sometimes with abandon (and were sometimes just as quickly abandoned).

Both 1:1 (or 1:many) chats and channel chats took off. Files were created and shared via OneDrive for Business (‘Files’ in the 1:1 chat area), or via the back-end SharePoint sites (‘Files’ in the channel chat area).

There was (and maybe still is) a belief that files were being saved to Teams but not SharePoint. ‘We are storing everything in Teams’ was not an uncommon expression, sometimes followed by ‘but we’re not using SharePoint or OneDrive’.

The year 2020 saw a huge increase in the volume of records stored in SharePoint sites linked with Teams, as well as a completely new set of records – chats (‘compliance’ copies of which are stored in Exchange mailboxes).

The diagram below provides an overview of the relationship between Teams, Microsoft 365 Groups, Exchange mailboxes, SharePoint and OneDrive for Business.

What about SharePoint?

As the diagram above shows, SharePoint has not disappeared. Many organisations will continue to use, and ask end-users to access, SharePoint sites directly to store and manage records.

But accessing SharePoint from SharePoint may become less necessary over time. At Ignite 2020, the ability to pin a ‘home site’ (such as an intranet) to Teams was demonstrated. Even the intranet may end up in Teams.

As Jeff Teper said, SharePoint is a foundation platform, one that does not get in the way of collaboration and productivity but powers it.

Implications for records managers

Records managers, who were likely already on a steep learning curve regarding SharePoint, need to continue to improve their knowledge of the SharePoint platform. On a positive note, SharePoint Online is a much easier application to learn and manage, compared with its earlier on-premise predecessors.

In organisations that have been using SharePoint for a while and/or have allowed the free-creation of Teams in MS Teams, there will some requirement for retrospective analysis, review, and cleaning up.

In all organisations, there will be a requirement to establish some form of governance and oversight of records (files and chats) that have been created, including for the purpose of retention and disposal/disposition.

Retrospective implications

Where MS Teams has been implemented with little thought given to naming conventions, SharePoint site provisioning, or access controls, records managers should been given access to and review the list of all SharePoint sites that have been created, including from MS Teams. This will provide an initial idea of the volume of content and activity on each site, and what action needs to be taken on things like inactive Teams.

Ideally, records managers should be added to the Site Collection Administrators (SCA) group of every SharePoint site, including MS Teams-based sites. This action will give records managers access to the content on every site and to help advise on the management of records in those sites (including Team-based sites).

  • The best way to do this is to add records managers to a Security Group and then add that Group to the SCA group of every site. This access could be deferred for sites that contain very sensitive information, although typically records managers would have access to all records, including if they had an EDRMS. And, access is always recorded in audit logs or the local site ‘viewers’ (where enabled) and ‘last modified by’ information.

Access to the chat content of Teams (including 1:1 chats) will not normally be required; some understanding of the content could be inferred from the name of the Team or the SharePoint content. If necessary, Global Admins or a Compliance Admin can run a Content Search across Teams to find chat content, and/or export that content by an individual person or subject.

Records managers will also need to advise on the appropriate retention policy or policies that need to be created and then applied to:

  • The chat content in 1:1 chats.
  • The chat content in the various Teams.
  • SharePoint sites linked with Teams.
  • Exchange mailboxes.
  • OneDrive for Business accounts. An additional consideration is how long the content of inactive ODfB acccounts should be retained via the ‘Storage’ policy (default is 30 days then permanent deletion).
  • SharePoint sites not linked with MS Teams. This includes whole sites as well as library-based retention policies.
  • Office 365 Groups (mailbox/SharePoint site). If linked with a Team, a second retention policy is required for the Team chat content retention (second dot point above). For example, one policy ‘GroupABC’ and a second policy ‘GroupABCTeamChat’.

As many of the above retention policies replace the need for backups, records managers need to discuss the options with their IT colleagues.

Forward looking implications

Ideally, there should be some form of governance around the creation of new Teams in MS Teams. These governance arrangements might include:

  • The necessary access for records managers. For example, Site Collection Administrator on every site, and/or a customised Compliance Admin role to create and access retention policies.
  • Controls around the creation of new Teams, including naming conventions. If not controlled, what processes will ensure that records are properly managed.
  • Retention implications. For example, can the new site and/or the channel chat content be covered by another retention policy – e.g., ‘All Teams with assessed low-level working content should be kept for 5 years’.
  • Simple best practice guidance for all new users, including on how to share and co-author.
  • Retention policies for all Microsoft 365 content, not just SharePoint.
  • Reviews of the content of OneDrive for Business accounts of departed end-users, especially for people in senior or decision making positions. It is relatively common practice for end-users to delete (and download) this content before they leave their jobs.
  • Monitoring and oversight of content, including access to reporting dashboards.

So, is Microsoft 365 just Teams, Outlook and Office (in Edge)?

Perhaps, yes.

For many, or not most information based end-users, MS Teams is likely to become the primary interface to Microsoft 365 collaboration team spaces including SharePoint and OneDrive. Just like Outlook, Teams will probably be left open all day.

In theory, the volume of low-value emails, and emails with attachments, should reduce over time.

The developing role of records managers

In this new world, the role of records managers will change from being the curators of records copied to and stored in a separate ‘records and document management’ system, to being records compliance analysts or perhaps, corporate knowledge and information managers and content analysts.

They will learn what the Graph can do, and help to guide AI tools including machine learning and machine teaching, Project Cortex and SharePoint Syntex. They will be responsible for monitoring content across the Microsoft 365 platform, creating and applying retention policies and managing the outcome of those policies, working more interactively with the Graph, and with a range of data.

In organisations that have a requirement to transfer records to archival institutions, the new knowledge and information managers will have a key role in ensuring that this data is suitable for transfer.

They might even have oversight of old paper records gathering dust until they can be destroyed.

Posted in Compliance, Governance, Information Management, Microsoft 365, Records management, Retention and disposal

Recordkeeping roles and permissions in Microsoft 365

(Updated 3 September 2020 with reference to customised admin roles)

Microsoft 365 is a cloud-based collaboration and content system that includes a wide range of functionality to create, capture and manage records, primarily in SharePoint Online but also in OneDrive for Business, Exchange Online and in MS Teams. 

This post outlines the roles and permissions required by records managers to manage records in Microsoft 365.

Whether all the roles and permissions will be granted may depend on a number of factors including technical competence, security and risk. Where they are not granted, records managers will need to ensure that the relevant IT resources can and will set up and manage the recordkeeping functionality as required.

Azure AD/Microsoft 365 Admin Center roles

There are around 50 roles that can be assigned to individuals in the Microsoft 365 admin center or the Azure Admin portal (which includes 11 more roles).

These roles may be grouped as follows:

  • Admin. For example, Global Admin and the Admins for Exchange Online, MS Teams, and SharePoint Online/OneDrive for Business.
  • Security and Compliance. For example, Security Admin, Compliance Admin, Compliance Data Admin
  • Identity management. For example, Authentication Admin, Guest Inviter, Licence Admin, Password Admin, User Admin
  • Device management. For example, InTune Admin, Printer Admin
  • Reader. For example, Global Reader, Message Center Reader, Reports Reader, Security Reader

There is no specific ‘records manager’ role in Microsoft 365. The closest in terms of functionality is the Compliance admin role that includes several several sub-roles including ‘RecordManagement’, ‘Disposition Management’ and ‘Retention Management’. Alternatively, a custom role may be created with those (and a couple of other) sub-roles, thereby restricted access to only the sub-roles that are specific to or required by records managers.

In addition to the role and sub-roles required to access the Compliance portal and carry out records management activities, records managers should also be assigned the Global Reader and Reports Reader roles so they can access and view the various dashboards on the Microsoft 365 admin center:

Example dashboard

Compliance admin portal roles and sub-roles

The Compliance admin portal (https://compliance.microsoft.com) includes the following sections that are all relevant for records managers:

  • Reports (dashboard)
  • Audit logs. These cover the entire Microsoft 365 environment, kept for only 3 months (E3) or 12 months (E5).
  • Content Search (effectively eDiscovery)
  • Information Governance (where retention labels and retention policies are created and managed)
  • Records Management (which is essentially an extended set of IG functionality, including auto-application of labels, available to E5 licence holders, and disposition management)

Access to the Compliance admin portal is restricted to the Global Admins and Compliance Admin and Compliance Data Admin roles. These two roles include various sub-roles (including sub-roles that are not relevant to records management) that are described in considerable detail in this Microsoft page ‘Permissions in the Security & Compliance center‘.

The sub-roles that are most relevant to records managers are:

  • RecordManagement (required to manage and dispose record content)
  • Retention Management (required to create retention labels)
  • Audit Logs
  • View-Only Audit Logs (audit logs cannot be modified)
  • Disposition Management (required to manage disposition)
  • Compliance Search (required to conduct a global ‘case’ search of anything anywhere in the Microsoft 365 platform, including ‘personal’ mailboxes and 1:1 Teams chats)
  • Case Management
  • Hold

It is recommended that records managers – or select individuals with higher compliance responsibilities, be assigned either to one of the two Compliance Admin roles, or a custom role group with just the sub-roles listed above. This will enable records managers to access the Compliance portal to create, apply and manage records retention policies. They will also have access to the audit logs and content search options. 

Note: The ‘Audit logs’ sub-role is actually assigned via a role group in the Exchange Online admin portal under the Permissions section. The three key roles in this section that contain these sub-roles are ‘Organisation Management’, ‘Compliance Management’ and ‘Records Management’. As the first two contain a very long list of sub-roles, it is recommended that the records manager/s be added to the ‘Records Management’ role group that includes the ‘Audit logs’ and ‘Retention Management’ sub-roles.

SharePoint Admin roles

From an admin point of view, there are essentially three SharePoint admin roles:

  • SharePoint administrator. This person has access to the SharePoint admin portal, manages the settings, creates and provisions new sites, and monitors the environment. They are usually also responsible for troubleshooting issues and may have some responsibility for development (including scripts) and customisations or integrations. Subject to the size and complexity of the environment, a records manager with good technical skills, including being an EDRM system admin, may be able to take on the role of SharePoint admin with some training. In most cases, however, this is likely to remain a specialised IT role.
  • Site Collection Administrator. This role sits between the SharePoint Admin and the Site Owner role and provides ‘back-end’ access to the SharePoint site. Generally speaking, the SharePoint Admin will always be a Site Collection Administrator, ideally added via an AD Security Group. If records managers are added to this AD Security Group, and that Group is added to the Site Collection Admin section of every SharePoint, they will have the ability to access every site (with all access and actions recorded in the audit logs). This access can be revoked on individual sites if necessary. 
  • SharePoint Site Owner. The person assigned to this role will usually be someone working in the business area or group responsible for day to day management of the site. Records managers should not be Site Owners as this suggests that the records managers have day to day responsibility for managing the site (creating libraries for example).

Other factors to consider

Any content stored in OneDrive for Business accounts, Exchange mailboxes and MS Teams will remain accessible via a Content Search as long as it exists. If no retention policy has been applied to these workloads and the end-users deletes that content, there is no way to retrieve the deleted content after minimum periods (90 days for ODfB, 14 days for Exchange mailbox content).

The OneDrive portal includes a Storage section that determines how long the content will be retained after the account becomes inactive. This is separate from any retention policy that may be applied to the accounts via the Compliance portal. Records managers should understand these two elements (retention and storage period).

The Exchange Online admin portal includes a number of legacy recordkeeping elements, in particular the Messaging Records Management (MRM) policies in the compliance/retention policies section. Records managers do not need to be assigned the role of Exchange Online admin but need to engage with the admins regarding the application of Microsoft 365 retention policies. While it is possible to apply label-based retention policies to Exchange mailboxes, including advanced auto-application with E5 licences, in practice it may be much simpler to apply a few broad non-label retention policies to mailboxes.

Screenshot of the MRM policy area

The MS Teams admin portal does not include any recordkeeping settings or elements. However, the records manager should discuss and determine suitable retention requirements for both 1:1 chats and channel chats with the relevant admin. These are created and added via the Compliance admin portal. It is not possible to apply a label-based retention policy to Teams chats, accordingly there is (currently) no disposition review record of what was destroyed.

Conclusion

Records managers need an appropriate level of access to the Microsoft 365 ecosystem to manage records that have been created, captured and stored across the system. The following is recommended:

  • Global reader and Reports reader. These two roles provide read-only access to dashboards in the Microsoft 365 Admin portal, allowing records managers to review volumes and activities in the various workloads. 

  • Compliance admin or a customised role group. The role group allows the creation and management of records retention policies and dispositions. It also provides access to audit logs and global content searches. 

  • SharePoint admin (optional). This role would be suitable for a records manager with the required level of technical competence to manage SharePoint. 

  • SharePoint Site Collection Admin (via a Security Group). This role allows records managers to access every site where the Security Group has been added to the Site Collection Admin group. 

 

 

Posted in Compliance, Data Loss Prevention - DLP, Electronic records, Governance, Information Classification, Information Management, Information Security, Legal, Microsoft 365, Records management, Retention and disposal, SharePoint Online

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

  • Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
  • Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.

Posted in Classification, Compliance, Electronic records, Governance, Information Management, Microsoft Teams, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

msteamschatteams-1

There are two types of chat message in MS Chat:

  • One-to-one/one-to-many ‘chat’ (top icon above).
  • Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

  • Knowing if chat messages on any given subject exist. 
  • Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

ContentSearchQuery

Searches can be configured to find content in any or all of the following locations:

  • Users, Groups, Teams
    • Exchange email
    • Office 365 group email
    • Skype for Business
    • Teams messages [the copy in the mailbox]
    • To-Do
    • Sway
    • Forms
  • SharePoint
    • SharePoint sites
    • OneDrive accounts
    • Office 365 group sites
    • Teams sites
  • Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

  • As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
  • There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search. But what happens if there is no Exchange Online mailbox? It seems the chats become […]

A modern way to manage the retention of digital records

In his April 2007 article titled ‘Useful Void: The Art of Forgetting in the Age of Ubiquitous Computing’ (Harvard University RWP07-022), Viktor Mayer-Schönberger noted that the default human behaviour for millenia was to forget. Only information that needed to be kept would be retained. He noted that the digital world had changed the default to […]

Posted in Classification, Electronic records, Information Management, Office 365, Records management, Retention and disposal

Planning for retention management in Microsoft 365

Fools rush to implement retention without thought‘ – Tony Redmond, 13 April 2017

Tony Redmond’s quote above, as well as the rest of the article in ‘Bringing Compliance to Office 365 Groups‘, is as relevant today as it was in 2017.

Tony is a contributing author to the e-book ‘Office 365 for IT Pros‘, essential reading for anyone doing anything with Microsoft 365. Page 921 of the May 2020 edition contains the following paragraph, which expands on the quote above and contains probably the best guidance ever required in relation to this subject:

It is sensible to write down each of the retention labels that you plan to use before creating anything. It is much easier to delay the release of a label and the training of users to use the label properly than it is to launch a label into general circulation only to discover that you later need to withdraw it. Another thing to consider is how easy it is for users to decide between different retention labels when the time comes for them to apply a label. Too many labels, misleading names, or too much choice can lead to frustration and bad decisions.

How do you go about writing down each of the retention labels as part of a plan – especially for a Microsoft 365 environment that is already in full swing?

This post provides some suggestions to help you do this.

What is your records retention and disposal status?

A good starting point is to establish the current records retention and disposal status for your organisation. Do you have a records retention schedule, also known as a disposal authority or records authority? 

If you have one of these documents, it would be useful to review it as a key part of the process is to ‘map’ the records retention classes to specific records across the various Microsoft 365 ‘workloads’ (e.g., Exchange, SharePoint, OneDrive, MS Teams etc), not just in one system (such as SharePoint).

You will need to know what and where these workloads are.

Where (and what) are the records in Microsoft 365?

If you are a records manager then there is a reasonably good chance that you have very little access to, or visibility of, all the content stored across Microsoft 365.

You may have access to one or more SharePoint sites, but unless you are a SharePoint Admin or Site Collection Admin on every site, your visibility will be very limited.

Most of the records in Microsoft 365 will be stored in Exchange, SharePoint, OneDrive for Business, or MS Teams.

  • Emails created and sent by users are stored in Exchange mailboxes. There may also be public mailboxes. Unless there is a plan (or third-party app) to copy these (or some of these) emails out of Exchange (e.g., to SharePoint), most email records will probably remain in user’s mailboxes.
  • Records that, in the past, would have been saved to a network file share (or EDRMS) will now be in SharePoint Online (corporate content) or OneDrive for Business (ODfB) (personal/working content).
  • Chat messages in MS Teams are stored in a hidden area of the Exchange mailbox of each user who participates in the chat. Any documents shared in this chat area are stored in the OneDrive for Business of the person who shared the document.
  • Channel-based Team chat messages in MS Teams are stored in a hidden area of the Exchange mailbox of the Office 365 Group linked with the Team. Any documents shared in this chat area are stored in the SharePoint site of the Office 365 Group linked with the Team.

So, fundamentally, records are stored in two primary workloads: Exchange mailboxes and SharePoint/OneDrive for Business.

What are the retention options?

There are two retention options in Microsoft 365. Both are configured in the Compliance portal of Microsoft 365. Access to this portal requires special privileges, which may not always be granted to records managers.

The two options are:

  • Retention labels published as retention policies and then applied to the various workloads (Exchange email, SharePoint, OneDrive, Office 365 Groups (Exchange/SharePoint content)). These are sometimes described as ‘explicit’ policies because they are visible to end users. Organisations with an E5 licence can extend the way these labels are applied and retention managed.
  • Retention policies that are applied directly to the various workloads (Exchange email, Exchange public folders, SharePoint, OneDrive, Office 365 Groups (Exchange/SharePoint content)). These are sometimes described as ‘implicit’ policies because they are not visible to end users. These policies automatically delete content at the end of a retention period, without any review possible.

Records managers will need to determine how to ‘translate’ each records retention class into one of the two options above, and how and where it will be applied in Microsoft 365.

Some of the options may also require the creation of new records retention classes – for example for the chat element in Microsoft Teams.

A suggested first model

Exchange mailboxes

Your IT probably already has some form of back-up regime (‘archive’) for mailboxes, used for disaster recovery and investigation purposes.

It might be worth creating two policies for mailboxes:

  • All end-user mailboxes could have a single ‘implicit’ retention policy (e.g., 7 years).
  • Mailboxes for specific staff (e.g., senior managers) could have a second, longer, ‘implicit’ retention policy. This policy will take over when the first one expires, but just for the mailboxes identified.

The use of retention policies in this way can replace the need for mailbox backups. No emails will ever actually be deleted while the retention policy is in place and all content can be retrieved via the Content Search option in the Compliance Portal. 

Content Searches can also be used to retrieve and export emails.

OneDrive for Business

As with end-user mailboxes, OneDrive for Business accounts are generally inaccessible to records managers. To ensure that the content in those accounts is not deleted, a single Microsoft implicit retention policy of, say, 7 years could be applied to all ODfB accounts.  This policy will create a hidden (to the user) ‘Preservation Hold’ library on the ODfB account.

Anything ‘deleted’ by the end user during the retention period will be moved to the Preservation Hold library, which is visible to the Global Admins and SharePoint Admins from this URL – /_layouts/15/viewlsts.aspx?view=14

In addition the OneDrive settings include the option (under ‘Storage’ in the ODfB admin portal) to retain OneDrive accounts for a period of time after they are inactive.

All content in these locations is accessible from a Content Search.

SharePoint

SharePoint is likely to be the most complicated in terms of retention policies if there is a requirement to keep content for different periods of time in accordance with the retention schedules/records disposal authorities.

There are likely to be three main options in relation to SharePoint content:

  • One or more implicit retention policy/ies applied to one or more sites. When applied to a SharePoint site, a ‘Preservation Hold’ library retains anything that is ‘deleted’ by end users.
  • One or more explicit label-based retention policies applied to one or more sites. When applied to a SharePoint site, the option to apply it appears for each document library on the site. Once applied (manually), end users cannot delete anything and if the library is synced to File Explorer, the File Explorer view of the library will be read only.
  • A combination of implicit and explicit retention policies.

The decision to apply what policy to what site will depend on your SharePoint architecture and the content stored in each site. For example:

  • A SharePoint site that only stores records that map to one records retention class could have either a single implicit policy (if there is no requirement for disposal review) or a single explicit policy that is applied manually to each library.
  • A SharePoint site that contains records that map to multiple retention classes, but for one business function and also ‘working papers’ could have (a) one implicit policy to cover the working papers and (b) one label-based retention policy with multiple labels – one for each class. This means, for (b), that a specific retention label can be applied to each library as required.
  • SharePoint sites linked with Office 365 Groups and Teams. Depending on the content in the site, it may be possible to apply a single retention policy for all M365 Groups (which covers both the SharePoint site and the mailbox), or a similar policy created for a Group of SharePoint sites (which excludes the mailbox).

MS Teams

As noted above, the chat content in MS Teams is stored in Exchange mailboxes – (a) the mailbox of each participant for one-to-one chat, and (b) the mailbox of the Office 365 Group for channel-based chat.

You may consider having a relatively short-term retention period for one-to-one chat. The retention period for the channel based chat will depend on the subject matter and should – ideally – be the same as for the linked SharePoint site. For example:

  • A Team set up for a specific business function and activity (or activities) will have channel based chat and a linked SharePoint site. Both should be subject to the same retention period.
  • A Team set up for low-level discussion about a subject that may be not be covered by any retention period could be subject to a general retention policy for the chat and the SharePoint content.

Bringing it together

As noted at the beginning of the post, if you are going to use retention policies in Microsoft 365 you need a plan and you need to document it. It doesn’t matter too much if the environment is already active.

However, you will need to have discussions with your Microsoft 365 Global Admins, Compliance Admins and SharePoint Admins and know where the content is stored.

  • The Global Admins can give you a list of every Office 365 Group and Team in MS Team (these are connected – every Team is based on an O365 Group).
  • The SharePoint Admins (or Global Admins) can give you a list of every SharePoint site.

There are some potential ‘quick wins’, such as agreement with IT regarding Exchange mailboxes, OneDrive for Business accounts, and MS Teams.

The more complex requirement is to map the classes in your records retention schedules/disposal authority to content stored in SharePoint, including for standard sites (not linked with Microsoft Groups), communication sites, and sites linked to Office 365 Groups.

You can start to do this by having a list of all the sites exported from the SharePoint Admin portal. This should allow you to see how many sites exist, how much content they hold, and if they are active or not.

It is probably a good idea for the records manager to be included as a Site Collection Administrator, including by being a member of a Security Group added to every SharePoint site. This will help the records manager gain visibility of the content of each site, however they should be very careful about browsing the content as everything is recorded in audit logs.

Document and plan

The outcome of all these actions should be one or more documents that describe (a) where records are stored and (b) the retention policy and action that will apply to those records.

  • For Exchange mailboxes, OneDrive for Business accounts, and MS Teams, this may be a single line for each policy.
  • For SharePoint, there should be a listing of every site and the retention policy or policies that apply to that site.
  • Additionally, for SharePoint sites where an explicit label-based retention policy is applied, the listing should show which libraries this has been applied to. If a disposal review option has been selected, there should be a process to ensure that the metadata of the library where the records are stored is exported and stored in a different location. The original library may then be deleted.