Posted in Classification, Electronic records, Information Management, Office 365, Records management, Retention and disposal

Planning for retention management in Microsoft 365

Fools rush to implement retention without thought‘ – Tony Redmond, 13 April 2017

Tony Redmond’s quote above, as well as the rest of the article in ‘Bringing Compliance to Office 365 Groups‘, is as relevant today as it was in 2017.

Tony is a contributing author to the e-book ‘Office 365 for IT Pros‘, essential reading for anyone doing anything with Microsoft 365. Page 921 of the May 2020 edition contains the following paragraph, which expands on the quote above and contains probably the best guidance ever required in relation to this subject:

It is sensible to write down each of the retention labels that you plan to use before creating anything. It is much easier to delay the release of a label and the training of users to use the label properly than it is to launch a label into general circulation only to discover that you later need to withdraw it. Another thing to consider is how easy it is for users to decide between different retention labels when the time comes for them to apply a label. Too many labels, misleading names, or too much choice can lead to frustration and bad decisions.

How do you go about writing down each of the retention labels as part of a plan – especially for a Microsoft 365 environment that is already in full swing?

This post provides some suggestions to help you do this.

What is your records retention and disposal status?

A good starting point is to establish the current records retention and disposal status for your organisation. Do you have a records retention schedule, also known as a disposal authority or records authority? 

If you have one of these documents, it would be useful to review it as a key part of the process is to ‘map’ the records retention classes to specific records across the various Microsoft 365 ‘workloads’ (e.g., Exchange, SharePoint, OneDrive, MS Teams etc), not just in one system (such as SharePoint).

You will need to know what and where these workloads are.

Where (and what) are the records in Microsoft 365?

If you are a records manager then there is a reasonably good chance that you have very little access to, or visibility of, all the content stored across Microsoft 365.

You may have access to one or more SharePoint sites, but unless you are a SharePoint Admin or Site Collection Admin on every site, your visibility will be very limited.

Most of the records in Microsoft 365 will be stored in Exchange, SharePoint, OneDrive for Business, or MS Teams.

  • Emails created and sent by users are stored in Exchange mailboxes. There may also be public mailboxes. Unless there is a plan (or third-party app) to copy these (or some of these) emails out of Exchange (e.g., to SharePoint), most email records will probably remain in user’s mailboxes.
  • Records that, in the past, would have been saved to a network file share (or EDRMS) will now be in SharePoint Online (corporate content) or OneDrive for Business (ODfB) (personal/working content).
  • Chat messages in MS Teams are stored in a hidden area of the Exchange mailbox of each user who participates in the chat. Any documents shared in this chat area are stored in the OneDrive for Business of the person who shared the document.
  • Channel-based Team chat messages in MS Teams are stored in a hidden area of the Exchange mailbox of the Office 365 Group linked with the Team. Any documents shared in this chat area are stored in the SharePoint site of the Office 365 Group linked with the Team.

So, fundamentally, records are stored in two primary workloads: Exchange mailboxes and SharePoint/OneDrive for Business.

What are the retention options?

There are two retention options in Microsoft 365. Both are configured in the Compliance portal of Microsoft 365. Access to this portal requires special privileges, which may not always be granted to records managers.

The two options are:

  • Retention labels published as retention policies and then applied to the various workloads (Exchange email, SharePoint, OneDrive, Office 365 Groups (Exchange/SharePoint content)). These are sometimes described as ‘explicit’ policies because they are visible to end users. Organisations with an E5 licence can extend the way these labels are applied and retention managed.
  • Retention policies that are applied directly to the various workloads (Exchange email, Exchange public folders, SharePoint, OneDrive, Office 365 Groups (Exchange/SharePoint content)). These are sometimes described as ‘implicit’ policies because they are not visible to end users. These policies automatically delete content at the end of a retention period, without any review possible.

Records managers will need to determine how to ‘translate’ each records retention class into one of the two options above, and how and where it will be applied in Microsoft 365.

Some of the options may also require the creation of new records retention classes – for example for the chat element in Microsoft Teams.

A suggested first model

Exchange mailboxes

Your IT probably already has some form of back-up regime (‘archive’) for mailboxes, used for disaster recovery and investigation purposes.

It might be worth creating two policies for mailboxes:

  • All end-user mailboxes could have a single ‘implicit’ retention policy (e.g., 7 years).
  • Mailboxes for specific staff (e.g., senior managers) could have a second, longer, ‘implicit’ retention policy. This policy will take over when the first one expires, but just for the mailboxes identified.

The use of retention policies in this way can replace the need for mailbox backups. No emails will ever actually be deleted while the retention policy is in place and all content can be retrieved via the Content Search option in the Compliance Portal. 

Content Searches can also be used to retrieve and export emails.

OneDrive for Business

As with end-user mailboxes, OneDrive for Business accounts are generally inaccessible to records managers. To ensure that the content in those accounts is not deleted, a single Microsoft implicit retention policy of, say, 7 years could be applied to all ODfB accounts.  This policy will create a hidden (to the user) ‘Preservation Hold’ library on the ODfB account.

Anything ‘deleted’ by the end user during the retention period will be moved to the Preservation Hold library, which is visible to the Global Admins and SharePoint Admins from this URL – /_layouts/15/viewlsts.aspx?view=14

In addition the OneDrive settings include the option (under ‘Storage’ in the ODfB admin portal) to retain OneDrive accounts for a period of time after they are inactive.

All content in these locations is accessible from a Content Search.

SharePoint

SharePoint is likely to be the most complicated in terms of retention policies if there is a requirement to keep content for different periods of time in accordance with the retention schedules/records disposal authorities.

There are likely to be three main options in relation to SharePoint content:

  • One or more implicit retention policy/ies applied to one or more sites. When applied to a SharePoint site, a ‘Preservation Hold’ library retains anything that is ‘deleted’ by end users.
  • One or more explicit label-based retention policies applied to one or more sites. When applied to a SharePoint site, the option to apply it appears for each document library on the site. Once applied (manually), end users cannot delete anything and if the library is synced to File Explorer, the File Explorer view of the library will be read only.
  • A combination of implicit and explicit retention policies.

The decision to apply what policy to what site will depend on your SharePoint architecture and the content stored in each site. For example:

  • A SharePoint site that only stores records that map to one records retention class could have either a single implicit policy (if there is no requirement for disposal review) or a single explicit policy that is applied manually to each library.
  • A SharePoint site that contains records that map to multiple retention classes, but for one business function and also ‘working papers’ could have (a) one implicit policy to cover the working papers and (b) one label-based retention policy with multiple labels – one for each class. This means, for (b), that a specific retention label can be applied to each library as required.
  • SharePoint sites linked with Office 365 Groups and Teams. Depending on the content in the site, it may be possible to apply a single retention policy for all M365 Groups (which covers both the SharePoint site and the mailbox), or a similar policy created for a Group of SharePoint sites (which excludes the mailbox).

MS Teams

As noted above, the chat content in MS Teams is stored in Exchange mailboxes – (a) the mailbox of each participant for one-to-one chat, and (b) the mailbox of the Office 365 Group for channel-based chat.

You may consider having a relatively short-term retention period for one-to-one chat. The retention period for the channel based chat will depend on the subject matter and should – ideally – be the same as for the linked SharePoint site. For example:

  • A Team set up for a specific business function and activity (or activities) will have channel based chat and a linked SharePoint site. Both should be subject to the same retention period.
  • A Team set up for low-level discussion about a subject that may be not be covered by any retention period could be subject to a general retention policy for the chat and the SharePoint content.

Bringing it together

As noted at the beginning of the post, if you are going to use retention policies in Microsoft 365 you need a plan and you need to document it. It doesn’t matter too much if the environment is already active.

However, you will need to have discussions with your Microsoft 365 Global Admins, Compliance Admins and SharePoint Admins and know where the content is stored.

  • The Global Admins can give you a list of every Office 365 Group and Team in MS Team (these are connected – every Team is based on an O365 Group).
  • The SharePoint Admins (or Global Admins) can give you a list of every SharePoint site.

There are some potential ‘quick wins’, such as agreement with IT regarding Exchange mailboxes, OneDrive for Business accounts, and MS Teams.

The more complex requirement is to map the classes in your records retention schedules/disposal authority to content stored in SharePoint, including for standard sites (not linked with Microsoft Groups), communication sites, and sites linked to Office 365 Groups.

You can start to do this by having a list of all the sites exported from the SharePoint Admin portal. This should allow you to see how many sites exist, how much content they hold, and if they are active or not.

It is probably a good idea for the records manager to be included as a Site Collection Administrator, including by being a member of a Security Group added to every SharePoint site. This will help the records manager gain visibility of the content of each site, however they should be very careful about browsing the content as everything is recorded in audit logs.

Document and plan

The outcome of all these actions should be one or more documents that describe (a) where records are stored and (b) the retention policy and action that will apply to those records.

  • For Exchange mailboxes, OneDrive for Business accounts, and MS Teams, this may be a single line for each policy.
  • For SharePoint, there should be a listing of every site and the retention policy or policies that apply to that site.
  • Additionally, for SharePoint sites where an explicit label-based retention policy is applied, the listing should show which libraries this has been applied to. If a disposal review option has been selected, there should be a process to ensure that the metadata of the library where the records are stored is exported and stored in a different location. The original library may then be deleted.
Posted in Classification, Compliance, Exchange Online, Information Management, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online, Training and education

Planning for records retention in Office 365

Office 365 is sometimes referred to as an ‘ecosystem’. In theory this means that records could be stored anywhere across that ecosystem.

Unlike the ‘old’ on-premise world of standalone servers for each Microsoft application (Exchange, SharePoint, Skype) – and where specific retention policies could apply (including the Exchange Messaging Records Management MRM policy), the various elements that make up Office 365 are interconnected.

The most obvious example of this interconnectivity is Microsoft Teams which stores chat content in Exchange and provides access to content stored in both SharePoint (primarily the SharePoint site of the linked Office 365 Group) and OneDrive, and has links to other elements such as Planner.

Records continue to be created and kept in the various applications but retention policies are set centrally and can apply to any or all of the content across the ecosystem.

Managing records in Office 365, and applying retention rules to those records, requires an understanding of at least the key parts of the ecosystem – Exchange, Teams, SharePoint and OneDrive and how they interrelate, and from there establishing a plan for the implementation of retention.

What types of records are created in Office 365?

Records are defined as ‘evidence of business activity’ and are often associated with some form of metadata.

Evidence of business activity is an overarching term that can include:

  • Emails
  • Calendars
  • Documents and notebooks (in the sense of text on a page)
  • Plans, including both project plans and architectural plans and diagrams
  • Images/photographs and video
  • Chat and/or messages
  • Conversations (audio and/or video based)
  • Social media posts

All digital records contain some form of metadata, usually displayed as ‘Properties’.

Where are the records stored in Office 365?

Most records created organisations using Office 365 are likely to be created or stored in the following parts of the ecosystem:

  • Exchange/Outlook – for emails and calendars.
  • SharePoint and OneDrive – for documents and notebooks (in the sense of text on a page), plans, images/photographs and video.
  • Stream – for audio and video recordings.
  • MS Teams – for chat and/or messages, conversations (audio and/or video based). Note that 1:1 chats are stored in a hidden folder of the Exchange mailbox of the end-user/s participating in the chat, while Teams channel chat is stored in a hidden folder of the linked Office 365 Group mailbox.
  • Yammer – for (internal) social media posts.

It is also possible to import and archive certain external content such as Twitter tweets and Facebook content in Office 365.

The diagram below provides a overview of the main Office 365 applications and locations where records are created or stored. Under SharePoint, the term ‘Sites’ refers to all types of SharePoint sites, including those associated with Office 365 Groups. Libraries are shown separately because of the potential to apply a retention policy to a library – see below.

O365WheretheRecordsare

Note also that this diagram does not include network file shares (NFS) as the assumption is made that (a) NFS content will be migrated to SharePoint and the NFS made read only, and (b) all new content that would previously have been stored on the NFS is instead saved either to OneDrive for Business (for ‘personal’ or working documents) or SharePoint only.

Creating a plan to manage records retention across Office 365

In previous posts I have recommended that organisations implementing Office 365 have the following:

  • A basic architecture design model for SharePoint sites, including SharePoint sites linked with Office 365 Groups (and Teams in MS Teams).
  • A plan for creating and applying retention policies across the ecosystem.

Because SharePoint is the most likely location for records to be stored (aside from Exchange mailboxes and OneDrive accounts), there should be at least one retention policy for every SharePoint site (or group of sites), as well as policies for specific document libraries if the retention for the content in those libraries may be different from the retention on the overall site.

For example, a ‘Management’ site may contain a range of general content as well as specific content that needs to be retained for longer. 

  • The site can be covered by a single implicit retention policy of (say) 7 years. This policy will delete content in the background, based on date created or data modified. 
  • The document library where specific types of records with longer or different retention requirements are stored may have one or more explicit label-based policies applied to those libraries. This content will be retained while the rest of the site content is deleted via the first policy.

Structure of a retention plan for records in Office 365

A basic plan for creating and applying retention policies might look something like the following:

  • User mailboxes – one ‘general’ (implicit) retention policy for all mailboxes (say, 7 years after creation) and another more specific retention policy for specific mailboxes that require longer retention.
  • SharePoint sites – multiple (implicit) retention policies targeting one or more sites.
  • SharePoint libraries – multiple (explicit) label-based retention policies that are applied manually. These policies will usually a retention policy that is longer than any implicit retention policy as any implicit site policy will prevent the deletion of content before it reaches the end of that retention period.
  • Office 365 Groups (includes the associated mailbox and SharePoint site) – one ‘general’ (implicit) retention policy. See also below.
  • Teams channel chat – one ‘general’ (implicit) retention policy. Note that this content is stored in a special folder of the Office 365 Group mailbox.
  • 1:1 chat – one ‘general’ (implicit) retention policy. This content is stored in a special folder of the participant mailboxes.
  • OneDrive documents – one ‘general’ (implicit) retention policy for all ODfB accounts, plus the configuration of retention after the account is inactive.

At a high level, the retention policy plan might look something like the following – ‘implicit’ policies are shown in yellow, SharePoint document libraries may be subject to ‘explicit’, label-based policies. The ‘+7 years’ for OneDrive relates to inactive accounts, a setting set in the OneDrive Admin portal.

O365WheretheRecordsare2

Regarding Microsoft Office 365 Groups, Microsoft notes the following on this page about managing retention in Office 365:

To retain content for a Microsoft 365 group, you need to use the Microsoft 365 groups location. Even though an Microsoft 365 group has an Exchange mailbox, a retention policy that includes the entire Exchange location won’t include content in Microsoft 365 group mailboxes. A retention policy applied to an Microsoft 365 group includes both the group mailbox and site. A retention policy applied to an Microsoft 365 group protects the resources created by an Microsoft 365 group, which would include Microsoft Teams.

The actual plan should contain more detail and included as part of other recordkeeping documentation (perhaps stored on a ‘Records Management’ SharePoint site). The plan should include details about (a) where the policies have been applied and (b) the expected outcomes or actions for the policies, including automatic deletion or disposition review (for document libraries).

Keep in mind that, unless the organisation decides to acquire this option, there is no default backup for content in Office 365 – once a record had been deleted, it is gone forever and there may be no record of this beyond 90 days.

Posted in Governance, Information Management, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, SharePoint Online

What happens when you create a Team in MS Teams

On 27 March 2020 I asked, via Twitter, whether organisations that rolled out MS Teams will wonder in the future who created all the random (and randomly-named) SharePoint sites.

20200414_122632

The reason for this question was because many organisations, scrambling to establish ways for staff to work from home, decided to make use of MS Teams in their (often newly implemented) Office 365 suite of apps.

I have seen multiple organisations since late 2019 ask ‘who created all those SharePoint sites?’ when they reviewed the list. The current COVID-19 work-from-home situation will only make this situation ‘worse’ and, without effective oversight or controls, result in the creation of multiple uncontrolled SharePoint sites.

Unlike other products like Zoom, Whatsapp, Facetime and Skype, however, MS Teams is not a standalone product, but a core element in the Microsoft Office 365 ecosystem.

The key point is this – every Team in MS Teams has a linked SharePoint site (and an Exchange mailbox, where all the chat content is stored). You can’t disable these options.

What happens if you create a Team in MS Teams?

The good thing about the one-to-one chat element of MS Teams is that it’s relatively intuitive and easy to use, including on the mobile app. You only need to tell users it’s like Skype or Whatsapp, but for internal user only, and most pick it up quickly.

The Teams part of MS Teams is not quite as intuitive, but early adopters generally understand the basic concepts – that a Team has members, and you can have multiple chat channels for each Team.

Once end-users understand how a Team works (and this can take some time because one-to-one chat can include multiple people), they might notice this option at the bottom left of the app:

JoinCreateTeam

Creating a new team sounds like a great idea, so end-users may try:

JoinCreateTeam2

My guess is that end-users are more likely to want to ‘build a team from scratch’ as shown below, because the second option doesn’t really make sense.

JoinCreateTeam3

There is a good chance they will want the Team to be ‘Private’, although may not fully understand what this means. A Public Team sounds like a Yammer Group (or Community).

JoinCreateTeam4

So far, so good, the end-user can give the Team any name they like:

JoinCreateTeam5

At the bottom of the naming screen is the option to ‘Create’. The end-user is then invited to add members to their new Team. This seems a fairly obvious step, and they can add whoever they want. New members are by default ‘Members’ but they can be changed to ‘Owners’ if necessary. There is no control over this process.

JoinCreateTeam6

The new team now appears on the left-hand menu of MS Teams:

JoinCreateTeam7

The new team opens at the default ‘General’ channel.

On the main part of the Team, the following options are offered:

  • Along the top, ‘Posts’, ‘Files’, ‘Wiki’ and a + to add more applications. (Hint – the ‘Files’ option points to the SharePoint site that has been created behind the scenes).
  • Across the middle, three options to ‘Add more people’, ‘Create more channels’ and ‘Open the FAQ’
  • At the bottom, the option to ‘Start a new conversation’ with various other options including the ‘Meet now’ video option.

The end-user can now get on with chatting, sharing files, and adding apps to do other things.

But what else has happened?

As noted above, the ‘Files’ tab in the General channel gives a clue to the existence of the connected SharePoint site. End-users may not care terribly much about this, for them it provides the option to create, upload, share and collaborate on files.

A new Office 365 Group is created

But before we get to the SharePoint site, it’s important to understand the one-to-one relationship between a Team in MS Teams and an Office 365 Group. If you do not know what an Office 365 Group is, please read this Microsoft guidance on Office 365 Groups.

In very simple terms:

  • Every new Team in MS Teams creates a new Office 365 Group.
  • The Owner of the Office 365 Group is the Owner of the team; the members of the Group are the Members of the team, as added by the person who created the Team.

The new Office 365 Group appears in the list of Groups in the Office 365 Admin portal, as shown below. Access to this part of the Admin portal is normally restricted to Global Admins (who would normally be responsible for creating other types of AD Groups, such as Security Groups and Distribution Lists.

A new Exchange mailbox has been created

Note that the process has also created an Exchange mailbox with a Group email address. The new Exchange mailbox will now appear in the Outlook client of everyone in the Team – something they are unlikely to notice.

JoinCreateTeam8

As noted above, all the chat messages in the Team are stored in a hidden folder in the Exchange mailbox for the Team.

A new SharePoint site has been created

If we go across to the SharePoint Admin portal, which is normally restricted to Global Admins and SharePoint Admins, we can see that a new SharePoint site has been created, and is owned by the ‘Group owners’.

JoinCreateTeam9

The SharePoint Admin has had no involvement in the creation, naming, or structure of this new site. And, just to add another factor, the SharePoint Admin cannot access the site – see below.

The Team owner may not realise it, but they now have a SharePoint site. The new site’s ‘Documents’ library appears in the ‘Files’ tab as shown below.

JoinCreateTeam11

And, just to add a confusing element, the site includes the invitation (at the bottom left) to create a new Team!

JoinCreateTeam10

As noted above the SharePoint Admin can ‘see’ that this site exists in the list of sites but cannot actually access it. The Global Admin, on the other hand, can access it.

JoinCreateTeam12

So the person responsible for managing SharePoint across the organisation cannot access the SharePoint site, which is not a good thing from an information governance point of view.

The reason they cannot access the site is because they were not added to the Site Collection Admin Group when the site was created. And, just to make it a bit more confusing, the ‘Users and Permissions’ section of Site Settings, where the ‘Site collection administrators’ section is found (see screenshot below), does not appear in Office 365 Group-based SharePoint sites.

SPOSiteSettings

So, how does the SharePoint Admin get access to this site to configure and manage it? There are two ways:

  • The Global Admin can go to /_layouts/15/mngsiteadmin.aspx (after the site name URL) and add them (or a Security Group with them in it) there.
  • The SharePoint Admin can click on the site details in the SharePoint admin portal and add him/herself as an Owner. This puts them in the Site Collection Admin section along with the Group Owner.

Summary

This post began with a simple question – if organisations allow end-users to create Teams to work from home, how will they manage all the SharePoint sites that are created through the process described above?

There is no one answer to this question but it’s worth understanding exactly what happens – and what else is created (including Planner) – when a Team is created. Organisations seem to go one of two ways:

  • Let end users create Teams and deal with the consequences later, including attempts at auto-classification and retention policy application across the various elements of the new Office 365 Group – mailbox, SharePoint site, Team chat. This is the Microsoft default and the preference of many organisations that are don’t have compliance issues or can accept the risks of uncontrolled information stores.
  • Control the creation of Teams, but make any controlled process as easy as possible for end-users to keep them working quickly, and manage the content in mailboxes, SharePoint and Teams proactively. While not the preferred option, it will help with the management of corporate information down the track.