Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

There are two types of chat message in MS Chat:

• One-to-one/one-to-many ‘chat’ (top icon above).
• Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

• Knowing if chat messages on any given subject exist. 
• Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

Searches can be configured to find content in any or all of the following locations:

• Users, Groups, Teams
  • Exchange email
  • Office 365 group email
  • Skype for Business
  • Teams messages [the copy in the mailbox]
  • To-Do
  • Sway
  • Forms
• SharePoint
  • SharePoint sites
  • OneDrive accounts
  • Office 365 group sites
  • Teams sites
• Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

• As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
• There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

Classification

Microsoft 365 – Options for collating, grouping and aggregating records

The international standard for records management, ISO 15489-1:2016 (‘Information and documentation – Records management – Part 1: Concepts and Principles’), defines records as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’. Among other things, the standard notes that […]

July 2, 2020

Classification

A Tale of Two Retention Policies in Microsoft 365

There are two ways to create retention policies in Microsoft 365 – (a) by creating a label and publishing it as retention policy, or (b) creating a retention policy without a label. Both are created from the Information Governance section of the Compliance portal in Microsoft 365. This post describes how the policies are created […]

June 22, 2020

Classification

What to use when – metadata, content types, folders or document sets

Most end-users will be familiar with using folders to ‘categorise’ content in SharePoint document libraries, but not everyone will be familiar with other options including document sets or the use of metadata or content types to achieve similar ‘grouping’ outcomes. Any (and all of) these options can be used to categorise content in a document […]

June 16, 2020

Planning for records retention in Office 365

Office 365 is sometimes referred to as an ‘ecosystem’. In theory this means that records could be stored anywhere across that ecosystem.

Unlike the ‘old’ on-premise world of standalone servers for each Microsoft application (Exchange, SharePoint, Skype) – and where specific retention policies could apply (including the Exchange Messaging Records Management MRM policy), the various elements that make up Office 365 are interconnected.

The most obvious example of this interconnectivity is Microsoft Teams which stores chat content in Exchange and provides access to content stored in both SharePoint (primarily the SharePoint site of the linked Office 365 Group) and OneDrive, and has links to other elements such as Planner.

Records continue to be created and kept in the various applications but retention policies are set centrally and can apply to any or all of the content across the ecosystem.

Managing records in Office 365, and applying retention rules to those records, requires an understanding of at least the key parts of the ecosystem – Exchange, Teams, SharePoint and OneDrive and how they interrelate, and from there establishing a plan for the implementation of retention.

What types of records are created in Office 365?

Records are defined as ‘evidence of business activity’ and are often associated with some form of metadata.

Evidence of business activity is an overarching term that can include:

• Emails
• Calendars
• Documents and notebooks (in the sense of text on a page)
• Plans, including both project plans and architectural plans and diagrams
• Images/photographs and video
• Chat and/or messages
• Conversations (audio and/or video based)
• Social media posts

All digital records contain some form of metadata, usually displayed as ‘Properties’.

Where are the records stored in Office 365?

Most records created organisations using Office 365 are likely to be created or stored in the following parts of the ecosystem:

• Exchange/Outlook – for emails and calendars.
• SharePoint and OneDrive – for documents and notebooks (in the sense of text on a page), plans, images/photographs and video.
• Stream – for audio and video recordings.
• MS Teams – for chat and/or messages, conversations (audio and/or video based). Note that 1:1 chats are stored in a hidden folder of the Exchange mailbox of the end-user/s participating in the chat, while Teams channel chat is stored in a hidden folder of the linked Office 365 Group mailbox.
• Yammer – for (internal) social media posts.

It is also possible to import and archive certain external content such as Twitter tweets and Facebook content in Office 365.

The diagram below provides a overview of the main Office 365 applications and locations where records are created or stored. Under SharePoint, the term ‘Sites’ refers to all types of SharePoint sites, including those associated with Office 365 Groups. Libraries are shown separately because of the potential to apply a retention policy to a library – see below.

Note also that this diagram does not include network file shares (NFS) as the assumption is made that (a) NFS content will be migrated to SharePoint and the NFS made read only, and (b) all new content that would previously have been stored on the NFS is instead saved either to OneDrive for Business (for ‘personal’ or working documents) or SharePoint only.

Creating a plan to manage records retention across Office 365

In previous posts I have recommended that organisations implementing Office 365 have the following:

• A basic architecture design model for SharePoint sites, including SharePoint sites linked with Office 365 Groups (and Teams in MS Teams).
• A plan for creating and applying retention policies across the ecosystem.

Because SharePoint is the most likely location for records to be stored (aside from Exchange mailboxes and OneDrive accounts), there should be at least one retention policy for every SharePoint site (or group of sites), as well as policies for specific document libraries if the retention for the content in those libraries may be different from the retention on the overall site.

For example, a ‘Management’ site may contain a range of general content as well as specific content that needs to be retained for longer. 

• The site can be covered by a single implicit retention policy of (say) 7 years. This policy will delete content in the background, based on date created or data modified. 
• The document library where specific types of records with longer or different retention requirements are stored may have one or more explicit label-based policies applied to those libraries. This content will be retained while the rest of the site content is deleted via the first policy.

Structure of a retention plan for records in Office 365

A basic plan for creating and applying retention policies might look something like the following:

• User mailboxes – one ‘general’ (implicit) retention policy for all mailboxes (say, 7 years after creation) and another more specific retention policy for specific mailboxes that require longer retention.
• SharePoint sites – multiple (implicit) retention policies targeting one or more sites.
• SharePoint libraries – multiple (explicit) label-based retention policies that are applied manually. These policies will usually a retention policy that is longer than any implicit retention policy as any implicit site policy will prevent the deletion of content before it reaches the end of that retention period.
• Office 365 Groups (includes the associated mailbox and SharePoint site) – one ‘general’ (implicit) retention policy. See also below.
• Teams channel chat – one ‘general’ (implicit) retention policy. Note that this content is stored in a special folder of the Office 365 Group mailbox.
• 1:1 chat – one ‘general’ (implicit) retention policy. This content is stored in a special folder of the participant mailboxes.
• OneDrive documents – one ‘general’ (implicit) retention policy for all ODfB accounts, plus the configuration of retention after the account is inactive.

At a high level, the retention policy plan might look something like the following – ‘implicit’ policies are shown in yellow, SharePoint document libraries may be subject to ‘explicit’, label-based policies. The ‘+7 years’ for OneDrive relates to inactive accounts, a setting set in the OneDrive Admin portal.

Regarding Microsoft Office 365 Groups, Microsoft notes the following on this page about managing retention in Office 365:

To retain content for a Microsoft 365 group, you need to use the Microsoft 365 groups location. Even though an Microsoft 365 group has an Exchange mailbox, a retention policy that includes the entire Exchange location won’t include content in Microsoft 365 group mailboxes. A retention policy applied to an Microsoft 365 group includes both the group mailbox and site. A retention policy applied to an Microsoft 365 group protects the resources created by an Microsoft 365 group, which would include Microsoft Teams.

The actual plan should contain more detail and included as part of other recordkeeping documentation (perhaps stored on a ‘Records Management’ SharePoint site). The plan should include details about (a) where the policies have been applied and (b) the expected outcomes or actions for the policies, including automatic deletion or disposition review (for document libraries).

Keep in mind that, unless the organisation decides to acquire this option, there is no default backup for content in Office 365 – once a record had been deleted, it is gone forever and there may be no record of this beyond 90 days.

What happens when you create a Team in MS Teams

On 27 March 2020 I asked, via Twitter, whether organisations that rolled out MS Teams will wonder in the future who created all the random (and randomly-named) SharePoint sites.

The reason for this question was because many organisations, scrambling to establish ways for staff to work from home, decided to make use of MS Teams in their (often newly implemented) Office 365 suite of apps.

I have seen multiple organisations since late 2019 ask ‘who created all those SharePoint sites?’ when they reviewed the list. The current COVID-19 work-from-home situation will only make this situation ‘worse’ and, without effective oversight or controls, result in the creation of multiple uncontrolled SharePoint sites.

Unlike other products like Zoom, Whatsapp, Facetime and Skype, however, MS Teams is not a standalone product, but a core element in the Microsoft Office 365 ecosystem.

The key point is this – every Team in MS Teams has a linked SharePoint site (and an Exchange mailbox, where all the chat content is stored). You can’t disable these options.

What happens if you create a Team in MS Teams?

The good thing about the one-to-one chat element of MS Teams is that it’s relatively intuitive and easy to use, including on the mobile app. You only need to tell users it’s like Skype or Whatsapp, but for internal user only, and most pick it up quickly.

The Teams part of MS Teams is not quite as intuitive, but early adopters generally understand the basic concepts – that a Team has members, and you can have multiple chat channels for each Team.

Once end-users understand how a Team works (and this can take some time because one-to-one chat can include multiple people), they might notice this option at the bottom left of the app:

Creating a new team sounds like a great idea, so end-users may try:

My guess is that end-users are more likely to want to ‘build a team from scratch’ as shown below, because the second option doesn’t really make sense.

There is a good chance they will want the Team to be ‘Private’, although may not fully understand what this means. A Public Team sounds like a Yammer Group (or Community).

So far, so good, the end-user can give the Team any name they like:

At the bottom of the naming screen is the option to ‘Create’. The end-user is then invited to add members to their new Team. This seems a fairly obvious step, and they can add whoever they want. New members are by default ‘Members’ but they can be changed to ‘Owners’ if necessary. There is no control over this process.

The new team now appears on the left-hand menu of MS Teams:

The new team opens at the default ‘General’ channel.

On the main part of the Team, the following options are offered:

• Along the top, ‘Posts’, ‘Files’, ‘Wiki’ and a + to add more applications. (Hint – the ‘Files’ option points to the SharePoint site that has been created behind the scenes).
• Across the middle, three options to ‘Add more people’, ‘Create more channels’ and ‘Open the FAQ’
• At the bottom, the option to ‘Start a new conversation’ with various other options including the ‘Meet now’ video option.

The end-user can now get on with chatting, sharing files, and adding apps to do other things.

But what else has happened?

As noted above, the ‘Files’ tab in the General channel gives a clue to the existence of the connected SharePoint site. End-users may not care terribly much about this, for them it provides the option to create, upload, share and collaborate on files.

A new Office 365 Group is created

But before we get to the SharePoint site, it’s important to understand the one-to-one relationship between a Team in MS Teams and an Office 365 Group. If you do not know what an Office 365 Group is, please read this Microsoft guidance on Office 365 Groups.

In very simple terms:

• Every new Team in MS Teams creates a new Office 365 Group.
• The Owner of the Office 365 Group is the Owner of the team; the members of the Group are the Members of the team, as added by the person who created the Team.

The new Office 365 Group appears in the list of Groups in the Office 365 Admin portal, as shown below. Access to this part of the Admin portal is normally restricted to Global Admins (who would normally be responsible for creating other types of AD Groups, such as Security Groups and Distribution Lists.

A new Exchange mailbox has been created

Note that the process has also created an Exchange mailbox with a Group email address. The new Exchange mailbox will now appear in the Outlook client of everyone in the Team – something they are unlikely to notice.

As noted above, all the chat messages in the Team are stored in a hidden folder in the Exchange mailbox for the Team.

A new SharePoint site has been created

If we go across to the SharePoint Admin portal, which is normally restricted to Global Admins and SharePoint Admins, we can see that a new SharePoint site has been created, and is owned by the ‘Group owners’.

The SharePoint Admin has had no involvement in the creation, naming, or structure of this new site. And, just to add another factor, the SharePoint Admin cannot access the site – see below.

The Team owner may not realise it, but they now have a SharePoint site. The new site’s ‘Documents’ library appears in the ‘Files’ tab as shown below.

And, just to add a confusing element, the site includes the invitation (at the bottom left) to create a new Team!

As noted above the SharePoint Admin can ‘see’ that this site exists in the list of sites but cannot actually access it. The Global Admin, on the other hand, can access it.

So the person responsible for managing SharePoint across the organisation cannot access the SharePoint site, which is not a good thing from an information governance point of view.

The reason they cannot access the site is because they were not added to the Site Collection Admin Group when the site was created. And, just to make it a bit more confusing, the ‘Users and Permissions’ section of Site Settings, where the ‘Site collection administrators’ section is found (see screenshot below), does not appear in Office 365 Group-based SharePoint sites.

So, how does the SharePoint Admin get access to this site to configure and manage it? There are two ways:

• The Global Admin can go to /_layouts/15/mngsiteadmin.aspx (after the site name URL) and add them (or a Security Group with them in it) there.
• The SharePoint Admin can click on the site details in the SharePoint admin portal and add him/herself as an Owner. This puts them in the Site Collection Admin section along with the Group Owner.

Summary

This post began with a simple question – if organisations allow end-users to create Teams to work from home, how will they manage all the SharePoint sites that are created through the process described above?

There is no one answer to this question but it’s worth understanding exactly what happens – and what else is created (including Planner) – when a Team is created. Organisations seem to go one of two ways:

• Let end users create Teams and deal with the consequences later, including attempts at auto-classification and retention policy application across the various elements of the new Office 365 Group – mailbox, SharePoint site, Team chat. This is the Microsoft default and the preference of many organisations that are don’t have compliance issues or can accept the risks of uncontrolled information stores.
• Control the creation of Teams, but make any controlled process as easy as possible for end-users to keep them working quickly, and manage the content in mailboxes, SharePoint and Teams proactively. While not the preferred option, it will help with the management of corporate information down the track.

 

Managing emails as records in Office 365

Records managers have been struggling with managing emails as records ever since they first appeared in the workplace.

For a long time the accepted practice, as with other digital records, was to print them out and put them on the appropriate file. With the introduction of electronic document and records management (EDRM) systems, end users were instead required to save or copy documents and emails to an electronic ‘file’ in that system.

The old paradigm where emails were copied to an EDRMS or a paper file

In both cases, the emails remained in the user’s ‘personal’ mailbox, where they remained inaccessible for ‘privacy’ reasons. End-users and business areas would (and still do) conduct business via the email system, without these records being available to anyone except the sender and recipient/s. Attachments to emails sent to individual recipients were (and continue to be) not managed as records unless they were printed out or saved to the EDRMS.

Microsoft Office 365 has changed the paradigm for keeping records as described in the linked post, away from the central storage and management of records in one system (while leaving the originals in place), to the decentralised ‘in place’ storage and centralised management of records across Office 365.

This post provides an overview of the three main options for managing email as records in Office 365, in both Exchange and SharePoint.

In summary the options are:

• Leave emails in place in Exchange mailboxes (personal and Office 365 Group mailboxes) and apply one or more Office 365 retention policies to mailboxes.
• Same as previous point, and use Content Search to retrieve emails as required.
• Same as previous point, and only copy specific emails to SharePoint

Keep in mind while reading this post that chat content from MS Teams is also stored in Exchange mailboxes but that content cannot be copied to SharePoint.

Option 1 – Leave emails in place and apply retention policy

In this option, emails remained stored in personal or Office 365 Group mailboxes. End users may create folders and ‘categorise’ the content as they wish, but no additional attempt is made to further categorise, add metadata to, or group the content according to recordkeeping requirements. The aggregation, from a recordkeeping point of view, is the end-user or Office 365 Group.

All mailboxes are subject to one or more retention policies set in the Office 365 Compliance portal to ensure that no emails are deleted before a pre-defined minimum period.

Note that retention policies can effectively replace a back-up regime used by IT for disaster recovery and investigation purposes purposes.

Positives

Emails are aggregated by user name or Office 365 Group and will remain in mailboxes for a minimum period of time as set by the retention policy. 

Office 365 Group mailboxes provide the ability to group emails by a more specific subject  (the Group name, which could map to a business function – e.g., ‘Correspondence Management’) and have the added positive of having an associated SharePoint site.

Negatives

The negative with this option, from a recordkeeping point of view, is that all emails – regardless of subject or importance – are grouped by the ‘personal’ or Office 365 Group mailbox, and kept for the period defined in the retention policy. That is, there is no differentiation between (email) records that may need to be kept for a long period of time and those that are transient in nature.

If there is a requirement to ensure that certain emails are kept in different aggregations or for different periods of time, then option 3 should be considered.

Option 2 – Same as option 1 and use Content Search to retrieve emails

This option is the same as the first option, but the business can make use of Content Search to identify and isolate emails as required. Content Search is more or less the same as the search part of an e-Discovery case.

The Content Search option is accessed from the left hand navigation of the Compliance portal: https://compliance.microsoft.com/homepage

Note that access to the Content Search area is restricted to Office 365 Global Admins and Compliance Admins. This is because, as can be seen in the screenshot below, a Content Search can be set up to search for any content in email, documents and much more.

Content Searches can be set up from the ‘New Search’ option, or the Administrator can make use of a Guided search or Search by ID List. For the purpose of this email, only the ‘New search’ will be examined.

Configuring a new Content Search

Each content search can be configured against three main options as shown in the screenshot below: Keywords, Conditions, and Locations. Some searches may require a combination of these three options.

Keywords can be any words that may be found anywhere in the email, including the content.

The available conditions are listed below:

Common

• Date
• Sender/Author
• Size (in bytes)
• Subject/Title
• Compliance label

Emails

• Message kind
• Participants
• Type
• Received
• Recipients
• Sender
• Sent
• Subject
• To

Documents

• Author
• Title
• Created
• Last modified
• File type

The available search locations include any or all of the options below:

• Exchange email
• Office 365 group email
• Skype for Business
• Teams messages
• To-Do
• Sway
• Forms
• SharePoint sites
• OneDrive accounts
• Office 365 Group sites
• Teams sites
• Exchange public folders

For more detail on how to use Content Search and all the options available, go to this Microsoft site.

Running a search

After the search has been configured, it must be run. The speed of the search will depend on the complexity of the search, conditions, locations and the volume of content. Every search will appear in the list of searches that have been saved.

When complete, the search result will show a ‘Status’, showing the number of:

• Items found
• Unindexed items
• Mailboxes
• Sites
• Public folders

Exporting results

Once the search has completed, the results of the search may be exported. There are two  configurable options for exported results.

Output options:

• All items, excluding ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• All items, including ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• Only items that have an unrecognized format, are encrypted, or weren’t indexed for other reasons

Exchange content export options:

• One PST file for each mailbox
• One PST file containing all messages
• One PST file containing all messages in a single folder
• Individual messages
• Enable de-duplication for Exchange content (check box)

Positives

Content searches are likely to find and retrieve more relevant emails than might be saved elsewhere, as it looks through all emails. Provided a retention policy has been applied to the mailboxes, the content should still be accessible. If the emails have been deleted at the end of a retention policy, they will not be accessible any more.

Emails can be exported and – if necessary – the PST copied to a different system (such as SharePoint) for long-term storage with additional metadata as required.

Negatives

Access to the Content Search option is restricted to Global Admins and Compliance admins, for good reason. Consideration might need to be given to governance or procedural rules. Note that Global Admins are always alerted when a new content search is created or run.

Each search must be pre-configured and run regularly to ensure that all emails are identified.

Content searches may retrieve too much unrelated content.

Option 3 – Same as option 2 and copy only select emails to SharePoint

This option mimics the legacy way of saving a record to a pre-defined separate aggregation, in this case to a SharePoint document library.

It differs from the first two options in that only certain select emails are copied (by end-users or using a third-party application) to specific SharePoint document libraries. It is still, however, possible (and preferable) to apply a retention policy to the original mailboxes.

Content search, which can be used at any time, will find the emails in both Exchange mailboxes and SharePoint as long as they have not been deleted via a retention policy expiry .

Positives

The positives with this option are that emails copied to a SharePoint document library:

• Are grouped with other related records. This may be important from an organisational recordkeeping point of view, for example for certain key records. Consideration might also be given to setting up an Office 365 Group instead for these specific records.
• Can have additional metadata.
• Can be retained for a period of time, different from the original mailbox.

Negatives

The problems with this option are that:

• It requires some kind of action to copy the email.
• It creates a copy of the email, it doesn’t remove the original.
• An email copied to another system may not be the most recent in a thread, especially if that thread is still active.
• Does not include the ‘chat’ elements from MS Teams.

Summing up the options

The idea of copying an email to a separate aggregation, container or file for recordkeeping purposes is a legacy concept inherited from the paper recordkeeping period. While attempts were made over the years to mimic that concept in EDRM systems, it has several weaknesses that mostly outweigh the alleged benefits.

Email (in Exchange) and documents (in SharePoint) continue to remain separate in Office 365 but there is now the potential to manage both equally through a combination of retention policies and pre-defined content searches.

The majority of business emails are never captured in separate recordkeeping systems. Microsoft’s centralised retention model and ability to apply to retrieve emails on the fly mean that it is more efficient and cost effective to leave emails in place. This does not exclude the potential to copy certain select emails to SharePoint.

Additionally, mailboxes associated with Office 365 Groups provide the ability to keep emails in a business context, away from inaccessible ‘personal’ email accounts. Records managers should consider the potential of using Office 365 Group mailboxes in this way for particular types of records.

Using Office 365 Groups to manage (some) records in context

The third part of my recent series of posts, titled Setting up SharePoint Online to manage records, included an example architecture model with several Office 365 Group-based sites, shown in green in the graphic below.

This post describes how Office 365 Groups, with their associated mailbox and SharePoint site, can be useful when there is a business requirement to manage aggregations of email and document-based records for a given function or subject, and why this option should be considered as part of your architecture model for managing records in Office 365 and SharePoint.

What are Office 365 Groups?

Office 365 Groups are a relatively new concept and still remain poorly understood, especially in organisations implementing Office 365.

Because they are not well understood, organisations may implement all the elements of Office 365 without necessarily being aware that:

• Every new Group in Outlook, Team in MS Teams, Planner, and Yammer group creates an Office 365 Group with an associated SharePoint site.
• End-users can create SharePoint sites from their individual SharePoint portal (the default ‘Team site’ option creates an Office 365 Group)
• SharePoint sites will proliferate without any controls over naming or content.

In very simple terms, Office 365 Groups:

• Are Azure Active Directory (AD) objects, similar to Security Groups (also known as AD Groups).
• Like Security Groups, have members and give those members access to certain (but usually different) IT resources.
• Have an associated Exchange mailbox and SharePoint site (that links these two in the same context), and can be linked with a Team in MS Teams. The members of the Group have access to both the mailbox and the SharePoint site. Members of Security Groups may also be added to the SharePoint site but won’t be able to see the mailbox or the Teams chat.
• Are similar to Distribution Lists or a shared mailbox in the sense that all the members of the Group can receive emails from the one mailbox.
• Include other functionality, including Planner, and can be linked with Yammer groups/communities.

For more information on Office 365 Groups, see this blog post ‘Office 365 Groups Explained‘ on the Microsoft Tech Community site, which points to a more detailed post on the same subject from ShareGate.

The records management problem that Office 365 Groups can help to resolve

One of the key recordkeeping problems in many organisations (aside from managing digital records generally) is the disconnect between email and other forms of records on the same subject or context.

• Emails are created, sent to and from, and stored in mostly inaccessible ‘personal’ accounts (with the odd shared mailbox). Emails may have attachments and the attachment may be the only version of the record. Shared mailboxes partially help this issue but there remains a disconnect between emails and other records.
• Other digital records (including saved email attachments) may be stored across network file shares (including ‘personal drives’ and the local C drive) and other locations, including USB drives and unofficial cloud-based systems. Unofficial and unapproved storage locations heighten information security risks both from storing official records in unofficial locations, but also the potential for malicious links.

In the early computer days the only way to keep these records together was to print them and put them on a file, a practice that (sadly) continues to exist. Over the last 20 years, electronic document and records management (EDRM) systems have provided a similar functionality by requiring end-users to copy original documents to a digital version of the file (leaving the originals in place in most cases).

The problem that most organisations face is that records about the same subject or  business context, in multiple forms (email, documents) and formats (including chat, messaging and social media) are stored across multiple systems (Outlook, network file shares, personal drives, personal apps on mobile devices). 

As a case in point, in 2018 I asked a business unit of so-called ‘mobile workers’ how they kept in touch and ‘collaborated’. Their responses included Facebook Messager, Whatsapp, DropBox (and others) and private emails.

How can Office 365 Groups help recordkeeping?

As noted above, Office 365 Groups have a mailbox, SharePoint site and a Team in MS Teams.

If set up correctly, a single Office 365 Group can provide a single point of context for both email and other digital records including chat, without end-users having to copy or move content anywhere else.

The following are examples (from real life experience) of how Office 365 Groups can be used to keep related records in context.

Correspondence management

Organisation often have a central point for the management of all incoming correspondence. In the past, this was likely to be a shared mailbox and correspondence might be uploaded (including after scanning of paper mail) to an EDRMS or other system for routing and responses.

Instead of using a shared mailbox, the Office 365 Group mailbox can be used to receive all emails (where they can remain), and its connected SharePoint site can be used for the storage of digital content including template responses (that could also be stored in Content Types created on the site). A Team in MS Teams may also be created to provide a forum for discussion about the correspondence or other matters – for example, a channel to discuss draft responses.

All the members of the Office 365 Group can access the mailbox, chat and collaborate on content within Team channels, and use the SharePoint site. They can use the ‘share’ option (rather than attaching document to emails) to share drafts with others, or use the ‘out of the box’ ‘Request Sign Off’ flow available in every document library to seek approval.

Additionally, if the Group’s SharePoint site front page has the default ‘Site Activity’ web app displayed, this will show new emails coming in to the Group’s mailbox as shown in the image below. These emails are only accessible to the members of the Group.

Example of an incoming email automatically appearing on the front page of a Group’s SharePoint site

 

 

 

 

 

 

 

 

 

Storing records by function/activity/subject

Office 365 Groups could be used to manage the records of a particular business function and activity, particularly those where there is a lot of email.

For example, the functions of ‘Fleet Management’ (or ‘Asset Management’), ‘Property Management’, or even ‘Financial Management’ are all likely to have both email and document type records.

• Emails relating to the function can be sent to and managed from the Group’s mailbox by the members of the Group, rather than using a shared mailbox (which is disconnected from the other records).
• Documents (including emails from the mailbox, if required) relating to the function can be stored in SharePoint document libraries that map to the activities that are being performed. For example, in a ‘Meetings’ library.
• There may be a single retention policy for the entire site or one or more label-based retention policies used on individual libraries, or a combination of both (the longest retention policy will take precedence).

In this way, emails and documents about the same subject can be managed from a single Office 365 Group.

Senior executive management

Office 365 Groups can be used to support and secure communication between senior executives. It provides them with a single restricted access mailbox and SharePoint site, access to both of which are controlled by membership of the Office 365 Group. It also provides them with a Team in MS Teams.

Additional security measures can be applied to more sensitive information including:

• More restricted security on some parts of the SharePoint site
• Data Loss Prevention policies for very sensitive information
• Retention policies to prevent the deletion of content (or capture ‘deleted’ content in a hidden Preservation Hold library).
• Stronger monitoring of all activity by end-user.

IT Operations/Service Desk

The IT Service Desk is a common point of contact in most organisations and most service desks will have a shared mailbox to review and triage incoming emails. They will also have a requirement to keep records relating to service support issues.

An Office 365 Group, perhaps named ‘ITServiceDesk’, can be established.

• The Group mailbox can be the central point of email contact for the Service Desk (ITServiceDesk@organisation.name).
• The associated SharePoint site can be used for the storage of service support documents and other content.
• The Group’s Team in MS Teams can have multiple channels to support each application or aspect of IT as required. The Service Desk can use the one-to-one chat and sharing screen capability to resolve service issues.

Office 365 Groups can change the way we work

As noted above, ‘personal’ emails and the use of mostly uncontrolled network file shares (and other storage locations) have been a common way of working for three decades.

Changing these work habits can be hard. However, change can be brought about with minimal impact, provided end-users are assured that the content they need to access will still be accessible and protected, and there is a tangible benefit in doing so in adopting the new ways of working.

These changes should start small, with a focus on mainly small business units that would benefit from being ‘converted’ to an Office 365 Group. The changes that come with an Office 365 Group include:

• Replacing shared mailboxes and some (mostly smaller) distribution lists with Office 365 Group mailboxes – still accessed from Outlook.
• Replacing network file shares with a File Explorer-based view of the Group’s SharePoint site libraries (via the sync option). This will allow end-users to continue to work in a familiar way. Note that added metadata is not visible from File Explorer and certain metadata options, such as making a column mandatory, will cause the File Explorer view to become read only.
• Introducing end-users to MS Teams initially for one-to-one chat, and pointing out how their Group also has a Team where they can chat and access the SharePoint site and other resources.
• Demonstrating how the browser-based version of a SharePoint site can show (via the ‘Activity’ web part) emails coming from the Group’s mailbox. And that document libraries have a range of additional document and records management functionality.
• Managing retention for all aspects of the Group, ensuring that content is kept (and can be recovered) for as long as required. This action can be hidden from end-users.

The end result should be better management of all records relating to specific subjects or needing to be kept in context.

What about other records stored in email and SharePoint sites?

This post has focused on the benefits of using Office 365 Groups to manage certain types of records within a given context. This model may not be suitable for all types of records. For example:

• End-users will probably, for years to come, create, send and manage emails in ‘personal’ email accounts. Email won’t go away as it provides a useful medium for certain types of communication.
• End-users will also continue to use their personal space in OneDrive to create and store records that should be stored in SharePoint. It is not easy to monitor this content and so end-user training is essential to ensure that final versions of records are stored in SharePoint.
• Security Groups are still valid, especially for groups of 30 or more and can be used in parallel with Office 365 Groups. For example, a small business area may have an Office 365-based SharePoint site and decide to give read only access to the members of the members of a Security Group (with different members), by adding that Security Group to the Site Visitors group. Any member of the Security Group who also happens to be a member of the Office 365 Group will continue to have Member (add/edit) access.
• High level business department/divisions may prefer to retain standard SharePoint sites with limited Member access and with Visitor (read only) access controlled via Security Groups (especially those with hundreds of members). An Office 365 Group with more than 30/50 members is still possible, but the benefits compared with using Security Groups is debatable, especially if a Team is linked with the Office 365 Group. Experience using Yammer since 2012 suggests that (a) Teams with more than 30 people can become very quiet and (b) a mailbox with more than 30 members is not useful.

Summary

Office 365 Groups:

• Should be part of the information architecture model for managing records in Office 365. They are well-suited for lower-level and/or small business units with fewer than 30 members as experience suggests that the more people in the group the less likely everyone will actively participate and contribute.
• Can and should, in many instances, replace existing functionality including shared mailboxes and network file shares.
• Will allow end-user to continue to work in familiar ways, via Outlook and File Explorer, while offering new options to communicate and collaborate.
• Should reduce the volume of ‘personal’ emails and attachments to emails.
• May enable the creation, and facilitate the storage and management of records relating to the same business context, including within a function/activity pair.
• May never replace the requirement for email or ‘standard’ high level business division or departmental SharePoint sites.

 

Setting up SharePoint Online to manage records (as part of Office 365) – Part 2/3

This is the second of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

The first post was focused on Office 365 admin: Licences, roles and Groups; Compliance Admin – Retention labels and policies.

This post focuses on SharePoint Online Admin centre configuration. 

The next and last post focuses on SharePoint site collection provisioning and configuration for the management of records. 

SharePoint Online admin centre configuration

The SharePoint Online admin centre contains a number of configuration options and settings. Most of these settings relate to the administration of SharePoint as a service and are not described further unless they relate to the management of records.

Active sites

The section named ‘Active sites’ lists all active sites, including details of storage used and when it was last modified. The list can be exported as a csv file.

The records management team should have a retention plan for every SharePoint site, including Office 365 Group-based sites and communication sites. The SharePoint Admin and the Records Manager/s to review the list from time to time to review where content is stored and if any sites could potentially be deleted. 

Creating new sites

As noted in the screenshot above, the SharePoint Admin can create a new site directly from this portal, or it may be scripted.

Organisations that are new to Office 365, and especially larger organisations that want to manage corporate records in SharePoint, might consider restricting – at least initially – the ability for end users to create new SharePoint sites, as well as new Teams in MS Teams, Groups in Outlook that also create SharePoint sites via the Office 365 Group.

If there is no control over the creation (at least initially) of SharePoint sites, the number of sites could grow exponentially with no regard to corporate recordkeeping requirements. Sites holding important records may abandoned or forgotten, or be invisible to people who need to see them.

As soon as there is sufficient critical mass in terms of SharePoint sites for business areas, and training and awareness for end users, these controls may be loosened.

There are three options to create new sites from this portal:

• Team site. These create an Office 365 Group with Members who become the Members (add/edit) of the SharePoint site. It is recommended that an Office 365 Group is created first to ensure consistency in Group naming and controls. These types of sites, with a Team in MS Teams, may work better for smaller business units or project teams with less than 30 staff. They are also more likely to contain ‘working documents’ or have content (including the connected mailbox) that can be covered by a single retention policy.
• Communication site.
• Other options (sites). The options here are Team Sites, Document Center, Enterprise Wiki, Publishing Site. Team sites created here are best for large departmental or divisional sites where access can be controlled through AD Security Groups. These types of sites are more likely to last for several years, contain formal, final versions of records stored in controlled and well-named document libraries, and be subject to more than one retention policy (including both site and library policies).

All new sites must be provisioned, which is described further below.

Admins

The SharePoint admin can only assign, from the admin portal, Site (Collection) Administrator permissions for individual SPO sites. Site Owners, Members and Visitors are assigned in the individual sites once they are created.

Generally speaking, Site Owners should work in the business unit that ‘owns’ the SharePoint site. Site Owners should not be the head of the business unit unless they are prepared to manage the SharePoint site.

Site Administrators are the Site Collection Administrators found in that section of the permissions ribbon menu for the site, under ‘Advanced permissions settings’.

Generally speaking:

• All SharePoint Admins should be Site Collection Admins
• Site Collection Admins should be grouped in a Security Group (so each site doesn’t have to be modified every time, only the SG)

If the SharePoint Admin is not listed in the Site Collection Admin group (including via the recommended SG), they may get ‘access denied’ if they try to open the site directly. They can, however, still see the site and modify the admins from the SP Admin portal.

The Primary Admin is by default ‘Company Administrator’. It is good practice to: (a) create a single SG for SharePoint Admins, and (b) remove Company Administrator as it doesn’t really need to be there – GAs can access the SP Admin portal anyway.

It is recommended that a key or senior records or information manager be added to the Site Collection Administrator Security Group added to all SharePoint sites to provide access to all to the content, if required. This can be removed on a case by case basis if there are concerns about the security of the content in those sites.

External Sharing

External Sharing is always disabled, even if it is enabled globally. A decision must be made for each site to allow external sharing.

External sharing allows records to be shared directly with external parties, rather than being attached to emails. This provides better security for those records as the ability to prevent the download the record can also be added.

Hub sites (or sub-sites?)

Hub sites (top level and ‘subsidiary’ sites) are effectively the replacement for sub-sites in SharePoint. See below regarding the architecture of SharePoint sites.

More features – Records Management

The SharePoint admin portal has a ‘classic’ setting under ‘More features’ called ‘Records Management’. This is not what it appears to be – it is in fact a way to set up ‘send to connections’ to ‘send’ (actually copy) content to a Records Center.

There are a number of problems with this (one of which is that it copies the most recent version and re-creates it in a new library) and it is not recommended for the management of records. 

OneDrive Admin

The OneDrive Admin portal includes a ‘Storage’ section that defines how much storage user’s will get as well as a setting for how long the content will be retained.

Records managers should be involved in discussions around the retention of OneDrive for Business content both while the account is active (via an Office 365 retention policy) and after the account is de-activated (via the setting here).

 

 

Setting up SharePoint Online to manage records (as part of Office 365) – Part 3/3

This is the third and last of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

The first post focused on the recordkeeping related elements in the Office 365 and Compliance admin portals, including user licences, Roles and AD Groups, and Compliance Admin – Retention labels and policies. 

The second post focused on SharePoint Online Admin centre configuration.

This post focuses on SharePoint site collection provisioning and configuration.

SharePoint Online Site collection architecture, provisioning and configuration

An example SharePoint architecture model

Architecture design model

The three primary architecture considerations are as follows.

URL path

All new SharePoint Online (SPO) sites are created under the tenant name:

• https://tenantname.sharepoint.com/

There are only two options after the tenant name element:

• /teams/. This would normally be used for logical organisational business units and projects. It includes team sites created for Office 365 Groups.
• /sites/. This would normally be used for cross organisational business units or subject areas and communication sites.

Site name and naming conventions

The site name comes after the URL path option (teams or sites).

• The URL name for the site should have no spaces (otherwise these are changed to ‘%20’), and should be limited to 14 characters. Common or not obvious acronyms should always be avoided. For example, the name ‘Business Development Management’ should not be ‘BDM’ but could be abbreviated to ‘BusDevMgt’.
• The display name for the site can have spaces, added back after the site is created. For example, ‘Business Development Management’.

The URL name and the site display name should always bear some similarity to each other.

Type of site

Three types can be created (as noted in the previous post):

• Modern. These are directly associated with an Office 365 Group which can be linked to a Team in MS Teams.
• Communication. These are the replacement for sites that were created using the ‘Publishing’ features. Generally speaking these sites store fewer records and contain or display information of an ‘informative’ nature, like the intranet.
• Standard. While this option provides the ability to create several types of sites, but the most common to use here will be a standard team site.

The type of site may be influenced by the type of content and records stored in it. Generally speaking:

• Higher level business units (department, division, with more than 30 staff) may be better as a standard site where formal records are stored. Membership of these sites can be via AD Security Groups.
• Lower level business units, often with fewer than 30 staff, may work better as modern team sites based on Office 365 Groups where all the members of the business units are Members of the Office 365 Group (and Team), rather than using AD Security Groups. These sites are more likely to contain content that is subject to change or be considered ‘working documents’.

Additional standard sites may be created ‘between’ these two levels, to meet the requirements of the organisational business unit hierarchy. The number of people in each business unit, and their need to collaborate via MS Teams, may provide a guide to the best type of site.

Keep in mind however that AD Security Groups are usually maintained by IT, while Office 365 Groups (that provide the same type of access control as SGs) are maintained by the O365 Group Owner.

Higher level standard team sites can have hyperlinks to lower level business units on the left hand navigation or in a links web part on the page. Sub-sites (except perhaps to control access to a smaller group, such as ‘Leadership’ or ‘Management’ of larger groups) should be avoided.

New site provisioning

All new sites need to be provisioned before they made available.

This usually means doing the following after they are created, noting that SPO sites that are created from Office 365 Groups may have fewer options initially in the ‘Site Settings’ area.

• Remove any left hand navigation options that are not required or could cause confusion via the ‘Edit’ menu. For example, consider removing ‘Notebook’ and ‘Pages’. Suggest you don’t remove Site Content or Recycle Bin unless really necessary – the end user can still access these via the cog/gear icon on the top right.
• Removing and adding any webparts on the main page. To do this, click on Edit and use the web part menus that appear on the left of each.
• Users and Permissions. For standard sites (non-Office 365 Group-based), go to Site Information – View all Site Settings, and click on ‘Site permissions’ This is where you add Site Owners, Members and Visitors. You can also add Site Collection Administrators here, if you forgot to do it from the admin portal. For O365 Group-based sites, click on Site Permissions from the cog/gear icon. You will see that the O365 Group Owners are now the Site Owners, and the O365 Group Members are the site Members. You can add anyone else via the “invite people’ option that includes the option to invite to the Group or just the site. You can get to the same other settings as for normal sites by clicking on ‘Advanced permission settings’.
• Look and feel. For normal sites, click here to change the display title and log. You won’t normally change anything else. On O365 Group-based sites you can change the display name from the cog icon menu – Site Information.
• Web Designer Galleries (both types). This is where you set up Site columns and Site content types. There are other options in normal sites.
• Site Collection Administration (do this before Site Actions). The only two things you will modify here are the Site collection features and the Document ID settings (when they appear, after enabling a feature). The features to enable are:
  • Document ID Service, Document Sets (optional), Open Documents in Client Application by Default, Reporting, SharePoint Server Enterprise Site Collection features, SharePoint Service Standard Site Collection Features, Three-state workflow (optional), Video and Rich Media (optional), Workflows (optional if you use SharePoint Designer to create workflows).
• Once you have enabled Document IDs, you will see that option in the settings; open this to change the default DocID prefix to be the same as the URL of the site or something very similar, to ensure it is unique (12 characters only). Document IDs are PREFIX-LibraryNumber-DocumentNumber. DO NOT enable the Site Publishing features; this is an old option that was for on-prem sites such as the intranet. It has been replaced by communication sites.
• Site Actions. Click on Manage site features and enable the following (you will see that a couple are already enabled):
  • SharePoint Service Enterprise site features; SharePoint Service Standard Site Features.
• Site Administration. Click on Regional settings to ensure the settings there are correct; they usually need to be changed for O365 Group-based sites.
• Search. There is usually no need to change anything here.

Site libraries and metadata

Every new SharePoint site has a default ‘Documents’ library.

Except perhaps for lower-level sites where the content is mostly working documents of little value, this library should be hidden or deleted and replaced with document libraries that have (a) more appropriate names, (b) metadata and if required, (c) either folders or document sets.

Document library names

As with site URL and display names, all document library have both a URL name and a display name. The URL name is the name given when the library is first created (from the gear/cog icon – ‘Add an app’).

• The URL name should ideally have fewer than 20 characters and no spaces (which will be replace by three characters ‘%20’). Consider including the year in the URL name instead of creating folders for each year, especially on team sites that may continue for many years (with a new library for each year). Separate year-based libraries may be easier to manage when it comes to retention and won’t contain an excessive number of items after several years, making it harder to isolate content. The URL name should also not repeat elements already in the site name, for example: https://tenantname.sharepoint/com//teams/tenantnamemeetings/
• The display name can be changed via the Library Settings after it is created. The display name should bear some similarity to the library’s URL name. If the library display name needs to change, consider creating a new library with and moving the content to that new library.

Standard and added metadata (columns)

Every new SharePoint site comes with a very rich set of metadata columns that can be added to each document library.

The standard set of columns available on every library are listed below:

• Type (icon linked to document)
• Name
• Modified
• Modified By
• Label applied by
• Label setting
• Retention label
• Retention label Applied
• Sensitivity
• App Created By
• App Modified By
• Check In Comment
• Checked Out To
• Comment count
• Compliance Asset Id
• Content Type
• Copy Source
• Created
• Created By
• File Size
• Folder Child Count
• ID
• Item Child Count
• Item is a Record
• Like count
• Title
• Version

The unique ‘Document ID’ metadata column will only appear when that feature is enabled from the Site Collection Administration section of the site. Document IDs have the format PREFIX-L-NNNNNN:

• Prefix. This should be changed in the Document ID set up to be the same as the site URL name, so any documents produced on that site have the same or similar name. Otherwise, the default prefix is a random set of 12 characters.
• The library ID (L). These IDs are sequential. As the ‘Documents’ library already exists, the next library will usually be ‘2’.
• The document ID (N). These IDs are also sequential and never re-used. If a document is deleted, the document ID is also removed from use. Accordingly, a gap in document IDs indicates that documents have been deleted.

Added or new metadata columns may also be created as either (a) site columns, which can be added to any library on the site, or (b) ‘local’ library columns. The advantage of using site columns is that the same column can be added to any library or list, ensuring consistency across the site.

Every new metadata column can be anyone of the following:

• Single line of text
• Multiple lines of text
• Choice (menu to choose from)
• Number (1, 1.0, 100)
• Currency ($, ¥, €)
• Date and Time
• Lookup (information already on this site)
• Yes/No (check box)
• Person or Group
• Hyperlink or Picture
• Calculated (calculation based on other columns)
• Task Outcome
• Full HTML content with formatting and constraints for publishing
• Image with formatting and constraints for publishing
• Hyperlink with formatting and constraints for publishing
• Summary Links data
• Rich media data for publishing
• Managed Metadata (open the Term Store Managed Metadata service)

Each of the above:

• Can have a description
• Is optional but can be mandatory
• Can be used to enforce unique values
• Can have a maximum length
• Can have a default value
• Can use validation formulas (for example, must have a certain form or content or an error will be display)

Site columns are added to document libraries manually from the Library settings page.

Metadata columns can be used to group content in a library instead of using folders or document sets, but this concept can be quite alien to users who generally prefer to work with folders.

Note – If metadata columns are made mandatory and a library is synced to File Explorer, the synced library will become read only in File Explorer.

Note also – Only the standard File Explorer metadata will appear in synced document libraries. If there is a requirement to see or enter more metadata, the end-user will need to work directly with the browser version.

Site Content Types

Site Content Types, from a recordkeeping point of view, allow the organisation to define a type of content that will be stored on a site. They usually have added metadata.

The main types of Content Type that are likely to be used from a recordkeeping point of view are document and document sets.

Document content types may also have a standard template attached.

SharePoint also includes the ability to use a Document Set content type. A document set is type of folder which, as the name suggests, is useful for grouping a common set of documents where additional metadata is required for the ‘folder’.

Document sets must be enabled as a feature from the Site Collection Administration section of the Site Settings, and then can be added ‘as is’, or new document sets created (from the ‘Site Content Types section of the Site Settings) based on the basic template.

An example of their use would be to store a set of documents relating to an object where there is a requirement to describe that object in the folder. For example:

• Employee. Metadata might be: name, employee ID, business area.
• Contract. Metadata might be: contract name, contract ID, expiry date.

Adding Content Types in a document library

Note – While Content Types can be very useful to ensure additional metadata is added or to group content (in document sets), end users will be required to choose the correct content type (for example, document type) and then add metadata as necessary.

This process can be off-putting and should, accordingly, only be used when there is a genuine requirement and no other obvious way to identify the ‘type’. For example, instead of forcing users to choose between two content types consider either:

• Including a metadata choice option (with a default setting) to select the option.
• Separating the library into the separate ‘types’, so that two different document types (especially with different metadata requirements) aren’t stored in the same library).
• Using file naming conventions to identify the type, e.g., ‘Contract-WilsonBrosFeb2020Final.docx’.

Library views

Standard ‘out of the box’ library list view

Every new document library displays content against a set of default columns:

• Type (icon)
• Name
• Modified (date)
• Modified by

Each item in the library has a Share option (if sharing is not disabled) as well as a three dot ellipsis menu (see below).

The list view may be modified (via All documents – Edit current view) to display more columns from the standard list as well as the document ID and any other added columns.

Views may be sorted, grouped, and filtered as required. SharePoint also includes a handy option to display all contents without folders.

Multiple views may be saved, each with a URL that can then be added as a link on the left navigation, on the page, or even sent to others. For example, a view might show a filtered view of all contracts due to expire in the coming six months.

Libraries, and views of libraries, can be embedded on the site pages. This could be useful if, for example, there is a need to see the most recent documents at a glance, or documents with an expiry date, etc.

Other document and records management functionality

All SharePoint document libraries include a range of document and records management functionality accessible from the three dot ellipsis menu, including:

• Permission/access control
• Sharing
• Copy to/Move to
• Check out/in
• Version history and the ability to restore versions
• Alerts

Every SharePoint site also includes a Recycle Bin that stores deleted content for 90 days. The Site Collection Administrator can review and restore documents deleted by anyone on the site for the previous 90 days. If there is concern about documents being deleted, any user with access can easily set an alert on the library for any changes that are made.

Retention policies

Retention policies, created in the Compliance Admin portal, work in two different ways in SharePoint document libraries.

See this post for a lot more detail on the creation and application of retention policies.

Explicit (visible) retention labels

These are published as retention policies and must be applied to each individual library.

It is not possible to delete content in libraries where this type of retention policy has been applied.

If the retention policy has the disposition review option enabled, the person or role nominated will be advised that the records are due for review; they will remain in place until a decision is made to delete them.

Implicit (invisible) retention policies

These are applied at the site level and may delete content when the retention period has ended.

There is no disposition review process, the documents are deleted by ‘System Account’ when the retention period has expired and the 90 day Recycle Bin period has finished.

Going live

If you have managed to reach the end of these three posts – well done! There is a lot to take in. Hopefully this information will help you going live with a new SharePoint implementation, a migration from SharePoint on-premise, or a clean up of either.

If you’d like me to write about other aspects of managing records in SharePoint, let me know via the feedback option here.

Setting up SharePoint Online to manage records (as part of Office 365) – Part 1/3

This is the first of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

This post focuses on the recordkeeping related elements in the Office 365 and Compliance admin portals:

• Office 365 Admin – Licences, Roles and AD Groups (including Office 365 Groups)
• Compliance Admin – Retention labels and policies (and some more options)

The second post focuses on SharePoint Online Admin centre configuration.

The third and last post focuses on SharePoint site collection provisioning and configuration to manage records. 

Office 365 admin center

The main elements that impact on the management of records in Office 365 are Users (for licences), Roles and Groups, as can be seen in the screenshot.

Users – licencing and applications

Organisations that acquire Office 365 will generally have the relevant licences required (a) to set up and administer SharePoint Online, and (b) for users to use it (and OneDrive for Business).

This post assumes that organisations will have at least an E3 licence which includes SharePoint for end users, visible as an app when they log on to https://office.com, along with all other applications included in the licence, for example Exchange/Outlook, OneDrive for Business, MS Teams and so on. End users with access to these items will also be able to download and use the equivalent mobile device apps.

Roles

The three key roles that impact on the management of records in SharePoint are as follows:

Global Admin (GA)

Global Admins:

• Are responsible for managing the entire Office 365 environment. This includes creating new Groups (Security Groups, Distribution Lists and Office 365 Groups).
• Are responsible for assigning key roles, including the SharePoint Administrator and Compliance Administrator (and other roles).
• May have responsibility for, and/or the skills and knowledge required to set up and administer SharePoint Online and create new sites for the organisation.
• May also be able to create and publish retention policies in the Compliance admin portal.

Note – Organisations that outsource the administration of Office 365 should always have at least one GA account to access the tenant if ever required. If they don’t have a log on, they should have or acquire a very good understanding of the access and privileges afforded to the outsourced company. 

SharePoint Administrator (SP Admin)

The SP Admin role will usually be a ‘system’ role that is responsible for managing the SharePoint environment, including OneDrive for Business. As noted above, a GA with the right skills can also manage the SharePoint environment. 

Generally speaking, SharePoint Administrators will focus on the technical and configuration aspects of SharePoint. They are not usually responsible for confirugint SharePoint to manage records, managing records, or creating and publishing retention policies. This role usually falls to either the GA or Compliance Administrator.

Compliance Administrator

The Compliance Admin role is responsible, among other things, for the creation and publishing of retention labels and policies in the Compliance Admin portal. A GA can perform this role (along with all other roles) if required.

Compliance Admins will usually be responsible for disposition reviews linked with retention labels, and be involved in eDiscovery cases.

The Compliance Admin can search and view the audit logs for all activity across Office 365 and can carry out broad content searches with the ability to export the content of those searches. As this role is relatively powerful, it should be limited to key senior individuals in the organisation.

Office 365 and Security Groups

Office 365 Groups are Azure/Exchange objects just like Security Groups and Distribution Lists. Accordingly, there should be controls around their creation, including naming conventions.

As every Office 365 Group has an associated SharePoint site, organisations should consider restricting the ability for end users to create Office 365 Groups, and only allowing Global Admins and members of a Security Group to do this. Neither SharePoint Admins or Compliance Admins would normally create AD Groups.

If the ability to create Office 365 Groups is not restricted, an Office 365 Group will be created with an associated SharePoint site whenever:

• A new Team is created in MS Teams.
• A new Group is created from Outlook.
• A new Yammer Group/Community is created.

External sharing

The ability to share content externally from SharePoint and OneDrive for Business is controlled from the Office 365 Admin portal. This is a global setting that can be disabled by the Global Admins if required.

It is assumed, for the purpose of this post, that that setting is enabled to allow external sharing.

Note that enabling external sharing at the global level does not enable it globally for all SharePoint sites; sites must be individually modified to allow it.

Compliance Admin

The Compliance admin portal can be accessed by the GAs and also the Compliance Admins (and some other roles). It is where retention labels and policies are created (in line with the corporate file plan/BCS) and published, and disposition reviews are undertaken, so records managers need access.

Other options in this section that relate to the management of records include the audit logs, content search and eDiscovery.

Retention policies

Retention policies may be applied to all the key workloads in Office 365 where records are stored:

• Exchange Online
• SharePoint Online
• OneDrive for Business
• MS Teams
• Office 365 Groups

Retention labels published as retention policies are visible to and can be applied by end-users. Generally these are more likely to be applied at the document library level rather than to individual records, or in mailboxes or OneDrive for Business.

Retention policies that are not based on labels may be applied to all, or parts of, the four workloads listed above. For example, they may be applied to all, or a sub-set of Exchange mailboxes or OneDrive for Business accounts, or SharePoint sites. Retention policies may also be applied to individual or team chats in MS Teams.

Organisations seeking to use retention policies in Office 365 should understand how these work, have a plan for their implementation, and keep track of what has been applied where.

• Retention policies for all mailboxes or all ODfB accounts may replace previous on-premise backup options for those workloads. It is unlikely that end-users will (or will want to) apply retention labels published as policies to individual emails or folders in mailboxes or OneDrive.
• SharePoint sites are likely to have either or a combination of explicit and implicit/invisible retention policies. Implicit, single period retention policies may be more suitable for entire smaller, short-lived SharePoint sites. Explicit retention policies may be more suitable for the diverse range of content on more complex and long-lasting sites. Some sites may be created and populated around the need to keep a particular type of record for a long period of time – for example, employee records.

Audit logs

The Office 365 audit logs are found in the Compliance admin portal. For an E3 licence, the content in the logs is stored for 90 days.

As audit logs are an important element in keeping records, organisations may need to consider ways to retain this content for a longer period.

Note – SharePoint document libraries record the name of anyone who edited a document (and also previous versions), but they don’t record the name of anyone who simply viewed it. SharePoint lists also include audit trails, making it possible to track changes in individual rows of a list.

Content searches and eDiscovery

The Compliance admin portal provides two similar options to search for content across Office 365. Both the Content Search and eDiscovery options provide the ability to establish a ‘case’ that can be run more than once.

The eDiscovery option provides the added ability to put content on Legal Hold. Advanced eDiscovery is available with a higher licence.

Next

Click on the links below to read the next two posts:

• SharePoint Online Admin centre configuration.
• SharePoint site collection provisioning and configuration to manage records.

Understanding and applying retention policies to content in MS Teams

This post highlights the need to understand how retention works in MS Teams, why it may be related to how long you keep emails (including for backup purposes), and why you need to consider all the elements that make up an Office 365 Group when considering how – and how long – to retain content in MS Teams.

Overview of retention in MS Teams

If you are unfamiliar with how retention works with MS Teams, these two related sites provide very useful detail.

• https://docs.microsoft.com/en-us/MicrosoftTeams/security-compliance-overview
• https://docs.microsoft.com/en-us/microsoftteams/retention-policies

Image from the first link above – Security Compliance Overview

The quote below from the second link is relevant to this post:

‘Teams chats are stored in a hidden SubstrateHolds folder in the mailbox of each user in the chat, and Teams channel messages are stored in a hidden SubstratesHolds folder in the group mailbox for a team. Teams uses an Azure-powered chat service that also stores this data, and by default this service stores the data forever. With a Teams retention policy, when you delete data, the data is permanently deleted from both the Exchange mailboxes and the underlying chat service.’

and

‘Teams chats and channel messages aren’t affected by retention policies applied to user or group mailboxes in the Exchange email or Office 365 groups locations. Even though Teams chats and channel messages are stored in Exchange, they’re only affected by retention policies applied to the Teams locations.’

In summary:

• One-to-one chat in MS Teams is stored in a hidden folder of the mailbox of each user in the chat. Documents shared in those chats are stored in the OneDrive for Business of the person who shared it.
• Group chat in Team channels is stored in a hidden folder of the mailbox of the associated Office 365 Group – and also in an Azure chat service. Documents are stored in the Office 365 Group’s SharePoint site (other SharePoint site libraries may also be linked in a channel).

Another quote from the same post:

‘In many cases, organizations consider private chat data as more of a liability than channel messages, which are typically more project-related conversations.’

Teams content is kept in mailboxes, retention may be similar

Typically, in the on-premise past, organisations will have backed up their Exchange mailboxes (and possibly also enabled journaling, to capture emails), for disaster recovery, ‘archiving’ and investigations. Unless a decision is made to invest in cloud back-ups, Office 365 retention policies may also be applied to Exchange mailboxes, effectively replacing the need to back them up. Retention policies applied to Exchange mailboxes don’t affect the teams chat folder.

Organisations should probably apply the same retention period to both emails and Teams chats as they do to email mailbox backups now. That is, if mailboxes are typically kept for 7 – 10 years after the person leaves the organisation, then keep the Teams chats for the same period.

Note that, even if a poster deletes an item (if that option is enabled), it will still be retained if there is a retention policy.

Suggestions for retention in MS Teams

As there can be different retention requirements, depending on the subject matter, here are some suggestions for retention:

• One-to-one chat is like email, you will never know everything that is being said or sent. So a single retention policy that mirrors email would be appropriate.
• Teams chat is more likely to be about the subject of the Team, which is based on an Office 365 Group, its own mailbox, and has a SharePoint site. In this case, you could consider a retention policy applied to all Office 365 Groups or specific Groups – for example ‘Project Groups’, then ensure that the retention policy or policies cover all aspects of the Office 365 Group (mailbox, team chat, SharePoint).
• If all the records relating to a particular subject matter (including email, chat and documents) must be retained for 25 years, then you need to understand all the options.

It underscores the need to plan carefully for retention management for all the key workloads in Office 365.

Creating a Team in MS Teams creates a SharePoint site – and more

In recent weeks a number of organisations with ‘default’ Office 365 configuration settings have told me they are not using SharePoint but they are using MS Teams, and have even created new Teams.

Every new Team in MS Teams creates a linked SharePoint site via the Office 365 Group that is created when the Team is created. If the ability to create Office 365 Groups is not restricted the following is likely to happen:

• Naming conventions go out the window. New Teams and SharePoint sites will probably be created with random names (eg ‘Andrews Team’, ‘Footy tipping’).
• The SharePoint environment will ‘go feral’; new sites will not be provisioned according to business requirements.

This post describes what happens when a Team is created and recommends the creation of new Teams by creating an Office 365 Group.

What happens when a Team is created

At the bottom left of the MS Teams client is the option to ‘Join or create a team’. This option will be visible even if the ability to create Teams is not enabled for end users (because the control is on the creation of Office 365 Groups).

The dialogue box that opens gives the option to ‘Create Team’.

The user now has the choice to build a new team from scratch or create it from an existing Office 365 group or team. For the purposes of this post, we will assume the user chooses the first option.

The user is then asked if the team should be private, public or organisation wide. The options will affect the visibility of the Team to others. For the purpose of this post, the new Team is ‘Private’.

The next option is to name the site (‘Footy Tipping’) and give it a description.

The user is then prompted to add members (people who have edit rights) to the new Team. They may add individuals by name, a distribution list, or a security group. If external access is allowed, they may also add people outside the organization as guests. People or groups that are added are made ‘Members’ by default but this may be changed to ‘Owners’.

A key point here is who will have access to the Team if there is a single Owner. What if that person leaves the organisation?

The new Team has been created with a ‘General’ channel. The three dots to the right of the name allow the Owner to modify the members of the Team, add channels, get a link to the Team (to send to others and delete the Team.

Along the top of the new Team are three default tab: Posts, Files, Wiki.

The ‘Files’ tab appears (for those who are new to this) to allow documents to be uploaded to the Team, Synced to their File Explorer and so on. This is actually the default Documents library of the SharePoint site that is created when the Office 365 Group is created when the Team is created.

What happens in Office 365 Groups

The end user is not likely to care much about what happens anywhere else, they have a new Team and can start chatting.

Meanwhile, in the Groups area of the Office 365 Admin portal, a new Office 365 Group appears. The Global Administrator should be keeping an eye on the creation of new Groups, if they are not controlled, especially if there is a requirement to adhere to naming conventions for all AD Groups (Distribution Lists, Security Groups, and Office 365 Groups).

The Group name has had the space removed in the Group’s email address (and, as we will see, in the SharePoint site). The Global Admin can review and change the Members.

The Global Admin may also changed the settings to allow external senders to email the Group and to send copies of Group conversations (in Outlook, see below) and events to Group members. (The Microsoft Teams settings takes the Global Admin to the MS Teams Admin portal).

So, an end user has ‘simply’ created a Team, but now there is a new Office 365 Group with a mailbox (not visible but can receive emails) and a SharePoint site.

What happens in Outlook

Every new Office 365 Group has an Exchange mailbox, similar to a shared mailbox, but when a new Team is created from MS Teams, the mailbox is not visible in Outlook. If the Global admin enables the ability to ‘send copies of group conversations and events to group members’, the group members may use that Group’s mailbox address.

The mailbox is visible when a Group is created first, which is a good reason to create a new Team by creating the Office 365 Group first.

Channel chat message are stored in a hidden folder in the Group’s mailbox, where they are subject to any retention policy applied to the chat messages, separate from any retention policy applied to the mailbox.

What happens in SharePoint

As noted already, every new Team gets a SharePoint site because the Team has created an Office 365 Group.

The SharePoint Admin will see the new site in the SharePoint admin portal:

The SharePoint Admin may, via the ‘Permissions’ section, view and update the Group Owner/s and also may add additional ‘Admins’. They may make the site a Hub site and decide whether the site can be shared externally or not (the default is not shared externally).

The SharePoint admin may also delete the site – but consider that it is not now just a site but a Team and also an Office 365 Group. Some care needs to be taken here – which should be deleted first, and what happens if a retention policy has been applied to the Teams channel or the Office 365 Group?

If the SharePoint admin opens the site they will see a standard ‘modern’ team site with a single default document library. This is the ‘Files’ library that appears as a tab in the Teams General channel.

In the Permissions section of the site, the Site Owners show as the Team owners group, and the Site members (add/edit rights) show as the Team members group. There are no site visitors.

If the SharePoint admin goes to Advanced permissions settings and clicks on Site Collection Administrators they will see that only the Footy Tipping Owners are in this section. Organisations should consider adding a Security Group, that includes any records or information managers, in this section. Otherwise, any records will be more difficult to manage and the records managers will need to request access from the SharePoint admin.

Two important points that are sometimes missed:

• Aside from the Global and SharePoint admin, only the Team Owners and Members can access the SharePoint site.
• The SharePoint site may be shared with another person (or Group) and given Member or Visitor access but this does NOT give them access to the Team channel. They need to be added to the Team Owners or Members to have access to the Team channel.

Summary

Allowing end users to create a Team in MS Teams has a flow-on effect:

• It creates an Office 365 Group with an associated SharePoint site
• It creates an Exchange mailbox
• It will (initially, unless this is changed) make the SharePoint site inaccessible to records managers.
• It gets complicated if it is decided to delete the Team, SharePoint site, or Office 365 Group.

It is recommended, in organisations rolling out MS Teams to end users, that the ability to create Office 365 Groups is disabled except for Global Admins, and any new Team is created from a new Office 365 Group that includes the option to ‘Add Microsoft Teams to your group’, as shown below:

This will result in the following outcomes:

• Controlled creation of Office 365 Groups, SharePoint sites and Teams, with appropriate naming conventions.
• A new and visible mailbox for both the Group and the Team.
• Stop SharePoint from ‘going feral’ and becoming uncontrolled.
• Establish better governance controls for recordkeeping.