A Tale of Two Retention Policies in Microsoft 365

There are two ways to create retention policies in Microsoft 365 – (a) by creating a label and publishing it as retention policy, or (b) creating a retention policy without a label. Both are created from the Information Governance section of the Compliance portal in Microsoft 365.

This post describes how the policies are created and applied, and the outcome for each.

Retention policy based on a label

Retention labels are created from the Labels tab of the Information Governance section in the Compliance portal.

Ideally, the name of the label would be based on a disposal class in a records retention schedule or records disposal authority. For the purpose of this post, the new label was named ‘Test One – Retain five years then delete‘.

Adding the details of the retention provided a quick visual clue to the retention details.

The label was configured to have a retention period of 5 years after the date modified after which the content could be deleted, with no disposition review. No record would be kept of what was deleted.

Alternatively, it would be possible to select the Disposition Review option. This option sends an alert to the account indicated so the records could be reviewed before disposal. However, keep in mind that if the trigger is left as ‘Date created’ or ‘Date modified’, records will be ready for disposition review based on those dates. It might be better to choose ‘Date applied’ instead so that a group of records could be reviewed at the same time – for example, in a SharePoint library.

Retention labels have no life unless they are published. This is achieved from the ‘Publish labels’ option in the same tab.

One or more labels can be published in a retention policy. This could be useful if the records disposal authority contained multiple labels for a single function.

As part of the publishing process, a decision has to be made to apply the policy to somewhere in Microsoft 365. In this case, the decision was made to apply it to a single SharePoint site because that site contained records that mapped to the retention label.

Finally, the new policy was given a name. In this case the policy had only one label so the policy had the same name as the label.

If the organisation’s records disposal authority/retention schedule had multiple retention classes mapped to a business function, the policy could be named after the function and include all the relevant classes created as labels.

All of these labels would appear in the drop down list from the Library Settings on every site where the policy was applied.

But even when it was published in this way and applied to a specific SharePoint site, the label does not do anything. The policy has to be applied to a document library in that site – but even then it wouldn’t do anything until the label was selected from a drop down list.

The screenshot below shows how the specific label is selected on the library and whether it should be applied to all existing items.

The label can now be seen in the ‘Retention Label’ column (when made visible in the view settings).

Anyone who uses the library to add and edit content (site members) will notice an obvious change if they try to delete anything, including from their synced document libraries in File Explorer.

They can’t.

There is, however, a workaround that end-users can use to delete documents with a label: (a) check the box next to the document, (b) open the information panel and (c) remove the label on individual documents.

Once the label is removed it no longer appears against the record.

The document can now be deleted. It will move to the Recycle Bin, from where it could be restored for up to 93 days if a mistake was made.

The lessons to be learned from this are that:

• It might be useful to make the library read only after the label is applied.
• The label should only be applied when the library became inactive, to allow people to get on with their work while it is still active.

If anyone was concerned about content being deleted from an active library, they could set an alert on the library.

In the meantime, for all the other records that retained their label, when the retention period ended the records in the library would be transferred to the Recycle Bin where they would sit for 93 days before complete and permanent deletion.

If no-one restored a document from the Recycle Bin, the content would be deleted, permanently, forever, without any record. The records managers might need to make a copy of the metadata for the documents to be deleted before they are deleted.

Alternatively, if they had selected the option for a ‘disposition review’ when the label was created, this would alert the records managers (and others) of the need to review the contents of the library.

Once everything had been deleted, all that will be left will be an empty library with no record of what used to be there.

Retention policy without a label

Non-label retention policies are created from the ‘Retention’ tab of the Information Governance portal.

For the purpose of this post, this label will be named ‘Label Two – Retain 5 years then destroy’.

This policy has the same retention period, trigger, and action as Label One – 5 years after date modified – and then the content could be deleted without any record of this being kept. [Note – yes the screenshot below shows 7 years].

As the policy was being created it was applied directly to a SharePoint site.

The policy could start working immediately.

Nobody, apart from the person who created and applied it knew it had been applied. It didn’t appear anywhere on the SharePoint site where it had been applied.

But, as part of the application process, it created a Preservation Hold (PH) library visible only to Admins, including the Site Collection Admins but not the Site Owners.

(Note – if end-users are allowed to create SharePoint sites, or Teams in MS Teams that also creates a SharePoint site, they are made Site Collection Admins by default and can accordingly see the Preservation Hold Library).

Anything that was deleted in advance of the retention policy expiry would be moved to the Preservation Hold library.

The PH library had a specific purpose – to ensure that everything on the site was retained until the end of the retention period.

It was not possible to restore any items from the Preservation Hold library. It was also not possible to ‘clear’ that library. If an administrator tried, they would be advised that ‘something went wrong’.

The only way to delete from the Preservation Hold library would be to remove the policy.

However, if a record was deleted it would be placed in the Recycle Bin for up to 93 days, from where it could be restored. If a record was restored in this way (from the Recycle Bin), the copy in the Preservation Hold library would remain.

For most people this type of policy was fine. They could (appear to) delete records, unaware that nothing was actually deleted if it were subject to a retention policy.

The same thing happened if a similar type of retention policy was applied to OneDrive accounts. End-users could ‘delete’, but nothing was actually deleted as long as the retention policy was active.

At the end of the retention period, the content that was subject to this policy was transferred to the Recycle Bin where it remained for 93 days. It would then be deleted, permanently, forever, without any record.

All that was left would be an empty library with no record of its previous content. The only clue would be in the ID or the ID part of the Document ID (if enabled). If anything new was added to the library, the next number in sequence would indicate how many records had previously been saved to that library.

Lessons to be learned

There are several lessons to be learned.

The first is that you need a plan to create and apply retention labels or retention policies to content across Microsoft 365. The configuration options and implications of each option need to be understood before they are applied.

You need to understand the difference between (a) a retention label published as a policy and applied to a SharePoint site and then a library on that site, and (b) a retention policy applied to an entire SharePoint site.

Retention labels can be mapped to retention schedule/disposal authority classes and then applied to specific libraries on SharePoint sites where the policy is applied. Accordingly, it may be useful to create document libraries that map to those classes, and only store content in those libraries that is covered by a single retention class.

If single SharePoint document libraries are used to store content that maps to different retention classes, then it will become very complicated to accurately map retention labels to content.

In many cases, it may be possible to apply a single non-label retention policy to a single SharePoint site or sites. For example, all project sites might have the same 7 year retention applied to all the content on the site as there may be no real need to apply policies to individual libraries.

Whatever you do, have a plan to act, document your settings, and keep it up to date.

And understand what happens at the end of the retention period. There is no back up.

Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

There are two types of chat message in MS Chat:

• One-to-one/one-to-many ‘chat’ (top icon above).
• Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

• Knowing if chat messages on any given subject exist. 
• Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

Searches can be configured to find content in any or all of the following locations:

• Users, Groups, Teams
  • Exchange email
  • Office 365 group email
  • Skype for Business
  • Teams messages [the copy in the mailbox]
  • To-Do
  • Sway
  • Forms
• SharePoint
  • SharePoint sites
  • OneDrive accounts
  • Office 365 group sites
  • Teams sites
• Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

• As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
• There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

Classification

Data-based records

(Note – the image above is a small ticket dated 1956 from my grandmother’s visit to Denmark. I used this because the word ‘Kontrolbillet’ seemed appropriate for this post.) In response to several queries about this following my previous post about whether it is possible to manage records as data, it seemed apparent that the […]

July 27, 2020

Classification

Managing digital records as data

(Note – while drafting this post I became aware of an MA Dissertation on the subject of ‘Artificial Intelligence and Record-keeping’ being developed by Mohamed Ben Tahayekt at University College London. I have not had access to this material but I believe some of the concepts may be similar to those outlined in this post.) […]

July 23, 2020

Classification

Microsoft 365 – Options for collating, grouping and aggregating records

The international standard for records management, ISO 15489-1:2016 (‘Information and documentation – Records management – Part 1: Concepts and Principles’), defines records as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’. Among other things, the standard notes that […]

July 2, 2020

Applying multiple retention policies to a SharePoint Online site

Many organisations have complex records retention requirements that are described in records retention schedules, disposal authorities or records authorities. For example:

• There may be different ‘levels’ of retention depending on the ‘state’ of a record. The final versions of certain records may have a longer retention requirement than the working versions.
• For each business function there may be multiple types of records, each with their own retention requirement or ‘class’.
• In some disposal authorities based on business functions, activities that produce records (for example ‘Meetings’) may appear in multiple functions with the same retention requirement.

This post describes multiple and different types of Microsoft 365 retention policies created with an E3 licence in the Information Governance section of the Compliance admin portal can be applied to a single SharePoint site.

Example retention schedules/disposal authorities

Most records retention schedules or disposal authorities list types (or ‘classes’) of records that are created or captured by the organisation, including through the completion of various activities or transactions, and define how long these records must be kept or retained by the organisation (or transferred to an archival institution).

These record types or classes are usually grouped, by business subject or function.

The following extract, from a private sector company records retention schedule, shows records grouped by subject type (‘Company records’).

In the example below, from the Victorian (Australia) government, records are grouped by function (‘Enquiries and Complaints’).

The diagram below presents a simple view of the examples above. For every subject type or business function, there may be one or more records description (based on the activity or transaction that creates or captures the record) with a corresponding retention period.

How does SharePoint manage records?

SharePoint Online team sites (including the sites linked with Microsoft 365 Groups and MS Teams) may be created to manage the records for a particular business area or function, or for a specific business activity.

Whether a single or multiple document libraries are used, SharePoint sites may contain a mix of record content. It may not always be possible to apply a single retention policy to the site.

Use case

For the purpose of this post, we will assume that the organisation has a business function named ‘Client Services’ – a generic name for a business unit that delivers client services.

The Client Services area has several SharePoint sites. One of these sites is named ‘Client Services’.

The ‘Client Services’ site, which has been active for several years, has multiple libraries for the activities it performs, including ‘Meetings’, ‘Procedures’, ‘Working papers’, ‘Rosters’, ‘Marketing’ and so on. Most of these libraries are created annually and consequently the year is added to the library name to help group content more efficiently – for example, ‘Meetings 2018’, ‘Meetings 2019’.

The organisation’s records retention authority has multiple classes for the Client Services function, including:

• Marketing – Retain for five years
• Meetings – Retain for seven years
• Procedures – Retain for seven years.
• Rosters – Retain for ten years

There is no class for general ‘working papers’ that may be created in support of the above activities, but the organisation would like to ensure that all content not otherwise covered by one of the ‘explicit’ retention policies above is retained by an ‘implicit’ or background policy.

Creating the Office 365 retention policy

Based on its requirements, the organisation will require two different options.

• A single retention policy with a minimum three year retention for content (including ‘working papers’) not covered by any other longer retention period. This will be created as an ‘implicit’ or background policy and applied to the site. Any content that is deleted by the end users will be moved to the invisible (to end users) Preservation Hold library. Records covered by this policy will be automatically deleted – via the Recycle Bin – at the end of the retention period.
• Multiple retention labels published in a single retention policy, that is applied on this site or other sites that can be mapped to the same function. This means that, when applied to a document library, every one of the labels will appear in the drop down menu in the library settings to apply a label. Depending on how the label has been configured, the records may be automatically deleted or subject to a disposition review.

Label-based retention policies – retention settings

Each retention label that is created will include a name and description, and then the label retention settings.

• How long it is to be kept (e.g., 7 years).
• What happens at the end of that period (delete automatically, disposition review, nothing).
• Trigger for disposal – date created, date modified, date labeled. The ‘Date labelled’ option is preferred as it will not prevent day-to-day actions on the library or make the synced version read-only.

This process is repeated for each label. Each label can include the ‘File Plan’ settings, for example any reference numbers, the Function and Activity, and so on.

Here are two of the labels that have been created:

Publishing the labels

After each label has been created, they can then be published together in a single (‘Client Services) retention policy that is applied to the site (Client Services).

The published policy now appears in the list of label-based retention policies. It also appears under the ‘Retention’ tab of the Information Governance section, along with all other published label-based policies and policies that are not based on policies.

Non-label retention policy

The ‘implicit’ or background policy is created directly as a retention policy, without the need for a label. This policy, named ‘Temporary records’, has a three-year retention. It is applied directly to the site (or multiple sites).

Applying the label-based policies to the site

The Client Services site has several libraries as shown below.

We want to apply the label-based policies to the libraries named ‘Meetings 2020’, ‘Rosters 2020’ and ‘Marketing’. The general ‘Documents’ library will be covered by the implicit retention policy for ‘Temporary records’.

To apply the label-based policies to the library, click on the library and navigate to Library Settings where the option to ‘Apply label to items in this list or library’ is found.

A drop down list shows all available label-based policies. As the ‘Client Services’ policy was only applied to this site, only those labels appear. Only one option can be selected for each library.

It is usually a good idea to check the box (hidden behind the list of policies in the screenshot below) to ensure that anything already stored in the library will be covered by the policy.

The Meetings 2020 library has now been assigned the Client Services Meetings – 7 years policy. As soon as this label as been applied:

• It will no longer be possible to delete any content.
• If the library has been synced to File Explorer, the library in File Explorer will become read only.

The only way to to remove this restriction is to remove the policy. Accordingly, it may be better to apply the label only when the library has become inactive.

Note – The Temporary records implicit policy will continue to operate in the background and will apply to any content in any library or list not covered by an explicit policy. Anything deleted will be moved to the Preservation Hold Library accessible only by the Site Collection Admins or higher.

The final model can be visualised as follows:

 

The longest retention option will always take precedence. So, if an explicit label-based policy has a retention period of 2 years, and the background implicit retention policy has a retention of 5 years, the content will be kept for 5 years.

Note also that only the content of the libraries or lists is deleted at the end of the retention period. The library or list – and the site – remain.

Conclusion

As described in this post, it is possible to create multiple retention policies and apply them only to a single SharePoint site.

This allow organisations to create targeted groups of retention policies which is likely to be useful in organisations with detailed or function/activity based retention schedules.

Planning is required to ensure that there is appropriate and effective retention coverage for all the content created and captured in all SharePoint sites.

Planning for records retention in Office 365

Office 365 is sometimes referred to as an ‘ecosystem’. In theory this means that records could be stored anywhere across that ecosystem.

Unlike the ‘old’ on-premise world of standalone servers for each Microsoft application (Exchange, SharePoint, Skype) – and where specific retention policies could apply (including the Exchange Messaging Records Management MRM policy), the various elements that make up Office 365 are interconnected.

The most obvious example of this interconnectivity is Microsoft Teams which stores chat content in Exchange and provides access to content stored in both SharePoint (primarily the SharePoint site of the linked Office 365 Group) and OneDrive, and has links to other elements such as Planner.

Records continue to be created and kept in the various applications but retention policies are set centrally and can apply to any or all of the content across the ecosystem.

Managing records in Office 365, and applying retention rules to those records, requires an understanding of at least the key parts of the ecosystem – Exchange, Teams, SharePoint and OneDrive and how they interrelate, and from there establishing a plan for the implementation of retention.

What types of records are created in Office 365?

Records are defined as ‘evidence of business activity’ and are often associated with some form of metadata.

Evidence of business activity is an overarching term that can include:

• Emails
• Calendars
• Documents and notebooks (in the sense of text on a page)
• Plans, including both project plans and architectural plans and diagrams
• Images/photographs and video
• Chat and/or messages
• Conversations (audio and/or video based)
• Social media posts

All digital records contain some form of metadata, usually displayed as ‘Properties’.

Where are the records stored in Office 365?

Most records created organisations using Office 365 are likely to be created or stored in the following parts of the ecosystem:

• Exchange/Outlook – for emails and calendars.
• SharePoint and OneDrive – for documents and notebooks (in the sense of text on a page), plans, images/photographs and video.
• Stream – for audio and video recordings.
• MS Teams – for chat and/or messages, conversations (audio and/or video based). Note that 1:1 chats are stored in a hidden folder of the Exchange mailbox of the end-user/s participating in the chat, while Teams channel chat is stored in a hidden folder of the linked Office 365 Group mailbox.
• Yammer – for (internal) social media posts.

It is also possible to import and archive certain external content such as Twitter tweets and Facebook content in Office 365.

The diagram below provides a overview of the main Office 365 applications and locations where records are created or stored. Under SharePoint, the term ‘Sites’ refers to all types of SharePoint sites, including those associated with Office 365 Groups. Libraries are shown separately because of the potential to apply a retention policy to a library – see below.

Note also that this diagram does not include network file shares (NFS) as the assumption is made that (a) NFS content will be migrated to SharePoint and the NFS made read only, and (b) all new content that would previously have been stored on the NFS is instead saved either to OneDrive for Business (for ‘personal’ or working documents) or SharePoint only.

Creating a plan to manage records retention across Office 365

In previous posts I have recommended that organisations implementing Office 365 have the following:

• A basic architecture design model for SharePoint sites, including SharePoint sites linked with Office 365 Groups (and Teams in MS Teams).
• A plan for creating and applying retention policies across the ecosystem.

Because SharePoint is the most likely location for records to be stored (aside from Exchange mailboxes and OneDrive accounts), there should be at least one retention policy for every SharePoint site (or group of sites), as well as policies for specific document libraries if the retention for the content in those libraries may be different from the retention on the overall site.

For example, a ‘Management’ site may contain a range of general content as well as specific content that needs to be retained for longer. 

• The site can be covered by a single implicit retention policy of (say) 7 years. This policy will delete content in the background, based on date created or data modified. 
• The document library where specific types of records with longer or different retention requirements are stored may have one or more explicit label-based policies applied to those libraries. This content will be retained while the rest of the site content is deleted via the first policy.

Structure of a retention plan for records in Office 365

A basic plan for creating and applying retention policies might look something like the following:

• User mailboxes – one ‘general’ (implicit) retention policy for all mailboxes (say, 7 years after creation) and another more specific retention policy for specific mailboxes that require longer retention.
• SharePoint sites – multiple (implicit) retention policies targeting one or more sites.
• SharePoint libraries – multiple (explicit) label-based retention policies that are applied manually. These policies will usually a retention policy that is longer than any implicit retention policy as any implicit site policy will prevent the deletion of content before it reaches the end of that retention period.
• Office 365 Groups (includes the associated mailbox and SharePoint site) – one ‘general’ (implicit) retention policy. See also below.
• Teams channel chat – one ‘general’ (implicit) retention policy. Note that this content is stored in a special folder of the Office 365 Group mailbox.
• 1:1 chat – one ‘general’ (implicit) retention policy. This content is stored in a special folder of the participant mailboxes.
• OneDrive documents – one ‘general’ (implicit) retention policy for all ODfB accounts, plus the configuration of retention after the account is inactive.

At a high level, the retention policy plan might look something like the following – ‘implicit’ policies are shown in yellow, SharePoint document libraries may be subject to ‘explicit’, label-based policies. The ‘+7 years’ for OneDrive relates to inactive accounts, a setting set in the OneDrive Admin portal.

Regarding Microsoft Office 365 Groups, Microsoft notes the following on this page about managing retention in Office 365:

To retain content for a Microsoft 365 group, you need to use the Microsoft 365 groups location. Even though an Microsoft 365 group has an Exchange mailbox, a retention policy that includes the entire Exchange location won’t include content in Microsoft 365 group mailboxes. A retention policy applied to an Microsoft 365 group includes both the group mailbox and site. A retention policy applied to an Microsoft 365 group protects the resources created by an Microsoft 365 group, which would include Microsoft Teams.

The actual plan should contain more detail and included as part of other recordkeeping documentation (perhaps stored on a ‘Records Management’ SharePoint site). The plan should include details about (a) where the policies have been applied and (b) the expected outcomes or actions for the policies, including automatic deletion or disposition review (for document libraries).

Keep in mind that, unless the organisation decides to acquire this option, there is no default backup for content in Office 365 – once a record had been deleted, it is gone forever and there may be no record of this beyond 90 days.

Managing emails as records in Office 365

Records managers have been struggling with managing emails as records ever since they first appeared in the workplace.

For a long time the accepted practice, as with other digital records, was to print them out and put them on the appropriate file. With the introduction of electronic document and records management (EDRM) systems, end users were instead required to save or copy documents and emails to an electronic ‘file’ in that system.

The old paradigm where emails were copied to an EDRMS or a paper file

In both cases, the emails remained in the user’s ‘personal’ mailbox, where they remained inaccessible for ‘privacy’ reasons. End-users and business areas would (and still do) conduct business via the email system, without these records being available to anyone except the sender and recipient/s. Attachments to emails sent to individual recipients were (and continue to be) not managed as records unless they were printed out or saved to the EDRMS.

Microsoft Office 365 has changed the paradigm for keeping records as described in the linked post, away from the central storage and management of records in one system (while leaving the originals in place), to the decentralised ‘in place’ storage and centralised management of records across Office 365.

This post provides an overview of the three main options for managing email as records in Office 365, in both Exchange and SharePoint.

In summary the options are:

• Leave emails in place in Exchange mailboxes (personal and Office 365 Group mailboxes) and apply one or more Office 365 retention policies to mailboxes.
• Same as previous point, and use Content Search to retrieve emails as required.
• Same as previous point, and only copy specific emails to SharePoint

Keep in mind while reading this post that chat content from MS Teams is also stored in Exchange mailboxes but that content cannot be copied to SharePoint.

Option 1 – Leave emails in place and apply retention policy

In this option, emails remained stored in personal or Office 365 Group mailboxes. End users may create folders and ‘categorise’ the content as they wish, but no additional attempt is made to further categorise, add metadata to, or group the content according to recordkeeping requirements. The aggregation, from a recordkeeping point of view, is the end-user or Office 365 Group.

All mailboxes are subject to one or more retention policies set in the Office 365 Compliance portal to ensure that no emails are deleted before a pre-defined minimum period.

Note that retention policies can effectively replace a back-up regime used by IT for disaster recovery and investigation purposes purposes.

Positives

Emails are aggregated by user name or Office 365 Group and will remain in mailboxes for a minimum period of time as set by the retention policy. 

Office 365 Group mailboxes provide the ability to group emails by a more specific subject  (the Group name, which could map to a business function – e.g., ‘Correspondence Management’) and have the added positive of having an associated SharePoint site.

Negatives

The negative with this option, from a recordkeeping point of view, is that all emails – regardless of subject or importance – are grouped by the ‘personal’ or Office 365 Group mailbox, and kept for the period defined in the retention policy. That is, there is no differentiation between (email) records that may need to be kept for a long period of time and those that are transient in nature.

If there is a requirement to ensure that certain emails are kept in different aggregations or for different periods of time, then option 3 should be considered.

Option 2 – Same as option 1 and use Content Search to retrieve emails

This option is the same as the first option, but the business can make use of Content Search to identify and isolate emails as required. Content Search is more or less the same as the search part of an e-Discovery case.

The Content Search option is accessed from the left hand navigation of the Compliance portal: https://compliance.microsoft.com/homepage

Note that access to the Content Search area is restricted to Office 365 Global Admins and Compliance Admins. This is because, as can be seen in the screenshot below, a Content Search can be set up to search for any content in email, documents and much more.

Content Searches can be set up from the ‘New Search’ option, or the Administrator can make use of a Guided search or Search by ID List. For the purpose of this email, only the ‘New search’ will be examined.

Configuring a new Content Search

Each content search can be configured against three main options as shown in the screenshot below: Keywords, Conditions, and Locations. Some searches may require a combination of these three options.

Keywords can be any words that may be found anywhere in the email, including the content.

The available conditions are listed below:

Common

• Date
• Sender/Author
• Size (in bytes)
• Subject/Title
• Compliance label

Emails

• Message kind
• Participants
• Type
• Received
• Recipients
• Sender
• Sent
• Subject
• To

Documents

• Author
• Title
• Created
• Last modified
• File type

The available search locations include any or all of the options below:

• Exchange email
• Office 365 group email
• Skype for Business
• Teams messages
• To-Do
• Sway
• Forms
• SharePoint sites
• OneDrive accounts
• Office 365 Group sites
• Teams sites
• Exchange public folders

For more detail on how to use Content Search and all the options available, go to this Microsoft site.

Running a search

After the search has been configured, it must be run. The speed of the search will depend on the complexity of the search, conditions, locations and the volume of content. Every search will appear in the list of searches that have been saved.

When complete, the search result will show a ‘Status’, showing the number of:

• Items found
• Unindexed items
• Mailboxes
• Sites
• Public folders

Exporting results

Once the search has completed, the results of the search may be exported. There are two  configurable options for exported results.

Output options:

• All items, excluding ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• All items, including ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• Only items that have an unrecognized format, are encrypted, or weren’t indexed for other reasons

Exchange content export options:

• One PST file for each mailbox
• One PST file containing all messages
• One PST file containing all messages in a single folder
• Individual messages
• Enable de-duplication for Exchange content (check box)

Positives

Content searches are likely to find and retrieve more relevant emails than might be saved elsewhere, as it looks through all emails. Provided a retention policy has been applied to the mailboxes, the content should still be accessible. If the emails have been deleted at the end of a retention policy, they will not be accessible any more.

Emails can be exported and – if necessary – the PST copied to a different system (such as SharePoint) for long-term storage with additional metadata as required.

Negatives

Access to the Content Search option is restricted to Global Admins and Compliance admins, for good reason. Consideration might need to be given to governance or procedural rules. Note that Global Admins are always alerted when a new content search is created or run.

Each search must be pre-configured and run regularly to ensure that all emails are identified.

Content searches may retrieve too much unrelated content.

Option 3 – Same as option 2 and copy only select emails to SharePoint

This option mimics the legacy way of saving a record to a pre-defined separate aggregation, in this case to a SharePoint document library.

It differs from the first two options in that only certain select emails are copied (by end-users or using a third-party application) to specific SharePoint document libraries. It is still, however, possible (and preferable) to apply a retention policy to the original mailboxes.

Content search, which can be used at any time, will find the emails in both Exchange mailboxes and SharePoint as long as they have not been deleted via a retention policy expiry .

Positives

The positives with this option are that emails copied to a SharePoint document library:

• Are grouped with other related records. This may be important from an organisational recordkeeping point of view, for example for certain key records. Consideration might also be given to setting up an Office 365 Group instead for these specific records.
• Can have additional metadata.
• Can be retained for a period of time, different from the original mailbox.

Negatives

The problems with this option are that:

• It requires some kind of action to copy the email.
• It creates a copy of the email, it doesn’t remove the original.
• An email copied to another system may not be the most recent in a thread, especially if that thread is still active.
• Does not include the ‘chat’ elements from MS Teams.

Summing up the options

The idea of copying an email to a separate aggregation, container or file for recordkeeping purposes is a legacy concept inherited from the paper recordkeeping period. While attempts were made over the years to mimic that concept in EDRM systems, it has several weaknesses that mostly outweigh the alleged benefits.

Email (in Exchange) and documents (in SharePoint) continue to remain separate in Office 365 but there is now the potential to manage both equally through a combination of retention policies and pre-defined content searches.

The majority of business emails are never captured in separate recordkeeping systems. Microsoft’s centralised retention model and ability to apply to retrieve emails on the fly mean that it is more efficient and cost effective to leave emails in place. This does not exclude the potential to copy certain select emails to SharePoint.

Additionally, mailboxes associated with Office 365 Groups provide the ability to keep emails in a business context, away from inaccessible ‘personal’ email accounts. Records managers should consider the potential of using Office 365 Group mailboxes in this way for particular types of records.

Recordkeeping metadata in Office 365 and SharePoint

Records management standards (see below) state that a defining feature of records is that they are associated with metadata – both ‘point of capture’ metadata and ‘process’ metadata that continues to evolve throughout the life of the record.

For at least two decades, the requirement to capture and store metadata for digital records has driven the implementation of centralised electronic document and records management EDRM systems, many of which began life as databases used to record metadata about physical records (files and boxes).

EDRM systems were (and still are) used to store copies of digital records created or captured natively in other systems, primarily network file shares and email. End-users were required to copy individual records to the EDRMS, a process that mirrored the storage of records (including printed digital records) in physical files.

Network file shares and email systems were not considered to be suitable as recordkeeping systems because they could not ensure the authenticity, integrity and reliability of records over time, including to manage and preserve metadata about the records stored in them.

The increasing implementation of Office 365, and in particular the use of SharePoint for the storage of records, has highlighted the extent to which recordkeeping metadata can – or even should – be applied to the content stored in that system.

This post discusses the need for metadata in records stored in Office 365, including in both Exchange/Outlook, MS Teams, and SharePoint/OneDrive for Business. It concludes that most records stored in Office 365 do not need additional metadata but, where such metadata is required, there is unlimited capability to add it.

Records and metadata

The international standard for records management, ISO 15489:2016, defines a record as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’.

Records are said to be different from ‘non-records’ because they are associated or described with (mostly added) metadata that describes ‘the context, content and structure of records and their management through time’.

The standard for recordkeeping metadata is ISO 23081:2017. One records management professional (link at the end of the post) noted that there has been reluctant adoption of this standard, mostly because it was ‘too complex’ and ‘academic’, and used ‘foreign terms’. Unspecified vendors were said to have been dismissive of the standard.

Standard for managing digital records – ISO 16175

Part 2 of the standard ISO 16175:2011, ‘Guidelines and functional requirements for digital records management systems’ contains multiple requirements relating to metadata, across three broad categories:

• Point of capture metadata. This includes metadata that forms part of the ‘metadata payload’ of the original record (e.g., date created, creator), other metadata added at point of capture, and metadata that provides additional context for the records.
• Process metadata. This is metadata that records activities and changes to both the record and metadata over the life of the record.
• The need to manage and control metadata over time.

This standard appears to reinforce the requirement for records to be stored and managed in dedicated recordkeeping systems.

Metadata versus enterprise ‘graphs’

While this post was in draft, James Lappin published a very interesting and informative post titled ‘Project Cortex and the future of document management in Office 365‘. The post highlighted a key difference between on-premise and cloud era document management, in relation to the way metadata is managed:

• On premise document and records management systems: These systems use metadata schema that specify metadata fields to be used in the system.
• Cloud systems including Office 365: These systems can make use of enterprise ‘graphs’ that map people to documents and topics. The graph is built from the interactions of people with content across the different workloads of the suite.

Microsoft’s depiction of a graph model and the dynamic interaction between objects. 

Most people now accept the algorithm capabilities of Facebook, LinkedIn, eBay, Amazon and similar online systems to automatically connect us with information relevant to us, without having to add any metadata.

Given the volume and types of digital content, almost all of which has metadata ‘payloads’, how can we ever hope to add the required recordkeeping metadata?

Can’t we just rely on the algorithms and graphs?  

How much metadata do you really need?

The answer to this question may depend largely on business, regulatory/compliance and/or government recordkeeping requirements relevant to the organisation and its jurisdiction. In my experience, across multiple very large and also very small organisations:

• Most private sector organisations will likely have minimal metadata requirements beyond basic ‘point of capture’ and ‘process’ metadata already recorded in the system where the records are created or captured (including email), unless this is required for specific compliance or regulatory purposes, or where there is risk associated with poor recordkeeping. For example, in a major food processing company, records relating to the manufacture of food were very well documented and managed, while corporate records were managed haphazardly.
• Most public sector organisations are required, for government accountability and transparency (and information retrieval) purposes, to apply a minimum set of both ‘point of capture’ and ‘process’ metadata for non-permanent records. Many government agencies have struggled to manage digital records effectively.
• A small percentage of records captured or created in government agencies may require more extensive metadata, especially if those records are to be transferred to archival institutions for permanent retention.

Office 365 ‘workloads’

In Office 365, most business records will be created or captured in either Exchange/Outlook (includes MS Teams chats), or SharePoint or OneDrive for Business (for ‘working’ or personal content).

Exchange is a recordkeeping system in that it stores records with consistent metadata. The primary ‘weakness’, in terms of recordkeeping, is that ‘personal’ Exchange mailboxes aggregate records on a range of subjects by an individual user rather than by business subject. The mailboxes of Office 365 Groups, on the other hand, can be used to aggregate records about a business function/activity or subject.

SharePoint is a recordkeeping system that has extensive default metadata and almost unlimited additional metadata capability (see below). OneDrive for Business is a SharePoint service that has the same extensive default metadata capability.

There is, generally speaking, no requirement for organisations that have implemented Office 365 to allow the continued use of network file shares because the ‘save’ and ‘save as’ options in Office/Windows 10 points to SharePoint and OneDrive as the default save locations.

Metadata in Exchange mailboxes/MS Teams

Emails have the same metadata options in the header of every email:

• Message ID
• Subject
• Sender (From)
• Recipients (To, including CC and BCC)
• Sent (date/time)
• Received (date/time)
• (Plus more with routing information and security controls including DKIM, SPF, DMARC etc)

However, no other metadata can be added and some (or most) emails may never form part of the collated record of a given subject.

Because of this ‘limitation’, there has been an assumption ever since email was introduced that emails identified as records would have to be copied to a (separate) recordkeeping system.

• In pre-digital days, this meant printing out emails and placing them on a paper file. 
• In organisations with EDRM systems, this meant copying the email to the EDRMs where additional metadata would be applied.

The original emails generally remained in place in individual mailboxes where they may be subject to backups and journaling in case they needed to be recovered for whatever reason including subpoenas (eDiscovery).

The Office Graph in Office 365 now provides the ability to connect the content in email with other content across that ecosystem, as noted in James Lappin’s post above. This is new – but it doesn’t rely on metadata or copying emails anywhere.

Metadata in SharePoint

As a SharePoint service, OneDrive for Business has the same default metadata columns. According it will not be described further here.

What metadata is required?

Organisations that plan to manage records in SharePoint should consider the following questions as part of their overall information architecture design to ensure records are kept in logical aggregations rather than randomly. This is important especially if end-users are allowed to create Office 365 Groups or Teams. 

• What point of capture and process metadata is required (for compliance, regulatory, recordkeeping purposes)? What is the source of this requirement?
• Is there a difference in the metadata requirements for short-term (retain in the organisation) and permanent records that are to be transferred to archival institutions?
• Do the required metadata columns already exist in SharePoint?
• If they don’t exist, should the additional metadata columns be added as site columns or library columns?
• Does any of the metadata need to be mandatory, and/or can it be a default setting – for example, a metadata column that has the default function and/or activity so the user doesn’t need to add this.
• Where is the process metadata and how do you view or manage it? (See also below on this subject).

Information architecture and metadata

The information architecture of SharePoint, in terms of managing records as objects (e.g., documents, spreadsheets, images, etc), is relatively simple:

• SharePoint site. The primary aggregation that can be linked to a business function (e.g., ‘Financial management’).
  • Document library/ies. Logical aggregations or containers of records that can be linked to business activities (e.g., ‘Meetings’).
    • Folders, document sets as content aggregations.
      • Documents/records.

An effective site architecture can replace the requirement for metadata. For example, the name of the SharePoint site can map to a business function, and library names can map to activities, instead of applying a function and activity pair to each record. The URL address for the record provides the context:

https://tenantname.sharepoint.com/teams/finance/invoices2020/record1.ext

If additional metadata is still required, SharePoint has extensive and almost unlimited capability.

• Every new SharePoint site comes with a standard set of around 240 metadata ‘site columns’. The metadata columns include the Dublin Core metadata items.
• New metadata columns can be created at the site level (‘site columns’). These are then can be used by all libraries and lists on the site. Here is a useful description of how to add new site columns from ShareGate: SharePoint 101: SharePoint Site Columns.
• Every new SharePoint library comes with a standard set of metadata columns – see below. New metadata columns can be created at the library (or list) level, but these columns are only available to that specific library or list.

Default SharePoint document library columns

The default library metadata columns are as follows. Dublin Core metadata items are shown with [DC]:

• App Created By
• App Modified By
• Check In Comment
• Checked Out To
• Comment count
• Compliance Asset Id
• Content Type
• Copy Source
• Created [DC]
• Created By [DC]
• Document ID (when enabled as a feature)
• File Size
• Folder Child Count
• ID
• Item Child Count
• Item is a Record
• Label applied by
• Label setting
• Like count
• Modified [DC]
• Modified By Name [DC]
• Retention label
• Retention label Applied
• Sensitivity
• Title [DC]
• Type [DC]
• Version

How metadata is added to records in SharePoint

Every digital record saved to SharePoint will have some form of native metadata (payload). Additional metadata may be added when the document is saved; this may be optional or mandatory.

• When a digital record is saved to SharePoint, SharePoint only copies the title or name of the record, not the original created date or author.
• When a Microsoft Office document is saved to a SharePoint document library, the Office document stores the library metadata (including the unique Document ID) in its own XML-based properties. This information is retained with the record even when the record is downloaded from SharePoint.

Viewing the metadata

The metadata that describes the content stored in the SharePoint document library may be viewed in multiple ways (via the edit view option), and may be exported (for example if records are to be destroyed or transferred).

Every record includes a version history that provides details of who modified the content, and when (but not what changes were made unless this is recorded).

Process metadata

Process metadata is metadata that records events relating to the record or the aggregation in which it is kept.

Examples of process metadata include when:

• Records are viewed or downloaded (date and by whom).
• Records are modified (date and by whom, and ideally what changes were made).
• Records are copied or moved (date and by whom).
• Security controls were changed (date, by whom, and what changes were made).
• Records are deleted/destroyed (date and by whom, with what authority).

While ISO 16175 describes the general requirement to keep process metadata, the actual requirement is likely to differ between organisations. Organisations with high compliance requirements, such as certain types of businesses or government, are more likely to want process metadata to be created, accessed when required, and protected against unauthorised modification.

Office 365 process metadata

Office 365 records process metadata in multiple ways in Exchange and SharePoint.

• Emails generally cannot be modified after they have been sent. Accordingly, the primary process metadata for emails and Teams chat is likely to be in the deletion records stored in the Office 365 Compliance admin portal audit logs.
• SharePoint/OneDrive process metadata is recorded as follows:
  • Viewed or downloaded, modified, copied or moved: This is recorded in the Office 365 Compliance admin portal audit logs.
  • Modified: This is recorded in the Date modified and Modified by metadata, as well as the version history (which also keeps the previous actual versions that can be compared if required).
  • Security changes: This is recorded in the Office 365 Compliance admin portal audit logs.
  • Destroyed: Depends, but generally this requires the capture of information manually, then stored elsewhere. For example, if the content of a document library is to be destroyed, then the metadata (along with details of the original library URL) should be exported (manually) first and saved somewhere. This is a manual process.

Note that audit log data in Office 365 is only retained for 90 days with an E3 licences, 365 days for an E5 licence.

Final thoughts

Exchange/Outlook email has basic metadata. It is unlikely that it will ever be possible to add other metadata, unless email is copied to SharePoint document libraries.

Chats from MS Teams are stored in hidden folders in Exchange mailboxes.

Organisations that need to keep certain emails for specific compliance, recordkeeping or archival purposes, should consider capturing these in SharePoint document libraries. Organisations might also consider making more use of Office 365 Group mailboxes for business-specific content as these Groups also include both MS Teams chat and have an associated SharePoint site.

The metadata capabilities of SharePoint are unlimited but not all records need the same degree of metadata.

The majority of records can probably be managed in standard SharePoint document libraries using the default metadata columns, or with one or two additional site or library metadata columns added, where required.

The Office Graph will increasingly be able to bring together records dynamically in the context of the business or the end-user via Project Cortex and Delve, respecting security controls that may be in place. The centralised content search and retention policy capability in Office 365 will also enable businesses to find, retrieve and manage content across both Exchange and SharePoint.

Reference

AS/NZS ISO 23081 series, Information and documentation – Records management processes – Metadata for records

Destroying digital records – are they really destroyed?

Most people should be aware that pressing the ‘delete’ option for a file stored on a computer doesn’t actually delete the item, it only makes the file ‘invisible’. The actual file is still accessible on the disk and can be retrieved relatively easily or using forensic tools until the space it was stored on is overwritten.

Traditional legacy electronic document and records management (EDRM) systems have two components:

• A database (e.g., SQL, Oracle) where the metadata about the records are stored
• A linked file share where the actual objects are stored, most of which are copies of emails or network file share files that remain in their original location.

In most on-premise systems, email mailboxes, network file shares, and the EDRMS database and linked file share are likely to be backed up.

When a digital record comes to the end of its retention and is subject to a ‘destruction’ process, how do you know if the record has actually been destroyed? And even if it is, how can you be sure that the original isn’t still stored in a mailbox, network file share, or a back up?

This post examines what actually happens when a file is ‘deleted’ from a Windows NT File System (NTFS), and questions whether digital records stored in an EDRMS are really destroyed at the end of the retention period.

The Windows NTFS Master File Table (MFT)

Details of every file stored on a computer drive will be found in the NTFS Master File Table (MFT).

In some ways, the MFT operates like a traditional electronic document management system – it is a kind of database that it records metadata about the attributes of the digital objects stored on the drive. These attributes include the following:

As noted in the diagram above, the details stored by the MFT include the $File_Name and $Data attributes.

• The $File_Name attributes include the actual name of the file as well as when it was created and modified, and its size.  This is the information that can be seen via File Explorer and is often copied to the EDRMS metadata.
• The $Data attribute contains details of where the actual data in the file is stored on the disk (in 0s and 1s) or the complete data if the file is small enough to fit in the MFT record.

If the MFT record has many attributes or the file data is stored in multiple fragments on a disk (for example as a file is being edited), additional MFT ‘extension’ records may be created.

When a file is deleted, the MFT records the deletion.

• If the file is simply deleted, the record will remain on the disk and can be recovered from the Recycle Bin.
• If the file is deleted through SHIFT-DEL or emptying the Recycle Bin, the MFT will be updated to the ‘Deleted’ state and update the cluster bitmap section to set the file’s cluster (where the data is stored) as being free for reuse. The MFT record remains until it is re-used or the data clusters are allocated in whole or part to another file.

So, in summary, ‘deleting’ a file does not actually delete it. It may either:

• Store the file in the Recycle Bin, making it relatively easy to recover, or
• Change the MFT record to show the file as being deleted but leave the file data on the desk until it is overwritten.

How does an EDRMS store and manage files?

The following summary relates to a well-known Electronic Document and Records Management System (EDRMS). Other systems may work differently but the point is that records managers should understand exactly how they work and what happens when electronic files are destroyed at the end of a retention period.

Most EDRM systems are made up of two parts:

• A database (SQL, Oracle etc) to store the metadata about the record.
• An attached file store that stores the actual digital objects.

When EDRM systems are used to register paper or physical records (files and boxes), only the database is used.

When digital records are uploaded to the EDRMS:

• The metadata in the original file, including the file type, original file name, date created, date modified and author are ‘captured’ by the system and recorded in the new database record.
• Additional metadata may be added, including a content or record ‘type’.
• The record will usually be associated with a ‘container’ (e.g., ‘file’). This containment makes the record appear to be ‘contained’ within that container, whereas in fact it is simply a metadata record of an object stored elsewhere.
• The original record filename is changed to random characters (to make it harder to find, in theory) and then stored on the attached (usually Windows NTFS) file store, often in a series of folders.
• A link is made between the database record and the record object stored in the file store (the MFT record).

When the end-user opens the EDRMS, they can search for or navigate to containers/files and see what appears to be the digital objects ‘stored’ in that container/file. In reality, they are seeing a link to the object stored (randomly) in the file store.

What happens when an EDRMS record is destroyed?

If there is no requirement to extend their retention, or keep them on a legal hold, records may be destroyed at the conclusion of a retention period.

For physical records, this usually means destroying the physical objects so they cannot be recovered, a process that may include bulk shredding or pulping.

For digital records, however, there may be less certainty about the outcome of the destruction. While the EDRMS may flag the record as being ‘destroyed’ it is not completely clear if the destruction process has actually destroyed the records and overwritten the digital records in a way that ensures its destruction to the same level as destroyed paper files. 

Also:

• If the original associated NTFS file share becomes full and a new one is used, the original is likely to be made read only.
• There is likely to be a backup of the EDRMS.
• The original records uploaded to the EDRMS probably continue to exist on network files shares, in email, or in back up tapes.
• Digital forensics can be used to recover ‘deleted’ files from the associated file share.

Consider this scenario:

• An email containing evidence of something is saved to a container in an EDRMS.
• The container of records is ‘destroyed’ after the retention period expires.
• A legal case arises after the container is ‘destroyed’
• A subpoena is made for all records, including those specific records.
• Has the record actually been destroyed, or could it still be recoverable, including from backups or the digital originals?

Is it really possible to destroy digital records, and does it matter?

Yes, records can be destroyed by overwriting the cluster where the record is kept, and some EDRM systems may offer this option.

But:

• Do EDRM systems overwrite the cluster when a digital record is destroyed in line with your records retention and disposal authorities, or simply mark the record as being deleted, when it is still technically recoverable?
• Could the record still exist in the network file shares or email, or in backups of these or the EDRMS?
• Might it be possible to recover the record with digital forensics tools?
• Does it matter?

It might be worth asking IT and your EDRMS vendor.

References:

• https://docs.microsoft.com/en-us/windows/win32/fileio/master-file-table
• http://www.kcall.co.uk/ntfs/index.html

 

 

Understanding and applying retention policies to content in MS Teams

This post highlights the need to understand how retention works in MS Teams, why it may be related to how long you keep emails (including for backup purposes), and why you need to consider all the elements that make up an Office 365 Group when considering how – and how long – to retain content in MS Teams.

Overview of retention in MS Teams

If you are unfamiliar with how retention works with MS Teams, these two related sites provide very useful detail.

• https://docs.microsoft.com/en-us/MicrosoftTeams/security-compliance-overview
• https://docs.microsoft.com/en-us/microsoftteams/retention-policies

Image from the first link above – Security Compliance Overview

The quote below from the second link is relevant to this post:

‘Teams chats are stored in a hidden SubstrateHolds folder in the mailbox of each user in the chat, and Teams channel messages are stored in a hidden SubstratesHolds folder in the group mailbox for a team. Teams uses an Azure-powered chat service that also stores this data, and by default this service stores the data forever. With a Teams retention policy, when you delete data, the data is permanently deleted from both the Exchange mailboxes and the underlying chat service.’

and

‘Teams chats and channel messages aren’t affected by retention policies applied to user or group mailboxes in the Exchange email or Office 365 groups locations. Even though Teams chats and channel messages are stored in Exchange, they’re only affected by retention policies applied to the Teams locations.’

In summary:

• One-to-one chat in MS Teams is stored in a hidden folder of the mailbox of each user in the chat. Documents shared in those chats are stored in the OneDrive for Business of the person who shared it.
• Group chat in Team channels is stored in a hidden folder of the mailbox of the associated Office 365 Group – and also in an Azure chat service. Documents are stored in the Office 365 Group’s SharePoint site (other SharePoint site libraries may also be linked in a channel).

Another quote from the same post:

‘In many cases, organizations consider private chat data as more of a liability than channel messages, which are typically more project-related conversations.’

Teams content is kept in mailboxes, retention may be similar

Typically, in the on-premise past, organisations will have backed up their Exchange mailboxes (and possibly also enabled journaling, to capture emails), for disaster recovery, ‘archiving’ and investigations. Unless a decision is made to invest in cloud back-ups, Office 365 retention policies may also be applied to Exchange mailboxes, effectively replacing the need to back them up. Retention policies applied to Exchange mailboxes don’t affect the teams chat folder.

Organisations should probably apply the same retention period to both emails and Teams chats as they do to email mailbox backups now. That is, if mailboxes are typically kept for 7 – 10 years after the person leaves the organisation, then keep the Teams chats for the same period.

Note that, even if a poster deletes an item (if that option is enabled), it will still be retained if there is a retention policy.

Suggestions for retention in MS Teams

As there can be different retention requirements, depending on the subject matter, here are some suggestions for retention:

• One-to-one chat is like email, you will never know everything that is being said or sent. So a single retention policy that mirrors email would be appropriate.
• Teams chat is more likely to be about the subject of the Team, which is based on an Office 365 Group, its own mailbox, and has a SharePoint site. In this case, you could consider a retention policy applied to all Office 365 Groups or specific Groups – for example ‘Project Groups’, then ensure that the retention policy or policies cover all aspects of the Office 365 Group (mailbox, team chat, SharePoint).
• If all the records relating to a particular subject matter (including email, chat and documents) must be retained for 25 years, then you need to understand all the options.

It underscores the need to plan carefully for retention management for all the key workloads in Office 365.

Understanding SharePoint Online in the context of Office 365 admin

On-premise versions of SharePoint were standalone systems, usually administered by a trained and qualified SharePoint Administrator. Records managers may and may not have had access to or a role in that environment.

Generally, the only other group that would typically have access to SharePoint on-premise were the DBAs who managed the (SQL) database.

SharePoint Online is no longer a standalone system but a core part of the Office 365 ecosystem.

This post describes, for records and information managers, how SharePoint Online needs to be understood in the context of the broader Office 365 administration, and how other admin roles can configure or change settings that can affect SharePoint and OneDrive for Business. 

See also my earlier post from 9 January 2020 titled ‘How many Office 365 Admins do you really need?‘

Office 365 Roles

Global admin

The highest level admin role in Office 365 is the Global Admin (GAs).

To protect the security of Office 365, there should be a very small number of GAs. GAs should have unique cloud-only log ons preferably using multi-factor authentication for added security. End-user accounts should NEVER be assigned the GA role.

GAs can access everything across Office 365, including the content of emails, SharePoint, OneDrive for Business and MS Teams. All activity carried out by GAs (and anyone else) is recorded in the audit logs.

Organisations that outsource the GA role to third-party companies need to be aware of the capability of the GA role and, ideally, also have at least one GA log-on account so they can, among other things, access the tenant and review the audit logs if required.

The key activities that GAs are responsible for, that impact on the management of SharePoint, are as follows.

• Assigning licences. Licences (e.g., E3) provide user access to the various applications in Office 365, including Exchange, SharePoint, OneDrive for Business, MS Teams and Office (via http://www.office.com). Generally speaking it is inadvisable to remove individual options from licences. Note that the SharePoint licence gives access to use the application, it is not the admin role (next point).
• Assigning roles. Roles provide admin access to the core applications (listed in the previous point) and to a range of activities (for example, Billing, Compliance, Security, User Admin). Office 365 Admin roles should always be cloud only and never assigned to normal end-user accounts. This ensure that the person logs on to perform an admin activity, as opposed to a general end-user activity. It is common (and good) practice for users may be logged on to two ;separate accounts at the same time.
• Creating Groups. Groups are Azure/Exchange objects. The three main types of groups are: (a) Security Groups that control access to resources but are not email enabled; (b) Distribution Lists that provide the ability to email multiple people but don’t control access to resources; and (c) Office 365 Groups that a cross between Security Groups and Distribution Lists with much more capability. Office 365 Groups are a core element across Office 365. Every O365 Group has (a) an email mailbox, (b) and a SharePoint site. If the ability to create these types of groups is not controlled, every new Team in MS Teams will create an O365 Group with a SharePoint site (with no controls on naming). Accordingly, there needs to be close cooperation between the GA, the SharePoint admin and/or the records/information manager in relation to the creation of O365 Groups.
• Enabling external access for SharePoint. This setting allows the GA to determine whether SharePoint sites and OneDrive for Business, and the content in them, can be shared externally. The setting only makes the option available for SharePoint sites but allows ODfB content to be shared externally. Individual sites must still be enabled (by the SharePoint admin) for external access.

SharePoint/OneDrive for Business Admin

The SharePoint Admin will normally be a qualified SharePoint administrator and may have administered earlier versions of SharePoint. They will also generally be the OneDrive for Business admin (as OneDrive is a SharePoint service).

The SharePoint Online admin role is much less complex in Office 365 than it was in the on-premise version. Records managers who currently manage an EDRMS could potentially become a SharePoint admin, with some training.

Additional training is required only if the organisation wishes to do additional customisation or development work, integration, or has third-party applications.

The SharePoint Admin has a number of roles:

• Configuring SharePoint settings in the admin portal. This is usually a one-off activity that may be reviewed from time to time. Configuration settings should be documented.
• Creating new SharePoint standard and communication sites – but NOT ‘modern’ team sites that are based on Office 365 Groups, as noted above. These should be created by the GAs who will need to be advised about (a) preferring naming conventions (if any) and (b) Group ownership and membership (which flows through to SharePoint site ownership and membership).
• Provisioning new sites. This activity involves changing site collection features and site features to enable things like Document IDs and Document Sets. It also includes assigning the initial Site Collection Admin and Site Owner permissions. It may also include some basic additional options such as a new document library or list.
• Assigning access and permissions. Records managers who have responsibility for managing records in SharePoint should be added to the Site Collection Admin section, ideally as part of a Security Group. This ensures that records managers can access all SharePoint sites as required (including the Preservation Hold library on sites where implicit retention policies have been applied) and, if they have the responsibility to do so, create and configure new document libraries to manage records. Both Site Collection Admins and Site Owners can apply explicit (visible) retention policies to document libraries and lists, if used.
• Monitoring and managing the SharePoint environment, including resolving issues and working with Site Owners.
• Managing the OneDrive for Business admin portal, including setting (a) the size of the ODfB storage and (b) the retention period for ODfB accounts after an end-user leaves.
• Providing training to Site Owners, if no other training is provided.

The relationship between the various Office 365 admin elements, SharePoint admin, and the end-user experience is described in the graphic below.

• SharePoint admins access the SharePoint admin portal by logging on to http://www.office.com, clicking on the ‘Admin’ option, the then SharePoint admin portal (or directly to that admin portal if they save it as a favorite).
• End-users access SharePoint by logging on to http://www.office.com and clicking on the ‘SharePoint’ app, or via the mobile app.

Exchange Online Admins

The primary role of the Exchange Online (EXO) admin is to manage that application. The EXO admin may also be the MS Teams admin – see below.

If the creation of Office 365 Groups is not controlled as noted above, both EXO admins and end users can create a new Office 365 Group from Exchange or Outlook which in turn creates a new SharePoint site.

The Outlook menu with the New Group option

 

While emails can be copied from Exchange to SharePoint, Microsoft’s model assumes that the vast majority of emails will remain in end-user mailboxes.

Records managers need to work closely with the EXO admin/s and the Compliance admin/s (see below) to ensure that an appropriate Office 365 retention policy is applied to the content of the mailboxes. There may also be a requirement to remove the default MRM policies.

An Office 365 retention policy may initially appear to conflict with, but can support and replace previous backup strategies deployed to recover mailboxes in case of disaster or for investigation purposes. This means that a single retention policy that keeps all emails for a specific period of time will be applied to all mailboxes.

MS Teams Admins

The role of the MS Teams admin is to configure and manage the MS Teams environment. As noted above, the EXO admin may also be assigned the role of MS Teams admin.

MS Teams includes two main component parts:

• Chat. One-to-one chats are stored in a hidden folder in the Exchange mailboxes of individual users. Channel chats are stored in a hidden folder in the mailbox of the linked Office 365 Group. These hidden folders are not subject to a retention policy applied to the rest of the mailbox.
• Documents. These are stored in either (a) OneDrive for Business for one-to-one chat, or (b) in the linked SharePoint site for Teams channels.

Records managers should work with the MS Teams admin and the Compliance Admin to identify how retention policies will be applied to both the chat and SharePoint content in MS Teams.

See this article titled ‘Retention policies in Microsoft Teams‘ for more information.

 

 

Compliance Admin

Microsoft separated the Security and Compliance portals in early 2020. Consequently, there may be an admin to manage each component part – one for Security and one for Compliance.

The Compliance admin portal includes a range of actions relating to the management of information. These actions include:

• Data classification. This option is still in preview but for E5 licence holders, will allow data to be classified automatically and retention policies applied to that content as an alternative to pre-defined (SharePoint site/library) ‘classification’.
• Setting and monitoring alerts.
• Viewing reports on various compliance matters, including the status of retention policies.
• Creating and monitoring retention labels and policies. This includes retention policies for Exchange mailboxes, SharePoint Online, OneDrive for Business, and MS Teams.
• Creating and monitoring data loss prevention policies.
• Assigning permissions to individuals.
• Managing GDPR data subject requests.
• Searching audit logs (90 days of history only).
• Searching for content across all of Office 365.
• Reviewing disposition for records covered by explicit retention label policies, where this option is enabled

Some or all of these roles may be performed by senior records or information managers.

The Security admin portal provides access to the following actions, some of which may impact on SharePoint (sensitivity labels in particular).

• Reviewing alerts
• Reviewing security related reports
• Creating and managing sensitivity labels and information types (and also creating and publishing retention labels)
• Creating a range of security-related policies including for devices, threat protection
• Assigning permissions

Conclusions

SharePoint on-premise was a standalone system that generally did not interact or integrate much with other systems.

SharePoint Online is a core part of the broader Office 365 ecosystem. A range of roles and configuration settings set across that ecosystem have – or can have – a direct impact on SharePoint Online.

Records managers who are involved with SharePoint Online need to understand this crucial difference and either learn or seek to be assigned key roles that impact on the management of records across the Office 365 ecosystem, not just in SharePoint.

Shifting the paradigm for managing records – from EDRMS to Office 365

Computer systems used to to manage electronic documents and records, commonly known as ‘EDRMS’, have been around for at least 20 years.

Many (but not all) of these systems developed from electronic databases originally used to register and manage only paper records, replacing the old paper registers (hence ‘Registries’).

How does an EDRMS work

A common theme with most EDRM systems is that they describe (via metadata) and provide some kind of visual ‘file’ or ‘folder’ structure for digital objects, almost always stored in a linked network file store.

To store records in this way, EDRM systems required end-users to upload a copy of a digital object (document, email, photograph) to a pre-defined digital container, corresponding to a ‘file’ or ‘folder’. The digital file might have be assigned a range of metadata including the classification (business function and activity) or file plan details, title, business owner or area, and retention information.

Once an object was uploaded, end users were required to add metadata about the object, including the object ‘title’ (if it didn’t copy the original title). Additional metadata fields, for example ‘Document Type’, might also be required.

The system recorded the date and time the object was uploaded and who uploaded it. As noted, the system might copy some of the uploaded object’s metadata, for example the default title, date created and author.

The uploaded document then ‘became’ a record, visible ‘within’ a digital container (‘file’) along with other records.

EDRM systems had (at least) three weaknesses:

• End-users were required to upload the records to the EDRMS, and to one correct container (file/folder)
• The EDRMS contained a copy of a digital object that almost always remained in the original storage location (email, network file share)
• The EDRMS tended to be based on records as documents (including emails, and sometimes photos). Newly evolving forms of record such as text messages, social media posts and new digital forms were difficult to upload without costly add-ons that didn’t necessarily capture everything

These weaknesses meant that:

• End users avoided uploading records because it was extra work (uploading and then adding metadata)
• The EDRMS contained only a percentage of all potential records stored in any location
• The original copies of records, remained in email and network file shares

There were exceptions to this situation, but most (and very much in the minority in terms of total volume) involved the requirement to meet compliance obligations to capture certain types of records.

The Office 365 model

Microsoft took a different approach with the approach to records management in Office 365.

Instead of centralising the storage of records in one system or location (with the weaknesses described above), records in the Office 365 environment generally remain in their original location (Exchange Online, SharePoint Online, OneDrive for Business, MS Teams), where they are covered by an overarching records management framework.

The Office 365 model for records management

What this means is that records can be stored in any of the above locations and managed in those locations through (among other things):

• User types, licences and roles set in the Office 365 admin portal
• Retention and other controls set in the Office 365 Security and Compliance admin portal/s (the two were split in early January 2020).

How the paradigm shifted

The paradigm has shifted from (a) an attempt to manage records in a single system where not everything is captured and originals remain in place in email and network file shares, to (b) the distributed management of records where originals remain in place (assuming SharePoint and OneDrive are used instead of network file shares and personal drives, and email remain in Exchange) and records are managed through ‘global’ settings.

The new paradigm does not exclude the ability to store (or aim to store) digital records in a single location – SharePoint Online (including for specific compliance reasons), but it provides the opportunity manage records wherever they are and use a range of additional tools to manage content from creation through to disposal.

Why the new paradigm matters

The new paradigm is likely to be counter-intuitive to many records (and other information) managers. Records management training for many years has been focused on the idea of storing and managing records in a central location with specific controls (classification, metadata and retention).

But the reality is that there are now too many digital records, and too many types of digital records, to ever expect these to be all stored in an EDRMS. And, even if only some are, what about all the others? Has a legal subpoena ever been focused only on records stored in the EDRMS?

Plan to manage records

Many organisations have acquired and are implementing Office 365, sometimes at the expense of the traditional EDRMS. It doesn’t take long for end-users to adopt the new technology because it is so easy to use.

Any suggestion that specific records now need to be copied to the EDRMS seems to be counter-intuitive. And yet, that is how some records managers continue to see Office 365 – as yet another source of records to be uploaded to the EDRMS. It is not a viable plan.

Records managers need to be at the forefront of planning for Office 365, in particular managing content across the four primary workloads. Records managers should be able to provide advice on:

• The architecture of SharePoint Online
• Controls around the creation of sites, including naming conventions and the ongoing management of sites
• The structure of SharePoint Online sites, document libraries and metadata in particular
• The retention model for Exchange Online, SharePoint Online, OneDrive for Business, MS Teams. This includes understanding existing disaster recovery arrangements and potentially replacing them with retention policies.
• Disposal actions
• Other compliance obligations

Plan for change

Moving away from the centralised management of records in an EDRMS to a less visible (for end-users) decentralised model, or even implementing Office 365 without any other previous document and records management system, requires careful change management.

End users (and records managers) used to the idea of uploading records to a central EDRMS may find the new ‘invisible’ and decentralised model of recordkeeping unusually simple (to the point of disbelief).

Consequently, additional re-assurance, training and awareness sessions, may be required to demonstrate and confirm how the records are managed in the new environment. There is potential for some ‘push back’ as, although it requires very little end-user effort, it manages more records than ever before, including in ‘personal’ spaces such as mailboxes and personal drives.

IT will also need to be involved as disaster recovery processes, such as backing up email and network file shares, may no longer be required.

For end users who have never had to use an EDRMS, change management activities might focus more on improving awareness and knowledge about how records will be managed in the future, including in ‘personal’ spaces.