Managing the retention of records in Microsoft 365 with an E3 licence

Microsoft released its ‘Records Management’ solution for Microsoft 365 during 2020. The solution is only accessible to organisations with an E5 licence (or an E5 Security and Compliance licence).

Some of the retention-related options previously available to E3 licences, such as disposition review, are now only available with an E5 licence. However, for cost and other reasons, many organisations have decided to stay with E3 and asked if it is still possible to manage the retention of records.

According its licencing guidance ‘Microsoft 365 licence guidance for security and compliance‘, E3/A3 (and E5/A5) licences provide the following ‘information governance’ capability:

• A basic organization-wide or location-wide Exchange mailbox retention policy and/or to manually apply a non-record retention labeling to mailbox data.
• A basic SharePoint or OneDrive retention policy and/or to manually apply a non-record retention label to files in SharePoint or OneDrive.
• A Teams retention policy.

This post describes how the retention of records could be managed with an E3 licence and recommends that organisations intending to deploy Microsoft’s retention policies (whether E3 or E5) develop a plan for their deployment based on a detailed understanding of the following:

• What records exist and where they are stored across the main Microsoft 365 workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.
• What retention is required, including for EXO mailboxes and personal/ODfB accounts for which retention was previously ‘managed’ through backups, and MS Teams chats.
• What type of retention policy will apply to what workloads.
• How will disposition be managed (including of original storage locations, not just the records), and what evidence of destruction is required?
• How will any limitations and shortcomings be addressed to minimise legal risk?

Where are the records?

A good starting point with retention planning is to establish where the records that will be subject to retention policies are stored.

Most records are created and stored in one of the primary four workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.

While SharePoint sites (including Teams-based sites and Teams channels) may be used to logically ‘group’ records (e.g., in a team site, a document library, or a channel), this is not the case for EXO mailboxes, ODfB accounts, MS Teams 1:1 or private channel chats (a compliance copy of these chats is stored in the personal EXO mailboxes of participants in the chat).

EXO mailboxes and ODfB accounts usually contain a range of records on different subjects, with different retention requirements. Personal chat messages could be about any subject.

For most of the past three decades, the requirement to keep specific emails or other records meant that end-users had to copy those records to another system, including an EDRMS or even SharePoint, or print and place them on a file. The ability to find old records depended more often than note on the ability to recover them from back-up tapes, a process ironically often referred to as ‘archiving’.

Action: Organisations should establish a list of where records are stored in Microsoft 365 and how retention (including via backups) is currently managed.

What retention is required?

The next step in the process is to establish how long these records need to be kept.

Most of the time this will be based on an organisational records retention policy or schedule. These policies or schedules describe groups of records and how long they must be retained – for example, financial records generally must be retained for seven years. Specific types of financial records may require shorter or longer retention.

This model generally works well when records have been stored in logical groups (e.g., a whole SPO site/Team, or document library), but are more difficult to apply for individual records stored with other records in personal EXO mailboxes, ODfB accounts, SPO sites, or MS Teams chats. If records with longer retention requirements in these locations are not stored elsewhere, or cannot be specifically identified, all the records may have to be kept for the longest retention period.

The start of retention is usually based on a trigger action. Typically these are based on (a) date created, (b) date modified, or (c) date of last action. However, they may also be based on less specific events, for example: (a) 7 years after a contract has expired, (b) 25 years after an employee leaves, (c) when a child turns 21. These less specific events need special attention.

Action: Organisations should ‘map’ retention requirements to the locations where the records are stored. This process will likely involve a discussion regarding the replacement (or supplementation) of email backups with retention policies, and may end up looking something like the following:

Workload	Retention
EXO mailboxes of senior managers	25 years
EXO mailboxes of all other employees	7 years
MS Teams 1:1 chats of senior managers	25 years
MS Teams 1:1 chats of all other employees	7 years
MS Teams private channel chats	As per retention policies
ODfB accounts of senior managers	10 years
ODfB accounts of all other employees	7 years
SPO sites that are not subject to more specific policies	7 years
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies
SPO sites with specific retention requirements, per site or library	As per retention policies

What are the retention options with an E3 licence?

The three main options available with an E3 licence – labels, label policies, and retention policies – are all set from the Compliance admin portal under the ‘Information Governance’ section.

The table below summarises the options:

Type of policy	Can be used for
‘Implicit’, ‘safety net’ retention policies. These policies: (a) work in the back end and cannot be changed by an end-user; (b) create a preservation hold library in SPO sites and ODfB accounts, and hold deleted emails in a hidden EXO mailbox folder; (c) provide an alternative to back-ups, although it should be kept in mind that all content that is retained in this way contributes to the overall storage quota; (d) do not retain record of what was destroyed.	EXO, SPO, MS Teams, ODfB
‘Explicit’ label-based retention policies. Labels must be published to the required workloads before they become visible. Records cannot be deleted once a label has been applied, but end-users can change or remove the label and then delete the record. Label-based policies do not retain a record of what was destroyed.	EXO, SPO, ODfB
‘Explicit’ label-based retention policies that are auto-applied based on three limited options – see below. Records cannot be deleted once a label has been applied. These policies do not retain a record of what was destroyed.	EXO, SPO, ODfB

The three auto-apply options are as follows:

• Apply label to content that contains sensitive information. Unlikely to be used as retention is never based on sensitivity.
• Apply label to content that contains specific words or phrases, or properties. Possibly useful. Recommend that organisations do a Content Search first to see what records may exist.
• Apply label to content that matches a trainable classifier. E3 licences provide six out of the box, limited classification options including ‘profanity’. These are unlikely to be of any value.

Mapping the E3 options

Given the options available, the following is a suggested mapping of records, retention and policy options:

Workload	Retention	Retention policy
EXO mailboxes of senior managers	25 years	Retention policy, plus possibly labels auto-applied to specific records
EXO mailboxes of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
MS Teams 1:1 chats of senior managers	25 years	Retention policy
MS Teams 1:1 chats of all other employees	7 years	Retention policy
MS Teams private channel chats	As per retention policies	Retention policy
ODfB accounts of senior managers	10 years	Retention policy, plus possibly labels auto-applied to specific records
ODfB accounts of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
SPO sites that are not subject to more specific policies	7 years	Retention policy
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies	Retention policy
SPO sites with specific retention requirements, per site or library	As per retention policies	Retention policy (safety net) plus label-based retention policy for specific records

Policy limits

Organisations planning to deploy retention policies should be aware of the limits on custom policies, as described on this Microsoft page, ‘Create and configure retention policies‘. There are no limits on policies that apply to an entire workload (e.g., all EXO mailboxes).

• 1,000 mailboxes
• 1,000 Microsoft 365 groups
• 1,000 users for Teams private chats
• 100 sites (OneDrive or SharePoint)

The page above notes also that ‘There is also a maximum number of policies that are supported for a tenant: 10,000. However, for Exchange Online, the maximum number is 1,800. The maximum number includes retention policies, retention label policies, and auto-apply retention policies.’

How will disposition be managed

Microsoft 365 retention policies retain records for a specified period and usually then delete the records automatically. No record is retained of what was deleted. Even with an E5 licence, only limited metadata is retained and only on those records subject to disposition review.

Organisations deploying retention policies with an E3 licence need to understand the potential risks associated with being unable to provide evidence of what was destroyed.

There are at least two ways to approach this point, but in all cases the approach and options must be legally defensible:

• Ensure that organisational policies clearly indicate what records will be destroyed without any record being kept. For example, with certain exceptions, most emails, Teams chats, SPO sites and the content of ODfB accounts will only be kept for 7 years and then destroyed.
• Establish a process to ensure that a record is kept of specific records that have been destroyed. For example, the metadata of records, due for disposal and stored in document libraries in specific SPO sites, will be captured (and stored separately) before the records are destroyed, and then the document library will be deleted. This is a labour-intensive process and may only be used on some sites.

Addressing E3 licence limitations and shortcomings

The limitations of E3 licence retention policies (and also E5 licences from an evidentiary point of view) should not put organisations off using the out of the box options or cause them to acquire third-party products.

The retention options available in Microsoft 365 now provide functionality that assists recordkeeping compliance, for example the ability to apply ‘back end, safety net’ retention policies to EXO mailboxes, MS Teams chats, ODfB accounts and whole SPO sites. Coupled with the ability to locate all records, including ‘deleted’ records via Content Search, this should be a boon to records managers and also to IT (saving the latter from having to recover records from backup tapes).

On the other hand, deploying the options can be complex and requires good planning and governance documentation. The lack of good governance can lead to the inadvertent loss of records as KPMG found in August 2020 (‘IT blunder permanently erases 145,000 users’ personal chats in KPMG’s MS Teams deployment‘).

Once retention policies have been deployed, the next most complex task may be the disposition approval and destruction process for those records for which a formal disposition review and approval process is required, including the requirement to establish a list of records to be destroyed.

As noted above, even the E5 licence options for disposition review and ‘proof of disposition’ are inadequate in terms of the metadata that is presented and retained. Until Microsoft provides the ability to record the metadata of every record that is due for disposal and/or destroyed, based on ANY type retention policy this process will continue to be a manual task for records managers.

Organisations may consider acquiring a limited number of E5 Security and Compliance licences to gain access to the E5 ‘Records Management’ capability to gain access to other options including File Plan, auto-classification options, and Disposition Review. However, aside from these options (and the ability to auto-apply labels more broadly) these options may not add much more capability to the existing options already available with an E3 licence.

Summarising the options

Based on the above details, organisations with E3 licences might do the following:

• Create several ‘safety net’ retention policies for EXO mailboxes, ODfB accounts, MS Teams chats and general SPO sites. EXO/ODfB retention policies may be (a) based on current back-up periods and (b) be split into ‘senior’ and all other employees. They may also create policies for Microsoft 365 Groups which will cover the Group’s EXO mailbox and SPO site. Some of these policies may map to existing retention classes in the records retention schedules, but others such as the EXO, ODfB and MS Teams policies may need to be added.
• Create label-based retention policies for content stored in specific SPO sites where more granular retention policies are required per library. However, keep in mind these can be changed or removed by end-users with add/edit access. Generally it is not recommended to apply label-based policies to EXO or ODfB unless end-users will apply these accurately and consistently.
• Auto-apply some labels to specific content stored in SPO, EXO and ODfB (having identified that these records exist first via a Content Search). These labels should have a retention period that is longer than the safety net policy or any other label-based policy.
• Establish a ‘manual’ disposition review process for records that are subject to label-based retention policies.

References:

Microsoft 365 licensing guidance for security and compliance

Auto-populating Microsoft Word templates – a no-workflow option

In August 2019 I published a post titled ‘Auto-populating document templates via a form in SharePoint‘. The suggested model was based on using a SharePoint 2010 workflow in SharePoint Designer.

In July 2020, Microsoft announced that it would turn off SharePoint 2010 workflows for newly created tenants from 1 August 2020, and would remove the ability to run or create these workflows from existing tenants from 1 November 2020. It recommended that organisations use Power Automate instead. (Source: Microsoft Ending Workflows for SharePoint 2010 Online Next Month).

This post details an alternative option, with no workflows required – but with the same set up – that may be sufficient in most cases. At this sage (September 2020), there does not appear to be an out of the box option with Power Automate that will create a document (from a content type) when a new document set is created.

Overview

The alternative option requires the following elements:

• The document set feature enabled.
• New site columns for each unique metadata required in the output documents.
• New document set and document content types, both with the same metadata.
• One or more Word document templates.
• A SPO site with a document library. It is recommended that a dedicated library (e.g., ‘Client Agreements’) is created for this purpose.

Note – the person who does this must be a Site Collection Administrator.

The main difference with the earlier model is that there is no workflow to create the document/s. Instead, the end user creates and saves (without modification except a name) the document into the library/document set where the custom metadata is stored. The method below then auto-populates those new documents with the required metadata.

This option does not use a separate list as the metadata details can be extracted from the document sets.

Enabling the document set feature

The document set feature must be enabled via Site Settings > Site Collection Administration > Site collection features. If you don’t see this option, you don’t have the required level of permission.

It may be a good idea to activate the Document ID Service at the same time as this can be a useful additional piece of metadata in the final documents.

New site columns

From the same Site Settings area, go to the Web Designer Galleries section to add the required new site columns. Directly below that option is where you create new site content types.

The Site Columns section has a very large number of default columns that come with every SPO site. Check to make sure the required columns don’t already exist, or are non-recommended column names (like ‘Library’).

Assuming the required columns do not exist, select the ‘Create’ option to create a new custom column. A couple of points to keep in mind here:

• Give the column a name that is obvious (easy to find).
• Don’t put spaces between words but capitalise each word (e.g., ‘ClientName’, ‘ClientAddress’). From experience, this space can cause the metadata element to wrap to a second line and cause problems.
• Keep in mind that the ‘Single line of text’ is free text. If you want controlled choices, use the Choice option instead.
• Date and Time may require some tweaking to get the correct format in the output Word document. It may be easier to use ‘Single line of text’ if some dates may vary or aren’t exact.
• Don’t use the ‘Yes/No’ option. Instead, use a Choice column with those options instead.
• Person or Group is only for internal names recorded in Active Directory (AD). If you need to insert any other name, use the single line of text.

The additional column settings that may be useful are:

• Require that this column contain information. If changed to ‘Yes’, this metadata field will be mandatory.
• Enforce unique values. Generally not used unless there is a requirement to enforce unique values (e.g., unique client IDs).
• Maximum number of characters. This can be useful to restrict the number of characters. For example, ‘Postcode’ will only have four characters. Don’t use the number field for postcodes.
• Default text. This option is useful if there is a requirement to have a default value, especially from a choice list, or if the value will usually always be the same (although that could also be simply added to the text of the document template.
• Column formatting. Don’t use this if the content is to be copied to a document template.
• Column validation. This may be useful to ensure that only certain characters are entered. For example, postcodes will never have a letter, only numbers.

Repeat this process until all the required metadata columns exist.

New document set content type

Create a document set content type so the output Word documents (and any other digital content) can be stored in the same ‘folder’.

To create a new document set content type, return to the Site Settings and, under Web Designer Galleries, select ‘Site content types’.

Give the new content type a name that is obvious for its purpose (e.g., ‘ClientAgreementsFolder’).

Importantly, change ‘Select parent content type from’ > Document Set Content Types, and ‘Parent Content Type’ > Document Set.

Open the content type that has been created. Under the ‘Columns’ section, add the columns ‘from existing site columns’. The example below shows a series of site columns that were added. Don’t remove any of the other columns.

In the above example. the ‘Title’ column is hidden (default setting). If you click this column, you will see the option under the ‘Change Content Type Column’ to make it required, optional or hidden. Generally, these settings should not be changed.

New Document Content Type

Now, create a new document content type. Give the content type a name that reflects its purpose. Ensure that ‘Select parent content type from’ is set to ‘Document Content Types’ and ‘Parent Content Type’ is set to ‘Document’ as shown below.

The next step is to add the two content types to a document library.

Adding the content types to a document library

All the steps below can be carried out by a Site Owner.

Open the library where the content types are to be added and go to Library Settings (via the Cog/Gear option). In the Library Settings, under ‘Advanced Settings’, click the option to ‘Allow management of content types’ (change from No to Yes).

If you don’t want other folders to be created in this library, change ‘Make ‘New Folder’ command available’ from Yes to No.

A new section will now appear in the Library Settings – ‘Content Types’.

To add the two content types that were created, click on ‘Add from existing site content types’, select each content type from the list that is displayed, and click OK.

The two content types should now appear in the list of Content Types and the custom site columns should appear in the ‘Columns’ list.

Click on the document set content type (e.g., ‘ClientAgreementsFolder’). The custom columns should be visible. Now, we need to share these columns with the custom document. Click on ‘Document Set Settings’.

• From the list of ‘Available Site Content Types’, chose the custom document content type that was created above. Ignore the ‘Document’ content type.
• In the ‘Shared Columns’ section, check the box against all the custom columns.
• Click OK.

The steps above will mean that all the custom columns in the document set will be shared with the the document content type. When you return to the Library Settings, you will see under the ‘Columns’ section that the columns are now shared with the two custom content types as well as the default Document content type (which can be ignored).

Add the Word template to the document content type

Now, add the Word template to the document content type. Note that this step is NOT done at the Site Content Types level, but at the local library level.

Ensure the Word document template is in the latest version of Word (e.g., docx) and has all the text required. It can be updated at any time (see below) but it is important to ensure that the document is as complete as possible, and the sections where the metadata is to be placed is obvious. One way to do this is to put text such as CLIENT NAME in the body of the document. Avoid using highlights as this can be troublesome (but not impossible) to remove.

From the Library Settings, Content Types section, click on the custom document content type (e.g., ‘ClientAgeements’). Click on Advanced settings.

Click ‘upload a new document template’ and add the document template. The name of the template should be obvious. It is possible to go back and edit the template at any time after it has been uploaded. This is usually much easier if a change is required, compared with uploading a new template and having to re-link all the metadata.

Connect the metadata to the document content

From the Document Template section above, click on ‘Edit Template’. This should open the document in the installed Word application. These steps cannot be carried out using Word Online.

Place the cursor in the body of the text where the metadata is to be placed. Click on the ‘Insert’ tab in Word, then click on ‘Quick Parts’ in the text section.

Click on ‘Document Property’ and the custom columns will appear.

When you select each property, the word document will display it as shown below. Continue this process until all the required metadata is added to the document:

As noted earlier, it is usually much easier to edit an existing template ‘in place’ instead of uploading a new version because the steps involved in adding the metadata to the template must be repeated if a new template is uploaded.

Auto-populating Word documents

Now that all the elements are in place, metadata is added automatically to the Word template as follows.

First, open the document library. Click on ‘+ New’ and choose the document set content type (e.g., Client Folder).

Fill in the metadata elements required, then click ‘Save’.

Now, open the document set/folder, and click on the document content type (e.g., ‘ClientAgreement’). This will open the Word template.

Don’t change the document, just click File – Save As. This should display the list of SharePoint sites on the right. Give the document a name, then choose the relevant site (e.g., ‘Client Agreements’), then the relevant library and document set/folder and press save. This is the most complicated part of the process once it is set up.

The template document, with all the metadata elements, should now appear in the document library under the document set/folder. As it is a Word document, it is possible to ask someone to sign on glass, then ‘save as’ a PDF.

If the document requires a sensitivity label and these are enabled, this can be added directly from the Word document menu when it is created.

Other relevant documents (e.g., emails) could be added to the same folder, including drag and drop from Outlook or other locations when the library is synced to File Explorer.

Note that the metadata elements that were added to the document in this way form part of properties of the document. These properties, which will remain with the document if it downloaded or attached to an email, can be viewed from the ‘File – Info’ section of the Word document.

Summary

The disabling of SharePoint 2010 workflows in SharePoint Designer has removed the ability to auto-create a document in SharePoint with the required metadata. However, it is not particularly difficult to set up a site, with two custom content types, one of which contains a document template, to achieve the same outcome without any workflow at all. The only complicating factor is the requirement to save the document template back to the library – without making any changes at all.

Data-based records

(Note – the image above is a small ticket dated 1956 from my grandmother’s visit to Denmark. I used this because the word ‘Kontrolbillet’ seemed appropriate for this post.)

In response to several queries about this following my previous post about whether it is possible to manage records as data, it seemed apparent that the data-based nature of contemporary modern digital content formats, especially Office documents, is not well known.

This post provides details of the data structure content of a typical Word document, to help explain why such records could be seen (and managed) as self-contained data sets.

Just to be clear, the idea of managing records as data does not remove the need or business requirement to store and manage records in ‘local’ aggregations or context – a SharePoint document library or a mailbox for example (less so a network file share because of the limited metadata, but still possible). These aggregations will generally map to business activities, can have specific metadata requirements and can be used to control access to and retention of records as long as they need to be managed.

Managing records as data is a more holistic data analytics concept that allows organisations to better understand and analyse records amidst the volume of all other digital content. It should should help to ensure that all records on a given subject or context are managed appropriately through time, and that, wherever possible, only one copy exists.

A document in a SharePoint Online library

For this example, a document is stored in a SharePoint Online document library called ‘Client Agreements’. The library has a set of metadata columns that must be added to every record. The library uses document sets but it could equally use metadata or folders, the important point is that metadata is added to the library.

The metadata added to the library can be anything, including terms from a business classification scheme. The metadata can be mandatory or optional, and can be set as default options – for example, you may want every document in a library to automatically have the same function and activity terms.

In the screenshot below, we can see the document library with two document sets (a type of folder). The library has four added metadata options: Client Name, Client Reference, ClientRef, and Date of Birth (not visible in the screenshot but we’ll see it later).

The metadata properties

Here are the metadata columns for the library. As we will see below in the actual data, metadata columns with a space between words results in additional characters (‘_0020_’) replacing the space.

The columns from ‘Created’ downwards are all default columns

When I open the Harpin ‘folder’, I can see the metadata columns next to a document. In this case they were added to the document automatically as the documents inherit the same metadata properties as the document set. This is set via the Document Set settings – ‘Shared Columns’:

Alternatively, the metadata can be added to each new individual document when the document is added.

If the Harpin document is selected as shown below …

… the information panel on the far right shows the metadata properties for the document (and also the activity – when the document was modified and by whom, and who viewed it):

As this particular document is a Word template added to a content type in the library, an end user can to select it when they create a new document in the library as shown in the screenshot below. Alternatively, the ‘Client Folder’ option allows them to create a new document set folder with all the metadata that relates to the client; this data is then inherited by any document created in the library:

If the document is opened, you can click on File – Info and see the metadata properties already added TO the document in the library. These properties remain with the document even if it is downloaded and/or attached to an email. If Document IDs have been enabled, that metadata value is also added to the document properties, meaning we can see that it came from a SharePoint library (and which one):

Because it is used as a template, the Word document can make use of the metadata added to the record in the body of the document, in addition to the metadata forming part of the properties for the document.

The metadata properties have been automatically added to the body of the document

The XML properties

Let’s now look at the XML of the document.

Download the document to an accessible location. Using the Command Prompt (CMD), rename the document to .zip (You cannot do this from File Explorer). From File Explorer, the original file will now have the extension .zip. In the list below, the other file with a similar name is a copy, but the size is identical.

Now, unzip the zip file (right click, Extract All).

Here is the top level output, which is standard for all Word documents.

Open the ‘customXml’ folder and you will see a set of XML files:

Open item1.xml, and you will see the custom properties which, as you can see, includes both the Document ID as well as the original path SharePoint site/library location. Just to be clear the Document ID ends in ‘119’, which is the actual document; the original document set folder’s ID ends in 118 (scroll up to check):

To avoid ‘_x0020_’ in the metadata properties, don’t use spaces in the metadata column names

As can be seen, the document that was downloaded has the unique Document ID embedded in the metadata. Note that this ID will change if the document is uploaded to a different library.

In the ‘docProps’ folder we find three sets of XML files:

In the ‘coreXML’ file we see the Dublin Core (DC) metadata that you see in the document Properties above. You can add all the Dublin Core metadata to the library, they are built-in to every library, which means that every document can have all that metadata.

The actual content (the body) of the email is found in the ‘word’ folder of the XML files. Here is the content of that ‘word’ folder:

In the screenshot below you can see some of the ‘document.xml’ content including the metadata that has been added in the body of the document (separately from the properties of the document).

All this metadata is accessible and is used by the Microsoft Graph.

Excel files

Excel files are interesting because, in a sense, they contain data within data. Here is some data in a spreadsheet:

This data is – strangely – stored in two different XML files. The text (including the column headings) is stored here: \xl\sharedStrings.xml:

The values are stored according to each worksheet. For example: \xl\worksheets\sheet1.xml (first two rows only)

A note about emails

Emails do not have the same XML-based structure as Office documents and generally cannot have additional metadata added (except as tags).

Emails in Outlook (sent or received) become ‘.msg’ files if saved to another location from Outlook.

The ‘.msg’ format is based on CFB_3, or compound file binary format, a format that was also used by earlier versions of Microsoft Office documents. It is ‘a general-purpose file format that provides a file-system-like structure within a file for the storage of arbitrary, application-specific streams of data’. (Source: Microsoft web page on Compound File Binary File Format).

Copies of Microsoft Teams chat messages are also stored in a hidden folder in Exchange mailboxes, as instant messages. They cannot be accessed directly but should be considered as a type of archive copy – the originals are stored in a separate database.

If emails are saved to a SharePoint document library, they can be described with additional metadata while stored in the library, but this metadata does not become part of the core metadata of the email or remain with it if it is downloaded, as it does with other Office documents.

In any case, whether they remain in Exchange/Outlook mailboxes, are copied and stored in SharePoint or other Microsoft-based locations, the metadata content in them is accessible via searches.

Active Directory completes the relationships

Every digital record has an author and is likely to have contributors (modified by). Every email is sent and received by someone. All of the internal names linked with digital content are recorded in an organisation’s Active Directory. Employees are also likely to be added to Security Groups (sometimes known as AD Groups) that provide a way to control access to IT resources.

The relationship between document-based content (documents, emails), and between people in AD Security Groups, provides the ability to establish relationships between content, people and business activities.

A final word

Importantly, managing records data does NOT remove or exclude the business need or requirement to aggregate (e.g., in document libraries, mailboxes), manage through time, and then destroy or transfer records according to business requirements. Instead, it enhances this capability by ensuring that all records about a given subject or context can be identified and that, as much as possible, only one copy of the record exists.

Managing digital records as data

(Note – while drafting this post I became aware of an MA Dissertation on the subject of ‘Artificial Intelligence and Record-keeping’ being developed by Mohamed Ben Tahayekt at University College London. I have not had access to this material but I believe some of the concepts may be similar to those outlined in this post.)

Digital records have long been thought of (and described) as being ‘unstructured’.

The reality, however, is that almost all contemporary text-based digital record is made up of a defined, structured and mostly open or accessible package of data that is based on standards. For example:

• Microsoft Word, PowerPoint and Excel documents are all based on an XML structure (indicated by the ‘x’ on the end of the file extension) described in ISO/IEC 29500 and ECMA 376.
• Google Docs exist only in an online format (described in this Google site); to access them offline they must be converted to one of the following formats ISO/IEC 29500 format, ODT (ISO/IEC 26300), PDF or html.
• Emails are now mostly based on the Internet Messaging Format (IMF), standardized by RFC 5322.
• PDFs are based on the open standard ISO 32000.

All of these standards support interoperability between systems (and devices). (See my post about Metadata Payloads for more information on this subject).

An exception to the above are binary objects, including digital photos and images and where these are embedded in text-based documents. But even so, most binary objects are stored with a range of metadata to describe them.

EXIF metadata automatically captured from a smartphone camera

Given that text-based digital records are already full of readable and accessible structured data (and binary objects come with a range of descriptive metadata), is it possible to manage digital records as self-contained data objects?

Records and context

Digital content (records and non-records) will be always be captured, saved to or stored somewhere:

• In email mailboxes. Emails of course may include attachments that duplicate records stored elsewhere in the system, or are not stored anywhere else – e.g., received from outside the organisation.
• In a drive/folder structure in a network file share location, including ‘personal’ drives.
• In a library/folder in online file storage and collaboration platforms, including ‘personal’ online storage locations.
• In corporate enterprise ‘social’ platforms such as the intranet.
• In corporate messaging and chat applications.

Some of the above may have well-defined ‘filing’ or storage structures (including folders) that are used to store or ‘file’ records. Some of these may include the ability to classify and categorise records, and add additional metadata.

An example document library in a SharePoint site

In an organisational setting, all of this digital content will be created, sent/received, or modified by someone listed in Active Directory (AD), a system that generally links employees through their organisational structure. Additionally, employees are likely to belong to several AD Groups that further define relationships between them.

These relationships are important as they help us to understand the context for records.

Isolating records from other content

But one of the challenges for any organisation is knowing what is a record and what isn’t. Perhaps that isn’t as important as it sounds, if all the digital content is considered a potential record.

Organisations create or receive and store a lot of digital content, and a lot of this content has traditionally been kept (on backup tapes) for a long time to support disaster recovery and investigation purposes.

Only a percentage of this content is likely to fit the standard definition of a record – ‘evidence of business activities’.

And some digital content may not obviously be a record until it is connected with or related to other content or activities. For example, a simple email that says ‘Yes’ or ‘OK’ may be the record of agreement to something that doesn’t form part of any other obvious records until it is identified as being a record.

Not uncommonly in traditional electronic recordkeeping systems, there could be no guarantee that everything copied there was a copy of every record that existed on a given subject. Additionally, a record stored in a recordkeeping system may be of relevance in other contexts.

The key to what a record might be is the word ‘evidence’; this is exactly what lawyers look for when they conduct eDiscovery activities.

Rather than assume all records can be accurately found and managed amidst the volume of all digital content, it may be more efficient and accurate to assume all digital content is a record and then apply rules and tools to manage that content, with the aim of identifying records and their potential context based on the data contained in the individual digital objects and their relationships with both other records and people.

In other words find records amongst the entire content, rather than seeking to isolate only those digital objects that are identified as records and copy them to another location – while leaving the originals and potentially other related records in place. Managing records this way avoids the problem of email threads or chats that continue after the copy has been made, or a new copy of a Word document appearing.

How can we achieve this outcome?

There are three potential ways to manage records as data.

The first is to understand, even in general terms, is that digital content is not unstructured, and to learn more about how they are structured. Some simple examples:

• Every email (and instant messages) has a sender, recipient, date sent, date received. They usually (but not always) have a subject. The text-based body of the email provides an additional form of accessible data. A quick look at email headers reveals a huge amount about the email.
• Every document (and web page) has an author, created dated, modified date and last modified by, and a name. They also have a large amount of other data, a lot of which is visible in the Properties section.
• Photographs are stored as binary objects but have a range of EXIF metadata that includes the creation date, information about the camera settings, and may also include the name of the person who created it, as well as a GPS location.

Word document properties showing data added to the record and storage location (Doc ID reference)

The second is to understand that digital content may include added data or metadata. This added data may relate to or derive from the location where the record is stored, or may be added by end-users as part of their work. It may include a unique identifier and information about the aggregation where it is stored, as well as recordkeeping classification terms. Additionally, it may include both process metadata (modified by, and when) and security or access control metadata. Depending on where it is stored, this additional metadata may be embedded with the document properties (the metadata payload).

The third is to have access to (ideally) all digital content across the organisation, and the necessary tools (or access to people with them who can provide usable output) to search and retrieve, relate, and manage all digital content on any given subject or context through to disposal. A very simple example of this is to run a PowerBI report across the network file shares.

And lastly, while there will always be some form of ‘local’ aggregation for specific records where all the records are stored in the one place (mailbox, document library, folder), the only way to establish an aggregation of all digital records on a given subject or context using data only is through the use of advanced searches and/or eDiscovery tools and/or data reporting or visualisations and/or artificial intelligence to find, link and relate content.

Linking and relating content

The diagram below from Microsoft about its Graph technology, provides a simple example of how content can be linked and related through its data.

There are now many data analytics and data visualisation tools that help to understand digital content. These tools are just one part of the picture.

Data analytics tools (such as ‘Constellation‘, a joint project between the Australian Signals Directorate and the Australian CSIRO) are a starting point to understand digital content – including digital content from line of business systems.

A data visualisation from the Constellation website (link above).

These tools might be used to identify content or people related to a given subject, through chat messages, emails, documents etc, including content that is already linked through its own context – the mailbox or a SharePoint library. From that information it would be possible to build a picture – types and volume of content, people, and the relationships between them.

A global search should be able to retrieve, and if necessary export, the content, keeping in mind always that the nature of digital content means it may continue to be modified or new content may added at any time.

As searches improve, narrower set of content allows more granular analysis and visualisation, allowing the identification of sub-sets of records within broader sets. For example, of the potentially large group of ‘everything about COVID’, just the narrower set ‘Vaccines’.

All of this could be achieved through the data that makes up the digital content. And many data-driven organisations are likely to be doing just this, using a range of business intelligence tools to understand the information available to them, in both line of business systems and other content.

Can we manage records as data

Perhaps ‘manage’ is not the right word, or at least not in the sense of expecting digital records to be managed as exceptions to the rest of the digital content.

But there is a lot more we can do to make this outcome possible. We can leave the records where they are stored or captured, we can apply local structure to those records, or security controls. We can keep records of changes that are made. We can apply retention rules that prevent the destruction of any record, or potential record, before it can be legally destroyed.

Instead of ‘managing’ records as exceptions, we can leave the data where it was created or stored, and use a range of tools to help us understand and manage it.

This will allow us to manage records as data and finally achieve the ‘semantic office‘ I wrote about in 2010.

Microsoft 365 – Options for collating, grouping and aggregating records

The international standard for records management, ISO 15489-1:2016 (‘Information and documentation – Records management – Part 1: Concepts and Principles’), defines records as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’.

Among other things, the standard notes that records systems may exist in a variety of forms, not necessary as or in a single or dedicated application. It also underlines the importance of appraisal; that is, the recurrent analysis of business context, business activity, processes and risk for the purpose of determining what records to make and keep and how to manage them over time – especially given the complexity of contemporary recordkeeping.

In terms of risks, the standard states that risk management is required to develop strategies for managing records and the management of records as a risk management strategy in itself.

Unlike traditional electronic document and records management (EDRM) systems that are used to store copies of records created and stored in other applications (‘exception management’), the Microsoft 365 environment is a single system in which records are a sub-set of the entire content (‘exception identification’).

This post discusses how records can be collated, grouped and aggregated in Microsoft 365 to meet requirements for management records. It emphases the point made in the international standard that the risk to records should be understood and minimised.

Records and context

Records are usually created or captured in some form of context – for example a business activity or project. This in turn provides the basis for collating, grouping or aggregating those records according to that context – commonly, a ‘subject’ or ‘topic’.

Records may be a subset of a broader subject (or series). They may be relevant or relate to more than one context or subject.

Digital records that may have no obvious context when they are first created or capture (for example a casual email about an ‘unusual virus outbreak’ in November 2019) may form part of a specific context only when their value is recognised (‘global pandemic’).

Grouping digital records

Grouping records in the digital world has up until now usually involved copying a digital record, created or captured in one system (such as email or a network file share), to a digital ‘file’ in another system such as an electronic document and records management (EDRM) system. The digital ‘file’ in those systems is a virtual representation; the records are actually stored in a file share, linked by metadata in the form of a file number.

The grouping of digital records as exceptions had (and continues to have) several flaws:

• It assumed that all types of digital records could be stored in a digital ‘file’ from where they could be faithfully and reliably rendered (and not just stored as zipped versions of exported content from the originating system).
• It relied on the willingness of end-users (often after training) and/or a technical third-party system, to copy a record to the system. This ‘exception management’ meant that some records were not copied to the EDRMS.
• It was a ‘point in time’ capture. The original digital record remained in the system where it was created or captured, and might also be attached to emails and from there saved to multiple other locations.
• There was no way of knowing if all the records in the file were all the records relating to the subject.

Where are the records created or captured in Microsoft 365

Most business records in Microsoft 365 will be created or captured in Outlook/Exchange mailboxes, SharePoint site libraries or MS Teams (which stores chat in Exchange mailboxes and documents in SharePoint or OneDrive). (For the purpose of this post, OneDrive is seen as a personal working space that should not be used to store business records.)

Regardless of whether they are created or captured in Exchange or SharePoint (including via Teams), all of the content – records and non records – created or captured in Microsoft 365 is stored in the Azure substrate. This effectively means that records in Microsoft 365 are a sub-set of all the other content stored in the Azure substrate.

Consequently, the management of records in Microsoft 365 involves exception identification. That is, identifying records and ensuring they are managed appropriately as much as possible where they are captured or created – and placing other controls over all the other content as necessary.

Everything created and stored in Microsoft 365 – including all the very rich metadata associated with every digital record – is subject to the Graph. The Graph identifies relationships and ‘signals’ not only between digital content but between people (agents) and business activities.

The Graph powers Delve and Discovery and the soon-to-be-released Project Cortex, presenting information (they have access to) to end-users that can sometimes be unsettling for people used to working in relative privacy. See below for further discussion about Project Cortex.

Additionally, as all the content in Microsoft 365 is stored in the Azure back-end, most of it can be searched and (where necessary) exported through the Content Search option in the Compliance portal, a capability that supports eDiscovery. This capability means that even when records are not ‘manually’ identified as records, there is a better chance they will be found.

How are records aggregated in Microsoft 365

There are three main ways that records are, or can be, aggregated in Microsoft 365: Exchange mailboxes, SharePoint site libraries, and Microsoft Groups that have a mailbox and a SharePoint site and can be linked to (or created from) a Team in MS Teams.

Exchange/Outlook

Exchange aggregates email records by:

• Personal mailboxes, accessible only the ‘owner’ (end-user).
• Shared mailboxes, accessible to those who have access.
• Microsoft 365 Group mailboxes, accessible to the members of the Group (including anyone added to the Group).

Although a mailbox is a form of aggregation, there is no way to relate or link emails stored there with other related records stored in SharePoint unless they are copied to a SharePoint document library, as can be seen in the example below. This is recommended if an organisation wants to keep emails together with other records.

A SharePoint library with an email and a document

Emails copied to a SharePoint document library are a ‘point in time’ copy; there may be additional replies to the email, forming a thread that isn’t captured.

The alternatives to copying emails to SharePoint are:

• Leave all emails in mailboxes and use Content Search to find and export them to SharePoint as a PST.
• Creating a Microsoft 365 Group with an associated mailbox and SharePoint site, so that the records are retained in the context of the Group.

In any case, all mailboxes should be subject to a minimum retention period to ensure that any email that might be a record is preserved for that period. Certain mailboxes (for example, senior or key staff members) may be kept for longer periods and then exported for permanent storage.

SharePoint

SharePoint document libraries are logical aggregations for the storage of records, including emails copied from Exchange mailboxes.

A SharePoint library configured to manage records

Ideally, individual libraries that are used for the storage of records should map to a business activity and/or records retention class; this mapping should be reflected in the library name.

NOTE: Individual document libraries should not be used to store records relating to multiple subjects or mapping to more than one retention class or policy.

Document libraries may be assigned as much metadata as required, and content stored in them can be defined through the use of metadata and/or content types.

Almost every type of digital file, with a (newly announced) 100GB single file limit, can be stored in SharePoint Online, as noted in the article ‘Types of files that cannot be added to a list or library‘ (restrictions only apply to on-premise versions).

Microsoft 365 Groups (including Teams in MS Teams)

Microsoft 365 Groups provide a way to group and manage records, including MS Teams channel chats, in the context of the Group.

The relationship between Teams and Groups, Exchange and SharePoint

Every Group includes a mailbox (visible in Outlook) and a SharePoint site, and can be linked to new Team in MS Teams. Teams channel chats are stored in a hidden folder in the Group mailbox. Any documents and records are stored in the ‘Files’ tab of the channel, which surfaces the default ‘Documents’ library in the connected SharePoint site.

If the creation of Teams is allowed from the MS Teams application, every new Team creates a Microsoft Group (with the same name) and a SharePoint site (with the same name), however the mailbox (with the hidden folder for channel chats) is not visible from Outlook.

(The exception here are private channels; if these are allowed: (a) the chat content is stored in the Exchange mailbox of the each participant, and (b) a new SharePoint site is created for the ‘Files’.

The relationship between the content created by the Group is most obviously visible from the ‘Activity’ web part of the SharePoint site of the Group as can be seen in the screenshot below. This shows (right to left), an original incoming email from Outlook in the Group’s mailbox, the copy saved to the SharePoint document library, and the Word document reply. The specific context of the record (= the ‘file’) – ‘Correspondence 2020’ – is defined by the document library.

What about records in 1:1 Teams chat

As with OneDrive, Teams 1:1 chat should not be used to create or capture records, but may be used as a ‘working’ space.

However, ‘should’ and ‘reality’ can be different things. There are two ways to address this:

• Explictly, through communication to end-users. Make it clear that Teams 1:1 chat and OneDrive are NOT to be used to create or capture records. Applying short-term retention policies to this content may assist with reducing (or increasing) this risk.
• Implicitly, through monitoring and retention policies. Apply longer-term retention policies to the content and use Content Search/eDiscovery to look for content that may be records. Additionally, review the content of the OneDrive of departed staff and ensure that any records are kept.

Implications for managing records

The implications for collating, grouping and aggregating records in Microsoft 365 are as follows.

• SharePoint document libraries will continue to be the primary aggregation for managing corporate records, including emails copied from Outlook.
• Organisations should establish an architecture model for SharePoint sites that are used to manage records. The model may include a mix of the following: (a) sites mapped to business functions with libraries mapped to business activities and retention classes, (b) entire sites used to create and capture records relating to a single activity, where the entire site is mapped to a retention class, and (c) MS Groups (and Teams) with an associated SharePoint site, where the Group (mailbox/SharePoint site) is subject to a single retention class (and the Team channel chat also).
• More effort, in terms of site/library set up, metadata, access controls, retention and end-of-retention process is likely to be required for the management of high-level, high-risk and permanent records.
• Personal mailboxes in Exchange will continue to exist as a form of aggregation, and consideration should be given to having different retention policies for different ‘types’ of mailbox, to ensure that any email that could be a record is not deleted too quickly.

Addendum – Other options that collate, group and aggregate content in Microsoft 365

Content Search/eDiscovery

As noted earlier, all of the content created or captured in Microsoft 365 is stored in the backend Azure substrate. Consequently, it is possible to search across all or part of that content to find related information and, where required, export it to a different location.

The global Content Search is accessed from the Compliance portal and access requires elevated privileges – Global Admin or Compliance Admin.

Searches are created as cases and are based on keywords, conditions (such as ‘Sender’ for emails), and locations – all or specific. When a new content search is created or run, the Global Admins are alerted, providing a form of oversight in addition to audit logs.

While content searches find content is related to the search parameters, and legal holds can then be applied to that content, they do not create any form of aggregation in a recordkeeping sense.

The Graph, Delve, Discovery

Microsoft describe the Graph as being ‘the gateway to data and intelligence in Microsoft 365 [that can be used via the Microsoft Graph API] to access the tremendous amount of data in Microsoft 365, Windows 10, and Enterprise Mobility + Security’ and ‘… build apps that support scenarios spanning across productivity, collaboration, education, people and workplace intelligence, and much more. (Source ‘Overview of Microsoft Graph‘)

The Graph is commonly represented in diagrams similar to the one below.

Most end-users will encounter the Graph through either Delve or the Discover option in both the office.com portal and their OneDrive for Business accounts.

It is not uncommon for end-users to express surprise at the content (that they have access to) that is presented. Commonly this will show documents that a colleague is working on, or connections between people. Disabling Delve does not fix permissions; if a person has access to a document that appears in Delve, they will be able to search for it and find it that way.

Over time, the Graph can also provide other information based on the relationships or ‘signals’ it finds between all the different content in Microsoft 365.

While the Graph can present groups of records that have some relationship to the end-user, it does not aggregate those records or maintain a single consistent view. However, the Graph powers the new Project Cortex that does do something similar.

Project Cortex

Project Cortex was announced by Microsoft in April 2019. To quote the announcement, Project Cortex:

• Uses advanced AI to deliver insights and expertise in the apps you use every day, to harness collective knowledge and to empower people and teams to learn, upskill and innovate faster.
• Uses AI to reason over content across teams and systems, recognizing content types, extracting important information, and automatically organizing content into shared topics like projects, products, processes and customers. Cortex then creates a knowledge network based on relationships among topics, content, and people.

From a recordkeeping aggregation point of view, a core functionality of Project Cortex is its ability to create ‘topic cards’ based on the rich metadata that makes up all the content in Microsoft 365. Again to quote the announcement:

• Project Cortex securely collects content that is created and shared every day in Microsoft 365—including files, conversations, recorded meetings and video—and it categorizes the content based on its type, and tags it with extracted metadata.
• AI then applies advanced topic mining logic—whether its content contained in Microsoft 365 or connected from external systems—to identify topics and relate content to those topics.
• Topics can reflect any knowledge that’s important, including customers, products, projects, policies and procedures. Technically, AI is creating knowledge entities, a new object class, in the Microsoft Graph. The relationships between those topics—those knowledge entities—and the experiences that connect this knowledge with people creates your knowledge network.

Topic cards – or ‘knowledge entities’ – are a form of AI-generated aggregation.

However, topic cards will only present information that an end-user has access to and so the nirvana of presenting emails or Teams 1:1 chats in these cards as a form of aggregation for recordkeeping purposes is not likely to be realised through Project Cortex.

A Tale of Two Retention Policies in Microsoft 365

There are two ways to create retention policies in Microsoft 365 – (a) by creating a label and publishing it as retention policy, or (b) creating a retention policy without a label. Both are created from the Information Governance section of the Compliance portal in Microsoft 365.

This post describes how the policies are created and applied, and the outcome for each.

Retention policy based on a label

Retention labels are created from the Labels tab of the Information Governance section in the Compliance portal.

Ideally, the name of the label would be based on a disposal class in a records retention schedule or records disposal authority. For the purpose of this post, the new label was named ‘Test One – Retain five years then delete‘.

Adding the details of the retention provided a quick visual clue to the retention details.

The label was configured to have a retention period of 5 years after the date modified after which the content could be deleted, with no disposition review. No record would be kept of what was deleted.

Alternatively, it would be possible to select the Disposition Review option. This option sends an alert to the account indicated so the records could be reviewed before disposal. However, keep in mind that if the trigger is left as ‘Date created’ or ‘Date modified’, records will be ready for disposition review based on those dates. It might be better to choose ‘Date applied’ instead so that a group of records could be reviewed at the same time – for example, in a SharePoint library.

Retention labels have no life unless they are published. This is achieved from the ‘Publish labels’ option in the same tab.

One or more labels can be published in a retention policy. This could be useful if the records disposal authority contained multiple labels for a single function.

As part of the publishing process, a decision has to be made to apply the policy to somewhere in Microsoft 365. In this case, the decision was made to apply it to a single SharePoint site because that site contained records that mapped to the retention label.

Finally, the new policy was given a name. In this case the policy had only one label so the policy had the same name as the label.

If the organisation’s records disposal authority/retention schedule had multiple retention classes mapped to a business function, the policy could be named after the function and include all the relevant classes created as labels.

All of these labels would appear in the drop down list from the Library Settings on every site where the policy was applied.

But even when it was published in this way and applied to a specific SharePoint site, the label does not do anything. The policy has to be applied to a document library in that site – but even then it wouldn’t do anything until the label was selected from a drop down list.

The screenshot below shows how the specific label is selected on the library and whether it should be applied to all existing items.

The label can now be seen in the ‘Retention Label’ column (when made visible in the view settings).

Anyone who uses the library to add and edit content (site members) will notice an obvious change if they try to delete anything, including from their synced document libraries in File Explorer.

They can’t.

There is, however, a workaround that end-users can use to delete documents with a label: (a) check the box next to the document, (b) open the information panel and (c) remove the label on individual documents.

Once the label is removed it no longer appears against the record.

The document can now be deleted. It will move to the Recycle Bin, from where it could be restored for up to 93 days if a mistake was made.

The lessons to be learned from this are that:

• It might be useful to make the library read only after the label is applied.
• The label should only be applied when the library became inactive, to allow people to get on with their work while it is still active.

If anyone was concerned about content being deleted from an active library, they could set an alert on the library.

In the meantime, for all the other records that retained their label, when the retention period ended the records in the library would be transferred to the Recycle Bin where they would sit for 93 days before complete and permanent deletion.

If no-one restored a document from the Recycle Bin, the content would be deleted, permanently, forever, without any record. The records managers might need to make a copy of the metadata for the documents to be deleted before they are deleted.

Alternatively, if they had selected the option for a ‘disposition review’ when the label was created, this would alert the records managers (and others) of the need to review the contents of the library.

Once everything had been deleted, all that will be left will be an empty library with no record of what used to be there.

Retention policy without a label

Non-label retention policies are created from the ‘Retention’ tab of the Information Governance portal.

For the purpose of this post, this label will be named ‘Label Two – Retain 5 years then destroy’.

This policy has the same retention period, trigger, and action as Label One – 5 years after date modified – and then the content could be deleted without any record of this being kept. [Note – yes the screenshot below shows 7 years].

As the policy was being created it was applied directly to a SharePoint site.

The policy could start working immediately.

Nobody, apart from the person who created and applied it knew it had been applied. It didn’t appear anywhere on the SharePoint site where it had been applied.

But, as part of the application process, it created a Preservation Hold (PH) library visible only to Admins, including the Site Collection Admins but not the Site Owners.

(Note – if end-users are allowed to create SharePoint sites, or Teams in MS Teams that also creates a SharePoint site, they are made Site Collection Admins by default and can accordingly see the Preservation Hold Library).

Anything that was deleted in advance of the retention policy expiry would be moved to the Preservation Hold library.

The PH library had a specific purpose – to ensure that everything on the site was retained until the end of the retention period.

It was not possible to restore any items from the Preservation Hold library. It was also not possible to ‘clear’ that library. If an administrator tried, they would be advised that ‘something went wrong’.

The only way to delete from the Preservation Hold library would be to remove the policy.

However, if a record was deleted it would be placed in the Recycle Bin for up to 93 days, from where it could be restored. If a record was restored in this way (from the Recycle Bin), the copy in the Preservation Hold library would remain.

For most people this type of policy was fine. They could (appear to) delete records, unaware that nothing was actually deleted if it were subject to a retention policy.

The same thing happened if a similar type of retention policy was applied to OneDrive accounts. End-users could ‘delete’, but nothing was actually deleted as long as the retention policy was active.

At the end of the retention period, the content that was subject to this policy was transferred to the Recycle Bin where it remained for 93 days. It would then be deleted, permanently, forever, without any record.

All that was left would be an empty library with no record of its previous content. The only clue would be in the ID or the ID part of the Document ID (if enabled). If anything new was added to the library, the next number in sequence would indicate how many records had previously been saved to that library.

Lessons to be learned

There are several lessons to be learned.

The first is that you need a plan to create and apply retention labels or retention policies to content across Microsoft 365. The configuration options and implications of each option need to be understood before they are applied.

You need to understand the difference between (a) a retention label published as a policy and applied to a SharePoint site and then a library on that site, and (b) a retention policy applied to an entire SharePoint site.

Retention labels can be mapped to retention schedule/disposal authority classes and then applied to specific libraries on SharePoint sites where the policy is applied. Accordingly, it may be useful to create document libraries that map to those classes, and only store content in those libraries that is covered by a single retention class.

If single SharePoint document libraries are used to store content that maps to different retention classes, then it will become very complicated to accurately map retention labels to content.

In many cases, it may be possible to apply a single non-label retention policy to a single SharePoint site or sites. For example, all project sites might have the same 7 year retention applied to all the content on the site as there may be no real need to apply policies to individual libraries.

Whatever you do, have a plan to act, document your settings, and keep it up to date.

And understand what happens at the end of the retention period. There is no back up.

What to use when – metadata, content types, folders or document sets

Most end-users will be familiar with using folders to ‘categorise’ content in SharePoint document libraries, but not everyone will be familiar with other options including document sets or the use of metadata or content types to achieve similar ‘grouping’ outcomes.

Any (and all of) these options can be used to categorise content in a document library. But, aside from the default folder option, some thought should be given to the actual purpose and the end-user experience for the other three options.

This post provides some suggestions for what to use when.

In summary:

• Use folders most of the time, but try to limit them (through end-user communication) to no more than two levels. Any more and it may be better to create a new library.
• Use document sets where there is a (genuine) need to group documents with additional metadata and security controls. In some cases, metadata may be a better option.
• Use content types (including document sets) sparingly and for specific business requirements, not just to categorise document types in a single library when a metadata choice, naming conventions, or separate libraries may provide a better outcome.
• Use metadata columns when it makes sense to display content grouped and filtered by that metadata.

Document libraries are logical aggregations for records

As a starting point, keep in mind that SharePoint document libraries are logical aggregations or ‘containers’ for records.

When a SharePoint site is mapped to a business function, the libraries in that site can be mapped to or named after the activities that produce records.

It is generally not a good idea, from a recordkeeping point of view, to mix records created by different business activities (which is not uncommon in the default ‘Documents’ SharePoint library, especially the site attached to a Team in MS Teams).

Access controls and records retention are much harder to manage in single libraries that contain records created from multiple different business activities.

As the library is the logical container, any option use to categorize or break up content in a library – whether it be metadata, content types, folders or documents sets – is similar to a divider in a lever-arch folder. Ideally, the entire content of the document library should contain records about the same subject.

Metadata

Content in a document library can be categorized using metadata.

SharePoint sites have more or less unlimited metadata that can be applied to content in document libraries. All document libraries include the default ‘Name’ and ‘Title’ metadata fields as well as system generated metadata.

SharePoint also includes the Managed Metadata Service (MMS) that allows for the creation of a centralised set of hierarchical metadata terms, based on taxonomies and thesauri, accessible to all sites.

Microsoft announced in April 2020 that it was modernizing the SharePoint Managed Metadata Service (MMS) and that the updates ‘will be essential to delivering new premium value to Project Cortex later this year’ (2020).

To add metadata to a document library, it must be either (a) added via a content type that contains one or more site columns, or (b) created as a ‘local’ library column. Both of these options can link to the MMS.

Once the metadata has been added, documents may be grouped in a view by the metadata. In the example below, they are grouped by the choice metadata column ‘Document Type’.

The benefits with using metadata include:

• Ability to draw on the global terms defined in the MMS.
• Ability to group and filter content, especially useful in libraries with a lot of content. Note that certain columns, including those that use MMS terms, cannot be grouped.
• Supports multiple pre-defined views that can replace search (especially useful since ‘faceted’ searching was deprecated).
• Support for recordkeeping requirements.
• Support for search.

The negatives with using metadata relate to usability and syncing.

• End-users generally don’t like to add more than two items of metadata when they upload a document.
• End-users generally don’t ‘get’ categorization or grouping by metadata (but generally they don’t seem to mind pre-defined views).
• Metadata columns are not visible in libraries that are synced to File Explorer.
• If a metadata column is mandatory, the synced library will become read only.
• Global metadata defined in the MMS may not suit all parts of the business. We found that even a simple set of terms to define ‘Document Type’ (e.g., ‘Meetings’, ‘Agenda’) varied across the organisation.

Suggested use case:

• Use metadata columns to categorise records (and the MMS) only when it meets a specific business need, not by default. For example, a document library for organisational policies that lists each policy (or group of policies and procedures) via the ‘Policy Name’ metadata field. The metadata can be used to describe the policies and related documents, categorize them in multiple ways. End users are unlikely to need to sync this library. This method also assists with cross-referencing related policies including by using links.

Content Types

Content types are a fundamental building block in SharePoint, and are used to define content.

SharePoint has long included the concept of a ‘Content Type Hub’ where, in theory, all Content Types are centralised. This will be replaced in the near future with the new ‘Content Types Gallery’ option accessible from the SharePoint Admin portal.

All content types derive from the top level ‘System’ content type that is used to define the ‘Item’ content type which in turn defines the ‘Document’ content type.

Every SharePoint document library includes (a) the default ‘Document’ content type with two metadata columns ‘Title’ and ‘Name’ and (b) the default ‘Folder’ content type. They also include the ‘Document Set’ content type when this is enabled as a site feature.

New content types are created from the Site Settings area of every SharePoint site but must be added to a library by first allowing content types (from Library Settings – Advanced) and then adding them from ‘Existing Site Content Types’. This process includes document sets which are discussed below.

Given their direct relationship with metadata, there may be a tendency to want to create a lot of content types whenever additional metadata is required and to apply multiple custom content types to a single document library to define a type of content when other options, such as a metadata choice or even folders, may work better.

In the screenshot below, the library column shows the Content Type for the particular object. This column can be grouped in a view, providing a way to show different content types in the library.

The benefits with using content types are as follows:

• Help to define content through the use of metadata.
• May include a template document which can use the metadata applied to the item. For example, automatically adding the metadata in the body of a Word document as a content control.

The negatives with using content types are as follows:

• There can be too many content types.
• End users don’t like having to choose a content type, especially if the only apparent purpose is to define a document type (e.g., ‘Meetings’ vs ‘Agenda’). In this case, a simple metadata column would be sufficient OR use document naming conventions instead.
• End users don’t like adding metadata after having to choose a content type.
• If any column in the content type is mandatory, it will cause the synced library to become read-only.

Suggested use case:

• A document library that is used to support a specific business purpose where it is necessary to define one or more documents using additional metadata, and a pre-defined document template is required. For example, a document library used to store client agreements based on a template. For each client agreement, there is a requirement to capture certain metadata (client name, address, phone, etc) which will then be auto-populated into the template agreement document.

Document Sets

Document sets are a like a super folder in that they behave like normal folders but have a lot more functionality.

As noted above, document sets must be enabled as a feature before they become visible in the list of available content types. Once enabled, and the option to ‘Allow management of content types’ is enabled in the Library Settings (under ‘Advanced’), they can be added to a document library from the list of available content types.

Document sets can be very useful where there is a need for (a) a unique identifier for a folder inside a library, (b) additional metadata to help define a group of documents and (c) access controls that may be different from the library. (Note – this last option is also possible with folders, see below).

While document sets may sound appealing to business areas that want to ‘hard categorize’ content in a document library (for example, every document set contains a policy, and related procedures, forms and so on), this can actually result in making it harder for end-users to find information. The reverse of this is that they can be very useful if there is a genuine requirement to restrict access to information.

Document sets are visible in the screenshot above regarding content types. When added to a document library, they are visible from the ‘New’ menu as shown below:

The positives of document sets include the following:

• They can have a unique ID (based on the Document ID being enabled in the library); the Document ID is the same used for the documents.
• Metadata can be added automatically to the body of template documents (via content controls) stored in document content types linked with the document set.
• They can be used to restrict access.

The negatives of document sets include the following:

• Can make it harder to find information, especially if there are more than around 30 in a library.
• Searches may retrieve only one document. It will also not retrieve the document set. There may be other related documents.
• Can be more difficult to replace with other options because of the relationship between document sets and linked document content types.
• In synced file libraries, document sets appear as folders. For this reason it is recommended that any library using document sets includes at least one mandatory column to make the synced version read-only. Ideally, end-users should not need to sync these libraries.

Suggested use cases:

• Document sets are a good way to group client-related or employee documents in a single library, especially when there is a need to restrict access between the document sets. In this sense, document sets work like ‘sub-files’ in a library. For example, Manager ‘Jim’ may need to see five document sets, while Manager ‘Mary’ may need to see eight others. Neither will be able to see the others files when they access the library.
• In smaller organisations, a SharePoint site might include a library called ‘Projects’, with one document set per project. Larger organisations might have a library per project, even larger organisations might have a single SharePoint site (perhaps based on a Microsoft 365 Group) per project.

Folders

Pretty much everyone knows what folders are and how they are used. Every SharePoint document library includes the option to create folders. When synced to File Explorer, end users can create folders there too.

It is possible to prevent the creation of folders in SharePoint via the Library Settings – Advanced – Folders option but this option should really only be enforced in exceptional circumstances.

It is also possible to view all the content in a library without folders by editing the view (or creating a new ‘No folders’ view) and changing the option in the ‘Folders’ section from ‘Display items in folders’ to ‘Show all items without folders’. It is not uncommon to find, when the no-folder view is displayed, to see fewer documents than folders. It is sometimes a good way for admins to demonstrate the benefit of using metadata columns instead.

The positives of using folders include the following:

• Everyone knows how to use them.
• They can be created from File Explorer.

The negatives of using folders including the following:

• Too many folders and levels.
• Folders that repeat other previous words in the URL path, including the name of the site and the library.
• One word folders – for example: Meetings – 2020 – June – 8.
• Folders with unique permissions.

Suggested use case:

• Folders are best for simple document libraries. It is a good idea to let end users know about the 400 character limit and suggest they might consider creating new libraries if the folders are clearly about entirely different subjects (unless this is very low level). For example, don’t mix records about meetings with records about policies or staff rosters all in the same library.
• Note: Try to avoid using unique permissions on folders. If unique permissions are really required, create a separate library.

Using all four (or a mix of the four) options

It is possible to use all four options in the same library:

• Folders
• Additional metadata colums, which may be used to group content in views without folders
• Added content types, including document sets with added metadata.

But it’s probably not a good idea to do this.

Keep it simple

If you are trying to categorise records in a way that makes some sense, it is not a bad idea to keep it simple and only deploy the option or options that best meets the business needs.

• Folders are best for most end users interacting with a library including via File Explorer. Limit the use of extra metadata – for example ‘Document Type’, keeping in mind that added metadata is not (yet) visible in libraries synced to File Explorer.
• Use metadata to create read-only views of records or other content stored in a library (for example, policies), and a small group of people is responsible for that metadata. Using metadata to group content in this way can be better than using folders or document sets. Keep in mind that not all columns (including MMS columns) can be grouped in views but they can be filtered.
• Use custom content types including document sets in situations where there is a need to add specific metadata to records (and use it in templates), or group them in ‘sub-files’ in a document library. Folders can be used in document sets to categorise content further.

Microsoft 365 – ‘Records Management’ solution

Microsoft announced the General Availability of its so-called ‘records management’ solution for Microsoft 365 on 30 April 2020. The announcement included a screenshot of the ‘Overview’ tab of the ‘records management’ section of the Microsoft 365 Compliance portal which contains a range of other options including the very similar looking ‘information governance’ section.

Screenshot of the ‘overview’ tab of the records management section

The announcement noted that organisations would be able to use the records management solution to:

• Classify, retain, review, dispose, and manage content without compromising productivity or data security.
• Leverage machine learning capabilities to identify and classify regulatory, legal, and business critical records at scale.
• Demonstrate compliance with regulations through defensible audit trails and proof of destruction.

It also noted that the solution would be limited to eligible Microsoft 365 E5 customers.

This post:

• Examines what the records management solution in Microsoft 365 can do and its limitations.
• Highlights the need for records managers to be more actively involved (a) at the very least in the governance of and provision of advice relating to the management of records in Microsoft 365, and (b) preferably in actually having a specific role to play in managing records in that environment.

Before going into more detail, it is important to understand the differences between an E3 and an E5 licence, and how this relates to managing records in Microsoft 365.

Licensing

Microsoft have published extensive information in its service description for the Security and Compliance elements of Microsoft 365. It is a good idea to discuss these options with a Microsoft licensing partner.

At a general level, the difference between the ‘information governance’ (E3/E5) and ‘records management’ (E5 only) options are as follows:

• Info Governance – ‘Broad policies to catch everything and keep it simple’. According to the service description (link above) this provides for ‘a single organization-wide or location-wide retention policy and/or manual retention labeling’.
• Records Management (E5) – ‘Deeper dives into classifying content, enforcing particular process and workflows’. According to the service description, this means ‘automatically applying retention labels or policies, starting the retention period of a retention label based on a custom event, triggering a manual disposition review at the end of the label’s retention period, importing third-party data through native data connectors, discovering labeled content and monitoring labeling activity.’ and ‘automatically applying retention labels based on trainable classifiers’.

Microsoft have not stated that the ‘information governance’ section is being deprecated but it is clear that ‘records management’ is an advanced version of ‘information governance’.

What it’s not

As a starting point, the new ‘records management’ solution is not a standalone recordkeeping system or application within Microsoft 365, along the lines of an electronic document and records management system (EDRMS).

It is not a place where records are stored. It is one of several options (in addition to the lesser ‘information governance’ section) within the Compliance portal as shown in the screenshot below of the left hand navigation for that portal:

The solution does not make reference to, and is not based on, recordkeeping standards such as ISO 15489 or ISO 16175 Part 2.

It appears to assume that records managers will have a role to play, at the very least in providing advice about the management of records, being part of the governance for Microsoft 365, or having a specific role.

What is ‘records management’ in Microsoft 365

In summary, the ‘records management’ solution (which is still separate from the ‘information governance’ set of options – see below) is a set of advanced options in Microsoft 365 that allow organisations to:

• group (‘classify’) records for retention purposes based on several options including search, auto-classification and pre-defined events;
• allow a review of certain records due for disposal;
• destroy records; and
• retain a record of what was destroyed.

As described, the solution appears to be designed to manage records with minimal manual intervention. To quote from this Microsoft article ‘Automate event-driven retention‘:

‘The explosion of content in organizations and how it can become ROT (redundant, obsolete, trivial) is serious business. To continue to meet legal, business, and regulatory compliance challenges, organizations must be able to keep and protect important information and quickly find what’s relevant. Retaining only important, pertinent information is key to an organization’s success.’

These options are described below.

Responsibilities for records management in Microsoft 365

As noted above, access to the ‘records management’ section of the Compliance portal requires having one of several roles in Microsoft 365:

• Global Admin – usually restricted to a very small (~3) group of individuals in IT, as it gives access to all parts of (and content in) the Microsoft 365 tenant.
• Compliance admin. Gives access to all parts of the Compliance portal including the audit logs and content search for all of the tenant, the Information Governance (E3) and Records Management (E5) sections, and a lot more.
• Customised admin. Gives access to some parts of the Compliance portal.

There is no dedicated ‘Records Management’ role that gives access just to the ‘Records Management’ section of the Compliance portal.

An indication of Microsoft’s thinking about who should have access to what part of Microsoft 365 is contained in this article on automating event-driven retention that defines specific roles in relation to records management:

• (Global/Compliance) Admin – Creates Retention Event types, Retention labels and Record repositories in SharePoint [my emphasis as it is not clear what this is intended to mean in relation to the Admin roles].
• Records Manager – Provides Retention Policies and Retention Schedules guidance and compliance details [presumably to the Admin].
• System Admin (business) – Sets up and manages external systems to work with Microsoft 365.
• Information Worker – Manages the lifecycle of their business process (HR, Finance, IT, and so on).

What does the records management solution actually do?

According to Microsoft, the records management solution allows organisations to:

• Classify, retain, review, dispose, and manage content without compromising productivity or data security.
• Leverage machine learning capabilities to identify and classify regulatory, legal, and business critical records at scale.
• Demonstrate compliance with regulations through defensible audit trails and proof of destruction.

Many of these options are (still) also available in the ‘information governance’ section available with an E3 licence. Where they are only available in the ‘records management’ section, ‘(E5 licence only)’ is indicated.

Let’s look at each of these options, which are also described in this article: ‘Records Management in Microsoft 365‘ and other links.

Classify (via a retention label and policy)

The article ‘Overview of Retention Labels‘ provides an insight into how Microsoft sees the classification of records. It states that ‘Auto-apply’ retention labels are powerful because:

• You don’t need to train your users on all of your classifications.
• You don’t need to rely on users to classify all content correctly.
• Users no longer need to know about data governance policies – they can focus on their work.

In Microsoft 365, ‘classify’ has the meaning of identifying and grouping related or ‘like’ records primarily for the purposes of managing retention. It does not mean applying recordkeeping classification labels to records across Microsoft 365; the only thing that is applied is a retention label that can map to the selected classification.

The Microsoft article ‘Records management in Microsoft 365‘ describes several ways that information can be ‘classified’ when linked with retention labels.

When a new retention label is created a decision must be made about:

• Retention period (e.g., 7 years)
• Retention trigger (date created, date modified, date label applied, an event)
• Disposal action (do nothing, review before disposal, just destroy)

After the label is created, it may then be auto-applied to content across Microsoft 365 or to specific locations (such as SharePoint only). The three auto-apply options are:

• Content that contains sensitive information.
• Content that contains specific words or phrases, or properties.
• Content that matches a built-in or trainable classifier (E5 licence only).

The following describes the three options:

• Sensitive information. This option uses the same set of policies used for Data Loss Prevention (DLP) including custom policies. For example, the policy ‘financial information’ that identifies credit card numbers. In practice in the Australian context, these built-in policies are not very accurate. (See this article for more information on DLP Policies).
• Specific words or phrases or properties. These match queries built via the Keyword Query Language (KQL) or other keyword searches – the same that would be used for Content Searches across Microsoft 365.
• This option also includes the metadata properties (including metadata from the Managed Metadata Service) contained in SharePoint Content Types added to a document library. These metadata elements are crawled properties that can be used to find and automatically apply a retention label. This option allows organisations to include their business classification scheme terms in the Managed Metadata Service (MMS) and use terms from this MMS as metadata options in custom Content Types that are applied to document libraries. A retention policy can then be automatically applied to content that has been classified in this way. In practice, this is a complex model to develop and manage overtime and should only be used for specific types of records.(See this Microsoft article for a very detailed description of how this is achieved: ‘Manage the lifecycle of SharePoint documents with retention labels‘). 
• Trainable classifiers. This refers to both (a) the built-in options of: Offensive Language, Resumes, SourceCode, Targeted Harassment, Profanity, and Threat and (b) additional trainable classifiers (E5 licence only). Both built-in classifiers and trainable classifiers are available as a condition when retention labels are auto-applied to content. (See further discussion below and these Microsoft articles for an Introduction to Trainable Classifiers and directions to ‘Create a trainable classifier‘.)

Note: The ‘auto-apply’ option may take up to 7 days to take effect.

Review

When a new retention label is created, one of three disposal actions must be selected for what will happen at the end of the retention period:

• Do nothing.
• Review disposition (currently E3 and E5). This option requires an email address to notify the person who will do the review. This person must have access to the Dispositions area of the Compliance portal to do this.
• Just delete the content without a review.

If the retention label includes the second option to review the disposition, the nominated reviewer (with the required access role) will receive a notification. They can also navigate at any time to the Dispositions tab (of the Information Governance and Records Management sections of the Compliance portal).

The dispositions section displays individual records that are due for disposal based on the retention label settings, with an option to view by ‘Documents’ or ‘Emails’.

Several points to note:

• Records are not displayed in their original containers (e.g., mailboxes or document libraries etc) but they can be grouped this way via the search option.
• The metadata that may have been assigned to records in a SharePoint library is not displayed and cannot be exported from the Dispositions area. The only way to export the metadata is to access the document library on the SharePoint site – and this would assume the review has the permissions to see everything on the site (Site Collection Admin).
• Only those records covered by the label in the document library will be displayed. This is another reason to review the original library.

If anything, the Dispositions area is useful to provide a heads-up for records managers to review the actual library content in SharePoint. The records manager may then export the metadata of the records to be destroyed.

Note – Any records that are subject to a label with the ‘just delete’ option selected will not appear in the Dispositions section. They will simply be deleted (via a 90 day period in the Recycle Bin).

Leverage machine learning capabilities to identify and classify records (E5 licence only)

This option has been described above in the section relating to the auto-application of a retention label based on the terms defined in a trainable classifier. The diagram below is from the Microsoft article ‘Create a trainable classifier‘:

The question arises whether it would be possible to develop a set of classification terms that (a) map to the records class descriptions contained in a records retention schedule and (b) can accurately identify content that matches those descriptions across the Microsoft 365 ecosystem. This would certainly be an ideal goal.

Demonstrate compliance with regulations through defensible audit trails and proof of destruction

The records management section includes a tab named ‘Dispositions’ as shown in the screenshot below (from the article ‘Disposition of content‘). This is currently the same for both E3 and E5 licences, but some functionality may be restricted to E5.

As noted in this article, ‘Items that are shown in the Disposed Items tab for record labels are kept for up to 7 years after the item was disposed, with a limit of one million items per record for that period.’ (It is not clear yet if this is for E5 only).

The data about records destroyed for each label can be exported.

The image below is from the same article and shows the limited amount of content provided for each item that is destroyed. It does not include any metadata from the original location and does not destroy the original document library. It is up to the organisation as to whether this simple form of disposition review will be suitable or if more details are required.

For most records of corporate value, the disposition review process is too limited in terms of the record it retains of what was destroyed. However, it does provide a heads-up for records managers (provided they can access it).

Most records that are of low value should never require a disposition review, however many organisations may be loathe to automatically delete content – even low-level content – that may be required beyond the minimum retention period.

What about Information Governance?

The alternative to the options provided in the records management section are the labels and retention policies in the ‘information governance’ section of the Compliance portal. These options have been described in a separate post but in summary allow organisations to use one or both of the following options:

• Label-based retention policies. These policies are based on labels that can include a disposition review. Once published, they must be manually applied to Microsoft 365 workloads but are most likely to be used on SharePoint document libraries.
• Retention policies. These policies are simple ‘background’ retention policies that can be applied to the main Microsoft 365 workloads (Exchange, SharePoint, MS Teams, OneDrive) They delete content automatically but have no disposition review and no record is kept beyond the period of the audit logs (90 days).

These policies can be combined on individual SharePoint sites for maximum effect.

It is assumed, but cannot be confirmed, that these options will continue to exist for some time.

Conclusion

The records management solution is not a solution to manage records across Microsoft 365. It has a specific purpose that would be more accurately described as ‘advanced information governance’.

The solution offers organisations with (more expensive) E5 licences a way to automatically identify (classify) and manage certain types of records through to the end of their retention period. It is designed to address the high volumes of both low-level digital content (‘ROT’ in particular) and specific high-value records in organisations.

For organisations that have E3 licences, the alternative options offered from the Information Governance section of the Compliance may be sufficient for as long as these options remain available.

Managing emails as records in Office 365

Records managers have been struggling with managing emails as records ever since they first appeared in the workplace.

For a long time the accepted practice, as with other digital records, was to print them out and put them on the appropriate file. With the introduction of electronic document and records management (EDRM) systems, end users were instead required to save or copy documents and emails to an electronic ‘file’ in that system.

The old paradigm where emails were copied to an EDRMS or a paper file

In both cases, the emails remained in the user’s ‘personal’ mailbox, where they remained inaccessible for ‘privacy’ reasons. End-users and business areas would (and still do) conduct business via the email system, without these records being available to anyone except the sender and recipient/s. Attachments to emails sent to individual recipients were (and continue to be) not managed as records unless they were printed out or saved to the EDRMS.

Microsoft Office 365 has changed the paradigm for keeping records as described in the linked post, away from the central storage and management of records in one system (while leaving the originals in place), to the decentralised ‘in place’ storage and centralised management of records across Office 365.

This post provides an overview of the three main options for managing email as records in Office 365, in both Exchange and SharePoint.

In summary the options are:

• Leave emails in place in Exchange mailboxes (personal and Office 365 Group mailboxes) and apply one or more Office 365 retention policies to mailboxes.
• Same as previous point, and use Content Search to retrieve emails as required.
• Same as previous point, and only copy specific emails to SharePoint

Keep in mind while reading this post that chat content from MS Teams is also stored in Exchange mailboxes but that content cannot be copied to SharePoint.

Option 1 – Leave emails in place and apply retention policy

In this option, emails remained stored in personal or Office 365 Group mailboxes. End users may create folders and ‘categorise’ the content as they wish, but no additional attempt is made to further categorise, add metadata to, or group the content according to recordkeeping requirements. The aggregation, from a recordkeeping point of view, is the end-user or Office 365 Group.

All mailboxes are subject to one or more retention policies set in the Office 365 Compliance portal to ensure that no emails are deleted before a pre-defined minimum period.

Note that retention policies can effectively replace a back-up regime used by IT for disaster recovery and investigation purposes purposes.

Positives

Emails are aggregated by user name or Office 365 Group and will remain in mailboxes for a minimum period of time as set by the retention policy. 

Office 365 Group mailboxes provide the ability to group emails by a more specific subject  (the Group name, which could map to a business function – e.g., ‘Correspondence Management’) and have the added positive of having an associated SharePoint site.

Negatives

The negative with this option, from a recordkeeping point of view, is that all emails – regardless of subject or importance – are grouped by the ‘personal’ or Office 365 Group mailbox, and kept for the period defined in the retention policy. That is, there is no differentiation between (email) records that may need to be kept for a long period of time and those that are transient in nature.

If there is a requirement to ensure that certain emails are kept in different aggregations or for different periods of time, then option 3 should be considered.

Option 2 – Same as option 1 and use Content Search to retrieve emails

This option is the same as the first option, but the business can make use of Content Search to identify and isolate emails as required. Content Search is more or less the same as the search part of an e-Discovery case.

The Content Search option is accessed from the left hand navigation of the Compliance portal: https://compliance.microsoft.com/homepage

Note that access to the Content Search area is restricted to Office 365 Global Admins and Compliance Admins. This is because, as can be seen in the screenshot below, a Content Search can be set up to search for any content in email, documents and much more.

Content Searches can be set up from the ‘New Search’ option, or the Administrator can make use of a Guided search or Search by ID List. For the purpose of this email, only the ‘New search’ will be examined.

Configuring a new Content Search

Each content search can be configured against three main options as shown in the screenshot below: Keywords, Conditions, and Locations. Some searches may require a combination of these three options.

Keywords can be any words that may be found anywhere in the email, including the content.

The available conditions are listed below:

Common

• Date
• Sender/Author
• Size (in bytes)
• Subject/Title
• Compliance label

Emails

• Message kind
• Participants
• Type
• Received
• Recipients
• Sender
• Sent
• Subject
• To

Documents

• Author
• Title
• Created
• Last modified
• File type

The available search locations include any or all of the options below:

• Exchange email
• Office 365 group email
• Skype for Business
• Teams messages
• To-Do
• Sway
• Forms
• SharePoint sites
• OneDrive accounts
• Office 365 Group sites
• Teams sites
• Exchange public folders

For more detail on how to use Content Search and all the options available, go to this Microsoft site.

Running a search

After the search has been configured, it must be run. The speed of the search will depend on the complexity of the search, conditions, locations and the volume of content. Every search will appear in the list of searches that have been saved.

When complete, the search result will show a ‘Status’, showing the number of:

• Items found
• Unindexed items
• Mailboxes
• Sites
• Public folders

Exporting results

Once the search has completed, the results of the search may be exported. There are two  configurable options for exported results.

Output options:

• All items, excluding ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• All items, including ones that have unrecognized format, are encrypted, or weren’t indexed for other reasons
• Only items that have an unrecognized format, are encrypted, or weren’t indexed for other reasons

Exchange content export options:

• One PST file for each mailbox
• One PST file containing all messages
• One PST file containing all messages in a single folder
• Individual messages
• Enable de-duplication for Exchange content (check box)

Positives

Content searches are likely to find and retrieve more relevant emails than might be saved elsewhere, as it looks through all emails. Provided a retention policy has been applied to the mailboxes, the content should still be accessible. If the emails have been deleted at the end of a retention policy, they will not be accessible any more.

Emails can be exported and – if necessary – the PST copied to a different system (such as SharePoint) for long-term storage with additional metadata as required.

Negatives

Access to the Content Search option is restricted to Global Admins and Compliance admins, for good reason. Consideration might need to be given to governance or procedural rules. Note that Global Admins are always alerted when a new content search is created or run.

Each search must be pre-configured and run regularly to ensure that all emails are identified.

Content searches may retrieve too much unrelated content.

Option 3 – Same as option 2 and copy only select emails to SharePoint

This option mimics the legacy way of saving a record to a pre-defined separate aggregation, in this case to a SharePoint document library.

It differs from the first two options in that only certain select emails are copied (by end-users or using a third-party application) to specific SharePoint document libraries. It is still, however, possible (and preferable) to apply a retention policy to the original mailboxes.

Content search, which can be used at any time, will find the emails in both Exchange mailboxes and SharePoint as long as they have not been deleted via a retention policy expiry .

Positives

The positives with this option are that emails copied to a SharePoint document library:

• Are grouped with other related records. This may be important from an organisational recordkeeping point of view, for example for certain key records. Consideration might also be given to setting up an Office 365 Group instead for these specific records.
• Can have additional metadata.
• Can be retained for a period of time, different from the original mailbox.

Negatives

The problems with this option are that:

• It requires some kind of action to copy the email.
• It creates a copy of the email, it doesn’t remove the original.
• An email copied to another system may not be the most recent in a thread, especially if that thread is still active.
• Does not include the ‘chat’ elements from MS Teams.

Summing up the options

The idea of copying an email to a separate aggregation, container or file for recordkeeping purposes is a legacy concept inherited from the paper recordkeeping period. While attempts were made over the years to mimic that concept in EDRM systems, it has several weaknesses that mostly outweigh the alleged benefits.

Email (in Exchange) and documents (in SharePoint) continue to remain separate in Office 365 but there is now the potential to manage both equally through a combination of retention policies and pre-defined content searches.

The majority of business emails are never captured in separate recordkeeping systems. Microsoft’s centralised retention model and ability to apply to retrieve emails on the fly mean that it is more efficient and cost effective to leave emails in place. This does not exclude the potential to copy certain select emails to SharePoint.

Additionally, mailboxes associated with Office 365 Groups provide the ability to keep emails in a business context, away from inaccessible ‘personal’ email accounts. Records managers should consider the potential of using Office 365 Group mailboxes in this way for particular types of records.

More than metadata – information management professionals and data modelling

Most information management professionals (in the context of this post – records managers, information managers, librarians) are familiar with the use and application of metadata.

The metadata in their domain of work may:

• Form part of the built-in properties of digital records, and remain with it wherever it is stored, as part of its metadata ‘payload’. Commonly this is the title or name, date created, and creator/author.
• Record (usually additional) details about, or provide the context for an object when it is captured or registered in a system (‘point of capture’ metadata). This metadata may include classification terms or numbers, object types, access and security controls, storage location, and the container or aggregation.
• Record various actions and events through the life of the object (‘process metadata’), including when the record was accessed/used, modified or deleted/destroyed, and by whom.

This post discusses how the metadata and the metadata skills and related knowledge of information management professionals are closely related to a broader set of skills and knowledge including enterprise data modelling. Information management professionals might consider learning more about this subject as a career path.

How metadata relates to data modelling (theory) and business systems (actual)

What is data modelling?

Most of the definitions for the term ‘data modelling’ have the same three ‘layers’ (or variations), usually shown in the form of a pyramid:

• Conceptual: A (usually simple) model that shows all the high-level data entities and their relationships across an organisation. For example ‘Customer’, ‘Employee’, ‘Property’, ‘Organisation’. See below for an example.
• Logical: A more detailed model of each entity in the conceptual model, showing the multiple logical entities that exist for each conceptual entity, their attributes and associations (relationships). The attributes for each level 2 entity are more or less metadata fields. See below for an example.
• Physical: A database schema or framework for how data is actually stored in a database. Physical data models are usually very complex and are often regarded as Intellectual Property (IP) by product vendors. 

This post focuses only on the first two layers, conceptual and logical.

Elements that make up data models

Both the conceptual and logical data models are made up of entities, attributes and relationships/associations.

• Entities.
  • At the conceptual level, entities are the highest level groupings that are related to each other. For example:  ‘Services’ or ‘Products’, ‘Vendors’, ‘Employers’, ‘Property’, ‘Customers’, ‘Accounts’ or ‘Organisation’.
  • At the logical level, entities are the various data elements that make up the level 1 entity). One way to think of this would be to consider a data entry screen where all the details about an entity must be recorded. For example, what data would or should be captured about an Employee, or a Service? In complex organisations, second level logical models may have several hundred entities or may need be broken down into related sub-entities.
• Attributes.
  • At the conceptual level, attributes may simply be the logical layer entity names and a definition for each.
  • At the logical level, entity attributes define each of the entities. Depending on the complexity of the model, level 2 entities may be single entities (for example, ‘Employment Type’) or grouped (for example, ‘Personal details’ or ‘Contact details’); in these cases, the attributes become the equivalent of metadata or field names, e.g., ‘Surname’, ‘First name’, ‘Gender’, ‘Date of Birth’.
• Relationships.
  • At the conceptual level, relationships are usually relatively simple. For example, an employee ‘works in’ the organisation; the organisation ‘sells’ services or products to customers who ‘pay’ for them.
  • At the logical level, relationships are also relatively simple but there will be more of them as there are more entities. For example, an employee ‘has’ a position in the organisation, and ‘has’ a salary level. The employee ‘has’ personal details (e.g., name, date of birth).

Example Level 1 conceptual model

The following diagram is an example of a conceptual model, showing the high-level entities and the relationships between them. What are the level 1 entities in your organisation?

In many organisations, line of business systems can be mapped to each of these entities because that is where the data for those entities is stored. For example:

• Employee information may be managed in a Human Resources Information System
• Accounts or financial information may be managed in a Financial Management Information System.
• Client information may be managed in a Client/Customer Relationship System and other client-specific systems.
• Property information management may be managed in a Property Management System.

Unstructured data, in the form of records, relating to each of these entities, may be managed in a centralised document and records management system, network file shares, email, or other alternatives.

Example level 2 logical model

The following diagram is an example of an actual logical model for the entity ‘Employee’, showing the multiple entities and the relationships between them. If you have identified the level 1 entities in your organisation, what are the level 2 entities? 

Note the ‘Common’ entity. This was used because these terms are common to multiple level 1 entities.

If you plan to create data models, it is a good idea to use appropriate software; neither Visio nor PowerPoint are in any way suitable for level 2 data modelling. The level 2 model above was created using the software application ‘Enterprise Architect’ from Sparx Systems. This system allows multiple entities to be created independently and then brought together in a model as required, with relationships automatically indicated. In the model above, the level 1 entity precedes the level 2 entity name. There are three entities: Employee, Common (as these entities are common to multiple level 1 entities), and Organisation.  The attributes for each entity indicate whether they form part of a group (e.g., ‘Contact details’), or are metadata attributes in their own right (for example ‘Family Name’ or ‘Date of Birth’ in the ‘Personal Details’ entity, which is a grouping related to the ‘Employee Details’ entity. 

In the same organisation, the ‘Client’ level 2 entity data model contained several hundred entities and so several sub-entities (based on the ‘Client type’ entity) were created.

Data models and data dictionaries

The metadata attributes depicted for each entity in the level 2 model show only the ‘field’ name. For example the Personal Details entity includes the field names ‘Family Name’, ‘First Name’, ‘Gender’ etc. The data model does not normally provide further detail. 

Instead, details for all entities, including their attributes and associations, can be defined in data dictionaries. The following are examples of the information that should be defined for each level 2 entity attribute:

• Name (e.g., ‘Gender’)
• Description
• Data Type
  • Integer/Number
  • Text/String (‘free text’).
  • Date
  • Currency
  • Choice
  • Boolean (yes/no)
• Data Format
  • Alphanumeric
  • Numeric
  • Currency
  • Choice (and the actual choice options)
• Length
• Relationship (e.g., with other attributes)
• References

This type of detail and form should be familiar to most information management professionals. Data dictionaries can also include other more specific metadata entity details as well, including things like Function, Activity, Document Type and so on, even if these are not represented in actual data models.

The value of data models and data dictionaries

Conceptual and logical data models – and the data dictionaries that describe the details in these models – are essential information artifacts, especially in larger organisations with multiple business systems. They:

• provide an easy to understand, visual conceptual and logical overview of all the data across the organisation; 
• can be used in discussions with third-party database vendors, as they define an ideal objective against which acquisition decisions may be made; 
• help to understand why there are issues or problems with data quality. For example, a database may allow free text data for important data elements that then cannot be easily analysed;
• help to define issues with or support analytical and business intelligence outputs;
• show that the organisation is serious about managing data consistently and appropriately;
• have the potential to help reduce costs or increase efficiencies. For example, a critical database may allow for free text entry (which takes time and is prone to error) when a choice option would be far more efficient and accurate.

The pathway from metadata to data modelling

As noted at the beginning of this post, most information management professionals have the skills and knowledge required to manage metadata. Information management professionals can draw on these core skills to develop or refine data models for the organisation.

In the first instance, knowing if data models even exist would be a good step. Even if they already exist, a discussion with IT (or the relevant person responsible for managing the data models) could be a good idea.