A common question asked by many organisations is whether Microsoft 365 (M365) retention policies – labels in particular – can be applied to folders in SharePoint document libraries so the content in those folders will have the same label.
The quick answer is yes, but it is a manual process and – for all its perceived benefits – is likely to be more of an administrative and support burden and not worth the effort. Folders should NOT be thought of as the replacement for ‘files’ (aggregations of individual records), but more like dividers in a lever arch (= the document library).
This post describes how labels can be applied to and work with folders, including in SharePoint sites linked with Teams. It also suggests alternative options.
How retention labels are applied to a library
Retention labels are created in the M365 Compliance portal under either the Information Governance > Labels or Records Management > File Plan sections.
Where labels are created in the Information Governance section of the Compliance portal
Labels created in the Compliance portal do not do anything once created; they must be applied to content in various ways to make them work. This includes by:
Publishing one or more labels as part of a retention policy to various locations including SharePoint sites, Exchange mailboxes and OneDrive (but not Teams – see screenshot of available locations below). In this scenario, each label will be visible to – and selectable by – end-users.
Auto-applying them to the same locations based on various options, including (for E5) trainable classifiers
Adding them to Content Types used in SharePoint Syntex.
Locations where labels can be published
Publishing labels to SharePoint sites
When one or more labels are published to SharePoint sites they don’t do anything until they are ‘manually’ enabled through one of the following options:
On each individual document library via the library settings option ‘Apply labels to items in this list or library’ (see screenshot below)
On each individual folder in a library (via the information panel, see screenshot below)
On each individual object in the library (also via the information panel)
Applying a label to the library
A label can be applied to the entire document library via Library Settings, as shown in the screenshot below.
Note that the ‘None’ option is shorter if no labels have been published here
If the drop-down option is set to ‘None’, and there are no options to choose from, it means that no labels have been published to this SharePoint site.
If labels exist, they will appear in the drop down list (below the default ‘None’). Note that only one label can be set as the default for the library. If the check box ‘Apply label to existing items in the library’ is selected, this will apply the label to all existing items. It will also likely override any existing label that may have been applied.
When the retention label has been applied to a library, the label only applies to the non-folder objects stored in the library as can be seen in the option below. That is, the retention label is NOT applied to the folders by default.
Document library folders without retention labels
The retention label can be seen when the folder is opened:
Content inside a folder, with retention labels applied
Applying a retention label to a folder or document
It is also possible to apply a retention label to a folder or object stored in the folder via the information panel, even when a default library label has been set, as shown below. This can be done on each individual folder including, for Teams-based sites, each folder that maps to a channel.
When applied to a folder in this way, any content stored in the folder will inherit that retention label.
Documents stored inside the June folder have inherited the folder’s label
If a default label has already been applied at the library level, the folder-based label will replace it, although in testing this, one of the original default labels wasn’t replaced automatically as shown below, but could be manually changed via the information panel.
Implications for Teams-based records (Files)
Every Team in MS Teams has an associated SharePoint site linked with the underlying Microsoft 365 Group.
Every non-private channel in the Team maps to a folder in the Documents library of the SharePoint site as can be seen in the two screenshots below. (Every private channel has a separate SharePoint site that would be covered by a separate retention policy).
The Team’s channelsFour channel-linked folders in a Teams-based SharePoint site
Keep in mind that retention labels remove the ability to delete objects stored in the library (including via the ‘Files’ tab in a Team). If end-users are working in Teams, this could be annoying and potentially put them off using Teams. However, end-users can remove the label by navigating to the SharePoint site and removing the label via the Information panel.
Why folder-based retention labels may not be a good idea
The default options to apply retention labels to content stored in SharePoint document libraries are:
By applying them at the library level. This can apply the label to all existing (and future) content stored in the library but does not apply to folders.
Through the auto-application of labels.
Via SharePoint Syntex using labels on Content Types.
Applying retention labels to individual folders in a document library is a manual-intensive process, one that may be a waste of time given the potential number of libraries that can exist and the ease with which they can be removed by end users.
Additionally, applying retention labels to the channel-linked folders of Teams may be pointless if end users:
Store documents at the same level as the channel-linked folders; that is, ‘above’ the folder structure.
Create new folders via a synced library or SharePoint. These folders are not linked to channels.
Create new libraries in the SharePoint site.
Keep it simple
It is very easy to deliberately or inadvertently establish over-complicated retention settings for content stored in SharePoint, especially as there is currently no simple way to see what label has been applied where.
Given the retention period linked with retention policies generally, there is a good chance that the person who applied the labels may not be around when the retention period expires, or to keep an eye on what has been applied or changed over time.
The best retention intentions may be overruled by practical necessity.
The best retention model, in my opinion, is a simple one that does not get in the way of end-users but ensures that records will be kept for a minimum period required. So, instead of applying retention labels to folders, especially on Teams-based SharePoint site libraries, it is recommended to:
Start by trying to avoid mixing content with different retention periods in the same SharePoint site or Team, or document library. That will make it easier to manage the retention outcomes. (If you can’t avoid mixing content, you may need to use auto-application of labels including via Syntex or trainable classifiers).
Use ‘back-end’ safety net retention policies applied to all SharePoint sites. This ensures a minimum retention period and does not get in the way of end-user activities.
Use retention labels on site libraries where more granular retention is required. Ideally, apply them as the default to all the content in a single document library (including the default library for all Teams-based SharePoint sites) and – preferably – only apply the labels when the content is inactive and the library can be made read only, to protect the records from that point.
Only use multiple labels on folders when (a) all the labels applied to the site relate to the same function/activity pair or subject matter, and (b) the content is largely inactive. Ideally, avoid folder-based retention to avoid complication in the future.
Microsoft 365 has become one of the world’s most accessed products for office collaboration. Jeff Teper, the ‘father of SharePoint’ at Microsoft, tweeted on 27 April 2021 that Teams had 145 million daily active users. (As reported in by the team at Office365ITPros.com.) According to the website ‘Microsoft Office Statistics and Facts (2021) | By the Numbers‘ , Microsoft Teams usage grew 40% during the COVID lockdowns.
Although organisations create, capture and store a range of records using the various Microsoft 365 applications, most records are likely to be created or captured in Exchange mailboxes or SharePoint (including OneDrive).
The volume and range of records has, in many respects, overwhelmed traditional standards-based models that required records (including emails) to be copied to a central electronic document and records management system (EDRMS) or identified and then ‘declared’ as records.
Given the reality of the new paradigm, organisations have tried various ways to manage records in Microsoft 365, including by retaining the EDRMS (for high value records), acquiring third-party products that promise to address the ‘gap’ in recordkeeping functionality, or working with the ‘out of the box’ capability.
Whichever method is chose, records managers need to have a very good understanding of where the records are in Microsoft 365 and how they can be managed. In many cases, leaving and managing them where they were created or captured (‘in place’ management), and using new and emerging capability to leverage the power of AI-based tools is likely to be the future state.
With the above in mind, and regardless of which method you follow, the following are ten points that I think are important to consider when managing records in Microsoft 365.
What are your recordkeeping obligations?
It is perhaps the most obvious question but organisations have sometimes rolled out Microsoft 365 without consideration of their obligations for managing records.
Records provide evidence of business activities and accordingly need to be protected to ensure their authenticity, integrity and reliability as evidence. The most common way of achieving this outcome until now has been to ‘lock’ digital records from change. Is this practical in the digital world? How do you lock Teams chats or stop a new thread in an email exchange? Could the same outcome be achieved by recording all changes and ensuring these are retained?
Records are usually subject to minimum retention requirements and understanding what these are is essential. Where there are minimum retention requirements, records cannot be destroyed before a specific period of time based on a particular trigger or event. These requirements are defined in legislation (sometimes based on statutes of limitation) or, for government agencies, in records disposal authorities or schedules (as shown in the example above).
As a starting point, look at these retention requirements and consider how these will be applied to Teams 1:1 chats, or team chats, or emails still in Exchange mailboxes, or OneDrive content. And then extend this to the content stored in Teams/Group-based SharePoint sites and sites not linked with Microsoft 365 Groups.
Consider also what is required to manage the outcome of retention. Do you need to review records due for destruction? Do you need to keep a record of what was destroyed? For all records?
There may be a requirement to categorise or classify records (especially to group them by context and/or for retention purposes where retention is based on classification). How will this outcome be achieved for Teams chats or emails that remain in Exchange mailboxes, or OneDrive content? What other metadata do you need for records?
Your recordkeeping obligations, in particular records retention requirements, should guide the management of records in Microsoft 365.
Where are the records created or captured in Microsoft 365?
Neither Microsoft 365 nor SharePoint is a dedicated recordkeeping system like an EDRMS (see my post ‘SharePoint is not an EDRMS’). Rather, it is an ecosystem of multiple applications that are used to create, capture, store and manage records.
Most records are likely to be stored in either SharePoint (OneDrive is a SharePoint service) or Exchange mailboxes.
A compliance copy of Teams chats are stored in a hidden folder of Exchange mailboxes. Content stored in the ‘Files’ tab is either stored in SharePoint or (for 1:1 chat) in OneDrive.
Of course, there will be some other records – Yammer conversations, tasks and plans, communication sites, calendar entries, forms and even WhiteBoard sessions. But, more than 95% are stored in Exchange mailboxes or SharePoint/OneDrive.
Knowing your recordkeeping obligations and the location of records are the main starting points. In fact, you can map your recordkeeping obligations (especially metadata and retention) to the location of the records.
Do you control the creation of Teams and SharePoint sites or not?
There two, broadly speaking, two approaches to controlling the creation of Teams and SharePoint sites:
Yes, we have controls – There is some kind of control or decision ‘gate’ for the creation of Teams and SharePoint sites.
No, we don’t have controls – End-users can create Teams and SharePoint sites whenever they want. In this case, the points below may not be of much use. You will likely rely on the built-in ‘records management’ capability to manage the records.
If your answer to the question above was ‘No’, then you will probably need third-party products and/or rely heavily on AI-based solutions to manage the records (which is the default Microsoft position).
Map sites and Teams to business functions – don’t mix content
Almost every organisation has a range of business functions. Some of these are common to all or most organisations (e.g., information technology, human resources, financial management, legal, property, etc) while others are ‘core’ (e.g., engineering, manufacturing, research, sales and marketing, etc).
Many organisations are structured around these business functions, and most records retention schedules are based on function (or business function).
If you can map new Teams and SharePoint sites to these functions, this will facilitate the management of records down the track. Mixing content across multiple functions – except where it makes sense to do this, such where there are related but smaller numbers of records in project sites – is going to make it harder to manage the records in the longer term – and more or less the same as letting end-users put whatever they want into a paper box for long-term storage.
A common example where records might be mixed are ‘Corporate Services’ areas that create or capture records across multiple functions, including property, IT, finance and so on. Unless all the records in a Team-based site can be kept for the same period of time, it may be a good idea to separate the records into different sites.
Also keep in mind that some business areas may exist for a long time; having a single (Teams-based) document library that has folders linked to channels may not be the best way to manage records long-term.
The suggestions above don’t take into account Exchange mailboxes, Teams chats or OneDrive accounts because these cannot be mapped to functions.
Naming conventions for sites and teams, and libraries
The main reasons for having naming conventions for SharePoint sites and Teams are:
It is good practice, to avoid acronyms and other less than useful names.
To prevent unnecessarily long names that end up creating very long URLs (e.g., ‘https://tenantname.sharepoint.com/sites/ExecutiveCommittee20202021MeetingsHeldDuringLockdownandrecordedviaMSTeamsSeniorManagersOnly‘.) It is important to know the difference between the URL name and the display name.
Ideally, the original names of Teams and SharePoint sites should be restricted to no more than 14 characters so that Document IDs (that have a 12 character prefix) can be the same as, or very close to, the URL name of the site. For example:
Aside from the default ‘Documents’ library of every Teams-based site, library names should also be subject to naming conventions and restricted to around 20 characters. There are several reasons for this.
The first is how they appear in the left hand navigation of a SharePoint site. There isn’t much point having multiple library names that aren’t easily visible (the two examples below have completely different names after ‘Financial Management’).
The third reason is that it is good practice to have some form of naming conventions.
Ideally, library names should map to the activities that produce the records AND include the year where this is relevant, e.g., ‘Meetings2021’.
There is NO need to repeat words in the tenant or site name – e.g., h**ps://tenantname.sharepoint.com/sites/TenantNameCorporateRecords/TenantNameFinancial%20Management%20%20Accounting%20%20Invoices%202021/Forms/AllItems.aspx
As noted above, this doesn’t apply to the default ‘Documents’ library in Teams-based SharePoint sites (the actual name is ‘Shared%20Documents’).
Metadata and Content Types
For many organisations, the minimum metadata requirements consist of (a) agent (e.g., the person who did something), (b) dates (when they did something) and (c) a unique identifier. That is, who did what and when?
If you need to add more metadata for certain types of records you can really only do this in SharePoint document libraries, including by adding them from the SharePoint Term Store (see below example). It can also be done in Outlook but these metadata terms are not linked with the SharePoint terms.
As for Content Types, do you really need them? SharePoint is made up of multiple Content Types already, including the default ‘Document’ Content Type. It is important to understand how Content Types work in SharePoint before making the assumption that they are required.
In many cases, choice metadata fields can replace the need for Content Types. Custom Content Types may only be needed for specific or high value record types.
Document retention policies and labels
In the first section about recordkeeping obligations, it was noted that most records will be subject to minimum retention requirements. Retention labels and policies are created in the Compliance admin portal of Microsoft 365.
Unfortunately, the current Compliance admin portal provides very little information to show what label or policy was applied where. The only way to do this is to document it yourself. One way to do this is to create a spreadsheet that lists on each row:
The business function and activity from a File Plan or Business Classification Scheme (e.g., Financial Management – Accounting)
Each retention class for that function/activity pair, including the reference number
If that class has been created as a label, what the label name is. If it has been created as a non-label retention policy, what that retention policy name is. (Generally speaking, disposal authority classes don’t refer to Exchange mailboxes, SharePoint sites, MS Teams chats or OneDrive content, so the organisation may need to determine what this minimum retention period should be and how it will manage the retention outcomes).
(Note, the above can be created in the File Plan section of the Records Management part of the Compliance admin portal, E5 licences only. However, it only documents the above information and does not show where the label has been applied.)
Where the label has been applied ‘manually’ – to which SharePoint site/document library, Exchange mailbox or OneDrive account. This point may have multiple location references.
Where the label has been auto-applied through the basic E3 option or, for E5 licences, the document understanding model (DUM) in SharePoint Syntex, or via trainable classifiers.
When the retention will expire.
Retention outcome – If a disposition review (E5 only) exists, the records will be destroyed automatically without any record kept, or ‘do nothing’. See also below.
Remember that retention labels and policies apply to individual items (emails, Teams chat, SharePoint or OneDrive content stored in libraries), not to aggregations (e.g., the entire library or site). The aggregation will continue to exist after the content has been destroyed and no ‘stub’ (a record of what used to exist) will remain.
How will you manage retention outcomes?
Generally speaking, Microsoft 365 retention policies destroy records when they are due for destruction unless they are subject to a label that has the disposition review option enabled or the ‘do nothing’ option has been selected.
Organisations need to understand how they will manage these retention outcomes especially as, in most cases, a review process is required. (See ‘Recordkeeping obligations’ above).
Even when retention label have the disposition review option enabled (E5 only), there are two points that need to be understood:
The ‘disposition review’ interface presents individual records with no context except for the original site URL name. Some additional (default) metadata is now included (from May 2021) but not any added metadata. In most cases, it will be necessary to return to the original library to view the context of the records presented for disposal, and if there are any others.
If records are destroyed through that review process, only basic metadata is retained about what was destroyed.
Organisations that have an obligation to undertake a full review of records due for disposal will likely need to consider establishing workarounds such as exporting the full set of metadata from a document library and then using that to review whether the content of the library can be destroyed. If approval is granted, that decision should be recorded, along with the metadata extract.
Allow end-users to get on with their work
End-users generally don’t have much interest in the management of records beyond the period of time they are important to them. They want to do whatever they want, whenever they want, using the applications they have available to them.
Collaboration no longer consists of email exchanges and document-based records. Creating control gates for the creation of Teams and sites, and insisting on naming conventions for sites and libraries (and folders) may be interpreted negatively.
There needs to be a fine balance between control and freedom and this can impact the creation, capture and management of records. Some of the ways to minimise the impact of recordkeeping requirements include:
Enabling Document IDs on every site.
Creating custom metadata columns on sites or libraries with default values.
Applying non-label ‘safety net’ retention policies to all workloads. Retention policies (along with the Recycle Bin for 90 days) helps with the recovery of accidentally deleted content.
Using various communication methods to highlight useful features including sharing (instead of attaching), the Recycle Bin, versioning in SharePoint/OneDrive, and the ability to have a ‘single source of truth’. These features can be used to ‘soften’ the impact of other recordkeeping obligations in some sites.
Pro-actively monitoring activity across the Microsoft 365 ecosystem, including by monitoring the various dashboards, searches, and audit logs, and responding to events.
Learning more about the Microsoft Graph Explorer and the potential to use AI-based options to manage records.
Use the system for other recordkeeping purposes
The Microsoft 365 environment can be used for other recordkeeping purposes as well. For example:
Managing physical records stored offsite in a SharePoint list.
Keeping a record of records (and SharePoint sites or other systems) that have been destroyed, as well as ongoing destruction review and approval processes.
Publishing policies and procedures (in a SharePoint site, not necessarily a communication site).
Communicating information about managing records (communication site).
Archiving social media content (to Exchange mailboxes).
Searching for content stored in other locations or systems including File Explorer and Line of Business systems (via connectors).
Archiving network file share content, where it can be better protected and then subject to retention and disposal outcomes.
Understanding where records are stored (via dashboards and Power BI reporting).
One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.
Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.
Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:
Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.
Decision tree
With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.
The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:
The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.
Note: the box outline colours represent the four different types of site that are likely to be created.
How will the content be accessed?
It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.
Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.
They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.
If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.
It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).
What type of SharePoint site will be required?
Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.
As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.
Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.
Creation method
The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.
Security Controls
Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.
Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.
Retention
Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.
As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.
Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.
Provisioning
Once the site has been created, it should be provisioned with three key elements shown below.
Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.
The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.
There are three main options in Microsoft 365 to apply recordkeeping classification terms to (some) records:
Metadata columns added to SharePoint sites, including those added to Content Types and/or added directly to document libraries.
Taxonomy terms stored in the central Term Store, including those added as site columns and added to site content types and/or added directly to document libraries. The only difference with the first option is that with the Term Store the classification terms are stored and managed centrally and are therefore available to every SharePoint site.
Retention labels that: (a) ‘map’ to classification terms; (b) are linked with a File Plan that includes the classification terms; (c) are either the same as (a) or (b) and are used in with a Document Understanding Model in SharePoint Syntex; or (d) the same as (a) or (b) and used with conjunction with Trainable Classifiers.
The first two options can only be applied to content stored in SharePoint. Retention labels may be applied to emails and OneDrive content. None of the three options can be applied to Teams chats. Also note that there is no connection between the SharePoint Term Store and the File Plan, both of which can be used to store classification terms.
This post:
Defines the meaning of classification from a recordkeeping point of view.
Describes each of the above options and their limits.
Discusses the requirement to classify records and other options in Microsoft 365.
What is classification?
Humans are natural-born classifiers. We see it in the way we store cutlery or linen, or other household items or personal records.
Business records also need some form of classification. But what does that mean? The 2002 version of the records management standard ISO 15489, defines classification as:
‘the systematic identification and arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules represented in a classification system’. (ISO 15489.1 2017 clause 3.5).
The standard also states (4.2.1) that a classification scheme based on business activities, along with a records disposition authority and a security and access classification scheme, were the principal instruments used in records management operations.
The classification of records in business is important to establish their context and help finding them.
Microsoft 365 includes various options to apply classification terms to records.
Metadata columns in SharePoint
The simplest way to classify records stored in SharePoint document libraries is to either create site columns containing the classification terms and add those columns to document libraries, or create them directly in those libraries.
Benefits
Adding site or library columns is relatively simple. As classification terms are usually in the form of a (hierarchical) list, it is simple to add one choice or lookup column for function and another for activities.
A lookup column can bring across a value from another column when an item is selected; for example, if the look up list places ‘Accounting’ (Activity) in the same list row as ‘Financial Management’, selection of ‘Accounting’ will bring across ‘Financial Management’ as a separate (linked) column.
Default values (or even one value) can be set meaning that records added to a library (that only contains records with those classification terms) can be assigned the same classification terms each time without user intervention.
Negatives
SharePoint choice or lookup columns do not allow for hierarchical views or values to be displayed from the list view so the context for the classification terms may not be obvious unless both function and activity are listed.
The Term Store
The Term Store, also known as the Managed Metadata Service (MMS) has existed in SharePoint as a option to create and centrally manage classification and taxonomy terms in SharePoint only for at least a decade.
The location of the Term Store in the SharePoint admin center
Organisations can create multiple sets of taxonomies or ‘term groups’ (e.g, ‘BCS’ or ‘People’) within the Term Store. Each Term Group consists of the following:
Term Sets. These generally could map to a business function. Each Term Set has a name and description, and four tabs with the following information: (a) General: Owner, Stakeholders, Contact, Unique ID (GUID); (b) Usage settings: Submission policy, Available for tagging, Sort order; (c) Navigation: Use term set for site navigation or faceted navigation – both disabled by default; (d) Advanced: Translation options, custom properties.
Terms. These generally could map to an activity. Each Term has a name and three tabs: (a) General: Language, translation, synonyms and description; (b) Usage settings: Available for tagging, Member of (Term Set), Unique ID (GUID); (c) Advanced: Shared custom properties, Local custom properties.
In the example below, the Term Set (function) of ‘Community Relations’ has three Terms (activities).
Once they have been created in the Term Store, term set or terms can be added to a SharePoint site, either as a new site or local library/list column, as shown in the two screenshots below:
First, create a new column and select Managed MetadataThen scroll down to Term Set Settings and choose the term set to be used.
Once added as a site column, the new column may be added to a Content Type that is added to a library, or directly on the library or list.
A Term Store-based column added to a library via a Content Type.
Benefits
The primary benefit using the central Term Store terms via a Managed Metadata column is that the Term Store is the ‘master’ classification scheme providing consistency in classification terms for all SharePoint sites.
As we will see below, Term Store terms may be used to help with the application of retention labels (which themselves may ‘map’ to classification terms in a function/activity-based retention schedule).
Negatives
Using metadata terms from the Term Store is almost identical with using a choice or lookup column. The only real difference is that the Term Store provides a ‘master’ and consistent list of classification (and other) terms.
Term store classification terms, including in Content Types, may only be used on a minority of SharePoint sites.
Additionally:
It is not possible to select a Term Set (e.g., the function level), only a Term within a Term Set.
Only the selected classification Term appears in the library metadata, without the parent Term Set or visual hierarchy reference to that Term Set – see screenshot below. Technically only that Term is searchable. It is not possible to view a global listing of all records classified according to function and activity.
If multiple choices are allowed, a record may be classified according to more than one Term. This may cause issues with grouping, sorting or filtering the content of a library in views.
How the Term appears in the library column.
As we will see below, there is no connection between the classification Terms in Term Sets and the categorisation options available when creating new retention labels via a File Plan. ‘Business Function’ or ‘Category’ choices in the File Plan do not connect with the Term Store.
Term Store terms and Content Types can only be used to classify content stored in SharePoint.
Retention labels
Retention labels in Microsoft 365 can be used in an indirect way to classify records in SharePoint, email and OneDrive because they can be ‘mapped’ to classification elements.
For example, a label may be based on the following elements:
Function: Financial Management
Activity: Accounting
Description: Accounting records
Retention: 7 years
Every retention label contains the following options:
Name. The name can provide simple details of the classification, for example: ‘Financial Management Accounting – 7 years’.
Description for users. This can be the full wording of the retention class.
Description for admins. This can contain details of how to apply or interpret the class, if required.
Retention settings (e.g., 7 years after date created/modified or label applied).
Benefits
Where the classification terms map to a retention class, the process of applying a retention label to an individual record, email or OneDrive content could potentially be seen as classifying those records against the classification scheme.
The Data Classification section in the Microsoft 365 Compliance portal provides an overview of the volume of records in SharePoint, OneDrive or Exchange that have a specific retention class:
Negatives
Not every record in every SharePoint document library may be subject to a retention label. Many records (for example in Teams-based SharePoint sites) may be subject to a ‘back end’ retention policy applied to the entire site (which creates a Preservation Hold library).
A retention label applied to a record doesn’t actually add any classification terms to the record.
Retention labels don’t map in any way to Term Store classification terms, except in SharePoint Syntex – see below (but this only applies to SharePoint content).
Retention labels/File Plan combination
The File Plan option (Records Management > File Plan, requires E5 licences) can also be used to add classification terms to a retention label as shown in the screenshot below. Note that there is no link with the Term Store.
Benefits
Records (including emails) that have been assigned a retention label could, in theory, be regarded as having been classified in this way because the label contains (or references) the classification terms.
Negatives
When applied to content in SharePoint, OneDrive or Exchange, retention labels linked with the File Plan do not show the File Plan classification terms. It may be possible to write a script that displays all records with the terms from the File Plan, but it may be easier to do this using the Data Classification option described above.
Retention labels/SharePoint Syntex combination
SharePoint Syntex provides a way to apply retention labels to records, stored in SharePoint, that have been identified through the Document Understanding Model process.
Benefits
As can be seen in the screenshot above, each new DU model allows similar types of records (in the example above, ‘Statements of Work’) to be associated with a new or existing Content Type that can include a Term Store Term – for SharePoint records only – and a retention label. This provides three types of ‘classification’:
Grouping by record type (e.g., Statement of Work, Invoice)
Linking (of sorts) between the records ‘classified’ in this way and a Term Store term added as a metadata column to the Content Type.
Assigning of a retention label. This provides the same form of retention label-based classification described above.
Furthermore, if the Extraction option is also used, data extracted to SharePoint columns can be based on choices listed in the Term Store metadata.
Negatives
SharePoint Syntex only works for records – and only those records that have some form of consistency – stored in SharePoint.
Trainable classifiers are another way that could be used to identify related records and apply a retention label to those records. Microsoft 365 includes six ‘out of the box’ trainable classifiers that will not be of much value to records managers for the classification of records:
Source code
Harassment
Profanity
Threat
Resume
Offensive language (to be deprecated)
The creation of new trainable classifiers requires an E5 licence; they are created through the Data Classification area of the Microsoft 365 Compliance admin portal. Machine Learning is used to identify related records to create the trainable classifiers.
The option to auto-apply a label based on a trainable classifier.
Benefits
The primary outcome (from a recordkeeping classification point of view) of using trainable classifiers is the application of a retention label to content stored in SharePoint and Exchange mailboxes. It can also be used to apply a sensitivity label to that content.
Negatives
It is unlikely that every record will be classified according to every classification option.
Trainable classifiers only work with SharePoint and Exchange mailboxes.
Classifying records per workload
The options are summarised below for each main workload:
SharePoint: Use local site or library columns, Term Store terms or retention labels (mapped to a File Plan as necessary), applied manually or automatically, including via SharePoint Syntex or trainable classifiers.
Exchange mailboxes: The only feasible option to classify these records is to manually or auto-apply retention labels that are mapped to a classification, including a trainable classifer.
OneDrive: Manually or auto-apply retention labels mapped to a classification.
Teams. It is not possible to classify Teams chats with the options available.
Is classification necessary?
The classification model described in ISO 15489 and other standards was based on the idea that records would be stored in a central recordkeeping system where they would be subject to and tagged by the terms contained a classification scheme, often applied at the aggregation level (e.g., a file).
Microsoft 365 is not a recordkeeping system but a collection of multiple applications that may create or capture records, primarily in Exchange mailboxes, SharePoint, OneDrive and MS Teams (and also Yammer).
There is no central option to classify records in the recordkeeping sense. The closest options are:
The grouping of records in SharePoint sites (and Teams, each of which has a SharePoint site) and libraries that map to business functions and activities.
The use of metadata, either terms set in the central Term Store or created in local sites/libraries, to ‘classify’ individual records (including emails) stored in SharePoint document libraries. Each item in the library might have a default classification, or could be classified differently.
The use of retention labels that ‘map’ to function/activity pairs in a records disposal authority/schedule. These may be applied, manually or automatically, to content stored in SharePoint, OneDrive and Exchange mailboxes.
Neither of the above may apply, or be applied consistently, to all SharePoint sites, Exchange mailboxes, OneDrive accounts. And neither can be applied to Teams chats.
A different approach to this problem is required, one that likely will likely involve greater use of Artificial Intelligence (AI) and Machine Learning (ML) methods to identify and enable the grouping of records, and provide visualisations of the records so-classified.
Image: Werribee Mansion, Victoria, Australia stairwell (Andrew Warland photo)
A recent ‘functional and efficiency’ review into the National Archives of Australia (also known as the ‘Tune Review’, published on 30 January 2021) noted the ‘rapid and ever-evolving challenges of the digital world’.
It stated that ‘the definition of a ‘record’ needs to reflect current international standards, be more directly applied to digital technologies, and more clearly provide for direct capture of records that are susceptible to deletion, such as emails, texts or online messages’.
The review also highlighted the difficulties associated with ingesting digital records ‘via manual intensive activities (due to lack of interoperable systems)’ and proposed a new model based on the ‘continuous automated appraisal of [Agency] digital records that would require a combination of artificial intelligence and skilled archivists’.
The review underlined the challenges of identifying and managing born-digital records, and the need for better solutions.
This post explores the challenges of accurately and identifying born-digital records in order to manage them.
Identifying and protecting records
Records usually provide evidence of something that happened – an action, an activity or process, a decision, or a current state (including a photograph or video record). They may have or be associated with descriptive metadata used to provide context to the records and guide or determine retention.
Like all other types of evidence, the authenticity, integrity and reliability should be protected for as long as they must be kept.
In the paper world, this outcome was achieved by storing physical records (including the printed version of born-digital records) on paper files or in physical storage spaces.
For the past twenty years or so, this outcome was achieved for (some) digital records by (mostly manually) copying them from a network drive or email system (or via a connector) to a dedicated electronic records management (ERM) system and then ‘locking’ them in that system to prevent unauthorised change or deletion. Most ERM systems consisted of a database for the metadata and an associated network drive file store for the objects.
The main problem with this centralised storage model – however good it might be at protecting copies of records stored in it – was that the original versions, along with all the other records that were not identified or could not be copied to the ERMS, remained where they were created or captured.
And the records stored ‘in’ the ERMS were actually stored on a network file share on a server that was (a) accessible to IT, and (b) almost always backed up. So, yet more copies existed.
The challenge of born-digital records
There are several key challenges with born-digital records:
Consistently and accurately identifying (or ‘declaring’) all records in all formats created or captured in all locations. For too long, the focus has primarily been on emails and anything that can be saved to a network drive with the onus of identifying a record on end-users.
Ensuring their authenticity, reliability and integrity over time. For records stored in the ERMS, this has usually involved locking them from edit, including through the ‘declaration’ process, or preventing deletion. But in almost all cases, the original version (in email, on the network drives), could continue to be modified. Other records that were not identified or stored in an ERMS may be deleted.
Ensuring that born-digital records will remain accessible for as long as they are required.
It is not possible to consistently and accurately manually (or even automatically) identify every born-digital record that an organisation creates or captures to ensure their authenticity, reliability, integrity or accessibility over time. Only a small percentage of born-digital records are copied to an ERMS.
Records remain hidden in personal mailboxes, personal drives and third-party (often unauthorised) systems. Records may exist in multiple forms and formats, sometimes created or stored in ‘private’ systems or on social media platforms. They may take the form of text or instant messages or social networking posts and threads. They may be drawings, images, voice or video recordings.
Even if a record is identified, it is not always possible to save it to an ERMS. Text or instant messages on mobile devices are a case in point that has been a problem for at least two decades. More recent examples include chat messages, reactions (emojis, comments), and recordings of online meetings.
And even if a high percentage of born-digital records could be stored in the ERMS, the original versions will almost always remain where they were created or captured.
A different approach is needed.
Triaging records?
One approach to the problem would be to accept that not all records have equal value. That is, not all records need to be managed the same way.
To some degree, this way of thinking is already reflected in classes in the structure of records retention schedules and the attention paid to each:
Records that have permanent or archival value and need to be transferred to archival institutions.
Specific types of records that must be created or kept by the organisation for a minimum periods (sometimes quite long but not ‘forever’), for legal, compliance or auditing purposes.
Records that are not subject to legal or compliance requirements but which the organisation decides to keep for a minimum period of time.
Everything else.
Triaging records means that they can be managed as required at each level, but nothing is missed. It requires a risk management approach.
For records of permanent value, or are subject to legal or compliance requirements, it means that ensuring that these records receive the most attention and every effort it made to ensure that they are and can be identified (declared) and managed accordingly. This would include ensuring that it is possible to identify and capture these records in the systems used to create or capture them, for example, key emails.
A similar approach would be taken to records that need to be kept for legal, compliance or auditing purposes but with an understanding that some of these records (e.g., emails) may remain in the original system where they were created or captured. Technological solutions may be used to identify or tag these records. The destruction of these records should be subject to some form of review and a record kept of the approval and what was destroyed.
For all other records would remain stored wherever they were created or captured and subject to minimum retention periods after which they can be destroyed without review – but a record kept of the basic metadata of each record (including original storage location).
Protecting – or proving – the authenticity, integrity and reliability of records
The assumption behind the protection of records is that they should not be changed or deleted.
The reality, with digital records, is that they may change at any time through new threads, new revisions, new chats, or even through photoshopping.
A more realistic approach may be to use information about what was changed, by whom, and when – not to protect the record but to provide an evidentiary trail to prove what it is or was. The ‘smoking gun’ evidence for most born-digital records is the metadata that is recorded when it was captured or modified, not (necessarily) the added descriptive metadata.
For example:
Someone may author a document (metadata records each revision, and each revision can be viewed).
The document may be approved electronically (recorded in metadata).
Someone then modifies the approved version.
All of the above is recorded in the ‘modified’, ‘modified by’ and approval metadata.
The record should (or may) also recorded who viewed the record, and when.
EXIF metadata stored on images provides a similar form of evidence (and may even include GPS information).
Which record is more likely to be accepted as evidence:
A record stored in an EDRMS, versions or revisions of which may exist in multiple other places, including on network file shares, email system and even backup tapes
A record stored in a system that shows the full set of metadata about access and changes, or the most recent thread of an email discussion?
Conclusions
At the end of the day, it should be possible to confirm the authenticity, reliability and integrity of records based on information/metadata that forms part of the born-digital record: who and when it was created, the context in which it was created and its relationship with other records.
Perhaps, instead of focussing on trying to identify and capture all born-digital objects that might be records and ‘protecting’ a version of that record, it may be more practical and easier to leave most records where they were created or captured (and retained by retention policies) and use change or revision metadata to provide evidence of authenticity.
This may, in the end, be a much easier way to protect the authenticity of records than having to rely on manual identification or declaration.
In late 2012, IDM magazine published an article I co-authored with Umi Asma Mokhtar in Malaysia titled ‘Can technology classify records better than a human?’
The article drew on research into recent advances in technology to assist in legal discovery, known as ‘computer-assisted coding’, or ‘predictive coding’, including the following two articles:
An article titled ‘Search, Forward‘ by Andrew Peck, then a United States magistrate judge published in Law Technology News in October 2011. Peck’s article made reference to ‘predictive coding’.
Grossman and Cormack’s article noted that ‘a technology-assisted review process involves the interplay of humans and computers to identify the documents in a collection that are responsive to a production request, or to identify those documents that should be withheld on the basis of privilege‘. By contrast, an ‘exhaustive manual review’ required ‘one or more humans to examine each and every document in the collection, and to code them as response (or privileged) or not‘.
The article noted, somewhat gently, that ‘relevant literature suggests that manual review is far from perfect’.
Peck’s article contained similar conclusions. He also noted how computer-based coding was based on a initial ‘seed set’ of documents identified by a human; the computer then identified the properties of those documents and used that to code other similar documents. ‘As the senior reviewer continues to code more sample documents, the computer predicts the reviewer’s coding‘ (hence predictive coding).
By 2011, this new technology was challenging old methods of manual review and classification. Despite some scepticism and slow uptake (for example, see this 2015 IDM article ‘Predictive Coding – What happened to the next big thing?‘), by 2021, it had become an accepted option to support discovery, sometimes involving offshore processing for high volumes of content.
‘… applies machine learning … enabling users to explore large, unstructured sets of data and quickly find what is relevant. It uses advanced text analytics to perform multi-dimensional analyses of data collections, intelligently sorting documents into themes, grouping near-duplicates, isolating unique data, and helping users quickly identify the documents they need. As part of this process, users train the system to identify documents relevant to a particular subject, such as a legal case or investigation. This iterative process is more accurate and cost effective than keyword searches and manual review of vast quantities of documents.’Â
It added that the product would be deployed in Office 365.
Classifying records
The concept of classification for records was defined in paragraph 7.3 of part 1 of the Australian Standard (AS) 4390, released in 1996. The standard defined classification as:
‘… the process of devising and applying schemes based on the business activities generating records, whereby they are categorised in systematic and consistent ways to facilitate their capture, retrieval, maintenance and disposal. Classification includes the determination of naming conventions, user permissions and security restrictions on records’.
The definition provided a number of examples of how the classification of business activities could act as a ‘powerful tool to assist in many of the processes involved in the management of records, resulting from those activities’. This included ‘determining appropriate retention periods for records’.
The only problem with the concept was the assumption that all records could be classified in this way, in a singular recordkeeping system. Unless they were copied to that system, emails largely escaped classification.
Fast forward to 2020
Managing all digital records according to recordkeeping standards has always been a problem. Electronic records management (ERM) systems managed the records that were copied into them, but a much higher percentage remained outside its control – in email systems, network files shares and, increasingly over the past 10 years, created and captured on host of alternative systems including third-party and social media platforms.
By the end of 2019, Microsoft had built a comprehensive single ecosystem to create, capture and manage digital content, including most of the records that would have been previously consigned to an ERMS. And then COVID appeared and working from home become common. All of a sudden (almost), it had to be possible to work online. Online meeting and collaboration systems such as Microsoft Teams took off, usually in parallel with email. Anything that required a VPN to access became a problem.
2021 – Automated classification for records (maybe)
The Microsoft 365 ecosystem generated a huge volume of new content scattered across four main workloads – Exchange/Outlook, SharePoint, OneDrive and Teams. A few other systems such as Yammer also added to the mix.
Most of this information was not subject to any form of classification in the recordkeeping sense. The Microsoft 365 platform included the ability to apply retention policies to content but there was a disconnect between classification and retention.
Microsoft announced Project Cortex at Ignite in 2019. According to the announcement, Project Cortex:
Uses advanced AI to deliver insights and expertise in the apps that are used every day, to harness collective knowledge and to empower people and teams to learn, upskill and innovate faster.
Uses AI to reason over content across teams and systems, recognizing content types, extracting important information, and automatically organizing content into shared topics like projects, products, processes and customers.
Creates a knowledge network based on relationships among topics, content, and people.
Project Cortex drew on technological capabilities present in Azure’s Cognitive Services and the Microsoft Graph. It is not known to what extent the Equivio product, acquired in 2015, was integrated with these solutions but, from all the available details, it appears the technology is at least connected in one way or another.
During Ignite 2020, Microsoft announced SharePoint Syntex and trainable classifiers, either of which could be deployed to classify information and apply retention rules.
Trainable classifiers sound very similar to the predictive coding capability that appeared from 2011. However, they:
Use the power of Machine Learning (ML) to identify categories of information. This is achieved by creating an initial ‘seed’ of data in a SharePoint library, creating a new trainable classifier and pointing it at the seed, then reviewing the outcomes. More content is added to ensure accuracy.
Can be used to identify similar content in Exchange mailboxes, SharePoint sites, OneDrive for Business accounts, and Microsoft 365 Groups and apply a pre-defined retention label to that content.
In theory, this means it might be possible to identify a set of similar records – for example, financial documents – and apply the same retention label to them. The Content Explorer in the Compliance admin portal will list the records that are subject to that label.
SharePoint Syntex
SharePoint Syntex was announced at Ignite in September 2020 and made generally available in early 2021.
The original version of Syntex (as part of Project Cortex) was targeted at the ability to extract metadata from forms, a capability that has existed with various other scanning/OCR products for at least a decade. The capability that was released in early 2021 included the base metadata extraction capability as well as a broader capability to classify content and apply a retention label.
Classification. This capability involves the following steps: (a) Creation of (SharePoint site) Content Center; (b) Creation of a Document Understanding Model (DUM) for each ‘type’ of record; the DUM can create a new content type or point to an existing one; the DUM can also link with the retention label to be applied; (c) Creation of an initial seed of records (positives and a couple of negatives); (d) Creation of Explanations that help the model find records by phrase, proximity, or pattern (matching, e.g., dates); (e) Training; (f) Applying the model to SharePoint sites or libraries. The outcome of the classification is that matching records in the location where it is pointed are assigned to the Content Type (replacing any previous one) and tagged with a retention label (also replacing any previous one).
Extraction. This capability has similar steps to the classification option except that the Explanations identify what metadata is to be extracted from where (again based on phrase, proximity or pattern) to what metadata column. The outcome of extraction is that the matching records include the extracted metadata in the library columns (in addition to the Content Type and retention label).
As with trainable classifiers, Syntex uses Machine Learning to classify records, but Syntex also has the ability to extract metadata. Syntex can only classify or extract data from SharePoint libraries.
Trainable classifiers or Syntex?
Both options require the organisation to create an initial seed of content and to use Machine Learning to develop an understanding of the content, in order to classify it.
The models are similar, the primary difference is that trainable classifiers can work on content stored in email, SharePoint and OneDrive, whereas Syntex is currently restricted to SharePoint.
Predictive coding
On 18 March 2021, Microsoft announced the pending (April 2021) preview release of an enhanced predictive coding module for advanced eDiscovery in Microsoft 365.
The announcement, pointing to this roadmap item, noted that eDiscovery managers would be able to create and train relevance models within Advanced eDiscovery using as few as 50 documents, to prioritize review.
So, can Microsoft technology classify records better than humans?
In their 1999 book ‘Sorting Things Out: Classification and its Consequences‘ (MIT Press), Geoffrey Bowker and Susan Leigh Star noted that ‘to classify is human’ and that classification was ‘the sleeping beauty of information science’ and ‘the scaffolding of information infrastructures’.
But they also noted how ‘each standard and category valorizes some point or view and silences another. Standards and classifications (can) produce advantage or suffering’ (quote from review in link above).
Technology-based classification in theory is impartial. It categorises what it finds through machine learning and algorithms. But, technology-based classification requires human review of the initial and subsequent seeds. Accordingly such classification has the potential to be skewed according to the way the reviewer’s bias or predilections, the selection of one set of preferred or ‘matching’ records over another.
Ultimately, a ‘match’ is based on a scoring ‘relevancy’ algorithm. Perhaps the technology can classify better than humans, but whether the classification is accurate may depend on the human to make accurate, consistent and impartial decisions.
Either way, the manual classification of records is likely to go the same way as the manual review of legal documents for discovery.
The classification of records is fundamental recordkeeping activity. It is defined in the international standard ISO 15489-1:2016 (Information and Documentation – Records Management) as the ‘systematic identification and/or arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules‘. (Terms and Definitions, 3.4)
The purpose of classification is defined by State Records NSW as follows: ‘In records management, records are classified according to the business functions and activities which generate the records. This functional approach to classification means that classification can be used for a range of records management purposes, including appraisal and disposal, determining handling, storage and security requirements, and setting user permissions, as well as providing a basis for titling and indexing‘. (Records Classification, accessed 13 January 2021.)
The ever-increasing volume of digital records, the many different ways to create them, and the multitude of record types that are created and storage locations, have made it more difficult to accurately and consistently manually classify records, including through the creation of pre-defined ‘containers’ or aggregations based on classification terms. Despite this, the requirement to link the classification of records with their retention and and disposal remains.
For over three decades, Microsoft’s applications and technology platforms have been used to create, capture, store and manage records. Some of these records (in the earlier period) were printed and placed on paper files, or stored (from around 2000) in dedicated electronic document and records management (EDRM) systems.
But the volume and type of digital content, including with new types of records (e.g., chat messages) and storage locations, continues to grow. In response, Microsoft invested heavily in addressing the need to classify records ‘at scale’.
This post looks at various ways to classify records, for retention and disposition purposes, in Microsoft 365.
The old-school, manual method – metadata
Most of the records in Microsoft 365 will be created, captured or stored in one of the four primary workloads: Exchange mailboxes, SharePoint sites/libraries, MS Teams chats (a ‘compliance copy’ of which is stored in Exchange mailboxes), and OneDrive for Business libraries. Some records may also exist in Yammer or other web page content (e.g., intranet).
Most SharePoint sites as well as Teams (that have a SharePoint site) will be created according to some form of business need to create, capture, store and share records; that is, the site or team purpose may be based on a business function or activity. This way of grouping records may in some ways be used as a way to classify records – by SharePoint site (e.g., function) or document library (e.g., activity).
Records may be stored in multiple document libraries, or within a folder structure of a single library.
A number of methods (some of which rely on others) can be used to add classification (and other) metadata to records stored in SharePoint document libraries:
1 – Creating the classification taxonomy in the Managed Metadata Service (MMS)/Term Store via the SharePoint Admin portal – Content Services – Term store, and then applying these terms in content types that are then deployed in SharePoint sites.
An example of Business Classification Terms in the MMS
2 – Creating global content types from the SharePoint Admin portal, in the Content Services – Content type gallery area (see ‘Finance Document’ example below) and then deploying these in specific SharePoint sites where site columns that contain classification terms will be added.
3 – Creating site columns that contain classification terms, including from the MMS, and adding these to global or site content types or document libraries where they can be applied to records.
In this example, the site column ‘BCS Function’ maps to the MMS BCS terms
4 – Creating site content types and adding site columns (including MMS-based columns), then adding these content types to document libraries.
In this example, the MMS-based column now appears in the library columns
But, most of the above is somewhat complicated and cumbersome and would normally only be used for and manually applied to specific types of records.
The simplest way to apply BCS/File Plan terms at the document (or document set) level is to (a) store records to the same BCS function or activity in the same library, (b) create site or library columns with default values and add these to the library. This way means that the default terms are applied automatically as soon as a new record is uploaded, including when shared/inherited from the site columns added to a document set that ‘contains’ a document content type.
Example metadata columns shared from the document set content type
However, keep in mind that SharePoint is just one of the workloads where records are stored.
Records in the form of emails, chats and ‘personal’ content (as well as Yammer messages and web pages) are created in and stored across the other workloads. Some attempt may be made to copy these other records (especially emails) in SharePoint sites but it starts to get complicated or impossible to do so with things like Teams chat messages.
In most cases (and according to Microsoft’s own recommendations), it is better to leave the records where they were created or captured (‘in place’), and apply centralised compliance controls (classification, retention labels and policies) to this content.
Leaving the records in place in this way does not exclude the ability to create SharePoint sites and document libraries in those sites that map to classification terms, and/or use the site column approach described above but these are more likely to be exceptions.
In fact, some form of logical structure is almost certain anyway as most end-users will probably want to access and manage information in their own specific work context (the Team/SharePoint site).
Trainable classifiers
Since not all records are stored in SharePoint and the ever-increasing volume of digital content stored across the Microsoft 365 platform, Microsoft needed to find a way to classify records ‘at scale’.
The solution was to use machine learning (ML) via trainable classifiers accessed in the ‘Data Classification’ section of the Microsoft 365 Compliance portal. This capability is only available to E5 licences.
‘This classification method is particularly well suited to content that isn’t easily identified by either the manual or automated pattern matching methods. This method of classification is more about training a classifier to identify an item based on what the item is, not by elements that are in the item (pattern matching).’
Organisations (including E3 licence holders) may make use of five pre-defined trainable classifiers (Resumes, Source Code, Targeted Harassment, Profanity or Threat. A sixth classifier ‘Offensive language’, is to be deprecated). Custom classifiers require an E5 licence.
Custom classifiers require ‘significantly more work’ than the pre-existing classifiers and the process is quite involved (see the process flow diagram in the ‘Learn about’ page link above) but in summary it involves the following steps:
Creating the custom classifier.
Creating a set of manually selected example records (50 to 500) in a dedicated SharePoint Online site as the ‘seed’. This would include a range of emails in the seed examples.
Testing the classifier with the seeded documents.
Re-training with additional content – both positive and negative matches.
Once the classifier is published, it can be used to identify and classify related content across SharePoint Online, Exchange, and OneDrive (but not Teams).
The page ‘Default crawled file name extensions and parsed file types‘ provides details of all the record types that can be classified in this way. Note it is not clear if trainable classifiers can crawl the compliance copy of Teams chat messages stored in hidden folders in Exchange mailboxes.
Label-based retention policies can then be automatically applied to content that has been identified through the trainable classifier.
However, note that the classifier does not ‘group’, aggregate or ‘present’ (list) the records for review (except broadly via the Content Explorer); however, the label applied to the records can be searched via the ‘Content Search’ option in the Compliance portal. This is a much better option than not having any idea how many records of a particular classification may exist in Exchange mailboxes, OneDrive accounts, or general SharePoint sites. It requires some degree of ‘letting go’ of the ability to view and browse content classified this way, and trusting the system.
The main limit with trainable classifiers is that it requires an E5 or E5 compliance licence.
The other limitation is the management of the disposition of records that have been identified with trainable classifiers and had a label-based retention policy applied. There are significant shortcomings with the current ‘Disposition Review’ process, specifically the lack of adequate metadata to review records due for disposal or the details of what has been destroyed.
SharePoint Syntex
Another (but limited) option might be to use SharePoint Syntex (see ‘Introduction to Microsoft SharePoint Syntex‘ for an overview), although its range is limited to SharePoint and – it seems – only records that have a relatively consistent structure and format.
SharePoint Syntex evolved out of Project Cortex’s ability to extract and capture metadata from records. It can also be used through its ‘Document Understanding Model‘ (DUM) to provide a way to classify records stored in SharePoint Online (only). It makes use of a ‘seeding’ model that is similar to trainable classifiers (and may be based on the same underlying AI engine).
Broadly speaking, the DUM works on the basis of loading a small ‘seed’ set of (relatively consistently formated) example files into a dedicated Content Center (or Centers). This is very similar to the process of using trainable classifiers, except that the latter does not require a ‘content center’ SharePoint site to be created.
The example files are ‘trained’ by being ‘classified’ through the document understanding process based on a set of ‘explanation types‘ that are used to help find the relevant content. The three explanation types are: (a) phrase list (a list of words, phrases, numbers, or other characters used in the document or information that you are extracting); (b) pattern list (patterns of numbers, letters, or other characters); and (c) proximity (describes how close other explanations are to each other).
The document understanding model (DUM) produced through the explanation types is associated (and deployed) with a new or existing content type.Â
Once applied to a SharePoint site library, the DUM/content type provides the basis for identifying and tagging (with metadata) other similar records in the location (e.g., the library) where the DUM has been deployed.Â
If the documents have consistent content such as invoices, certain data from those documents can be extracted as metadata.Â
The answer to this question will depend on your compliance requirements.
Smaller organisations may be able to set up SharePoint sites and document libraries with site columns/metadata that maps to their business classification scheme or file plan, and copy emails to those libraries. There may be little need to use AI-based classification methods.
In large and more complex organisations (with E5 licences), especially those with a lot of content stored across Exchange mailboxes and SharePoint sites (including Teams-based sites) there will most certainly be a need for some form of AI-based classification in addition to classification-mapped SharePoint sites (and Teams).
Organisations with E3 licences might use the manual methods described above for specific types of records, and consider acquiring additional E5 Compliance licences to make use of trainable classifiers or SharePoint Syntex for other records.
Microsoft released its ‘Records Management’ solution for Microsoft 365 during 2020. The solution is only accessible to organisations with an E5 licence (or an E5 Security and Compliance licence).
Some of the retention-related options previously available to E3 licences, such as disposition review, are now only available with an E5 licence. However, for cost and other reasons, many organisations have decided to stay with E3 and asked if it is still possible to manage the retention of records.
A basic organization-wide or location-wide Exchangemailbox retention policy and/or to manually apply a non-record retention labeling to mailbox data.
A basic SharePoint or OneDrive retention policy and/or to manually apply a non-record retention label to files in SharePoint or OneDrive.
A Teams retention policy.
This post describes how the retention of records could be managed with an E3 licence and recommends that organisations intending to deploy Microsoft’s retention policies (whether E3 or E5) develop a plan for their deployment based on a detailed understanding of the following:
What records exist and where they are stored across the main Microsoft 365 workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.
What retention is required, including for EXO mailboxes and personal/ODfB accounts for which retention was previously ‘managed’ through backups, and MS Teams chats.
What type of retention policy will apply to what workloads.
How will disposition be managed (including of original storage locations, not just the records), and what evidence of destruction is required?
How will any limitations and shortcomings be addressed to minimise legal risk?
Where are the records?
A good starting point with retention planning is to establish where the records that will be subject to retention policies are stored.
Most records are created and stored in one of the primary four workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.
While SharePoint sites (including Teams-based sites and Teams channels) may be used to logically ‘group’ records (e.g., in a team site, a document library, or a channel), this is not the case for EXO mailboxes, ODfB accounts, MS Teams 1:1 or private channel chats (a compliance copy of these chats is stored in the personal EXO mailboxes of participants in the chat).
EXO mailboxes and ODfB accounts usually contain a range of records on different subjects, with different retention requirements. Personal chat messages could be about any subject.
For most of the past three decades, the requirement to keep specific emails or other records meant that end-users had to copy those records to another system, including an EDRMS or even SharePoint, or print and place them on a file. The ability to find old records depended more often than note on the ability to recover them from back-up tapes, a process ironically often referred to as ‘archiving’.
Action: Organisations should establish a list of where records are stored in Microsoft 365 and how retention (including via backups) is currently managed.
What retention is required?
The next step in the process is to establish how long these records need to be kept.
Most of the time this will be based on an organisational records retention policy or schedule. These policies or schedules describe groups of records and how long they must be retained – for example, financial records generally must be retained for seven years. Specific types of financial records may require shorter or longer retention.
This model generally works well when records have been stored in logical groups (e.g., a whole SPO site/Team, or document library), but are more difficult to apply for individual records stored with other records in personal EXO mailboxes, ODfB accounts, SPO sites, or MS Teams chats. If records with longer retention requirements in these locations are not stored elsewhere, or cannot be specifically identified, all the records may have to be kept for the longest retention period.
The start of retention is usually based on a trigger action. Typically these are based on (a) date created, (b) date modified, or (c) date of last action. However, they may also be based on less specific events, for example: (a) 7 years after a contract has expired, (b) 25 years after an employee leaves, (c) when a child turns 21. These less specific events need special attention.
Action: Organisations should ‘map’ retention requirements to the locations where the records are stored. This process will likely involve a discussion regarding the replacement (or supplementation) of email backups with retention policies, and may end up looking something like the following:
Workload
Retention
EXO mailboxes of senior managers
25 years
EXO mailboxes of all other employees
7 years
MS Teams 1:1 chats of senior managers
25 years
MS Teams 1:1 chats of all other employees
7 years
MS Teams private channel chats
As per retention policies
ODfB accounts of senior managers
10 years
ODfB accounts of all other employees
7 years
SPO sites that are not subject to more specific policies
7 years
Microsoft 365 Groups (includes mailbox and SPO site)
As per retention policies
SPO sites with specific retention requirements, per site or library
As per retention policies
What are the retention options with an E3 licence?
The three main options available with an E3 licence – labels, label policies, and retention policies – are all set from the Compliance admin portal under the ‘Information Governance’ section.
The table below summarises the options:
Type of policy
Can be used for
‘Implicit’, ‘safety net’ retention policies. These policies: (a) work in the back end and cannot be changed by an end-user; (b) create a preservation hold library in SPO sites and ODfB accounts, and hold deleted emails in a hidden EXO mailbox folder; (c) provide an alternative to back-ups, although it should be kept in mind that all content that is retained in this way contributes to the overall storage quota; (d) do not retain record of what was destroyed.
EXO, SPO, MS Teams, ODfB
‘Explicit’ label-based retention policies. Labels must be published to the required workloads before they become visible. Records cannot be deleted once a label has been applied, but end-users can change or remove the label and then delete the record. Label-based policies do not retain a record of what was destroyed.
EXO, SPO, ODfB
‘Explicit’ label-based retention policies that are auto-applied based on three limited options – see below. Records cannot be deleted once a label has been applied. These policies do not retain a record of what was destroyed.
EXO, SPO, ODfB
The three auto-apply options are as follows:
Apply label to content that contains sensitive information. Unlikely to be used as retention is never based on sensitivity.
Apply label to content that contains specific words or phrases, or properties. Possibly useful. Recommend that organisations do a Content Search first to see what records may exist.
Apply label to content that matches a trainable classifier. E3 licences provide six out of the box, limited classification options including ‘profanity’. These are unlikely to be of any value.
Mapping the E3 options
Given the options available, the following is a suggested mapping of records, retention and policy options:
Workload
Retention
Retention policy
EXO mailboxes of senior managers
25 years
Retention policy, plus possibly labels auto-applied to specific records
EXO mailboxes of all other employees
7 years
Retention policy, plus possibly labels auto-applied to specific records
MS Teams 1:1 chats of senior managers
25 years
Retention policy
MS Teams 1:1 chats of all other employees
7 years
Retention policy
MS Teams private channel chats
As per retention policies
Retention policy
ODfB accounts of senior managers
10 years
Retention policy, plus possibly labels auto-applied to specific records
ODfB accounts of all other employees
7 years
Retention policy, plus possibly labels auto-applied to specific records
SPO sites that are not subject to more specific policies
7 years
Retention policy
Microsoft 365 Groups (includes mailbox and SPO site)
As per retention policies
Retention policy
SPO sites with specific retention requirements, per site or library
As per retention policies
Retention policy (safety net) plus label-based retention policy for specific records
Policy limits
Organisations planning to deploy retention policies should be aware of the limits on custom policies, as described on this Microsoft page, ‘Create and configure retention policies‘. There are no limits on policies that apply to an entire workload (e.g., all EXO mailboxes).
1,000 mailboxes
1,000 Microsoft 365 groups
1,000 users for Teams private chats
100 sites (OneDrive or SharePoint)
The page above notes also that ‘There is also a maximum number of policies that are supported for a tenant: 10,000. However, for Exchange Online, the maximum number is 1,800. The maximum number includes retention policies, retention label policies, and auto-apply retention policies.’
How will disposition be managed
Microsoft 365 retention policies retain records for a specified period and usually then delete the records automatically. No record is retained of what was deleted. Even with an E5 licence, only limited metadata is retained and only on those records subject to disposition review.
Organisations deploying retention policies with an E3 licence need to understand the potential risks associated with being unable to provide evidence of what was destroyed.
There are at least two ways to approach this point, but in all cases the approach and options must be legally defensible:
Ensure that organisational policies clearly indicate what records will be destroyed without any record being kept. For example, with certain exceptions, most emails, Teams chats, SPO sites and the content of ODfB accounts will only be kept for 7 years and then destroyed.
Establish a process to ensure that a record is kept of specific records that have been destroyed. For example, the metadata of records, due for disposal and stored in document libraries in specific SPO sites, will be captured (and stored separately) before the records are destroyed, and then the document library will be deleted. This is a labour-intensive process and may only be used on some sites.
Addressing E3 licence limitations and shortcomings
The limitations of E3 licence retention policies (and also E5 licences from an evidentiary point of view) should not put organisations off using the out of the box options or cause them to acquire third-party products.
The retention options available in Microsoft 365 now provide functionality that assists recordkeeping compliance, for example the ability to apply ‘back end, safety net’ retention policies to EXO mailboxes, MS Teams chats, ODfB accounts and whole SPO sites. Coupled with the ability to locate all records, including ‘deleted’ records via Content Search, this should be a boon to records managers and also to IT (saving the latter from having to recover records from backup tapes).
Once retention policies have been deployed, the next most complex task may be the disposition approval and destruction process for those records for which a formal disposition review and approval process is required, including the requirement to establish a list of records to be destroyed.
As noted above, even the E5 licence options for disposition review and ‘proof of disposition’ are inadequate in terms of the metadata that is presented and retained. Until Microsoft provides the ability to record the metadata of every record that is due for disposal and/or destroyed, based on ANY type retention policy this process will continue to be a manual task for records managers.
Organisations may consider acquiring a limited number of E5 Security and Compliance licences to gain access to the E5 ‘Records Management’ capability to gain access to other options including File Plan, auto-classification options, and Disposition Review. However, aside from these options (and the ability to auto-apply labels more broadly) these options may not add much more capability to the existing options already available with an E3 licence.
Summarising the options
Based on the above details, organisations with E3 licences might do the following:
Create several ‘safety net’ retention policies for EXO mailboxes, ODfB accounts, MS Teams chats and general SPO sites. EXO/ODfB retention policies may be (a) based on current back-up periods and (b) be split into ‘senior’ and all other employees. They may also create policies for Microsoft 365 Groups which will cover the Group’s EXO mailbox and SPO site. Some of these policies may map to existing retention classes in the records retention schedules, but others such as the EXO, ODfB and MS Teams policies may need to be added.
Create label-based retention policies for content stored in specific SPO sites where more granular retention policies are required per library. However, keep in mind these can be changed or removed by end-users with add/edit access. Generally it is not recommended to apply label-based policies to EXO or ODfB unless end-users will apply these accurately and consistently.
Auto-apply some labels to specific content stored in SPO, EXO and ODfB (having identified that these records exist first via a Content Search). These labels should have a retention period that is longer than the safety net policy or any other label-based policy.
Establish a ‘manual’ disposition review process for records that are subject to label-based retention policies.
SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.
While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.
Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.
So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.
Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.
The following is a list of the main ways a person can create a SharePoint site. Have I missed any?
This option also allows the administrator to provision new SharePoint sites.
2. Via the SharePoint Admin portal (+ Create)
This option allows the creation of three main types of sites: modern team sites (Team site), communication sites, and non-Microsoft 365 Group-linked sites (Other options).
3. By creating a Microsoft 365 Group
Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.
Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.
The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.
Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.
4. By an end-user creating a new Team in MS Teams
Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.
Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an Exchange Online mailbox that is not visible in Outlook.
5. By creating a Private Channel in MS Teams
If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.
For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.
A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.
6. By the Teams Admin creating a new Team
The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.
As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.
7. From the end-user SharePoint portal (+ Create site)
This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.
It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.
8. From OneDrive for Business as a ‘shared library’
This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.
9. When a new Plan is created in Planner
If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.
10. When a new Yammer community is created
End users with access to Yammer can click on ‘Create a Community’ from Yammer.
To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’
Why have Microsoft allowed this?
It’s a smarter way to manage access.
Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.
The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.
But, some of the decisions don’t seem to make sense.
Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
And why does a new Plan create a SharePoint site? For what purpose?
Perhaps there is a reason for it. It’s just not clear.
Microsoft 365 includes a range of connectors, in three categories, that can be used to support the management of records created by other applications. The three categories are:
Search connectors, that find content created by and/or stored in a range of internal and external applications, including social media.
Archive connectors, that import and archive content created by third-party applications.
API connectors, that support business processes such as capturing email attachments.
This post how these connectors can assist with the management of records.
The recordkeeping dilemma
Finding, capturing and managing records across an ever increasing volume of digital content and content types has been one of the biggest challenges for recordkeeping since the early 2000s.
The primary method of managing digital records for most of the past 20 years has been to require digital records (mostly emails and other digital content created on file shares) to be saved to or stored in an electronic document and records management system (EDRMS). The EDRMS was established as ‘the’ recordkeeping system for the organisation.
EDRM systems were also used to manage paper records which, over the past 20 years, have mostly contained the printed version of born-digital records that remain stored in the systems where they were created or captured.
There were two fundamental flaws in the EDRMS model. The first was an expectation that end-users would be willing to save digital records to the EDRMS. The second was that the original digital record remained in place where it was created or captured, usually ignored but often the source of rich pickings for eDiscovery.
The introduction of web-based email and document storage systems, smart phones, social media and personal messaging applications from around 2005 (in addition to already existing text messaging/SMS messages) further challenged the concept of a centralised recordkeeping system; in many cases, the only option to save these records was to print and scan, screenshot and save the image, or save to PDF, none of which were particularly effective in capturing the full set of records.
The hasty introduction from early 2020 of ‘work from home’ applications such as Zoom and Microsoft Teams has been a further blow to these methods.
In place records management
To the chagrin of records managers around the world, Microsoft never made it easy to save an email from Outlook to another system. Emails stubbornly remained stored in Exchange mailboxes with no sign of integration with file shares.
And for good reason – they have a different purpose and architecture to support that purpose. It would be similar to asking when it would be possible to create and send an email in Word.
The introduction of Office 365 (later Microsoft 365) from the mid 2010s changed the paradigm from a centralised model – where records were all copied to a central location and the originals left where they were created or captured, to a de-centralised or ‘in place’ model – where records are mostly left where they were created or captured.
The decentralised model does not exclude the ability to store copies of some records (e.g., emails) in other applications (e.g., SharePoint document libraries), but these are exceptions to the general rule.
It also does not exclude the ability to import or migrate content from third-party applications where necessary for recordkeeping purposes.
Microsoft 365 connectors
Microsoft 365 includes a wide range of options to connect with both internal and external systems. Many of these connectors simplify business processes and support integration models.
Connectors may also be used to support recordkeeping requirements, in three broad categories.
The three connectors
Archive connectors
Archive connectors allow organisations to import and archive data from third-party systems such as social media, instant messaging and document collaboration* platforms. Most of this data will be stored in Exchange mailboxes, where it can be subject to retention policies, eDiscovery and legal holds.
(*This option is still limited via connectors, but also see below under Search).
The social media and instant messaging data that can be archived in this way currently includes Facebook (business pages), LinkedIn company page data, Twitter, Webex Teams, Webpages, WhatsApp, Workplace from Facebook, Zoom Meetings. For the full listing, and a detailed description of what is required to connect each service, see this Microsoft description ‘Archive third-party data‘.
An important thing to keep in mind is that the data will be archived to an Exchange mailbox; this will require an account to be created for the purpose. Any data archived ot the mailbox will contribute to the overall storage quotas.
Search connectors
Search connectors (also known as Microsoft Graph connectors) index third-party data that then appears in Microsoft search results, including via Bing (the ‘Work’ tab), from http://www.office.com, and via SharePoint Online.
Most ECM/EDRM systems are listed, which means that organisations that continue to use those systems can allow end-users to find content from a single search point, only surfacing content that users are permitted to see.
The following is an example of what a Bing search looks like in the ‘Work’ tab (when enabled).
Example Bing search showing the Work tab
Note: as at 17 November 2020, Microsoft’s page ‘Overview of Microsoft Graph connectors‘ (which includes a very helpful architecture diagram) states that these are ‘currently in preview status available for tenants in Targeted release.’
There are two main types of search connector:
Microsoft built: Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL, Enterprise websites, MediaWiki, Microsoft SQL, and ServiceNow.
Partner built. Includes the following on-premise and online document management/ECM/EDRM connectors – Alfresco, Alfresco Content Services, Box, Confluence, Documentum, Facebook Workplace, File Share (on prem), File System (on prem), Google Drive, IBM Connections, Lotus Notes, iManage, MicroFocus Content Manager (HPE Records Manager, HP TRIM), Objective, OneDrive, Open Text, Oracle, SharePoint (on prem), Slack, Twitter, Xerox DocuShare, Yammer
A consideration when deploying search connectors is the quality of the data that will be surfaced via searches. Duplicate content is likely to be a problem in identifying the single – or most recent – source of truth of any particular digital record, especially when the organisation has required records to be copied from one system (mailbox/file share) to another (EDRMS).
API Connectors
API connectors provide a way for Microsoft 365 to access and use content, including in third-party applications. To quote from the Microsoft ‘Connectors‘ web page:
‘A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.’
To see the complete list and for more information about each connector, see the Microsoft web page ‘Connector reference overview‘.
Each connector provides two things:
Actions. These are changes initiated by an end-user.
Triggers. There are two types of triggers: Polling and Push. Triggers may notify the app when a specific event occurs, resulting in an action. See the above web page for more details.
API connectors can support records management requirements in different ways (such as triggering an action when a specific event occurs) but they should not be confused with archiving or search connectors.
Summing up
The connectors available in Microsoft 365 support the model of keeping records in place where they were first created or captured. They enable the ability to archive data from third-party cloud applications, search for data in those (and on-premise) applications, and triggers actions based on events.
The use of connectors should be part of an overall strategic plan for managing records across the organisation. This may include a business decision to continue using an ECM/EDRMS in addition to the content created and captured in Microsoft 365. Ideally, however, the content in the ECM/EDRMS should not be a copy of what already exists in Microsoft 365.
