The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.
This post explains the difference between the two ‘in place’ options. In brief:
The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.
The Microsoft in-place model
The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.
Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:
The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
Mandatory record metadata.
Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
Unique computer-generated record identifiersfor each record, regardless of where that record is stored.
The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
Not permitting modification of certain metadata fields.
The capability to support multiple renditions of a record.
The capability to increment versions of records when filing. Linking the record metadata to the record so that it can be accessed for display, export.
Enforcement of data integrity, referential integrity, and relational integrity.
Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):
Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.
The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’
The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’
A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.
One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.
Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.
The new ‘in place’ model
A newer form of in-place records management has appeared with Microsoft 365.
Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.
The newer model of in place records management is based on the assumptions that:
It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.
Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.
In place co-existence
Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.
Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021
SharePoint Online document libraries include the option to move content to a different location. But what happens to the metadata when this content is moved?
This post explains what happens to the metadata when different types of content stored in a SharePoint Online document library is moved (not copied) using the ‘Move to‘ option to a different library in either the same site or a different site (both in SharePoint Online).
If you don’t have time to read all the details, here is a summary. When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:
The original system generated metadata (Created, Modified) is retained for all types of content.
The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
Information Protection labels (set on Office document only) are retained unless a label setting prevents the move.
Retention labels are retained for all types of content.
Any added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.
The same outcomes occur if documents are moved between two synced document libraries in File Explorer.
Metadata in SharePoint libraries
All SharePoint document libraries include four types of metadata:
System generated. Examples include the Created [in SharePoint, not the original item created date] and Modified dates, the Created by and Modified by names, File size, Content Type, sequential ID, and Version History (which is a collection of metadata objects). If Document IDs are enabled, it also includes the Document ID (PREFIX-LibraryID-sequential ID, eg ‘RECORDS-12312322-234). Note that every object saved to SharePoint is also assigned a unique GUID.
Name. This is either the original name of the item (in the case of an email, the email subject line), or added by an end-user. Note that this is NOT the same column as the Title column which is blank by default.
Added metadata. That is, pre-existing or custom site, or ‘local’, metadata columns added to the library. In the examples used in this post, the added metadata is ‘Document Type’ (choice, default =’Agenda’), ‘Event Type’ (choice, default =’Indoor’), and ‘Random Text’ (text, default = ‘Elephant’).
It is important to note that metadata in added metadata columns in one library will NOT appear in the destination library unless that (destination) library has exactly the same columns.
For the purpose of this post, four different types of content (Office/Word document, PDF, email (.msg and .eml), image) were moved from a library with multiple metadata elements shown below to an empty document library in brand new site with nothing configured at the site or library level.
The metadata elements set on the source site library were as follows:
Document Type (choice)
Event Type (choice)
Random Text (default set to ‘Elephant’)
Retention label applied to the library and a different label applied at the folder level
Information Protection label. This was only applied to Word documents (when they were edited) as there was no option to apply it to the emails, PDFs or images.
Moving a PDF, email (msg) and image
As the outcome is likely to be the same for all non-Office content, a PDF, email (msg) and image were all moved at the same time to the destination library. Immediately a notification appeared that metadata properties on all three items would be lost if they were moved. The option to ‘Move anyway’ was selected and the items were moved.
The destination library shows the three items without the Document IDs (which were not enabled) or any of the added metadata columns. However, the items have retained the Retention label – no retention has been set anywhere on the destination site.
Were the Document IDs moved?
No. The original Document IDs did NOT appear because the destination library did not have the Document ID feature enabled.
Once that feature was enabled and the column was made visible in the list view, we can see that the three items that were already moved have lost their original Document IDs. Two new items moved have retained their original source Document ID (CORPRECORDS-667097513-n).
Moving Office documents
The metadata applied to Office documents (Word, Excel, PowerPoint with an ‘x’ at the end) works in a similar way but those documents also embed the metadata in their underlying XML structure.
Some of this metadata is visible in, and can be directly edited from, the Info properties of the document. For example, in this document …
… we can see the Document ID, Document Type, Random Text and Event Type in the Info properties. These metadata properties remain with the document even when it is downloaded and so can be very useful to determine the origin of a document if, for example, it is found as an email attachment or on a network file share.
Where is this metadata embedded?
All Office documents with a document extension that ends in ‘x’ (e.g., docx) are based on an XML structure that can be viewed by converting the original to a zip file (using the CMD shell and REN to rename it) and then extracting the resulting zip file and opening it.
The XML structure of a standard Word document looks like the following. The XML documents contained in the structure can be viewed with simple text viewers like NotePad.
The system, added and embedded metadata properties are stored in one of the XML documents named item1 to item 4 in the customXml folder. The following shows the values applied in item1, while item 4 contains the other values that can be assigned to choice fields.
The metadata for the Sensitivity label (including any formatting details) is stored in the ‘custom’ XML document in the docProps folder.
BUT – details of the Retention label are not stored in the XML properties of the downloaded document. This may be because once the document is downloaded, the retention label no longer applies.
What happens to that metadata when Word documents are moved?
As with the previous examples, a warning will indicate that metadata properties may be lost. If ‘Move anyway’ is selected, the document is moved.
If the document has an Information Protection label setting that prevents it, it may not be possible to move it. The following message will be displayed. The only way to move it is to change the label to one that allows it, or remove the label.
For any other Office document, the original Document ID and Retention label are retained but any added metadata properties will be lost if those columns do not exist in the destination library.
However, it seems that the added properties are not completely lost; they are moved to the ‘custom’ XML document in the docProps folder.
However, if the same metadata column is added to the library, documents that have this added metadata in the ‘custom’ XML document do not display that metadata. It must be added back.
What happens if the destination library has the same metadata columns?
If the destination library has the same added metadata columns, as shown in the example below, that metadata will be copied to the destination library. The first Word document in the example below was copied before the metadata column was added.
When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:
The original system generated metadata (Created, Modified) is retained for all types of content.
The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
Information Protection labels set on Office document only are retained unless a label setting prevents the move.
Retention labels are retained for all types of content.
Added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.
A common question asked by many organisations is whether Microsoft 365 (M365) retention policies – labels in particular – can be applied to folders in SharePoint document libraries so the content in those folders will have the same label.
The quick answer is yes, but it is a manual process and – for all its perceived benefits – is likely to be more of an administrative and support burden and not worth the effort. Folders should NOT be thought of as the replacement for ‘files’ (aggregations of individual records), but more like dividers in a lever arch (= the document library).
This post describes how labels can be applied to and work with folders, including in SharePoint sites linked with Teams. It also suggests alternative options.
How retention labels are applied to a library
Retention labels are created in the M365 Compliance portal under either the Information Governance > Labels or Records Management > File Plan sections.
Labels created in the Compliance portal do not do anything once created; they must be applied to content in various ways to make them work. This includes by:
Publishing one or more labels as part of a retention policy to various locations including SharePoint sites, Exchange mailboxes and OneDrive (but not Teams – see screenshot of available locations below). In this scenario, each label will be visible to – and selectable by – end-users.
Auto-applying them to the same locations based on various options, including (for E5) trainable classifiers
Adding them to Content Types used in SharePoint Syntex.
Publishing labels to SharePoint sites
When one or more labels are published to SharePoint sites they don’t do anything until they are ‘manually’ enabled through one of the following options:
On each individual document library via the library settings option ‘Apply labels to items in this list or library’ (see screenshot below)
On each individual folder in a library (via the information panel, see screenshot below)
On each individual object in the library (also via the information panel)
Applying a label to the library
A label can be applied to the entire document library via Library Settings, as shown in the screenshot below.
If the drop-down option is set to ‘None’, and there are no options to choose from, it means that no labels have been published to this SharePoint site.
If labels exist, they will appear in the drop down list (below the default ‘None’). Note that only one label can be set as the default for the library. If the check box ‘Apply label to existing items in the library’ is selected, this will apply the label to all existing items. It will also likely override any existing label that may have been applied.
When the retention label has been applied to a library, the label only applies to the non-folder objects stored in the library as can be seen in the option below. That is, the retention label is NOT applied to the folders by default.
The retention label can be seen when the folder is opened:
Applying a retention label to a folder or document
It is also possible to apply a retention label to a folder or object stored in the folder via the information panel, even when a default library label has been set, as shown below. This can be done on each individual folder including, for Teams-based sites, each folder that maps to a channel.
When applied to a folder in this way, any content stored in the folder will inherit that retention label.
If a default label has already been applied at the library level, the folder-based label will replace it, although in testing this, one of the original default labels wasn’t replaced automatically as shown below, but could be manually changed via the information panel.
Implications for Teams-based records (Files)
Every Team in MS Teams has an associated SharePoint site linked with the underlying Microsoft 365 Group.
Every non-private channel in the Team maps to a folder in the Documents library of the SharePoint site as can be seen in the two screenshots below. (Every private channel has a separate SharePoint site that would be covered by a separate retention policy).
Keep in mind that retention labels remove the ability to delete objects stored in the library (including via the ‘Files’ tab in a Team). If end-users are working in Teams, this could be annoying and potentially put them off using Teams. However, end-users can remove the label by navigating to the SharePoint site and removing the label via the Information panel.
Why folder-based retention labels may not be a good idea
The default options to apply retention labels to content stored in SharePoint document libraries are:
By applying them at the library level. This can apply the label to all existing (and future) content stored in the library but does not apply to folders.
Through the auto-application of labels.
Via SharePoint Syntex using labels on Content Types.
Applying retention labels to individual folders in a document library is a manual-intensive process, one that may be a waste of time given the potential number of libraries that can exist and the ease with which they can be removed by end users.
Additionally, applying retention labels to the channel-linked folders of Teams may be pointless if end users:
Store documents at the same level as the channel-linked folders; that is, ‘above’ the folder structure.
Create new folders via a synced library or SharePoint. These folders are not linked to channels.
Create new libraries in the SharePoint site.
Keep it simple
It is very easy to deliberately or inadvertently establish over-complicated retention settings for content stored in SharePoint, especially as there is currently no simple way to see what label has been applied where.
Given the retention period linked with retention policies generally, there is a good chance that the person who applied the labels may not be around when the retention period expires, or to keep an eye on what has been applied or changed over time.
The best retention intentions may be overruled by practical necessity.
The best retention model, in my opinion, is a simple one that does not get in the way of end-users but ensures that records will be kept for a minimum period required. So, instead of applying retention labels to folders, especially on Teams-based SharePoint site libraries, it is recommended to:
Start by trying to avoid mixing content with different retention periods in the same SharePoint site or Team, or document library. That will make it easier to manage the retention outcomes. (If you can’t avoid mixing content, you may need to use auto-application of labels including via Syntex or trainable classifiers).
Use ‘back-end’ safety net retention policies applied to all SharePoint sites. This ensures a minimum retention period and does not get in the way of end-user activities.
Use retention labels on site libraries where more granular retention is required. Ideally, apply them as the default to all the content in a single document library (including the default library for all Teams-based SharePoint sites) and – preferably – only apply the labels when the content is inactive and the library can be made read only, to protect the records from that point.
Only use multiple labels on folders when (a) all the labels applied to the site relate to the same function/activity pair or subject matter, and (b) the content is largely inactive. Ideally, avoid folder-based retention to avoid complication in the future.
Microsoft 365 has become one of the world’s most accessed products for office collaboration. Jeff Teper, the ‘father of SharePoint’ at Microsoft, tweeted on 27 April 2021 that Teams had 145 million daily active users. (As reported in by the team at Office365ITPros.com.) According to the website ‘Microsoft Office Statistics and Facts (2021) | By the Numbers‘ , Microsoft Teams usage grew 40% during the COVID lockdowns.
Although organisations create, capture and store a range of records using the various Microsoft 365 applications, most records are likely to be created or captured in Exchange mailboxes or SharePoint (including OneDrive).
The volume and range of records has, in many respects, overwhelmed traditional standards-based models that required records (including emails) to be copied to a central electronic document and records management system (EDRMS) or identified and then ‘declared’ as records.
Given the reality of the new paradigm, organisations have tried various ways to manage records in Microsoft 365, including by retaining the EDRMS (for high value records), acquiring third-party products that promise to address the ‘gap’ in recordkeeping functionality, or working with the ‘out of the box’ capability.
Whichever method is chose, records managers need to have a very good understanding of where the records are in Microsoft 365 and how they can be managed. In many cases, leaving and managing them where they were created or captured (‘in place’ management), and using new and emerging capability to leverage the power of AI-based tools is likely to be the future state.
With the above in mind, and regardless of which method you follow, the following are ten points that I think are important to consider when managing records in Microsoft 365.
What are your recordkeeping obligations?
It is perhaps the most obvious question but organisations have sometimes rolled out Microsoft 365 without consideration of their obligations for managing records.
Records provide evidence of business activities and accordingly need to be protected to ensure their authenticity, integrity and reliability as evidence. The most common way of achieving this outcome until now has been to ‘lock’ digital records from change. Is this practical in the digital world? How do you lock Teams chats or stop a new thread in an email exchange? Could the same outcome be achieved by recording all changes and ensuring these are retained?
Records are usually subject to minimum retention requirements and understanding what these are is essential. Where there are minimum retention requirements, records cannot be destroyed before a specific period of time based on a particular trigger or event. These requirements are defined in legislation (sometimes based on statutes of limitation) or, for government agencies, in records disposal authorities or schedules (as shown in the example above).
As a starting point, look at these retention requirements and consider how these will be applied to Teams 1:1 chats, or team chats, or emails still in Exchange mailboxes, or OneDrive content. And then extend this to the content stored in Teams/Group-based SharePoint sites and sites not linked with Microsoft 365 Groups.
Consider also what is required to manage the outcome of retention. Do you need to review records due for destruction? Do you need to keep a record of what was destroyed? For all records?
There may be a requirement to categorise or classify records (especially to group them by context and/or for retention purposes where retention is based on classification). How will this outcome be achieved for Teams chats or emails that remain in Exchange mailboxes, or OneDrive content? What other metadata do you need for records?
Your recordkeeping obligations, in particular records retention requirements, should guide the management of records in Microsoft 365.
Where are the records created or captured in Microsoft 365?
Neither Microsoft 365 nor SharePoint is a dedicated recordkeeping system like an EDRMS (see my post ‘SharePoint is not an EDRMS’). Rather, it is an ecosystem of multiple applications that are used to create, capture, store and manage records.
Most records are likely to be stored in either SharePoint (OneDrive is a SharePoint service) or Exchange mailboxes.
A compliance copy of Teams chats are stored in a hidden folder of Exchange mailboxes. Content stored in the ‘Files’ tab is either stored in SharePoint or (for 1:1 chat) in OneDrive.
Of course, there will be some other records – Yammer conversations, tasks and plans, communication sites, calendar entries, forms and even WhiteBoard sessions. But, more than 95% are stored in Exchange mailboxes or SharePoint/OneDrive.
Knowing your recordkeeping obligations and the location of records are the main starting points. In fact, you can map your recordkeeping obligations (especially metadata and retention) to the location of the records.
Do you control the creation of Teams and SharePoint sites or not?
There two, broadly speaking, two approaches to controlling the creation of Teams and SharePoint sites:
Yes, we have controls – There is some kind of control or decision ‘gate’ for the creation of Teams and SharePoint sites.
No, we don’t have controls – End-users can create Teams and SharePoint sites whenever they want. In this case, the points below may not be of much use. You will likely rely on the built-in ‘records management’ capability to manage the records.
If your answer to the question above was ‘No’, then you will probably need third-party products and/or rely heavily on AI-based solutions to manage the records (which is the default Microsoft position).
Map sites and Teams to business functions – don’t mix content
Almost every organisation has a range of business functions. Some of these are common to all or most organisations (e.g., information technology, human resources, financial management, legal, property, etc) while others are ‘core’ (e.g., engineering, manufacturing, research, sales and marketing, etc).
Many organisations are structured around these business functions, and most records retention schedules are based on function (or business function).
If you can map new Teams and SharePoint sites to these functions, this will facilitate the management of records down the track. Mixing content across multiple functions – except where it makes sense to do this, such where there are related but smaller numbers of records in project sites – is going to make it harder to manage the records in the longer term – and more or less the same as letting end-users put whatever they want into a paper box for long-term storage.
A common example where records might be mixed are ‘Corporate Services’ areas that create or capture records across multiple functions, including property, IT, finance and so on. Unless all the records in a Team-based site can be kept for the same period of time, it may be a good idea to separate the records into different sites.
Also keep in mind that some business areas may exist for a long time; having a single (Teams-based) document library that has folders linked to channels may not be the best way to manage records long-term.
The suggestions above don’t take into account Exchange mailboxes, Teams chats or OneDrive accounts because these cannot be mapped to functions.
Naming conventions for sites and teams, and libraries
The main reasons for having naming conventions for SharePoint sites and Teams are:
It is good practice, to avoid acronyms and other less than useful names.
To prevent unnecessarily long names that end up creating very long URLs (e.g., ‘https://tenantname.sharepoint.com/sites/ExecutiveCommittee20202021MeetingsHeldDuringLockdownandrecordedviaMSTeamsSeniorManagersOnly‘.) It is important to know the difference between the URL name and the display name.
Ideally, the original names of Teams and SharePoint sites should be restricted to no more than 14 characters so that Document IDs (that have a 12 character prefix) can be the same as, or very close to, the URL name of the site. For example:
Aside from the default ‘Documents’ library of every Teams-based site, library names should also be subject to naming conventions and restricted to around 20 characters. There are several reasons for this.
The first is how they appear in the left hand navigation of a SharePoint site. There isn’t much point having multiple library names that aren’t easily visible (the two examples below have completely different names after ‘Financial Management’).
The third reason is that it is good practice to have some form of naming conventions.
Ideally, library names should map to the activities that produce the records AND include the year where this is relevant, e.g., ‘Meetings2021’.
There is NO need to repeat words in the tenant or site name – e.g., h**ps://tenantname.sharepoint.com/sites/TenantNameCorporateRecords/TenantNameFinancial%20Management%20%20Accounting%20%20Invoices%202021/Forms/AllItems.aspx
As noted above, this doesn’t apply to the default ‘Documents’ library in Teams-based SharePoint sites (the actual name is ‘Shared%20Documents’).
Metadata and Content Types
For many organisations, the minimum metadata requirements consist of (a) agent (e.g., the person who did something), (b) dates (when they did something) and (c) a unique identifier. That is, who did what and when?
If you need to add more metadata for certain types of records you can really only do this in SharePoint document libraries, including by adding them from the SharePoint Term Store (see below example). It can also be done in Outlook but these metadata terms are not linked with the SharePoint terms.
As for Content Types, do you really need them? SharePoint is made up of multiple Content Types already, including the default ‘Document’ Content Type. It is important to understand how Content Types work in SharePoint before making the assumption that they are required.
In many cases, choice metadata fields can replace the need for Content Types. Custom Content Types may only be needed for specific or high value record types.
Document retention policies and labels
In the first section about recordkeeping obligations, it was noted that most records will be subject to minimum retention requirements. Retention labels and policies are created in the Compliance admin portal of Microsoft 365.
Unfortunately, the current Compliance admin portal provides very little information to show what label or policy was applied where. The only way to do this is to document it yourself. One way to do this is to create a spreadsheet that lists on each row:
The business function and activity from a File Plan or Business Classification Scheme (e.g., Financial Management – Accounting)
Each retention class for that function/activity pair, including the reference number
If that class has been created as a label, what the label name is. If it has been created as a non-label retention policy, what that retention policy name is. (Generally speaking, disposal authority classes don’t refer to Exchange mailboxes, SharePoint sites, MS Teams chats or OneDrive content, so the organisation may need to determine what this minimum retention period should be and how it will manage the retention outcomes).
(Note, the above can be created in the File Plan section of the Records Management part of the Compliance admin portal, E5 licences only. However, it only documents the above information and does not show where the label has been applied.)
Where the label has been applied ‘manually’ – to which SharePoint site/document library, Exchange mailbox or OneDrive account. This point may have multiple location references.
Where the label has been auto-applied through the basic E3 option or, for E5 licences, the document understanding model (DUM) in SharePoint Syntex, or via trainable classifiers.
When the retention will expire.
Retention outcome – If a disposition review (E5 only) exists, the records will be destroyed automatically without any record kept, or ‘do nothing’. See also below.
Remember that retention labels and policies apply to individual items (emails, Teams chat, SharePoint or OneDrive content stored in libraries), not to aggregations (e.g., the entire library or site). The aggregation will continue to exist after the content has been destroyed and no ‘stub’ (a record of what used to exist) will remain.
How will you manage retention outcomes?
Generally speaking, Microsoft 365 retention policies destroy records when they are due for destruction unless they are subject to a label that has the disposition review option enabled or the ‘do nothing’ option has been selected.
Organisations need to understand how they will manage these retention outcomes especially as, in most cases, a review process is required. (See ‘Recordkeeping obligations’ above).
Even when retention label have the disposition review option enabled (E5 only), there are two points that need to be understood:
The ‘disposition review’ interface presents individual records with no context except for the original site URL name. Some additional (default) metadata is now included (from May 2021) but not any added metadata. In most cases, it will be necessary to return to the original library to view the context of the records presented for disposal, and if there are any others.
If records are destroyed through that review process, only basic metadata is retained about what was destroyed.
Organisations that have an obligation to undertake a full review of records due for disposal will likely need to consider establishing workarounds such as exporting the full set of metadata from a document library and then using that to review whether the content of the library can be destroyed. If approval is granted, that decision should be recorded, along with the metadata extract.
Allow end-users to get on with their work
End-users generally don’t have much interest in the management of records beyond the period of time they are important to them. They want to do whatever they want, whenever they want, using the applications they have available to them.
Collaboration no longer consists of email exchanges and document-based records. Creating control gates for the creation of Teams and sites, and insisting on naming conventions for sites and libraries (and folders) may be interpreted negatively.
There needs to be a fine balance between control and freedom and this can impact the creation, capture and management of records. Some of the ways to minimise the impact of recordkeeping requirements include:
Enabling Document IDs on every site.
Creating custom metadata columns on sites or libraries with default values.
Applying non-label ‘safety net’ retention policies to all workloads. Retention policies (along with the Recycle Bin for 90 days) helps with the recovery of accidentally deleted content.
Using various communication methods to highlight useful features including sharing (instead of attaching), the Recycle Bin, versioning in SharePoint/OneDrive, and the ability to have a ‘single source of truth’. These features can be used to ‘soften’ the impact of other recordkeeping obligations in some sites.
Pro-actively monitoring activity across the Microsoft 365 ecosystem, including by monitoring the various dashboards, searches, and audit logs, and responding to events.
Learning more about the Microsoft Graph Explorer and the potential to use AI-based options to manage records.
Use the system for other recordkeeping purposes
The Microsoft 365 environment can be used for other recordkeeping purposes as well. For example:
Managing physical records stored offsite in a SharePoint list.
Keeping a record of records (and SharePoint sites or other systems) that have been destroyed, as well as ongoing destruction review and approval processes.
Publishing policies and procedures (in a SharePoint site, not necessarily a communication site).
Communicating information about managing records (communication site).
Archiving social media content (to Exchange mailboxes).
Searching for content stored in other locations or systems including File Explorer and Line of Business systems (via connectors).
Archiving network file share content, where it can be better protected and then subject to retention and disposal outcomes.
Understanding where records are stored (via dashboards and Power BI reporting).
One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.
Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.
Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:
Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.
With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.
The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:
The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.
Note: the box outline colours represent the four different types of site that are likely to be created.
How will the content be accessed?
It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.
Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.
They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.
If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.
It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).
What type of SharePoint site will be required?
Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.
As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.
Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.
The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.
Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.
Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.
Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.
As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.
Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.
Once the site has been created, it should be provisioned with three key elements shown below.
Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.
The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.
SharePoint has a long, twenty-year legacy. Over that period, organisations and third-party developers have developed and implemented a range of solutions and options across SharePoint. Many of these have been in place for a long time.
The arrival of the new look, ‘modern’ SharePoint Online (SPO) in the last five years has created a dilemma for organisations – migrate to SPO and lose the older functionality, or retain some of the functionality by creating ‘classic’ sites in SPO?
This post was inspired from discussions with a number of organisations that were either creating or using classic site templates in SPO – sometimes as the default option – or indicated a resistance to moving to SPO because of the ‘loss of functionality’.
What are modern sites in SPO?
The default options to create in SharePoint Online are either (a) modern team sites linked with a Microsoft 365 (M365) Group (most common) or (b) communication sites.
Modern team sites linked with M365 Groups are created:
From the SP Admin portal (first option ‘Team site’ – above)
By end users from their SharePoint portal (if that feature is not disabled)
When a new Team is created via MS Teams or the MS Teams admin portal
When a new Group is created from Outlook or from the Microsoft 365 admin portal
When a new ‘shared folder’ is created from OneDrive
Modern team and communication sites are almost identical in terms of core functionality. The primary visible difference is that: (a) team sites have the navigation bar at the left and are usually used to store documents and other digital objects; whereas (b) communication sites have the navigation bar along the top (which can be made into a mega menu easily) and are usually used to store information on pages. They may also store documents in document libraries.
Note that M365 Group-based sites use the template ‘GROUP#0’.
What are classic sites in SPO?
Other, ‘classic’ SharePoint site templates can be created from the ‘Other Options’ section of the SP Admin portal.
The options, as grouped, are as follows. Note that for all of them, the SharePoint 2013 experience (look and feel) is used.
Collaboration: Team site (classic experience), Developer Site, Project Site, Community Site
Enterprise: Document Center, eDiscovery Center, Records Center, Team Site – SharePoint Online configuration, Business Intelligence Center, Compliance Policy Center, Enterprise Search Center, My Site Host, Community Portal, Basic Search Center, Visio Process Repository
Publishing: Publishing Portal, Enterprise Wiki
Custom: (if a custom template exists)
SharePoint sites, including using classic templates, can also be created using PowerShell and the ‘New-SPOSite’ command (with additional configuration details, see this Microsoft docs page). The PowerShell command will needs to include the template to be used:
The PowerShell script to create a new classic site is slightly different from a modern Group-based siste, using ‘New-SPOSite’:
Organisations may have genuine technical reasons to want (or need) to create and continue to create some SharePoint Online sites using one of the classic templates. For example:
They may have many sites that were created in the legacy on-premise environment. Upgrading these sites will take time, so it may be easier simply to migrate from one to another ‘as is’.
A site or sites may have integration points with other systems that require the older functionality.
Classic sites should not normally be created simply because of loss of (or changed) legacy functionality (e.g., a preference for older style web parts), or simply a preference to continue to use older functionality.
As much as possible, organisations should aim to create modern sites. In some – or many – cases, this may require the organisation to adopt the new modern options.
Classic to modern functionality changes
Examples of broader-level functionality that has largely been replaced with new functionality, or may be deprecated in the future, include:
Publishing sites were often used for customised intranets, often with multiple levels of sub-sites. These have been replaced by communication sites and hub sites.
Publishing features, including features sometimes used for navigation purposes and linked with terms in the Term Store were a common ‘classic’ option. These have been replaced by new mega menu functionality.
Older web parts on pages have been replaced by modern web parts.
The Document Center and Records Center templates are more or less now superseded by in-place records management functionality, leaving records where they were created or captured.
The Business Intelligence Center has mostly been replaced by PowerBI or similar tools and interfaces.
Community Sites and Community Portals have been replaced by communication sites, Teams and Yammer.
The need for sub-sites has been replaced by creating one or more sites as hub sites and linking other sites to the hub.
Many of these classic options have been around for at least a decade, making it sometimes hard to move to the newer options.
Functionality differences between classic and modern sites
Possibly the most common of the classic sites that continue to be created, however, are classic team sites (‘STS#0’). In some cases, these site templates are the default option. It is not clear why these continue to be created when the STS03 (non M365 Group) template has the same basic functionality:
The same Site Settings.
The same ways to add a page, library or list (from Site Contents or from the gear icon – Add an app).
The same library and list settings (including the ‘Information management policy settings’).
One key difference is the look and feel, something that may become less of an issue with Teams becoming the primary UI to SharePoint.
It is still possible on classic (STS#0) sites to create classic style pages using classic web parts, as can be seen in the screenshot below from a classic (STS#0) site:
What’s the problem with creating new classic sites?
As noted above, however, just because it remains possible to create and use classic functionality isn’t a reason to keep it in face of newer and often better options – especially when end-users won’t even see the site.
From my own experience, anything that remains in an older version of any technology becomes (or is likely to become) technology debt.
It may be appropriate – temporarily – to migrate some older on-premise sites to the classic template until they can be upgraded, but there is no valid reason in my opinion to create new classic sites as a default option for all new sites.
What do you think? Do you continue to create classic sites?
Photographs (or, for that matter, any visual work) can be more powerful as a record of something than a written record. Some of the iconic events in history are only known from the visual record.
Photos (and videos) are ubiquitous in the digital age. They are captured in multiple ways and stored on a range of media including computers, portable drives, mobile devices and the cloud (including online storage and social media).
Digital photos can be easily changed or ‘photoshopped’, to use the more common term.
But records management requires us to ensure the authenticity, reliability and integrity of records. How can we ensure this for photographic records?
This post examines how SharePoint can be used to manage (some) photos as records.
What is a digital photo?
For the purpose of this post, a digital photo is any image that is captured and stored electronically as bits in a recognised image format. It excludes digital photos (including scanned or digitised paper records) that are embedded in PDF files.
To quote from the Wikipedia article titled ‘Digital Image‘, a digital photo is:
‘… an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively. Depending on whether the image resolution is fixed, it may be of vector or raster type. By itself, the term ‘digital image’ usually refers to raster images or bitmapped images (as opposed to vector images).’
According to the definition, a digital photo is a collection of bits set out in a map that a computer interprets to display it on a screen. For more detail on bitmaps, see this Microsoft page ‘Bitmaps‘.
When to use SharePoint to store digital photos
Almost any digital object that can be stored electronically can be captured in SharePoint. But, just because they can doesn’t mean they should.
It is important to keep in mind that SharePoint may be not be the most suitable option for storing digital photographs in general. The volume, size and intended usage of the photos are all likely to influence the decision to use SharePoint. Organisations may need to consider other ways to store and manage digital photos (and other digital media) such as dedicated digital asset management systems.
However, SharePoint may be a good option to store digital photos as records if those photos:
Need to be kept as a record of something.
Support or relate to other content in the same document library. For example, photos that relate to or support building plans (which themselves may be scanned images in PDF form) or construction.
Are about a specific subject. For example, photos of damage to a physical object.
Are relatively low in volume and file size.
Factors to consider before storing digital photos in SharePoint
The following points should be considered before storing any digital photo (or digital media) in SharePoint.
Digital photos are usually stored on devices that capture them with meaningless, device-generated names such as ‘20210423_123192321’, ‘DSC_0330’, or ‘n594015825_1706121_4959’. These types of names provide no clue to the content when the image is saved to SharePoint, as shown below.
Ideally, the device-generated name should be replaced with a more meaningful name as shown below.
We can see from the unique Document ID that it is the same record. The version history also tells us that something was changed but the file size hasn’t changed.
Keep in mind that every change to the file name or any other metadata added to the library will create a new copy (‘version’) of the image. In the version history above, there are now four versions each 3 MB in size.
The Activity section of the information panel to the right (which also provides a preview of the image) also provides some key information to support the version history information:
Does changing the name potentially compromise a key element of metadata? Who should change the name, and when?
Organisations should consider establishing naming conventions for photographs to help with findability and context.
Almost any type of digital object can be stored in SharePoint. SharePoint also supports the ability to view almost every type of contemporary digital photo format (as described in this Microsoft page) so format should not be a problem when it comes to viewing the file.
However, if the organisation plans to store digital photos in older or less common formats, it will need to ensure that these can be accessed for as long as required. This will generally mean ensuring that the appropriate software application is available to anyone who needs to access (open/view) the photos.
Alternatively, consideration should be given to saving the photos in different, more common formats.
The size of a digital photograph may affect its usability as a record. The two photos below are of the same scene but the first photo is very low resolution (= small size, 63 KB), while the second is a section of a much large photo (= large size, 3.7 MB). The second version is clearly a more accurate record.
Digital photos used as records should provide sufficient detail to be useful as records. Accordingly, in most instances, the original photo, not a re-sized version, should be saved.
Reliability as a record
As noted above, photos can be easily modified, sometimes making it difficult to know if it accurately reflects the image it purports to capture. This is also an issue for any form of visual record, including drawings.
Metadata created when the photo was taken and stored in the photo’s properties (including EXIF metadata) can provide evidence of the reliability of the photo as a record, even when the record was stored in SharePoint. The following are the key metadata properties that are usually created and stored with a digital photo:
(Date) Created. The date and time when the photograph was created as a photo.
Date taken. This is, for most digital photos, the same date and time as the ‘created’ date.
(Date) Modified. This should be very close to the original date and time created on the original photograph, but may be several seconds different. If the photograph has been modified (including simple photo adjustments or ‘photoshopped’), it will show a different date and time which might give a clue to its reliability as a record.
Image dimensions, width, height, horizontal and vertical resolution, bit depth
Camera details including F-stop, exposure time, ISO speed, flash mode
Storing digital photos in SharePoint Online
Note that SharePoint previously had the option to create a dedicated ‘Picture Library’. This is no longer necessary as any document library can be used to store any digital object, and SharePoint Online document libraries now have additional options to view the photos.
Any digital photo can now be saved to a standard SharePoint document library, including via the ‘Files’ tab in MS Teams.
Unlike some EDRM systems, SharePoint does not ‘capture’ or extract the details of digital objects when they are saved to a document library. Instead, these remain with the original digital object.
If digital photos are to be saved to a SharePoint Online document library, consideration should be given to using existing or additional metadata columns to describe the image, for example, the ‘Image’ column (small version preview that is stored separately in the Site Assets library >’Lists’ folder > GUID-named folder).
The following screenshot shows a preview image of the main photo.
SharePoint includes three options to view the content in a document library list view: List, Compact List, or Tiles.
The following is an example of a single digital photo stored in a document library (same as the one in the other options earlier) set to the ‘Tiles’ view:
Reliability as a record
Once saved to SharePoint, the photo’s reliability as a record can be determined from version history.
Additional protections may include access/permission controls, retention and/or information security labels.
SharePoint can be used to store (some) digital photos as records, but it should not generally be used as a general storage location for digital photos. Other dedicated digital asset management systems may be more suitable for that purpose.
Before storing digital photos in SharePoint, organisations should establish procedural rules or principles including (re-) naming conventions, format and file size requirements.
SharePoint does not extract and store the metadata from digital objects, which means that digital photos should retain metadata that shows when they were created or modified, and other details about the photograph which will provide evidence to support the authenticity of the photo as a record. Some consideration should be given to adding additional metadata to help describe digital photos as records.
A combination of version history, access controls, information protection and retention labels should provide sufficient controls to ensure the reliability, integrity and authenticity of records – or at the very least provide the evidence of changes that may be made.
There are two sides to the question of authenticity, reliability and integrity:
Knowing if the photo that has been uploaded is a correct record of what it purports to be.
Preventing the photo from modification or deletion or tracking any modifications that may occur.
It might seem impossible to know if a photograph is what is purports to be, but its metadata payload may provide the detail required.
There are three main options in Microsoft 365 to apply recordkeeping classification terms to (some) records:
Metadata columns added to SharePoint sites, including those added to Content Types and/or added directly to document libraries.
Taxonomy terms stored in the central Term Store, including those added as site columns and added to site content types and/or added directly to document libraries. The only difference with the first option is that with the Term Store the classification terms are stored and managed centrally and are therefore available to every SharePoint site.
Retention labels that: (a) ‘map’ to classification terms; (b) are linked with a File Plan that includes the classification terms; (c) are either the same as (a) or (b) and are used in with a Document Understanding Model in SharePoint Syntex; or (d) the same as (a) or (b) and used with conjunction with Trainable Classifiers.
The first two options can only be applied to content stored in SharePoint. Retention labels may be applied to emails and OneDrive content. None of the three options can be applied to Teams chats. Also note that there is no connection between the SharePoint Term Store and the File Plan, both of which can be used to store classification terms.
Defines the meaning of classification from a recordkeeping point of view.
Describes each of the above options and their limits.
Discusses the requirement to classify records and other options in Microsoft 365.
What is classification?
Humans are natural-born classifiers. We see it in the way we store cutlery or linen, or other household items or personal records.
Business records also need some form of classification. But what does that mean? The 2002 version of the records management standard ISO 15489, defines classification as:
‘the systematic identification and arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules represented in a classification system’. (ISO 15489.1 2017 clause 3.5).
The standard also states (4.2.1) that a classification scheme based on business activities, along with a records disposition authority and a security and access classification scheme, were the principal instruments used in records management operations.
The classification of records in business is important to establish their context and help finding them.
Microsoft 365 includes various options to apply classification terms to records.
Metadata columns in SharePoint
The simplest way to classify records stored in SharePoint document libraries is to either create site columns containing the classification terms and add those columns to document libraries, or create them directly in those libraries.
Adding site or library columns is relatively simple. As classification terms are usually in the form of a (hierarchical) list, it is simple to add one choice or lookup column for function and another for activities.
A lookup column can bring across a value from another column when an item is selected; for example, if the look up list places ‘Accounting’ (Activity) in the same list row as ‘Financial Management’, selection of ‘Accounting’ will bring across ‘Financial Management’ as a separate (linked) column.
Default values (or even one value) can be set meaning that records added to a library (that only contains records with those classification terms) can be assigned the same classification terms each time without user intervention.
SharePoint choice or lookup columns do not allow for hierarchical views or values to be displayed from the list view so the context for the classification terms may not be obvious unless both function and activity are listed.
The Term Store
The Term Store, also known as the Managed Metadata Service (MMS) has existed in SharePoint as a option to create and centrally manage classification and taxonomy terms in SharePoint only for at least a decade.
Organisations can create multiple sets of taxonomies or ‘term groups’ (e.g, ‘BCS’ or ‘People’) within the Term Store. Each Term Group consists of the following:
Term Sets. These generally could map to a business function. Each Term Set has a name and description, and four tabs with the following information: (a) General: Owner, Stakeholders, Contact, Unique ID (GUID); (b) Usage settings: Submission policy, Available for tagging, Sort order; (c) Navigation: Use term set for site navigation or faceted navigation – both disabled by default; (d) Advanced: Translation options, custom properties.
Terms. These generally could map to an activity. Each Term has a name and three tabs: (a) General: Language, translation, synonyms and description; (b) Usage settings: Available for tagging, Member of (Term Set), Unique ID (GUID); (c) Advanced: Shared custom properties, Local custom properties.
In the example below, the Term Set (function) of ‘Community Relations’ has three Terms (activities).
Once they have been created in the Term Store, term set or terms can be added to a SharePoint site, either as a new site or local library/list column, as shown in the two screenshots below:
Once added as a site column, the new column may be added to a Content Type that is added to a library, or directly on the library or list.
The primary benefit using the central Term Store terms via a Managed Metadata column is that the Term Store is the ‘master’ classification scheme providing consistency in classification terms for all SharePoint sites.
As we will see below, Term Store terms may be used to help with the application of retention labels (which themselves may ‘map’ to classification terms in a function/activity-based retention schedule).
Using metadata terms from the Term Store is almost identical with using a choice or lookup column. The only real difference is that the Term Store provides a ‘master’ and consistent list of classification (and other) terms.
Term store classification terms, including in Content Types, may only be used on a minority of SharePoint sites.
It is not possible to select a Term Set (e.g., the function level), only a Term within a Term Set.
Only the selected classification Term appears in the library metadata, without the parent Term Set or visual hierarchy reference to that Term Set – see screenshot below. Technically only that Term is searchable. It is not possible to view a global listing of all records classified according to function and activity.
If multiple choices are allowed, a record may be classified according to more than one Term. This may cause issues with grouping, sorting or filtering the content of a library in views.
As we will see below, there is no connection between the classification Terms in Term Sets and the categorisation options available when creating new retention labels via a File Plan. ‘Business Function’ or ‘Category’ choices in the File Plan do not connect with the Term Store.
Term Store terms and Content Types can only be used to classify content stored in SharePoint.
Retention labels in Microsoft 365 can be used in an indirect way to classify records in SharePoint, email and OneDrive because they can be ‘mapped’ to classification elements.
For example, a label may be based on the following elements:
Function: Financial Management
Description: Accounting records
Retention: 7 years
Every retention label contains the following options:
Name. The name can provide simple details of the classification, for example: ‘Financial Management Accounting – 7 years’.
Description for users. This can be the full wording of the retention class.
Description for admins. This can contain details of how to apply or interpret the class, if required.
Retention settings (e.g., 7 years after date created/modified or label applied).
Where the classification terms map to a retention class, the process of applying a retention label to an individual record, email or OneDrive content could potentially be seen as classifying those records against the classification scheme.
The Data Classification section in the Microsoft 365 Compliance portal provides an overview of the volume of records in SharePoint, OneDrive or Exchange that have a specific retention class:
Not every record in every SharePoint document library may be subject to a retention label. Many records (for example in Teams-based SharePoint sites) may be subject to a ‘back end’ retention policy applied to the entire site (which creates a Preservation Hold library).
A retention label applied to a record doesn’t actually add any classification terms to the record.
Retention labels don’t map in any way to Term Store classification terms, except in SharePoint Syntex – see below (but this only applies to SharePoint content).
Retention labels/File Plan combination
The File Plan option (Records Management > File Plan, requires E5 licences) can also be used to add classification terms to a retention label as shown in the screenshot below. Note that there is no link with the Term Store.
Records (including emails) that have been assigned a retention label could, in theory, be regarded as having been classified in this way because the label contains (or references) the classification terms.
When applied to content in SharePoint, OneDrive or Exchange, retention labels linked with the File Plan do not show the File Plan classification terms. It may be possible to write a script that displays all records with the terms from the File Plan, but it may be easier to do this using the Data Classification option described above.
Retention labels/SharePoint Syntex combination
SharePoint Syntex provides a way to apply retention labels to records, stored in SharePoint, that have been identified through the Document Understanding Model process.
As can be seen in the screenshot above, each new DU model allows similar types of records (in the example above, ‘Statements of Work’) to be associated with a new or existing Content Type that can include a Term Store Term – for SharePoint records only – and a retention label. This provides three types of ‘classification’:
Grouping by record type (e.g., Statement of Work, Invoice)
Linking (of sorts) between the records ‘classified’ in this way and a Term Store term added as a metadata column to the Content Type.
Assigning of a retention label. This provides the same form of retention label-based classification described above.
Furthermore, if the Extraction option is also used, data extracted to SharePoint columns can be based on choices listed in the Term Store metadata.
SharePoint Syntex only works for records – and only those records that have some form of consistency – stored in SharePoint.
Trainable classifiers are another way that could be used to identify related records and apply a retention label to those records. Microsoft 365 includes six ‘out of the box’ trainable classifiers that will not be of much value to records managers for the classification of records:
Offensive language (to be deprecated)
The creation of new trainable classifiers requires an E5 licence; they are created through the Data Classification area of the Microsoft 365 Compliance admin portal. Machine Learning is used to identify related records to create the trainable classifiers.
The primary outcome (from a recordkeeping classification point of view) of using trainable classifiers is the application of a retention label to content stored in SharePoint and Exchange mailboxes. It can also be used to apply a sensitivity label to that content.
It is unlikely that every record will be classified according to every classification option.
Trainable classifiers only work with SharePoint and Exchange mailboxes.
Classifying records per workload
The options are summarised below for each main workload:
SharePoint: Use local site or library columns, Term Store terms or retention labels (mapped to a File Plan as necessary), applied manually or automatically, including via SharePoint Syntex or trainable classifiers.
Exchange mailboxes: The only feasible option to classify these records is to manually or auto-apply retention labels that are mapped to a classification, including a trainable classifer.
OneDrive: Manually or auto-apply retention labels mapped to a classification.
Teams. It is not possible to classify Teams chats with the options available.
Is classification necessary?
The classification model described in ISO 15489 and other standards was based on the idea that records would be stored in a central recordkeeping system where they would be subject to and tagged by the terms contained a classification scheme, often applied at the aggregation level (e.g., a file).
Microsoft 365 is not a recordkeeping system but a collection of multiple applications that may create or capture records, primarily in Exchange mailboxes, SharePoint, OneDrive and MS Teams (and also Yammer).
There is no central option to classify records in the recordkeeping sense. The closest options are:
The grouping of records in SharePoint sites (and Teams, each of which has a SharePoint site) and libraries that map to business functions and activities.
The use of metadata, either terms set in the central Term Store or created in local sites/libraries, to ‘classify’ individual records (including emails) stored in SharePoint document libraries. Each item in the library might have a default classification, or could be classified differently.
The use of retention labels that ‘map’ to function/activity pairs in a records disposal authority/schedule. These may be applied, manually or automatically, to content stored in SharePoint, OneDrive and Exchange mailboxes.
Neither of the above may apply, or be applied consistently, to all SharePoint sites, Exchange mailboxes, OneDrive accounts. And neither can be applied to Teams chats.
A different approach to this problem is required, one that likely will likely involve greater use of Artificial Intelligence (AI) and Machine Learning (ML) methods to identify and enable the grouping of records, and provide visualisations of the records so-classified.
Image: Werribee Mansion, Victoria, Australia stairwell (Andrew Warland photo)
Unlike Tasks, To Do items are essentially a personal list of things to do that is accessed from the separate To Do section of Outlook. They are not included in the calendar.
There are two ways to create a new item in the To Do list. The first is to click on the ‘My Day’ option in Outlook and then add a task in the To Do section.
The second is to click on the ‘To Do’ option at the bottom left of the Outlook app, which will open the ‘My Day’ section and allow a new (To Do) task to be created.
A bit confusingly, the Planned section of ‘To Do’ displays tasks that are:
Personal tasks, created as an Outlook calendar tasks but which don’t appear in the individual’s Outlook calendar, only in the To Do calendar.
Assigned to the individual from a Microsoft 365 Group or Team (including ‘Tasks by Planner’).
The difference between the two can be seen below; the first two are personal tasks from To Do, the second two are Group/Team-based tasks. The ‘Assigned to you’ items are the second two under ‘Later’. Note that the Planned section does not include any simple, non-calendar-based, ‘To Do’ items.
Planner is a task-based service originally linked directly with Office 365 Groups (announced in 2014 ‘Delivering the first chapter of Office 365 Groups‘). It was described as ‘a simple and highly visual way to organize teamwork’ within a team – which meant initially an Office 365 Group. It seemed that Microsoft’s vision was to move the creation of Tasks away from Outlook to Planner.
Initially, when a new Office 365 Group was created (including when a new Team was created in MS Teams), it created a Plan. This connection was later removed and so a new Group or Team no longer creates a new Plan.
The following is an example of an empty plan for an Office 365 Group called ‘SharePoint Admin Group’. All the members of the Group (or Team if one exists) would have access to the plan. Plans contain ‘buckets’ or groupings of tasks. The default bucket is ‘To do’ which, in the example below contains a single task ‘Create two new tasks’ (which is outstanding) A separate bucket was created named ‘New Sites’, and it has one completed task.
Changes to Planner
Several changes happened with Planner since 2018:
New Office 365 Groups and Teams did not automatically create a Plan.
Multiple plans could be created for every Office 365 Group or Team.
Tasks by Planner was introduced to Teams (see below) so that every channel can use either the ‘parent’ Office 365 Group Plan or create a new one.
These changes have created some confusing content in the Planner app.
Tasks by Planner in Teams
Microsoft announced in April 2020 that Planner would be renamed Tasks (it is still named Planner a year later).
As noted in the link (by onmsft.com), ‘… the change means that Teams users will soon be able to see their individual tasks and team tasks in a single app from across Teams channels, Planner and Outlook. For mobile users, the change also means that both a list view and a new mobile tasks experience will soon be available in within the Teams app.’
The new Tasks by Planner and To Do was visible from Teams in early 2021 but the relationship between Outlook To Do tasks and Planner Tasks via Teams remained a bit confusing.
When a Team channel is opened, the ‘Tasks by Planner and To Do’ can be added as a tab.
When ‘Tasks by Planner and To Do’ is selected, two options are visible and this is where some of the confusion starts.
Most people are likely to simply click ‘Save’, which creates a ‘Tasks’ tab in the channel and a new Plan. Ideally, they should use an existing plan, if it exists (which it probably won’t – as a result, multiple Plans may be created for each Team and channel.
This is how the new Tasks tab looks like in the Team:
Perhaps it doesn’t matter (for some organisations) how many Plans or Tasks that are set up.
We can now see there are three Plans for the SharePoint Admin Group Team, two in the same (General) channel. The end user can also see tasks that were assigned to him/her. If they click on the ‘Tasks’ option they will see the list of personal calendar- and To Do-based tasks as we saw above in Outlook:
Behind the scenes, in Planner, we can see four plans for the SharePoint Admin Group. Three of these map to the three above, but not the one titled ‘Tasks – SharePoint Admin Group’ which has two completed tasks. But where is it?
Here are the two completed tasks in SharePoint Admin Group ‘Tasks’ plan that don’t exist in the main SharePoint Admin Group Plan, or the other two.
Where are these two completed tasks?
Or, more specifically, why do they not appear in many of the Teams Tasks tabs? There are no private channels in this Team, so I know it’s not hidden in one – and, in any case, you cannot create a new Task list in a private channel.
Just to try to work this out, I created a new Task in that list of Tasks, assigned it to myself. The only place I could find it was in both Teams and Outlook in the ‘Assigned to you’ area.
Tasks, either to remind yourself of things you need to do, or what others need to do, are probably good for specific purposes or Teams. But the ability to create multiple Task lists in Teams channels is just going to create more and more confusion.
But it’s confusing and will likely result in multiple random Tasks/Plans in Planner, even for the same channel.
An electronic document management system (EDMS)supported day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
An electronic records management system (ERMS) was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted.
It could be argued that SharePoint began its life in 2001 as an EDMS and began to include ERMS functionality when SharePoint 2010 was released. So, how is it not an EDRMS?
In simple terms, unlike a traditional EDRMS model that was intended to be the repository for all business records, SharePoint is only one of several repositories that are used to store and manage business records. Instead of centralising the storage and management of records, systems such as Microsoft 365 provide centralised management of records wherever they are stored.
The EDRMS model
Almost all EDRM systems since the mid-1990s have had the following characteristics:
Compliance with a standard. For example, DOD 5015.2 (1997), AS 4390 (1996)/ISO 15489 (2001), ISO 16175 (2010), VERS (1999), MoReq (2001)/MoReq2, etc.
A relational database (for the various functions, metadata and controls (see diagram below) and a separate file share (for the actual content/records), accessed via a user interface.
An expectation that most or all (digital) records would be copied from other systems used to create or capture them to the EDRMS for ongoing storage and protection. This outcome might be achieved through integration with other systems, for example to copy emails to the EDRMS.
The integrated EDM/ERM system model was described in the following diagram in 2008 by the International Standards Organisation committee, ISO/TC171/SC2, in its document ‘Document management applications’:
There are two key problems with the EDRMS model:
They do not (and cannot) realistically capture, store or classify all business records. Emails have remained a problem in this regard for at least two decades. Additionally, there is now a much wider range of born-digital records that remain uncaptured.
Many of the born-digital records that were copied to the EDRMS continue to exist where they were first created or captured.
Any organisation that has been subject to a legal subpoena or Freedom of Information (FOI) request for records will know the limited extent to which EDRM systems provide evidence of all business activities or decision making.
And yet, almost every organisation has a requirement to manage records. There is clearly a disconnect between the various requirements and standards to keep records and the ability to keep them.
To paraphrase from Tony Redmond’s recent (21 March 2021) post ‘SharePoint Hits 20: Memories and Reflections‘, SharePoint was originally designed to allow content (including page-based content in the form of intranets) to be accessed via the web, thereby replacing network file shares.
In practice, most organisations retained their network file shares, built their intranet using SharePoint, and otherwise used SharePoint to manage only some digital content.
It could be argued SharePoint started out as an EDM system and became more like an ERMS when more recordkeeping capability was introduced in SharePoint 2011. But that doesn’t make it an EDRMS in the traditional sense of being a central repository of business records.
The reality is that records are, have been, and will continue to be created or captured in many places:
Email systems. In Microsoft 365, Exchange Online mailboxes are also used to store the ‘compliance copy’ of Teams chats messages.
Network file shares.
SharePoint, including OneDrive.
Other online document management systems, including Google Drive.
Text, chat and instant messages often created in third-party systems, often completely inaccessible or encrypted.
The type and format of a record can vary considerably. For example:
A calendar entry.
A photograph or video recording (including CCTV recording).
The recording of a meeting, in video or transcript form, or both.
Virtual reality simulations.
Social media posts.
3D and digital drawings (e.g., via digital whiteboards).
And, of course, all the data in line of business systems.
Instead of trying to save records into a single EDRM system, Microsoft 365 provides the ability to apply controls over the management of most records where they were created or captured – in email (including archived social media and other records), Teams chat, SharePoint, or OneDrive, and Yammer.
Is there a need for an EDRMS?
There is nothing stopping organisations acquiring and implementing traditional EDRM systems, or even setting up some SharePoint sites, to manage certain high value or permanent records, including some records that can be copied from other create/capture systems (such as email).
But there is also a need to address the management of all other records that remain in the systems where they were created or captured.
In most modern organisations, this requires broader controls such as applying minimum retention periods at the backend, monitoring usage and activity, and the proactive management of disposals. Not trying to copy them all to SharePoint.