From time to time I’m asked about the way records retention policies ‘work’ in SharePoint. A common criticism has been that SharePoint’s retention model is based on applying retention policies to individual records (e.g., documents in a library or individual emails) rather than to aggregations of records, the most obvious of which is a document library.
The idea of storing and managing related records together in a single aggregation derives from the management of paper records – in files, boxes, and series. This model (of aggregations containing all records relating to a given subject) was largely replicated in electronic document management systems (EDMS – many of which were used to register paper files and boxes previously) when they appeared or were modified to manage digital records in the late 1990s.
In fact, many EDM systems did not actually manage records in an aggregation; the actual digital records were stored in a secure network file stored, and presented in the EDMS user interface though a common ‘file number’ (or similar) ID.
In any case, the ability to store all digital records on the same subject together in the one system (e.g., EDMS) was always hampered by the fact that (a) email and documents were created by different systems, (b) stored in different locations (servers), and (c) use of network file shares continued more or less unabated.
The increasing complexity and types of digital records underlines the difficulty of ever storing, let alone managing or applying retention and disposal actions, to them in a single aggregation.
Until recently, Microsoft’s retention and disposal options reflected the fact that applications used to create digital records stored them in different locations (servers) – Exchange and SharePoint. Retention policies targeted individual records stored in those applications, rather than aggregations.
In March 2017, Microsoft introduced a new, single central way to create and apply retention and disposal policies to most Office 365 content, wherever it was stored – Exchange, SharePoint, OneDrive for Business, Office 365 Groups, and Skype for Business.
This post:
- Summarizes the existing ‘out of the box’ retention and disposal options in SharePoint, but not Exchange (see my earlier post on this subject).
- Discusses issues with existing retention and disposal options in SharePoint.
- Describes how the new centrally-managed retention policies and labels can be applied to most content in Office 365.
- Discusses why applying retention policies to individual records rather than aggregations may be a better option in the digital world.
Records managers working in organisations that use Office 365 to manage records should familiarize themselves with the way these new retention policies work.
Note: The details in this post are based on the Australian recordkeeping context, which may be different from your specific location.
SharePoint out of the box (OOTB) retention and disposal options
Until recently, the only available OOTB options to apply retention and disposal actions to SharePoint were to:
- Apply an information management policy to an entire site via the Site Collection Settings. This option is suitable for short-lived sites such as project or closed, archived sites, but less suitable for long-lived team sites which might have a range of different content.
- Create a retention policy using the information management policy settings in Content Types. This option applies the policy to individual records. Content Types also include the ability to ‘transfer’ (actually copy) records after a defined period to another location, such as a Records Center.
- Use a folder-based information management policy. This option requires the default Content Type-based policy on a document library to be changed via Library Settings – Information Management Policy Settings, to Library and Folders.
Another option was to adopt a form of ‘retention in place’ and regard each library as a logical aggregation of records, the equivalent of a ‘file’, and manage retention and disposal manually or using PowerShell scripts to identify libraries for potential disposal based on the last modified date of the records. Some vendors have developed a similar model to manage retention policies on libraries using a central ‘console’.
Applying retention and disposal actions to individual records
Both the Content Type and folder-based options noted above apply the retention policy to individual records in the library, not the library (aggregation/container) as a whole.
That is, disposal was based on a time period after which each individual record was created, modified, or declared a record. The logic behind this model appears to be that a document library may store multiple record types each with different retention requirements. This may not be true for all document libraries, but it usually is for many.
Applying automated disposal actions on individual records (rather than an aggregation of records) is probably counter-intuitive for most records managers. The main concerns, from a recordkeeping (and possibly also archival) point of view are the absence of (a) a documented review and approval process before the records are destroyed, and (b) a metadata record of what was destroyed. That is, the records simple disappear from the document library, removing records that may would be relevant to the context of the original aggregation. This, of course, assumes that all records relating to the subject were stored in a single aggregation which, as noted above, may not always be the case.
Global Retention Policies and Labels in Office 365
In March 2017, Microsoft introduced two new ‘global’ retention options – retention policies and labels – to Office 365. The two options allow organisations to apply centrally set and apply retention policies to the same type of record, in whatever form and wherever they are stored – emails in Exchange, documents and lists in SharePoint, conversations (in Office 365 Groups and Skype)..
Examples of ‘types’ of information could include:
- Corporate records that must be kept for the life of the company.
- Financial records that need to be kept for 7 years.
- ‘Working records’ that could be deleted after a minimum period of time.
- Personnel records or staff files that had to be kept indefinitely.
As Tony Redmond noted in this recent article, these new retention policies build on the type of retention policies first released in Exchange 2010 using folder, system, personal and default tags. The article suggests that organisations that have applied Exchange retention policies may need to consider the impact of these new types of policies. In particular, the ability to move email to archive mailboxes is lost, replaced with a retention policy.
How Retention Policies work
Retention policies in Office 365 are created by authorized users (ideally, records managers) in the Retention section of the Security and Compliance Center.
Creating a new retention policy
Each policy has the following options: Name, Settings, Locations and Preservation Lock.
Name
The name of the retention policy should reflect the class name or number in the records retention schedules so that it can be easily identified and applied to content wherever it can be applied in Office 365 (see below for ‘Locations’).
Settings
The two Settings options are based on two questions:
- Do you want to retain the content?
- If ‘Yes, I want to retain it’ is selected, the choices are either ‘Forever’ or a configurable ‘n days/months/years’ (e.g. 7 years). The administrator must then decide if, once it reaches that point, the record should be deleted or not. If ‘Yes’ is selected, the content will be deleted from where it is currently stored as described in the next two points.
- >>For SharePoint content there are two options when the retention period expires. (1) If the record has not been modified or deleted it will be deleted from the original library where it was stored, and then remain in the two-stage Recycle Bin for up to 90 days. (2) If the content has been modified or deleted, it is transferred to the hidden Preservation Hold library that is created when the retention policy is applied to a SharePoint site and deleted from that library. In this case, the administrator has only 7 days to recover the content before it is deleted permanently.
- >>For Exchange content there are also two options. (1) If the item is modified or permanently deleted by the user during the retention period, the item is copied (if modified) or moved (if deleted) to the Recoverable Items folder. The retention policy process identifies and deletes items whose retention period has expired within 14 to 30 (configurable) days of the end of the retention period. (2) If the item is not modified or deleted during the retention period, the same process runs on all folders in the mailbox and identifies items whose retention period has expired. These items are also permanently deleted within 14 to 30 days of the end of the retention period. (Note: If a user leaves the organization, and their mailbox is included in a retention policy, the mailbox becomes an inactive mailbox. ‘The contents of an inactive mailbox are still subject to any retention policy that was placed on the mailbox before it was made inactive.)
- If ‘No’ is selected, the content will be left in place and must be manually deleted at some point.
- No, just delete the content that’s older than … The options are to delete: (a) after ‘n days/months/years’, and (b) based on when it was created or modified.
The (subtle) difference between these two options is that the first option (Yes) ensures that records are not permanently deleted before the end of the retention period, while the second option (No) just deletes records permanently at the end of the retention period.
Advanced retention settings are also available these allow the administrator to create a search query with specific words phrases, or link the policy with the same sensitive information options found under DLP policies, e.g., financial, medical and health, privacy, and custom.
Locations
The Locations section sets where the policy will be applied. By default this is all locations across Office 365, including content in Exchange, SharePoint, OneDrive, Office 365 Groups and Skype for Business.
- Office 365 has a limit of 10 organisation-wide policies and entire-location policies combined per tenant. Therefore, careful consideration should be given to what specific types of record need a global policy, especially given that not all types of records will be found globally across the organisation.
The alternative option is to apply the policy only to specific locations or users. In most cases this is likely to be Exchange and SharePoint where the majority of key records are created and stored.
- A retention policy that includes or excludes over 1,000 specific users can contain no more than 1,000 mailboxes and 100 sites. A tenant can contain no more than 1,000 such retention policies. According to Microsoft ‘… you can get over these limits by applying either an org-wide policy or a policy that applies to entire locations’.
Retention policies applied to a SharePoint site or OneDrive account result in the creation of a hidden Preservation Hold library as noted above.
Retention policies applied to Exchange user mailboxes apply the policy to the mailbox. For public folders, the retention policy is applied at the folder level.
Preservation Lock
Finally, the administrator has the option to apply a Preservation Lock, which prevents anyone from changing or deleting the policy after it is turned on. This option should only be applied in specific circumstances as it cannot be turned off or made less restricted (by anyone, including the administrator) after it has been applied. .
Review and save
Finally, the new retention policy should be reviewed, may be saved for later, or published.
Labels
A separate option for managing retention and disposal is to use (retention) labels, which should not be confused with security labels. This option is designed to replace the following:
- Exchange Online retention tags and retention policies, also known as messaging records management (MRM).
- In SharePoint Online and OneDrive for Business: (a) in-place records management, (b) the Records Center, and (c) information management policies.
Labels are used to manage retention policies for specific types of content across the Office 365 environment. Labels can be applied automatically to content if it matches certain conditions or keywords (E5 licence only), or manually by users to emails, documents, or Office 365 Group conversations.
See below for the relationship and priority between retention policies and labels.
Who can create labels
Labels are created by individuals (ideally records managers or similar) assigned to a compliance role in the Security and Compliance Admin portal in Office 365.
Creating Labels
Labels are created in the Security and Compliance Admin Portal under ‘Classifications’. Labels may also be created without having an associated retention policy; that is, a label can be created and applied to content as no more than a visual ‘tag’. A policy can be added to it at a later stage.
If the ‘Retention’ option is enabled for labels (on/off switch), a new section appears titled ‘When users apply this label to content’. This section is where the retention policy is defined with two options:
- Retain the content. The choices are either ‘Forever’ or ‘n days/months/years’ (e.g., 7 years). The administrator must decide if, once it reaches that point, the labelled record should be deleted or not. The ‘Yes’ and ‘No’ options are the same as for retention policies, described above.
- If ‘Yes’ is selected, the record will be deleted from where it is stored. Administrators have 93 days to recover records that have not been edited or deleted, or 7 days to records that have been edited or deleted (and moved to the Preservation Hold library).
- If ‘No’ is selected, the content will be left in place and must be manually deleted.
- Don’t retain the content. The choices are to delete (a) after ‘n days/months/years’, and (b) based on when the record was created, modified, or labelled.
If the first option (‘Retain the content’) above is selected a check box option allows the administrator to use the label to classify content as a record. If the content is classified as a record, users are unable to change or delete the content or change or remove the label. They may still, however, edit the metadata.
The final step in the process is to review the settings. Once created, the administrator is returned to the main Labels screen which displays the label that has been created, allowing the administrator to then publish it.
Label limitations when used on a SharePoint document library
There are some limitations to applying a default label to a SharePoint document library:
- It applies the label to all records except those that already have a label and those contained in document sets.
- If the default label is removed, it removes the label from all records except those that have a label and those contained in document sets.
- Labels cannot be applied to folders in SharePoint or OneDrive (but can be applied to folders in Exchange).
- If the record is moved to a different library that has a different default label, it will inherit that label. Conversely, if it is moved to a library with no label, the existing label will be removed.
Note: When labels are published to an Office 365 group, the labels appear in both the group site and group mailbox in Outlook on the web. The experience of applying a label to content is identical to that shown above for email and documents.
What about legal holds?
eDiscovery in Office 365 is based around the creation of ‘cases’ in a SharePoint eDiscovery site. Cases are generally established in response to litigation (or potential litigation) and can be used to search across a range of sources. Once found, the information that forms part of the case can then be placed on hold, overriding any retention policy. However, once the hold is released, retention policies on records continue.
For more information on this subject, see:
What’s the relationship between retention policies and labels?
Retention policies and labels do the same thing but the former is more likely to be set centrally, while the latter is set by the end user. This means that a record could have more than one retention policy applied to it.
According to Microsoft’s documentation (link below), records will be retained until the end of the longest retention period applied to it, regardless of whether that policy was based on the retention policy or the label.
Are retention policies and labels better than previous retention options?
One of the primary benefits of the new retention policy regime in Office 365 is that it enables organisations to apply retention policies centrally rather than do this separately for each application (e.g., Exchange, SharePoint) as was the case until recently. It also allows end users to apply retention policies via labels.
Retention and disposal continues to be based on the individual record, or type of record (as defined by the policy or label), not logical aggregations or containers of records such as a document library.
As noted above, the concept of an aggregation that contains all the records on a given subject is ill-suited to the digital world. The reality is that records may be created using different applications (e.g., email in Exchange, document, list item or page in SharePoint, conversation in Groups, discussions in Skype etc) and stored in multiple application locations (e.g. in Exchange folders, SharePoint libraries, etc).
The dilemma for many records managers using Office 365 is how to store or manage records together in context, including based on the organisation’s File Plan or Business Classification Scheme (BCS) terms. The need to keep records together has been the driver behind the integration of EDRM systems with email applications, allowing email to be ‘captured’ in the EDRM along with other types of documents. This has rarely been successful in practice and, in most cases, emails are duplicated and remain stored in the email server.
The new Office 365 retention policies, including those applied as labels to specific types of content, may well be the answer to this dilemma. Rather than try to capture all types of records (e.g, document email, list item, conversation) in a single aggregation or container, Office 365 allows the option for them to be stored wherever the user prefers, with the same retention policy applied.
If necessary, all records with the same label can then be found using a content search in the ‘Search and Investigation’ section of Office 365.
In my view, there are still some shortcomings in basing retention policies on individual record types:
- Individual documents, rather than logical aggregations of documents, will be continue to be subject to disposal actions.
- Records that may provide context to other records (including those stored in different locations) may be destroyed.
- Appraisal options may be limited and appropriate review and approval steps before disposal may not be possible.
- Disposal actions may be automatic and unrecoverable.
- There may be no record kept, including the metadata, of the individual records that were destroyed.
- It is not known how courts might view the automatic disposal of records without prior review and approval.
Final thoughts
The new Office 365 records retention policy and label options centralise the management of retention and disposal for most types of records across Office 365, reducing complexity.
Retention and disposal continues to be based on individual records rather than aggregations, but this may be better suited to the digital world in which aggregations of records may not always be achievable.
Records managers working in organisations using Office 365 need to understand and provide guidance to IT on how records retention schedules can be applied as retention policies, and how they can be directly involved in decisions regarding the new options.
For more information: –
https://support.office.com/en-us/article/Overview-of-labels-af398293-c69d-465e-a249-d74561552d30
