Posted in Compliance, Conservation and preservation, Electronic records, Governance, Information Management, Information Security, Legal, Records management, Retention and disposal, Security

Destroying digital records – are they really destroyed?

Most people should be aware that pressing the ‘delete’ option for a file stored on a computer doesn’t actually delete the item, it only makes the file ‘invisible’. The actual file is still accessible on the disk and can be retrieved relatively easily or using forensic tools until the space it was stored on is overwritten.

Traditional legacy electronic document and records management (EDRM) systems have two components:

  • A database (e.g., SQL, Oracle) where the metadata about the records are stored
  • A linked file share where the actual objects are stored, most of which are copies of emails or network file share files that remain in their original location.

In most on-premise systems, email mailboxes, network file shares, and the EDRMS database and linked file share are likely to be backed up.

When a digital record comes to the end of its retention and is subject to a ‘destruction’ process, how do you know if the record has actually been destroyed? And even if it is, how can you be sure that the original isn’t still stored in a mailbox, network file share, or a back up?

This post examines what actually happens when a file is ‘deleted’ from a Windows NT File System (NTFS), and questions whether digital records stored in an EDRMS are really destroyed at the end of the retention period.

The Windows NTFS Master File Table (MFT)

Details of every file stored on a computer drive will be found in the NTFS Master File Table (MFT).

In some ways, the MFT operates like a traditional electronic document management system – it is a kind of database that it records metadata about the attributes of the digital objects stored on the drive. These attributes include the following:

attriblist

As noted in the diagram above, the details stored by the MFT include the $File_Name and $Data attributes.

  • The $File_Name attributes include the actual name of the file as well as when it was created and modified, and its size.  This is the information that can be seen via File Explorer and is often copied to the EDRMS metadata.
  • The $Data attribute contains details of where the actual data in the file is stored on the disk (in 0s and 1s) or the complete data if the file is small enough to fit in the MFT record.

If the MFT record has many attributes or the file data is stored in multiple fragments on a disk (for example as a file is being edited), additional MFT ‘extension’ records may be created.

When a file is deleted, the MFT records the deletion.

  • If the file is simply deleted, the record will remain on the disk and can be recovered from the Recycle Bin.
  • If the file is deleted through SHIFT-DEL or emptying the Recycle Bin, the MFT will be updated to the ‘Deleted’ state and update the cluster bitmap section to set the file’s cluster (where the data is stored) as being free for reuse. The MFT record remains until it is re-used or the data clusters are allocated in whole or part to another file.

So, in summary, ‘deleting’ a file does not actually delete it. It may either:

  • Store the file in the Recycle Bin, making it relatively easy to recover, or
  • Change the MFT record to show the file as being deleted but leave the file data on the desk until it is overwritten.

How does an EDRMS store and manage files?

The following summary relates to a well-known Electronic Document and Records Management System (EDRMS). Other systems may work differently but the point is that records managers should understand exactly how they work and what happens when electronic files are destroyed at the end of a retention period.

Most EDRM systems are made up of two parts:

  • A database (SQL, Oracle etc) to store the metadata about the record.
  • An attached file store that stores the actual digital objects.

When EDRM systems are used to register paper or physical records (files and boxes), only the database is used.

When digital records are uploaded to the EDRMS:

  • The metadata in the original file, including the file type, original file name, date created, date modified and author are ‘captured’ by the system and recorded in the new database record.
  • Additional metadata may be added, including a content or record ‘type’.
  • The record will usually be associated with a ‘container’ (e.g., ‘file’). This containment makes the record appear to be ‘contained’ within that container, whereas in fact it is simply a metadata record of an object stored elsewhere.
  • The original record filename is changed to random characters (to make it harder to find, in theory) and then stored on the attached (usually Windows NTFS) file store, often in a series of folders.
  • A link is made between the database record and the record object stored in the file store (the MFT record).

When the end-user opens the EDRMS, they can search for or navigate to containers/files and see what appears to be the digital objects ‘stored’ in that container/file. In reality, they are seeing a link to the object stored (randomly) in the file store.

What happens when an EDRMS record is destroyed?

If there is no requirement to extend their retention, or keep them on a legal hold, records may be destroyed at the conclusion of a retention period.

For physical records, this usually means destroying the physical objects so they cannot be recovered, a process that may include bulk shredding or pulping.

For digital records, however, there may be less certainty about the outcome of the destruction. While the EDRMS may flag the record as being ‘destroyed’ it is not completely clear if the destruction process has actually destroyed the records and overwritten the digital records in a way that ensures its destruction to the same level as destroyed paper files. 

Also:

  • If the original associated NTFS file share becomes full and a new one is used, the original is likely to be made read only.
  • There is likely to be a backup of the EDRMS.
  • The original records uploaded to the EDRMS probably continue to exist on network files shares, in email, or in back up tapes.
  • Digital forensics can be used to recover ‘deleted’ files from the associated file share.

Consider this scenario:

  • An email containing evidence of something is saved to a container in an EDRMS.
  • The container of records is ‘destroyed’ after the retention period expires.
  • A legal case arises after the container is ‘destroyed’
  • A subpoena is made for all records, including those specific records.
  • Has the record actually been destroyed, or could it still be recoverable, including from backups or the digital originals?

Is it really possible to destroy digital records, and does it matter?

Yes, records can be destroyed by overwriting the cluster where the record is kept, and some EDRM systems may offer this option.

But:

  • Do EDRM systems overwrite the cluster when a digital record is destroyed in line with your records retention and disposal authorities, or simply mark the record as being deleted, when it is still technically recoverable?
  • Could the record still exist in the network file shares or email, or in backups of these or the EDRMS?
  • Might it be possible to recover the record with digital forensics tools?
  • Does it matter?

It might be worth asking IT and your EDRMS vendor.

References:

 

 

Posted in Electronic records, Exchange Online, Governance, Information Management, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Setting up SharePoint Online to manage records (as part of Office 365) – Part 1/3

This is the first of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

This post focuses on the recordkeeping related elements in the Office 365 and Compliance admin portals:

  • Office 365 Admin – Licences, Roles and AD Groups (including Office 365 Groups)
  • Compliance Admin – Retention labels and policies (and some more options)

The second post focuses on SharePoint Online Admin centre configuration.

The third and last post focuses on SharePoint site collection provisioning and configuration to manage records

Office 365 admin center

O365AdminPortalUsersRolesGroups

The main elements that impact on the management of records in Office 365 are Users (for licences), Roles and Groups, as can be seen in the screenshot.

Users – licencing and applications

Organisations that acquire Office 365 will generally have the relevant licences required (a) to set up and administer SharePoint Online, and (b) for users to use it (and OneDrive for Business).

This post assumes that organisations will have at least an E3 licence which includes SharePoint for end users, visible as an app when they log on to https://office.com, along with all other applications included in the licence, for example Exchange/Outlook, OneDrive for Business, MS Teams and so on. End users with access to these items will also be able to download and use the equivalent mobile device apps.

Roles

The three key roles that impact on the management of records in SharePoint are as follows:

Global Admin (GA)

Global Admins:

  • Are responsible for managing the entire Office 365 environment. This includes creating new Groups (Security Groups, Distribution Lists and Office 365 Groups).
  • Are responsible for assigning key roles, including the SharePoint Administrator and Compliance Administrator (and other roles).
  • May have responsibility for, and/or the skills and knowledge required to set up and administer SharePoint Online and create new sites for the organisation.
  • May also be able to create and publish retention policies in the Compliance admin portal.

Note – Organisations that outsource the administration of Office 365 should always have at least one GA account to access the tenant if ever required. If they don’t have a log on, they should have or acquire a very good understanding of the access and privileges afforded to the outsourced company. 

SharePoint Administrator (SP Admin)

The SP Admin role will usually be a ‘system’ role that is responsible for managing the SharePoint environment, including OneDrive for Business. As noted above, a GA with the right skills can also manage the SharePoint environment. 

Generally speaking, SharePoint Administrators will focus on the technical and configuration aspects of SharePoint. They are not usually responsible for confirugint SharePoint to manage records, managing records, or creating and publishing retention policies. This role usually falls to either the GA or Compliance Administrator.

Compliance Administrator

The Compliance Admin role is responsible, among other things, for the creation and publishing of retention labels and policies in the Compliance Admin portal. A GA can perform this role (along with all other roles) if required.

Compliance Admins will usually be responsible for disposition reviews linked with retention labels, and be involved in eDiscovery cases.

The Compliance Admin can search and view the audit logs for all activity across Office 365 and can carry out broad content searches with the ability to export the content of those searches. As this role is relatively powerful, it should be limited to key senior individuals in the organisation.

Office 365 and Security Groups

Office 365 Groups are Azure/Exchange objects just like Security Groups and Distribution Lists. Accordingly, there should be controls around their creation, including naming conventions.

As every Office 365 Group has an associated SharePoint site, organisations should consider restricting the ability for end users to create Office 365 Groups, and only allowing Global Admins and members of a Security Group to do this. Neither SharePoint Admins or Compliance Admins would normally create AD Groups.

If the ability to create Office 365 Groups is not restricted, an Office 365 Group will be created with an associated SharePoint site whenever:

  • A new Team is created in MS Teams.
  • A new Group is created from Outlook.
  • A new Yammer Group/Community is created.

External sharing

The ability to share content externally from SharePoint and OneDrive for Business is controlled from the Office 365 Admin portal. This is a global setting that can be disabled by the Global Admins if required.

It is assumed, for the purpose of this post, that that setting is enabled to allow external sharing.

Note that enabling external sharing at the global level does not enable it globally for all SharePoint sites; sites must be individually modified to allow it.

Compliance Admin

The Compliance admin portal can be accessed by the GAs and also the Compliance Admins (and some other roles). It is where retention labels and policies are created (in line with the corporate file plan/BCS) and published, and disposition reviews are undertaken, so records managers need access.

Other options in this section that relate to the management of records include the audit logs, content search and eDiscovery.

Retention policies

Retention policies may be applied to all the key workloads in Office 365 where records are stored:

  • Exchange Online
  • SharePoint Online
  • OneDrive for Business
  • MS Teams
  • Office 365 Groups

Retention labels published as retention policies are visible to and can be applied by end-users. Generally these are more likely to be applied at the document library level rather than to individual records, or in mailboxes or OneDrive for Business.

Retention policies that are not based on labels may be applied to all, or parts of, the four workloads listed above. For example, they may be applied to all, or a sub-set of Exchange mailboxes or OneDrive for Business accounts, or SharePoint sites. Retention policies may also be applied to individual or team chats in MS Teams.

Organisations seeking to use retention policies in Office 365 should understand how these work, have a plan for their implementation, and keep track of what has been applied where.

  • Retention policies for all mailboxes or all ODfB accounts may replace previous on-premise backup options for those workloads. It is unlikely that end-users will (or will want to) apply retention labels published as policies to individual emails or folders in mailboxes or OneDrive.
  • SharePoint sites are likely to have either or a combination of explicit and implicit/invisible retention policies. Implicit, single period retention policies may be more suitable for entire smaller, short-lived SharePoint sites. Explicit retention policies may be more suitable for the diverse range of content on more complex and long-lasting sites. Some sites may be created and populated around the need to keep a particular type of record for a long period of time – for example, employee records.

Audit logs

The Office 365 audit logs are found in the Compliance admin portal. For an E3 licence, the content in the logs is stored for 90 days.

As audit logs are an important element in keeping records, organisations may need to consider ways to retain this content for a longer period.

Note – SharePoint document libraries record the name of anyone who edited a document (and also previous versions), but they don’t record the name of anyone who simply viewed it. SharePoint lists also include audit trails, making it possible to track changes in individual rows of a list.

Content searches and eDiscovery

The Compliance admin portal provides two similar options to search for content across Office 365. Both the Content Search and eDiscovery options provide the ability to establish a ‘case’ that can be run more than once.

The eDiscovery option provides the added ability to put content on Legal Hold. Advanced eDiscovery is available with a higher licence.

Next

Click on the links below to read the next two posts:

  • SharePoint Online Admin centre configuration.
  • SharePoint site collection provisioning and configuration to manage records.
Posted in Compliance, Electronic records, Exchange Online, Information Management, Microsoft Teams, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Understanding and applying retention policies to content in MS Teams

This post highlights the need to understand how retention works in MS Teams, why it may be related to how long you keep emails (including for backup purposes), and why you need to consider all the elements that make up an Office 365 Group when considering how – and how long – to retain content in MS Teams.

Overview of retention in MS Teams

If you are unfamiliar with how retention works with MS Teams, these two related sites provide very useful detail.

overview_of_security_and_compliance_in_microsoft_teams_image1
Image from the first link above – Security Compliance Overview

The quote below from the second link is relevant to this post:

‘Teams chats are stored in a hidden SubstrateHolds folder in the mailbox of each user in the chat, and Teams channel messages are stored in a hidden SubstratesHolds folder in the group mailbox for a team. Teams uses an Azure-powered chat service that also stores this data, and by default this service stores the data forever. With a Teams retention policy, when you delete data, the data is permanently deleted from both the Exchange mailboxes and the underlying chat service.’

and

‘Teams chats and channel messages aren’t affected by retention policies applied to user or group mailboxes in the Exchange email or Office 365 groups locations. Even though Teams chats and channel messages are stored in Exchange, they’re only affected by retention policies applied to the Teams locations.’

In summary:

  • One-to-one chat in MS Teams is stored in a hidden folder of the mailbox of each user in the chat. Documents shared in those chats are stored in the OneDrive for Business of the person who shared it.
  • Group chat in Team channels is stored in a hidden folder of the mailbox of the associated Office 365 Group – and also in an Azure chat service. Documents are stored in the Office 365 Group’s SharePoint site (other SharePoint site libraries may also be linked in a channel).

Another quote from the same post:

‘In many cases, organizations consider private chat data as more of a liability than channel messages, which are typically more project-related conversations.’

Teams content is kept in mailboxes, retention may be similar

Typically, in the on-premise past, organisations will have backed up their Exchange mailboxes (and possibly also enabled journaling, to capture emails), for disaster recovery, ‘archiving’ and investigations. Unless a decision is made to invest in cloud back-ups, Office 365 retention policies may also be applied to Exchange mailboxes, effectively replacing the need to back them up. Retention policies applied to Exchange mailboxes don’t affect the teams chat folder.

Organisations should probably apply the same retention period to both emails and Teams chats as they do to email mailbox backups now. That is, if mailboxes are typically kept for 7 – 10 years after the person leaves the organisation, then keep the Teams chats for the same period.

Note that, even if a poster deletes an item (if that option is enabled), it will still be retained if there is a retention policy.

Suggestions for retention in MS Teams

As there can be different retention requirements, depending on the subject matter, here are some suggestions for retention:

  • One-to-one chat is like email, you will never know everything that is being said or sent. So a single retention policy that mirrors email would be appropriate.
  • Teams chat is more likely to be about the subject of the Team, which is based on an Office 365 Group, its own mailbox, and has a SharePoint site. In this case, you could consider a retention policy applied to all Office 365 Groups or specific Groups – for example ‘Project Groups’, then ensure that the retention policy or policies cover all aspects of the Office 365 Group (mailbox, team chat, SharePoint).
  • If all the records relating to a particular subject matter (including email, chat and documents) must be retained for 25 years, then you need to understand all the options.

It underscores the need to plan carefully for retention management for all the key workloads in Office 365.

Posted in Classification, Compliance, Electronic records, Governance, Information Management, Office 365, Products and applications, Records management, Retention and disposal, SharePoint Online

Shifting the paradigm for managing records – from EDRMS to Office 365

Computer systems used to to manage electronic documents and records, commonly known as ‘EDRMS’, have been around for at least 20 years.

Many (but not all) of these systems developed from electronic databases originally used to register and manage only paper records, replacing the old paper registers (hence ‘Registries’).

How does an EDRMS work

A common theme with most EDRM systems is that they describe (via metadata) and provide some kind of visual ‘file’ or ‘folder’ structure for digital objects, almost always stored in a linked network file store.

To store records in this way, EDRM systems required end-users to upload a copy of a digital object (document, email, photograph) to a pre-defined digital container, corresponding to a ‘file’ or ‘folder’. The digital file might have be assigned a range of metadata including the classification (business function and activity) or file plan details, title, business owner or area, and retention information.

Once an object was uploaded, end users were required to add metadata about the object, including the object ‘title’ (if it didn’t copy the original title). Additional metadata fields, for example ‘Document Type’, might also be required.

The system recorded the date and time the object was uploaded and who uploaded it. As noted, the system might copy some of the uploaded object’s metadata, for example the default title, date created and author.

The uploaded document then ‘became’ a record, visible ‘within’ a digital container (‘file’) along with other records.

EDRMModel2

EDRM systems had (at least) three weaknesses:

  • End-users were required to upload the records to the EDRMS, and to one correct container (file/folder)
  • The EDRMS contained a copy of a digital object that almost always remained in the original storage location (email, network file share)
  • The EDRMS tended to be based on records as documents (including emails, and sometimes photos). Newly evolving forms of record such as text messages, social media posts and new digital forms were difficult to upload without costly add-ons that didn’t necessarily capture everything

These weaknesses meant that:

  • End users avoided uploading records because it was extra work (uploading and then adding metadata)
  • The EDRMS contained only a percentage of all potential records stored in any location
  • The original copies of records, remained in email and network file shares

There were exceptions to this situation, but most (and very much in the minority in terms of total volume) involved the requirement to meet compliance obligations to capture certain types of records.

The Office 365 model

Microsoft took a different approach with the approach to records management in Office 365.

Instead of centralising the storage of records in one system or location (with the weaknesses described above), records in the Office 365 environment generally remain in their original location (Exchange Online, SharePoint Online, OneDrive for Business, MS Teams), where they are covered by an overarching records management framework.

O365RMModel
The Office 365 model for records management

What this means is that records can be stored in any of the above locations and managed in those locations through (among other things):

  • User types, licences and roles set in the Office 365 admin portal
  • Retention and other controls set in the Office 365 Security and Compliance admin portal/s (the two were split in early January 2020).

How the paradigm shifted

The paradigm has shifted from (a) an attempt to manage records in a single system where not everything is captured and originals remain in place in email and network file shares, to (b) the distributed management of records where originals remain in place (assuming SharePoint and OneDrive are used instead of network file shares and personal drives, and email remain in Exchange) and records are managed through ‘global’ settings.

The new paradigm does not exclude the ability to store (or aim to store) digital records in a single location – SharePoint Online (including for specific compliance reasons), but it provides the opportunity manage records wherever they are and use a range of additional tools to manage content from creation through to disposal.

Why the new paradigm matters

The new paradigm is likely to be counter-intuitive to many records (and other information) managers. Records management training for many years has been focused on the idea of storing and managing records in a central location with specific controls (classification, metadata and retention).

But the reality is that there are now too many digital records, and too many types of digital records, to ever expect these to be all stored in an EDRMS. And, even if only some are, what about all the others? Has a legal subpoena ever been focused only on records stored in the EDRMS?

Plan to manage records

Many organisations have acquired and are implementing Office 365, sometimes at the expense of the traditional EDRMS. It doesn’t take long for end-users to adopt the new technology because it is so easy to use.

Any suggestion that specific records now need to be copied to the EDRMS seems to be counter-intuitive. And yet, that is how some records managers continue to see Office 365 – as yet another source of records to be uploaded to the EDRMS. It is not a viable plan.

Records managers need to be at the forefront of planning for Office 365, in particular managing content across the four primary workloads. Records managers should be able to provide advice on:

  • The architecture of SharePoint Online
  • Controls around the creation of sites, including naming conventions and the ongoing management of sites
  • The structure of SharePoint Online sites, document libraries and metadata in particular
  • The retention model for Exchange Online, SharePoint Online, OneDrive for Business, MS Teams. This includes understanding existing disaster recovery arrangements and potentially replacing them with retention policies.
  • Disposal actions
  • Other compliance obligations

Plan for change

Moving away from the centralised management of records in an EDRMS to a less visible (for end-users) decentralised model, or even implementing Office 365 without any other previous document and records management system, requires careful change management.

End users (and records managers) used to the idea of uploading records to a central EDRMS may find the new ‘invisible’ and decentralised model of recordkeeping unusually simple (to the point of disbelief).

Consequently, additional re-assurance, training and awareness sessions, may be required to demonstrate and confirm how the records are managed in the new environment. There is potential for some ‘push back’ as, although it requires very little end-user effort, it manages more records than ever before, including in ‘personal’ spaces such as mailboxes and personal drives.

IT will also need to be involved as disaster recovery processes, such as backing up email and network file shares, may no longer be required.

For end users who have never had to use an EDRMS, change management activities might focus more on improving awareness and knowledge about how records will be managed in the future, including in ‘personal’ spaces.

 

Posted in Digital preservation, Disasters, Electronic records, Governance, Information Management, OneDrive for Business, Records management, Retention and disposal

Managing the retention of content stored in OneDrive for Business accounts

The methods available to manage the retention of content stored by end-users in their Office 365 OneDrive for Business (ODfB) accounts are not always well understood.

Organisations may initially default (in their thinking) to backing up the content because that’s what was always done in the past. A change of thinking may be required.

This post:

  • Explains some of the key differences between ‘home drives’ and ODfB accounts.
  • Highlights the need for organisations to understand their business requirements for retention of ‘personal’ content, and not assume traditional backup methods are the only option.
  • Also highlights the need for organisations to understand the potential risks (and potentially unnecessary additional costs) associated with backing up Office 365 content.
  • Describes two simple options for the retention of content stored in ODfB accounts.
  • Suggests that organisations can probably use a combination of a single Office 365 retention policy and a change to the storage retention period for inactive accounts, instead of backups to achieve the same outcome.

What are ‘home drives’?

In many organisations, home drives are usually a dedicated area on a network file share designed to allow end-users to store ‘working’ documents and ‘personal’ content.

Using the network file shares for home drives ensures that the content stored in them is backed up as part of standard disaster recovery processes while the user is still active (for disaster recovery and to recover deleted items) and still accessible (as an ‘archive’) after they leave the organisation.

In some organisations, home drives may instead be an area on the user’s computer (C drive). Any content stored on local computers is not backed up.

Generally speaking, home drives – whether in the NFS or on the user’s computer, are not accessible once the end-user leaves the office. This has given rise to the fairly regular use of USB storage devices or uncontrolled, internet-based, file storage systems such as DropBox.

How is ODfB different from home drives?

In organisations that implement Office 365, ODfB is the replacement for ‘home’ or ‘personal’ drives.

Although they offer similar functionality for end-users (in terms of the ability to access the content from File Explorer), ODfB accounts are fundamentally different in several ways.

  • The content can be accessed on almost any device. No VPN is required.
  • With Windows 10 devices, the content is synced to and can be accessed via File Explorer. This makes ODfB an almost identical replacement for existing home drives in terms of look and feel, and functionality (plus even more functionality, such as the ability to share directly).
  • There is no accessible back up – Microsoft is entirely responsible for disaster recovery. If organisations want to back up ODfB accounts from Office 365, they will need to acquire a third-party product. The ability to establish retention for the content (last two dot points below) may make the need for back up redundant.
  • There is a 90 day Recycle Bin accessible via the browser-based interface. This allows end-users to restore the content they deleted themselves within that time-frame.
  • Organisations can set a storage retention period that will apply once the end-user leaves and their account is deactivated.
  • Organisations can also set a retention policy that will prevent the deletion of content while the user remains active.

Both the last two options are the subject of this post.

Access to and retention of home drives vs ODfB accounts

In many organisations, the content stored by end-users in their home drives is considered to be ‘private’ to them, despite the system being owned by the organisation.

While they can be accessed easily by network administrators with elevated privileges, it is not uncommon (often for audit purposes) for IT to have to seek special approval from someone senior to access the content of a home drive either while the end-user is still employed or after they have left. In these cases, IT will either access the active drive or request the back up tape to restore the content. 

The content in home drives, when backed up, remains as long as the backup media is accessible.

In Office 365, Global Administrators can access the ODfB accounts of any active user. They do this by going to the Office 365 Admin portal and, under the ‘Users’ section, clicking the end-user account name and then going to the ‘OneDrive’ tab where the option to ‘Get access to files’ is displayed’. Any access to ODfB accounts, by anyone (including Global Admins) is recorded in the audit logs.

[Note: At at January 2020, the old ‘My Sites’ options in SharePoint still exists. These options allow the Global Admins or SharePoint Admins to assign someone, or a Security Group, as a Secondary Admin for all ODfB accounts. This option is largely redundant because Global Admins can access the content anyway.]

The default retention period for ODfB content is 30 days after the end user’s account is disabled.

What exactly are you trying to achieve?

As noted, there are some fundamental differences between ‘home drives’ and ODfB.

Consequently, organisations ideally should re-examine their business requirements for access to and the retention of ‘personal content’ both while the user account is active and when it is made inactive, and not assume that old backup option remain valid.

For example, consider the use of backup tapes:

  • The primary purpose of backup tapes is to support disaster recovery. These made sense when IT owned the servers, but it makes less sense when Microsoft own them and are responsible for disaster recovery. Is Microsoft’s disaster recovery capability sufficient or suitable?
  • Backup tapes were (and still are) often used as a type of ‘archive’, allowing organisations to recover data from active and inactive home drives for an indefinite period of time.

The bottom line is – what business outcome/s do you want? Generally, these are likely to be:

  • The ability to recover content stored on personal drives after a disaster (not just when the end-user has deleted something).
  • The ability to access and retain content while the user is active or after they become inactive.

An additional business requirement might be to reduce the use of ‘home drives’ for business related content.

Retention options for content stored in ODfB

ODfB ships with two default retention options:

  • Recycle Bin. Any ODfB content deleted by an end-user goes to the Recycle Bin for 90 days.
  • Inactive content retention. When an end-user accounts is deactivated, the content remains accessible for a default period of 30 days.

Neither of these two options on their own, without modification, is likely to meet business requirements to achieve some form of back-up equivalent capability and the ability to access content in ODfB for a period of time.

It is likely that most business requirements (to replace backups) will be met instead via a combination of the following:

  • Creating a single Office 365 retention policy applied to all ODfB accounts that prevents content in those accounts from being deleted for a given period of time.
  • Extending the default retention period for the content in deactivated accounts from 30 days to a much longer period, for example 7 years.

Office 365 Retention Policy

To ensure that content is kept (and accessible, even after being ‘deleted’ by the user) while the user is active, and after they leave, (a) create a single Retention Policy in the Office 365 Compliance portal, ‘Information Governance’ section and (b) apply it to all ODfB accounts by choosing ‘https://tenantname-mysharepoint.com’.

ODfBRetentionPolicy.JPG

Once published, the retention policy creates a ‘Preservation Hold library’, visible only to the Global Admins, that stores any content that is modified or deleted by the end-user during the retention period.

At the end of the retention period, the content in the Preservation Hold library and anything else that has reached the end of the retention period is sent to the Recycle Bin where it is kept for 90 days before being permanently deleted.

ODfBPresHoldLib.JPG

This type of retention policy effectively replaces the need for a back up of home drives, provided the organisation:

  • Accepts the risk that Microsoft may not be able to recover all or some of the content in the case of a disaster. Note that this risk also applies to Exchange, SharePoint and MS Teams content.
  • Understands that, if it decides to attempt to back up ODfB, restoring from back up may not be as simple as it used to be when the organisation owned and managed the relevant servers. What, exactly, will you back up to, and how will you read the data?

ODfB Storage Retention

The second retention option relates to the ODfB accounts of departed users, or inactive accounts.

ODfB includes the option to retain files in ODfB for a specific period of time after the end-user account is deactivated. This is set in the ODfB Admin portal under ‘Storage’.

ODfBStorage.JPG

At the end of the period of time specified, the content is sent to the Recycle Bin after which it is deleted permanently.

Summary

Many organisations are likely to approach the retention of ODfB content in the same way they did for home drive content, by considering backup options first, often ‘because that’s what we’ve always done’.

Organisations implementing Office 365 should:

  • Define their business requirements for the retention of home drive/ODfB content
  • Examine, understand and consider if retention options in Office 365 result in the same outcome
  • Understand the potential risks of relying on Microsoft to provide a reliable service including in a disaster situation
  • Understand the complexity (and risks) of backing up (and recovering) content from Office 365.

In many cases, retention options in Office 365 may provide the required outcome at a much lower cost.

 

 

 

 

 

Posted in Compliance, Electronic records, Governance, Information Management, Office 365, Products and applications, Records management, Retention and disposal, SharePoint Online

Planning for records retention in Office 365

Almost ten years ago, in January 2010, I published a post here about the origins of the statute of limitations. The post had the following introduction:

The retention – and eventual disposal – of records is a common business practice, despite occasional concerns about what gets destroyed. Justice Scalia, in Arthur Andersen LLP v United States (No. 04-368, 2004) said as much about the destruction of records relating to Enron by Arthur Anderson:

‘… we all know that what are euphemistically termed “record-retention programs” are, in fact, record-destruction programs, and that one of the purposes of the destruction is to eliminate from the files information that private individuals can use for lawsuits and that Government investigators can use for investigations.’

Almost ten years on, it still seems an appropriate way to introduce a post about the retention of records, this time in relation to records stored in Office 365.

Bottom line – you need to plan for it, and make sure your legal team is consulted.

Blatant plug for a great book

Before you read further, I recommend you have a look at the comprehensive e-book ‘Office 365 for IT Pros‘. This 1000+ page ebook includes, in Chapter 19 – Office 365 Data Governance, a comprehensive description of how to create, apply and manage retention policies in Office 365.

What do we mean by retention?

The retention of records is generally based on business, legal or regulatory requirements to keep certain records for a minimum period of time. In the case of government records, there may also be an archival requirement.

The retention period may relate to or be based on a statute of limitations that governs when potential legal actions expire. For example, simple contracts generally need to be kept for a minimum of six or seven years (depending on jurisdiction after they expire. More complex contracts (including deeds) may need to be kept for much longer.

Records that need to be retained should not be deleted and must remain accessible for the period of time during which the integrity, authenticity and reliability of – and often the context for – the retained records must remain inviolate.

Retention is not IT back up or (or for) disaster recovery

Retention management is not about IT backups or ‘archiving’, or disaster recovery programs. These activities are focused more on the ability to recover data and records.

On-premise to online – a different paradigm

Many organisations have (or should have) records retention schedules, also known as disposal or disposition authorities. Records retention schedules and authorities define what needs to be kept and for how long.

Most records are managed in similar ways:

  • As paper (usually printed from digital records), stored in files and/or boxes. These records may be tracked in a database.
  • As digital records, uploaded to a third-party electronic document management system (EDMS), while leaving the original records stored in Exchange mailboxes or File Explorer.
  • As entire business systems (with little thought given to individual records).
Goonellabah_2.jpg
An example of old-style paper file storage. These records are around 20 years old and are well overdue for disposal. 

Into the on-premise mix:

  • MRM policies may be enabled on Exchange mailboxes, allowing end users to apply retention tags to emails. An archiving policy may be in place as well.
  • Individual user mailboxes and ‘home drives’ may be retained for a period of time after the user account is deactivated.
  • There is a backup regime.

Many of the options above will not, or may not, exist (at least in the same way) in Office 365.

On the other hand, Office 365 now includes a range of records retention options that can be used to better manage retention.

Why you need a plan for retention in Office 365

Implementing Office 365 retention policies without a good plan to transition from the on-premise environment, is fraught with potential failure, potential confusion, uncertainty, and legal risk.

To quote from page 882 of Office 365 for IT Pros:

‘… it is wise to take time to chart out how retention will work across the tenant for all workloads before you create any policies. Fools rush to implement retention without thought!

A good starting point is to contact the records management team and get a copy of the organisation’s records retention schedules or authorities to understand what needs to be kept and for how long and also – importantly – where the records are currently stored.

Retention options in Office 365

In simple terms there are two types of retention that can be applied to records in Office 365. The following paraphrases parts of chapter 19 of the book Office 365 for IT Pros.

Explicit (visible) retention policies

This option involves (a) creating retention labels that define a retention period (and if a disposition review is required), (b) publishing the labels as retention policies to specific Office 365 workloads, and (c) applying them manually (including via PS scripts) to content that needs to be retained.

Retention labels published as explicit retention policies can be applied (including automatically, in certain circumstances and/or with an E5 licence) to the following workloads

  • Exchange email (all/select)
  • SPO sites (all/select)
  • ODfB accounts (all/select)
  • O365 groups (all/select)

ExplicitRetentionPolicy.JPG

Implicit (invisible) retention policies

This option involves (a) creating retention policies (that do not include a disposition review) and then (b) applying them to all or specific Office 365 workloads.

Implicit retention policies can be applied to:

  • Exchange email (all/select)
  • SPO sites inc O365 Group sites (all/select)
  • ODfB accounts (all/select)
  • O365 groups (all/select)
  • Skype for Business (specific users)
  • Exchange public folders (all)
  • Teams channel messages (specific teams)
  • Team chats (specific users)

The first four workloads are the same for both types. Which one should you choose – explicit or implicit?

ImplicitRetentionPolicy.JPG
Implicit retention policy

Keep in mind that explicit policies take priority over implicit policies.

Retention policy limits

Both explicit and implicit retention policies have specific limits. You can create:

  • Up to 10 organisation-wide retention policies. For example, all mailboxes, all OneDrive accounts, all SharePoint sites, all Office 365 Groups. 
  • Up to 1000 narrow/specific retention policies. Each of these can point to up to 100 sites, 100 ODfB/O365 group accounts, and 1000 mailboxes. 

These limits, and the difference between explicit and implicit, show why planning for retention is essential.

Questions you might want to ask

Some questions to consider as part of the planning process:

  • For Exchange mailboxes, is it better to have (a) a single implicit policy to keep all  mailboxes for x years, or (b) a single implicit policy that targets only certain mailboxes (e.g., senior managers only), or (c) multiple explicit policies that end-users can apply? What do you do now with Exchange on-premise? Do you journal emails? How will you do that if you go completely online? Could a single implicit policy achieve the same outcome as backing up mailboxes?
  • For OneDrive accounts, is it better to (a) have a single implicit policy to keep all ODfB account for x years, or (b) rely on the ODfB admin storage setting to keep the content after an account becomes inactive (default is 30 days), or (c) have an explicit policy that end users can apply themselves?
  • For SharePoint sites, is it better to (a) have a single implicit policy to keep all SPO sites for x years, or (b) have a single implicit policy applied to up to 100 sites at a time, or (c) create multiple explicit policies that end-users can apply?
  • For Office 365 Groups, is it better to (a) have a single implicit policy, or (b) focus instead on MS Teams channel retention and/or (c) a retention policy for the associated SharePoint sites? Do you have AD premium where Group expiry can be implemented? If yes, should you enable or disable it?

And for all of the above – how will you keep track of what has been applied where?

Recommendations

My suggested recommendations would be:

  • Exchange mailboxes. If the organisation keeps back ups of on-premise mailboxes so these can be recovered after a period of time, remove the default MRM policies and create a single organisation-wide implicit retention policy.
  • OneDrive for Business. For similar reasons to the Exchange mailboxes, create a single organisation-wide implicit retention policy to retain content in user accounts for a given period of time (say, 7 years). Also change the default storage period from 30 days to the same period of time.
  • SharePoint Online. My sense is that a single implicit retention policy for all SharePoint sites is unwise. Retention options should be considered when sites are created. For example, try not to mix (or allows users to mix) content that may have different retention requirements in the same document library – aim to apply the aggregation to the highest level of ‘aggregation’ – site or library. Create labels for and publish a small number of specific explicit retention policies (mapped to the organisation’s records retention schedule). Create one or more implicit policies for specific groups of sites (for example, all inactive project sites). Whatever model you implement, ensure you know if you need to record what was destroyed, including unique metadata from document libraries.
  • Office 365 Groups. The SPO part of Groups can be covered by the SPO retention policy, the email by the Exchange mailbox policy. You may need to consider if you need to create teams chat or channel retention policies.

What happens at the end of the retention period?

So far this post has raised questions about the type of policy you might apply to different workloads.

But what happens when the retention period ends? Should you allow records to be deleted without any kind of disposition review, or check first? This single point may be a key factor in your decision around what type of policy to create and implement, and where.

Some questions to ask:

  • Is is appropriate in your environment for end users to destroy business records by applying a policy?
  • Should someone review the content before it is disposed of?
  • Do you need to keep the metadata associated with content stored in SharePoint document libraries? If yes, where is it going to be stored?
  • Do you need a permanent record of what was destroyed?
  • What do you do if you dispose of something that should not be disposed of?

The answers to these questions may differ depending on the workload to which the retention policy has been applied and whether the retention policy is explicit or implicit.

What does a retention plan look like?

Pages 880 and 881 of Office 365 for IT Pros has an excellent model plan for retention in Office 365. The following (slightly edited) points should be documented for every retention policy that is created and published.

  • Name: It seems obvious, but naming can be important especially for explicit policies. Consider, for explicit policies, adding the retention period to the name, e.g., ‘Company Financial Records – 7 years’.
  • Purpose: What is the purpose of the policy?
  • Retention settings: How long should the content be kept for? Should this be based on when it was created, modified, or when the label was applied? Should there be a disposition review? Who will review the content when it is due for disposition?
  • File Plan: Explicit retention policies can be mapped to a file plan and thereby linked to the retention schedules. If they do, what part of the File Plan should the policy map to?
  • Type: Will the policy be implicit (invisible to users) or explicit (visible to users)?
  • Scope: What Office 365 workloads will this policy cover?
  • Broad or Narrow: Will this policy be applied across an entire workload or to specific mailboxes, sites, accounts? If an explict policy uses retention labels, what are those labels?
  • End of retention: What should happen when the retention period expires? For example, does the metadata need to be kept, should a record be kept of what was destroyed?
  • Lock (optional): Is the content that comes under the scope of the policy considered to be a formal record for the company and if so, is a preservation lock needed? (This option requires additional PowerShell work)

Joint planning is a must

Your organisation is almost certainly going to have business, legal or regulatory requirements for keeping records.

Records managers know how to interpret and apply these to records, and what to do when records reach the end of their retention period. It’s a good idea to consult with these experts, and with your legal team.

It would be a very brave IT shop that unilaterally applies retention policies to records stored in Office 365 without reference to or consultation with records managers, legal teams, or records retention schedules.