Posted in Classification, Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, SharePoint Online, Training and education

AI curated chaos or control – the equally valid but opposite ends of the SharePoint spectrum

There are, broadly speaking, two ‘bookend’ options when it comes to creating new SharePoint Online sites and the document libraries in those sites:

  • ‘Controlled’ model: The creation of new sites is restricted to a small group of individuals with admin rights, who also oversee the creation of document libraries and application of metadata. A combination of controlled and manually applied classification and metadata and retention policies are used to access and manage content over time. Artificial intelligence (AI) tools can also be used to manage content.
  • ‘Chaos/uncontrolled’ model: The creation of new sites, including the creation of document libraries is not restricted. AI tools (including auto-classification) and auto-applied retention policies are used to classify, access and manage content over time. This model assumes that any form of random categorisation applied by end users (e.g., library names, metadata) is mostly ignored by AI tools.

From a traditional information governance and records management (ISO 15498/ISO 16175) point of view, the second ‘chaos’ or uncontrolled model option seems to run counter to conventional wisdom and agreed standards.

From a practical point of view, the first ‘control’ model option seems to run counter to common sense given the volume and range of digital information and the difficulty of classifying or categorising information and records correctly.

Which option is better?

Confusingly, perhaps, the answer may be a combination of both.

  • Certain types of more formal records, such as those required for corporate compliance, formal policies, staff files, accounting information not stored in a finance system, property information, and/or product information, is almost certainly going to be better off in a controlled SharePoint sites with pre-defined libraries and metadata. These types of documents are more likely to be subject to records retention requirements and almost certainly may be subject to eDiscovery and legal holds.
  • Other types of less formal records, including ‘working’ documents, chats and conversations may be better off stored in uncontrolled SharePoint sites, including SharePoint sites linked with Office 365 Groups and Teams, and in MS Teams/Outlook. These types of records are less likely to be subject to records retention requirements but may be subject to eDiscovery and legal holds.

Ultimately, the way the organisation needs to implement Office 365, including SharePoint Online and apply retention policies and other options will depend on its need to comply with oversight and legal requirements (including minimum retention periods), and/or its tolerance for risk.

How does this work in Office 365/SharePoint Online?

If both options Organisations need to make a conscious decision to allow both options, and be prepared to manage both.

The key features of Office 365 and SharePoint to allow both options are listed below:

  • Office 365 retention policies apply to all of Exchange Online, all OneDrive for Business accounts, entire sites (invisible to users) or parts of sites (visible to users).
  • Some retention policies may be applied based on the auto-classification of records, subject to review.
  • The creation of SharePoint sites is either controlled (requested and provisioned) or uncontrolled (created by end users) via either (a) ‘Create sites’ in the end-user SharePoint portal or (b) when a new Team is created in MS Teams.
  • All sites, including Office 365 Group/Team sites are reviewed regularly for activity and inactive sites with no content of value deleted.
  • All controlled sites are assigned either an invisible retention policy or individual visible retention policies (with disposal review), depending on their content.
  • All uncontrolled sites are assigned an invisible retention policy. Uncontrolled and inactive sites with content are also made read only.

Features of controlled and uncontrolled SharePoint sites

SharePoint Online is quite different from older versions of the application and those who dismiss it based on previous experience should consider having another look as a lot has changed in the past couple of years.

SharePoint Online allows the creation of sites that contain important content that needs to be controlled of managed as records, as well as sites created and managed entirely by end-users. And, as an added bonus, all the content is stored in the one place, not in multiple locations (network drives, email servers, EDRM system, etc).

The elements that make up both types of sites, as well as ‘informational’ sites, are described below:

  • Controlled sites
    • Where the organisation’s official records are stored and managed.
    • Created by SharePoint Administrators.
    • More formal in nature, containing the official records.
    • Structure decided by business areas – for example, document libraries using agreed naming conventions.
    • Use of Content Types and site column or local library metadata to define the content.
    • Application of Office 365 retention policies to entire sites or individual document libraries, with disposal reviews. Auto-classification is less likely to be required as the content has already been structured as required.
  • Uncontrolled sites
    • Usually based on end-user created Office 365 Groups or MS Teams.
    • Where ‘working documents’ are created and managed, with the emphasis on allowing end-users collaborate and communicate easily and effectively – and move content to formal sites when required.
    • Created by end-users but naming monitored by SharePoint administrators (or using rules).
    • Informal in nature, used for working documents (effectively replacing personal and network file shares, and other unapproved systems).
    • A fluid structure for document libraries, driven by end-user requirements (not imposed by others).
    • Little if any use of Content Types or metadata.
    • Retention based on Group activity (E5 licences), otherwise based on Office 365 site retention policies and/or auto-classification options.
    • No disposal reviews – content is deleted after a given period of time.
  • Informative
    • Communication sites (e.g., ‘intranet’)
    • Used to publish information to the organisation

Things to watch out for

It is largely true that if you give people an option, someone is bound to try it, sooner or later, especially if it says ‘Create site’, ‘Create team’, or ‘Create group’. Early adopters learn quickly and can just as quickly abandon something that provides no benefit. 

In a ‘free for all’ SharePoint environment, where end-users can create new sites, teams or groups (both of the latter have a SharePoint site), the most likely issues will include:

  • Sites with names that are very similar to ones that already exist, created because the end-user didn’t know another existed (it may not be obvious) or didn’t like the name.
  • Sites with names that make no sense (including common acronyms) or are just ‘wrong’ or contrary to preferred naming conventions.
  • Sites used to create and store content that really should be stored in a more formal site or, conversely, doesn’t belong in the organisation’s official information systems (e.g., photos of someone’s wedding).

All of these issues require some general rules about the creation of new sites (or Office 365 Groups or Teams or Yammer Groups), including suggested naming.

Global and SharePoint admins can monitor the environment and fix issues when they arise rather than wielding a big stick.

What’s great about it

You can have the best of both worlds with SharePoint Online.

  • Keep formal official records in ‘formal’ sites with controlled structures and metadata.
  • Allow end-users to get on with creating, collaborating, sharing (one copy, not attachments), chatting, on any device.

If your communications and change management are good, end-users will soon learn how much fun it can be to use Teams, or access their content from File Explorer (or both!), without having to having to be trained how to save records. All they need to know is how to use the ‘Move’ option to move the final version of records to a formal site.

The foundation of any compliance program is knowing where all of your data lives and then classifying, labeling, and governing it appropriately.

Posted in Compliance, Electronic records, Governance, Information Management, Office 365, Products and applications, Records management, Retention and disposal, SharePoint Online

Planning for records retention in Office 365

Almost ten years ago, in January 2010, I published a post here about the origins of the statute of limitations. The post had the following introduction:

The retention – and eventual disposal – of records is a common business practice, despite occasional concerns about what gets destroyed. Justice Scalia, in Arthur Andersen LLP v United States (No. 04-368, 2004) said as much about the destruction of records relating to Enron by Arthur Anderson:

‘… we all know that what are euphemistically termed “record-retention programs” are, in fact, record-destruction programs, and that one of the purposes of the destruction is to eliminate from the files information that private individuals can use for lawsuits and that Government investigators can use for investigations.’

Almost ten years on, it still seems an appropriate way to introduce a post about the retention of records, this time in relation to records stored in Office 365.

Bottom line – you need to plan for it, and make sure your legal team is consulted.

Blatant plug for a great book

Before you read further, I recommend you have a look at the comprehensive e-book ‘Office 365 for IT Pros‘. This 1000+ page ebook includes, in Chapter 19 – Office 365 Data Governance, a comprehensive description of how to create, apply and manage retention policies in Office 365.

What do we mean by retention?

The retention of records is generally based on business, legal or regulatory requirements to keep certain records for a minimum period of time. In the case of government records, there may also be an archival requirement.

The retention period may relate to or be based on a statute of limitations that governs when potential legal actions expire. For example, simple contracts generally need to be kept for a minimum of six or seven years (depending on jurisdiction after they expire. More complex contracts (including deeds) may need to be kept for much longer.

Records that need to be retained should not be deleted and must remain accessible for the period of time during which the integrity, authenticity and reliability of – and often the context for – the retained records must remain inviolate.

Retention is not IT back up or (or for) disaster recovery

Retention management is not about IT backups or ‘archiving’, or disaster recovery programs. These activities are focused more on the ability to recover data and records.

On-premise to online – a different paradigm

Many organisations have (or should have) records retention schedules, also known as disposal or disposition authorities. Records retention schedules and authorities define what needs to be kept and for how long.

Most records are managed in similar ways:

  • As paper (usually printed from digital records), stored in files and/or boxes. These records may be tracked in a database.
  • As digital records, uploaded to a third-party electronic document management system (EDMS), while leaving the original records stored in Exchange mailboxes or File Explorer.
  • As entire business systems (with little thought given to individual records).
Goonellabah_2.jpg
An example of old-style paper file storage. These records are around 20 years old and are well overdue for disposal. 

Into the on-premise mix:

  • MRM policies may be enabled on Exchange mailboxes, allowing end users to apply retention tags to emails. An archiving policy may be in place as well.
  • Individual user mailboxes and ‘home drives’ may be retained for a period of time after the user account is deactivated.
  • There is a backup regime.

Many of the options above will not, or may not, exist (at least in the same way) in Office 365.

On the other hand, Office 365 now includes a range of records retention options that can be used to better manage retention.

Why you need a plan for retention in Office 365

Implementing Office 365 retention policies without a good plan to transition from the on-premise environment, is fraught with potential failure, potential confusion, uncertainty, and legal risk.

To quote from page 882 of Office 365 for IT Pros:

‘… it is wise to take time to chart out how retention will work across the tenant for all workloads before you create any policies. Fools rush to implement retention without thought!

A good starting point is to contact the records management team and get a copy of the organisation’s records retention schedules or authorities to understand what needs to be kept and for how long and also – importantly – where the records are currently stored.

Retention options in Office 365

In simple terms there are two types of retention that can be applied to records in Office 365. The following paraphrases parts of chapter 19 of the book Office 365 for IT Pros.

Explicit (visible) retention policies

This option involves (a) creating retention labels that define a retention period (and if a disposition review is required), (b) publishing the labels as retention policies to specific Office 365 workloads, and (c) applying them manually (including via PS scripts) to content that needs to be retained.

Retention labels published as explicit retention policies can be applied (including automatically, in certain circumstances and/or with an E5 licence) to the following workloads

  • Exchange email (all/select)
  • SPO sites (all/select)
  • ODfB accounts (all/select)
  • O365 groups (all/select)

ExplicitRetentionPolicy.JPG

Implicit (invisible) retention policies

This option involves (a) creating retention policies (that do not include a disposition review) and then (b) applying them to all or specific Office 365 workloads.

Implicit retention policies can be applied to:

  • Exchange email (all/select)
  • SPO sites inc O365 Group sites (all/select)
  • ODfB accounts (all/select)
  • O365 groups (all/select)
  • Skype for Business (specific users)
  • Exchange public folders (all)
  • Teams channel messages (specific teams)
  • Team chats (specific users)

The first four workloads are the same for both types. Which one should you choose – explicit or implicit?

ImplicitRetentionPolicy.JPG
Implicit retention policy

Keep in mind that explicit policies take priority over implicit policies.

Retention policy limits

Both explicit and implicit retention policies have specific limits. You can create:

  • Up to 10 organisation-wide retention policies. For example, all mailboxes, all OneDrive accounts, all SharePoint sites, all Office 365 Groups. 
  • Up to 1000 narrow/specific retention policies. Each of these can point to up to 100 sites, 100 ODfB/O365 group accounts, and 1000 mailboxes. 

These limits, and the difference between explicit and implicit, show why planning for retention is essential.

Questions you might want to ask

Some questions to consider as part of the planning process:

  • For Exchange mailboxes, is it better to have (a) a single implicit policy to keep all  mailboxes for x years, or (b) a single implicit policy that targets only certain mailboxes (e.g., senior managers only), or (c) multiple explicit policies that end-users can apply? What do you do now with Exchange on-premise? Do you journal emails? How will you do that if you go completely online? Could a single implicit policy achieve the same outcome as backing up mailboxes?
  • For OneDrive accounts, is it better to (a) have a single implicit policy to keep all ODfB account for x years, or (b) rely on the ODfB admin storage setting to keep the content after an account becomes inactive (default is 30 days), or (c) have an explicit policy that end users can apply themselves?
  • For SharePoint sites, is it better to (a) have a single implicit policy to keep all SPO sites for x years, or (b) have a single implicit policy applied to up to 100 sites at a time, or (c) create multiple explicit policies that end-users can apply?
  • For Office 365 Groups, is it better to (a) have a single implicit policy, or (b) focus instead on MS Teams channel retention and/or (c) a retention policy for the associated SharePoint sites? Do you have AD premium where Group expiry can be implemented? If yes, should you enable or disable it?

And for all of the above – how will you keep track of what has been applied where?

Recommendations

My suggested recommendations would be:

  • Exchange mailboxes. If the organisation keeps back ups of on-premise mailboxes so these can be recovered after a period of time, remove the default MRM policies and create a single organisation-wide implicit retention policy.
  • OneDrive for Business. For similar reasons to the Exchange mailboxes, create a single organisation-wide implicit retention policy to retain content in user accounts for a given period of time (say, 7 years). Also change the default storage period from 30 days to the same period of time.
  • SharePoint Online. My sense is that a single implicit retention policy for all SharePoint sites is unwise. Retention options should be considered when sites are created. For example, try not to mix (or allows users to mix) content that may have different retention requirements in the same document library – aim to apply the aggregation to the highest level of ‘aggregation’ – site or library. Create labels for and publish a small number of specific explicit retention policies (mapped to the organisation’s records retention schedule). Create one or more implicit policies for specific groups of sites (for example, all inactive project sites). Whatever model you implement, ensure you know if you need to record what was destroyed, including unique metadata from document libraries.
  • Office 365 Groups. The SPO part of Groups can be covered by the SPO retention policy, the email by the Exchange mailbox policy. You may need to consider if you need to create teams chat or channel retention policies.

What happens at the end of the retention period?

So far this post has raised questions about the type of policy you might apply to different workloads.

But what happens when the retention period ends? Should you allow records to be deleted without any kind of disposition review, or check first? This single point may be a key factor in your decision around what type of policy to create and implement, and where.

Some questions to ask:

  • Is is appropriate in your environment for end users to destroy business records by applying a policy?
  • Should someone review the content before it is disposed of?
  • Do you need to keep the metadata associated with content stored in SharePoint document libraries? If yes, where is it going to be stored?
  • Do you need a permanent record of what was destroyed?
  • What do you do if you dispose of something that should not be disposed of?

The answers to these questions may differ depending on the workload to which the retention policy has been applied and whether the retention policy is explicit or implicit.

What does a retention plan look like?

Pages 880 and 881 of Office 365 for IT Pros has an excellent model plan for retention in Office 365. The following (slightly edited) points should be documented for every retention policy that is created and published.

  • Name: It seems obvious, but naming can be important especially for explicit policies. Consider, for explicit policies, adding the retention period to the name, e.g., ‘Company Financial Records – 7 years’.
  • Purpose: What is the purpose of the policy?
  • Retention settings: How long should the content be kept for? Should this be based on when it was created, modified, or when the label was applied? Should there be a disposition review? Who will review the content when it is due for disposition?
  • File Plan: Explicit retention policies can be mapped to a file plan and thereby linked to the retention schedules. If they do, what part of the File Plan should the policy map to?
  • Type: Will the policy be implicit (invisible to users) or explicit (visible to users)?
  • Scope: What Office 365 workloads will this policy cover?
  • Broad or Narrow: Will this policy be applied across an entire workload or to specific mailboxes, sites, accounts? If an explict policy uses retention labels, what are those labels?
  • End of retention: What should happen when the retention period expires? For example, does the metadata need to be kept, should a record be kept of what was destroyed?
  • Lock (optional): Is the content that comes under the scope of the policy considered to be a formal record for the company and if so, is a preservation lock needed? (This option requires additional PowerShell work)

Joint planning is a must

Your organisation is almost certainly going to have business, legal or regulatory requirements for keeping records.

Records managers know how to interpret and apply these to records, and what to do when records reach the end of their retention period. It’s a good idea to consult with these experts, and with your legal team.

It would be a very brave IT shop that unilaterally applies retention policies to records stored in Office 365 without reference to or consultation with records managers, legal teams, or records retention schedules.

Posted in Classification, Compliance, Information Classification, Information Management, Office 365, Products and applications, Records management, Retention and disposal, Security

Office 365 Security and Compliance – classification label changes

Microsoft have improved the Classification section in the Office 365 Security and Compliance centre. The change will help to reduce confusion and make it easier for records managers and security administrators to focus on their individual needs.

Previous user interface

The primary change is to the menu interface. The previous menu options, shown in the screenshot below, showed only ‘Labels’ and ‘Label policies’.

O365_Classifications_Labels

When the previous ‘Labels’ option was selected, a new screen with two tabs ‘Sensitivity’ (default) and ‘Retention’ was displayed, as shown below.

O365_Classifications_Labels

The sensitivity or retention tab had to be selected to create or publish a new label. The user interface was unclear and the difference between creating and publishing a label was not obvious.

New user interface

The sensitivity and retention elements have now been separated and placed under the primary ‘Classification’ menu option as shown below.

O365_Compliance_ClassificationLabels23Aug19.JPG

Now, ‘Labels’ and ‘Label policies’ are two tabs under the relevant section as can be seen below.

 

O365_Compliance_ClassificationLabelsRetention23Aug19

O365_Compliance_ClassificationLabelsSensitivity23Aug19

The options to create and publish labels remain the same.

Posted in Compliance, Exchange Online, Governance, Information Management, Office 365, Office 365 Groups, OneDrive for Business, Records management, Retention and disposal, SharePoint Online

Managing the outcomes of records retention in Office 365

The retention of records in Exchange Online (EXO), SharePoint Online (SPO), OneDrive for Business (ODfB) and Office 365 (O365) groups can be achieved through the application of retention labels published in the O365 Security and Compliance admin portal.

This post describes:

  • How retention labels work (in summary), including the ‘per record’ rather than the container/aggregation retention model.
  • What happens to content in Office 365 when a retention period expires.
  • The options and actions that may influence the way retention labels/policies are configured, where and how they are applied, and the outcomes required.

The post highlights the need for information and records managers to be involved in all aspects of governance, site architecture and design, and decisions around specific settings and configuration, as well as being assigned specific roles, when Office 365 is implemented.

A quick summary of how O365 retention labels work

Records retention policies in O365 are based on ‘retention labels’ that are created in the O365 Security and Compliance admin portal under the ‘Classifications’ section. Multiple labels can be applied to a single policy.

  • Click this link to read Microsoft’s detailed guidance on retention labels.
  • Click this link to read Microsoft’s detailed guidance on retention policies.

Retention outcomes

Each retention label defines one of three potential outcomes at the end of the retention period, if retention is enabled, ‘keep forever’ is not selected, and the label is not used to classify the content as a record*:

  • The content will be automatically deleted. If the content is in SharePoint, it will first be sent to the Recycle Bin, from which it can be recovered within 90 days.
    • This option may be suitable for certain types of low value records.
  • A disposition review will be triggered to notify specific people. As with the previous point, SharePoint content will be sent to the Recycle Bin if a decision is made to delete it.
    • This option will require additional, human-intervention actions, as described below, if standard records management disposal review processes are followed.
  • Nothing.

Triggers

The date when the above action will occur is based on one of four triggers:

  • Date created
  • Date last modified
  • When labelled applied
  • A event. The ‘out of the box’ (OOTB) event types are:
    • Employee activity. (Processes related to hiring, performance and termination of an employee)
    • Expiration or termination of contracts and agreements.
    • Product lifetime. (Processes relating to last manufacturing date of products).
    • A new event can also be added.
    • See this post for Microsoft guidance on event-driven retention.

An additional alternative option is available: ‘Don’t retain the content, just delete it if it’s older than n days/months/years.’ This is similar to the automatic deletion option above and may be suitable for certain types of records. 

Declaring content as records

* The option to classify or ‘declare’ content as a record is not discussed further as relates to the way records are managed in the US. Microsoft’s guidance on labels notes that: ‘At a high level, records management means that: (a) Important content is classified as a record by users. (b) A record can’t be modified or deleted. (c) Records are finally disposed of after their stated lifetime is past.’ The standard on records management, ISO 15489, defines a record as ‘evidence of business activities, often (but not exclusively) in the form of a document or object, in any form’. This means that anything can be a record. The record may continue to be modified throughout its life. 

When do retention labels become active?

Retention labels become active only when they are published. As part of the publishing process, a decision must be made if the label will apply to all (a single option) or selected parts of the O365 ecosystem:

  • The Exchange Online (EXO) mailboxes of all or specific recipients, or excluding specific recipients.
  • All or specific SharePoint Online (SPO) sites, or excluding specific sites.
  • All or specific OneDrive for Business (ODfB) accounts, or excluding specific accounts.
  • All or specific O365 Groups, or excluding specific groups. Note that content in Microsoft Teams (MS Teams) is included in the O365 Groups options that include both the SharePoint content and email/Teams chat content.

Auto-applying retention labels

Both the retention label and policy sections include the ability to auto-apply a retention policy if certain conditions are met.

  • Sensitive information types. These are the same types that appear in the Data Loss Prevention (DLP) section, for example ‘Financial data’ or ‘Privacy data’.
  • Specific keywords.
  • Content types and metadata (E5 licences only). See this post by Joanne Klein for a description of these options.

The ability of the first two options to accurately identify content and apply a retention policy should be investigated before they are relied on.

When do retention policies start working

According to Microsoft’s guidance Overview of Retention Labels:

If you publish retention labels to SharePoint or OneDrive, it can take one day for those retention labels to appear for end users. In addition, if you publish retention labels to Exchange, it can take 7 days for those retention labels to appear for end users, and the mailbox needs to contain at least 10 MB of data.

  • In EXO, the default MRM policy needs to be removed before the new policy applies.
  • In ODfB, the policy is available to be manually applied on folders or documents. It does not automatically apply to content.
  • In SPO, the policy can be applied to document libraries or documents. To avoid removing the ability for users to legitimately need to delete documents in an active library it is recommended to apply the policy after the document library has ceased to be active.
  • Content in Office 365 Groups is covered by either the EXO (for email/teams chat content) or the SPO policy (applied to libraries).

Retention labels apply to individual records within aggregations

Records labels can be applied to aggregations of records (an entire email mailbox or folder, a SharePoint library or list, an ODfB account, O365 Groups) or individual records. However, the disposal process targets individual records (e.g., individual emails, single documents in SharePoint libraries, individual list items). 

That is, even when all the individual records are disposed of, the parent aggregation remains in place without any indication that the records previously stored in it (sometimes known as a ‘stub’) have been destroyed. 

This outcome has implications for the way the outcome of a retention label is set. It requires a choice between (a) delete automatically without review or (b) review before delete.

The latter option is made complicated by the requirement to review individual documents, including potentially in the original container (document library in SPO) and export metadata relating to those records if a record of the deletion is to be retained.

What happens when records reach the end of their retention period

As noted above, the outcome at the end of the retention period (trigger date + n days/months/years) will depend on the settings on the label.

  • Where the label was applied (EXO mailbox, SPO library or list, ODfB folder or document, O365 Group)
  • Whether the records would be deleted automatically or be subject to a disposition review.

If the records are to be deleted automatically:

  • SPO and ODfB records will be sent to the site/ODfB Recycle Bin for 90 days
  • EXO emails will be moved to a ‘Cleanup’ area for 14 days, before permanent deletion.
  • Aside from the audit logs (which by default only go back 90 days), no other record will be kept of the destroyed records.

If the records are subject to a disposition review, an email is sent to the person nominated. When that person clicks on the link in the email they are taken directly to the ‘Dispositions’ sub-section of the Records Management section of the O365 Security and Compliance centre.

It is arguable that retention policies with disposition review should not be applied to ODfB content as this will require the reviewer to review all the content that has been labelled by a user in their ODfB account.

  • For more information about this subject see this Microsoft page ‘Overview of disposition reviews‘. Microsoft note, on that page ‘To get access to the Disposition page, reviewers must be members of the Disposition Management role and the View-Only Audit Logs role. We recommend creating a new role group called Disposition Reviewers, adding these two roles to that role group, and then adding members to the role group.’

The dispositions dashboard shows the number of records that are pending disposition against each retention policy label:

O365_RM_DispositionsDashboard

Pending disposition tab

When the reviewer clicks on one of the retention policies listed, the following view opens for records ‘Pending disposition’:

O365_Dispositions_Pending

An important point to note here is that records are listed individually, not in logical aggregations or collections. It is possible however to use the Search option on the left to filter by author (emails) or SharePoint site and/or site library. It is also possible to export the details (which does not include any unique metadata applied to documents in SharePoint libraries).

All the records displayed may then be selected and a ‘Finalise decision’ dialogue box appears with the following options:

  • Dispose of the records.
  • Extend the retention.
  • Re-label the records.

Disposed items tab

The Dispositions dashboard includes a ‘Disposed items’ tab.

Microsoft note that this tab ‘… shows dispositions [that] were approved for deletion during a disposition review and are now in the process of being permanently deleted. Items that had a different retention label applied or their retention period extended as part of a review won’t appear here.’

Importantly, once records are permanently deleted, they no longer appear in the ‘Disposed Items’ tab. This means that no record will be kept of the records that were destroyed.

Shortcomings of the O365 dispositions/disposal model for records stored in SPO

Only individual records appear, not all the items in a document library

If the retention outcome is based on the ‘created’ or ‘last modified’ date, individual records in SPO document libraries will start to appear as soon as they reach the retention end date. The reviewer may need (or want) to view the original library, which they can identify from the link is in the dispositions review pane.

Retention policies prevent deletion

As a retention label prevents the deletion of content by users, and this may put them off using SharePoint, it is recommended that retention in SPO document libraries be based on when the label was applied NOT when it was created or last modified. This will help to ensure that all documents appear in the disposition review area at the same time.

Event based triggers may not be suitable for disposition review

If the retention outcome is based on an event, or is auto applied and a disposition review is required, those records will appear randomly when the event is triggered. It could be difficult for records managers to decide the disposal outcome in this way without referring back to the library.

The dispositions review pane does not display the original metadata

The  dispositions review pane displays only very basic metadata from the original library. Again, the reviewer may need to view the original library, export the metadata and store that in a secure location. Note that the exported metadata includes the URL of each original record including the library name.

The document library remains even when all contained records are destroyed

If the reviewer chooses to dispose of the records listed, only the content of the library (the individual documents stored in it) is deleted, not the actual library itself. No record (e.g., a ‘stub’ of the deleted item) is kept in the library of the deleted content.

The ‘Disposed items’ tab only shows records being destroyed

The ‘Disposed items’ tab only shows records in the process of being destroyed. It does not keep a record of what was destroyed. Records managers will need to retain the metadata of what was destroyed, when, based on what disposal authority, and with whose approval.

Dispositions really only provides a ‘heads up’ for further action

The Dispositions process may be instead used as a form of ‘heads up’ that records are starting to be due for disposal in a document library. This would allow the records managers (who should be Site Collection administrators) to review the library, export the complete set of metadata, and decide if the entire library can be deleted since it is no longer required.

Conclusions

Retention labels in O365 are an effective way of managing the retention and disposal of records in that environment, subject to the following points.

Email

Emails will likely continue to be managed as complete aggregations of records – the mailbox. Users cannot be expected to create logical groupings and apply individual retention labels to those records.

Organisational records policies may mandate specific timeframes for the retention of email (e.g., 1 year), while HR/IT security policies may mandate that whole mailboxes are retained for a period of time after employees leave. It is important to understand the difference between these two models

Options to automatically transfer emails to SharePoint document libraries via rules may be possible using Flow but these rely on individual users to set up.

Consideration should instead be given to using O365 Group mailboxes, rather than individual personal mailboxes, for specific work related matters. For example, ‘Customer Complaints’, or ‘XYZ Project’.

OneDrive for Business Accounts

ODfB accounts may be covered by two forms of retention:

  • Retention labels that apply to all ODfB accounts while the account is active. These must be manually applied by users.
  • A separate retention period set for ODfB accounts after a user leaves the organisation.

If there is a requirement to prevent the deletion of content by a user from their ODfB account, the better way to achieve this is using an eDiscovery case with Legal Hold applied.

SharePoint Online

As most records will be stored in SharePoint document libraries (including Office 365 Group-based SP libraries), multiple retention labels will be required to address different types of content or retention requirements.

Careful consideration should be given to whether records can be deleted automatically at the end of the retention period or should be subject to disposition review, noting that the automatic deletion provides no opportunity to capture the metadata of the records.

The ‘auto-apply’ or event-based retention option should be used sparingly to avoid a trickle of records for disposal – unless there is enough trust that these can be accurately marked and deleted without review.

Shortcomings in the disposition review process support the following decisions for SharePoint Online content:

  • The number of retention labels should be minimised to avoid a very long drop-down menu when a label is applied. If current record retention or disposal authorities contain a lot of classes, some of these could potential be combined into a single class (e.g., ‘Company Records – 7 years’), while the site name and document library name should provide some context to the content to ‘map’ back to the original classes.
  • Retention labels should be applied when document libraries (or lists) become inactive as this will avoid conflict with users who want to delete content and also ensure that documents are ready for disposition review at the same time.
  • Retention labels applied to SPO document libraries should include the disposition review option unless a ‘delete only’ label is considered suitable for certain document libraries that clearly contain working documents or Redundant, Outdated and Trivial (ROT) content.
  • Records managers should review the content of all or most original SPO document libraries, and export the metadata of those libraries for storage in a separate location (such as an ‘archives’ site), or in the original library with the retention label changed to ‘Never Delete’. The original document library can then be deleted.
Posted in Compliance, Electronic records, Exchange Online, Governance, Information Management, Products and applications, Records management, Retention and disposal

Records retention in Exchange Online

Retention policies created as labels in the classification section of  the Office 365 (O365) Security & Compliance admin centre can be applied to content in Exchange Online (EXO) mailboxes.

It may not be possible to apply more than one Office 365 retention policy to EXO mailboxes because, unless the mailbox is dedicated to a specific subject (for example, ‘Customer Complaints’), or using a dedicated Office 365 Group’s mailbox:

  • Emails generally contain content about multiple subjects.
  • The way the content is categorised in mailboxes , including through the use of rules and/or folders, varies between users.
  • The retention and disposal of records relies largely on the ability to assign retention policies to categories or groups of records, not individual records.
  • Organisational policies may require all user emails to be kept (‘archived’) for a period of time after they leave the organisation.

Unless emails are moved to a different storage location such as SharePoint, it may be necessary to continue apply a single, but shorter, O365 retention policy to mailboxes.

Exchange Messaging Records Management (MRM) policies

Until Office 365 retention policies appeared as an option, MRM policies applied in EXO were likely based on an organisational business requirement to keep the mailboxes (and other content) of departed users for potential legal or compliance reasons.

MRM policies in EXO are found under the ‘Compliance Management’ section of the EXO admin portal.

EXO_Compliance_RetentionMenu.JPG

When this section is opened, the following message may appear:

EXO_Retention_Message

The default MRM policy has the following options. These may be modified, or additional retention tags created, as required.

EXO_Default_MRMPolicySettingsA.JPG

If the default MRM policies have not been changed (by the Exchange administrator), the default policy/ies will apply. This means that users can use the ‘Assign Policy’ option on folders and emails to decide how long they should be kept.

Emails that are deleted before a backup is made may not be retained.

Some organisations may decide to retain all emails and the mailboxes of departed users ‘forever’. They can do this by removing all the options except ‘Never Delete’.

How O365 Retention Policies are applied to Exchange

Retention labels created in Office 365 can be used to manage the retention of emails, including (to some degree) emails that have content that meets certain pre-defined conditions.

Retention labels are created in the Office 365 Security and Compliance admin portal under the ‘Classifications’ section. This section has three options:

  • Labels. This section is used to create both ‘Sensitivity’ and ‘Retention’ labels. There is also an ‘Auto-apply’ option in the Retention section.
  • Label policies. This section partially duplicates the options in the previous option (except the ‘Create’ option), and lists the labels that have been published.
  • Sensitive info types.

Auto-apply, as its name suggests, auto-applies an existing label based on certain conditions. The conditions are as follows:

  • Apply label to content that contains sensitive info. The sensitive info types are pre-defined options for (a) Financial data (e.g., credit card numbers), (b) Medical and Health (e.g., predefined health records), (c) Privacy (e.g., personal and sensitive information. There is also the option to create a Custom setting.
  • Apply label to content that contains specific words or phrases, or properties. This option works by looking for specific words or phrases.

New labels must be published before they appear or apply anywhere in Office 365.

During the publish process, policies must specify where (in the ‘Locations’ section) the policy is to be applied.

The default option is ‘All locations. Includes content in Exchange email, Office 365 groups, OneDrive and SharePoint documents.’ Alternatively, the policy may be set to specific locations including

  • The Exchange mailboxes of all or specific recipients, or excluding specific recipients.
  • All or specific SharePoint sites, or exluding specific sites.
  • All or specific OneDrive accounts, or excluding specific accounts.
  • All or specific Office 365 Groups, or excluding specific groups.

Note that content in Microsoft Teams is included in the Office 365 Groups options which includes both the SharePoint content and email/Teams chat content.

Mixing MRM and O365 retention policies – maybe not a good idea

If the default MRM policies are not removed, any O365 retention policy that is applied to EXO will appear in the list of retention tags under the default MRM policy, as can be seen in the screenshot below which shows three options in addition to the original MRM policies: ‘Temporary records – 7 days’, ‘Financial Records’, and ‘Company records – 7 years’. If nothing is changed in the environment, these policies can be applied by users to folders and emails. 

O365_MRM_ExchangeRetentionoptions

If the organisation has decided to remove all retention tags except ‘Never Delete’ and a new O365 retention policy is applied to EXO, the ‘Never Delete’ option will prevail and the O365 policy will not work.

Accordingly, careful consideration needs to be given to the creation of O365 retention policies that may be applied to EXO records.

Should user mailboxes be kept ‘forever’?

Many IT departments keep user mailboxes of departed staff (and most other content on the network) for a long time, usually on backups, ‘just in case’ they may be required for legal or compliance requirements, including investigations into misconduct.

Recent personal experience with subpoenas for mailboxes of departed staff indicates that 10 years is likely to be the maximum retention requirement for these types of records. There may be a case to keep certain individual mailboxes for much longer, which the O365 policy allows for.

 

What happens when emails reach the end of their O365 retention policy period?

O365 retention policies define how long records are to be retained before they are either deleted or ready for review (via the Records Management – Dispositions section of the O365 Security and Compliance admin portal).

The following options define what happens when retention is enabled:

  • Retain the content (a) for a specific period (n days/months/years) or (b) forever. Option (b) is the same as the MRM policy ‘Never Delete’.
    • Action to be taken at the end of the period (except ‘forever’): (a) Delete the content automatically, (b) Trigger a disposition review (i.e., notify specific people), or (c) Do nothing, leave the content as is.
  • Don’t retain the content, just delete it if it’s older than n days/months/years.
  • Retain or delete the content based on: (a) When it was created, (b) When it was last modified, (c) When the label was applied, (d) based on an event.

The three actions above define the options for records managers:

  • Allow the emails to be deleted automatically. This is possibly the easiest and most efficient option but it will result in the deletion of any emails when they reach the end of the retention period – if they are kept in Exchange. Importantly, if a specific period of time (e.g., 7 years) is set for email retention, this could start to delete the emails of users who are still with the organisation after that period expires. This fact may affect the retention period that is set. 
  • Trigger a disposition review – see below. This option would be onerous to implement; it would take a lot of effort to review the individual emails of a departed user as part of a disposition review. It would, however, allow for selective review by using the ‘filter’ option in the Dispositions area.
  • Do nothing. This option may be useful for specific types of records, but not emails.

Disposition Reviews

Emails that are subject to a disposition review will appear in the Records Management – Dispositions section of the O365 Security and Compliance centre. Note that the ‘Type’ must be changed from ‘Documents’ to ‘Emails’ to see the emails that are due for disposal. As noted above, while it is possible to filter by user to review the emails, this process could be quite onerous.

O365_EXO_DispositionsA.JPG

Summary

The nature of email makes it almost impossible to categorise them into categories that map to different retention and disposal policies.

Most mailboxes will be subject to a single retention policy.

Office 365 retention policies can and probably should replace the default EXO MRM policies that govern the retention of emails.

Retaining emails in the mailboxes they are stored in ‘forever’ is not a practical retention model. 10 years is a reasonable maximum period, but exceptions may be required.

If O365 retention policies replace EXO MRM policies, records managers need to specify (a) how long emails need to be kept for and (b) whether they can simply be deleted when they reach the end of the retention period or need to be reviewed before deletion.

References

‘Overview of Retention Policies’ https://docs.microsoft.com/en-au/office365/securitycompliance/retention-policies (accessed 9 August 2019)

‘Set up an archive and deletion policy for mailboxes in your Office 365 organization’
https://docs.microsoft.com/en-au/office365/securitycompliance/set-up-an-archive-and-deletion-policy-for-mailboxes (accessed 6 August 2019)

Posted in Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

SharePoint Online – records management options and settings

This post summarises the primary records management options, settings and ideas that can be applied in SharePoint Online to manage records.

This post should be read as the second part of my previous post on the records management options and settings available in the Office 365 admin and security and compliance portals. Some of these settings will be referred to in this post.

The options and settings described in this post should ideally form part of your SharePoint governance documentation.

SharePoint Governance

We have already seen in the previous post that Office 365 Global Admins (GAs) have access to all parts of the Office 365 ecosystem. But they should rarely solely be responsible for SharePoint Online (SPO).

Some form of governance arrangement is necessary for SPO, especially if you plan to manage records in that application.

Some of the key considerations are as follows.

  • Who is responsible for ‘marketing’ or promoting SharePoint in the organisation, and making sure it is used correctly? The area responsible in IT for change management should probably take the lead on this as SPO is only one part of the O365 ecosystem. Records managers should have a role too, or be consulted.
  • SharePoint Administrator. You should already have a SharePoint Administrator and that person (or persons) is likely to be sitting in your IT department. Records managers will rarely also be SharePoint administrators; the two need to work closely together.
  • Who is responsible for training people to use SharePoint, especially to highlight the recordkeeping aspects of the application?
  • Who are the Site Collection Administrators? See next point.
  • Who are the Site Owners?
  • Who can create Office 365 Groups?

Answers to these questions should all be documented in your governance documentation.

SharePoint Online Admin Portal

SharePoint Online customised administrator

The SPO administrator role, a ‘customised administrator’ set in the Office 365 (O365) portal, should normally have a log on that is separate from that person’s O365 user log on. The SPO administrator account should not be a generic one (and generic accounts should generally be avoided).

The SPO administrator accesses the SPO admin portal from the Office 365 admin portal. They will also have access to the O365 Message Centre and Service Health sections.

SharePoint Online Architecture

Why a design model is good to have

Organisations should have some sort of design model for their SPO architecture. Most records will be kept in document libraries SharePoint team sites under the /teams path but some could also be under the /sites path.

The design model should include naming conventions for sites to avoid site names that have unknown acronyms or complex names. Site names form part of the total 400 characters allowed from https to the document suffix (e.g., .docx) so site names should ideally be no longer than around 16 characters. For example:

https://tenantname.sharepoint.com/teams/sitename

Records managers should be involved in designing this architecture model and could also be part of any approval process for new sites, to ensure the proposed names are suitable.

The names of SPO sites should generally map to business functions. Where the main function is very large (e.g., Financial Management is very large, you may decide to create sites based on the ‘sub-function’. That is, under the broader Financial Management (or simply ‘Finance’) site, you could have a separate site for Finance AP and another for Finance AR). These can be linked to a hub site that could be the ‘parent’ function site.

Don’t mix functions (such as personnel and IT) in the same site if only because this site is likely to become very large.

Try to aim for team site coverage of all business areas as all areas are likely to create or maintain records.

One relatively easy way to do this is to consult with the business area and understand how they use their current Network File Share location. This has the additional benefits of ‘mapping’ their SPO site to their existing NFS structure (generally or very specific) so it is familiar to them, and assisting with the migration of NFS to SPO later on.

Creating new Site Collections

Generally speaking there now are three types of SPO site:

  • A team site not linked to an O365 Group (but can be retrospectively linked)
  • A team site linked to an O365 Group
  • A communication site

Again, generally speaking:

  • SPO team sites (linked or not with O365 Groups) are the functional replacement for network file shares and, accordingly, contain most of the ‘document’ type records.
  • SPO communication sites are used for publishing purposes, including the intranet. They may contain documents in document libraries that, again, replace network file shares previously used for this purpose.

New sites can be created:

  • Directly from the SPO admin portal.
  • Via the ‘Create Site’ dialogue available in each user’s SharePoint portal, when this option is enabled. When this option is enabled, users can create either a Team site (linked with an O365 Group), a Communication site, or a ‘classic’ site (not linked with an O365 Group).
  • When a new Office 365 Group is created. This includes, if enabled, when a new Team in MS Teams is created, a Yammer group is enabled, or the person choose to create a new group from Outlook. If this option is allowed, whoever creates the O365 Group becomes the Site Collection Administrator and the SharePoint admin will be unable to access the site. For this reason, organisations that want to control their SPO environment may wish to limit who can create Office 365 Groups.
  • Via a PowerShell script.

SharePoint Sites

Site Collection Administrators (SCA)

Every SPO site has Site Collection Administrators. To ensure that records managers can access every site to manage records, it may be useful to add them to the membership of a Security Group that is in turn added to every site’s Site Collection Administrators after it is created.

Site Collection Administrators are added and managed in Advanced Permission Settings.

SPO_AdvancedPermissions

When you click on Site Collection Administrators, this dialogue appears:

SPO_SCAs.JPG

As noted above, if the ability to create O365 Groups is not controlled, the person who creates the O365 Group (as noted in the screenshot above) will become the SCA. The SharePoint administrator will be able to see the site in the SPO admin portal but may not be able to change the SCA settings. They may need to ask a Global Admin to do this.

Being a Site Owner only is not sufficient for records managers. Site Owners should be someone in the business area that ‘owns’ and will manage the SPO site on a day to day basis.

Site collection features – document IDs and Document Sets

Site collection features are only accessible to Site Collection Administrators. The list below expands as new features are activated; as can be seen, the ‘Document ID service’ feature has been enabled on this site. (Note, ‘Site features’ are activated from the Site Administration section, see below).

SPO_SiteCollectionAdminOptions.JPG

The Document ID feature is required for recordkeeping purposes as it assigns a unique Document ID to every object (including document sets but not folders) stored in a library.

If they are to be used in the site, the Document Set feature is also enabled in the Site collection features section.

SPO_DocIDDocSetSiteCollectionFeatures.JPG

After the Document ID service is enabled a new option appears in the Site Collection Administration section called ‘Document ID Settings’ (as noted above).

As can be seen in the screenshot below, all Document IDs begin with a unique set of up to 12 characters. Ideally, the Site name should be used as this will immediately give a clue to the site name on the document.

SPO_DocIDSettings.JPG

Document IDs take the form:

  • Prefix (e.g., ‘SITENAME’)
  • Library number. This is a unique and un-modifiable number of the library where the document is stored. It is not based on the library GUID.
  • Next sequential number.

If a document is deleted or moved from the library, the document ID (the sequential number) is not re-used.

Note that Document Sets use the same Document IDs. These cannot be separately modified.

Site collection features – Site Audit logs

The option ‘Site collection audit settings’ will already be visible in the Site Collection Administration section of all new sites, however (a) the options in the audit settings need to be enabled and (b) the ‘Reporting’ Site collection feature must be activated to enable the production of Site Audit Logs as required.

Note, the Site collection audit sections settings notes that ‘If you’d like to keep audit data for longer than this, please specify a document library where we can store audit reports before trimming occurs’. The default storage location is /_catalogs/MaintenanceLogs. However, the various options shown below must be selected for anything to be saved.

SPO_SiteCollectionAuditOptions.JPG

Enabling ‘Reporting’ results in a new section in the Site collection administration sections called ‘Audit log reports’. This section allows the Site Collection Administrators to create one-off audit logs for a range of activity on the site, going back 90 days.

  • Content Activity
    • Content viewing
    • Content modifications
    • Deletion
    • Content type and list modifications
  • Information Management
    • Policy modifications
    • Expiration and Disposition
  • Security and Site Settings
    • Auditing settings
    • Security settings
  • Custom
    • Run a custom report

The 90 day time period is the same as the O365 audit logs accessible from the Security and Compliance ‘Search’ section. If audit logs are required for longer periods, an add on may be required.

Metadata – Site columns or the Managed Metadata Service

The architecture model and/or business requirements may require the use of specific metadata across your environment. Metadata may be set in three ways.

Managed Metadata Service. This option is effective if you need to use the same metadata columns on multiple sites. Experience suggests that this option will be used selectively.

Site columns. These are in addition to the many columns that already exist by default on every site. This option is very effective if the same metadata needs to be used in multiple document libraries or lists on the same site. It is not accessible on any other site. In document libraries or lists, it must be added as an existing site column (i.e., not via the Create new column option).

Library columns. These columns are created in individual libraries or lists and are not accessible on any other library or list.

All new Site and Library columns have the following options:

SPO_NewColumnOptions.JPG

Each new column may be created in an existing or new group. They may also be (a) made mandatory and/or (b) enforce unique values. Note that making a Site Column mandatory and adding it to a document library will make the library read only in File Explorer if it is synced there.

Columns may have default values and may also include JSON formatting codes.

When Site columns are added to a document library, including via a Content Type (see next section), users may be required to fill in the required metadata (especially if it is mandatory).

Site Content Types

Site Content Types are a way to define metadata requirements for different types of documents, using Site columns. The default ‘document’ Content Type on every new SPO site is simply ‘Document’; all new document-based Content Types will be created using that one as a template.

Site Content  Types may also incorporate standard document templates (via the ‘Advanced section). These templates can be auto-populated using the library metadata. In any case, all metadata in a document library is added to any Office document as its metadata ‘payload’.

Once created, Site Content Types must be added to each individual library where they are to be used. To do this, the individual library must have the setting ‘Allow management of content types’ enabled in the ‘Advanced’ section of the document library settings.

SPO_DocLib_ContentTypesYesNo.JPG

When Content Types are enabled in this way, some other drop down features in the ‘+ New’ option on the library disappear, such as the ability to create Word, Excel or PowerPoint documents as can be seen below (the option on the right shows when Content Types are allowed).

Aggregations, containers, ‘files’ – Site Libraries

SharePoint document libraries are the container, aggregation or ‘file’ (if you will) in which records are stored. They are the functional replacement for network file shares. You may end up migrating from those NFS to SharePoint.

Naming conventions for new document libraries are useful to have but the extent to which you require people to follow them (if Site Owners create them) may differ between organisations.

Document libraries ideally should contain only a year of content; including the year in the library name is a good way to maintain year-based content, which in turn makes it easier to manage at the end of the record’s life.

Avoid using the generic ‘Documents’ library that comes with every new library because users will create folders with uncontrolled names and content.

All SPO document libraries and lists have default views of the metadata. These views can be modified as required (via the option on the top right of the menu bar) with a range of additional metadata that is by default hidden from view. Multiple views can be created; pre-defined views may sometimes be easier than expecting users to depend on searches.

Document libraries include all the usual and expected document management functions including check out/in, copy to or move to and versioning.

SPO_DocumentOptionsincCheckOut.JPG

Users with Contribute or Edit permissions can view and restore versions.

SPO_DocLib_VersionHistory.JPG

If there is a requirement to know who modified what part of a document, it is recommended to enable track changes on that document.

Note, with co-authoring now available, the last person to edit the document will create the last version.

Folders and document sets

Folders  should be seen as visual ‘dividers’ within a file, not as ‘hard-coded’ structures as they are in file shares.

Document sets can include additional metadata (including a document ID), making them suitable for use in breaking down a document library. However, for most of the time, folders are a more logical ‘divider’ for users.

Note that both document sets and folders look the same in a synced library.

Both folders and document sets can have unique permissions.

Create and capture records

One of the best reasons for using SharePoint is the ability to create a single source of truth. That is, a single record stored on a library that multiple people can access and work on at the same time.

Having a single source of truth avoids the requirement to (a) create a initial copy on a personal drive or network file share, (b) attach that copy to an email and send it to multiple people who are all likely to save it somewhere and also send back a changed version.

In SPO, users can create a new record directly within a document library (or in the synced library on a drive). Anyone with access to that library can access it; alternatively the document can be shared. Co-authoring means that anyone with edit access can edit the document. Every time it is edited and closed a new version is created.

If it is necessary to refer to the original from another library, the ‘Link’ option can be used.

Access controls and permissions

All SharePoint site contain three default permission groups. Individuals will usually be added to one of these groups only, depending on their access requirements:

  • Site Owner – Full Control across the site but cannot see the Site Collection Administration section (shown above). There will normally be only two to four Site Owners. Site Owners are responsible for managing their sites.
  • Site Member – Update and edit.
  • Site Visitor – Read only.

All content on a SharePoint site inherits the default permissions above however at any point the default permission inheritance can be broken and unique permissions applied. This is a manual process for document libraries (via Advanced Permissions) but automatically applied if a folder or document is shared with someone who is not in a default permission group.

Note, one of the leading support issues in SharePoint is understanding and unravelling complex permissions, especially when applied to individual documents that are placed ‘under’ folders with unique permissions, in libraries with unique permissions.

Retention and disposal

Generally speaking, a SPO site collection will consist of multiple libraries, each (ideally) containing content that is specific to the activity that it relates to. For example, a library for ‘Meetings’ and one for local forms. Consequently, the records stored in document libraries may require different retention.

If O365 Classification labels are used for retention, and depending on how these are configured, these must be applied per library; the individual documents stored in the library – not the entire library as such, are then governed by the retention requirement.

It is also possible to apply a policy to an entire site collection via site policies. This option will only be useful if the entire site can be subject to a single retention requirement – for example, inactive old sites that have a range of content all likely to be covered by the same retention period, or project sites.

Once retention policies are applied to a library, users cannot delete any content in the library so it may be prurient to apply them when the libraries are no longer used instead. Hopefully you will have implemented year-based libraries, which will facilitate this. Alternatively, the retention period trigger can start when the actual policy is applied.

It may be useful for the records manager to review the content of document libraries, and perhaps export the metadata of the library, before the content is disposed of via the O365 Security and Compliance area as any unique metadata is not visible in the Dispositions area.

When records are due to be disposed, an email is automatically send to whoever is in the Records Management role in the O365 Security and Compliance admin portal. The activity of reviewing and approving disposals happens in the O365 Security and Admin portal.

It may be useful to set up an ‘Archives’ SPO site to keep records of all disposal activities, including metadata from document libraries.

Note the library will remain even after the documents are destroyed. An alternative and perhaps better disposal model would be to use the notifications to alert the records manager to the records due for disposal; the records manager may then export and save the metadata in a SharePoint archives site, and then delete the library entirely.

Note that the retention of records in Exchange Online mailboxes and OneDrive may be managed differently by the organisation.

Minimising duplication of content

SharePoint allows organisations to have a single source of truth, to avoid the duplication of using NFS and then uploading to a document management system.

Users can create the record within a SharePoint library, upload it there, use the ‘save as’ option (where you will see all your SPO sites to choose from).

The ability to share with external users (when this is enabled) also helps to reduce duplication and email attachments.

Hybrid records

As noted above, links can be created in any SPO document library to point to resources in a different location. If paper records are managed in a SPO list, the document library can include a link to that SPO list.

Syncing document libraries to File Explorer

Users with an O365 licence and Windows 10 may use the ‘sync’ option available on the ribbon menu of every library. This option syncs the document library to the user’s File Explorer from where they may continue to access and work on the documents.

Note, as discussed in this post, if there is any mandatory metadata in the library, the synced library will become read only.

End users like using the sync option as, although it doesn’t (yet) display any unique metadata on the library, it allows them to work the way they have always worked and they get the added bonus of being able to do it on any device.

eDiscovery

eDiscovery cases are created in the O365 Security and Compliance portal. Essentially, an eDiscovery case uses search and other options to find records. Once found, these records can be placed on Legal Hold, which prevents their disposal.

If a document library has no retention label applied, and all or some of the content is identified as part of the eDiscovery case with a Legal Hold, and a user deletes a record, that record remains in a hidden library but still visible to the eDiscovery case manager. Once the Legal Hold is lifted, the record will resume the 90 day deletion process after which it will no longer be available.

Search

Search in SPO, and across all of Office 365, is very powerful. A single click in the Search box in the user’s SPO portal will result in suggestions before anything is entered.

Searches will return anything the user has access to. The access limits plus the Artificial Intelligence (AI) engine will return different search results for different users.

Users may also take advantage of the Office Graph-powered Delve (E3 licences and above) or the Discovery option in OneDrive to see information that may be of interest to the user. This works on the basis of the various ‘signals’ between users and objects, as depicted in the graphic below.

microsoft-graph_hero-image.png

 

Posted in Compliance, Exchange Online, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal

Managing the retention and disposal of emails in Office 365

In a recent blog post (https://thinkingrecords.co.uk/2019/02/14/managing-email-in-office-365/), James Lappin provided a good overview of the direction that Microsoft have gone with retention and disposal in Office 365.

A key point with almost anything to do with SharePoint Online (that differentiates it from on-premise) is that SharePoint Online (and its ‘personal’ end-user service, OneDrive for Business) is just one element of the Office 365 ecosystem. That is, you can no longer really regard SharePoint as a standalone service that can be managed independently of the other services you may or may not decide to use.

For example:

  • Office 365 Groups (which are an Exchange object, similar to Distribution Lists and Security Groups) all have an associated SharePoint site. O365 Groups are in many way at the ‘heart’ of the Office 365 security/permission model. You cannot create an O365 Group without a SharePoint site.
  • Teams in MIcrosoft Teams (yes the duplication of wording is unfortunate) create an Office 365 Group (which in turn creates a SharePoint site). Alternatively, you can create an Office 365 Group (with a SharePoint site) and link that Group to the new Team. So, Teams in Microsoft Teams have their own Team site.
  • If you enable the ‘Create Site’ option in the end-user SharePoint portal, and the user selects ‘Team site’, this creates an Office 365 Group also.
  • If you allow anyone to create an Office 365 Group, then any new Yammer group creates an Office 365 Groups and – yes, you got it right – a SharePoint.
  • Retention policies for Exchange and SharePoint are set as ‘classification policies’ in the Office 365 Security and Compliance admin portal. This, by the way, is also where you set the new Information Security policies that have only recently appeared. They are both a type of label.

It can be quite overwhelming at first, but the key point is that you cannot regard SharePoint as an isolation application any more. However, most IT shops are pretty ‘hardened’ to the idea that the Exchange ‘box’ (the server) and the SharePoint box are managed by different teams, and one challenge in the new Office 365 world may be to convince the Exchange admins that they should be friends with the SharePoint admins AND the records team.

Backing up as a retention option

It is important to understand that IT departments often regarding ‘backing up’ as a form of retention (or ‘archiving’). Your IT department will almost always have a back-up regime for its on-premises servers.

However, you cannot (easily, cost efficiently) back up SharePoint Online or Exchange Online like you could back up your on-premises environments, but there are many vendors in the market who will offer you a solution to this.

Most IT shops consider back ups to be an archive from which they can retrieve content, a kind of alternative records retention regime. This factor may impact on any decisions that may need to be made with retention policies applied to both Exchange Online and SharePoint Online.

The problem with applying retention and disposal policies to email

It almost goes without saying that, while retention policies can be applied to Exchange Online, typically (a) the content is structured (in multiple folders) differently by every person and (b) the content is mixed together so no retention policy can normally apply to all emails in a single folder.

It is why, generally speaking, we ask users to copy emails into SharePoint (or other EDRM) containers or aggregations (document libraries, files), to keep the content in context.

But in most cases the content (the emails) still remains in Exchange too.

Challenges when applying retention and disposal actions to emails

There are several challenges for the application of records retention and disposal policies in Exchange/Outlook.

  • Do you have a blanket approach to all email, disallowing the deletion of any email for say 7 years?
  • Or do you apply a much shorter retention policy to all emails (say 12 months or less)? (Cue – ‘but what if I want to get my email back after 5 years’ from a user with a labyrinthine email folder structure)
  • Do you rely on users to copy emails to SharePoint or other EDRM containers where they will be stored in context?

The core problem with email is that it’s personal to each user. While it may be good to be able to apply a retention policy to emails, my sense is that anything that is optional will almost always fail to be taken up.

Having a single retention policy (e.g., 7 years) applied to the email accounts of departed users may be a good option (similar to the same policy applied to the OneDrive accounts of departed users).

Another newish option is to use the new Microsoft Flow options to automatically move emails and/or attachments to SharePoint document libraries.

Every organisation is likely to be different and all options need to be considered, understood and then applied – along with the question: ‘Do we (really) need to back this up’?

Posted in Classification, Compliance, Digital preservation, Electronic records, Governance, Information Management, Information Security, Office 365, Records management, Retention and disposal, SharePoint Online

Office 365 – Applying retention periods to SharePoint document libraries and disposal/disposition actions

Records retention policies are created in the Security and Compliance Admin portal, Classifications section of Office 365, as noted in my previous post of 9 March 2018 on the subject.

This post describes how these are applied to document libraries and what happens when the records reach their disposal/disposition period.

Note: In Australia we refer to the disposal of records. In the US this is called disposition.

Setting up retention policies

Organisations may have complex or quite simple records retention policies. An important point to keep in mind in Office 365 is how many policies should be displayed to the end user to choose from.

Ideally, there should be fewer than a dozen classes so they are easy to choose from (see below). There is nothing stopping you creating 100 or 500 policies, but all of them will appear in the drop down list to choose from. Microsoft say they are working on ‘grouping’ policies, so this may help to fix the issue.

For some organisations, it may be useful to distill or group retention policies down to a smaller number.

  • For example, specific retention policies for certain types of records, and one (or two) for ‘all other’ records. The key, as we will see below, is naming them so they are obvious and easy to apply.

Viewing available retention policies

Retention policies that have been created appear in the Security and Compliance Admin portal, under Classifications > Labels.

O365_Classifications_Labels

Note: Labels must be published before they become visible to end users.

When you click on Labels, you can then see all the retention policies that have been created (but not necessarily published).

The screenshot below shows just the very top policy (a test/demonstration policy with a 7 day retention period) in a list of policies.

O365_Classifications_Labels_List.png

Note: Policies can be auto-applied, provided the policy has sufficient ability to identify what records they should be applied to.

Published policies appear in the Data Governance, Dispositions section:

O365_DataGovernance_Dispositions.png

The Dispositions section displays policies that have been published and are visible to end users in the Office 365 areas selected when the policy was created (e.g., Exchange, SharePoint, OneDrive etc).

O365_DataGovernance_Dispositions_List.png

Applying the policy in a SharePoint document library

To apply the policy to a SharePoint document library, go to the document library, library settings, and you will see the option to add the retention policy: ‘Apply label to items in this list or library’.

O365_RetentionPolicy_LibrarySet1.PNG

The ‘Apply Label’ dialogue shows the option to apply the label to existing items (recommended) and a drop down which shows all the published retention policies.

O365_RetentionPolicy_LibrarySet2.PNG

In this example below, there are four policies including the test policy.

O365_RetentionPolicy_LibrarySet3

The policy now applies to all records stored in that document library.

Managing disposal/disposition

When the records reach the end of the retention period configured in the policy, the person designated to be informed about the retention will receive an email notifying them of the need to review the dispositions.

O365_Dispositions_EmailNotification.pngNote, the person (or mailbox) receiving this email MUST be assigned to the Records Management role in the Security and Compliance Admin portal, Permissions section. No-one else will see the records due for disposal otherwise (not even the Global Admins, unless they have also been delegated to that role).

The records person clicks on the link ‘Go there now’ and it opens the following section in the Office 365, Security and Compliance Admin portal, showing the documents that are pending disposition. A number of options are available to sort by Type, to search, and to filter by several options.

 

O365_Dispositions_DocListing

The following options appear if a single document is selected. Note the option to extend the retention period or apply a different label, as well as the ability to delete the item permanently.

O365_Dispositions_Doc_OneDocument

Filtering options are displayed below.

O365_DataGovernance_Dispositions_Filters

Finally, the records manager can choose all the documents in the list and complete three bulk actions as shown.

O365_DataGovernance_Dispositions_BulkActions.png

Positives and negatives

The positives of this method of disposing of documents are that all records from any location will appear in a single view that can be filtered and actions taken as required.

The negatives are that potentially thousands of documents might appear in this listing every single day making it difficult to decide what can deleted or not.

However, as it’s possible to filter by the retention policy, that at least should make it relatively easy to identify what can be destroyed. The more fine-grained the policies, the fewer records should appear.

Organisations that have function-based disposal classes should find that all records relating to the same function appear for disposal under that function.

Another potential negative is that records may not always appear in the same context, whether it be subject- or function-based. For example, a collection of documents (often known as a ‘file’) may not appear in the disposition listing as a collection but as a set of records that are only connected by the disposal policy name. Does this matter?

Recording disposal actions

A key requirement for most organisations is keeping a record of what was destroyed.

At the moment the only apparent option to do this is to apply filters and export the list, using the handy ‘Export’ option to keep a record of what was destroyed. That csv file can then be stored in a control library to ensure a record is kept. This type of action requires a degree of control to ensure it happens every time.

It may also be possible to identify what was destroyed – and by whom – in the audit logs. This is being investigated.

 

Posted in Compliance, Data Loss Prevention - DLP, Exchange Online, Governance, Information Classification, Information Management, Information Security, Legal, Office 365, OneDrive for Business, Products and applications, Security, SharePoint Online, Training and education

SharePoint Online and OneDrive for Business – Preventing external sharing of data

A recent (September 2017) article suggested that OneDrive for Business (ODfB) (and by extension SharePoint Online (SPO); ODfB is a SharePoint-based service), a key application in Office 365 was a potential source of data leaks and/or target for hacking attacks.

I don’t disagree that, if not configured correctly, any online document management system – not just ODfB/SPO – could be the source of leaks or the target of external attacks. Especially if these systems, and the security controls that can protect the data in them, are not properly configured, governed, administered, and monitored.

But, I would ask, what controls do most organisations have in place now for documents stored in file shares and personal file folders, not to mention USB sticks, and the ability to send document via Bluetooth to mobile devices or upload corporate data to third-party document storage systems? Probably not many, because users have no other way to access the data out of the office.

As we will see, the controls available in Office 365 are likely to be more than sufficient to allow users to access to their documents out of the office, while at the same time reducing (if not eliminating) the sharing of documents with unauthorised users.

How to stop or minimise sharing from OneDrive for Business and SharePoint Online

There is one simple way to prevent the sharing of data stored in SPO and ODfB with external people – don’t allow it.

There are several ways to control what can be shared, each allowing the user a bit more capability. All these options should be based on business requirements and information security risk assessments, and Office 365 configured accordingly.

In this article I will start with no sharing allowed, and then show how the controls can be reduced as necessary.

External sharing – on or off

This is the primary setting, found in the main Office 365 Admin centre under Settings > Services & add-ins > Sites. If you turn this off, no-one can share anything stored in SPO or ODfB.

The option is shown below:

O365_SC_Sites_SharingOnOff

If you do allow sharing, you need to decide (as shown above) if sharing will be with:

  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users

The second option is recommended because it doesn’t restrict the ability to share with new users. The last option is unlikely to be used in most organisations and comes with some risks.

The next place to set these options are in the SPO and ODfB Admin centres.

OneDrive admin center

If the previous option is enabled, the following options are available for ODfB. Note that BOTH SharePoint and OneDrive are included here because the latter is a part of the SharePoint environment.

  • Let users share SharePoint content with external users: ON or OFF.
    • NOTE: If this option is turned OFF, all the following options disappear.
  • If sharing with external users is enabled, the following three options are offered:
    • Only existing external users
    • New and existing external users [Recommended]
    • Anyone, including anonymous users
  • Let users share OneDrive content with external users: ON or OFF
    • This setting must be at least as restrictive as the SharePoint setting.
  • If sharing with external users is enabled, the following three options are offered
    • Only existing external users
    • New and existing external users [Recommended]
    • Anyone, including anonymous users

If sharing is allowed, there are three sharing link options:

  • Direct – only people who already have permission [Recommended]
  • Internal – only people in the organisation
  • Anonymous access – anyone with the link

You can limit external sharing by domain, by allowing or blocking sharing with people on selected domains.

External users have two options:

  • External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]
  • Let external users share items they don’t own. [This should normally be disabled]

A final ‘Share recipients’ checkbox allow the owners to see who viewed their files.

SharePoint admin center

The SPO admin center (to be upgraded in late 2017) has two options for sharing.

The first option is under the ‘sharing’ section which currently has the following options:

Sharing outside your organization

Control how users share content with people outside your organization.

  • Don’t allow sharing outside your organization
  • Allow sharing only with the external users that already exist in your organization’s directory
  • Allow users to invite and share with authenticated external users [Recommended]
  • Allow sharing to authenticated external users and using anonymous access links

Who can share outside your organization

  • [Checkbox] Let only users in selected security groups share with authenticated external users

Default link type

Choose the type of link that is created by default when users get links.

  • Direct – only people who have permission [Recommended, same as above]
  • Internal – people in the organization only
  • Anonymous Access – anyone with the link

Default link permission

Choose the default permission that is selected when users share. This applies to anonymous access, internal and direct links.

  • View [Recommended]
  • Edit

Additional settings (Checkboxes)

  • Limit external sharing using domains (applies to all future sharing invitations). Separate multiple domains with spaces.
  • Prevent external users from sharing files, folders, and sites that they don’t own [Recommended]
  • External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]

Notifications (Checkboxes)

E-mail OneDrive for Business owners when

  • Other users invite additional external users to shared files [Recommended]
  • External users accept invitations to access files [Recommended]
  • An anonymous access link is created or changed [Recommended]

Sharing via the Site Collections option

In addition to the options above, sharing options for each SharePoint site are set in the ‘site collections’ section as follows. Note that the default is ‘no sharing allowed’. A conscious decision must be taken to allow sharing, and what type of sharing.

O365_SPO_Sharing1

When a site collection name is checked, the following options are displayed.

Sharing outside your company

Control how users invite people outside your organisation to access content

  • Don’t allowing sharing outside your organisation (default)
  • Allow sharing only with the external users that already exist in your organization’s directory
  • Allow external users who accept sharing invitations and sign in as authenticated users
  • Allow sharing with all external users, and by using anonymous access links

If anonymous access is not permitted (setting above), a message in red is displayed:

Anonymous access links aren’t allowed in your organization

SharePoint Sharing option

The SharePoint Admin Centre has an additional ‘Sharing’ section with the same settings as shown above for ODfB. It is expected that these multiple options will be merged in the new SharePoint Admin Centre due for release in late 2017.

Additional security controls

In addition to all the above settings, there are a range of additional controls available:

  • All user activities related to SPO and ODfB, including who accessed, viewed, edited, deleted, or shared files is accessible in the audit logs.
  • SPO and ODfB content may be picked up by Data Loss Prevention (DLP) policies and users prevented from sending them externally. This is of course subject to the DLP policies being able to identify the content correctly.
  • SPO and ODfB content may be subject to records retention policies set by preservation policies. These may impact on the ability to send documents externally.
  • SPO and ODfB content may be subject to an eDiscovery case.
  • Administrators can be notified when users perform specific activities in both SPO and ODfB.
  • Sharing (and access to the documents once shared) may be subject to security controls enforced through Microsoft Information Protection.

Conclusion

In summary, the settings above allow an organisation to strongly control what can be shared. If sharing is allowed, certain additional controls determine whether the sharing is for internal users or for users external to the organisation. If the latter is chosen, there are further controls on what external users can do. Audit controls and policies may also control how users can share information externally.

The key takeaway is that organisations should ensure that the sharing options available in Office 365 are based on the organisation’s business requirements and security risk framework.