Managing the retention and disposal of emails in Office 365

In a recent blog post (https://thinkingrecords.co.uk/2019/02/14/managing-email-in-office-365/), James Lappin provided a good overview of the direction that Microsoft have gone with retention and disposal in Office 365.

A key point with almost anything to do with SharePoint Online (that differentiates it from on-premise) is that SharePoint Online (and its ‘personal’ end-user service, OneDrive for Business) is just one element of the Office 365 ecosystem. That is, you can no longer really regard SharePoint as a standalone service that can be managed independently of the other services you may or may not decide to use.

For example:

• Office 365 Groups (which are an Exchange object, similar to Distribution Lists and Security Groups) all have an associated SharePoint site. O365 Groups are in many way at the ‘heart’ of the Office 365 security/permission model. You cannot create an O365 Group without a SharePoint site.
• Teams in MIcrosoft Teams (yes the duplication of wording is unfortunate) create an Office 365 Group (which in turn creates a SharePoint site). Alternatively, you can create an Office 365 Group (with a SharePoint site) and link that Group to the new Team. So, Teams in Microsoft Teams have their own Team site.
• If you enable the ‘Create Site’ option in the end-user SharePoint portal, and the user selects ‘Team site’, this creates an Office 365 Group also.
• If you allow anyone to create an Office 365 Group, then any new Yammer group creates an Office 365 Groups and – yes, you got it right – a SharePoint.
• Retention policies for Exchange and SharePoint are set as ‘classification policies’ in the Office 365 Security and Compliance admin portal. This, by the way, is also where you set the new Information Security policies that have only recently appeared. They are both a type of label.

It can be quite overwhelming at first, but the key point is that you cannot regard SharePoint as an isolation application any more. However, most IT shops are pretty ‘hardened’ to the idea that the Exchange ‘box’ (the server) and the SharePoint box are managed by different teams, and one challenge in the new Office 365 world may be to convince the Exchange admins that they should be friends with the SharePoint admins AND the records team.

Backing up as a retention option

It is important to understand that IT departments often regarding ‘backing up’ as a form of retention (or ‘archiving’). Your IT department will almost always have a back-up regime for its on-premises servers.

However, you cannot (easily, cost efficiently) back up SharePoint Online or Exchange Online like you could back up your on-premises environments, but there are many vendors in the market who will offer you a solution to this.

Most IT shops consider back ups to be an archive from which they can retrieve content, a kind of alternative records retention regime. This factor may impact on any decisions that may need to be made with retention policies applied to both Exchange Online and SharePoint Online.

The problem with applying retention and disposal policies to email

It almost goes without saying that, while retention policies can be applied to Exchange Online, typically (a) the content is structured (in multiple folders) differently by every person and (b) the content is mixed together so no retention policy can normally apply to all emails in a single folder.

It is why, generally speaking, we ask users to copy emails into SharePoint (or other EDRM) containers or aggregations (document libraries, files), to keep the content in context.

But in most cases the content (the emails) still remains in Exchange too.

Challenges when applying retention and disposal actions to emails

There are several challenges for the application of records retention and disposal policies in Exchange/Outlook.

• Do you have a blanket approach to all email, disallowing the deletion of any email for say 7 years?
• Or do you apply a much shorter retention policy to all emails (say 12 months or less)? (Cue – ‘but what if I want to get my email back after 5 years’ from a user with a labyrinthine email folder structure)
• Do you rely on users to copy emails to SharePoint or other EDRM containers where they will be stored in context?

The core problem with email is that it’s personal to each user. While it may be good to be able to apply a retention policy to emails, my sense is that anything that is optional will almost always fail to be taken up.

Having a single retention policy (e.g., 7 years) applied to the email accounts of departed users may be a good option (similar to the same policy applied to the OneDrive accounts of departed users).

Another newish option is to use the new Microsoft Flow options to automatically move emails and/or attachments to SharePoint document libraries.

Every organisation is likely to be different and all options need to be considered, understood and then applied – along with the question: ‘Do we (really) need to back this up’?

Advertisements

Office 365 – Applying retention periods to SharePoint document libraries and disposal/disposition actions

Records retention policies are created in the Security and Compliance Admin portal, Classifications section of Office 365, as noted in my previous post of 9 March 2018 on the subject.

This post describes how these are applied to document libraries and what happens when the records reach their disposal/disposition period.

Note: In Australia we refer to the disposal of records. In the US this is called disposition.

Setting up retention policies

Organisations may have complex or quite simple records retention policies. An important point to keep in mind in Office 365 is how many policies should be displayed to the end user to choose from.

Ideally, there should be fewer than a dozen classes so they are easy to choose from (see below). There is nothing stopping you creating 100 or 500 policies, but all of them will appear in the drop down list to choose from. Microsoft say they are working on ‘grouping’ policies, so this may help to fix the issue.

For some organisations, it may be useful to distill or group retention policies down to a smaller number.

• For example, specific retention policies for certain types of records, and one (or two) for ‘all other’ records. The key, as we will see below, is naming them so they are obvious and easy to apply.

Viewing available retention policies

Retention policies that have been created appear in the Security and Compliance Admin portal, under Classifications > Labels.

Note: Labels must be published before they become visible to end users.

When you click on Labels, you can then see all the retention policies that have been created (but not necessarily published).

The screenshot below shows just the very top policy (a test/demonstration policy with a 7 day retention period) in a list of policies.

Note: Policies can be auto-applied, provided the policy has sufficient ability to identify what records they should be applied to.

Published policies appear in the Data Governance, Dispositions section:

The Dispositions section displays policies that have been published and are visible to end users in the Office 365 areas selected when the policy was created (e.g., Exchange, SharePoint, OneDrive etc).

Applying the policy in a SharePoint document library

To apply the policy to a SharePoint document library, go to the document library, library settings, and you will see the option to add the retention policy: ‘Apply label to items in this list or library’.

The ‘Apply Label’ dialogue shows the option to apply the label to existing items (recommended) and a drop down which shows all the published retention policies.

In this example below, there are four policies including the test policy.

The policy now applies to all records stored in that document library.

Managing disposal/disposition

When the records reach the end of the retention period configured in the policy, the person designated to be informed about the retention will receive an email notifying them of the need to review the dispositions.

Note, the person (or mailbox) receiving this email MUST be assigned to the Records Management role in the Security and Compliance Admin portal, Permissions section. No-one else will see the records due for disposal otherwise (not even the Global Admins, unless they have also been delegated to that role).

The records person clicks on the link ‘Go there now’ and it opens the following section in the Office 365, Security and Compliance Admin portal, showing the documents that are pending disposition. A number of options are available to sort by Type, to search, and to filter by several options.

 

The following options appear if a single document is selected. Note the option to extend the retention period or apply a different label, as well as the ability to delete the item permanently.

Filtering options are displayed below.

Finally, the records manager can choose all the documents in the list and complete three bulk actions as shown.

Positives and negatives

The positives of this method of disposing of documents are that all records from any location will appear in a single view that can be filtered and actions taken as required.

The negatives are that potentially thousands of documents might appear in this listing every single day making it difficult to decide what can deleted or not.

However, as it’s possible to filter by the retention policy, that at least should make it relatively easy to identify what can be destroyed. The more fine-grained the policies, the fewer records should appear.

Organisations that have function-based disposal classes should find that all records relating to the same function appear for disposal under that function.

Another potential negative is that records may not always appear in the same context, whether it be subject- or function-based. For example, a collection of documents (often known as a ‘file’) may not appear in the disposition listing as a collection but as a set of records that are only connected by the disposal policy name. Does this matter?

Recording disposal actions

A key requirement for most organisations is keeping a record of what was destroyed.

At the moment the only apparent option to do this is to apply filters and export the list, using the handy ‘Export’ option to keep a record of what was destroyed. That csv file can then be stored in a control library to ensure a record is kept. This type of action requires a degree of control to ensure it happens every time.

It may also be possible to identify what was destroyed – and by whom – in the audit logs. This is being investigated.

 

SharePoint Online and OneDrive for Business – Preventing external sharing of data

A recent (September 2017) article suggested that OneDrive for Business (ODfB) (and by extension SharePoint Online (SPO); ODfB is a SharePoint-based service), a key application in Office 365 was a potential source of data leaks and/or target for hacking attacks.

I don’t disagree that, if not configured correctly, any online document management system – not just ODfB/SPO – could be the source of leaks or the target of external attacks. Especially if these systems, and the security controls that can protect the data in them, are not properly configured, governed, administered, and monitored.

But, I would ask, what controls do most organisations have in place now for documents stored in file shares and personal file folders, not to mention USB sticks, and the ability to send document via Bluetooth to mobile devices or upload corporate data to third-party document storage systems? Probably not many, because users have no other way to access the data out of the office.

As we will see, the controls available in Office 365 are likely to be more than sufficient to allow users to access to their documents out of the office, while at the same time reducing (if not eliminating) the sharing of documents with unauthorised users.

How to stop or minimise sharing from OneDrive for Business and SharePoint Online

There is one simple way to prevent the sharing of data stored in SPO and ODfB with external people – don’t allow it.

There are several ways to control what can be shared, each allowing the user a bit more capability. All these options should be based on business requirements and information security risk assessments, and Office 365 configured accordingly.

In this article I will start with no sharing allowed, and then show how the controls can be reduced as necessary.

External sharing – on or off

This is the primary setting, found in the main Office 365 Admin centre under Settings > Services & add-ins > Sites. If you turn this off, no-one can share anything stored in SPO or ODfB.

The option is shown below:

If you do allow sharing, you need to decide (as shown above) if sharing will be with:

• Only existing external users
• New and existing external users [Recommended]
• Anyone, including anonymous users

The second option is recommended because it doesn’t restrict the ability to share with new users. The last option is unlikely to be used in most organisations and comes with some risks.

The next place to set these options are in the SPO and ODfB Admin centres.

OneDrive admin center

If the previous option is enabled, the following options are available for ODfB. Note that BOTH SharePoint and OneDrive are included here because the latter is a part of the SharePoint environment.

• Let users share SharePoint content with external users: ON or OFF.
  • NOTE: If this option is turned OFF, all the following options disappear.
• If sharing with external users is enabled, the following three options are offered:
  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users
• Let users share OneDrive content with external users: ON or OFF
  • This setting must be at least as restrictive as the SharePoint setting.
• If sharing with external users is enabled, the following three options are offered
  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users

If sharing is allowed, there are three sharing link options:

• Direct – only people who already have permission [Recommended]
• Internal – only people in the organisation
• Anonymous access – anyone with the link

You can limit external sharing by domain, by allowing or blocking sharing with people on selected domains.

External users have two options:

• External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]
• Let external users share items they don’t own. [This should normally be disabled]

A final ‘Share recipients’ checkbox allow the owners to see who viewed their files.

SharePoint admin center

The SPO admin center (to be upgraded in late 2017) has two options for sharing.

The first option is under the ‘sharing’ section which currently has the following options:

Sharing outside your organization

Control how users share content with people outside your organization.

• Don’t allow sharing outside your organization
• Allow sharing only with the external users that already exist in your organization’s directory
• Allow users to invite and share with authenticated external users [Recommended]
• Allow sharing to authenticated external users and using anonymous access links

Who can share outside your organization

• [Checkbox] Let only users in selected security groups share with authenticated external users

Default link type

Choose the type of link that is created by default when users get links.

• Direct – only people who have permission [Recommended, same as above]
• Internal – people in the organization only
• Anonymous Access – anyone with the link

Default link permission

Choose the default permission that is selected when users share. This applies to anonymous access, internal and direct links.

• View [Recommended]
• Edit

Additional settings (Checkboxes)

• Limit external sharing using domains (applies to all future sharing invitations). Separate multiple domains with spaces.
• Prevent external users from sharing files, folders, and sites that they don’t own [Recommended]
• External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]

Notifications (Checkboxes)

E-mail OneDrive for Business owners when

• Other users invite additional external users to shared files [Recommended]
• External users accept invitations to access files [Recommended]
• An anonymous access link is created or changed [Recommended]

Sharing via the Site Collections option

In addition to the options above, sharing options for each SharePoint site are set in the ‘site collections’ section as follows. Note that the default is ‘no sharing allowed’. A conscious decision must be taken to allow sharing, and what type of sharing.

When a site collection name is checked, the following options are displayed.

Sharing outside your company

Control how users invite people outside your organisation to access content

• Don’t allowing sharing outside your organisation (default)
• Allow sharing only with the external users that already exist in your organization’s directory
• Allow external users who accept sharing invitations and sign in as authenticated users
• Allow sharing with all external users, and by using anonymous access links

If anonymous access is not permitted (setting above), a message in red is displayed:

Anonymous access links aren’t allowed in your organization

SharePoint Sharing option

The SharePoint Admin Centre has an additional ‘Sharing’ section with the same settings as shown above for ODfB. It is expected that these multiple options will be merged in the new SharePoint Admin Centre due for release in late 2017.

Additional security controls

In addition to all the above settings, there are a range of additional controls available:

• All user activities related to SPO and ODfB, including who accessed, viewed, edited, deleted, or shared files is accessible in the audit logs.
• SPO and ODfB content may be picked up by Data Loss Prevention (DLP) policies and users prevented from sending them externally. This is of course subject to the DLP policies being able to identify the content correctly.
• SPO and ODfB content may be subject to records retention policies set by preservation policies. These may impact on the ability to send documents externally.
• SPO and ODfB content may be subject to an eDiscovery case.
• Administrators can be notified when users perform specific activities in both SPO and ODfB.
• Sharing (and access to the documents once shared) may be subject to security controls enforced through Microsoft Information Protection.

Conclusion

In summary, the settings above allow an organisation to strongly control what can be shared. If sharing is allowed, certain additional controls determine whether the sharing is for internal users or for users external to the organisation. If the latter is chosen, there are further controls on what external users can do. Audit controls and policies may also control how users can share information externally.

The key takeaway is that organisations should ensure that the sharing options available in Office 365 are based on the organisation’s business requirements and security risk framework.

Applying (new) Retention Policies to Office 365 Content

From time to time I’m asked about the way records retention policies ‘work’ in SharePoint. A common criticism has been that SharePoint’s retention model is based on applying retention policies to individual records (e.g., documents in a library or individual emails) rather than to aggregations of records, the most obvious of which is a document library.

The idea of storing and managing related records together in a single aggregation derives from the management of paper records – in files, boxes, and series. This model (of aggregations containing all records relating to a given subject) was largely replicated in electronic document management systems (EDMS – many of which were used to register paper files and boxes previously) when they appeared or were modified to manage digital records in the late 1990s.

In fact, many EDM systems did not actually manage records in an aggregation; the actual digital records were stored in a secure network file stored, and presented in the EDMS user interface though a common ‘file number’ (or similar) ID.

In any case, the ability to store all digital records on the same subject together in the one system (e.g., EDMS) was always hampered by the fact that (a) email and documents were created by different systems, (b) stored in different locations (servers), and (c) use of network file shares continued more or less unabated.

The increasing complexity and types of digital records underlines the difficulty of ever storing, let alone managing or applying retention and disposal actions, to them in a single aggregation.

Until recently, Microsoft’s retention and disposal options reflected the fact that applications used to create digital records stored them in different locations (servers) – Exchange and SharePoint. Retention policies targeted individual records stored in those applications, rather than aggregations.

In March 2017, Microsoft introduced a new, single central way to create and apply retention and disposal policies to most Office 365 content, wherever it was stored – Exchange, SharePoint, OneDrive for Business, Office 365 Groups, and Skype for Business.

This post:

• Summarizes the existing ‘out of the box’ retention and disposal options in SharePoint, but not Exchange (see my earlier post on this subject).
• Discusses issues with existing retention and disposal options in SharePoint.
• Describes how the new centrally-managed retention policies and labels can be applied to most content in Office 365.
• Discusses why applying retention policies to individual records rather than aggregations may be a better option in the digital world.

Records managers working in organisations that use Office 365 to manage records should familiarize themselves with the way these new retention policies work.

Note: The details in this post are based on the Australian recordkeeping context, which may be different from your specific location.

SharePoint out of the box (OOTB) retention and disposal options

Until recently, the only available OOTB options to apply retention and disposal actions to SharePoint were to:

• Apply an information management policy to an entire site via the Site Collection Settings. This option is suitable for short-lived sites such as project or closed, archived sites, but less suitable for long-lived team sites which might have a range of different content.
• Create a retention policy using the information management policy settings in Content Types. This option applies the policy to individual records. Content Types also include the ability to ‘transfer’ (actually copy) records after a defined period to another location, such as a Records Center.
• Use a folder-based information management policy. This option requires the default Content Type-based policy on a document library to be changed via Library Settings – Information Management Policy Settings, to Library and Folders.

Another option was to adopt a form of ‘retention in place’ and regard each library as a logical aggregation of records, the equivalent of a ‘file’, and manage retention and disposal manually or using PowerShell scripts to identify libraries for potential disposal based on the last modified date of the records. Some vendors have developed a similar model to manage retention policies on libraries using a central ‘console’.

Applying retention and disposal actions to individual records

Both the Content Type and folder-based options noted above apply the retention policy to individual records in the library, not the library (aggregation/container) as a whole.

That is, disposal was based on a time period after which each individual record was created, modified, or declared a record. The logic behind this model appears to be that a document library may store multiple record types each with different retention requirements. This may not be true for all document libraries, but it usually is for many.

Applying automated disposal actions on individual records (rather than an aggregation of records) is probably counter-intuitive for most records managers. The main concerns, from a recordkeeping (and possibly also archival) point of view are the absence of (a) a documented review and approval process before the records are destroyed, and (b) a metadata record of what was destroyed. That is, the records simple disappear from the document library, removing records that may would be relevant to the context of the original aggregation. This, of course, assumes that all records relating to the subject were stored in a single aggregation which, as noted above, may not always be the case.

Global Retention Policies and Labels in Office 365

In March 2017, Microsoft introduced two new ‘global’ retention options – retention policies and labels – to Office 365. The two options allow organisations to apply centrally set and apply retention policies to the same type of record, in whatever form and wherever they are stored – emails in Exchange, documents and lists in SharePoint, conversations (in Office 365 Groups and Skype)..

Examples of ‘types’ of information could include:

• Corporate records that must be kept for the life of the company.
• Financial records that need to be kept for 7 years.
• ‘Working records’ that could be deleted after a minimum period of time.
• Personnel records or staff files that had to be kept indefinitely.

As Tony Redmond noted in this recent article, these new retention policies build on the type of retention policies first released in Exchange 2010 using folder, system, personal and default tags. The article suggests that organisations that have applied Exchange retention policies may need to consider the impact of these new types of policies. In particular, the ability to move email to archive mailboxes is lost, replaced with a retention policy.

How Retention Policies work

Retention policies in Office 365 are created by authorized users (ideally, records managers) in the Retention section of the Security and Compliance Center.

Creating a new retention policy

Each policy has the following options: Name, Settings, Locations and Preservation Lock.

Name

The name of the retention policy should reflect the class name or number in the records retention schedules so that it can be easily identified and applied to content wherever it can be applied in Office 365 (see below for ‘Locations’).

Settings

The two Settings options are based on two questions:

• Do you want to retain the content? 
  • If ‘Yes, I want to retain it’ is selected, the choices are either ‘Forever’ or a configurable ‘n days/months/years’ (e.g. 7 years). The administrator must then decide if, once it reaches that point, the record should be deleted or not. If ‘Yes’ is selected, the content will be deleted from where it is currently stored as described in the next two points.
  • >>For SharePoint content there are two options when the retention period expires. (1) If the record has not been modified or deleted it will be deleted from the original library where it was stored, and then remain in the two-stage Recycle Bin for up to 90 days. (2) If the content has been modified or deleted, it is transferred to the hidden Preservation Hold library that is created when the retention policy is applied to a SharePoint site and deleted from that library. In this case, the administrator has only 7 days to recover the content before it is deleted permanently.
  • >>For Exchange content there are also two options. (1) If the item is modified or permanently deleted by the user during the retention period, the item is copied (if modified) or moved (if deleted) to the Recoverable Items folder. The retention policy process identifies and deletes items whose retention period has expired within 14 to 30 (configurable) days of the end of the retention period.  (2) If the item is not modified or deleted during the retention period, the same process runs on all folders in the mailbox and identifies items whose retention period has expired. These items are also permanently deleted within 14 to 30 days of the end of the retention period. (Note: If a user leaves the organization, and their mailbox is included in a retention policy, the mailbox becomes an inactive mailbox. ‘The contents of an inactive mailbox are still subject to any retention policy that was placed on the mailbox before it was made inactive.)
  • If ‘No’ is selected, the content will be left in place and must be manually deleted at some point.
• No, just delete the content that’s older than … The options are to delete: (a) after ‘n days/months/years’, and (b) based on when it was created or modified.

The (subtle) difference between these two options is that the first option (Yes) ensures that records are not permanently deleted before the end of the retention period, while the second option (No) just deletes records permanently at the end of the retention period.

Advanced retention settings are also available these allow the administrator to create a search query with specific words phrases, or link the policy with the same sensitive information options found under DLP policies, e.g., financial, medical and health, privacy, and custom.

Locations

The Locations section sets where the policy will be applied. By default this is all locations across Office 365, including content in Exchange, SharePoint, OneDrive, Office 365 Groups and Skype for Business.

• Office 365 has a limit of 10 organisation-wide policies and entire-location policies combined per tenant. Therefore, careful consideration should be given to what specific types of record need a global policy, especially given that not all types of records will be found globally across the organisation.

The alternative option is to apply the policy only to specific locations or users. In most cases this is likely to be Exchange and SharePoint where the majority of key records are created and stored.

• A retention policy that includes or excludes over 1,000 specific users can contain no more than 1,000 mailboxes and 100 sites. A tenant can contain no more than 1,000 such retention policies. According to Microsoft ‘… you can get over these limits by applying either an org-wide policy or a policy that applies to entire locations’.

Retention policies applied to a SharePoint site or OneDrive account result in the creation of a hidden Preservation Hold library as noted above.

Retention policies applied to Exchange user mailboxes apply the policy to the mailbox. For public folders, the retention policy is applied at the folder level.

Preservation Lock

Finally, the administrator has the option to apply a Preservation Lock, which prevents anyone from changing or deleting the policy after it is turned on. This option should only be applied in specific circumstances as it cannot be turned off or made less restricted (by anyone, including the administrator) after it has been applied. .

Review and save

Finally, the new retention policy should be reviewed, may be saved for later, or published.

Labels

A separate option for managing retention and disposal is to use (retention) labels, which should not be confused with security labels. This option is designed to replace the following:

• Exchange Online retention tags and retention policies, also known as messaging records management (MRM).
• In SharePoint Online and OneDrive for Business: (a) in-place records management, (b) the Records Center, and (c) information management policies.

Labels are used to manage retention policies for specific types of content across the Office 365 environment. Labels can be applied automatically to content if it matches certain conditions or keywords (E5 licence only), or manually by users to emails, documents, or Office 365 Group conversations.

See below for the relationship and priority between retention policies and labels.

Who can create labels

Labels are created by individuals (ideally records managers or similar) assigned to a compliance role in the Security and Compliance Admin portal in Office 365.

Creating Labels

Labels are created in the Security and Compliance Admin Portal under ‘Classifications’. Labels may also be created without having an associated retention policy; that is, a label can be created and applied to content as no more than a visual ‘tag’. A policy can be added to it at a later stage.

If the ‘Retention’ option is enabled for labels (on/off switch), a new section appears titled ‘When users apply this label to content’. This section is where the retention policy is defined with two options:

• Retain the content. The choices are either ‘Forever’ or ‘n days/months/years’ (e.g., 7 years). The administrator must decide if, once it reaches that point, the labelled record should be deleted or not. The ‘Yes’ and ‘No’ options are the same as for retention policies, described above.
  • If ‘Yes’ is selected, the record will be deleted from where it is stored. Administrators have 93 days to recover records that have not been edited or deleted, or 7 days to records that have been edited or deleted (and moved to the Preservation Hold library).
  • If ‘No’ is selected, the content will be left in place and must be manually deleted.
• Don’t retain the content. The choices are to delete (a) after ‘n days/months/years’, and (b) based on when the record was created, modified, or labelled.

If the first option (‘Retain the content’) above is selected a check box option allows the administrator to use the label to classify content as a record. If the content is classified as a record, users are unable to change or delete the content or change or remove the label. They may still, however, edit the metadata.

The final step in the process is to review the settings. Once created, the administrator is returned to the main Labels screen which displays the label that has been created, allowing the administrator to then publish it.

Label limitations when used on a SharePoint document library

There are some limitations to applying a default label to a SharePoint document library:

• It applies the label to all records except those that already have a label and those contained in document sets.
• If the default label is removed, it removes the label from all records except those that have a label and those contained in document sets.
• Labels cannot be applied to folders in SharePoint or OneDrive (but can be applied to folders in Exchange).
• If the record is moved to a different library that has a different default label, it will inherit that label. Conversely, if it is moved to a library with no label, the existing label will be removed.

Note: When labels are published to an Office 365 group, the labels appear in both the group site and group mailbox in Outlook on the web. The experience of applying a label to content is identical to that shown above for email and documents.

What about legal holds?

eDiscovery in Office 365 is based around the creation of ‘cases’ in a SharePoint eDiscovery site. Cases are generally established in response to litigation (or potential litigation) and can be used to search across a range of sources. Once found, the information that forms part of the case can then be placed on hold, overriding any retention policy. However, once the hold is released, retention policies on records continue.

For more information on this subject, see:

https://support.office.com/en-gb/article/Add-content-to-a-case-and-place-sources-on-hold-in-the-eDiscovery-Center-54d70de9-1ec2-4325-84f3-aeb588554479?ui=en-US&rs=en-GB&ad=GB

What’s the relationship between retention policies and labels?

Retention policies and labels do the same thing but the former is more likely to be set centrally, while the latter is set by the end user. This means that a record could have more than one retention policy applied to it.

According to Microsoft’s documentation (link below), records will be retained until the end of the longest retention period applied to it, regardless of whether that policy was based on the retention policy or the label.

Are retention policies and labels better than previous retention options?

One of the primary benefits of the new retention policy regime in Office 365 is that it enables organisations to apply retention policies centrally rather than do this separately for each application (e.g., Exchange, SharePoint) as was the case until recently. It also allows end users to apply retention policies via labels.

Retention and disposal continues to be based on the individual record, or type of record (as defined by the policy or label), not logical aggregations or containers of records such as a document library.

As noted above, the concept of an aggregation that contains all the records on a given subject is ill-suited to the digital world. The reality is that records may be created using different applications (e.g., email in Exchange, document, list item or page in SharePoint, conversation in Groups, discussions in Skype etc) and stored in multiple application locations (e.g. in Exchange folders, SharePoint libraries, etc).

The dilemma for many records managers using Office 365 is how to store or manage records together in context, including based on the organisation’s File Plan or Business Classification Scheme (BCS) terms. The need to keep records together has been the driver behind the integration of EDRM systems with email applications, allowing email to be ‘captured’ in the EDRM along with other types of documents. This has rarely been successful in practice and, in most cases, emails are duplicated and remain stored in the email server.

The new Office 365 retention policies, including those applied as labels to specific types of content, may well be the answer to this dilemma. Rather than try to capture all types of records (e.g, document email, list item, conversation) in a single aggregation or container, Office 365 allows the option for them to be stored wherever the user prefers, with the same retention policy applied.

If necessary, all records with the same label can then be found using a content search in the ‘Search and Investigation’ section of Office 365.

In my view, there are still some shortcomings in basing retention policies on individual record types:

• Individual documents, rather than logical aggregations of documents, will be continue to be subject to disposal actions.
• Records that may provide context to other records (including those stored in different locations) may be destroyed.
• Appraisal options may be limited and appropriate review and approval steps before disposal may not be possible.
• Disposal actions may be automatic and unrecoverable.
• There may be no record kept, including the metadata, of the individual records that were destroyed.
• It is not known how courts might view the automatic disposal of records without prior review and approval.

Final thoughts

The new Office 365 records retention policy and label options centralise the management of retention and disposal for most types of records across Office 365, reducing complexity.

Retention and disposal continues to be based on individual records rather than aggregations, but this may be better suited to the digital world in which aggregations of records may not always be achievable.

Records managers working in organisations using Office 365 need to understand and provide guidance to IT on how records retention schedules can be applied as retention policies, and how they can be directly involved in decisions regarding the new options.

For more information: –

https://support.office.com/en-us/article/Overview-of-retention-policies-5e377752-700d-4870-9b6d-12bfc12d2423

https://support.office.com/en-us/article/Overview-of-labels-af398293-c69d-465e-a249-d74561552d30

 

Office 365 & SharePoint Online – Data Loss Prevention (DLP)

Summary

Office 365 includes a range of information security and protection capabilities. This post focusses on the configuration and implementation of Data Loss Prevention (‘DLP’) capabilities in SharePoint Online and OneDrive for Business (ODfB).

Note: Microsoft have advised that the Office 365 DLP framework will apply to both Exchange and SharePoint/ODfB in the near future. For Exchange settings see https://technet.microsoft.com/en-us/library/jj200706(v=exchg.150).aspx and https://technet.microsoft.com/en-us/library/jj150527(v=exchg.150).aspx for more information.

Purpose of DLP

The purpose of DLP is to protect specific and definable types of sensitive company or agency information by preventing (or monitoring) its deliberate or inadvertent exfiltration from the organisation.

Examples of exfiltration methods where DLP can be used include:

• Attachments to emails.
• Uploads to web-based systems.

Examples of the types of sensitive information that can be protected with DLP include:

• Financial data. For example, bank account numbers, tax file numbers, credit/debit card numbers.
• Personal and sensitive information (PSI). For example, driver’s licence numbers, tax file numbers, passport numbers.
• Medical and Health records. For example, medical account numbers.

The requirement to protect sensitive information is the subject of legislation in a number of countries.

Enabling Data Loss Prevention in Office 365

DLP in Office 365 enabled through policies that are set in the Security and Compliance Admin Centre of the Office 365 Admin Portal, under ‘Threat Management’ > ‘Data Loss Prevention’.

DLP policies are set by the Office 365 Global Administrator, as well as the Compliance Administrator and/or the Security Administrator if these roles have been configured in the Security and Compliance Admin Centre.

To create a DLP policy, the Administrator clicks on the + icon in the Data Loss Prevention screen. This opens a new window with the following options displayed.

A custom policy is one that is defined by the organisation. It would normally be for content that contains specific values.

The options ‘Financial regulations’, ‘Medical and health regulations’, and ‘Privacy regulations’ include default Microsoft-provided policies. Each of these default policies includes a description, coverage (e.g., what information is protected), and where the information is to be protected (e.g., in SharePoint Online, OneDrive for Business, and Exchange Online).

Enabling and modifying default policies

After selecting a default policy, the authorised user must then identify the services that may store the information that need to be protected – SharePoint Online, OneDrive.

Note: The option to choose Exchange Online is (as of 13 March 2017) still unselectable.

The next option allows the Administrator to customise the rule that has been chosen. If a default policy has been selected in the previous dialog, options for that policy will display; these may include ‘count sensitivity’ (i.e., how many times the sensitive content is identified. Low count means high sensitivity to sensitive content.

The Administrator may add a new rule or edit one of the default options.

The Administrator may modify the conditions, actions and what happens when there is an incident for each of the default policies – see below for further details.

Defining custom DLP policies

If a custom policy is required, the Administrator clicks on ‘Custom Policy’ from the ‘Data Loss Prevention’ opening dialog screen, and then ‘Next’ at the bottom of the screen. The Administrator must define which services are to be protected (same as for default policies, above).

The next screen allows the Administrator to create a new policy, via the + icon.

In the new window that opens, the Administrator can then must define the new DLP rule through Conditions, Actions, Incident reports and General.

Conditions, Actions, Incident reports, General options

For either default or custom policies, the Administrator must set the following rules:

• Conditions – what will cause the policy to run?
• Actions – what will happen when the policy runs?
• Incident reports – how is reporting managed?
• General – any other points.

Conditions

For default policies, conditions are pre-defined and are based on (a) the type of content (e.g., credit card numbers, bank account numbers) and (b) whether the content is shared internally or externally.

These pre-defined conditions may be removed or edited, and new conditions may be added. Editing options include the number of times the sensitive content is found (‘Min count’, ‘Max count’), and both maximum and minimum percentage-based ‘confidence levels’.

For custom policies, the Administrator must define which conditions are to be met:

• If you choose ‘Content contains sensitive information’, you must define the information through a + option. This brings up all the default choices provided by Microsoft.
• If you choose ‘Content is shared with’, it allows you define if the information is shared with people inside or outside the organisation.
• If you choose ‘Document properties contain any of these values’, you must define the values that would be found in a document. Note that, if this option is selected, the property must be configured in the SharePoint Online search settings.

Actions

For default policies, the actions to be taken are pre-defined and are based on sending a notification.

For custom policies, the Administrator must first decide whether the action will be to (a) block the content or (b) send a notification.

If ‘Block the content’ is selected the user will be unable to send an email or access the shared content.

If ‘Send a notification’ is selected it offers the same options as for custom policies. Note the ability to customise the email notification.

Incident Reports

When ‘Incident Reports’ is selected for both custom and default policies, the following options are available. Incident reports should be sent to the Administrator/s.

General

Default policies are pre-named but the name can be modified. This is also where the policy can be disabled.

Custom policies must be named and a decision made whether to enable it, test it, or turn it off. As noted below it is possible to test the policy first, to collect data.

DLP Reporting

Reporting from the DLP policies is accessed from the Security and Compliance Centre > Reports > Dashboard.

Document Audit Trails in SharePoint 2010

Part 2 of ISO 16175 (‘Principles and functional requirements for records in electronic office environments’) includes a section (5.4.7) on records management process metadata. This section contains 13 mandatory requirements including:

• To create unalterable metadata of records management actions on records, aggregations of records, or the classification scheme.
• To track events without manual intervention.
• To maintain that metadata ‘for as long as required’.
• Provide metadata of all changes.
• Document all changes made to administrative parameters.
• Ensure that the metadata is available for inspection on request.
• Allow the metadata to be exported.
• Record any violations, or attempted violations (e.g., access attempts).
• Provide reporting.
• Allow configuration.
• Allow an administrator to make authorised changes, which are also recorded.

SharePoint 2010 provides audit trails on all sites but, unlike some other systems, audit trail information is not included in the metadata about the document itself. That is, records management process metadata is not included with other metadata about the record, but must be accessed separately, and only by a Site Owner or someone with higher privileges.

Records management process metadata is captured in two ways in SharePoint:

• By enabling it in a Content Type that is enabled in a Site Library.
• By enabling it in the Site Collection settings.

Audit trails may be deleted regularly as a default setting

Process metadata (‘audit trails’, or ‘audit logs’) are not kept permanently by default but are ‘trimmed’ according to a schedule for audit logs that is configured by the SharePoint (or server) Administrator in Central Administration.

The reason for not keeping the logs is to do with performance. According to Microsoft’s online help ‘… auditing can potentially generate a large number of audit events, creating a large audit logs. This could fill the hard drive, affecting performance and other aspects of a site collection’.

The default trim schedule is ‘end of the month’, which means that all previous metadata is deleted and a new log is started. Alternative options include a given period, for example 30 days or 7 days.

This means that audit trails for documents will not exist beyond the configured time period.

Turning off the default schedule

To disable the default schedule, the Site Collection Administrator must change the configuration settings of the ‘Audit Log Trimming’ section of the ‘Configure Audit Settings’ option under ‘Site Collection Administration’.

The Site Administrator may also opt to store the audit reports in a site collection library before they are trimmed. The three options are:

• Automatically trim the audit log for this site? This is set to ‘Yes’ by default, which means it will trim the logs in accordance with the Farm default setting.
• Specify the number of days of audit log data to retain. The Site Administrator may set the days when the audit log will be trimmed, regardless of the Farm default setting.
• Specify a location to store audit reports before trimming the audit log. This option allows the Site Administrator to automatically save audit logs to the location identified so the logs are not lost.

Other configuration options in the ‘Configure Audit Settings’ section are:

• Documents and Items. There are five events that can be audited: (a) opening or downloading documents, viewing items in lists, or viewing item properties; (b) editing items; (c) checking out or checking in items; (d) moving or copying items to another location in the site; and (e) deleting or restoring items.
• Lists, Libraries, and Sites. There are three events that can be audited: (a) editing content types and columns; (b) searching site content; and (c) editing users and permissions.

Audit Log Reports

Once auditing is enabled, the audit logs are available via Site Settings – Site Collection Administration – Audit Log Reports.

There are usually four options available:

• Content Activity Reports. This includes: (a) content modifications (‘all events that modified content’ – this includes editing, checking out/in, moving/copying); (b) content type and list modifications; (c) content viewing (‘all events where a user viewed content in this site’); and (d) deletion (‘all events that caused content in this site to be deleted or restored’).

• Custom Reports. This allows the Administrator to manually specify filters for a report.

• Information Management Policy Reports. This includes reports on: (a) expiration and disposition and (b) policy modifications.

• S

  ecurity and Site Settings Reports

  .

  This includes: (a) auditing settings (‘all events that change the auditing settings’) and (b) security settings (‘all events that change the security configuration of SharePoint’).

Audit trail report content

Audit trails in SharePoint are exported to Excel and include two worksheets, a summary and the detail.

The summary worksheet shows the number of actions (e.g., view, edit) on a document. The detailed worksheet includes a row of information similar to the following:

Site Id: {fffaf4fd-b5d0-4940-a615-06e0026a855d}
Item Id: {373b3b5e-43a7-46d8-8d44-9a36fad6a4da}
Item Type: Document
User Id: John Smith <DOMAIN\jsmith>
Document Location: _catalogs/masterpage/v4.master
Occurred (GMT): 2013-02-13T04:02:45
Event: View
Custom Event Name: (may be blank)
Event Source: SharePoint
Source Name: (may be blank)
Event Data: <Version><Major>1</Major><Minor>0</Minor></Version>

Compliance with ISO 16175

While SharePoint 2010 does create audit trails for records management actions, it does not store these audit trails in an end-user visible form with the record itself; that is, when an end user views the metadata describing the record, he/she cannot view the audit trail history of the record. Only a Site Administrator can access and/or view the audit trails.

The other main weakness, in terms of compliance with ISO 16175, is that the audit trails could be deleted according to the default schedule set by the farm administrator. To get around this, a choice needs to be made whether to override the default trimming, or (to reduce storage requirements) to export the audit logs regularly and store them on the same site.

Enforcing Information Management Systems … and Strategies that Work

Extracted from a speech given to the 4th Information Management & E-Discovery Summit, Sydney, Australia, 10th June 2010.

Digital information is everywhere. One of the biggest problems for organisations is the volume of so-called unstructured information commonly found on network drives and in email folders.

Electronic document management systems (EDMS), often now known as enterprise content management (ECM) systems, are seen as the panacea to this problem, however unless they are mandated for all unstructured information, users will continue to have some level of discretion about what goes into them.

In response to this, organisations may implement an email archiving system to at least capture emails, or use back up tapes as a form of archive tape, or move information to some other form of long-term storage.

All of these options have shortcomings, not the least of which is that they may end up storing information for much longer than necessary, thereby exposing the organisation to the risk of this information being discovered.

So, you are probably a bit confused about what to do, and what direction to take.

Do you:

(a) Acquire and implement a system in the hope that you can get users to put unstructured information into it, and somehow manage the information that’s not captured, including by using email archiving or network drive archiving/duplication?

(b) Acquire, implement and mandate a system across the organisation for all unstructured information?

(c) Don’t acquire anything, and risk manage it all?

If you decide to go with option (a), here are some suggestions for managing unstructured information more effectively in organisations.

Have a good information management strategy

Not everything (usually) needs to be managed in the same way.  It would be an unusual organisation that wanted to manage everything in exactly the same way.  Some information must be managed well, some less so.

A good strategy will show a good understanding of the following elements:

• Your organisation’s digital information assets.
• Your compliance requirements and standards, including records management best practice, standards and compliance needs.
• The way your users, and your organisation, currently do their work.
• Your current technological environment (your technological limits or opportunities).
• The external information management and electronic document management environment (what is happening out there).
• How contemporary electronic document management systems work, as they are not all the same.

And then develop your information management strategy and EDMS implementation plan accordingly.

Communicate this strategy through information provided via the intranet, all staff emails, desk drops, information provided through managerial meetings.

Make sure there is effective communication at all phases of the project

This is essential, almost critical to success.  More is better.  Best projects have very good communication.

Keep people informed and they will keep up with you.  They may even use your system.

When considering change management, remember the technology adoption lifecycle and the five broad categories of users.

There will always be innovators, early adopters, the early majority and late majority and finally the laggards.

And yes, unfortunately, these are often (but not always) reflective of the ages of your staff.  Target your changes, and the required training, accordingly.

Have realistic and understandable policies and procedures for managing electronic documents

A key element in any information strategy, therefore, is having and implementing an effective policy that addresses the management and retention of information.

But, make sure it is a policy that everyone can access, understand, and put into practice.  Recent research conducted in Malaysia in 2004 and 2008 indicated that less than 50% of organisations had a policy for managing electronic documents, and of the 50% that did, less than 25% of respondents complied with it.  The one’s who didn’t comply said they didn’t have the time, or found it too difficult, to do so.

Interestingly, the research also showed that most respondents said they wanted such policies in the form of guidelines for managing electronic records.

What does this research indicate to us?  That good communication and effective change management are critical to support policies and help users to use systems to keep records; if you don’t have one, you probably won’t succeed.

Have a good, realistic defensible, and well considered project scope document

Don’t underestimate just how hard it is to implement these systems.  After all, you are making a fundamental and sometimes unintuitive change to the way people manage information.

This is a key part of any proposed project.  What are you trying to achieve?

Installing a new product is easy, changing staff behaviours is very difficult.

What’s your objective?  Why is it your objective?  Is it compliance, if so, compliance with what?  There would be few organisations in Australia for which compliance with something is the only driver for an organisation-wide EDMS.

When defining the project scope and project plan, you should have a really good handle on the following:

• What information are we actually trying to manage?
• What are the actual compliance drivers?  (see below)
• How do people actually manage digital AND paper information now?
• What are the risks with current processes and practices?
• What risks are you trying (or needing) to mitigate against?
• What is happening externally?

A simple rule of thumb that seems to work in most cases is this: the greater your risks of not complying or needing to be accountable (including because of anticipated litigation), the more likely your users will manage your information well.

Or, to put it another way, you are more likely – and probably should – spend most of your effort managing the information that you have to manage well.

There are several risks associated with users continuing to use uncontrolled systems such as network drives and email folders to manage records.  Making them aware of these risks through policies, procedures or guidelines can be a good idea.

The risks are:

• Being unable to find and produce information required except through costly data recovery or forensic processes.
• Not meeting compliance requirements, that is, managing information poorly.
• Keeping information for longer than required.

The longer we keep information that doesn’t need to kept, the more we expose the organisation to potential embarrassment and expense through e-discovery and other forms of information retrieval.

Understand your real compliance requirements

What are your actual compliance mandates?  If users know what they are, they may respond better to a new system.  Often this is not well communicated.

It is very important to have a very thorough understanding of your actual compliance requirements.

And then, based on that understanding, take a risk based approach to the management of your information.  This will in turn inform how you approach the issue of user uptake of a system.

In August 2007 the Management Advisory Committee (MAC) of the Federal Government’s Public Service Board published one of the most useful common sense documents to come out of Canberra, ‘Note for File’.

The report acknowledged the key difficulties faced by agencies in managing information, the problems of information stored unofficially on network drives and in email and other electronic systems.  It recommended agencies take a risk based approach and prioritise their recordkeeping attention on activities that pose the greatest level of risk.

Get senior management support

Will your CEO use the system? If not, why not? My CEO uses our system.  It’s a great selling point for staff.

Have end user participation and contribution from the beginning

This is part of good business analysis.

Understand your organisation and its business needs, preferably with some product knowledge behind you.

Find the key users, the early adopters. Consult. Listen. Identify problem areas.

BUT! Recognise that users are not always right and remember that customisation can be expensive.

Have well defined system requirements – but don’t over do it

Understand your business needs and define your requirements for a system carefully.

Understand and define what the relationship will be between the existing systems, drives and email folders and the new system, and how end users will work in the future.  Storybook it even.

Many products probably already do what you want.  These systems offer users folders containing documents, wrapped up in access, security and audit requirements, sharing and collaboration models, with retention and disposal capability.

The more you customise the base product, the more expensive and complicated it gets.

Find the product that best meets your needs

Find a product that is the best match for your business needs, not one that has the best marketing team, or is flavour of the month.  Many products have been around for years.  They started out as DM, became EDM, then EDRM, and are now ECM systems.

There is a lot of maturity in this market and, in recent years, a great deal of upheaval and acquisitions.

TRIM was bought by HP.  Hummingbird and Vignette were acquired by Open Text.  The old iManage was bought by Interwoven and became Worksite (and Mailsite), then was acquired by Autonomy.  Documentum by ECM, and in the wake of those acquisitions Alfresco appeared.

Really get to know the product and what it does, including your infrastructure requirements

So, what’s the best system for users to use?  One that meets your objectives.  One that will:

• Meet your business, regulatory and litigation compliance requirements and standards for keeping, storing or managing information and records.
• Be used by users, with minimal training.
• At a cost that is affordable and gives true return on investment.

It doesn’t, necessarily, have to manage everything.

Don’t forget about recordkeeping requirements.  If you follow the principles outlined in ISO 15489 you are more likely to have good records that will meet compliance requirements.

A realistic project plan AND change plan

Have a separate change and project plan, although they may be in the same document.

I’ve seen extraordinarily complex project plans for projects that didn’t succeed, and very simple project plans that did succeed.

Success can be as much about a great plan as it is about the people and resources doing the work, and the capability of the users in the organisation to change.

Failure, on the other hand, is often about poor scoping and/or scope creep, poor understanding of the organisation’s capability for change, poor change management, poor communication, user rejection, and, frankly, the wrong choice of technology.

As they say, successful projects have proud parents, the unsuccessful ones are orphans.

Have a great change management plan

You need a good change plan, especially for the late majority and late adopters.  In fact, some people suggest you should only target these groups for change.

Change is not popular.  People fear change and tend to be somewhat parochial and protective of the technology they have got used to.

Moving from a paper-based to an electronic-based way of managing information is, perhaps, one of the greatest challenges over the past decade.

Understand the technology adoption lifecycle.

Software – know your proposed or acquired product – well.

I mean, really well.  How often do we see organisations evaluating and acquiring a system, and then engaging a systems integrator to implement a product, without having a good idea of how the product really works in the first place?

This is as much about the acquisition process as it is about the implementation process.

In one case I was involved in recently, the organisation spent a lot of money getting external consultants to help define the business requirements of the system, which were then passed to an integrator who did exactly what was required.

Problem was, the system then failed because it was not designed to work in the way that the users had requested, the consultant had defined, and the integrator had made it work.  Find out how other organisations use the same product.

You should also know your vendor very well, too, along with any proposed systems integrator.

Customise only when necessary

This is an interesting one and in many ways related to the previous point. Clearly it is a good idea to engage staff in how they want a system to work.

However, many systems are, by default, capable of doing most of what the majority of users want without customisation.

You need to tread a very fine line between listening to users and implementing what they want.

The more complex your project in terms of customisation (in particular), the more likely it is the majority of potential early adopters will miss out.

Maintain sufficient flexibility to get your project over the line within time and budget.

In a case I was involved in last year, the organisation persisted in developing a complex and expensive set of customisations, when the product out of the box actually met 98% of user requirements.

Recordkeeping

Don’t forget about recordkeeping issues, including retention periods for the information.  This is closely linked to your policy.

Remember, keeping records for too long can be just as risky as not keeping them, because it exposes the organisation to legacy information that should probably have been deleted.

Use qualified resources

Use qualified resources, people who have a good track record.

Post implementation evaluation of the project

Look around at other projects. Learn the lessons. Communicate them!