Office 365 – Applying retention periods to SharePoint document libraries and disposal/disposition actions

Records retention policies are created in the Security and Compliance Admin portal, Classifications section of Office 365, as noted in my previous post of 9 March 2018 on the subject.

This post describes how these are applied to document libraries and what happens when the records reach their disposal/disposition period.

Note: In Australia we refer to the disposal of records. In the US this is called disposition.

Setting up retention policies

Organisations may have complex or quite simple records retention policies. An important point to keep in mind in Office 365 is how many policies should be displayed to the end user to choose from.

Ideally, there should be fewer than a dozen classes so they are easy to choose from (see below). There is nothing stopping you creating 100 or 500 policies, but all of them will appear in the drop down list to choose from. Microsoft say they are working on ‘grouping’ policies, so this may help to fix the issue.

For some organisations, it may be useful to distill or group retention policies down to a smaller number.

• For example, specific retention policies for certain types of records, and one (or two) for ‘all other’ records. The key, as we will see below, is naming them so they are obvious and easy to apply.

Viewing available retention policies

Retention policies that have been created appear in the Security and Compliance Admin portal, under Classifications > Labels.

Note: Labels must be published before they become visible to end users.

When you click on Labels, you can then see all the retention policies that have been created (but not necessarily published).

The screenshot below shows just the very top policy (a test/demonstration policy with a 7 day retention period) in a list of policies.

Note: Policies can be auto-applied, provided the policy has sufficient ability to identify what records they should be applied to.

Published policies appear in the Data Governance, Dispositions section:

The Dispositions section displays policies that have been published and are visible to end users in the Office 365 areas selected when the policy was created (e.g., Exchange, SharePoint, OneDrive etc).

Applying the policy in a SharePoint document library

To apply the policy to a SharePoint document library, go to the document library, library settings, and you will see the option to add the retention policy: ‘Apply label to items in this list or library’.

The ‘Apply Label’ dialogue shows the option to apply the label to existing items (recommended) and a drop down which shows all the published retention policies.

In this example below, there are four policies including the test policy.

The policy now applies to all records stored in that document library.

Managing disposal/disposition

When the records reach the end of the retention period configured in the policy, the person designated to be informed about the retention will receive an email notifying them of the need to review the dispositions.

Note, the person (or mailbox) receiving this email MUST be assigned to the Records Management role in the Security and Compliance Admin portal, Permissions section. No-one else will see the records due for disposal otherwise (not even the Global Admins, unless they have also been delegated to that role).

The records person clicks on the link ‘Go there now’ and it opens the following section in the Office 365, Security and Compliance Admin portal, showing the documents that are pending disposition. A number of options are available to sort by Type, to search, and to filter by several options.

 

The following options appear if a single document is selected. Note the option to extend the retention period or apply a different label, as well as the ability to delete the item permanently.

Filtering options are displayed below.

Finally, the records manager can choose all the documents in the list and complete three bulk actions as shown.

Positives and negatives

The positives of this method of disposing of documents are that all records from any location will appear in a single view that can be filtered and actions taken as required.

The negatives are that potentially thousands of documents might appear in this listing every single day making it difficult to decide what can deleted or not.

However, as it’s possible to filter by the retention policy, that at least should make it relatively easy to identify what can be destroyed. The more fine-grained the policies, the fewer records should appear.

Organisations that have function-based disposal classes should find that all records relating to the same function appear for disposal under that function.

Another potential negative is that records may not always appear in the same context, whether it be subject- or function-based. For example, a collection of documents (often known as a ‘file’) may not appear in the disposition listing as a collection but as a set of records that are only connected by the disposal policy name. Does this matter?

Recording disposal actions

A key requirement for most organisations is keeping a record of what was destroyed.

At the moment the only apparent option to do this is to apply filters and export the list, using the handy ‘Export’ option to keep a record of what was destroyed. That csv file can then be stored in a control library to ensure a record is kept. This type of action requires a degree of control to ensure it happens every time.

It may also be possible to identify what was destroyed – and by whom – in the audit logs. This is being investigated.

 

Advertisements

Changes to security classification and records retention in Office 365

In May 2016, I wrote about the creation of security classification labels in the Azure Information Protection (AIP) portal (old post here). Quite a bit has changed since that post, in particular the naming of policies, away from ‘High’ to ‘Low’ Business Impact (e.g., HBI – LBI) to real-world words such as ‘General’ and ‘Highly Confidential’.

In October 2017, I wrote about the new retention policies that could be applied to all Exchange, SharePoint and OneDrive content in Office 365.

Changes to the Security and Compliance admin portal – Classifications section

On 23 February 2018, Microsoft’s Adam Jung posted a new article to the Microsoft Tech Community titled ‘Consistent labeling and protection policies coming to Office 365 and Azure Information Protection’.

The main outcome of this change is that information security protection and records retention policies, linked with Data Loss Prevention (DLP policies) are created from a single interface in the Security and Compliance admin centre > Classifications section (Labels). These policies are set in Office 365 are then synced to Azure (and vice versa).

To quote the Microsoft blog: ‘The upcoming experience means that the same default labels can be used in both Office 365 and Azure Information Protection, and the labels you create in either of these services will automatically be synchronized across the other service – no need to create labels in two different places!’

This post looks at the changes and some potential issues that may arise.

Security and Compliance Admin Portal – Classifications

Records retention policies for Office 365 content are set as labels in the Security & Compliance Admin portal of Office 365 under Classifications – Labels.

The Classifications area also includes a section for ‘Sensitive Information Types’, which simply lists a range of information types that are also used for DLP policies.

Note: Access to that Admin portal is restricted by default to Global Admins and anyone assigned to a specific security role. Records managers in organisations that have or are deploying Office 365 should have access to this feature.

Setting (Records Retention) Classification Labels

The options for setting a records retention label were described in detail in my post above, but for reference again, they are:

• Name
• Label settings
  • Disabled or enabled (off/on)
  • When enabled, the ability to set (a) a retention period, and (b) an action when the period expires.
  • Alternatively, it is possible to just delete content when it’s older than a given time.
  • An option also allows the content be to be classified as a ‘record’ when the label was applied, providing further protection against deletion, for example.
• Review your settings

Merging of label options – Retention and Security together in a single label

The primary change to classifications is the inclusion of new options when you choose to ‘Create a Label’.

These options are now:

• Label name
• Protection settings (e.g., information security)
• Retention settings
• Advanced options settings
• Review your settings

These options are described below.

The ‘Protection settings’ section includes the following options:

• Enabled or disabled. (If disabled the next check box options do not appear)
• Block users from sending email messages or sharing documents with this label
• Show policy tip to users if they send or share labeled content (The text of the policy tip is editable)
• Send incident reports in email
• Advanced protection for content with this label (Customise settings option)

The ‘Retention settings’ are identical with the options already described above:

• Disabled or enabled
• Various settings when enabled.

The ‘Advanced options settings’ section includes the following options:

• Enabled or disabled. (If disabled the next check box options do not appear)
• Add a watermark (text can be customised)
• Add a header (text can be customised)
•  Add a footer (text can be customised)

The Microsoft article notes: ‘We are building labeling capabilities natively into the core Office apps – including Word, PowerPoint, Excel, and Outlook, and soon there will be no need to download or install any additional plug-ins.’ This comment references the problem of having to download a plug-in for the classification options to appear in installed versions of Office.

Does it make sense to merge security classifications and records retention?

In my opinion, putting information security and records retention policies in the same label doesn’t make sense.

Retention is almost never linked with the confidentiality (or otherwise) of the records but based on government or legislative requirements or business needs.

But that was probably not Microsoft’s intention; it was probably to make it as simple as possible to create and apply these policies.

It would have made more sense to have separate label options for ‘Retention policies’ and ‘Security policies’. This would potentially mean, however, having two labels (if a label is in fact required for retention purposes).

Organisations with complex retention policies might find that the mixing of both policies in the one view makes it harder to find the individual security related policies, and have the potential to cause some confusion.

For example, it is could be hard to spot the Highly Confidential label in this listing if there were more than (say) 50 retention classes:

• Client records – 7 years
• Confidential
• Financial Records – 7 years
• Highly Confidential
• Internal Use Only
• Meeting Records – 3 years
• Working Paper – 1 year

It also raises the question (which I have asked and will update this post if I receive a response) as to whether two policies can (or should) be applied on a document.

If two labels cannot be applied, this could mean that organisations have to have even more labels to take account of the various combinations. For example:

• General Financial Records – 7 years
• Confidential Financial Records – 7 years
• Highly Confidential Financial Records – 7 years

Not to mention the link to DLP policies, although that doesn’t appear as a label.

In my opinion, combining these two options, while perhaps making it easier at the ‘front end’, has the potential to create confusion for users, let alone complicate the administration of retention management.

Read the full Microsoft blog article in the link below

https://techcommunity.microsoft.com/t5/Security-Privacy-and-Compliance/Consistent-labeling-and-protection-policies-coming-to-Office-365/ba-p/161553

Office 365 – new data governance and records retention management features

At the September 2017 Ignite conference in Orlando, Florida, Microsoft announced a range of new features coming soon to data governance in Office 365.

These new features build on the options already available in the Security and Compliance section of the Office 365 Admin portal. You can watch the video of the slide presentation here.

Both information technology and records management professionals working in organisations that have Office 365 need to work together to understand these new features and how they will be implemented.

Some of the key catch-phrases to come out of the presentation included ‘keep information in place’, ‘don’t horde everything’, ‘no more moving everything to one bucket’, ‘three-zone policy’, and ‘defensible deletion process’. The last one is probably the most important.

How do you manage the retention of digital content?

If your organisation is like most others, you will have no effective records retention policy or process for emails or content stored across network file shares and in ‘personal’ drives.

If you have an old-style EDRM system you may have acquired a third-party product and/or tried to encourage users (with some success, perhaps) to store emails in that system, in ‘containers’ set up by records managers.

The problem with most of these traditional methods is that it assumes there should be one place to store records relating to a given subject. In reality, attempts to get all related records in the one place conjures up the ‘herding cats’ problem. It’s not easy.

What is Microsoft’s take on this?

For many years now, Microsoft have adopted an alternative approach, one that is not dissimilar to the view taken by eDiscovery vendors such as Recommind. Instead of trying to force users to put records in a single location, it makes more sense to use powerful search and tagging tools to find and manage the retention of records wherever they are stored.

Office 365 already comes with powerful eDiscovery capability, allowing the organisation to search for and put on hold records relating to a given subject, or ‘case’. But it also now has very powerful records retention tools that are about to get even better.

This post extends my previous posting ‘Applying New Retention Policies to Office 365 Content‘, and won’t repeat all of it as a result.

Where do you start?

A standard starting point for the management of the retention and disposal of records is a records retention schedule. These are also known in the Australian recordkeeping context as disposal authorities, general disposal authorities, and records authorities. They may be very granular and contain hundreds of classes, or ‘big bucket’ (for example, Australian Federal government RAs).

Records retention schedules usually describe types of records (sometimes grouped by ‘function’ and ‘activity’, or by business area) and how long they must be retained before they can be disposed of, unless they must be kept for a very long time as archival records.

The classes contained in records retention schedules or similar documents become retention policies in Office 365.

Records retention in Office 365

It is really important to understand that records retention management in Office 365 covers the entire environment – Exchange (EXO), SharePoint (SPO), OneDrive for Business (OD), Office 365 Groups (O365G), Skype for Business. Coverage for Microsoft Teams and OneNote is coming soon. Yammer will not be included until at least the second half of 2018.

That is, records retention is not just about documents stored in SharePoint. It’s everything except as noted.

Records managers working in organisations that have implemented (or are implementing) Office 365 need to be on top of this, to understand this way of approaching and managing the records retention process.

Retention policies in Office 365 are set up in the Security and Compliance Admin Centre, a part of the Office 365 Admin portal. Ideally, records managers should be allocated a role to allow them to access this area.

There are two retention policy subsections:

• Data Governance > Retention > Policy
• Classification > Labels > Policy

The settings in both are almost identical but have slightly different settings and purposes. However, note all retention policies that are set up are visible in both locations.

The difference between the two options is that:

• Retention-based policies are (according to Microsoft) meant for IT to be used more for ‘global’ policies. For example, a global policy for the retention of emails not subject to any other retention policy.
• Label-based policies map to the individual classes in a retention schedule or disposal authority.

Note: Organisations that have many hundreds or even thousands of records retention classes will need to create them using Powershell.

Creating a retention-based policy

Retention-based policies have the following options:

Directly underneath this are two options:

• Find specific types of records based on keyword searches [COMING > also label-based]
• Find Data Loss Prevention (DLP) sensitive information types. [COMING > label-based DLP-related polices can be auto-applied]

A decision must then be made as to where this policy will be applied – see below.

Creating a label-based policy

To create a classification label manually, click on ‘Create a label’.

Note:

• Labels are not available until they are published.
• Labels can be auto-applied

The screenshot below shows the options for creating a new label.

Label- based policies have the following settings:

• Retain the content for n days/months/years
• Based on Created or Last Modified [COMING > when labelled, an event*]
• Then three options: (a) delete it after n days/months/years (b) subject it to a disposition review process (labels only), or (c) don’t delete.

* Such as when certain actions take place on the system.

 

Applying the policies

Once a policy has been created it can then be applied to the entire Office 365 environment or to only specific elements, for example EXO, SPO, OD, O365G.

• IT may want to establish a specific global policy
• Most other policies will be based on the organisation’s records retention schedule

Once they have been published, labels may then be applied automatically or users can have the option to apply them manually.

In EXO, a user may create a folder and apply the policy there. All emails dragged into that folder will be subject to the same policy.

In SPO, retention policies may be applied to a document library and can be applied automatically as the default setting to all new documents. [COMING > also to a folder and a document set]. Adding a label-based policy to a library also creates a new column so the user can easily see what policy the documents are subject to.

Note: Individual documents stored in the library will be subject to disposal, not the library. 

What about Content Types?

Organisations that have used content types to manage groups of records including for retention management will be able to continue to do so, but Microsoft appears to take the view (in the presentation above) that this method should probably replaced by labelling. This points needs further consideration as content types are usually used as a way to apply metadata to records.

Note: If the ability to delete content (emails, documents) is enabled, any deleted content subject to a retention policy will be retained in a hidden location. The option also exists when a label-based policy is created to ‘declare’ records based on the application of a label. 

What happens when records are due for disposal?

Once the records reach the end of their retention period, they will be:

• Deleted
• Subject to a new disposition review process [COMING in 2017 – see below]
• Remain in place (i.e., nothing happens)

In relation to the second option above, a new ‘Disposition’ section under Data Governance will allow the records manager or other authorised person to review records (tagged for Disposition Review) that have become due for disposal.

This is an important point – only records that had a label with the option ‘Disposition Review’ checked will be subject to review. All other records will be destroyed. Therefore, if the organisation needs to keep a record of what was destroyed, then the classification label must have ‘Disposition Review’ selected.

Records that are reviewed and approved to be destroyed are marked as ‘Completed’. This means there is a record of everything (subject to disposition review) that has been destroyed, a key requirement for records managers.

Other new or coming features

A number of other new features demonstrated at the Ignite conference, are coming.

• Labels will have a new ‘Advanced’ check box. This option will allow records marked with that label to have any of the following: watermark, header/footer, subject line suffix, colour.
• Data Governance > Records Management Dashboard. The dashboard will provide an overview of all disposition activity.
• Data Governance > Access Governance. This dashboard, which supports data leakage controls, will show any items that (a) appear to contain sensitive content and (b) can be accessed by ‘too many’ people.
• Auto-suggested records retention policies. The system may identify groups of records that do not seem to be subject to a suitable retention policy and make a recommendation to create one.
• For those parts of the world who need it, new General Data Protection Regulations (GDPR) controls
• Microsoft Information Protection, to replace Azure Information Protection and provide a single set of controls over all of Microsoft’s platforms.

The Recordkeeping World of Office 365 – More than just SharePoint

I’m often asked (and sometimes challenged to confirm) if SharePoint can manage records. The question is often based on a unspoken second element – like ‘xyz’ system?

It might be (and sometimes is) argued that any system can manage records if the records stored in the system provide evidence of business activity and are considered information assets. But recordkeeping is more than just keeping any records in any system.

The international standard for records management, ISO 15489-1-2016 states: ‘They (records) can be distinguished from other information assets by their role as evidence in the transaction of business and by their reliance on metadata. Metadata for records is used to indicate and preserve context and apply appropriate rules for managing records.’

So, records are not just distinguished by their evidentiary role, but also by their reliance on metadata.

Record types

In my opinion, it is a mistake to focus solely on SharePoint when the question is asked – can it manage records? The question assumes that one application will be used to store all the records – or at least the so-called ‘unstructured’ records – created by the organisation. Any discussion of SharePoint must include and take account of SharePoint Online as part of the Office 365 ecosystem.

The term ‘unstructured’ in this sense generally means email and any kind of digital record that can be saved to network file shares. The latter generally includes documents (in multiple formats), images, and other digital record types.

However, ‘what gets saved on a network files share’ generally overlooks the fast-increasing volumes of information that would not normally be ever stored in a file share. Simple examples of records that are never (or may never be) saved in network file shares (or dedicated recordkeeping systems) include tweets on Twitter, messages sent by personal messaging apps, and social network type information.

I’m always impressed (albeit a bit sceptical) when I hear that organisations state they are capturing all these types of information in their recordkeeping system. That’s pretty impressive.

Records in Office 365

Many organisations have moved (or are moving) their Microsoft enterprise licencing to Office 365, Microsoft’s subscription-based service. Office 365 includes a range of applications that create or store records including:

• Exchange (email, calendars, Groups, Planner)
• Office (used to create the content)
• SharePoint / OneDrive for Business* (document libraries and lists)
• Microsoft Teams
• Sway
• Skype for Business
• Stream (video)

*OneDrive for Business is a SharePoint-based service.

SharePoint is only one part of this information rich ecosystem, and really shouldn’t be thought of as a single destination for the storage of records. Yes, it can be used to store and manage records, but you need to stand back a little to appreciate the full picture.

How Microsoft (may) have approached recordkeeping in Office 365

Until recently, and in the on-premise world, records were stored and managed separately in Exchange and SharePoint, each with their own recordkeeping capabilities, quite independent of each other.

Over the past two years, Microsoft developed a unified strategy for recordkeeping in both systems, presumably based on the likelihood that most corporate records would continue to be stored separately. The requirement for additional recordkeeping metadata in either system would remain optional – see below.

In mid 2017, Microsoft introduced a centralised way to create, manage and apply retention policies to content stored across (no longer separately in) Exchange, SharePoint and OneDrive for Business. These new policies are created as labels in the Office 365 – Security and Compliance Portal under the slightly misleading section called ‘Classifications’.

These new retention policies can be applied across Exchange, SharePoint and OneDrive for Business. In SharePoint, they can be applied to a site, a document library (preferred), list, or individual documents. They remove the requirement to manage retention separately in Exchange and SharePoint.

But what about the metadata?

As noted above, the international standard for recordkeeping states that metadata ‘is used to indicate and preserve context and apply appropriate rules for managing records’.

So why or how is metadata optional in Office 365?  I think this is for two reasons:

• Making any form of metadata mandatory will turn users off using the system.
• Metadata may not be the ‘be all and end all’ for context-based discovery.

‘Context’ essentially means that records relating to a given context (e.g., ‘noise complaints’) can be identified, retrieved and managed in that context. For example, emails relating to a meeting; the meeting agenda and minutes may be stored in one location but more often than not the emails remain on the Exchange server. Another example might be emails relating to the development of a new policy; again, these are more often than not stored separately from the system used to store and manage documents.

Regardless of where they may be stored, metadata should provide and indicate the context of any records that may be created. Years of EDRMS use suggests that users generally don’t like to add additional metadata to records.

So how does Office 365 do this?

In most organisations, the only ways to apply recordkeeping metadata to an email is to save it to an EDRMS or in SharePoint. Most organisations will rarely configure Exchange to include the capture of metadata.

As with an EDRMS, the metadata options in SharePoint are more or less unlimited but careful thought needs to go into what metadata should be applied, and how. For example, metadata can be set:

• In the Managed Metadata Service (MMS)/Term Store, including with hierarchical models
• As site columns
• As library columns

Regardless of the option selected, metadata may be set as a default on each SharePoint document library column. That is, when a record (including an email) is saved to a specific library, it can be assigned specific metadata that is to be assigned to all documents in that library.

Applying metadata in this way, especially as site columns, means that information can be retrieved in context.

It should also be kept in mind that Microsoft Office documents saved to a SharePoint document library also retain their metadata in the document properties, even when the document is exported, a kind of ‘metadata payload’.

Is metadata still relevant?

In the mid 1990s, Yahoo introduced a new portal that allowed users to browse the nascent internet based on pre-defined categories. That is, a form of metadata tagging was applied to all content that allowed the user to browse to where they wanted to go to.

The problem with this idea was that it assumed everybody would understand the categories. Google’s response to this was to provide a single search box allowing users to retrieve whatever they were looking for – subject, of course, to the way the algorithm presented the information to the user based on their understood context.

Adding metadata to indicate the context of a record works as long as the context is still valid – both for the content and the user. Or, to put it another way, the way in which I might describe a record with metadata may be different from the way you want to access that record, because your context is not the same as mine. There may be a range of information that I want to find that hasn’t necessarily been recorded in the context in which I am looking for it.

Some years ago I was curious why users in one business area could not find many records relating to a specific subject – noise complaints in a city area well known for its nightlife. In most cases, they were searching for records containing complaints about noise in that specific area, recorded in the title or metadata of the record.

When we asked them to ignore the metadata and search by the content of the records they found thousands of records, all described in different contexts – building approvals and inspections, delivery of services, police liaison, visitor numbers and public feedback. All these contexts were quite valid, but they were not the context of the user searching for the records.

The lesson learned was simple – my context is not necessarily your context. Records, especially digital records, could relate to any context including future and unpredictable contexts.

Context-based information and eDiscovery

For some users, one of the most ‘startling’ features of Office 365 is Delve and the related Discover option in the user’s OneDrive for Business. Both are based on the underlying Office Graph that learns a user’s context based on their interactions (or ‘signals’) across the Office 365 environment and presents potentially relevant content (to which they have access) from SharePoint or another user’s OneDrive.

I used the term ‘startling’ because, for most users, the idea that you can find out what others are working on seems intuitively to be some kind of breach of privacy (even though they have access to that content already). And yet, what it is doing is letting a user know, based on her or his context, what may be of interest from potentially quite different contexts. It does this based on the interactions between users.

Office 365 also includes a powerful eDiscovery capability that allows the user (if licenced to do so) to find all information across Exchange and SharePoint relating to a specific context regardless of where it is stored, and quarantine that information as required in a case file. While metadata may assist in the process, it is not essential.

But what about all the other records?

So far I have not said anything about the records produced by and stored in the other Office 365 applications such as Teams, Planner, Skype for Business and so on. Or about the management of records produced in third-party social media or messaging applications.

The Office Graph already takes into account the interactions between users to present potentially relevant information stored in SharePoint or OneDrive for Business. At some point in the future, Microsoft may include the information in the various other Office 365 applications.

As for social media, the preferred model may be to capture the feed of that information in an Office 365 service – Microsoft Teams, for example, can receive a feed from third-party applications including Twitter. The answer to the use of third-party messaging applications is to use applications that have at least the same or, preferably, better functionality. Teams and Skype for Business are in this space.

Summary

If you have got to the end of this article, thank you for reading.

In summary, my main point is that when thinking about SharePoint for recordkeeping it is a good idea to consider it in the context of the broader Office 365 ecosystem and its recordkeeping capabilities, not as an isolated application capable of storing and managing records.

Applying (new) Retention Policies to Office 365 Content

From time to time I’m asked about the way records retention policies ‘work’ in SharePoint. A common criticism has been that SharePoint’s retention model is based on applying retention policies to individual records (e.g., documents in a library or individual emails) rather than to aggregations of records, the most obvious of which is a document library.

The idea of storing and managing related records together in a single aggregation derives from the management of paper records – in files, boxes, and series. This model (of aggregations containing all records relating to a given subject) was largely replicated in electronic document management systems (EDMS – many of which were used to register paper files and boxes previously) when they appeared or were modified to manage digital records in the late 1990s.

In fact, many EDM systems did not actually manage records in an aggregation; the actual digital records were stored in a secure network file stored, and presented in the EDMS user interface though a common ‘file number’ (or similar) ID.

In any case, the ability to store all digital records on the same subject together in the one system (e.g., EDMS) was always hampered by the fact that (a) email and documents were created by different systems, (b) stored in different locations (servers), and (c) use of network file shares continued more or less unabated.

The increasing complexity and types of digital records underlines the difficulty of ever storing, let alone managing or applying retention and disposal actions, to them in a single aggregation.

Until recently, Microsoft’s retention and disposal options reflected the fact that applications used to create digital records stored them in different locations (servers) – Exchange and SharePoint. Retention policies targeted individual records stored in those applications, rather than aggregations.

In March 2017, Microsoft introduced a new, single central way to create and apply retention and disposal policies to most Office 365 content, wherever it was stored – Exchange, SharePoint, OneDrive for Business, Office 365 Groups, and Skype for Business.

This post:

• Summarizes the existing ‘out of the box’ retention and disposal options in SharePoint, but not Exchange (see my earlier post on this subject).
• Discusses issues with existing retention and disposal options in SharePoint.
• Describes how the new centrally-managed retention policies and labels can be applied to most content in Office 365.
• Discusses why applying retention policies to individual records rather than aggregations may be a better option in the digital world.

Records managers working in organisations that use Office 365 to manage records should familiarize themselves with the way these new retention policies work.

Note: The details in this post are based on the Australian recordkeeping context, which may be different from your specific location.

SharePoint out of the box (OOTB) retention and disposal options

Until recently, the only available OOTB options to apply retention and disposal actions to SharePoint were to:

• Apply an information management policy to an entire site via the Site Collection Settings. This option is suitable for short-lived sites such as project or closed, archived sites, but less suitable for long-lived team sites which might have a range of different content.
• Create a retention policy using the information management policy settings in Content Types. This option applies the policy to individual records. Content Types also include the ability to ‘transfer’ (actually copy) records after a defined period to another location, such as a Records Center.
• Use a folder-based information management policy. This option requires the default Content Type-based policy on a document library to be changed via Library Settings – Information Management Policy Settings, to Library and Folders.

Another option was to adopt a form of ‘retention in place’ and regard each library as a logical aggregation of records, the equivalent of a ‘file’, and manage retention and disposal manually or using PowerShell scripts to identify libraries for potential disposal based on the last modified date of the records. Some vendors have developed a similar model to manage retention policies on libraries using a central ‘console’.

Applying retention and disposal actions to individual records

Both the Content Type and folder-based options noted above apply the retention policy to individual records in the library, not the library (aggregation/container) as a whole.

That is, disposal was based on a time period after which each individual record was created, modified, or declared a record. The logic behind this model appears to be that a document library may store multiple record types each with different retention requirements. This may not be true for all document libraries, but it usually is for many.

Applying automated disposal actions on individual records (rather than an aggregation of records) is probably counter-intuitive for most records managers. The main concerns, from a recordkeeping (and possibly also archival) point of view are the absence of (a) a documented review and approval process before the records are destroyed, and (b) a metadata record of what was destroyed. That is, the records simple disappear from the document library, removing records that may would be relevant to the context of the original aggregation. This, of course, assumes that all records relating to the subject were stored in a single aggregation which, as noted above, may not always be the case.

Global Retention Policies and Labels in Office 365

In March 2017, Microsoft introduced two new ‘global’ retention options – retention policies and labels – to Office 365. The two options allow organisations to apply centrally set and apply retention policies to the same type of record, in whatever form and wherever they are stored – emails in Exchange, documents and lists in SharePoint, conversations (in Office 365 Groups and Skype)..

Examples of ‘types’ of information could include:

• Corporate records that must be kept for the life of the company.
• Financial records that need to be kept for 7 years.
• ‘Working records’ that could be deleted after a minimum period of time.
• Personnel records or staff files that had to be kept indefinitely.

As Tony Redmond noted in this recent article, these new retention policies build on the type of retention policies first released in Exchange 2010 using folder, system, personal and default tags. The article suggests that organisations that have applied Exchange retention policies may need to consider the impact of these new types of policies. In particular, the ability to move email to archive mailboxes is lost, replaced with a retention policy.

How Retention Policies work

Retention policies in Office 365 are created by authorized users (ideally, records managers) in the Retention section of the Security and Compliance Center.

Creating a new retention policy

Each policy has the following options: Name, Settings, Locations and Preservation Lock.

Name

The name of the retention policy should reflect the class name or number in the records retention schedules so that it can be easily identified and applied to content wherever it can be applied in Office 365 (see below for ‘Locations’).

Settings

The two Settings options are based on two questions:

• Do you want to retain the content? 
  • If ‘Yes, I want to retain it’ is selected, the choices are either ‘Forever’ or a configurable ‘n days/months/years’ (e.g. 7 years). The administrator must then decide if, once it reaches that point, the record should be deleted or not. If ‘Yes’ is selected, the content will be deleted from where it is currently stored as described in the next two points.
  • >>For SharePoint content there are two options when the retention period expires. (1) If the record has not been modified or deleted it will be deleted from the original library where it was stored, and then remain in the two-stage Recycle Bin for up to 90 days. (2) If the content has been modified or deleted, it is transferred to the hidden Preservation Hold library that is created when the retention policy is applied to a SharePoint site and deleted from that library. In this case, the administrator has only 7 days to recover the content before it is deleted permanently.
  • >>For Exchange content there are also two options. (1) If the item is modified or permanently deleted by the user during the retention period, the item is copied (if modified) or moved (if deleted) to the Recoverable Items folder. The retention policy process identifies and deletes items whose retention period has expired within 14 to 30 (configurable) days of the end of the retention period.  (2) If the item is not modified or deleted during the retention period, the same process runs on all folders in the mailbox and identifies items whose retention period has expired. These items are also permanently deleted within 14 to 30 days of the end of the retention period. (Note: If a user leaves the organization, and their mailbox is included in a retention policy, the mailbox becomes an inactive mailbox. ‘The contents of an inactive mailbox are still subject to any retention policy that was placed on the mailbox before it was made inactive.)
  • If ‘No’ is selected, the content will be left in place and must be manually deleted at some point.
• No, just delete the content that’s older than … The options are to delete: (a) after ‘n days/months/years’, and (b) based on when it was created or modified.

The (subtle) difference between these two options is that the first option (Yes) ensures that records are not permanently deleted before the end of the retention period, while the second option (No) just deletes records permanently at the end of the retention period.

Advanced retention settings are also available these allow the administrator to create a search query with specific words phrases, or link the policy with the same sensitive information options found under DLP policies, e.g., financial, medical and health, privacy, and custom.

Locations

The Locations section sets where the policy will be applied. By default this is all locations across Office 365, including content in Exchange, SharePoint, OneDrive, Office 365 Groups and Skype for Business.

• Office 365 has a limit of 10 organisation-wide policies and entire-location policies combined per tenant. Therefore, careful consideration should be given to what specific types of record need a global policy, especially given that not all types of records will be found globally across the organisation.

The alternative option is to apply the policy only to specific locations or users. In most cases this is likely to be Exchange and SharePoint where the majority of key records are created and stored.

• A retention policy that includes or excludes over 1,000 specific users can contain no more than 1,000 mailboxes and 100 sites. A tenant can contain no more than 1,000 such retention policies. According to Microsoft ‘… you can get over these limits by applying either an org-wide policy or a policy that applies to entire locations’.

Retention policies applied to a SharePoint site or OneDrive account result in the creation of a hidden Preservation Hold library as noted above.

Retention policies applied to Exchange user mailboxes apply the policy to the mailbox. For public folders, the retention policy is applied at the folder level.

Preservation Lock

Finally, the administrator has the option to apply a Preservation Lock, which prevents anyone from changing or deleting the policy after it is turned on. This option should only be applied in specific circumstances as it cannot be turned off or made less restricted (by anyone, including the administrator) after it has been applied. .

Review and save

Finally, the new retention policy should be reviewed, may be saved for later, or published.

Labels

A separate option for managing retention and disposal is to use (retention) labels, which should not be confused with security labels. This option is designed to replace the following:

• Exchange Online retention tags and retention policies, also known as messaging records management (MRM).
• In SharePoint Online and OneDrive for Business: (a) in-place records management, (b) the Records Center, and (c) information management policies.

Labels are used to manage retention policies for specific types of content across the Office 365 environment. Labels can be applied automatically to content if it matches certain conditions or keywords (E5 licence only), or manually by users to emails, documents, or Office 365 Group conversations.

See below for the relationship and priority between retention policies and labels.

Who can create labels

Labels are created by individuals (ideally records managers or similar) assigned to a compliance role in the Security and Compliance Admin portal in Office 365.

Creating Labels

Labels are created in the Security and Compliance Admin Portal under ‘Classifications’. Labels may also be created without having an associated retention policy; that is, a label can be created and applied to content as no more than a visual ‘tag’. A policy can be added to it at a later stage.

If the ‘Retention’ option is enabled for labels (on/off switch), a new section appears titled ‘When users apply this label to content’. This section is where the retention policy is defined with two options:

• Retain the content. The choices are either ‘Forever’ or ‘n days/months/years’ (e.g., 7 years). The administrator must decide if, once it reaches that point, the labelled record should be deleted or not. The ‘Yes’ and ‘No’ options are the same as for retention policies, described above.
  • If ‘Yes’ is selected, the record will be deleted from where it is stored. Administrators have 93 days to recover records that have not been edited or deleted, or 7 days to records that have been edited or deleted (and moved to the Preservation Hold library).
  • If ‘No’ is selected, the content will be left in place and must be manually deleted.
• Don’t retain the content. The choices are to delete (a) after ‘n days/months/years’, and (b) based on when the record was created, modified, or labelled.

If the first option (‘Retain the content’) above is selected a check box option allows the administrator to use the label to classify content as a record. If the content is classified as a record, users are unable to change or delete the content or change or remove the label. They may still, however, edit the metadata.

The final step in the process is to review the settings. Once created, the administrator is returned to the main Labels screen which displays the label that has been created, allowing the administrator to then publish it.

Label limitations when used on a SharePoint document library

There are some limitations to applying a default label to a SharePoint document library:

• It applies the label to all records except those that already have a label and those contained in document sets.
• If the default label is removed, it removes the label from all records except those that have a label and those contained in document sets.
• Labels cannot be applied to folders in SharePoint or OneDrive (but can be applied to folders in Exchange).
• If the record is moved to a different library that has a different default label, it will inherit that label. Conversely, if it is moved to a library with no label, the existing label will be removed.

Note: When labels are published to an Office 365 group, the labels appear in both the group site and group mailbox in Outlook on the web. The experience of applying a label to content is identical to that shown above for email and documents.

What about legal holds?

eDiscovery in Office 365 is based around the creation of ‘cases’ in a SharePoint eDiscovery site. Cases are generally established in response to litigation (or potential litigation) and can be used to search across a range of sources. Once found, the information that forms part of the case can then be placed on hold, overriding any retention policy. However, once the hold is released, retention policies on records continue.

For more information on this subject, see:

https://support.office.com/en-gb/article/Add-content-to-a-case-and-place-sources-on-hold-in-the-eDiscovery-Center-54d70de9-1ec2-4325-84f3-aeb588554479?ui=en-US&rs=en-GB&ad=GB

What’s the relationship between retention policies and labels?

Retention policies and labels do the same thing but the former is more likely to be set centrally, while the latter is set by the end user. This means that a record could have more than one retention policy applied to it.

According to Microsoft’s documentation (link below), records will be retained until the end of the longest retention period applied to it, regardless of whether that policy was based on the retention policy or the label.

Are retention policies and labels better than previous retention options?

One of the primary benefits of the new retention policy regime in Office 365 is that it enables organisations to apply retention policies centrally rather than do this separately for each application (e.g., Exchange, SharePoint) as was the case until recently. It also allows end users to apply retention policies via labels.

Retention and disposal continues to be based on the individual record, or type of record (as defined by the policy or label), not logical aggregations or containers of records such as a document library.

As noted above, the concept of an aggregation that contains all the records on a given subject is ill-suited to the digital world. The reality is that records may be created using different applications (e.g., email in Exchange, document, list item or page in SharePoint, conversation in Groups, discussions in Skype etc) and stored in multiple application locations (e.g. in Exchange folders, SharePoint libraries, etc).

The dilemma for many records managers using Office 365 is how to store or manage records together in context, including based on the organisation’s File Plan or Business Classification Scheme (BCS) terms. The need to keep records together has been the driver behind the integration of EDRM systems with email applications, allowing email to be ‘captured’ in the EDRM along with other types of documents. This has rarely been successful in practice and, in most cases, emails are duplicated and remain stored in the email server.

The new Office 365 retention policies, including those applied as labels to specific types of content, may well be the answer to this dilemma. Rather than try to capture all types of records (e.g, document email, list item, conversation) in a single aggregation or container, Office 365 allows the option for them to be stored wherever the user prefers, with the same retention policy applied.

If necessary, all records with the same label can then be found using a content search in the ‘Search and Investigation’ section of Office 365.

In my view, there are still some shortcomings in basing retention policies on individual record types:

• Individual documents, rather than logical aggregations of documents, will be continue to be subject to disposal actions.
• Records that may provide context to other records (including those stored in different locations) may be destroyed.
• Appraisal options may be limited and appropriate review and approval steps before disposal may not be possible.
• Disposal actions may be automatic and unrecoverable.
• There may be no record kept, including the metadata, of the individual records that were destroyed.
• It is not known how courts might view the automatic disposal of records without prior review and approval.

Final thoughts

The new Office 365 records retention policy and label options centralise the management of retention and disposal for most types of records across Office 365, reducing complexity.

Retention and disposal continues to be based on individual records rather than aggregations, but this may be better suited to the digital world in which aggregations of records may not always be achievable.

Records managers working in organisations using Office 365 need to understand and provide guidance to IT on how records retention schedules can be applied as retention policies, and how they can be directly involved in decisions regarding the new options.

For more information: –

https://support.office.com/en-us/article/Overview-of-retention-policies-5e377752-700d-4870-9b6d-12bfc12d2423

https://support.office.com/en-us/article/Overview-of-labels-af398293-c69d-465e-a249-d74561552d30

 

Sentencing and disposing of records in SharePoint 2010

This article applies only to on-premises installations of SharePoint 2010. 

SharePoint’s ‘Records Centre’ was, in theory, a place to send records from various sites for long term storage, sentencing and disposal. The idea was that you could automatically, via Content Type rules, or directly, via the ‘Send to Records Centre’ option, transfer records out of team (and other) sites into the Records Centre. 

While the theory made sense, in practice there were several problems with the model, not the least being that ‘transferred’ records were not actually transferred but copied and effectively re-registered as new records in the Records Centre. They also lost previous versions, and so on. 

We needed to find another way to manage the sentencing and disposal of records. SharePoint 2013 has the (information management) option to apply a sentence on an entire site, which is great if you want to do that, but in most cases the requirement is to sentence parts of a site, including whole libraries or lists. 

As our document-based records are stored in document libraries, and these libraries generally (but not always) have names that make sense (if you can stop people using the generic ‘Shared Documents’ default library), it seemed a good idea to focus on how we could apply retention and disposal policies to document libraries. 

The first problem is visibility of all those libraries, created across multiple site collections. The only way to see all of them was to be the SharePoint Administrator or have Site Collection Administration privileges across all sites. But it was cumbersome to have to open every site to see what was stored in them. 

I put this problem to our SharePoint Administrator and developer (Eric Fang – blog here: http://fangdahai.blogspot.com.au/) . Using PowerShell scripts, Eric developed a method to display all document libraries across all Sharepoint sites in a list. The list updates on a regular basis.

NOTE: You cannot create this type of list ‘out of the box’, it requires PowerShell scripts, and that will need to be maintained over time. This is not a skill that is normally found with most SharePoint Administrators. 

The list displays:

• The library GUID
• The library name and URL
• The site collection and sites, including URLs
• The number of items stored in each library
• The date the library was created and who created it
• The date any item in the library was last modified

The first, immediate, benefit we could see from this method of displaying libraries was the number of libraries that had been created but not used. We could immediately see the ability to reduce the number of libraries, especially if these had not been used for a given period (say, one year).

The next benefit was the ability to group libraries by site collection. As many of our site collections map to business functions, we could start to see the volume of content that was stored for each function, by library – many of which were created to store documents created through various activities.

For example, a common document library is ‘Meetings’. We can now filter the view to see all libraries that contain documents relating to meetings. We can also see types of libraries that have specific retention and disposal requirements.

While the list of all libraries has provided an excellent way to view all our libraries, we are now working on a method to apply retention and disposal actions to these libraries. One way to do this would be to add an extra column in the list for retention and disposal information (class number, class decription, disposal action, expected disposal date, approved by, etc).

Once a disposal action had been applied to the list, we can then review it when the disposal action became due to determine if the library could in fact be destroyed. If it can be destroyed, it would be possible to export the original library metadata columns to a spreadsheet to keep a record of what was destroyed, and when.

SharePoint 2013 Site Disposal Policies

SharePoint 2013 includes the option to set a disposal date on site collections. This article describes how to configure a SharePoint 2013 site collection to include a site disposal policy.

Default settings

A site cannot be deleted (either manually or automatically) unless a Site Policy has been set up (exception – the SharePoint Administrator has permissions to do this).

Without a Site Policy, the default settings under the Site Closure and Deletion option (see below) are as follows:

• Site Closure – ‘Close this site now’ click box default: greyed out.
• Site Deletion – ‘This site will be deleted on:’ Default: ‘Never’.
• Site Policy – Default:  ‘No Site Policy’.

Setting up a Site Policy

New site policies are created under Site Settings – Site Collection Administration – Site Policies. Once created, the policy is applied under Site Settings – Site Administration – Site Closure and Deletion. While you can create multiple policies, only one policy can be selected at a time under the Site Closure and Deletion option.

There are no default policies; the first time Site Policies is opened, the Site Policies section provides only one option – ‘Create’. Each policy must have a Name and may have a Description. The name and description can be the class description from a records retention schedule, using ‘after date created’ or ‘after date closed’ as the triggers (see below).

Site Closure and Deletion options

There are three options under Site Closure and Deletion:

• Do not close or delete site automatically. The default option.
• Delete sites automatically. This option deletes a site on a pre-defined date after it was created or closed.
• Close and delete sites automatically. This option first closes the site and then deletes it on pre-defined dates.

In addition there is a check box ‘Site Collection Closure’ that allows the site collection to be made read only when it is closed.

Delete sites automatically

When this option is selected the following appears:

• Set Deletion Event. The two options provided are ‘Site closed date’ and ‘Site created date’, plus n days, months, or years.
• (Check box) ‘Send an email notification to site owners this far in advance of deletion:’ (i.e., to warn them of the pending deletion) – n days, months or years. Default setting is 3 months.
• (Check box) ‘Send follow-up notifications every:’ (i.e., to remind site owners of the pending deletion) – n days, months, or years. Default setting is 14 days.
• (Check box) ‘Owners can postpone imminent deletion for:’ (i.e., to postpone the proposed deletion) – n days, months or years. Default setting is 1 month.

Close and delete sites automatically

This option is identical to Delete Sites Automatically except that it also includes a date when the site can be closed – after which a deletion event date is set followed by the same three options above.

Site Closure and Deletion

As noted above, a Site Policy must exist before a site can be closed and deleted using these options. The Site Policy must be selected otherwise the default options (see above) apply.

• If the Site Policy is based on the Delete Sites Automatically option, the option to ‘Close this site now’ becomes available. If the option ‘Site Closed Date’ was selected, the site will not be deleted (at the pre-defined time) until this option is selected. If the option ‘Site Created Date’ was selected there is no requirement to ‘manually’ close the site.
• If the policy is based on the Close and Delete Sites Automatically option, the option to ‘Close this site now’ becomes available. This allows the site to be closed earlier, otherwise the deletion date will be automatically calculated from the site policy setting and displayed next to the Site Closure and the Site Deletion options.

• If no policy is selected, the default settings will apply; this means that the site cannot be closed.

Further reading

Overview of site policies in SharePoint 2013 (Microsoft).

Managing Physical Records in SharePoint 2010

It is sometimes said by records managers that one of the drawbacks of SharePoint 2010 (and SharePoint 2013) is that it doesn’t manage physical records, or at least not very well.

This article describes how to manage physical records in SharePoint 2010/2013 using a list that includes barcodes.

Up front and out of the box, the main limitations are the ability to: (a) print labels for paper files and boxes (including labels with barcodes) (b) attach a portable scanner to scan barcodes and (c) maintain a chronological history of file movements. These limitations may not be a problem for many organisations seeking a low-cost way to manage mostly inactive files and boxes, especially if they currently use an Access database or Excel spreadsheet to do this now.

Options to address each of these limitations are discussed in this article.

Metadata options

There are almost no limitations on the type of metadata you can use to describe physical records. I would suggest having the following metadata:

• Title (e.g., name on the file or box)
• Record Type (drop down list – file, box, etc)
• Reference ID (manually entered unless you use the default ‘ID’ option, possibly with a separate ‘year’ column)
• Date created (date first registered, default system date)
• Start date (start of date range)
• End date (end of date range)
• Business Owner
• Current location (can be a custom list of pre-defined locations)
• Date at current location
• Record status (drop down list: active, inactive, missing, destroyed etc)
• Internal barcode (if one exists)
• External barcode (usually the one provided by a storage provider)
• Records description (description of contents, usually of a box)
• Box number
• Box barcode (may be the same as the Internal or External Barcode)
• Retention schedule (can be from a pre-defined list)
• Disposal action (can be from a pre-defined list)
• Date to be destroyed (manually entered based on retention schedule and disposal action)
• Destroyed? (checkbox)
• Date destroyed (manually entered date)
• Legal hold? (checkbox)

If physical files are registered in the system, then only details of the box and disposal action would need to be entered. When the box is ready to be sent offsite, details from the offsite storage provider’s label (often provided in advance) can be entered in the external barcode and other metadata (including current location) updated.

An optional barcode can be obtained by checking ‘Barcode’ in the Information Management Policy settings for the list item. Once checked, the barcode appears when the item is displayed in view but not when it is edited or exported.

Printing labels

All list metadata (except the barcode) can be exported to an Excel spreadsheet, which can then be used to print labels. Microsoft provide details how to create and print labels for a list in Excel here:

Scanning

Using a portable (or attached) scanner to scan barcodes and update your records register, especially with a new location (or during a physical records audit) saves a great deal of time instead of entering details manually. It is possible to scan directly to an Excel spreadsheet however several steps may be required to match or validate the barcode with the details in the SharePoint list.

For example, if series of files are being scanned to a box (or location), the usual practice is to scan the box (or location) barcode first, then scan each individual file. If scanning directly to an Excel spreadsheet, you would do it the other way around – have the list of existing files in the spreadsheet, then scan the box or location barcode to the cell next to the file number. This detail can then be copied and pasted back to the SharePoint list.

Chronological movement history

The only option to maintain a chronological movement history, out of the box, would be to add versioning to the list. This means that each time the item is edited, a new version is created, keeping the previous details. The movement history will not display in a single view in the list but clicking on ‘Version History’ will show any changes made in single view. Only ‘major’ version numbers are supported in lists (i.e., 1, 2, 3, 4).

SharePoint 2010 – Send to Records Center / Centre

SharePoint 2010 includes the option to send a document from the site where it was stored to the Records Center (Records Centre in Australia), and other locations. The option is set at the Central Administration Web site under General Applications – Configure Send to Connections. See http://technet.microsoft.com/en-us/library/ee424395(v=office.14).aspx for the details of how to configure this option.

The ‘Send to action’ list offers three options when a document is sent to the Records Centre (options which are then available across the farm):

• Copy the document (which simply makes a copy)
• Move the document (which appears to remove the document entirely. Microsoft state that ‘users will no longer be able to access the document from its original location’, suggesting it is not actually deleted from the original location)
• Move and leave a link (see below)

Microsoft state (in the link above) that this option will ‘… delete the document from its current location, move it to the destination repository, and leave a link at the current location indicating that the document has been moved’. If Document IDs have been enabled, the Document ID will disappear next to the document icon, which then shows a hyperlink ‘hook’ to the document in the Records Centre. A user clicking on that link will be advised that the document has been moved.

The impact of versioning, and metadata, on ‘Send to’ moves

There are a number of issues, and potential problems, with the ‘Send to ‘ move option that need to be understood. Some of these might be sufficient to decide against using the Send to Records Centre option (or even using the Records Centre).

• Versioning. If versioning has been applied to the document library, and versions exist, only the most current version will appear in the Records Centre. The Document ID will be the same as the current version. However, any previous versions remain in the original library. In one sense, it is good to show previous versions, however any of these previous versions can be restored. When this happens, the version that is restored now appears, with the original document number (and a ‘hook’ still on the icon). The logic of this appears to be that, if a previous version is restored, the version that has been sent to the Records Centre is no longer the current version. However, it remains – with the same Document ID – in the Records Centre.
• Metadata. Any additional, local metadata that has been added to the Content Type in the original Site Library will disappear when the document is moved. Metadata in Content Types stored in the Content Type Hub will remain.
• Created date. The ‘Date Created’ field in SharePoint is the date the document was created in SharePoint. When the document is sent to the Records Centre, it is assigned a new ‘Date Created’. The date created field on the original document remains unchanged.
• Permissions. Any permissions that may have been applied to the document, via inheritance from the Library or Site, are removed and the document inherits the permissions of the relevant library in the Records Centre.

The implications for ‘Send to Records Centre’

The issues documented above could affect planning and decision making regarding use of the Records Centre and the ‘Send to’ functionality. Some issues to consider include:

• Loss of versions when the document is moved
• Ability to restore a previous version (and confusion about which is the current version)
• Loss of metadata applied locally to content types
• Changes to access permissions

If any of these are concerns for the organisation, consideration might be given to leaving documents in place where they were stored and managing them in that way. The Record Centre could instead be used to archive and apply retention policies to content from network drives.

Document Audit Trails in SharePoint 2010

Part 2 of ISO 16175 (‘Principles and functional requirements for records in electronic office environments’) includes a section (5.4.7) on records management process metadata. This section contains 13 mandatory requirements including:

• To create unalterable metadata of records management actions on records, aggregations of records, or the classification scheme.
• To track events without manual intervention.
• To maintain that metadata ‘for as long as required’.
• Provide metadata of all changes.
• Document all changes made to administrative parameters.
• Ensure that the metadata is available for inspection on request.
• Allow the metadata to be exported.
• Record any violations, or attempted violations (e.g., access attempts).
• Provide reporting.
• Allow configuration.
• Allow an administrator to make authorised changes, which are also recorded.

SharePoint 2010 provides audit trails on all sites but, unlike some other systems, audit trail information is not included in the metadata about the document itself. That is, records management process metadata is not included with other metadata about the record, but must be accessed separately, and only by a Site Owner or someone with higher privileges.

Records management process metadata is captured in two ways in SharePoint:

• By enabling it in a Content Type that is enabled in a Site Library.
• By enabling it in the Site Collection settings.

Audit trails may be deleted regularly as a default setting

Process metadata (‘audit trails’, or ‘audit logs’) are not kept permanently by default but are ‘trimmed’ according to a schedule for audit logs that is configured by the SharePoint (or server) Administrator in Central Administration.

The reason for not keeping the logs is to do with performance. According to Microsoft’s online help ‘… auditing can potentially generate a large number of audit events, creating a large audit logs. This could fill the hard drive, affecting performance and other aspects of a site collection’.

The default trim schedule is ‘end of the month’, which means that all previous metadata is deleted and a new log is started. Alternative options include a given period, for example 30 days or 7 days.

This means that audit trails for documents will not exist beyond the configured time period.

Turning off the default schedule

To disable the default schedule, the Site Collection Administrator must change the configuration settings of the ‘Audit Log Trimming’ section of the ‘Configure Audit Settings’ option under ‘Site Collection Administration’.

The Site Administrator may also opt to store the audit reports in a site collection library before they are trimmed. The three options are:

• Automatically trim the audit log for this site? This is set to ‘Yes’ by default, which means it will trim the logs in accordance with the Farm default setting.
• Specify the number of days of audit log data to retain. The Site Administrator may set the days when the audit log will be trimmed, regardless of the Farm default setting.
• Specify a location to store audit reports before trimming the audit log. This option allows the Site Administrator to automatically save audit logs to the location identified so the logs are not lost.

Other configuration options in the ‘Configure Audit Settings’ section are:

• Documents and Items. There are five events that can be audited: (a) opening or downloading documents, viewing items in lists, or viewing item properties; (b) editing items; (c) checking out or checking in items; (d) moving or copying items to another location in the site; and (e) deleting or restoring items.
• Lists, Libraries, and Sites. There are three events that can be audited: (a) editing content types and columns; (b) searching site content; and (c) editing users and permissions.

Audit Log Reports

Once auditing is enabled, the audit logs are available via Site Settings – Site Collection Administration – Audit Log Reports.

There are usually four options available:

• Content Activity Reports. This includes: (a) content modifications (‘all events that modified content’ – this includes editing, checking out/in, moving/copying); (b) content type and list modifications; (c) content viewing (‘all events where a user viewed content in this site’); and (d) deletion (‘all events that caused content in this site to be deleted or restored’).

• Custom Reports. This allows the Administrator to manually specify filters for a report.

• Information Management Policy Reports. This includes reports on: (a) expiration and disposition and (b) policy modifications.

• S

  ecurity and Site Settings Reports

  .

  This includes: (a) auditing settings (‘all events that change the auditing settings’) and (b) security settings (‘all events that change the security configuration of SharePoint’).

Audit trail report content

Audit trails in SharePoint are exported to Excel and include two worksheets, a summary and the detail.

The summary worksheet shows the number of actions (e.g., view, edit) on a document. The detailed worksheet includes a row of information similar to the following:

Site Id: {fffaf4fd-b5d0-4940-a615-06e0026a855d}
Item Id: {373b3b5e-43a7-46d8-8d44-9a36fad6a4da}
Item Type: Document
User Id: John Smith <DOMAIN\jsmith>
Document Location: _catalogs/masterpage/v4.master
Occurred (GMT): 2013-02-13T04:02:45
Event: View
Custom Event Name: (may be blank)
Event Source: SharePoint
Source Name: (may be blank)
Event Data: <Version><Major>1</Major><Minor>0</Minor></Version>

Compliance with ISO 16175

While SharePoint 2010 does create audit trails for records management actions, it does not store these audit trails in an end-user visible form with the record itself; that is, when an end user views the metadata describing the record, he/she cannot view the audit trail history of the record. Only a Site Administrator can access and/or view the audit trails.

The other main weakness, in terms of compliance with ISO 16175, is that the audit trails could be deleted according to the default schedule set by the farm administrator. To get around this, a choice needs to be made whether to override the default trimming, or (to reduce storage requirements) to export the audit logs regularly and store them on the same site.