Posted in Records management, Electronic records, Products and applications, Legal, Training and education, Information Management, SharePoint Online, SharePoint Designer

Auto-populating document templates via a form in SharePoint

Most organisations have standard agreements or contracts or similar types of documents.

The common factor between them is that the original template remains the same while elements within the document change. For example, a client name, address and phone number, or differing contract terms.

There are several different ways this is achieved, including:

  • Printing the form and completing it manually.
    • This is time-consuming, handwriting can be difficult to read or require the form to be re-completed, and there is no easy way to extract the data. These types of forms are often scanned for storage.
  • Completing a digital version in Word (and sometimes printing/scanning or saving as a PDF).
    • This is also time consuming and in many cases it can be faster to print the form to fill it in by hand. Errors and omissions are possible and if the metadata appears in more than one place it must be re-typed. There is no easy way to extract the data.
  • Using editable PDF forms, sometimes using (Adobe or other) digital signatures.
    • These are very common (and very useful for specific purposes such as simple forms, less so for common agreements). They are time consuming, errors and omissions are possible and metadata must be re-typed. There is no easy (or cheap) way to extract the data.

Common factors in all of the above are that they are time-consuming and the data is hard to extract from the form.

A better and more efficient option

This post describes how to create a form in SharePoint that, via a very simple workflow:

  • Auto-creates one or more Word documents (multiple based on metadata choices contained in the form).
  • Auto-populates the Word documents where required with the metadata in the form. Where the same metadata value (e.g., ‘Client Name’) appears more than once, that value appears throughout the document where required at the same time.
  • Stores that document (or documents) in a folder (actually a document set) that can be used to add other content.

Additional benefits are that:

  • The metadata is easily accessible for export and other uses.
  • The Word document can be ‘signed’ with a touchscreen computer.
  • The Word document can be saved as a PDF.
  • Other documents can be added to the same folder.

This post is based on several actual examples that I developed (with the assistance of our SharePoint Developer) in a very large (9,000 staff) organisation.

The primary uses were for client agreements based on standard templates, including up to 10 different documents per client. We also deployed other designs that used a similar methodology, but the underlying principle was the same.

Note that, while the model is actually simple to implement, this post contains all of the details to follow step by step. I’m not a fan of posts that only provide part of the details and leave the rest to the imagination.

Setting up the model

Important note: The SharePoint site MUST have the document set feature enabled in the Site Collection Administration settings. Otherwise, the option to create a custom document set will not appear.

The model consists of the following elements that can be created by a SharePoint Administrator, a Site Collection Administrator or a Site Owner.

  • New site columns that will map to the elements in the form. For example, ‘Client Name’, ‘Client Address’, ‘Client Phone Number’. Note that every SP site has a lot of standard site columns so some of these can be used instead of creating new ones.
  • A new document set site content type containing all the site columns that should appear in the form. (‘Add from existing site columns’ option). It is recommended you give the document set a name that will be clear to end users as they will select this from a list. For example ‘Client Folder’ or ‘Agreement folder’.
  • A new document site content type for every template that is needed. The actual document template are not added now, only after the content type has been added to the document library – see below. It is recommended that you give each of these document CTs a name that is similar to the name of the document template.
  • A document library. It is recommended that you create a dedicated library for this purpose with a name that makes it very clear what it houses, for example ‘Client Agreements’. See below for the set up of the library.

Once all of these options are in place, the SharePoint Designer workflow can be set up – see below.

Setting up the document library

Library settings

The document library needs to be set up as follows in the Library Settings section.

  • In Advanced settings, enable the option ‘Enable management of content types’. This will make a new section ‘Content Types’ appear in the Library Settings.
  • In the newly visible ‘Content Types‘ section and choose ‘Add from existing site content types’ and add all the new site Content Types that were created.
  • The newly added CTs will now be visible, along with the default ‘document’ content type.

Document set CT settings

Click on the new document set CT. The metadata site columns that were added should be visible in the ‘Columns’ section.

Click on ‘Document Set settings’. In the section ‘Allowed content types’, click then use the ‘Add’ option to add all the document CTs that are required. These will now appear in the right-hand section.

 

O365_SPO_AddDocCTtoDocSetCT

Scroll down to ‘Shared columns’ and select all the document set columns. It does not matter that these will be shared with document CTs that don’t use the columns, as we will see below.

Click OK and return to the library settings area.

Adding the templates

At this point it is assumed that you have one or more document templates ready to upload. The template/s should be in a newer version of Word (e.g., .docx NOT .doc).

The ‘Content Types’ section of Library Settings displays a list of all the CTs that were added, including the document set CT (which will not be changed).

To add the template, click on the name of the (document) CT. In the new page that opens, you will see the list of site columns that have been shared from the document set.

Click on ‘Advanced settings’, where you will see the ‘Document Template’ section. Click the ‘upload a new document template’ option, choose your document template, and click OK.

O365_SPO_AddTemplatetoDocCT

Link the metadata columns with the template

Now, return back to the document CT ‘Advanced Settings’ (if you are not still there) and click on ‘Edit Template’ to open the template document in Word.

Now, add the metadata site columns where they are required in the template. For example, next to ‘Client Name’, place the cursor where you want the metadata to appear (don’t forget to include a space!).

In Word, go to the ‘Insert’ option on the ribbon menu and then go to the ‘Text’ section. Choose the ‘Quick Parts’ > Document Property and you should see the metadata columns as shown below.

O365_SPO_InsertMetadatainDocTemplate

Add the relevant document metadata where it should appear in the Word template. You will notice that the same metadata element can be used in multiple locations throughout the document. You can also use these in the header and footer and apply different formatting as required.

If you have made an error, do not ‘delete’ the added metadata in square brackets, instead right click and choose ‘Remove content control’. Be careful of formatting too especially different fonts and font sizes. Some of these will be more visible once you create the first document (see below).

The finished template will look something like the screenshot below.

 

 

O365_SPO_MetadataInWordTemplate3

Repeat for each content type template.

Summary and outcomes of the first stage

The site and library set up stage is now complete. The new content types now appear in the ‘New’ menu as shown below. You may want to edit the new menu options to remove any option you don’t want to appear, such as ‘Folder’ and ‘Document’ (you cannot remove ‘Link’).

O365_SPO_CustomCTDocLibNew.JPG

If the end user selects ‘Client Agreements’, they will be presented with a form to complete such as the example below – but this does NOT yet create the template document. That’s the next step below.

 

O365_SPO_CT_DocSet_NewForm3.JPG

 

Note that the order of these metadata elements can be moved around as required via the document set settings.

Create the workflow

You will need access to and be able to use SharePoint Designer to complete this section.

Remember: The workflow is based on the end user selecting and completing a new (document set by completing the form as shown above. The workflow is triggered by the fact that a new item has been created, which in turn creates and saves a new document (or documents as required) with the metadata populated automatically ‘inside’ the new document set.

Open SharePoint Designer

First, click on ‘Lists and Libraries’, choose the library that the workflow will be associated with, then click on ‘List Workflow’ as shown in the ribbon menu below.

SPD_NewListWorkflow

Give the workflow a name that will help to identify it in future – in this example, ‘Create Client Agreement’ would be a suitable name. Note:

  • You must create this as a SharePoint 2010 workflow.
  • The workflow can create one or more documents. In this example, only one document is created.

New workflow settings

A new tab will open. On the top right of the ribbon menu, click on ‘Workflow Settings’.

In the ‘Start Options’ section, check the box to start the workflow automatically when an item is created. The manual start checkbox should already be checked. This will allow the end user to run it again if required.

SPD_StartOptionsSP2010

Note – Some organisations may prefer not to allow the workflow to start automatically because they want to check the form first. In this case, the document set-based form can be created, but only after it is created the end-user must choose to run the workflow via the ‘More – Workflows’ option from the 3-dot menu.

 

Create local variables

Click on the ‘Local variables’ option on the top right of the ribbon menu to create (Add) two local variables:

  • DocSetName < this one is used to record the name on the document set.
  • DocumentPathforClientAgreement < this one is used to save the new document ‘under’ the document set.

Create the workflow

In the Workflow settings, click on ‘Edit workflow’ to create the workflow. For this example, there are two steps.

Click on ‘Step’ to change the name to something like ‘Initialisation’ or ‘Initialise variables’.

SPD_WorkflowNewStep1Blank.JPG

In this part we add and configure the two local variables that were created.

Click where it says (‘Start typing …’), click on on ‘Action’ in the ribbon menu, and choose ‘Set workflow variable’ to set the two variables.

  • Set Variable: DocSetName to Current Item:Name
  • Set Variable: DocumentPathforClientAgreement to [%Variable: DocSetName%]

Both of these will be set as a String value.

SPD_SetSP2010WorkflowVariable1.JPG

Click just underneath the step; a short orange line should appear. Click on ‘Step’ from the ribbon menu to create the next step.

(Note – a screenshot of all the following steps can be seen below)

  • Rename the step if required (e.g., to ‘Create Agreement’).
  • Click in this new step where it says (‘Start typing …’), then click on ‘Action’ (ribbon menu) and choose ‘Create List Item‘.
  • Click where the new action says ‘this list‘. A new dialogue box opens ‘Create new list item’. Select the name of the library from the drop down list in that dialogue box.
  • As soon as you do this, ‘Path and Name (*)’ appears below ‘Content Type ID’. You must complete the second part of this command before it can be saved.
  • Click on Path and Name (*) and click ‘Modify’. The ‘Set this field’ option should not be changed, only the option ‘To this value’. To the right of the blank field click the ‘fx’ option, then do the following.
    • For ‘Data Source’, choose ‘Workflow variables and parameters’.
    • For ‘Field from Source’, choose ‘Variable: DocumentPathforClientAgreement’
    • For ‘Return field as’, leave it as a ‘string’ value.
  • After you click save, the ‘Value assignment’ dialogue box should still be open. If not on the ‘Path and Name (*)’ option, then Modify, which will open the ‘Value assignment’ dialogue.
  • Click on the three dot menu option (to the left of fx) to open the ‘String Builder’ dialogue. Modify it as shown below by adding the prefix text. This puts the name given in the document as the first part of the document name: [%Current Item:Name%]/[%Variable: DocumentPathforClientAgreement%]
  • Note, you can add anything else you want after the last ‘]’, for example ‘- Client Agreement’, as a suffix to the document name.

SPD_WorkflowNewStep2CreateListItemA.JPG

Click OK (several times) to close the dialogue.

Add a ‘Stop the workflow and log’ option from the Action menu.

The final workflow is shown below:

SPD_TwoStepWorkflowFinal

Publish the workflow

Finally, publish the workflow. You can also press ‘Save’ to save without publishing. Publishing also saves any changes.

Allow some time for the workflow to appear in the document library. Generally this is fairly quick – refreshing the site page may assist.

Confirm the workflow is ready

To confirm the workflow is ready, click the three dot menu to the right of the document set and click on ‘More’, then ‘Workflow’.

The new workflow should appear similar to the screenshot below.

Note that this is the primary interface for most actions relating to the workflow. From here you can click the workflow to run it again any time (Manual start). If the workflow has a problem you will see that message here under ‘Running Workflows’; from there you can terminate the workflow if it has a problem (which sometimes happens – the clue is that the document was not created).

SPD_LibraryWorkflowReady

End result

When the end user completes and saves the form, the workflow will run, creating one or more documents (based on the template) ‘inside’ the document set. Each document will have the correct metadata based on the template.

O365_SPO_MetadataInWordTemplateFilled3.JPG

 

Benefits

There are many benefits to creating this model to manage common document agreements, contracts and other templates.

  • The document template always remains the same and can be updated at any time (but note that entire template updates require re-connecting all the metadata elements).
  • If a mistake is made in the metadata, the end user can simply delete the documents that were created and re-run the workflow as many times as required, saving a lot of effort in having to re-populate an entire document. If there is concern about deleting documents, the manager can set an alert on the library. The Recycle Bin keeps deleted documents for 90 days.
  • All Word documents created this way include the metadata from the library in their properties (the ‘metadata payload’). This includes the Document ID (if enabled).
  • Once the Word document has been created it can be ‘signed’ electronically using touch screen technology. If you really need a more sophisticated signing process, consider acquiring a third-party product.
  • Once the Word document has been signed in this way, it can be saved as a PDF, preventing changes.
  • If saved as a PDF, the defaults save location is the same location. Saving to PDF is a three step process: Open the Word document, click ‘Save as’, and change the option to PDF.
  • All the metadata site columns can be exported for analysis and reporting purposes. It may be also be used to created groupings of records for example ‘All contracts created by users’, or ‘All contracts that have a specific metadata choice option’.
  • The newly created Word or PDF documents can be shared, including with external people if required.

Negatives

In practice we found that there were not many negatives associated with this model and it brought considerable productivity benefits to the business areas that regularly created multiple agreements with clients, based on standard templates.

The primary negatives we found were:

  • Poor bandwidth meant that the new Word agreement may not create as quickly as required. Business areas with this problem kept both digital copies of the agreement to complete or printed versions.
  • If the entire template had to be changed, all the metadata links had to be re-connected. It was usually much easier only to update the part of the document that needed to be updated, including by adding new pages.
  • Every once in a while the workflow would not work. Our first clue to this was that an end user would call to say the document was not created or a metadata field was blank. We could usually track this problem down to either a network ‘glitch’ or other minor issue.
  • If metadata fields are left blank in the form, the square brackets metadata option remained visible. This then had to be deleted from the final.
  • From time to time, for various reasons, the end user would create a second copy of the document template without deleting the first. This simply creates a new document with the date and time as a suffix to the document name.
Posted in Classification, Conservation and preservation, Electronic records, Information Management, Records management, Retention and disposal, SharePoint Online

SharePoint Content Types – A records management view

SharePoint Content Types (CTs) have been a fundamental element in SharePoint architecture since they were introduced in 2007. They are also the cause of some divided opinions, in favour of (a) using multiple, ‘custom’ CTs to control content, or (b) minimising their use in favour of alternative options such as metadata choices.

A recent tweet from Nate Chambelain (@chambernate) read as follows:

You can have multiple content types in lists and libraries. They can all have unique metadata, templates, forms, retention policies, workflows, etc. Content types are EVERYTHING.

While CTs can do a lot as suggested by Nate, it is important to understand what they are, and what they can do, as part of an overall architecture model if you are planning to use SharePoint to manage records.

SharePoint administrators (and information architects and others) may have different views on this subject. There is, on one hand, the view that multiple ‘custom’ CTs are a good thing as they provide more control over content. On the other hand, there is the view that too many CTs are a bad thing because they are not easy to implement (see below for details) and it makes the environment harder to manage.

The key is getting the balance right. In my own opinion based on a decade of working with SharePoint, it is better to create custom CTs only where they provide specific useful functionality and otherwise using metadata columns and/or folders.

Content Types have been around since 2007

Content Types (CTs) were introduced in SharePoint 2007 (SP2007 usually known as MOSS2007), the immediate precursor to SP2010. CTs, it was said, would allow organisation to:

  • Have multiple document types associated with a single library.
  • Have a different document template in each CT.
  • Have different workflows and metadata for each CT.

List CTs could also be used to capture different metadata in the same list.

What are Document Types?

Document types examples might include:

  • Contracts
  • HR document (e.g., specific to a staff member)
  • Invoices
  • Plans
  • Minutes

Most organisations will have (and share) a relatively standard set of document types.

While it is tempting to create a CT for each document type (similar to the way some document management systems do this), in my experience this can create considerable overhead for very little benefit.

In my view, document types should only ‘map’ to custom ‘content types’ if there is a reason to use CTs for a specific purpose (see below for examples).

Otherwise, it is far easier and more efficient (in terms of management) to create additional metadata columns including, if required, a ‘choice’ metadata column to define the document type.

Every Content Type has a parent

Every new site collection includes a range of default CTs and each has a parent. The primary default CTs are:

  • ‘Document’, ‘Folder’ (and, where enabled ‘Document Set’) for document libraries.
  • ‘Item’ for lists.
  • ‘Event’ (parent is ‘Item’) for calendars

Your starting point, when thinking about CTs, is whether you will (a) use the default CTs or (b) create custom CTs – or (c) a combination of both. Most organisations will have a combination.

By way of reference, we had 500 site collections and fewer than 30 custom CTs, each used for a specific purpose. We did not use the CT Hub but created CTs directly on the sites where they were required.

How many custom Content Types do you really need?

The answer is, ‘it depends’.

The case for multiple custom CTs:

  • Custom CTs allows organisations control of specific types of content at a more granular level than the document library.
  • Having multiple custom CTs requires more overhead to create, implement and manage. First, each custom CT must be created via Site Settings. In each document library setting where the CT is to be used, a setting must be enabled to allow management of CTs. And finally, each CT must be added to each document library (or list).
  • Questions to consider: Who will enable the library setting then add each CT that is required? Who will manage metadata conflicts if different CTs use the same or very similar metadata? How will end users react to have to select a CT every time they just want to save a document (including via synced libraries in File Explorer)?

The case for fewer CTs:

  • Fewer or more selective use of custom CTs allows organisations to manage content at the logical aggregation of a document library rather than CT-linked content within that aggregation, using custom CTs only where necessary. This requires less overhead to manage.
  • Site Collection Administrators or Site Owners can create CTs when and only if they are needed.
  • Site Owners (and end users) can more easily manage content within a document library by using folders and/or metadata to group content and/or apply workflows.
  • Choice metadata columns (e.g., for ‘Document type’) may be a better option.
  • Site Owners can create document libraries with names the comply with naming conventions and apply retention policies to the entire library.

The bottom line – just because you can create multiple custom CTs, it doesn’t mean you should. It is important to understand the broader architecture of the entire SP environment and what you are seeking to achieve.

As a general recommendation, I would suggest creating only the CTs that you really need (based on a content management design), and consider using separate libraries (or sites) and/or metadata to define content types. For example, consider having a single library called ‘Meetings’ or ‘Policies’ with useful metadata columns.

More custom CTs will mean a more complex environment to manage, especially from a records management point of view.

How are Content Types created and then applied

New custom CTs are created via Site Settings – Site Content Types.

O365_SPO_ContentTypeCreate1

As can be seen above, every new CT is based on an existing (‘parent’) CT that exists in a grouping of ‘parent content types’. For example:

  • The ‘Document’ CT is found in the ‘Document Content Types’ group. Surprisingly that group also includes ‘Master Page’, ‘Site Page’ among others.
  • The ‘Document Set’ CT is the only option in the ‘Document Set Content Types’ group.
  • The ‘List’ CT is one of multiple options in the ‘List Content Types. It includes ‘Event’ found in calendars – ‘Event’ is the child of ‘Item’.

Once the new CT is created, additional columns can be added to it, including from the Managed Metadata Service (MMS), if used. The new, custom CT can now be added anywhere on the site.

Two steps are required to add a CT to a document library or list:

  • Go to the library or list settings and click on ‘Advanced’.
  • In the ‘Advanced’ section, click on the option to ‘Allow management of content types’.

O365_SPO_ContentTypeEnable

Note that enabling this option has an impact on the ‘+ New’ option in the library ribbon menu – see below.

Once enabled, a new ‘Content Types’ section appears in the Library Settings, displaying all the CTs now available in the library. Only the standard ‘Document’ CT will appear and show as the default option.

Someone (SP Admin, Site Collection Admin or Site Owner) must now click on ‘Add from existing site content types’, and choose the CT to be added. This action could probably be scripted however it is a one-off event.

Once the custom CTs are added, another (but only one) CT may be set as the default. The others are visible by default but can be made invisible if required.

Any unique columns in any of the CTs will now appear in the ‘Columns’ section, showing the source of the column (‘Used in’).

Folders or document sets?

While it is possible to create folder-based CTs, with unique metadata, to group documents, it may be more useful to create document set-based CTs as these provide much more flexibility. Unlike folders, document sets also have a unique document ID (that same one that is applied to all documents in the library, with a different sequential number).

How CTs are used

Once they are enabled, the new CTs now appear in the ‘+ New’ drop down menu for the library. End users may now choose which CT they want to add to the library (or it may be the default).

O365_SPO_ContentTypeChoiceDocLibrary

As can be seen above, the option to create an Office Online document is no longer available. (These options can, however, be added back via the ‘Edit New Menu’ option, however there may be reasons not to want to allow this).

Each document-based CT includes the default ‘Document’ template. This means that if an end-user clicks a custom CT from the ‘New’ menu, it opens Word Online. Should it be necessary for the end user to, say, create an Excel document the following steps must be followed:

  • Click on library settings
  • Click on the custom CT
  • Click on ‘Advanced settings’ (which will show ‘template.dotx’ as the default template)
  • Upload a new document and choose the file – e.g., a blank spreadsheet with a sensible name, or a given template that must be used every time. Note that the template must be able to be opened in the online versions of the Office applications.

If the document library contains mandatory metadata, and the end user is working directly from the library in SPO, they will be required to add the mandatory metadata.

If the end-user saves a document to a synced library in File Explorer, and that library includes CTs, the end user will be asked to select the CT from a drop down list in a separate dialogue. If the CT includes any mandatory metadata, it will not be possible to add a new document via File Explorer.

What about retention policies?

The ability to apply retention policies in CTs is only available as an ‘auto-apply’ option to those who have an E5 licence. Essentially, when a new retention policy is created (based on a label that defines the retention options), it can be auto-applied to specific CTs. For more details, see this article by Joanne Klein.

What this means is that individual documents contained in the CT (not the CT itself) will be subject to disposal at the end of the retention period. As noted in my earlier post on managing retention outcomes, disposal may be automatic or subject to a review.

Specific use cases and careful consideration of the options and implications need to go into planning for the use of retention policies on custom CTs. While the option may appear to be a simple way to manage the retention outcomes of specific types of content, in reality it could be very hard to manage in large organisations, especially as individual documents are the subject of disposal, not the aggregation (CT or library).

Joanne Klein’s article above refers to a CT named ‘Contract document’. While this is just an example, most contract documents will be subject to disposal review ‘n’ years after the contract has expired. Therefore, retention should be based on a metadata column named something like ‘Contract Expiry Date’. Many organisations, in my experience, keep contract documents well after the contract expiry date. The use of retention policies on contract-related CTs needs to be considered carefully.

Where custom CTs could be useful

Custom CTs can be useful to manage ‘files’ within a broader grouping or aggregation. The following are actual examples from my organisation:

  • To manage individual staff files (created as custom document sets), in a document library that is used to manage all staff files.
  • To manage a collection of contract documents (stored within a custom document sets), in a document library (or libraries) that has retention enabled at the library level. In this example, folders would probably be just as suitable.
  • To manage building/property documents (again stored within a custom document set) in one or more document libraries.

The primary reason that using custom CTs here is better than using simple folders is that the custom document sets can contain specific metadata that is needed, such as ‘Staff ID’ or ‘Staff Name’, ‘Building Number’ or ‘Building address’, or ‘Company’.

Conclusion

Content types are, indeed, a core part of SharePoint. However, unless your organisation is small or you only want to control selected content via the use of custom CTs, it may be easier and more efficient to minimise their use and use metadata choice columns instead.

Applying retention policies to CTs – assuming that is considered to be a viable records management option – is only available to organisations that have an E5 licence. This option should be considered carefully, especially the outcomes when the retention period expires.

Custom CTs can be very useful when used correctly. It is important to have a good understanding of how they work and where they can be applied most usefully. Otherwise, there is a high potential to add considerable complexity and management overhead to your SharePoint environment.

Posted in Classification, Compliance, Information Classification, Information Management, Office 365, Products and applications, Records management, Retention and disposal, Security

Office 365 Security and Compliance – classification label changes

Microsoft have improved the Classification section in the Office 365 Security and Compliance centre. The change will help to reduce confusion and make it easier for records managers and security administrators to focus on their individual needs.

Previous user interface

The primary change is to the menu interface. The previous menu options, shown in the screenshot below, showed only ‘Labels’ and ‘Label policies’.

O365_Classifications_Labels

When the previous ‘Labels’ option was selected, a new screen with two tabs ‘Sensitivity’ (default) and ‘Retention’ was displayed, as shown below.

O365_Classifications_Labels

The sensitivity or retention tab had to be selected to create or publish a new label. The user interface was unclear and the difference between creating and publishing a label was not obvious.

New user interface

The sensitivity and retention elements have now been separated and placed under the primary ‘Classification’ menu option as shown below.

O365_Compliance_ClassificationLabels23Aug19.JPG

Now, ‘Labels’ and ‘Label policies’ are two tabs under the relevant section as can be seen below.

 

O365_Compliance_ClassificationLabelsRetention23Aug19

O365_Compliance_ClassificationLabelsSensitivity23Aug19

The options to create and publish labels remain the same.

Posted in Compliance, Exchange Online, Governance, Information Management, Office 365, Office 365 Groups, OneDrive for Business, Records management, Retention and disposal, SharePoint Online

Managing the outcomes of records retention in Office 365

The retention of records in Exchange Online (EXO), SharePoint Online (SPO), OneDrive for Business (ODfB) and Office 365 (O365) groups can be achieved through the application of retention labels published in the O365 Security and Compliance admin portal.

This post describes:

  • How retention labels work (in summary), including the ‘per record’ rather than the container/aggregation retention model.
  • What happens to content in Office 365 when a retention period expires.
  • The options and actions that may influence the way retention labels/policies are configured, where and how they are applied, and the outcomes required.

The post highlights the need for information and records managers to be involved in all aspects of governance, site architecture and design, and decisions around specific settings and configuration, as well as being assigned specific roles, when Office 365 is implemented.

A quick summary of how O365 retention labels work

Records retention policies in O365 are based on ‘retention labels’ that are created in the O365 Security and Compliance admin portal under the ‘Classifications’ section. Multiple labels can be applied to a single policy.

  • Click this link to read Microsoft’s detailed guidance on retention labels.
  • Click this link to read Microsoft’s detailed guidance on retention policies.

Retention outcomes

Each retention label defines one of three potential outcomes at the end of the retention period, if retention is enabled, ‘keep forever’ is not selected, and the label is not used to classify the content as a record*:

  • The content will be automatically deleted. If the content is in SharePoint, it will first be sent to the Recycle Bin, from which it can be recovered within 90 days.
    • This option may be suitable for certain types of low value records.
  • A disposition review will be triggered to notify specific people. As with the previous point, SharePoint content will be sent to the Recycle Bin if a decision is made to delete it.
    • This option will require additional, human-intervention actions, as described below, if standard records management disposal review processes are followed.
  • Nothing.

Triggers

The date when the above action will occur is based on one of four triggers:

  • Date created
  • Date last modified
  • When labelled applied
  • A event. The ‘out of the box’ (OOTB) event types are:
    • Employee activity. (Processes related to hiring, performance and termination of an employee)
    • Expiration or termination of contracts and agreements.
    • Product lifetime. (Processes relating to last manufacturing date of products).
    • A new event can also be added.
    • See this post for Microsoft guidance on event-driven retention.

An additional alternative option is available: ‘Don’t retain the content, just delete it if it’s older than n days/months/years.’ This is similar to the automatic deletion option above and may be suitable for certain types of records. 

Declaring content as records

* The option to classify or ‘declare’ content as a record is not discussed further as relates to the way records are managed in the US. Microsoft’s guidance on labels notes that: ‘At a high level, records management means that: (a) Important content is classified as a record by users. (b) A record can’t be modified or deleted. (c) Records are finally disposed of after their stated lifetime is past.’ The standard on records management, ISO 15489, defines a record as ‘evidence of business activities, often (but not exclusively) in the form of a document or object, in any form’. This means that anything can be a record. The record may continue to be modified throughout its life. 

When do retention labels become active?

Retention labels become active only when they are published. As part of the publishing process, a decision must be made if the label will apply to all (a single option) or selected parts of the O365 ecosystem:

  • The Exchange Online (EXO) mailboxes of all or specific recipients, or excluding specific recipients.
  • All or specific SharePoint Online (SPO) sites, or excluding specific sites.
  • All or specific OneDrive for Business (ODfB) accounts, or excluding specific accounts.
  • All or specific O365 Groups, or excluding specific groups. Note that content in Microsoft Teams (MS Teams) is included in the O365 Groups options that include both the SharePoint content and email/Teams chat content.

Auto-applying retention labels

Both the retention label and policy sections include the ability to auto-apply a retention policy if certain conditions are met.

  • Sensitive information types. These are the same types that appear in the Data Loss Prevention (DLP) section, for example ‘Financial data’ or ‘Privacy data’.
  • Specific keywords.
  • Content types and metadata (E5 licences only). See this post by Joanne Klein for a description of these options.

The ability of the first two options to accurately identify content and apply a retention policy should be investigated before they are relied on.

When do retention policies start working

According to Microsoft’s guidance Overview of Retention Labels:

If you publish retention labels to SharePoint or OneDrive, it can take one day for those retention labels to appear for end users. In addition, if you publish retention labels to Exchange, it can take 7 days for those retention labels to appear for end users, and the mailbox needs to contain at least 10 MB of data.

  • In EXO, the default MRM policy needs to be removed before the new policy applies.
  • In ODfB, the policy is available to be manually applied on folders or documents. It does not automatically apply to content.
  • In SPO, the policy can be applied to document libraries or documents. To avoid removing the ability for users to legitimately need to delete documents in an active library it is recommended to apply the policy after the document library has ceased to be active.
  • Content in Office 365 Groups is covered by either the EXO (for email/teams chat content) or the SPO policy (applied to libraries).

Retention labels apply to individual records within aggregations

Records labels can be applied to aggregations of records (an entire email mailbox or folder, a SharePoint library or list, an ODfB account, O365 Groups) or individual records. However, the disposal process targets individual records (e.g., individual emails, single documents in SharePoint libraries, individual list items). 

That is, even when all the individual records are disposed of, the parent aggregation remains in place without any indication that the records previously stored in it (sometimes known as a ‘stub’) have been destroyed. 

This outcome has implications for the way the outcome of a retention label is set. It requires a choice between (a) delete automatically without review or (b) review before delete.

The latter option is made complicated by the requirement to review individual documents, including potentially in the original container (document library in SPO) and export metadata relating to those records if a record of the deletion is to be retained.

What happens when records reach the end of their retention period

As noted above, the outcome at the end of the retention period (trigger date + n days/months/years) will depend on the settings on the label.

  • Where the label was applied (EXO mailbox, SPO library or list, ODfB folder or document, O365 Group)
  • Whether the records would be deleted automatically or be subject to a disposition review.

If the records are to be deleted automatically:

  • SPO and ODfB records will be sent to the site/ODfB Recycle Bin for 90 days
  • EXO emails will be moved to a ‘Cleanup’ area for 14 days, before permanent deletion.
  • Aside from the audit logs (which by default only go back 90 days), no other record will be kept of the destroyed records.

If the records are subject to a disposition review, an email is sent to the person nominated. When that person clicks on the link in the email they are taken directly to the ‘Dispositions’ sub-section of the Records Management section of the O365 Security and Compliance centre.

It is arguable that retention policies with disposition review should not be applied to ODfB content as this will require the reviewer to review all the content that has been labelled by a user in their ODfB account.

  • For more information about this subject see this Microsoft page ‘Overview of disposition reviews‘. Microsoft note, on that page ‘To get access to the Disposition page, reviewers must be members of the Disposition Management role and the View-Only Audit Logs role. We recommend creating a new role group called Disposition Reviewers, adding these two roles to that role group, and then adding members to the role group.’

The dispositions dashboard shows the number of records that are pending disposition against each retention policy label:

O365_RM_DispositionsDashboard

Pending disposition tab

When the reviewer clicks on one of the retention policies listed, the following view opens for records ‘Pending disposition’:

O365_Dispositions_Pending

An important point to note here is that records are listed individually, not in logical aggregations or collections. It is possible however to use the Search option on the left to filter by author (emails) or SharePoint site and/or site library. It is also possible to export the details (which does not include any unique metadata applied to documents in SharePoint libraries).

All the records displayed may then be selected and a ‘Finalise decision’ dialogue box appears with the following options:

  • Dispose of the records.
  • Extend the retention.
  • Re-label the records.

Disposed items tab

The Dispositions dashboard includes a ‘Disposed items’ tab.

Microsoft note that this tab ‘… shows dispositions [that] were approved for deletion during a disposition review and are now in the process of being permanently deleted. Items that had a different retention label applied or their retention period extended as part of a review won’t appear here.’

Importantly, once records are permanently deleted, they no longer appear in the ‘Disposed Items’ tab. This means that no record will be kept of the records that were destroyed.

Shortcomings of the O365 dispositions/disposal model for records stored in SPO

Only individual records appear, not all the items in a document library

If the retention outcome is based on the ‘created’ or ‘last modified’ date, individual records in SPO document libraries will start to appear as soon as they reach the retention end date. The reviewer may need (or want) to view the original library, which they can identify from the link is in the dispositions review pane.

Retention policies prevent deletion

As a retention label prevents the deletion of content by users, and this may put them off using SharePoint, it is recommended that retention in SPO document libraries be based on when the label was applied NOT when it was created or last modified. This will help to ensure that all documents appear in the disposition review area at the same time.

Event based triggers may not be suitable for disposition review

If the retention outcome is based on an event, or is auto applied and a disposition review is required, those records will appear randomly when the event is triggered. It could be difficult for records managers to decide the disposal outcome in this way without referring back to the library.

The dispositions review pane does not display the original metadata

The  dispositions review pane displays only very basic metadata from the original library. Again, the reviewer may need to view the original library, export the metadata and store that in a secure location. Note that the exported metadata includes the URL of each original record including the library name.

The document library remains even when all contained records are destroyed

If the reviewer chooses to dispose of the records listed, only the content of the library (the individual documents stored in it) is deleted, not the actual library itself. No record (e.g., a ‘stub’ of the deleted item) is kept in the library of the deleted content.

The ‘Disposed items’ tab only shows records being destroyed

The ‘Disposed items’ tab only shows records in the process of being destroyed. It does not keep a record of what was destroyed. Records managers will need to retain the metadata of what was destroyed, when, based on what disposal authority, and with whose approval.

Dispositions really only provides a ‘heads up’ for further action

The Dispositions process may be instead used as a form of ‘heads up’ that records are starting to be due for disposal in a document library. This would allow the records managers (who should be Site Collection administrators) to review the library, export the complete set of metadata, and decide if the entire library can be deleted since it is no longer required.

Conclusions

Retention labels in O365 are an effective way of managing the retention and disposal of records in that environment, subject to the following points.

Email

Emails will likely continue to be managed as complete aggregations of records – the mailbox. Users cannot be expected to create logical groupings and apply individual retention labels to those records.

Organisational records policies may mandate specific timeframes for the retention of email (e.g., 1 year), while HR/IT security policies may mandate that whole mailboxes are retained for a period of time after employees leave. It is important to understand the difference between these two models

Options to automatically transfer emails to SharePoint document libraries via rules may be possible using Flow but these rely on individual users to set up.

Consideration should instead be given to using O365 Group mailboxes, rather than individual personal mailboxes, for specific work related matters. For example, ‘Customer Complaints’, or ‘XYZ Project’.

OneDrive for Business Accounts

ODfB accounts may be covered by two forms of retention:

  • Retention labels that apply to all ODfB accounts while the account is active. These must be manually applied by users.
  • A separate retention period set for ODfB accounts after a user leaves the organisation.

If there is a requirement to prevent the deletion of content by a user from their ODfB account, the better way to achieve this is using an eDiscovery case with Legal Hold applied.

SharePoint Online

As most records will be stored in SharePoint document libraries (including Office 365 Group-based SP libraries), multiple retention labels will be required to address different types of content or retention requirements.

Careful consideration should be given to whether records can be deleted automatically at the end of the retention period or should be subject to disposition review, noting that the automatic deletion provides no opportunity to capture the metadata of the records.

The ‘auto-apply’ or event-based retention option should be used sparingly to avoid a trickle of records for disposal – unless there is enough trust that these can be accurately marked and deleted without review.

Shortcomings in the disposition review process support the following decisions for SharePoint Online content:

  • The number of retention labels should be minimised to avoid a very long drop-down menu when a label is applied. If current record retention or disposal authorities contain a lot of classes, some of these could potential be combined into a single class (e.g., ‘Company Records – 7 years’), while the site name and document library name should provide some context to the content to ‘map’ back to the original classes.
  • Retention labels should be applied when document libraries (or lists) become inactive as this will avoid conflict with users who want to delete content and also ensure that documents are ready for disposition review at the same time.
  • Retention labels applied to SPO document libraries should include the disposition review option unless a ‘delete only’ label is considered suitable for certain document libraries that clearly contain working documents or Redundant, Outdated and Trivial (ROT) content.
  • Records managers should review the content of all or most original SPO document libraries, and export the metadata of those libraries for storage in a separate location (such as an ‘archives’ site), or in the original library with the retention label changed to ‘Never Delete’. The original document library can then be deleted.
Posted in Compliance, Electronic records, Exchange Online, Governance, Information Management, Products and applications, Records management, Retention and disposal

Records retention in Exchange Online

Retention policies created as labels in the classification section of  the Office 365 (O365) Security & Compliance admin centre can be applied to content in Exchange Online (EXO) mailboxes.

It may not be possible to apply more than one Office 365 retention policy to EXO mailboxes because, unless the mailbox is dedicated to a specific subject (for example, ‘Customer Complaints’), or using a dedicated Office 365 Group’s mailbox:

  • Emails generally contain content about multiple subjects.
  • The way the content is categorised in mailboxes , including through the use of rules and/or folders, varies between users.
  • The retention and disposal of records relies largely on the ability to assign retention policies to categories or groups of records, not individual records.
  • Organisational policies may require all user emails to be kept (‘archived’) for a period of time after they leave the organisation.

Unless emails are moved to a different storage location such as SharePoint, it may be necessary to continue apply a single, but shorter, O365 retention policy to mailboxes.

Exchange Messaging Records Management (MRM) policies

Until Office 365 retention policies appeared as an option, MRM policies applied in EXO were likely based on an organisational business requirement to keep the mailboxes (and other content) of departed users for potential legal or compliance reasons.

MRM policies in EXO are found under the ‘Compliance Management’ section of the EXO admin portal.

EXO_Compliance_RetentionMenu.JPG

When this section is opened, the following message may appear:

EXO_Retention_Message

The default MRM policy has the following options. These may be modified, or additional retention tags created, as required.

EXO_Default_MRMPolicySettingsA.JPG

If the default MRM policies have not been changed (by the Exchange administrator), the default policy/ies will apply. This means that users can use the ‘Assign Policy’ option on folders and emails to decide how long they should be kept.

Emails that are deleted before a backup is made may not be retained.

Some organisations may decide to retain all emails and the mailboxes of departed users ‘forever’. They can do this by removing all the options except ‘Never Delete’.

How O365 Retention Policies are applied to Exchange

Retention labels created in Office 365 can be used to manage the retention of emails, including (to some degree) emails that have content that meets certain pre-defined conditions.

Retention labels are created in the Office 365 Security and Compliance admin portal under the ‘Classifications’ section. This section has three options:

  • Labels. This section is used to create both ‘Sensitivity’ and ‘Retention’ labels. There is also an ‘Auto-apply’ option in the Retention section.
  • Label policies. This section partially duplicates the options in the previous option (except the ‘Create’ option), and lists the labels that have been published.
  • Sensitive info types.

Auto-apply, as its name suggests, auto-applies an existing label based on certain conditions. The conditions are as follows:

  • Apply label to content that contains sensitive info. The sensitive info types are pre-defined options for (a) Financial data (e.g., credit card numbers), (b) Medical and Health (e.g., predefined health records), (c) Privacy (e.g., personal and sensitive information. There is also the option to create a Custom setting.
  • Apply label to content that contains specific words or phrases, or properties. This option works by looking for specific words or phrases.

New labels must be published before they appear or apply anywhere in Office 365.

During the publish process, policies must specify where (in the ‘Locations’ section) the policy is to be applied.

The default option is ‘All locations. Includes content in Exchange email, Office 365 groups, OneDrive and SharePoint documents.’ Alternatively, the policy may be set to specific locations including

  • The Exchange mailboxes of all or specific recipients, or excluding specific recipients.
  • All or specific SharePoint sites, or exluding specific sites.
  • All or specific OneDrive accounts, or excluding specific accounts.
  • All or specific Office 365 Groups, or excluding specific groups.

Note that content in Microsoft Teams is included in the Office 365 Groups options which includes both the SharePoint content and email/Teams chat content.

Mixing MRM and O365 retention policies – maybe not a good idea

If the default MRM policies are not removed, any O365 retention policy that is applied to EXO will appear in the list of retention tags under the default MRM policy, as can be seen in the screenshot below which shows three options in addition to the original MRM policies: ‘Temporary records – 7 days’, ‘Financial Records’, and ‘Company records – 7 years’. If nothing is changed in the environment, these policies can be applied by users to folders and emails. 

O365_MRM_ExchangeRetentionoptions

If the organisation has decided to remove all retention tags except ‘Never Delete’ and a new O365 retention policy is applied to EXO, the ‘Never Delete’ option will prevail and the O365 policy will not work.

Accordingly, careful consideration needs to be given to the creation of O365 retention policies that may be applied to EXO records.

Should user mailboxes be kept ‘forever’?

Many IT departments keep user mailboxes of departed staff (and most other content on the network) for a long time, usually on backups, ‘just in case’ they may be required for legal or compliance requirements, including investigations into misconduct.

Recent personal experience with subpoenas for mailboxes of departed staff indicates that 10 years is likely to be the maximum retention requirement for these types of records. There may be a case to keep certain individual mailboxes for much longer, which the O365 policy allows for.

 

What happens when emails reach the end of their O365 retention policy period?

O365 retention policies define how long records are to be retained before they are either deleted or ready for review (via the Records Management – Dispositions section of the O365 Security and Compliance admin portal).

The following options define what happens when retention is enabled:

  • Retain the content (a) for a specific period (n days/months/years) or (b) forever. Option (b) is the same as the MRM policy ‘Never Delete’.
    • Action to be taken at the end of the period (except ‘forever’): (a) Delete the content automatically, (b) Trigger a disposition review (i.e., notify specific people), or (c) Do nothing, leave the content as is.
  • Don’t retain the content, just delete it if it’s older than n days/months/years.
  • Retain or delete the content based on: (a) When it was created, (b) When it was last modified, (c) When the label was applied, (d) based on an event.

The three actions above define the options for records managers:

  • Allow the emails to be deleted automatically. This is possibly the easiest and most efficient option but it will result in the deletion of any emails when they reach the end of the retention period – if they are kept in Exchange. Importantly, if a specific period of time (e.g., 7 years) is set for email retention, this could start to delete the emails of users who are still with the organisation after that period expires. This fact may affect the retention period that is set. 
  • Trigger a disposition review – see below. This option would be onerous to implement; it would take a lot of effort to review the individual emails of a departed user as part of a disposition review. It would, however, allow for selective review by using the ‘filter’ option in the Dispositions area.
  • Do nothing. This option may be useful for specific types of records, but not emails.

Disposition Reviews

Emails that are subject to a disposition review will appear in the Records Management – Dispositions section of the O365 Security and Compliance centre. Note that the ‘Type’ must be changed from ‘Documents’ to ‘Emails’ to see the emails that are due for disposal. As noted above, while it is possible to filter by user to review the emails, this process could be quite onerous.

O365_EXO_DispositionsA.JPG

Summary

The nature of email makes it almost impossible to categorise them into categories that map to different retention and disposal policies.

Most mailboxes will be subject to a single retention policy.

Office 365 retention policies can and probably should replace the default EXO MRM policies that govern the retention of emails.

Retaining emails in the mailboxes they are stored in ‘forever’ is not a practical retention model. 10 years is a reasonable maximum period, but exceptions may be required.

If O365 retention policies replace EXO MRM policies, records managers need to specify (a) how long emails need to be kept for and (b) whether they can simply be deleted when they reach the end of the retention period or need to be reviewed before deletion.

References

‘Overview of Retention Policies’ https://docs.microsoft.com/en-au/office365/securitycompliance/retention-policies (accessed 9 August 2019)

‘Set up an archive and deletion policy for mailboxes in your Office 365 organization’
https://docs.microsoft.com/en-au/office365/securitycompliance/set-up-an-archive-and-deletion-policy-for-mailboxes (accessed 6 August 2019)

Posted in Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

SharePoint Online – records management options and settings

This post summarises the primary records management options, settings and ideas that can be applied in SharePoint Online to manage records.

This post should be read as the second part of my previous post on the records management options and settings available in the Office 365 admin and security and compliance portals. Some of these settings will be referred to in this post.

The options and settings described in this post should ideally form part of your SharePoint governance documentation.

SharePoint Governance

We have already seen in the previous post that Office 365 Global Admins (GAs) have access to all parts of the Office 365 ecosystem. But they should rarely solely be responsible for SharePoint Online (SPO).

Some form of governance arrangement is necessary for SPO, especially if you plan to manage records in that application.

Some of the key considerations are as follows.

  • Who is responsible for ‘marketing’ or promoting SharePoint in the organisation, and making sure it is used correctly? The area responsible in IT for change management should probably take the lead on this as SPO is only one part of the O365 ecosystem. Records managers should have a role too, or be consulted.
  • SharePoint Administrator. You should already have a SharePoint Administrator and that person (or persons) is likely to be sitting in your IT department. Records managers will rarely also be SharePoint administrators; the two need to work closely together.
  • Who is responsible for training people to use SharePoint, especially to highlight the recordkeeping aspects of the application?
  • Who are the Site Collection Administrators? See next point.
  • Who are the Site Owners?
  • Who can create Office 365 Groups?

Answers to these questions should all be documented in your governance documentation.

SharePoint Online Admin Portal

SharePoint Online customised administrator

The SPO administrator role, a ‘customised administrator’ set in the Office 365 (O365) portal, should normally have a log on that is separate from that person’s O365 user log on. The SPO administrator account should not be a generic one (and generic accounts should generally be avoided).

The SPO administrator accesses the SPO admin portal from the Office 365 admin portal. They will also have access to the O365 Message Centre and Service Health sections.

SharePoint Online Architecture

Why a design model is good to have

Organisations should have some sort of design model for their SPO architecture. Most records will be kept in document libraries SharePoint team sites under the /teams path but some could also be under the /sites path.

The design model should include naming conventions for sites to avoid site names that have unknown acronyms or complex names. Site names form part of the total 400 characters allowed from https to the document suffix (e.g., .docx) so site names should ideally be no longer than around 16 characters. For example:

https://tenantname.sharepoint.com/teams/sitename

Records managers should be involved in designing this architecture model and could also be part of any approval process for new sites, to ensure the proposed names are suitable.

The names of SPO sites should generally map to business functions. Where the main function is very large (e.g., Financial Management is very large, you may decide to create sites based on the ‘sub-function’. That is, under the broader Financial Management (or simply ‘Finance’) site, you could have a separate site for Finance AP and another for Finance AR). These can be linked to a hub site that could be the ‘parent’ function site.

Don’t mix functions (such as personnel and IT) in the same site if only because this site is likely to become very large.

Try to aim for team site coverage of all business areas as all areas are likely to create or maintain records.

One relatively easy way to do this is to consult with the business area and understand how they use their current Network File Share location. This has the additional benefits of ‘mapping’ their SPO site to their existing NFS structure (generally or very specific) so it is familiar to them, and assisting with the migration of NFS to SPO later on.

Creating new Site Collections

Generally speaking there now are three types of SPO site:

  • A team site not linked to an O365 Group (but can be retrospectively linked)
  • A team site linked to an O365 Group
  • A communication site

Again, generally speaking:

  • SPO team sites (linked or not with O365 Groups) are the functional replacement for network file shares and, accordingly, contain most of the ‘document’ type records.
  • SPO communication sites are used for publishing purposes, including the intranet. They may contain documents in document libraries that, again, replace network file shares previously used for this purpose.

New sites can be created:

  • Directly from the SPO admin portal.
  • Via the ‘Create Site’ dialogue available in each user’s SharePoint portal, when this option is enabled. When this option is enabled, users can create either a Team site (linked with an O365 Group), a Communication site, or a ‘classic’ site (not linked with an O365 Group).
  • When a new Office 365 Group is created. This includes, if enabled, when a new Team in MS Teams is created, a Yammer group is enabled, or the person choose to create a new group from Outlook. If this option is allowed, whoever creates the O365 Group becomes the Site Collection Administrator and the SharePoint admin will be unable to access the site. For this reason, organisations that want to control their SPO environment may wish to limit who can create Office 365 Groups.
  • Via a PowerShell script.

SharePoint Sites

Site Collection Administrators (SCA)

Every SPO site has Site Collection Administrators. To ensure that records managers can access every site to manage records, it may be useful to add them to the membership of a Security Group that is in turn added to every site’s Site Collection Administrators after it is created.

Site Collection Administrators are added and managed in Advanced Permission Settings.

SPO_AdvancedPermissions

When you click on Site Collection Administrators, this dialogue appears:

SPO_SCAs.JPG

As noted above, if the ability to create O365 Groups is not controlled, the person who creates the O365 Group (as noted in the screenshot above) will become the SCA. The SharePoint administrator will be able to see the site in the SPO admin portal but may not be able to change the SCA settings. They may need to ask a Global Admin to do this.

Being a Site Owner only is not sufficient for records managers. Site Owners should be someone in the business area that ‘owns’ and will manage the SPO site on a day to day basis.

Site collection features – document IDs and Document Sets

Site collection features are only accessible to Site Collection Administrators. The list below expands as new features are activated; as can be seen, the ‘Document ID service’ feature has been enabled on this site. (Note, ‘Site features’ are activated from the Site Administration section, see below).

SPO_SiteCollectionAdminOptions.JPG

The Document ID feature is required for recordkeeping purposes as it assigns a unique Document ID to every object (including document sets but not folders) stored in a library.

If they are to be used in the site, the Document Set feature is also enabled in the Site collection features section.

SPO_DocIDDocSetSiteCollectionFeatures.JPG

After the Document ID service is enabled a new option appears in the Site Collection Administration section called ‘Document ID Settings’ (as noted above).

As can be seen in the screenshot below, all Document IDs begin with a unique set of up to 12 characters. Ideally, the Site name should be used as this will immediately give a clue to the site name on the document.

SPO_DocIDSettings.JPG

Document IDs take the form:

  • Prefix (e.g., ‘SITENAME’)
  • Library number. This is a unique and un-modifiable number of the library where the document is stored. It is not based on the library GUID.
  • Next sequential number.

If a document is deleted or moved from the library, the document ID (the sequential number) is not re-used.

Note that Document Sets use the same Document IDs. These cannot be separately modified.

Site collection features – Site Audit logs

The option ‘Site collection audit settings’ will already be visible in the Site Collection Administration section of all new sites, however (a) the options in the audit settings need to be enabled and (b) the ‘Reporting’ Site collection feature must be activated to enable the production of Site Audit Logs as required.

Note, the Site collection audit sections settings notes that ‘If you’d like to keep audit data for longer than this, please specify a document library where we can store audit reports before trimming occurs’. The default storage location is /_catalogs/MaintenanceLogs. However, the various options shown below must be selected for anything to be saved.

SPO_SiteCollectionAuditOptions.JPG

Enabling ‘Reporting’ results in a new section in the Site collection administration sections called ‘Audit log reports’. This section allows the Site Collection Administrators to create one-off audit logs for a range of activity on the site, going back 90 days.

  • Content Activity
    • Content viewing
    • Content modifications
    • Deletion
    • Content type and list modifications
  • Information Management
    • Policy modifications
    • Expiration and Disposition
  • Security and Site Settings
    • Auditing settings
    • Security settings
  • Custom
    • Run a custom report

The 90 day time period is the same as the O365 audit logs accessible from the Security and Compliance ‘Search’ section. If audit logs are required for longer periods, an add on may be required.

Metadata – Site columns or the Managed Metadata Service

The architecture model and/or business requirements may require the use of specific metadata across your environment. Metadata may be set in three ways.

Managed Metadata Service. This option is effective if you need to use the same metadata columns on multiple sites. Experience suggests that this option will be used selectively.

Site columns. These are in addition to the many columns that already exist by default on every site. This option is very effective if the same metadata needs to be used in multiple document libraries or lists on the same site. It is not accessible on any other site. In document libraries or lists, it must be added as an existing site column (i.e., not via the Create new column option).

Library columns. These columns are created in individual libraries or lists and are not accessible on any other library or list.

All new Site and Library columns have the following options:

SPO_NewColumnOptions.JPG

Each new column may be created in an existing or new group. They may also be (a) made mandatory and/or (b) enforce unique values. Note that making a Site Column mandatory and adding it to a document library will make the library read only in File Explorer if it is synced there.

Columns may have default values and may also include JSON formatting codes.

When Site columns are added to a document library, including via a Content Type (see next section), users may be required to fill in the required metadata (especially if it is mandatory).

Site Content Types

Site Content Types are a way to define metadata requirements for different types of documents, using Site columns. The default ‘document’ Content Type on every new SPO site is simply ‘Document’; all new document-based Content Types will be created using that one as a template.

Site Content  Types may also incorporate standard document templates (via the ‘Advanced section). These templates can be auto-populated using the library metadata. In any case, all metadata in a document library is added to any Office document as its metadata ‘payload’.

Once created, Site Content Types must be added to each individual library where they are to be used. To do this, the individual library must have the setting ‘Allow management of content types’ enabled in the ‘Advanced’ section of the document library settings.

SPO_DocLib_ContentTypesYesNo.JPG

When Content Types are enabled in this way, some other drop down features in the ‘+ New’ option on the library disappear, such as the ability to create Word, Excel or PowerPoint documents as can be seen below (the option on the right shows when Content Types are allowed).

Aggregations, containers, ‘files’ – Site Libraries

SharePoint document libraries are the container, aggregation or ‘file’ (if you will) in which records are stored. They are the functional replacement for network file shares. You may end up migrating from those NFS to SharePoint.

Naming conventions for new document libraries are useful to have but the extent to which you require people to follow them (if Site Owners create them) may differ between organisations.

Document libraries ideally should contain only a year of content; including the year in the library name is a good way to maintain year-based content, which in turn makes it easier to manage at the end of the record’s life.

Avoid using the generic ‘Documents’ library that comes with every new library because users will create folders with uncontrolled names and content.

All SPO document libraries and lists have default views of the metadata. These views can be modified as required (via the option on the top right of the menu bar) with a range of additional metadata that is by default hidden from view. Multiple views can be created; pre-defined views may sometimes be easier than expecting users to depend on searches.

Document libraries include all the usual and expected document management functions including check out/in, copy to or move to and versioning.

SPO_DocumentOptionsincCheckOut.JPG

Users with Contribute or Edit permissions can view and restore versions.

SPO_DocLib_VersionHistory.JPG

If there is a requirement to know who modified what part of a document, it is recommended to enable track changes on that document.

Note, with co-authoring now available, the last person to edit the document will create the last version.

Folders and document sets

Folders  should be seen as visual ‘dividers’ within a file, not as ‘hard-coded’ structures as they are in file shares.

Document sets can include additional metadata (including a document ID), making them suitable for use in breaking down a document library. However, for most of the time, folders are a more logical ‘divider’ for users.

Note that both document sets and folders look the same in a synced library.

Both folders and document sets can have unique permissions.

Create and capture records

One of the best reasons for using SharePoint is the ability to create a single source of truth. That is, a single record stored on a library that multiple people can access and work on at the same time.

Having a single source of truth avoids the requirement to (a) create a initial copy on a personal drive or network file share, (b) attach that copy to an email and send it to multiple people who are all likely to save it somewhere and also send back a changed version.

In SPO, users can create a new record directly within a document library (or in the synced library on a drive). Anyone with access to that library can access it; alternatively the document can be shared. Co-authoring means that anyone with edit access can edit the document. Every time it is edited and closed a new version is created.

If it is necessary to refer to the original from another library, the ‘Link’ option can be used.

Access controls and permissions

All SharePoint site contain three default permission groups. Individuals will usually be added to one of these groups only, depending on their access requirements:

  • Site Owner – Full Control across the site but cannot see the Site Collection Administration section (shown above). There will normally be only two to four Site Owners. Site Owners are responsible for managing their sites.
  • Site Member – Update and edit.
  • Site Visitor – Read only.

All content on a SharePoint site inherits the default permissions above however at any point the default permission inheritance can be broken and unique permissions applied. This is a manual process for document libraries (via Advanced Permissions) but automatically applied if a folder or document is shared with someone who is not in a default permission group.

Note, one of the leading support issues in SharePoint is understanding and unravelling complex permissions, especially when applied to individual documents that are placed ‘under’ folders with unique permissions, in libraries with unique permissions.

Retention and disposal

Generally speaking, a SPO site collection will consist of multiple libraries, each (ideally) containing content that is specific to the activity that it relates to. For example, a library for ‘Meetings’ and one for local forms. Consequently, the records stored in document libraries may require different retention.

If O365 Classification labels are used for retention, and depending on how these are configured, these must be applied per library; the individual documents stored in the library – not the entire library as such, are then governed by the retention requirement.

It is also possible to apply a policy to an entire site collection via site policies. This option will only be useful if the entire site can be subject to a single retention requirement – for example, inactive old sites that have a range of content all likely to be covered by the same retention period, or project sites.

Once retention policies are applied to a library, users cannot delete any content in the library so it may be prurient to apply them when the libraries are no longer used instead. Hopefully you will have implemented year-based libraries, which will facilitate this. Alternatively, the retention period trigger can start when the actual policy is applied.

It may be useful for the records manager to review the content of document libraries, and perhaps export the metadata of the library, before the content is disposed of via the O365 Security and Compliance area as any unique metadata is not visible in the Dispositions area.

When records are due to be disposed, an email is automatically send to whoever is in the Records Management role in the O365 Security and Compliance admin portal. The activity of reviewing and approving disposals happens in the O365 Security and Admin portal.

It may be useful to set up an ‘Archives’ SPO site to keep records of all disposal activities, including metadata from document libraries.

Note the library will remain even after the documents are destroyed. An alternative and perhaps better disposal model would be to use the notifications to alert the records manager to the records due for disposal; the records manager may then export and save the metadata in a SharePoint archives site, and then delete the library entirely.

Note that the retention of records in Exchange Online mailboxes and OneDrive may be managed differently by the organisation.

Minimising duplication of content

SharePoint allows organisations to have a single source of truth, to avoid the duplication of using NFS and then uploading to a document management system.

Users can create the record within a SharePoint library, upload it there, use the ‘save as’ option (where you will see all your SPO sites to choose from).

The ability to share with external users (when this is enabled) also helps to reduce duplication and email attachments.

Hybrid records

As noted above, links can be created in any SPO document library to point to resources in a different location. If paper records are managed in a SPO list, the document library can include a link to that SPO list.

Syncing document libraries to File Explorer

Users with an O365 licence and Windows 10 may use the ‘sync’ option available on the ribbon menu of every library. This option syncs the document library to the user’s File Explorer from where they may continue to access and work on the documents.

Note, as discussed in this post, if there is any mandatory metadata in the library, the synced library will become read only.

End users like using the sync option as, although it doesn’t (yet) display any unique metadata on the library, it allows them to work the way they have always worked and they get the added bonus of being able to do it on any device.

eDiscovery

eDiscovery cases are created in the O365 Security and Compliance portal. Essentially, an eDiscovery case uses search and other options to find records. Once found, these records can be placed on Legal Hold, which prevents their disposal.

If a document library has no retention label applied, and all or some of the content is identified as part of the eDiscovery case with a Legal Hold, and a user deletes a record, that record remains in a hidden library but still visible to the eDiscovery case manager. Once the Legal Hold is lifted, the record will resume the 90 day deletion process after which it will no longer be available.

Search

Search in SPO, and across all of Office 365, is very powerful. A single click in the Search box in the user’s SPO portal will result in suggestions before anything is entered.

Searches will return anything the user has access to. The access limits plus the Artificial Intelligence (AI) engine will return different search results for different users.

Users may also take advantage of the Office Graph-powered Delve (E3 licences and above) or the Discovery option in OneDrive to see information that may be of interest to the user. This works on the basis of the various ‘signals’ between users and objects, as depicted in the graphic below.

microsoft-graph_hero-image.png

 

Posted in Electronic records, Exchange Online, Governance, Information Management, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Office 365 admin and Security and Compliance portals – records management options and settings

If you plan, or want to understand how, to manage records ‘out of the box’ in the Office 365 ecosystem including in SharePoint Online, Exchange Online and MS Teams, you will need to know the available options and settings. These would normally be set by the Office 365 Global Admins (GAs) or, in some cases, devolved to Customised Administrators. GAs have access to all parts of the Office 365 environment including SharePoint Online, Exchange, OneDrive and Microsoft Teams.

See the next post for a list of the options and settings available in SharePoint Online to manage records.

Note, the description below is for a typical E3 licenced level organisation. E5 licences provide additional capability some of which is referenced below with a comment.

Office 365 admin portal options and settings

The options and settings in the Office 365 admin portal required to manage records are listed below.

Customised administrator

In addition to the GAs, the Office 365 admin portal is where customised administrators are set up. Typically these admins will have log ons that are different from their normal user log on and will not need the full range of licence options. The SharePoint Admin role is a customised administrator.

Records managers could potentially be SharePoint Admins if they are suitably skilled. Otherwise, at the very least they should be Site Collection Administrators and work closely with the SharePoint Admins to ensure that SharePoint Online (SPO) is configured correctly.

Office 365 Groups

Records managers need to understand how Office 365 Groups work.

Most people know that Distribution Lists (DL) are used to send emails to multiple people. However, DLs cannot be used to control access to IT resources; this is achieved by using Security Groups (SG). SGs, on the other hand, are not email enabled.

Office 365 (O365) Groups are ‘kind of’ a mix of DG and SG functionality in that they can be used to control access to certain resources in Office 365 (including SPO) AND they can be used to contact all members of the Group.

O365_AllGroups
Image source: https://techcommunity.microsoft.com/t5/Microsoft-Teams/NEW-Teams-IT-architecture-posters-published/m-p/480928#M30306

But O365 Groups are much more. They are in many respects central to Office 365.

  • Every new O365 Group creates a SharePoint site (this is not optional).
  • If the creation of O365 Groups is not controlled, every new Team in MS Teams creates an O365 Group that in turn creates a SPO site.
  • If you use Yammer, every new Yammer group also creates an O365 Group that creates a SPO site.
  • Again, if not controlled, any user can create a new O365 Group from Outlook.

In short, you need to either allow their creation and expect to see multiple uncontrolled SPO sites, or control their creation. There is no middle path.

Additionally, if the creation of O365 Groups is not controlled, the Owners of the new O365 Group (usually the person who created it and anyone else they invite) will become the Site Collection Administrators, locking the SharePoint Admins out of the site. They will need to call on the O365 GAs to give them access to the site.

External Sharing for SharePoint and O365 Groups

Although it relates more to security, external sharing is a option and setting that may require input from the information or records management area. External sharing is initially enabled in the O365 Admin portal in the Settings – Services and Add-ins section.

O365_Admin_SPOExternalSharing

Note, even if this setting is enabled, SPO sites don’t have this enabled by default. The setting is controlled from the SharePoint Admin portal.

External access for Office 365 Groups is set in the following setting:

O365_O365GroupsExternalSharing.JPG

Office 365 Security and Compliance admin portal options and settings

The options and settings in the Office 365 Security and Compliance admin portal required to manage records are listed below.

Permissions – Roles – Records Management (and others)

The Security and Compliance admin centre includes several roles in the ‘Permissions’ section that may be required by records and/or information management staff, especially to establish records retention schedules, manage dispositions, check audit logs and manage eDiscovery cases and legal holds.

Classification – Labels (Records Retention labels)

Records retention policies in O365 are set in the O365 Security and Compliance Portal in the Classifications section. These retention policies may be applied across SPO, Exchange Online, Teams.

Some thought needs to go into this including potentially grouping policies that have the same retention requirement (e.g., 7 years), or using the File Plan (see below) and other options now available to group them. This requires records management input.

O365_FilePlanDescriptors

Classification policies used for records retention will be applied across all of the O365 environment, not just SPO. However, your IT department may want to implement different rules for Exchange (e.g., using the default MRM policy to keep all emails ‘forever’) or OneDrive (e.g., a 7 year retention for everyone’s content after they leave).

Click this link for more details about Retention Policies.

Records Management – Dispositions

The O365 Security and Compliance Centre includes a ‘Records Management’ section that has three options: File Plan, Events, Dispositions. Records Managers should have access to these areas; this is achieved by them having the ‘Records Management Role’ in the ‘Permissions’ section.

The ‘File Plan’ section displays a list of retention policies (labels) with any details added to the ‘File Plan’ section (shown above), thereby providing the records manager with a view of all labels and any added details, for example by numbering, citation and so on.

The ‘Events’ section shows any events that have been defined for use in retention policies.

The Dispositions section has two parts, a basic dashboard that shows all retention policies and the number of records covered by those policies:

O365_RM_DispositionsDashboard

If the records manager clicks on any of the policies it displays the records due for disposal and provides the various options for disposal. It also shows records that have been disposed on a separate tab.

O365_Dispositions_Pending

Search

The search section has two options: Content search, and Audit log Search. Access to both may be controlled but records managers may need to have the ability to ask for information from either from the GAs.

eDiscovery

The eDiscovery section is where eDiscovery cases are established. Cases are a form of content search that, once completed, puts any retention policies on hold (legal hold) under the case has been removed.

eDiscovery cases may includes searches across all of Office 365 (Exchange email, O365 Group email, Teams messages, To-Do, Sway, Forms, SPO, OneDrive, O365 Group SPO sites, Teams sites, Exchange public folders) or selected parts only. They may also be used to search mailboxes for specific individuals or selected SPO sites.

Governance

All of the above (and all other settings) should form part of a governance document that details the O365 environment. Settings should only be changed with agreement of everyone in a governance team.