Posted in Classification, Compliance, Exchange Online, Information Management, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online, Training and education

Planning for records retention in Office 365

Office 365 is sometimes referred to as an ‘ecosystem’. In theory this means that records could be stored anywhere across that ecosystem.

Unlike the ‘old’ on-premise world of standalone servers for each Microsoft application (Exchange, SharePoint, Skype) – and where specific retention policies could apply (including the Exchange Messaging Records Management MRM policy), the various elements that make up Office 365 are interconnected.

The most obvious example of this interconnectivity is Microsoft Teams which stores chat content in Exchange and provides access to content stored in both SharePoint (primarily the SharePoint site of the linked Office 365 Group) and OneDrive, and has links to other elements such as Planner.

Records continue to be created and kept in the various applications but retention policies are set centrally and can apply to any or all of the content across the ecosystem.

Managing records in Office 365, and applying retention rules to those records, requires an understanding of at least the key parts of the ecosystem – Exchange, Teams, SharePoint and OneDrive and how they interrelate, and from there establishing a plan for the implementation of retention.

What types of records are created in Office 365?

Records are defined as ‘evidence of business activity’ and are often associated with some form of metadata.

Evidence of business activity is an overarching term that can include:

  • Emails
  • Calendars
  • Documents and notebooks (in the sense of text on a page)
  • Plans, including both project plans and architectural plans and diagrams
  • Images/photographs and video
  • Chat and/or messages
  • Conversations (audio and/or video based)
  • Social media posts

All digital records contain some form of metadata, usually displayed as ‘Properties’.

Where are the records stored in Office 365?

Most records created organisations using Office 365 are likely to be created or stored in the following parts of the ecosystem:

  • Exchange/Outlook – for emails and calendars.
  • SharePoint and OneDrive – for documents and notebooks (in the sense of text on a page), plans, images/photographs and video.
  • Stream – for audio and video recordings.
  • MS Teams – for chat and/or messages, conversations (audio and/or video based). Note that 1:1 chats are stored in a hidden folder of the Exchange mailbox of the end-user/s participating in the chat, while Teams channel chat is stored in a hidden folder of the linked Office 365 Group mailbox.
  • Yammer – for (internal) social media posts.

It is also possible to import and archive certain external content such as Twitter tweets and Facebook content in Office 365.

The diagram below provides a overview of the main Office 365 applications and locations where records are created or stored. Under SharePoint, the term ‘Sites’ refers to all types of SharePoint sites, including those associated with Office 365 Groups. Libraries are shown separately because of the potential to apply a retention policy to a library – see below.

O365WheretheRecordsare

Note also that this diagram does not include network file shares (NFS) as the assumption is made that (a) NFS content will be migrated to SharePoint and the NFS made read only, and (b) all new content that would previously have been stored on the NFS is instead saved either to OneDrive for Business (for ‘personal’ or working documents) or SharePoint only.

Creating a plan to manage records retention across Office 365

In previous posts I have recommended that organisations implementing Office 365 have the following:

  • A basic architecture design model for SharePoint sites, including SharePoint sites linked with Office 365 Groups (and Teams in MS Teams).
  • A plan for creating and applying retention policies across the ecosystem.

Because SharePoint is the most likely location for records to be stored (aside from Exchange mailboxes and OneDrive accounts), there should be at least one retention policy for every SharePoint site (or group of sites), as well as policies for specific document libraries if the retention for the content in those libraries may be different from the retention on the overall site.

For example, a ‘Management’ site may contain a range of general content as well as specific content that needs to be retained for longer. 

  • The site can be covered by a single implicit retention policy of (say) 7 years. This policy will delete content in the background, based on date created or data modified. 
  • The document library where specific types of records with longer or different retention requirements are stored may have one or more explicit label-based policies applied to those libraries. This content will be retained while the rest of the site content is deleted via the first policy.

Structure of a retention plan for records in Office 365

A basic plan for creating and applying retention policies might look something like the following:

  • User mailboxes – one ‘general’ (implicit) retention policy for all mailboxes (say, 7 years after creation) and another more specific retention policy for specific mailboxes that require longer retention.
  • SharePoint sites – multiple (implicit) retention policies targeting one or more sites.
  • SharePoint libraries – multiple (explicit) label-based retention policies that are applied manually. These policies will usually a retention policy that is longer than any implicit retention policy as any implicit site policy will prevent the deletion of content before it reaches the end of that retention period.
  • Office 365 Groups (includes the associated mailbox and SharePoint site) – one ‘general’ (implicit) retention policy. See also below.
  • Teams channel chat – one ‘general’ (implicit) retention policy. Note that this content is stored in a special folder of the Office 365 Group mailbox.
  • 1:1 chat – one ‘general’ (implicit) retention policy. This content is stored in a special folder of the participant mailboxes.
  • OneDrive documents – one ‘general’ (implicit) retention policy for all ODfB accounts, plus the configuration of retention after the account is inactive.

At a high level, the retention policy plan might look something like the following – ‘implicit’ policies are shown in yellow, SharePoint document libraries may be subject to ‘explicit’, label-based policies. The ‘+7 years’ for OneDrive relates to inactive accounts, a setting set in the OneDrive Admin portal.

O365WheretheRecordsare2

Regarding Microsoft Office 365 Groups, Microsoft notes the following on this page about managing retention in Office 365:

To retain content for a Microsoft 365 group, you need to use the Microsoft 365 groups location. Even though an Microsoft 365 group has an Exchange mailbox, a retention policy that includes the entire Exchange location won’t include content in Microsoft 365 group mailboxes. A retention policy applied to an Microsoft 365 group includes both the group mailbox and site. A retention policy applied to an Microsoft 365 group protects the resources created by an Microsoft 365 group, which would include Microsoft Teams.

The actual plan should contain more detail and included as part of other recordkeeping documentation (perhaps stored on a ‘Records Management’ SharePoint site). The plan should include details about (a) where the policies have been applied and (b) the expected outcomes or actions for the policies, including automatic deletion or disposition review (for document libraries).

Keep in mind that, unless the organisation decides to acquire this option, there is no default backup for content in Office 365 – once a record had been deleted, it is gone forever and there may be no record of this beyond 90 days.

Posted in Classification, Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, SharePoint Online, Training and education

AI curated chaos or control – the equally valid but opposite ends of the SharePoint spectrum

There are, broadly speaking, two ‘bookend’ options when it comes to creating new SharePoint Online sites and the document libraries in those sites:

  • ‘Controlled’ model: The creation of new sites is restricted to a small group of individuals with admin rights, who also oversee the creation of document libraries and application of metadata. A combination of controlled and manually applied classification and metadata and retention policies are used to access and manage content over time. Artificial intelligence (AI) tools can also be used to manage content.
  • ‘Chaos/uncontrolled’ model: The creation of new sites, including the creation of document libraries is not restricted. AI tools (including auto-classification) and auto-applied retention policies are used to classify, access and manage content over time. This model assumes that any form of random categorisation applied by end users (e.g., library names, metadata) is mostly ignored by AI tools.

From a traditional information governance and records management (ISO 15498/ISO 16175) point of view, the second ‘chaos’ or uncontrolled model option seems to run counter to conventional wisdom and agreed standards.

From a practical point of view, the first ‘control’ model option seems to run counter to common sense given the volume and range of digital information and the difficulty of classifying or categorising information and records correctly.

Which option is better?

Confusingly, perhaps, the answer may be a combination of both.

  • Certain types of more formal records, such as those required for corporate compliance, formal policies, staff files, accounting information not stored in a finance system, property information, and/or product information, is almost certainly going to be better off in a controlled SharePoint sites with pre-defined libraries and metadata. These types of documents are more likely to be subject to records retention requirements and almost certainly may be subject to eDiscovery and legal holds.
  • Other types of less formal records, including ‘working’ documents, chats and conversations may be better off stored in uncontrolled SharePoint sites, including SharePoint sites linked with Office 365 Groups and Teams, and in MS Teams/Outlook. These types of records are less likely to be subject to records retention requirements but may be subject to eDiscovery and legal holds.

Ultimately, the way the organisation needs to implement Office 365, including SharePoint Online and apply retention policies and other options will depend on its need to comply with oversight and legal requirements (including minimum retention periods), and/or its tolerance for risk.

How does this work in Office 365/SharePoint Online?

If both options Organisations need to make a conscious decision to allow both options, and be prepared to manage both.

The key features of Office 365 and SharePoint to allow both options are listed below:

  • Office 365 retention policies apply to all of Exchange Online, all OneDrive for Business accounts, entire sites (invisible to users) or parts of sites (visible to users).
  • Some retention policies may be applied based on the auto-classification of records, subject to review.
  • The creation of SharePoint sites is either controlled (requested and provisioned) or uncontrolled (created by end users) via either (a) ‘Create sites’ in the end-user SharePoint portal or (b) when a new Team is created in MS Teams.
  • All sites, including Office 365 Group/Team sites are reviewed regularly for activity and inactive sites with no content of value deleted.
  • All controlled sites are assigned either an invisible retention policy or individual visible retention policies (with disposal review), depending on their content.
  • All uncontrolled sites are assigned an invisible retention policy. Uncontrolled and inactive sites with content are also made read only.

Features of controlled and uncontrolled SharePoint sites

SharePoint Online is quite different from older versions of the application and those who dismiss it based on previous experience should consider having another look as a lot has changed in the past couple of years.

SharePoint Online allows the creation of sites that contain important content that needs to be controlled of managed as records, as well as sites created and managed entirely by end-users. And, as an added bonus, all the content is stored in the one place, not in multiple locations (network drives, email servers, EDRM system, etc).

The elements that make up both types of sites, as well as ‘informational’ sites, are described below:

  • Controlled sites
    • Where the organisation’s official records are stored and managed.
    • Created by SharePoint Administrators.
    • More formal in nature, containing the official records.
    • Structure decided by business areas – for example, document libraries using agreed naming conventions.
    • Use of Content Types and site column or local library metadata to define the content.
    • Application of Office 365 retention policies to entire sites or individual document libraries, with disposal reviews. Auto-classification is less likely to be required as the content has already been structured as required.
  • Uncontrolled sites
    • Usually based on end-user created Office 365 Groups or MS Teams.
    • Where ‘working documents’ are created and managed, with the emphasis on allowing end-users collaborate and communicate easily and effectively – and move content to formal sites when required.
    • Created by end-users but naming monitored by SharePoint administrators (or using rules).
    • Informal in nature, used for working documents (effectively replacing personal and network file shares, and other unapproved systems).
    • A fluid structure for document libraries, driven by end-user requirements (not imposed by others).
    • Little if any use of Content Types or metadata.
    • Retention based on Group activity (E5 licences), otherwise based on Office 365 site retention policies and/or auto-classification options.
    • No disposal reviews – content is deleted after a given period of time.
  • Informative
    • Communication sites (e.g., ‘intranet’)
    • Used to publish information to the organisation

Things to watch out for

It is largely true that if you give people an option, someone is bound to try it, sooner or later, especially if it says ‘Create site’, ‘Create team’, or ‘Create group’. Early adopters learn quickly and can just as quickly abandon something that provides no benefit. 

In a ‘free for all’ SharePoint environment, where end-users can create new sites, teams or groups (both of the latter have a SharePoint site), the most likely issues will include:

  • Sites with names that are very similar to ones that already exist, created because the end-user didn’t know another existed (it may not be obvious) or didn’t like the name.
  • Sites with names that make no sense (including common acronyms) or are just ‘wrong’ or contrary to preferred naming conventions.
  • Sites used to create and store content that really should be stored in a more formal site or, conversely, doesn’t belong in the organisation’s official information systems (e.g., photos of someone’s wedding).

All of these issues require some general rules about the creation of new sites (or Office 365 Groups or Teams or Yammer Groups), including suggested naming.

Global and SharePoint admins can monitor the environment and fix issues when they arise rather than wielding a big stick.

What’s great about it

You can have the best of both worlds with SharePoint Online.

  • Keep formal official records in ‘formal’ sites with controlled structures and metadata.
  • Allow end-users to get on with creating, collaborating, sharing (one copy, not attachments), chatting, on any device.

If your communications and change management are good, end-users will soon learn how much fun it can be to use Teams, or access their content from File Explorer (or both!), without having to having to be trained how to save records. All they need to know is how to use the ‘Move’ option to move the final version of records to a formal site.

The foundation of any compliance program is knowing where all of your data lives and then classifying, labeling, and governing it appropriately.

Posted in Disasters, Electronic records, Information Management, Information Security, Legal, Office 365, Records management, Retention and disposal, SharePoint Online, Training and education

Why is it so hard to ‘go digital’?

I visited a local fast-food outlet recently and could not help but notice the ‘Lever Arch’ binders in the small office behind the counter. A small two-drawer filing cabinet was also located below the desk.

20191002_125518

It made me wonder – in this day and age when pretty much everyone has access to the internet including via their smart phone, why are there any paper records?

And, why is it so hard to ‘go digital’, when so many better and safer digital options are available?

Reasons for not going digital

People probably want to keep paper records in this digital age for a few fairly common reasons, all of which I’ve encountered over the years.

  • Ease of access. It is much ‘easier’ to access a record if it’s in the folder with an obvious name, like ‘Rosters’.
  • Speed of access. You can access a paper record in a couple of seconds. Accessing the same record on a computer means logging on then searching or navigating to where it is stored (potentially including on personal removable storage devices).
  • Easier to archive. At the end of a given period the records can ‘simply’ be placed in an archive box and sent off for archiving.
  • Keeping digital records is too ‘hard’.
  • The company doesn’t offer any other option.
  • ‘Computers are hard’.
  • No obvious or pressing business reason to go digital.
  • A preference for paper, or belief that paper records must be kept.

Which of the above have you encountered? Let me know via this anonymous Form:

Or click this link:

https://forms.office.com/Pages/ResponsePage.aspx?id=DQSIkWdsW0yxEjajBLZtrQAAAAAAAAAAAAN__td1WRVUM0hJM0g2Q1NCWFdLS0JYM0k5QUlOUVUxRC4u

Keeping paper records can be risky

Keeping paper records can be all well and good, unless this sort of thing happens:

burger-king-fire-hed-2017-1260x840
Source: https://finance.yahoo.com/news/burger-king-used-photos-real-105654804.html

If you keep paper records when better digital options exist, you are taking a calculated risk that doing so is ‘OK’.

Of course, not all businesses (a) store the only copy of their physical records locally or (b) burn down (including by being constructed in fire-prone areas). However, these are not the only risks. Other risks include:

  • Flooding, from burst pipes, storms, or floodwaters. Water-damaged records are not easy to recover.
  • Damage from falling objects, including trees or other objects falling from the sky.
  • Theft or vandalism.
  • Business closure and leaving records behind in the abandoned building.
  • Any combination of the above.

What’s the back up for physical records?

What’s the back up for these paper records when disaster strikes?

Generally, unless the physical records have been transferred off-site, or they are the printed version of a digital original that can still be accessed, there isn’t one.

Is there a better, digital way?

Yes.

Printed records are likely to fall into several broad categories, each of which can be managed in their own way. For example, in the business above:

  • Policies and procedures, including ‘operating manuals’ and similar types of instructions are likely to be the printed version of digital originals. They can be made available on the company intranet or, if one doesn’t exist, sent via email.
  • Financial records (e.g., invoices). Again, these are likely to be the printed version of a digital original. If they were in printed form when received (e.g., by mail, with a delivery), the company should (a) ask for digital copies to be sent by email, or (b) scan them and store them digitally.
  • Rosters and general documents relating to groups of employees (as opposed to individual staff ‘files’). Rosters could still be printed for display purposes, but the original should be kept in digital form.
  • Staff files. The format of these may depend on the organisation, but there should be no reason for ‘local’ staff files to be kept in an organisation that has a centralised HR system.
  • Other types of business documents. If necessary, these could be scanned and kept in digital form.

And, of course, all of these could be kept in Office 365, including SharePoint for document storage and MS Teams for teams chat, including for front line workers.

Additional training and support may be required to help these areas ‘go digital’.

 

 

Posted in Records management, Electronic records, Products and applications, Legal, Training and education, Information Management, SharePoint Online, SharePoint Designer

Auto-populating document templates via a form in SharePoint

Most organisations have standard agreements or contracts or similar types of documents.

The common factor between them is that the original template remains the same while elements within the document change. For example, a client name, address and phone number, or differing contract terms.

There are several different ways this is achieved, including:

  • Printing the form and completing it manually.
    • This is time-consuming, handwriting can be difficult to read or require the form to be re-completed, and there is no easy way to extract the data. These types of forms are often scanned for storage.
  • Completing a digital version in Word (and sometimes printing/scanning or saving as a PDF).
    • This is also time consuming and in many cases it can be faster to print the form to fill it in by hand. Errors and omissions are possible and if the metadata appears in more than one place it must be re-typed. There is no easy way to extract the data.
  • Using editable PDF forms, sometimes using (Adobe or other) digital signatures.
    • These are very common (and very useful for specific purposes such as simple forms, less so for common agreements). They are time consuming, errors and omissions are possible and metadata must be re-typed. There is no easy (or cheap) way to extract the data.

Common factors in all of the above are that they are time-consuming and the data is hard to extract from the form.

A better and more efficient option

This post describes how to create a form in SharePoint that, via a very simple workflow:

  • Auto-creates one or more Word documents (multiple based on metadata choices contained in the form).
  • Auto-populates the Word documents where required with the metadata in the form. Where the same metadata value (e.g., ‘Client Name’) appears more than once, that value appears throughout the document where required at the same time.
  • Stores that document (or documents) in a folder (actually a document set) that can be used to add other content.

Additional benefits are that:

  • The metadata is easily accessible for export and other uses.
  • The Word document can be ‘signed’ with a touchscreen computer.
  • The Word document can be saved as a PDF.
  • Other documents can be added to the same folder.

This post is based on several actual examples that I developed (with the assistance of our SharePoint Developer) in a very large (9,000 staff) organisation.

The primary uses were for client agreements based on standard templates, including up to 10 different documents per client. We also deployed other designs that used a similar methodology, but the underlying principle was the same.

Note that, while the model is actually simple to implement, this post contains all of the details to follow step by step. I’m not a fan of posts that only provide part of the details and leave the rest to the imagination.

Setting up the model

Important note: The SharePoint site MUST have the document set feature enabled in the Site Collection Administration settings. Otherwise, the option to create a custom document set will not appear.

The model consists of the following elements that can be created by a SharePoint Administrator, a Site Collection Administrator or a Site Owner.

  • New site columns that will map to the elements in the form. For example, ‘Client Name’, ‘Client Address’, ‘Client Phone Number’. Note that every SP site has a lot of standard site columns so some of these can be used instead of creating new ones.
  • A new document set site content type containing all the site columns that should appear in the form. (‘Add from existing site columns’ option). It is recommended you give the document set a name that will be clear to end users as they will select this from a list. For example ‘Client Folder’ or ‘Agreement folder’.
  • A new document site content type for every template that is needed. The actual document template are not added now, only after the content type has been added to the document library – see below. It is recommended that you give each of these document CTs a name that is similar to the name of the document template.
  • A document library. It is recommended that you create a dedicated library for this purpose with a name that makes it very clear what it houses, for example ‘Client Agreements’. See below for the set up of the library.

Once all of these options are in place, the SharePoint Designer workflow can be set up – see below.

Setting up the document library

Library settings

The document library needs to be set up as follows in the Library Settings section.

  • In Advanced settings, enable the option ‘Enable management of content types’. This will make a new section ‘Content Types’ appear in the Library Settings.
  • In the newly visible ‘Content Types‘ section and choose ‘Add from existing site content types’ and add all the new site Content Types that were created.
  • The newly added CTs will now be visible, along with the default ‘document’ content type.

Document set CT settings

Click on the new document set CT. The metadata site columns that were added should be visible in the ‘Columns’ section.

Click on ‘Document Set settings’. In the section ‘Allowed content types’, click then use the ‘Add’ option to add all the document CTs that are required. These will now appear in the right-hand section.

 

O365_SPO_AddDocCTtoDocSetCT

Scroll down to ‘Shared columns’ and select all the document set columns. It does not matter that these will be shared with document CTs that don’t use the columns, as we will see below.

Click OK and return to the library settings area.

Adding the templates

At this point it is assumed that you have one or more document templates ready to upload. The template/s should be in a newer version of Word (e.g., .docx NOT .doc).

The ‘Content Types’ section of Library Settings displays a list of all the CTs that were added, including the document set CT (which will not be changed).

To add the template, click on the name of the (document) CT. In the new page that opens, you will see the list of site columns that have been shared from the document set.

Click on ‘Advanced settings’, where you will see the ‘Document Template’ section. Click the ‘upload a new document template’ option, choose your document template, and click OK.

O365_SPO_AddTemplatetoDocCT

Link the metadata columns with the template

Now, return back to the document CT ‘Advanced Settings’ (if you are not still there) and click on ‘Edit Template’ to open the template document in Word.

Now, add the metadata site columns where they are required in the template. For example, next to ‘Client Name’, place the cursor where you want the metadata to appear (don’t forget to include a space!).

In Word, go to the ‘Insert’ option on the ribbon menu and then go to the ‘Text’ section. Choose the ‘Quick Parts’ > Document Property and you should see the metadata columns as shown below.

O365_SPO_InsertMetadatainDocTemplate

Add the relevant document metadata where it should appear in the Word template. You will notice that the same metadata element can be used in multiple locations throughout the document. You can also use these in the header and footer and apply different formatting as required.

If you have made an error, do not ‘delete’ the added metadata in square brackets, instead right click and choose ‘Remove content control’. Be careful of formatting too especially different fonts and font sizes. Some of these will be more visible once you create the first document (see below).

The finished template will look something like the screenshot below.

 

 

O365_SPO_MetadataInWordTemplate3

Repeat for each content type template.

Summary and outcomes of the first stage

The site and library set up stage is now complete. The new content types now appear in the ‘New’ menu as shown below. You may want to edit the new menu options to remove any option you don’t want to appear, such as ‘Folder’ and ‘Document’ (you cannot remove ‘Link’).

O365_SPO_CustomCTDocLibNew.JPG

If the end user selects ‘Client Agreements’, they will be presented with a form to complete such as the example below – but this does NOT yet create the template document. That’s the next step below.

 

O365_SPO_CT_DocSet_NewForm3.JPG

 

Note that the order of these metadata elements can be moved around as required via the document set settings.

Create the workflow

You will need access to and be able to use SharePoint Designer to complete this section.

Remember: The workflow is based on the end user selecting and completing a new (document set by completing the form as shown above. The workflow is triggered by the fact that a new item has been created, which in turn creates and saves a new document (or documents as required) with the metadata populated automatically ‘inside’ the new document set.

Open SharePoint Designer

First, click on ‘Lists and Libraries’, choose the library that the workflow will be associated with, then click on ‘List Workflow’ as shown in the ribbon menu below.

SPD_NewListWorkflow

Give the workflow a name that will help to identify it in future – in this example, ‘Create Client Agreement’ would be a suitable name. Note:

  • You must create this as a SharePoint 2010 workflow.
  • The workflow can create one or more documents. In this example, only one document is created.

New workflow settings

A new tab will open. On the top right of the ribbon menu, click on ‘Workflow Settings’.

In the ‘Start Options’ section, check the box to start the workflow automatically when an item is created. The manual start checkbox should already be checked. This will allow the end user to run it again if required.

SPD_StartOptionsSP2010

Note – Some organisations may prefer not to allow the workflow to start automatically because they want to check the form first. In this case, the document set-based form can be created, but only after it is created the end-user must choose to run the workflow via the ‘More – Workflows’ option from the 3-dot menu.

 

Create local variables

Click on the ‘Local variables’ option on the top right of the ribbon menu to create (Add) two local variables:

  • DocSetName < this one is used to record the name on the document set.
  • DocumentPathforClientAgreement < this one is used to save the new document ‘under’ the document set.

Create the workflow

In the Workflow settings, click on ‘Edit workflow’ to create the workflow. For this example, there are two steps.

Click on ‘Step’ to change the name to something like ‘Initialisation’ or ‘Initialise variables’.

SPD_WorkflowNewStep1Blank.JPG

In this part we add and configure the two local variables that were created.

Click where it says (‘Start typing …’), click on on ‘Action’ in the ribbon menu, and choose ‘Set workflow variable’ to set the two variables.

  • Set Variable: DocSetName to Current Item:Name
  • Set Variable: DocumentPathforClientAgreement to [%Variable: DocSetName%]

Both of these will be set as a String value.

SPD_SetSP2010WorkflowVariable1.JPG

Click just underneath the step; a short orange line should appear. Click on ‘Step’ from the ribbon menu to create the next step.

(Note – a screenshot of all the following steps can be seen below)

  • Rename the step if required (e.g., to ‘Create Agreement’).
  • Click in this new step where it says (‘Start typing …’), then click on ‘Action’ (ribbon menu) and choose ‘Create List Item‘.
  • Click where the new action says ‘this list‘. A new dialogue box opens ‘Create new list item’. Select the name of the library from the drop down list in that dialogue box.
  • As soon as you do this, ‘Path and Name (*)’ appears below ‘Content Type ID’. You must complete the second part of this command before it can be saved.
  • Click on Path and Name (*) and click ‘Modify’. The ‘Set this field’ option should not be changed, only the option ‘To this value’. To the right of the blank field click the ‘fx’ option, then do the following.
    • For ‘Data Source’, choose ‘Workflow variables and parameters’.
    • For ‘Field from Source’, choose ‘Variable: DocumentPathforClientAgreement’
    • For ‘Return field as’, leave it as a ‘string’ value.
  • After you click save, the ‘Value assignment’ dialogue box should still be open. If not on the ‘Path and Name (*)’ option, then Modify, which will open the ‘Value assignment’ dialogue.
  • Click on the three dot menu option (to the left of fx) to open the ‘String Builder’ dialogue. Modify it as shown below by adding the prefix text. This puts the name given in the document as the first part of the document name: [%Current Item:Name%]/[%Variable: DocumentPathforClientAgreement%]
  • Note, you can add anything else you want after the last ‘]’, for example ‘- Client Agreement’, as a suffix to the document name.

SPD_WorkflowNewStep2CreateListItemA.JPG

Click OK (several times) to close the dialogue.

Add a ‘Stop the workflow and log’ option from the Action menu.

The final workflow is shown below:

SPD_TwoStepWorkflowFinal

Publish the workflow

Finally, publish the workflow. You can also press ‘Save’ to save without publishing. Publishing also saves any changes.

Allow some time for the workflow to appear in the document library. Generally this is fairly quick – refreshing the site page may assist.

Confirm the workflow is ready

To confirm the workflow is ready, click the three dot menu to the right of the document set and click on ‘More’, then ‘Workflow’.

The new workflow should appear similar to the screenshot below.

Note that this is the primary interface for most actions relating to the workflow. From here you can click the workflow to run it again any time (Manual start). If the workflow has a problem you will see that message here under ‘Running Workflows’; from there you can terminate the workflow if it has a problem (which sometimes happens – the clue is that the document was not created).

SPD_LibraryWorkflowReady

End result

When the end user completes and saves the form, the workflow will run, creating one or more documents (based on the template) ‘inside’ the document set. Each document will have the correct metadata based on the template.

O365_SPO_MetadataInWordTemplateFilled3.JPG

 

Benefits

There are many benefits to creating this model to manage common document agreements, contracts and other templates.

  • The document template always remains the same and can be updated at any time (but note that entire template updates require re-connecting all the metadata elements).
  • If a mistake is made in the metadata, the end user can simply delete the documents that were created and re-run the workflow as many times as required, saving a lot of effort in having to re-populate an entire document. If there is concern about deleting documents, the manager can set an alert on the library. The Recycle Bin keeps deleted documents for 90 days.
  • All Word documents created this way include the metadata from the library in their properties (the ‘metadata payload’). This includes the Document ID (if enabled).
  • Once the Word document has been created it can be ‘signed’ electronically using touch screen technology. If you really need a more sophisticated signing process, consider acquiring a third-party product.
  • Once the Word document has been signed in this way, it can be saved as a PDF, preventing changes.
  • If saved as a PDF, the defaults save location is the same location. Saving to PDF is a three step process: Open the Word document, click ‘Save as’, and change the option to PDF.
  • All the metadata site columns can be exported for analysis and reporting purposes. It may be also be used to created groupings of records for example ‘All contracts created by users’, or ‘All contracts that have a specific metadata choice option’.
  • The newly created Word or PDF documents can be shared, including with external people if required.

Negatives

In practice we found that there were not many negatives associated with this model and it brought considerable productivity benefits to the business areas that regularly created multiple agreements with clients, based on standard templates.

The primary negatives we found were:

  • Poor bandwidth meant that the new Word agreement may not create as quickly as required. Business areas with this problem kept both digital copies of the agreement to complete or printed versions.
  • If the entire template had to be changed, all the metadata links had to be re-connected. It was usually much easier only to update the part of the document that needed to be updated, including by adding new pages.
  • Every once in a while the workflow would not work. Our first clue to this was that an end user would call to say the document was not created or a metadata field was blank. We could usually track this problem down to either a network ‘glitch’ or other minor issue.
  • If metadata fields are left blank in the form, the square brackets metadata option remained visible. This then had to be deleted from the final.
  • From time to time, for various reasons, the end user would create a second copy of the document template without deleting the first. This simply creates a new document with the date and time as a suffix to the document name.
Posted in Classification, Electronic records, Governance, Information Management, Legal, Office 365, Products and applications, Records management, SharePoint Online, Training and education

Using the Sync option to work smarter and reduce duplication, and increase end user acceptance of SharePoint

Note: A correction was made to this post on 20 July 2019, relating to if a document library contains mandatory metadata.

Perhaps the single most common complaint about using electronic document management (EDM) systems over the last two decades has been the requirement to save a copy of a record stored on a network file share to the EDM system.

Network file shares are littered with documents, many of them duplicated in other locations, on personal drives (and removable drives), and attached to email messages. Some of these documents may also have been saved in the EDM system. 

It is a known fact that legal discovery activities rarely focus solely on the records in an EDM system, no matter how good that system may be. As long as network file shares (and personal drives) have existed (and continue to exist) alongside EDM systems, the latter has always been the poorer sibling in terms of information value.

Various attempts over the years by EDM vendors to ‘integrate’ their products with network file shares (often via WebDAV – see below) have rarely been successful not the least because the folder structure of the network file share is inevitably more useful and flexible than the often rigid structure of the EDM.

*WebDAV, or ‘Web Distributed Authoring and Versioning’ (RFC 4918) is ‘an extension to HTTP, the protocol that web-browsers and web servers use to communicate with each other’. WebDAV facilitates collaborative authoring, editing and file management. The most common usage of WebDAV is to map cloud storage as a network drive. (Source: WebDAV: What it is, where it turns up, and its alternatives, retrieved 18 July 2019)

The old ‘Groove-y’ way

Microsoft Office Groove 2007, or ‘Groove’, was a Microsoft Office component that used WebDAV to synchronise with a SharePoint library, allowing the library to be opened from Windows Explorer. (Source: Understanding and troubleshooting the SharePoint Files tool in Groove 2007, retrieved 18 July 2019)

While this method worked, it was clumsy and difficult to use. Duplication on network file shares continued.

2018 – The new OneDrive for Business sync client

The previous Groove OneDrive for Business sync client (Groove.exe) was included with the Windows 10 Operating System that was released in mid 2015.

The new SharePoint Online became widely available from 2016 and has continued to evolve. Initially, it was only possible to synchronise a SharePoint Online document library using WebDAV methods.

The new OneDrive sync client (OneDrive.exe), also known as the Next Generation Sync Client (NGSC), appeared in early 2018. The new sync client allowed users (with Windows 10 devices) to sync their SharePoint document libraries to File Explorer.

A mostly unnoticed but significant change

The sync option on SharePoint document libraries (in addition to OneDrive and OneDrive for Business) is possibly one of the least noticed changes that has the potential to have – ironically – both a major and also minor impact on the way people work.

It is a minor impact because – provided the synced document library does not have mandatory metadata (see below) – the change effectively allows users to continue working the way they always have, in File Explorer, going only to SharePoint Online when they need to.

It is a major impact because, coupled with the ability to ‘share’ content easily (directly from File Explorer), the potential for duplication – except for the duplication between ‘work’ and ‘personal’ spaces – has been removed. Everyone with access to it can sync the same document library and multiple people can work on documents in the library at the same time.

Instead of creating a ‘working’ document on a drive and perhaps emailing it to everyone, there now only needs to be a single copy that multiple people can access – via File Explorer, at the same time. Everyone with access can see when any other person is editing.

That is, end users can continue to work in File Explorer, the way they have always done. In that sense, the ability to sync a document libraries makes redundant the need to open a browser and access SharePoint that way. (This in turn impacts on the way change is managed and perhaps how each SharePoint site might be configured).

How it works

As a start it should be emphasized that this works best with Windows 10 as Windows 7 devices may still have the old ‘Groove’ client installed.

Please note also that this only works if there is no mandatory metadata on the document library. If there is, the users will be unable to add new content to the synced library, or edit existing documents. See below for more information.

Users need to go to the SharePoint site first and click on the library they want to sync. Users need to have edit rights on the library to sync it.

They should then see the Sync option:

O365_SyncRibbon

The OneDrive for Business client notifies the user that the library will be synced.

O365_Sync_ODfBClientB

The library is then synced to the user’s File Explorer.

Note: If the document library has any mandatory metadata, the user will be notified via a pop-up that the library has been synced in ‘read only’ mode.

A new icon (with the Office 365 tenant name) appears on the left, and each document library that is synced is shown as a folder beneath it.

If the document library has any mandatory metadata columns OR the library requires check out (via Versioning settings), an additional ‘lock’ appears to the right of the sync status. This means the documents cannot be edited and new documents cannot be added. (Source: Sync SharePoint files with the new OneDrive sync client)

SPO_FileExplorerLockMandatoryMetadata

If neither condition applies, end users can work directly in the synced document library in File Explorer, including adding new folders and documents.

End users may also select which folders they wish to sync either by opening a folder in SharePoint and syncing from there, or by right clicking on the folder that was synced, clicking on ‘Settings’ and removing any unwanted folders. This, of course, could mean that users don’t see new folders they really should see and may as a result attempt to create one with the same name (which will be rejected).

Documents are not downloaded to the user’s computer until they open them. This can be seen below in the first document with a circle/tick icon (downloaded) and the three others with cloud icons (not downloaded).

O365_Sync_FileExp_Docs.JPG

The user can right-click and use the Share option (the same as in SharePoint Online) to share the document with colleagues which (as long as the person sharing has the permission to do so) gives the other person access if they didn’t have it before. The three dots at the top right of the dialogue box provide the option to manage access to the document.

O365_Sync_FileExpl_ShareOption

Note: End users cannot copy and paste a link to sync a library, the sync runs from a user’s computer and is personal to their log on and their device.

End user reactions

Personal experience supporting thousands of end-users with access to SharePoint Onine indicated that this was perhaps one of the most useful features ever released.

Several people noted that they regarded the sync option as a ‘cloud-based backup’. Some indicated that they rarely returned to the browser version of SharePoint for their key document libraries (which may be problem).

What about metadata and content types?

Presently, document libraries synced to File Explorer do not display any metadata associated with the document or document library, only the icon, name, date, type and size.

However, Microsoft Office documents (Word, Excel, and PowerPoint) retain any original metadata in the document properties (the ‘metadata payload’) and these properties may be changed on the document itself via the ‘File’ option.

Any metadata columns are also ignored; a user may add a document directly to the synced document library in File Explorer without having to add metadata. Note that this is the same behavior in SharePoint Online; if a document is added to a library with a metadata column, a warning appears (see screenshot below) but the document can still be uploaded. (This paragraph was corrected on 20 July 2019 to remove reference to mandatory columns, which make the synced library read only).

SPO_ExampleMissingMetadata

Note also that new options coming soon to SharePoint Online, which will also be seen via the ‘Share’ option in File Explorer, is the ability to set restrictions such as the ability to print or download, or expiry dates.

The new way of working

The old way of working was to create and manage documents on network file shares and personal drives, emailing copies as required. Adding documents to EDM systems was an additional and disliked step that in most cases created a copy of a document that still remained on a drive somewhere. (And, in many cases, the EDM system had a linked file share where the documents were stored).

The new way of working minimises the need for duplication.

  • Users create a new Office document (including directly from OneDrive or SharePoint, where it is automatically saved in the library from which it was created)
    • If the document was not created from OneDrive or SharePoint, the ‘save’ dialogue presents the following locations by default: OneDrive (personal); SharePoint (any SharePoint site the user has access to – including the synced document library on File Explorer); or ‘browse’ to another location.
    • If the document is saved to the synced document library in File Explorer, it is then automatically copied to the SharePoint Online document library (and a green circle and tick appears).
    • If the document is saved to a SharePoint Online library directly, it will appear in a synced folder in File Explorer initially with a cloud icon.
  • The document may then be shared, either from File Explorer or in SharePoint Online (the same Share dialogue on both).
  • The recipient of the Share invitation can then open the document directly and edit it (if given those rights).
  • Any edits of the document will be recorded in the version history of the document. Other actions (e.g., changes to security) will be recorded in the audit logs.

However, if the library contains any mandatory metadata, the synced library will read only.

One document, stored in a single location, accessed by many. A new, much smarter, way of working.

Posted in Compliance, Data Loss Prevention - DLP, Exchange Online, Governance, Information Classification, Information Management, Information Security, Legal, Office 365, OneDrive for Business, Products and applications, Security, SharePoint Online, Training and education

SharePoint Online and OneDrive for Business – Preventing external sharing of data

A recent (September 2017) article suggested that OneDrive for Business (ODfB) (and by extension SharePoint Online (SPO); ODfB is a SharePoint-based service), a key application in Office 365 was a potential source of data leaks and/or target for hacking attacks.

I don’t disagree that, if not configured correctly, any online document management system – not just ODfB/SPO – could be the source of leaks or the target of external attacks. Especially if these systems, and the security controls that can protect the data in them, are not properly configured, governed, administered, and monitored.

But, I would ask, what controls do most organisations have in place now for documents stored in file shares and personal file folders, not to mention USB sticks, and the ability to send document via Bluetooth to mobile devices or upload corporate data to third-party document storage systems? Probably not many, because users have no other way to access the data out of the office.

As we will see, the controls available in Office 365 are likely to be more than sufficient to allow users to access to their documents out of the office, while at the same time reducing (if not eliminating) the sharing of documents with unauthorised users.

How to stop or minimise sharing from OneDrive for Business and SharePoint Online

There is one simple way to prevent the sharing of data stored in SPO and ODfB with external people – don’t allow it.

There are several ways to control what can be shared, each allowing the user a bit more capability. All these options should be based on business requirements and information security risk assessments, and Office 365 configured accordingly.

In this article I will start with no sharing allowed, and then show how the controls can be reduced as necessary.

External sharing – on or off

This is the primary setting, found in the main Office 365 Admin centre under Settings > Services & add-ins > Sites. If you turn this off, no-one can share anything stored in SPO or ODfB.

The option is shown below:

O365_SC_Sites_SharingOnOff

If you do allow sharing, you need to decide (as shown above) if sharing will be with:

  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users

The second option is recommended because it doesn’t restrict the ability to share with new users. The last option is unlikely to be used in most organisations and comes with some risks.

The next place to set these options are in the SPO and ODfB Admin centres.

OneDrive admin center

If the previous option is enabled, the following options are available for ODfB. Note that BOTH SharePoint and OneDrive are included here because the latter is a part of the SharePoint environment.

  • Let users share SharePoint content with external users: ON or OFF.
    • NOTE: If this option is turned OFF, all the following options disappear.
  • If sharing with external users is enabled, the following three options are offered:
    • Only existing external users
    • New and existing external users [Recommended]
    • Anyone, including anonymous users
  • Let users share OneDrive content with external users: ON or OFF
    • This setting must be at least as restrictive as the SharePoint setting.
  • If sharing with external users is enabled, the following three options are offered
    • Only existing external users
    • New and existing external users [Recommended]
    • Anyone, including anonymous users

If sharing is allowed, there are three sharing link options:

  • Direct – only people who already have permission [Recommended]
  • Internal – only people in the organisation
  • Anonymous access – anyone with the link

You can limit external sharing by domain, by allowing or blocking sharing with people on selected domains.

External users have two options:

  • External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]
  • Let external users share items they don’t own. [This should normally be disabled]

A final ‘Share recipients’ checkbox allow the owners to see who viewed their files.

SharePoint admin center

The SPO admin center (to be upgraded in late 2017) has two options for sharing.

The first option is under the ‘sharing’ section which currently has the following options:

Sharing outside your organization

Control how users share content with people outside your organization.

  • Don’t allow sharing outside your organization
  • Allow sharing only with the external users that already exist in your organization’s directory
  • Allow users to invite and share with authenticated external users [Recommended]
  • Allow sharing to authenticated external users and using anonymous access links

Who can share outside your organization

  • [Checkbox] Let only users in selected security groups share with authenticated external users

Default link type

Choose the type of link that is created by default when users get links.

  • Direct – only people who have permission [Recommended, same as above]
  • Internal – people in the organization only
  • Anonymous Access – anyone with the link

Default link permission

Choose the default permission that is selected when users share. This applies to anonymous access, internal and direct links.

  • View [Recommended]
  • Edit

Additional settings (Checkboxes)

  • Limit external sharing using domains (applies to all future sharing invitations). Separate multiple domains with spaces.
  • Prevent external users from sharing files, folders, and sites that they don’t own [Recommended]
  • External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]

Notifications (Checkboxes)

E-mail OneDrive for Business owners when

  • Other users invite additional external users to shared files [Recommended]
  • External users accept invitations to access files [Recommended]
  • An anonymous access link is created or changed [Recommended]

Sharing via the Site Collections option

In addition to the options above, sharing options for each SharePoint site are set in the ‘site collections’ section as follows. Note that the default is ‘no sharing allowed’. A conscious decision must be taken to allow sharing, and what type of sharing.

O365_SPO_Sharing1

When a site collection name is checked, the following options are displayed.

Sharing outside your company

Control how users invite people outside your organisation to access content

  • Don’t allowing sharing outside your organisation (default)
  • Allow sharing only with the external users that already exist in your organization’s directory
  • Allow external users who accept sharing invitations and sign in as authenticated users
  • Allow sharing with all external users, and by using anonymous access links

If anonymous access is not permitted (setting above), a message in red is displayed:

Anonymous access links aren’t allowed in your organization

SharePoint Sharing option

The SharePoint Admin Centre has an additional ‘Sharing’ section with the same settings as shown above for ODfB. It is expected that these multiple options will be merged in the new SharePoint Admin Centre due for release in late 2017.

Additional security controls

In addition to all the above settings, there are a range of additional controls available:

  • All user activities related to SPO and ODfB, including who accessed, viewed, edited, deleted, or shared files is accessible in the audit logs.
  • SPO and ODfB content may be picked up by Data Loss Prevention (DLP) policies and users prevented from sending them externally. This is of course subject to the DLP policies being able to identify the content correctly.
  • SPO and ODfB content may be subject to records retention policies set by preservation policies. These may impact on the ability to send documents externally.
  • SPO and ODfB content may be subject to an eDiscovery case.
  • Administrators can be notified when users perform specific activities in both SPO and ODfB.
  • Sharing (and access to the documents once shared) may be subject to security controls enforced through Microsoft Information Protection.

Conclusion

In summary, the settings above allow an organisation to strongly control what can be shared. If sharing is allowed, certain additional controls determine whether the sharing is for internal users or for users external to the organisation. If the latter is chosen, there are further controls on what external users can do. Audit controls and policies may also control how users can share information externally.

The key takeaway is that organisations should ensure that the sharing options available in Office 365 are based on the organisation’s business requirements and security risk framework.