Posted in Classification, Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, SharePoint Online, Training and education

AI curated chaos or control – the equally valid but opposite ends of the SharePoint spectrum

There are, broadly speaking, two ‘bookend’ options when it comes to creating new SharePoint Online sites and the document libraries in those sites:

  • ‘Controlled’ model: The creation of new sites is restricted to a small group of individuals with admin rights, who also oversee the creation of document libraries and application of metadata. A combination of controlled and manually applied classification and metadata and retention policies are used to access and manage content over time. Artificial intelligence (AI) tools can also be used to manage content.
  • ‘Chaos/uncontrolled’ model: The creation of new sites, including the creation of document libraries is not restricted. AI tools (including auto-classification) and auto-applied retention policies are used to classify, access and manage content over time. This model assumes that any form of random categorisation applied by end users (e.g., library names, metadata) is mostly ignored by AI tools.

From a traditional information governance and records management (ISO 15498/ISO 16175) point of view, the second ‘chaos’ or uncontrolled model option seems to run counter to conventional wisdom and agreed standards.

From a practical point of view, the first ‘control’ model option seems to run counter to common sense given the volume and range of digital information and the difficulty of classifying or categorising information and records correctly.

Which option is better?

Confusingly, perhaps, the answer may be a combination of both.

  • Certain types of more formal records, such as those required for corporate compliance, formal policies, staff files, accounting information not stored in a finance system, property information, and/or product information, is almost certainly going to be better off in a controlled SharePoint sites with pre-defined libraries and metadata. These types of documents are more likely to be subject to records retention requirements and almost certainly may be subject to eDiscovery and legal holds.
  • Other types of less formal records, including ‘working’ documents, chats and conversations may be better off stored in uncontrolled SharePoint sites, including SharePoint sites linked with Office 365 Groups and Teams, and in MS Teams/Outlook. These types of records are less likely to be subject to records retention requirements but may be subject to eDiscovery and legal holds.

Ultimately, the way the organisation needs to implement Office 365, including SharePoint Online and apply retention policies and other options will depend on its need to comply with oversight and legal requirements (including minimum retention periods), and/or its tolerance for risk.

How does this work in Office 365/SharePoint Online?

If both options Organisations need to make a conscious decision to allow both options, and be prepared to manage both.

The key features of Office 365 and SharePoint to allow both options are listed below:

  • Office 365 retention policies apply to all of Exchange Online, all OneDrive for Business accounts, entire sites (invisible to users) or parts of sites (visible to users).
  • Some retention policies may be applied based on the auto-classification of records, subject to review.
  • The creation of SharePoint sites is either controlled (requested and provisioned) or uncontrolled (created by end users) via either (a) ‘Create sites’ in the end-user SharePoint portal or (b) when a new Team is created in MS Teams.
  • All sites, including Office 365 Group/Team sites are reviewed regularly for activity and inactive sites with no content of value deleted.
  • All controlled sites are assigned either an invisible retention policy or individual visible retention policies (with disposal review), depending on their content.
  • All uncontrolled sites are assigned an invisible retention policy. Uncontrolled and inactive sites with content are also made read only.

Features of controlled and uncontrolled SharePoint sites

SharePoint Online is quite different from older versions of the application and those who dismiss it based on previous experience should consider having another look as a lot has changed in the past couple of years.

SharePoint Online allows the creation of sites that contain important content that needs to be controlled of managed as records, as well as sites created and managed entirely by end-users. And, as an added bonus, all the content is stored in the one place, not in multiple locations (network drives, email servers, EDRM system, etc).

The elements that make up both types of sites, as well as ‘informational’ sites, are described below:

  • Controlled sites
    • Where the organisation’s official records are stored and managed.
    • Created by SharePoint Administrators.
    • More formal in nature, containing the official records.
    • Structure decided by business areas – for example, document libraries using agreed naming conventions.
    • Use of Content Types and site column or local library metadata to define the content.
    • Application of Office 365 retention policies to entire sites or individual document libraries, with disposal reviews. Auto-classification is less likely to be required as the content has already been structured as required.
  • Uncontrolled sites
    • Usually based on end-user created Office 365 Groups or MS Teams.
    • Where ‘working documents’ are created and managed, with the emphasis on allowing end-users collaborate and communicate easily and effectively – and move content to formal sites when required.
    • Created by end-users but naming monitored by SharePoint administrators (or using rules).
    • Informal in nature, used for working documents (effectively replacing personal and network file shares, and other unapproved systems).
    • A fluid structure for document libraries, driven by end-user requirements (not imposed by others).
    • Little if any use of Content Types or metadata.
    • Retention based on Group activity (E5 licences), otherwise based on Office 365 site retention policies and/or auto-classification options.
    • No disposal reviews – content is deleted after a given period of time.
  • Informative
    • Communication sites (e.g., ‘intranet’)
    • Used to publish information to the organisation

Things to watch out for

It is largely true that if you give people an option, someone is bound to try it, sooner or later, especially if it says ‘Create site’, ‘Create team’, or ‘Create group’. Early adopters learn quickly and can just as quickly abandon something that provides no benefit. 

In a ‘free for all’ SharePoint environment, where end-users can create new sites, teams or groups (both of the latter have a SharePoint site), the most likely issues will include:

  • Sites with names that are very similar to ones that already exist, created because the end-user didn’t know another existed (it may not be obvious) or didn’t like the name.
  • Sites with names that make no sense (including common acronyms) or are just ‘wrong’ or contrary to preferred naming conventions.
  • Sites used to create and store content that really should be stored in a more formal site or, conversely, doesn’t belong in the organisation’s official information systems (e.g., photos of someone’s wedding).

All of these issues require some general rules about the creation of new sites (or Office 365 Groups or Teams or Yammer Groups), including suggested naming.

Global and SharePoint admins can monitor the environment and fix issues when they arise rather than wielding a big stick.

What’s great about it

You can have the best of both worlds with SharePoint Online.

  • Keep formal official records in ‘formal’ sites with controlled structures and metadata.
  • Allow end-users to get on with creating, collaborating, sharing (one copy, not attachments), chatting, on any device.

If your communications and change management are good, end-users will soon learn how much fun it can be to use Teams, or access their content from File Explorer (or both!), without having to having to be trained how to save records. All they need to know is how to use the ‘Move’ option to move the final version of records to a formal site.

The foundation of any compliance program is knowing where all of your data lives and then classifying, labeling, and governing it appropriately.

Posted in Correspondence Management, Flow, Information Management, Microsoft Teams, Office 365 Groups, Products and applications, SharePoint Online, Solutions

Managing correspondence in SharePoint Online including using Teams and Flow

Many (if not most) organisations receive and respond to correspondence in the form of letters or emails. They may also respond to social media messages.

Correspondence may be managed in different ways. For example, some organisations may have a dedicated business correspondence unit while in others individuals or business areas may respond.

This post describes how SharePoint Online with MS Teams and MS Flow could be used to manage letters and emails that require a formal ‘organisational’ response. It does not look at managing responses to social media messages.

Core elements

The management of correspondence that requires a formal response generally involves the following elements:

  • Somewhere to store the incoming correspondence (including scanned paper documents) and responses.
  • A description of the correspondence (naming conventions and/or metadata).
  • Some form of workflow, including emails, used to notify people of actions they must take.
  • Various methods use to report on and track progress, and closure/completion rates.

SharePoint includes all these elements. The requirement for workflow can be met using emails, the built-in workflows, workflows built in SharePoint Designer, or MS Flow. MS Teams provides the opportunity to add additional value to the communication response process by allowing messaging, co-authoring of responses, approval processes, and more.

A model correspondence system

The model correspondence system described below is based on a correspondence management system that I developed for a large public sector organisation a few years ago, but using a different system.

The core elements of the model system are:

  • An Office 365 (O365) Group. The O365 Group has an email address and an associated SharePoint site.
  • The Office 365 Group’s SharePoint site.
  • A Team in MS Teams, connected to the O365 Group.
  • Microsoft Flow.

Office 365 Group

The O365 Group should have a name that reflects its purpose and makes it easy to recognise as it also becomes the email address name.

  • For example ‘GRP_Correspondence’ – the prefix ‘GRP’ is used as it defines it as an O365 Group, as opposed to a Security Group (SG) or Distribution List (DL) created under the same ‘Groups’ section in the O365 Admin portal.

The Members of the Group should be the group of people primarily responsible for managing responses to the correspondence. Other people can be invited to the SharePoint site directly (without having access to the Group’s Team) as Members or Visitors.

SharePoint site

The O365 Group’s SharePoint site has the same name in the URL – GRP_Correspondence.

The SPO site should have at least one document library with an obvious name, for example ‘Correspondence 2019’.

If there is concern about potentially long URL lengths, the library name could be reduced to, say ‘Corro2019’; the display name can still be ‘Correspondence 2019’.

Consideration might also be given to having a ‘drop off library’ in the Group site where anyone can save correspondence that may require a response.

The metadata required in the document library will vary between organisations. The core default metadata (for every new document library) already includes all the following:

  • Title
  • Created (date)
  • Created by
  • Modified (date)
  • Modified by
  • Version
  • Document ID (when enabled as a Site Collection feature, which is recommended)
  • Shared with
  • Shared with details
  • Check In Comment
  • Checked Out To
  • File Size
  • Folder Child Count
  • Item Child Count (shows how many documents in a folder)
  • Label setting
  • Retention label
  • Retention label Applied
  • Sensitivity

Additional metadata may include any of the following:

  • Sender (free text or potentially drop down choice)
  • Response Due date (Date)
  • Urgency (Choice – Routine, Urgent)
  • Description (free text)
  • Reply Status (Choice – Not required, Draft, Approved, Sent, Cancelled)
  • Response Type (Not required, By email, By letter)
  • Approved by (internal Active Directory name)
  • Date Completed (Date) < This date should correspond to the date on any reply.

Once the library has been created, content can be added.

  • Paper correspondence can be scanned and saved to the library. Many MFD (printers) now have the ability to save directly to SharePoint, perhaps to a drop-off library for categorisation and review before being moved to the primary library. The original paper can then be boxed and destroyed after a given period (3 – 6 months).
  • Emails can (and should) be copied from the Group’s inbox to the library (see screenshot below). To do this, sync the library to File Explorer and drag and drop.

EmailtoSyncLibrary

A new folder can be created for each new correspondence, as indicated in the screenshot below:

SPOCorroLibrary.JPG

Naming conventions

Both the correspondence and the folders where it is stored should be named according to naming conventions. Naming conventions can also be used instead of folders, to indicate the connection between the original and the reply. My preference is to use folders to group the original and the reply, and also because they are ubiquitous in the digital workplace.

Suggested naming conventions:

  • Folders: Surname-Subject-Date
  • Emails: EMAIL-Surname-Subject-Date
  • Documents: LETTER-Surname-Subject-Date
  • Replies: REPLY-Surname-Subject-Date-DRAFT or FINAL (the final version may be ‘signed’ and then saved as a PDF)

Always exclude spaces in names, as these will be replaced by ‘%20’ in the URL path.

If one or more standard templates are required for replies:

  • Create the new Site Content Type in the Site Content Types area.
  • Enable the management of content types in the ‘Advanced’ section of the document library settings.
  • Add the new Site CT to the library
  • Add or edit the Word template to be used by clicking on the Content Type in the ‘Content Types’ section, then clicking on ‘Advanced settings’ and ‘upload’. The template can continue to be modified as required directly from this area and ensures consistency in replies.

The new Content Type will appear in the ‘New’ menu in the document library but not in the MS Teams ‘New’ dialogue (see below). This means that standard replies using a template can only (currently) be created via the SharePoint library. Drafts, however, could be created with a standard document template first (same with emails, see below).

StandardReplyCT
SharePoint New options

MS Team

The Team in MS Teams can be connected to the O365 Group (it’s a one to one relationship, you cannot connected multiple O365 Groups with a single MS Team).

The Team can have multiple channels, and the SPO document library can be presented in a channel. For example, the channel named ‘Correspondence 2019’ includes the ‘Correspondence 2019’ document library as a tab, as shown below.

MSTeams_CorroExample.JPG

The use of Teams means allows drafts to be co-authored and chats to take place about the correspondence and other matters.

If the SharePoint site includes an ‘incoming correspondence’ drop off library, the Team could have a channel with that library as a tab. The channel could then be used to review and decide on what to do with the correspondence.

Routing incoming correspondence for a reply

Once the correspondence is saved in a SharePoint document library, a decision must be taken by someone whether a reply is required.

  • If no reply is required, this can be reflected in the metadata (‘No reply required’).
  • If a reply is required, the correspondence must be ‘sent’ or otherwise made available (see below) to someone to draft a reply. This can be a simple or complex process.
    • In some cases, a standard reply may be possible. The SharePoint site should contain at least one library that contains examples of standard replies to certain types of, or common, questions.

Simple routing

The easiest way to ‘send’ someone the correspondence for action is to use the ‘Share’ option on the folder where the incoming correspondence is stored. As the same ‘Share’ option appears in File Explorer when the library is synced, the sharing process can also be managed from File Explorer.

This means that the recipient only has to click on the folder link in the email they receive to see the content of the folder. As they have access to the folder, they can then use the ‘New’ option to create a draft reply, including to an email.

FolderLinkSent.JPG

Once the draft has been finalised, it can also be sent via the ‘Share’ option. Alternatively, an alert on the library will notify anyone who needs to be notified that a change has been made to the library.

Routing using MS Flow

A more complex routing process may be required if the draft requires several steps, for example:

  • Send to someone to create the draft reply.
  • Draft reply gets sent to someone else for approval.
    • If not approved, goes back to the person who created it. (This can loop several times)
    • If approved, a message is sent to the person who needs to finalise it
  • Reply is finalised, metadata is updated, and reply sent.

All SharePoint Online libraries include an in-built Flow workflow ‘Request Sign Off’. When a document is selected and the ‘Request Sign Off’ option is selected the first time, the person must select the option to ‘Create Flow’. The ‘Run Flow’ dialogue then appears, requiring someone to be identified as the Approver and a Message to be included. The approver can be anyone in the organisation, for example the person’s manager.

 

RequestSignOffWho.JPG

The Approver receives an email, allowing them to approve or reject, and add a comment. If rejected, the ‘Sign Off Status’ column in the SharePoint library is updated to ‘Rejected’, and the sender receives a message to advice them that approval was rejected.

RequestSignOffEmail.JPG

If approved, the sender receives an email to notify them, and the ‘Sign off status’ column changes to ‘Approved’.

Once the reply has been approved, it can be finalised and sent to the person who wrote the correspondence. All versions of the draft reply are kept in the same folder, along with the final.

Email replies

Once approved, any email reply could be sent directly to the correspondent from the O365 Group’s mailbox.

Reporting

Metadata from the document library can be exported to Excel, or to business intelligence systems (or PowerBI) for analysis and reporting purposes.

Summing up

Correspondence can be managed in SharePoint, with MS Teams used to provide additional co-authoring and ‘chat’ options for the team, and MS Flow used for more complex approval requirements.

 

 

Posted in Compliance, Exchange Online, Governance, Information Management, Office 365, Office 365 Groups, OneDrive for Business, Records management, Retention and disposal, SharePoint Online

Managing the outcomes of records retention in Office 365

The retention of records in Exchange Online (EXO), SharePoint Online (SPO), OneDrive for Business (ODfB) and Office 365 (O365) groups can be achieved through the application of retention labels published in the O365 Security and Compliance admin portal.

This post describes:

  • How retention labels work (in summary), including the ‘per record’ rather than the container/aggregation retention model.
  • What happens to content in Office 365 when a retention period expires.
  • The options and actions that may influence the way retention labels/policies are configured, where and how they are applied, and the outcomes required.

The post highlights the need for information and records managers to be involved in all aspects of governance, site architecture and design, and decisions around specific settings and configuration, as well as being assigned specific roles, when Office 365 is implemented.

A quick summary of how O365 retention labels work

Records retention policies in O365 are based on ‘retention labels’ that are created in the O365 Security and Compliance admin portal under the ‘Classifications’ section. Multiple labels can be applied to a single policy.

  • Click this link to read Microsoft’s detailed guidance on retention labels.
  • Click this link to read Microsoft’s detailed guidance on retention policies.

Retention outcomes

Each retention label defines one of three potential outcomes at the end of the retention period, if retention is enabled, ‘keep forever’ is not selected, and the label is not used to classify the content as a record*:

  • The content will be automatically deleted. If the content is in SharePoint, it will first be sent to the Recycle Bin, from which it can be recovered within 90 days.
    • This option may be suitable for certain types of low value records.
  • A disposition review will be triggered to notify specific people. As with the previous point, SharePoint content will be sent to the Recycle Bin if a decision is made to delete it.
    • This option will require additional, human-intervention actions, as described below, if standard records management disposal review processes are followed.
  • Nothing.

Triggers

The date when the above action will occur is based on one of four triggers:

  • Date created
  • Date last modified
  • When labelled applied
  • A event. The ‘out of the box’ (OOTB) event types are:
    • Employee activity. (Processes related to hiring, performance and termination of an employee)
    • Expiration or termination of contracts and agreements.
    • Product lifetime. (Processes relating to last manufacturing date of products).
    • A new event can also be added.
    • See this post for Microsoft guidance on event-driven retention.

An additional alternative option is available: ‘Don’t retain the content, just delete it if it’s older than n days/months/years.’ This is similar to the automatic deletion option above and may be suitable for certain types of records. 

Declaring content as records

* The option to classify or ‘declare’ content as a record is not discussed further as relates to the way records are managed in the US. Microsoft’s guidance on labels notes that: ‘At a high level, records management means that: (a) Important content is classified as a record by users. (b) A record can’t be modified or deleted. (c) Records are finally disposed of after their stated lifetime is past.’ The standard on records management, ISO 15489, defines a record as ‘evidence of business activities, often (but not exclusively) in the form of a document or object, in any form’. This means that anything can be a record. The record may continue to be modified throughout its life. 

When do retention labels become active?

Retention labels become active only when they are published. As part of the publishing process, a decision must be made if the label will apply to all (a single option) or selected parts of the O365 ecosystem:

  • The Exchange Online (EXO) mailboxes of all or specific recipients, or excluding specific recipients.
  • All or specific SharePoint Online (SPO) sites, or excluding specific sites.
  • All or specific OneDrive for Business (ODfB) accounts, or excluding specific accounts.
  • All or specific O365 Groups, or excluding specific groups. Note that content in Microsoft Teams (MS Teams) is included in the O365 Groups options that include both the SharePoint content and email/Teams chat content.

Auto-applying retention labels

Both the retention label and policy sections include the ability to auto-apply a retention policy if certain conditions are met.

  • Sensitive information types. These are the same types that appear in the Data Loss Prevention (DLP) section, for example ‘Financial data’ or ‘Privacy data’.
  • Specific keywords.
  • Content types and metadata (E5 licences only). See this post by Joanne Klein for a description of these options.

The ability of the first two options to accurately identify content and apply a retention policy should be investigated before they are relied on.

When do retention policies start working

According to Microsoft’s guidance Overview of Retention Labels:

If you publish retention labels to SharePoint or OneDrive, it can take one day for those retention labels to appear for end users. In addition, if you publish retention labels to Exchange, it can take 7 days for those retention labels to appear for end users, and the mailbox needs to contain at least 10 MB of data.

  • In EXO, the default MRM policy needs to be removed before the new policy applies.
  • In ODfB, the policy is available to be manually applied on folders or documents. It does not automatically apply to content.
  • In SPO, the policy can be applied to document libraries or documents. To avoid removing the ability for users to legitimately need to delete documents in an active library it is recommended to apply the policy after the document library has ceased to be active.
  • Content in Office 365 Groups is covered by either the EXO (for email/teams chat content) or the SPO policy (applied to libraries).

Retention labels apply to individual records within aggregations

Records labels can be applied to aggregations of records (an entire email mailbox or folder, a SharePoint library or list, an ODfB account, O365 Groups) or individual records. However, the disposal process targets individual records (e.g., individual emails, single documents in SharePoint libraries, individual list items). 

That is, even when all the individual records are disposed of, the parent aggregation remains in place without any indication that the records previously stored in it (sometimes known as a ‘stub’) have been destroyed. 

This outcome has implications for the way the outcome of a retention label is set. It requires a choice between (a) delete automatically without review or (b) review before delete.

The latter option is made complicated by the requirement to review individual documents, including potentially in the original container (document library in SPO) and export metadata relating to those records if a record of the deletion is to be retained.

What happens when records reach the end of their retention period

As noted above, the outcome at the end of the retention period (trigger date + n days/months/years) will depend on the settings on the label.

  • Where the label was applied (EXO mailbox, SPO library or list, ODfB folder or document, O365 Group)
  • Whether the records would be deleted automatically or be subject to a disposition review.

If the records are to be deleted automatically:

  • SPO and ODfB records will be sent to the site/ODfB Recycle Bin for 90 days
  • EXO emails will be moved to a ‘Cleanup’ area for 14 days, before permanent deletion.
  • Aside from the audit logs (which by default only go back 90 days), no other record will be kept of the destroyed records.

If the records are subject to a disposition review, an email is sent to the person nominated. When that person clicks on the link in the email they are taken directly to the ‘Dispositions’ sub-section of the Records Management section of the O365 Security and Compliance centre.

It is arguable that retention policies with disposition review should not be applied to ODfB content as this will require the reviewer to review all the content that has been labelled by a user in their ODfB account.

  • For more information about this subject see this Microsoft page ‘Overview of disposition reviews‘. Microsoft note, on that page ‘To get access to the Disposition page, reviewers must be members of the Disposition Management role and the View-Only Audit Logs role. We recommend creating a new role group called Disposition Reviewers, adding these two roles to that role group, and then adding members to the role group.’

The dispositions dashboard shows the number of records that are pending disposition against each retention policy label:

O365_RM_DispositionsDashboard

Pending disposition tab

When the reviewer clicks on one of the retention policies listed, the following view opens for records ‘Pending disposition’:

O365_Dispositions_Pending

An important point to note here is that records are listed individually, not in logical aggregations or collections. It is possible however to use the Search option on the left to filter by author (emails) or SharePoint site and/or site library. It is also possible to export the details (which does not include any unique metadata applied to documents in SharePoint libraries).

All the records displayed may then be selected and a ‘Finalise decision’ dialogue box appears with the following options:

  • Dispose of the records.
  • Extend the retention.
  • Re-label the records.

Disposed items tab

The Dispositions dashboard includes a ‘Disposed items’ tab.

Microsoft note that this tab ‘… shows dispositions [that] were approved for deletion during a disposition review and are now in the process of being permanently deleted. Items that had a different retention label applied or their retention period extended as part of a review won’t appear here.’

Importantly, once records are permanently deleted, they no longer appear in the ‘Disposed Items’ tab. This means that no record will be kept of the records that were destroyed.

Shortcomings of the O365 dispositions/disposal model for records stored in SPO

Only individual records appear, not all the items in a document library

If the retention outcome is based on the ‘created’ or ‘last modified’ date, individual records in SPO document libraries will start to appear as soon as they reach the retention end date. The reviewer may need (or want) to view the original library, which they can identify from the link is in the dispositions review pane.

Retention policies prevent deletion

As a retention label prevents the deletion of content by users, and this may put them off using SharePoint, it is recommended that retention in SPO document libraries be based on when the label was applied NOT when it was created or last modified. This will help to ensure that all documents appear in the disposition review area at the same time.

Event based triggers may not be suitable for disposition review

If the retention outcome is based on an event, or is auto applied and a disposition review is required, those records will appear randomly when the event is triggered. It could be difficult for records managers to decide the disposal outcome in this way without referring back to the library.

The dispositions review pane does not display the original metadata

The  dispositions review pane displays only very basic metadata from the original library. Again, the reviewer may need to view the original library, export the metadata and store that in a secure location. Note that the exported metadata includes the URL of each original record including the library name.

The document library remains even when all contained records are destroyed

If the reviewer chooses to dispose of the records listed, only the content of the library (the individual documents stored in it) is deleted, not the actual library itself. No record (e.g., a ‘stub’ of the deleted item) is kept in the library of the deleted content.

The ‘Disposed items’ tab only shows records being destroyed

The ‘Disposed items’ tab only shows records in the process of being destroyed. It does not keep a record of what was destroyed. Records managers will need to retain the metadata of what was destroyed, when, based on what disposal authority, and with whose approval.

Dispositions really only provides a ‘heads up’ for further action

The Dispositions process may be instead used as a form of ‘heads up’ that records are starting to be due for disposal in a document library. This would allow the records managers (who should be Site Collection administrators) to review the library, export the complete set of metadata, and decide if the entire library can be deleted since it is no longer required.

Conclusions

Retention labels in O365 are an effective way of managing the retention and disposal of records in that environment, subject to the following points.

Email

Emails will likely continue to be managed as complete aggregations of records – the mailbox. Users cannot be expected to create logical groupings and apply individual retention labels to those records.

Organisational records policies may mandate specific timeframes for the retention of email (e.g., 1 year), while HR/IT security policies may mandate that whole mailboxes are retained for a period of time after employees leave. It is important to understand the difference between these two models

Options to automatically transfer emails to SharePoint document libraries via rules may be possible using Flow but these rely on individual users to set up.

Consideration should instead be given to using O365 Group mailboxes, rather than individual personal mailboxes, for specific work related matters. For example, ‘Customer Complaints’, or ‘XYZ Project’.

OneDrive for Business Accounts

ODfB accounts may be covered by two forms of retention:

  • Retention labels that apply to all ODfB accounts while the account is active. These must be manually applied by users.
  • A separate retention period set for ODfB accounts after a user leaves the organisation.

If there is a requirement to prevent the deletion of content by a user from their ODfB account, the better way to achieve this is using an eDiscovery case with Legal Hold applied.

SharePoint Online

As most records will be stored in SharePoint document libraries (including Office 365 Group-based SP libraries), multiple retention labels will be required to address different types of content or retention requirements.

Careful consideration should be given to whether records can be deleted automatically at the end of the retention period or should be subject to disposition review, noting that the automatic deletion provides no opportunity to capture the metadata of the records.

The ‘auto-apply’ or event-based retention option should be used sparingly to avoid a trickle of records for disposal – unless there is enough trust that these can be accurately marked and deleted without review.

Shortcomings in the disposition review process support the following decisions for SharePoint Online content:

  • The number of retention labels should be minimised to avoid a very long drop-down menu when a label is applied. If current record retention or disposal authorities contain a lot of classes, some of these could potential be combined into a single class (e.g., ‘Company Records – 7 years’), while the site name and document library name should provide some context to the content to ‘map’ back to the original classes.
  • Retention labels should be applied when document libraries (or lists) become inactive as this will avoid conflict with users who want to delete content and also ensure that documents are ready for disposition review at the same time.
  • Retention labels applied to SPO document libraries should include the disposition review option unless a ‘delete only’ label is considered suitable for certain document libraries that clearly contain working documents or Redundant, Outdated and Trivial (ROT) content.
  • Records managers should review the content of all or most original SPO document libraries, and export the metadata of those libraries for storage in a separate location (such as an ‘archives’ site), or in the original library with the retention label changed to ‘Never Delete’. The original document library can then be deleted.
Posted in Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

SharePoint Online – records management options and settings

This post summarises the primary records management options, settings and ideas that can be applied in SharePoint Online to manage records.

This post should be read as the second part of my previous post on the records management options and settings available in the Office 365 admin and security and compliance portals. Some of these settings will be referred to in this post.

The options and settings described in this post should ideally form part of your SharePoint governance documentation.

SharePoint Governance

We have already seen in the previous post that Office 365 Global Admins (GAs) have access to all parts of the Office 365 ecosystem. But they should rarely solely be responsible for SharePoint Online (SPO).

Some form of governance arrangement is necessary for SPO, especially if you plan to manage records in that application.

Some of the key considerations are as follows.

  • Who is responsible for ‘marketing’ or promoting SharePoint in the organisation, and making sure it is used correctly? The area responsible in IT for change management should probably take the lead on this as SPO is only one part of the O365 ecosystem. Records managers should have a role too, or be consulted.
  • SharePoint Administrator. You should already have a SharePoint Administrator and that person (or persons) is likely to be sitting in your IT department. Records managers will rarely also be SharePoint administrators; the two need to work closely together.
  • Who is responsible for training people to use SharePoint, especially to highlight the recordkeeping aspects of the application?
  • Who are the Site Collection Administrators? See next point.
  • Who are the Site Owners?
  • Who can create Office 365 Groups?

Answers to these questions should all be documented in your governance documentation.

SharePoint Online Admin Portal

SharePoint Online customised administrator

The SPO administrator role, a ‘customised administrator’ set in the Office 365 (O365) portal, should normally have a log on that is separate from that person’s O365 user log on. The SPO administrator account should not be a generic one (and generic accounts should generally be avoided).

The SPO administrator accesses the SPO admin portal from the Office 365 admin portal. They will also have access to the O365 Message Centre and Service Health sections.

SharePoint Online Architecture

Why a design model is good to have

Organisations should have some sort of design model for their SPO architecture. Most records will be kept in document libraries SharePoint team sites under the /teams path but some could also be under the /sites path.

The design model should include naming conventions for sites to avoid site names that have unknown acronyms or complex names. Site names form part of the total 400 characters allowed from https to the document suffix (e.g., .docx) so site names should ideally be no longer than around 16 characters. For example:

https://tenantname.sharepoint.com/teams/sitename

Records managers should be involved in designing this architecture model and could also be part of any approval process for new sites, to ensure the proposed names are suitable.

The names of SPO sites should generally map to business functions. Where the main function is very large (e.g., Financial Management is very large, you may decide to create sites based on the ‘sub-function’. That is, under the broader Financial Management (or simply ‘Finance’) site, you could have a separate site for Finance AP and another for Finance AR). These can be linked to a hub site that could be the ‘parent’ function site.

Don’t mix functions (such as personnel and IT) in the same site if only because this site is likely to become very large.

Try to aim for team site coverage of all business areas as all areas are likely to create or maintain records.

One relatively easy way to do this is to consult with the business area and understand how they use their current Network File Share location. This has the additional benefits of ‘mapping’ their SPO site to their existing NFS structure (generally or very specific) so it is familiar to them, and assisting with the migration of NFS to SPO later on.

Creating new Site Collections

Generally speaking there now are three types of SPO site:

  • A team site not linked to an O365 Group (but can be retrospectively linked)
  • A team site linked to an O365 Group
  • A communication site

Again, generally speaking:

  • SPO team sites (linked or not with O365 Groups) are the functional replacement for network file shares and, accordingly, contain most of the ‘document’ type records.
  • SPO communication sites are used for publishing purposes, including the intranet. They may contain documents in document libraries that, again, replace network file shares previously used for this purpose.

New sites can be created:

  • Directly from the SPO admin portal.
  • Via the ‘Create Site’ dialogue available in each user’s SharePoint portal, when this option is enabled. When this option is enabled, users can create either a Team site (linked with an O365 Group), a Communication site, or a ‘classic’ site (not linked with an O365 Group).
  • When a new Office 365 Group is created. This includes, if enabled, when a new Team in MS Teams is created, a Yammer group is enabled, or the person choose to create a new group from Outlook. If this option is allowed, whoever creates the O365 Group becomes the Site Collection Administrator and the SharePoint admin will be unable to access the site. For this reason, organisations that want to control their SPO environment may wish to limit who can create Office 365 Groups.
  • Via a PowerShell script.

SharePoint Sites

Site Collection Administrators (SCA)

Every SPO site has Site Collection Administrators. To ensure that records managers can access every site to manage records, it may be useful to add them to the membership of a Security Group that is in turn added to every site’s Site Collection Administrators after it is created.

Site Collection Administrators are added and managed in Advanced Permission Settings.

SPO_AdvancedPermissions

When you click on Site Collection Administrators, this dialogue appears:

SPO_SCAs.JPG

As noted above, if the ability to create O365 Groups is not controlled, the person who creates the O365 Group (as noted in the screenshot above) will become the SCA. The SharePoint administrator will be able to see the site in the SPO admin portal but may not be able to change the SCA settings. They may need to ask a Global Admin to do this.

Being a Site Owner only is not sufficient for records managers. Site Owners should be someone in the business area that ‘owns’ and will manage the SPO site on a day to day basis.

Site collection features – document IDs and Document Sets

Site collection features are only accessible to Site Collection Administrators. The list below expands as new features are activated; as can be seen, the ‘Document ID service’ feature has been enabled on this site. (Note, ‘Site features’ are activated from the Site Administration section, see below).

SPO_SiteCollectionAdminOptions.JPG

The Document ID feature is required for recordkeeping purposes as it assigns a unique Document ID to every object (including document sets but not folders) stored in a library.

If they are to be used in the site, the Document Set feature is also enabled in the Site collection features section.

SPO_DocIDDocSetSiteCollectionFeatures.JPG

After the Document ID service is enabled a new option appears in the Site Collection Administration section called ‘Document ID Settings’ (as noted above).

As can be seen in the screenshot below, all Document IDs begin with a unique set of up to 12 characters. Ideally, the Site name should be used as this will immediately give a clue to the site name on the document.

SPO_DocIDSettings.JPG

Document IDs take the form:

  • Prefix (e.g., ‘SITENAME’)
  • Library number. This is a unique and un-modifiable number of the library where the document is stored. It is not based on the library GUID.
  • Next sequential number.

If a document is deleted or moved from the library, the document ID (the sequential number) is not re-used.

Note that Document Sets use the same Document IDs. These cannot be separately modified.

Site collection features – Site Audit logs

The option ‘Site collection audit settings’ will already be visible in the Site Collection Administration section of all new sites, however (a) the options in the audit settings need to be enabled and (b) the ‘Reporting’ Site collection feature must be activated to enable the production of Site Audit Logs as required.

Note, the Site collection audit sections settings notes that ‘If you’d like to keep audit data for longer than this, please specify a document library where we can store audit reports before trimming occurs’. The default storage location is /_catalogs/MaintenanceLogs. However, the various options shown below must be selected for anything to be saved.

SPO_SiteCollectionAuditOptions.JPG

Enabling ‘Reporting’ results in a new section in the Site collection administration sections called ‘Audit log reports’. This section allows the Site Collection Administrators to create one-off audit logs for a range of activity on the site, going back 90 days.

  • Content Activity
    • Content viewing
    • Content modifications
    • Deletion
    • Content type and list modifications
  • Information Management
    • Policy modifications
    • Expiration and Disposition
  • Security and Site Settings
    • Auditing settings
    • Security settings
  • Custom
    • Run a custom report

The 90 day time period is the same as the O365 audit logs accessible from the Security and Compliance ‘Search’ section. If audit logs are required for longer periods, an add on may be required.

Metadata – Site columns or the Managed Metadata Service

The architecture model and/or business requirements may require the use of specific metadata across your environment. Metadata may be set in three ways.

Managed Metadata Service. This option is effective if you need to use the same metadata columns on multiple sites. Experience suggests that this option will be used selectively.

Site columns. These are in addition to the many columns that already exist by default on every site. This option is very effective if the same metadata needs to be used in multiple document libraries or lists on the same site. It is not accessible on any other site. In document libraries or lists, it must be added as an existing site column (i.e., not via the Create new column option).

Library columns. These columns are created in individual libraries or lists and are not accessible on any other library or list.

All new Site and Library columns have the following options:

SPO_NewColumnOptions.JPG

Each new column may be created in an existing or new group. They may also be (a) made mandatory and/or (b) enforce unique values. Note that making a Site Column mandatory and adding it to a document library will make the library read only in File Explorer if it is synced there.

Columns may have default values and may also include JSON formatting codes.

When Site columns are added to a document library, including via a Content Type (see next section), users may be required to fill in the required metadata (especially if it is mandatory).

Site Content Types

Site Content Types are a way to define metadata requirements for different types of documents, using Site columns. The default ‘document’ Content Type on every new SPO site is simply ‘Document’; all new document-based Content Types will be created using that one as a template.

Site Content  Types may also incorporate standard document templates (via the ‘Advanced section). These templates can be auto-populated using the library metadata. In any case, all metadata in a document library is added to any Office document as its metadata ‘payload’.

Once created, Site Content Types must be added to each individual library where they are to be used. To do this, the individual library must have the setting ‘Allow management of content types’ enabled in the ‘Advanced’ section of the document library settings.

SPO_DocLib_ContentTypesYesNo.JPG

When Content Types are enabled in this way, some other drop down features in the ‘+ New’ option on the library disappear, such as the ability to create Word, Excel or PowerPoint documents as can be seen below (the option on the right shows when Content Types are allowed).

Aggregations, containers, ‘files’ – Site Libraries

SharePoint document libraries are the container, aggregation or ‘file’ (if you will) in which records are stored. They are the functional replacement for network file shares. You may end up migrating from those NFS to SharePoint.

Naming conventions for new document libraries are useful to have but the extent to which you require people to follow them (if Site Owners create them) may differ between organisations.

Document libraries ideally should contain only a year of content; including the year in the library name is a good way to maintain year-based content, which in turn makes it easier to manage at the end of the record’s life.

Avoid using the generic ‘Documents’ library that comes with every new library because users will create folders with uncontrolled names and content.

All SPO document libraries and lists have default views of the metadata. These views can be modified as required (via the option on the top right of the menu bar) with a range of additional metadata that is by default hidden from view. Multiple views can be created; pre-defined views may sometimes be easier than expecting users to depend on searches.

Document libraries include all the usual and expected document management functions including check out/in, copy to or move to and versioning.

SPO_DocumentOptionsincCheckOut.JPG

Users with Contribute or Edit permissions can view and restore versions.

SPO_DocLib_VersionHistory.JPG

If there is a requirement to know who modified what part of a document, it is recommended to enable track changes on that document.

Note, with co-authoring now available, the last person to edit the document will create the last version.

Folders and document sets

Folders  should be seen as visual ‘dividers’ within a file, not as ‘hard-coded’ structures as they are in file shares.

Document sets can include additional metadata (including a document ID), making them suitable for use in breaking down a document library. However, for most of the time, folders are a more logical ‘divider’ for users.

Note that both document sets and folders look the same in a synced library.

Both folders and document sets can have unique permissions.

Create and capture records

One of the best reasons for using SharePoint is the ability to create a single source of truth. That is, a single record stored on a library that multiple people can access and work on at the same time.

Having a single source of truth avoids the requirement to (a) create a initial copy on a personal drive or network file share, (b) attach that copy to an email and send it to multiple people who are all likely to save it somewhere and also send back a changed version.

In SPO, users can create a new record directly within a document library (or in the synced library on a drive). Anyone with access to that library can access it; alternatively the document can be shared. Co-authoring means that anyone with edit access can edit the document. Every time it is edited and closed a new version is created.

If it is necessary to refer to the original from another library, the ‘Link’ option can be used.

Access controls and permissions

All SharePoint site contain three default permission groups. Individuals will usually be added to one of these groups only, depending on their access requirements:

  • Site Owner – Full Control across the site but cannot see the Site Collection Administration section (shown above). There will normally be only two to four Site Owners. Site Owners are responsible for managing their sites.
  • Site Member – Update and edit.
  • Site Visitor – Read only.

All content on a SharePoint site inherits the default permissions above however at any point the default permission inheritance can be broken and unique permissions applied. This is a manual process for document libraries (via Advanced Permissions) but automatically applied if a folder or document is shared with someone who is not in a default permission group.

Note, one of the leading support issues in SharePoint is understanding and unravelling complex permissions, especially when applied to individual documents that are placed ‘under’ folders with unique permissions, in libraries with unique permissions.

Retention and disposal

Generally speaking, a SPO site collection will consist of multiple libraries, each (ideally) containing content that is specific to the activity that it relates to. For example, a library for ‘Meetings’ and one for local forms. Consequently, the records stored in document libraries may require different retention.

If O365 Classification labels are used for retention, and depending on how these are configured, these must be applied per library; the individual documents stored in the library – not the entire library as such, are then governed by the retention requirement.

It is also possible to apply a policy to an entire site collection via site policies. This option will only be useful if the entire site can be subject to a single retention requirement – for example, inactive old sites that have a range of content all likely to be covered by the same retention period, or project sites.

Once retention policies are applied to a library, users cannot delete any content in the library so it may be prurient to apply them when the libraries are no longer used instead. Hopefully you will have implemented year-based libraries, which will facilitate this. Alternatively, the retention period trigger can start when the actual policy is applied.

It may be useful for the records manager to review the content of document libraries, and perhaps export the metadata of the library, before the content is disposed of via the O365 Security and Compliance area as any unique metadata is not visible in the Dispositions area.

When records are due to be disposed, an email is automatically send to whoever is in the Records Management role in the O365 Security and Compliance admin portal. The activity of reviewing and approving disposals happens in the O365 Security and Admin portal.

It may be useful to set up an ‘Archives’ SPO site to keep records of all disposal activities, including metadata from document libraries.

Note the library will remain even after the documents are destroyed. An alternative and perhaps better disposal model would be to use the notifications to alert the records manager to the records due for disposal; the records manager may then export and save the metadata in a SharePoint archives site, and then delete the library entirely.

Note that the retention of records in Exchange Online mailboxes and OneDrive may be managed differently by the organisation.

Minimising duplication of content

SharePoint allows organisations to have a single source of truth, to avoid the duplication of using NFS and then uploading to a document management system.

Users can create the record within a SharePoint library, upload it there, use the ‘save as’ option (where you will see all your SPO sites to choose from).

The ability to share with external users (when this is enabled) also helps to reduce duplication and email attachments.

Hybrid records

As noted above, links can be created in any SPO document library to point to resources in a different location. If paper records are managed in a SPO list, the document library can include a link to that SPO list.

Syncing document libraries to File Explorer

Users with an O365 licence and Windows 10 may use the ‘sync’ option available on the ribbon menu of every library. This option syncs the document library to the user’s File Explorer from where they may continue to access and work on the documents.

Note, as discussed in this post, if there is any mandatory metadata in the library, the synced library will become read only.

End users like using the sync option as, although it doesn’t (yet) display any unique metadata on the library, it allows them to work the way they have always worked and they get the added bonus of being able to do it on any device.

eDiscovery

eDiscovery cases are created in the O365 Security and Compliance portal. Essentially, an eDiscovery case uses search and other options to find records. Once found, these records can be placed on Legal Hold, which prevents their disposal.

If a document library has no retention label applied, and all or some of the content is identified as part of the eDiscovery case with a Legal Hold, and a user deletes a record, that record remains in a hidden library but still visible to the eDiscovery case manager. Once the Legal Hold is lifted, the record will resume the 90 day deletion process after which it will no longer be available.

Search

Search in SPO, and across all of Office 365, is very powerful. A single click in the Search box in the user’s SPO portal will result in suggestions before anything is entered.

Searches will return anything the user has access to. The access limits plus the Artificial Intelligence (AI) engine will return different search results for different users.

Users may also take advantage of the Office Graph-powered Delve (E3 licences and above) or the Discovery option in OneDrive to see information that may be of interest to the user. This works on the basis of the various ‘signals’ between users and objects, as depicted in the graphic below.

microsoft-graph_hero-image.png

 

Posted in Electronic records, Exchange Online, Governance, Information Management, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Office 365 admin and Security and Compliance portals – records management options and settings

If you plan, or want to understand how, to manage records ‘out of the box’ in the Office 365 ecosystem including in SharePoint Online, Exchange Online and MS Teams, you will need to know the available options and settings. These would normally be set by the Office 365 Global Admins (GAs) or, in some cases, devolved to Customised Administrators. GAs have access to all parts of the Office 365 environment including SharePoint Online, Exchange, OneDrive and Microsoft Teams.

See the next post for a list of the options and settings available in SharePoint Online to manage records.

Note, the description below is for a typical E3 licenced level organisation. E5 licences provide additional capability some of which is referenced below with a comment.

Office 365 admin portal options and settings

The options and settings in the Office 365 admin portal required to manage records are listed below.

Customised administrator

In addition to the GAs, the Office 365 admin portal is where customised administrators are set up. Typically these admins will have log ons that are different from their normal user log on and will not need the full range of licence options. The SharePoint Admin role is a customised administrator.

Records managers could potentially be SharePoint Admins if they are suitably skilled. Otherwise, at the very least they should be Site Collection Administrators and work closely with the SharePoint Admins to ensure that SharePoint Online (SPO) is configured correctly.

Office 365 Groups

Records managers need to understand how Office 365 Groups work.

Most people know that Distribution Lists (DL) are used to send emails to multiple people. However, DLs cannot be used to control access to IT resources; this is achieved by using Security Groups (SG). SGs, on the other hand, are not email enabled.

Office 365 (O365) Groups are ‘kind of’ a mix of DG and SG functionality in that they can be used to control access to certain resources in Office 365 (including SPO) AND they can be used to contact all members of the Group.

O365_AllGroups
Image source: https://techcommunity.microsoft.com/t5/Microsoft-Teams/NEW-Teams-IT-architecture-posters-published/m-p/480928#M30306

But O365 Groups are much more. They are in many respects central to Office 365.

  • Every new O365 Group creates a SharePoint site (this is not optional).
  • If the creation of O365 Groups is not controlled, every new Team in MS Teams creates an O365 Group that in turn creates a SPO site.
  • If you use Yammer, every new Yammer group also creates an O365 Group that creates a SPO site.
  • Again, if not controlled, any user can create a new O365 Group from Outlook.

In short, you need to either allow their creation and expect to see multiple uncontrolled SPO sites, or control their creation. There is no middle path.

Additionally, if the creation of O365 Groups is not controlled, the Owners of the new O365 Group (usually the person who created it and anyone else they invite) will become the Site Collection Administrators, locking the SharePoint Admins out of the site. They will need to call on the O365 GAs to give them access to the site.

External Sharing for SharePoint and O365 Groups

Although it relates more to security, external sharing is a option and setting that may require input from the information or records management area. External sharing is initially enabled in the O365 Admin portal in the Settings – Services and Add-ins section.

O365_Admin_SPOExternalSharing

Note, even if this setting is enabled, SPO sites don’t have this enabled by default. The setting is controlled from the SharePoint Admin portal.

External access for Office 365 Groups is set in the following setting:

O365_O365GroupsExternalSharing.JPG

Office 365 Security and Compliance admin portal options and settings

The options and settings in the Office 365 Security and Compliance admin portal required to manage records are listed below.

Permissions – Roles – Records Management (and others)

The Security and Compliance admin centre includes several roles in the ‘Permissions’ section that may be required by records and/or information management staff, especially to establish records retention schedules, manage dispositions, check audit logs and manage eDiscovery cases and legal holds.

Classification – Labels (Records Retention labels)

Records retention policies in O365 are set in the O365 Security and Compliance Portal in the Classifications section. These retention policies may be applied across SPO, Exchange Online, Teams.

Some thought needs to go into this including potentially grouping policies that have the same retention requirement (e.g., 7 years), or using the File Plan (see below) and other options now available to group them. This requires records management input.

O365_FilePlanDescriptors

Classification policies used for records retention will be applied across all of the O365 environment, not just SPO. However, your IT department may want to implement different rules for Exchange (e.g., using the default MRM policy to keep all emails ‘forever’) or OneDrive (e.g., a 7 year retention for everyone’s content after they leave).

Click this link for more details about Retention Policies.

Records Management – Dispositions

The O365 Security and Compliance Centre includes a ‘Records Management’ section that has three options: File Plan, Events, Dispositions. Records Managers should have access to these areas; this is achieved by them having the ‘Records Management Role’ in the ‘Permissions’ section.

The ‘File Plan’ section displays a list of retention policies (labels) with any details added to the ‘File Plan’ section (shown above), thereby providing the records manager with a view of all labels and any added details, for example by numbering, citation and so on.

The ‘Events’ section shows any events that have been defined for use in retention policies.

The Dispositions section has two parts, a basic dashboard that shows all retention policies and the number of records covered by those policies:

O365_RM_DispositionsDashboard

If the records manager clicks on any of the policies it displays the records due for disposal and provides the various options for disposal. It also shows records that have been disposed on a separate tab.

O365_Dispositions_Pending

Search

The search section has two options: Content search, and Audit log Search. Access to both may be controlled but records managers may need to have the ability to ask for information from either from the GAs.

eDiscovery

The eDiscovery section is where eDiscovery cases are established. Cases are a form of content search that, once completed, puts any retention policies on hold (legal hold) under the case has been removed.

eDiscovery cases may includes searches across all of Office 365 (Exchange email, O365 Group email, Teams messages, To-Do, Sway, Forms, SPO, OneDrive, O365 Group SPO sites, Teams sites, Exchange public folders) or selected parts only. They may also be used to search mailboxes for specific individuals or selected SPO sites.

Governance

All of the above (and all other settings) should form part of a governance document that details the O365 environment. Settings should only be changed with agreement of everyone in a governance team.

Posted in Compliance, Exchange Online, Office 365, Office 365 Groups, Products and applications, Records management, Retention and disposal

Managing the retention and disposal of emails in Office 365

In a recent blog post (https://thinkingrecords.co.uk/2019/02/14/managing-email-in-office-365/), James Lappin provided a good overview of the direction that Microsoft have gone with retention and disposal in Office 365.

A key point with almost anything to do with SharePoint Online (that differentiates it from on-premise) is that SharePoint Online (and its ‘personal’ end-user service, OneDrive for Business) is just one element of the Office 365 ecosystem. That is, you can no longer really regard SharePoint as a standalone service that can be managed independently of the other services you may or may not decide to use.

For example:

  • Office 365 Groups (which are an Exchange object, similar to Distribution Lists and Security Groups) all have an associated SharePoint site. O365 Groups are in many way at the ‘heart’ of the Office 365 security/permission model. You cannot create an O365 Group without a SharePoint site.
  • Teams in MIcrosoft Teams (yes the duplication of wording is unfortunate) create an Office 365 Group (which in turn creates a SharePoint site). Alternatively, you can create an Office 365 Group (with a SharePoint site) and link that Group to the new Team. So, Teams in Microsoft Teams have their own Team site.
  • If you enable the ‘Create Site’ option in the end-user SharePoint portal, and the user selects ‘Team site’, this creates an Office 365 Group also.
  • If you allow anyone to create an Office 365 Group, then any new Yammer group creates an Office 365 Groups and – yes, you got it right – a SharePoint.
  • Retention policies for Exchange and SharePoint are set as ‘classification policies’ in the Office 365 Security and Compliance admin portal. This, by the way, is also where you set the new Information Security policies that have only recently appeared. They are both a type of label.

It can be quite overwhelming at first, but the key point is that you cannot regard SharePoint as an isolation application any more. However, most IT shops are pretty ‘hardened’ to the idea that the Exchange ‘box’ (the server) and the SharePoint box are managed by different teams, and one challenge in the new Office 365 world may be to convince the Exchange admins that they should be friends with the SharePoint admins AND the records team.

Backing up as a retention option

It is important to understand that IT departments often regarding ‘backing up’ as a form of retention (or ‘archiving’). Your IT department will almost always have a back-up regime for its on-premises servers.

However, you cannot (easily, cost efficiently) back up SharePoint Online or Exchange Online like you could back up your on-premises environments, but there are many vendors in the market who will offer you a solution to this.

Most IT shops consider back ups to be an archive from which they can retrieve content, a kind of alternative records retention regime. This factor may impact on any decisions that may need to be made with retention policies applied to both Exchange Online and SharePoint Online.

The problem with applying retention and disposal policies to email

It almost goes without saying that, while retention policies can be applied to Exchange Online, typically (a) the content is structured (in multiple folders) differently by every person and (b) the content is mixed together so no retention policy can normally apply to all emails in a single folder.

It is why, generally speaking, we ask users to copy emails into SharePoint (or other EDRM) containers or aggregations (document libraries, files), to keep the content in context.

But in most cases the content (the emails) still remains in Exchange too.

Challenges when applying retention and disposal actions to emails

There are several challenges for the application of records retention and disposal policies in Exchange/Outlook.

  • Do you have a blanket approach to all email, disallowing the deletion of any email for say 7 years?
  • Or do you apply a much shorter retention policy to all emails (say 12 months or less)? (Cue – ‘but what if I want to get my email back after 5 years’ from a user with a labyrinthine email folder structure)
  • Do you rely on users to copy emails to SharePoint or other EDRM containers where they will be stored in context?

The core problem with email is that it’s personal to each user. While it may be good to be able to apply a retention policy to emails, my sense is that anything that is optional will almost always fail to be taken up.

Having a single retention policy (e.g., 7 years) applied to the email accounts of departed users may be a good option (similar to the same policy applied to the OneDrive accounts of departed users).

Another newish option is to use the new Microsoft Flow options to automatically move emails and/or attachments to SharePoint document libraries.

Every organisation is likely to be different and all options need to be considered, understood and then applied – along with the question: ‘Do we (really) need to back this up’?

Posted in Access controls, Electronic records, Office 365, Office 365 Groups, OneDrive for Business, Products and applications, SharePoint 2013, SharePoint Online

Migrating to SharePoint Online Part 2 (implementation)

In my previous post I described what we did to prepare for the migration of our SharePoint 2013 (SP2013) environment to SharePoint Online (SPO). In this post I describe the process we undertook and the lessons we learned along the way.

By August 2017 we had around 245 SP2013 sites across seven web applications: apps (13 ‘purpose-built’ sites); intranet (1 site); ipform (an old site that was closed several years back); projects; publication (30 sites); sptest (used for testing sites); and team.

The bulk of our sites (around 210 sites split almost equally) containing most of our corporate records were in either the teams and projects web applications.

The details of our root sites were recorded in several key artefacts:

  • A SharePoint Online list in our SharePoint Admin site, used for new site requests. This was always our ‘master’ listing of sites and included a range of additional metadata, including the business owner and a ‘yes/no’ if the site had been migrated or not. This was one of the first sites we migrated.
  • Another SharePoint list that recorded the details of site collections and subsites.
  • An Excel spreadsheet used to ‘map’ of all our root sites (one per cell_ grouped under business areas. This map provided a simple, printable visual map of all our SP2013 sites grouped by business area. We used colours to indicate when sites were migrated to SPO, providing an immediate visual reference.

Configuring and learning about Office 365 Admin and SharePoint Online (and OneDrive) admin

By August 2017 we had access to our Office 365 tenant admin environment, access to which is required to get to the SharePoint Admin portal initially (subsequently, the SharePoint Service Administrator could access it directly).

After setting up and configuring the Office 365 elements and SharePoint Online (SPO) environment, we created some initial test sites (via the Admin portal) to understand the new environment.

One of the early SPO sites was a re-created SharePoint User Group site, used to store general training and other useful information about the new environment. This site has remained our primary go-to point for all SharePoint users.

Our SharePoint developer also re-created various scripts, including scripts to automatically create and configure new sites from a request in a SharePoint Online list.

Monitoring changes in Office 365 and SharePoint Online

We learned the importance of monitoring – daily – the Office 365 message centre and also the Microsoft techcommunity site (which had been moved off a Yammer platform a year or so earlier), as well as the Microsoft Office 365 roadmap to ensure we were aware of any likely changes – many of which were introduced during the time we were migrating.

We quickly learned a few things:

  • Customisation was not a friend of migration. Fortunately, almost none of our sites were customised. However, page-based content would need to be re-created.
  • Any complex workflows, integration or data extraction (e.g., ETL for business intelligence purposes which needed to be re-linked) could delay migrations. As it turned out, these sites ended up at the very end of the migration process and a couple were still waiting for migration as at the date of this post.
  • New ‘modern’ sites based on Office 365 Groups needed careful planning to get them right early on. We decided that any request for an Office 365 Group would go through the same request process as SharePoint requests.

Towards the end of 2018, when they were introduced, we also learned that hub sites were preferred over subsites.

Project management

While the migration of SharePoint sites was included as part of an internal IT project, that project was focussed for most of the first year on a range of other core networking elements including the broader network architecture model and high level designs required for our new cloud environment.

It would only later start work on the migration of Exchange mailboxes to Exchange Online and personal drives to OneDrive.

SharePoint migrations continued through the life of the project.

Migration tool

There were a number of options to migrate our SP2013 sites to SPO. We decided against going with an external provider for a number of reasons and instead – after reviewing the market – acquired the ShareGate migration tool in September 2017.

Final architecture

By September 2017 we had finalised the architecture for our SPO environment. As we had been using web applications in our on-premise environment we needed to ‘map’ this to the new environment.

The new model was based on the following:

  • All team and project sites would be created under the ‘/teams’ path. Project sites would have the prefix ‘PRJ’ to identify them. (Some would also be created as Office 365 Group-based sites, with ‘O365_PRJ’ as the prefix). ‘Team’ sites had to be based on a logical business unit or team.
  • All other sites, including communication sites and sites that crossed over multiple business areas would be created under ‘/sites/’.

SharePoint migrations were ready to go.

First batch

Our earlier analysis indicated that around 50 of our SP2013 project sites were inactive (because the projects had since closed). As no-one was accessing any of these sites we decided to use the ShareGate tool to test the migration process and learn from the experience.

We migrated 51 project sites in October 2017. The migrations initially took place during the day but we then changed to an early morning migration (before 9 AM usually) to avoid any network traffic issues.

First lessons learned

The ShareGate tool worked as expected, and proved to be a very useful tool for other reasons too, such as moving libraries and lists between sites.

The early batch of SP2013 sites were migrated ‘as is’ in terms of their look and feel. They looked exactly the same but were now in SPO. That is, they did not get the new ‘modern’ page look and were not mobile friendly. This didn’t matter too much as the sites were rarely accessed.

After the first batch, we did the following for all new sites:

  • Added a new page (named ‘home2’) and swapped over the old ‘classic’ page (renamed to ‘homex’) with the new one.
  • Edited the replacement home page and add any text/images from the old site home page.
  • Fixed up the left hand navigation; indented libraries and lists on old sites were now under a heading with a drop down menu option. In some cases we left them ‘as is’, in other cases we promoted the indented libraries via the left hand ‘Edit’ option.

For any sites that had the publishing feature enabled on the site collection and site settings, we disabled this on the SPO site post-migration as there was generally no need to have these settings enabled.

Some sites had Active Directory security groups in their SharePoint permission groups. As these were not migrated to our Office 365 environment (for multiple reasons, including the complexity of this legacy environment), these had to be added back in a different way to provide the same level of access. In almost all cases, existing SP permission groups (Owner, Member, Visitor) were sufficient. The primary one we had to re-create was the AD Group for ‘everyone’; this was replaced by the ‘Everyone except external users’ option.

The other factor we had to consider were Office 365 licences. By October 2017 few people had these licences. Anyone who needed to access SharePoint would need a licence, but these might not be issued to everyone until mid 2018. This limited the number of sites that we could migrate. By mid 2018, more or less anyone who accessed SharePoint 2013 before could now access the SPO environment.

Next batch – to end June 2018

From November 2017 to end June 2018 we migrated another 57 sites, including a further 16 inactive project sites in December 2017. The primary reason for the low number was (as noted above) the need for staff to have Office 365 licences, which were not allocated more broadly until mid 2018.

At the same time, however, we were also starting to create a range of new SPO sites, including Office 365 Group-based sites and new communication sites.

Publication sites re-created as communication sites

Almost none of our publication sites could be migrated ‘as is’ to SharePoint Online because the page-based content, while it could be migrated, was not in modern pages or mobile friendly.

Accordingly, it was decided to re-create all the publication sites as SPO communication sites. In almost all cases this was a relatively simple process of creating pages and copying content from the old to the new.

Our intranet was the only except to this process and, as of end 2018, remains as a SP2013 site because of heavy customisation.

Other project impacts

From July 2018 two key projects impacted on the SharePoint migrations.

The first was the roll-out of new Windows 10 devices. While a Windows 10 device was not required to access SharePoint, some of our older Windows 7 devices had Internet Explorer 9 that had issues with SPO. These users were asked to use Chrome instead.

The second related project work (which was part of the overall Office 365 project that included SharePoint migrations) was the migration of Exchange mailboxes and personal drives to Exchange Online and OneDrive respectively. This part of the project encountered a problem with Windows 7 devices and as a result that part of the project activity was delayed.

Final migrations – from July 2018

From July to the end of 2018 we migrated all except around 10 of the remaining sites, at an average rate of around 20 per month. Many of these were simple migrations.

For each migration we followed the same process:

  • Engaged with the business area to provide awareness of, and where required, training in, the new environment. (By the end of 2018, this training included information about Office 365 and MS Teams to help site owners understand that SharePoint is not an ‘isolated’ product as it was before, but part of a much larger ecosystem)
  • Identified a suitable date and time to migrate the site (most of these were completed before 8 AM on a working day, but some were done over a weekend)
  • Alerted our Service Desk to the proposed change
  • Migrated the site
  • Made minor changes to the site (home page swaps mostly)
  • Made the old site read only with a pointer to the new site
  • ‘Released’ the migrated site to the business area, usually before 9 AM.
  • Provide post-migration support to the business area. In most cases the business area was able to use the new site immediately as the new site was very similar to the old one in look and layout.

The last sites to migrate included sites with complex workflows, integration or ETIL elements and several large, complex and sensitive sites.

New site request process

While we had always had a SharePoint-based site request form, the new environment meant that we needed an updated form. The (SPO) online form changed several times during 2018 as we learned from experience what worked and what didn’t.

The current form captures the following (not all columns/fields are listed):

  • Site Acronym (up to 12 characters – becomes the DocID prefix)
  • Site Type: (a) team or project site (no Office 365 Group), (b) team or project site (with Office 365 Group, (c) communication site
  • Approver
  • Business area owner
  • Owner/s
  • Member/s
  • Sensitivity (information security)
  • Suggested site URL

Each of these requests was reviewed by the SharePoint administrators who, if the site request is OK, then run a workflow for non Office 365 Group-based sites only. For Office 365 Group based sites, these were created by creating the Office 365 Group.

By the end of 2018 we had approximately the same number of migrated sites as new sites and our SPO environment was ‘live’ and active.

Almost all the old SP2013 sites were made read only. We expect that these will be deleted by June 2019.

Posted in Electronic records, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, Records management, SharePoint Online

Is Microsoft Teams the future for office communications?

At a recent presentation on Office 365, the presenter started with Microsoft Teams and spent the next half hour or so demonstrating how it, not Outlook, had become the centre of his daily life. He didn’t mention the connection with Office 365 Groups until asked.

Is Microsoft Teams the future of office communications, replacing Outlook?

Teams was introduced to the Office 365 environment in late 2016. (See this video). At the time, it was described as ‘a true chat-based hub for teamwork and give customers the opportunity to create a more open, fluid, and digital environment.’ (https://docs.microsoft.com/en-us/microsoftteams/teams-overview)

Many early reviews suggested that Teams was Microsoft’s response to Slack, but this comparison is simplistic. Teams has much more functionality than Slack.

How do Teams link into the Office 365 environment?

Teams is not an isolated application in the Office 365 (O365) environment. It has direct links with O365 Groups.

This means that, unless your organisation controls the creation of O365 Groups, every new Team will create a new O365 Group – which in turn creates a Group mailbox and calendar, a SharePoint site, and a Planner.

If your organisation controls Group creation (which is not a bad idea), a Team cannot be created by users using the ‘Create Team’ option.

Instead, whoever controls the creation of Groups (ideally a defined Admin role) can create a Team through the ‘Create Team’ option or, preferably, by linking an existing Group to a new Team. That is, a Team is created (from the Teams interface) with the same name as the O365 Group.

The linkage with O365 Groups is important to understand. Both the Exchange and/or SharePoint Administrators should have a role in the creation of both O365 Groups, SharePoint sites and Teams in environments where this is controlled.

Where Group, Team and SharePoint site creation is not controlled, there is potential for their proliferation. There is some debate as to which is the best option but my own recommendation is to maintain controls, at least as the new Office 365 environment is being rolled out. Otherwise, the SharePoint Admin may have to deal with a plethora of similarly or poorly named SharePoint sites, and the Exchange Admin will also have a job on their hands.

The Outlook paradigm – 30 years of poorly managed records

Almost every office worker for the past 30 years has used Outlook as the primary communication medium, using folders to categorise content. Distribution Lists (DLs) helped to provide a way to communicate (in a single direction) with a known group of users.

The primary way to share a document in the Outlook environment was been to attach it to a new email. Email attachments may be left in Outlook and/or saved to a drive somewhere. Multiple copies probably exist.

Organisations that have deployed SharePoint over the last decade have learned that links in emails to documents are a much more effective way of controlling document versions and reducing copies, but this is a hard change for many users to accept.

The idea that there can be one version of a document in a globally accessible location seems counter-intuitive to users who prefer to squirrel information away in ‘personal’ email or network drive folders.

The rise of social networking and messaging

A range of social network applications, including MySpace, began to appear from the early 2000s (Facebook was open generally from September 2006). Originally browser-based, the general popularity of these applications took off once smartphones included those apps.

It wasn’t long before messaging apps such as Yahoo Messenger started to replace SMS messaging as the default way to communicate with others via phones.

Social networking and messaging apps began to change the way we communicated and connected and began to move personal communications away from email. Instead of emailing each other photos, we could now share them in a single location for all of our friends to view, like and comment.

Email has persisted, however, as the primary ‘formal’ way to communicate.

Probably the main reason for this was its recognition and persistence as a ‘record’ – many document and records management systems integrated with email systems, allowing emails to be captured as records.

Instant messaging, on the other hand, remained largely (and artificially) outside the formally accepted recordkeeping world despite the efforts of records managers to try to capture all this ephemeral content.

Enter Microsoft Teams

Microsoft Teams is an interesting technology from a social change point of view, and one that Microsoft seems to believe will be a game changer for business communications.

To understand Teams, it is important to understand what it’s not. It’s not ‘just’ an alternative to Slack. It’s not ‘just’ a replacement for Skype for Business. It’s not ‘just’ a messaging app. It’s a new way to connect, communicate, and collaborate any device.

Teams:

  • Is accessible on almost any device or browser.
  • Includes 1:1 messaging and group messaging.
  • Includes a range of emojis and gifs.
  • Includes voice and video calling.
  • Has its own Office 365 Group (which has its own mailbox in Outlook).
  • Has an email address for anyone who still prefers to use email to connect.
  • Has its own dedicated (O365 Group) SharePoint site.
  • Allows (and in fact encourages) users to share and work on a document at the same time in the Teams interface (rather than attaching it).
  • Allows a team to communicate in multiple channels.
  • Has cool ‘toast’ notifications.
  • Includes a range of connectors to other services.
  • Allows a user to see where other people fit into the organisation.
  • Saves all the chat content to a hidden folder in the associated Group’s mailbox.
  • Allows external (guest) access.

The Teams interface is, in fact, so useful, that some users might find it more useful than Outlook. If you use it for long enough, you may soon find yourself checking Teams instead of Outlook. In fact, Outlook looks a bit dated by comparison.

Is the end near for email?

I don’t think so, at least not for a few years.

Email is a heavily ingrained way of communicating for many people and is still seen as the ‘official’ communication medium for many organisations (having replaced the old paper Memo or Minute).

But, just as Facebook and Instragram (and other applications) replaced email because they were a more effecient and effective way for people to keep in touch (despite all the security issues), Teams – or its natural successor – has the potential to move a lot of communication traffic (and attachments) away from Outlook.

This change has already happened in part. Many (if not most) people – including government officials (allegedly) – already use a range of ‘unofficial’ applications such as Whatsapp, Facebook Messenger, Signal and so on, for both personal and professional use. The use of email is, slowly, being eroded in favour of more instant ways to communicate and share information.

Why? Because it’s faster and easier to use and meets the new paradigm of limited attention spans and interest in reading long sentences (TL;DR).

Is Microsoft really the game changer?

Perhaps, but it may not be the only one.

It is a relatively new app, and one that will probably get a lot of traction with lots of marketing by Microsoft, its inclusion in O365 licences, and the very recent ability to connect with external ‘guests’.

Whether users will use its full, Team-based collaboration functionality or remain more a Skype-replacement will remain to be seen. But for now, Outlook is looking like an ‘old’ person’s way to communicate.

Learn more here:

https://www.microsoft.com/en-us/education/products/teams/default.aspx