Posted in Archiving third party content, Connectors, Conservation and preservation, Electronic records, Information Management, Microsoft 365, Microsoft Graph, Records management, Retention and disposal, Solutions

Using Microsoft 365 connectors to support records management

Microsoft 365 includes a range of connectors, in three categories, that can be used to support the management of records created by other applications. The three categories are:

  • Search connectors, that find content created by and/or stored in a range of internal and external applications, including social media.
  • Archive connectors, that import and archive content created by third-party applications.
  • API connectors, that support business processes such as capturing email attachments.

This post how these connectors can assist with the management of records.

The recordkeeping dilemma

Finding, capturing and managing records across an ever increasing volume of digital content and content types has been one of the biggest challenges for recordkeeping since the early 2000s.

The primary method of managing digital records for most of the past 20 years has been to require digital records (mostly emails and other digital content created on file shares) to be saved to or stored in an electronic document and records management system (EDRMS). The EDRMS was established as ‘the’ recordkeeping system for the organisation.

EDRM systems were also used to manage paper records which, over the past 20 years, have mostly contained the printed version of born-digital records that remain stored in the systems where they were created or captured.

There were two fundamental flaws in the EDRMS model. The first was an expectation that end-users would be willing to save digital records to the EDRMS. The second was that the original digital record remained in place where it was created or captured, usually ignored but often the source of rich pickings for eDiscovery.

The introduction of web-based email and document storage systems, smart phones, social media and personal messaging applications from around 2005 (in addition to already existing text messaging/SMS messages) further challenged the concept of a centralised recordkeeping system; in many cases, the only option to save these records was to print and scan, screenshot and save the image, or save to PDF, none of which were particularly effective in capturing the full set of records.

The hasty introduction from early 2020 of ‘work from home’ applications such as Zoom and Microsoft Teams has been a further blow to these methods.

In place records management

To the chagrin of records managers around the world, Microsoft never made it easy to save an email from Outlook to another system. Emails stubbornly remained stored in Exchange mailboxes with no sign of integration with file shares.

And for good reason – they have a different purpose and architecture to support that purpose. It would be similar to asking when it would be possible to create and send an email in Word.

The introduction of Office 365 (later Microsoft 365) from the mid 2010s changed the paradigm from a centralised model – where records were all copied to a central location and the originals left where they were created or captured, to a de-centralised or ‘in place’ model – where records are mostly left where they were created or captured.

The decentralised model does not exclude the ability to store copies of some records (e.g., emails) in other applications (e.g., SharePoint document libraries), but these are exceptions to the general rule.

It also does not exclude the ability to import or migrate content from third-party applications where necessary for recordkeeping purposes.

Microsoft 365 connectors

Microsoft 365 includes a wide range of options to connect with both internal and external systems. Many of these connectors simplify business processes and support integration models.

Connectors may also be used to support recordkeeping requirements, in three broad categories.

The three connectors

Archive connectors

Archive connectors allow organisations to import and archive data from third-party systems such as social media, instant messaging and document collaboration* platforms. Most of this data will be stored in Exchange mailboxes, where it can be subject to retention policies, eDiscovery and legal holds.

(*This option is still limited via connectors, but also see below under Search).

The social media and instant messaging data that can be archived in this way currently includes Facebook (business pages), LinkedIn company page data, Twitter, Webex Teams, Webpages, WhatsApp, Workplace from Facebook, Zoom Meetings. For the full listing, and a detailed description of what is required to connect each service, see this Microsoft description ‘Archive third-party data‘.

An important thing to keep in mind is that the data will be archived to an Exchange mailbox; this will require an account to be created for the purpose. Any data archived ot the mailbox will contribute to the overall storage quotas.

Search connectors

Search connectors (also known as Microsoft Graph connectors) index third-party data that then appears in Microsoft search results, including via Bing (the ‘Work’ tab), from http://www.office.com, and via SharePoint Online.

Most ECM/EDRM systems are listed, which means that organisations that continue to use those systems can allow end-users to find content from a single search point, only surfacing content that users are permitted to see.

The following is an example of what a Bing search looks like in the ‘Work’ tab (when enabled).

Example Bing search showing the Work tab

Note: as at 17 November 2020, Microsoft’s page ‘Overview of Microsoft Graph connectors‘ (which includes a very helpful architecture diagram) states that these are ‘currently in preview status available for tenants in Targeted release.’

There are two main types of search connector:

  • Microsoft built: Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL, Enterprise websites, MediaWiki, Microsoft SQL, and ServiceNow.
  • Partner built. Includes the following on-premise and online document management/ECM/EDRM connectors – Alfresco, Alfresco Content Services, Box, Confluence, Documentum, Facebook Workplace, File Share (on prem), File System (on prem), Google Drive, IBM Connections, Lotus Notes, iManage, MicroFocus Content Manager (HPE Records Manager, HP TRIM), Objective, OneDrive, Open Text, Oracle, SharePoint (on prem), Slack, Twitter, Xerox DocuShare, Yammer

See the ‘Microsft Graph connectors gallery‘ web page for the full set of current connectors.

A consideration when deploying search connectors is the quality of the data that will be surfaced via searches. Duplicate content is likely to be a problem in identifying the single – or most recent – source of truth of any particular digital record, especially when the organisation has required records to be copied from one system (mailbox/file share) to another (EDRMS).

API Connectors

API connectors provide a way for Microsoft 365 to access and use content, including in third-party applications. To quote from the Microsoft ‘Connectors‘ web page:

‘A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.’

To see the complete list and for more information about each connector, see the Microsoft web page ‘Connector reference overview‘.

Each connector provides two things:

  • Actions. These are changes initiated by an end-user.
  • Triggers. There are two types of triggers: Polling and Push. Triggers may notify the app when a specific event occurs, resulting in an action. See the above web page for more details.

API connectors can support records management requirements in different ways (such as triggering an action when a specific event occurs) but they should not be confused with archiving or search connectors.

Summing up

The connectors available in Microsoft 365 support the model of keeping records in place where they were first created or captured. They enable the ability to archive data from third-party cloud applications, search for data in those (and on-premise) applications, and triggers actions based on events.

The use of connectors should be part of an overall strategic plan for managing records across the organisation. This may include a business decision to continue using an ECM/EDRMS in addition to the content created and captured in Microsoft 365. Ideally, however, the content in the ECM/EDRMS should not be a copy of what already exists in Microsoft 365.

Posted in Correspondence Management, Flow, Information Management, Microsoft Teams, Office 365 Groups, Products and applications, SharePoint Online, Solutions

Managing correspondence in SharePoint Online including using Teams and Flow

Many (if not most) organisations receive and respond to correspondence in the form of letters or emails. They may also respond to social media messages.

Correspondence may be managed in different ways. For example, some organisations may have a dedicated business correspondence unit while in others individuals or business areas may respond.

This post describes how SharePoint Online with MS Teams and MS Flow could be used to manage letters and emails that require a formal ‘organisational’ response. It does not look at managing responses to social media messages.

Core elements

The management of correspondence that requires a formal response generally involves the following elements:

  • Somewhere to store the incoming correspondence (including scanned paper documents) and responses.
  • A description of the correspondence (naming conventions and/or metadata).
  • Some form of workflow, including emails, used to notify people of actions they must take.
  • Various methods use to report on and track progress, and closure/completion rates.

SharePoint includes all these elements. The requirement for workflow can be met using emails, the built-in workflows, workflows built in SharePoint Designer, or MS Flow. MS Teams provides the opportunity to add additional value to the communication response process by allowing messaging, co-authoring of responses, approval processes, and more.

A model correspondence system

The model correspondence system described below is based on a correspondence management system that I developed for a large public sector organisation a few years ago, but using a different system.

The core elements of the model system are:

  • An Office 365 (O365) Group. The O365 Group has an email address and an associated SharePoint site.
  • The Office 365 Group’s SharePoint site.
  • A Team in MS Teams, connected to the O365 Group.
  • Microsoft Flow.

Office 365 Group

The O365 Group should have a name that reflects its purpose and makes it easy to recognise as it also becomes the email address name.

  • For example ‘GRP_Correspondence’ – the prefix ‘GRP’ is used as it defines it as an O365 Group, as opposed to a Security Group (SG) or Distribution List (DL) created under the same ‘Groups’ section in the O365 Admin portal.

The Members of the Group should be the group of people primarily responsible for managing responses to the correspondence. Other people can be invited to the SharePoint site directly (without having access to the Group’s Team) as Members or Visitors.

SharePoint site

The O365 Group’s SharePoint site has the same name in the URL – GRP_Correspondence.

The SPO site should have at least one document library with an obvious name, for example ‘Correspondence 2019’.

If there is concern about potentially long URL lengths, the library name could be reduced to, say ‘Corro2019’; the display name can still be ‘Correspondence 2019’.

Consideration might also be given to having a ‘drop off library’ in the Group site where anyone can save correspondence that may require a response.

The metadata required in the document library will vary between organisations. The core default metadata (for every new document library) already includes all the following:

  • Title
  • Created (date)
  • Created by
  • Modified (date)
  • Modified by
  • Version
  • Document ID (when enabled as a Site Collection feature, which is recommended)
  • Shared with
  • Shared with details
  • Check In Comment
  • Checked Out To
  • File Size
  • Folder Child Count
  • Item Child Count (shows how many documents in a folder)
  • Label setting
  • Retention label
  • Retention label Applied
  • Sensitivity

Additional metadata may include any of the following:

  • Sender (free text or potentially drop down choice)
  • Response Due date (Date)
  • Urgency (Choice – Routine, Urgent)
  • Description (free text)
  • Reply Status (Choice – Not required, Draft, Approved, Sent, Cancelled)
  • Response Type (Not required, By email, By letter)
  • Approved by (internal Active Directory name)
  • Date Completed (Date) < This date should correspond to the date on any reply.

Once the library has been created, content can be added.

  • Paper correspondence can be scanned and saved to the library. Many MFD (printers) now have the ability to save directly to SharePoint, perhaps to a drop-off library for categorisation and review before being moved to the primary library. The original paper can then be boxed and destroyed after a given period (3 – 6 months).
  • Emails can (and should) be copied from the Group’s inbox to the library (see screenshot below). To do this, sync the library to File Explorer and drag and drop.

EmailtoSyncLibrary

A new folder can be created for each new correspondence, as indicated in the screenshot below:

SPOCorroLibrary.JPG

Naming conventions

Both the correspondence and the folders where it is stored should be named according to naming conventions. Naming conventions can also be used instead of folders, to indicate the connection between the original and the reply. My preference is to use folders to group the original and the reply, and also because they are ubiquitous in the digital workplace.

Suggested naming conventions:

  • Folders: Surname-Subject-Date
  • Emails: EMAIL-Surname-Subject-Date
  • Documents: LETTER-Surname-Subject-Date
  • Replies: REPLY-Surname-Subject-Date-DRAFT or FINAL (the final version may be ‘signed’ and then saved as a PDF)

Always exclude spaces in names, as these will be replaced by ‘%20’ in the URL path.

If one or more standard templates are required for replies:

  • Create the new Site Content Type in the Site Content Types area.
  • Enable the management of content types in the ‘Advanced’ section of the document library settings.
  • Add the new Site CT to the library
  • Add or edit the Word template to be used by clicking on the Content Type in the ‘Content Types’ section, then clicking on ‘Advanced settings’ and ‘upload’. The template can continue to be modified as required directly from this area and ensures consistency in replies.

The new Content Type will appear in the ‘New’ menu in the document library but not in the MS Teams ‘New’ dialogue (see below). This means that standard replies using a template can only (currently) be created via the SharePoint library. Drafts, however, could be created with a standard document template first (same with emails, see below).

StandardReplyCT
SharePoint New options

MS Team

The Team in MS Teams can be connected to the O365 Group (it’s a one to one relationship, you cannot connected multiple O365 Groups with a single MS Team).

The Team can have multiple channels, and the SPO document library can be presented in a channel. For example, the channel named ‘Correspondence 2019’ includes the ‘Correspondence 2019’ document library as a tab, as shown below.

MSTeams_CorroExample.JPG

The use of Teams means allows drafts to be co-authored and chats to take place about the correspondence and other matters.

If the SharePoint site includes an ‘incoming correspondence’ drop off library, the Team could have a channel with that library as a tab. The channel could then be used to review and decide on what to do with the correspondence.

Routing incoming correspondence for a reply

Once the correspondence is saved in a SharePoint document library, a decision must be taken by someone whether a reply is required.

  • If no reply is required, this can be reflected in the metadata (‘No reply required’).
  • If a reply is required, the correspondence must be ‘sent’ or otherwise made available (see below) to someone to draft a reply. This can be a simple or complex process.
    • In some cases, a standard reply may be possible. The SharePoint site should contain at least one library that contains examples of standard replies to certain types of, or common, questions.

Simple routing

The easiest way to ‘send’ someone the correspondence for action is to use the ‘Share’ option on the folder where the incoming correspondence is stored. As the same ‘Share’ option appears in File Explorer when the library is synced, the sharing process can also be managed from File Explorer.

This means that the recipient only has to click on the folder link in the email they receive to see the content of the folder. As they have access to the folder, they can then use the ‘New’ option to create a draft reply, including to an email.

FolderLinkSent.JPG

Once the draft has been finalised, it can also be sent via the ‘Share’ option. Alternatively, an alert on the library will notify anyone who needs to be notified that a change has been made to the library.

Routing using MS Flow

A more complex routing process may be required if the draft requires several steps, for example:

  • Send to someone to create the draft reply.
  • Draft reply gets sent to someone else for approval.
    • If not approved, goes back to the person who created it. (This can loop several times)
    • If approved, a message is sent to the person who needs to finalise it
  • Reply is finalised, metadata is updated, and reply sent.

All SharePoint Online libraries include an in-built Flow workflow ‘Request Sign Off’. When a document is selected and the ‘Request Sign Off’ option is selected the first time, the person must select the option to ‘Create Flow’. The ‘Run Flow’ dialogue then appears, requiring someone to be identified as the Approver and a Message to be included. The approver can be anyone in the organisation, for example the person’s manager.

 

RequestSignOffWho.JPG

The Approver receives an email, allowing them to approve or reject, and add a comment. If rejected, the ‘Sign Off Status’ column in the SharePoint library is updated to ‘Rejected’, and the sender receives a message to advice them that approval was rejected.

RequestSignOffEmail.JPG

If approved, the sender receives an email to notify them, and the ‘Sign off status’ column changes to ‘Approved’.

Once the reply has been approved, it can be finalised and sent to the person who wrote the correspondence. All versions of the draft reply are kept in the same folder, along with the final.

Email replies

Once approved, any email reply could be sent directly to the correspondent from the O365 Group’s mailbox.

Reporting

Metadata from the document library can be exported to Excel, or to business intelligence systems (or PowerBI) for analysis and reporting purposes.

Summing up

Correspondence can be managed in SharePoint, with MS Teams used to provide additional co-authoring and ‘chat’ options for the team, and MS Flow used for more complex approval requirements.