Recordkeeping metadata in Office 365 and SharePoint

Records management standards (see below) state that a defining feature of records is that they are associated with metadata – both ‘point of capture’ metadata and ‘process’ metadata that continues to evolve throughout the life of the record.

For at least two decades, the requirement to capture and store metadata for digital records has driven the implementation of centralised electronic document and records management EDRM systems, many of which began life as databases used to record metadata about physical records (files and boxes).

EDRM systems were (and still are) used to store copies of digital records created or captured natively in other systems, primarily network file shares and email. End-users were required to copy individual records to the EDRMS, a process that mirrored the storage of records (including printed digital records) in physical files.

Network file shares and email systems were not considered to be suitable as recordkeeping systems because they could not ensure the authenticity, integrity and reliability of records over time, including to manage and preserve metadata about the records stored in them.

The increasing implementation of Office 365, and in particular the use of SharePoint for the storage of records, has highlighted the extent to which recordkeeping metadata can – or even should – be applied to the content stored in that system.

This post discusses the need for metadata in records stored in Office 365, including in both Exchange/Outlook, MS Teams, and SharePoint/OneDrive for Business. It concludes that most records stored in Office 365 do not need additional metadata but, where such metadata is required, there is unlimited capability to add it.

Records and metadata

The international standard for records management, ISO 15489:2016, defines a record as ‘information created, received, and maintained as evidence and as an asset by an organization or person, in pursuit of legal obligations or in the transaction of business’.

Records are said to be different from ‘non-records’ because they are associated or described with (mostly added) metadata that describes ‘the context, content and structure of records and their management through time’.

The standard for recordkeeping metadata is ISO 23081:2017. One records management professional (link at the end of the post) noted that there has been reluctant adoption of this standard, mostly because it was ‘too complex’ and ‘academic’, and used ‘foreign terms’. Unspecified vendors were said to have been dismissive of the standard.

Standard for managing digital records – ISO 16175

Part 2 of the standard ISO 16175:2011, ‘Guidelines and functional requirements for digital records management systems’ contains multiple requirements relating to metadata, across three broad categories:

• Point of capture metadata. This includes metadata that forms part of the ‘metadata payload’ of the original record (e.g., date created, creator), other metadata added at point of capture, and metadata that provides additional context for the records.
• Process metadata. This is metadata that records activities and changes to both the record and metadata over the life of the record.
• The need to manage and control metadata over time.

This standard appears to reinforce the requirement for records to be stored and managed in dedicated recordkeeping systems.

Metadata versus enterprise ‘graphs’

While this post was in draft, James Lappin published a very interesting and informative post titled ‘Project Cortex and the future of document management in Office 365‘. The post highlighted a key difference between on-premise and cloud era document management, in relation to the way metadata is managed:

• On premise document and records management systems: These systems use metadata schema that specify metadata fields to be used in the system.
• Cloud systems including Office 365: These systems can make use of enterprise ‘graphs’ that map people to documents and topics. The graph is built from the interactions of people with content across the different workloads of the suite.

Microsoft’s depiction of a graph model and the dynamic interaction between objects. 

Most people now accept the algorithm capabilities of Facebook, LinkedIn, eBay, Amazon and similar online systems to automatically connect us with information relevant to us, without having to add any metadata.

Given the volume and types of digital content, almost all of which has metadata ‘payloads’, how can we ever hope to add the required recordkeeping metadata?

Can’t we just rely on the algorithms and graphs?  

How much metadata do you really need?

The answer to this question may depend largely on business, regulatory/compliance and/or government recordkeeping requirements relevant to the organisation and its jurisdiction. In my experience, across multiple very large and also very small organisations:

• Most private sector organisations will likely have minimal metadata requirements beyond basic ‘point of capture’ and ‘process’ metadata already recorded in the system where the records are created or captured (including email), unless this is required for specific compliance or regulatory purposes, or where there is risk associated with poor recordkeeping. For example, in a major food processing company, records relating to the manufacture of food were very well documented and managed, while corporate records were managed haphazardly.
• Most public sector organisations are required, for government accountability and transparency (and information retrieval) purposes, to apply a minimum set of both ‘point of capture’ and ‘process’ metadata for non-permanent records. Many government agencies have struggled to manage digital records effectively.
• A small percentage of records captured or created in government agencies may require more extensive metadata, especially if those records are to be transferred to archival institutions for permanent retention.

Office 365 ‘workloads’

In Office 365, most business records will be created or captured in either Exchange/Outlook (includes MS Teams chats), or SharePoint or OneDrive for Business (for ‘working’ or personal content).

Exchange is a recordkeeping system in that it stores records with consistent metadata. The primary ‘weakness’, in terms of recordkeeping, is that ‘personal’ Exchange mailboxes aggregate records on a range of subjects by an individual user rather than by business subject. The mailboxes of Office 365 Groups, on the other hand, can be used to aggregate records about a business function/activity or subject.

SharePoint is a recordkeeping system that has extensive default metadata and almost unlimited additional metadata capability (see below). OneDrive for Business is a SharePoint service that has the same extensive default metadata capability.

There is, generally speaking, no requirement for organisations that have implemented Office 365 to allow the continued use of network file shares because the ‘save’ and ‘save as’ options in Office/Windows 10 points to SharePoint and OneDrive as the default save locations.

Metadata in Exchange mailboxes/MS Teams

Emails have the same metadata options in the header of every email:

• Message ID
• Subject
• Sender (From)
• Recipients (To, including CC and BCC)
• Sent (date/time)
• Received (date/time)
• (Plus more with routing information and security controls including DKIM, SPF, DMARC etc)

However, no other metadata can be added and some (or most) emails may never form part of the collated record of a given subject.

Because of this ‘limitation’, there has been an assumption ever since email was introduced that emails identified as records would have to be copied to a (separate) recordkeeping system.

• In pre-digital days, this meant printing out emails and placing them on a paper file. 
• In organisations with EDRM systems, this meant copying the email to the EDRMs where additional metadata would be applied.

The original emails generally remained in place in individual mailboxes where they may be subject to backups and journaling in case they needed to be recovered for whatever reason including subpoenas (eDiscovery).

The Office Graph in Office 365 now provides the ability to connect the content in email with other content across that ecosystem, as noted in James Lappin’s post above. This is new – but it doesn’t rely on metadata or copying emails anywhere.

Metadata in SharePoint

As a SharePoint service, OneDrive for Business has the same default metadata columns. According it will not be described further here.

What metadata is required?

Organisations that plan to manage records in SharePoint should consider the following questions as part of their overall information architecture design to ensure records are kept in logical aggregations rather than randomly. This is important especially if end-users are allowed to create Office 365 Groups or Teams. 

• What point of capture and process metadata is required (for compliance, regulatory, recordkeeping purposes)? What is the source of this requirement?
• Is there a difference in the metadata requirements for short-term (retain in the organisation) and permanent records that are to be transferred to archival institutions?
• Do the required metadata columns already exist in SharePoint?
• If they don’t exist, should the additional metadata columns be added as site columns or library columns?
• Does any of the metadata need to be mandatory, and/or can it be a default setting – for example, a metadata column that has the default function and/or activity so the user doesn’t need to add this.
• Where is the process metadata and how do you view or manage it? (See also below on this subject).

Information architecture and metadata

The information architecture of SharePoint, in terms of managing records as objects (e.g., documents, spreadsheets, images, etc), is relatively simple:

• SharePoint site. The primary aggregation that can be linked to a business function (e.g., ‘Financial management’).
  • Document library/ies. Logical aggregations or containers of records that can be linked to business activities (e.g., ‘Meetings’).
    • Folders, document sets as content aggregations.
      • Documents/records.

An effective site architecture can replace the requirement for metadata. For example, the name of the SharePoint site can map to a business function, and library names can map to activities, instead of applying a function and activity pair to each record. The URL address for the record provides the context:

https://tenantname.sharepoint.com/teams/finance/invoices2020/record1.ext

If additional metadata is still required, SharePoint has extensive and almost unlimited capability.

• Every new SharePoint site comes with a standard set of around 240 metadata ‘site columns’. The metadata columns include the Dublin Core metadata items.
• New metadata columns can be created at the site level (‘site columns’). These are then can be used by all libraries and lists on the site. Here is a useful description of how to add new site columns from ShareGate: SharePoint 101: SharePoint Site Columns.
• Every new SharePoint library comes with a standard set of metadata columns – see below. New metadata columns can be created at the library (or list) level, but these columns are only available to that specific library or list.

Default SharePoint document library columns

The default library metadata columns are as follows. Dublin Core metadata items are shown with [DC]:

• App Created By
• App Modified By
• Check In Comment
• Checked Out To
• Comment count
• Compliance Asset Id
• Content Type
• Copy Source
• Created [DC]
• Created By [DC]
• Document ID (when enabled as a feature)
• File Size
• Folder Child Count
• ID
• Item Child Count
• Item is a Record
• Label applied by
• Label setting
• Like count
• Modified [DC]
• Modified By Name [DC]
• Retention label
• Retention label Applied
• Sensitivity
• Title [DC]
• Type [DC]
• Version

How metadata is added to records in SharePoint

Every digital record saved to SharePoint will have some form of native metadata (payload). Additional metadata may be added when the document is saved; this may be optional or mandatory.

• When a digital record is saved to SharePoint, SharePoint only copies the title or name of the record, not the original created date or author.
• When a Microsoft Office document is saved to a SharePoint document library, the Office document stores the library metadata (including the unique Document ID) in its own XML-based properties. This information is retained with the record even when the record is downloaded from SharePoint.

Viewing the metadata

The metadata that describes the content stored in the SharePoint document library may be viewed in multiple ways (via the edit view option), and may be exported (for example if records are to be destroyed or transferred).

Every record includes a version history that provides details of who modified the content, and when (but not what changes were made unless this is recorded).

Process metadata

Process metadata is metadata that records events relating to the record or the aggregation in which it is kept.

Examples of process metadata include when:

• Records are viewed or downloaded (date and by whom).
• Records are modified (date and by whom, and ideally what changes were made).
• Records are copied or moved (date and by whom).
• Security controls were changed (date, by whom, and what changes were made).
• Records are deleted/destroyed (date and by whom, with what authority).

While ISO 16175 describes the general requirement to keep process metadata, the actual requirement is likely to differ between organisations. Organisations with high compliance requirements, such as certain types of businesses or government, are more likely to want process metadata to be created, accessed when required, and protected against unauthorised modification.

Office 365 process metadata

Office 365 records process metadata in multiple ways in Exchange and SharePoint.

• Emails generally cannot be modified after they have been sent. Accordingly, the primary process metadata for emails and Teams chat is likely to be in the deletion records stored in the Office 365 Compliance admin portal audit logs.
• SharePoint/OneDrive process metadata is recorded as follows:
  • Viewed or downloaded, modified, copied or moved: This is recorded in the Office 365 Compliance admin portal audit logs.
  • Modified: This is recorded in the Date modified and Modified by metadata, as well as the version history (which also keeps the previous actual versions that can be compared if required).
  • Security changes: This is recorded in the Office 365 Compliance admin portal audit logs.
  • Destroyed: Depends, but generally this requires the capture of information manually, then stored elsewhere. For example, if the content of a document library is to be destroyed, then the metadata (along with details of the original library URL) should be exported (manually) first and saved somewhere. This is a manual process.

Note that audit log data in Office 365 is only retained for 90 days with an E3 licences, 365 days for an E5 licence.

Final thoughts

Exchange/Outlook email has basic metadata. It is unlikely that it will ever be possible to add other metadata, unless email is copied to SharePoint document libraries.

Chats from MS Teams are stored in hidden folders in Exchange mailboxes.

Organisations that need to keep certain emails for specific compliance, recordkeeping or archival purposes, should consider capturing these in SharePoint document libraries. Organisations might also consider making more use of Office 365 Group mailboxes for business-specific content as these Groups also include both MS Teams chat and have an associated SharePoint site.

The metadata capabilities of SharePoint are unlimited but not all records need the same degree of metadata.

The majority of records can probably be managed in standard SharePoint document libraries using the default metadata columns, or with one or two additional site or library metadata columns added, where required.

The Office Graph will increasingly be able to bring together records dynamically in the context of the business or the end-user via Project Cortex and Delve, respecting security controls that may be in place. The centralised content search and retention policy capability in Office 365 will also enable businesses to find, retrieve and manage content across both Exchange and SharePoint.

Reference

AS/NZS ISO 23081 series, Information and documentation – Records management processes – Metadata for records

Using Office 365 Groups to manage (some) records in context

The third part of my recent series of posts, titled Setting up SharePoint Online to manage records, included an example architecture model with several Office 365 Group-based sites, shown in green in the graphic below.

This post describes how Office 365 Groups, with their associated mailbox and SharePoint site, can be useful when there is a business requirement to manage aggregations of email and document-based records for a given function or subject, and why this option should be considered as part of your architecture model for managing records in Office 365 and SharePoint.

What are Office 365 Groups?

Office 365 Groups are a relatively new concept and still remain poorly understood, especially in organisations implementing Office 365.

Because they are not well understood, organisations may implement all the elements of Office 365 without necessarily being aware that:

• Every new Group in Outlook, Team in MS Teams, Planner, and Yammer group creates an Office 365 Group with an associated SharePoint site.
• End-users can create SharePoint sites from their individual SharePoint portal (the default ‘Team site’ option creates an Office 365 Group)
• SharePoint sites will proliferate without any controls over naming or content.

In very simple terms, Office 365 Groups:

• Are Azure Active Directory (AD) objects, similar to Security Groups (also known as AD Groups).
• Like Security Groups, have members and give those members access to certain (but usually different) IT resources.
• Have an associated Exchange mailbox and SharePoint site (that links these two in the same context), and can be linked with a Team in MS Teams. The members of the Group have access to both the mailbox and the SharePoint site. Members of Security Groups may also be added to the SharePoint site but won’t be able to see the mailbox or the Teams chat.
• Are similar to Distribution Lists or a shared mailbox in the sense that all the members of the Group can receive emails from the one mailbox.
• Include other functionality, including Planner, and can be linked with Yammer groups/communities.

For more information on Office 365 Groups, see this blog post ‘Office 365 Groups Explained‘ on the Microsoft Tech Community site, which points to a more detailed post on the same subject from ShareGate.

The records management problem that Office 365 Groups can help to resolve

One of the key recordkeeping problems in many organisations (aside from managing digital records generally) is the disconnect between email and other forms of records on the same subject or context.

• Emails are created, sent to and from, and stored in mostly inaccessible ‘personal’ accounts (with the odd shared mailbox). Emails may have attachments and the attachment may be the only version of the record. Shared mailboxes partially help this issue but there remains a disconnect between emails and other records.
• Other digital records (including saved email attachments) may be stored across network file shares (including ‘personal drives’ and the local C drive) and other locations, including USB drives and unofficial cloud-based systems. Unofficial and unapproved storage locations heighten information security risks both from storing official records in unofficial locations, but also the potential for malicious links.

In the early computer days the only way to keep these records together was to print them and put them on a file, a practice that (sadly) continues to exist. Over the last 20 years, electronic document and records management (EDRM) systems have provided a similar functionality by requiring end-users to copy original documents to a digital version of the file (leaving the originals in place in most cases).

The problem that most organisations face is that records about the same subject or  business context, in multiple forms (email, documents) and formats (including chat, messaging and social media) are stored across multiple systems (Outlook, network file shares, personal drives, personal apps on mobile devices). 

As a case in point, in 2018 I asked a business unit of so-called ‘mobile workers’ how they kept in touch and ‘collaborated’. Their responses included Facebook Messager, Whatsapp, DropBox (and others) and private emails.

How can Office 365 Groups help recordkeeping?

As noted above, Office 365 Groups have a mailbox, SharePoint site and a Team in MS Teams.

If set up correctly, a single Office 365 Group can provide a single point of context for both email and other digital records including chat, without end-users having to copy or move content anywhere else.

The following are examples (from real life experience) of how Office 365 Groups can be used to keep related records in context.

Correspondence management

Organisation often have a central point for the management of all incoming correspondence. In the past, this was likely to be a shared mailbox and correspondence might be uploaded (including after scanning of paper mail) to an EDRMS or other system for routing and responses.

Instead of using a shared mailbox, the Office 365 Group mailbox can be used to receive all emails (where they can remain), and its connected SharePoint site can be used for the storage of digital content including template responses (that could also be stored in Content Types created on the site). A Team in MS Teams may also be created to provide a forum for discussion about the correspondence or other matters – for example, a channel to discuss draft responses.

All the members of the Office 365 Group can access the mailbox, chat and collaborate on content within Team channels, and use the SharePoint site. They can use the ‘share’ option (rather than attaching document to emails) to share drafts with others, or use the ‘out of the box’ ‘Request Sign Off’ flow available in every document library to seek approval.

Additionally, if the Group’s SharePoint site front page has the default ‘Site Activity’ web app displayed, this will show new emails coming in to the Group’s mailbox as shown in the image below. These emails are only accessible to the members of the Group.

Example of an incoming email automatically appearing on the front page of a Group’s SharePoint site

 

 

 

 

 

 

 

 

 

Storing records by function/activity/subject

Office 365 Groups could be used to manage the records of a particular business function and activity, particularly those where there is a lot of email.

For example, the functions of ‘Fleet Management’ (or ‘Asset Management’), ‘Property Management’, or even ‘Financial Management’ are all likely to have both email and document type records.

• Emails relating to the function can be sent to and managed from the Group’s mailbox by the members of the Group, rather than using a shared mailbox (which is disconnected from the other records).
• Documents (including emails from the mailbox, if required) relating to the function can be stored in SharePoint document libraries that map to the activities that are being performed. For example, in a ‘Meetings’ library.
• There may be a single retention policy for the entire site or one or more label-based retention policies used on individual libraries, or a combination of both (the longest retention policy will take precedence).

In this way, emails and documents about the same subject can be managed from a single Office 365 Group.

Senior executive management

Office 365 Groups can be used to support and secure communication between senior executives. It provides them with a single restricted access mailbox and SharePoint site, access to both of which are controlled by membership of the Office 365 Group. It also provides them with a Team in MS Teams.

Additional security measures can be applied to more sensitive information including:

• More restricted security on some parts of the SharePoint site
• Data Loss Prevention policies for very sensitive information
• Retention policies to prevent the deletion of content (or capture ‘deleted’ content in a hidden Preservation Hold library).
• Stronger monitoring of all activity by end-user.

IT Operations/Service Desk

The IT Service Desk is a common point of contact in most organisations and most service desks will have a shared mailbox to review and triage incoming emails. They will also have a requirement to keep records relating to service support issues.

An Office 365 Group, perhaps named ‘ITServiceDesk’, can be established.

• The Group mailbox can be the central point of email contact for the Service Desk (ITServiceDesk@organisation.name).
• The associated SharePoint site can be used for the storage of service support documents and other content.
• The Group’s Team in MS Teams can have multiple channels to support each application or aspect of IT as required. The Service Desk can use the one-to-one chat and sharing screen capability to resolve service issues.

Office 365 Groups can change the way we work

As noted above, ‘personal’ emails and the use of mostly uncontrolled network file shares (and other storage locations) have been a common way of working for three decades.

Changing these work habits can be hard. However, change can be brought about with minimal impact, provided end-users are assured that the content they need to access will still be accessible and protected, and there is a tangible benefit in doing so in adopting the new ways of working.

These changes should start small, with a focus on mainly small business units that would benefit from being ‘converted’ to an Office 365 Group. The changes that come with an Office 365 Group include:

• Replacing shared mailboxes and some (mostly smaller) distribution lists with Office 365 Group mailboxes – still accessed from Outlook.
• Replacing network file shares with a File Explorer-based view of the Group’s SharePoint site libraries (via the sync option). This will allow end-users to continue to work in a familiar way. Note that added metadata is not visible from File Explorer and certain metadata options, such as making a column mandatory, will cause the File Explorer view to become read only.
• Introducing end-users to MS Teams initially for one-to-one chat, and pointing out how their Group also has a Team where they can chat and access the SharePoint site and other resources.
• Demonstrating how the browser-based version of a SharePoint site can show (via the ‘Activity’ web part) emails coming from the Group’s mailbox. And that document libraries have a range of additional document and records management functionality.
• Managing retention for all aspects of the Group, ensuring that content is kept (and can be recovered) for as long as required. This action can be hidden from end-users.

The end result should be better management of all records relating to specific subjects or needing to be kept in context.

What about other records stored in email and SharePoint sites?

This post has focused on the benefits of using Office 365 Groups to manage certain types of records within a given context. This model may not be suitable for all types of records. For example:

• End-users will probably, for years to come, create, send and manage emails in ‘personal’ email accounts. Email won’t go away as it provides a useful medium for certain types of communication.
• End-users will also continue to use their personal space in OneDrive to create and store records that should be stored in SharePoint. It is not easy to monitor this content and so end-user training is essential to ensure that final versions of records are stored in SharePoint.
• Security Groups are still valid, especially for groups of 30 or more and can be used in parallel with Office 365 Groups. For example, a small business area may have an Office 365-based SharePoint site and decide to give read only access to the members of the members of a Security Group (with different members), by adding that Security Group to the Site Visitors group. Any member of the Security Group who also happens to be a member of the Office 365 Group will continue to have Member (add/edit) access.
• High level business department/divisions may prefer to retain standard SharePoint sites with limited Member access and with Visitor (read only) access controlled via Security Groups (especially those with hundreds of members). An Office 365 Group with more than 30/50 members is still possible, but the benefits compared with using Security Groups is debatable, especially if a Team is linked with the Office 365 Group. Experience using Yammer since 2012 suggests that (a) Teams with more than 30 people can become very quiet and (b) a mailbox with more than 30 members is not useful.

Summary

Office 365 Groups:

• Should be part of the information architecture model for managing records in Office 365. They are well-suited for lower-level and/or small business units with fewer than 30 members as experience suggests that the more people in the group the less likely everyone will actively participate and contribute.
• Can and should, in many instances, replace existing functionality including shared mailboxes and network file shares.
• Will allow end-user to continue to work in familiar ways, via Outlook and File Explorer, while offering new options to communicate and collaborate.
• Should reduce the volume of ‘personal’ emails and attachments to emails.
• May enable the creation, and facilitate the storage and management of records relating to the same business context, including within a function/activity pair.
• May never replace the requirement for email or ‘standard’ high level business division or departmental SharePoint sites.

 

Setting up SharePoint Online to manage records (as part of Office 365) – Part 2/3

This is the second of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

The first post was focused on Office 365 admin: Licences, roles and Groups; Compliance Admin – Retention labels and policies.

This post focuses on SharePoint Online Admin centre configuration. 

The next and last post focuses on SharePoint site collection provisioning and configuration for the management of records. 

SharePoint Online admin centre configuration

The SharePoint Online admin centre contains a number of configuration options and settings. Most of these settings relate to the administration of SharePoint as a service and are not described further unless they relate to the management of records.

Active sites

The section named ‘Active sites’ lists all active sites, including details of storage used and when it was last modified. The list can be exported as a csv file.

The records management team should have a retention plan for every SharePoint site, including Office 365 Group-based sites and communication sites. The SharePoint Admin and the Records Manager/s to review the list from time to time to review where content is stored and if any sites could potentially be deleted. 

Creating new sites

As noted in the screenshot above, the SharePoint Admin can create a new site directly from this portal, or it may be scripted.

Organisations that are new to Office 365, and especially larger organisations that want to manage corporate records in SharePoint, might consider restricting – at least initially – the ability for end users to create new SharePoint sites, as well as new Teams in MS Teams, Groups in Outlook that also create SharePoint sites via the Office 365 Group.

If there is no control over the creation (at least initially) of SharePoint sites, the number of sites could grow exponentially with no regard to corporate recordkeeping requirements. Sites holding important records may abandoned or forgotten, or be invisible to people who need to see them.

As soon as there is sufficient critical mass in terms of SharePoint sites for business areas, and training and awareness for end users, these controls may be loosened.

There are three options to create new sites from this portal:

• Team site. These create an Office 365 Group with Members who become the Members (add/edit) of the SharePoint site. It is recommended that an Office 365 Group is created first to ensure consistency in Group naming and controls. These types of sites, with a Team in MS Teams, may work better for smaller business units or project teams with less than 30 staff. They are also more likely to contain ‘working documents’ or have content (including the connected mailbox) that can be covered by a single retention policy.
• Communication site.
• Other options (sites). The options here are Team Sites, Document Center, Enterprise Wiki, Publishing Site. Team sites created here are best for large departmental or divisional sites where access can be controlled through AD Security Groups. These types of sites are more likely to last for several years, contain formal, final versions of records stored in controlled and well-named document libraries, and be subject to more than one retention policy (including both site and library policies).

All new sites must be provisioned, which is described further below.

Admins

The SharePoint admin can only assign, from the admin portal, Site (Collection) Administrator permissions for individual SPO sites. Site Owners, Members and Visitors are assigned in the individual sites once they are created.

Generally speaking, Site Owners should work in the business unit that ‘owns’ the SharePoint site. Site Owners should not be the head of the business unit unless they are prepared to manage the SharePoint site.

Site Administrators are the Site Collection Administrators found in that section of the permissions ribbon menu for the site, under ‘Advanced permissions settings’.

Generally speaking:

• All SharePoint Admins should be Site Collection Admins
• Site Collection Admins should be grouped in a Security Group (so each site doesn’t have to be modified every time, only the SG)

If the SharePoint Admin is not listed in the Site Collection Admin group (including via the recommended SG), they may get ‘access denied’ if they try to open the site directly. They can, however, still see the site and modify the admins from the SP Admin portal.

The Primary Admin is by default ‘Company Administrator’. It is good practice to: (a) create a single SG for SharePoint Admins, and (b) remove Company Administrator as it doesn’t really need to be there – GAs can access the SP Admin portal anyway.

It is recommended that a key or senior records or information manager be added to the Site Collection Administrator Security Group added to all SharePoint sites to provide access to all to the content, if required. This can be removed on a case by case basis if there are concerns about the security of the content in those sites.

External Sharing

External Sharing is always disabled, even if it is enabled globally. A decision must be made for each site to allow external sharing.

External sharing allows records to be shared directly with external parties, rather than being attached to emails. This provides better security for those records as the ability to prevent the download the record can also be added.

Hub sites (or sub-sites?)

Hub sites (top level and ‘subsidiary’ sites) are effectively the replacement for sub-sites in SharePoint. See below regarding the architecture of SharePoint sites.

More features – Records Management

The SharePoint admin portal has a ‘classic’ setting under ‘More features’ called ‘Records Management’. This is not what it appears to be – it is in fact a way to set up ‘send to connections’ to ‘send’ (actually copy) content to a Records Center.

There are a number of problems with this (one of which is that it copies the most recent version and re-creates it in a new library) and it is not recommended for the management of records. 

OneDrive Admin

The OneDrive Admin portal includes a ‘Storage’ section that defines how much storage user’s will get as well as a setting for how long the content will be retained.

Records managers should be involved in discussions around the retention of OneDrive for Business content both while the account is active (via an Office 365 retention policy) and after the account is de-activated (via the setting here).

 

 

Setting up SharePoint Online to manage records (as part of Office 365) – Part 3/3

This is the third and last of three posts that describe the main elements involved in setting up SharePoint Online to manage records.

The first post focused on the recordkeeping related elements in the Office 365 and Compliance admin portals, including user licences, Roles and AD Groups, and Compliance Admin – Retention labels and policies. 

The second post focused on SharePoint Online Admin centre configuration.

This post focuses on SharePoint site collection provisioning and configuration.

SharePoint Online Site collection architecture, provisioning and configuration

An example SharePoint architecture model

Architecture design model

The three primary architecture considerations are as follows.

URL path

All new SharePoint Online (SPO) sites are created under the tenant name:

• https://tenantname.sharepoint.com/

There are only two options after the tenant name element:

• /teams/. This would normally be used for logical organisational business units and projects. It includes team sites created for Office 365 Groups.
• /sites/. This would normally be used for cross organisational business units or subject areas and communication sites.

Site name and naming conventions

The site name comes after the URL path option (teams or sites).

• The URL name for the site should have no spaces (otherwise these are changed to ‘%20’), and should be limited to 14 characters. Common or not obvious acronyms should always be avoided. For example, the name ‘Business Development Management’ should not be ‘BDM’ but could be abbreviated to ‘BusDevMgt’.
• The display name for the site can have spaces, added back after the site is created. For example, ‘Business Development Management’.

The URL name and the site display name should always bear some similarity to each other.

Type of site

Three types can be created (as noted in the previous post):

• Modern. These are directly associated with an Office 365 Group which can be linked to a Team in MS Teams.
• Communication. These are the replacement for sites that were created using the ‘Publishing’ features. Generally speaking these sites store fewer records and contain or display information of an ‘informative’ nature, like the intranet.
• Standard. While this option provides the ability to create several types of sites, but the most common to use here will be a standard team site.

The type of site may be influenced by the type of content and records stored in it. Generally speaking:

• Higher level business units (department, division, with more than 30 staff) may be better as a standard site where formal records are stored. Membership of these sites can be via AD Security Groups.
• Lower level business units, often with fewer than 30 staff, may work better as modern team sites based on Office 365 Groups where all the members of the business units are Members of the Office 365 Group (and Team), rather than using AD Security Groups. These sites are more likely to contain content that is subject to change or be considered ‘working documents’.

Additional standard sites may be created ‘between’ these two levels, to meet the requirements of the organisational business unit hierarchy. The number of people in each business unit, and their need to collaborate via MS Teams, may provide a guide to the best type of site.

Keep in mind however that AD Security Groups are usually maintained by IT, while Office 365 Groups (that provide the same type of access control as SGs) are maintained by the O365 Group Owner.

Higher level standard team sites can have hyperlinks to lower level business units on the left hand navigation or in a links web part on the page. Sub-sites (except perhaps to control access to a smaller group, such as ‘Leadership’ or ‘Management’ of larger groups) should be avoided.

New site provisioning

All new sites need to be provisioned before they made available.

This usually means doing the following after they are created, noting that SPO sites that are created from Office 365 Groups may have fewer options initially in the ‘Site Settings’ area.

• Remove any left hand navigation options that are not required or could cause confusion via the ‘Edit’ menu. For example, consider removing ‘Notebook’ and ‘Pages’. Suggest you don’t remove Site Content or Recycle Bin unless really necessary – the end user can still access these via the cog/gear icon on the top right.
• Removing and adding any webparts on the main page. To do this, click on Edit and use the web part menus that appear on the left of each.
• Users and Permissions. For standard sites (non-Office 365 Group-based), go to Site Information – View all Site Settings, and click on ‘Site permissions’ This is where you add Site Owners, Members and Visitors. You can also add Site Collection Administrators here, if you forgot to do it from the admin portal. For O365 Group-based sites, click on Site Permissions from the cog/gear icon. You will see that the O365 Group Owners are now the Site Owners, and the O365 Group Members are the site Members. You can add anyone else via the “invite people’ option that includes the option to invite to the Group or just the site. You can get to the same other settings as for normal sites by clicking on ‘Advanced permission settings’.
• Look and feel. For normal sites, click here to change the display title and log. You won’t normally change anything else. On O365 Group-based sites you can change the display name from the cog icon menu – Site Information.
• Web Designer Galleries (both types). This is where you set up Site columns and Site content types. There are other options in normal sites.
• Site Collection Administration (do this before Site Actions). The only two things you will modify here are the Site collection features and the Document ID settings (when they appear, after enabling a feature). The features to enable are:
  • Document ID Service, Document Sets (optional), Open Documents in Client Application by Default, Reporting, SharePoint Server Enterprise Site Collection features, SharePoint Service Standard Site Collection Features, Three-state workflow (optional), Video and Rich Media (optional), Workflows (optional if you use SharePoint Designer to create workflows).
• Once you have enabled Document IDs, you will see that option in the settings; open this to change the default DocID prefix to be the same as the URL of the site or something very similar, to ensure it is unique (12 characters only). Document IDs are PREFIX-LibraryNumber-DocumentNumber. DO NOT enable the Site Publishing features; this is an old option that was for on-prem sites such as the intranet. It has been replaced by communication sites.
• Site Actions. Click on Manage site features and enable the following (you will see that a couple are already enabled):
  • SharePoint Service Enterprise site features; SharePoint Service Standard Site Features.
• Site Administration. Click on Regional settings to ensure the settings there are correct; they usually need to be changed for O365 Group-based sites.
• Search. There is usually no need to change anything here.

Site libraries and metadata

Every new SharePoint site has a default ‘Documents’ library.

Except perhaps for lower-level sites where the content is mostly working documents of little value, this library should be hidden or deleted and replaced with document libraries that have (a) more appropriate names, (b) metadata and if required, (c) either folders or document sets.

Document library names

As with site URL and display names, all document library have both a URL name and a display name. The URL name is the name given when the library is first created (from the gear/cog icon – ‘Add an app’).

• The URL name should ideally have fewer than 20 characters and no spaces (which will be replace by three characters ‘%20’). Consider including the year in the URL name instead of creating folders for each year, especially on team sites that may continue for many years (with a new library for each year). Separate year-based libraries may be easier to manage when it comes to retention and won’t contain an excessive number of items after several years, making it harder to isolate content. The URL name should also not repeat elements already in the site name, for example: https://tenantname.sharepoint/com//teams/tenantnamemeetings/
• The display name can be changed via the Library Settings after it is created. The display name should bear some similarity to the library’s URL name. If the library display name needs to change, consider creating a new library with and moving the content to that new library.

Standard and added metadata (columns)

Every new SharePoint site comes with a very rich set of metadata columns that can be added to each document library.

The standard set of columns available on every library are listed below:

• Type (icon linked to document)
• Name
• Modified
• Modified By
• Label applied by
• Label setting
• Retention label
• Retention label Applied
• Sensitivity
• App Created By
• App Modified By
• Check In Comment
• Checked Out To
• Comment count
• Compliance Asset Id
• Content Type
• Copy Source
• Created
• Created By
• File Size
• Folder Child Count
• ID
• Item Child Count
• Item is a Record
• Like count
• Title
• Version

The unique ‘Document ID’ metadata column will only appear when that feature is enabled from the Site Collection Administration section of the site. Document IDs have the format PREFIX-L-NNNNNN:

• Prefix. This should be changed in the Document ID set up to be the same as the site URL name, so any documents produced on that site have the same or similar name. Otherwise, the default prefix is a random set of 12 characters.
• The library ID (L). These IDs are sequential. As the ‘Documents’ library already exists, the next library will usually be ‘2’.
• The document ID (N). These IDs are also sequential and never re-used. If a document is deleted, the document ID is also removed from use. Accordingly, a gap in document IDs indicates that documents have been deleted.

Added or new metadata columns may also be created as either (a) site columns, which can be added to any library on the site, or (b) ‘local’ library columns. The advantage of using site columns is that the same column can be added to any library or list, ensuring consistency across the site.

Every new metadata column can be anyone of the following:

• Single line of text
• Multiple lines of text
• Choice (menu to choose from)
• Number (1, 1.0, 100)
• Currency ($, ¥, €)
• Date and Time
• Lookup (information already on this site)
• Yes/No (check box)
• Person or Group
• Hyperlink or Picture
• Calculated (calculation based on other columns)
• Task Outcome
• Full HTML content with formatting and constraints for publishing
• Image with formatting and constraints for publishing
• Hyperlink with formatting and constraints for publishing
• Summary Links data
• Rich media data for publishing
• Managed Metadata (open the Term Store Managed Metadata service)

Each of the above:

• Can have a description
• Is optional but can be mandatory
• Can be used to enforce unique values
• Can have a maximum length
• Can have a default value
• Can use validation formulas (for example, must have a certain form or content or an error will be display)

Site columns are added to document libraries manually from the Library settings page.

Metadata columns can be used to group content in a library instead of using folders or document sets, but this concept can be quite alien to users who generally prefer to work with folders.

Note – If metadata columns are made mandatory and a library is synced to File Explorer, the synced library will become read only in File Explorer.

Note also – Only the standard File Explorer metadata will appear in synced document libraries. If there is a requirement to see or enter more metadata, the end-user will need to work directly with the browser version.

Site Content Types

Site Content Types, from a recordkeeping point of view, allow the organisation to define a type of content that will be stored on a site. They usually have added metadata.

The main types of Content Type that are likely to be used from a recordkeeping point of view are document and document sets.

Document content types may also have a standard template attached.

SharePoint also includes the ability to use a Document Set content type. A document set is type of folder which, as the name suggests, is useful for grouping a common set of documents where additional metadata is required for the ‘folder’.

Document sets must be enabled as a feature from the Site Collection Administration section of the Site Settings, and then can be added ‘as is’, or new document sets created (from the ‘Site Content Types section of the Site Settings) based on the basic template.

An example of their use would be to store a set of documents relating to an object where there is a requirement to describe that object in the folder. For example:

• Employee. Metadata might be: name, employee ID, business area.
• Contract. Metadata might be: contract name, contract ID, expiry date.

Adding Content Types in a document library

Note – While Content Types can be very useful to ensure additional metadata is added or to group content (in document sets), end users will be required to choose the correct content type (for example, document type) and then add metadata as necessary.

This process can be off-putting and should, accordingly, only be used when there is a genuine requirement and no other obvious way to identify the ‘type’. For example, instead of forcing users to choose between two content types consider either:

• Including a metadata choice option (with a default setting) to select the option.
• Separating the library into the separate ‘types’, so that two different document types (especially with different metadata requirements) aren’t stored in the same library).
• Using file naming conventions to identify the type, e.g., ‘Contract-WilsonBrosFeb2020Final.docx’.

Library views

Standard ‘out of the box’ library list view

Every new document library displays content against a set of default columns:

• Type (icon)
• Name
• Modified (date)
• Modified by

Each item in the library has a Share option (if sharing is not disabled) as well as a three dot ellipsis menu (see below).

The list view may be modified (via All documents – Edit current view) to display more columns from the standard list as well as the document ID and any other added columns.

Views may be sorted, grouped, and filtered as required. SharePoint also includes a handy option to display all contents without folders.

Multiple views may be saved, each with a URL that can then be added as a link on the left navigation, on the page, or even sent to others. For example, a view might show a filtered view of all contracts due to expire in the coming six months.

Libraries, and views of libraries, can be embedded on the site pages. This could be useful if, for example, there is a need to see the most recent documents at a glance, or documents with an expiry date, etc.

Other document and records management functionality

All SharePoint document libraries include a range of document and records management functionality accessible from the three dot ellipsis menu, including:

• Permission/access control
• Sharing
• Copy to/Move to
• Check out/in
• Version history and the ability to restore versions
• Alerts

Every SharePoint site also includes a Recycle Bin that stores deleted content for 90 days. The Site Collection Administrator can review and restore documents deleted by anyone on the site for the previous 90 days. If there is concern about documents being deleted, any user with access can easily set an alert on the library for any changes that are made.

Retention policies

Retention policies, created in the Compliance Admin portal, work in two different ways in SharePoint document libraries.

See this post for a lot more detail on the creation and application of retention policies.

Explicit (visible) retention labels

These are published as retention policies and must be applied to each individual library.

It is not possible to delete content in libraries where this type of retention policy has been applied.

If the retention policy has the disposition review option enabled, the person or role nominated will be advised that the records are due for review; they will remain in place until a decision is made to delete them.

Implicit (invisible) retention policies

These are applied at the site level and may delete content when the retention period has ended.

There is no disposition review process, the documents are deleted by ‘System Account’ when the retention period has expired and the 90 day Recycle Bin period has finished.

Going live

If you have managed to reach the end of these three posts – well done! There is a lot to take in. Hopefully this information will help you going live with a new SharePoint implementation, a migration from SharePoint on-premise, or a clean up of either.

If you’d like me to write about other aspects of managing records in SharePoint, let me know via the feedback option here.

Understanding SharePoint Online in the context of Office 365 admin

On-premise versions of SharePoint were standalone systems, usually administered by a trained and qualified SharePoint Administrator. Records managers may and may not have had access to or a role in that environment.

Generally, the only other group that would typically have access to SharePoint on-premise were the DBAs who managed the (SQL) database.

SharePoint Online is no longer a standalone system but a core part of the Office 365 ecosystem.

This post describes, for records and information managers, how SharePoint Online needs to be understood in the context of the broader Office 365 administration, and how other admin roles can configure or change settings that can affect SharePoint and OneDrive for Business. 

See also my earlier post from 9 January 2020 titled ‘How many Office 365 Admins do you really need?‘

Office 365 Roles

Global admin

The highest level admin role in Office 365 is the Global Admin (GAs).

To protect the security of Office 365, there should be a very small number of GAs. GAs should have unique cloud-only log ons preferably using multi-factor authentication for added security. End-user accounts should NEVER be assigned the GA role.

GAs can access everything across Office 365, including the content of emails, SharePoint, OneDrive for Business and MS Teams. All activity carried out by GAs (and anyone else) is recorded in the audit logs.

Organisations that outsource the GA role to third-party companies need to be aware of the capability of the GA role and, ideally, also have at least one GA log-on account so they can, among other things, access the tenant and review the audit logs if required.

The key activities that GAs are responsible for, that impact on the management of SharePoint, are as follows.

• Assigning licences. Licences (e.g., E3) provide user access to the various applications in Office 365, including Exchange, SharePoint, OneDrive for Business, MS Teams and Office (via http://www.office.com). Generally speaking it is inadvisable to remove individual options from licences. Note that the SharePoint licence gives access to use the application, it is not the admin role (next point).
• Assigning roles. Roles provide admin access to the core applications (listed in the previous point) and to a range of activities (for example, Billing, Compliance, Security, User Admin). Office 365 Admin roles should always be cloud only and never assigned to normal end-user accounts. This ensure that the person logs on to perform an admin activity, as opposed to a general end-user activity. It is common (and good) practice for users may be logged on to two ;separate accounts at the same time.
• Creating Groups. Groups are Azure/Exchange objects. The three main types of groups are: (a) Security Groups that control access to resources but are not email enabled; (b) Distribution Lists that provide the ability to email multiple people but don’t control access to resources; and (c) Office 365 Groups that a cross between Security Groups and Distribution Lists with much more capability. Office 365 Groups are a core element across Office 365. Every O365 Group has (a) an email mailbox, (b) and a SharePoint site. If the ability to create these types of groups is not controlled, every new Team in MS Teams will create an O365 Group with a SharePoint site (with no controls on naming). Accordingly, there needs to be close cooperation between the GA, the SharePoint admin and/or the records/information manager in relation to the creation of O365 Groups.
• Enabling external access for SharePoint. This setting allows the GA to determine whether SharePoint sites and OneDrive for Business, and the content in them, can be shared externally. The setting only makes the option available for SharePoint sites but allows ODfB content to be shared externally. Individual sites must still be enabled (by the SharePoint admin) for external access.

SharePoint/OneDrive for Business Admin

The SharePoint Admin will normally be a qualified SharePoint administrator and may have administered earlier versions of SharePoint. They will also generally be the OneDrive for Business admin (as OneDrive is a SharePoint service).

The SharePoint Online admin role is much less complex in Office 365 than it was in the on-premise version. Records managers who currently manage an EDRMS could potentially become a SharePoint admin, with some training.

Additional training is required only if the organisation wishes to do additional customisation or development work, integration, or has third-party applications.

The SharePoint Admin has a number of roles:

• Configuring SharePoint settings in the admin portal. This is usually a one-off activity that may be reviewed from time to time. Configuration settings should be documented.
• Creating new SharePoint standard and communication sites – but NOT ‘modern’ team sites that are based on Office 365 Groups, as noted above. These should be created by the GAs who will need to be advised about (a) preferring naming conventions (if any) and (b) Group ownership and membership (which flows through to SharePoint site ownership and membership).
• Provisioning new sites. This activity involves changing site collection features and site features to enable things like Document IDs and Document Sets. It also includes assigning the initial Site Collection Admin and Site Owner permissions. It may also include some basic additional options such as a new document library or list.
• Assigning access and permissions. Records managers who have responsibility for managing records in SharePoint should be added to the Site Collection Admin section, ideally as part of a Security Group. This ensures that records managers can access all SharePoint sites as required (including the Preservation Hold library on sites where implicit retention policies have been applied) and, if they have the responsibility to do so, create and configure new document libraries to manage records. Both Site Collection Admins and Site Owners can apply explicit (visible) retention policies to document libraries and lists, if used.
• Monitoring and managing the SharePoint environment, including resolving issues and working with Site Owners.
• Managing the OneDrive for Business admin portal, including setting (a) the size of the ODfB storage and (b) the retention period for ODfB accounts after an end-user leaves.
• Providing training to Site Owners, if no other training is provided.

The relationship between the various Office 365 admin elements, SharePoint admin, and the end-user experience is described in the graphic below.

• SharePoint admins access the SharePoint admin portal by logging on to http://www.office.com, clicking on the ‘Admin’ option, the then SharePoint admin portal (or directly to that admin portal if they save it as a favorite).
• End-users access SharePoint by logging on to http://www.office.com and clicking on the ‘SharePoint’ app, or via the mobile app.

Exchange Online Admins

The primary role of the Exchange Online (EXO) admin is to manage that application. The EXO admin may also be the MS Teams admin – see below.

If the creation of Office 365 Groups is not controlled as noted above, both EXO admins and end users can create a new Office 365 Group from Exchange or Outlook which in turn creates a new SharePoint site.

The Outlook menu with the New Group option

 

While emails can be copied from Exchange to SharePoint, Microsoft’s model assumes that the vast majority of emails will remain in end-user mailboxes.

Records managers need to work closely with the EXO admin/s and the Compliance admin/s (see below) to ensure that an appropriate Office 365 retention policy is applied to the content of the mailboxes. There may also be a requirement to remove the default MRM policies.

An Office 365 retention policy may initially appear to conflict with, but can support and replace previous backup strategies deployed to recover mailboxes in case of disaster or for investigation purposes. This means that a single retention policy that keeps all emails for a specific period of time will be applied to all mailboxes.

MS Teams Admins

The role of the MS Teams admin is to configure and manage the MS Teams environment. As noted above, the EXO admin may also be assigned the role of MS Teams admin.

MS Teams includes two main component parts:

• Chat. One-to-one chats are stored in a hidden folder in the Exchange mailboxes of individual users. Channel chats are stored in a hidden folder in the mailbox of the linked Office 365 Group. These hidden folders are not subject to a retention policy applied to the rest of the mailbox.
• Documents. These are stored in either (a) OneDrive for Business for one-to-one chat, or (b) in the linked SharePoint site for Teams channels.

Records managers should work with the MS Teams admin and the Compliance Admin to identify how retention policies will be applied to both the chat and SharePoint content in MS Teams.

See this article titled ‘Retention policies in Microsoft Teams‘ for more information.

 

 

Compliance Admin

Microsoft separated the Security and Compliance portals in early 2020. Consequently, there may be an admin to manage each component part – one for Security and one for Compliance.

The Compliance admin portal includes a range of actions relating to the management of information. These actions include:

• Data classification. This option is still in preview but for E5 licence holders, will allow data to be classified automatically and retention policies applied to that content as an alternative to pre-defined (SharePoint site/library) ‘classification’.
• Setting and monitoring alerts.
• Viewing reports on various compliance matters, including the status of retention policies.
• Creating and monitoring retention labels and policies. This includes retention policies for Exchange mailboxes, SharePoint Online, OneDrive for Business, and MS Teams.
• Creating and monitoring data loss prevention policies.
• Assigning permissions to individuals.
• Managing GDPR data subject requests.
• Searching audit logs (90 days of history only).
• Searching for content across all of Office 365.
• Reviewing disposition for records covered by explicit retention label policies, where this option is enabled

Some or all of these roles may be performed by senior records or information managers.

The Security admin portal provides access to the following actions, some of which may impact on SharePoint (sensitivity labels in particular).

• Reviewing alerts
• Reviewing security related reports
• Creating and managing sensitivity labels and information types (and also creating and publishing retention labels)
• Creating a range of security-related policies including for devices, threat protection
• Assigning permissions

Conclusions

SharePoint on-premise was a standalone system that generally did not interact or integrate much with other systems.

SharePoint Online is a core part of the broader Office 365 ecosystem. A range of roles and configuration settings set across that ecosystem have – or can have – a direct impact on SharePoint Online.

Records managers who are involved with SharePoint Online need to understand this crucial difference and either learn or seek to be assigned key roles that impact on the management of records across the Office 365 ecosystem, not just in SharePoint.

Office 365/SharePoint Online – Accounts, roles and groups for records and information managers

Organisations have stored and managed records in on-premise versions of SharePoint for many years. Generally speaking, these records had to be managed in – and disposed from – individual site collections.

Office 365 now includes a range of roles, access controls and settings that can be used to manage records across the ecosystem, making it much easier to manage records centrally.

This post describes the suggested user accounts and roles, and the options they provide access to, that can be used by records and information managers to support the management and administration of records in Office 365.

The short version

If you’d like to avoid reading the entire post, the details in summary are:

• User Account: A dedicated ‘records manager admin’ Office 365 user account (separate from a normal end-user account), with its own O365 licence (E3 recommended).
• Admin role: EITHER the ‘Compliance Admin’ role set in the Office 365/Azure admin portals OR the same role in the Security and Compliance portal OR a custom role group (e.g., ‘Records Management’).
  • The Compliance Admin role provides additional options that may not be required, hence the suggestion to create a custom role group.
• Security Group: An Office 365 Security Group that includes the records management admin account (and others).
  • The addition of the O365 Security Group to the Site Collection Admin section of every SharePoint site.

User account

Anyone with any kind of admin access or role in Office 365 should have an admin account, in addition to their normal end-user account.

Admin accounts will usually (but not always) require an Office 365 licence.

Office 365 admin accounts should exist only in the cloud and their naming convention should reflect that difference. For example:

• Normal AD end user account name: username@tenantdomain.org
• Office 365 SharePoint admin account name: O365_ADM_SPO_username@tenantname.onmicrosoft.com

To access the records management features in the Office 365 Security and Compliance portal, records and information managers should have either (a) an appropriate Office 365/Azure admin account (see below) or (b) an ‘admin’ user account. The admin user account name could be something like:

• O365_ADM_Records_username@tenantname.onmicrosoft.com

Office 365 (and Azure) admin role

There are around 35 admin roles in Office 365/Azure, in three broad role types:

• Global admins. This role has unlimited access and can edit any setting.
• Application-based roles such as Dynamics 365 admin, Exchange admin, SharePoint admin, and Teams admin.
• Activity-based roles such as Billing admin, Compliance admin, Helpdesk admin, License admin, Security admin, Service admin, and User admin.

There is NO specific Office 365 admin role for ‘records management’ – the closest in terms of access and functionality required is the ‘Compliance Admin’ role.

Office 365 admin roles are assigned to user accounts either (a) from the individual user account or (b) from the ‘Roles’ section in either the Azure Active Directory admin centre or the Office 365 Admin portal.

Except for the Global Admin role which can access anything, all other roles provide access to specific parts of the Office 365 admin portal or to other admin portals. For example:

• The SharePoint Admin role provides access to the SharePoint admin portal.
• The Exchange admin role provides access to the Exchange admin portal.
• The Security Admin and Compliance admin roles provide access to the Security and Compliance admin portal BUT NOTE that these two roles can also be assigned directly in the Security and Compliance portal which only give access to that portal – it does not give the person access to the Office 365 Admin portal. See below for what this role provides access to.

Users who are assigned and logged on with Office 365 admin roles will see the ‘Admin’ icon in the list of Office 365 apps in their office.com apps listing. If they are granted either the ‘Security Admin’ or the ‘Compliance Admin’ role, they will also see the ‘Compliance’ app as shown below.

Security and compliance portal – records management roles

The Security and Compliance centre (https://protection.office.com) includes a number of sections and options, including several that a records manager should probably have access to or manage:

• Classification
  • Retention labels (published as retention policies)
  • (Two other options specific to security/sensitive information)
• Data loss prevention
  • Policy
• Records management
  • File Plan (if used when creating retention labels)
  • Events (that can be used to trigger retention policies)
  • Dispositions (where disposal is managed)
• Information governance
  • Dashboard
  • Import (PST files and social media content)
  • Archive (for Exchange mailboxes)
  • Retention (create labels and publish them as policies (which replicates the options under ‘Classification’; create retention policies (without the option of a disposition review) for specific SharePoint sites)
• Search
  • Content search
  • Audit logs (up to 90 days)
• eDiscovery
  • eDiscovery (cases, and legal holds)
  • Advanced eDiscovery

All the options above, and the ones listed below, are accessible to anyone assigned the Office 365 ‘Compliance admin’ role. Not all of these may be relevant or required by records managers – and indeed may provide too much access.

• Permissions (to manage roles)
• Supervision
  • Use to define policies that capture email and 3rd-party communications
• Threat management
  • Dashboard, Submissions, Review, Policy
• Mail Flow
  • Dashboard, Message trace
• Data Privacy
  • GDPR Dashboard, Data subject requests
• Data Investigations (new)
• Reports
  • Dashboard, Manage Schedules, Reports for download,
• Service Assurance
  • Four options with information about the Security and Compliance centre.

Because the Office 365 Compliance admin role provides so much access, it may be better to create a custom role group in the Security and Compliance Centre with fewer roles. 

The minimum set of roles most likely to be required by a records manager, and the options they provide access to, are Disposition Management and Retention Management which (together) provide access to the options listed below. The ‘RecordsManagement’ role is redundant because it includes the same last three main dot points listed below that are included with the above two roles. 

• Classification
  • Retention labels (create and publish)
  • Sensitivity labels (create and publish)
  • BUT NOT Sensitive info types
• Records Management
  • File Plan (review)
  • Events (create)
  • Dispositions (review and manage)
• Information Governance
  • Retention (create and publish)
  • BUT NOT Import, Archive, or Dashboard
• Threat management
  • Review
• Mail flow
  • Dashboard
• Service assurance
  • Four options

If records managers need to search and view audit logs and also access the eDiscovery section, it may be better to assign the Compliance Admin role to the records manager (or at least one of them. The portal warns that because the results of these logs could contain sensitive information, the role should be used only by those people with a specific need to know to details.

Organisations will therefore need to consider if records managers should be assigned the Compliance admin role (with more options), or a ‘lesser’ custom role with fewer options created with the Compliance centre directly.

SharePoint-specific roles and access

Security Group to manage SharePoint Site Collections

In addition to the SharePoint Administrator (Office 365 admin) role, every SharePoint site has two key local ‘admin’ role groups:

• Site Collection Administrator (SCA)
• Site Owner

SCA access allows records managers to:

• Access all parts of a SharePoint site (for review purposes, and also to guide Site Owners on best practice records management)
• Apply label-based retention policies to specific lists and libraries
• Export metadata for document libraries before the library content is disposed of as part of a retention policy
• Manage inactive SharePoint sites

Adding the Security Group to the Site Collection Administration section of every SharePoint site

As SCAs can configure settings and access all parts of SharePoint sites, records managers should be in the SCA group on every SharePoint site, but not by name.

Instead, they should be added to a Security Group created in the Office 365 admin portal (under ‘Groups’) or Azure AD. This SG should be added to the SCA group on every site.

The screenshot below shows where and how the Security Group is added to the SCA group in SharePoint:

If the ability to create Office 365 Groups is not controlled, end users will be able to create SharePoint sites either directly from their SharePoint portal or by creating a Team, Plan, or Yammer Group; the O365 Group Owners are added to the SCA group.

Accordingly it may be necessary to retrospectively add the Security Group to SharePoint sites created by end users who create Office 365 or Yammer Groups, or Teams in MS Teams.

Footnotes

Expiry rules for Office 365 Groups and associated content

Organisations that have AD Premium can create expiry rules for Office 365 Groups that can result in the deletion of all aspects of an Office 365 Group, including its SharePoint site, if the Group becomes completely inactive (including no views of any content) within a given period of time.

Generally speaking the action of creating such rules would be restricted to the Azure Admins/Global Admins. In organisations that have AD premium, the records administrators will need to know if an expiry policy is in place OR direct what that policy should be. It is understood that any content in an Office 365 Group’s SharePoint site will not be deleted in this way if the SharePoint site (including via a retention policy applied to the Group) requires the content to be kept for longer.

Auto-application of retention policies

As noted above, records administrators may need to be able to use the ‘auto apply’ option in retention policies based on certain conditions being met. An E3 licence only allows two conditions:

• When the content contains certain (machine identifiable) sensitive information (same as the DLP options)
• When specific words, phrases or other properties can be identified via a keyword query.

An E5 licence allows additional conditions including based on the auto-classification (machine learning/teaching) of content.

Migrating network file shares to SharePoint

Updated 10 November 2019 with additional details based on feedback received.

As part of the implementation of Office 365, organisations may consider migrating content from network file shares (NFS) to SharePoint Online (SPO).

Closing down file shares and moving all the content to a system that can manage records has been an objective for many years. SharePoint can help organisations achieve this goal but it needs to be planned well, and some compromises may be required along the way. For example, you may end up migrating most of the content on drives ‘as is’ – but at least you can subject the content to recordkeeping controls including disposal/disposition.

This post describes the main activities or outcomes associated with migration projects.

Assumptions

For the purpose of this post, it is assumed that:

• The organisation has established a change program for Office 365 including SharePoint Online, and that end-users will be familiar with using SharePoint by the time the file shares are migrated.
• The implementation of Office 365 will includes a sub-project specifically to migrate the file shares to SharePoint Online.
• On-premise Exchange mailboxes and personal ‘home’ drives will be migrated as a separate sub-project but likely at around the same time.

Project artifacts and outcomes

The primary project activities, artifacts or outcomes may include the following:

• Detailed design process flow summary.
• Initial description and analysis of the network file shares that may be migrated.
• Creation of a ‘decision’ tree to help decide what will be migrated.
• Initial and ongoing consultation with the business areas that ‘own’ parts of the file shares.
• Creation of a SharePoint site destination architecture model.
• Training and awareness sessions for end-users, if required.
• Development of a run sheet for every migration batch.
• Migration and application of retention policies, as required
• End-user support and guidance post-migration.

The activities involved in producing these outcomes are described below.

Detailed design process flow summary

A detailed design process flow diagram provides an overview summary of what is expected to happen before, during and after the migration, as described below.

Initial description and analysis of the network file shares

The first (and perhaps obvious) stage of any migration project is to document what is on the network file shares – the various mapped drives on file servers. This exercise – which may be more difficult than it looks – provides the raw information needed to make decisions.

Only file servers that are used to store content are examined; servers that house line of business applications, even if they stored documents, are usually excluded from this exercise. However, at some point there may or will be a requirement to examine these drives as well when the application is upgraded and/or migrated to another system.

Documenting the network file share environment requires full access to the servers and drives and the ability to export the raw data in readable (likely csv) format. This means that the work will be done either by a network administrator and/or using a third-party tool that has the access required. If a third-party tool is used, care needs to be taken in terms of who will be able to access the information on the drive.

The documentation should include the following for each network file share (one column per dot point):

• The server names and folder paths only down to a level that can be ‘mapped’ (in terms of ownership and use) to business areas (divisions/departments, or perhaps the next business level down if required), or defunct business areas.
  • Note that terms like ‘P drive’ or ‘N drive’ are irrelevant to this exercise, except to record it as the ‘public’ name that the business area uses. Multiple business areas may actually use the same server but refer to ‘their’ part of the drive by the drive letter allocated to them.
  • This detail will be confirmed with the business area in the next stage. It may also include paths that were used by now defunct business areas.
• The business area known to ‘own’ the path content, or the defunct business area.
• A list of all folders and file names (which will give the file types). This part will obviously take up the most room in the csv output file – if it fits at all. Another output format may be required.
• Folder/file sizes.
• Date created
• Date modified.
• Who (AD Security Group or by name) has access to the folders – inherited or unique.
• Does the file share contain or appear to contain any obvious sensitive or personal information (usually obvious from folder or file names).

This information can, potentially, be used as a snapshot of what was on the drives before it was migrated, both to cross-check the migration process but also as an archival record.

At this stage, identifying duplicate (or potentially duplicate) documents is not a high priority as it could get very complicated to work out which version was the ‘master’ or original (even with last modified dates).

Based on this initial documentation, it should be possible to determine:

• The total size of all the content to be migrated.
  • The overall size will impact on how long it may take to migrate the content, and also how long end-users may be affected during the migration (as you don’t want anyone modifying content on the drive after it has been migrated).
  • The size may also determine the method or application used to migrate the content. Keep in mind that the integrity of the metadata on each record (especially date created, date modified, author) needs to be maintained.
• The file types that may not be migrated.
  • For example, downloaded movies or personal photographs. There may also be some executable (exe) files and other unusual content.
• Activity levels (by date modified), which will help determine if content will be migrated as ‘active’ or ‘inactive’ content.
• Any security access controls that may need to be considered as part of the migration.
  • Determining if these controls exists can be time-consuming and may impact on how the migration is undertaken. For example, if a part of the NFS is used by multiple parts of the business or business area and, for whatever reason, access controls are used to restrict access, you could either (a) attempt to replicate the access controls in SharePoint or (b) split the migration into multiple SharePoint sites so that only those people who can see the content on the NFS will see the same content on the SharePoint site.

Creation of a decision tree

After the drive content is reviewed and documented, a basic decision tree will help determine what – in theory – will be migrated where.

Note that these initial high-level decision trees are very similar in most organisations. Each organisation should create its own based on the content uncovered and things like records retention requirements.

The following is an example diagram of an initial decision tree.

Is it worth identifying the ROT?

The effort involved in positively identifying redundant, outdated or trivial (ROT) content – apart from the obvious downloaded movies and user photographs – can become a major exercise and can bog down the project very quickly.

It may be a more efficient use of resources to focus on what needs to be migrated. That is, identifying live, active content (even if some of it is ‘ROT’) moved into SharePoint sites and then applying retention policies to that content.

Inactive content could be moved ‘as is’ to archive, read only, SharePoint sites with Office 365 retention policies applied on libraries in those locations.

Anything that isn’t to be migrated (whether it is ROT or just very old and not required) can be left on the drives and deleted when the NFS is decommissioned.

Consultation with business areas (initial and ongoing)

Once the details of each network file share is established and the decision tree has been agreed, business areas need to be engaged to gain a broad understanding of the following:

• Confirmation of the details found in the analysis, and modification of the listing if it is incorrect.
• Areas of the network drive/s that are active (sometimes in parallel with an existing SharePoint site) and inactive.
• Areas of the drive that do not need to be kept.
• Areas of the drive that don’t have an owner.

The original listing for the drives is then updated.

Consultation with business areas should be an ongoing activity (e.g., regular meetings). It should result in a much more accurate map of the network drive areas and identification of what will go where – to existing or new SharePoint sites, or simply deleted.

A note of caution – the devil is in the detail of network file shares. The lower down the folder structure you go to try to identify duplicates or ROT for example, the more work it may be to migrate them. However counter-intuitive it may seem, it may be much easier in many instances to migrate as much as content possible ‘as is’ and then ‘aim to’ do a clean up post-migration. This can include the application of O365 retention policies to sites or libraries.

Development of a SharePoint destination architecture

Try to avoid mixing active and inactive ‘messy’ NFS content on the same SharePoint site. The benefits of not mixing the two include:

• Ensuring that active sites have a cleaner structure.
• Fewer document libraries on each site.
• More useful search results.
• Messy legacy folder structures are kept out of site on inactive sites.
• Inactive content is subject to retention and disposal actions away from active sites.
• Better end-user engagement.

Try not to confuse end-users too much by changing the way their content re-appears in SharePoint. End-users generally know where the content is and can navigate there reasonably easily.

The following diagram is an example that shows how NFS folder paths might map to SharePoint sites and document libraries.

The following two Microsoft principles are worth noting  here:

An effective site collection consists of groups of individuals and teams that share common goals. A site collection is one where the content is useful to those sharing the site.

This means having as many sites (and libraries) as the business areas need. There is likely to be a one-to-one relationship between the NFS drive/folder path used by ‘lower-level’ business units and SharePoint site document libraries.

A secure site is one that is open to those who need the information, but where information is blocked from those who should not see it.

That is, instead of creating many sites and restricting access to each one, make all sites open (at least read only Visitor access) and restrict only what needs to be restricted on the site. There will of course be some exceptions to this rule, but open access should be the norm.

Post-migration, end-users will want to be able to re-find the information they need easily, and create and share new content in the same (or similar) way as before. If the migrated content is more complicated, users may reject it.

SharePoint training and awareness sessions

A common question is how much training do end-users need to use SharePoint. The simple answer should be – not much. It should be as easy to use as DropBox, or Facebook and no-one ever got training for those applications.

When you migrate content from a network file share to SharePoint, end-users need to know how to use the new environment from the moment they log in. Rather than formal training, the majority of end-users may only need simple guidance on how to find the content that used to be on the drive and how to add and edit new content.

Some suggestions:

• Communicate regularly with the business area and ensure they know who to contact.
• Identify early adopters (who can support the migration) and potential late adopters (who may need additional help).
• As much as possible, use the same names for the site URL and document libraries as on the network file share folders.
• Be conscious of, and let users know about, limits with URL paths. Or, to put it another way, communicate clearly the need to avoid complex folder hierarchies.
• Don’t move content around unnecessarily when migrating from network file shares to SharePoint.
• Don’t introduce new concepts when they didn’t exist before. For example, don’t add content types (e.g., to choose from when a new document is created) or mandatory columns to document libraries when these did not previously exist. Restrict these to new document libraries.
• Be aware that Office 365 retention policies prevent deletion from document libraries. If they could delete before, end-users might react negatively to this ‘new’ restriction.

The following are some points that can be used in conversations and meetings with business areas and end-users to help them ease into the SharePoint world.

• After an initial and very short overview (including perhaps and explanation of how ‘this’ Office 365 differs from their home version), ask end-users to log on to the Office 365 log on screen. Outlook, Word, Excel and PowerPoint should all be visible, as is OneDrive and SharePoint.
• As the focus is on SharePoint, open that application and talk them through the portal. Introduce them to the parts of the portal, including search.
• As their drives will be migrated to SharePoint, use an example site for them to play with and have a look around. Note – ensure that they have a clickable list of the new sites that will contain the migrated drive content.
• Demonstrate how to create a new document and save it, noting how SharePoint and OneDrive are the new default options.
• Demonstrate the Sync option in a document library, and the options available (such as versions) when you click the three dot ellipsis menu. Demonstrate how to Share instead of attach a document.
• Open MS Teams and demonstrate how SharePoint works in that application.

Migration run sheet

It is worth running a number of ‘test migrations’ to test SharePoint sites, to understand the process and outcomes.

A migration run sheet should exist for every batch that is migrated to SharePoint Online. The run sheet can be created in a spreadsheet with following seven dot points as column headings.

• Action (see the dot point steps below – Pre-migration to Post-migration)
• Event / Task (e.g., what will actually happen in a few words)
• Communication method if required (e.g., ‘By email’)
• Responsible (who is responsible for this action)
• Decision Point (up to what point can the task be cancelled)
• Testing Required (mostly no)
• Communications, if required (to whom, in what form)

The following is a list of the actions that will take place for each migration batch.

Pre migration (1 week or more)

• Identify NFS source path/paths and any specific permissions that are required.
• Pre-migration analysis of NFS
• Fix or address any NFS issues

Pre migration (72 hours/3 days in advance)

• Communicate change and migration date and time to affected business area
  • Keep in mind that very large migrations may take several days to complete.
• Communicate change to Change Manager and Service Desk

Pre migration (24 hours in advance)

• Re-communicate change details to affected business area
• Establish SPO destination site or identify existing target SPO site. If the latter, confirm destination site URL.
• Establish or change permissions on the SPO destination site, as required.

Migration

A migration tool (or a third-party with a migration tool) will be needed to move content from the NFS drives/folder paths in batches to SharePoint document libraries.

The time it takes to migrate the content will depend on total size and volume of files to be migrated, and the network bandwidth. The migration process may completely quickly or take several days. Files in the source NFS folder path will continue to be accessible and editable but this should be discouraged as modified content may not be migrated.

The following steps are required after each migration batch has completed.

• Test migration outcome
  • Testing can include object numbers and total size comparisons.
  • Note that some files may not migrate, for whatever reason and this can result in having to re-do the migration. The reasons for the non-migration will need to be identified quickly before the migration is confirmed. In some cases it may be possible to re-migrate just those files after the problem is fixed.
• Decision Gate – OK or NOT.
  • If OK, confirm with business area and ensure they are aware of the SPO site/s URL.
  • If NOT, alert business area (drive remains accessible)
• Make NFS source path/s read-only.
• Delete NFS (at an agreed date)

Applying retention policies

Retention policies can be applied after the content has been migrated. Retention requirements will depend on the content that is migrated and business needs:

• Active content migrated to new libraries or existing SharePoint sites.
  • May not initially have any retention policy applied (as it could prevent the deletion of content in the libraries), or may have retention policies applied only to specific libraries.
  • May also have an O365 site policy, but this would need to be confirmed with the business area.
• Most inactive content migrated to ‘archive’ SharePoint sites.
  • May have site-level Office 365 policies applied, based on when the content was created or modified
  • May have other policies auto-applied (including based on auto-classification (subject to licencing)).
• Some inactive content migrated to ‘archive’ SharePoint sites.
  • May have specific and different library-level policies applied, with a disposition review.

Post migration support

It almost goes without saying that you should provide appropriate support for end-users if you change the way they are expected to work. It requires empathy, patience and understanding, while focussed on a speedy resolution of any issues that arise.

Effective support for end-users has three main elements:

• The ability to show empathy and be patient. For many end-users, this will be the first time they are using SharePoint.
• ‘Mini-training’ and self-empowerment, by talking the end-user through the issue and potential resolutions and/or sending follow up help and guidance or links for further help. This will help them to learn and pass the learning on to others.
• Quick resolutions, to allow the end-user to get on with their job as soon as possible. If a quick resolution is not possible, regular updates and options to allow the end-user to continue working.

Good luck with your migrations!