Office 365/SharePoint Online – Accounts, roles and groups for records and information managers

Organisations have stored and managed records in on-premise versions of SharePoint for many years. Generally speaking, these records had to be managed in – and disposed from – individual site collections.

Office 365 now includes a range of roles, access controls and settings that can be used to manage records across the ecosystem, making it much easier to manage records centrally.

This post describes the suggested user accounts and roles, and the options they provide access to, that can be used by records and information managers to support the management and administration of records in Office 365.

The short version

If you’d like to avoid reading the entire post, the details in summary are:

• User Account: A dedicated ‘records manager admin’ Office 365 user account (separate from a normal end-user account), with its own O365 licence (E3 recommended).
• Admin role: EITHER the ‘Compliance Admin’ role set in the Office 365/Azure admin portals OR the same role in the Security and Compliance portal OR a custom role group (e.g., ‘Records Management’).
  • The Compliance Admin role provides additional options that may not be required, hence the suggestion to create a custom role group.
• Security Group: An Office 365 Security Group that includes the records management admin account (and others).
  • The addition of the O365 Security Group to the Site Collection Admin section of every SharePoint site.

User account

Anyone with any kind of admin access or role in Office 365 should have an admin account, in addition to their normal end-user account.

Admin accounts will usually (but not always) require an Office 365 licence.

Office 365 admin accounts should exist only in the cloud and their naming convention should reflect that difference. For example:

• Normal AD end user account name: username@tenantdomain.org
• Office 365 SharePoint admin account name: O365_ADM_SPO_username@tenantname.onmicrosoft.com

To access the records management features in the Office 365 Security and Compliance portal, records and information managers should have either (a) an appropriate Office 365/Azure admin account (see below) or (b) an ‘admin’ user account. The admin user account name could be something like:

• O365_ADM_Records_username@tenantname.onmicrosoft.com

Office 365 (and Azure) admin role

There are around 35 admin roles in Office 365/Azure, in three broad role types:

• Global admins. This role has unlimited access and can edit any setting.
• Application-based roles such as Dynamics 365 admin, Exchange admin, SharePoint admin, and Teams admin.
• Activity-based roles such as Billing admin, Compliance admin, Helpdesk admin, License admin, Security admin, Service admin, and User admin.

There is NO specific Office 365 admin role for ‘records management’ – the closest in terms of access and functionality required is the ‘Compliance Admin’ role.

Office 365 admin roles are assigned to user accounts either (a) from the individual user account or (b) from the ‘Roles’ section in either the Azure Active Directory admin centre or the Office 365 Admin portal.

Except for the Global Admin role which can access anything, all other roles provide access to specific parts of the Office 365 admin portal or to other admin portals. For example:

• The SharePoint Admin role provides access to the SharePoint admin portal.
• The Exchange admin role provides access to the Exchange admin portal.
• The Security Admin and Compliance admin roles provide access to the Security and Compliance admin portal BUT NOTE that these two roles can also be assigned directly in the Security and Compliance portal which only give access to that portal – it does not give the person access to the Office 365 Admin portal. See below for what this role provides access to.

Users who are assigned and logged on with Office 365 admin roles will see the ‘Admin’ icon in the list of Office 365 apps in their office.com apps listing. If they are granted either the ‘Security Admin’ or the ‘Compliance Admin’ role, they will also see the ‘Compliance’ app as shown below.

Security and compliance portal – records management roles

The Security and Compliance centre (https://protection.office.com) includes a number of sections and options, including several that a records manager should probably have access to or manage:

• Classification
  • Retention labels (published as retention policies)
  • (Two other options specific to security/sensitive information)
• Data loss prevention
  • Policy
• Records management
  • File Plan (if used when creating retention labels)
  • Events (that can be used to trigger retention policies)
  • Dispositions (where disposal is managed)
• Information governance
  • Dashboard
  • Import (PST files and social media content)
  • Archive (for Exchange mailboxes)
  • Retention (create labels and publish them as policies (which replicates the options under ‘Classification’; create retention policies (without the option of a disposition review) for specific SharePoint sites)
• Search
  • Content search
  • Audit logs (up to 90 days)
• eDiscovery
  • eDiscovery (cases, and legal holds)
  • Advanced eDiscovery

All the options above, and the ones listed below, are accessible to anyone assigned the Office 365 ‘Compliance admin’ role. Not all of these may be relevant or required by records managers – and indeed may provide too much access.

• Permissions (to manage roles)
• Supervision
  • Use to define policies that capture email and 3rd-party communications
• Threat management
  • Dashboard, Submissions, Review, Policy
• Mail Flow
  • Dashboard, Message trace
• Data Privacy
  • GDPR Dashboard, Data subject requests
• Data Investigations (new)
• Reports
  • Dashboard, Manage Schedules, Reports for download,
• Service Assurance
  • Four options with information about the Security and Compliance centre.

Because the Office 365 Compliance admin role provides so much access, it may be better to create a custom role group in the Security and Compliance Centre with fewer roles. 

The minimum set of roles most likely to be required by a records manager, and the options they provide access to, are Disposition Management and Retention Management which (together) provide access to the options listed below. The ‘RecordsManagement’ role is redundant because it includes the same last three main dot points listed below that are included with the above two roles. 

• Classification
  • Retention labels (create and publish)
  • Sensitivity labels (create and publish)
  • BUT NOT Sensitive info types
• Records Management
  • File Plan (review)
  • Events (create)
  • Dispositions (review and manage)
• Information Governance
  • Retention (create and publish)
  • BUT NOT Import, Archive, or Dashboard
• Threat management
  • Review
• Mail flow
  • Dashboard
• Service assurance
  • Four options

If records managers need to search and view audit logs and also access the eDiscovery section, it may be better to assign the Compliance Admin role to the records manager (or at least one of them. The portal warns that because the results of these logs could contain sensitive information, the role should be used only by those people with a specific need to know to details.

Organisations will therefore need to consider if records managers should be assigned the Compliance admin role (with more options), or a ‘lesser’ custom role with fewer options created with the Compliance centre directly.

SharePoint-specific roles and access

Security Group to manage SharePoint Site Collections

In addition to the SharePoint Administrator (Office 365 admin) role, every SharePoint site has two key local ‘admin’ role groups:

• Site Collection Administrator (SCA)
• Site Owner

SCA access allows records managers to:

• Access all parts of a SharePoint site (for review purposes, and also to guide Site Owners on best practice records management)
• Apply label-based retention policies to specific lists and libraries
• Export metadata for document libraries before the library content is disposed of as part of a retention policy
• Manage inactive SharePoint sites

Adding the Security Group to the Site Collection Administration section of every SharePoint site

As SCAs can configure settings and access all parts of SharePoint sites, records managers should be in the SCA group on every SharePoint site, but not by name.

Instead, they should be added to a Security Group created in the Office 365 admin portal (under ‘Groups’) or Azure AD. This SG should be added to the SCA group on every site.

The screenshot below shows where and how the Security Group is added to the SCA group in SharePoint:

If the ability to create Office 365 Groups is not controlled, end users will be able to create SharePoint sites either directly from their SharePoint portal or by creating a Team, Plan, or Yammer Group; the O365 Group Owners are added to the SCA group.

Accordingly it may be necessary to retrospectively add the Security Group to SharePoint sites created by end users who create Office 365 or Yammer Groups, or Teams in MS Teams.

Footnotes

Expiry rules for Office 365 Groups and associated content

Organisations that have AD Premium can create expiry rules for Office 365 Groups that can result in the deletion of all aspects of an Office 365 Group, including its SharePoint site, if the Group becomes completely inactive (including no views of any content) within a given period of time.

Generally speaking the action of creating such rules would be restricted to the Azure Admins/Global Admins. In organisations that have AD premium, the records administrators will need to know if an expiry policy is in place OR direct what that policy should be. It is understood that any content in an Office 365 Group’s SharePoint site will not be deleted in this way if the SharePoint site (including via a retention policy applied to the Group) requires the content to be kept for longer.

Auto-application of retention policies

As noted above, records administrators may need to be able to use the ‘auto apply’ option in retention policies based on certain conditions being met. An E3 licence only allows two conditions:

• When the content contains certain (machine identifiable) sensitive information (same as the DLP options)
• When specific words, phrases or other properties can be identified via a keyword query.

An E5 licence allows additional conditions including based on the auto-classification (machine learning/teaching) of content.

Migrating network file shares to SharePoint

Updated 10 November 2019 with additional details based on feedback received.

As part of the implementation of Office 365, organisations may consider migrating content from network file shares (NFS) to SharePoint Online (SPO).

Closing down file shares and moving all the content to a system that can manage records has been an objective for many years. SharePoint can help organisations achieve this goal but it needs to be planned well, and some compromises may be required along the way. For example, you may end up migrating most of the content on drives ‘as is’ – but at least you can subject the content to recordkeeping controls including disposal/disposition.

This post describes the main activities or outcomes associated with migration projects.

Assumptions

For the purpose of this post, it is assumed that:

• The organisation has established a change program for Office 365 including SharePoint Online, and that end-users will be familiar with using SharePoint by the time the file shares are migrated.
• The implementation of Office 365 will includes a sub-project specifically to migrate the file shares to SharePoint Online.
• On-premise Exchange mailboxes and personal ‘home’ drives will be migrated as a separate sub-project but likely at around the same time.

Project artifacts and outcomes

The primary project activities, artifacts or outcomes may include the following:

• Detailed design process flow summary.
• Initial description and analysis of the network file shares that may be migrated.
• Creation of a ‘decision’ tree to help decide what will be migrated.
• Initial and ongoing consultation with the business areas that ‘own’ parts of the file shares.
• Creation of a SharePoint site destination architecture model.
• Training and awareness sessions for end-users, if required.
• Development of a run sheet for every migration batch.
• Migration and application of retention policies, as required
• End-user support and guidance post-migration.

The activities involved in producing these outcomes are described below.

Detailed design process flow summary

A detailed design process flow diagram provides an overview summary of what is expected to happen before, during and after the migration, as described below.

Initial description and analysis of the network file shares

The first (and perhaps obvious) stage of any migration project is to document what is on the network file shares – the various mapped drives on file servers. This exercise – which may be more difficult than it looks – provides the raw information needed to make decisions.

Only file servers that are used to store content are examined; servers that house line of business applications, even if they stored documents, are usually excluded from this exercise. However, at some point there may or will be a requirement to examine these drives as well when the application is upgraded and/or migrated to another system.

Documenting the network file share environment requires full access to the servers and drives and the ability to export the raw data in readable (likely csv) format. This means that the work will be done either by a network administrator and/or using a third-party tool that has the access required. If a third-party tool is used, care needs to be taken in terms of who will be able to access the information on the drive.

The documentation should include the following for each network file share (one column per dot point):

• The server names and folder paths only down to a level that can be ‘mapped’ (in terms of ownership and use) to business areas (divisions/departments, or perhaps the next business level down if required), or defunct business areas.
  • Note that terms like ‘P drive’ or ‘N drive’ are irrelevant to this exercise, except to record it as the ‘public’ name that the business area uses. Multiple business areas may actually use the same server but refer to ‘their’ part of the drive by the drive letter allocated to them.
  • This detail will be confirmed with the business area in the next stage. It may also include paths that were used by now defunct business areas.
• The business area known to ‘own’ the path content, or the defunct business area.
• A list of all folders and file names (which will give the file types). This part will obviously take up the most room in the csv output file – if it fits at all. Another output format may be required.
• Folder/file sizes.
• Date created
• Date modified.
• Who (AD Security Group or by name) has access to the folders – inherited or unique.
• Does the file share contain or appear to contain any obvious sensitive or personal information (usually obvious from folder or file names).

This information can, potentially, be used as a snapshot of what was on the drives before it was migrated, both to cross-check the migration process but also as an archival record.

At this stage, identifying duplicate (or potentially duplicate) documents is not a high priority as it could get very complicated to work out which version was the ‘master’ or original (even with last modified dates).

Based on this initial documentation, it should be possible to determine:

• The total size of all the content to be migrated.
  • The overall size will impact on how long it may take to migrate the content, and also how long end-users may be affected during the migration (as you don’t want anyone modifying content on the drive after it has been migrated).
  • The size may also determine the method or application used to migrate the content. Keep in mind that the integrity of the metadata on each record (especially date created, date modified, author) needs to be maintained.
• The file types that may not be migrated.
  • For example, downloaded movies or personal photographs. There may also be some executable (exe) files and other unusual content.
• Activity levels (by date modified), which will help determine if content will be migrated as ‘active’ or ‘inactive’ content.
• Any security access controls that may need to be considered as part of the migration.
  • Determining if these controls exists can be time-consuming and may impact on how the migration is undertaken. For example, if a part of the NFS is used by multiple parts of the business or business area and, for whatever reason, access controls are used to restrict access, you could either (a) attempt to replicate the access controls in SharePoint or (b) split the migration into multiple SharePoint sites so that only those people who can see the content on the NFS will see the same content on the SharePoint site.

Creation of a decision tree

After the drive content is reviewed and documented, a basic decision tree will help determine what – in theory – will be migrated where.

Note that these initial high-level decision trees are very similar in most organisations. Each organisation should create its own based on the content uncovered and things like records retention requirements.

The following is an example diagram of an initial decision tree.

Is it worth identifying the ROT?

The effort involved in positively identifying redundant, outdated or trivial (ROT) content – apart from the obvious downloaded movies and user photographs – can become a major exercise and can bog down the project very quickly.

It may be a more efficient use of resources to focus on what needs to be migrated. That is, identifying live, active content (even if some of it is ‘ROT’) moved into SharePoint sites and then applying retention policies to that content.

Inactive content could be moved ‘as is’ to archive, read only, SharePoint sites with Office 365 retention policies applied on libraries in those locations.

Anything that isn’t to be migrated (whether it is ROT or just very old and not required) can be left on the drives and deleted when the NFS is decommissioned.

Consultation with business areas (initial and ongoing)

Once the details of each network file share is established and the decision tree has been agreed, business areas need to be engaged to gain a broad understanding of the following:

• Confirmation of the details found in the analysis, and modification of the listing if it is incorrect.
• Areas of the network drive/s that are active (sometimes in parallel with an existing SharePoint site) and inactive.
• Areas of the drive that do not need to be kept.
• Areas of the drive that don’t have an owner.

The original listing for the drives is then updated.

Consultation with business areas should be an ongoing activity (e.g., regular meetings). It should result in a much more accurate map of the network drive areas and identification of what will go where – to existing or new SharePoint sites, or simply deleted.

A note of caution – the devil is in the detail of network file shares. The lower down the folder structure you go to try to identify duplicates or ROT for example, the more work it may be to migrate them. However counter-intuitive it may seem, it may be much easier in many instances to migrate as much as content possible ‘as is’ and then ‘aim to’ do a clean up post-migration. This can include the application of O365 retention policies to sites or libraries.

Development of a SharePoint destination architecture

Try to avoid mixing active and inactive ‘messy’ NFS content on the same SharePoint site. The benefits of not mixing the two include:

• Ensuring that active sites have a cleaner structure.
• Fewer document libraries on each site.
• More useful search results.
• Messy legacy folder structures are kept out of site on inactive sites.
• Inactive content is subject to retention and disposal actions away from active sites.
• Better end-user engagement.

Try not to confuse end-users too much by changing the way their content re-appears in SharePoint. End-users generally know where the content is and can navigate there reasonably easily.

The following diagram is an example that shows how NFS folder paths might map to SharePoint sites and document libraries.

The following two Microsoft principles are worth noting  here:

An effective site collection consists of groups of individuals and teams that share common goals. A site collection is one where the content is useful to those sharing the site.

This means having as many sites (and libraries) as the business areas need. There is likely to be a one-to-one relationship between the NFS drive/folder path used by ‘lower-level’ business units and SharePoint site document libraries.

A secure site is one that is open to those who need the information, but where information is blocked from those who should not see it.

That is, instead of creating many sites and restricting access to each one, make all sites open (at least read only Visitor access) and restrict only what needs to be restricted on the site. There will of course be some exceptions to this rule, but open access should be the norm.

Post-migration, end-users will want to be able to re-find the information they need easily, and create and share new content in the same (or similar) way as before. If the migrated content is more complicated, users may reject it.

SharePoint training and awareness sessions

A common question is how much training do end-users need to use SharePoint. The simple answer should be – not much. It should be as easy to use as DropBox, or Facebook and no-one ever got training for those applications.

When you migrate content from a network file share to SharePoint, end-users need to know how to use the new environment from the moment they log in. Rather than formal training, the majority of end-users may only need simple guidance on how to find the content that used to be on the drive and how to add and edit new content.

Some suggestions:

• Communicate regularly with the business area and ensure they know who to contact.
• Identify early adopters (who can support the migration) and potential late adopters (who may need additional help).
• As much as possible, use the same names for the site URL and document libraries as on the network file share folders.
• Be conscious of, and let users know about, limits with URL paths. Or, to put it another way, communicate clearly the need to avoid complex folder hierarchies.
• Don’t move content around unnecessarily when migrating from network file shares to SharePoint.
• Don’t introduce new concepts when they didn’t exist before. For example, don’t add content types (e.g., to choose from when a new document is created) or mandatory columns to document libraries when these did not previously exist. Restrict these to new document libraries.
• Be aware that Office 365 retention policies prevent deletion from document libraries. If they could delete before, end-users might react negatively to this ‘new’ restriction.

The following are some points that can be used in conversations and meetings with business areas and end-users to help them ease into the SharePoint world.

• After an initial and very short overview (including perhaps and explanation of how ‘this’ Office 365 differs from their home version), ask end-users to log on to the Office 365 log on screen. Outlook, Word, Excel and PowerPoint should all be visible, as is OneDrive and SharePoint.
• As the focus is on SharePoint, open that application and talk them through the portal. Introduce them to the parts of the portal, including search.
• As their drives will be migrated to SharePoint, use an example site for them to play with and have a look around. Note – ensure that they have a clickable list of the new sites that will contain the migrated drive content.
• Demonstrate how to create a new document and save it, noting how SharePoint and OneDrive are the new default options.
• Demonstrate the Sync option in a document library, and the options available (such as versions) when you click the three dot ellipsis menu. Demonstrate how to Share instead of attach a document.
• Open MS Teams and demonstrate how SharePoint works in that application.

Migration run sheet

It is worth running a number of ‘test migrations’ to test SharePoint sites, to understand the process and outcomes.

A migration run sheet should exist for every batch that is migrated to SharePoint Online. The run sheet can be created in a spreadsheet with following seven dot points as column headings.

• Action (see the dot point steps below – Pre-migration to Post-migration)
• Event / Task (e.g., what will actually happen in a few words)
• Communication method if required (e.g., ‘By email’)
• Responsible (who is responsible for this action)
• Decision Point (up to what point can the task be cancelled)
• Testing Required (mostly no)
• Communications, if required (to whom, in what form)

The following is a list of the actions that will take place for each migration batch.

Pre migration (1 week or more)

• Identify NFS source path/paths and any specific permissions that are required.
• Pre-migration analysis of NFS
• Fix or address any NFS issues

Pre migration (72 hours/3 days in advance)

• Communicate change and migration date and time to affected business area
  • Keep in mind that very large migrations may take several days to complete.
• Communicate change to Change Manager and Service Desk

Pre migration (24 hours in advance)

• Re-communicate change details to affected business area
• Establish SPO destination site or identify existing target SPO site. If the latter, confirm destination site URL.
• Establish or change permissions on the SPO destination site, as required.

Migration

A migration tool (or a third-party with a migration tool) will be needed to move content from the NFS drives/folder paths in batches to SharePoint document libraries.

The time it takes to migrate the content will depend on total size and volume of files to be migrated, and the network bandwidth. The migration process may completely quickly or take several days. Files in the source NFS folder path will continue to be accessible and editable but this should be discouraged as modified content may not be migrated.

The following steps are required after each migration batch has completed.

• Test migration outcome
  • Testing can include object numbers and total size comparisons.
  • Note that some files may not migrate, for whatever reason and this can result in having to re-do the migration. The reasons for the non-migration will need to be identified quickly before the migration is confirmed. In some cases it may be possible to re-migrate just those files after the problem is fixed.
• Decision Gate – OK or NOT.
  • If OK, confirm with business area and ensure they are aware of the SPO site/s URL.
  • If NOT, alert business area (drive remains accessible)
• Make NFS source path/s read-only.
• Delete NFS (at an agreed date)

Applying retention policies

Retention policies can be applied after the content has been migrated. Retention requirements will depend on the content that is migrated and business needs:

• Active content migrated to new libraries or existing SharePoint sites.
  • May not initially have any retention policy applied (as it could prevent the deletion of content in the libraries), or may have retention policies applied only to specific libraries.
  • May also have an O365 site policy, but this would need to be confirmed with the business area.
• Most inactive content migrated to ‘archive’ SharePoint sites.
  • May have site-level Office 365 policies applied, based on when the content was created or modified
  • May have other policies auto-applied (including based on auto-classification (subject to licencing)).
• Some inactive content migrated to ‘archive’ SharePoint sites.
  • May have specific and different library-level policies applied, with a disposition review.

Post migration support

It almost goes without saying that you should provide appropriate support for end-users if you change the way they are expected to work. It requires empathy, patience and understanding, while focussed on a speedy resolution of any issues that arise.

Effective support for end-users has three main elements:

• The ability to show empathy and be patient. For many end-users, this will be the first time they are using SharePoint.
• ‘Mini-training’ and self-empowerment, by talking the end-user through the issue and potential resolutions and/or sending follow up help and guidance or links for further help. This will help them to learn and pass the learning on to others.
• Quick resolutions, to allow the end-user to get on with their job as soon as possible. If a quick resolution is not possible, regular updates and options to allow the end-user to continue working.

Good luck with your migrations!