Posted in Classification, Compliance, Electronic records, Governance, Information Management, Office 365, Products and applications, Records management, Retention and disposal, SharePoint Online

Shifting the paradigm for managing records – from EDRMS to Office 365

Computer systems used to to manage electronic documents and records, commonly known as ‘EDRMS’, have been around for at least 20 years.

Many (but not all) of these systems developed from electronic databases originally used to register and manage only paper records, replacing the old paper registers (hence ‘Registries’).

How does an EDRMS work

A common theme with most EDRM systems is that they describe (via metadata) and provide some kind of visual ‘file’ or ‘folder’ structure for digital objects, almost always stored in a linked network file store.

To store records in this way, EDRM systems required end-users to upload a copy of a digital object (document, email, photograph) to a pre-defined digital container, corresponding to a ‘file’ or ‘folder’. The digital file might have be assigned a range of metadata including the classification (business function and activity) or file plan details, title, business owner or area, and retention information.

Once an object was uploaded, end users were required to add metadata about the object, including the object ‘title’ (if it didn’t copy the original title). Additional metadata fields, for example ‘Document Type’, might also be required.

The system recorded the date and time the object was uploaded and who uploaded it. As noted, the system might copy some of the uploaded object’s metadata, for example the default title, date created and author.

The uploaded document then ‘became’ a record, visible ‘within’ a digital container (‘file’) along with other records.

EDRMModel2

EDRM systems had (at least) three weaknesses:

  • End-users were required to upload the records to the EDRMS, and to one correct container (file/folder)
  • The EDRMS contained a copy of a digital object that almost always remained in the original storage location (email, network file share)
  • The EDRMS tended to be based on records as documents (including emails, and sometimes photos). Newly evolving forms of record such as text messages, social media posts and new digital forms were difficult to upload without costly add-ons that didn’t necessarily capture everything

These weaknesses meant that:

  • End users avoided uploading records because it was extra work (uploading and then adding metadata)
  • The EDRMS contained only a percentage of all potential records stored in any location
  • The original copies of records, remained in email and network file shares

There were exceptions to this situation, but most (and very much in the minority in terms of total volume) involved the requirement to meet compliance obligations to capture certain types of records.

The Office 365 model

Microsoft took a different approach with the approach to records management in Office 365.

Instead of centralising the storage of records in one system or location (with the weaknesses described above), records in the Office 365 environment generally remain in their original location (Exchange Online, SharePoint Online, OneDrive for Business, MS Teams), where they are covered by an overarching records management framework.

O365RMModel
The Office 365 model for records management

What this means is that records can be stored in any of the above locations and managed in those locations through (among other things):

  • User types, licences and roles set in the Office 365 admin portal
  • Retention and other controls set in the Office 365 Security and Compliance admin portal/s (the two were split in early January 2020).

How the paradigm shifted

The paradigm has shifted from (a) an attempt to manage records in a single system where not everything is captured and originals remain in place in email and network file shares, to (b) the distributed management of records where originals remain in place (assuming SharePoint and OneDrive are used instead of network file shares and personal drives, and email remain in Exchange) and records are managed through ‘global’ settings.

The new paradigm does not exclude the ability to store (or aim to store) digital records in a single location – SharePoint Online (including for specific compliance reasons), but it provides the opportunity manage records wherever they are and use a range of additional tools to manage content from creation through to disposal.

Why the new paradigm matters

The new paradigm is likely to be counter-intuitive to many records (and other information) managers. Records management training for many years has been focused on the idea of storing and managing records in a central location with specific controls (classification, metadata and retention).

But the reality is that there are now too many digital records, and too many types of digital records, to ever expect these to be all stored in an EDRMS. And, even if only some are, what about all the others? Has a legal subpoena ever been focused only on records stored in the EDRMS?

Plan to manage records

Many organisations have acquired and are implementing Office 365, sometimes at the expense of the traditional EDRMS. It doesn’t take long for end-users to adopt the new technology because it is so easy to use.

Any suggestion that specific records now need to be copied to the EDRMS seems to be counter-intuitive. And yet, that is how some records managers continue to see Office 365 – as yet another source of records to be uploaded to the EDRMS. It is not a viable plan.

Records managers need to be at the forefront of planning for Office 365, in particular managing content across the four primary workloads. Records managers should be able to provide advice on:

  • The architecture of SharePoint Online
  • Controls around the creation of sites, including naming conventions and the ongoing management of sites
  • The structure of SharePoint Online sites, document libraries and metadata in particular
  • The retention model for Exchange Online, SharePoint Online, OneDrive for Business, MS Teams. This includes understanding existing disaster recovery arrangements and potentially replacing them with retention policies.
  • Disposal actions
  • Other compliance obligations

Plan for change

Moving away from the centralised management of records in an EDRMS to a less visible (for end-users) decentralised model, or even implementing Office 365 without any other previous document and records management system, requires careful change management.

End users (and records managers) used to the idea of uploading records to a central EDRMS may find the new ‘invisible’ and decentralised model of recordkeeping unusually simple (to the point of disbelief).

Consequently, additional re-assurance, training and awareness sessions, may be required to demonstrate and confirm how the records are managed in the new environment. There is potential for some ‘push back’ as, although it requires very little end-user effort, it manages more records than ever before, including in ‘personal’ spaces such as mailboxes and personal drives.

IT will also need to be involved as disaster recovery processes, such as backing up email and network file shares, may no longer be required.

For end users who have never had to use an EDRMS, change management activities might focus more on improving awareness and knowledge about how records will be managed in the future, including in ‘personal’ spaces.

 

Posted in Classification, Compliance, Electronic records, Governance, Information Management, Legal, Office 365, Office 365 Groups, Products and applications, Records management, SharePoint Online, Training and education

AI curated chaos or control – the equally valid but opposite ends of the SharePoint spectrum

There are, broadly speaking, two ‘bookend’ options when it comes to creating new SharePoint Online sites and the document libraries in those sites:

  • ‘Controlled’ model: The creation of new sites is restricted to a small group of individuals with admin rights, who also oversee the creation of document libraries and application of metadata. A combination of controlled and manually applied classification and metadata and retention policies are used to access and manage content over time. Artificial intelligence (AI) tools can also be used to manage content.
  • ‘Chaos/uncontrolled’ model: The creation of new sites, including the creation of document libraries is not restricted. AI tools (including auto-classification) and auto-applied retention policies are used to classify, access and manage content over time. This model assumes that any form of random categorisation applied by end users (e.g., library names, metadata) is mostly ignored by AI tools.

From a traditional information governance and records management (ISO 15498/ISO 16175) point of view, the second ‘chaos’ or uncontrolled model option seems to run counter to conventional wisdom and agreed standards.

From a practical point of view, the first ‘control’ model option seems to run counter to common sense given the volume and range of digital information and the difficulty of classifying or categorising information and records correctly.

Which option is better?

Confusingly, perhaps, the answer may be a combination of both.

  • Certain types of more formal records, such as those required for corporate compliance, formal policies, staff files, accounting information not stored in a finance system, property information, and/or product information, is almost certainly going to be better off in a controlled SharePoint sites with pre-defined libraries and metadata. These types of documents are more likely to be subject to records retention requirements and almost certainly may be subject to eDiscovery and legal holds.
  • Other types of less formal records, including ‘working’ documents, chats and conversations may be better off stored in uncontrolled SharePoint sites, including SharePoint sites linked with Office 365 Groups and Teams, and in MS Teams/Outlook. These types of records are less likely to be subject to records retention requirements but may be subject to eDiscovery and legal holds.

Ultimately, the way the organisation needs to implement Office 365, including SharePoint Online and apply retention policies and other options will depend on its need to comply with oversight and legal requirements (including minimum retention periods), and/or its tolerance for risk.

How does this work in Office 365/SharePoint Online?

If both options Organisations need to make a conscious decision to allow both options, and be prepared to manage both.

The key features of Office 365 and SharePoint to allow both options are listed below:

  • Office 365 retention policies apply to all of Exchange Online, all OneDrive for Business accounts, entire sites (invisible to users) or parts of sites (visible to users).
  • Some retention policies may be applied based on the auto-classification of records, subject to review.
  • The creation of SharePoint sites is either controlled (requested and provisioned) or uncontrolled (created by end users) via either (a) ‘Create sites’ in the end-user SharePoint portal or (b) when a new Team is created in MS Teams.
  • All sites, including Office 365 Group/Team sites are reviewed regularly for activity and inactive sites with no content of value deleted.
  • All controlled sites are assigned either an invisible retention policy or individual visible retention policies (with disposal review), depending on their content.
  • All uncontrolled sites are assigned an invisible retention policy. Uncontrolled and inactive sites with content are also made read only.

Features of controlled and uncontrolled SharePoint sites

SharePoint Online is quite different from older versions of the application and those who dismiss it based on previous experience should consider having another look as a lot has changed in the past couple of years.

SharePoint Online allows the creation of sites that contain important content that needs to be controlled of managed as records, as well as sites created and managed entirely by end-users. And, as an added bonus, all the content is stored in the one place, not in multiple locations (network drives, email servers, EDRM system, etc).

The elements that make up both types of sites, as well as ‘informational’ sites, are described below:

  • Controlled sites
    • Where the organisation’s official records are stored and managed.
    • Created by SharePoint Administrators.
    • More formal in nature, containing the official records.
    • Structure decided by business areas – for example, document libraries using agreed naming conventions.
    • Use of Content Types and site column or local library metadata to define the content.
    • Application of Office 365 retention policies to entire sites or individual document libraries, with disposal reviews. Auto-classification is less likely to be required as the content has already been structured as required.
  • Uncontrolled sites
    • Usually based on end-user created Office 365 Groups or MS Teams.
    • Where ‘working documents’ are created and managed, with the emphasis on allowing end-users collaborate and communicate easily and effectively – and move content to formal sites when required.
    • Created by end-users but naming monitored by SharePoint administrators (or using rules).
    • Informal in nature, used for working documents (effectively replacing personal and network file shares, and other unapproved systems).
    • A fluid structure for document libraries, driven by end-user requirements (not imposed by others).
    • Little if any use of Content Types or metadata.
    • Retention based on Group activity (E5 licences), otherwise based on Office 365 site retention policies and/or auto-classification options.
    • No disposal reviews – content is deleted after a given period of time.
  • Informative
    • Communication sites (e.g., ‘intranet’)
    • Used to publish information to the organisation

Things to watch out for

It is largely true that if you give people an option, someone is bound to try it, sooner or later, especially if it says ‘Create site’, ‘Create team’, or ‘Create group’. Early adopters learn quickly and can just as quickly abandon something that provides no benefit. 

In a ‘free for all’ SharePoint environment, where end-users can create new sites, teams or groups (both of the latter have a SharePoint site), the most likely issues will include:

  • Sites with names that are very similar to ones that already exist, created because the end-user didn’t know another existed (it may not be obvious) or didn’t like the name.
  • Sites with names that make no sense (including common acronyms) or are just ‘wrong’ or contrary to preferred naming conventions.
  • Sites used to create and store content that really should be stored in a more formal site or, conversely, doesn’t belong in the organisation’s official information systems (e.g., photos of someone’s wedding).

All of these issues require some general rules about the creation of new sites (or Office 365 Groups or Teams or Yammer Groups), including suggested naming.

Global and SharePoint admins can monitor the environment and fix issues when they arise rather than wielding a big stick.

What’s great about it

You can have the best of both worlds with SharePoint Online.

  • Keep formal official records in ‘formal’ sites with controlled structures and metadata.
  • Allow end-users to get on with creating, collaborating, sharing (one copy, not attachments), chatting, on any device.

If your communications and change management are good, end-users will soon learn how much fun it can be to use Teams, or access their content from File Explorer (or both!), without having to having to be trained how to save records. All they need to know is how to use the ‘Move’ option to move the final version of records to a formal site.

The foundation of any compliance program is knowing where all of your data lives and then classifying, labeling, and governing it appropriately.

Posted in Governance, Information Management, Office 365, Products and applications, Records management

How many Office 365 Admins do you really need?

Organisations implementing Office 365 probably already have some form of IT support. This support may range from single individuals through to complex IT ‘shops’, or it may be outsourced.

This post:

  • Describes the change from traditional on-premise IT admin to the new world of Office 365 administration, and the admins that are needed.
  • Highlights that Office 365 includes new applications or features that were never part of the on-premise environment and so new skills may be required.
  • Describes the various Office 365 admin roles, and why these should having distinct naming conventions and be cloud-only.
  • Recommends that admin roles are defined in Office 365 governance documentation.

Transitioning from on-premise IT to Office 365

Traditional on-premise IT tended to be focussed on network infrastructure (servers and networks) and Exchange. Depending on the size of the organisation, it might have a single IT specialist or a more complex IT structure with a range of administrators managing different aspects of the network.

When Office 365 is implemented some on-premise infrastructure will likely remain and need to be supported, while some of the previous on-premise administration skills will ‘move’ to the respective online version, for example Exchange to Exchange Online.

Office 365 includes a range of applications some of which are likely to be new to previous on-premise IT administrators. For example:

  • Home or personal drives will be migrated to OneDrive for Business
  • All or some parts of the network file shares will be migrated to SharePoint Online
  • MS Teams is likely to replace any existing messaging apps such as Skype
  • Retention policies can be applied across the core Office 365 workloads
  • Security policies can be implemented
  • Yammer may be used as the enterprise social networking platform
  • Audit logs and eDiscovery are now available

Each of these (and other options) require new skills that may not exist in the existing IT support structures.

Additionally, some traditional IT activities such as backup don’t exist in Office 365. There is likely to be a tendency to try to re-create those on-premise solutions when other options (such as retention management) may be just as effective.

Office 365 Admin roles – Cloud only

Office 365 is entirely based in the Microsoft cloud environment. Office 365 admin roles have no access to any on-premise environment.

Accordingly, key Office 365 admin roles (Exchange, SharePoint/OneDrive, MS Teams, Security and Compliance) should exist only in the cloud and be named accordingly.

Examples of cloud-only admin account names:

  • ADM_O365_GA_username@tenantname.onmicrosoft.com (Global Admin)
  • ADM_O365_EXO_username@tenantname.onmicrosoft.com (Exchange Admin)
  • ADM_O365_SPO_username@tenantname.onmicrosoft.com (SharePoint Admin)
  • ADM_O365_SEC_username@tenantname.onmicrosoft.com (Security Admin)

A person with global records management responsibility might also need elevated privileges. The account could be:

  • ADM_O365_REC_username@tenantname.onmicrosoft.com (Records Admin)

There are a couple of exceptions to this model:

  • Reader admin. Office 365 includes various ‘reader’ admin accounts that give an account read-only access to things like the Office 365 Message Centre only. It may be acceptable to assign reader admin to a standard user account.
  • Yammer Admin. Yammer admin has limited funcionality. A Yammer admin sees additional options via the cog/gear icon but otherwise has no elevated privileges. Therefore, a Yammer admin could be assigned to a standard user account.

In every case, every Admin role must have the user name included; there should be no generic admin accounts, ever.

Primary Admin role – Global Administrators

The highest level admin role in Office 365 is the Global Administrator (GA). GAs have access to everything across Office 365 in the same way that an on-premise administrator had access to everything.

A common mistake in organisations that are new to Office 365 is to assign a GA role to a standard user account, often the person or individuals with similar privileges in the on-premise environment.

Microsoft recommend the following:

  • There should only be two or three GAs, no more than five (maximum) with very strong passwords. GA admin activity should be minimal. 
  • Use multi-factor authentication for GA accounts.
  • Only sign in with the dedicated global administrator accounts when carrying out tasks that require global administrator privileges.
  • Carry out other Office 365 administration (Exchange, SharePoint etc) with other administration roles (see below).

Other Office 365 admin roles

A range of other admin roles can be assigned in Office 365. The primary admin roles are for Exchange, SharePoint and MS Teams. Secondary admin roles (that may be performed by the GA in smaller IT shops) are the global reader, help desk and service support admins, and user admin.

O365AdminRoles.JPG

In addition to the primary roles above, there are also a range of other admin roles that can be assigned, including the following (plus at least the same amount again).

  • Dynamics 365 admin
  • Groups admin
  • Kaizala admin
  • Office apps admin
  • PowerBI admin
  • Power Platform admin
  • Search admin
  • Cloud device admin
  • Intune admin
  • Licence admin
  • Password admin
  • Billing admin
  • Compliance admin
  • Security admin

The need for each of these admins will depend on the size and structure of the organisation.

So, who does what?

Despite the multiplicity of admin roles, the reality is that most organisations will only have the following and create and assign other admin roles (e.g., Licence Admin, Help Desk admin, Billing Admin) or access (e.g., Global Reader), as required.

  • Global Admins (2 – 5)
  • Exchange Admin
  • SharePoint Admin
  • MS Teams Admin
  • Security/Compliance Admin

There is likely to be some overlap between these roles, especially in relation to the creation and management of Office 365 Groups (if their creation, and therefore also the creation of new Teams in MS Teams, is controlled) and the creation and implementation of retention policies from the Security and Compliance admin centre, as shown in the diagram below.

O365AdminOverlap.JPG

Governance documentation

However the admin roles are configured, these should be defined in the organisation’s Office 365 governance documentation which should define, for each role:

  • Naming convention
  • Responsibilities
  • Name of actual person with that role
Posted in Digital preservation, Disasters, Electronic records, Governance, Information Management, OneDrive for Business, Records management, Retention and disposal

Managing the retention of content stored in OneDrive for Business accounts

The methods available to manage the retention of content stored by end-users in their Office 365 OneDrive for Business (ODfB) accounts are not always well understood.

Organisations may initially default (in their thinking) to backing up the content because that’s what was always done in the past. A change of thinking may be required.

This post:

  • Explains some of the key differences between ‘home drives’ and ODfB accounts.
  • Highlights the need for organisations to understand their business requirements for retention of ‘personal’ content, and not assume traditional backup methods are the only option.
  • Also highlights the need for organisations to understand the potential risks (and potentially unnecessary additional costs) associated with backing up Office 365 content.
  • Describes two simple options for the retention of content stored in ODfB accounts.
  • Suggests that organisations can probably use a combination of a single Office 365 retention policy and a change to the storage retention period for inactive accounts, instead of backups to achieve the same outcome.

What are ‘home drives’?

In many organisations, home drives are usually a dedicated area on a network file share designed to allow end-users to store ‘working’ documents and ‘personal’ content.

Using the network file shares for home drives ensures that the content stored in them is backed up as part of standard disaster recovery processes while the user is still active (for disaster recovery and to recover deleted items) and still accessible (as an ‘archive’) after they leave the organisation.

In some organisations, home drives may instead be an area on the user’s computer (C drive). Any content stored on local computers is not backed up.

Generally speaking, home drives – whether in the NFS or on the user’s computer, are not accessible once the end-user leaves the office. This has given rise to the fairly regular use of USB storage devices or uncontrolled, internet-based, file storage systems such as DropBox.

How is ODfB different from home drives?

In organisations that implement Office 365, ODfB is the replacement for ‘home’ or ‘personal’ drives.

Although they offer similar functionality for end-users (in terms of the ability to access the content from File Explorer), ODfB accounts are fundamentally different in several ways.

  • The content can be accessed on almost any device. No VPN is required.
  • With Windows 10 devices, the content is synced to and can be accessed via File Explorer. This makes ODfB an almost identical replacement for existing home drives in terms of look and feel, and functionality (plus even more functionality, such as the ability to share directly).
  • There is no accessible back up – Microsoft is entirely responsible for disaster recovery. If organisations want to back up ODfB accounts from Office 365, they will need to acquire a third-party product. The ability to establish retention for the content (last two dot points below) may make the need for back up redundant.
  • There is a 90 day Recycle Bin accessible via the browser-based interface. This allows end-users to restore the content they deleted themselves within that time-frame.
  • Organisations can set a storage retention period that will apply once the end-user leaves and their account is deactivated.
  • Organisations can also set a retention policy that will prevent the deletion of content while the user remains active.

Both the last two options are the subject of this post.

Access to and retention of home drives vs ODfB accounts

In many organisations, the content stored by end-users in their home drives is considered to be ‘private’ to them, despite the system being owned by the organisation.

While they can be accessed easily by network administrators with elevated privileges, it is not uncommon (often for audit purposes) for IT to have to seek special approval from someone senior to access the content of a home drive either while the end-user is still employed or after they have left. In these cases, IT will either access the active drive or request the back up tape to restore the content. 

The content in home drives, when backed up, remains as long as the backup media is accessible.

In Office 365, Global Administrators can access the ODfB accounts of any active user. They do this by going to the Office 365 Admin portal and, under the ‘Users’ section, clicking the end-user account name and then going to the ‘OneDrive’ tab where the option to ‘Get access to files’ is displayed’. Any access to ODfB accounts, by anyone (including Global Admins) is recorded in the audit logs.

[Note: At at January 2020, the old ‘My Sites’ options in SharePoint still exists. These options allow the Global Admins or SharePoint Admins to assign someone, or a Security Group, as a Secondary Admin for all ODfB accounts. This option is largely redundant because Global Admins can access the content anyway.]

The default retention period for ODfB content is 30 days after the end user’s account is disabled.

What exactly are you trying to achieve?

As noted, there are some fundamental differences between ‘home drives’ and ODfB.

Consequently, organisations ideally should re-examine their business requirements for access to and the retention of ‘personal content’ both while the user account is active and when it is made inactive, and not assume that old backup option remain valid.

For example, consider the use of backup tapes:

  • The primary purpose of backup tapes is to support disaster recovery. These made sense when IT owned the servers, but it makes less sense when Microsoft own them and are responsible for disaster recovery. Is Microsoft’s disaster recovery capability sufficient or suitable?
  • Backup tapes were (and still are) often used as a type of ‘archive’, allowing organisations to recover data from active and inactive home drives for an indefinite period of time.

The bottom line is – what business outcome/s do you want? Generally, these are likely to be:

  • The ability to recover content stored on personal drives after a disaster (not just when the end-user has deleted something).
  • The ability to access and retain content while the user is active or after they become inactive.

An additional business requirement might be to reduce the use of ‘home drives’ for business related content.

Retention options for content stored in ODfB

ODfB ships with two default retention options:

  • Recycle Bin. Any ODfB content deleted by an end-user goes to the Recycle Bin for 90 days.
  • Inactive content retention. When an end-user accounts is deactivated, the content remains accessible for a default period of 30 days.

Neither of these two options on their own, without modification, is likely to meet business requirements to achieve some form of back-up equivalent capability and the ability to access content in ODfB for a period of time.

It is likely that most business requirements (to replace backups) will be met instead via a combination of the following:

  • Creating a single Office 365 retention policy applied to all ODfB accounts that prevents content in those accounts from being deleted for a given period of time.
  • Extending the default retention period for the content in deactivated accounts from 30 days to a much longer period, for example 7 years.

Office 365 Retention Policy

To ensure that content is kept (and accessible, even after being ‘deleted’ by the user) while the user is active, and after they leave, (a) create a single Retention Policy in the Office 365 Compliance portal, ‘Information Governance’ section and (b) apply it to all ODfB accounts by choosing ‘https://tenantname-mysharepoint.com’.

ODfBRetentionPolicy.JPG

Once published, the retention policy creates a ‘Preservation Hold library’, visible only to the Global Admins, that stores any content that is modified or deleted by the end-user during the retention period.

At the end of the retention period, the content in the Preservation Hold library and anything else that has reached the end of the retention period is sent to the Recycle Bin where it is kept for 90 days before being permanently deleted.

ODfBPresHoldLib.JPG

This type of retention policy effectively replaces the need for a back up of home drives, provided the organisation:

  • Accepts the risk that Microsoft may not be able to recover all or some of the content in the case of a disaster. Note that this risk also applies to Exchange, SharePoint and MS Teams content.
  • Understands that, if it decides to attempt to back up ODfB, restoring from back up may not be as simple as it used to be when the organisation owned and managed the relevant servers. What, exactly, will you back up to, and how will you read the data?

ODfB Storage Retention

The second retention option relates to the ODfB accounts of departed users, or inactive accounts.

ODfB includes the option to retain files in ODfB for a specific period of time after the end-user account is deactivated. This is set in the ODfB Admin portal under ‘Storage’.

ODfBStorage.JPG

At the end of the period of time specified, the content is sent to the Recycle Bin after which it is deleted permanently.

Summary

Many organisations are likely to approach the retention of ODfB content in the same way they did for home drive content, by considering backup options first, often ‘because that’s what we’ve always done’.

Organisations implementing Office 365 should:

  • Define their business requirements for the retention of home drive/ODfB content
  • Examine, understand and consider if retention options in Office 365 result in the same outcome
  • Understand the potential risks of relying on Microsoft to provide a reliable service including in a disaster situation
  • Understand the complexity (and risks) of backing up (and recovering) content from Office 365.

In many cases, retention options in Office 365 may provide the required outcome at a much lower cost.