Posted in Classification, Governance, Information Management, Microsoft 365, Records management, Retention and disposal

Classifying records in Microsoft 365

There are three main options in Microsoft 365 to apply recordkeeping classification terms to (some) records:

  • Metadata columns added to SharePoint sites, including those added to Content Types and/or added directly to document libraries.
  • Taxonomy terms stored in the central Term Store, including those added as site columns and added to site content types and/or added directly to document libraries. The only difference with the first option is that with the Term Store the classification terms are stored and managed centrally and are therefore available to every SharePoint site.
  • Retention labels that: (a) ‘map’ to classification terms; (b) are linked with a File Plan that includes the classification terms; (c) are either the same as (a) or (b) and are used in with a Document Understanding Model in SharePoint Syntex; or (d) the same as (a) or (b) and used with conjunction with Trainable Classifiers.

The first two options can only be applied to content stored in SharePoint. Retention labels may be applied to emails and OneDrive content. None of the three options can be applied to Teams chats. Also note that there is no connection between the SharePoint Term Store and the File Plan, both of which can be used to store classification terms.

This post:

  • Defines the meaning of classification from a recordkeeping point of view.
  • Describes each of the above options and their limits.
  • Discusses the requirement to classify records and other options in Microsoft 365.

What is classification?

Humans are natural-born classifiers. We see it in the way we store cutlery or linen, or other household items or personal records.

Business records also need some form of classification. But what does that mean? The 2002 version of the records management standard ISO 15489, defines classification as:

‘the systematic identification and arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules represented in a classification system’. (ISO 15489.1 2017 clause 3.5).

The standard also states (4.2.1) that a classification scheme based on business activities, along with a records disposition authority and a security and access classification scheme, were the principal instruments used in records management operations.

The classification of records in business is important to establish their context and help finding them.

Microsoft 365 includes various options to apply classification terms to records.

Metadata columns in SharePoint

The simplest way to classify records stored in SharePoint document libraries is to either create site columns containing the classification terms and add those columns to document libraries, or create them directly in those libraries.

Benefits

Adding site or library columns is relatively simple. As classification terms are usually in the form of a (hierarchical) list, it is simple to add one choice or lookup column for function and another for activities.

A lookup column can bring across a value from another column when an item is selected; for example, if the look up list places ‘Accounting’ (Activity) in the same list row as ‘Financial Management’, selection of ‘Accounting’ will bring across ‘Financial Management’ as a separate (linked) column.

Default values (or even one value) can be set meaning that records added to a library (that only contains records with those classification terms) can be assigned the same classification terms each time without user intervention.

Negatives

SharePoint choice or lookup columns do not allow for hierarchical views or values to be displayed from the list view so the context for the classification terms may not be obvious unless both function and activity are listed.

The Term Store

The Term Store, also known as the Managed Metadata Service (MMS) has existed in SharePoint as a option to create and centrally manage classification and taxonomy terms in SharePoint only for at least a decade.

In 2020, access to the Term Store was re-located from its previous location (https://tenantname-admin.sharepoint.com/_layouts/15/TermStoreManager.aspx) to the SharePoint Online admin portal under the ‘Content Services’ section:

The location of the Term Store in the SharePoint admin center

Organisations can create multiple sets of taxonomies or ‘term groups’ (e.g, ‘BCS’ or ‘People’) within the Term Store. Each Term Group consists of the following:

  • Term Sets. These generally could map to a business function. Each Term Set has a name and description, and four tabs with the following information: (a) General: Owner, Stakeholders, Contact, Unique ID (GUID); (b) Usage settings: Submission policy, Available for tagging, Sort order; (c) Navigation: Use term set for site navigation or faceted navigation – both disabled by default; (d) Advanced: Translation options, custom properties.
  • Terms. These generally could map to an activity. Each Term has a name and three tabs: (a) General: Language, translation, synonyms and description; (b) Usage settings: Available for tagging, Member of (Term Set), Unique ID (GUID); (c) Advanced: Shared custom properties, Local custom properties.

In the example below, the Term Set (function) of ‘Community Relations’ has three Terms (activities).

Once they have been created in the Term Store, term set or terms can be added to a SharePoint site, either as a new site or local library/list column, as shown in the two screenshots below:

First, create a new column and select Managed Metadata

Then scroll down to Term Set Settings and choose the term set to be used.

Once added as a site column, the new column may be added to a Content Type that is added to a library, or directly on the library or list.

A Term Store-based column added to a library via a Content Type.

Benefits

The primary benefit using the central Term Store terms via a Managed Metadata column is that the Term Store is the ‘master’ classification scheme providing consistency in classification terms for all SharePoint sites.

As we will see below, Term Store terms may be used to help with the application of retention labels (which themselves may ‘map’ to classification terms in a function/activity-based retention schedule).

Negatives

Using metadata terms from the Term Store is almost identical with using a choice or lookup column. The only real difference is that the Term Store provides a ‘master’ and consistent list of classification (and other) terms.

Term store classification terms, including in Content Types, may only be used on a minority of SharePoint sites.

Additionally:

  • It is not possible to select a Term Set (e.g., the function level), only a Term within a Term Set.
  • Only the selected classification Term appears in the library metadata, without the parent Term Set or visual hierarchy reference to that Term Set – see screenshot below. Technically only that Term is searchable. It is not possible to view a global listing of all records classified according to function and activity.
  • If multiple choices are allowed, a record may be classified according to more than one Term. This may cause issues with grouping, sorting or filtering the content of a library in views.
How the Term appears in the library column.

As we will see below, there is no connection between the classification Terms in Term Sets and the categorisation options available when creating new retention labels via a File Plan. ‘Business Function’ or ‘Category’ choices in the File Plan do not connect with the Term Store.

Term Store terms and Content Types can only be used to classify content stored in SharePoint.

Retention labels

Retention labels in Microsoft 365 can be used in an indirect way to classify records in SharePoint, email and OneDrive because they can be ‘mapped’ to classification elements.

For example, a label may be based on the following elements:

  • Function: Financial Management
  • Activity: Accounting
  • Description: Accounting records
  • Retention: 7 years

Every retention label contains the following options:

  • Name. The name can provide simple details of the classification, for example: ‘Financial Management Accounting – 7 years’.
  • Description for users. This can be the full wording of the retention class.
  • Description for admins. This can contain details of how to apply or interpret the class, if required.
  • Retention settings (e.g., 7 years after date created/modified or label applied).

Benefits

Where the classification terms map to a retention class, the process of applying a retention label to an individual record, email or OneDrive content could potentially be seen as classifying those records against the classification scheme.

The Data Classification section in the Microsoft 365 Compliance portal provides an overview of the volume of records in SharePoint, OneDrive or Exchange that have a specific retention class:

Negatives

Not every record in every SharePoint document library may be subject to a retention label. Many records (for example in Teams-based SharePoint sites) may be subject to a ‘back end’ retention policy applied to the entire site (which creates a Preservation Hold library).

A retention label applied to a record doesn’t actually add any classification terms to the record.

Retention labels don’t map in any way to Term Store classification terms, except in SharePoint Syntex – see below (but this only applies to SharePoint content).

Retention labels/File Plan combination

The File Plan option (Records Management > File Plan, requires E5 licences) can also be used to add classification terms to a retention label as shown in the screenshot below. Note that there is no link with the Term Store.

Benefits

Records (including emails) that have been assigned a retention label could, in theory, be regarded as having been classified in this way because the label contains (or references) the classification terms.

Negatives

When applied to content in SharePoint, OneDrive or Exchange, retention labels linked with the File Plan do not show the File Plan classification terms. It may be possible to write a script that displays all records with the terms from the File Plan, but it may be easier to do this using the Data Classification option described above.

Retention labels/SharePoint Syntex combination

SharePoint Syntex provides a way to apply retention labels to records, stored in SharePoint, that have been identified through the Document Understanding Model process.

Benefits

As can be seen in the screenshot above, each new DU model allows similar types of records (in the example above, ‘Statements of Work’) to be associated with a new or existing Content Type that can include a Term Store Term – for SharePoint records only – and a retention label. This provides three types of ‘classification’:

  • Grouping by record type (e.g., Statement of Work, Invoice)
  • Linking (of sorts) between the records ‘classified’ in this way and a Term Store term added as a metadata column to the Content Type.
  • Assigning of a retention label. This provides the same form of retention label-based classification described above.

Furthermore, if the Extraction option is also used, data extracted to SharePoint columns can be based on choices listed in the Term Store metadata.

Negatives

SharePoint Syntex only works for records – and only those records that have some form of consistency – stored in SharePoint.

Retention labels/trainable classifiers combination

Trainable classifiers are another way that could be used to identify related records and apply a retention label to those records. Microsoft 365 includes six ‘out of the box’ trainable classifiers that will not be of much value to records managers for the classification of records:

  • Source code
  • Harassment
  • Profanity
  • Threat
  • Resume
  • Offensive language (to be deprecated)

The creation of new trainable classifiers requires an E5 licence; they are created through the Data Classification area of the Microsoft 365 Compliance admin portal. Machine Learning is used to identify related records to create the trainable classifiers.

Once created, a retention label may be auto-applied to content stored in SharePoint or Exchange mailboxes using the classifier.

The option to auto-apply a label based on a trainable classifier.

Benefits

The primary outcome (from a recordkeeping classification point of view) of using trainable classifiers is the application of a retention label to content stored in SharePoint and Exchange mailboxes. It can also be used to apply a sensitivity label to that content.

Negatives

It is unlikely that every record will be classified according to every classification option.

Trainable classifiers only work with SharePoint and Exchange mailboxes.

Classifying records per workload

The options are summarised below for each main workload:

  • SharePoint: Use local site or library columns, Term Store terms or retention labels (mapped to a File Plan as necessary), applied manually or automatically, including via SharePoint Syntex or trainable classifiers.
  • Exchange mailboxes: The only feasible option to classify these records is to manually or auto-apply retention labels that are mapped to a classification, including a trainable classifer.
  • OneDrive: Manually or auto-apply retention labels mapped to a classification.
  • Teams. It is not possible to classify Teams chats with the options available.

Is classification necessary?

The classification model described in ISO 15489 and other standards was based on the idea that records would be stored in a central recordkeeping system where they would be subject to and tagged by the terms contained a classification scheme, often applied at the aggregation level (e.g., a file).

Microsoft 365 is not a recordkeeping system but a collection of multiple applications that may create or capture records, primarily in Exchange mailboxes, SharePoint, OneDrive and MS Teams (and also Yammer).

There is no central option to classify records in the recordkeeping sense. The closest options are:

  • The grouping of records in SharePoint sites (and Teams, each of which has a SharePoint site) and libraries that map to business functions and activities.
  • The use of metadata, either terms set in the central Term Store or created in local sites/libraries, to ‘classify’ individual records (including emails) stored in SharePoint document libraries. Each item in the library might have a default classification, or could be classified differently.
  • The use of retention labels that ‘map’ to function/activity pairs in a records disposal authority/schedule. These may be applied, manually or automatically, to content stored in SharePoint, OneDrive and Exchange mailboxes.

Neither of the above may apply, or be applied consistently, to all SharePoint sites, Exchange mailboxes, OneDrive accounts. And neither can be applied to Teams chats.

A different approach to this problem is required, one that likely will likely involve greater use of Artificial Intelligence (AI) and Machine Learning (ML) methods to identify and enable the grouping of records, and provide visualisations of the records so-classified.

Image: Werribee Mansion, Victoria, Australia stairwell (Andrew Warland photo)

Posted in Compliance, Electronic records, Governance, Information Management, Legal, Microsoft 365, Microsoft Teams, Products and applications, Records management, Retention and disposal, SharePoint Online, Yammer

All the ways SharePoint sites can be created

SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.

While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.

Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.

So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.

Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.

The following is a list of the main ways a person can create a SharePoint site. Have I missed any?

1. Via a PowerShell script

As described in the Microsoft docs web page ‘Create SharePoint Online sites and add users with PowerShell‘. The script is based on a csv file (‘sitecollections.csv) and looks something like the following (see the link for more details):

Import-Csv C:\users\MyAlias\desktop\SiteCollections.csv | ForEach-Object {New-SPOSite -Owner $_.Owner -StorageQuota $_.StorageQuota -Url $_.Url -NoWait -ResourceQuota $_.ResourceQuota -Template $_.Template -TimeZoneID $_.TimeZoneID -Title $_.Name}

This option also allows the administrator to provision new SharePoint sites.

2. Via the SharePoint Admin portal (+ Create)

This option allows the creation of three main types of sites: modern team sites (Team site),
communication sites, and non-Microsoft 365 Group-linked sites (Other options).

3. By creating a Microsoft 365 Group

Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.


Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.

The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.

Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.

4. By an end-user creating a new Team in MS Teams

Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.

Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an
Exchange Online mailbox that is not visible in Outlook.

5. By creating a Private Channel in MS Teams

If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.

For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.

A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.

6. By the Teams Admin creating a new Team

The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.

As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.

7. From the end-user SharePoint portal (+ Create site)

If not disabled, end users can create a new SharePoint site by clicking on ‘+ Create site’ from the SharePoint portal – https://tenantname.sharepoint.com/_layouts/15/sharepoint.aspx

This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.

It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.

8. From OneDrive for Business as a ‘shared library’

This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.

9. When a new Plan is created in Planner

If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.

10. When a new Yammer community is created

End users with access to Yammer can click on ‘Create a Community’ from Yammer.

To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’

Why have Microsoft allowed this?

It’s a smarter way to manage access.

Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.

The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.

But, some of the decisions don’t seem to make sense.

  • Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
  • Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
  • Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
  • And why does a new Plan create a SharePoint site? For what purpose?

Perhaps there is a reason for it. It’s just not clear.

Posted in Compliance, Governance, Information Management, Microsoft 365, Records management, Retention and disposal

Recordkeeping roles and permissions in Microsoft 365

(Updated 3 September 2020 with reference to customised admin roles)

Microsoft 365 is a cloud-based collaboration and content system that includes a wide range of functionality to create, capture and manage records, primarily in SharePoint Online but also in OneDrive for Business, Exchange Online and in MS Teams. 

This post outlines the roles and permissions required by records managers to manage records in Microsoft 365.

Whether all the roles and permissions will be granted may depend on a number of factors including technical competence, security and risk. Where they are not granted, records managers will need to ensure that the relevant IT resources can and will set up and manage the recordkeeping functionality as required.

Azure AD/Microsoft 365 Admin Center roles

There are around 50 roles that can be assigned to individuals in the Microsoft 365 admin center or the Azure Admin portal (which includes 11 more roles).

These roles may be grouped as follows:

  • Admin. For example, Global Admin and the Admins for Exchange Online, MS Teams, and SharePoint Online/OneDrive for Business.
  • Security and Compliance. For example, Security Admin, Compliance Admin, Compliance Data Admin
  • Identity management. For example, Authentication Admin, Guest Inviter, Licence Admin, Password Admin, User Admin
  • Device management. For example, InTune Admin, Printer Admin
  • Reader. For example, Global Reader, Message Center Reader, Reports Reader, Security Reader

There is no specific ‘records manager’ role in Microsoft 365. The closest in terms of functionality is the Compliance admin role that includes several several sub-roles including ‘RecordManagement’, ‘Disposition Management’ and ‘Retention Management’. Alternatively, a custom role may be created with those (and a couple of other) sub-roles, thereby restricted access to only the sub-roles that are specific to or required by records managers.

In addition to the role and sub-roles required to access the Compliance portal and carry out records management activities, records managers should also be assigned the Global Reader and Reports Reader roles so they can access and view the various dashboards on the Microsoft 365 admin center:

Example dashboard

Compliance admin portal roles and sub-roles

The Compliance admin portal (https://compliance.microsoft.com) includes the following sections that are all relevant for records managers:

  • Reports (dashboard)
  • Audit logs. These cover the entire Microsoft 365 environment, kept for only 3 months (E3) or 12 months (E5).
  • Content Search (effectively eDiscovery)
  • Information Governance (where retention labels and retention policies are created and managed)
  • Records Management (which is essentially an extended set of IG functionality, including auto-application of labels, available to E5 licence holders, and disposition management)

Access to the Compliance admin portal is restricted to the Global Admins and Compliance Admin and Compliance Data Admin roles. These two roles include various sub-roles (including sub-roles that are not relevant to records management) that are described in considerable detail in this Microsoft page ‘Permissions in the Security & Compliance center‘.

The sub-roles that are most relevant to records managers are:

  • RecordManagement (required to manage and dispose record content)
  • Retention Management (required to create retention labels)
  • Audit Logs
  • View-Only Audit Logs (audit logs cannot be modified)
  • Disposition Management (required to manage disposition)
  • Compliance Search (required to conduct a global ‘case’ search of anything anywhere in the Microsoft 365 platform, including ‘personal’ mailboxes and 1:1 Teams chats)
  • Case Management
  • Hold

It is recommended that records managers – or select individuals with higher compliance responsibilities, be assigned either to one of the two Compliance Admin roles, or a custom role group with just the sub-roles listed above. This will enable records managers to access the Compliance portal to create, apply and manage records retention policies. They will also have access to the audit logs and content search options. 

Note: The ‘Audit logs’ sub-role is actually assigned via a role group in the Exchange Online admin portal under the Permissions section. The three key roles in this section that contain these sub-roles are ‘Organisation Management’, ‘Compliance Management’ and ‘Records Management’. As the first two contain a very long list of sub-roles, it is recommended that the records manager/s be added to the ‘Records Management’ role group that includes the ‘Audit logs’ and ‘Retention Management’ sub-roles.

SharePoint Admin roles

From an admin point of view, there are essentially three SharePoint admin roles:

  • SharePoint administrator. This person has access to the SharePoint admin portal, manages the settings, creates and provisions new sites, and monitors the environment. They are usually also responsible for troubleshooting issues and may have some responsibility for development (including scripts) and customisations or integrations. Subject to the size and complexity of the environment, a records manager with good technical skills, including being an EDRM system admin, may be able to take on the role of SharePoint admin with some training. In most cases, however, this is likely to remain a specialised IT role.
  • Site Collection Administrator. This role sits between the SharePoint Admin and the Site Owner role and provides ‘back-end’ access to the SharePoint site. Generally speaking, the SharePoint Admin will always be a Site Collection Administrator, ideally added via an AD Security Group. If records managers are added to this AD Security Group, and that Group is added to the Site Collection Admin section of every SharePoint, they will have the ability to access every site (with all access and actions recorded in the audit logs). This access can be revoked on individual sites if necessary. 
  • SharePoint Site Owner. The person assigned to this role will usually be someone working in the business area or group responsible for day to day management of the site. Records managers should not be Site Owners as this suggests that the records managers have day to day responsibility for managing the site (creating libraries for example).

Other factors to consider

Any content stored in OneDrive for Business accounts, Exchange mailboxes and MS Teams will remain accessible via a Content Search as long as it exists. If no retention policy has been applied to these workloads and the end-users deletes that content, there is no way to retrieve the deleted content after minimum periods (90 days for ODfB, 14 days for Exchange mailbox content).

The OneDrive portal includes a Storage section that determines how long the content will be retained after the account becomes inactive. This is separate from any retention policy that may be applied to the accounts via the Compliance portal. Records managers should understand these two elements (retention and storage period).

The Exchange Online admin portal includes a number of legacy recordkeeping elements, in particular the Messaging Records Management (MRM) policies in the compliance/retention policies section. Records managers do not need to be assigned the role of Exchange Online admin but need to engage with the admins regarding the application of Microsoft 365 retention policies. While it is possible to apply label-based retention policies to Exchange mailboxes, including advanced auto-application with E5 licences, in practice it may be much simpler to apply a few broad non-label retention policies to mailboxes.

Screenshot of the MRM policy area

The MS Teams admin portal does not include any recordkeeping settings or elements. However, the records manager should discuss and determine suitable retention requirements for both 1:1 chats and channel chats with the relevant admin. These are created and added via the Compliance admin portal. It is not possible to apply a label-based retention policy to Teams chats, accordingly there is (currently) no disposition review record of what was destroyed.

Conclusion

Records managers need an appropriate level of access to the Microsoft 365 ecosystem to manage records that have been created, captured and stored across the system. The following is recommended:

  • Global reader and Reports reader. These two roles provide read-only access to dashboards in the Microsoft 365 Admin portal, allowing records managers to review volumes and activities in the various workloads. 

  • Compliance admin or a customised role group. The role group allows the creation and management of records retention policies and dispositions. It also provides access to audit logs and global content searches. 

  • SharePoint admin (optional). This role would be suitable for a records manager with the required level of technical competence to manage SharePoint. 

  • SharePoint Site Collection Admin (via a Security Group). This role allows records managers to access every site where the Security Group has been added to the Site Collection Admin group. 

 

 

Posted in Compliance, Data Loss Prevention - DLP, Electronic records, Governance, Information Classification, Information Management, Information Security, Legal, Microsoft 365, Records management, Retention and disposal, SharePoint Online

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

  • Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
  • Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.

Posted in Classification, Compliance, Electronic records, Governance, Information Management, Microsoft Teams, Office 365 Groups, Products and applications, Records management, Retention and disposal, SharePoint Online

Managing MS Teams chat as records

(The image above was part of collector’s album issued in 1930 by Echte Wagner, a German margarine company. Source – https://flashbak.com/wonderful-futuristic-visions-of-germany-by-artists-in-1930-381451/)

On 19 May 2020, Tony Redmond published a very helpful article on the Office 365 for IT Pros website titled ‘Using Teams Compliance Data for eDiscovery‘.

In the article, Tony describes where and how the chat component of MS Teams is stored and how this might affect eDiscovery.

He also makes the important point that, while it may be possible ‘… to backup Teams by copying the compliance records in an Exchange Online backup … you’ll never be able to restore those items into Teams.’ In other words, it is better to leave the data where it was created – in MS Teams. The post explains why this is the case. 

This post draws on the article to describe the factors involving in managing the chat element of Teams as records. It notes that, while is is technically possible to export chat messages (in various ways), it may be much better from a recordkeeping point of view to leave them where they are and subject them to a retention policy.

Two key reasons for leaving chat messages in place are: (a) chat messages are dynamic and may not always be a static ‘thread’, and (b) the chat messages exported from Exchange may not contain the full content of the message. 

What is a Teams chat?

A Teams chat consists of one or more electronic messages with at least two participants – a sender and a receiver. 

msteamschatteams-1

There are two types of chat message in MS Chat:

  • One-to-one/one-to-many ‘chat’ (top icon above).
  • Channel-based Teams chat (second icon above). Teams chat is visible to all members of the Team. Within channel-based chats, a person may create a private channel which is visible only the person who created the private channel and any participants.

Messages created in both options could be regarded as records because they may contain evidence of business activity.

However, one-to-one chats have no logical subject or grouping. Only the chat messages in Team channel chat are connected through the context of the Team/channel. 

Where and how are chat messages stored?

The following is a summary from Tony Redmond’s article.

Chat messages are stored directly in the backend Azure Cosmos DB (part of the so-called Microsoft 365 ‘substrate’). The version in the database is the complete version of the chat message.

The messages are then copied, less some content elements (for example: reactions, audio records, code snippets), to a hidden folder in either (a) end-user mailboxes for one-to-one chat and private channel chats, and (b) M365 Group mailboxes for channel chat.

Most export options, including the export option in Content Search and eDiscovery, draw their content from the mailbox version of the message. This has potential implications for the completeness of the chat message as a record.

Additionally, any export can only be a ‘point in time’ record unless there is absolute certainty that all chat on a given subject have ceased. 

Implications for records managers

In addition to the concerns about a chat message (or exports of them) being complete, there are (at least) two other points relating to the management of chat messages as records in MS Teams:

  • Knowing if chat messages on any given subject exist. 
  • Applying an appropriate retention policy. 

Both of these points are discussed below. 

Finding content

The primary way to locate content on any given subject across Microsoft 365 is via the Content Search option in the Compliance portal. Access to the Content Search option is likely to be restricted. So, if records managers do not have access, they will need to ask the Global Administrators to conduct a search. 

Content searches are very powerful. This Microsoft article, ‘Keyword queries and search conditions for Microsoft 365‘ provides details on how to search. The screenshot below shows an example of a very simple keyword queries with the option to add conditions. 

ContentSearchQuery

Searches can be configured to find content in any or all of the following locations:

  • Users, Groups, Teams
    • Exchange email
    • Office 365 group email
    • Skype for Business
    • Teams messages [the copy in the mailbox]
    • To-Do
    • Sway
    • Forms
  • SharePoint
    • SharePoint sites
    • OneDrive accounts
    • Office 365 group sites
    • Teams sites
  • Exchange public folders

Note that content search only works on the copies of the items in the Exchange mailboxes, not the backend Teams database. Accordingly, there is some potential for it to not find some content.

Both the mailbox content and the content discovered by the search can be exported.  Teams chat messages can be exported as individual items or as a PST – but note that these message may exclude the elements as described in Tony’s article.

The problem with exporting the content either this way or via other export options (such as described in this post ‘How to export MS Teams chat to html (for backup)‘ (using the Microsoft Graph API) is that it creates a single ‘point in time’ copy; additional content could be added at any time and, if the chats were subject to a retention policy, they may already be deleted.

Managing chat messages ‘in place’ as records

As any export only creates a ‘point in time’ version, it makes more sense from a recordkeeping point of view to leave the chat messages where they are and apply one or more retention policies to ensure the records are preserved. 

Ideally, organisations that may create or capture records on a given subject will have taken the time to establish a way for users to do this, including through the creation of a dedicated Microsoft 365 Group with an associated SharePoint site and Team in MS Teams. 

For example, if there is a requirement to store all records relating to COVID-19, it would make sense (at the very least) to create a Microsoft 365 Group with that name; this will create: (a) a linked mailbox accessible by all members of the Group, (b) a SharePoint site with the same name, and (c) a Team in MS Teams. All of the content – emails, documents, chat, is linked via the same (subject) Group. 

This model makes it easier to aggregate ‘like’ information and apply a single retention policy. It assumes there is (or will be) some degree of control over the creation of Teams (or very good communication to users) to prevent the creation of random Teams, Groups and SharePoint sites – AND to ensure that end-users chat about a given subject within a Team channel, not in one-to-one chat. 

What retention period should be applied to chat messages?

The retention period applied to either one-to-one or Team channel messages will depend largely on the organisation’s business or regulatory requirements to keep records. There are two potential models. 

The simplest model is to have a single retention policy for one-to-one chats, and a separate retention policy for all Teams channel chats.

As one-to-one chats are stored in the mailboxes of chat participants, it makes sense to retain the chat content for as long as the mailboxes. However, some organisations may seek to minimise the use of chat and have a much reduced retention period – even as little as a few days. 

The creation and application of retention policies to Teams channel chat may require additional considerations. For example:

  • As every Team is based on a Microsoft Group that has its own SharePoint site, it is probably a good idea to establish Teams based on subjects that logically map to a retention class. For example, if ‘customer correspondence’ needs to be kept for a minimum 5 years, and there is a Group/SharePoint site/Team for that subject, then all the content should have the same retention policy – although the Group mailbox and SharePoint site may have a policy applied to the Group, with a separate (but same retention period) applied to the Team. 
  • There may be a number of Teams that contain trivial content that does not need to be retained as records. These Teams could be subject to a specific implicit policy that deletes content after a given period – say 3 years. 

In all cases, there is a requirement to plan for retention for records across all the Microsoft 365 workloads. 

What happens to chat messages at the end of a retention period?

At the end of a Microsoft 365 retention policy period, both the mailbox version and the database version of the Teams chat message are deleted. To paraphrase Tony’s article, the Exchange Managed Folder Assistant removes expired records from mailboxes. Those deletions are synchronized back to Teams, which then removes the real messages from the backend database.

No record is kept of this deletion action except in the audit logs. Accordingly, if there is a requirement to keep a record of what was destroyed, this will need to be factored in to whatever retention policy is created. 

 

In place vs in place management of records in Microsoft 365

The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’. This post explains the difference between the two ‘in place’ options. In brief: The Microsoft ‘in place’ model […]

Posted in Governance, Information Management, Microsoft Teams, Office 365, Office 365 Groups, Products and applications, SharePoint Online

What happens when you create a Team in MS Teams

On 27 March 2020 I asked, via Twitter, whether organisations that rolled out MS Teams will wonder in the future who created all the random (and randomly-named) SharePoint sites.

20200414_122632

The reason for this question was because many organisations, scrambling to establish ways for staff to work from home, decided to make use of MS Teams in their (often newly implemented) Office 365 suite of apps.

I have seen multiple organisations since late 2019 ask ‘who created all those SharePoint sites?’ when they reviewed the list. The current COVID-19 work-from-home situation will only make this situation ‘worse’ and, without effective oversight or controls, result in the creation of multiple uncontrolled SharePoint sites.

Unlike other products like Zoom, Whatsapp, Facetime and Skype, however, MS Teams is not a standalone product, but a core element in the Microsoft Office 365 ecosystem.

The key point is this – every Team in MS Teams has a linked SharePoint site (and an Exchange mailbox, where all the chat content is stored). You can’t disable these options.

What happens if you create a Team in MS Teams?

The good thing about the one-to-one chat element of MS Teams is that it’s relatively intuitive and easy to use, including on the mobile app. You only need to tell users it’s like Skype or Whatsapp, but for internal user only, and most pick it up quickly.

The Teams part of MS Teams is not quite as intuitive, but early adopters generally understand the basic concepts – that a Team has members, and you can have multiple chat channels for each Team.

Once end-users understand how a Team works (and this can take some time because one-to-one chat can include multiple people), they might notice this option at the bottom left of the app:

JoinCreateTeam

Creating a new team sounds like a great idea, so end-users may try:

JoinCreateTeam2

My guess is that end-users are more likely to want to ‘build a team from scratch’ as shown below, because the second option doesn’t really make sense.

JoinCreateTeam3

There is a good chance they will want the Team to be ‘Private’, although may not fully understand what this means. A Public Team sounds like a Yammer Group (or Community).

JoinCreateTeam4

So far, so good, the end-user can give the Team any name they like:

JoinCreateTeam5

At the bottom of the naming screen is the option to ‘Create’. The end-user is then invited to add members to their new Team. This seems a fairly obvious step, and they can add whoever they want. New members are by default ‘Members’ but they can be changed to ‘Owners’ if necessary. There is no control over this process.

JoinCreateTeam6

The new team now appears on the left-hand menu of MS Teams:

JoinCreateTeam7

The new team opens at the default ‘General’ channel.

On the main part of the Team, the following options are offered:

  • Along the top, ‘Posts’, ‘Files’, ‘Wiki’ and a + to add more applications. (Hint – the ‘Files’ option points to the SharePoint site that has been created behind the scenes).
  • Across the middle, three options to ‘Add more people’, ‘Create more channels’ and ‘Open the FAQ’
  • At the bottom, the option to ‘Start a new conversation’ with various other options including the ‘Meet now’ video option.

The end-user can now get on with chatting, sharing files, and adding apps to do other things.

But what else has happened?

As noted above, the ‘Files’ tab in the General channel gives a clue to the existence of the connected SharePoint site. End-users may not care terribly much about this, for them it provides the option to create, upload, share and collaborate on files.

A new Office 365 Group is created

But before we get to the SharePoint site, it’s important to understand the one-to-one relationship between a Team in MS Teams and an Office 365 Group. If you do not know what an Office 365 Group is, please read this Microsoft guidance on Office 365 Groups.

In very simple terms:

  • Every new Team in MS Teams creates a new Office 365 Group.
  • The Owner of the Office 365 Group is the Owner of the team; the members of the Group are the Members of the team, as added by the person who created the Team.

The new Office 365 Group appears in the list of Groups in the Office 365 Admin portal, as shown below. Access to this part of the Admin portal is normally restricted to Global Admins (who would normally be responsible for creating other types of AD Groups, such as Security Groups and Distribution Lists.

A new Exchange mailbox has been created

Note that the process has also created an Exchange mailbox with a Group email address. The new Exchange mailbox will now appear in the Outlook client of everyone in the Team – something they are unlikely to notice.

JoinCreateTeam8

As noted above, all the chat messages in the Team are stored in a hidden folder in the Exchange mailbox for the Team.

A new SharePoint site has been created

If we go across to the SharePoint Admin portal, which is normally restricted to Global Admins and SharePoint Admins, we can see that a new SharePoint site has been created, and is owned by the ‘Group owners’.

JoinCreateTeam9

The SharePoint Admin has had no involvement in the creation, naming, or structure of this new site. And, just to add another factor, the SharePoint Admin cannot access the site – see below.

The Team owner may not realise it, but they now have a SharePoint site. The new site’s ‘Documents’ library appears in the ‘Files’ tab as shown below.

JoinCreateTeam11

And, just to add a confusing element, the site includes the invitation (at the bottom left) to create a new Team!

JoinCreateTeam10

As noted above the SharePoint Admin can ‘see’ that this site exists in the list of sites but cannot actually access it. The Global Admin, on the other hand, can access it.

JoinCreateTeam12

So the person responsible for managing SharePoint across the organisation cannot access the SharePoint site, which is not a good thing from an information governance point of view.

The reason they cannot access the site is because they were not added to the Site Collection Admin Group when the site was created. And, just to make it a bit more confusing, the ‘Users and Permissions’ section of Site Settings, where the ‘Site collection administrators’ section is found (see screenshot below), does not appear in Office 365 Group-based SharePoint sites.

SPOSiteSettings

So, how does the SharePoint Admin get access to this site to configure and manage it? There are two ways:

  • The Global Admin can go to /_layouts/15/mngsiteadmin.aspx (after the site name URL) and add them (or a Security Group with them in it) there.
  • The SharePoint Admin can click on the site details in the SharePoint admin portal and add him/herself as an Owner. This puts them in the Site Collection Admin section along with the Group Owner.

Summary

This post began with a simple question – if organisations allow end-users to create Teams to work from home, how will they manage all the SharePoint sites that are created through the process described above?

There is no one answer to this question but it’s worth understanding exactly what happens – and what else is created (including Planner) – when a Team is created. Organisations seem to go one of two ways:

  • Let end users create Teams and deal with the consequences later, including attempts at auto-classification and retention policy application across the various elements of the new Office 365 Group – mailbox, SharePoint site, Team chat. This is the Microsoft default and the preference of many organisations that are don’t have compliance issues or can accept the risks of uncontrolled information stores.
  • Control the creation of Teams, but make any controlled process as easy as possible for end-users to keep them working quickly, and manage the content in mailboxes, SharePoint and Teams proactively. While not the preferred option, it will help with the management of corporate information down the track.