Posted in Compliance, Exchange Online, In Place Records, Information Management, Records management, SharePoint Online

In place vs in place management of records in Microsoft 365

The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.

This post explains the difference between the two ‘in place’ options. In brief:

  • The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
  • The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.

The Microsoft in-place model

SharePoint

The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.

Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:

  • The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
  • Mandatory record metadata.
  • Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
  • Unique computer-generated record identifiers for each record, regardless of where that record is stored.
  • The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
  • The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
  • Not permitting modification of certain metadata fields.
  • The capability to support multiple renditions of a record.
  • The capability to increment versions of records when filing.
    Linking the record metadata to the record so that it can be accessed for display, export.
  • Enforcement of data integrity, referential integrity, and relational integrity.

Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):

Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.

The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’

The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’

Exchange

A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.

One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.

Archive mailboxes, including ‘auto-expanding archives’, also exist in Exchange Online. In the Exchange Online archiving service description, it is noted that:

Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.

The new ‘in place’ model

A newer form of in-place records management has appeared with Microsoft 365.

Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.

The newer model of in place records management is based on the assumptions that:

  • It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
  • Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
  • The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.

Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.

The relationship between Microsoft’s ‘in place’ focussed records management, and managing everything (including records) in place.

In place co-existence

Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.

Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021

Posted in In Place Records, Records management, Retention and disposal

Setting retention labels on folders in SharePoint document libraries

A common question asked by many organisations is whether Microsoft 365 (M365) retention policies – labels in particular – can be applied to folders in SharePoint document libraries so the content in those folders will have the same label.

The quick answer is yes, but it is a manual process and – for all its perceived benefits – is likely to be more of an administrative and support burden and not worth the effort. Folders should NOT be thought of as the replacement for ‘files’ (aggregations of individual records), but more like dividers in a lever arch (= the document library).

This post describes how labels can be applied to and work with folders, including in SharePoint sites linked with Teams. It also suggests alternative options.

How retention labels are applied to a library

Retention labels are created in the M365 Compliance portal under either the Information Governance > Labels or Records Management > File Plan sections.

Where labels are created in the Information Governance section of the Compliance portal

Labels created in the Compliance portal do not do anything once created; they must be applied to content in various ways to make them work. This includes by:

  • Publishing one or more labels as part of a retention policy to various locations including SharePoint sites, Exchange mailboxes and OneDrive (but not Teams – see screenshot of available locations below). In this scenario, each label will be visible to – and selectable by – end-users.
  • Auto-applying them to the same locations based on various options, including (for E5) trainable classifiers
  • Adding them to Content Types used in SharePoint Syntex.
Locations where labels can be published

Publishing labels to SharePoint sites

When one or more labels are published to SharePoint sites they don’t do anything until they are ‘manually’ enabled through one of the following options:

  • On each individual document library via the library settings option ‘Apply labels to items in this list or library’ (see screenshot below)
  • On each individual folder in a library (via the information panel, see screenshot below)
  • On each individual object in the library (also via the information panel)

Applying a label to the library

A label can be applied to the entire document library via Library Settings, as shown in the screenshot below.

Note that the ‘None’ option is shorter if no labels have been published here

If the drop-down option is set to ‘None’, and there are no options to choose from, it means that no labels have been published to this SharePoint site.

If labels exist, they will appear in the drop down list (below the default ‘None’). Note that only one label can be set as the default for the library. If the check box ‘Apply label to existing items in the library’ is selected, this will apply the label to all existing items. It will also likely override any existing label that may have been applied.

When the retention label has been applied to a library, the label only applies to the non-folder objects stored in the library as can be seen in the option below. That is, the retention label is NOT applied to the folders by default.

Document library folders without retention labels

The retention label can be seen when the folder is opened:

Content inside a folder, with retention labels applied

Applying a retention label to a folder or document

It is also possible to apply a retention label to a folder or object stored in the folder via the information panel, even when a default library label has been set, as shown below. This can be done on each individual folder including, for Teams-based sites, each folder that maps to a channel.

When applied to a folder in this way, any content stored in the folder will inherit that retention label.

Documents stored inside the June folder have inherited the folder’s label

If a default label has already been applied at the library level, the folder-based label will replace it, although in testing this, one of the original default labels wasn’t replaced automatically as shown below, but could be manually changed via the information panel.

Implications for Teams-based records (Files)

Every Team in MS Teams has an associated SharePoint site linked with the underlying Microsoft 365 Group.

Every non-private channel in the Team maps to a folder in the Documents library of the SharePoint site as can be seen in the two screenshots below. (Every private channel has a separate SharePoint site that would be covered by a separate retention policy).

The Team’s channels
Four channel-linked folders in a Teams-based SharePoint site

Keep in mind that retention labels remove the ability to delete objects stored in the library (including via the ‘Files’ tab in a Team). If end-users are working in Teams, this could be annoying and potentially put them off using Teams. However, end-users can remove the label by navigating to the SharePoint site and removing the label via the Information panel.

Why folder-based retention labels may not be a good idea

The default options to apply retention labels to content stored in SharePoint document libraries are:

  • By applying them at the library level. This can apply the label to all existing (and future) content stored in the library but does not apply to folders.
  • Through the auto-application of labels.
  • Via SharePoint Syntex using labels on Content Types.

Applying retention labels to individual folders in a document library is a manual-intensive process, one that may be a waste of time given the potential number of libraries that can exist and the ease with which they can be removed by end users.

Additionally, applying retention labels to the channel-linked folders of Teams may be pointless if end users:

  • Store documents at the same level as the channel-linked folders; that is, ‘above’ the folder structure.
  • Create new folders via a synced library or SharePoint. These folders are not linked to channels.
  • Create new libraries in the SharePoint site.

Keep it simple

It is very easy to deliberately or inadvertently establish over-complicated retention settings for content stored in SharePoint, especially as there is currently no simple way to see what label has been applied where.

Given the retention period linked with retention policies generally, there is a good chance that the person who applied the labels may not be around when the retention period expires, or to keep an eye on what has been applied or changed over time.

The best retention intentions may be overruled by practical necessity.

The best retention model, in my opinion, is a simple one that does not get in the way of end-users but ensures that records will be kept for a minimum period required. So, instead of applying retention labels to folders, especially on Teams-based SharePoint site libraries, it is recommended to:

  • Start by trying to avoid mixing content with different retention periods in the same SharePoint site or Team, or document library. That will make it easier to manage the retention outcomes. (If you can’t avoid mixing content, you may need to use auto-application of labels including via Syntex or trainable classifiers).
  • Use ‘back-end’ safety net retention policies applied to all SharePoint sites. This ensures a minimum retention period and does not get in the way of end-user activities.
  • Use retention labels on site libraries where more granular retention is required. Ideally, apply them as the default to all the content in a single document library (including the default library for all Teams-based SharePoint sites) and – preferably – only apply the labels when the content is inactive and the library can be made read only, to protect the records from that point.
  • Only use multiple labels on folders when (a) all the labels applied to the site relate to the same function/activity pair or subject matter, and (b) the content is largely inactive. Ideally, avoid folder-based retention to avoid complication in the future.

Posted in Products and applications, SharePoint Online

Modern vs classic sites in SharePoint Online?

SharePoint has a long, twenty-year legacy. Over that period, organisations and third-party developers have developed and implemented a range of solutions and options across SharePoint. Many of these have been in place for a long time.

The arrival of the new look, ‘modern’ SharePoint Online (SPO) in the last five years has created a dilemma for organisations – migrate to SPO and lose the older functionality, or retain some of the functionality by creating ‘classic’ sites in SPO?

This post was inspired from discussions with a number of organisations that were either creating or using classic site templates in SPO – sometimes as the default option – or indicated a resistance to moving to SPO because of the ‘loss of functionality’.

What are modern sites in SPO?

The default options to create in SharePoint Online are either (a) modern team sites linked with a Microsoft 365 (M365) Group (most common) or (b) communication sites.

The ‘Create a site’ options in the SharePoint admin portal. These are just two of several ways to create new sites.

Modern team sites linked with M365 Groups are created:

  • From the SP Admin portal (first option ‘Team site’ – above)
  • By end users from their SharePoint portal (if that feature is not disabled)
  • When a new Team is created via MS Teams or the MS Teams admin portal
  • When a new Group is created from Outlook or from the Microsoft 365 admin portal
  • When a new ‘shared folder’ is created from OneDrive

Modern team and communication sites are almost identical in terms of core functionality. The primary visible difference is that: (a) team sites have the navigation bar at the left and are usually used to store documents and other digital objects; whereas (b) communication sites have the navigation bar along the top (which can be made into a mega menu easily) and are usually used to store information on pages. They may also store documents in document libraries.

The default page web parts are the same on both; for more information about web parts, see this Microsoft guidance ‘Using web parts on SharePoint pages‘.

For more information about the modern experience see this Microsoft page ‘Guide to the modern experience in SharePoint‘.

The PowerShell script to create a new Group-based site uses the command ‘New-SPOSiteGroup’, which creates both the Group, the SharePoint site, and a Group-based mailbox in Exchange Online.

Note that M365 Group-based sites use the template ‘GROUP#0’.

What are classic sites in SPO?

Other, ‘classic’ SharePoint site templates can be created from the ‘Other Options’ section of the SP Admin portal.

‘Other options’ section of the Create Site section of the SP admin portal

The options, as grouped, are as follows. Note that for all of them, the SharePoint 2013 experience (look and feel) is used.

  • Collaboration: Team site (classic experience), Developer Site, Project Site, Community Site
  • Enterprise: Document Center, eDiscovery Center, Records Center, Team Site – SharePoint Online configuration, Business Intelligence Center, Compliance Policy Center, Enterprise Search Center, My Site Host, Community Portal, Basic Search Center, Visio Process Repository
  • Publishing: Publishing Portal, Enterprise Wiki
  • Custom: (if a custom template exists)

SharePoint sites, including using classic templates, can also be created using PowerShell and the ‘New-SPOSite’ command (with additional configuration details, see this Microsoft docs page). The PowerShell command will needs to include the template to be used:

SharePoint site templates

The PowerShell script to create a new classic site is slightly different from a modern Group-based siste, using ‘New-SPOSite’:

Why create SP sites using the classic templates?

Organisations may have genuine technical reasons to want (or need) to create and continue to create some SharePoint Online sites using one of the classic templates. For example:

  • They may have many sites that were created in the legacy on-premise environment. Upgrading these sites will take time, so it may be easier simply to migrate from one to another ‘as is’.
  • A site or sites may have integration points with other systems that require the older functionality.

Classic sites should not normally be created simply because of loss of (or changed) legacy functionality (e.g., a preference for older style web parts), or simply a preference to continue to use older functionality.

As much as possible, organisations should aim to create modern sites. In some – or many – cases, this may require the organisation to adopt the new modern options.

Classic to modern functionality changes

Examples of broader-level functionality that has largely been replaced with new functionality, or may be deprecated in the future, include:

  • Publishing sites were often used for customised intranets, often with multiple levels of sub-sites. These have been replaced by communication sites and hub sites.
  • Publishing features, including features sometimes used for navigation purposes and linked with terms in the Term Store were a common ‘classic’ option. These have been replaced by new mega menu functionality.
  • Older web parts on pages have been replaced by modern web parts.
  • The Document Center and Records Center templates are more or less now superseded by in-place records management functionality, leaving records where they were created or captured.
  • The Business Intelligence Center has mostly been replaced by PowerBI or similar tools and interfaces.
  • Community Sites and Community Portals have been replaced by communication sites, Teams and Yammer.
  • The need for sub-sites has been replaced by creating one or more sites as hub sites and linking other sites to the hub.

Many of these classic options have been around for at least a decade, making it sometimes hard to move to the newer options.

Functionality differences between classic and modern sites

Possibly the most common of the classic sites that continue to be created, however, are classic team sites (‘STS#0’). In some cases, these site templates are the default option. It is not clear why these continue to be created when the STS03 (non M365 Group) template has the same basic functionality:

  • The same Site Settings.
  • The same ways to add a page, library or list (from Site Contents or from the gear icon – Add an app).
  • The same library and list settings (including the ‘Information management policy settings’).

One key difference is the look and feel, something that may become less of an issue with Teams becoming the primary UI to SharePoint.

It is still possible on classic (STS#0) sites to create classic style pages using classic web parts, as can be seen in the screenshot below from a classic (STS#0) site:

What’s the problem with creating new classic sites?

As noted above, however, just because it remains possible to create and use classic functionality isn’t a reason to keep it in face of newer and often better options – especially when end-users won’t even see the site.

From my own experience, anything that remains in an older version of any technology becomes (or is likely to become) technology debt.

It may be appropriate – temporarily – to migrate some older on-premise sites to the classic template until they can be upgraded, but there is no valid reason in my opinion to create new classic sites as a default option for all new sites.

What do you think? Do you continue to create classic sites?

Featured image: A classic VW Combi.

Posted in Information Management, Records management, SharePoint Online

Managing digital photos as records in SharePoint Online

Photographs (or, for that matter, any visual work) can be more powerful as a record of something than a written record. Some of the iconic events in history are only known from the visual record.

Photos (and videos) are ubiquitous in the digital age. They are captured in multiple ways and stored on a range of media including computers, portable drives, mobile devices and the cloud (including online storage and social media).

Digital photos can be easily changed or ‘photoshopped’, to use the more common term.

But records management requires us to ensure the authenticity, reliability and integrity of records. How can we ensure this for photographic records?

This post examines how SharePoint can be used to manage (some) photos as records.

What is a digital photo?

For the purpose of this post, a digital photo is any image that is captured and stored electronically as bits in a recognised image format. It excludes digital photos (including scanned or digitised paper records) that are embedded in PDF files.

To quote from the Wikipedia article titled ‘Digital Image‘, a digital photo is:

‘… an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively. Depending on whether the image resolution is fixed, it may be of vector or raster type. By itself, the term ‘digital image’ usually refers to raster images or bitmapped images (as opposed to vector images).’

According to the definition, a digital photo is a collection of bits set out in a map that a computer interprets to display it on a screen. For more detail on bitmaps, see this Microsoft page ‘Bitmaps‘.

When to use SharePoint to store digital photos

Almost any digital object that can be stored electronically can be captured in SharePoint. But, just because they can doesn’t mean they should.

It is important to keep in mind that SharePoint may be not be the most suitable option for storing digital photographs in general. The volume, size and intended usage of the photos are all likely to influence the decision to use SharePoint. Organisations may need to consider other ways to store and manage digital photos (and other digital media) such as dedicated digital asset management systems.

However, SharePoint may be a good option to store digital photos as records if those photos:

  • Need to be kept as a record of something.
  • Support or relate to other content in the same document library. For example, photos that relate to or support building plans (which themselves may be scanned images in PDF form) or construction.
  • Are about a specific subject. For example, photos of damage to a physical object.
  • Are relatively low in volume and file size.

Factors to consider before storing digital photos in SharePoint

The following points should be considered before storing any digital photo (or digital media) in SharePoint.

File names

Digital photos are usually stored on devices that capture them with meaningless, device-generated names such as ‘20210423_123192321’, ‘DSC_0330’, or ‘n594015825_1706121_4959’. These types of names provide no clue to the content when the image is saved to SharePoint, as shown below.

Ideally, the device-generated name should be replaced with a more meaningful name as shown below.

We can see from the unique Document ID that it is the same record. The version history also tells us that something was changed but the file size hasn’t changed.

Keep in mind that every change to the file name or any other metadata added to the library will create a new copy (‘version’) of the image. In the version history above, there are now four versions each 3 MB in size.

The Activity section of the information panel to the right (which also provides a preview of the image) also provides some key information to support the version history information:

Does changing the name potentially compromise a key element of metadata? Who should change the name, and when?

Organisations should consider establishing naming conventions for photographs to help with findability and context.

Format

Almost any type of digital object can be stored in SharePoint. SharePoint also supports the ability to view almost every type of contemporary digital photo format (as described in this Microsoft page) so format should not be a problem when it comes to viewing the file.

However, if the organisation plans to store digital photos in older or less common formats, it will need to ensure that these can be accessed for as long as required. This will generally mean ensuring that the appropriate software application is available to anyone who needs to access (open/view) the photos.

Alternatively, consideration should be given to saving the photos in different, more common formats.

Size/Resolution

The size of a digital photograph may affect its usability as a record. The two photos below are of the same scene but the first photo is very low resolution (= small size, 63 KB), while the second is a section of a much large photo (= large size, 3.7 MB). The second version is clearly a more accurate record.

Digital photos used as records should provide sufficient detail to be useful as records. Accordingly, in most instances, the original photo, not a re-sized version, should be saved.

Reliability as a record

As noted above, photos can be easily modified, sometimes making it difficult to know if it accurately reflects the image it purports to capture. This is also an issue for any form of visual record, including drawings.

Metadata created when the photo was taken and stored in the photo’s properties (including EXIF metadata) can provide evidence of the reliability of the photo as a record, even when the record was stored in SharePoint. The following are the key metadata properties that are usually created and stored with a digital photo:

  • Size.
  • (Date) Created. The date and time when the photograph was created as a photo.
  • Date taken. This is, for most digital photos, the same date and time as the ‘created’ date.
  • (Date) Modified. This should be very close to the original date and time created on the original photograph, but may be several seconds different. If the photograph has been modified (including simple photo adjustments or ‘photoshopped’), it will show a different date and time which might give a clue to its reliability as a record.
  • Image dimensions, width, height, horizontal and vertical resolution, bit depth
  • Camera details including F-stop, exposure time, ISO speed, flash mode

Storing digital photos in SharePoint Online

Note that SharePoint previously had the option to create a dedicated ‘Picture Library’. This is no longer necessary as any document library can be used to store any digital object, and SharePoint Online document libraries now have additional options to view the photos.

Any digital photo can now be saved to a standard SharePoint document library, including via the ‘Files’ tab in MS Teams.

Metadata

Unlike some EDRM systems, SharePoint does not ‘capture’ or extract the details of digital objects when they are saved to a document library. Instead, these remain with the original digital object.

If digital photos are to be saved to a SharePoint Online document library, consideration should be given to using existing or additional metadata columns to describe the image, for example, the ‘Image’ column (small version preview that is stored separately in the Site Assets library >’Lists’ folder > GUID-named folder).

The following screenshot shows a preview image of the main photo.

Viewing options

SharePoint includes three options to view the content in a document library list view: List, Compact List, or Tiles.

The following is an example of a single digital photo stored in a document library (same as the one in the other options earlier) set to the ‘Tiles’ view:

Reliability as a record

Once saved to SharePoint, the photo’s reliability as a record can be determined from version history.

Additional protections may include access/permission controls, retention and/or information security labels.

Summary

SharePoint can be used to store (some) digital photos as records, but it should not generally be used as a general storage location for digital photos. Other dedicated digital asset management systems may be more suitable for that purpose.

Before storing digital photos in SharePoint, organisations should establish procedural rules or principles including (re-) naming conventions, format and file size requirements.

SharePoint does not extract and store the metadata from digital objects, which means that digital photos should retain metadata that shows when they were created or modified, and other details about the photograph which will provide evidence to support the authenticity of the photo as a record. Some consideration should be given to adding additional metadata to help describe digital photos as records.

A combination of version history, access controls, information protection and retention labels should provide sufficient controls to ensure the reliability, integrity and authenticity of records – or at the very least provide the evidence of changes that may be made.

There are two sides to the question of authenticity, reliability and integrity:

  • Knowing if the photo that has been uploaded is a correct record of what it purports to be.
  • Preventing the photo from modification or deletion or tracking any modifications that may occur.

It might seem impossible to know if a photograph is what is purports to be, but its metadata payload may provide the detail required.

Possibly photoshopped photo

Feature Image: Source not known.

Posted in Classification, Governance, Information Management, Microsoft 365, Records management, Retention and disposal

Classifying records in Microsoft 365

There are three main options in Microsoft 365 to apply recordkeeping classification terms to (some) records:

  • Metadata columns added to SharePoint sites, including those added to Content Types and/or added directly to document libraries.
  • Taxonomy terms stored in the central Term Store, including those added as site columns and added to site content types and/or added directly to document libraries. The only difference with the first option is that with the Term Store the classification terms are stored and managed centrally and are therefore available to every SharePoint site.
  • Retention labels that: (a) ‘map’ to classification terms; (b) are linked with a File Plan that includes the classification terms; (c) are either the same as (a) or (b) and are used in with a Document Understanding Model in SharePoint Syntex; or (d) the same as (a) or (b) and used with conjunction with Trainable Classifiers.

The first two options can only be applied to content stored in SharePoint. Retention labels may be applied to emails and OneDrive content. None of the three options can be applied to Teams chats. Also note that there is no connection between the SharePoint Term Store and the File Plan, both of which can be used to store classification terms.

This post:

  • Defines the meaning of classification from a recordkeeping point of view.
  • Describes each of the above options and their limits.
  • Discusses the requirement to classify records and other options in Microsoft 365.

What is classification?

Humans are natural-born classifiers. We see it in the way we store cutlery or linen, or other household items or personal records.

Business records also need some form of classification. But what does that mean? The 2002 version of the records management standard ISO 15489, defines classification as:

‘the systematic identification and arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules represented in a classification system’. (ISO 15489.1 2017 clause 3.5).

The standard also states (4.2.1) that a classification scheme based on business activities, along with a records disposition authority and a security and access classification scheme, were the principal instruments used in records management operations.

The classification of records in business is important to establish their context and help finding them.

Microsoft 365 includes various options to apply classification terms to records.

Metadata columns in SharePoint

The simplest way to classify records stored in SharePoint document libraries is to either create site columns containing the classification terms and add those columns to document libraries, or create them directly in those libraries.

Benefits

Adding site or library columns is relatively simple. As classification terms are usually in the form of a (hierarchical) list, it is simple to add one choice or lookup column for function and another for activities.

A lookup column can bring across a value from another column when an item is selected; for example, if the look up list places ‘Accounting’ (Activity) in the same list row as ‘Financial Management’, selection of ‘Accounting’ will bring across ‘Financial Management’ as a separate (linked) column.

Default values (or even one value) can be set meaning that records added to a library (that only contains records with those classification terms) can be assigned the same classification terms each time without user intervention.

Negatives

SharePoint choice or lookup columns do not allow for hierarchical views or values to be displayed from the list view so the context for the classification terms may not be obvious unless both function and activity are listed.

The Term Store

The Term Store, also known as the Managed Metadata Service (MMS) has existed in SharePoint as a option to create and centrally manage classification and taxonomy terms in SharePoint only for at least a decade.

In 2020, access to the Term Store was re-located from its previous location (https://tenantname-admin.sharepoint.com/_layouts/15/TermStoreManager.aspx) to the SharePoint Online admin portal under the ‘Content Services’ section:

The location of the Term Store in the SharePoint admin center

Organisations can create multiple sets of taxonomies or ‘term groups’ (e.g, ‘BCS’ or ‘People’) within the Term Store. Each Term Group consists of the following:

  • Term Sets. These generally could map to a business function. Each Term Set has a name and description, and four tabs with the following information: (a) General: Owner, Stakeholders, Contact, Unique ID (GUID); (b) Usage settings: Submission policy, Available for tagging, Sort order; (c) Navigation: Use term set for site navigation or faceted navigation – both disabled by default; (d) Advanced: Translation options, custom properties.
  • Terms. These generally could map to an activity. Each Term has a name and three tabs: (a) General: Language, translation, synonyms and description; (b) Usage settings: Available for tagging, Member of (Term Set), Unique ID (GUID); (c) Advanced: Shared custom properties, Local custom properties.

In the example below, the Term Set (function) of ‘Community Relations’ has three Terms (activities).

Once they have been created in the Term Store, term set or terms can be added to a SharePoint site, either as a new site or local library/list column, as shown in the two screenshots below:

First, create a new column and select Managed Metadata

Then scroll down to Term Set Settings and choose the term set to be used.

Once added as a site column, the new column may be added to a Content Type that is added to a library, or directly on the library or list.

A Term Store-based column added to a library via a Content Type.

Benefits

The primary benefit using the central Term Store terms via a Managed Metadata column is that the Term Store is the ‘master’ classification scheme providing consistency in classification terms for all SharePoint sites.

As we will see below, Term Store terms may be used to help with the application of retention labels (which themselves may ‘map’ to classification terms in a function/activity-based retention schedule).

Negatives

Using metadata terms from the Term Store is almost identical with using a choice or lookup column. The only real difference is that the Term Store provides a ‘master’ and consistent list of classification (and other) terms.

Term store classification terms, including in Content Types, may only be used on a minority of SharePoint sites.

Additionally:

  • It is not possible to select a Term Set (e.g., the function level), only a Term within a Term Set.
  • Only the selected classification Term appears in the library metadata, without the parent Term Set or visual hierarchy reference to that Term Set – see screenshot below. Technically only that Term is searchable. It is not possible to view a global listing of all records classified according to function and activity.
  • If multiple choices are allowed, a record may be classified according to more than one Term. This may cause issues with grouping, sorting or filtering the content of a library in views.
How the Term appears in the library column.

As we will see below, there is no connection between the classification Terms in Term Sets and the categorisation options available when creating new retention labels via a File Plan. ‘Business Function’ or ‘Category’ choices in the File Plan do not connect with the Term Store.

Term Store terms and Content Types can only be used to classify content stored in SharePoint.

Retention labels

Retention labels in Microsoft 365 can be used in an indirect way to classify records in SharePoint, email and OneDrive because they can be ‘mapped’ to classification elements.

For example, a label may be based on the following elements:

  • Function: Financial Management
  • Activity: Accounting
  • Description: Accounting records
  • Retention: 7 years

Every retention label contains the following options:

  • Name. The name can provide simple details of the classification, for example: ‘Financial Management Accounting – 7 years’.
  • Description for users. This can be the full wording of the retention class.
  • Description for admins. This can contain details of how to apply or interpret the class, if required.
  • Retention settings (e.g., 7 years after date created/modified or label applied).

Benefits

Where the classification terms map to a retention class, the process of applying a retention label to an individual record, email or OneDrive content could potentially be seen as classifying those records against the classification scheme.

The Data Classification section in the Microsoft 365 Compliance portal provides an overview of the volume of records in SharePoint, OneDrive or Exchange that have a specific retention class:

Negatives

Not every record in every SharePoint document library may be subject to a retention label. Many records (for example in Teams-based SharePoint sites) may be subject to a ‘back end’ retention policy applied to the entire site (which creates a Preservation Hold library).

A retention label applied to a record doesn’t actually add any classification terms to the record.

Retention labels don’t map in any way to Term Store classification terms, except in SharePoint Syntex – see below (but this only applies to SharePoint content).

Retention labels/File Plan combination

The File Plan option (Records Management > File Plan, requires E5 licences) can also be used to add classification terms to a retention label as shown in the screenshot below. Note that there is no link with the Term Store.

Benefits

Records (including emails) that have been assigned a retention label could, in theory, be regarded as having been classified in this way because the label contains (or references) the classification terms.

Negatives

When applied to content in SharePoint, OneDrive or Exchange, retention labels linked with the File Plan do not show the File Plan classification terms. It may be possible to write a script that displays all records with the terms from the File Plan, but it may be easier to do this using the Data Classification option described above.

Retention labels/SharePoint Syntex combination

SharePoint Syntex provides a way to apply retention labels to records, stored in SharePoint, that have been identified through the Document Understanding Model process.

Benefits

As can be seen in the screenshot above, each new DU model allows similar types of records (in the example above, ‘Statements of Work’) to be associated with a new or existing Content Type that can include a Term Store Term – for SharePoint records only – and a retention label. This provides three types of ‘classification’:

  • Grouping by record type (e.g., Statement of Work, Invoice)
  • Linking (of sorts) between the records ‘classified’ in this way and a Term Store term added as a metadata column to the Content Type.
  • Assigning of a retention label. This provides the same form of retention label-based classification described above.

Furthermore, if the Extraction option is also used, data extracted to SharePoint columns can be based on choices listed in the Term Store metadata.

Negatives

SharePoint Syntex only works for records – and only those records that have some form of consistency – stored in SharePoint.

Retention labels/trainable classifiers combination

Trainable classifiers are another way that could be used to identify related records and apply a retention label to those records. Microsoft 365 includes six ‘out of the box’ trainable classifiers that will not be of much value to records managers for the classification of records:

  • Source code
  • Harassment
  • Profanity
  • Threat
  • Resume
  • Offensive language (to be deprecated)

The creation of new trainable classifiers requires an E5 licence; they are created through the Data Classification area of the Microsoft 365 Compliance admin portal. Machine Learning is used to identify related records to create the trainable classifiers.

Once created, a retention label may be auto-applied to content stored in SharePoint or Exchange mailboxes using the classifier.

The option to auto-apply a label based on a trainable classifier.

Benefits

The primary outcome (from a recordkeeping classification point of view) of using trainable classifiers is the application of a retention label to content stored in SharePoint and Exchange mailboxes. It can also be used to apply a sensitivity label to that content.

Negatives

It is unlikely that every record will be classified according to every classification option.

Trainable classifiers only work with SharePoint and Exchange mailboxes.

Classifying records per workload

The options are summarised below for each main workload:

  • SharePoint: Use local site or library columns, Term Store terms or retention labels (mapped to a File Plan as necessary), applied manually or automatically, including via SharePoint Syntex or trainable classifiers.
  • Exchange mailboxes: The only feasible option to classify these records is to manually or auto-apply retention labels that are mapped to a classification, including a trainable classifer.
  • OneDrive: Manually or auto-apply retention labels mapped to a classification.
  • Teams. It is not possible to classify Teams chats with the options available.

Is classification necessary?

The classification model described in ISO 15489 and other standards was based on the idea that records would be stored in a central recordkeeping system where they would be subject to and tagged by the terms contained a classification scheme, often applied at the aggregation level (e.g., a file).

Microsoft 365 is not a recordkeeping system but a collection of multiple applications that may create or capture records, primarily in Exchange mailboxes, SharePoint, OneDrive and MS Teams (and also Yammer).

There is no central option to classify records in the recordkeeping sense. The closest options are:

  • The grouping of records in SharePoint sites (and Teams, each of which has a SharePoint site) and libraries that map to business functions and activities.
  • The use of metadata, either terms set in the central Term Store or created in local sites/libraries, to ‘classify’ individual records (including emails) stored in SharePoint document libraries. Each item in the library might have a default classification, or could be classified differently.
  • The use of retention labels that ‘map’ to function/activity pairs in a records disposal authority/schedule. These may be applied, manually or automatically, to content stored in SharePoint, OneDrive and Exchange mailboxes.

Neither of the above may apply, or be applied consistently, to all SharePoint sites, Exchange mailboxes, OneDrive accounts. And neither can be applied to Teams chats.

A different approach to this problem is required, one that likely will likely involve greater use of Artificial Intelligence (AI) and Machine Learning (ML) methods to identify and enable the grouping of records, and provide visualisations of the records so-classified.

Image: Werribee Mansion, Victoria, Australia stairwell (Andrew Warland photo)

Posted in Information Management, Microsoft 365, Planner, Tasks

The complicated world of Tasks and To Do

We all have different ways to remind ourselves (and others) of things we (and they) need to do. In Outlook, we could create a task, something we needed to do.

In the Microsoft 365 world, personal tasks are now things we need to assign in the To Do app. In Groups or Teams, tasks are Tasks.

This post describes the difference between To Do and Tasks, and how and why Tasks has become a bit confusing.

Outlook tasks become (things) To Do

For a long time, it was possible to create and assign tasks in the calendar part of email. You could assign tasks to others.

Image source: Turn Emails into Tasks in Outlook- Instructions – TeachUcomp, Inc

In mid 2015, Microsoft acquired 6Wunderkinder. That company’s primary offering was the ‘to do’ app called Wunderlist.

In early 2020, Microsoft released the ‘To Do’ app, built on Wunderlist. This CNET article that describes alternatives to Wunderlist includes information about the new Microsoft ‘To Do’ app.

Unlike Tasks, To Do items are essentially a personal list of things to do that is accessed from the separate To Do section of Outlook. They are not included in the calendar.

The ‘To Do’ option is on the far right of the Outlook options

There are two ways to create a new item in the To Do list. The first is to click on the ‘My Day’ option in Outlook and then add a task in the To Do section.

The second is to click on the ‘To Do’ option at the bottom left of the Outlook app, which will open the ‘My Day’ section and allow a new (To Do) task to be created.

A bit confusingly, the Planned section of ‘To Do’ displays tasks that are:

  • Personal tasks, created as an Outlook calendar tasks but which don’t appear in the individual’s Outlook calendar, only in the To Do calendar.
  • Assigned to the individual from a Microsoft 365 Group or Team (including ‘Tasks by Planner’).

The difference between the two can be seen below; the first two are personal tasks from To Do, the second two are Group/Team-based tasks. The ‘Assigned to you’ items are the second two under ‘Later’. Note that the Planned section does not include any simple, non-calendar-based, ‘To Do’ items.

Planner

Microsoft announced the Office 365 service called Planner in September 2015.

The Microsoft 365 Planner app on 1 April 2021

Planner is a task-based service originally linked directly with Office 365 Groups (announced in 2014 ‘Delivering the first chapter of Office 365 Groups‘). It was described as ‘a simple and highly visual way to organize teamwork’ within a team – which meant initially an Office 365 Group. It seemed that Microsoft’s vision was to move the creation of Tasks away from Outlook to Planner.

Initially, when a new Office 365 Group was created (including when a new Team was created in MS Teams), it created a Plan. This connection was later removed and so a new Group or Team no longer creates a new Plan.

The following is an example of an empty plan for an Office 365 Group called ‘SharePoint Admin Group’. All the members of the Group (or Team if one exists) would have access to the plan. Plans contain ‘buckets’ or groupings of tasks. The default bucket is ‘To do’ which, in the example below contains a single task ‘Create two new tasks’ (which is outstanding) A separate bucket was created named ‘New Sites’, and it has one completed task.

Example plan in Planner

Changes to Planner

Several changes happened with Planner since 2018:

  • New Office 365 Groups and Teams did not automatically create a Plan.
  • Multiple plans could be created for every Office 365 Group or Team.
  • Tasks by Planner was introduced to Teams (see below) so that every channel can use either the ‘parent’ Office 365 Group Plan or create a new one.

These changes have created some confusing content in the Planner app.

Tasks by Planner in Teams

Microsoft announced in April 2020 that Planner would be renamed Tasks (it is still named Planner a year later).

As noted in the link (by onmsft.com), ‘… the change means that Teams users will soon be able to see their individual tasks and team tasks in a single app from across Teams channels, Planner and Outlook. For mobile users, the change also means that both a list view and a new mobile tasks experience will soon be available in within the Teams app.’

The new Tasks by Planner and To Do was visible from Teams in early 2021 but the relationship between Outlook To Do tasks and Planner Tasks via Teams remained a bit confusing.

Tasks by Planner and To Do in Teams

When a Team channel is opened, the ‘Tasks by Planner and To Do’ can be added as a tab.

Tasks by Planner and To Do can be added as a tab in any ‘public’ channel (not private channels)

When ‘Tasks by Planner and To Do’ is selected, two options are visible and this is where some of the confusion starts.

Most people are likely to simply click ‘Save’, which creates a ‘Tasks’ tab in the channel and a new Plan. Ideally, they should use an existing plan, if it exists (which it probably won’t – as a result, multiple Plans may be created for each Team and channel.

This is how the new Tasks tab looks like in the Team:

Perhaps it doesn’t matter (for some organisations) how many Plans or Tasks that are set up.

We can now see there are three Plans for the SharePoint Admin Group Team, two in the same (General) channel. The end user can also see tasks that were assigned to him/her. If they click on the ‘Tasks’ option they will see the list of personal calendar- and To Do-based tasks as we saw above in Outlook:

Behind the scenes, in Planner, we can see four plans for the SharePoint Admin Group. Three of these map to the three above, but not the one titled ‘Tasks – SharePoint Admin Group’ which has two completed tasks. But where is it?

Here are the two completed tasks in SharePoint Admin Group ‘Tasks’ plan that don’t exist in the main SharePoint Admin Group Plan, or the other two.

Where are these two completed tasks?

Or, more specifically, why do they not appear in many of the Teams Tasks tabs? There are no private channels in this Team, so I know it’s not hidden in one – and, in any case, you cannot create a new Task list in a private channel.

Just to try to work this out, I created a new Task in that list of Tasks, assigned it to myself. The only place I could find it was in both Teams and Outlook in the ‘Assigned to you’ area.

Assigned to me in Teams
Assigned to me in Outlook

Summing up

Tasks, either to remind yourself of things you need to do, or what others need to do, are probably good for specific purposes or Teams. But the ability to create multiple Task lists in Teams channels is just going to create more and more confusion.

But it’s confusing and will likely result in multiple random Tasks/Plans in Planner, even for the same channel.

Posted in EDRMS, In Place Records, Products and applications, Records management, SharePoint Online, Solutions

SharePoint is not an EDRMS

In my February 2021 post A brief history of electronic document and records management systems and related standards, I quoted from a presentation by Philip Bantin in 2001 that summarised the difference between the two systems.

  • An electronic document management system (EDMS) supported day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
  • An electronic records management system (ERMS) was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted.

It could be argued that SharePoint began its life in 2001 as an EDMS and began to include ERMS functionality when SharePoint 2010 was released. So, how is it not an EDRMS?

In simple terms, unlike a traditional EDRMS model that was intended to be the repository for all business records, SharePoint is only one of several repositories that are used to store and manage business records. Instead of centralising the storage and management of records, systems such as Microsoft 365 provide centralised management of records wherever they are stored.

The EDRMS model

Almost all EDRM systems since the mid-1990s have had the following characteristics:

  • Compliance with a standard. For example, DOD 5015.2 (1997), AS 4390 (1996)/ISO 15489 (2001), ISO 16175 (2010), VERS (1999), MoReq (2001)/MoReq2, etc.
  • A relational database (for the various functions, metadata and controls (see diagram below) and a separate file share (for the actual content/records), accessed via a user interface.
  • An expectation that most or all (digital) records would be copied from other systems used to create or capture them to the EDRMS for ongoing storage and protection. This outcome might be achieved through integration with other systems, for example to copy emails to the EDRMS.

The integrated EDM/ERM system model was described in the following diagram in 2008 by the International Standards Organisation committee, ISO/TC171/SC2, in its document ‘Document management applications’:

There are two key problems with the EDRMS model:

  • They do not (and cannot) realistically capture, store or classify all business records. Emails have remained a problem in this regard for at least two decades. Additionally, there is now a much wider range of born-digital records that remain uncaptured.
  • Many of the born-digital records that were copied to the EDRMS continue to exist where they were first created or captured.

Any organisation that has been subject to a legal subpoena or Freedom of Information (FOI) request for records will know the limited extent to which EDRM systems provide evidence of all business activities or decision making.

And yet, almost every organisation has a requirement to manage records. There is clearly a disconnect between the various requirements and standards to keep records and the ability to keep them.

But SharePoint (on its own) is not the answer.

Why isn’t SharePoint an EDRMS?

SharePoint was never promoted or marketed as an EDRM system. It is not even a relational database (see this 2019 post by Sergey Golubenko ‘Why SharePoint is no good as a database‘ for details).

To paraphrase from Tony Redmond’s recent (21 March 2021) post ‘SharePoint Hits 20: Memories and Reflections‘, SharePoint was originally designed to allow content (including page-based content in the form of intranets) to be accessed via the web, thereby replacing network file shares.

In practice, most organisations retained their network file shares, built their intranet using SharePoint, and otherwise used SharePoint to manage only some digital content.

It could be argued SharePoint started out as an EDM system and became more like an ERMS when more recordkeeping capability was introduced in SharePoint 2011. But that doesn’t make it an EDRMS in the traditional sense of being a central repository of business records.

The reality is that records are, have been, and will continue to be created or captured in many places:

  • Email systems. In Microsoft 365, Exchange Online mailboxes are also used to store the ‘compliance copy’ of Teams chats messages.
  • Network file shares.
  • SharePoint, including OneDrive.
  • Other online document management systems, including Google Drive.
  • Text, chat and instant messages often created in third-party systems, often completely inaccessible or encrypted.

The type and format of a record can vary considerably. For example:

  • An emoji.
  • A calendar entry.
  • A photograph or video recording (including CCTV recording).
  • The recording of a meeting, in video or transcript form, or both.
  • Virtual reality simulations.
  • Social media posts.
  • 3D and digital drawings (e.g., via digital whiteboards).

And, of course, all the data in line of business systems.

Instead of trying to save records into a single EDRM system, Microsoft 365 provides the ability to apply controls over the management of most records where they were created or captured – in email (including archived social media and other records), Teams chat, SharePoint, or OneDrive, and Yammer.

Is there a need for an EDRMS?

There is nothing stopping organisations acquiring and implementing traditional EDRM systems, or even setting up some SharePoint sites, to manage certain high value or permanent records, including some records that can be copied from other create/capture systems (such as email).

But there is also a need to address the management of all other records that remain in the systems where they were created or captured.

In most modern organisations, this requires broader controls such as applying minimum retention periods at the backend, monitoring usage and activity, and the proactive management of disposals. Not trying to copy them all to SharePoint.

Featured image: Lorenz Stoer, late 1500s.