How records managers can manage retention labels and policies in Microsoft 365

Microsoft 365 includes the capability, subject to licencing, to apply retention labels and non-label retention policies to content stored in Exchange mailboxes, SharePoint Online sites, Microsoft 365 Groups and OneDrive for Business accounts.

Non-label retention policies may also be applied to Teams chats, Teams channel (including private channel) posts, and Yammer messages. For Teams chats and posts, these policies apply to the ‘compliance copy’ of the messages stored in a hidden folder in Exchange mailboxes.

Both retention labels and retention policies work against individual items stored in these location, not the containers (e.g., Exchange mailbox, SharePoint document libraries, OneDrive account, Team chats and channels. This means that individual items stored in those aggregations will disappear (when destroyed) leaving the container still in place.

Creating and applying retention labels and policies is a relatively straightforward matter. For records and information managers, knowing where the labels and policies have been applied is more complex, as this post explains.

Creating retention labels and policies

Both retention labels and retention policies are created in and applied from the Microsoft 365 Compliance admin portal.

The Information Governance section is used to create both: (a) retention labels that can be published as label policies, or auto-applied (E5 licence required), and (b) retention policies.

The Records Management section is used to create retention labels, based on a file plan that can be published as label policies or auto-applied (E5 licence required).

The two other tabs in the records management section are (a) Events and (b) Dispositions. The latter area is where basic details or records due for disposition, and records that have been destroyed, are displayed.

Publishing retention labels

Retention labels do not do anything until they are either:

• Published individually or in groups as label policies to all or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
• Auto-applied (with an E5 licence) including: (a) via keyword searches, (b) based on information sensitivity, (c), based on basic classifiers, (d) as part of trainable classifiers or (e) SharePoint Syntex (linked with a Content Type).

Labels cannot be published to or applied to Teams chats or channel posts, or to Yammer content.

Applying retention labels

Retention policies may be published (applied) to either:

• All or selected Exchange Online mailboxes, SharePoint Online sites, OneDrive accounts, or Microsoft 365 Groups.
• All or none of: Teams chats, Teams channel posts, Teams private channel posts, Skype for Business chats, Yammer community posts and chat messages.

Viewing labels, label policies, and retention labels

Once they have been created, the basic details of labels, label policies and retention policies can be seen in the Information Governance and Records Management sections.

Information Governance > Labels

Labels listing

The ‘Labels‘ tab displays all labels that have been created (including via the Records Management – File Plan section, see below), with the following columns (bold indicates what is visible in the default view for this listing – see also below for the list of labels in the Records Management section). The listing (and individual items when clicked) does not include any information to indicate if the label has been published to what label policy, or auto-applied.

• Name
• Status (active, inactive)
• Based on (trigger – when created, last modified, label applied, event, none)
• Is record (if ‘records declaration’ is checked)
• Retention duration (days, months, years or none)
• Disposition type (auto delete, (disposition) review required, no action)
• Reference ID (File Plan detail)
• Function/department (File Plan detail)
• Category (File Plan detail)
• Subcategory (File Plan detail)
• Authority type (File Plan detail)
• Provision/citation (File Plan detail)
• Created date
• Created by
• Last modified
• Last modified by

Information Governance > Label policies

Label policies listing

The ‘Label policies‘ tab displays all label policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, e.g., Exchange mailboxes, SharePoint sites, not specific locations), the labels that have been included, and if a preservation lock has been enabled.

• Name
• Type (Publish, other options may also be listed)
• Created by
• Last modified by
• Last modified date

Information Governance > Retention policies

Retention policies listing

The ‘Retention policies‘ tab displays all retention policies with the following columns. Clicking on any retention policy displays the status, description, where the policy has been applied (generally, not specifically), retention period and if a preservation lock has been enabled.

• Name
• Created by
• Last modified

Records management section

Records Management > File Plan

File plan details (cut off at right)

The ‘File Plan‘ tab displays all labels (including those created via the Information Governance – Labels section above) with all the columns shown above for the Information Governance labels. Again, there is no information to indicate if the label has been auto-applied or what label policy it has been published in.

Records Management > Label policies

Records management label policies

The ‘Label policies‘ tab displays all label policies that have been created. It contains the same content as the Information Governance ‘Label policies’ details above.

How do you know where retention labels and retention policies have been applied?

As noted above, both retention labels (published as label policies or auto-applied, but not to Teams or Yammer) and retention policies are applied to individual items stored in containers (mailboxes, sites/document libraries, OneDrive accounts, Team chats, Teams channels) in the various workloads.

This model, and the ability for end-users with edit rights to change or remove labels on content stored in SharePoint that is not declared as a record or change a label on emails, can make it very difficult to know (for records managers in particular) what has been applied where.

Microsoft 365 does not provide any out of the box option to view where retention policies have been applied (to individual records). There are currently only two ways to know where retention labels have been applied:

• Post facto – via Content Explorer in the Data Classification section (requires E5 licence), for labels only.
• As part of retention planning.

Both of these options are described below.

Data Classification > Content Explorer

For those with E5 licences, the ‘Content Explorer’ area in the Data Classification section of the Compliance admin portal can provide details of where labels (only) have been (or are currently) applied after the fact.

According to the Microsoft guidance Get started with content explorer, (which includes helpful screenshots) ‘Content explorer shows a current snapshot of the items that have a sensitivity label, a retention label or have been classified as a sensitive information type in your organization.’

It also notes that ‘Access to content explorer is highly restricted because it lets you read the contents of scanned files’, via the second of the two roles required to access this capability:

• Content Explorer List viewer: Membership in this role group allows you to see each item and its location in list view.
• Content Explorer Content viewer: Membership in this role group allows you to view the contents of each item in the list.

So, Content Explorer is helpful, but only to know where retention labels have been applied after the fact. And it requires an E5 licence (or E5 Compliance licence add on).

Documenting retention via retention planning

Planning for retention in Microsoft 365 is an essential requirement for records and information managers. It should always be based on regulatory, archival or business requirements for keeping records.

One of the challenges for the retention model in Microsoft 365 is the application of retention labels and/or retention policies to Exchange Online mailboxes, OneDrive accounts, Teams chats and Teams channel posts (including private channel posts). It is a challenge because the traditional recordkeeping model involved copying content identified as records from the create/capture application to a central recordkeeping system where the records would be subject to retention, whereas in Microsoft 365 the reality is that not all records may (or can) be captured.

Retention and disposal authorities/schedules typically do not include coverage for entire mailboxes or OneDrive accounts, let alone Teams content. In response to this, the United States National Archives and Records Administration (NARA) developed the ‘Capstone‘ (PDF) approach for email, whereby ‘final disposition is determined by the role or position of the account user, rather than the content of each individual email.’ Something similar to this model, also for OneDrive accounts and Teams chats and channel posts (both of which have compliance copies stored in Exchange mailboxes), may also be required.

Tackle retention policies first

Retention policies are sometimes referred to as ‘back-end’ or ‘safety net’ policies because they work in the background and help to capture deleted content that needs to be retained for minimum periods of time after which it can be deleted unless a longer retention (e.g., label-based) applies. Retention policies cannot normally be mapped to retention classes because they target entire workloads not specific aggregations in those workloads (e.g., they target all content in SharePoint sites, not the content in the libraries of those sites).

Retention policies do not include any form of disposal review. The content is either destroyed or ‘nothing happens’ (allowing the content to be destroyed after the retention period expires).

Retention policies can be applied to the following parts of Microsoft 365:

• All or selected groups of Exchange mailboxes
• All or selected groups of SharePoint sites
• All or selected groups of OneDrive accounts
• All or selected groups of Microsoft 365 Groups
• All (or none of) Teams chats
• All (or none of) Teams channel posts
• All (or none of) Teams private channel posts
• All (or none of) Skype for Business chats
• All (or none of) Yammer community posts
• All (or none of) Yammer chats

(Keep in mind that Teams chats and Teams private channel posts are stored in the personal Exchange mailboxes of participants, while Teams channel posts are stored in the Exchange mailboxes of the Microsoft 365 Group linked with the Team, BUT Teams retention policies can only apply to all or none.)

Once a decision is made to create retention policies, this detail should be documented separately from Microsoft 365 – in a spreadsheet or SharePoint list perhaps. The following is a basic example of what might be included in that documentation.

• Senior executive EXO mailboxes – 15 years / date modified / do nothing
• All other EXO mailboxes – 7 years / date modified / destroy
• All SharePoint sites – 7 years / date modified / destroy
• M365 Groups – 7 years / date modified / destroy
• All ODfB accounts – 7 years / date modified / destroy (see also note below)
• All MS Teams chats – 7 years / date created / destroy
• MS Teams channel posts – 7 years / date created / destroy
• MS Teams private channel posts – 7 years / date created / destroy
• Yammer user messages – 1 year / date created / destroy
• Yammer community messages – 2 years / date created / destroy

Keep in mind that retention is based on either date created or date modified, so the content stored in these locations will begin to disappear when the retention period expires. This will leave empty mailboxes, empty SharePoint document libraries, OneDrive accounts and empty Teams. Part of the retention planning needs to include a process for destroying those containers once the content inside them has been destroyed – and documenting that destruction also.

For OneDrive accounts, consideration might be given instead to actively reviewing the accounts of departed employees to removing any content considered valuable to another location (e.g., SharePoint) or account, rather than simply leaving it hidden from view for the period of the retention. This would allow much shorter retention after the employee leaves.

Documenting retention labels

Retention labels will typically be based on individual classes in a records retention schedule or disposal authority (DA). Accordingly, it will be useful to have a listing of all classes in a spreadsheet or SharePoint list with the following details as a starting point:

• Retention schedule class ID
• Retention schedule class name
• Retention schedule class description (or abbreviated version).

Retention labels can then be created based on those classes. Every label includes the following details (see also screenshot below):

• A name that should be obvious to end users who may end up having to choose it or know from a glance what it means. For example ‘5.1.1 Financial records – 7 years’.
• A description – same as the original retention class description, or an abbreviated version.
• A retention period. For example, 7 years.
• A trigger. The options are: ‘When items were created’, ‘When items were modified’, or ‘When label was applied’. (Note that triggers may also be based on events, but these should be used sparingly and only after the full implication of using them is understood.)
• An action at the end of the retention. For example: ‘Delete items automatically’, ‘Trigger a disposition review’ (E5 only), ‘Do nothing’. ‘Do nothing’ is a good option for permanent records as it will stop deletion and tag a record making it easier to find later. There are three other options instead of retain: ‘Retain forever’, ‘Only delete items when they reach a certain age’, and ‘Don’t retain or delete items’.
• (Note: When labels are created from the Records Management section, two additional options appear: (a) Mark items as a record and (b) Mark items as a regulatory record. The implications of using either option needs to be understood).

Options when a new label is created. Note that two additional items appear in the Records Management labels.

As each retention label is created the following details should be recorded next to the classes in new columns in the spreadsheet or list:

• Retention label name (may be a combination of the ID and name, and retention)
• Retention period. For example, 7 years.
• Retention trigger. For example, ‘When created’, ‘When modified’, ‘When label applied’.
• End of retention action. For example, ‘Destroy’, ‘Review’, ‘Do nothing’
• Declare as a record: Yes/No
• Regulatory record: Yes/No

As labels are published, individually or in groups, this information should be added to columns in the spreadsheet or list:

• Retention policy. That is, what is the name of the policy where this policy has been published.
• Policy location. That is, the location/s were the policy has been applied.
• Date applied?
• Date for destruction. (Date applied plus retention period)

If the label is being used via trainable classifiers, SharePoint Syntex, or otherwise auto-applied, that information should also be added to the spreadsheet or list.

• Trainable classifier? Yes/No
• SharePoint Syntex? Yes/No
• Auto applied? Yes/No
• Auto applied location.

End product – for all retention labels and policies

The end of product of is either a spreadsheet with two workbooks, or two SharePoint lists (one for the retention policies, one for the labels), that describe what retention labels and retention policies have been created and where they have been applied across Microsoft 365.

Records and/or information managers will need to proactively monitor the Microsoft 365 environment and provide advice to IT about the destruction of content and records, as well as the containers in which that content was stored.

There may be other ways to achieve this outcome, but records and information managers need to understand what capability is available (or not) out of the box to make informed decisions.

Featured image source: https://www.pexels.com/photo/architecture-black-and-white-challenge-chance-277593/

Combining metadata with folders in SharePoint

As organisations increasingly move content from network file shares to SharePoint, many have found that folders in SharePoint (and via the Files tab in Teams channels) are a bit less friendly.

For example, there is no way to ‘expand’ the folder view, so it is not possible to see the folder hierarchy (unless the library is synced to File Explorer). And although it is possible to add more or less unlimited metadata to content stored in document libraries, this may be a pointless exercise given that the metadata is not yet visible in File Explorer and folder-based structures are more familiar.

Fortunately, SharePoint not only has the ability to use both folders and metadata, but also to see all the content in a document library without the folders AND using that metadata to group, filter and sort the content regardless of how the folders were structured. This capability can be (surprisingly) useful for records managers.

This post explains how it works, and why this option doesn’t work in the File Explorer view of a synced library.

A typical folder-based library

The screenshot below shows three folders (out of many more) in a fairly typical document library for a SharePoint site.

A typical folder-based library

SharePoint (via a browser) does not show a hierarchical view of folders, so the end-user must click on every folder to find out what they contain. (And yes, you can enable the ‘Item child count’ option but that only tells you how many items are directly under the folder, there is no expanded view).

The ‘Files’ tab in Teams also presents a the same view of the Team’s linked SharePoint site, except that each channel maps to a folder in the default ‘Documents’ library, so that folder (content, if any) is displayed.

Adding metadata

SharePoint allows for almost unlimited metadata on document libraries and lists. (Whether you need to add unlimited metadata is a separate question).

Metadata (known as ‘site columns’ or ‘library columns’) may be created at the site or library level, and then added to any library on the site via the Library Settings > Columns > ‘Create column’ OR ‘Add from existing site columns’.

The screenshot below shows an example of a site or library column named ‘Document Type’ to be added to the ‘Meetings’ library. This column has three choice options because there are typically three types of document in this particular library: Agendas, Drafts, and Minutes. You can of course have as many choices as you like but consider the end-user experience. No-one wants to scroll through a hundred choices.

The column above has three choices (selected from a drop down menu), is not mandatory, does not enforce unique values, does not allow end-users to add their own options, and has a default value set. This means that every new document added to the library will be assigned the ‘Agenda’ choice.

Default values may also be useful to automatically add recordkeeping terms such as ‘Function’ and/or ‘Activity’, without the end-user having to do anything.

Once added to the library (via the Library Settings > ‘Create column’ or ‘Add from existing site column’), the metadata column can be displayed in the view, and the metadata options may be added to the documents via ‘Edit in Grid View’ or by adding or changing the metadata on individual documents via the information properties pane on the right.

Adding or changing the metadata via ‘Edit in Grid View’

(Note that there is no option to add or change additional metadata on individual items via the Teams ‘Files’ tab interface.)

But, adding or changing metadata by navigating through the folder structure would be onerous and a waste of time when there is a much better way of doing this – by making the folders disappear.

How to make folders disappear

Every SharePoint document library (including those in sites connected to Teams) has the ability to hide the folders in a document library. This – and the ability to add metadata and more – is achieved via the library view options.

The screenshot below shows the location of default ‘All documents’ view of a library, accessed from the top right of the document library in either SharePoint or via the Files tab in Teams.

Generally the ‘All Documents’ view should be left ‘as is’ and a new view created.

To do this, select ‘Create new view’ or ‘Save view as’ to create the view required. Give the new view a name that reflects what it is, for example ‘NoFolders-Minutes’ (and don’t use a long name or the text may not be visible).

Every view has a range of options, including the following.

Choose columns to be displayed

Select which (system or added) metadata columns are to be displayed in the view.

Choose the columns to appear in the view, and where they appear.

Sort columns

Select how the content will be sorted. For example, sort by ‘Created’ date.

Sort by options

Filter columns

Select how the content will be filtered by the metadata columns. For example, only show content that meets a specific criteria such as Document Type = ‘Minutes’.

Filtering options. In this case, the view would only show content tagged with the metadata value ‘Minutes’

Group columns

Select how the content will be grouped by the metadata columns. This groups the content by a particular site column. For example, Group by ‘Created by’ or other options. Grouping (and sub-grouping, two levels are allowed) content in this way is often a much better approach than using folders.

The screenshot below shows an example of two-level grouping of metadata.

Example of two-level grouping

Folders

A bit further down is the ‘magic’ button to make folders vanish – ‘show all items without folders’. This options also hides document sets.

Making the folders disappear.

What this looks like

In the document library that was originally folder-based, we have created a new view without any folders. This now allows us to see all the documents in the library without folders. We can see from the view below that some of the items have been mistagged (Minutes with ‘Agenda’ for example). These can now be easily fixed via the ‘Edit in Grid View’ option.

(Note – if the library has more than 5,000 items without folders, the view won’t work and additional filtering may be required; when this happens, press the back space and fix the view, otherwise you may be locked out).

The view can be further modified. In the example below, the view has now been grouped by ‘Created by’ and also filtered to only anything created after 1 August 2021.

Views created in SharePoint are also visible via the Teams Files tab, and can be created in that tab. The screenshot below is of a Teams-based SharePoint library that had multiple folders under the ‘General’ folder and one document (‘Cover Letter’) under General. We can now see that there were in fact no documents in the other folders.

Benefits of views without folders

The primary benefit of creating views without folders is that (subject to the 5,000 item view limit) you can easily see what content (if anything) is hidden in the folder structure of a document library. The metadata that is displayed can be easily exported to a csv or Excel file.

It also makes it easier to apply and/or change metadata; this metadata can in turn be used to sort, filter and group the content in a library.

Note that, if there are more than 5,000 items in the library, additional views can be created to reduce the volume that is displayed at any one time.

As every view has a URL, they can also be added as links in the left hand or page navigation or emailed, making them a kind of pre-canned search.

In the example below, there is only one library ‘Policy Library’; both the ‘Aged Care Policies’ and ‘Due for Review’ are views based on metadata assigned to documents. Additional views (not visible) can group policies based on business areas or relevancy, or various other options. This means that policies (in this case) are not restricted to a single ‘folder’, but can be viewed in multiple ways depending on their context.

Multiple views showing as links on the left hand navigation pane

Are there any drawbacks doing this?

Aside from the 5,000 view item limit, the other things to be aware of are (a) what happens when the library is synced, and (b) potential confusion.

Libraries (including Teams-based libraries, via the ‘Files’ tab) synced to File Explorer do not currently display any additional metadata added to the SharePoint library, although this may be coming soon. Even if this metadata is displayed, the views are not available. So hiding folders and grouping/filtering may only be useful via the Teams client or browser.

In terms of the potential for confusion, folders on network file shares are very good at hiding duplicates or versions of the same document. When folders are removed from SharePoint, the same document may appear multiple times without any details of the original folder structure or path, resulting in confusion or uncertainty. What if someone always accessed the document via a particular path, then discovers it is missing? How do they know it is hiding in another folder path? If you are going to turn off folders, make sure that end-users about this and how to continue to access their documents via the (preferably unchanged) ‘All documents’ view.

Can we get rid of folders, yet?

Folders have been around for a long time and are a very hard habit to break. They are unlikely to go away any time soon.

However, metadata-based views provide a useful way to ‘see through’ folders and can also be used to apply additional metadata that, in turn, can be used – in conjunction with folders – to group, sort and filter the content hidden away in the folders. Perhaps, in some cases, metadata-based grouping could replace folders, but this will take time and may not be always useful – for example in synced document libraries.

Either way, a combination of folders and metadata can be useful to support recordkeeping activities.

Metadata in old film rolls from Japan – a personal story

Metadata is often thought of as either describing objects (e.g., paper or digital records, library books, physical assets, etc), or forming part of a digital object (e.g., system generated metadata or in the case of most digital photographs, EXIF metadata).

Before the arrival of computer-based databases (and even after), many companies embedded codes in their manufactured products, a kind of embedded physical metadata. These codes usually recorded (and continue to record) information about batches, allowing them to identify when and where it was made. This information can assist with manufacturing recalls.

This post describes, through a personal story, how metadata embedded in two very old rolls of negatives were used to identify their origins. It also underlines the importance of asking older people about objects they may have in their possession before they pass on.

Starting point – two unprinted rolls of negatives

Among my father’s possessions when he died in 2012 were three old AGFA aluminium film canisters containing 35 mm black and white negatives.

One of these canisters contained two rolls of Eastman Kodak film. One of them showed a family and their garden, the other roll was of a parade in Japan. A small western girl appeared in both, linking them somehow.

The original negatives

It was obvious that at least one of the rolls contained photos taken in Japan. My father had been stationed in Japan from November 1945 to March 1946 so I assumed that was the connection. However, none of the photos in those two rolls had been printed or placed in my father’s Japan album. The album contained a range of photos from various locations in Japan that he had bought from other servicemen as he didn’t have a camera at the time.

In early 2021, a military historian based in Japan identified that the parade appeared to be in Yokohama, where my father had been stationed.

But something wasn’t making sense. The photographs showed a well-dressed crowd on what appeared to be a warm day in a normal city. My father’s letters from November to December 1945 pained a different picture – he said that it was a bitterly cold winter and the locals were mostly destitute and living in shanties as most of the city had been reduced to rubble:

Soon we saw bomb damage. Piles of smashed cars, and flattened buildings. There are acres of flattened and bare spaces … There are no shops, all having been smashed up and flattened. (Extract from a letter dated 19 November 1945)

A photograph in 1945 showing the extent of bomb damage. The red circle is the area where the parade photos were taken. The large building immediately below the red circle was used as a hospital during WW2 (and after, by American occupation forces).

To try to find more information about the parade, I made contact with a local Yokohama resident who blogs at Weekend Walks in Yokohama, Shinya Ota. He also expressed doubt that these photos were taken in 1945 (especially given the photo above). He noted that the annual port opening parades did not re-commence until the early 1950s.

So when were these photos taken, and why did my father have them?

Looking for clues in the metadata

I decided to look more closely at the negatives and found some initial clues. Both rolls of film were made by Eastman Kodak and both were nitrate film.

According to the website Preservation and Treatment: ‘The use of nitrate base in 35 mm still camera film ended in 1938. Eastman ended the use of nitrate base in commercial and professional sheet film in 1939.’

Kodak phased out all nitrate base film by 1951. So it seemed likely that the negatives were dated from before 1939.

The next clue was the small code next to the word ‘Kodak’. One roll had a ‘+’ and a small square, the other roll had a circle. Note also the perforations in the film and the indentation above the word ‘Kodak’; this would provide a clue to the type of camera that was likely used.

The table below, part of a larger table that can be found on the internet, shows the manufacturing year of Kodak film and also where they were made – all the film on the left column was made in Rochester USA.

Kodak manufacturing date codes

So now I had two dates for the rolls of negatives – 1935 and 1936. These rolls were likely imported to Japan from the USA at some point, either by the person with the camera or a local importer/re-seller. My father could not have taken these photos (he was only 10 in 1935) and there was no obvious family connection.

Additional clues on the film

I sought advice about the film from photographic specialists at photrio.com. They picked up two additional clues.

The first was the perforation or ‘sprocket holes’ along the edge of the film roll. These appeared to be very similar in dimension to the Bell and Howell (BH) design described in the Wikipedia page ‘Film Perforations‘. Although a different Kodak Standard (KS) was introduced in the 1920s, it apparently failed to displace the BH perforations. It is not known why the two rolls of Kodak negatives did not use the Kodak Standard but it might have something to do with the camera that was used.

Film perforations

The second clue, which may be also reflected in the yellowish circular marking on the film, was the camera that was used. According to the Wikipedia page about 135 film: ‘In 1934, Kodak introduced a 135 daylight-loading single-use cassette. This cassette was engineered so that it could be used in both Leica and Zeiss Ikon Contax cameras along with the camera for which it was invented, namely the Kodak Retina camera.’

It seemed very likely that my negatives were taken on one of these cameras using the relatively new single-use cassette film. The perforations allowed the film to be used on two other popular cameras at the time.

Additional clues in the photos

There were two other sources of information in the negatives, or rather in the photographs themselves.

The roll of photographs showing the family was dated 1935, while the roll of parade photographs was dated 1936. Of course, the 1935 roll could have been used in 1936.

One of my sisters reminded me about a single sentence in my father’s life story that suggested he had found these negatives in the British Consulate in Yokohama when he was assigned there as a guard in December 1945. The Consulate building had been closed since 1940 and was now re-occupied by some British military personnel. I had probably overlooked this line and just assumed that the negatives dated from when he was there.

I now realised that these rolls were already around 10 years old when my father obtained them.

Based on various visual clues, the photographs of the family were almost certainly taken at the British Consul-General’s then residence, likely at Yamate-cho 18 in 1935 or 1936 where the Consul-General had lived since 1920 (and possibly also after the 1923 earthquake). A new residence was opened in 1939 and these photos were definitely not taken in that building.

So, the people in the photographs are likely to be the British Consul-General, his wife and other family members (all yet to be confirmed). A small girl in these photos also appears in three of the parade photos (with a group of Japanese, likely the Consul-General’s staff), providing a link between the two.

The photographs of the parade offered a range of additional clues, including buildings, banners and flags, uniforms and the way onlookers were dressed, all indicating that the photographs were taken some time between 1935 and 1938 and likely in April or May when the weather was warmer.

A photograph from the parade with typical ‘daimyo gyoretsu’ performers. The banner on the left has the word ‘Yokohama’ on it. One building the background has an advertisement from 1935.

Lessons learned

The main lesson from this exercise was the reminder to ask grandparents, parents or other elderly relatives about the details in or of any objects and photographs that may be handed down – who is who in photographs, where were they taken, where is the source of a object (painting, vase, clock, other art work) and so on.

The second lesson was that rolls of film may contain metadata clues, beyond the content of the actual photos, that can help identify their date, source and perhaps even the camera that was used.

Feature image: Tsuchiya Koitsu, ‘Yokohama Sankei Garden’, 1936 (various sources)

Metadata persistence when records are moved in SharePoint

SharePoint Online document libraries include the option to move content to a different location. But what happens to the metadata when this content is moved?

This post explains what happens to the metadata when different types of content stored in a SharePoint Online document library is moved (not copied) using the ‘Move to‘ option to a different library in either the same site or a different site (both in SharePoint Online).

TL:DR

If you don’t have time to read all the details, here is a summary. When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:

• The original system generated metadata (Created, Modified) is retained for all types of content.
• The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
• Information Protection labels (set on Office document only) are retained unless a label setting prevents the move.
• Retention labels are retained for all types of content.
• Any added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.
• The same outcomes occur if documents are moved between two synced document libraries in File Explorer.

Metadata in SharePoint libraries

All SharePoint document libraries include four types of metadata:

• System generated. Examples include the Created [in SharePoint, not the original item created date] and Modified dates, the Created by and Modified by names, File size, Content Type, sequential ID, and Version History (which is a collection of metadata objects). If Document IDs are enabled, it also includes the Document ID (PREFIX-LibraryID-sequential ID, eg ‘RECORDS-12312322-234). Note that every object saved to SharePoint is also assigned a unique GUID.
• Name. This is either the original name of the item (in the case of an email, the email subject line), or added by an end-user. Note that this is NOT the same column as the Title column which is blank by default.
• Added metadata. That is, pre-existing or custom site, or ‘local’, metadata columns added to the library. In the examples used in this post, the added metadata is ‘Document Type’ (choice, default =’Agenda’), ‘Event Type’ (choice, default =’Indoor’), and ‘Random Text’ (text, default = ‘Elephant’).
• Compliance metadata (when applied). Retention label, Label applied by, Label setting, Sensitivity (for Information Protection labels).

It is important to note that metadata in added metadata columns in one library will NOT appear in the destination library unless that (destination) library has exactly the same columns.

For the purpose of this post, four different types of content (Office/Word document, PDF, email (.msg and .eml), image) were moved from a library with multiple metadata elements shown below to an empty document library in brand new site with nothing configured at the site or library level.

The metadata elements set on the source site library were as follows:

• Document IDs
• Document Type (choice)
• Event Type (choice)
• Random Text (default set to ‘Elephant’)
• Retention label applied to the library and a different label applied at the folder level
• Information Protection label. This was only applied to Word documents (when they were edited) as there was no option to apply it to the emails, PDFs or images.

Metadata in the source library

Moving a PDF, email (msg) and image

As the outcome is likely to be the same for all non-Office content, a PDF, email (msg) and image were all moved at the same time to the destination library. Immediately a notification appeared that metadata properties on all three items would be lost if they were moved. The option to ‘Move anyway’ was selected and the items were moved.

Items not moved with the ‘Move anyway’ option

The destination library shows the three items without the Document IDs (which were not enabled) or any of the added metadata columns. However, the items have retained the Retention label – no retention has been set anywhere on the destination site.

Destination library

Were the Document IDs moved?

No. The original Document IDs did NOT appear because the destination library did not have the Document ID feature enabled.

Once that feature was enabled and the column was made visible in the list view, we can see that the three items that were already moved have lost their original Document IDs. Two new items moved have retained their original source Document ID (CORPRECORDS-667097513-n).

Moving Office documents

The metadata applied to Office documents (Word, Excel, PowerPoint with an ‘x’ at the end) works in a similar way but those documents also embed the metadata in their underlying XML structure.

Some of this metadata is visible in, and can be directly edited from, the Info properties of the document. For example, in this document …

… we can see the Document ID, Document Type, Random Text and Event Type in the Info properties. These metadata properties remain with the document even when it is downloaded and so can be very useful to determine the origin of a document if, for example, it is found as an email attachment or on a network file share.

Where is this metadata embedded?

All Office documents with a document extension that ends in ‘x’ (e.g., docx) are based on an XML structure that can be viewed by converting the original to a zip file (using the CMD shell and REN to rename it) and then extracting the resulting zip file and opening it.

The XML structure of a standard Word document looks like the following. The XML documents contained in the structure can be viewed with simple text viewers like NotePad.

The system, added and embedded metadata properties are stored in one of the XML documents named item1 to item 4 in the customXml folder. The following shows the values applied in item1, while item 4 contains the other values that can be assigned to choice fields.

Metadata properties in the item1 XML document

Metadata values and options in the item4 XML document

The metadata for the Sensitivity label (including any formatting details) is stored in the ‘custom’ XML document in the docProps folder.

BUT – details of the Retention label are not stored in the XML properties of the downloaded document. This may be because once the document is downloaded, the retention label no longer applies.

What happens to that metadata when Word documents are moved?

As with the previous examples, a warning will indicate that metadata properties may be lost. If ‘Move anyway’ is selected, the document is moved.

If the document has an Information Protection label setting that prevents it, it may not be possible to move it. The following message will be displayed. The only way to move it is to change the label to one that allows it, or remove the label.

For any other Office document, the original Document ID and Retention label are retained but any added metadata properties will be lost if those columns do not exist in the destination library.

Moved document without the added metadata columns

However, it seems that the added properties are not completely lost; they are moved to the ‘custom’ XML document in the docProps folder.

Added metadata properties in the custom XML document

However, if the same metadata column is added to the library, documents that have this added metadata in the ‘custom’ XML document do not display that metadata. It must be added back.

What happens if the destination library has the same metadata columns?

If the destination library has the same added metadata columns, as shown in the example below, that metadata will be copied to the destination library. The first Word document in the example below was copied before the metadata column was added.

Summing up

When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:

• The original system generated metadata (Created, Modified) is retained for all types of content.
• The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
• Information Protection labels set on Office document only are retained unless a label setting prevents the move.
• Retention labels are retained for all types of content.
• Added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.

Ten things to consider when managing records in Microsoft 365

Microsoft 365 has become one of the world’s most accessed products for office collaboration. Jeff Teper, the ‘father of SharePoint’ at Microsoft, tweeted on 27 April 2021 that Teams had 145 million daily active users. (As reported in by the team at Office365ITPros.com.) According to the website ‘Microsoft Office Statistics and Facts (2021) | By the Numbers‘ , Microsoft Teams usage grew 40% during the COVID lockdowns.

Although organisations create, capture and store a range of records using the various Microsoft 365 applications, most records are likely to be created or captured in Exchange mailboxes or SharePoint (including OneDrive).

The volume and range of records has, in many respects, overwhelmed traditional standards-based models that required records (including emails) to be copied to a central electronic document and records management system (EDRMS) or identified and then ‘declared’ as records.

Given the reality of the new paradigm, organisations have tried various ways to manage records in Microsoft 365, including by retaining the EDRMS (for high value records), acquiring third-party products that promise to address the ‘gap’ in recordkeeping functionality, or working with the ‘out of the box’ capability.

Whichever method is chose, records managers need to have a very good understanding of where the records are in Microsoft 365 and how they can be managed. In many cases, leaving and managing them where they were created or captured (‘in place’ management), and using new and emerging capability to leverage the power of AI-based tools is likely to be the future state.

With the above in mind, and regardless of which method you follow, the following are ten points that I think are important to consider when managing records in Microsoft 365.

What are your recordkeeping obligations?

It is perhaps the most obvious question but organisations have sometimes rolled out Microsoft 365 without consideration of their obligations for managing records.

Records provide evidence of business activities and accordingly need to be protected to ensure their authenticity, integrity and reliability as evidence. The most common way of achieving this outcome until now has been to ‘lock’ digital records from change. Is this practical in the digital world? How do you lock Teams chats or stop a new thread in an email exchange? Could the same outcome be achieved by recording all changes and ensuring these are retained?

Records are usually subject to minimum retention requirements and understanding what these are is essential. Where there are minimum retention requirements, records cannot be destroyed before a specific period of time based on a particular trigger or event. These requirements are defined in legislation (sometimes based on statutes of limitation) or, for government agencies, in records disposal authorities or schedules (as shown in the example above).

As a starting point, look at these retention requirements and consider how these will be applied to Teams 1:1 chats, or team chats, or emails still in Exchange mailboxes, or OneDrive content. And then extend this to the content stored in Teams/Group-based SharePoint sites and sites not linked with Microsoft 365 Groups.

Consider also what is required to manage the outcome of retention. Do you need to review records due for destruction? Do you need to keep a record of what was destroyed? For all records?

There may be a requirement to categorise or classify records (especially to group them by context and/or for retention purposes where retention is based on classification). How will this outcome be achieved for Teams chats or emails that remain in Exchange mailboxes, or OneDrive content? What other metadata do you need for records?

Your recordkeeping obligations, in particular records retention requirements, should guide the management of records in Microsoft 365.

Where are the records created or captured in Microsoft 365?

Neither Microsoft 365 nor SharePoint is a dedicated recordkeeping system like an EDRMS (see my post ‘SharePoint is not an EDRMS’). Rather, it is an ecosystem of multiple applications that are used to create, capture, store and manage records.

Most records are likely to be stored in either SharePoint (OneDrive is a SharePoint service) or Exchange mailboxes.

A compliance copy of Teams chats are stored in a hidden folder of Exchange mailboxes. Content stored in the ‘Files’ tab is either stored in SharePoint or (for 1:1 chat) in OneDrive.

Of course, there will be some other records – Yammer conversations, tasks and plans, communication sites, calendar entries, forms and even WhiteBoard sessions. But, more than 95% are stored in Exchange mailboxes or SharePoint/OneDrive.

Knowing your recordkeeping obligations and the location of records are the main starting points. In fact, you can map your recordkeeping obligations (especially metadata and retention) to the location of the records.

Do you control the creation of Teams and SharePoint sites or not?

There two, broadly speaking, two approaches to controlling the creation of Teams and SharePoint sites:

• Yes, we have controls – There is some kind of control or decision ‘gate’ for the creation of Teams and SharePoint sites.
• No, we don’t have controls – End-users can create Teams and SharePoint sites whenever they want. In this case, the points below may not be of much use. You will likely rely on the built-in ‘records management’ capability to manage the records.

If your answer to the question above was ‘No’, then you will probably need third-party products and/or rely heavily on AI-based solutions to manage the records (which is the default Microsoft position).

Map sites and Teams to business functions – don’t mix content

Almost every organisation has a range of business functions. Some of these are common to all or most organisations (e.g., information technology, human resources, financial management, legal, property, etc) while others are ‘core’ (e.g., engineering, manufacturing, research, sales and marketing, etc).

Many organisations are structured around these business functions, and most records retention schedules are based on function (or business function).

If you can map new Teams and SharePoint sites to these functions, this will facilitate the management of records down the track. Mixing content across multiple functions – except where it makes sense to do this, such where there are related but smaller numbers of records in project sites – is going to make it harder to manage the records in the longer term – and more or less the same as letting end-users put whatever they want into a paper box for long-term storage.

A common example where records might be mixed are ‘Corporate Services’ areas that create or capture records across multiple functions, including property, IT, finance and so on. Unless all the records in a Team-based site can be kept for the same period of time, it may be a good idea to separate the records into different sites.

Also keep in mind that some business areas may exist for a long time; having a single (Teams-based) document library that has folders linked to channels may not be the best way to manage records long-term.

The suggestions above don’t take into account Exchange mailboxes, Teams chats or OneDrive accounts because these cannot be mapped to functions.

Naming conventions for sites and teams, and libraries

The main reasons for having naming conventions for SharePoint sites and Teams are:

• It is good practice, to avoid acronyms and other less than useful names.
• To prevent unnecessarily long names that end up creating very long URLs (e.g., ‘https://tenantname.sharepoint.com/sites/ExecutiveCommittee20202021MeetingsHeldDuringLockdownandrecordedviaMSTeamsSeniorManagersOnly‘.) It is important to know the difference between the URL name and the display name.

Ideally, the original names of Teams and SharePoint sites should be restricted to no more than 14 characters so that Document IDs (that have a 12 character prefix) can be the same as, or very close to, the URL name of the site. For example:

• URL (original) name: ExecMgtCtte
• Display name: Organisation Executive Management Committee
• Document ID: EXECMGTCTTE

Aside from the default ‘Documents’ library of every Teams-based site, library names should also be subject to naming conventions and restricted to around 20 characters. There are several reasons for this.

• The first is how they appear in the left hand navigation of a SharePoint site. There isn’t much point having multiple library names that aren’t easily visible (the two examples below have completely different names after ‘Financial Management’).

• The second is that long library names, especially if they include spaces, look terrible if they are sent as a URL. Spaces are replaced by ‘%20’:- h**ps://tenantname.sharepoint.com/sites/CorporateRecords/Financial%20Management%20%20Accounting%20%20Invoices%202021

• The third reason is that it is good practice to have some form of naming conventions.

Ideally, library names should map to the activities that produce the records AND include the year where this is relevant, e.g., ‘Meetings2021’.

There is NO need to repeat words in the tenant or site name – e.g., h**ps://tenantname.sharepoint.com/sites/TenantNameCorporateRecords/TenantNameFinancial%20Management%20%20Accounting%20%20Invoices%202021/Forms/AllItems.aspx

As noted above, this doesn’t apply to the default ‘Documents’ library in Teams-based SharePoint sites (the actual name is ‘Shared%20Documents’).

Metadata and Content Types

For many organisations, the minimum metadata requirements consist of (a) agent (e.g., the person who did something), (b) dates (when they did something) and (c) a unique identifier. That is, who did what and when?

If you need to add more metadata for certain types of records you can really only do this in SharePoint document libraries, including by adding them from the SharePoint Term Store (see below example). It can also be done in Outlook but these metadata terms are not linked with the SharePoint terms.

As for Content Types, do you really need them? SharePoint is made up of multiple Content Types already, including the default ‘Document’ Content Type. It is important to understand how Content Types work in SharePoint before making the assumption that they are required.

In many cases, choice metadata fields can replace the need for Content Types. Custom Content Types may only be needed for specific or high value record types.

Document retention policies and labels

In the first section about recordkeeping obligations, it was noted that most records will be subject to minimum retention requirements. Retention labels and policies are created in the Compliance admin portal of Microsoft 365.

Unfortunately, the current Compliance admin portal provides very little information to show what label or policy was applied where. The only way to do this is to document it yourself. One way to do this is to create a spreadsheet that lists on each row:

• The business function and activity from a File Plan or Business Classification Scheme (e.g., Financial Management – Accounting)
• Each retention class for that function/activity pair, including the reference number
• If that class has been created as a label, what the label name is. If it has been created as a non-label retention policy, what that retention policy name is. (Generally speaking, disposal authority classes don’t refer to Exchange mailboxes, SharePoint sites, MS Teams chats or OneDrive content, so the organisation may need to determine what this minimum retention period should be and how it will manage the retention outcomes).
• (Note, the above can be created in the File Plan section of the Records Management part of the Compliance admin portal, E5 licences only. However, it only documents the above information and does not show where the label has been applied.)
• Where the label has been applied ‘manually’ – to which SharePoint site/document library, Exchange mailbox or OneDrive account. This point may have multiple location references.
• Where the label has been auto-applied through the basic E3 option or, for E5 licences, the document understanding model (DUM) in SharePoint Syntex, or via trainable classifiers.
• When the retention will expire.
• Retention outcome – If a disposition review (E5 only) exists, the records will be destroyed automatically without any record kept, or ‘do nothing’. See also below.

Remember that retention labels and policies apply to individual items (emails, Teams chat, SharePoint or OneDrive content stored in libraries), not to aggregations (e.g., the entire library or site). The aggregation will continue to exist after the content has been destroyed and no ‘stub’ (a record of what used to exist) will remain.

How will you manage retention outcomes?

Generally speaking, Microsoft 365 retention policies destroy records when they are due for destruction unless they are subject to a label that has the disposition review option enabled or the ‘do nothing’ option has been selected.

Organisations need to understand how they will manage these retention outcomes especially as, in most cases, a review process is required. (See ‘Recordkeeping obligations’ above).

Even when retention label have the disposition review option enabled (E5 only), there are two points that need to be understood:

• The ‘disposition review’ interface presents individual records with no context except for the original site URL name. Some additional (default) metadata is now included (from May 2021) but not any added metadata. In most cases, it will be necessary to return to the original library to view the context of the records presented for disposal, and if there are any others.
• If records are destroyed through that review process, only basic metadata is retained about what was destroyed.

Organisations that have an obligation to undertake a full review of records due for disposal will likely need to consider establishing workarounds such as exporting the full set of metadata from a document library and then using that to review whether the content of the library can be destroyed. If approval is granted, that decision should be recorded, along with the metadata extract.

Allow end-users to get on with their work

End-users generally don’t have much interest in the management of records beyond the period of time they are important to them. They want to do whatever they want, whenever they want, using the applications they have available to them.

Collaboration no longer consists of email exchanges and document-based records. Creating control gates for the creation of Teams and sites, and insisting on naming conventions for sites and libraries (and folders) may be interpreted negatively.

There needs to be a fine balance between control and freedom and this can impact the creation, capture and management of records. Some of the ways to minimise the impact of recordkeeping requirements include:

• Enabling Document IDs on every site.
• Creating custom metadata columns on sites or libraries with default values.
• Applying non-label ‘safety net’ retention policies to all workloads. Retention policies (along with the Recycle Bin for 90 days) helps with the recovery of accidentally deleted content.
• Using various communication methods to highlight useful features including sharing (instead of attaching), the Recycle Bin, versioning in SharePoint/OneDrive, and the ability to have a ‘single source of truth’. These features can be used to ‘soften’ the impact of other recordkeeping obligations in some sites.
• Pro-actively monitoring activity across the Microsoft 365 ecosystem, including by monitoring the various dashboards, searches, and audit logs, and responding to events.
• Learning more about the Microsoft Graph Explorer and the potential to use AI-based options to manage records.

Use the system for other recordkeeping purposes

The Microsoft 365 environment can be used for other recordkeeping purposes as well. For example:

• Managing physical records stored offsite in a SharePoint list.
• Keeping a record of records (and SharePoint sites or other systems) that have been destroyed, as well as ongoing destruction review and approval processes.
• Publishing policies and procedures (in a SharePoint site, not necessarily a communication site).
• Communicating information about managing records (communication site).
• Archiving social media content (to Exchange mailboxes).
• Searching for content stored in other locations or systems including File Explorer and Line of Business systems (via connectors).
• Archiving network file share content, where it can be better protected and then subject to retention and disposal outcomes.
• Understanding where records are stored (via dashboards and Power BI reporting).

A decision tree for managing records in SharePoint and Teams

One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.

Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.

Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:

• Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
• The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.

Decision tree

With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.

The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:

The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.

Note: the box outline colours represent the four different types of site that are likely to be created.

How will the content be accessed?

It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.

Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.

They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.

If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.

It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).

What type of SharePoint site will be required?

Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.

As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.

Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.

Creation method

The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.

Security Controls

Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.

Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.

Retention

Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.

As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.

Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.

Provisioning

Once the site has been created, it should be provisioned with three key elements shown below.

• Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
• Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
• The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.

The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.

Managing the retention of records in Microsoft 365 with an E3 licence

Microsoft released its ‘Records Management’ solution for Microsoft 365 during 2020. The solution is only accessible to organisations with an E5 licence (or an E5 Security and Compliance licence).

Some of the retention-related options previously available to E3 licences, such as disposition review, are now only available with an E5 licence. However, for cost and other reasons, many organisations have decided to stay with E3 and asked if it is still possible to manage the retention of records.

According its licencing guidance ‘Microsoft 365 licence guidance for security and compliance‘, E3/A3 (and E5/A5) licences provide the following ‘information governance’ capability:

• A basic organization-wide or location-wide Exchange mailbox retention policy and/or to manually apply a non-record retention labeling to mailbox data.
• A basic SharePoint or OneDrive retention policy and/or to manually apply a non-record retention label to files in SharePoint or OneDrive.
• A Teams retention policy.

This post describes how the retention of records could be managed with an E3 licence and recommends that organisations intending to deploy Microsoft’s retention policies (whether E3 or E5) develop a plan for their deployment based on a detailed understanding of the following:

• What records exist and where they are stored across the main Microsoft 365 workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.
• What retention is required, including for EXO mailboxes and personal/ODfB accounts for which retention was previously ‘managed’ through backups, and MS Teams chats.
• What type of retention policy will apply to what workloads.
• How will disposition be managed (including of original storage locations, not just the records), and what evidence of destruction is required?
• How will any limitations and shortcomings be addressed to minimise legal risk?

Where are the records?

A good starting point with retention planning is to establish where the records that will be subject to retention policies are stored.

Most records are created and stored in one of the primary four workloads – Exchange Online (EXO) mailboxes, SharePoint Online (SPO) sites, Microsoft Teams (MS Teams), and OneDrive for Business (ODfB) accounts.

While SharePoint sites (including Teams-based sites and Teams channels) may be used to logically ‘group’ records (e.g., in a team site, a document library, or a channel), this is not the case for EXO mailboxes, ODfB accounts, MS Teams 1:1 or private channel chats (a compliance copy of these chats is stored in the personal EXO mailboxes of participants in the chat).

EXO mailboxes and ODfB accounts usually contain a range of records on different subjects, with different retention requirements. Personal chat messages could be about any subject.

For most of the past three decades, the requirement to keep specific emails or other records meant that end-users had to copy those records to another system, including an EDRMS or even SharePoint, or print and place them on a file. The ability to find old records depended more often than note on the ability to recover them from back-up tapes, a process ironically often referred to as ‘archiving’.

Action: Organisations should establish a list of where records are stored in Microsoft 365 and how retention (including via backups) is currently managed.

What retention is required?

The next step in the process is to establish how long these records need to be kept.

Most of the time this will be based on an organisational records retention policy or schedule. These policies or schedules describe groups of records and how long they must be retained – for example, financial records generally must be retained for seven years. Specific types of financial records may require shorter or longer retention.

This model generally works well when records have been stored in logical groups (e.g., a whole SPO site/Team, or document library), but are more difficult to apply for individual records stored with other records in personal EXO mailboxes, ODfB accounts, SPO sites, or MS Teams chats. If records with longer retention requirements in these locations are not stored elsewhere, or cannot be specifically identified, all the records may have to be kept for the longest retention period.

The start of retention is usually based on a trigger action. Typically these are based on (a) date created, (b) date modified, or (c) date of last action. However, they may also be based on less specific events, for example: (a) 7 years after a contract has expired, (b) 25 years after an employee leaves, (c) when a child turns 21. These less specific events need special attention.

Action: Organisations should ‘map’ retention requirements to the locations where the records are stored. This process will likely involve a discussion regarding the replacement (or supplementation) of email backups with retention policies, and may end up looking something like the following:

Workload	Retention
EXO mailboxes of senior managers	25 years
EXO mailboxes of all other employees	7 years
MS Teams 1:1 chats of senior managers	25 years
MS Teams 1:1 chats of all other employees	7 years
MS Teams private channel chats	As per retention policies
ODfB accounts of senior managers	10 years
ODfB accounts of all other employees	7 years
SPO sites that are not subject to more specific policies	7 years
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies
SPO sites with specific retention requirements, per site or library	As per retention policies

What are the retention options with an E3 licence?

The three main options available with an E3 licence – labels, label policies, and retention policies – are all set from the Compliance admin portal under the ‘Information Governance’ section.

The table below summarises the options:

Type of policy	Can be used for
‘Implicit’, ‘safety net’ retention policies. These policies: (a) work in the back end and cannot be changed by an end-user; (b) create a preservation hold library in SPO sites and ODfB accounts, and hold deleted emails in a hidden EXO mailbox folder; (c) provide an alternative to back-ups, although it should be kept in mind that all content that is retained in this way contributes to the overall storage quota; (d) do not retain record of what was destroyed.	EXO, SPO, MS Teams, ODfB
‘Explicit’ label-based retention policies. Labels must be published to the required workloads before they become visible. Records cannot be deleted once a label has been applied, but end-users can change or remove the label and then delete the record. Label-based policies do not retain a record of what was destroyed.	EXO, SPO, ODfB
‘Explicit’ label-based retention policies that are auto-applied based on three limited options – see below. Records cannot be deleted once a label has been applied. These policies do not retain a record of what was destroyed.	EXO, SPO, ODfB

The three auto-apply options are as follows:

• Apply label to content that contains sensitive information. Unlikely to be used as retention is never based on sensitivity.
• Apply label to content that contains specific words or phrases, or properties. Possibly useful. Recommend that organisations do a Content Search first to see what records may exist.
• Apply label to content that matches a trainable classifier. E3 licences provide six out of the box, limited classification options including ‘profanity’. These are unlikely to be of any value.

Mapping the E3 options

Given the options available, the following is a suggested mapping of records, retention and policy options:

Workload	Retention	Retention policy
EXO mailboxes of senior managers	25 years	Retention policy, plus possibly labels auto-applied to specific records
EXO mailboxes of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
MS Teams 1:1 chats of senior managers	25 years	Retention policy
MS Teams 1:1 chats of all other employees	7 years	Retention policy
MS Teams private channel chats	As per retention policies	Retention policy
ODfB accounts of senior managers	10 years	Retention policy, plus possibly labels auto-applied to specific records
ODfB accounts of all other employees	7 years	Retention policy, plus possibly labels auto-applied to specific records
SPO sites that are not subject to more specific policies	7 years	Retention policy
Microsoft 365 Groups (includes mailbox and SPO site)	As per retention policies	Retention policy
SPO sites with specific retention requirements, per site or library	As per retention policies	Retention policy (safety net) plus label-based retention policy for specific records

Policy limits

Organisations planning to deploy retention policies should be aware of the limits on custom policies, as described on this Microsoft page, ‘Create and configure retention policies‘. There are no limits on policies that apply to an entire workload (e.g., all EXO mailboxes).

• 1,000 mailboxes
• 1,000 Microsoft 365 groups
• 1,000 users for Teams private chats
• 100 sites (OneDrive or SharePoint)

The page above notes also that ‘There is also a maximum number of policies that are supported for a tenant: 10,000. However, for Exchange Online, the maximum number is 1,800. The maximum number includes retention policies, retention label policies, and auto-apply retention policies.’

How will disposition be managed

Microsoft 365 retention policies retain records for a specified period and usually then delete the records automatically. No record is retained of what was deleted. Even with an E5 licence, only limited metadata is retained and only on those records subject to disposition review.

Organisations deploying retention policies with an E3 licence need to understand the potential risks associated with being unable to provide evidence of what was destroyed.

There are at least two ways to approach this point, but in all cases the approach and options must be legally defensible:

• Ensure that organisational policies clearly indicate what records will be destroyed without any record being kept. For example, with certain exceptions, most emails, Teams chats, SPO sites and the content of ODfB accounts will only be kept for 7 years and then destroyed.
• Establish a process to ensure that a record is kept of specific records that have been destroyed. For example, the metadata of records, due for disposal and stored in document libraries in specific SPO sites, will be captured (and stored separately) before the records are destroyed, and then the document library will be deleted. This is a labour-intensive process and may only be used on some sites.

Addressing E3 licence limitations and shortcomings

The limitations of E3 licence retention policies (and also E5 licences from an evidentiary point of view) should not put organisations off using the out of the box options or cause them to acquire third-party products.

The retention options available in Microsoft 365 now provide functionality that assists recordkeeping compliance, for example the ability to apply ‘back end, safety net’ retention policies to EXO mailboxes, MS Teams chats, ODfB accounts and whole SPO sites. Coupled with the ability to locate all records, including ‘deleted’ records via Content Search, this should be a boon to records managers and also to IT (saving the latter from having to recover records from backup tapes).

On the other hand, deploying the options can be complex and requires good planning and governance documentation. The lack of good governance can lead to the inadvertent loss of records as KPMG found in August 2020 (‘IT blunder permanently erases 145,000 users’ personal chats in KPMG’s MS Teams deployment‘).

Once retention policies have been deployed, the next most complex task may be the disposition approval and destruction process for those records for which a formal disposition review and approval process is required, including the requirement to establish a list of records to be destroyed.

As noted above, even the E5 licence options for disposition review and ‘proof of disposition’ are inadequate in terms of the metadata that is presented and retained. Until Microsoft provides the ability to record the metadata of every record that is due for disposal and/or destroyed, based on ANY type retention policy this process will continue to be a manual task for records managers.

Organisations may consider acquiring a limited number of E5 Security and Compliance licences to gain access to the E5 ‘Records Management’ capability to gain access to other options including File Plan, auto-classification options, and Disposition Review. However, aside from these options (and the ability to auto-apply labels more broadly) these options may not add much more capability to the existing options already available with an E3 licence.

Summarising the options

Based on the above details, organisations with E3 licences might do the following:

• Create several ‘safety net’ retention policies for EXO mailboxes, ODfB accounts, MS Teams chats and general SPO sites. EXO/ODfB retention policies may be (a) based on current back-up periods and (b) be split into ‘senior’ and all other employees. They may also create policies for Microsoft 365 Groups which will cover the Group’s EXO mailbox and SPO site. Some of these policies may map to existing retention classes in the records retention schedules, but others such as the EXO, ODfB and MS Teams policies may need to be added.
• Create label-based retention policies for content stored in specific SPO sites where more granular retention policies are required per library. However, keep in mind these can be changed or removed by end-users with add/edit access. Generally it is not recommended to apply label-based policies to EXO or ODfB unless end-users will apply these accurately and consistently.
• Auto-apply some labels to specific content stored in SPO, EXO and ODfB (having identified that these records exist first via a Content Search). These labels should have a retention period that is longer than the safety net policy or any other label-based policy.
• Establish a ‘manual’ disposition review process for records that are subject to label-based retention policies.

References:

Microsoft 365 licensing guidance for security and compliance

Auto-populating Microsoft Word templates – a no-workflow option

In August 2019 I published a post titled ‘Auto-populating document templates via a form in SharePoint‘. The suggested model was based on using a SharePoint 2010 workflow in SharePoint Designer.

In July 2020, Microsoft announced that it would turn off SharePoint 2010 workflows for newly created tenants from 1 August 2020, and would remove the ability to run or create these workflows from existing tenants from 1 November 2020. It recommended that organisations use Power Automate instead. (Source: Microsoft Ending Workflows for SharePoint 2010 Online Next Month).

This post details an alternative option, with no workflows required – but with the same set up – that may be sufficient in most cases. At this sage (September 2020), there does not appear to be an out of the box option with Power Automate that will create a document (from a content type) when a new document set is created.

Overview

The alternative option requires the following elements:

• The document set feature enabled.
• New site columns for each unique metadata required in the output documents.
• New document set and document content types, both with the same metadata.
• One or more Word document templates.
• A SPO site with a document library. It is recommended that a dedicated library (e.g., ‘Client Agreements’) is created for this purpose.

Note – the person who does this must be a Site Collection Administrator.

The main difference with the earlier model is that there is no workflow to create the document/s. Instead, the end user creates and saves (without modification except a name) the document into the library/document set where the custom metadata is stored. The method below then auto-populates those new documents with the required metadata.

This option does not use a separate list as the metadata details can be extracted from the document sets.

Enabling the document set feature

The document set feature must be enabled via Site Settings > Site Collection Administration > Site collection features. If you don’t see this option, you don’t have the required level of permission.

It may be a good idea to activate the Document ID Service at the same time as this can be a useful additional piece of metadata in the final documents.

New site columns

From the same Site Settings area, go to the Web Designer Galleries section to add the required new site columns. Directly below that option is where you create new site content types.

The Site Columns section has a very large number of default columns that come with every SPO site. Check to make sure the required columns don’t already exist, or are non-recommended column names (like ‘Library’).

Assuming the required columns do not exist, select the ‘Create’ option to create a new custom column. A couple of points to keep in mind here:

• Give the column a name that is obvious (easy to find).
• Don’t put spaces between words but capitalise each word (e.g., ‘ClientName’, ‘ClientAddress’). From experience, this space can cause the metadata element to wrap to a second line and cause problems.
• Keep in mind that the ‘Single line of text’ is free text. If you want controlled choices, use the Choice option instead.
• Date and Time may require some tweaking to get the correct format in the output Word document. It may be easier to use ‘Single line of text’ if some dates may vary or aren’t exact.
• Don’t use the ‘Yes/No’ option. Instead, use a Choice column with those options instead.
• Person or Group is only for internal names recorded in Active Directory (AD). If you need to insert any other name, use the single line of text.

The additional column settings that may be useful are:

• Require that this column contain information. If changed to ‘Yes’, this metadata field will be mandatory.
• Enforce unique values. Generally not used unless there is a requirement to enforce unique values (e.g., unique client IDs).
• Maximum number of characters. This can be useful to restrict the number of characters. For example, ‘Postcode’ will only have four characters. Don’t use the number field for postcodes.
• Default text. This option is useful if there is a requirement to have a default value, especially from a choice list, or if the value will usually always be the same (although that could also be simply added to the text of the document template.
• Column formatting. Don’t use this if the content is to be copied to a document template.
• Column validation. This may be useful to ensure that only certain characters are entered. For example, postcodes will never have a letter, only numbers.

Repeat this process until all the required metadata columns exist.

New document set content type

Create a document set content type so the output Word documents (and any other digital content) can be stored in the same ‘folder’.

To create a new document set content type, return to the Site Settings and, under Web Designer Galleries, select ‘Site content types’.

Give the new content type a name that is obvious for its purpose (e.g., ‘ClientAgreementsFolder’).

Importantly, change ‘Select parent content type from’ > Document Set Content Types, and ‘Parent Content Type’ > Document Set.

Open the content type that has been created. Under the ‘Columns’ section, add the columns ‘from existing site columns’. The example below shows a series of site columns that were added. Don’t remove any of the other columns.

In the above example. the ‘Title’ column is hidden (default setting). If you click this column, you will see the option under the ‘Change Content Type Column’ to make it required, optional or hidden. Generally, these settings should not be changed.

New Document Content Type

Now, create a new document content type. Give the content type a name that reflects its purpose. Ensure that ‘Select parent content type from’ is set to ‘Document Content Types’ and ‘Parent Content Type’ is set to ‘Document’ as shown below.

The next step is to add the two content types to a document library.

Adding the content types to a document library

All the steps below can be carried out by a Site Owner.

Open the library where the content types are to be added and go to Library Settings (via the Cog/Gear option). In the Library Settings, under ‘Advanced Settings’, click the option to ‘Allow management of content types’ (change from No to Yes).

If you don’t want other folders to be created in this library, change ‘Make ‘New Folder’ command available’ from Yes to No.

A new section will now appear in the Library Settings – ‘Content Types’.

To add the two content types that were created, click on ‘Add from existing site content types’, select each content type from the list that is displayed, and click OK.

The two content types should now appear in the list of Content Types and the custom site columns should appear in the ‘Columns’ list.

Click on the document set content type (e.g., ‘ClientAgreementsFolder’). The custom columns should be visible. Now, we need to share these columns with the custom document. Click on ‘Document Set Settings’.

• From the list of ‘Available Site Content Types’, chose the custom document content type that was created above. Ignore the ‘Document’ content type.
• In the ‘Shared Columns’ section, check the box against all the custom columns.
• Click OK.

The steps above will mean that all the custom columns in the document set will be shared with the the document content type. When you return to the Library Settings, you will see under the ‘Columns’ section that the columns are now shared with the two custom content types as well as the default Document content type (which can be ignored).

Add the Word template to the document content type

Now, add the Word template to the document content type. Note that this step is NOT done at the Site Content Types level, but at the local library level.

Ensure the Word document template is in the latest version of Word (e.g., docx) and has all the text required. It can be updated at any time (see below) but it is important to ensure that the document is as complete as possible, and the sections where the metadata is to be placed is obvious. One way to do this is to put text such as CLIENT NAME in the body of the document. Avoid using highlights as this can be troublesome (but not impossible) to remove.

From the Library Settings, Content Types section, click on the custom document content type (e.g., ‘ClientAgeements’). Click on Advanced settings.

Click ‘upload a new document template’ and add the document template. The name of the template should be obvious. It is possible to go back and edit the template at any time after it has been uploaded. This is usually much easier if a change is required, compared with uploading a new template and having to re-link all the metadata.

Connect the metadata to the document content

From the Document Template section above, click on ‘Edit Template’. This should open the document in the installed Word application. These steps cannot be carried out using Word Online.

Place the cursor in the body of the text where the metadata is to be placed. Click on the ‘Insert’ tab in Word, then click on ‘Quick Parts’ in the text section.

Click on ‘Document Property’ and the custom columns will appear.

When you select each property, the word document will display it as shown below. Continue this process until all the required metadata is added to the document:

As noted earlier, it is usually much easier to edit an existing template ‘in place’ instead of uploading a new version because the steps involved in adding the metadata to the template must be repeated if a new template is uploaded.

Auto-populating Word documents

Now that all the elements are in place, metadata is added automatically to the Word template as follows.

First, open the document library. Click on ‘+ New’ and choose the document set content type (e.g., Client Folder).

Fill in the metadata elements required, then click ‘Save’.

Now, open the document set/folder, and click on the document content type (e.g., ‘ClientAgreement’). This will open the Word template.

Don’t change the document, just click File – Save As. This should display the list of SharePoint sites on the right. Give the document a name, then choose the relevant site (e.g., ‘Client Agreements’), then the relevant library and document set/folder and press save. This is the most complicated part of the process once it is set up.

The template document, with all the metadata elements, should now appear in the document library under the document set/folder. As it is a Word document, it is possible to ask someone to sign on glass, then ‘save as’ a PDF.

If the document requires a sensitivity label and these are enabled, this can be added directly from the Word document menu when it is created.

Other relevant documents (e.g., emails) could be added to the same folder, including drag and drop from Outlook or other locations when the library is synced to File Explorer.

Note that the metadata elements that were added to the document in this way form part of properties of the document. These properties, which will remain with the document if it downloaded or attached to an email, can be viewed from the ‘File – Info’ section of the Word document.

Summary

The disabling of SharePoint 2010 workflows in SharePoint Designer has removed the ability to auto-create a document in SharePoint with the required metadata. However, it is not particularly difficult to set up a site, with two custom content types, one of which contains a document template, to achieve the same outcome without any workflow at all. The only complicating factor is the requirement to save the document template back to the library – without making any changes at all.

Data-based records

(Note – the image above is a small ticket dated 1956 from my grandmother’s visit to Denmark. I used this because the word ‘Kontrolbillet’ seemed appropriate for this post.)

In response to several queries about this following my previous post about whether it is possible to manage records as data, it seemed apparent that the data-based nature of contemporary modern digital content formats, especially Office documents, is not well known.

This post provides details of the data structure content of a typical Word document, to help explain why such records could be seen (and managed) as self-contained data sets.

Just to be clear, the idea of managing records as data does not remove the need or business requirement to store and manage records in ‘local’ aggregations or context – a SharePoint document library or a mailbox for example (less so a network file share because of the limited metadata, but still possible). These aggregations will generally map to business activities, can have specific metadata requirements and can be used to control access to and retention of records as long as they need to be managed.

Managing records as data is a more holistic data analytics concept that allows organisations to better understand and analyse records amidst the volume of all other digital content. It should should help to ensure that all records on a given subject or context are managed appropriately through time, and that, wherever possible, only one copy exists.

A document in a SharePoint Online library

For this example, a document is stored in a SharePoint Online document library called ‘Client Agreements’. The library has a set of metadata columns that must be added to every record. The library uses document sets but it could equally use metadata or folders, the important point is that metadata is added to the library.

The metadata added to the library can be anything, including terms from a business classification scheme. The metadata can be mandatory or optional, and can be set as default options – for example, you may want every document in a library to automatically have the same function and activity terms.

In the screenshot below, we can see the document library with two document sets (a type of folder). The library has four added metadata options: Client Name, Client Reference, ClientRef, and Date of Birth (not visible in the screenshot but we’ll see it later).

The metadata properties

Here are the metadata columns for the library. As we will see below in the actual data, metadata columns with a space between words results in additional characters (‘_0020_’) replacing the space.

The columns from ‘Created’ downwards are all default columns

When I open the Harpin ‘folder’, I can see the metadata columns next to a document. In this case they were added to the document automatically as the documents inherit the same metadata properties as the document set. This is set via the Document Set settings – ‘Shared Columns’:

Alternatively, the metadata can be added to each new individual document when the document is added.

If the Harpin document is selected as shown below …

… the information panel on the far right shows the metadata properties for the document (and also the activity – when the document was modified and by whom, and who viewed it):

As this particular document is a Word template added to a content type in the library, an end user can to select it when they create a new document in the library as shown in the screenshot below. Alternatively, the ‘Client Folder’ option allows them to create a new document set folder with all the metadata that relates to the client; this data is then inherited by any document created in the library:

If the document is opened, you can click on File – Info and see the metadata properties already added TO the document in the library. These properties remain with the document even if it is downloaded and/or attached to an email. If Document IDs have been enabled, that metadata value is also added to the document properties, meaning we can see that it came from a SharePoint library (and which one):

Because it is used as a template, the Word document can make use of the metadata added to the record in the body of the document, in addition to the metadata forming part of the properties for the document.

The metadata properties have been automatically added to the body of the document

The XML properties

Let’s now look at the XML of the document.

Download the document to an accessible location. Using the Command Prompt (CMD), rename the document to .zip (You cannot do this from File Explorer). From File Explorer, the original file will now have the extension .zip. In the list below, the other file with a similar name is a copy, but the size is identical.

Now, unzip the zip file (right click, Extract All).

Here is the top level output, which is standard for all Word documents.

Open the ‘customXml’ folder and you will see a set of XML files:

Open item1.xml, and you will see the custom properties which, as you can see, includes both the Document ID as well as the original path SharePoint site/library location. Just to be clear the Document ID ends in ‘119’, which is the actual document; the original document set folder’s ID ends in 118 (scroll up to check):

To avoid ‘_x0020_’ in the metadata properties, don’t use spaces in the metadata column names

As can be seen, the document that was downloaded has the unique Document ID embedded in the metadata. Note that this ID will change if the document is uploaded to a different library.

In the ‘docProps’ folder we find three sets of XML files:

In the ‘coreXML’ file we see the Dublin Core (DC) metadata that you see in the document Properties above. You can add all the Dublin Core metadata to the library, they are built-in to every library, which means that every document can have all that metadata.

The actual content (the body) of the email is found in the ‘word’ folder of the XML files. Here is the content of that ‘word’ folder:

In the screenshot below you can see some of the ‘document.xml’ content including the metadata that has been added in the body of the document (separately from the properties of the document).

All this metadata is accessible and is used by the Microsoft Graph.

Excel files

Excel files are interesting because, in a sense, they contain data within data. Here is some data in a spreadsheet:

This data is – strangely – stored in two different XML files. The text (including the column headings) is stored here: \xl\sharedStrings.xml:

The values are stored according to each worksheet. For example: \xl\worksheets\sheet1.xml (first two rows only)

A note about emails

Emails do not have the same XML-based structure as Office documents and generally cannot have additional metadata added (except as tags).

Emails in Outlook (sent or received) become ‘.msg’ files if saved to another location from Outlook.

The ‘.msg’ format is based on CFB_3, or compound file binary format, a format that was also used by earlier versions of Microsoft Office documents. It is ‘a general-purpose file format that provides a file-system-like structure within a file for the storage of arbitrary, application-specific streams of data’. (Source: Microsoft web page on Compound File Binary File Format).

Copies of Microsoft Teams chat messages are also stored in a hidden folder in Exchange mailboxes, as instant messages. They cannot be accessed directly but should be considered as a type of archive copy – the originals are stored in a separate database.

If emails are saved to a SharePoint document library, they can be described with additional metadata while stored in the library, but this metadata does not become part of the core metadata of the email or remain with it if it is downloaded, as it does with other Office documents.

In any case, whether they remain in Exchange/Outlook mailboxes, are copied and stored in SharePoint or other Microsoft-based locations, the metadata content in them is accessible via searches.

Active Directory completes the relationships

Every digital record has an author and is likely to have contributors (modified by). Every email is sent and received by someone. All of the internal names linked with digital content are recorded in an organisation’s Active Directory. Employees are also likely to be added to Security Groups (sometimes known as AD Groups) that provide a way to control access to IT resources.

The relationship between document-based content (documents, emails), and between people in AD Security Groups, provides the ability to establish relationships between content, people and business activities.

A final word

Importantly, managing records data does NOT remove or exclude the business need or requirement to aggregate (e.g., in document libraries, mailboxes), manage through time, and then destroy or transfer records according to business requirements. Instead, it enhances this capability by ensuring that all records about a given subject or context can be identified and that, as much as possible, only one copy of the record exists.

Managing digital records as data

(Note – while drafting this post I became aware of an MA Dissertation on the subject of ‘Artificial Intelligence and Record-keeping’ being developed by Mohamed Ben Tahayekt at University College London. I have not had access to this material but I believe some of the concepts may be similar to those outlined in this post.)

Digital records have long been thought of (and described) as being ‘unstructured’.

The reality, however, is that almost all contemporary text-based digital record is made up of a defined, structured and mostly open or accessible package of data that is based on standards. For example:

• Microsoft Word, PowerPoint and Excel documents are all based on an XML structure (indicated by the ‘x’ on the end of the file extension) described in ISO/IEC 29500 and ECMA 376.
• Google Docs exist only in an online format (described in this Google site); to access them offline they must be converted to one of the following formats ISO/IEC 29500 format, ODT (ISO/IEC 26300), PDF or html.
• Emails are now mostly based on the Internet Messaging Format (IMF), standardized by RFC 5322.
• PDFs are based on the open standard ISO 32000.

All of these standards support interoperability between systems (and devices). (See my post about Metadata Payloads for more information on this subject).

An exception to the above are binary objects, including digital photos and images and where these are embedded in text-based documents. But even so, most binary objects are stored with a range of metadata to describe them.

EXIF metadata automatically captured from a smartphone camera

Given that text-based digital records are already full of readable and accessible structured data (and binary objects come with a range of descriptive metadata), is it possible to manage digital records as self-contained data objects?

Records and context

Digital content (records and non-records) will be always be captured, saved to or stored somewhere:

• In email mailboxes. Emails of course may include attachments that duplicate records stored elsewhere in the system, or are not stored anywhere else – e.g., received from outside the organisation.
• In a drive/folder structure in a network file share location, including ‘personal’ drives.
• In a library/folder in online file storage and collaboration platforms, including ‘personal’ online storage locations.
• In corporate enterprise ‘social’ platforms such as the intranet.
• In corporate messaging and chat applications.

Some of the above may have well-defined ‘filing’ or storage structures (including folders) that are used to store or ‘file’ records. Some of these may include the ability to classify and categorise records, and add additional metadata.

An example document library in a SharePoint site

In an organisational setting, all of this digital content will be created, sent/received, or modified by someone listed in Active Directory (AD), a system that generally links employees through their organisational structure. Additionally, employees are likely to belong to several AD Groups that further define relationships between them.

These relationships are important as they help us to understand the context for records.

Isolating records from other content

But one of the challenges for any organisation is knowing what is a record and what isn’t. Perhaps that isn’t as important as it sounds, if all the digital content is considered a potential record.

Organisations create or receive and store a lot of digital content, and a lot of this content has traditionally been kept (on backup tapes) for a long time to support disaster recovery and investigation purposes.

Only a percentage of this content is likely to fit the standard definition of a record – ‘evidence of business activities’.

And some digital content may not obviously be a record until it is connected with or related to other content or activities. For example, a simple email that says ‘Yes’ or ‘OK’ may be the record of agreement to something that doesn’t form part of any other obvious records until it is identified as being a record.

Not uncommonly in traditional electronic recordkeeping systems, there could be no guarantee that everything copied there was a copy of every record that existed on a given subject. Additionally, a record stored in a recordkeeping system may be of relevance in other contexts.

The key to what a record might be is the word ‘evidence’; this is exactly what lawyers look for when they conduct eDiscovery activities.

Rather than assume all records can be accurately found and managed amidst the volume of all digital content, it may be more efficient and accurate to assume all digital content is a record and then apply rules and tools to manage that content, with the aim of identifying records and their potential context based on the data contained in the individual digital objects and their relationships with both other records and people.

In other words find records amongst the entire content, rather than seeking to isolate only those digital objects that are identified as records and copy them to another location – while leaving the originals and potentially other related records in place. Managing records this way avoids the problem of email threads or chats that continue after the copy has been made, or a new copy of a Word document appearing.

How can we achieve this outcome?

There are three potential ways to manage records as data.

The first is to understand, even in general terms, is that digital content is not unstructured, and to learn more about how they are structured. Some simple examples:

• Every email (and instant messages) has a sender, recipient, date sent, date received. They usually (but not always) have a subject. The text-based body of the email provides an additional form of accessible data. A quick look at email headers reveals a huge amount about the email.
• Every document (and web page) has an author, created dated, modified date and last modified by, and a name. They also have a large amount of other data, a lot of which is visible in the Properties section.
• Photographs are stored as binary objects but have a range of EXIF metadata that includes the creation date, information about the camera settings, and may also include the name of the person who created it, as well as a GPS location.

Word document properties showing data added to the record and storage location (Doc ID reference)

The second is to understand that digital content may include added data or metadata. This added data may relate to or derive from the location where the record is stored, or may be added by end-users as part of their work. It may include a unique identifier and information about the aggregation where it is stored, as well as recordkeeping classification terms. Additionally, it may include both process metadata (modified by, and when) and security or access control metadata. Depending on where it is stored, this additional metadata may be embedded with the document properties (the metadata payload).

The third is to have access to (ideally) all digital content across the organisation, and the necessary tools (or access to people with them who can provide usable output) to search and retrieve, relate, and manage all digital content on any given subject or context through to disposal. A very simple example of this is to run a PowerBI report across the network file shares.

And lastly, while there will always be some form of ‘local’ aggregation for specific records where all the records are stored in the one place (mailbox, document library, folder), the only way to establish an aggregation of all digital records on a given subject or context using data only is through the use of advanced searches and/or eDiscovery tools and/or data reporting or visualisations and/or artificial intelligence to find, link and relate content.

Linking and relating content

The diagram below from Microsoft about its Graph technology, provides a simple example of how content can be linked and related through its data.

There are now many data analytics and data visualisation tools that help to understand digital content. These tools are just one part of the picture.

Data analytics tools (such as ‘Constellation‘, a joint project between the Australian Signals Directorate and the Australian CSIRO) are a starting point to understand digital content – including digital content from line of business systems.

A data visualisation from the Constellation website (link above).

These tools might be used to identify content or people related to a given subject, through chat messages, emails, documents etc, including content that is already linked through its own context – the mailbox or a SharePoint library. From that information it would be possible to build a picture – types and volume of content, people, and the relationships between them.

A global search should be able to retrieve, and if necessary export, the content, keeping in mind always that the nature of digital content means it may continue to be modified or new content may added at any time.

As searches improve, narrower set of content allows more granular analysis and visualisation, allowing the identification of sub-sets of records within broader sets. For example, of the potentially large group of ‘everything about COVID’, just the narrower set ‘Vaccines’.

All of this could be achieved through the data that makes up the digital content. And many data-driven organisations are likely to be doing just this, using a range of business intelligence tools to understand the information available to them, in both line of business systems and other content.

Can we manage records as data

Perhaps ‘manage’ is not the right word, or at least not in the sense of expecting digital records to be managed as exceptions to the rest of the digital content.

But there is a lot more we can do to make this outcome possible. We can leave the records where they are stored or captured, we can apply local structure to those records, or security controls. We can keep records of changes that are made. We can apply retention rules that prevent the destruction of any record, or potential record, before it can be legally destroyed.

Instead of ‘managing’ records as exceptions, we can leave the data where it was created or stored, and use a range of tools to help us understand and manage it.

This will allow us to manage records as data and finally achieve the ‘semantic office‘ I wrote about in 2010.