A basic retention model for Microsoft Teams

In my previous post about managing inactive Teams, the third option listed was to apply retention policies to those Teams. It included the graphic below.

This post provides more details of a basic retention model that can be applied to both active and inactive Teams.

Key takeaways

Key takeaways from this post for records and information managers:

• Every Team has a ‘Posts’ (group chat messages) and ‘Files’ (documents etc) tab, and usually also starts with a Wiki tab (which can be removed). Other tabs may be added via the + option.
• A Team in Microsoft Teams is not a single container or aggregation for the capture and storage of records. Almost all the records in a Team are stored in a hidden folder in Exchange Online (EXO) mailboxes (posts) or SharePoint Online (SPO) (files). Some records (conversations) may also be created and captured in the EXO mailbox of the associated Microsoft 365 (M365) Group.
• It is not possible to apply a single retention policy to a Team; at least two separate policies will be required – one policy for the Team channel posts of EVERY team, and one or more policies for the content captured in SPO sites (files) or groups of sites.
• Some records, created in and accessible from Teams, may be stored in other M365 applications (e.g., Tasks, Forms, WhiteBoard, etc) or third-party applications. It is not possible to apply any Microsoft 365 retention policy to records created by or captured in these applications.
• Records and information managers should have access to the details (not necessarily the content) of every M365 Group, Team, and SPO site in order to establish a plan for the creation and application of retention policies to Teams. At a minimum, they should be assigned the Global Reader role (for details of M365 Groups and SPO sites) and the Compliance admin role (for retention policies).
• It is relatively easy to overcomplicate the retention model for Teams, for example by applying separate retention labels to different folders and sub-folders in each channel ‘files’ tab.
• Try to keep the model simple for as long as possible.

Core components of a Team

The main components of every Team are shown in the diagram below. If private channels are not allowed in the organisation, ignore the top two left and right elements.

The relationship of a Team to its M365 Group, Exchange mailbox and SharePoint site, showing where the content is stored (dotted lines).

As shown in the diagram above:

• Every Team is directly linked with an M365 Group. Every M365 Group has an Exchange Online (EXO) mailbox and a SharePoint Online (SPO) site.
  • The Team, M365 Group, SPO site, and mailbox address (teamname@) all share the same name. The original name (which should be brief, <20 characters if possible) and the display name may be different.
  • The Owners and Members of the Team are the Owners and Members of the M365 Group and those Groups are added to the SPO site Owners and Members permission groups respectively.
• A ‘compliance copy’ of every post in a normal channel is copied from the Azure-based Teams chat service (which is always inaccessible) to a hidden folder of the EXO mailbox of the M365 Group linked with the Team.
  • Where private channels are allowed, a ‘compliance copy’ of every post in a private channel is copied to a hidden folder of the ‘personal’ EXO mailboxes of all participants in the private channel.
• Any content created or captured in the ‘Files’ tab of the Team channels is stored in the SPO site of the M365 Group linked with the Team. If any lists are created, they are either stored on the same SPO site or are linked from another site.
  • Where private channels are allowed, a separate SPO site is created (using the name of the ‘parent’ site followed by a hyphen then the private channel name, e.g., parentsitename-privatechannelnamesite). Any content created or captured in the ‘Files’ tab is stored in that SPO site.

So, a Team is a combination of at least four elements: the Teams user-interface (and back-end database), an M365 Group, a SPO site, and an EXO mailbox. The mailbox is used for three main purposes:

• Email-based ‘conversations’ (when used).
• Calendaring.
• Storage of Teams posts.

This is why it is not possible to apply a single retention policy to a Team.

The basic retention model

The basic retention model for Teams assumes the following:

• If the organisation’s retention schedule/disposal authority does not include coverage for Team posts (chat messages) and also general Team chats, there is a legally defensible policy that defines how long Team channel (including private channel) posts (and chats) will be retained. Note: This policy will define a single retention period for ALL posts and and a separate policy for ALL chats.
• Records and information managers know the details of every M365 Group, Team (including number of private channels) and SPO site (including last activity and number of files).
• One or more retention policies will be created for SPO sites.
• One or more retention policies may be created for M365 Groups.
• Unless it is done ‘manually’, there will be no review process before the content is destroyed at the end of the retention period.
• No label-based retention policies will be applied (at this point). They may be added later as required (see below).
• Unless the option to auto-expiry M365 Groups is used, there will be a manual process to delete inactive and empty M365 Groups or Teams; deleting either will also delete the linked SPO site.

Creating retention policies

Retention policies are created in the Information Governance section of the M365 Compliance admin portal under ‘Retention policies’.

Generally speaking, organisations should not create many of these policies as they should ideally target entire workloads (all SPO sites, all EXO mailboxes, etc) or in some cases major groupings (e.g., EXO mailboxes of senior executives, all other mailboxes).

And remember, these policies do NOT destroy the container (Team, SPO site, EXO mailbox), only the content in those containers.

Every new retention policy has three parts.

Name

The name of the retention policy should be easily recognisable, for example ‘Teams channel posts 7 years’ (all encompassing, for all channel posts, see next dot point), or ‘General SPO site retention 7 years’. The name section also includes a description that should always be used to link the policy to details in a retention schedule/disposal authority or corporate policy.

Location

The ‘location’ element is where the complexity arises as it is not possible to create a single retention policy for all the elements in a Team. Selecting either ‘Teams channel messages’ or ‘Teams private channel messages’ will disable all other options. It is not possible to select ‘SharePoint sites’ or ‘Microsoft 365 Groups’ AND any of the Teams options in the same policy.

Because of this limitation, at least two separate retention policies will be required for a basic retention model, with an additional one for private channels (if required):

• A retention policy for either all or selected SharePoint sites, including private channel sites. The simplest model is to create a single retention policy for all SharePoint sites. This creates a preservation hold library on every site, retaining all deleted content for the minimum period required. Alternatively, and especially if there is a way to ‘group’ SPO sites (e.g., all project team sites), create retention policies for those groups and add in the site names. Always keep in mind that a retention policy applied to the SPO site has no connection with or impact on the channel posts.
• A retention policy for all Teams channel messages. Note that this cannot include or exclude any Teams – it’s all or none. Depending on the retention selected for channel posts (next point), this could mean that channel posts are destroyed before (or after) the Team’s SPO content.
• A retention policy for all Teams private channel posts. Similar to the previous point, this is an ‘all or none’ policy.

If the Team is also making use of the M365 Group’s ‘conversations’ in Outlook, consideration may also be given to creating a retention policy for M365 Groups (or included/excluded Groups). This policy will cover (a) Group ‘conversations’ and (b) the SharePoint site linked with the Group/Team. It will NOT cover the Team channel posts that may be stored in the M365 Group EXO mailbox. Note: It is possible to select just the M365 Group mailbox OR the M365 Group’s SPO site in this policy via a PowerShell script.

Retention period

Retention options are shown in the screenshot below. These options are the same for every retention policy.

Retention policies either automatically delete content after a minimum period or do nothing (includes the ‘retain items forever’ option). There is no disposition review. This means that the content in the SPO site and Team channel (including any ‘deleted’ content, which is not actually deleted, just hidden) simply disappears when the retention period expires.

Retention variations

Organisations may of course have different requirements or decide to apply retention differently. Each of these will still be some variation on the above model.

In most cases, there should be at least one retention policy in place for each of the different elements that make up a Team – the M365 Group, the SPO site, the channel posts, the private channel posts. Whether those policies have the same retention period will be up the organisation to determine, but in all cases, the details should be documented somewhere as currently this information is not easily available.

Retention labels

It is not possible to apply retention labels to Teams channel or private channel posts (or chats). There is only one option, and that is a single retention policy for each of these.

Retention labels may be applied to the content stored in the Teams linked SPO site, and these may be applied instead of using retention policies. This may be an effective model when combined with auto-expiry of M365 Groups as this (auto-expiry) will not occur if the content is subject to an active retention policy or retention label.

However, applying labels to the content stored in each Team channel ‘files’ tab has the potential to be a very complicated model that will become almost impossible to monitor or manage in time.

Each channel ‘files’ tab maps to a folder with the same name in the Documents library of the linked SPO site. As each Team channel may have been created for the records of a different subject with a different retention requirement, this means that each folder (or potentially even sub-folders) in the library may have a different label.

As retention labels (and policies) apply to individual items in the library (but not the folder), this means that individual items, stored in folders, that are subject to disposition review will come up for review in the future.

The application of multiple retention labels to folders within the single Document library of the SPO site is already complicated; having to review some of the individual items as part of a disposition review in the future is just adding to the complexity.

My view is that Teams should, as far as possible, ‘contain’ records relating to the same subject with the same single retention period that can be applied to the entire SPO site. Applying individual labels to folders or sub-folders within a single document library is a complex model both to apply and manage into the future.

What do to with empty Teams?

As noted already, retention policies (and labels) do not delete the SPO site, Team or M365 Group, only the content stored in them. Each of these ‘containers’ remain after the content has been destroyed within them.

Accordingly, it is advisable for records and information managers to (a) have access to the details of every SPO site, Team and M365 Group and (b) work closely with IT to determine when these containers can be deleted (and document that activity). Otherwise, the M365 environment will be left with the hollow shells of sites, Teams and Groups.

Further reading

The following Microsoft links provide further details on this subject.

Learn about retention policies and retention labels

Learn about retention for Microsoft Teams

Learn about retention for SharePoint and OneDrive

Create and configure retention policies

Apply retention labels to files in SharePoint or OneDrive

Teams messages about retention policies

Featured image: http://www.pexels.com

Combining metadata with folders in SharePoint

As organisations increasingly move content from network file shares to SharePoint, many have found that folders in SharePoint (and via the Files tab in Teams channels) are a bit less friendly.

For example, there is no way to ‘expand’ the folder view, so it is not possible to see the folder hierarchy (unless the library is synced to File Explorer). And although it is possible to add more or less unlimited metadata to content stored in document libraries, this may be a pointless exercise given that the metadata is not yet visible in File Explorer and folder-based structures are more familiar.

Fortunately, SharePoint not only has the ability to use both folders and metadata, but also to see all the content in a document library without the folders AND using that metadata to group, filter and sort the content regardless of how the folders were structured. This capability can be (surprisingly) useful for records managers.

This post explains how it works, and why this option doesn’t work in the File Explorer view of a synced library.

A typical folder-based library

The screenshot below shows three folders (out of many more) in a fairly typical document library for a SharePoint site.

A typical folder-based library

SharePoint (via a browser) does not show a hierarchical view of folders, so the end-user must click on every folder to find out what they contain. (And yes, you can enable the ‘Item child count’ option but that only tells you how many items are directly under the folder, there is no expanded view).

The ‘Files’ tab in Teams also presents a the same view of the Team’s linked SharePoint site, except that each channel maps to a folder in the default ‘Documents’ library, so that folder (content, if any) is displayed.

Adding metadata

SharePoint allows for almost unlimited metadata on document libraries and lists. (Whether you need to add unlimited metadata is a separate question).

Metadata (known as ‘site columns’ or ‘library columns’) may be created at the site or library level, and then added to any library on the site via the Library Settings > Columns > ‘Create column’ OR ‘Add from existing site columns’.

The screenshot below shows an example of a site or library column named ‘Document Type’ to be added to the ‘Meetings’ library. This column has three choice options because there are typically three types of document in this particular library: Agendas, Drafts, and Minutes. You can of course have as many choices as you like but consider the end-user experience. No-one wants to scroll through a hundred choices.

The column above has three choices (selected from a drop down menu), is not mandatory, does not enforce unique values, does not allow end-users to add their own options, and has a default value set. This means that every new document added to the library will be assigned the ‘Agenda’ choice.

Default values may also be useful to automatically add recordkeeping terms such as ‘Function’ and/or ‘Activity’, without the end-user having to do anything.

Once added to the library (via the Library Settings > ‘Create column’ or ‘Add from existing site column’), the metadata column can be displayed in the view, and the metadata options may be added to the documents via ‘Edit in Grid View’ or by adding or changing the metadata on individual documents via the information properties pane on the right.

Adding or changing the metadata via ‘Edit in Grid View’

(Note that there is no option to add or change additional metadata on individual items via the Teams ‘Files’ tab interface.)

But, adding or changing metadata by navigating through the folder structure would be onerous and a waste of time when there is a much better way of doing this – by making the folders disappear.

How to make folders disappear

Every SharePoint document library (including those in sites connected to Teams) has the ability to hide the folders in a document library. This – and the ability to add metadata and more – is achieved via the library view options.

The screenshot below shows the location of default ‘All documents’ view of a library, accessed from the top right of the document library in either SharePoint or via the Files tab in Teams.

Generally the ‘All Documents’ view should be left ‘as is’ and a new view created.

To do this, select ‘Create new view’ or ‘Save view as’ to create the view required. Give the new view a name that reflects what it is, for example ‘NoFolders-Minutes’ (and don’t use a long name or the text may not be visible).

Every view has a range of options, including the following.

Choose columns to be displayed

Select which (system or added) metadata columns are to be displayed in the view.

Choose the columns to appear in the view, and where they appear.

Sort columns

Select how the content will be sorted. For example, sort by ‘Created’ date.

Sort by options

Filter columns

Select how the content will be filtered by the metadata columns. For example, only show content that meets a specific criteria such as Document Type = ‘Minutes’.

Filtering options. In this case, the view would only show content tagged with the metadata value ‘Minutes’

Group columns

Select how the content will be grouped by the metadata columns. This groups the content by a particular site column. For example, Group by ‘Created by’ or other options. Grouping (and sub-grouping, two levels are allowed) content in this way is often a much better approach than using folders.

The screenshot below shows an example of two-level grouping of metadata.

Example of two-level grouping

Folders

A bit further down is the ‘magic’ button to make folders vanish – ‘show all items without folders’. This options also hides document sets.

Making the folders disappear.

What this looks like

In the document library that was originally folder-based, we have created a new view without any folders. This now allows us to see all the documents in the library without folders. We can see from the view below that some of the items have been mistagged (Minutes with ‘Agenda’ for example). These can now be easily fixed via the ‘Edit in Grid View’ option.

(Note – if the library has more than 5,000 items without folders, the view won’t work and additional filtering may be required; when this happens, press the back space and fix the view, otherwise you may be locked out).

The view can be further modified. In the example below, the view has now been grouped by ‘Created by’ and also filtered to only anything created after 1 August 2021.

Views created in SharePoint are also visible via the Teams Files tab, and can be created in that tab. The screenshot below is of a Teams-based SharePoint library that had multiple folders under the ‘General’ folder and one document (‘Cover Letter’) under General. We can now see that there were in fact no documents in the other folders.

Benefits of views without folders

The primary benefit of creating views without folders is that (subject to the 5,000 item view limit) you can easily see what content (if anything) is hidden in the folder structure of a document library. The metadata that is displayed can be easily exported to a csv or Excel file.

It also makes it easier to apply and/or change metadata; this metadata can in turn be used to sort, filter and group the content in a library.

Note that, if there are more than 5,000 items in the library, additional views can be created to reduce the volume that is displayed at any one time.

As every view has a URL, they can also be added as links in the left hand or page navigation or emailed, making them a kind of pre-canned search.

In the example below, there is only one library ‘Policy Library’; both the ‘Aged Care Policies’ and ‘Due for Review’ are views based on metadata assigned to documents. Additional views (not visible) can group policies based on business areas or relevancy, or various other options. This means that policies (in this case) are not restricted to a single ‘folder’, but can be viewed in multiple ways depending on their context.

Multiple views showing as links on the left hand navigation pane

Are there any drawbacks doing this?

Aside from the 5,000 view item limit, the other things to be aware of are (a) what happens when the library is synced, and (b) potential confusion.

Libraries (including Teams-based libraries, via the ‘Files’ tab) synced to File Explorer do not currently display any additional metadata added to the SharePoint library, although this may be coming soon. Even if this metadata is displayed, the views are not available. So hiding folders and grouping/filtering may only be useful via the Teams client or browser.

In terms of the potential for confusion, folders on network file shares are very good at hiding duplicates or versions of the same document. When folders are removed from SharePoint, the same document may appear multiple times without any details of the original folder structure or path, resulting in confusion or uncertainty. What if someone always accessed the document via a particular path, then discovers it is missing? How do they know it is hiding in another folder path? If you are going to turn off folders, make sure that end-users about this and how to continue to access their documents via the (preferably unchanged) ‘All documents’ view.

Can we get rid of folders, yet?

Folders have been around for a long time and are a very hard habit to break. They are unlikely to go away any time soon.

However, metadata-based views provide a useful way to ‘see through’ folders and can also be used to apply additional metadata that, in turn, can be used – in conjunction with folders – to group, sort and filter the content hidden away in the folders. Perhaps, in some cases, metadata-based grouping could replace folders, but this will take time and may not be always useful – for example in synced document libraries.

Either way, a combination of folders and metadata can be useful to support recordkeeping activities.

Different approaches for managing records with Microsoft 365

The COVID pandemic from early 2020 led to the requirement for many employees to work from home (WFH). IT Departments scrambled to enable this capability, many making use of Microsoft (MS) Teams that was already bundled with their Microsoft 365 licences.

The rapid enabling and uptake (rather than an actual ‘implementation’) of MS Teams was more often than not achieved without much consideration for recordkeeping requirements or an overall plan for using Microsoft 365.

MS Teams became popular quickly, increasing from around 30 million active users daily in early 2020 to around 250 million by mid 2021 (Source: ZDNet quoting Microsoft latest results). End-users could chat with each other and with external people (and on their phones too!), have video meetings, create new teams with channels and private channels, share and collaborate on content via the ‘Files’ tab in Teams, create and manage tasks, and more. They also continued to use email.

Anecdotal evidence suggests that the capture of records to on-premise electronic document and records management system (EDRMS) declined from early 2020. One reason suggested for this was that it was too hard to save some cloud records such as Teams chats or content from the Files tab to an on-premise system. Alternative approaches for managing records with Microsoft 365 began to evolve.

This post discusses four approaches to managing records in Microsoft 365, summarised in the diagram below.

Which approach have you taken? Answer my (anonymous) short survey here (Microsoft Forms).

Approach 1 – EDRMS + key Microsoft 365 applications to create and capture

Approach 1 – EDRMS plus the main Microsoft 365 applications

This model has two elements:

• Retaining an existing centralised recordkeeping system (the EDRMS) for the storage of records.
• Using email, Teams, SharePoint or OneDrive to create or capture records to be copied to the EDRMS, and leaving other content (in theory non-records) ‘in place’.

The main positive aspect of this model is that records are (in theory) captured and managed in the EDRMS with all the traditional recordkeeping options. Some leading EDRMS vendors now offer solutions that integrate with Microsoft 365 and make it easier to capture records from Microsoft 365. But the model is still based on a centralised recordkeeping system and the requirement for end-users to copy content identified as records.

The main negative aspects of this model include the following points:

• End-users still have to identify and copy records to the EDRMS.
• Not all records created or captured in Microsoft 365 can be copied to the EDRMS.
• Additional products or add-ons may be required to enable the copying.
• The record is copied to the EDRMS, not moved, so remains in place with no controls.
• Records that remain stored in Microsoft 365 applications may not be subject to the same degree of recordkeeping controls available in the EDRMS. Unless they acquire a third-party product (see next approach) to overcome this problem (which is unlikely for cost reasons), organisations must use the out of the box recordkeeping capability in Microsoft 365. This capability may not meet all requirements for keeping records if not properly configured.
• There is a real risk that some records that remain in Microsoft 365 may be lost, especially if settings allow content to be deleted and there is no retention policy or backup.
• EDRM system admins and records managers will need to learn a lot more about Microsoft 365.
• The unified logs in Microsoft 365 only retain the details for 3 months (E3) or 12 months (E5) – although SharePoint’s versioning history can provide a lot of ‘modified’ event metadata for the life of the document (up to the the maximum number of versions allowed). (Update: Microsoft 365 customers can retain the audit log for up to 10 years with an add-on license. Many export audit data to a SEIM such as Azure Sentinel where they can retain the log for as long as they want.)

On a positive note, however, Microsoft 365 includes a wide range of search, audit, monitoring and reporting tools, as well as security and protection controls, that improve the ability for records managers to find, manage and protect records (or potential records) in Exchange mailboxes, MS Teams chats and posts, SharePoint sites and OneDrive accounts AND put that content on a legal hold. So, as long as those options are enabled, the risk of losing records is reduced.

Approach 2 – Third-party application + Microsoft 365 applications for creation, capture and storage

Approach 2 – Third-party product plus Microsoft 365

A number of Microsoft partners have developed applications to manage records in Microsoft 365. Several have been available for a decade or more, originally designed to manage records primarily in on-premise SharePoint environments.

Most of these third-party applications were developed to comply with the same recordkeeping standards used by EDRMS vendors. These applications are generally either:

• Replacements for EDRM systems (often requiring migration from the EDRMS).
• New implementations where there was no EDRMS beforehand.

It is not common to see both an EDRMS and one of these third-party products being used together, because of licensing cost reasons.

The main positive aspect of using a third-party dedicated application is that records created or captured in Microsoft 365 can be stay there and be managed according to recordkeeping requirements. Some of these applications are invisible to end-users, making them even more attractive.

The main potential negative aspect of using a third-party application, which is the same for any other vendor product, is that it creates a dependency on the vendor to maintain the product. Microsoft 365 continues to evolve and any third-party application must keep up with these changes. Two questions might be asked:

• Will this dependency become a ‘tech debt’ liability in the future, if a ‘better’ option comes along?
• How hard will it be to transfer to a different vendor in the future? Generally speaking this is less likely if the vendor is an established Microsoft partner, but the question should still be asked. For example, many organisations decided to use the Google suite of products but have now decided to use Microsoft 365.

Organisations seeking to implement third-party applications to manage records in Microsoft 365 should have a very detailed understanding of the underlying Microsoft 365 environment beforehand and the impact the third-party application might have on this environment. Some of the considerations might include:

• The requirement to provide the third-party vendor with admin (including global admin) access to the Microsoft 365 tenant. Is this a security concern?
• The location of records – in some cases, third-party vendors may use, move or back up content to one of their Microsoft 365 tenants. Is this a security concern? How can you monitor activity on your content if it’s not in your tenant?
• The use of the central Term Store or Content Types to support the application. Will this create a dependency or make it harder for people to work, for example by requiring end-users to select Content Types or add metadata.
• Changes to SharePoint settings and architecture, including the addition of hidden columns. Will these changes be consistent with your own architecture model?
• How and where event metadata (audit logs) will be captured and managed.
• How retention outcomes will be managed.

Approach 3 – One or more Microsoft 365 applications are the default ‘recordkeeping systems’ (no EDRMS or other application)

Approach 3 – Individual systems highlighted are the ‘recordkeeping’ systems

This approach focuses on the applications where most records are likely to be created or captured in Microsoft 365 – Exchange mailboxes, MS Teams, SharePoint, and OneDrive for Business – and therefore considers other content created and/or stored in other Microsoft 365 applications (e.g., Yammer, Forms, Planner/Tasks, etc) as being non-records.

There are several variations on this model including the following:

• Outlook and Teams are the primary ‘recordkeeping systems’ as they are the two applications that are most used. Teams has been positioned as the primary interface for both SharePoint and OneDrive (via the ‘Files’ tab). The ability to also access both SharePoint and OneDrive from File Explorer via the sync option makes it even less likely that SharePoint or OneDrive will be accessed by end-users.
• All four applications are the recordkeeping systems, using the various controls and settings available in the various admin portals, as well as the Compliance admin portal for retention policies.
• SharePoint is the primary recordkeeping system, configured to mimic EDRMS capability. In this case, end-users would be expected to copy emails from Outlook or records from OneDrive, similar to the way they would have to do this for an EDRMS. Various controls and settings, such as ‘back end’ retention policies, might be applied to the other main applications to ensure that any records in those systems (such as Teams chats or emails) are not destroyed before a given period.

The main positive aspects of this approach are (a) simplicity and (b) cost savings, mostly by not having to purchase an EDRMS or third-party application.

However, these potential positives should not compromise the requirement for both IT and records management to have a very good understanding of, detailed approach to, and governance for, managing records in Microsoft 365. In other words, simply saying that one or more of these four applications is the recordkeeping system is not sufficient; additional work is required to ensure that records stored in them are managed appropriately.

There are several potential negative aspects of this model:

• With the exception of SharePoint, none of the other three systems can be configured to manage records based on standards used for EDRM systems. Given that SharePoint has been positioned behind the Teams user interface, and SharePoint document libraries can be synced via Teams to File Explorer, any recordkeeping functionality configured in SharePoint should in theory be accessible or useable via Teams and possibly also File Explorer, but this is mostly not the case. So, SharePoint on its own, accessed via the browser only, is not really an option. Additionally, without effective controls, the Files (SharePoint) element of Teams has the potential to become the future equivalent of legacy network file shares full of redundant, outdated and trivial content.
• If only one or two systems are considered to be the only recordkeeping systems, there is a risk that records may not be saved and/or could be lost, especially if end-users can delete records and there is no back up option.
• Managing records in this way requires both access to and a very good understanding of the applications designated to be the recordkeeping systems by both IT and records managers.
• Retention policies (either the base level information governance or more expensive records management) may not be adequate, in terms of both application and coverage, and retention outcome management.
• Exporting the records to another system or transferring them to another organisation, could become a complex task.
• Accessing audit logs over a long period (see first approach, last dot point, above).

Approach 4 – All of Microsoft 365 is the recordkeeping system

Approach 4 – All of Microsoft 365 is the recordkeeping system

This approach is similar to the previous one except that it takes a broader approach and requires a degree of ‘letting go’ of the standards used by EDRMS systems (and third-party products). It is also the Microsoft default.

The approach assumes that records may be created or captured anywhere in Microsoft 365, saved to Microsoft 365 via archive connectors, or accessed (subject to access controls) via search connectors. Records are managed ‘in place’, meaning wherever they are created or captured, using a range of tools already available in Microsoft 365. Additional ‘in place’ controls allow certain items to be declared as records.

The approach requires both a very good technical understanding of the Microsoft 365 environment and effective governance by IT and records managers. If internal skills are lacking, it may also require a third-party organisation to implement the system – but based on what recordkeeping model? A reliance on a third-party to implement the recordkeeping elements has several risks (see below).

The main positives of this approach include the following:

• Records that are created or captured in the Microsoft 365 environment remain there. There is no requirement to copy them to a separate system.
• Some records, such as emails, can be copied to SharePoint if required.
• The combination of Teams and SharePoint sites allows for multiple models to manage records – for example, high value records could be managed in a dedicated SharePoint site with multiple dedicated libraries and additional controls (metadata, retention, permissions etc), whereas low level records could be managed in the single ‘Documents’ library presented as the Files tab in a Team, or via File Explorer.
• All the content (records and non-records) stored across Exchange, Teams, SharePoint/One drive can be searched (subject to roles and permissions). This allows records managers (and others such as Legal) to identify if records may be hidden in personal mailboxes or Teams chat or OneDrive accounts.
• Minimum retention periods can be applied to all the content (not just records), ensuring that records that may be hidden in Teams chats, OneDrive accounts, or personal mailboxes, will be retained for minimum periods. This option also helps to reduce the volume of redundant, outdated and trivial content that may build up over time otherwise.
• Retention labels can be applied, including automatically (and using machine learning), to records in mailboxes, SharePoint sites and OneDrive accounts (but not Teams chats or posts, yet).

The main negatives of this model are the same as those listed for the previous model with more focus on the need for both IT and records managers to have a very detailed understanding of and establish effective governance for the entire environment where records may be created or captured, not just the main four applications. This requires some effort to achieve and should not be understated. It is not uncommon to see IT staff with Global Admin managing the entire Microsoft 365 environment using default settings and/or records managers will little technical knowledge or appropriate access struggling to understand how the environment works and drawing on experience with EDRM systems.

Some organisations may engage third-party implementation specialists to configure and set up the environment. Organisations that decide to go down this path should ensure they have the details of this configuration and can support it in the longer run, or the environment (or parts of it) could end up becoming difficult to manage or support over time.

Approach 5 – A potential future model

Microsoft 365 includes a wide range of settings, options and capabilities that have a significant impact on the way records can and will be managed across Microsoft 365 in the future.

Microsoft 365 will continue to evolve over time, including in ways that will support how records are managed. But it is important to keep in mind that Microsoft 365, or its component applications, is not and will never be an EDRMS based on standards such as DOD 5015.2. Microsoft 365 is too complex, and the volume and type of content stored in it too large, for any part of it to be considered the ‘records management’ system.

A new approach is required for the identification and management of records. This approach may draw on existing recordkeeping standards and concepts but is likely to rely more heavily on new and evolving ways to work with information, including records.

Some of these ways have been around for a decade or so in the form of graph-based machine learning (ML), process automation, artificial intelligence (AI). Examples include Google, Facebook, LinkedIn, Netflix, Amazon, eBay and so on. These examples have one thing in common – they all take advantage of the various ‘signals’ and ‘digital exhaust’ voluntarily offered by their users to identify and present things that match your interests – jobs, friends, things to purchase, movies. Post something on Facebook or (perhaps) talk about a particular subject near your phone, and related ads will appear.

So, what is different about Microsoft 365? End-users are related to each other thanks to Active Directory, they connect and communicate with others via email or Teams, they share content, they attend meetings. All of these (and a lot more) signals feed into the underlying Graph and allow connections to be made and suggestions.

There is nothing stopping organisations setting up dedicated SharePoint sites with multiple well-named libraries to manage certain records and leaving other content and records to the world of Teams Files. But all of this information can be related based on context, including who created it, what team that person was in, who they connect with, what access do they have and so on.

Perhaps by 2035, the primary approach to records management will be relying on all the digital connections and signals, machine learning, the Graph and AI to identify all related records in context, not just the ones neatly placed in a SharePoint document library. Records may be automatically identified as important and needing stronger controls based on this context – who created, sent or received it, whether it relates to a subject that is trending (or was in the past).

Instead of just a simple pre-defined aggregation of records (which will still be a valid way to aggregate records), future aggregations will include a wider range of content, created automatically, likely presented in the form of ‘cards’.

Viva Topics is an interesting pre-cursor to this possible future model.

Viva Topics presented in Teams

The following text is from the Microsoft page ‘Alexandria in Microsoft Viva Topics: from big data to big knowledge‘:

Looking further ahead, Alexandria’s ability to extract information automatically gives us the opportunity to customize the knowledge discovery process. By automatically retrieving the set of types and properties being talked about in an organization’s documents, Alexandria can create a knowledge base with a bespoke schema exactly tailored to the needs of each organization and using the familiar language and terminology that people in the organization are used to. Read more about the proposed schema-based design in our research paper.

We are only beginning to dream of the experiences that an automatically created and updated knowledge base can enable, but it is already clear that it could transform the future of how we work. The era of big knowledge is coming sooner than you might think.

Whatever the new approach is, managing records in Microsoft 365 will require new skills on the part of information and records managers.

Tips to reduce complexity when implementing SharePoint

Over a period of almost 10 years of providing support for SharePoint (2010, 2013, Online), one of my key learnings has been to avoid complexity in its implementation. The more complicated the implementation is, the harder it will be to support, fix and maintain.

This post contains several tips for implementing SharePoint to minimise complexity and therefore the need for support.

Before the tips, it is important to understand the original SharePoint architecture model as this model (a) likely continues to exist in many organisations and (b) can influence decisions – not necessarily in a positive way – about how to implement SharePoint Online.

Understand the (historic) dual nature of SharePoint

SharePoint was first released in 2001. There continues to be a lot of legacy SharePoint implementations and approaches to SharePoint Online that reflect these older implementation models.

SharePoint has always had two sides – the publishing functionality (usually in the form of an intranet) and the document management functionality (where documents are stored). Sometimes these two are conflated into the single term – ‘intranet’ when in fact they are two quite different things in terms of architecture and usage.

Older (pre Online) on-premise versions of SharePoint typically consisted of multiple web applications in several app pools, as the diagram below shows.

(Source: Microsoft ‘SPS 2010 Design Sample Corporate Portal with Classic Authentication architecture’ model)

Within each of these web applications, organisations would have one or more site collections, often with sub-sites. Almost none of this architecture model remains in SharePoint Online – there are now only sites, which are still technically site collections, but preferably without any subsites – see below. This is sometimes described as a ‘flat’ implementation as every site (formerly a site collection) is on the same ‘level’.

The publishing functionality, enabled through the publishing features, was typically used for intranets. Most information was presented on pages, typically across multiple subsites, with some use of the document libraries (e.g., to store common forms or policies).

But the publishing User Interface (UI) was poor; most organisations engaged a third-party to create more engaging (and usually heavily customised) intranets. The outcome has often been that organisations have had to maintain old legacy SharePoint servers to keep the ‘old’ intranet running. In most cases, these older SharePoint sites cannot be migrated ‘as is’ to SharePoint Online.

The former publishing functionality was replaced in SharePoint Online with communication sites.

The document management functionality was (and continues to remain) the default ‘out of the box’ (OOTB) option for all new team sites. Most older SharePoint sites had to be accessed via a browser and the older UI was often seen as not very interesting or engaging. Business areas would sometimes customise the site pages, sometimes to the point of making them appear to be ‘local’ (business area) intranets.

Although SharePoint Online has the same dual nature, it does not use web applications. Instead, all site templates (communication or team) must be created under one of two paths: /sites/ and /teams/. Organisations may default to just one of these two paths for all sites, or could decide to make use of them to differentiate between, e.g., enterprise sites (cross organisational, e.g., the intranet) and team sites (the name of the business area/team comes after /teams/).

Tip 1 – Try to avoid or minimise customisation

As noted above, customisation was common in on-premise versions of SharePoint. Microsoft have done a lot to improve the UI for both communication and team sites.

Microsoft have released a range of pre-canned templates for communication sites that can be downloaded from the ‘lookbook‘ site. (These templates will soon be available as a choice when new sites are created as well.) These templates and new web parts allow organisations to set up a nice-looking and engaging intranet, with no customisation, (other than via the web parts on each page), that is easy to support and manage.

Organisations that require more complex intranets can of course continue to customise their communication sites and/or use a wide range of third-party tools. These are likely to require a higher level of support to maintain and troubleshoot issues, including from third-party vendors.

The customisation of SharePoint team sites used primarily to store and manage document-type content (e.g., to replace network file share storage) is now a different matter. The rapid adoption of MS Teams as the primary UI to access this content over the past 18 months and the ability to sync document libraries to File Explorer and access the content on a mobile device, has in most cases made browser-based access redundant.

There is no longer much point in customising the browser (pages) view of SharePoint team sites used to store documents. Customisation in these sites is more likely to take the form of the site structure, content types and columns, permissions, or the types of information stored in them. Any customisation of team site pages should be restricted to the out of the box options.

Tip 2 – Don’t overcomplicate the structure

Older versions of SharePoint could have quite complicated structures – from the web application through to the site collection, sub-sites, libraries, content types and columns.

These structures would inevitably cause issues over time as organisations re-structured or were renamed, or wanted content moved to different parts of a site or a different site. The easiest sites to manage were always the simple stand-alone sites which is, fortunately, the preferred model in SharePoint Online.

To avoid complexity with all SharePoint Online sites, don’t create sub-sites unless absolutely necessary. Microsoft don’t recommend sub-sites any more – see ‘Planning Your SharePoint Hub sites‘. To quote from that page:

Subsites don’t give any room for flexibility and change. Since subsites are a physical construct reflected in the URL for content, if you reorganize your business relationships, you break all the intranet relationships in your content. Subsites can also create challenges when it comes to governance because many features (including policy features like retention and classification) in SharePoint apply to all sites within the site collection, whether you want them to or not. This means that you must frequently enable a feature for the entire site collection, even if it’s only applicable to one subsite.

Instead of using sub-sites in communication sites, create multiple pages linked via the out of the box mega menu or links.

Avoid using sub-sites at all for team sites unless there is a genuine requirement to restrict access to part of the site. The same outcome can usually be achieved by changing access controls on a document library.

Tip 3 – Use Content Types and columns selectively

SharePoint is built on content types, including the common ‘item’, ‘document’ and ‘folder’ types. Older SharePoint implementation models sometimes made extensive use of Content Types and site (or library) columns to define content added to document libraries. This meant that end-users would have to select a Content Type and/or add or choose metadata when content was saved, a process that was seen as off-putting for most.

To minimise complexity, deploy custom Content Types and additional site or library columns only where really necessary, not as a default approach. Don’t use Content Types to differentiate between (ironically) content or document types but consider using site or library choice columns with default values instead. For example, if a document library is used to store records about meetings or committees, a drop down choice value for ‘document type’ is more likely to be used (and useful) than Content Types.

Tip 4 – Avoid complex permissions

Possibly up to 70% of all support queries over a decade related to permissions or access restrictions relating to a team site and usually started with ‘Why can’t I access …?’ or ‘Why can (so and so) see (or not see) this content?’ They rarely related to permissions on a publishing or communication site, which tend to have simple permission structures.

Almost all of the queries about team site permissions could be tracked back to someone changing the default inherited permissions. In one case, on a site with very complex permissions, a site owner deleted the site owners permission group, locking them out not only of the site but on every single part of the site that had unique permissions. It took a couple of weeks to restore these permissions.

Try to avoid changing the way permission inheritance works. The permissions set at the site level (example shown below) flow down to all the content. It is all too easy to lose sight of (and then find) what permissions have been changed.

If there is a genuine requirement to restrict access, consider moving that specific content to a separate library or even a separate site, rather than restricting access on items within a library, especially one that has a complex folder-based structure.

Also, keep in mind that over the past decade, the access model for SharePoint content has changed from (a) restrict access everywhere and only grant access to those who can access it, to (b) open access everywhere and only restrict access to what needs to be restricted. This is a much easier model to support.

The easiest way to give ‘everyone’ internally access to all SharePoint content is to add ‘Everyone except external users’ to the Visitors group of every SharePoint site, including sites linked with MS Teams. This access does NOT give them access to the Team channel posts in MS Teams.

Whether or not external users should be allowed access via external sharing will depend on each organisation, but it is a good way to reduce the volume of duplication that happens when content is attached to emails. It can also be monitored.

Avoiding access control complexity should also help to improve information security, by ensuring that controls are more easily understood.

Tip 5 – Avoid storing too much and/or unrelated content together

SharePoint sites (including those linked with Teams) are a form of logical aggregation or container. And you can store a LOT of content on those sites. According to the Microsoft site ‘SharePoint Online limits‘:

A library can have up to 30 million files and folders. When a list, library, or folder contains more than 100,000 items, you can’t break permissions inheritance on the list, library, or folder.

But, as the saying goes, just because you can doesn’t mean you should.

It is all too easy to see the ‘Documents’ library of every SharePoint site (especially Teams-linked sites) as the equivalent of a top level folder in a network file share with multiple folder hierarchies. This might seen quite appealing and simple to use at first, but after several years, the ‘Documents’ library can end up a lot of content, not all of which is related to the site’s original intended purpose.

After 10 years, the content is likely to resemble an uncontrolled network file share, with redundant, trivial and outdated content all over the place. With potentially up to 30 million items, this will make managing the content so much harder.

As much as possible, try to ensure that SharePoint sites are based on logical business functions and the libraries in those sites mapped to business activities.

This is not always easy; for example, many smaller organisations have a ‘corporate services’ type function that includes a range of business activities, including managing property, general admin, legal, HR/personnel, fleet and corporate meetings etc. While all of this content could be stored in a single SharePoint site, there is likely to be a requirement to restrict access to content (see previous point) which will only result in increasingly complex permission controls and support issues over time. Mixing unrelated content could also become a nightmare for compliance and records management.

A better, more sustainable, option would be to create multiple SharePoint sites (or Teams) with multiple libraries, linked via a hub site as necessary. This approach will also make it easier to manage records in the longer-term.

Feature image – original source unknown

In place vs in place management of records in Microsoft 365

The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.

This post explains the difference between the two ‘in place’ options. In brief:

• The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
• The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.

The Microsoft in-place model

SharePoint

The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.

Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:

• The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
• Mandatory record metadata.
• Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
• Unique computer-generated record identifiers for each record, regardless of where that record is stored.
• The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
• The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
• Not permitting modification of certain metadata fields.
• The capability to support multiple renditions of a record.
• The capability to increment versions of records when filing.
  Linking the record metadata to the record so that it can be accessed for display, export.
• Enforcement of data integrity, referential integrity, and relational integrity.

Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):

Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.

The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’

The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’

Exchange

A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.

One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.

Archive mailboxes, including ‘auto-expanding archives’, also exist in Exchange Online. In the Exchange Online archiving service description, it is noted that:

Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.

The new ‘in place’ model

A newer form of in-place records management has appeared with Microsoft 365.

Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.

The newer model of in place records management is based on the assumptions that:

• It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
• Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
• The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.

Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.

The relationship between Microsoft’s ‘in place’ focussed records management, and managing everything (including records) in place.

In place co-existence

Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.

Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021

A decision tree for managing records in SharePoint and Teams

One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.

Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.

Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:

• Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
• The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.

Decision tree

With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.

The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:

The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.

Note: the box outline colours represent the four different types of site that are likely to be created.

How will the content be accessed?

It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.

Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.

They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.

If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.

It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).

What type of SharePoint site will be required?

Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.

As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.

Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.

Creation method

The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.

Security Controls

Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.

Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.

Retention

Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.

As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.

Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.

Provisioning

Once the site has been created, it should be provisioned with three key elements shown below.

• Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
• Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
• The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.

The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.

Modern vs classic sites in SharePoint Online?

SharePoint has a long, twenty-year legacy. Over that period, organisations and third-party developers have developed and implemented a range of solutions and options across SharePoint. Many of these have been in place for a long time.

The arrival of the new look, ‘modern’ SharePoint Online (SPO) in the last five years has created a dilemma for organisations – migrate to SPO and lose the older functionality, or retain some of the functionality by creating ‘classic’ sites in SPO?

This post was inspired from discussions with a number of organisations that were either creating or using classic site templates in SPO – sometimes as the default option – or indicated a resistance to moving to SPO because of the ‘loss of functionality’.

What are modern sites in SPO?

The default options to create in SharePoint Online are either (a) modern team sites linked with a Microsoft 365 (M365) Group (most common) or (b) communication sites.

The ‘Create a site’ options in the SharePoint admin portal. These are just two of several ways to create new sites.

Modern team sites linked with M365 Groups are created:

• From the SP Admin portal (first option ‘Team site’ – above)
• By end users from their SharePoint portal (if that feature is not disabled)
• When a new Team is created via MS Teams or the MS Teams admin portal
• When a new Group is created from Outlook or from the Microsoft 365 admin portal
• When a new ‘shared folder’ is created from OneDrive

Modern team and communication sites are almost identical in terms of core functionality. The primary visible difference is that: (a) team sites have the navigation bar at the left and are usually used to store documents and other digital objects; whereas (b) communication sites have the navigation bar along the top (which can be made into a mega menu easily) and are usually used to store information on pages. They may also store documents in document libraries.

The default page web parts are the same on both; for more information about web parts, see this Microsoft guidance ‘Using web parts on SharePoint pages‘.

For more information about the modern experience see this Microsoft page ‘Guide to the modern experience in SharePoint‘.

The PowerShell script to create a new Group-based site uses the command ‘New-SPOSiteGroup’, which creates both the Group, the SharePoint site, and a Group-based mailbox in Exchange Online.

• New-SPOSiteGroup -Site https://example.sharepoint.com/sites/GroupSiteExample -Group “GroupSiteExample Owners” -PermissionLevels “Full Control”

Note that M365 Group-based sites use the template ‘GROUP#0’.

What are classic sites in SPO?

Other, ‘classic’ SharePoint site templates can be created from the ‘Other Options’ section of the SP Admin portal.

‘Other options’ section of the Create Site section of the SP admin portal

The options, as grouped, are as follows. Note that for all of them, the SharePoint 2013 experience (look and feel) is used.

• Collaboration: Team site (classic experience), Developer Site, Project Site, Community Site
• Enterprise: Document Center, eDiscovery Center, Records Center, Team Site – SharePoint Online configuration, Business Intelligence Center, Compliance Policy Center, Enterprise Search Center, My Site Host, Community Portal, Basic Search Center, Visio Process Repository
• Publishing: Publishing Portal, Enterprise Wiki
• Custom: (if a custom template exists)

SharePoint sites, including using classic templates, can also be created using PowerShell and the ‘New-SPOSite’ command (with additional configuration details, see this Microsoft docs page). The PowerShell command will needs to include the template to be used:

SharePoint site templates

The PowerShell script to create a new classic site is slightly different from a modern Group-based siste, using ‘New-SPOSite’:

• New-SPOSite -Url https://example.sharepoint.com/sites/ClassicTeamSite -Owner peter.smith -StorageQuota 1000 -CompatibilityLevel 15 -LocaleID 1033 -ResourceQuota 300 -Template “STS#0” -TimeZoneId 13 -Title “A classic team site”

Why create SP sites using the classic templates?

Organisations may have genuine technical reasons to want (or need) to create and continue to create some SharePoint Online sites using one of the classic templates. For example:

• They may have many sites that were created in the legacy on-premise environment. Upgrading these sites will take time, so it may be easier simply to migrate from one to another ‘as is’.
• A site or sites may have integration points with other systems that require the older functionality.

Classic sites should not normally be created simply because of loss of (or changed) legacy functionality (e.g., a preference for older style web parts), or simply a preference to continue to use older functionality.

As much as possible, organisations should aim to create modern sites. In some – or many – cases, this may require the organisation to adopt the new modern options.

Classic to modern functionality changes

Examples of broader-level functionality that has largely been replaced with new functionality, or may be deprecated in the future, include:

• Publishing sites were often used for customised intranets, often with multiple levels of sub-sites. These have been replaced by communication sites and hub sites.
• Publishing features, including features sometimes used for navigation purposes and linked with terms in the Term Store were a common ‘classic’ option. These have been replaced by new mega menu functionality.
• Older web parts on pages have been replaced by modern web parts.
• The Document Center and Records Center templates are more or less now superseded by in-place records management functionality, leaving records where they were created or captured.
• The Business Intelligence Center has mostly been replaced by PowerBI or similar tools and interfaces.
• Community Sites and Community Portals have been replaced by communication sites, Teams and Yammer.
• The need for sub-sites has been replaced by creating one or more sites as hub sites and linking other sites to the hub.

Many of these classic options have been around for at least a decade, making it sometimes hard to move to the newer options.

Functionality differences between classic and modern sites

Possibly the most common of the classic sites that continue to be created, however, are classic team sites (‘STS#0’). In some cases, these site templates are the default option. It is not clear why these continue to be created when the STS03 (non M365 Group) template has the same basic functionality:

• The same Site Settings.

• The same ways to add a page, library or list (from Site Contents or from the gear icon – Add an app).
• The same library and list settings (including the ‘Information management policy settings’).

One key difference is the look and feel, something that may become less of an issue with Teams becoming the primary UI to SharePoint.

It is still possible on classic (STS#0) sites to create classic style pages using classic web parts, as can be seen in the screenshot below from a classic (STS#0) site:

What’s the problem with creating new classic sites?

As noted above, however, just because it remains possible to create and use classic functionality isn’t a reason to keep it in face of newer and often better options – especially when end-users won’t even see the site.

From my own experience, anything that remains in an older version of any technology becomes (or is likely to become) technology debt.

It may be appropriate – temporarily – to migrate some older on-premise sites to the classic template until they can be upgraded, but there is no valid reason in my opinion to create new classic sites as a default option for all new sites.

What do you think? Do you continue to create classic sites?

Featured image: A classic VW Combi.

Managing digital photos as records in SharePoint Online

Photographs (or, for that matter, any visual work) can be more powerful as a record of something than a written record. Some of the iconic events in history are only known from the visual record.

Photos (and videos) are ubiquitous in the digital age. They are captured in multiple ways and stored on a range of media including computers, portable drives, mobile devices and the cloud (including online storage and social media).

Digital photos can be easily changed or ‘photoshopped’, to use the more common term.

But records management requires us to ensure the authenticity, reliability and integrity of records. How can we ensure this for photographic records?

This post examines how SharePoint can be used to manage (some) photos as records.

What is a digital photo?

For the purpose of this post, a digital photo is any image that is captured and stored electronically as bits in a recognised image format. It excludes digital photos (including scanned or digitised paper records) that are embedded in PDF files.

To quote from the Wikipedia article titled ‘Digital Image‘, a digital photo is:

‘… an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively. Depending on whether the image resolution is fixed, it may be of vector or raster type. By itself, the term ‘digital image’ usually refers to raster images or bitmapped images (as opposed to vector images).’

According to the definition, a digital photo is a collection of bits set out in a map that a computer interprets to display it on a screen. For more detail on bitmaps, see this Microsoft page ‘Bitmaps‘.

When to use SharePoint to store digital photos

Almost any digital object that can be stored electronically can be captured in SharePoint. But, just because they can doesn’t mean they should.

It is important to keep in mind that SharePoint may be not be the most suitable option for storing digital photographs in general. The volume, size and intended usage of the photos are all likely to influence the decision to use SharePoint. Organisations may need to consider other ways to store and manage digital photos (and other digital media) such as dedicated digital asset management systems.

However, SharePoint may be a good option to store digital photos as records if those photos:

• Need to be kept as a record of something.
• Support or relate to other content in the same document library. For example, photos that relate to or support building plans (which themselves may be scanned images in PDF form) or construction.
• Are about a specific subject. For example, photos of damage to a physical object.
• Are relatively low in volume and file size.

Factors to consider before storing digital photos in SharePoint

The following points should be considered before storing any digital photo (or digital media) in SharePoint.

File names

Digital photos are usually stored on devices that capture them with meaningless, device-generated names such as ‘20210423_123192321’, ‘DSC_0330’, or ‘n594015825_1706121_4959’. These types of names provide no clue to the content when the image is saved to SharePoint, as shown below.

Ideally, the device-generated name should be replaced with a more meaningful name as shown below.

We can see from the unique Document ID that it is the same record. The version history also tells us that something was changed but the file size hasn’t changed.

Keep in mind that every change to the file name or any other metadata added to the library will create a new copy (‘version’) of the image. In the version history above, there are now four versions each 3 MB in size.

The Activity section of the information panel to the right (which also provides a preview of the image) also provides some key information to support the version history information:

Does changing the name potentially compromise a key element of metadata? Who should change the name, and when?

Organisations should consider establishing naming conventions for photographs to help with findability and context.

Format

Almost any type of digital object can be stored in SharePoint. SharePoint also supports the ability to view almost every type of contemporary digital photo format (as described in this Microsoft page) so format should not be a problem when it comes to viewing the file.

However, if the organisation plans to store digital photos in older or less common formats, it will need to ensure that these can be accessed for as long as required. This will generally mean ensuring that the appropriate software application is available to anyone who needs to access (open/view) the photos.

Alternatively, consideration should be given to saving the photos in different, more common formats.

Size/Resolution

The size of a digital photograph may affect its usability as a record. The two photos below are of the same scene but the first photo is very low resolution (= small size, 63 KB), while the second is a section of a much large photo (= large size, 3.7 MB). The second version is clearly a more accurate record.

Digital photos used as records should provide sufficient detail to be useful as records. Accordingly, in most instances, the original photo, not a re-sized version, should be saved.

Reliability as a record

As noted above, photos can be easily modified, sometimes making it difficult to know if it accurately reflects the image it purports to capture. This is also an issue for any form of visual record, including drawings.

Metadata created when the photo was taken and stored in the photo’s properties (including EXIF metadata) can provide evidence of the reliability of the photo as a record, even when the record was stored in SharePoint. The following are the key metadata properties that are usually created and stored with a digital photo:

• Size.
• (Date) Created. The date and time when the photograph was created as a photo.
• Date taken. This is, for most digital photos, the same date and time as the ‘created’ date.
• (Date) Modified. This should be very close to the original date and time created on the original photograph, but may be several seconds different. If the photograph has been modified (including simple photo adjustments or ‘photoshopped’), it will show a different date and time which might give a clue to its reliability as a record.
• Image dimensions, width, height, horizontal and vertical resolution, bit depth
• Camera details including F-stop, exposure time, ISO speed, flash mode

Storing digital photos in SharePoint Online

Note that SharePoint previously had the option to create a dedicated ‘Picture Library’. This is no longer necessary as any document library can be used to store any digital object, and SharePoint Online document libraries now have additional options to view the photos.

Any digital photo can now be saved to a standard SharePoint document library, including via the ‘Files’ tab in MS Teams.

Metadata

Unlike some EDRM systems, SharePoint does not ‘capture’ or extract the details of digital objects when they are saved to a document library. Instead, these remain with the original digital object.

If digital photos are to be saved to a SharePoint Online document library, consideration should be given to using existing or additional metadata columns to describe the image, for example, the ‘Image’ column (small version preview that is stored separately in the Site Assets library >’Lists’ folder > GUID-named folder).

The following screenshot shows a preview image of the main photo.

Viewing options

SharePoint includes three options to view the content in a document library list view: List, Compact List, or Tiles.

The following is an example of a single digital photo stored in a document library (same as the one in the other options earlier) set to the ‘Tiles’ view:

Reliability as a record

Once saved to SharePoint, the photo’s reliability as a record can be determined from version history.

Additional protections may include access/permission controls, retention and/or information security labels.

Summary

SharePoint can be used to store (some) digital photos as records, but it should not generally be used as a general storage location for digital photos. Other dedicated digital asset management systems may be more suitable for that purpose.

Before storing digital photos in SharePoint, organisations should establish procedural rules or principles including (re-) naming conventions, format and file size requirements.

SharePoint does not extract and store the metadata from digital objects, which means that digital photos should retain metadata that shows when they were created or modified, and other details about the photograph which will provide evidence to support the authenticity of the photo as a record. Some consideration should be given to adding additional metadata to help describe digital photos as records.

A combination of version history, access controls, information protection and retention labels should provide sufficient controls to ensure the reliability, integrity and authenticity of records – or at the very least provide the evidence of changes that may be made.

There are two sides to the question of authenticity, reliability and integrity:

• Knowing if the photo that has been uploaded is a correct record of what it purports to be.
• Preventing the photo from modification or deletion or tracking any modifications that may occur.

It might seem impossible to know if a photograph is what is purports to be, but its metadata payload may provide the detail required.

Possibly photoshopped photo

Feature Image: Source not known.

SharePoint is not an EDRMS

In my February 2021 post A brief history of electronic document and records management systems and related standards, I quoted from a presentation by Philip Bantin in 2001 that summarised the difference between the two systems.

• An electronic document management system (EDMS) supported day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
• An electronic records management system (ERMS) was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted.

It could be argued that SharePoint began its life in 2001 as an EDMS and began to include ERMS functionality when SharePoint 2010 was released. So, how is it not an EDRMS?

In simple terms, unlike a traditional EDRMS model that was intended to be the repository for all business records, SharePoint is only one of several repositories that are used to store and manage business records. Instead of centralising the storage and management of records, systems such as Microsoft 365 provide centralised management of records wherever they are stored.

The EDRMS model

Almost all EDRM systems since the mid-1990s have had the following characteristics:

• Compliance with a standard. For example, DOD 5015.2 (1997), AS 4390 (1996)/ISO 15489 (2001), ISO 16175 (2010), VERS (1999), MoReq (2001)/MoReq2, etc.
• A relational database (for the various functions, metadata and controls (see diagram below) and a separate file share (for the actual content/records), accessed via a user interface.
• An expectation that most or all (digital) records would be copied from other systems used to create or capture them to the EDRMS for ongoing storage and protection. This outcome might be achieved through integration with other systems, for example to copy emails to the EDRMS.

The integrated EDM/ERM system model was described in the following diagram in 2008 by the International Standards Organisation committee, ISO/TC171/SC2, in its document ‘Document management applications’:

There are two key problems with the EDRMS model:

• They do not (and cannot) realistically capture, store or classify all business records. Emails have remained a problem in this regard for at least two decades. Additionally, there is now a much wider range of born-digital records that remain uncaptured.
• Many of the born-digital records that were copied to the EDRMS continue to exist where they were first created or captured.

Any organisation that has been subject to a legal subpoena or Freedom of Information (FOI) request for records will know the limited extent to which EDRM systems provide evidence of all business activities or decision making.

And yet, almost every organisation has a requirement to manage records. There is clearly a disconnect between the various requirements and standards to keep records and the ability to keep them.

But SharePoint (on its own) is not the answer.

Why isn’t SharePoint an EDRMS?

SharePoint was never promoted or marketed as an EDRM system. It is not even a relational database (see this 2019 post by Sergey Golubenko ‘Why SharePoint is no good as a database‘ for details).

To paraphrase from Tony Redmond’s recent (21 March 2021) post ‘SharePoint Hits 20: Memories and Reflections‘, SharePoint was originally designed to allow content (including page-based content in the form of intranets) to be accessed via the web, thereby replacing network file shares.

In practice, most organisations retained their network file shares, built their intranet using SharePoint, and otherwise used SharePoint to manage only some digital content.

It could be argued SharePoint started out as an EDM system and became more like an ERMS when more recordkeeping capability was introduced in SharePoint 2011. But that doesn’t make it an EDRMS in the traditional sense of being a central repository of business records.

The reality is that records are, have been, and will continue to be created or captured in many places:

• Email systems. In Microsoft 365, Exchange Online mailboxes are also used to store the ‘compliance copy’ of Teams chats messages.
• Network file shares.
• SharePoint, including OneDrive.
• Other online document management systems, including Google Drive.
• Text, chat and instant messages often created in third-party systems, often completely inaccessible or encrypted.

The type and format of a record can vary considerably. For example:

• An emoji.
• A calendar entry.
• A photograph or video recording (including CCTV recording).
• The recording of a meeting, in video or transcript form, or both.
• Virtual reality simulations.
• Social media posts.
• 3D and digital drawings (e.g., via digital whiteboards).

And, of course, all the data in line of business systems.

Instead of trying to save records into a single EDRM system, Microsoft 365 provides the ability to apply controls over the management of most records where they were created or captured – in email (including archived social media and other records), Teams chat, SharePoint, or OneDrive, and Yammer.

Is there a need for an EDRMS?

There is nothing stopping organisations acquiring and implementing traditional EDRM systems, or even setting up some SharePoint sites, to manage certain high value or permanent records, including some records that can be copied from other create/capture systems (such as email).

But there is also a need to address the management of all other records that remain in the systems where they were created or captured.

In most modern organisations, this requires broader controls such as applying minimum retention periods at the backend, monitoring usage and activity, and the proactive management of disposals. Not trying to copy them all to SharePoint.

Featured image: Lorenz Stoer, late 1500s.

Understanding permission groups in Teams and SharePoint

One of the most confusing aspects of Teams and SharePoint in Microsoft 365 is the relationship between permission groups used to control access to both of these resources. This is especially the case as every Team in MS Teams has an associated SharePoint site (the ‘Files’ tab).

This post explains how permission groups work between MS Teams, Microsoft 365 Groups and SharePoint.

SharePoint permission groups

Before discussing how Teams permissions relate to SharePoint, here is a brief reminder of how SharePoint permissions work.

SharePoint has always had three default permission groups, prefixed by the URL name of the site, as shown in the screenshot below (the name of the site always prefixes the words Owners, Members and Visitors).

Site Owners

• People (including in a Group, see below) added to the Owners permission group have full access (full control) to all parts of the site and are usually responsible for managing the SharePoint site. There would normally be two or three site owners.

Site Members

• People (including in a Group, see below) added to the Members permission group have add/edit (contribute) rights.

Site Visitors

• People added to the Visitors permission group have read-only (view) rights.

These permissions are set at the site level and inherited on everything in the site, unless that inheritance is broken and unique permission are applied. Additional permission groups can be created as necessary but most SharePoint sites only use the default Owners, Members and Visitors groups.

Microsoft 365 Groups

Microsoft 365 Groups were introduced in 2017 and control access to resources, like Security Groups.

However, unlike Security Groups, which usually provide access to individual resources (such as a single SharePoint site, or Line of Business (LOB) system), Microsoft 365 Groups control access to multiple linked Microsoft 365 resources.

Microsoft 365 groups, distribution lists, mail-enabled security groups, and security groups (collectively referred to as Active Directory (AD) groups, are all created in ‘Groups’ area of the Microsoft 365 Admin portal.

When a new group is created, the following options appear.

As noted above, Microsoft 365 groups are recommended. It is important to understand the relationship between Microsoft 365 groups, Teams and SharePoint.

A new group has a visible mailbox and a Team is created by default

When a new Microsoft 365 group is created (from the dialogue above), it creates:

• At least one Owner must be specified. The Owner/s are responsible for managing the Members group.
• An Exchange mailbox with the same email @ name as the Microsoft 365 group. The mailbox is visible in Outlook to the members of the Group.
• A SharePoint site with the same URL name as the Microsoft 365 group.
• By default (unless the checkbox is unchecked), a new Team is also created in MS Teams.

When a new Team is created from MS Teams, or a new SharePoint Team site is created, it creates:

• A Microsoft 365 Group with an Exchange mailbox and a SharePoint site (‘Files’ tab).
• The name of the Team becomes the name of the Group and the SharePoint site.
• The mailbox is not visible in Outlook and is only used for calendaring and for the storage of Teams chats (in a hidden folder).

Importantly, when a new Microsoft 365 group or Team is created (which creates a Microsoft 365 group), the Group Owners: (a) are the same as the Team Owners and (b) are added to the SharePoint Owners permission group, as explained below. .

Group/Team Owners and Members

In other words, the Microsoft 365 group owners (group) is added to the SharePoint site owners permission group – a ‘group within a group’.

That is, the Microsoft 365 group controls access to the Team and the SharePoint site as shown in the diagram below. Security Groups may also be added to the Microsoft 365 Group site, but this does not provide access to the Team.

The relationship between Microsoft 365 Groups, Teams and SharePoint

This ‘group within a group’ model is visible from the ‘Site Permissions’ section of the gear/cog icon as shown below (the name of the Microsoft 365 Group/Team/SharePoint site is ‘SharePoint Admin’). The SharePoint Admin Group Owners (group) is in the SharePoint site owners group, and the SharePoint Admin Group Members (group) is in the Site members group.

If a mouse hovers over the Group ‘icon’ (in the above example, GO or GM), it is possible to view the members of the Group and, for Owners, to modify that list. Confusingly, the ‘GM’ in the SharePoint site permissions group becomes ‘SG’ in the drop down list.

You can also see the ‘group within group’ model from the back-end ‘Advanced permissions’ section of the SharePoint site, but you cannot manage the Microsoft 365 Group members here.

Implementing the model

As with Security Groups, the members of Microsoft 365 Groups will usually be a logical group of people who require access to something, in this case access to the SharePoint site or the Team (for chat, files, or other resources).

The main thing to remember is that membership of the (backend) Microsoft 365 Group provides access to BOTH the Team and the Team’s SharePoint site (the ‘Files’ tab in a Team).

• Every Team in MS Teams will usually consist of the members of a logical group with a common interest – a business unit, project team, or with some other work relationship, for example, the members of a committee. The Team Owners are responsible for managing the Team Members.
• The Team Owners are the SharePoint site owners and are responsible for managing the site if they decide to access it directly. The Team Members are the SharePoint site members and have the ability to add or edit content, usually via the ‘Files’ tab in Teams.

Note: Security Groups with the same members as Microsoft 365 Groups (and Teams) may already exist. There is no need to add a Security Group if it has the same members as a Microsoft 365 Group.

As noted earlier, a Group/Team does not have visitors with read-only rights. Every Member of the Team has add/edit access to both the Team and its associated SharePoint site.

• If there is a requirement to give specific other people either add/edit or read-only access to the SharePoint site, that outcome is achieved by adding people by name, or a Security Group, to either the SharePoint Members or Visitors group.
• If there is a requirement to give everyone in the organisation either add/edit rights, or read only access, to the SharePoint site, that outcome is achieved by adding ‘Everyone except external users’ to either the SharePoint Members or Visitors group.

External guests may also be added to the Team and the Team’s SharePoint site.