The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.
This post explains the difference between the two ‘in place’ options. In brief:
The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.
The Microsoft in-place model
The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.
Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:
The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
Mandatory record metadata.
Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
Unique computer-generated record identifiersfor each record, regardless of where that record is stored.
The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
Not permitting modification of certain metadata fields.
The capability to support multiple renditions of a record.
The capability to increment versions of records when filing. Linking the record metadata to the record so that it can be accessed for display, export.
Enforcement of data integrity, referential integrity, and relational integrity.
Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):
Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.
The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’
The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’
A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.
One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.
Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.
The new ‘in place’ model
A newer form of in-place records management has appeared with Microsoft 365.
Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.
The newer model of in place records management is based on the assumptions that:
It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.
Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.
In place co-existence
Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.
Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021
SharePoint Online document libraries include the option to move content to a different location. But what happens to the metadata when this content is moved?
This post explains what happens to the metadata when different types of content stored in a SharePoint Online document library is moved (not copied) using the ‘Move to‘ option to a different library in either the same site or a different site (both in SharePoint Online).
If you don’t have time to read all the details, here is a summary. When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:
The original system generated metadata (Created, Modified) is retained for all types of content.
The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
Information Protection labels (set on Office document only) are retained unless a label setting prevents the move.
Retention labels are retained for all types of content.
Any added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.
The same outcomes occur if documents are moved between two synced document libraries in File Explorer.
Metadata in SharePoint libraries
All SharePoint document libraries include four types of metadata:
System generated. Examples include the Created [in SharePoint, not the original item created date] and Modified dates, the Created by and Modified by names, File size, Content Type, sequential ID, and Version History (which is a collection of metadata objects). If Document IDs are enabled, it also includes the Document ID (PREFIX-LibraryID-sequential ID, eg ‘RECORDS-12312322-234). Note that every object saved to SharePoint is also assigned a unique GUID.
Name. This is either the original name of the item (in the case of an email, the email subject line), or added by an end-user. Note that this is NOT the same column as the Title column which is blank by default.
Added metadata. That is, pre-existing or custom site, or ‘local’, metadata columns added to the library. In the examples used in this post, the added metadata is ‘Document Type’ (choice, default =’Agenda’), ‘Event Type’ (choice, default =’Indoor’), and ‘Random Text’ (text, default = ‘Elephant’).
It is important to note that metadata in added metadata columns in one library will NOT appear in the destination library unless that (destination) library has exactly the same columns.
For the purpose of this post, four different types of content (Office/Word document, PDF, email (.msg and .eml), image) were moved from a library with multiple metadata elements shown below to an empty document library in brand new site with nothing configured at the site or library level.
The metadata elements set on the source site library were as follows:
Document Type (choice)
Event Type (choice)
Random Text (default set to ‘Elephant’)
Retention label applied to the library and a different label applied at the folder level
Information Protection label. This was only applied to Word documents (when they were edited) as there was no option to apply it to the emails, PDFs or images.
Moving a PDF, email (msg) and image
As the outcome is likely to be the same for all non-Office content, a PDF, email (msg) and image were all moved at the same time to the destination library. Immediately a notification appeared that metadata properties on all three items would be lost if they were moved. The option to ‘Move anyway’ was selected and the items were moved.
The destination library shows the three items without the Document IDs (which were not enabled) or any of the added metadata columns. However, the items have retained the Retention label – no retention has been set anywhere on the destination site.
Were the Document IDs moved?
No. The original Document IDs did NOT appear because the destination library did not have the Document ID feature enabled.
Once that feature was enabled and the column was made visible in the list view, we can see that the three items that were already moved have lost their original Document IDs. Two new items moved have retained their original source Document ID (CORPRECORDS-667097513-n).
Moving Office documents
The metadata applied to Office documents (Word, Excel, PowerPoint with an ‘x’ at the end) works in a similar way but those documents also embed the metadata in their underlying XML structure.
Some of this metadata is visible in, and can be directly edited from, the Info properties of the document. For example, in this document …
… we can see the Document ID, Document Type, Random Text and Event Type in the Info properties. These metadata properties remain with the document even when it is downloaded and so can be very useful to determine the origin of a document if, for example, it is found as an email attachment or on a network file share.
Where is this metadata embedded?
All Office documents with a document extension that ends in ‘x’ (e.g., docx) are based on an XML structure that can be viewed by converting the original to a zip file (using the CMD shell and REN to rename it) and then extracting the resulting zip file and opening it.
The XML structure of a standard Word document looks like the following. The XML documents contained in the structure can be viewed with simple text viewers like NotePad.
The system, added and embedded metadata properties are stored in one of the XML documents named item1 to item 4 in the customXml folder. The following shows the values applied in item1, while item 4 contains the other values that can be assigned to choice fields.
The metadata for the Sensitivity label (including any formatting details) is stored in the ‘custom’ XML document in the docProps folder.
BUT – details of the Retention label are not stored in the XML properties of the downloaded document. This may be because once the document is downloaded, the retention label no longer applies.
What happens to that metadata when Word documents are moved?
As with the previous examples, a warning will indicate that metadata properties may be lost. If ‘Move anyway’ is selected, the document is moved.
If the document has an Information Protection label setting that prevents it, it may not be possible to move it. The following message will be displayed. The only way to move it is to change the label to one that allows it, or remove the label.
For any other Office document, the original Document ID and Retention label are retained but any added metadata properties will be lost if those columns do not exist in the destination library.
However, it seems that the added properties are not completely lost; they are moved to the ‘custom’ XML document in the docProps folder.
However, if the same metadata column is added to the library, documents that have this added metadata in the ‘custom’ XML document do not display that metadata. It must be added back.
What happens if the destination library has the same metadata columns?
If the destination library has the same added metadata columns, as shown in the example below, that metadata will be copied to the destination library. The first Word document in the example below was copied before the metadata column was added.
When content is moved between SharePoint document libraries (same site or other sites) using the built-in ‘Move to’ functionality, the following occurs for all types of content:
The original system generated metadata (Created, Modified) is retained for all types of content.
The original Document ID is retained for all types of content only if the destination site has the Document ID feature enabled.
Information Protection labels set on Office document only are retained unless a label setting prevents the move.
Retention labels are retained for all types of content.
Added metadata is lost for all content unless the destination library has exactly the same metadata columns. However, Office documents may retain those values in their XML structure.
A common question asked by many organisations is whether Microsoft 365 (M365) retention policies – labels in particular – can be applied to folders in SharePoint document libraries so the content in those folders will have the same label.
The quick answer is yes, but it is a manual process and – for all its perceived benefits – is likely to be more of an administrative and support burden and not worth the effort. Folders should NOT be thought of as the replacement for ‘files’ (aggregations of individual records), but more like dividers in a lever arch (= the document library).
This post describes how labels can be applied to and work with folders, including in SharePoint sites linked with Teams. It also suggests alternative options.
How retention labels are applied to a library
Retention labels are created in the M365 Compliance portal under either the Information Governance > Labels or Records Management > File Plan sections.
Labels created in the Compliance portal do not do anything once created; they must be applied to content in various ways to make them work. This includes by:
Publishing one or more labels as part of a retention policy to various locations including SharePoint sites, Exchange mailboxes and OneDrive (but not Teams – see screenshot of available locations below). In this scenario, each label will be visible to – and selectable by – end-users.
Auto-applying them to the same locations based on various options, including (for E5) trainable classifiers
Adding them to Content Types used in SharePoint Syntex.
Publishing labels to SharePoint sites
When one or more labels are published to SharePoint sites they don’t do anything until they are ‘manually’ enabled through one of the following options:
On each individual document library via the library settings option ‘Apply labels to items in this list or library’ (see screenshot below)
On each individual folder in a library (via the information panel, see screenshot below)
On each individual object in the library (also via the information panel)
Applying a label to the library
A label can be applied to the entire document library via Library Settings, as shown in the screenshot below.
If the drop-down option is set to ‘None’, and there are no options to choose from, it means that no labels have been published to this SharePoint site.
If labels exist, they will appear in the drop down list (below the default ‘None’). Note that only one label can be set as the default for the library. If the check box ‘Apply label to existing items in the library’ is selected, this will apply the label to all existing items. It will also likely override any existing label that may have been applied.
When the retention label has been applied to a library, the label only applies to the non-folder objects stored in the library as can be seen in the option below. That is, the retention label is NOT applied to the folders by default.
The retention label can be seen when the folder is opened:
Applying a retention label to a folder or document
It is also possible to apply a retention label to a folder or object stored in the folder via the information panel, even when a default library label has been set, as shown below. This can be done on each individual folder including, for Teams-based sites, each folder that maps to a channel.
When applied to a folder in this way, any content stored in the folder will inherit that retention label.
If a default label has already been applied at the library level, the folder-based label will replace it, although in testing this, one of the original default labels wasn’t replaced automatically as shown below, but could be manually changed via the information panel.
Implications for Teams-based records (Files)
Every Team in MS Teams has an associated SharePoint site linked with the underlying Microsoft 365 Group.
Every non-private channel in the Team maps to a folder in the Documents library of the SharePoint site as can be seen in the two screenshots below. (Every private channel has a separate SharePoint site that would be covered by a separate retention policy).
Keep in mind that retention labels remove the ability to delete objects stored in the library (including via the ‘Files’ tab in a Team). If end-users are working in Teams, this could be annoying and potentially put them off using Teams. However, end-users can remove the label by navigating to the SharePoint site and removing the label via the Information panel.
Why folder-based retention labels may not be a good idea
The default options to apply retention labels to content stored in SharePoint document libraries are:
By applying them at the library level. This can apply the label to all existing (and future) content stored in the library but does not apply to folders.
Through the auto-application of labels.
Via SharePoint Syntex using labels on Content Types.
Applying retention labels to individual folders in a document library is a manual-intensive process, one that may be a waste of time given the potential number of libraries that can exist and the ease with which they can be removed by end users.
Additionally, applying retention labels to the channel-linked folders of Teams may be pointless if end users:
Store documents at the same level as the channel-linked folders; that is, ‘above’ the folder structure.
Create new folders via a synced library or SharePoint. These folders are not linked to channels.
Create new libraries in the SharePoint site.
Keep it simple
It is very easy to deliberately or inadvertently establish over-complicated retention settings for content stored in SharePoint, especially as there is currently no simple way to see what label has been applied where.
Given the retention period linked with retention policies generally, there is a good chance that the person who applied the labels may not be around when the retention period expires, or to keep an eye on what has been applied or changed over time.
The best retention intentions may be overruled by practical necessity.
The best retention model, in my opinion, is a simple one that does not get in the way of end-users but ensures that records will be kept for a minimum period required. So, instead of applying retention labels to folders, especially on Teams-based SharePoint site libraries, it is recommended to:
Start by trying to avoid mixing content with different retention periods in the same SharePoint site or Team, or document library. That will make it easier to manage the retention outcomes. (If you can’t avoid mixing content, you may need to use auto-application of labels including via Syntex or trainable classifiers).
Use ‘back-end’ safety net retention policies applied to all SharePoint sites. This ensures a minimum retention period and does not get in the way of end-user activities.
Use retention labels on site libraries where more granular retention is required. Ideally, apply them as the default to all the content in a single document library (including the default library for all Teams-based SharePoint sites) and – preferably – only apply the labels when the content is inactive and the library can be made read only, to protect the records from that point.
Only use multiple labels on folders when (a) all the labels applied to the site relate to the same function/activity pair or subject matter, and (b) the content is largely inactive. Ideally, avoid folder-based retention to avoid complication in the future.
Microsoft 365 has become one of the world’s most accessed products for office collaboration. Jeff Teper, the ‘father of SharePoint’ at Microsoft, tweeted on 27 April 2021 that Teams had 145 million daily active users. (As reported in by the team at Office365ITPros.com.) According to the website ‘Microsoft Office Statistics and Facts (2021) | By the Numbers‘ , Microsoft Teams usage grew 40% during the COVID lockdowns.
Although organisations create, capture and store a range of records using the various Microsoft 365 applications, most records are likely to be created or captured in Exchange mailboxes or SharePoint (including OneDrive).
The volume and range of records has, in many respects, overwhelmed traditional standards-based models that required records (including emails) to be copied to a central electronic document and records management system (EDRMS) or identified and then ‘declared’ as records.
Given the reality of the new paradigm, organisations have tried various ways to manage records in Microsoft 365, including by retaining the EDRMS (for high value records), acquiring third-party products that promise to address the ‘gap’ in recordkeeping functionality, or working with the ‘out of the box’ capability.
Whichever method is chose, records managers need to have a very good understanding of where the records are in Microsoft 365 and how they can be managed. In many cases, leaving and managing them where they were created or captured (‘in place’ management), and using new and emerging capability to leverage the power of AI-based tools is likely to be the future state.
With the above in mind, and regardless of which method you follow, the following are ten points that I think are important to consider when managing records in Microsoft 365.
What are your recordkeeping obligations?
It is perhaps the most obvious question but organisations have sometimes rolled out Microsoft 365 without consideration of their obligations for managing records.
Records provide evidence of business activities and accordingly need to be protected to ensure their authenticity, integrity and reliability as evidence. The most common way of achieving this outcome until now has been to ‘lock’ digital records from change. Is this practical in the digital world? How do you lock Teams chats or stop a new thread in an email exchange? Could the same outcome be achieved by recording all changes and ensuring these are retained?
Records are usually subject to minimum retention requirements and understanding what these are is essential. Where there are minimum retention requirements, records cannot be destroyed before a specific period of time based on a particular trigger or event. These requirements are defined in legislation (sometimes based on statutes of limitation) or, for government agencies, in records disposal authorities or schedules (as shown in the example above).
As a starting point, look at these retention requirements and consider how these will be applied to Teams 1:1 chats, or team chats, or emails still in Exchange mailboxes, or OneDrive content. And then extend this to the content stored in Teams/Group-based SharePoint sites and sites not linked with Microsoft 365 Groups.
Consider also what is required to manage the outcome of retention. Do you need to review records due for destruction? Do you need to keep a record of what was destroyed? For all records?
There may be a requirement to categorise or classify records (especially to group them by context and/or for retention purposes where retention is based on classification). How will this outcome be achieved for Teams chats or emails that remain in Exchange mailboxes, or OneDrive content? What other metadata do you need for records?
Your recordkeeping obligations, in particular records retention requirements, should guide the management of records in Microsoft 365.
Where are the records created or captured in Microsoft 365?
Neither Microsoft 365 nor SharePoint is a dedicated recordkeeping system like an EDRMS (see my post ‘SharePoint is not an EDRMS’). Rather, it is an ecosystem of multiple applications that are used to create, capture, store and manage records.
Most records are likely to be stored in either SharePoint (OneDrive is a SharePoint service) or Exchange mailboxes.
A compliance copy of Teams chats are stored in a hidden folder of Exchange mailboxes. Content stored in the ‘Files’ tab is either stored in SharePoint or (for 1:1 chat) in OneDrive.
Of course, there will be some other records – Yammer conversations, tasks and plans, communication sites, calendar entries, forms and even WhiteBoard sessions. But, more than 95% are stored in Exchange mailboxes or SharePoint/OneDrive.
Knowing your recordkeeping obligations and the location of records are the main starting points. In fact, you can map your recordkeeping obligations (especially metadata and retention) to the location of the records.
Do you control the creation of Teams and SharePoint sites or not?
There two, broadly speaking, two approaches to controlling the creation of Teams and SharePoint sites:
Yes, we have controls – There is some kind of control or decision ‘gate’ for the creation of Teams and SharePoint sites.
No, we don’t have controls – End-users can create Teams and SharePoint sites whenever they want. In this case, the points below may not be of much use. You will likely rely on the built-in ‘records management’ capability to manage the records.
If your answer to the question above was ‘No’, then you will probably need third-party products and/or rely heavily on AI-based solutions to manage the records (which is the default Microsoft position).
Map sites and Teams to business functions – don’t mix content
Almost every organisation has a range of business functions. Some of these are common to all or most organisations (e.g., information technology, human resources, financial management, legal, property, etc) while others are ‘core’ (e.g., engineering, manufacturing, research, sales and marketing, etc).
Many organisations are structured around these business functions, and most records retention schedules are based on function (or business function).
If you can map new Teams and SharePoint sites to these functions, this will facilitate the management of records down the track. Mixing content across multiple functions – except where it makes sense to do this, such where there are related but smaller numbers of records in project sites – is going to make it harder to manage the records in the longer term – and more or less the same as letting end-users put whatever they want into a paper box for long-term storage.
A common example where records might be mixed are ‘Corporate Services’ areas that create or capture records across multiple functions, including property, IT, finance and so on. Unless all the records in a Team-based site can be kept for the same period of time, it may be a good idea to separate the records into different sites.
Also keep in mind that some business areas may exist for a long time; having a single (Teams-based) document library that has folders linked to channels may not be the best way to manage records long-term.
The suggestions above don’t take into account Exchange mailboxes, Teams chats or OneDrive accounts because these cannot be mapped to functions.
Naming conventions for sites and teams, and libraries
The main reasons for having naming conventions for SharePoint sites and Teams are:
It is good practice, to avoid acronyms and other less than useful names.
To prevent unnecessarily long names that end up creating very long URLs (e.g., ‘https://tenantname.sharepoint.com/sites/ExecutiveCommittee20202021MeetingsHeldDuringLockdownandrecordedviaMSTeamsSeniorManagersOnly‘.) It is important to know the difference between the URL name and the display name.
Ideally, the original names of Teams and SharePoint sites should be restricted to no more than 14 characters so that Document IDs (that have a 12 character prefix) can be the same as, or very close to, the URL name of the site. For example:
Aside from the default ‘Documents’ library of every Teams-based site, library names should also be subject to naming conventions and restricted to around 20 characters. There are several reasons for this.
The first is how they appear in the left hand navigation of a SharePoint site. There isn’t much point having multiple library names that aren’t easily visible (the two examples below have completely different names after ‘Financial Management’).
The third reason is that it is good practice to have some form of naming conventions.
Ideally, library names should map to the activities that produce the records AND include the year where this is relevant, e.g., ‘Meetings2021’.
There is NO need to repeat words in the tenant or site name – e.g., h**ps://tenantname.sharepoint.com/sites/TenantNameCorporateRecords/TenantNameFinancial%20Management%20%20Accounting%20%20Invoices%202021/Forms/AllItems.aspx
As noted above, this doesn’t apply to the default ‘Documents’ library in Teams-based SharePoint sites (the actual name is ‘Shared%20Documents’).
Metadata and Content Types
For many organisations, the minimum metadata requirements consist of (a) agent (e.g., the person who did something), (b) dates (when they did something) and (c) a unique identifier. That is, who did what and when?
If you need to add more metadata for certain types of records you can really only do this in SharePoint document libraries, including by adding them from the SharePoint Term Store (see below example). It can also be done in Outlook but these metadata terms are not linked with the SharePoint terms.
As for Content Types, do you really need them? SharePoint is made up of multiple Content Types already, including the default ‘Document’ Content Type. It is important to understand how Content Types work in SharePoint before making the assumption that they are required.
In many cases, choice metadata fields can replace the need for Content Types. Custom Content Types may only be needed for specific or high value record types.
Document retention policies and labels
In the first section about recordkeeping obligations, it was noted that most records will be subject to minimum retention requirements. Retention labels and policies are created in the Compliance admin portal of Microsoft 365.
Unfortunately, the current Compliance admin portal provides very little information to show what label or policy was applied where. The only way to do this is to document it yourself. One way to do this is to create a spreadsheet that lists on each row:
The business function and activity from a File Plan or Business Classification Scheme (e.g., Financial Management – Accounting)
Each retention class for that function/activity pair, including the reference number
If that class has been created as a label, what the label name is. If it has been created as a non-label retention policy, what that retention policy name is. (Generally speaking, disposal authority classes don’t refer to Exchange mailboxes, SharePoint sites, MS Teams chats or OneDrive content, so the organisation may need to determine what this minimum retention period should be and how it will manage the retention outcomes).
(Note, the above can be created in the File Plan section of the Records Management part of the Compliance admin portal, E5 licences only. However, it only documents the above information and does not show where the label has been applied.)
Where the label has been applied ‘manually’ – to which SharePoint site/document library, Exchange mailbox or OneDrive account. This point may have multiple location references.
Where the label has been auto-applied through the basic E3 option or, for E5 licences, the document understanding model (DUM) in SharePoint Syntex, or via trainable classifiers.
When the retention will expire.
Retention outcome – If a disposition review (E5 only) exists, the records will be destroyed automatically without any record kept, or ‘do nothing’. See also below.
Remember that retention labels and policies apply to individual items (emails, Teams chat, SharePoint or OneDrive content stored in libraries), not to aggregations (e.g., the entire library or site). The aggregation will continue to exist after the content has been destroyed and no ‘stub’ (a record of what used to exist) will remain.
How will you manage retention outcomes?
Generally speaking, Microsoft 365 retention policies destroy records when they are due for destruction unless they are subject to a label that has the disposition review option enabled or the ‘do nothing’ option has been selected.
Organisations need to understand how they will manage these retention outcomes especially as, in most cases, a review process is required. (See ‘Recordkeeping obligations’ above).
Even when retention label have the disposition review option enabled (E5 only), there are two points that need to be understood:
The ‘disposition review’ interface presents individual records with no context except for the original site URL name. Some additional (default) metadata is now included (from May 2021) but not any added metadata. In most cases, it will be necessary to return to the original library to view the context of the records presented for disposal, and if there are any others.
If records are destroyed through that review process, only basic metadata is retained about what was destroyed.
Organisations that have an obligation to undertake a full review of records due for disposal will likely need to consider establishing workarounds such as exporting the full set of metadata from a document library and then using that to review whether the content of the library can be destroyed. If approval is granted, that decision should be recorded, along with the metadata extract.
Allow end-users to get on with their work
End-users generally don’t have much interest in the management of records beyond the period of time they are important to them. They want to do whatever they want, whenever they want, using the applications they have available to them.
Collaboration no longer consists of email exchanges and document-based records. Creating control gates for the creation of Teams and sites, and insisting on naming conventions for sites and libraries (and folders) may be interpreted negatively.
There needs to be a fine balance between control and freedom and this can impact the creation, capture and management of records. Some of the ways to minimise the impact of recordkeeping requirements include:
Enabling Document IDs on every site.
Creating custom metadata columns on sites or libraries with default values.
Applying non-label ‘safety net’ retention policies to all workloads. Retention policies (along with the Recycle Bin for 90 days) helps with the recovery of accidentally deleted content.
Using various communication methods to highlight useful features including sharing (instead of attaching), the Recycle Bin, versioning in SharePoint/OneDrive, and the ability to have a ‘single source of truth’. These features can be used to ‘soften’ the impact of other recordkeeping obligations in some sites.
Pro-actively monitoring activity across the Microsoft 365 ecosystem, including by monitoring the various dashboards, searches, and audit logs, and responding to events.
Learning more about the Microsoft Graph Explorer and the potential to use AI-based options to manage records.
Use the system for other recordkeeping purposes
The Microsoft 365 environment can be used for other recordkeeping purposes as well. For example:
Managing physical records stored offsite in a SharePoint list.
Keeping a record of records (and SharePoint sites or other systems) that have been destroyed, as well as ongoing destruction review and approval processes.
Publishing policies and procedures (in a SharePoint site, not necessarily a communication site).
Communicating information about managing records (communication site).
Archiving social media content (to Exchange mailboxes).
Searching for content stored in other locations or systems including File Explorer and Line of Business systems (via connectors).
Archiving network file share content, where it can be better protected and then subject to retention and disposal outcomes.
Understanding where records are stored (via dashboards and Power BI reporting).
An electronic document management system (EDMS)supported day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
An electronic records management system (ERMS) was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted.
It could be argued that SharePoint began its life in 2001 as an EDMS and began to include ERMS functionality when SharePoint 2010 was released. So, how is it not an EDRMS?
In simple terms, unlike a traditional EDRMS model that was intended to be the repository for all business records, SharePoint is only one of several repositories that are used to store and manage business records. Instead of centralising the storage and management of records, systems such as Microsoft 365 provide centralised management of records wherever they are stored.
The EDRMS model
Almost all EDRM systems since the mid-1990s have had the following characteristics:
Compliance with a standard. For example, DOD 5015.2 (1997), AS 4390 (1996)/ISO 15489 (2001), ISO 16175 (2010), VERS (1999), MoReq (2001)/MoReq2, etc.
A relational database (for the various functions, metadata and controls (see diagram below) and a separate file share (for the actual content/records), accessed via a user interface.
An expectation that most or all (digital) records would be copied from other systems used to create or capture them to the EDRMS for ongoing storage and protection. This outcome might be achieved through integration with other systems, for example to copy emails to the EDRMS.
The integrated EDM/ERM system model was described in the following diagram in 2008 by the International Standards Organisation committee, ISO/TC171/SC2, in its document ‘Document management applications’:
There are two key problems with the EDRMS model:
They do not (and cannot) realistically capture, store or classify all business records. Emails have remained a problem in this regard for at least two decades. Additionally, there is now a much wider range of born-digital records that remain uncaptured.
Many of the born-digital records that were copied to the EDRMS continue to exist where they were first created or captured.
Any organisation that has been subject to a legal subpoena or Freedom of Information (FOI) request for records will know the limited extent to which EDRM systems provide evidence of all business activities or decision making.
And yet, almost every organisation has a requirement to manage records. There is clearly a disconnect between the various requirements and standards to keep records and the ability to keep them.
To paraphrase from Tony Redmond’s recent (21 March 2021) post ‘SharePoint Hits 20: Memories and Reflections‘, SharePoint was originally designed to allow content (including page-based content in the form of intranets) to be accessed via the web, thereby replacing network file shares.
In practice, most organisations retained their network file shares, built their intranet using SharePoint, and otherwise used SharePoint to manage only some digital content.
It could be argued SharePoint started out as an EDM system and became more like an ERMS when more recordkeeping capability was introduced in SharePoint 2011. But that doesn’t make it an EDRMS in the traditional sense of being a central repository of business records.
The reality is that records are, have been, and will continue to be created or captured in many places:
Email systems. In Microsoft 365, Exchange Online mailboxes are also used to store the ‘compliance copy’ of Teams chats messages.
Network file shares.
SharePoint, including OneDrive.
Other online document management systems, including Google Drive.
Text, chat and instant messages often created in third-party systems, often completely inaccessible or encrypted.
The type and format of a record can vary considerably. For example:
A calendar entry.
A photograph or video recording (including CCTV recording).
The recording of a meeting, in video or transcript form, or both.
Virtual reality simulations.
Social media posts.
3D and digital drawings (e.g., via digital whiteboards).
And, of course, all the data in line of business systems.
Instead of trying to save records into a single EDRM system, Microsoft 365 provides the ability to apply controls over the management of most records where they were created or captured – in email (including archived social media and other records), Teams chat, SharePoint, or OneDrive, and Yammer.
Is there a need for an EDRMS?
There is nothing stopping organisations acquiring and implementing traditional EDRM systems, or even setting up some SharePoint sites, to manage certain high value or permanent records, including some records that can be copied from other create/capture systems (such as email).
But there is also a need to address the management of all other records that remain in the systems where they were created or captured.
In most modern organisations, this requires broader controls such as applying minimum retention periods at the backend, monitoring usage and activity, and the proactive management of disposals. Not trying to copy them all to SharePoint.
Sources for this information are listed where this is known.
1973 – Plato Notes
A history of ERM and EDM systems must include reference to Lotus Notes.
Lotus Notes began its life in 1973 as Plato Notes, developed by the Computer-based Education Research Laboratory (CERL) at the University of Illinois in 1973. Elements of the Plato Notes system would be developed for PC by Ray Ozzie during the late 1970s. This was picked up by Lotus Development Corporation and in 1984 became Lotus Notes.
An early version of Lotus Notes was released (under contract to Lotus) in 1984. The original vision included on-line discussion, email, phone books and document databases. Eventually the product fell into the ‘groupware’ category. The capability of the product continued to grow and some organisations only used Notes.
Lotus acquired all the rights to Lotus Notes in 1987 and version 1.0 was released on 7 December 1989.
1974 – Compulink Management Center/Laserfiche founded
Compulink Management Center was founded in the US in 1974. It created Laserfiche, the first DOS-based document imaging system, in 1987.
1976 – Micro Focus founded
Micro Focus was founded in the UK in 1976. Its first software product was CIS COBOL, a solution for micro computers. It entered the EDRM market in 2017, see below.
1981 – Enterprise Informatics founded
Enterprise Informatics, a privately-held software company, was founded in 1981 by early pioneers of the document management industry. (Source: LinkedIn company profile) It would later be acquired by Spescom, a South African company.
1982 – FileNet founded
FileNet was founded in 1982 by Ted Smith, formerly of Basic 4. FileNet’s original focus of attention was the storage and management of scanned images but it also developed a workflow software. (Source: Wikipedia article on FileNet)
1983 – GMB/DocFind founded
GMB (named after the original founders, Gillett, Frank McKenna, and Bachmann) was formed in Australia in 1983. In 1984, GMB released DocFind 1.0. DocFind was renamed RecFind in 1986.
Tower Software was founded by Brand Hoff in Canberra in 1985 as a software development company. The company provided and supported enterprise content management software, notably its TRIM (Tower Records and Information Management) product line for electronic records management.
The ‘Tower’ in the company name derives from the telecommunications tower on top of Black Mountain (technically a hill, 812 m high) overlooking Canberra. A graphic of the tower was used in the TRIM logo until the company was acquired by HP’s Software Division in 2008 (see also below).
1986 – Autonomy founded (UK)
Autonomy was founded by Michael Lynch, David Tabizel and Richard Gaunt in Cambridge, UK in 1986 ‘as a spin-off from Cambridge Neurodynamics, a firm specializing in computer-based finger print recognition’.
Before 1987 – Saros Corp
Saros Corp was established in Washington by Mike Kennewick (a former Microsoft employee) before 1987. Saros Corp produced Saros Mezzanine, a client-server document management engine. In 1993, released Saros Document Manager.
1989 – Ymijs (later Valid Information Systems) founded – R/KYV (UK)
Ymijs was founded in the UK in 1989. It sold the R/KYV software initially as a basic document imaging processing system. The company name was changed to Valid Information Systems and R/KYV was further developed as a compliance and records management system ‘… that is widely used by major corporations as well as central and local government authorities and related governmental agencies’ (in the UK).
1989 – Provenance Systems (later TrueArc) founded (Canada)
Bruce Miller, sometimes noted as ‘the inventor of modern electronic recordkeeping software’, founded Provenance Systems in 1989 where he created ForeMost. The company name was changed to TrueArc. Bruce would go on to found Tarian Software as well in 1999.
TrueArc ForeMost RM would be acquired in 2002 by Documentum (which which it had a long-standing technology partnership).
1990 – Documentum founded (US)
According to this Wikipedia article, Documentum was founded in June 1990 by Howard Shao and John Newton who had previously worked at Ingres (a relational database vendor). They sought to solve the problem of unstructured information.
The first Documentum EDMS was released in 1993. According to the Wikipedia article, ‘This product managed access to unstructured information stored within a shared repository, running on a central server. End users connected to the repository through PC, Macintosh, and Unix Motif desktop client applications.’
1992 – Altris Software (UK)
Altris, established in 1992 (Source: Rob Liddell\’s LinkedIn profile. Rob was one of the co-founders of Altris), developed document management systems, including (according to this South African ITWeb post of 26 October 2001), eB, a ‘configuration management’ application.
This article titled ‘The Case for 11g‘ (referring to Oracle’s product, see below) noted that Optika’s original software development focus was Image and Process Management (IPM).
An undated (but likely mid to late 1990s) webpage on the Property and Casuality website titled ‘Optika and Xerox Package FilePower with Document Centre‘ noted that ‘Optika Imaging Systems, Inc. and Xerox announced that the two companies will jointly work to integrate Optika’s FilePower with Document Centre digital systems products from Xerox. The combination of the Document Centre and FilePower will provide a complete solution for capturing, managing and distributing large volumes of documents, increasing users’ productivity and significantly reducing labor and capital costs. Optika’s integrated product suite — FilePower — combines imaging, workflow and COLD technology into a unified software package. The Xerox Document Centre 220ST and 230ST combine network scanning, printing, faxing and copying into one hardware device.’
1993 – Workflow Management Coalition formed
The Workflow Management Coalition (WfMC), ‘a consortium formed to define standards for the interoperability of workflow management systems’, was founded in May 1993. Original members included IBM, Hewlett-Packard, Fujitsu, ICL, Staffware and approximately 300 software and services firms in the business software sector.
The WfMC’s Workflow Reference Model was published first in 1995 and still forms the basis of most BPM and workflow software systems in use today. (Source: Undated Gutenberg article)
1993 – Kainos Meridio (UK)
Meridio was developed in 1993 by Kainos (a Northern Ireland company and joint venture between Fujitsu and The Queens University in Belfast) as an electronic document and records management (EDRM) system based on Microsoft products. It would be acquired by HP Autonomy in 2007.
1993 – Saros (US) Document Manager
Saros Corp released Saros Document Manager in mid 1993. The product was said ‘to act as a front-end to the Bellevue, Washington-based firm’s client-server document management engine, Saros Mezzanine’. (Source: Computer Business Review article ‘Saros Sets Document Manager‘ )
ERM before the mid 1990s
Before the arrival of personal computers in offices in the early 1990s, computer mainframes and databases were the regarded by some observers as the only places where electronic ‘records’ (in the form of data in tables) were stored and managed.
A report by the United States General Account Office in July 1999 (GAO/GGD-99-94) titled ‘Preserving Electronic Records in an Era of Rapidly Changing Technology’) stated that, historically (as far back as 1972), NARA’s Electronic Records Management (ERM) guidance (GRS 20) was geared towards mainframes and databases, not personal computers.
The GAO report noted that until at least the late 1990s, there was a general expectation that all other electronic records not created or captured in ERM systems would be printed and placed on a paper file or another system. The original (electronic) records could then be destroyed.
Some early ERM (database) systems, such as TRIM from Tower Software in Australia, were originally developed in the mid 1980s to manage paper files and boxes. Similar systems were developed to manage library catalogues and the old card catalogues started to disappear.
But, although some of it was printed and filed, the volume of electronic records in email systems and stored across network file shares continued to grow. Several vendors released systems that could be used to manage electronic documents (EDM) more effectively than network drives but there was no agreed standard for managing that content as records.
1994 – The DLM Forum and MoReq
From the early 1990s, the European Council sought to promote greater cooperation between European governments on the management of archives. One of the outcomes of a meeting in 1996 was the creation of the DLM Forum. DLM is the acronym of the French term ‘Données Lisibles par Machine’, or ‘machine-readable data’.
One of the ten action points arising from the June 1994 DLM meeting was the creation of ‘Model Requirements for the Management of Electronic Records’, or MoReq, first published in 2001 (see below).
According to its website, ‘the InterPARES Project was borne out of previous research carried out at the University of British Columbia’s School of Library, Archival and Information Studies. “The Preservation of the Integrity of Electronic Records” (a.k.a. “The UBC Project”) defined the requirements for creating, handling and preserving reliable and authentic electronic records in active recordkeeping systems.’
‘The UBC Project researchers, Dr. Luciana Duranti and Professor Terry Eastwood, worked in close collaboration with the U.S. Department of Defense Records Management Task Force to identify requirements for Records Management Applications (RMA).
The work of the UBC team influenced the development of DOD 5015.2 published in 1997 (see below) and the subsequent development of a range of electronic document and records management (EDRM) systems.
Australia – intervention in business applications model
In 1996, the University of Pittsburgh published the ‘Functional Requirements for Evidence in Recordkeeping Project’, led by David Bearman. This work would influence the development of both MoReq2010 and the ICA standards that became ISO 16175-2010, both of which attempted to define a minimum set of functional requirements for a business application to be able to manage its own records. (Lappin)
1995 – IBM Acquires Notes
Lotus Notes was acquired by IBM in July 1995. By December 1996 it had 20 million users. By the end of 1999, Lotus Notes had extensive capability including ERM and EDM.
Lotus Notes continued to retain a strong presence in the market but its dominance began to be reduced by the arrival of Microsoft’s broader capabilities and other EDM solutions.
1995 – Alpharel (US) acquires Trimco (UK)
According to this Computer Business Review article of 23 November 1995, Alpharel Inc, San Diego was expected to acquire Trimco Group Plc of Ealing, London, a supplier of enterprise-wide document management systems.
1995 – FileNet acquires Saros
FileNet acquired Saros Corporation in 1995 to acquire its electronic document management capability. It was said to have pioneered ‘integrated document management’ (IDM), through a suite that offered document imaging, electronic document management, COLD and workflow. (Source: Wikipedia article on FileNet)
1996 – Australian Standard AS 4390
In February 1996 Australia issued the world’s first national records management standard, AS4390 ‘Records Management – General‘. The standard provided guidance for the implementation of records management strategies, procedures and practices.
Tower Software, the Canberra-based developers of TRIM, contributed to the development of the standard (according to its Wikipedia entry) although the standard did not prescribe requirements for the management of electronic records.
AS 4390 would become internationalised through ISO 15489 in 2002.
1996 – OpenText Corporation (US) – Livelink
OpenText Corporation was founded in 1991 from OpenText Systems. It released Livelink in 1996.
1996 – EDM solutions (UK listing)
The following is a list of EDM systems taken from the Document Management Resource Guide, 1995/96 Edition, kindly provided by Reynold Lemming in 2021. (^ = Original software author entry, all others are system resellers)
QStar: Axxess / Server / Worksgroup / Enterprise ^
1996 – Various EDM solutions
The March 1996 edition of Engineering Data Management included a number of updates on electronic document management solutions in the market at that time. Note that Trimco and Alpharel are listed separately; this may because Alpharel’s acquisition of Trimco had not been completed by that time.
Alpharel (San Diego, CA): Document Management solutions – Enabler, FlexFolder, RIPS, Toolkit API. Wisdom, a product that facilitated internet access to participating electronic document vaults.
Auto-trol Technology (Denver, CO): CENTRA 2000, document management, workflow, PDM, change management and messaging.
Cimage Enterprise Systems (Bracknell, UK): Document Manager for Windows.
Documentum, Inc. (Pleasanton, CA): Documentum Accelera for the World Wide Web and Documentum UnaLink for Lotus Notes.
Interleaf (Waltham, MA): Interleaft 6 SGML, a solution for publishing SGML doocuments. Intellecte/BusinessWeb, a document management solution that allowed organisations to access enterprise document repositories from the internet.
Trimco (Ealing, UK): Document management systems.
Alpharel changed its name to Altris Software (US) in October 1996, according to this Telecompaper article published the same month.
From 1996 – Germany’s DOMEA project
In 1996, the Coordinating and Advising Agency of the Federal Government for Information Technology in the Federal Administration (KBSt) introduced a pilot project named Document Management and Electronic Archiving in computer-assisted business processes (DOMEA).
Under the framework of DOMEA, a project group was set up in 1998 to find solutions for the disposition and archiving of electronic records. The goal was to find a suitable and efficient way for the disposition of electronic records created and maintained in office systems. Its “Concept for the Disposition and Archiving of Electronic Records in Federal Agencies,” containing recommendations for managing electronic records was published in September 1998. (Source: The Free Library article)
Late 1990s – EDMS vs ERMS
Electronic document management systems (EDMS) and electronic records management systems (ERMS) were regarded as separate types of system from the late 1990s until at least 2008.
According to Philip Bantin in August 2002:
An EDMS was said to support day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
An ERMS was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted. (The concept of ‘declaring a record’ may be related to this point).
(Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001)
The difference between the two types of system endured for at least a decade. By the end of the 1990s, four main EDRMS options had emerged:
Extending an existing EDM product capability to include ERM.
Extending an existing ERM capability to include EDM.
Creating new ERM products (technically also with some EDM capability).
Integrating separate EDM and ERM products.
1997 – DOD 5015.2
According to the 1999 GAO report quoted above, for several years prior to 1997, NARA worked with the US Department of Defense, considered ‘one of the agencies that is most advanced in its ERM efforts’.
The outcome of this work was the release in November 1997 of the DOD standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’ usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, 11 April 1997.
The GAO report stated that ‘ERM information systems that were in place before the approval of this standard must comply with the standard by November 1999’.
It added that US agencies ‘were confronted with many ERM challenges’ from the ever-increasing volume of digital records, including the ability to preserve and access those records over time. The ‘Year 2000 problem’ was drawing attention away from the issue.
Nevertheless, by 2 June 1999, nine companies were certified as compliant with the DOD standard. Some, it noted, were standalone ERM software, while others were an integrated solution.
An interesting small note on page 11 of the GAO report noted that ‘it is important that ERMS software requires users to make no more than two or three extra keystrokes, and that users realize there is a benefit to this additional ‘burden’.
From 1997 – SER eGovernment (Germany)
SER eGovernment was developed for the German/Austrian market following the release of the German eGovernment standard, DOMEA in 1997.
1998 – Documentum goes online
In 1998, Documentum released its Web Application Environment, a set of internet extensions for EDMS, offering web access to documents stored within an EDMS repository. Various additional products were acquired and their functionality added to the Documentum system.
1998 – Optika eMedia released
Optika released eMedia, ‘a software and methodology product designed to manage business transactions within an organization, across extranets, and throughout the supply chain’, in late 1998. (Source ‘Optika Delivers App to Manage Business Transactions‘) Optika eMedia was said to be ‘a workflow enabled replacement for an imaging solution named FilePower’.
1998 – FileNet Panagon suite released
In 1998, FileNet released its Panagon suite of products. This included Panagon Content Services that was previous Saros Mezzanine. (Source: Wikipedia article on FileNet)
1999 – International differences
The 1999 GAO report noted differences between the US, UK, Australia and Canada on their approach to ‘common ERM challenges’.
Australia was said to have ‘strong central authority (including for compliance audits) and decentralised custody’ (except when the records are transferred to permanent retention).
Canada had ‘vision statements rather than specific policies’ and also had decentralised custody, but agencies could transfer records at any time to the archives.
The UK had broad guidelines put into practice by individual agencies.
1999 – the UK PRO standard released
The UK Public Records Office (PRO, later The National Archives, TNA) released a standard in 1999 designed ‘to provide a tool for benchmarking the ability of government departments to support electronic records management’. This standard would be replaced by TNA 2003. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)
End of the 1900s – XML
By the end of the 20th century it was becoming clear (to some) that XML would likely play a strong role in the standardisation of electronic record formats and their management over time.
XML-based record structures meant that electronic records could contain their own ‘metadata payloads’ rather than being independent objects defined in a separate system (like a library catalogue describes books on shelves).
The establishment of XML-based formats would (after about 20 years) begin to change the way in which records would be managed, although paper records and the paradigm of managing records in pre-defined containers would continue to persist, largely because of the standards developed to manage electronic records – in particular DOD 5015.2.
1999 – EDM/early ERM products
The following is a collated list of EDM (and related) products collated in November 1999:
Autonomy Portal in a Box
CompuTechnics (1990 to 1999)/Objective (from 1999)
Hummingbird (from 1999 with acquisition of PC DOCS)
Insight Technologies Knowledge Server (IKS) / Document Management System (DMS)
Intraspect Knowledge Server (IKS) (KM)
Onyx Enterprise Portal, with integration to various EDM applications
Open Text Livelink
Pitney Bowes Digital Document Delivery (D3)
PC DOCS (acquired by Hummingbird in early 1999)
ReadSoft (OCR processing)
Tower Software / TRIM Captura
1999 – Tarian Software founded
Tarian Software was founded in Canada in 1999 by Bruce Miller, the founder of Provenance Systems (later TrueArc) and creator of ForeMost. Tarian developed the Tarian eRecordsEngine, an embedded electronic recordkeeping technology for business application software. Tarian was the first e-Records technology in the world to be certified against the revised 5015.2 June 2002 standard. Tarian was acquired by IBM in 2002.
1999 – The Victorian Electronic Records Strategy (VERS)
The (Australian) Victorian government’s Public Records Office (PROV) published a standard for the management of electronic records in 1999, Standard 99/007 ‘Standard for the Management of Electronic Records’. The standard, usually known as VERS, defined the (XML-based) format required for the transfer of permanent records to the PROV.
The Standard noted that:
Records must be self-documenting. It is possible to interpret and understand the content of the record without needing to refer to documentation about the system in which it was produced
Records must be self-contained. All the information about the record is contained within the record itself
The record structure must be extensible. It must be possible to extend the structure of the record to add new metadata or new record types without affecting the interoperability of the basic structure.
Several EDRMS vendors developed the capability to create VERS encapsulated objects (VEOs) as required by the standard.
2000 – Spescom (South Africa) acquires Altris (UK)
The South African company Spescom acquired the UK firm Altris Software in 2000, as noted in this (South Africa) ITWeb article of 3 May 2000. Altris was described in the article as ‘a global leader in integrated electronic document management software, with well established channels to international markets’. As a result of this acquisition, Altris UK was renamed Spescom Ltd (UK).
The same journal announced in 2001 that Spescom KMS was ‘the UK operation of Spescom Limited’s US based subsidiary, Altris Software Inc, which specialises in the provision of asset information management software to markets including transportation, utilities and telcos’.
From 2000 – Microsoft adopts XML for Office documents
In 2000, Microsoft released an initial version of an XML-based format for Microsoft Excel, which was incorporated in Office XP.
In 2002, a new file format for Microsoft Word followed. The Excel and Word formats, known as the Microsoft Office XML formats (with an ‘x’ on the end of the document extension), were later incorporated into the 2003 release of Microsoft Office.
Microsoft’s XML formats, known as Open Office XML, later became ECMA 376 in 2006 and later ISO 29500 in 2008 ‘amid some controversy’ over the need for another XML format (see below).
Before 2001 – Intranet Solutions (later Stellent)
Intranet Solutions had developed software called’IntraDoc!’. The product was briefly renamed Xpedio! before the company and product were renamed Stellent in 2001. (Source: ‘Wikipedia article on Oracle Acquisitions‘)
2001 – EDM systems with RM functionality
The following is a list of ‘EDM systems with records management’ functionality available by early 2001:
TRIM (Tower Software, Australia) – integrated ERM and EDM.
In addition to the EDMS/ERMS differences, organisations were also seeking solutions for knowledge management (KM) and content management (CM).
CM solutions were usually portal-based options that mostly became some form of intranet.
Some of the options in the early 2000s included:
Hummingbird’s PowerDOCS for DM and CyberDOCS as the web client for the DM solution, along with Hummingbird’s (formerly Fulcrum) Knowledge Server for KM and PD Accord for web-based collaboration, with the Hummingbird Enterprise Information Portal (EIP) as the portal solution. Plumtree Corporate Portal could also be used as an Enterprise Portal.
iManage’s DeskSite for DM and WorkTeam for collaboration. For KM, WorkKnowledge Server and Concept Search (based on the Autonomy Server). The portal to link all of these was called WorkPortal.
Open Text’s Livelink for DM, KM and collaboration.
Elite’s Encompass, built on Microsoft’s new SharePoint Portal Server (SPS).
Autonomy Server for KM, with Plumtree Corporate Portal as the portal.
Documentum’s DM and CM product coupled with Plumtree Corporate Portal.
Digital Asset Management (DAM) systems, used to manage other types of digital content such as photographs, also appeared around this time.
Information Technology Decisions published a paper on DOD 5015.2 certified products in November 2001 (original source/location has been lost). It noted that there were two types of products:
Products that started life as electronic document management (EDM) systems. Examples included Documentum, Livelink, and DOCS Open.
Products that started life as electronic recordkeeping (ERK/ERM) systems. Examples given included Tower Software’s TRIM, Foremost, iRIMS, Cuadra Star.
The presentation noted that DOD 5015.2 certification was based on alternative options:
Standalone. For example, True Arc Foremost, TRIM, iRIMS, Cuadra Associates Star, Relativity Records Manager, Hummingbird RM 4.0, Tarian eRecords, MDY/FileSurf, Cimage and Access Systems, Highland Technologies Highview-RM, Open Text, Livelink
Partnership. For example, Saperion with e-Manage 2000, Impact Systems eRecords Manager, FileNet with Foremost.
The report included three interesting points:
Both EDMS and ERKS required an enterprise view of information.
An EDMS is driven by business process requirements.
An ERKS (ERMS) is driven by enterprise requirements for the long-term preservation of information.
2001 – The first MoReq
The first version of MoReq was published in 2001. Volume 1 was 500 pages long.
MoReq emphasised the central importance of an electronic records management system, or ERMS. Its stated purpose was:
To provide guidance to organisations wishing to acquire ERMS.
As a tool to audit or check an existing ERMS.
As a reference document for use in training or teaching.
To guide product development by ERMS suppliers and developers.
To help define the nature of outsourced records management solutions.
Few, if any, products were certified against this version of MoReq.
2002 – Optika Acorde
Optika eMedia was rebranded to Optika Acorde in 2002, according to this website ‘The Case for 11g‘.
A June 2002 Gartner report titled ‘Optika Acorde Document Imaging, Workflow and Collaboration Suite‘ noted that Optika Acorde was an ‘integrated software family for managing the content associated with business transactions’ leveraging ‘Optika’s core strengths in document imaging, workflow and enterprise report management.’
2002 – FileNet BrightSpire, later P8 ECM
FileNet released BrightSpire in 2002. This product ‘leveraged the experience gained from integrated document management, web content management and workflow into what became ECM. (Source: Wikipedia article on FileNet)
By 2002 – Enterprise Content Management (ECM)
The term ‘Enterprise Content Management’ (ECM) began to appear more frequently by 2002. The Wikipedia post on ECM noted that ECM technologies descended from ‘electronic Document Management Systems (DMS) of the late 1980s and early 1990s’.
The integration of records management (RM) with business practices.
The capability for integration between RM products, EDM, various other digital products (such as OCR/character recognition technologies), and web publishing products.
Incorporation of Knowledge Management (KM) concepts.
The key word here was ‘integration’ with EDM and other systems, rather than standalone systems. Web-based access became increasingly essential. IBM’s acquisition of Tarian, Documentum’s acquisition of TrueArc’s Foremost were examples of these integrations. (see below)
According to the Wikipedia article on ECM: ‘Before 2003, the ECM market was dominated by medium-sized independent vendors which fell into two categories: those who originated as document management companies (Laserfiche, Saros, Documentum, docStar, and OpenText) and began adding the management of other business content, and those who started as web content management providers (Interwoven, Vignette, and Stellent) tried to branch out into managing business documents and rich media’.
The emergence of ECM quite possibly created the first challenge to centralised ERM through the integration of multiple elements, some of which created, captured or stored records in ever increasing formats.
2002 – OpenDocument XML format
According to the Wikipedia article on the OpenDocument standard, the OpenDocument standard was developed by a Technical Committee (TC) under the Organization for the Advancement of Structured Information Standards (OASIS) industry consortium. Sun and IBM apparently had a large voting influence but the standardization process involved the developers of many office suites or related document systems. The first ODF-TC meeting was held in December 2002.
2002 – An updated GAO report into electronic records
The US General Accounting Office (GAO) released a new report in June 2002 titled ‘Information Management: Challenges in Managing and Preserving Electronic Records’ (GAO-02-586). This report, which was more detailed than the earlier 1999 one, noted among other things that:
The DOD had by March 2002 certified 31 applications against standard 5015.2.
Progress had been made on the development of the Open Archival Information System (OAIS) model which, while initially developed by NASA for archiving the large volume of data produced by space missions, could be applied to ‘any archive, digital library or repository’. XML-based solutions were considered the most likely to be accepted.
From that date, IBM released the IBM Records Manager Version 2.0 (IRM), previously known as the Tarian eRecords Engine (TeRe). Tarian’s e-Records management technology was integrated into IBM’s software offerings, including IBM Content Manager, DB2 database and Lotus software. (Source: IBM press release)
2002 – Documentum 5 and TrueArc Foremost acquisition
Documentum released Documentum 4i, its first Web-native platform, in 2000. In 2002, it launched Documentum 5 as ‘a unified enterprise content management (ECM) platform for storing a virtually unlimited range of content types within a shared repository’.
Documentum acquired TruArc’s Foremost product in October 2002. The Documentum Wikipage above noted that this acquisition ‘added records management capabilities and augmented Documentum’s offerings for compliance solutions.’ The press release cited in this paragraph noted that ‘Documentum and TrueArc are existing technology partners and have worked together to provide an integration for TrueArc’s enterprise-scalable records management solution with the Documentum ECM platform.’
2003 – TNA 2003
The National Archives (TNA) released an updated version of its PRO standard in 2003, known as TNA 2003. This standard would be superseded by MoReq2. (Source: @ZenInformation on Twitter, 12 February 2021).
In October 2003, Open Text acquired the (German) DOMEA-certified SER eGovernment Deutschland GmbH, based in Berlin, Germany as well as SER Solutions Software GmbH, based in Salzburg, Austria. (Source: Open Text Acquires SER eGovernment)
From 2003 – CNIPA (Italy)
The Italian Centro Nazionale per l’Informatica nella Pubblica Amministrazione (CNIPA) published a protocol for the management of electronic records, Protocollo Informatico in 2003.
CNIPA was renamed DigitPA in 2009 and Agenzia per I’Italia digitale (AGID) in 2012. AGID is responsible for defining standards for the management of electronic records in Italian government agencies. (Source: Protocollo Informatico)
Mid 2003 – The challenges of Enterprise Records Management
In Industry Trend Reports of May 2003, Bruce Silver (of Bruce Silver Associates) made the case for Enterprise Records Management in the wake of various ‘scandals’ involving the management of records at the time, including Enron/Anderson.
Silver argued that EDM, email archive, and back-up solutions did not meet the ‘new statutory and regulatory records management requirements’ – DOD 5015.2, SEC Rules 17a-3 and 17a-4, NASD Rules 2210, 3010, and 3110, NYSE Rules 342 and 440, ISO 15489 and MoReq.
Silver also noted that an effective (‘total’) ERM solution would ‘be implemented as an extension of the company’s ECM infrastructure’, providing for a single interface for all records stored in multiple locations ‘including third-party document management repositories in addition to the email system and network file system’.
2003 – Key integrated EDM/RM vendors
The following is a list of ‘key vendors in the (Integrated Document Management) IDM Market Space’ in October 2003. The list is believed to have come from a Butler Group report:
The report of the sale in The Register stated that Stellent’s CEO said that ‘customers are looking to consolidate their content management needs, including imaging, business process management, web content management and record management with one vendor.’ The new product line was named Stellent Imaging and Business Procss Management (IBPM). The article also noted that Oracle would probably acquire Stellent following this acquisition (see 2006, below).
2004 – ReMANO (Netherlands)
In 2004, the Netherlands government established a catalogue of software specifications for ERM systems (ReMANO) used by Dutch government bodies. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)
ReMANO was replaced by NEN2082 – Eisen voor Functionaliteit van Informatie- en Archiefmanagement in programmatuur” in 2008. (NEN 2082:2008 nl)
2005 – C6 (France) builds D2 on top of EMC Documentum
The French ECM company C6 built a solution named D2, ‘a fully configurable web application for creating, managing, storing and delivering any type of information’, on top of EMC’s Documentum. (Source: C6 website ‘Company’ tab).
2006 – The National Archives of Australia ERMS standard
The National Archives of Australia (NAA) released its ‘Functional Specifications for Electronic Records Management Systems Software in February 2006. (ISBN 1 920807 34 9). The introduction noted that:
(The document) provided Australian Government agencies with a set of generic requirements for ensuring adequate recordkeeping functionality within electronic records management systems (ERMS) software.
Agencies were encouraged to make use of the ERMS specifications when designing or purchasing new, or upgrading existing, ERMS software. They could also be used when auditing, assessing or reviewing an agency’s existing ERMS software.
The requirements were not intended to be a complete specification, but rather provide a template of key functional requirements that agencies may incorporate into their tender documentation when preparing to select and purchase new ERMS software. Agencies were expected to assess and amend the functional requirements, and select requirements that best suit their own business and technical requirements and constraints.
Very few products met the specific requirements of the ERMS specifications which led to some suggestion at the time that it limited choice.
2006 – Rival XML Office document standards
The OpenDocument (ODF) standard was published as ISO/IEC 26300 in 2006.
Microsoft submitted initial material to the Ecma International Technical Committee TC45, where it was standardized to become ECMA-376, approved in December 2006. It was released as ISO 29500 in 2008.
According to the Wikipedia article on Open Office XML (OOXML), ‘The ISO standardization of Office Open XML was controversial and embittered’, as it seemed unnecessary to have two rival XML standards.
2006 – The world of collaboration
The Butler Group published a paper titled ‘Document Collaboration – Linking People, Process and Content’ in December 2006. The report noted that EDM systems had helped improve internal efficiency but there was now a need to ‘extend these systems to partners and stakeholders’ and deliver ‘sophisticated collaborative experiences’.
The paper listed the following EDM products:
Adobe Acrobat family
IBM Notes/Domino, Workplace collaboration services, QuickPlace
Microsoft Office 2007
Open Text Livelink ECM – eDOCS (incorporating the former Hummingbird product suite acquired by Open Text in 2006)
Oracle Collaboration Suite, Content DB and Records DB
Stellent Collaboration Management
2006 – Spescom Software Inc
A US SEC submission in January 2006 noted that Spescom Software Inc, a San Diego-based provider of computer integrated systems was the successor to Alpharel Inc and Altris Software Inc.
2006 – Oracle acquires Stellent
A 2006 Oracle press release titled ‘Oracle Buys Stellent‘ stated that Stellent was a global provider of enterprise content management (ECM) software solutions that included Document and Records Management, Web Content Management, Digital Asset Management, Imaging and Business Process Management, and Risk and Compliance. It also noted that the acquisition would ‘complement and extend Oracle’s existing content management solution portfolio’. Despite the acquisition, the ‘Stellent’ name persisted.
2006 – Google enters the online EDM productivity and collaboration market
In 2006, Google launched Google Apps for Your Domain, a collection of cloud computing, productivity and collaboration tools, software and products. Various apps and elements were acquired and/or added over the years but a key one from an EDM point of view was Google Docs (Wikipedia article). However, Google Docs had no RM capability.
A ZDNet article in June 2007 noted that Google Apps offered a tool for switching from Exchange Server and Lotus Notes, making Google a real alternative to Microsoft and IBM. Google Apps would later be rebranded G-Suite in 2016.
2007 – Spescom exits the EDM market / Enterprise Informatics
In 2007, Spescom exiting the enterprise software sector with the sale of its US operation Enterprise Informatics. (Source – Wikipedia article on Spescom, original reference no longer accessible).
Enterprise Informatics, originally founded in 1981, continued in existence as a subsidiary of Bentley Systems, Incorporated. It continued to market a suite of integrated document, configuration, and records management software products, mostly under the name eB.
2007 – Zoho enters the online EDM collaboration market
The India-based Zoho Corporation, known as AdventNet Inc from 1996 to 2009, released Zoho Docs in 2007.
2007 – HP Autonomy acquires Meridio
Meridio was acquired by HP Autonomy (a company that had had a long business partnership with Kainos) in 2007. The parent company Kainos continued to work with SharePoint-based solutions.
2007 – EDRMS vendors
Forrester released a report into electronic records management vendors in early 2007. The products that it evaluated were as follows:
CA MDY FileSurf v7.5 ^
EMC Records Manager 5.3
IBM FileNet P8 Records Manager v3.7 ^
IBM Records Manager v4.1.3 #
Interwoven Records Manager v 5.1 *
Meridio Document and Records Manager v4.4 *
Open Text Livelink ECM – Records Management v3.8 ^
Open Text Livelink – eDOCS RM (formerly Hummingbird) v6.0.1
Oracle (formerly Stellent) Universal Records Management v7.1 #
Oracle Records DB v1.0 ~
Tower Software TRIM Context v6.0 *
Vignette Records & Documents v 7.0.5
(Forrester assessment: ^ = leaders, # = close behind leaders, * = have hurdles to remain competitive’, ~ = basic functionality only)
2008 – NEN 2082
The Dutch government replaced ReMANO with NEN 2082 ‘Eisen voor functionaliteit van informatie- en archiefmanagement in programmatuur’ (‘Requirements for functionality of information and archive management in software’) in 2008 (NEN 2082:2008 nl). NEN 2082 was derived from MoReq, DOD 5015.2 and Australian standards. (See Eric Burger’s blog post ‘Nee, NEN 2082 is geen wettelijke verplichting‘ about its legal standing)
2008 – MoReq2
MoReq2 was published in 2008. It included new sections to support the testing of ERMS software for compliance with the standard. MoReq2 included the following vendors on its panel (Acknowledgements section):
EDRM Solutions, USA
ErgoGroup AS, Norway
Haessler Information, Germany
ICZ, Czech Republic
Lockheed Martin, USA
Objective Corporation, UK
Open Text Corporation, UK
SER Solutions Deutschland, Germany
Tower Software, UK
Both MoReq and MoReq2 were based on the premise of a central ERMS being acquired and implemented by organisations to manage unstructured records, the types of records that are stored across network drives and in email systems. MoReq2 specifically clearly excluded the management of ‘structured data … stored under the management of a data processing application’. (Source: MoReq2, section 1.2 ‘Emphasis and Limitations of this Specification’, page 12.)
The first software product certified against MoReq2 was Fabasoft Folio. It was the only certified product until June 2014.
2008 – EDMS and ERMS
In 2008, the International Standards Organisation, under ISO/TC171/SC2 ‘Document management applications’ proposed a framework for the integration of EDM and ERM systems. The definitions contained in that framework document noted that:
An EDMS was used to manage, control, locate and retrieve information in an electronic system.
An ERMS was used to manage electronic and non-electronic records according to accepted principles and practices of records management.
An integrated EDRMS would combine both capabilities.
Section 6 of the report described general (but fairly detailed) functional requirements for an integrated EDMS/ERMS, outlined in the following diagram:
2009 – Autonomy acquires Interwoven
In 2009, HP Autonomy acquired Interwoven, a niche provider of enterprise content management software mostly to the legal industry. It primarily competed with Documentum in this space. Interwoven became Autonomy Interwoven and Autonomy iManage.
2010 – MoReq2010
MoReq was completely revised and published as MoReq2010 in 2010. There were key differences with its predecessor versions.
It de-emphasised, but did not remove, the idea of an ERMS being the central or sole recordkeeping system or repository for organisations.
It emphasised the need for line of business systems to incorporate a minimum, defined level of recordkeeping functionality.
It brought a degree of practicality about the management of records in other systems.
It provided for interoperability between all MoReq compliant systems, based on a common XML language.
MoReq2010 established ‘… a definition of a common set of core services that are shared by many different types of records systems’. It provided a set of modules that could be incorporated into any software solution, including line of business applications, so they can be ‘MoReq compliant records systems’ (MCRS).
2010 – Google’s DM capability enhanced
In March 2010, Google acquired DocVerse, an online document collaboration company. DocVerse allowed multiple user online collaboration on Microsoft Word documents, as well as other Microsoft Office formats, such as Excel and PowerPoint. (Source – Wikipedia article on Google Docs)
‘Microsoft SharePoint 2010 is a software product with a range of uses, including website development, content management and collaboration. SharePoint allows users to collaborate on the creation, review and approval of various types of content, including documents, lists, discussions, wiki pages, web pages and blog posts. SharePoint is not a recordkeeping system (i.e. a system purposely designed to capture, maintain and provide access to records over time). When implemented ‘out of the box’, SharePoint has limited capacities for capturing and keeping records in a way that supports their ability to function as authentic evidence of business.’
Adam Harmetz, the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft said in a recent online interview about Records in SharePoint 2010, “We constantly get questions from around the world about how to deal with local government and industry standards for information management. Let me throw just a few at you… MOREQ2, VERS, ISO 15489, DOMEA, TNA, ERKS, the list goes on. Some of these standards are loosely based on one another and some have contradictory elements. Rather than focus our engineering efforts on addressing each of these standards in turn, we made the choice to deliver the usability and innovation required to make records management deployments successful and allow our partner ecosystem to build out the SharePoint platform to deal with specific requirements for those customers that are mandated to adhere to a specific standard.”
Despite these comments, SharePoint 2010 was assessed by at least one consultant (Wise Technology Solutions) to meet 88% of the requirements of the then ICA Standard that became ISO 16175 Part 2.
On 16 December 2011, State Records NSW published a blog post titled ‘Initial advice on implementing recordkeeping in SharePoint 2010‘. The post noted that the Wise report had concluded that ‘SharePoint is 88% compliant with the ICA requirements’. It added that the areas where full compliance could not be achieved relate to:
ease of email capture
native security classification and access control
physical and hybrid records management
The report states that third party providers are able to offer products that plug SharePoint’s gaps in these areas.
The blog posted also stated that ‘… the report very clearly makes the point that ‘we note that the achievement of these results is reliant on appropriate design and governance of implementation, configuration and set up to ensure consistency with desired records management outcomes’.’
Early 2010 – Microsoft launches Office 365
Microsoft launched Office 365 on 28 June 2011. Office 365 was designed to be a successor to Microsoft’s Business Productivity Online Suite (BPOS). (Source: Wikipedia article on Office 365). It would not be until the mid 2010s that Office 365 would become an effective counter-solution to the G Suite.
2011 – HP acquires Autonomy
In 2011, Hewlett-Packard acquired Autonomy, a deal that resulted in some interesting subsequent legal issues reading the value of the company.
As noted in the scope section of the standard, ‘(The) specification provides models for software services to support management activities for electronic records’. Further, ‘… models are provided that describe the platform independent model (PIM) that defines the business domain of Records Management and the RM services to be provided’. Three technology-specific implementations are specified:
PSM-1 – Web Services definition for Records Management Services in Web Service Description Language (WSDL). This is actually supplied as ten WSDL files; one for each Records Management Service.
PSM-2 – A Records Management Service XSD. The XSD is for use in creating XML files for import/export of Managed Records from compliant environments and to use as a basis for forming XQuery/XPath statements for the query service.
PSM-3 – An Attribute Profile XSD. The XSD is for capturing and communicating attribute profiles to permit flexible attribution of certain types of Records Management Objects.
2011 – The death of ERM systems?
In a May 2011 blog post on MoReq2010, James Lappin suggested that traditional systems used to manage electronic documents and records, while not being entirely dead in the water, had ‘lost momentum’.
James proposed two specific reasons for this situation:
The global financial crisis (GFC) from 2008 that limited the ability of organisations to acquire and implement hugely expensive ERMS solutions.
The rise of Microsoft SharePoint and particularly SharePoint 2010. In some ways, Sharepoint 2010 had the potential to take – and may have already taken – the ERMS wind from the records managers sails.
He also noted that a series of interrelated user-environment issues may have also played a part in the loss of momentum.
Usability and take up rates of the ERMS. These solutions are sometimes seen as ‘yet another system’ to manage the same records, using a classification structure that doesn’t make sense to most end users and is different from the way end users see and categorise their world.
The ongoing availability of and access to alternative places to store information, including network drives and email folders, and cloud-based storage and email solutions.
The rise and general availability of social networking tools and mobile applications used to create and share new forms of information content, and collaborate and communicate, including wikis, blogs, Twitter, Facebook, and similar solutions, often in an almost parallel ‘personal’ world to the official record.
The inability of ERMS solutions to manage structured data or to maintain and reproduce easily the diverse range of content created and stored in products like SharePoint. Indeed, one reasonably well known product has been described as an archive for SharePoint, even though the latter can quite easily manage its own archives.
The rise of search as a tool to find relevant information in context, and the related change from unstructured to structured in XML-based documents generated by products such as Microsoft Office 2007 and 2010.
From 2013 – GEVER (Switzerland)
From 2013, the Swiss Federal Chancellery was responsible for managing all activities relating to electronic records and process management (Elektronische Geschäftsverwaltung), or GEVER. GEVER consisted of a collection of five standards for the management of electronic records. (Source with current update: Gever Bund)
2015 – Hewlett Packard separates
In October 2015, the software products previously under the Autonomy banner were divided between HP Inc and Hewlett Packard Enterprise (HPE). HP Inc was assigned Autonomy’s content management software components including TeamSite, Qfiniti, Qfiniti Managed Services, MediaBin, Optimost, and Explore.
2015 – EDRMS vendors
Despite the alleged death of ERMS products in around 2010, many continued to thrive and grow. Some were acquired by others.
The following is a list of EDRMS vendors in December 2015 taken from a Gartner report diagram titled ‘Product or Service Scores for Trusted System of Record’ (with the scores included). Many of these products also appeared in the October 2016 ‘Magic Quadrant’ for Enterprise Content Management Systems as indicated)
Alfresco (2.47) (also ECM)
Open Text (4.07) (also ECM)
EMC Documentum (3.94) (as Dell EMC)(Acquired by Open Text)
IBM (3.91) (also ECM)
Oracle (3.45) (also ECM)
Laserfiche (2.45) (also ECM)
Microsoft (SharePoint) (2.38) (also ECM)
Hyland OnBase (2.37) (also ECM)
Lexmark (2.32) (also ECM)
Newgen (2.18) (also ECM)
Objective (2/10) (also ECM)
By 2015 – Oracle departing the scene?
In a July 2015 article titled ‘Looking for an Oracle IPM replacement‘ in the blog softwaredevelopmentforECM, it was noted that Oracle was ‘clearly, and publically, going in a different direction and moving away from traditional enterprise imaging and transactional content management’.
2016 – OpenText, Micro Focus
In May 2016, OpenText acquired HP TeamSite, HP MediaBin, HP Qfiniti, HP Explore, HP Aurasma, and HP Optimost from HP Inc.
The following is a list of products identified by the Victorian Public Records Office (PROV) in 2020. These products were all certified against the VERS standard, that required organisations to be able to create XML-based VERS Encapsulated Objects (VEOs) for long-term preservation.
AvePoint RevIM, Records, Cloud Records
Bluepoint Content Manager
Canon Therefore 2012
ELOprofessional / ELOenterprise
HP Records Manager
IBM Enterprise Records
IBM FileNet P8 Records Manager
MicroFocus Content Manager
OpenText eDOCS RM
OpenText Records Management
Oracle WebCentre Content
Technology One ECM
2021 – EDRMS vendors
The following is a list of dedicated vendors that offered EDRMS solutions (and more in most cases) by early 2021. Many of these vendors have a long history not necessarily reflected in the above text. Most of these vendors provide Enterprise Content Management (ECM) services, including EDM and ERM capabilities.
Alfresco ECM (alfresco.com)
Hyland OnBase (hyland.com)
IBM ECM (ibm.com)
Knowledgeone RecFind EDRM (knowledgeonecorp.com)
Laserfiche RME (laserfiche.com)
Lexmark RIM (lexmark.com)
Micro Focus Content Manager (microfocus.com)
Microsoft 365 (microsoft.com)
Newgen RMS (newgensoft.com)
Objective ECM (objective.com)
Open Text ECM (opentext.com)
Oracle ECM (docs.oracle.com)
TechnologyOne ECM (technologyonecorp.com)
2021 – Dedicated EDMS vendors
EDM vendors never went away, but many – like Google Drive, DropBox and Box – were built in and for the cloud. This Capterra website has a fairly detailed listing of current EDMS vendors.
The future of standards-based ERM/EDRM/ECM systems
Although the definition of a record has remained largely intact for the past two decades – ‘evidence of business activity’ (ISO 15489) – the form of records has evolved and continues to do so.
The ever-expanded world of digital content has made it increasingly difficult to accurately and consistently identify, capture and manage records in all forms, a challenge to the notion that all records can be stored in a single system.
The ‘in place’ approach to managing electronic records – wherever they are stored – has strong appeal. But where will we be in another 20 years? Some thoughts:
Electronic databases, whether on-premise or cloud-based (including subscription based), will be the primary method of capturing and storing a wide range of digital content rather than network file shares.
Metadata will be automatically captured or auto-generated for all digital content based on the content itself.
Artificial Intelligence (AI) will continue to grow in maturity, allowing records to be identified from all other digital content, classified, aggregated, and managed through to disposal/disposition or transfer to archives.
Email will, slowly, disappear as the current workforce transitions to chat- and video-based communication methods.
Microsoft 365 includes a range of connectors, in three categories, that can be used to support the management of records created by other applications. The three categories are:
Search connectors, that find content created by and/or stored in a range of internal and external applications, including social media.
Archive connectors, that import and archive content created by third-party applications.
API connectors, that support business processes such as capturing email attachments.
This post how these connectors can assist with the management of records.
The recordkeeping dilemma
Finding, capturing and managing records across an ever increasing volume of digital content and content types has been one of the biggest challenges for recordkeeping since the early 2000s.
The primary method of managing digital records for most of the past 20 years has been to require digital records (mostly emails and other digital content created on file shares) to be saved to or stored in an electronic document and records management system (EDRMS). The EDRMS was established as ‘the’ recordkeeping system for the organisation.
EDRM systems were also used to manage paper records which, over the past 20 years, have mostly contained the printed version of born-digital records that remain stored in the systems where they were created or captured.
There were two fundamental flaws in the EDRMS model. The first was an expectation that end-users would be willing to save digital records to the EDRMS. The second was that the original digital record remained in place where it was created or captured, usually ignored but often the source of rich pickings for eDiscovery.
The introduction of web-based email and document storage systems, smart phones, social media and personal messaging applications from around 2005 (in addition to already existing text messaging/SMS messages) further challenged the concept of a centralised recordkeeping system; in many cases, the only option to save these records was to print and scan, screenshot and save the image, or save to PDF, none of which were particularly effective in capturing the full set of records.
The hasty introduction from early 2020 of ‘work from home’ applications such as Zoom and Microsoft Teams has been a further blow to these methods.
In place records management
To the chagrin of records managers around the world, Microsoft never made it easy to save an email from Outlook to another system. Emails stubbornly remained stored in Exchange mailboxes with no sign of integration with file shares.
And for good reason – they have a different purpose and architecture to support that purpose. It would be similar to asking when it would be possible to create and send an email in Word.
The introduction of Office 365 (later Microsoft 365) from the mid 2010s changed the paradigm from a centralised model – where records were all copied to a central location and the originals left where they were created or captured, to a de-centralised or ‘in place’ model – where records are mostly left where they were created or captured.
The decentralised model does not exclude the ability to store copies of some records (e.g., emails) in other applications (e.g., SharePoint document libraries), but these are exceptions to the general rule.
It also does not exclude the ability to import or migrate content from third-party applications where necessary for recordkeeping purposes.
Microsoft 365 connectors
Microsoft 365 includes a wide range of options to connect with both internal and external systems. Many of these connectors simplify business processes and support integration models.
Connectors may also be used to support recordkeeping requirements, in three broad categories.
Archive connectors allow organisations to import and archive data from third-party systems such as social media, instant messaging and document collaboration* platforms. Most of this data will be stored in Exchange mailboxes, where it can be subject to retention policies, eDiscovery and legal holds.
(*This option is still limited via connectors, but also see below under Search).
The social media and instant messaging data that can be archived in this way currently includes Facebook (business pages), LinkedIn company page data, Twitter, Webex Teams, Webpages, WhatsApp, Workplace from Facebook, Zoom Meetings. For the full listing, and a detailed description of what is required to connect each service, see this Microsoft description ‘Archive third-party data‘.
An important thing to keep in mind is that the data will be archived to an Exchange mailbox; this will require an account to be created for the purpose. Any data archived ot the mailbox will contribute to the overall storage quotas.
Search connectors (also known as Microsoft Graph connectors) index third-party data that then appears in Microsoft search results, including via Bing (the ‘Work’ tab), from http://www.office.com, and via SharePoint Online.
Most ECM/EDRM systems are listed, which means that organisations that continue to use those systems can allow end-users to find content from a single search point, only surfacing content that users are permitted to see.
The following is an example of what a Bing search looks like in the ‘Work’ tab (when enabled).
Note: as at 17 November 2020, Microsoft’s page ‘Overview of Microsoft Graph connectors‘ (which includes a very helpful architecture diagram) states that these are ‘currently in preview status available for tenants in Targeted release.’
There are two main types of search connector:
Microsoft built: Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL, Enterprise websites, MediaWiki, Microsoft SQL, and ServiceNow.
Partner built. Includes the following on-premise and online document management/ECM/EDRM connectors – Alfresco, Alfresco Content Services, Box, Confluence, Documentum, Facebook Workplace, File Share (on prem), File System (on prem), Google Drive, IBM Connections, Lotus Notes, iManage, MicroFocus Content Manager (HPE Records Manager, HP TRIM), Objective, OneDrive, Open Text, Oracle, SharePoint (on prem), Slack, Twitter, Xerox DocuShare, Yammer
A consideration when deploying search connectors is the quality of the data that will be surfaced via searches. Duplicate content is likely to be a problem in identifying the single – or most recent – source of truth of any particular digital record, especially when the organisation has required records to be copied from one system (mailbox/file share) to another (EDRMS).
API connectors provide a way for Microsoft 365 to access and use content, including in third-party applications. To quote from the Microsoft ‘Connectors‘ web page:
‘A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.’
Actions. These are changes initiated by an end-user.
Triggers. There are two types of triggers: Polling and Push. Triggers may notify the app when a specific event occurs, resulting in an action. See the above web page for more details.
API connectors can support records management requirements in different ways (such as triggering an action when a specific event occurs) but they should not be confused with archiving or search connectors.
The connectors available in Microsoft 365 support the model of keeping records in place where they were first created or captured. They enable the ability to archive data from third-party cloud applications, search for data in those (and on-premise) applications, and triggers actions based on events.
The use of connectors should be part of an overall strategic plan for managing records across the organisation. This may include a business decision to continue using an ECM/EDRMS in addition to the content created and captured in Microsoft 365. Ideally, however, the content in the ECM/EDRMS should not be a copy of what already exists in Microsoft 365.
In July 2020, Microsoft announced that it would turn off SharePoint 2010 workflows for newly created tenants from 1 August 2020, and would remove the ability to run or create these workflows from existing tenants from 1 November 2020. It recommended that organisations use Power Automate instead. (Source: Microsoft Ending Workflows for SharePoint 2010 Online Next Month).
This post details an alternative option, with no workflows required – but with the same set up – that may be sufficient in most cases. At this sage (September 2020), there does not appear to be an out of the box option with Power Automate that will create a document (from a content type) when a new document set is created.
The alternative option requires the following elements:
The document set feature enabled.
New site columns for each unique metadata required in the output documents.
New document set and document content types, both with the same metadata.
One or more Word document templates.
A SPO site with a document library. It is recommended that a dedicated library (e.g., ‘Client Agreements’) is created for this purpose.
Note – the person who does this must be a Site Collection Administrator.
The main difference with the earlier model is that there is no workflow to create the document/s. Instead, the end user creates and saves (without modification except a name) the document into the library/document set where the custom metadata is stored. The method below then auto-populates those new documents with the required metadata.
This option does not use a separate list as the metadata details can be extracted from the document sets.
Enabling the document set feature
The document set feature must be enabled via Site Settings > Site Collection Administration > Site collection features. If you don’t see this option, you don’t have the required level of permission.
It may be a good idea to activate the Document ID Service at the same time as this can be a useful additional piece of metadata in the final documents.
New site columns
From the same Site Settings area, go to the Web Designer Galleries section to add the required new site columns. Directly below that option is where you create new site content types.
The Site Columns section has a very large number of default columns that come with every SPO site. Check to make sure the required columns don’t already exist, or are non-recommended column names (like ‘Library’).
Assuming the required columns do not exist, select the ‘Create’ option to create a new custom column. A couple of points to keep in mind here:
Give the column a name that is obvious (easy to find).
Don’t put spaces between words but capitalise each word (e.g., ‘ClientName’, ‘ClientAddress’). From experience, this space can cause the metadata element to wrap to a second line and cause problems.
Keep in mind that the ‘Single line of text’ is free text. If you want controlled choices, use the Choice option instead.
Date and Time may require some tweaking to get the correct format in the output Word document. It may be easier to use ‘Single line of text’ if some dates may vary or aren’t exact.
Don’t use the ‘Yes/No’ option. Instead, use a Choice column with those options instead.
Person or Group is only for internal names recorded in Active Directory (AD). If you need to insert any other name, use the single line of text.
The additional column settings that may be useful are:
Require that this column contain information. If changed to ‘Yes’, this metadata field will be mandatory.
Enforce unique values. Generally not used unless there is a requirement to enforce unique values (e.g., unique client IDs).
Maximum number of characters. This can be useful to restrict the number of characters. For example, ‘Postcode’ will only have four characters. Don’t use the number field for postcodes.
Default text. This option is useful if there is a requirement to have a default value, especially from a choice list, or if the value will usually always be the same (although that could also be simply added to the text of the document template.
Column formatting. Don’t use this if the content is to be copied to a document template.
Column validation. This may be useful to ensure that only certain characters are entered. For example, postcodes will never have a letter, only numbers.
Repeat this process until all the required metadata columns exist.
New document set content type
Create a document set content type so the output Word documents (and any other digital content) can be stored in the same ‘folder’.
To create a new document set content type, return to the Site Settings and, under Web Designer Galleries, select ‘Site content types’.
Give the new content type a name that is obvious for its purpose (e.g., ‘ClientAgreementsFolder’).
Importantly, change ‘Select parent content type from’ > Document Set Content Types, and ‘Parent Content Type’ > Document Set.
Open the content type that has been created. Under the ‘Columns’ section, add the columns ‘from existing site columns’. The example below shows a series of site columns that were added. Don’t remove any of the other columns.
In the above example. the ‘Title’ column is hidden (default setting). If you click this column, you will see the option under the ‘Change Content Type Column’ to make it required, optional or hidden. Generally, these settings should not be changed.
New Document Content Type
Now, create a new document content type. Give the content type a name that reflects its purpose. Ensure that ‘Select parent content type from’ is set to ‘Document Content Types’ and ‘Parent Content Type’ is set to ‘Document’ as shown below.
The next step is to add the two content types to a document library.
Adding the content types to a document library
All the steps below can be carried out by a Site Owner.
Open the library where the content types are to be added and go to Library Settings (via the Cog/Gear option). In the Library Settings, under ‘Advanced Settings’, click the option to ‘Allow management of content types’ (change from No to Yes).
If you don’t want other folders to be created in this library, change ‘Make ‘New Folder’ command available’ from Yes to No.
A new section will now appear in the Library Settings – ‘Content Types’.
To add the two content types that were created, click on ‘Add from existing site content types’, select each content type from the list that is displayed, and click OK.
The two content types should now appear in the list of Content Types and the custom site columns should appear in the ‘Columns’ list.
Click on the document set content type (e.g., ‘ClientAgreementsFolder’). The custom columns should be visible. Now, we need to share these columns with the custom document. Click on ‘Document Set Settings’.
From the list of ‘Available Site Content Types’, chose the custom document content type that was created above. Ignore the ‘Document’ content type.
In the ‘Shared Columns’ section, check the box against all the custom columns.
The steps above will mean that all the custom columns in the document set will be shared with the the document content type. When you return to the Library Settings, you will see under the ‘Columns’ section that the columns are now shared with the two custom content types as well as the default Document content type (which can be ignored).
Add the Word template to the document content type
Now, add the Word template to the document content type. Note that this step is NOT done at the Site Content Types level, but at the local library level.
Ensure the Word document template is in the latest version of Word (e.g., docx) and has all the text required. It can be updated at any time (see below) but it is important to ensure that the document is as complete as possible, and the sections where the metadata is to be placed is obvious. One way to do this is to put text such as CLIENT NAME in the body of the document. Avoid using highlights as this can be troublesome (but not impossible) to remove.
From the Library Settings, Content Types section, click on the custom document content type (e.g., ‘ClientAgeements’). Click on Advanced settings.
Click ‘upload a new document template’ and add the document template. The name of the template should be obvious. It is possible to go back and edit the template at any time after it has been uploaded. This is usually much easier if a change is required, compared with uploading a new template and having to re-link all the metadata.
Connect the metadata to the document content
From the Document Template section above, click on ‘Edit Template’. This should open the document in the installed Word application. These steps cannot be carried out using Word Online.
Place the cursor in the body of the text where the metadata is to be placed. Click on the ‘Insert’ tab in Word, then click on ‘Quick Parts’ in the text section.
Click on ‘Document Property’ and the custom columns will appear.
When you select each property, the word document will display it as shown below. Continue this process until all the required metadata is added to the document:
As noted earlier, it is usually much easier to edit an existing template ‘in place’ instead of uploading a new version because the steps involved in adding the metadata to the template must be repeated if a new template is uploaded.
Auto-populating Word documents
Now that all the elements are in place, metadata is added automatically to the Word template as follows.
First, open the document library. Click on ‘+ New’ and choose the document set content type (e.g., Client Folder).
Fill in the metadata elements required, then click ‘Save’.
Now, open the document set/folder, and click on the document content type (e.g., ‘ClientAgreement’). This will open the Word template.
Don’t change the document, just click File – Save As. This should display the list of SharePoint sites on the right. Give the document a name, then choose the relevant site (e.g., ‘Client Agreements’), then the relevant library and document set/folder and press save. This is the most complicated part of the process once it is set up.
The template document, with all the metadata elements, should now appear in the document library under the document set/folder. As it is a Word document, it is possible to ask someone to sign on glass, then ‘save as’ a PDF.
If the document requires a sensitivity label and these are enabled, this can be added directly from the Word document menu when it is created.
Other relevant documents (e.g., emails) could be added to the same folder, including drag and drop from Outlook or other locations when the library is synced to File Explorer.
Note that the metadata elements that were added to the document in this way form part of properties of the document. These properties, which will remain with the document if it downloaded or attached to an email, can be viewed from the ‘File – Info’ section of the Word document.
The disabling of SharePoint 2010 workflows in SharePoint Designer has removed the ability to auto-create a document in SharePoint with the required metadata. However, it is not particularly difficult to set up a site, with two custom content types, one of which contains a document template, to achieve the same outcome without any workflow at all. The only complicating factor is the requirement to save the document template back to the library – without making any changes at all.
Updated 10 November 2019 with additional details based on feedback received.
As part of the implementation of Office 365, organisations may consider migrating content from network file shares (NFS) to SharePoint Online (SPO).
Closing down file shares and moving all the content to a system that can manage records has been an objective for many years. SharePoint can help organisations achieve this goal but it needs to be planned well, and some compromises may be required along the way. For example, you may end up migrating most of the content on drives ‘as is’ – but at least you can subject the content to recordkeeping controls including disposal/disposition.
This post describes the main activities or outcomes associated with migration projects.
For the purpose of this post, it is assumed that:
The organisation has established a change program for Office 365 including SharePoint Online, and that end-users will be familiar with using SharePoint by the time the file shares are migrated.
The implementation of Office 365 will includes a sub-project specifically to migrate the file shares to SharePoint Online.
On-premise Exchange mailboxes and personal ‘home’ drives will be migrated as a separate sub-project but likely at around the same time.
Project artifacts and outcomes
The primary project activities, artifacts or outcomes may include the following:
Detailed design process flow summary.
Initial description and analysis of the network file shares that may be migrated.
Creation of a ‘decision’ tree to help decide what will be migrated.
Initial and ongoing consultation with the business areas that ‘own’ parts of the file shares.
Creation of a SharePoint site destination architecture model.
Training and awareness sessions for end-users, if required.
Development of a run sheet for every migration batch.
Migration and application of retention policies, as required
End-user support and guidance post-migration.
The activities involved in producing these outcomes are described below.
Detailed design process flow summary
A detailed design process flow diagram provides an overview summary of what is expected to happen before, during and after the migration, as described below.
Initial description and analysis of the network file shares
The first (and perhaps obvious) stage of any migration project is to document what is on the network file shares – the various mapped drives on file servers. This exercise – which may be more difficult than it looks – provides the raw information needed to make decisions.
Only file servers that are used to store content are examined; servers that house line of business applications, even if they stored documents, are usually excluded from this exercise. However, at some point there may or will be a requirement to examine these drives as well when the application is upgraded and/or migrated to another system.
Documenting the network file share environment requires full access to the servers and drives and the ability to export the raw data in readable (likely csv) format. This means that the work will be done either by a network administrator and/or using a third-party tool that has the access required. If a third-party tool is used, care needs to be taken in terms of who will be able to access the information on the drive.
The documentation should include the following for each network file share (one column per dot point):
The server names and folder paths only down to a level that can be ‘mapped’ (in terms of ownership and use) to business areas (divisions/departments, or perhaps the next business level down if required), or defunct business areas.
Note that terms like ‘P drive’ or ‘N drive’ are irrelevant to this exercise, except to record it as the ‘public’ name that the business area uses. Multiple business areas may actually use the same server but refer to ‘their’ part of the drive by the drive letter allocated to them.
This detail will be confirmed with the business area in the next stage. It may also include paths that were used by now defunct business areas.
The business area known to ‘own’ the path content, or the defunct business area.
A list of all folders and file names (which will give the file types). This part will obviously take up the most room in the csv output file – if it fits at all. Another output format may be required.
Who (AD Security Group or by name) has access to the folders – inherited or unique.
Does the file share contain or appear to contain any obvious sensitive or personal information (usually obvious from folder or file names).
This information can, potentially, be used as a snapshot of what was on the drives before it was migrated, both to cross-check the migration process but also as an archival record.
At this stage, identifying duplicate (or potentially duplicate) documents is not a high priority as it could get very complicated to work out which version was the ‘master’ or original (even with last modified dates).
Based on this initial documentation, it should be possible to determine:
The total size of all the content to be migrated.
The overall size will impact on how long it may take to migrate the content, and also how long end-users may be affected during the migration (as you don’t want anyone modifying content on the drive after it has been migrated).
The size may also determine the method or application used to migrate the content. Keep in mind that the integrity of the metadata on each record (especially date created, date modified, author) needs to be maintained.
The file types that may not be migrated.
For example, downloaded movies or personal photographs. There may also be some executable (exe) files and other unusual content.
Activity levels (by date modified), which will help determine if content will be migrated as ‘active’ or ‘inactive’ content.
Any security access controls that may need to be considered as part of the migration.
Determining if these controls exists can be time-consuming and may impact on how the migration is undertaken. For example, if a part of the NFS is used by multiple parts of the business or business area and, for whatever reason, access controls are used to restrict access, you could either (a) attempt to replicate the access controls in SharePoint or (b) split the migration into multiple SharePoint sites so that only those people who can see the content on the NFS will see the same content on the SharePoint site.
Creation of a decision tree
After the drive content is reviewed and documented, a basic decision tree will help determine what – in theory – will be migrated where.
Note that these initial high-level decision trees are very similar in most organisations. Each organisation should create its own based on the content uncovered and things like records retention requirements.
The following is an example diagram of an initial decision tree.
Is it worth identifying the ROT?
The effort involved in positively identifying redundant, outdated or trivial (ROT) content – apart from the obvious downloaded movies and user photographs – can become a major exercise and can bog down the project very quickly.
It may be a more efficient use of resources to focus on what needs to be migrated. That is, identifying live, active content (even if some of it is ‘ROT’) moved into SharePoint sites and then applying retention policies to that content.
Inactive content could be moved ‘as is’ to archive, read only, SharePoint sites with Office 365 retention policies applied on libraries in those locations.
Anything that isn’t to be migrated (whether it is ROT or just very old and not required) can be left on the drives and deleted when the NFS is decommissioned.
Consultation with business areas (initial and ongoing)
Once the details of each network file share is established and the decision tree has been agreed, business areas need to be engaged to gain a broad understanding of the following:
Confirmation of the details found in the analysis, and modification of the listing if it is incorrect.
Areas of the network drive/s that are active (sometimes in parallel with an existing SharePoint site) and inactive.
Areas of the drive that do not need to be kept.
Areas of the drive that don’t have an owner.
The original listing for the drives is then updated.
Consultation with business areas should be an ongoing activity (e.g., regular meetings). It should result in a much more accurate map of the network drive areas and identification of what will go where – to existing or new SharePoint sites, or simply deleted.
A note of caution – the devil is in the detail of network file shares. The lower down the folder structure you go to try to identify duplicates or ROT for example, the more work it may be to migrate them. However counter-intuitive it may seem, it may be much easier in many instances to migrate as much as content possible ‘as is’ and then ‘aim to’ do a clean up post-migration. This can include the application of O365 retention policies to sites or libraries.
Development of a SharePoint destination architecture
Try to avoid mixing active and inactive ‘messy’ NFS content on the same SharePoint site. The benefits of not mixing the two include:
Ensuring that active sites have a cleaner structure.
Fewer document libraries on each site.
More useful search results.
Messy legacy folder structures are kept out of site on inactive sites.
Inactive content is subject to retention and disposal actions away from active sites.
Better end-user engagement.
Try not to confuse end-users too much by changing the way their content re-appears in SharePoint. End-users generally know where the content is and can navigate there reasonably easily.
The following diagram is an example that shows how NFS folder paths might map to SharePoint sites and document libraries.
The following two Microsoft principles are worth noting here:
An effective site collection consists of groups of individuals and teams that share common goals. A site collection is one where the content is useful to those sharing the site.
This means having as many sites (and libraries) as the business areas need. There is likely to be a one-to-one relationship between the NFS drive/folder path used by ‘lower-level’ business units and SharePoint site document libraries.
A secure site is one that is open to those who need the information, but where information is blocked from those who should not see it.
That is, instead of creating many sites and restricting access to each one, make all sites open (at least read only Visitor access) and restrict only what needs to be restricted on the site. There will of course be some exceptions to this rule, but open access should be the norm.
Post-migration, end-users will want to be able to re-find the information they need easily, and create and share new content in the same (or similar) way as before. If the migrated content is more complicated, users may reject it.
SharePoint training and awareness sessions
A common question is how much training do end-users need to use SharePoint. The simple answer should be – not much. It should be as easy to use as DropBox, or Facebook and no-one ever got training for those applications.
When you migrate content from a network file share to SharePoint, end-users need to know how to use the new environment from the moment they log in. Rather than formal training, the majority of end-users may only need simple guidance on how to find the content that used to be on the drive and how to add and edit new content.
Communicate regularly with the business area and ensure they know who to contact.
Identify early adopters (who can support the migration) and potential late adopters (who may need additional help).
As much as possible, use the same names for the site URL and document libraries as on the network file share folders.
Be conscious of, and let users know about, limits with URL paths. Or, to put it another way, communicate clearly the need to avoid complex folder hierarchies.
Don’t move content around unnecessarily when migrating from network file shares to SharePoint.
Don’t introduce new concepts when they didn’t exist before. For example, don’t add content types (e.g., to choose from when a new document is created) or mandatory columns to document libraries when these did not previously exist. Restrict these to new document libraries.
Be aware that Office 365 retention policies prevent deletion from document libraries. If they could delete before, end-users might react negatively to this ‘new’ restriction.
The following are some points that can be used in conversations and meetings with business areas and end-users to help them ease into the SharePoint world.
After an initial and very short overview (including perhaps and explanation of how ‘this’ Office 365 differs from their home version), ask end-users to log on to the Office 365 log on screen. Outlook, Word, Excel and PowerPoint should all be visible, as is OneDrive and SharePoint.
As the focus is on SharePoint, open that application and talk them through the portal. Introduce them to the parts of the portal, including search.
As their drives will be migrated to SharePoint, use an example site for them to play with and have a look around. Note – ensure that they have a clickable list of the new sites that will contain the migrated drive content.
Demonstrate how to create a new document and save it, noting how SharePoint and OneDrive are the new default options.
Demonstrate the Sync option in a document library, and the options available (such as versions) when you click the three dot ellipsis menu. Demonstrate how to Share instead of attach a document.
Open MS Teams and demonstrate how SharePoint works in that application.
Migration run sheet
It is worth running a number of ‘test migrations’ to test SharePoint sites, to understand the process and outcomes.
A migration run sheet should exist for every batch that is migrated to SharePoint Online. The run sheet can be created in a spreadsheet with following seven dot points as column headings.
Action (see the dot point steps below – Pre-migration to Post-migration)
Event / Task (e.g., what will actually happen in a few words)
Communication method if required (e.g., ‘By email’)
Responsible (who is responsible for this action)
Decision Point (up to what point can the task be cancelled)
Testing Required (mostly no)
Communications, if required (to whom, in what form)
The following is a list of the actions that will take place for each migration batch.
Pre migration (1 week or more)
Identify NFS source path/paths and any specific permissions that are required.
Pre-migration analysis of NFS
Fix or address any NFS issues
Pre migration (72 hours/3 days in advance)
Communicate change and migration date and time to affected business area
Keep in mind that very large migrations may take several days to complete.
Communicate change to Change Manager and Service Desk
Pre migration (24 hours in advance)
Re-communicate change details to affected business area
Establish SPO destination site or identify existing target SPO site. If the latter, confirm destination site URL.
Establish or change permissions on the SPO destination site, as required.
A migration tool (or a third-party with a migration tool) will be needed to move content from the NFS drives/folder paths in batches to SharePoint document libraries.
The time it takes to migrate the content will depend on total size and volume of files to be migrated, and the network bandwidth. The migration process may completely quickly or take several days. Files in the source NFS folder path will continue to be accessible and editable but this should be discouraged as modified content may not be migrated.
The following steps are required after each migration batch has completed.
Test migration outcome
Testing can include object numbers and total size comparisons.
Note that some files may not migrate, for whatever reason and this can result in having to re-do the migration. The reasons for the non-migration will need to be identified quickly before the migration is confirmed. In some cases it may be possible to re-migrate just those files after the problem is fixed.
Decision Gate – OK or NOT.
If OK, confirm with business area and ensure they are aware of the SPO site/s URL.
If NOT, alert business area (drive remains accessible)
Make NFS source path/s read-only.
Delete NFS (at an agreed date)
Applying retention policies
Retention policies can be applied after the content has been migrated. Retention requirements will depend on the content that is migrated and business needs:
Active content migrated to new libraries or existing SharePoint sites.
May not initially have any retention policy applied (as it could prevent the deletion of content in the libraries), or may have retention policies applied only to specific libraries.
May also have an O365 site policy, but this would need to be confirmed with the business area.
Most inactive content migrated to ‘archive’ SharePoint sites.
May have site-level Office 365 policies applied, based on when the content was created or modified
May have other policies auto-applied (including based on auto-classification (subject to licencing)).
Some inactive content migrated to ‘archive’ SharePoint sites.
May have specific and different library-level policies applied, with a disposition review.
Post migration support
It almost goes without saying that you should provide appropriate support for end-users if you change the way they are expected to work. It requires empathy, patience and understanding, while focussed on a speedy resolution of any issues that arise.
Effective support for end-users has three main elements:
The ability to show empathy and be patient. For many end-users, this will be the first time they are using SharePoint.
‘Mini-training’ and self-empowerment, by talking the end-user through the issue and potential resolutions and/or sending follow up help and guidance or links for further help. This will help them to learn and pass the learning on to others.
Quick resolutions, to allow the end-user to get on with their job as soon as possible. If a quick resolution is not possible, regular updates and options to allow the end-user to continue working.
Many (if not most) organisations receive and respond to correspondence in the form of letters or emails. They may also respond to social media messages.
Correspondence may be managed in different ways. For example, some organisations may have a dedicated business correspondence unit while in others individuals or business areas may respond.
This post describes how SharePoint Online with MS Teams and MS Flow could be used to manage letters and emails that require a formal ‘organisational’ response. It does not look at managing responses to social media messages.
The management of correspondence that requires a formal response generally involves the following elements:
Somewhere to store the incoming correspondence (including scanned paper documents) and responses.
A description of the correspondence (naming conventions and/or metadata).
Some form of workflow, including emails, used to notify people of actions they must take.
Various methods use to report on and track progress, and closure/completion rates.
SharePoint includes all these elements. The requirement for workflow can be met using emails, the built-in workflows, workflows built in SharePoint Designer, or MS Flow. MS Teams provides the opportunity to add additional value to the communication response process by allowing messaging, co-authoring of responses, approval processes, and more.
A model correspondence system
The model correspondence system described below is based on a correspondence management system that I developed for a large public sector organisation a few years ago, but using a different system.
The core elements of the model system are:
An Office 365 (O365) Group. The O365 Group has an email address and an associated SharePoint site.
The Office 365 Group’s SharePoint site.
A Team in MS Teams, connected to the O365 Group.
Office 365 Group
The O365 Group should have a name that reflects its purpose and makes it easy to recognise as it also becomes the email address name.
For example ‘GRP_Correspondence’ – the prefix ‘GRP’ is used as it defines it as an O365 Group, as opposed to a Security Group (SG) or Distribution List (DL) created under the same ‘Groups’ section in the O365 Admin portal.
The Members of the Group should be the group of people primarily responsible for managing responses to the correspondence. Other people can be invited to the SharePoint site directly (without having access to the Group’s Team) as Members or Visitors.
The O365 Group’s SharePoint site has the same name in the URL – GRP_Correspondence.
The SPO site should have at least one document library with an obvious name, for example ‘Correspondence 2019’.
If there is concern about potentially long URL lengths, the library name could be reduced to, say ‘Corro2019’; the display name can still be ‘Correspondence 2019’.
Consideration might also be given to having a ‘drop off library’ in the Group site where anyone can save correspondence that may require a response.
The metadata required in the document library will vary between organisations. The core default metadata (for every new document library) already includes all the following:
Document ID (when enabled as a Site Collection feature, which is recommended)
Shared with details
Check In Comment
Checked Out To
Folder Child Count
Item Child Count (shows how many documents in a folder)
Retention label Applied
Additional metadata may include any of the following:
Sender (free text or potentially drop down choice)
Response Due date (Date)
Urgency (Choice – Routine, Urgent)
Description (free text)
Reply Status (Choice – Not required, Draft, Approved, Sent, Cancelled)
Response Type (Not required, By email, By letter)
Approved by (internal Active Directory name)
Date Completed (Date) < This date should correspond to the date on any reply.
Once the library has been created, content can be added.
Paper correspondence can be scanned and saved to the library. Many MFD (printers) now have the ability to save directly to SharePoint, perhaps to a drop-off library for categorisation and review before being moved to the primary library. The original paper can then be boxed and destroyed after a given period (3 – 6 months).
Emails can (and should) be copied from the Group’s inbox to the library (see screenshot below). To do this, sync the library to File Explorer and drag and drop.
A new folder can be created for each new correspondence, as indicated in the screenshot below:
Both the correspondence and the folders where it is stored should be named according to naming conventions. Naming conventions can also be used instead of folders, to indicate the connection between the original and the reply. My preference is to use folders to group the original and the reply, and also because they are ubiquitous in the digital workplace.
Suggested naming conventions:
Replies: REPLY-Surname-Subject-Date-DRAFT or FINAL (the final version may be ‘signed’ and then saved as a PDF)
Always exclude spaces in names, as these will be replaced by ‘%20’ in the URL path.
If one or more standard templates are required for replies:
Create the new Site Content Type in the Site Content Types area.
Enable the management of content types in the ‘Advanced’ section of the document library settings.
Add the new Site CT to the library
Add or edit the Word template to be used by clicking on the Content Type in the ‘Content Types’ section, then clicking on ‘Advanced settings’ and ‘upload’. The template can continue to be modified as required directly from this area and ensures consistency in replies.
The new Content Type will appear in the ‘New’ menu in the document library but not in the MS Teams ‘New’ dialogue (see below). This means that standard replies using a template can only (currently) be created via the SharePoint library. Drafts, however, could be created with a standard document template first (same with emails, see below).
The Team in MS Teams can be connected to the O365 Group (it’s a one to one relationship, you cannot connected multiple O365 Groups with a single MS Team).
The Team can have multiple channels, and the SPO document library can be presented in a channel. For example, the channel named ‘Correspondence 2019’ includes the ‘Correspondence 2019’ document library as a tab, as shown below.
The use of Teams means allows drafts to be co-authored and chats to take place about the correspondence and other matters.
If the SharePoint site includes an ‘incoming correspondence’ drop off library, the Team could have a channel with that library as a tab. The channel could then be used to review and decide on what to do with the correspondence.
Routing incoming correspondence for a reply
Once the correspondence is saved in a SharePoint document library, a decision must be taken by someone whether a reply is required.
If no reply is required, this can be reflected in the metadata (‘No reply required’).
If a reply is required, the correspondence must be ‘sent’ or otherwise made available (see below) to someone to draft a reply. This can be a simple or complex process.
In some cases, a standard reply may be possible. The SharePoint site should contain at least one library that contains examples of standard replies to certain types of, or common, questions.
The easiest way to ‘send’ someone the correspondence for action is to use the ‘Share’ option on the folder where the incoming correspondence is stored. As the same ‘Share’ option appears in File Explorer when the library is synced, the sharing process can also be managed from File Explorer.
This means that the recipient only has to click on the folder link in the email they receive to see the content of the folder. As they have access to the folder, they can then use the ‘New’ option to create a draft reply, including to an email.
Once the draft has been finalised, it can also be sent via the ‘Share’ option. Alternatively, an alert on the library will notify anyone who needs to be notified that a change has been made to the library.
Routing using MS Flow
A more complex routing process may be required if the draft requires several steps, for example:
Send to someone to create the draft reply.
Draft reply gets sent to someone else for approval.
If not approved, goes back to the person who created it. (This can loop several times)
If approved, a message is sent to the person who needs to finalise it
Reply is finalised, metadata is updated, and reply sent.
All SharePoint Online libraries include an in-built Flow workflow ‘Request Sign Off’. When a document is selected and the ‘Request Sign Off’ option is selected the first time, the person must select the option to ‘Create Flow’. The ‘Run Flow’ dialogue then appears, requiring someone to be identified as the Approver and a Message to be included. The approver can be anyone in the organisation, for example the person’s manager.
The Approver receives an email, allowing them to approve or reject, and add a comment. If rejected, the ‘Sign Off Status’ column in the SharePoint library is updated to ‘Rejected’, and the sender receives a message to advice them that approval was rejected.
If approved, the sender receives an email to notify them, and the ‘Sign off status’ column changes to ‘Approved’.
Once the reply has been approved, it can be finalised and sent to the person who wrote the correspondence. All versions of the draft reply are kept in the same folder, along with the final.
Once approved, any email reply could be sent directly to the correspondent from the O365 Group’s mailbox.
Metadata from the document library can be exported to Excel, or to business intelligence systems (or PowerBI) for analysis and reporting purposes.
Correspondence can be managed in SharePoint, with MS Teams used to provide additional co-authoring and ‘chat’ options for the team, and MS Flow used for more complex approval requirements.