Posted in Artificial Intelligence, Classification, Electronic records, Information Management, Microsoft Viva, Products and applications, Records management

Are auto-generated topic cards the future for aggregations of records?

Humans have natural instinct for grouping, classification and categorisation of things. It helps us find what we are looking for and gives us a sense of satisfaction, whether it be household items, computer storage, or much broader social and population groupings.

Humans have created and kept records ever since we developed a way to record them, on stones, clay shards, papyrus, bamboo sheets, velum, paper and various other means. Multiple records were aggregated in ways that made sense to the people who created or kept them and wanted to find them again.

The introduction of computers at work from the late 1980s/early 1990s began the decline of traditional ways of aggregating records about a particular subject together in a physical ‘file’, although that practice has persisted to the present day because it was and still is easier to refer to. Lawyers (or more often the legal clerks) still attend digital courtrooms armed with printed copies of (usually digital) evidence and other materials for this reason.

Lawyers off to court – Image credit Sven Vik (NYC TV News Videographer)

The ‘problem’ of digital aggregations

While physical files provided the ability to store anything (printable) about a given subject in the one location, digital ‘files’ (or aggregations) suffered from the fact that emails and other content are created or stored in completely different locations.

The only way to keep emails together with other content about the same subject was for end-users to copy them to a network file share folder location or a digital recordkeeping system. In almost every case, the original email remained in the mailbox where it might still have an active life. Some email mailboxes became a primary (or alternative) storage location for both emails and attachments (as did some desktops!).

Keeping all digital records about a given subject in a single aggregation was never an easy task. It was never possible to be sure that everything was captured because it relied on end-users.

The email mailbox – SharePoint conundrum

In the same way that organisations decided to store copies of emails in network file shares or EDRM system, it was easy to see SharePoint as the replacement for both.

But Microsoft have never made it easy to ‘natively’ copy an email from Outlook to SharePoint. There isn’t even a download option for emails. Emails can be dragged and dropped to synced document libraries, and various third-party products exist, but the process usually relies on end-users (a) to copy the emails and (b) to copy them consistently. Neither of these can be guaranteed.

And, of course, the records created and captured in Microsoft 365 is not just in Outlook mailboxes and SharePoint. A number of other apps create content that could records (for example Yammer conversations, Teams chats, calendar entries, Planner tasks, even Whiteboard diagrams). Few of these records can be saved to SharePoint.

So, are digital aggregations impossible?

There is nothing stopping organisations doing whatever they can or want to group related records together. In Microsoft 365, the most logical way to do this is in SharePoint document libraries (the ‘Files’ tab in Teams channels). An entire SharePoint site (the ‘Files’ tab in MS Teams channels) provides a form of meta-grouping; that is, multiple document libraries grouped by the SharePoint site/Team.

But if we stand back for a moment, to look at the (Microsoft 365) forest, what we see is not just individual trees (SharePoint sites, Exchange mailboxes and so on). Just as in a forest the roots of all the trees connect via mycorrhiza networks, sometimes known as ‘wood wide webs’, something similar happens in Microsoft 365 (and many other online systems, including Facebook).

Trees networking

The equivalent of networks in these systems are the ‘graphs’.

Like other graphs, the Microsoft Graph draws on all the rich data created and stored by end-users, in this case across the Microsoft 365 ecosystem – our corporate relationships, who we connect with and how, what we are communicating or writing, what we like, the way we use our time and so on. The graph learns what is popular or trending and makes suggestions (while respecting permissions) as to what we might want to see or know about.

Project Alexandria and Viva

According to a post in the Microsoft Research blog published in April 2021 and titled ‘Alexandria in Microsoft Viva Topics: from big data to big knowledge‘, Project Alexandria is ‘a research project within Microsoft Research Cambridge dedicated to discovering entities, or topics of information, and their associated properties from unstructured documents’.

The blog post also noted that ‘Alexandria technology plays a central role in the recently announced Microsoft Viva Topics, an AI product that automatically organizes large amounts of content and expertise, making it easier for people to find information and act on it’.

The Alexandria pipeline – from unstructured text to structured knowledge (From the blog post above)

The outcomes sound similar to traditional ‘manually’ created aggregations, although they don’t replace them. In fact, the more that content is manually curated, the more likely that Viva Topics can accurately connect them and other related content that might otherwise be missed.

While Viva Topics might appear to primarily focussed on supporting knowledge management outcomes and is currently limited to content stored in SharePoint, the technology has potential implications for records management. In particular, the age-old issue of how to find all information about a given subject (or know that a pre-defined aggregation contains all relevant information).

Viva Topic cards

As noted already, there is nothing stopping organisations from creating aggregations in ways that make sense to them and their end-users. SharePoint document libraries are the most logical form of aggregation that also happen to allow complex metadata, versioning and other features typically associated with EDRM systems. SharePoint document libraries are just one of several ways that content may be aggregated; Exchange mailboxes are another.

But, in most organisations, potentially relevant information AND records is frequently hidden from view in personal mailboxes and OneDrive accounts, in Teams chats, and in other applications (e.g., Planner). Viva Topics has the potential to leverage this information.

Once set up (as described in Set up Microsoft Viva Topics) , Microsoft Viva begins to work its magic, discovering topics. An example of a discovered topic (from ‘Manage topics at scale in Microsoft Viva Topics‘ is shown below.

While Topics are still limited to SharePoint content and people, there is potential to extend this model even further by including details about emails, chat messages or other content across the Microsoft 365 ecosystem – even if that information cannot be seen. For example:

  • Topic Name
  • Suggested people (perhaps grouped by AD manager or business area)
  • Suggested files and pages (you can see)
  • Authors of (n number of) emails that are related to the topic with an indication of volume over given periods (e.g., ‘251 emails in the past 6 months’) or a graphic representing this activity
  • Names of Teams that contain (n number of) chat messages related to the topic.
  • Participants in Teams 1:1 chats that contain (n number of) messages related to the topic.
  • Volume and date range of other related content (e.g., Tasks, Whiteboards, Forms, Yammer conversations).

Could Topic cards be the new aggregations?

Topic cards have the potential to resolve the age-old dilemma of digital aggregations, but they are unlikely to replace pre-defined ways to aggregate records including by copying emails to SharePoint document libraries. Those older methods will continue to exist for a long time.

But more importantly, they have the potential to draw out or highlight content that would otherwise be hidden from view – even if that content remains inaccessible.

When configured, Viva Topics already appear in search results, enhancing search outcomes.

It is only a matter of time before the probabilistic programming techniques of Project Alexandria, with expert human curation, begins to provide the type of high precision knowledge base construction for all relevant content about a given subject, first described by Microsoft researchers in May 2019.

Perhaps they may even support or link with retention and disposal processes, highlighting records due for disposal within a given period or even preventing their premature disposal.

Posted in Artificial Intelligence, EDRMS, Electronic records, Exchange Online, Information Management, Microsoft 365, Microsoft Teams, Records management, SharePoint Online

Different approaches for managing records with Microsoft 365

The COVID pandemic from early 2020 led to the requirement for many employees to work from home (WFH). IT Departments scrambled to enable this capability, many making use of Microsoft (MS) Teams that was already bundled with their Microsoft 365 licences.

The rapid enabling and uptake (rather than an actual ‘implementation’) of MS Teams was more often than not achieved without much consideration for recordkeeping requirements or an overall plan for using Microsoft 365.

MS Teams became popular quickly, increasing from around 30 million active users daily in early 2020 to around 250 million by mid 2021 (Source: ZDNet quoting Microsoft latest results). End-users could chat with each other and with external people (and on their phones too!), have video meetings, create new teams with channels and private channels, share and collaborate on content via the ‘Files’ tab in Teams, create and manage tasks, and more. They also continued to use email.

Anecdotal evidence suggests that the capture of records to on-premise electronic document and records management system (EDRMS) declined from early 2020. One reason suggested for this was that it was too hard to save some cloud records such as Teams chats or content from the Files tab to an on-premise system. Alternative approaches for managing records with Microsoft 365 began to evolve.

This post discusses four approaches to managing records in Microsoft 365, summarised in the diagram below.

Which approach have you taken? Answer my (anonymous) short survey here (Microsoft Forms).

Approach 1 – EDRMS + key Microsoft 365 applications to create and capture

Approach 1 – EDRMS plus the main Microsoft 365 applications

This model has two elements:

  • Retaining an existing centralised recordkeeping system (the EDRMS) for the storage of records.
  • Using email, Teams, SharePoint or OneDrive to create or capture records to be copied to the EDRMS, and leaving other content (in theory non-records) ‘in place’.

The main positive aspect of this model is that records are (in theory) captured and managed in the EDRMS with all the traditional recordkeeping options. Some leading EDRMS vendors now offer solutions that integrate with Microsoft 365 and make it easier to capture records from Microsoft 365. But the model is still based on a centralised recordkeeping system and the requirement for end-users to copy content identified as records.

The main negative aspects of this model include the following points:

  • End-users still have to identify and copy records to the EDRMS.
  • Not all records created or captured in Microsoft 365 can be copied to the EDRMS.
  • Additional products or add-ons may be required to enable the copying.
  • The record is copied to the EDRMS, not moved, so remains in place with no controls.
  • Records that remain stored in Microsoft 365 applications may not be subject to the same degree of recordkeeping controls available in the EDRMS. Unless they acquire a third-party product (see next approach) to overcome this problem (which is unlikely for cost reasons), organisations must use the out of the box recordkeeping capability in Microsoft 365. This capability may not meet all requirements for keeping records if not properly configured.
  • There is a real risk that some records that remain in Microsoft 365 may be lost, especially if settings allow content to be deleted and there is no retention policy or backup.
  • EDRM system admins and records managers will need to learn a lot more about Microsoft 365.
  • The unified logs in Microsoft 365 only retain the details for 3 months (E3) or 12 months (E5) – although SharePoint’s versioning history can provide a lot of ‘modified’ event metadata for the life of the document (up to the the maximum number of versions allowed). (Update: Microsoft 365 customers can retain the audit log for up to 10 years with an add-on license. Many export audit data to a SEIM such as Azure Sentinel where they can retain the log for as long as they want.)

On a positive note, however, Microsoft 365 includes a wide range of search, audit, monitoring and reporting tools, as well as security and protection controls, that improve the ability for records managers to find, manage and protect records (or potential records) in Exchange mailboxes, MS Teams chats and posts, SharePoint sites and OneDrive accounts AND put that content on a legal hold. So, as long as those options are enabled, the risk of losing records is reduced.

Approach 2 – Third-party application + Microsoft 365 applications for creation, capture and storage

Approach 2 – Third-party product plus Microsoft 365

A number of Microsoft partners have developed applications to manage records in Microsoft 365. Several have been available for a decade or more, originally designed to manage records primarily in on-premise SharePoint environments.

Most of these third-party applications were developed to comply with the same recordkeeping standards used by EDRMS vendors. These applications are generally either:

  • Replacements for EDRM systems (often requiring migration from the EDRMS).
  • New implementations where there was no EDRMS beforehand.

It is not common to see both an EDRMS and one of these third-party products being used together, because of licensing cost reasons.

The main positive aspect of using a third-party dedicated application is that records created or captured in Microsoft 365 can be stay there and be managed according to recordkeeping requirements. Some of these applications are invisible to end-users, making them even more attractive.

The main potential negative aspect of using a third-party application, which is the same for any other vendor product, is that it creates a dependency on the vendor to maintain the product. Microsoft 365 continues to evolve and any third-party application must keep up with these changes. Two questions might be asked:

  • Will this dependency become a ‘tech debt’ liability in the future, if a ‘better’ option comes along?
  • How hard will it be to transfer to a different vendor in the future? Generally speaking this is less likely if the vendor is an established Microsoft partner, but the question should still be asked. For example, many organisations decided to use the Google suite of products but have now decided to use Microsoft 365.

Organisations seeking to implement third-party applications to manage records in Microsoft 365 should have a very detailed understanding of the underlying Microsoft 365 environment beforehand and the impact the third-party application might have on this environment. Some of the considerations might include:

  • The requirement to provide the third-party vendor with admin (including global admin) access to the Microsoft 365 tenant. Is this a security concern?
  • The location of records – in some cases, third-party vendors may use, move or back up content to one of their Microsoft 365 tenants. Is this a security concern? How can you monitor activity on your content if it’s not in your tenant?
  • The use of the central Term Store or Content Types to support the application. Will this create a dependency or make it harder for people to work, for example by requiring end-users to select Content Types or add metadata.
  • Changes to SharePoint settings and architecture, including the addition of hidden columns. Will these changes be consistent with your own architecture model?
  • How and where event metadata (audit logs) will be captured and managed.
  • How retention outcomes will be managed.

Approach 3 – One or more Microsoft 365 applications are the default ‘recordkeeping systems’ (no EDRMS or other application)

Approach 3 – Individual systems highlighted are the ‘recordkeeping’ systems

This approach focuses on the applications where most records are likely to be created or captured in Microsoft 365 – Exchange mailboxes, MS Teams, SharePoint, and OneDrive for Business – and therefore considers other content created and/or stored in other Microsoft 365 applications (e.g., Yammer, Forms, Planner/Tasks, etc) as being non-records.

There are several variations on this model including the following:

  • Outlook and Teams are the primary ‘recordkeeping systems’ as they are the two applications that are most used. Teams has been positioned as the primary interface for both SharePoint and OneDrive (via the ‘Files’ tab). The ability to also access both SharePoint and OneDrive from File Explorer via the sync option makes it even less likely that SharePoint or OneDrive will be accessed by end-users.
  • All four applications are the recordkeeping systems, using the various controls and settings available in the various admin portals, as well as the Compliance admin portal for retention policies.
  • SharePoint is the primary recordkeeping system, configured to mimic EDRMS capability. In this case, end-users would be expected to copy emails from Outlook or records from OneDrive, similar to the way they would have to do this for an EDRMS. Various controls and settings, such as ‘back end’ retention policies, might be applied to the other main applications to ensure that any records in those systems (such as Teams chats or emails) are not destroyed before a given period.

The main positive aspects of this approach are (a) simplicity and (b) cost savings, mostly by not having to purchase an EDRMS or third-party application.

However, these potential positives should not compromise the requirement for both IT and records management to have a very good understanding of, detailed approach to, and governance for, managing records in Microsoft 365. In other words, simply saying that one or more of these four applications is the recordkeeping system is not sufficient; additional work is required to ensure that records stored in them are managed appropriately.

There are several potential negative aspects of this model:

  • With the exception of SharePoint, none of the other three systems can be configured to manage records based on standards used for EDRM systems. Given that SharePoint has been positioned behind the Teams user interface, and SharePoint document libraries can be synced via Teams to File Explorer, any recordkeeping functionality configured in SharePoint should in theory be accessible or useable via Teams and possibly also File Explorer, but this is mostly not the case. So, SharePoint on its own, accessed via the browser only, is not really an option. Additionally, without effective controls, the Files (SharePoint) element of Teams has the potential to become the future equivalent of legacy network file shares full of redundant, outdated and trivial content.
  • If only one or two systems are considered to be the only recordkeeping systems, there is a risk that records may not be saved and/or could be lost, especially if end-users can delete records and there is no back up option.
  • Managing records in this way requires both access to and a very good understanding of the applications designated to be the recordkeeping systems by both IT and records managers.
  • Retention policies (either the base level information governance or more expensive records management) may not be adequate, in terms of both application and coverage, and retention outcome management.
  • Exporting the records to another system or transferring them to another organisation, could become a complex task.
  • Accessing audit logs over a long period (see first approach, last dot point, above).

Approach 4 – All of Microsoft 365 is the recordkeeping system

Approach 4 – All of Microsoft 365 is the recordkeeping system

This approach is similar to the previous one except that it takes a broader approach and requires a degree of ‘letting go’ of the standards used by EDRMS systems (and third-party products). It is also the Microsoft default.

The approach assumes that records may be created or captured anywhere in Microsoft 365, saved to Microsoft 365 via archive connectors, or accessed (subject to access controls) via search connectors. Records are managed ‘in place’, meaning wherever they are created or captured, using a range of tools already available in Microsoft 365. Additional ‘in place’ controls allow certain items to be declared as records.

The approach requires both a very good technical understanding of the Microsoft 365 environment and effective governance by IT and records managers. If internal skills are lacking, it may also require a third-party organisation to implement the system – but based on what recordkeeping model? A reliance on a third-party to implement the recordkeeping elements has several risks (see below).

The main positives of this approach include the following:

  • Records that are created or captured in the Microsoft 365 environment remain there. There is no requirement to copy them to a separate system.
  • Some records, such as emails, can be copied to SharePoint if required.
  • The combination of Teams and SharePoint sites allows for multiple models to manage records – for example, high value records could be managed in a dedicated SharePoint site with multiple dedicated libraries and additional controls (metadata, retention, permissions etc), whereas low level records could be managed in the single ‘Documents’ library presented as the Files tab in a Team, or via File Explorer.
  • All the content (records and non-records) stored across Exchange, Teams, SharePoint/One drive can be searched (subject to roles and permissions). This allows records managers (and others such as Legal) to identify if records may be hidden in personal mailboxes or Teams chat or OneDrive accounts.
  • Minimum retention periods can be applied to all the content (not just records), ensuring that records that may be hidden in Teams chats, OneDrive accounts, or personal mailboxes, will be retained for minimum periods. This option also helps to reduce the volume of redundant, outdated and trivial content that may build up over time otherwise.
  • Retention labels can be applied, including automatically (and using machine learning), to records in mailboxes, SharePoint sites and OneDrive accounts (but not Teams chats or posts, yet).

The main negatives of this model are the same as those listed for the previous model with more focus on the need for both IT and records managers to have a very detailed understanding of and establish effective governance for the entire environment where records may be created or captured, not just the main four applications. This requires some effort to achieve and should not be understated. It is not uncommon to see IT staff with Global Admin managing the entire Microsoft 365 environment using default settings and/or records managers will little technical knowledge or appropriate access struggling to understand how the environment works and drawing on experience with EDRM systems.

Some organisations may engage third-party implementation specialists to configure and set up the environment. Organisations that decide to go down this path should ensure they have the details of this configuration and can support it in the longer run, or the environment (or parts of it) could end up becoming difficult to manage or support over time.

Approach 5 – A potential future model

Microsoft 365 includes a wide range of settings, options and capabilities that have a significant impact on the way records can and will be managed across Microsoft 365 in the future.

Microsoft 365 will continue to evolve over time, including in ways that will support how records are managed. But it is important to keep in mind that Microsoft 365, or its component applications, is not and will never be an EDRMS based on standards such as DOD 5015.2. Microsoft 365 is too complex, and the volume and type of content stored in it too large, for any part of it to be considered the ‘records management’ system.

A new approach is required for the identification and management of records. This approach may draw on existing recordkeeping standards and concepts but is likely to rely more heavily on new and evolving ways to work with information, including records.

Some of these ways have been around for a decade or so in the form of graph-based machine learning (ML), process automation, artificial intelligence (AI). Examples include Google, Facebook, LinkedIn, Netflix, Amazon, eBay and so on. These examples have one thing in common – they all take advantage of the various ‘signals’ and ‘digital exhaust’ voluntarily offered by their users to identify and present things that match your interests – jobs, friends, things to purchase, movies. Post something on Facebook or (perhaps) talk about a particular subject near your phone, and related ads will appear.

So, what is different about Microsoft 365? End-users are related to each other thanks to Active Directory, they connect and communicate with others via email or Teams, they share content, they attend meetings. All of these (and a lot more) signals feed into the underlying Graph and allow connections to be made and suggestions.

There is nothing stopping organisations setting up dedicated SharePoint sites with multiple well-named libraries to manage certain records and leaving other content and records to the world of Teams Files. But all of this information can be related based on context, including who created it, what team that person was in, who they connect with, what access do they have and so on.

Perhaps by 2035, the primary approach to records management will be relying on all the digital connections and signals, machine learning, the Graph and AI to identify all related records in context, not just the ones neatly placed in a SharePoint document library. Records may be automatically identified as important and needing stronger controls based on this context – who created, sent or received it, whether it relates to a subject that is trending (or was in the past).

Instead of just a simple pre-defined aggregation of records (which will still be a valid way to aggregate records), future aggregations will include a wider range of content, created automatically, likely presented in the form of ‘cards’.

Viva Topics is an interesting pre-cursor to this possible future model.

Viva Topics presented in Teams

The following text is from the Microsoft page ‘Alexandria in Microsoft Viva Topics: from big data to big knowledge‘:

Looking further ahead, Alexandria’s ability to extract information automatically gives us the opportunity to customize the knowledge discovery process. By automatically retrieving the set of types and properties being talked about in an organization’s documents, Alexandria can create a knowledge base with a bespoke schema exactly tailored to the needs of each organization and using the familiar language and terminology that people in the organization are used to. Read more about the proposed schema-based design in our research paper.

We are only beginning to dream of the experiences that an automatically created and updated knowledge base can enable, but it is already clear that it could transform the future of how we work. The era of big knowledge is coming sooner than you might think.

Whatever the new approach is, managing records in Microsoft 365 will require new skills on the part of information and records managers.

Posted in Artificial Intelligence, Classification, Electronic records, Information Management, Microsoft 365, Records management, Retention and disposal

Can Microsoft technology classify records better than a human?

In late 2012, IDM magazine published an article I co-authored with Umi Asma Mokhtar in Malaysia titled ‘Can technology classify records better than a human?’

The article drew on research into recent advances in technology to assist in legal discovery, known as ‘computer-assisted coding’, or ‘predictive coding’, including the following two articles:

Grossman and Cormack’s article noted that ‘a technology-assisted review process involves the interplay of humans and computers to identify the documents in a collection that are responsive to a production request, or to identify those documents that should be withheld on the basis of privilege‘. By contrast, an ‘exhaustive manual review’ required ‘one or more humans to examine each and every document in the collection, and to code them as response (or privileged) or not‘.

The article noted, somewhat gently, that ‘relevant literature suggests that manual review is far from perfect’.

Peck’s article contained similar conclusions. He also noted how computer-based coding was based on a initial ‘seed set’ of documents identified by a human; the computer then identified the properties of those documents and used that to code other similar documents. ‘As the senior reviewer continues to code more sample documents, the computer predicts the reviewer’s coding‘ (hence predictive coding).

By 2011, this new technology was challenging old methods of manual review and classification. Despite some scepticism and slow uptake (for example, see this 2015 IDM article ‘Predictive Coding – What happened to the next big thing?‘), by 2021, it had become an accepted option to support discovery, sometimes involving offshore processing for high volumes of content.

Meanwhile, in an almost unnoticed part of the technology woods, Microsoft acquired Equivio in January 2015. In its press release ‘Microsoft acquires Equivio, provider of machine learning-powered compliance solutions‘, Microsoft stated that the product:

‘… applies machine learning … enabling users to explore large, unstructured sets of data and quickly find what is relevant. It uses advanced text analytics to perform multi-dimensional analyses of data collections, intelligently sorting documents into themes, grouping near-duplicates, isolating unique data, and helping users quickly identify the documents they need. As part of this process, users train the system to identify documents relevant to a particular subject, such as a legal case or investigation. This iterative process is more accurate and cost effective than keyword searches and manual review of vast quantities of documents.’ 

It added that the product would be deployed in Office 365.

Classifying records

The concept of classification for records was defined in paragraph 7.3 of part 1 of the Australian Standard (AS) 4390, released in 1996. The standard defined classification as:

‘… the process of devising and applying schemes based on the business activities generating records, whereby they are categorised in systematic and consistent ways to facilitate their capture, retrieval, maintenance and disposal. Classification includes the determination of naming conventions, user permissions and security restrictions on records’.

The definition provided a number of examples of how the classification of business activities could act as a ‘powerful tool to assist in many of the processes involved in the management of records, resulting from those activities’. This included ‘determining appropriate retention periods for records’.

The only problem with the concept was the assumption that all records could be classified in this way, in a singular recordkeeping system. Unless they were copied to that system, emails largely escaped classification.

Fast forward to 2020

Managing all digital records according to recordkeeping standards has always been a problem. Electronic records management (ERM) systems managed the records that were copied into them, but a much higher percentage remained outside its control – in email systems, network files shares and, increasingly over the past 10 years, created and captured on host of alternative systems including third-party and social media platforms.

By the end of 2019, Microsoft had built a comprehensive single ecosystem to create, capture and manage digital content, including most of the records that would have been previously consigned to an ERMS. And then COVID appeared and working from home become common. All of a sudden (almost), it had to be possible to work online. Online meeting and collaboration systems such as Microsoft Teams took off, usually in parallel with email. Anything that required a VPN to access became a problem.

2021 – Automated classification for records (maybe)

The Microsoft 365 ecosystem generated a huge volume of new content scattered across four main workloads – Exchange/Outlook, SharePoint, OneDrive and Teams. A few other systems such as Yammer also added to the mix.

Most of this information was not subject to any form of classification in the recordkeeping sense. The Microsoft 365 platform included the ability to apply retention policies to content but there was a disconnect between classification and retention.

Microsoft announced Project Cortex at Ignite in 2019. According to the announcement, Project Cortex:

  • Uses advanced AI to deliver insights and expertise in the apps that are used every day, to harness collective knowledge and to empower people and teams to learn, upskill and innovate faster.
  • Uses AI to reason over content across teams and systems, recognizing content types, extracting important information, and automatically organizing content into shared topics like projects, products, processes and customers.
  • Creates a knowledge network based on relationships among topics, content, and people.

Project Cortex drew on technological capabilities present in Azure’s Cognitive Services and the Microsoft Graph. It is not known to what extent the Equivio product, acquired in 2015, was integrated with these solutions but, from all the available details, it appears the technology is at least connected in one way or another.

During Ignite 2020, Microsoft announced SharePoint Syntex and trainable classifiers, either of which could be deployed to classify information and apply retention rules.

Trainable classifiers

Trainable classifiers were made generally available (GA) in January 2021.

Trainable classifiers sound very similar to the predictive coding capability that appeared from 2011. However, they:

  • Use the power of Machine Learning (ML) to identify categories of information. This is achieved by creating an initial ‘seed’ of data in a SharePoint library, creating a new trainable classifier and pointing it at the seed, then reviewing the outcomes. More content is added to ensure accuracy.
  • Can be used to identify similar content in Exchange mailboxes, SharePoint sites, OneDrive for Business accounts, and Microsoft 365 Groups and apply a pre-defined retention label to that content.

In theory, this means it might be possible to identify a set of similar records – for example, financial documents – and apply the same retention label to them. The Content Explorer in the Compliance admin portal will list the records that are subject to that label.

SharePoint Syntex

SharePoint Syntex was announced at Ignite in September 2020 and made generally available in early 2021.

The original version of Syntex (as part of Project Cortex) was targeted at the ability to extract metadata from forms, a capability that has existed with various other scanning/OCR products for at least a decade. The capability that was released in early 2021 included the base metadata extraction capability as well as a broader capability to classify content and apply a retention label.

The two Syntex capabilities, described in a YouTube video from Microsoft titled ‘Step-by-Step: How to Build a Document Understanding Model using Project Cortex‘, are:

  • Classification. This capability involves the following steps: (a) Creation of (SharePoint site) Content Center; (b) Creation of a Document Understanding Model (DUM) for each ‘type’ of record; the DUM can create a new content type or point to an existing one; the DUM can also link with the retention label to be applied; (c) Creation of an initial seed of records (positives and a couple of negatives); (d) Creation of Explanations that help the model find records by phrase, proximity, or pattern (matching, e.g., dates); (e) Training; (f) Applying the model to SharePoint sites or libraries. The outcome of the classification is that matching records in the location where it is pointed are assigned to the Content Type (replacing any previous one) and tagged with a retention label (also replacing any previous one).
  • Extraction. This capability has similar steps to the classification option except that the Explanations identify what metadata is to be extracted from where (again based on phrase, proximity or pattern) to what metadata column. The outcome of extraction is that the matching records include the extracted metadata in the library columns (in addition to the Content Type and retention label).

As with trainable classifiers, Syntex uses Machine Learning to classify records, but Syntex also has the ability to extract metadata. Syntex can only classify or extract data from SharePoint libraries.

Trainable classifiers or Syntex?

Both options require the organisation to create an initial seed of content and to use Machine Learning to develop an understanding of the content, in order to classify it.

The models are similar, the primary difference is that trainable classifiers can work on content stored in email, SharePoint and OneDrive, whereas Syntex is currently restricted to SharePoint.

Predictive coding

On 18 March 2021, Microsoft announced the pending (April 2021) preview release of an enhanced predictive coding module for advanced eDiscovery in Microsoft 365.

The announcement, pointing to this roadmap item, noted that eDiscovery managers would be able to create and train relevance models within Advanced eDiscovery using as few as 50 documents, to prioritize review.

So, can Microsoft technology classify records better than humans?

In their 1999 book ‘Sorting Things Out: Classification and its Consequences‘ (MIT Press), Geoffrey Bowker and Susan Leigh Star noted that ‘to classify is human’ and that classification was ‘the sleeping beauty of information science’ and ‘the scaffolding of information infrastructures’.

But they also noted how ‘each standard and category valorizes some point or view and silences another. Standards and classifications (can) produce advantage or suffering’ (quote from review in link above).

Technology-based classification in theory is impartial. It categorises what it finds through machine learning and algorithms. But, technology-based classification requires human review of the initial and subsequent seeds. Accordingly such classification has the potential to be skewed according to the way the reviewer’s bias or predilections, the selection of one set of preferred or ‘matching’ records over another.

Ultimately, a ‘match’ is based on a scoring ‘relevancy’ algorithm. Perhaps the technology can classify better than humans, but whether the classification is accurate may depend on the human to make accurate, consistent and impartial decisions.

Either way, the manual classification of records is likely to go the same way as the manual review of legal documents for discovery.

Image source: Providence Public Library Flickr

Posted in Artificial Intelligence, EDRMS, Electronic records, Information Management, Microsoft 365, Products and applications, Records management, SharePoint Online, Solutions

A brief history of electronic document and records management systems and related standards

[NOTE – This post may be updated with additional content – last updated 2 March 2021 with information about DMS systems available in the UK in 1996]

This post provides a brief history of electronic document and records management systems, and standards that had an impact on the latter.

It notes the following:

  • Early electronic records management systems (ERMS) were mostly mainframe-based databases.
  • Electronic document management systems (EDMS) were developed from the 1980s to manage unstructured electronic information.
  • ERM systems began to include unstructured information from the 1990s, especially with the introduction of DOD 5015.2 and MoReq.
  • The rise of XML from the late 1990s had an impact on methods to preserve electronic information.
  • Content management (CM) and knowledge management (KM) systems appeared in the 1990s.
  • Cloud-based EDM systems, and enterprise content management (ECM) appeared from the early 2000s.
  • Cloud based systems challenged installed on-premise EDRM systems.
  • There is a history of acquisitions. (See Mergers & Acquisitions – This Week – MandAsoft to find more details of any acquisitions).

Sources for this information are listed where this is known.

1973 – Plato Notes

A history of ERM and EDM systems must include reference to Lotus Notes.

Lotus Notes began its life in 1973 as Plato Notes, developed by the Computer-based Education Research Laboratory (CERL) at the University of Illinois in 1973. Elements of the Plato Notes system would be developed for PC by Ray Ozzie during the late 1970s. This was picked up by Lotus Development Corporation and in 1984 became Lotus Notes.

An early version of Lotus Notes was released (under contract to Lotus) in 1984. The original vision included on-line discussion, email, phone books and document databases. Eventually the product fell into the ‘groupware’ category. The capability of the product continued to grow and some organisations only used Notes.

Lotus acquired all the rights to Lotus Notes in 1987 and version 1.0 was released on 7 December 1989.

1974 – Compulink Management Center/Laserfiche founded

Compulink Management Center was founded in the US in 1974. It created Laserfiche, the first DOS-based document imaging system, in 1987.

1976 – Micro Focus founded

Micro Focus was founded in the UK in 1976. Its first software product was CIS COBOL, a solution for micro computers. It entered the EDRM market in 2017, see below.

1981 – Enterprise Informatics founded

Enterprise Informatics, a privately-held software company, was founded in 1981 by early pioneers of the document management industry. (Source: LinkedIn company profile) It would later be acquired by Spescom, a South African company.

1982 – FileNet founded

FileNet was founded in 1982 by Ted Smith, formerly of Basic 4. FileNet’s original focus of attention was the storage and management of scanned images but it also developed a workflow software. (Source: Wikipedia article on FileNet)

1983 – GMB/DocFind founded

GMB (named after the original founders, Gillett, Frank McKenna, and Bachmann) was formed in Australia in 1983. In 1984, GMB released DocFind 1.0. DocFind was renamed RecFind in 1986.

For a very detailed history of this company, see The History of GMB/KnowledgeOne.

1984 – The first EDRMS standards – Noark (Norway)

The Norwegian government developed model requirements for EDRM systems in 1984, the first organisation globally to do so. Noark

(Source: ‘The Norwegian Noark Model requirements for EDRMS in the context of open government and access to governmental information‘, by Olav Hagen Sataslaaten, National Archives of Norway, published in the Records Management Journal 11 November 2014.)

Noark has remained active to the present day. (National Archives of Norway – E-Ark Project (

1985 – Tower Software founded (Australia)

Tower Software was founded by Brand Hoff in Canberra in 1985 as a software development company. The company provided and supported enterprise content management software, notably its TRIM (Tower Records and Information Management) product line for electronic records management.

The ‘Tower’ in the company name derives from the telecommunications tower on top of Black Mountain (technically a hill, 812 m high) overlooking Canberra. A graphic of the tower was used in the TRIM logo until the company was acquired by HP’s Software Division in 2008 (see also below).

1986 – Autonomy founded (UK)

Autonomy was founded by Michael Lynch, David Tabizel and Richard Gaunt in Cambridge, UK in 1986 ‘as a spin-off from Cambridge Neurodynamics, a firm specializing in computer-based finger print recognition’.

Before 1987 – Saros Corp

Saros Corp was established in Washington by Mike Kennewick (a former Microsoft employee) before 1987. Saros Corp produced Saros Mezzanine, a client-server document management engine. In 1993, released Saros Document Manager.

1989 – Ymijs (later Valid Information Systems) founded – R/KYV (UK)

Ymijs was founded in the UK in 1989. It sold the R/KYV software initially as a basic document imaging processing system. The company name was changed to Valid Information Systems and R/KYV was further developed as a compliance and records management system ‘… that is widely used by major corporations as well as central and local government authorities and related governmental agencies’ (in the UK).

Valid Information Systems was acquired by Hummingbird in 2003. (Source: ‘Legal Technology Insider‘, 23 July 2003.

1989 – Provenance Systems (later TrueArc) founded (Canada)

Bruce Miller, sometimes noted as ‘the inventor of modern electronic recordkeeping software’, founded Provenance Systems in 1989 where he created ForeMost. The company name was changed to TrueArc. Bruce would go on to found Tarian Software as well in 1999.

TrueArc ForeMost RM would be acquired in 2002 by Documentum (which which it had a long-standing technology partnership).

1990 – Documentum founded (US)

According to this Wikipedia article, Documentum was founded in June 1990 by Howard Shao and John Newton who had previously worked at Ingres (a relational database vendor). They sought to solve the problem of unstructured information.

The first Documentum EDMS was released in 1993. According to the Wikipedia article, ‘This product managed access to unstructured information stored within a shared repository, running on a central server. End users connected to the repository through PC, Macintosh, and Unix Motif desktop client applications.’

1992 – Altris Software (UK)

Altris, established in 1992 (Source: Rob Liddell\’s LinkedIn profile. Rob was one of the co-founders of Altris), developed document management systems, including (according to this South African ITWeb post of 26 October 2001), eB, a ‘configuration management’ application. 

Altris would be acquired by the South African Spescom in 2000.

1992 – Optika ImageFiler / FilePower

Optika Imaging Systems (Colorado, US) sought a trademark for a product called ImageFiler in September 1992. The request was abandoned according to information on the Bizapedia site.

This article titled ‘The Case for 11g‘ (referring to Oracle’s product, see below) noted that Optika’s original software development focus was Image and Process Management (IPM).

An undated (but likely mid to late 1990s) webpage on the Property and Casuality website titled ‘Optika and Xerox Package FilePower with Document Centre‘ noted that ‘Optika Imaging Systems, Inc. and Xerox announced that the two companies will jointly work to integrate Optika’s FilePower with Document Centre digital systems products from Xerox. The combination of the Document Centre and FilePower will provide a complete solution for capturing, managing and distributing large volumes of documents, increasing users’ productivity and significantly reducing labor and capital costs. Optika’s integrated product suite — FilePower — combines imaging, workflow and COLD technology into a unified software package. The Xerox Document Centre 220ST and 230ST combine network scanning, printing, faxing and copying into one hardware device.’

1993 – Workflow Management Coalition formed

The Workflow Management Coalition (WfMC), ‘a consortium formed to define standards for the interoperability of workflow management systems’, was founded in May 1993. Original members included IBM, Hewlett-Packard, Fujitsu, ICL, Staffware and approximately 300 software and services firms in the business software sector.

The WfMC’s Workflow Reference Model was published first in 1995 and still forms the basis of most BPM and workflow software systems in use today. (Source: Undated Gutenberg article)

1993 – Kainos Meridio (UK)

Meridio was developed in 1993 by Kainos (a Northern Ireland company and joint venture between Fujitsu and The Queens University in Belfast) as an electronic document and records management (EDRM) system based on Microsoft products. It would be acquired by HP Autonomy in 2007.

1993 – Saros (US) Document Manager

Saros Corp released Saros Document Manager in mid 1993. The product was said ‘to act as a front-end to the Bellevue, Washington-based firm’s client-server document management engine, Saros Mezzanine’. (Source: Computer Business Review article ‘Saros Sets Document Manager‘ )

ERM before the mid 1990s

Before the arrival of personal computers in offices in the early 1990s, computer mainframes and databases were the regarded by some observers as the only places where electronic ‘records’ (in the form of data in tables) were stored and managed.

A report by the United States General Account Office in July 1999 (GAO/GGD-99-94) titled ‘Preserving Electronic Records in an Era of Rapidly Changing Technology’) stated that, historically (as far back as 1972), NARA’s Electronic Records Management (ERM) guidance (GRS 20) was geared towards mainframes and databases, not personal computers.

The GAO report noted that until at least the late 1990s, there was a general expectation that all other electronic records not created or captured in ERM systems would be printed and placed on a paper file or another system. The original (electronic) records could then be destroyed.

Some early ERM (database) systems, such as TRIM from Tower Software in Australia, were originally developed in the mid 1980s to manage paper files and boxes. Similar systems were developed to manage library catalogues and the old card catalogues started to disappear.

But, although some of it was printed and filed, the volume of electronic records in email systems and stored across network file shares continued to grow. Several vendors released systems that could be used to manage electronic documents (EDM) more effectively than network drives but there was no agreed standard for managing that content as records.

1994 – The DLM Forum and MoReq

From the early 1990s, the European Council sought to promote greater cooperation between European governments on the management of archives. One of the outcomes of a meeting in 1996 was the creation of the DLM Forum. DLM is the acronym of the French term ‘Données Lisibles par Machine’, or ‘machine-readable data’.

One of the ten action points arising from the June 1994 DLM meeting was the creation of ‘Model Requirements for the Management of Electronic Records’, or MoReq, first published in 2001 (see below).

Mid 1990s – Two rival records management models

According to James Lappin, in his paper ‘Rival records management models in an era of partial automation‘ published on 22 January 2021, two rival records management models appeared in the mid 1990s.

North America – separate records repository model 

According to its website, ‘the InterPARES Project was borne out of previous research carried out at the University of British Columbia’s School of Library, Archival and Information Studies. “The Preservation of the Integrity of Electronic Records” (a.k.a. “The UBC Project”) defined the requirements for creating, handling and preserving reliable and authentic electronic records in active recordkeeping systems.’

‘The UBC Project researchers, Dr. Luciana Duranti and Professor Terry Eastwood, worked in close collaboration with the U.S. Department of Defense Records Management Task Force to identify requirements for Records Management Applications (RMA).

The work of the UBC team influenced the development of DOD 5015.2 published in 1997 (see below) and the subsequent development of a range of electronic document and records management (EDRM) systems.

Australia – intervention in business applications model

In 1996, the University of Pittsburgh published the ‘Functional Requirements for Evidence in Recordkeeping Project’, led by David Bearman. This work would influence the development of both MoReq2010 and the ICA standards that became ISO 16175-2010, both of which attempted to define a minimum set of functional requirements for a business application to be able to manage its own records. (Lappin)

1995 – IBM Acquires Notes

Lotus Notes was acquired by IBM in July 1995. By December 1996 it had 20 million users. By the end of 1999, Lotus Notes had extensive capability including ERM and EDM.

Lotus Notes continued to retain a strong presence in the market but its dominance began to be reduced by the arrival of Microsoft’s broader capabilities and other EDM solutions.

1995 – Alpharel (US) acquires Trimco (UK)

According to this Computer Business Review article of 23 November 1995, Alpharel Inc, San Diego was expected to acquire Trimco Group Plc of Ealing, London, a supplier of enterprise-wide document management systems.

1995 – FileNet acquires Saros

FileNet acquired Saros Corporation in 1995 to acquire its electronic document management capability. It was said to have pioneered ‘integrated document management’ (IDM), through a suite that offered document imaging, electronic document management, COLD and workflow. (Source: Wikipedia article on FileNet)

1996 – Australian Standard AS 4390

In February 1996 Australia issued the world’s first national records management standard, AS4390 ‘Records Management – General‘. The standard provided guidance for the implementation of records management strategies, procedures and practices.

Tower Software, the Canberra-based developers of TRIM, contributed to the development of the standard (according to its Wikipedia entry) although the standard did not prescribe requirements for the management of electronic records.

AS 4390 would become internationalised through ISO 15489 in 2002.

1996 – OpenText Corporation (US) – Livelink

OpenText Corporation was founded in 1991 from OpenText Systems. It released Livelink in 1996.

1996 – EDM solutions (UK listing)

The following is a list of EDM systems taken from the Document Management Resource Guide, 1995/96 Edition, kindly provided by Reynold Lemming in 2021. (^ = Original software author entry, all others are system resellers)

ADOS ArchiveDocStore ^Legato Networker, Arcada, DocuBaseRecFind
Adobe Acrobat, Adobe Capture ^Docarus.DMS ^Lotus NotesResmanager ^
Alpharel Image Enabler, Toolkits, API, TurnkeyDocsOpenM-Image ^Retriever
AutoFORM ^Documation ^MacroFicheRoleModel ^
Autodesk WorkCenterE-DOCMacroImageSAROS Mezzanine ^
B-ScanEcoStore ^MatrixSTOR/QM *
CADEXnet ^Epson ImageWise ^MaxDoc ManagerSaros Document Management ^
CAIRS-TMS ^FileMagicMaxDoc and TargetSimplifier
COLDbase ^FilePlusMicroStation TeamMate ^Soft Solutions
CanofileFlowMan ^Myriad ^Software Integration ^
CaptureBase, CaptureFlow, CapturePrint ^FORMTEK. TDM ^NovellSound Practice ^
Cimage: Document Manager, Image Master workflowFrame Technology Document Management ^Oasis RSD, COLD v.2.1 ^Staffware ^
ColdStore ^HDK2Odesta Livelink ^Status/IQ, inSign ^
Complete PMEHyperdrive ^OptechStorofile 650 ^
Concept Suite: Inquisitor, Interrogator, Detective ^ImageTekOptiFicheSydneyPLUS ^
Cyco Automanager workflowImagetreeOptiIMAGETRIP
D.A.R.T.S. ^Imagewise ^Optigraphics LDSTRIPscan
DB/TextWorks ^ImviewOptika FilePower, FileMagic, ImageSoftTeleForm OCR/ICR *
DCS PowerNET ^InConcert ^Optika File Power FP MultiTrimco
DM2 ^Intelligent Information Capture (IIC)PAFEC EDM ^Ultra
DOCS Open ^Interleaf: 6, RDM, World View, Cyberleaf ^PC DOCS OpenViewStar System Release 4 ^
Datascan ^IsysPageNet ^Visual Recall ^
DataWise ^KeyfilePaperClipVisual Workflow
Deimos Document Management System ^Kodak ImageryPaperoute ^WinWork ^
Displaywise ^LaserImage, LaserArcPaperwiseWorkCenter Issue Control, Revision Control ^
DocFind ^Lasercom ^PowerVision ^
DocPRO ^Lasertrack Imaging System ^QStar: Axxess / Server / Worksgroup / Enterprise ^

1996 – Various EDM solutions

The March 1996 edition of Engineering Data Management included a number of updates on electronic document management solutions in the market at that time. Note that Trimco and Alpharel are listed separately; this may because Alpharel’s acquisition of Trimco had not been completed by that time.

  • Alpharel (San Diego, CA): Document Management solutions – Enabler, FlexFolder, RIPS, Toolkit API. Wisdom, a product that facilitated internet access to participating electronic document vaults.
  • Auto-trol Technology (Denver, CO): CENTRA 2000, document management, workflow, PDM, change management and messaging.
  • Cimage Enterprise Systems (Bracknell, UK): Document Manager for Windows.
  • Documentum, Inc. (Pleasanton, CA): Documentum Accelera for the World Wide Web and Documentum UnaLink for Lotus Notes.
  • Interleaf (Waltham, MA): Interleaft 6 SGML, a solution for publishing SGML doocuments. Intellecte/BusinessWeb, a document management solution that allowed organisations to access enterprise document repositories from the internet.
  • Trimco (Ealing, UK): Document management systems.

Alpharel changed its name to Altris Software (US) in October 1996, according to this Telecompaper article published the same month.

From 1996 – Germany’s DOMEA project

In 1996, the Coordinating and Advising Agency of the Federal Government for Information Technology in the Federal Administration (KBSt) introduced a pilot project named Document Management and Electronic Archiving in computer-assisted business processes (DOMEA).

Under the framework of DOMEA, a project group was set up in 1998 to find solutions for the disposition and archiving of electronic records. The goal was to find a suitable and efficient way for the disposition of electronic records created and maintained in office systems. Its “Concept for the Disposition and Archiving of Electronic Records in Federal Agencies,” containing recommendations for managing electronic records was published in September 1998. (Source: The Free Library article)

Late 1990s – EDMS vs ERMS

Electronic document management systems (EDMS) and electronic records management systems (ERMS) were regarded as separate types of system from the late 1990s until at least 2008.

According to Philip Bantin in August 2002:

  • An EDMS was said to support day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
  • An ERMS was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted. (The concept of ‘declaring a record’ may be related to this point).

(Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001)

The difference between the two types of system endured for at least a decade. By the end of the 1990s, four main EDRMS options had emerged:

  • Extending an existing EDM product capability to include ERM.
  • Extending an existing ERM capability to include EDM.
  • Creating new ERM products (technically also with some EDM capability).
  • Integrating separate EDM and ERM products.

1997 – DOD 5015.2

According to the 1999 GAO report quoted above, for several years prior to 1997, NARA worked with the US Department of Defense, considered ‘one of the agencies that is most advanced in its ERM efforts’.

The outcome of this work was the release in November 1997 of the DOD standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’ usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, 11 April 1997.

The GAO report stated that ‘ERM information systems that were in place before the approval of this standard must comply with the standard by November 1999’.

It added that US agencies ‘were confronted with many ERM challenges’ from the ever-increasing volume of digital records, including the ability to preserve and access those records over time. The ‘Year 2000 problem’ was drawing attention away from the issue.

Nevertheless, by 2 June 1999, nine companies were certified as compliant with the DOD standard. Some, it noted, were standalone ERM software, while others were an integrated solution.

An interesting small note on page 11 of the GAO report noted that ‘it is important that ERMS software requires users to make no more than two or three extra keystrokes, and that users realize there is a benefit to this additional ‘burden’.

From 1997 – SER eGovernment (Germany)

SER eGovernment was developed for the German/Austrian market following the release of the German eGovernment standard, DOMEA in 1997.

1998 – Documentum goes online

In 1998, Documentum released its Web Application Environment, a set of internet extensions for EDMS, offering web access to documents stored within an EDMS repository. Various additional products were acquired and their functionality added to the Documentum system.

1998 – Optika eMedia released

Optika released eMedia, ‘a software and methodology product designed to manage business transactions within an organization, across extranets, and throughout the supply chain’, in late 1998. (Source ‘Optika Delivers App to Manage Business Transactions‘) Optika eMedia was said to be ‘a workflow enabled replacement for an imaging solution named FilePower’.

1998 – FileNet Panagon suite released

In 1998, FileNet released its Panagon suite of products. This included Panagon Content Services that was previous Saros Mezzanine. (Source: Wikipedia article on FileNet)

1999 – International differences

The 1999 GAO report noted differences between the US, UK, Australia and Canada on their approach to ‘common ERM challenges’.

  • Australia was said to have ‘strong central authority (including for compliance audits) and decentralised custody’ (except when the records are transferred to permanent retention).
  • Canada had ‘vision statements rather than specific policies’ and also had decentralised custody, but agencies could transfer records at any time to the archives.
  • The UK had broad guidelines put into practice by individual agencies.

1999 – the UK PRO standard released

The UK Public Records Office (PRO, later The National Archives, TNA) released a standard in 1999 designed ‘to provide a tool for benchmarking the ability of government departments to support electronic records management’. This standard would be replaced by TNA 2003. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)

End of the 1900s – XML

By the end of the 20th century it was becoming clear (to some) that XML would likely play a strong role in the standardisation of electronic record formats and their management over time.

XML-based record structures meant that electronic records could contain their own ‘metadata payloads’ rather than being independent objects defined in a separate system (like a library catalogue describes books on shelves).

The establishment of XML-based formats would (after about 20 years) begin to change the way in which records would be managed, although paper records and the paradigm of managing records in pre-defined containers would continue to persist, largely because of the standards developed to manage electronic records – in particular DOD 5015.2.

1999 – EDM/early ERM products

The following is a collated list of EDM (and related) products collated in November 1999:

  • Autonomy Portal in a Box
  • CompuTechnics (1990 to 1999)/Objective (from 1999)
  • DOCS
  • Eastman Software (Imaging, OCR)
  • EDM Software (UK) and SER (Germany)
  • Excalibur RetrievalWare
  • FileNet, incorporating Panagon Integrated Document Management
  • Fulcrum (acquired by PC DOCS in 1997)
  • GMB (later KnowledgeOne) RecFind/RecQuery
  • Hummingbird (from 1999 with acquisition of PC DOCS)
  • Insight Technologies Knowledge Server (IKS) / Document Management System (DMS)
  • Intraspect Knowledge Server (IKS) (KM)
  • Lotus Notes
  • Onyx Enterprise Portal, with integration to various EDM applications
  • Open Text Livelink
  • Pitney Bowes Digital Document Delivery (D3)
  • Oracle
  • Osyssey (ISYS)
  • PC DOCS (acquired by Hummingbird in early 1999)
  • ReadSoft (OCR processing)
  • RecordManager
  • Tower Software / TRIM Captura
  • Tower Technology
  • Verity (KM)

1999 – Tarian Software founded

Tarian Software was founded in Canada in 1999 by Bruce Miller, the founder of Provenance Systems (later TrueArc) and creator of ForeMost. Tarian developed the Tarian eRecordsEngine, an embedded electronic recordkeeping technology for business application software. Tarian was the first e-Records technology in the world to be certified against the revised 5015.2 June 2002 standard. Tarian was acquired by IBM in 2002.

1999 – The Victorian Electronic Records Strategy (VERS)

The (Australian) Victorian government’s Public Records Office (PROV) published a standard for the management of electronic records in 1999, Standard 99/007 ‘Standard for the Management of Electronic Records’. The standard, usually known as VERS, defined the (XML-based) format required for the transfer of permanent records to the PROV.

The Standard noted that:

  • Records must be self-documenting. It is possible to interpret and understand the
    content of the record without needing to refer to documentation about the system in
    which it was produced
  • Records must be self-contained. All the information about the record is contained
    within the record itself
  • The record structure must be extensible. It must be possible to extend the structure of
    the record to add new metadata or new record types without affecting the
    interoperability of the basic structure.

Several EDRMS vendors developed the capability to create VERS encapsulated objects (VEOs) as required by the standard.

2000 – Spescom (South Africa) acquires Altris (UK)

The South African company Spescom acquired the UK firm Altris Software in 2000, as noted in this (South Africa) ITWeb article of 3 May 2000. Altris was described in the article as ‘a global leader in integrated electronic document management software, with well established channels to international markets’. As a result of this acquisition, Altris UK was renamed Spescom Ltd (UK).

The same journal announced in 2001 that Spescom KMS was ‘the UK operation of Spescom Limited’s US based subsidiary, Altris Software Inc, which specialises in the provision of asset information management software to markets including transportation, utilities and telcos’.

From 2000 – Microsoft adopts XML for Office documents

In 2000, Microsoft released an initial version of an XML-based format for Microsoft Excel, which was incorporated in Office XP.

In 2002, a new file format for Microsoft Word followed. The Excel and Word formats, known as the Microsoft Office XML formats (with an ‘x’ on the end of the document extension), were later incorporated into the 2003 release of Microsoft Office.

Microsoft’s XML formats, known as Open Office XML, later became ECMA 376 in 2006 and later ISO 29500 in 2008 ‘amid some controversy’ over the need for another XML format (see below).

Before 2001 – Intranet Solutions (later Stellent)

Intranet Solutions had developed software called’IntraDoc!’. The product was briefly renamed Xpedio! before the company and product were renamed Stellent in 2001. (Source: ‘Wikipedia article on Oracle Acquisitions‘)

2001 – EDM systems with RM functionality

The following is a list of ‘EDM systems with records management’ functionality available by early 2001:

  • TRIM (Tower Software, Australia) – integrated ERM and EDM.
  • ForeMost Enterprise v2 (TrueArc Inc) – ERMS
  • Tarian eRecords Engine v1.0 (formerly e-Records v1.0)(Tarian) – ERMS
  • iRIMS 2001 (Open Text Corp) – ERMS
  • Objective 2000 (Objective Corp, Australia) – ERMS
  • Hummingbird RM Family 4.0 (Hummingbird, Canada) – ERMS (later acquired by Open Text)
  • FileSurf 7.0 (MDY Advanced Technologies) – ERMS
  • R2M (Relativity Records Manager v2 (Relativity Inc) – EDMS with added records management functionality.
  • eManage 2000 (ByteQuest Technologies) – knowledge management (KM) integrated with records management.
  • FileNET IDM Content Services 5.1.1/ForeMost Enterprise 2.0 – EDMS with integrated records management.
  • IBM e-Records Solution (IeRS) v1.0 – IBM’s content management (CM) solution integrated with Tarian eRecords v1.0.
  • Work Manager/ForeMost (Eastman Software) – combined the DM, imaging and workflow of Work manager with the RM capabilities of ForeMost.
  • (Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001).

Many of the products above were sold internationally on the basis of their compliance with DOD 5015.2 even though that was a US-specific standard.

2001 – Birth of SharePoint

Microsoft released SharePoint in 2001. It was born out of a combination of Office Server and Tahoe.

  • Office Server provided the simplest of network administration actions
  • Tahoe expanded on the email, calendaring, contacts, and to-do list technology contained within Exchange.

However, it was not until the release of Office 2003 that SharePoint (rebranded as Windows SharePoint Services) provided collaboration and other functionality.

(Source: History of SharePoint from PortalFront hosting)

Early 2000s – The DM, KM and CM mix

In addition to the EDMS/ERMS differences, organisations were also seeking solutions for knowledge management (KM) and content management (CM).

CM solutions were usually portal-based options that mostly became some form of intranet.

Some of the options in the early 2000s included:

  • Hummingbird’s PowerDOCS for DM and CyberDOCS as the web client for the DM solution, along with Hummingbird’s (formerly Fulcrum) Knowledge Server for KM and PD Accord for web-based collaboration, with the Hummingbird Enterprise Information Portal (EIP) as the portal solution. Plumtree Corporate Portal could also be used as an Enterprise Portal.
  • iManage’s DeskSite for DM and WorkTeam for collaboration. For KM, WorkKnowledge Server and Concept Search (based on the Autonomy Server). The portal to link all of these was called WorkPortal.
  • Open Text’s Livelink for DM, KM and collaboration.
  • Elite’s Encompass, built on Microsoft’s new SharePoint Portal Server (SPS).
  • Autonomy Server for KM, with Plumtree Corporate Portal as the portal.
  • Documentum’s DM and CM product coupled with Plumtree Corporate Portal.

Digital Asset Management (DAM) systems, used to manage other types of digital content such as photographs, also appeared around this time.

Early 2000s – RM meets AI

Don Lueders noted in his post An Industry Pioneer on Records Management and Artificial Intelligence ( that ‘in the early 2000’s TruArc released a product called AutoRecords. AutoRecords was the very first records management solution brought to market that leveraged artificial intelligence to automatically classify electronic records.’

2001 – EDMS Vendors

Information Technology Decisions published a paper on DOD 5015.2 certified products in November 2001 (original source/location has been lost). It noted that there were two types of products:

  • Products that started life as electronic document management (EDM) systems. Examples included Documentum, Livelink, and DOCS Open.
  • Products that started life as electronic recordkeeping (ERK/ERM) systems. Examples given included Tower Software’s TRIM, Foremost, iRIMS, Cuadra Star.

The presentation noted that DOD 5015.2 certification was based on alternative options:

  • Standalone. For example, True Arc Foremost, TRIM, iRIMS, Cuadra Associates Star, Relativity Records Manager, Hummingbird RM 4.0, Tarian eRecords, MDY/FileSurf, Cimage and Access Systems, Highland Technologies Highview-RM, Open Text, Livelink
  • Partnership. For example, Saperion with e-Manage 2000, Impact Systems eRecords Manager, FileNet with Foremost.

The report included three interesting points:

  • Both EDMS and ERKS required an enterprise view of information.
  • An EDMS is driven by business process requirements.
  • An ERKS (ERMS) is driven by enterprise requirements for the long-term preservation of information.

2001 – The first MoReq

The first version of MoReq was published in 2001. Volume 1 was 500 pages long.

MoReq emphasised the central importance of an electronic records management system, or ERMS. Its stated purpose was:

  • To provide guidance to organisations wishing to acquire ERMS.
  • As a tool to audit or check an existing ERMS.
  • As a reference document for use in training or teaching.
  • To guide product development by ERMS suppliers and developers.
  • To help define the nature of outsourced records management solutions.

Few, if any, products were certified against this version of MoReq.

2002 – Optika Acorde

Optika eMedia was rebranded to Optika Acorde in 2002, according to this website ‘The Case for 11g‘.

A June 2002 Gartner report titled ‘Optika Acorde Document Imaging, Workflow and Collaboration Suite‘ noted that Optika Acorde was an ‘integrated software family for managing the content associated with business transactions’ leveraging ‘Optika’s core strengths in document imaging, workflow and enterprise report management.’

2002 – FileNet BrightSpire, later P8 ECM

FileNet released BrightSpire in 2002. This product ‘leveraged the experience gained from integrated document management, web content management and workflow into what became ECM. (Source: Wikipedia article on FileNet)

By 2002 – Enterprise Content Management (ECM)

The term ‘Enterprise Content Management’ (ECM) began to appear more frequently by 2002. The Wikipedia post on ECM noted that ECM technologies descended from ‘electronic Document Management Systems (DMS) of the late 1980s and early 1990s’.

ECM promised:

  • The integration of records management (RM) with business practices.
  • The capability for integration between RM products, EDM, various other digital products (such as OCR/character recognition technologies), and web publishing products.
  • Incorporation of Knowledge Management (KM) concepts.

The key word here was ‘integration’ with EDM and other systems, rather than standalone systems. Web-based access became increasingly essential. IBM’s acquisition of Tarian, Documentum’s acquisition of TrueArc’s Foremost were examples of these integrations. (see below)

According to the Wikipedia article on ECM: ‘Before 2003, the ECM market was dominated by medium-sized independent vendors which fell into two categories: those who originated as document management companies (Laserfiche, Saros, Documentum, docStar, and OpenText) and began adding the management of other business content, and those who started as web content management providers (Interwoven, Vignette, and Stellent) tried to branch out into managing business documents and rich media’.

The emergence of ECM quite possibly created the first challenge to centralised ERM through the integration of multiple elements, some of which created, captured or stored records in ever increasing formats.

2002 – OpenDocument XML format

According to the Wikipedia article on the OpenDocument standard, the OpenDocument standard was developed by a Technical Committee (TC) under the Organization for the Advancement of Structured Information Standards (OASIS) industry consortium. Sun and IBM apparently had a large voting influence but the standardization process involved the developers of many office suites or related document systems. The first ODF-TC meeting was held in December 2002.

2002 – An updated GAO report into electronic records

The US General Accounting Office (GAO) released a new report in June 2002 titled ‘Information Management: Challenges in Managing and Preserving Electronic Records’ (GAO-02-586). This report, which was more detailed than the earlier 1999 one, noted among other things that:

  • The DOD had by March 2002 certified 31 applications against standard 5015.2.
  • Progress had been made on the development of the Open Archival Information System (OAIS) model which, while initially developed by NASA for archiving the large volume of data produced by space missions, could be applied to ‘any archive, digital library or repository’. XML-based solutions were considered the most likely to be accepted.

2002 – IBM acquires Tarian Software

IBM acquired Tarian Software in late 2002, ‘bolstering its leadership in ECM’.

From that date, IBM released the IBM Records Manager Version 2.0 (IRM), previously known as the Tarian eRecords Engine (TeRe). Tarian’s e-Records management technology was integrated into IBM’s software offerings, including IBM Content Manager, DB2 database and Lotus software. (Source: IBM press release)

2002 – Documentum 5 and TrueArc Foremost acquisition

Documentum released Documentum 4i, its first Web-native platform, in 2000. In 2002, it launched Documentum 5 as ‘a unified enterprise content management (ECM) platform for storing a virtually unlimited range of content types within a shared repository’.

Documentum acquired TruArc’s Foremost product in October 2002. The Documentum Wikipage above noted that this acquisition ‘added records management capabilities and augmented Documentum’s offerings for compliance solutions.’ The press release cited in this paragraph noted that ‘Documentum and TrueArc are existing technology partners and have worked together to provide an integration for TrueArc’s enterprise-scalable records management solution with the Documentum ECM platform.’

2003 – TNA 2003

The National Archives (TNA) released an updated version of its PRO standard in 2003, known as TNA 2003. This standard would be superseded by MoReq2. (Source: @ZenInformation on Twitter, 12 February 2021).

2003 – EMC acquires Documentum

EMC acquired Documentum in 2003. (Source: EMC offers $1.7 billion in stock for Documentum, October 2003))

2003 – FileNet P8 Records Manager

FileNet released P8 Records Manager in 2003. (Source: Wikipedia article on FileNet)

2003 – Hummingbird acquires Valid Information Systems R/KYV

Hummingbird acquired Valid Information Systems R/KYV product in 2003.

(Source: ‘Legal Technology Insider‘, 23 July 2003)

2003 – Open Text acquires SER eGovernment

In October 2003, Open Text acquired the (German) DOMEA-certified SER eGovernment Deutschland GmbH, based in Berlin, Germany as well as SER Solutions Software GmbH, based in Salzburg, Austria. (Source: Open Text Acquires SER eGovernment)

From 2003 – CNIPA (Italy)

The Italian Centro Nazionale per l’Informatica nella Pubblica Amministrazione (CNIPA) published a protocol for the management of electronic records, Protocollo Informatico in 2003.

CNIPA was renamed DigitPA in 2009 and Agenzia per I’Italia digitale (AGID) in 2012. AGID is responsible for defining standards for the management of electronic records in Italian government agencies. (Source: Protocollo Informatico)

Mid 2003 – The challenges of Enterprise Records Management

In Industry Trend Reports of May 2003, Bruce Silver (of Bruce Silver Associates) made the case for Enterprise Records Management in the wake of various ‘scandals’ involving the management of records at the time, including Enron/Anderson.

Silver argued that EDM, email archive, and back-up solutions did not meet the ‘new statutory and regulatory records management requirements’ – DOD 5015.2, SEC Rules 17a-3 and 17a-4, NASD Rules 2210, 3010, and 3110, NYSE Rules 342 and 440, ISO 15489 and MoReq.

Silver also noted that an effective (‘total’) ERM solution would ‘be implemented as an extension of the company’s ECM infrastructure’, providing for a single interface for all records stored in multiple locations ‘including third-party document management repositories in addition to the email system and network file system’.

2003 – Key integrated EDM/RM vendors

The following is a list of ‘key vendors in the (Integrated Document Management) IDM Market Space’ in October 2003. The list is believed to have come from a Butler Group report:

  • Documentum 5. EDMS, web content, ERM, RM (through acquisition), collaboration, DAM.
  • FileNet P8 Image Services. Content management, web content, image management, RM.
  • Gauss VIP Enterprise 8. Web content, DM, Imaging, ERM. (Acquired by Open Text)
  • Hummingbird Enterprise 5. Full IDM suite (DM, KM, portal, collaboration, workflow, imaging, RM)
  • Hyland Software OnBase. Integrated IDM but lacked strong web content, RM and collaboration.
  • IBM Content Manager. ERM and worfklow.
  • Identitech FYI. Integrated IDM.
  • Information Management Research (IMR) Alchemy. IDM.
  • Laserfiche Document Imaging. Integrated IDM including EDM, ERM.
  • Legato ApplicationXtender, EmailXtender, DiskXtender. Integrated IDM.
  • Open Text Livelink.
  • Optical Image Technology (OIT) DocFinity. Integrated IDM.
  • Optika Acorde. Primarily for images and ERM. Not strong CM, RM or collaboration. Acquired Select Technologies for RM.
  • Tower Technology IDM, Seraph. Integrated IDM.
  • Westbrook Fortis Suite, File Magic. Integrated IDM.

2004 – Stellent acquires Optika

Stellent acquired Optika in early 2004. At this stage, Stellent’s primary product was Universal Content Management (UCM). (Source: Wikipedia article on Oracle Acquisitions)

The report of the sale in The Register stated that Stellent’s CEO said that ‘customers are looking to consolidate their content management needs, including imaging, business process management, web content management and record management with one vendor.’ The new product line was named Stellent Imaging and Business Procss Management (IBPM). The article also noted that Oracle would probably acquire Stellent following this acquisition (see 2006, below).

2004 – ReMANO (Netherlands)

In 2004, the Netherlands government established a catalogue of software specifications for ERM systems (ReMANO) used by Dutch government bodies. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)

ReMANO was replaced by NEN2082 – Eisen voor Functionaliteit van Informatie- en Archiefmanagement in programmatuur” in 2008. (NEN 2082:2008 nl)

2005 – C6 (France) builds D2 on top of EMC Documentum

The French ECM company C6 built a solution named D2, ‘a fully configurable web application for creating, managing, storing and delivering any type of information’, on top of EMC’s Documentum. (Source: C6 website ‘Company’ tab).

2005 – ELAK (Austria)

A project to introduce electronic filing (ELAK) commenced in Austria late 2001 and was completed in 2005. The solution was based on standard software products (e.g. Fabasoft e-Gov-Suite, MS-Office). (Source (ELAK – The e-filing system of the Austrian Federal Ministries)

2006 – The National Archives of Australia ERMS standard

The National Archives of Australia (NAA) released its ‘Functional Specifications for Electronic Records Management Systems Software in February 2006. (ISBN 1 920807 34 9). The introduction noted that:

  • (The document) provided Australian Government agencies with a set of generic requirements for ensuring adequate recordkeeping functionality within electronic records management systems (ERMS) software.
  • Agencies were encouraged to make use of the ERMS specifications when designing or
    purchasing new, or upgrading existing, ERMS software. They could also
    be used when auditing, assessing or reviewing an agency’s existing ERMS software.
  • The requirements were not intended to be a complete specification, but
    rather provide a template of key functional requirements that agencies may
    incorporate into their tender documentation when preparing to select and purchase
    new ERMS software. Agencies were expected to assess and amend the
    functional requirements, and select requirements that best suit their own business
    and technical requirements and constraints.

Very few products met the specific requirements of the ERMS specifications which led to some suggestion at the time that it limited choice.

2006 – Rival XML Office document standards

The OpenDocument (ODF) standard was published as ISO/IEC 26300 in 2006.

Microsoft submitted initial material to the Ecma International Technical Committee TC45, where it was standardized to become ECMA-376, approved in December 2006. It was released as ISO 29500 in 2008.

According to the Wikipedia article on Open Office XML (OOXML), ‘The ISO standardization of Office Open XML was controversial and embittered’, as it seemed unnecessary to have two rival XML standards.

2006 – The world of collaboration

The Butler Group published a paper titled ‘Document Collaboration – Linking People, Process and Content’ in December 2006. The report noted that EDM systems had helped improve internal efficiency but there was now a need to ‘extend these systems to partners and stakeholders’ and deliver ‘sophisticated collaborative experiences’.

The paper listed the following EDM products:

  • Adobe Acrobat family
  • EMC eRoom
  • IBM Notes/Domino, Workplace collaboration services, QuickPlace
  • Microsoft Office 2007
  • Open Text Livelink ECM – eDOCS (incorporating the former Hummingbird product suite acquired by Open Text in 2006)
  • Oracle Collaboration Suite, Content DB and Records DB
  • Stellent Collaboration Management
  • Vignette

2006 – Spescom Software Inc

A US SEC submission in January 2006 noted that Spescom Software Inc, a San Diego-based provider of computer integrated systems was the successor to Alpharel Inc and Altris Software Inc.

2006 – Oracle acquires Stellent

A 2006 Oracle press release titled ‘Oracle Buys Stellent‘ stated that Stellent was a global provider of enterprise content management (ECM) software solutions that included Document and Records Management, Web Content Management, Digital Asset Management, Imaging and Business Process Management, and Risk and Compliance. It also noted that the acquisition would ‘complement and extend Oracle’s existing content management solution portfolio’. Despite the acquisition, the ‘Stellent’ name persisted.

2006 – IBM acquires FileNet

IBM acquired FileNet in 2006. (Source: Wikipedia article on FileNet)

2006 – Google enters the online EDM productivity and collaboration market

In 2006, Google launched Google Apps for Your Domain, a collection of cloud computing, productivity and collaboration tools, software and products. Various apps and elements were acquired and/or added over the years but a key one from an EDM point of view was Google Docs (Wikipedia article). However, Google Docs had no RM capability.

A ZDNet article in June 2007 noted that Google Apps offered a tool for switching from Exchange Server and Lotus Notes, making Google a real alternative to Microsoft and IBM. Google Apps would later be rebranded G-Suite in 2016.

2007 – Spescom exits the EDM market / Enterprise Informatics

In 2007, Spescom exiting the enterprise software sector with the sale of its US operation Enterprise Informatics. (Source – Wikipedia article on Spescom, original reference no longer accessible).

Enterprise Informatics, originally founded in 1981, continued in existence as a subsidiary of Bentley Systems, Incorporated. It continued to market a suite of integrated document, configuration, and records management software products, mostly under the name eB.

2007 – Zoho enters the online EDM collaboration market

The India-based Zoho Corporation, known as AdventNet Inc from 1996 to 2009, released Zoho Docs in 2007.

2007 – HP Autonomy acquires Meridio

Meridio was acquired by HP Autonomy (a company that had had a long business partnership with Kainos) in 2007. The parent company Kainos continued to work with SharePoint-based solutions.

2007 – EDRMS vendors

Forrester released a report into electronic records management vendors in early 2007. The products that it evaluated were as follows:

  • CA MDY FileSurf v7.5 ^
  • EMC Records Manager 5.3
  • IBM FileNet P8 Records Manager v3.7 ^
  • IBM Records Manager v4.1.3 #
  • Interwoven Records Manager v 5.1 *
  • Meridio Document and Records Manager v4.4 *
  • Open Text Livelink ECM – Records Management v3.8 ^
  • Open Text Livelink – eDOCS RM (formerly Hummingbird) v6.0.1
  • Oracle (formerly Stellent) Universal Records Management v7.1 #
  • Oracle Records DB v1.0 ~
  • Tower Software TRIM Context v6.0 *
  • Vignette Records & Documents v 7.0.5
  • (Forrester assessment: ^ = leaders, # = close behind leaders, * = have hurdles to remain competitive’, ~ = basic functionality only)

2008 – NEN 2082

The Dutch government replaced ReMANO with NEN 2082 ‘Eisen voor functionaliteit van informatie- en archiefmanagement in programmatuur’ (‘Requirements for functionality of information and archive management in software’) in 2008 (NEN 2082:2008 nl). NEN 2082 was derived from MoReq, DOD 5015.2 and Australian standards. (See Eric Burger’s blog post ‘Nee, NEN 2082 is geen wettelijke verplichting‘ about its legal standing)

2008 – MoReq2

MoReq2 was published in 2008. It included new sections to support the testing of ERMS software for compliance with the standard. MoReq2 included the following vendors on its panel (Acknowledgements section):

  • Docuware, Germany
  • EDRM Solutions, USA
  • EMC, Canada
  • ErgoGroup AS, Norway
  • Fabasoft, UK
  • FileNet, UK
  • Fujitsu, UK
  • Getronics, UK
  • Haessler Information, Germany
  • IBM, UK
  • ICZ, Czech Republic
  • Lockheed Martin, USA
  • Meridio, UK
  • Objective Corporation, UK
  • Open Text Corporation, UK
  • SAPERION, Germany
  • SER Solutions Deutschland, Germany
  • Tower Software, UK

Both MoReq and MoReq2 were based on the premise of a central ERMS being acquired and implemented by organisations to manage unstructured records, the types of records that are stored across network drives and in email systems. MoReq2 specifically clearly excluded the management of ‘structured data … stored under the management of a data processing application’. (Source: MoReq2, section 1.2 ‘Emphasis and Limitations of this Specification’, page 12.)

The first software product certified against MoReq2 was Fabasoft Folio. It was the only certified product until June 2014.

2008 – EDMS and ERMS

In 2008, the International Standards Organisation, under ISO/TC171/SC2 ‘Document management applications’ proposed a framework for the integration of EDM and ERM systems. The definitions contained in that framework document noted that:

  • An EDMS was used to manage, control, locate and retrieve information in an electronic system.
  • An ERMS was used to manage electronic and non-electronic records according to accepted principles and practices of records management.
  • An integrated EDRMS would combine both capabilities.

Section 6 of the report described general (but fairly detailed) functional requirements for an integrated EDMS/ERMS, outlined in the following diagram:

2009 – Autonomy acquires Interwoven

In 2009, HP Autonomy acquired Interwoven, a niche provider of enterprise content management software mostly to the legal industry. It primarily competed with Documentum in this space. Interwoven became Autonomy Interwoven and Autonomy iManage.

2010 – MoReq2010

MoReq was completely revised and published as MoReq2010 in 2010. There were key differences with its predecessor versions.

  • It de-emphasised, but did not remove, the idea of an ERMS being the central or sole recordkeeping system or repository for organisations.
  • It emphasised the need for line of business systems to incorporate a minimum, defined level of recordkeeping functionality.
  • It brought a degree of practicality about the management of records in other systems.
  • It provided for interoperability between all MoReq compliant systems, based on a common XML language.

MoReq2010 established ‘… a definition of a common set of core services that are shared by many different types of records systems’. It provided a set of modules that could be incorporated into any software solution, including line of business applications, so they can be ‘MoReq compliant records systems’ (MCRS).

2010 – Google’s DM capability enhanced

In March 2010, Google acquired DocVerse, an online document collaboration company. DocVerse allowed multiple user online collaboration on Microsoft Word documents, as well as other Microsoft Office formats, such as Excel and PowerPoint. (Source – Wikipedia article on Google Docs)

2010 – SharePoint gains (some) RM wings

SharePoint 2010 included a range of recordkeeping functionality. According to a web page titled ‘SharePoint 2010: Recordkeeping Considerations‘ by State Records NSW:

‘Microsoft SharePoint 2010 is a software product with a range of uses, including website development, content management and collaboration. SharePoint allows users to collaborate on the creation, review and approval of various types of content, including documents, lists, discussions, wiki pages, web pages and blog posts. SharePoint is not a recordkeeping system (i.e. a system purposely designed to capture, maintain and provide access to records over time). When implemented ‘out of the box’, SharePoint has limited capacities for capturing and keeping records in a way that supports their ability to function as authentic evidence of business.’

In the article ‘Is SharePoint 2010 a compliant records solution‘ in the 30 June 2010 edition of IDM Magazine, the author states the following (emphasis added):

Adam Harmetz, the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft said in a recent online interview about Records in SharePoint 2010, “We constantly get questions from around the world about how to deal with local government and industry standards for information management. Let me throw just a few at you… MOREQ2, VERS, ISO 15489, DOMEA, TNA, ERKS, the list goes on. Some of these standards are loosely based on one another and some have contradictory elements. Rather than focus our engineering efforts on addressing each of these standards in turn, we made the choice to deliver the usability and innovation required to make records management deployments successful and allow our partner ecosystem to build out the SharePoint platform to deal with specific requirements for those customers that are mandated to adhere to a specific standard.”

Despite these comments, SharePoint 2010 was assessed by at least one consultant (Wise Technology Solutions) to meet 88% of the requirements of the then ICA Standard that became ISO 16175 Part 2.

On 16 December 2011, State Records NSW published a blog post titled ‘Initial advice on implementing recordkeeping in SharePoint 2010‘. The post noted that the Wise report had concluded that ‘SharePoint is 88% compliant with the ICA requirements’. It added that the areas where full compliance could not be achieved relate to:

  • ease of email capture
  • native security classification and access control
  • physical and hybrid records management

The report states that third party providers are able to offer products that plug SharePoint’s gaps in these areas.

The blog posted also stated that ‘… the report very clearly makes the point that ‘we note that the achievement of these results is reliant on appropriate design and governance of implementation, configuration and set up to ensure consistency with desired records management outcomes’.’

Early 2010 – Microsoft launches Office 365

Microsoft launched Office 365 on 28 June 2011. Office 365 was designed to be a successor to Microsoft’s Business Productivity Online Suite (BPOS). (Source: Wikipedia article on Office 365). It would not be until the mid 2010s that Office 365 would become an effective counter-solution to the G Suite.

2011 – HP acquires Autonomy

In 2011, Hewlett-Packard acquired Autonomy, a deal that resulted in some interesting subsequent legal issues reading the value of the company.

2011 – OMG Records Management Services standard

The Object Management Group (OMG) released its Records Management Standard (RMS) in September 2011.

As noted in the scope section of the standard, ‘(The) specification provides models for software services to support management activities for electronic records’. Further, ‘… models are provided that describe the platform independent model (PIM) that defines the business domain of Records Management and the RM services to be provided’. Three technology-specific implementations are specified:

  • PSM-1 – Web Services definition for Records Management Services in Web Service Description Language (WSDL). This is actually supplied as ten WSDL files; one for each Records Management Service.
  • PSM-2 – A Records Management Service XSD. The XSD is for use in creating XML files for import/export of Managed Records from compliant environments and to use as a basis for forming XQuery/XPath statements for the query service.
  • PSM-3 – An Attribute Profile XSD. The XSD is for capturing and communicating attribute profiles to permit flexible attribution of certain types of Records Management Objects.

2011 – The death of ERM systems?

In a May 2011 blog post on MoReq2010, James Lappin suggested that traditional systems used to manage electronic documents and records, while not being entirely dead in the water, had ‘lost momentum’.

James proposed two specific reasons for this situation:

  • The global financial crisis (GFC) from 2008 that limited the ability of organisations to acquire and implement hugely expensive ERMS solutions.
  • The rise of Microsoft SharePoint and particularly SharePoint 2010. In some ways, Sharepoint 2010 had the potential to take – and may have already taken – the ERMS wind from the records managers sails.

He also noted that a series of interrelated user-environment issues may have also played a part in the loss of momentum.

  • Usability and take up rates of the ERMS. These solutions are sometimes seen as ‘yet another system’ to manage the same records, using a classification structure that doesn’t make sense to most end users and is different from the way end users see and categorise their world.
  • The ongoing availability of and access to alternative places to store information, including network drives and email folders, and cloud-based storage and email solutions.
  • The rise and general availability of social networking tools and mobile applications used to create and share new forms of information content, and collaborate and communicate, including wikis, blogs, Twitter, Facebook, and similar solutions, often in an almost parallel ‘personal’ world to the official record.
  • The inability of ERMS solutions to manage structured data or to maintain and reproduce easily the diverse range of content created and stored in products like SharePoint. Indeed, one reasonably well known product has been described as an archive for SharePoint, even though the latter can quite easily manage its own archives.
  • The rise of search as a tool to find relevant information in context, and the related change from unstructured to structured in XML-based documents generated by products such as Microsoft Office 2007 and 2010.

From 2013 – GEVER (Switzerland)

From 2013, the Swiss Federal Chancellery was responsible for managing all activities relating to electronic records and process management (Elektronische Geschäftsverwaltung), or GEVER. GEVER consisted of a collection of five standards for the management of electronic records. (Source with current update: Gever Bund)

2015 – Hewlett Packard separates

In October 2015, the software products previously under the Autonomy banner were divided between HP Inc and Hewlett Packard Enterprise (HPE). HP Inc was assigned Autonomy’s content management software components including TeamSite, Qfiniti, Qfiniti Managed Services, MediaBin, Optimost, and Explore.

2015 – EDRMS vendors

Despite the alleged death of ERMS products in around 2010, many continued to thrive and grow. Some were acquired by others.

The following is a list of EDRMS vendors in December 2015 taken from a Gartner report diagram titled ‘Product or Service Scores for Trusted System of Record’ (with the scores included). Many of these products also appeared in the October 2016 ‘Magic Quadrant’ for Enterprise Content Management Systems as indicated)

  • Alfresco (2.47) (also ECM)
  • Open Text (4.07) (also ECM)
  • EMC Documentum (3.94) (as Dell EMC)(Acquired by Open Text)
  • IBM (3.91) (also ECM)
  • HPE (3.85)
  • Oracle (3.45) (also ECM)
  • Laserfiche (2.45) (also ECM)
  • Microsoft (SharePoint) (2.38) (also ECM)
  • Hyland OnBase (2.37) (also ECM)
  • Lexmark (2.32) (also ECM)
  • Newgen (2.18) (also ECM)
  • Objective (2/10) (also ECM)

By 2015 – Oracle departing the scene?

In a July 2015 article titled ‘Looking for an Oracle IPM replacement‘ in the blog softwaredevelopmentforECM, it was noted that Oracle was ‘clearly, and publically, going in a different direction and moving away from traditional enterprise imaging and transactional content management’.

2016 – OpenText, Micro Focus

In May 2016, OpenText acquired HP TeamSite, HP MediaBin, HP Qfiniti, HP Explore, HP Aurasma, and HP Optimost from HP Inc.

2017 – Micro Focus merges with HPE

Micro Focus merged with HPE in September 2017 and in doing so acquired the Content Manager EDRMS (formerly TRIM).

Pre 2021 – EDRM vendors (Australia)

The following is a list of products identified by the Victorian Public Records Office (PROV) in 2020. These products were all certified against the VERS standard, that required organisations to be able to create XML-based VERS Encapsulated Objects (VEOs) for long-term preservation.

  • Alfresco
  • Altus ECM
  • AvePoint RevIM, Records, Cloud Records
  • Bluepoint Content Manager
  • Canon Therefore 2012
  • Docbureau
  • ELOprofessional / ELOenterprise
  • HP Records Manager
  • IBM Enterprise Records
  • IBM FileNet P8 Records Manager
  • Info-Organiser
  • Laserfiche
  • MAGIQ Documents
  • MicroFocus Content Manager
  • Objective
  • Objective ECM
  • OpenText eDOCS RM
  • OpenText Records Management
  • Oracle WebCentre Content
  • RecFind 6
  • RecordPoint
  • Technology One ECM
  • Technosoft OfficeTech

2021 – EDRMS vendors

The following is a list of dedicated vendors that offered EDRMS solutions (and more in most cases) by early 2021. Many of these vendors have a long history not necessarily reflected in the above text. Most of these vendors provide Enterprise Content Management (ECM) services, including EDM and ERM capabilities.

  • Alfresco ECM (
  • Hyland OnBase (
  • IBM ECM (
  • Knowledgeone RecFind EDRM (
  • Laserfiche RME (
  • Lexmark RIM (
  • Micro Focus Content Manager (
  • Microsoft 365 (
  • Newgen RMS (
  • Objective ECM (
  • Open Text ECM (
  • Oracle ECM (
  • TechnologyOne ECM (

2021 – Dedicated EDMS vendors

EDM vendors never went away, but many – like Google Drive, DropBox and Box – were built in and for the cloud. This Capterra website has a fairly detailed listing of current EDMS vendors.

2021 – NEN 2082 withdrawn

In January 2021, the Dutch government withdraw NEN 2082. (NEN 2082:2008 nl)

The future of standards-based ERM/EDRM/ECM systems

Although the definition of a record has remained largely intact for the past two decades – ‘evidence of business activity’ (ISO 15489) – the form of records has evolved and continues to do so.

The ever-expanded world of digital content has made it increasingly difficult to accurately and consistently identify, capture and manage records in all forms, a challenge to the notion that all records can be stored in a single system.

The ‘in place’ approach to managing electronic records – wherever they are stored – has strong appeal. But where will we be in another 20 years? Some thoughts:

  • Electronic databases, whether on-premise or cloud-based (including subscription based), will be the primary method of capturing and storing a wide range of digital content rather than network file shares.
  • Metadata will be automatically captured or auto-generated for all digital content based on the content itself.
  • Artificial Intelligence (AI) will continue to grow in maturity, allowing records to be identified from all other digital content, classified, aggregated, and managed through to disposal/disposition or transfer to archives.
  • Email will, slowly, disappear as the current workforce transitions to chat- and video-based communication methods.