Posted in Electronic records, Office 365 Groups, Products and applications, SharePoint Online, Training and education

Different ways to access content stored in SharePoint

SharePoint Online (SPO) is the primary location to store digital objects and documents in Microsoft 365.

In this sense, it replaces on premise network file shares and drives as a location to store information although a bit ironically, it can also be accessed from File Explorer.

In Microsoft Teams channels, SPO sits behind the scenes via the ‘Files’ tab. This tab presents the content of the folder from the default Documents library that has the same name as the channel (General channel = General folder in the Documents library).

SPO can also be accessed via the SPO app on a mobile device and even directly from Outlook Online.

So, which one is the best way to access your information stored in SPO? The answer is – it depends on what you need to do. You could use all five options.

This post describes five ways to access content stored in SPO, with positives and negatives.

1- For day to day use – synced via File Explorer

Most people use File Explorer to store, organise and access their content, and generally only work from a few folder locations They don’t want to have to open a browser or other application.

The good news is that they can sync SPO document libraries to File Explorer and work directly from there. They can sync a document library from the SPO site or via Teams.

Syncing a library from SPO
Syncing the same library in Teams

So, for most users, the main change will be a different location to access their content from File Explorer. It looks a bit like the image below. The first words after the folder are the site name, then the hyphen, then the library name. To make it as easy as possible, the library should ideally be the same as the old top level folder on the network file share.

Syncing downloads the metadata about the content that is stored the SPO library. The content is not downloaded to the location device (C: drive usually) until it is opened. From that point on, there is a ‘local’ copy. If that local copy is modified, the changes are synced back to the SPO site.

The content of a synced library (example)

End users can share directly from File Explorer. The dialogue box is exactly the same as the SPO/Teams option. This makes it much easier to share rather than attach a document to email.

End users can co-author an Office document opened from File Explorer provided they have the most recent version of Office installed. Other end-users may be accessing the same document at the same time via the online versions of Office in Teams or SPO.

The main negatives about the sync option are as follows:

  • Only basic File Explorer metadata is visible – Name, Date modified, etc. If end-users need to add or see added metadata columns, they will have to access this via the Teams Files tab or the SPO library.
  • More restrictive, granular-level permissions cannot be set (e.g., on a folder or a document). These have to be set from SPO.
  • End-users cannot access the version history or Recycle Bin (to restore deleted items)
  • A folder added at the same level as a Teams channel-mapped folder, will not create a new channel. However, folders created under the Teams channel-mapped folder will be visible.

2 – To collaborate – access via MS Teams

Microsoft have positioned Teams to be an ‘all in one’ collaboration application, allowing end-users to chat, upload and store files, have video calls and more.

In the 1:1 chat area of Teams, the ‘Files’ tab presents documents shared from the OneDrive account of a participant in the chat. OneDrive is, of course, a SharePoint service.

In the Teams area of Teams, the ‘Files’ tab displays, for each channel, a folder with the same name in the Documents library of the Team’s SPO site (every Team has a SPO site). End-users can create, capture and manage content in the channel’s Files tab, as shown in the example below.

Files and folders in the ‘General’ folder of the SPO Documents library

There are several positives of accessing SPO via Teams:

  • Teams includes additional collaboration options, including the ability to chat at the same time as a document is viewed or edited. Content (and folders) can be shared easily.
  • If this library is synced to File Explorer, any changes made in either location will be automatically updated in the other.
  • Any metadata columns added to the SPO document library will be visible here.
  • The SPO site can be accessed directly from a link on the menu bar.

The main negatives of accessing content via the Teams ‘Files’ tab are as follows:

  • End-users have to open and use an unfamiliar interface (although it has become more common)
  • The three dot ‘ellipsis’ menu is limited compared with the full SPO version. For example, it does not include versions. See the screenshot below.
  • The Recycle Bin is not accessible – you have to click on ‘Open in SharePoint’ to access it there.
Ellipsis menu in the Teams Files tab

3 – To see Group emails and files – access via Outlook

Microsoft 365 Groups are a key element in Microsoft 365 and provide a range of functionality that can replace or supplement existing access control and collaboration purposes.

Every Microsoft 365 Group has an Exchange mailbox and a SPO site, and can be linked to a new Team.

  • If the Microsoft 365 Group is created first, the Exchange mailbox is visible to the Group members in their Outlook.
  • If the Team is created first, a Microsoft 365 Group with a SPO site and mailbox are created, but the Exchange mailbox is not visible via Outlook. It is there only to store the compliance copies of chats and for calendaring purposes.

Microsoft 365 Groups can be used to replace shared mailboxes or to give business areas the ability to access both email and SPO-stored content from the same location (as well as via File Explorer and Teams).

In the first screenshot below from Outlook Online, you can see a square ‘documents’ icon to the right of the words ‘Send email’. This square icon opens the Group’s SPO documents library (next screenshot). In the installed version of Outlook, clicking this link opens the SPO site in the browser window.

The Group mailbox in Outlook Online

In the screenshot below, you can see the Group’s files from the Documents library, General folder in Outlook Online.

The Group’s files from Outlook Online

The main positives of accessing SPO content from Outlook Online is that it is relatively easy to move between the Group’s emails and document stored in SPO. End-users can open and documents directly from Outlook, although this (currently) opens Word Online.

The main negative of accessing SPO content from Outlook Online is the limited functionality available from the ellipsis menu, including the inability to see previous versions or access the Recycle Bin. It is also not possible to modify the view (display columns). However, any changes made to the view in either Teams or SPO will be visible in the Outlook Online view.

The ellipsis menu in Outlook

4 – Anywhere, anytime – via mobile devices

Both Google and Apple provide the SharePoint mobile app, as well as apps for OneDrive, MS Teams, Outlook and Microsoft Office.

This means that mobile users can access their SharePoint content directly from a mobile device. They can also use the SharePoint app to search for any content they have access to.

The document library from a mobile device

The main negative with accessing SharePoint from a mobile device is the functionality is very limited. End-users can access and edit the content (if they have the relevant app installed), and can share the documents, but that’s all.

On the other hand, they can access the content anywhere, any time. That makes it very useful.

5 – From the browser – the full SharePoint experience

Of course, the end-user may also access SharePoint from the browser and it is usually a good idea to let them know they can do this for the reasons below.

They can access the browser version in multiple ways:

  • From the end-user SharePoint portal and clicking on the site – https://tenantname.sharepoint.com/_layouts/15/sharepoint.aspx
  • By clicking on ‘Open in SharePoint’ from the Files tab in Teams.
  • By saving the site as a favorite in their browser.
  • By clicking on the files option in a Group’s email inbox area in Outlook installed on the desktop.

The main or common reasons they might want to access the browser version of SharePoint are:

  • To recover a file they deleted (from any of the other locations, including File Explorer), from the Recycle Bin. This option is available for 93 days after the file was deleted. After that point, unless a retention policy has been applied (in which case the document will be in the Preservation Hold library, accessible to admins), the file is gone forever.
  • To see who has been working on a file, from the version history.
  • To see who has viewed a file (when this feature is enabled).
  • To seek approval for, or see who has approved which version of, a document. This functionality comes with every SharePoint library and list.
  • To add additional metadata to the content.
  • To use the full functionality of document sets. Note that these appear as normal folders in a synced document library.
  • To copy or move documents.
  • To check out a document.
  • To search for content and to view content in multiple ways through views.
  • And more.

Summing up

Access to SharePoint has never been easier, but it is a good idea to let end-users know that they can access their SharePoint content in multiple ways.

Some users may rarely access it via Teams (unless they are interested in collaborating more effectively than attaching documents to emails), and even less so via the full SharePoint browser interface. In summary:

OptionBest for
File ExplorerDay to day use where no additional metadata or labels are needed. Sharing (instead of attaching)
MS TeamsCollaboration.
OutlookFor Groups that also use the Group mailbox.
Mobile deviceAnywhere, anytime access
BrowserFor the full set of functionality, including the Recycle Bin, versioning history, viewing, usage information, searching and more.

It is important to let end users know the functionality that they can access in the different areas, especially the version history and Recycle Bin in the browser version of SPO. These alone can be ‘life savers’.

Posted in Artificial Intelligence, EDRMS, Electronic records, Information Management, Microsoft 365, Products and applications, Records management, SharePoint Online, Solutions

A brief history of electronic document and records management systems and related standards

[NOTE – This post may be updated with additional content – last updated 17 February 2021 with information about Optika, Stellent and Oracle]

This post provides a brief history of electronic document and records management systems, and standards that had an impact on the latter.

It notes the following:

  • Early electronic records management systems (ERMS) were mostly mainframe-based databases.
  • Electronic document management systems (EDMS) were developed from the 1980s to manage unstructured electronic information.
  • ERM systems began to include unstructured information from the 1990s, especially with the introduction of DOD 5015.2 and MoReq.
  • The rise of XML from the late 1990s had an impact on methods to preserve electronic information.
  • Content management (CM) and knowledge management (KM) systems appeared in the 1990s.
  • Cloud-based EDM systems, and enterprise content management (ECM) appeared from the early 2000s.
  • Cloud based systems challenged installed on-premise EDRM systems.
  • There is a history of acquisitions. (See Mergers & Acquisitions – This Week – MandAsoft to find more details of any acquisitions).

Sources for this information are listed where this is known.

1973 – Plato Notes

A history of ERM and EDM systems must include reference to Lotus Notes.

Lotus Notes began its life in 1973 as Plato Notes, developed by the Computer-based Education Research Laboratory (CERL) at the University of Illinois in 1973. Elements of the Plato Notes system would be developed for PC by Ray Ozzie during the late 1970s. This was picked up by Lotus Development Corporation and in 1984 became Lotus Notes.

An early version of Lotus Notes was released (under contract to Lotus) in 1984. The original vision included on-line discussion, email, phone books and document databases. Eventually the product fell into the ‘groupware’ category. The capability of the product continued to grow and some organisations only used Notes.

Lotus acquired all the rights to Lotus Notes in 1987 and version 1.0 was released on 7 December 1989.

1974 – Compulink Management Center/Laserfiche founded

Compulink Management Center was founded in the US in 1974. It created Laserfiche, the first DOS-based document imaging system, in 1987.

1976 – Micro Focus founded

Micro Focus was founded in the UK in 1976. Its first software product was CIS COBOL, a solution for micro computers. It entered the EDRM market in 2017, see below.

1981 – Enterprise Informatics founded

Enterprise Informatics, a privately-held software company, was founded in 1981 by early pioneers of the document management industry. (Source: LinkedIn company profile) It would later be acquired by Spescom, a South African company.

1982 – FileNet founded

FileNet was founded in 1982 by Ted Smith, formerly of Basic 4. FileNet’s original focus of attention was the storage and management of scanned images but it also developed a workflow software. (Source: Wikipedia article on FileNet)

1983 – GMB/DocFind founded

GMB (named after the original founders, Gillett, Frank McKenna, and Bachmann) was formed in Australia in 1983. In 1984, GMB released DocFind 1.0. DocFind was renamed RecFind in 1986.

For a very detailed history of this company, see The History of GMB/KnowledgeOne.

1984 – The first EDRMS standards – Noark (Norway)

The Norwegian government developed model requirements for EDRM systems in 1984, the first organisation globally to do so. Noark

(Source: ‘The Norwegian Noark Model requirements for EDRMS in the context of open government and access to governmental information‘, by Olav Hagen Sataslaaten, National Archives of Norway, published in the Records Management Journal 11 November 2014.)

Noark has remained active to the present day. (National Archives of Norway – E-Ark Project (eark-project.com)

1985 – Tower Software founded (Australia)

Tower Software was founded by Brand Hoff in Canberra in 1985 as a software development company. The company provided and supported enterprise content management software, notably its TRIM (Tower Records and Information Management) product line for electronic records management.

The ‘Tower’ in the company name derives from the telecommunications tower on top of Black Mountain (technically a hill, 812 m high) overlooking Canberra. A graphic of the tower was used in the TRIM logo until the company was acquired by HP’s Software Division in 2008 (see also below).

1986 – Autonomy founded (UK)

Autonomy was founded by Michael Lynch, David Tabizel and Richard Gaunt in Cambridge, UK in 1986 ‘as a spin-off from Cambridge Neurodynamics, a firm specializing in computer-based finger print recognition’.

Before 1987 – Saros Corp

Saros Corp was established in Washington by Mike Kennewick (a former Microsoft employee) before 1987. Saros Corp produced Saros Mezzanine, a client-server document management engine. In 1993, released Saros Document Manager.

1989 – Ymijs (later Valid Information Systems) founded – R/KYV (UK)

Ymijs was founded in the UK in 1989. It sold the R/KYV software initially as a basic document imaging processing system. The company name was changed to Valid Information Systems and R/KYV was further developed as a compliance and records management system ‘… that is widely used by major corporations as well as central and local government authorities and related governmental agencies’ (in the UK).

Valid Information Systems was acquired by Hummingbird in 2003. (Source: ‘Legal Technology Insider‘, 23 July 2003.

1989 – Provenance Systems (later TrueArc) founded (Canada)

Bruce Miller, sometimes noted as ‘the inventor of modern electronic recordkeeping software’, founded Provenance Systems in 1989 where he created ForeMost. The company name was changed to TrueArc. Bruce would go on to found Tarian Software as well in 1999.

TrueArc ForeMost RM would be acquired in 2002 by Documentum (which which it had a long-standing technology partnership).

1990 – Documentum founded (US)

According to this Wikipedia article, Documentum was founded in June 1990 by Howard Shao and John Newton who had previously worked at Ingres (a relational database vendor). They sought to solve the problem of unstructured information.

The first Documentum EDMS was released in 1993. According to the Wikipedia article, ‘This product managed access to unstructured information stored within a shared repository, running on a central server. End users connected to the repository through PC, Macintosh, and Unix Motif desktop client applications.’

1992 – Altris Software (UK)

Altris, established in 1992 (Source: Rob Liddell\’s LinkedIn profile. Rob was one of the co-founders of Altris), developed document management systems, including (according to this South African ITWeb post of 26 October 2001), eB, a ‘configuration management’ application. 

Altris would be acquired by the South African Spescom in 2000.

1992 – Optika ImageFiler / FilePower

Optika Imaging Systems (Colorado, US) sought a trademark for a product called ImageFiler in September 1992. The request was abandoned according to information on the Bizapedia site.

This article titled ‘The Case for 11g‘ (referring to Oracle’s product, see below) noted that Optika’s original software development focus was Image and Process Management (IPM).

An undated (but likely mid to late 1990s) webpage on the Property and Casuality website titled ‘Optika and Xerox Package FilePower with Document Centre‘ noted that ‘Optika Imaging Systems, Inc. and Xerox announced that the two companies will jointly work to integrate Optika’s FilePower with Document Centre digital systems products from Xerox. The combination of the Document Centre and FilePower will provide a complete solution for capturing, managing and distributing large volumes of documents, increasing users’ productivity and significantly reducing labor and capital costs. Optika’s integrated product suite — FilePower — combines imaging, workflow and COLD technology into a unified software package. The Xerox Document Centre 220ST and 230ST combine network scanning, printing, faxing and copying into one hardware device.’

1993 – Workflow Management Coalition formed

The Workflow Management Coalition (WfMC), ‘a consortium formed to define standards for the interoperability of workflow management systems’, was founded in May 1993. Original members included IBM, Hewlett-Packard, Fujitsu, ICL, Staffware and approximately 300 software and services firms in the business software sector.

The WfMC’s Workflow Reference Model was published first in 1995 and still forms the basis of most BPM and workflow software systems in use today. (Source: Undated Gutenberg article)

1993 – Kainos Meridio (UK)

Meridio was developed in 1993 by Kainos (a Northern Ireland company and joint venture between Fujitsu and The Queens University in Belfast) as an electronic document and records management (EDRM) system based on Microsoft products. It would be acquired by HP Autonomy in 2007.

1993 – Saros (US) Document Manager

Saros Corp released Saros Document Manager in mid 1993. The product was said ‘to act as a front-end to the Bellevue, Washington-based firm’s client-server document management engine, Saros Mezzanine’. (Source: Computer Business Review article ‘Saros Sets Document Manager‘ )

ERM before the mid 1990s

Before the arrival of personal computers in offices in the early 1990s, computer mainframes and databases were the regarded by some observers as the only places where electronic ‘records’ (in the form of data in tables) were stored and managed.

A report by the United States General Account Office in July 1999 (GAO/GGD-99-94) titled ‘Preserving Electronic Records in an Era of Rapidly Changing Technology’) stated that, historically (as far back as 1972), NARA’s Electronic Records Management (ERM) guidance (GRS 20) was geared towards mainframes and databases, not personal computers.

The GAO report noted that until at least the late 1990s, there was a general expectation that all other electronic records not created or captured in ERM systems would be printed and placed on a paper file or another system. The original (electronic) records could then be destroyed.

Some early ERM (database) systems, such as TRIM from Tower Software in Australia, were originally developed in the mid 1980s to manage paper files and boxes. Similar systems were developed to manage library catalogues and the old card catalogues started to disappear.

But, although some of it was printed and filed, the volume of electronic records in email systems and stored across network file shares continued to grow. Several vendors released systems that could be used to manage electronic documents (EDM) more effectively than network drives but there was no agreed standard for managing that content as records.

1994 – The DLM Forum and MoReq

From the early 1990s, the European Council sought to promote greater cooperation between European governments on the management of archives. One of the outcomes of a meeting in 1996 was the creation of the DLM Forum. DLM is the acronym of the French term ‘Données Lisibles par Machine’, or ‘machine-readable data’.

One of the ten action points arising from the June 1994 DLM meeting was the creation of ‘Model Requirements for the Management of Electronic Records’, or MoReq, first published in 2001 (see below).

Mid 1990s – Two rival records management models

According to James Lappin, in his paper ‘Rival records management models in an era of partial automation‘ published on 22 January 2021, two rival records management models appeared in the mid 1990s.

North America – separate records repository model 

According to its website, ‘the InterPARES Project was borne out of previous research carried out at the University of British Columbia’s School of Library, Archival and Information Studies. “The Preservation of the Integrity of Electronic Records” (a.k.a. “The UBC Project”) defined the requirements for creating, handling and preserving reliable and authentic electronic records in active recordkeeping systems.’

‘The UBC Project researchers, Dr. Luciana Duranti and Professor Terry Eastwood, worked in close collaboration with the U.S. Department of Defense Records Management Task Force to identify requirements for Records Management Applications (RMA).

The work of the UBC team influenced the development of DOD 5015.2 published in 1997 (see below) and the subsequent development of a range of electronic document and records management (EDRM) systems.

Australia – intervention in business applications model

In 1996, the University of Pittsburgh published the ‘Functional Requirements for Evidence in Recordkeeping Project’, led by David Bearman. This work would influence the development of both MoReq2010 and the ICA standards that became ISO 16175-2010, both of which attempted to define a minimum set of functional requirements for a business application to be able to manage its own records. (Lappin)

1995 – IBM Acquires Notes

Lotus Notes was acquired by IBM in July 1995. By December 1996 it had 20 million users. By the end of 1999, Lotus Notes had extensive capability including ERM and EDM.

Lotus Notes continued to retain a strong presence in the market but its dominance began to be reduced by the arrival of Microsoft’s broader capabilities and other EDM solutions.

1995 – Alpharel (US) acquires Trimco (UK)

According to this Computer Business Review article of 23 November 1995, Alpharel Inc, San Diego was expected to acquire Trimco Group Plc of Ealing, London, a supplier of enterprise-wide document management systems.

1995 – FileNet acquires Saros

FileNet acquired Saros Corporation in 1995 to acquire its electronic document management capability. It was said to have pioneered ‘integrated document management’ (IDM), through a suite that offered document imaging, electronic document management, COLD and workflow. (Source: Wikipedia article on FileNet)

1996 – Australian Standard AS 4390

In February 1996 Australia issued the world’s first national records management standard, AS4390 ‘Records Management – General‘. The standard provided guidance for the implementation of records management strategies, procedures and practices.

Tower Software, the Canberra-based developers of TRIM, contributed to the development of the standard (according to its Wikipedia entry) although the standard did not prescribe requirements for the management of electronic records.

AS 4390 would become internationalised through ISO 15489 in 2002.

1996 – OpenText Corporation (US) – Livelink

OpenText Corporation was founded in 1991 from OpenText Systems. It released Livelink in 1996.

1996 – Various EDM solutions

The March 1996 edition of Engineering Data Management included a number of updates on electronic document management solutions in the market at that time. Note that Trimco and Alpharel are listed separately; this may because Alpharel’s acquisition of Trimco had not been completed by that time.

  • Alpharel (San Diego, CA): Document Management solutions – Enabler, FlexFolder, RIPS, Toolkit API. Wisdom, a product that facilitated internet access to participating electronic document vaults.
  • Auto-trol Technology (Denver, CO): CENTRA 2000, document management, workflow, PDM, change management and messaging.
  • Cimage Enterprise Systems (Bracknell, UK): Document Manager for Windows.
  • Documentum, Inc. (Pleasanton, CA): Documentum Accelera for the World Wide Web and Documentum UnaLink for Lotus Notes.
  • Interleaf (Waltham, MA): Interleaft 6 SGML, a solution for publishing SGML doocuments. Intellecte/BusinessWeb, a document management solution that allowed organisations to access enterprise document repositories from the internet.
  • Trimco (Ealing, UK): Document management systems.

Alpharel changed its name to Altris Software (US) in October 1996, according to this Telecompaper article published the same month.

From 1996 – Germany’s DOMEA project

In 1996, the Coordinating and Advising Agency of the Federal Government for Information Technology in the Federal Administration (KBSt) introduced a pilot project named Document Management and Electronic Archiving in computer-assisted business processes (DOMEA).

Under the framework of DOMEA, a project group was set up in 1998 to find solutions for the disposition and archiving of electronic records. The goal was to find a suitable and efficient way for the disposition of electronic records created and maintained in office systems. Its “Concept for the Disposition and Archiving of Electronic Records in Federal Agencies,” containing recommendations for managing electronic records was published in September 1998. (Source: The Free Library article)

Late 1990s – EDMS vs ERMS

Electronic document management systems (EDMS) and electronic records management systems (ERMS) were regarded as separate types of system from the late 1990s until at least 2008.

According to Philip Bantin in August 2002:

  • An EDMS was said to support day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
  • An ERMS was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted. (The concept of ‘declaring a record’ may be related to this point).

(Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001)

The difference between the two types of system endured for at least a decade. By the end of the 1990s, four main EDRMS options had emerged:

  • Extending an existing EDM product capability to include ERM.
  • Extending an existing ERM capability to include EDM.
  • Creating new ERM products (technically also with some EDM capability).
  • Integrating separate EDM and ERM products.

1997 – DOD 5015.2

According to the 1999 GAO report quoted above, for several years prior to 1997, NARA worked with the US Department of Defense, considered ‘one of the agencies that is most advanced in its ERM efforts’.

The outcome of this work was the release in November 1997 of the DOD standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’ usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, 11 April 1997.

The GAO report stated that ‘ERM information systems that were in place before the approval of this standard must comply with the standard by November 1999’.

It added that US agencies ‘were confronted with many ERM challenges’ from the ever-increasing volume of digital records, including the ability to preserve and access those records over time. The ‘Year 2000 problem’ was drawing attention away from the issue.

Nevertheless, by 2 June 1999, nine companies were certified as compliant with the DOD standard. Some, it noted, were standalone ERM software, while others were an integrated solution.

An interesting small note on page 11 of the GAO report noted that ‘it is important that ERMS software requires users to make no more than two or three extra keystrokes, and that users realize there is a benefit to this additional ‘burden’.

From 1997 – SER eGovernment (Germany)

SER eGovernment was developed for the German/Austrian market following the release of the German eGovernment standard, DOMEA in 1997.

1998 – Documentum goes online

In 1998, Documentum released its Web Application Environment, a set of internet extensions for EDMS, offering web access to documents stored within an EDMS repository. Various additional products were acquired and their functionality added to the Documentum system.

1998 – Optika eMedia released

Optika released eMedia, ‘a software and methodology product designed to manage business transactions within an organization, across extranets, and throughout the supply chain’, in late 1998. (Source ‘Optika Delivers App to Manage Business Transactions‘) Optika eMedia was said to be ‘a workflow enabled replacement for an imaging solution named FilePower’.

1998 – FileNet Panagon suite released

In 1998, FileNet released its Panagon suite of products. This included Panagon Content Services that was previous Saros Mezzanine. (Source: Wikipedia article on FileNet)

1999 – International differences

The 1999 GAO report noted differences between the US, UK, Australia and Canada on their approach to ‘common ERM challenges’.

  • Australia was said to have ‘strong central authority (including for compliance audits) and decentralised custody’ (except when the records are transferred to permanent retention).
  • Canada had ‘vision statements rather than specific policies’ and also had decentralised custody, but agencies could transfer records at any time to the archives.
  • The UK had broad guidelines put into practice by individual agencies.

1999 – the UK PRO standard released

The UK Public Records Office (PRO, later The National Archives, TNA) released a standard in 1999 designed ‘to provide a tool for benchmarking the ability of government departments to support electronic records management’. This standard would be replaced by TNA 2003. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)

End of the 1900s – XML

By the end of the 20th century it was becoming clear (to some) that XML would likely play a strong role in the standardisation of electronic record formats and their management over time.

XML-based record structures meant that electronic records could contain their own ‘metadata payloads’ rather than being independent objects defined in a separate system (like a library catalogue describes books on shelves).

The establishment of XML-based formats would (after about 20 years) begin to change the way in which records would be managed, although paper records and the paradigm of managing records in pre-defined containers would continue to persist, largely because of the standards developed to manage electronic records – in particular DOD 5015.2.

1999 – EDM/early ERM products

The following is a collated list of EDM (and related) products collated in November 1999:

  • Autonomy Portal in a Box
  • CompuTechnics (1990 to 1999)/Objective (from 1999)
  • DOCS
  • Eastman Software (Imaging, OCR)
  • EDM Software (UK) and SER (Germany)
  • Excalibur RetrievalWare
  • FileNet, incorporating Panagon Integrated Document Management
  • Fulcrum (acquired by PC DOCS in 1997)
  • GMB (later KnowledgeOne) RecFind/RecQuery
  • Hummingbird (from 1999 with acquisition of PC DOCS)
  • Insight Technologies Knowledge Server (IKS) / Document Management System (DMS)
  • Intraspect Knowledge Server (IKS) (KM)
  • Lotus Notes
  • Onyx Enterprise Portal, with integration to various EDM applications
  • Open Text Livelink
  • Pitney Bowes Digital Document Delivery (D3)
  • Oracle
  • Osyssey (ISYS)
  • PC DOCS (acquired by Hummingbird in early 1999)
  • ReadSoft (OCR processing)
  • RecordManager
  • Tower Software / TRIM Captura
  • Tower Technology
  • Verity (KM)

1999 – Tarian Software founded

Tarian Software was founded in Canada in 1999 by Bruce Miller, the founder of Provenance Systems (later TrueArc) and creator of ForeMost. Tarian developed the Tarian eRecordsEngine, an embedded electronic recordkeeping technology for business application software. Tarian was the first e-Records technology in the world to be certified against the revised 5015.2 June 2002 standard. Tarian was acquired by IBM in 2002.

1999 – The Victorian Electronic Records Strategy (VERS)

The (Australian) Victorian government’s Public Records Office (PROV) published a standard for the management of electronic records in 1999, Standard 99/007 ‘Standard for the Management of Electronic Records’. The standard, usually known as VERS, defined the (XML-based) format required for the transfer of permanent records to the PROV.

The Standard noted that:

  • Records must be self-documenting. It is possible to interpret and understand the
    content of the record without needing to refer to documentation about the system in
    which it was produced
  • Records must be self-contained. All the information about the record is contained
    within the record itself
  • The record structure must be extensible. It must be possible to extend the structure of
    the record to add new metadata or new record types without affecting the
    interoperability of the basic structure.

Several EDRMS vendors developed the capability to create VERS encapsulated objects (VEOs) as required by the standard.

2000 – Spescom (South Africa) acquires Altris (UK)

The South African company Spescom acquired the UK firm Altris Software in 2000, as noted in this (South Africa) ITWeb article of 3 May 2000. Altris was described in the article as ‘a global leader in integrated electronic document management software, with well established channels to international markets’. As a result of this acquisition, Altris UK was renamed Spescom Ltd (UK).

The same journal announced in 2001 that Spescom KMS was ‘the UK operation of Spescom Limited’s US based subsidiary, Altris Software Inc, which specialises in the provision of asset information management software to markets including transportation, utilities and telcos’.

From 2000 – Microsoft adopts XML for Office documents

In 2000, Microsoft released an initial version of an XML-based format for Microsoft Excel, which was incorporated in Office XP.

In 2002, a new file format for Microsoft Word followed. The Excel and Word formats, known as the Microsoft Office XML formats (with an ‘x’ on the end of the document extension), were later incorporated into the 2003 release of Microsoft Office.

Microsoft’s XML formats, known as Open Office XML, later became ECMA 376 in 2006 and later ISO 29500 in 2008 ‘amid some controversy’ over the need for another XML format (see below).

Before 2001 – Intranet Solutions (later Stellent)

Intranet Solutions had developed software called’IntraDoc!’. The product was briefly renamed Xpedio! before the company and product were renamed Stellent in 2001. (Source: ‘Wikipedia article on Oracle Acquisitions‘)

2001 – EDM systems with RM functionality

The following is a list of ‘EDM systems with records management’ functionality available by early 2001:

  • TRIM (Tower Software, Australia) – integrated ERM and EDM.
  • ForeMost Enterprise v2 (TrueArc Inc) – ERMS
  • Tarian eRecords Engine v1.0 (formerly e-Records v1.0)(Tarian) – ERMS
  • iRIMS 2001 (Open Text Corp) – ERMS
  • Objective 2000 (Objective Corp, Australia) – ERMS
  • Hummingbird RM Family 4.0 (Hummingbird, Canada) – ERMS (later acquired by Open Text)
  • FileSurf 7.0 (MDY Advanced Technologies) – ERMS
  • R2M (Relativity Records Manager v2 (Relativity Inc) – EDMS with added records management functionality.
  • eManage 2000 (ByteQuest Technologies) – knowledge management (KM) integrated with records management.
  • FileNET IDM Content Services 5.1.1/ForeMost Enterprise 2.0 – EDMS with integrated records management.
  • IBM e-Records Solution (IeRS) v1.0 – IBM’s content management (CM) solution integrated with Tarian eRecords v1.0.
  • Work Manager/ForeMost (Eastman Software) – combined the DM, imaging and workflow of Work manager with the RM capabilities of ForeMost.
  • (Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001).

Many of the products above were sold internationally on the basis of their compliance with DOD 5015.2 even though that was a US-specific standard.

2001 – Birth of SharePoint

Microsoft released SharePoint in 2001. It was born out of a combination of Office Server and Tahoe.

  • Office Server provided the simplest of network administration actions
  • Tahoe expanded on the email, calendaring, contacts, and to-do list technology contained within Exchange.

However, it was not until the release of Office 2003 that SharePoint (rebranded as Windows SharePoint Services) provided collaboration and other functionality.

(Source: History of SharePoint from PortalFront hosting)

Early 2000s – The DM, KM and CM mix

In addition to the EDMS/ERMS differences, organisations were also seeking solutions for knowledge management (KM) and content management (CM).

CM solutions were usually portal-based options that mostly became some form of intranet.

Some of the options in the early 2000s included:

  • Hummingbird’s PowerDOCS for DM and CyberDOCS as the web client for the DM solution, along with Hummingbird’s (formerly Fulcrum) Knowledge Server for KM and PD Accord for web-based collaboration, with the Hummingbird Enterprise Information Portal (EIP) as the portal solution. Plumtree Corporate Portal could also be used as an Enterprise Portal.
  • iManage’s DeskSite for DM and WorkTeam for collaboration. For KM, WorkKnowledge Server and Concept Search (based on the Autonomy Server). The portal to link all of these was called WorkPortal.
  • Open Text’s Livelink for DM, KM and collaboration.
  • Elite’s Encompass, built on Microsoft’s new SharePoint Portal Server (SPS).
  • Autonomy Server for KM, with Plumtree Corporate Portal as the portal.
  • Documentum’s DM and CM product coupled with Plumtree Corporate Portal.

Digital Asset Management (DAM) systems, used to manage other types of digital content such as photographs, also appeared around this time.

Early 2000s – RM meets AI

Don Lueders noted in his post An Industry Pioneer on Records Management and Artificial Intelligence (nextgenrm.com) that ‘in the early 2000’s TruArc released a product called AutoRecords. AutoRecords was the very first records management solution brought to market that leveraged artificial intelligence to automatically classify electronic records.’

2001 – EDMS Vendors

Information Technology Decisions published a paper on DOD 5015.2 certified products in November 2001 (original source/location has been lost). It noted that there were two types of products:

  • Products that started life as electronic document management (EDM) systems. Examples included Documentum, Livelink, and DOCS Open.
  • Products that started life as electronic recordkeeping (ERK/ERM) systems. Examples given included Tower Software’s TRIM, Foremost, iRIMS, Cuadra Star.

The presentation noted that DOD 5015.2 certification was based on alternative options:

  • Standalone. For example, True Arc Foremost, TRIM, iRIMS, Cuadra Associates Star, Relativity Records Manager, Hummingbird RM 4.0, Tarian eRecords, MDY/FileSurf, Cimage and Access Systems, Highland Technologies Highview-RM, Open Text, Livelink
  • Partnership. For example, Saperion with e-Manage 2000, Impact Systems eRecords Manager, FileNet with Foremost.

The report included three interesting points:

  • Both EDMS and ERKS required an enterprise view of information.
  • An EDMS is driven by business process requirements.
  • An ERKS (ERMS) is driven by enterprise requirements for the long-term preservation of information.

2001 – The first MoReq

The first version of MoReq was published in 2001. Volume 1 was 500 pages long.

MoReq emphasised the central importance of an electronic records management system, or ERMS. Its stated purpose was:

  • To provide guidance to organisations wishing to acquire ERMS.
  • As a tool to audit or check an existing ERMS.
  • As a reference document for use in training or teaching.
  • To guide product development by ERMS suppliers and developers.
  • To help define the nature of outsourced records management solutions.

Few, if any, products were certified against this version of MoReq.

2002 – Optika Acorde

Optika eMedia was rebranded to Optika Acorde in 2002, according to this website ‘The Case for 11g‘.

A June 2002 Gartner report titled ‘Optika Acorde Document Imaging, Workflow and Collaboration Suite‘ noted that Optika Acorde was an ‘integrated software family for managing the content associated with business transactions’ leveraging ‘Optika’s core strengths in document imaging, workflow and enterprise report management.’

2002 – FileNet BrightSpire, later P8 ECM

FileNet released BrightSpire in 2002. This product ‘leveraged the experience gained from integrated document management, web content management and workflow into what became ECM. (Source: Wikipedia article on FileNet)

By 2002 – Enterprise Content Management (ECM)

The term ‘Enterprise Content Management’ (ECM) began to appear more frequently by 2002. The Wikipedia post on ECM noted that ECM technologies descended from ‘electronic Document Management Systems (DMS) of the late 1980s and early 1990s’.

ECM promised:

  • The integration of records management (RM) with business practices.
  • The capability for integration between RM products, EDM, various other digital products (such as OCR/character recognition technologies), and web publishing products.
  • Incorporation of Knowledge Management (KM) concepts.

The key word here was ‘integration’ with EDM and other systems, rather than standalone systems. Web-based access became increasingly essential. IBM’s acquisition of Tarian, Documentum’s acquisition of TrueArc’s Foremost were examples of these integrations. (see below)

According to the Wikipedia article on ECM: ‘Before 2003, the ECM market was dominated by medium-sized independent vendors which fell into two categories: those who originated as document management companies (Laserfiche, Saros, Documentum, docStar, and OpenText) and began adding the management of other business content, and those who started as web content management providers (Interwoven, Vignette, and Stellent) tried to branch out into managing business documents and rich media’.

The emergence of ECM quite possibly created the first challenge to centralised ERM through the integration of multiple elements, some of which created, captured or stored records in ever increasing formats.

2002 – OpenDocument XML format

According to the Wikipedia article on the OpenDocument standard, the OpenDocument standard was developed by a Technical Committee (TC) under the Organization for the Advancement of Structured Information Standards (OASIS) industry consortium. Sun and IBM apparently had a large voting influence but the standardization process involved the developers of many office suites or related document systems. The first ODF-TC meeting was held in December 2002.

2002 – An updated GAO report into electronic records

The US General Accounting Office (GAO) released a new report in June 2002 titled ‘Information Management: Challenges in Managing and Preserving Electronic Records’ (GAO-02-586). This report, which was more detailed than the earlier 1999 one, noted among other things that:

  • The DOD had by March 2002 certified 31 applications against standard 5015.2.
  • Progress had been made on the development of the Open Archival Information System (OAIS) model which, while initially developed by NASA for archiving the large volume of data produced by space missions, could be applied to ‘any archive, digital library or repository’. XML-based solutions were considered the most likely to be accepted.

2002 – IBM acquires Tarian Software

IBM acquired Tarian Software in late 2002, ‘bolstering its leadership in ECM’.

From that date, IBM released the IBM Records Manager Version 2.0 (IRM), previously known as the Tarian eRecords Engine (TeRe). Tarian’s e-Records management technology was integrated into IBM’s software offerings, including IBM Content Manager, DB2 database and Lotus software. (Source: IBM press release)

2002 – Documentum 5 and TrueArc Foremost acquisition

Documentum released Documentum 4i, its first Web-native platform, in 2000. In 2002, it launched Documentum 5 as ‘a unified enterprise content management (ECM) platform for storing a virtually unlimited range of content types within a shared repository’.

Documentum acquired TruArc’s Foremost product in October 2002. The Documentum Wikipage above noted that this acquisition ‘added records management capabilities and augmented Documentum’s offerings for compliance solutions.’ The press release cited in this paragraph noted that ‘Documentum and TrueArc are existing technology partners and have worked together to provide an integration for TrueArc’s enterprise-scalable records management solution with the Documentum ECM platform.’

2003 – TNA 2003

The National Archives (TNA) released an updated version of its PRO standard in 2003, known as TNA 2003. This standard would be superseded by MoReq2. (Source: @ZenInformation on Twitter, 12 February 2021).

2003 – EMC acquires Documentum

EMC acquired Documentum in 2003. (Source: EMC offers $1.7 billion in stock for Documentum, October 2003))

2003 – FileNet P8 Records Manager

FileNet released P8 Records Manager in 2003. (Source: Wikipedia article on FileNet)

2003 – Hummingbird acquires Valid Information Systems R/KYV

Hummingbird acquired Valid Information Systems R/KYV product in 2003.

(Source: ‘Legal Technology Insider‘, 23 July 2003)

2003 – Open Text acquires SER eGovernment

In October 2003, Open Text acquired the (German) DOMEA-certified SER eGovernment Deutschland GmbH, based in Berlin, Germany as well as SER Solutions Software GmbH, based in Salzburg, Austria. (Source: Open Text Acquires SER eGovernment)

From 2003 – CNIPA (Italy)

The Italian Centro Nazionale per l’Informatica nella Pubblica Amministrazione (CNIPA) published a protocol for the management of electronic records, Protocollo Informatico in 2003.

CNIPA was renamed DigitPA in 2009 and Agenzia per I’Italia digitale (AGID) in 2012. AGID is responsible for defining standards for the management of electronic records in Italian government agencies. (Source: Protocollo Informatico)

Mid 2003 – The challenges of Enterprise Records Management

In Industry Trend Reports of May 2003, Bruce Silver (of Bruce Silver Associates) made the case for Enterprise Records Management in the wake of various ‘scandals’ involving the management of records at the time, including Enron/Anderson.

Silver argued that EDM, email archive, and back-up solutions did not meet the ‘new statutory and regulatory records management requirements’ – DOD 5015.2, SEC Rules 17a-3 and 17a-4, NASD Rules 2210, 3010, and 3110, NYSE Rules 342 and 440, ISO 15489 and MoReq.

Silver also noted that an effective (‘total’) ERM solution would ‘be implemented as an extension of the company’s ECM infrastructure’, providing for a single interface for all records stored in multiple locations ‘including third-party document management repositories in addition to the email system and network file system’.

2003 – Key integrated EDM/RM vendors

The following is a list of ‘key vendors in the (Integrated Document Management) IDM Market Space’ in October 2003. The list is believed to have come from a Butler Group report:

  • Documentum 5. EDMS, web content, ERM, RM (through acquisition), collaboration, DAM.
  • FileNet P8 Image Services. Content management, web content, image management, RM.
  • Gauss VIP Enterprise 8. Web content, DM, Imaging, ERM. (Acquired by Open Text)
  • Hummingbird Enterprise 5. Full IDM suite (DM, KM, portal, collaboration, workflow, imaging, RM)
  • Hyland Software OnBase. Integrated IDM but lacked strong web content, RM and collaboration.
  • IBM Content Manager. ERM and worfklow.
  • Identitech FYI. Integrated IDM.
  • Information Management Research (IMR) Alchemy. IDM.
  • Laserfiche Document Imaging. Integrated IDM including EDM, ERM.
  • Legato ApplicationXtender, EmailXtender, DiskXtender. Integrated IDM.
  • Open Text Livelink.
  • Optical Image Technology (OIT) DocFinity. Integrated IDM.
  • Optika Acorde. Primarily for images and ERM. Not strong CM, RM or collaboration. Acquired Select Technologies for RM.
  • Tower Technology IDM, Seraph. Integrated IDM.
  • Westbrook Fortis Suite, File Magic. Integrated IDM.

2004 – Stellent acquires Optika

Stellent acquired Optika in early 2004. At this stage, Stellent’s primary product was Universal Content Management (UCM). (Source: Wikipedia article on Oracle Acquisitions)

The report of the sale in The Register stated that Stellent’s CEO said that ‘customers are looking to consolidate their content management needs, including imaging, business process management, web content management and record management with one vendor.’ The new product line was named Stellent Imaging and Business Procss Management (IBPM). The article also noted that Oracle would probably acquire Stellent following this acquisition (see 2006, below).

2004 – ReMANO (Netherlands)

In 2004, the Netherlands government established a catalogue of software specifications for ERM systems (ReMANO) used by Dutch government bodies. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)

ReMANO was replaced by NEN2082 – Eisen voor Functionaliteit van Informatie- en Archiefmanagement in programmatuur” in 2008. (NEN 2082:2008 nl)

2005 – C6 (France) builds D2 on top of EMC Documentum

The French ECM company C6 built a solution named D2, ‘a fully configurable web application for creating, managing, storing and delivering any type of information’, on top of EMC’s Documentum. (Source: C6 website ‘Company’ tab).

2005 – ELAK (Austria)

A project to introduce electronic filing (ELAK) commenced in Austria late 2001 and was completed in 2005. The solution was based on standard software products (e.g. Fabasoft e-Gov-Suite, MS-Office). (Source (ELAK – The e-filing system of the Austrian Federal Ministries)

2006 – The National Archives of Australia ERMS standard

The National Archives of Australia (NAA) released its ‘Functional Specifications for Electronic Records Management Systems Software in February 2006. (ISBN 1 920807 34 9). The introduction noted that:

  • (The document) provided Australian Government agencies with a set of generic requirements for ensuring adequate recordkeeping functionality within electronic records management systems (ERMS) software.
  • Agencies were encouraged to make use of the ERMS specifications when designing or
    purchasing new, or upgrading existing, ERMS software. They could also
    be used when auditing, assessing or reviewing an agency’s existing ERMS software.
  • The requirements were not intended to be a complete specification, but
    rather provide a template of key functional requirements that agencies may
    incorporate into their tender documentation when preparing to select and purchase
    new ERMS software. Agencies were expected to assess and amend the
    functional requirements, and select requirements that best suit their own business
    and technical requirements and constraints.

Very few products met the specific requirements of the ERMS specifications which led to some suggestion at the time that it limited choice.

2006 – Rival XML Office document standards

The OpenDocument (ODF) standard was published as ISO/IEC 26300 in 2006.

Microsoft submitted initial material to the Ecma International Technical Committee TC45, where it was standardized to become ECMA-376, approved in December 2006. It was released as ISO 29500 in 2008.

According to the Wikipedia article on Open Office XML (OOXML), ‘The ISO standardization of Office Open XML was controversial and embittered’, as it seemed unnecessary to have two rival XML standards.

2006 – The world of collaboration

The Butler Group published a paper titled ‘Document Collaboration – Linking People, Process and Content’ in December 2006. The report noted that EDM systems had helped improve internal efficiency but there was now a need to ‘extend these systems to partners and stakeholders’ and deliver ‘sophisticated collaborative experiences’.

The paper listed the following EDM products:

  • Adobe Acrobat family
  • EMC eRoom
  • IBM Notes/Domino, Workplace collaboration services, QuickPlace
  • Microsoft Office 2007
  • Open Text Livelink ECM – eDOCS (incorporating the former Hummingbird product suite acquired by Open Text in 2006)
  • Oracle Collaboration Suite, Content DB and Records DB
  • Stellent Collaboration Management
  • Vignette

2006 – Spescom Software Inc

A US SEC submission in January 2006 noted that Spescom Software Inc, a San Diego-based provider of computer integrated systems was the successor to Alpharel Inc and Altris Software Inc.

2006 – Oracle acquires Stellent

A 2006 Oracle press release titled ‘Oracle Buys Stellent‘ stated that Stellent was a global provider of enterprise content management (ECM) software solutions that included Document and Records Management, Web Content Management, Digital Asset Management, Imaging and Business Process Management, and Risk and Compliance. It also noted that the acquisition would ‘complement and extend Oracle’s existing content management solution portfolio’. Despite the acquisition, the ‘Stellent’ name persisted.

2006 – IBM acquires FileNet

IBM acquired FileNet in 2006. (Source: Wikipedia article on FileNet)

2006 – Google enters the online EDM productivity and collaboration market

In 2006, Google launched Google Apps for Your Domain, a collection of cloud computing, productivity and collaboration tools, software and products. Various apps and elements were acquired and/or added over the years but a key one from an EDM point of view was Google Docs (Wikipedia article). However, Google Docs had no RM capability.

A ZDNet article in June 2007 noted that Google Apps offered a tool for switching from Exchange Server and Lotus Notes, making Google a real alternative to Microsoft and IBM. Google Apps would later be rebranded G-Suite in 2016.

2007 – Spescom exits the EDM market / Enterprise Informatics

In 2007, Spescom exiting the enterprise software sector with the sale of its US operation Enterprise Informatics. (Source – Wikipedia article on Spescom, original reference no longer accessible).

Enterprise Informatics, originally founded in 1981, continued in existence as a subsidiary of Bentley Systems, Incorporated. It continued to market a suite of integrated document, configuration, and records management software products, mostly under the name eB.

2007 – Zoho enters the online EDM collaboration market

The India-based Zoho Corporation, known as AdventNet Inc from 1996 to 2009, released Zoho Docs in 2007.

2007 – HP Autonomy acquires Meridio

Meridio was acquired by HP Autonomy (a company that had had a long business partnership with Kainos) in 2007. The parent company Kainos continued to work with SharePoint-based solutions.

2007 – EDRMS vendors

Forrester released a report into electronic records management vendors in early 2007. The products that it evaluated were as follows:

  • CA MDY FileSurf v7.5 ^
  • EMC Records Manager 5.3
  • IBM FileNet P8 Records Manager v3.7 ^
  • IBM Records Manager v4.1.3 #
  • Interwoven Records Manager v 5.1 *
  • Meridio Document and Records Manager v4.4 *
  • Open Text Livelink ECM – Records Management v3.8 ^
  • Open Text Livelink – eDOCS RM (formerly Hummingbird) v6.0.1
  • Oracle (formerly Stellent) Universal Records Management v7.1 #
  • Oracle Records DB v1.0 ~
  • Tower Software TRIM Context v6.0 *
  • Vignette Records & Documents v 7.0.5
  • (Forrester assessment: ^ = leaders, # = close behind leaders, * = have hurdles to remain competitive’, ~ = basic functionality only)

2008 – NEN 2082

The Dutch government replaced ReMANO with NEN 2082 ‘Eisen voor functionaliteit van informatie- en archiefmanagement in programmatuur’ (‘Requirements for functionality of information and archive management in software’) in 2008 (NEN 2082:2008 nl). NEN 2082 was derived from MoReq, DOD 5015.2 and Australian standards. (See Eric Burger’s blog post ‘Nee, NEN 2082 is geen wettelijke verplichting‘ about its legal standing)

2008 – MoReq2

MoReq2 was published in 2008. It included new sections to support the testing of ERMS software for compliance with the standard. MoReq2 included the following vendors on its panel (Acknowledgements section):

  • Docuware, Germany
  • EDRM Solutions, USA
  • EMC, Canada
  • ErgoGroup AS, Norway
  • Fabasoft, UK
  • FileNet, UK
  • Fujitsu, UK
  • Getronics, UK
  • Haessler Information, Germany
  • IBM, UK
  • ICZ, Czech Republic
  • Lockheed Martin, USA
  • Meridio, UK
  • Objective Corporation, UK
  • Open Text Corporation, UK
  • SAPERION, Germany
  • SER Solutions Deutschland, Germany
  • Tower Software, UK

Both MoReq and MoReq2 were based on the premise of a central ERMS being acquired and implemented by organisations to manage unstructured records, the types of records that are stored across network drives and in email systems. MoReq2 specifically clearly excluded the management of ‘structured data … stored under the management of a data processing application’. (Source: MoReq2, section 1.2 ‘Emphasis and Limitations of this Specification’, page 12.)

The first software product certified against MoReq2 was Fabasoft Folio. It was the only certified product until June 2014.

2008 – EDMS and ERMS

In 2008, the International Standards Organisation, under ISO/TC171/SC2 ‘Document management applications’ proposed a framework for the integration of EDM and ERM systems. The definitions contained in that framework document noted that:

  • An EDMS was used to manage, control, locate and retrieve information in an electronic system.
  • An ERMS was used to manage electronic and non-electronic records according to accepted principles and practices of records management.
  • An integrated EDRMS would combine both capabilities.

Section 6 of the report described general (but fairly detailed) functional requirements for an integrated EDMS/ERMS, outlined in the following diagram:

2009 – Autonomy acquires Interwoven

In 2009, HP Autonomy acquired Interwoven, a niche provider of enterprise content management software mostly to the legal industry. It primarily competed with Documentum in this space. Interwoven became Autonomy Interwoven and Autonomy iManage.

2010 – MoReq2010

MoReq was completely revised and published as MoReq2010 in 2010. There were key differences with its predecessor versions.

  • It de-emphasised, but did not remove, the idea of an ERMS being the central or sole recordkeeping system or repository for organisations.
  • It emphasised the need for line of business systems to incorporate a minimum, defined level of recordkeeping functionality.
  • It brought a degree of practicality about the management of records in other systems.
  • It provided for interoperability between all MoReq compliant systems, based on a common XML language.

MoReq2010 established ‘… a definition of a common set of core services that are shared by many different types of records systems’. It provided a set of modules that could be incorporated into any software solution, including line of business applications, so they can be ‘MoReq compliant records systems’ (MCRS).

2010 – Google’s DM capability enhanced

In March 2010, Google acquired DocVerse, an online document collaboration company. DocVerse allowed multiple user online collaboration on Microsoft Word documents, as well as other Microsoft Office formats, such as Excel and PowerPoint. (Source – Wikipedia article on Google Docs)

2010 – SharePoint gains (some) RM wings

SharePoint 2010 included a range of recordkeeping functionality. According to a web page titled ‘SharePoint 2010: Recordkeeping Considerations‘ by State Records NSW:

‘Microsoft SharePoint 2010 is a software product with a range of uses, including website development, content management and collaboration. SharePoint allows users to collaborate on the creation, review and approval of various types of content, including documents, lists, discussions, wiki pages, web pages and blog posts. SharePoint is not a recordkeeping system (i.e. a system purposely designed to capture, maintain and provide access to records over time). When implemented ‘out of the box’, SharePoint has limited capacities for capturing and keeping records in a way that supports their ability to function as authentic evidence of business.’

In the article ‘Is SharePoint 2010 a compliant records solution‘ in the 30 June 2010 edition of IDM Magazine, the author states the following (emphasis added):

Adam Harmetz, the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft said in a recent online interview about Records in SharePoint 2010, “We constantly get questions from around the world about how to deal with local government and industry standards for information management. Let me throw just a few at you… MOREQ2, VERS, ISO 15489, DOMEA, TNA, ERKS, the list goes on. Some of these standards are loosely based on one another and some have contradictory elements. Rather than focus our engineering efforts on addressing each of these standards in turn, we made the choice to deliver the usability and innovation required to make records management deployments successful and allow our partner ecosystem to build out the SharePoint platform to deal with specific requirements for those customers that are mandated to adhere to a specific standard.”

Despite these comments, SharePoint 2010 was assessed by at least one consultant (Wise Technology Solutions) to meet 88% of the requirements of the then ICA Standard that became ISO 16175 Part 2.

On 16 December 2011, State Records NSW published a blog post titled ‘Initial advice on implementing recordkeeping in SharePoint 2010‘. The post noted that the Wise report had concluded that ‘SharePoint is 88% compliant with the ICA requirements’. It added that the areas where full compliance could not be achieved relate to:

  • ease of email capture
  • native security classification and access control
  • physical and hybrid records management

The report states that third party providers are able to offer products that plug SharePoint’s gaps in these areas.

The blog posted also stated that ‘… the report very clearly makes the point that ‘we note that the achievement of these results is reliant on appropriate design and governance of implementation, configuration and set up to ensure consistency with desired records management outcomes’.’

Early 2010 – Microsoft launches Office 365

Microsoft launched Office 365 on 28 June 2011. Office 365 was designed to be a successor to Microsoft’s Business Productivity Online Suite (BPOS). (Source: Wikipedia article on Office 365). It would not be until the mid 2010s that Office 365 would become an effective counter-solution to the G Suite.

2011 – HP acquires Autonomy

In 2011, Hewlett-Packard acquired Autonomy, a deal that resulted in some interesting subsequent legal issues reading the value of the company.

2011 – OMG Records Management Services standard

The Object Management Group (OMG) released its Records Management Standard (RMS) in September 2011.

As noted in the scope section of the standard, ‘(The) specification provides models for software services to support management activities for electronic records’. Further, ‘… models are provided that describe the platform independent model (PIM) that defines the business domain of Records Management and the RM services to be provided’. Three technology-specific implementations are specified:

  • PSM-1 – Web Services definition for Records Management Services in Web Service Description Language (WSDL). This is actually supplied as ten WSDL files; one for each Records Management Service.
  • PSM-2 – A Records Management Service XSD. The XSD is for use in creating XML files for import/export of Managed Records from compliant environments and to use as a basis for forming XQuery/XPath statements for the query service.
  • PSM-3 – An Attribute Profile XSD. The XSD is for capturing and communicating attribute profiles to permit flexible attribution of certain types of Records Management Objects.

2011 – The death of ERM systems?

In a May 2011 blog post on MoReq2010, James Lappin suggested that traditional systems used to manage electronic documents and records, while not being entirely dead in the water, had ‘lost momentum’.

James proposed two specific reasons for this situation:

  • The global financial crisis (GFC) from 2008 that limited the ability of organisations to acquire and implement hugely expensive ERMS solutions.
  • The rise of Microsoft SharePoint and particularly SharePoint 2010. In some ways, Sharepoint 2010 had the potential to take – and may have already taken – the ERMS wind from the records managers sails.

He also noted that a series of interrelated user-environment issues may have also played a part in the loss of momentum.

  • Usability and take up rates of the ERMS. These solutions are sometimes seen as ‘yet another system’ to manage the same records, using a classification structure that doesn’t make sense to most end users and is different from the way end users see and categorise their world.
  • The ongoing availability of and access to alternative places to store information, including network drives and email folders, and cloud-based storage and email solutions.
  • The rise and general availability of social networking tools and mobile applications used to create and share new forms of information content, and collaborate and communicate, including wikis, blogs, Twitter, Facebook, and similar solutions, often in an almost parallel ‘personal’ world to the official record.
  • The inability of ERMS solutions to manage structured data or to maintain and reproduce easily the diverse range of content created and stored in products like SharePoint. Indeed, one reasonably well known product has been described as an archive for SharePoint, even though the latter can quite easily manage its own archives.
  • The rise of search as a tool to find relevant information in context, and the related change from unstructured to structured in XML-based documents generated by products such as Microsoft Office 2007 and 2010.

From 2013 – GEVER (Switzerland)

From 2013, the Swiss Federal Chancellery was responsible for managing all activities relating to electronic records and process management (Elektronische Geschäftsverwaltung), or GEVER. GEVER consisted of a collection of five standards for the management of electronic records. (Source with current update: Gever Bund)

2015 – Hewlett Packard separates

In October 2015, the software products previously under the Autonomy banner were divided between HP Inc and Hewlett Packard Enterprise (HPE). HP Inc was assigned Autonomy’s content management software components including TeamSite, Qfiniti, Qfiniti Managed Services, MediaBin, Optimost, and Explore.

2015 – EDRMS vendors

Despite the alleged death of ERMS products in around 2010, many continued to thrive and grow. Some were acquired by others.

The following is a list of EDRMS vendors in December 2015 taken from a Gartner report diagram titled ‘Product or Service Scores for Trusted System of Record’ (with the scores included). Many of these products also appeared in the October 2016 ‘Magic Quadrant’ for Enterprise Content Management Systems as indicated)

  • Alfresco (2.47) (also ECM)
  • Open Text (4.07) (also ECM)
  • EMC Documentum (3.94) (as Dell EMC)(Acquired by Open Text)
  • IBM (3.91) (also ECM)
  • HPE (3.85)
  • Oracle (3.45) (also ECM)
  • Laserfiche (2.45) (also ECM)
  • Microsoft (SharePoint) (2.38) (also ECM)
  • Hyland OnBase (2.37) (also ECM)
  • Lexmark (2.32) (also ECM)
  • Newgen (2.18) (also ECM)
  • Objective (2/10) (also ECM)

By 2015 – Oracle departing the scene?

In a July 2015 article titled ‘Looking for an Oracle IPM replacement‘ in the blog softwaredevelopmentforECM, it was noted that Oracle was ‘clearly, and publically, going in a different direction and moving away from traditional enterprise imaging and transactional content management’.

2016 – OpenText, Micro Focus

In May 2016, OpenText acquired HP TeamSite, HP MediaBin, HP Qfiniti, HP Explore, HP Aurasma, and HP Optimost from HP Inc.

2017 – Micro Focus merges with HPE

Micro Focus merged with HPE in September 2017 and in doing so acquired the Content Manager EDRMS (formerly TRIM).

Pre 2021 – EDRM vendors (Australia)

The following is a list of products identified by the Victorian Public Records Office (PROV) in 2020. These products were all certified against the VERS standard, that required organisations to be able to create XML-based VERS Encapsulated Objects (VEOs) for long-term preservation.

  • Alfresco
  • Altus ECM
  • AvePoint RevIM, Records, Cloud Records
  • Bluepoint Content Manager
  • Canon Therefore 2012
  • Docbureau
  • ELOprofessional / ELOenterprise
  • HP Records Manager
  • HP TRIM
  • IBM Enterprise Records
  • IBM FileNet P8 Records Manager
  • Info-Organiser
  • Laserfiche
  • MAGIQ Documents
  • MicroFocus Content Manager
  • Objective
  • Objective ECM
  • OpenText eDOCS RM
  • OpenText Records Management
  • Oracle WebCentre Content
  • RecFind 6
  • RecordPoint
  • Technology One ECM
  • Technosoft OfficeTech

2021 – EDRMS vendors

The following is a list of dedicated vendors that offered EDRMS solutions (and more in most cases) by early 2021. Many of these vendors have a long history not necessarily reflected in the above text. Most of these vendors provide Enterprise Content Management (ECM) services, including EDM and ERM capabilities.

  • Alfresco ECM (alfresco.com)
  • Hyland OnBase (hyland.com)
  • IBM ECM (ibm.com)
  • Knowledgeone RecFind EDRM (knowledgeonecorp.com)
  • Laserfiche RME (laserfiche.com)
  • Lexmark RIM (lexmark.com)
  • Micro Focus Content Manager (microfocus.com)
  • Microsoft 365 (microsoft.com)
  • Newgen RMS (newgensoft.com)
  • Objective ECM (objective.com)
  • Open Text ECM (opentext.com)
  • Oracle ECM (docs.oracle.com)
  • TechnologyOne ECM (technologyonecorp.com)

2021 – Dedicated EDMS vendors

EDM vendors never went away, but many – like Google Drive, DropBox and Box – were built in and for the cloud. This Capterra website has a fairly detailed listing of current EDMS vendors.

2021 – NEN 2082 withdrawn

In January 2021, the Dutch government withdraw NEN 2082. (NEN 2082:2008 nl)

The future of standards-based ERM/EDRM/ECM systems

Although the definition of a record has remained largely intact for the past two decades – ‘evidence of business activity’ (ISO 15489) – the form of records has evolved and continues to do so.

The ever-expanded world of digital content has made it increasingly difficult to accurately and consistently identify, capture and manage records in all forms, a challenge to the notion that all records can be stored in a single system.

The ‘in place’ approach to managing electronic records – wherever they are stored – has strong appeal. But where will we be in another 20 years? Some thoughts:

  • Electronic databases, whether on-premise or cloud-based (including subscription based), will be the primary method of capturing and storing a wide range of digital content rather than network file shares.
  • Metadata will be automatically captured or auto-generated for all digital content based on the content itself.
  • Artificial Intelligence (AI) will continue to grow in maturity, allowing records to be identified from all other digital content, classified, aggregated, and managed through to disposal/disposition or transfer to archives.
  • Email will, slowly, disappear as the current workforce transitions to chat- and video-based communication methods.
Posted in Classification, Compliance, Information Management, Records management, Retention and disposal

Classifying records in Microsoft 365

The classification of records is fundamental recordkeeping activity. It is defined in the international standard ISO 15489-1:2016 (Information and Documentation – Records Management) as the ‘systematic identification and/or arrangement of business activities and/or records into categories according to logically structured conventions, methods and procedural rules‘. (Terms and Definitions, 3.4)

The purpose of classification is defined by State Records NSW as follows: ‘In records management, records are classified according to the business functions and activities which generate the records. This functional approach to classification means that classification can be used for a range of records management purposes, including appraisal and disposal, determining handling, storage and security requirements, and setting user permissions, as well as providing a basis for titling and indexing‘. (Records Classification, accessed 13 January 2021.)

The ever-increasing volume of digital records, the many different ways to create them, and the multitude of record types that are created and storage locations, have made it more difficult to accurately and consistently manually classify records, including through the creation of pre-defined ‘containers’ or aggregations based on classification terms. Despite this, the requirement to link the classification of records with their retention and and disposal remains.

For over three decades, Microsoft’s applications and technology platforms have been used to create, capture, store and manage records. Some of these records (in the earlier period) were printed and placed on paper files, or stored (from around 2000) in dedicated electronic document and records management (EDRM) systems.

But the volume and type of digital content, including with new types of records (e.g., chat messages) and storage locations, continues to grow. In response, Microsoft invested heavily in addressing the need to classify records ‘at scale’.

This post looks at various ways to classify records, for retention and disposition purposes, in Microsoft 365.

The old-school, manual method – metadata

Most of the records in Microsoft 365 will be created, captured or stored in one of the four primary workloads: Exchange mailboxes, SharePoint sites/libraries, MS Teams chats (a ‘compliance copy’ of which is stored in Exchange mailboxes), and OneDrive for Business libraries. Some records may also exist in Yammer or other web page content (e.g., intranet).

Most SharePoint sites as well as Teams (that have a SharePoint site) will be created according to some form of business need to create, capture, store and share records; that is, the site or team purpose may be based on a business function or activity. This way of grouping records may in some ways be used as a way to classify records – by SharePoint site (e.g., function) or document library (e.g., activity).

Records may be stored in multiple document libraries, or within a folder structure of a single library.

A number of methods (some of which rely on others) can be used to add classification (and other) metadata to records stored in SharePoint document libraries:

1 – Creating the classification taxonomy in the Managed Metadata Service (MMS)/Term Store via the SharePoint Admin portal – Content Services – Term store, and then applying these terms in content types that are then deployed in SharePoint sites.

An example of Business Classification Terms in the MMS

2 – Creating global content types from the SharePoint Admin portal, in the Content Services – Content type gallery area (see ‘Finance Document’ example below) and then deploying these in specific SharePoint sites where site columns that contain classification terms will be added.

3 – Creating site columns that contain classification terms, including from the MMS, and adding these to global or site content types or document libraries where they can be applied to records.

In this example, the site column ‘BCS Function’ maps to the MMS BCS terms

4 – Creating site content types and adding site columns (including MMS-based columns), then adding these content types to document libraries.

In this example, the MMS-based column now appears in the library columns

But, most of the above is somewhat complicated and cumbersome and would normally only be used for and manually applied to specific types of records.

The simplest way to apply BCS/File Plan terms at the document (or document set) level is to (a) store records to the same BCS function or activity in the same library, (b) create site or library columns with default values and add these to the library. This way means that the default terms are applied automatically as soon as a new record is uploaded, including when shared/inherited from the site columns added to a document set that ‘contains’ a document content type.

Example metadata columns shared from the document set content type

However, keep in mind that SharePoint is just one of the workloads where records are stored.

Records in the form of emails, chats and ‘personal’ content (as well as Yammer messages and web pages) are created in and stored across the other workloads. Some attempt may be made to copy these other records (especially emails) in SharePoint sites but it starts to get complicated or impossible to do so with things like Teams chat messages.

In most cases (and according to Microsoft’s own recommendations), it is better to leave the records where they were created or captured (‘in place’), and apply centralised compliance controls (classification, retention labels and policies) to this content.

Leaving the records in place in this way does not exclude the ability to create SharePoint sites and document libraries in those sites that map to classification terms, and/or use the site column approach described above but these are more likely to be exceptions.

In fact, some form of logical structure is almost certain anyway as most end-users will probably want to access and manage information in their own specific work context (the Team/SharePoint site).

Trainable classifiers

Since not all records are stored in SharePoint and the ever-increasing volume of digital content stored across the Microsoft 365 platform, Microsoft needed to find a way to classify records ‘at scale’.

The solution was to use machine learning (ML) via trainable classifiers accessed in the ‘Data Classification’ section of the Microsoft 365 Compliance portal. This capability is only available to E5 licences.

The trainable classifiers solution was released to General Availability on 12 January 2021 (‘Announcing GA of machine learning trainable classifiers for your compliance needs‘, accessed 13 January 2021).

See the Microsoft web page ‘Learn about trainable classifiers‘ to learn more about this option. To quote from that page:

This classification method is particularly well suited to content that isn’t easily identified by either the manual or automated pattern matching methods. This method of classification is more about training a classifier to identify an item based on what the item is, not by elements that are in the item (pattern matching).’

Organisations (including E3 licence holders) may make use of five pre-defined trainable classifiers (Resumes, Source Code, Targeted Harassment, Profanity or Threat. A sixth classifier ‘Offensive language’, is to be deprecated). Custom classifiers require an E5 licence.

Custom classifiers require ‘significantly more work’ than the pre-existing classifiers and the process is quite involved (see the process flow diagram in the ‘Learn about’ page link above) but in summary it involves the following steps:

  • Creating the custom classifier.
  • Creating a set of manually selected example records (50 to 500) in a dedicated SharePoint Online site as the ‘seed’. This would include a range of emails in the seed examples.
  • Testing the classifier with the seeded documents.
  • Re-training with additional content – both positive and negative matches.

Once the classifier is published, it can be used to identify and classify related content across SharePoint Online, Exchange, and OneDrive (but not Teams).

The page ‘Default crawled file name extensions and parsed file types‘ provides details of all the record types that can be classified in this way. Note it is not clear if trainable classifiers can crawl the compliance copy of Teams chat messages stored in hidden folders in Exchange mailboxes.

Label-based retention policies can then be automatically applied to content that has been identified through the trainable classifier.

However, note that the classifier does not ‘group’, aggregate or ‘present’ (list) the records for review (except broadly via the Content Explorer); however, the label applied to the records can be searched via the ‘Content Search’ option in the Compliance portal. This is a much better option than not having any idea how many records of a particular classification may exist in Exchange mailboxes, OneDrive accounts, or general SharePoint sites. It requires some degree of ‘letting go’ of the ability to view and browse content classified this way, and trusting the system.

The main limit with trainable classifiers is that it requires an E5 or E5 compliance licence.

The other limitation is the management of the disposition of records that have been identified with trainable classifiers and had a label-based retention policy applied. There are significant shortcomings with the current ‘Disposition Review’ process, specifically the lack of adequate metadata to review records due for disposal or the details of what has been destroyed.

SharePoint Syntex

Another (but limited) option might be to use SharePoint Syntex (see ‘Introduction to Microsoft SharePoint Syntex‘ for an overview), although its range is limited to SharePoint and – it seems – only records that have a relatively consistent structure and format.

SharePoint Syntex evolved out of Project Cortex’s ability to extract and capture metadata from records. It can also be used through its ‘Document Understanding Model‘ (DUM) to provide a way to classify records stored in SharePoint Online (only). It makes use of a ‘seeding’ model that is similar to trainable classifiers (and may be based on the same underlying AI engine).

Broadly speaking, the DUM works on the basis of loading a small ‘seed’ set of (relatively consistently formated) example files into a dedicated Content Center (or Centers). This is very similar to the process of using trainable classifiers, except that the latter does not require a ‘content center’ SharePoint site to be created.

  • The example files are ‘trained’ by being ‘classified’ through the document understanding process based on a set of ‘explanation types‘ that are used to help find the relevant content. The three explanation types are: (a) phrase list (a list of words, phrases, numbers, or other characters used in the document or information that you are extracting); (b) pattern list (patterns of numbers, letters, or other characters); and (c) proximity (describes how close other explanations are to each other).
  • The document understanding model (DUM) produced through the explanation types is associated (and deployed) with a new or existing content type. 
  • Once applied to a SharePoint site library, the DUM/content type provides the basis for identifying and tagging (with metadata) other similar records in the location (e.g., the library) where the DUM has been deployed. 
  • If the documents have consistent content such as invoices, certain data from those documents can be extracted as metadata. 

Retention labels may be applied to records classified using SharePoint Syntex, as described on this page ‘Apply a retention label to a document understanding model‘.

Summing up – which one should be used?

The answer to this question will depend on your compliance requirements.

Smaller organisations may be able to set up SharePoint sites and document libraries with site columns/metadata that maps to their business classification scheme or file plan, and copy emails to those libraries. There may be little need to use AI-based classification methods.

In large and more complex organisations (with E5 licences), especially those with a lot of content stored across Exchange mailboxes and SharePoint sites (including Teams-based sites) there will most certainly be a need for some form of AI-based classification in addition to classification-mapped SharePoint sites (and Teams).

Organisations with E3 licences might use the manual methods described above for specific types of records, and consider acquiring additional E5 Compliance licences to make use of trainable classifiers or SharePoint Syntex for other records.

Posted in Compliance, Electronic records, Governance, Information Management, Legal, Microsoft 365, Microsoft Teams, Products and applications, Records management, Retention and disposal, SharePoint Online, Yammer

All the ways SharePoint sites can be created

SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.

While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.

Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.

So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.

Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.

The following is a list of the main ways a person can create a SharePoint site. Have I missed any?

1. Via a PowerShell script

As described in the Microsoft docs web page ‘Create SharePoint Online sites and add users with PowerShell‘. The script is based on a csv file (‘sitecollections.csv) and looks something like the following (see the link for more details):

Import-Csv C:\users\MyAlias\desktop\SiteCollections.csv | ForEach-Object {New-SPOSite -Owner $_.Owner -StorageQuota $_.StorageQuota -Url $_.Url -NoWait -ResourceQuota $_.ResourceQuota -Template $_.Template -TimeZoneID $_.TimeZoneID -Title $_.Name}

This option also allows the administrator to provision new SharePoint sites.

2. Via the SharePoint Admin portal (+ Create)

This option allows the creation of three main types of sites: modern team sites (Team site),
communication sites, and non-Microsoft 365 Group-linked sites (Other options).

3. By creating a Microsoft 365 Group

Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.


Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.

The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.

Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.

4. By an end-user creating a new Team in MS Teams

Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.

Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an
Exchange Online mailbox that is not visible in Outlook.

5. By creating a Private Channel in MS Teams

If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.

For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.

A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.

6. By the Teams Admin creating a new Team

The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.

As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.

7. From the end-user SharePoint portal (+ Create site)

If not disabled, end users can create a new SharePoint site by clicking on ‘+ Create site’ from the SharePoint portal – https://tenantname.sharepoint.com/_layouts/15/sharepoint.aspx

This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.

It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.

8. From OneDrive for Business as a ‘shared library’

This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.

9. When a new Plan is created in Planner

If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.

10. When a new Yammer community is created

End users with access to Yammer can click on ‘Create a Community’ from Yammer.

To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’

Why have Microsoft allowed this?

It’s a smarter way to manage access.

Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.

The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.

But, some of the decisions don’t seem to make sense.

  • Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
  • Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
  • Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
  • And why does a new Plan create a SharePoint site? For what purpose?

Perhaps there is a reason for it. It’s just not clear.

Posted in Archiving third party content, Connectors, Conservation and preservation, Electronic records, Information Management, Microsoft 365, Microsoft Graph, Records management, Retention and disposal, Solutions

Using Microsoft 365 connectors to support records management

Microsoft 365 includes a range of connectors, in three categories, that can be used to support the management of records created by other applications. The three categories are:

  • Search connectors, that find content created by and/or stored in a range of internal and external applications, including social media.
  • Archive connectors, that import and archive content created by third-party applications.
  • API connectors, that support business processes such as capturing email attachments.

This post how these connectors can assist with the management of records.

The recordkeeping dilemma

Finding, capturing and managing records across an ever increasing volume of digital content and content types has been one of the biggest challenges for recordkeeping since the early 2000s.

The primary method of managing digital records for most of the past 20 years has been to require digital records (mostly emails and other digital content created on file shares) to be saved to or stored in an electronic document and records management system (EDRMS). The EDRMS was established as ‘the’ recordkeeping system for the organisation.

EDRM systems were also used to manage paper records which, over the past 20 years, have mostly contained the printed version of born-digital records that remain stored in the systems where they were created or captured.

There were two fundamental flaws in the EDRMS model. The first was an expectation that end-users would be willing to save digital records to the EDRMS. The second was that the original digital record remained in place where it was created or captured, usually ignored but often the source of rich pickings for eDiscovery.

The introduction of web-based email and document storage systems, smart phones, social media and personal messaging applications from around 2005 (in addition to already existing text messaging/SMS messages) further challenged the concept of a centralised recordkeeping system; in many cases, the only option to save these records was to print and scan, screenshot and save the image, or save to PDF, none of which were particularly effective in capturing the full set of records.

The hasty introduction from early 2020 of ‘work from home’ applications such as Zoom and Microsoft Teams has been a further blow to these methods.

In place records management

To the chagrin of records managers around the world, Microsoft never made it easy to save an email from Outlook to another system. Emails stubbornly remained stored in Exchange mailboxes with no sign of integration with file shares.

And for good reason – they have a different purpose and architecture to support that purpose. It would be similar to asking when it would be possible to create and send an email in Word.

The introduction of Office 365 (later Microsoft 365) from the mid 2010s changed the paradigm from a centralised model – where records were all copied to a central location and the originals left where they were created or captured, to a de-centralised or ‘in place’ model – where records are mostly left where they were created or captured.

The decentralised model does not exclude the ability to store copies of some records (e.g., emails) in other applications (e.g., SharePoint document libraries), but these are exceptions to the general rule.

It also does not exclude the ability to import or migrate content from third-party applications where necessary for recordkeeping purposes.

Microsoft 365 connectors

Microsoft 365 includes a wide range of options to connect with both internal and external systems. Many of these connectors simplify business processes and support integration models.

Connectors may also be used to support recordkeeping requirements, in three broad categories.

The three connectors

Archive connectors

Archive connectors allow organisations to import and archive data from third-party systems such as social media, instant messaging and document collaboration* platforms. Most of this data will be stored in Exchange mailboxes, where it can be subject to retention policies, eDiscovery and legal holds.

(*This option is still limited via connectors, but also see below under Search).

The social media and instant messaging data that can be archived in this way currently includes Facebook (business pages), LinkedIn company page data, Twitter, Webex Teams, Webpages, WhatsApp, Workplace from Facebook, Zoom Meetings. For the full listing, and a detailed description of what is required to connect each service, see this Microsoft description ‘Archive third-party data‘.

An important thing to keep in mind is that the data will be archived to an Exchange mailbox; this will require an account to be created for the purpose. Any data archived ot the mailbox will contribute to the overall storage quotas.

Search connectors

Search connectors (also known as Microsoft Graph connectors) index third-party data that then appears in Microsoft search results, including via Bing (the ‘Work’ tab), from http://www.office.com, and via SharePoint Online.

Most ECM/EDRM systems are listed, which means that organisations that continue to use those systems can allow end-users to find content from a single search point, only surfacing content that users are permitted to see.

The following is an example of what a Bing search looks like in the ‘Work’ tab (when enabled).

Example Bing search showing the Work tab

Note: as at 17 November 2020, Microsoft’s page ‘Overview of Microsoft Graph connectors‘ (which includes a very helpful architecture diagram) states that these are ‘currently in preview status available for tenants in Targeted release.’

There are two main types of search connector:

  • Microsoft built: Azure Data Lake Storage Gen2, Azure DevOps, Azure SQL, Enterprise websites, MediaWiki, Microsoft SQL, and ServiceNow.
  • Partner built. Includes the following on-premise and online document management/ECM/EDRM connectors – Alfresco, Alfresco Content Services, Box, Confluence, Documentum, Facebook Workplace, File Share (on prem), File System (on prem), Google Drive, IBM Connections, Lotus Notes, iManage, MicroFocus Content Manager (HPE Records Manager, HP TRIM), Objective, OneDrive, Open Text, Oracle, SharePoint (on prem), Slack, Twitter, Xerox DocuShare, Yammer

See the ‘Microsft Graph connectors gallery‘ web page for the full set of current connectors.

A consideration when deploying search connectors is the quality of the data that will be surfaced via searches. Duplicate content is likely to be a problem in identifying the single – or most recent – source of truth of any particular digital record, especially when the organisation has required records to be copied from one system (mailbox/file share) to another (EDRMS).

API Connectors

API connectors provide a way for Microsoft 365 to access and use content, including in third-party applications. To quote from the Microsoft ‘Connectors‘ web page:

‘A connector is a proxy or a wrapper around an API that allows the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. It provides a way for users to connect their accounts and leverage a set of pre-built actions and triggers to build their apps and workflows.’

To see the complete list and for more information about each connector, see the Microsoft web page ‘Connector reference overview‘.

Each connector provides two things:

  • Actions. These are changes initiated by an end-user.
  • Triggers. There are two types of triggers: Polling and Push. Triggers may notify the app when a specific event occurs, resulting in an action. See the above web page for more details.

API connectors can support records management requirements in different ways (such as triggering an action when a specific event occurs) but they should not be confused with archiving or search connectors.

Summing up

The connectors available in Microsoft 365 support the model of keeping records in place where they were first created or captured. They enable the ability to archive data from third-party cloud applications, search for data in those (and on-premise) applications, and triggers actions based on events.

The use of connectors should be part of an overall strategic plan for managing records across the organisation. This may include a business decision to continue using an ECM/EDRMS in addition to the content created and captured in Microsoft 365. Ideally, however, the content in the ECM/EDRMS should not be a copy of what already exists in Microsoft 365.

Posted in Records management, Electronic records, Digital preservation, XML, Retention and disposal, Conservation and preservation, Exchange 2010, Information Management, Exchange 2013, Access controls, Exchange Online

The enduring problem of emails as records

Ever since emails first appeared as a way to communicate more than 30 years ago they have been a problem for records management, for two main reasons.

  • Emails (and attachments) are created and captured in a separate (email) system, and are stored in mailboxes that are inaccessible to records managers (a bit like ‘personal’ drives).
  • The only way to manage them in the context of other records was/is to print and file or copy them to a separate recordkeeping system, leaving the originals in place.

Thirty-plus years of email has left a trail of mostly inaccessible digital debris. An unknown volume of records remains locked away in ‘personal’ and archived mailboxes. Often, the only way to find these records is via legal eDiscovery, but even that can be limited in terms of how back you can go.

Options for the preservation of legacy emails

The Council on Information and Library Resources (CLIR) published a detailed report in August 2018 titled ‘The Future of Email Archives: A Report from the Task Force on Technical Approaches to Email Archives‘.

The report noted (from page 58) three common approaches to the preservation of legacy emails:

  • Bit-Level Preservation
  • Migration (to MBOX, EML or even XML)
  • Emulation

In a follow up article, the Australian IDM magazine published an article in March 2020 by one of the CLIR report authors (Chris Prom). The article, titled ‘The Future of Past Email is PDF‘, suggested that PDF may be (or become) a more suitable long-term solution for preservation of legacy emails.

Preservation is one thing, what about access

There is little point in preserving important records if they cannot be accessed. The two must go together. In fact, preservation without the ability access a record is not a long different from destruction through negligence.

Assuming emails can be migrated to a long-term and accessible format, what then?

No-one (except possible well-funded archival institutions perhaps) is seriously likely to attempt to move or copy individual legacy emails to pre-defined and pre-existing containers or aggregations of other records. This would be like printing individual emails and storing them in the same paper file or box that other records on the same subject are stored.

Access to legacy emails in an digitally accessible, metadata-rich format like PDF provides a range of potential opportunities to ‘harvest’ and make use of the content, including through machine learning and artificial intelligence.

These options have been available for close to twenty years in the eDiscovery world, but to support specific legal requirements.

Search, discovery and retention/disposal tools available in the Microsoft 365 Compliance portal, along with the underlying Graph and AI tools (including SharePoint Syntex) provide the potential to manage legacy content, including emails.

The starting point is migrating all those old legacy emails to an accessible format.

Posted in Compliance, Electronic records, Exchange Online, Information Management, Microsoft Teams, Records management, Retention and disposal, Security

Using MS Teams without an Exchange Online mailbox

When people chat in Microsoft Teams (MS Teams), a ‘compliance’ copy of the chat is saved to either personal or (Microsoft 365) Group mailboxes. This copy is subject to retention policies, and can be found and exported via Content Search.

But what happens if there is no Exchange Online mailbox? It seems the chats become inaccessible which could be an issue from a recordkeeping and compliance point of view.

This post explains what happens, and why it may not be a good idea (from a compliance and recordkeeping point of view) not to disable the Exchange Online mailbox option as part of licence provisioning.

Licences and Exchange Online mailboxes

When an end-user is allocated a licence for Microsoft 365, a decision (sometimes incorporated into a script) is made about which of the purchased licences – and apps in those licences – will be assigned to that person.

E1, E3 and E5 licences include ‘Exchange Online’ as an option under ‘Apps’. This option is checked by default (along with many of the other options), but it can be disabled (as shown below).

If the checkbox option is disabled as part of the licence assigning process (not after), the end-user won’t have an Exchange mailbox and so won’t see the Outlook option when they log on to office.com portal. (Note – If they have an on-premise mailbox, that will continue to exist, nothing changes).

Having an Exchange Online mailbox is important if end-users are using MS Teams, because the ‘compliance’ copy of 1:1 chat messages in MS Teams are stored in a hidden folder (/Conversation History/Team Chat) in the Exchange Online mailbox of every participant in the chat. If the mailbox doesn’t exist, those copies aren’t made and so aren’t accessible and may be deleted.

If end-users chat with other end-users who don’t have an Exchange mailbox as shown in the example below, the same thing happen – no compliance copy is kept. The chat remains inaccessible (unless the Global Admins take over the account).

The exchange above, between Roger Bond and Charles, includes some specific key words. As we will see below, these chats cannot be found via a Content Search.

(On a related note, if the ability to create private channels is enabled and they create a private channel and chat there, the chats are also not saved because a compliance copy of private channel chats are stored in the mailboxes of the individual participants.)

Searching for chats when no mailbox exists

As we can see above, the word ‘mosquito’ was contained in the chat messages between Roger and Charles.

Content Searches are carried out via the Compliance portal and are more or less the same as eDiscovery searches in that they are created as cases.

From the Content Search option, a new search is created by clicking on ‘+New Search’, as shown below. The word ‘mosquito’ has been added as a keyword.

We then need to determine where the search will look. In this case the search will look through all the options shown below, including all mailboxes and Teams messages.

When the search was run, the results area shows the words ‘No results found’.

Clicking on ‘Status details’ in the search results, the following information is displayed – ‘0 items’ found. The ‘5 unindexed items’ is unrelated to this search and simply indicates that there are 5 unindexed items.

Double-checking the results

To confirm the results were accurate, another search was conducted where the end-user originally did not have a mailbox, and then was assigned one.

If the end-user didn’t have a mailbox but the other recipient/s of the message did, the Content Search found one copy of the chat message in the mailbox of the other participants. Only one item was found.

When the Exchange Online option was enabled for the end-user who previously did not have a mailbox (so they were now assigned a mailbox), a copy of the chat was found in the mailbox of both participants, as shown in the details below (‘2 items’).

Summary and implications

In summary:

  • If end users chat in the 1:1 area of MS Teams and don’t have an Exchange Online mailbox, no compliance copy of the chat will be saved, and so it will not be found via Content Search.
  • If any of the participants in the 1:1 chat have an Exchange Online mailbox, the chat will appear in the mailboxes of those participants.
  • If all participants in the 1:1 chat have an Exchange Online mailbox, the chat will be found in the mailbox of all participants.

Further to the above:

  • If end users can delete chats (via Teams policies) and don’t have a mailbox, no copy of the chat will exist.
  • If end-users with a mailbox can delete Teams chats, but a retention policy has been applied to the chats, the chats will be retained as per the retention policy (in a hidden folder).

And finally, if you allow private channels, end-users can create private channels in the Organisation Team. The chats in these private channels are usually stored in the personal mailboxes of participants (not the Group mailbox) – so these chats will also be inaccessible and cannot be found via Content Search.

The implications for the above are that, if you need to ensure that personal chat messages can be accessed (from Content Search), then the participants in the chat must have an Exchange Online mailbox.

Further, if you allow deletion of chats but need to be able to recover them for compliance purposes, a retention policy should be applied to Teams 1:1 chat.

Posted in Electronic records, Information Management, Microsoft 365, Records management, Retention and disposal

A modern way to manage the retention of digital records

In his April 2007 article titled ‘Useful Void: The Art of Forgetting in the Age of Ubiquitous Computing’ (Harvard University RWP07-022), Viktor Mayer-Schönberger noted that the default human behaviour for millenia was to forget. Only information that needed to be kept would be retained. He noted that the digital world had changed the default to remembering, and that the concept of forgetting needed to be re-introduced through the active deletion of digital content that does not need to be retained.

The harsh reality is that there is now so much digital information in the world, including digital content created and captured by individual organisations, that active deletion of content that does not need to be retained, seems an almost impossible task.

This post explores issues with the traditional model of records retention in the digital world, and why newer options such as the records retention capability of Microsoft 365 is a more effective way to manage the retention and disposal of records, and all other digital content.

The traditional retention model

The traditional model of managing the retention and disposal/disposition of records was based on the ability to apply a retention policy to a group or aggregation of information identified as records. For the most part, those paper records were the only copy that existed (with some allowance for working and carbon copies).

The model worked reasonably well for paper records, but started to falter when paper records became the printed versions of born-digital records, and where the original digital versions remained where they were created or captured – on network files shares, in email systems, and on backups. Although, technically, the official record was on a file, a digital version was likely to remain on network file shares or in an email mailbox after the paper version was destroyed at the end of the retention period, and remain overlooked.

How many of us have had to wade through the content of old network file shares to examine the content, determine its value, and perhaps see if it can even still be accessed? Or do the same with old backup tapes?

The volume of unmanaged digital content, not subject to any retention policy, only continued to increase. This situation continued to worsen when electronic document and records management (EDRM) systems were introduced from the late 1990s. End-users had to copy records to the EDRMS, thereby creating yet another digital copy, in addition to the born-digital originals stored in mailboxes or file shares.

Even if the record in the EDRMS were destroyed, there was a good chance the original ‘uncontrolled’ version of the digital record – along with an unknown volume of digital records that probably should have been consigned to the EDRMS but weren’t – remained in email mailboxs, on file shares, or on a backup tape somewhere.

eDiscovery was born.

The emergence of new forms of digital records, including instant messages, social media, and smart-phone based chat and other apps from the early 2000s only added to the volume of digital content, much of which was stored in third-party cloud-based and mobile-device accessible applications completely out of the reach and ability of the organisation trying to manage records.

Modern retention management

A modern approach to retention management should be based on the following principles:

  • Information, not just records, should only be kept for as long as it is required.
  • It is no longer possible to accurately and/or consistently identify and capture all records in a single recordkeeping system.
  • Duplication of digital content can be reduced by creating and capturing records in place, promoting ‘working out loud’, co-authoring and sharing (no more attachments and private copies).

None of the above points excludes the ability to manage certain types of records at a more granular level where this is required. But these records, or the location in which they are created or captured, should not be regarded as the only form of record.

Ideally, these records should be created (or captured) directly in the system where they are to be managed – not copied to it.

Change management is necessary

Some of these new ways of working are likely to come up against deeply ingrained behaviours, many of which go back several decades and have contributed to a reluctance to ‘forget’ and destroy old digital content, including:

  • hiding/hoarding content in personal drives (and personal cloud-based systems or on USB drives);
  • communicating by email, the content is which is inaccessible to anyone else;
  • attaching documents to emails;
  • printing and filing born-digital content; and
  • sometimes, scanning/digitising the printed copies of born-digital records and saving them back to a digital system.

What about destruction?

Records managers in organisations moving away from the authorised destruction of digital content identified as records, to the destruction of all digital content (including identified records) need to consider what is required to achieve this outcome, and the implications for existing process and practices (including those described above).

  • Some activities will remain unchanged. For example, the need to review certain types of records before they are destroyed (aka ‘disposition review’), to seek approval for that destruction, and to keep a record of what was destroyed.
  • Some activities are new and can replace other existing actions and activities. For example, the application of retention policies to mailboxes can remove the requirement to backup those mailboxes.
  • Some of activities or outcomes may be challenging. For example, the automatic destruction without review of digital content that is not the subject of more granular retention requirements, such as emails out of mailboxes, documents in personal working drives. This content will simply disappear after the retention period expires.

How Microsoft 365 can support modern retention management

Microsoft recognised some time ago that it was becoming increasingly difficult to manage the volumes and types of digital content that was being created every day by organisations.

Exisiting and newly released functionality in the Compliance portal of Microsoft 365 includes the ability to create and apply both label-based retention policies to specific types of records, including automatically based on machine learning capabilities, and broader ‘workload’ specific (e.g., mailboxes, SharePoint sites, OneDrive accounts, MS Teams chats) retention policies. This capability helps organisations to focus retention requirements on the records that need to be retained, while destroying digital content that is no longer relevant and can be forgotten.

Instead of directing end-users to identify records and copy them from one system to another (thereby creating two versions), Microsoft 365 allows end-users to create and capture records in place, providing a single source of truth that can be shared (rather than attached), be the subject of co-authoring, and protected from unauthorised changes (and even downloads).

Limitations with Microsoft 365

It is important to keep in mind that there are some limitations with the current (October 2020) retention capability in Microsoft 365.

  • Retention and disposal is based on individual digital objects, not aggregations. There are limited ways to group individual records by the original aggregations in which they may have been stored (e.g., document libraries in SharePoint).
  • Only the (minimal) details of records that were subject to a disposition review are recorded in the ‘disposed items’ listing, and this is only kept for a year (but can be exported). No record is kept of any other destroyed record, except in audit logs (for a limited period).
  • The metadata details of records subject to a disposition review that were destroyed is minimal – the document type and name, date destroyed, destroyed by whom.
  • When records are destroyed from SharePoint document libraries or lists, the library or list remains with no record kept of what was previously stored there. It is not possible to leave a ‘stub’ for a destroyed record.

Summing up

The primary outcome from introducing modern ways to manage retention will be that all digital content, not just content that has been identified as records or copied to a recordkeeping system, will be subject to some form of retention and disposal management.

In other words, a change from exception-based retention (where all the other digital content is overlooked), to a more holistic method of retention with both granular controls on certain types of records where this is required, and broader retention capability allowing us to forget the content that is no longer relevant – the ‘redundant, trivial and outdated’ (ROT) content often scattered across network file shares.

Posted in Electronic records, Information Management, Microsoft 365, Microsoft Teams, Records management, Retention and disposal, SharePoint Online

The Microsoft 365 experience – Teams, Exchange, Outlook, Edge: Where did SharePoint Go?

At the 2020 Microsoft Ignite conference, Jeff Teper presented a diagram titled ‘Microsoft 365’. The diagram showed only four icons: Teams, Outlook, Office and Edge.

The implication of this diagram was that, for most end-users, Teams is now (or will become) their primary portal into Microsoft 365. As stated by Jeff Teper, SharePoint is a foundation platform, the out of sight content engine. Edge’s ability to serve up search results from Microsoft 365 further reduces the need to go to SharePoint.

So, what are the implications for managing records?

SharePoint as a recordkeeping system

For a long time, records have been created, captured and stored in recordkeeping systems.

In the paper world, the recordkeeping system consisted of paper records stored in files and boxes and detailed in registers. With the introduction of computers in the 1980s, registers were transferred to databases, making it a bit easier to find records. In the late 1990s, recordkeeping databases were linked with (separate) file stores and became electronic document and records management (EDRM) systems that continued to manage paper records (the so-called ‘hybrid’ systems).

For almost a decade (since SharePoint 2010 was introduced), SharePoint has contended with files shares and EDRM systems as an alternative recordkeeping system, providing almost all the same core functionality.

The ability to create a record in a single location, then share and co-author it from that location, has completely removed the requirement to copy a record to a separate recordkeeping system.

And then came Teams

Someone at Microsoft had incredible foresight to see the potential for a new user interface that would replace products like Lync and Skype for chat and conferencing, and would also provide access to files stored in SharePoint.

SharePoint has been a core part of the Microsoft productivity offerings for a very long time and people have built careers around developing functionality on the SharePoint platform to appeal to end-users, the intranet being the most common case in point, with customised team sites close behind.

The arrival of Microsoft 365 Groups and then Teams in 2017 was perhaps not widely noticed. One could argue that end by the beginning of 2020, it was still largely unnoticed.

And then came a pandemic and working from home. Teams – which may have been largely ignored or overlooked until then – was already ready to take its place next to Outlook, Office and Edge as a primary end-user interface.

New Teams were created, sometimes with abandon (and were sometimes just as quickly abandoned).

Both 1:1 (or 1:many) chats and channel chats took off. Files were created and shared via OneDrive for Business (‘Files’ in the 1:1 chat area), or via the back-end SharePoint sites (‘Files’ in the channel chat area).

There was (and maybe still is) a belief that files were being saved to Teams but not SharePoint. ‘We are storing everything in Teams’ was not an uncommon expression, sometimes followed by ‘but we’re not using SharePoint or OneDrive’.

The year 2020 saw a huge increase in the volume of records stored in SharePoint sites linked with Teams, as well as a completely new set of records – chats (‘compliance’ copies of which are stored in Exchange mailboxes).

The diagram below provides an overview of the relationship between Teams, Microsoft 365 Groups, Exchange mailboxes, SharePoint and OneDrive for Business.

What about SharePoint?

As the diagram above shows, SharePoint has not disappeared. Many organisations will continue to use, and ask end-users to access, SharePoint sites directly to store and manage records.

But accessing SharePoint from SharePoint may become less necessary over time. At Ignite 2020, the ability to pin a ‘home site’ (such as an intranet) to Teams was demonstrated. Even the intranet may end up in Teams.

As Jeff Teper said, SharePoint is a foundation platform, one that does not get in the way of collaboration and productivity but powers it.

Implications for records managers

Records managers, who were likely already on a steep learning curve regarding SharePoint, need to continue to improve their knowledge of the SharePoint platform. On a positive note, SharePoint Online is a much easier application to learn and manage, compared with its earlier on-premise predecessors.

In organisations that have been using SharePoint for a while and/or have allowed the free-creation of Teams in MS Teams, there will some requirement for retrospective analysis, review, and cleaning up.

In all organisations, there will be a requirement to establish some form of governance and oversight of records (files and chats) that have been created, including for the purpose of retention and disposal/disposition.

Retrospective implications

Where MS Teams has been implemented with little thought given to naming conventions, SharePoint site provisioning, or access controls, records managers should been given access to and review the list of all SharePoint sites that have been created, including from MS Teams. This will provide an initial idea of the volume of content and activity on each site, and what action needs to be taken on things like inactive Teams.

Ideally, records managers should be added to the Site Collection Administrators (SCA) group of every SharePoint site, including MS Teams-based sites. This action will give records managers access to the content on every site and to help advise on the management of records in those sites (including Team-based sites).

  • The best way to do this is to add records managers to a Security Group and then add that Group to the SCA group of every site. This access could be deferred for sites that contain very sensitive information, although typically records managers would have access to all records, including if they had an EDRMS. And, access is always recorded in audit logs or the local site ‘viewers’ (where enabled) and ‘last modified by’ information.

Access to the chat content of Teams (including 1:1 chats) will not normally be required; some understanding of the content could be inferred from the name of the Team or the SharePoint content. If necessary, Global Admins or a Compliance Admin can run a Content Search across Teams to find chat content, and/or export that content by an individual person or subject.

Records managers will also need to advise on the appropriate retention policy or policies that need to be created and then applied to:

  • The chat content in 1:1 chats.
  • The chat content in the various Teams.
  • SharePoint sites linked with Teams.
  • Exchange mailboxes.
  • OneDrive for Business accounts. An additional consideration is how long the content of inactive ODfB acccounts should be retained via the ‘Storage’ policy (default is 30 days then permanent deletion).
  • SharePoint sites not linked with MS Teams. This includes whole sites as well as library-based retention policies.
  • Office 365 Groups (mailbox/SharePoint site). If linked with a Team, a second retention policy is required for the Team chat content retention (second dot point above). For example, one policy ‘GroupABC’ and a second policy ‘GroupABCTeamChat’.

As many of the above retention policies replace the need for backups, records managers need to discuss the options with their IT colleagues.

Forward looking implications

Ideally, there should be some form of governance around the creation of new Teams in MS Teams. These governance arrangements might include:

  • The necessary access for records managers. For example, Site Collection Administrator on every site, and/or a customised Compliance Admin role to create and access retention policies.
  • Controls around the creation of new Teams, including naming conventions. If not controlled, what processes will ensure that records are properly managed.
  • Retention implications. For example, can the new site and/or the channel chat content be covered by another retention policy – e.g., ‘All Teams with assessed low-level working content should be kept for 5 years’.
  • Simple best practice guidance for all new users, including on how to share and co-author.
  • Retention policies for all Microsoft 365 content, not just SharePoint.
  • Reviews of the content of OneDrive for Business accounts of departed end-users, especially for people in senior or decision making positions. It is relatively common practice for end-users to delete (and download) this content before they leave their jobs.
  • Monitoring and oversight of content, including access to reporting dashboards.

So, is Microsoft 365 just Teams, Outlook and Office (in Edge)?

Perhaps, yes.

For many, or not most information based end-users, MS Teams is likely to become the primary interface to Microsoft 365 collaboration team spaces including SharePoint and OneDrive. Just like Outlook, Teams will probably be left open all day.

In theory, the volume of low-value emails, and emails with attachments, should reduce over time.

The developing role of records managers

In this new world, the role of records managers will change from being the curators of records copied to and stored in a separate ‘records and document management’ system, to being records compliance analysts or perhaps, corporate knowledge and information managers and content analysts.

They will learn what the Graph can do, and help to guide AI tools including machine learning and machine teaching, Project Cortex and SharePoint Syntex. They will be responsible for monitoring content across the Microsoft 365 platform, creating and applying retention policies and managing the outcome of those policies, working more interactively with the Graph, and with a range of data.

In organisations that have a requirement to transfer records to archival institutions, the new knowledge and information managers will have a key role in ensuring that this data is suitable for transfer.

They might even have oversight of old paper records gathering dust until they can be destroyed.