The ability to manage records ‘in place’ in SharePoint has existed since around 2013. But this is not the same thing as leaving records where they were created or captured and managing them there – ‘in place’.
This post explains the difference between the two ‘in place’ options. In brief:
The Microsoft ‘in place’ model is based on making the distinction between non-records content and content declared as records (as per DOD 5015.2), that may be stored in the same SharePoint site, or using Exchange in-place options.
The other ‘in place’ model is simply based on leaving records and other content where they were created or captured, and managing it there – including (where necessary) by applying the ‘in place’ options in the previous point.
The Microsoft in-place model
SharePoint
The Microsoft in-place model for managing records in SharePoint is based on the requirement to comply with the US Department of Defense (DOD) standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’, usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, originally published in 11 April 1997.
Section C2.2.3 ‘Declaring and Filing Records’ of the standard defines 26 specific requirements for declaring and filing records, including the following points:
The capability to associate the attributes of one or more record folder(s) to a record, or for categories to be managed at the record level, and to provide the capability to associate a record category to a record
Mandatory record metadata.
Restrictions on who can create, edit, and delete record metadata components, and their associated selection lists.
Unique computer-generated record identifiersfor each record, regardless of where that record is stored.
The capability to create, view, save, and print the complete record metadata, or user-specified portions thereof, in user-selectable order.
The ability to prevent subsequent changes to electronic records stored in its supported repositories and preserving the content of the record, once filed
Not permitting modification of certain metadata fields.
The capability to support multiple renditions of a record.
The capability to increment versions of records when filing. Linking the record metadata to the record so that it can be accessed for display, export.
Enforcement of data integrity, referential integrity, and relational integrity.
Microsoft’s initial guidance on configuring in place records management describes how to activate and apply this functionality primarily in SharePoint on-premise. It is still possible to apply this in SharePoint Online (but see below). The SharePoint in place model refers to a mixed content approach where both records and non-records can be managed in the same location (an EDMS with RM capability):
Managing records ‘in place’ also enables these records to be part of a collaborative workspace, living alongside other documents you are working on.
The same link above, however, also refers to newer capability that was introduced with the Microsoft 365 Records Management solution in the Compliance admin portal. This new capability allows organisations to use retention labels instead to declare content as records when the label is applied, which ‘effectively replaces the need to use the Records Center or in-place records management features.’
The guidance also noted that, ‘… moving forward, for the purpose of records management, we recommend using the Compliance Center solution instead of the Records Center.’
Exchange
A form of in-place management has also been available for Exchange on-premise mailboxes, with in place archiving based on using archive mailboxes – see the Microsoft guidance ‘In-Place Archiving for in Exchange Server‘.
One draw-back of this model is that the (email) records in these mailboxes were not covered by the same DOD 5015.2 rigor as those in SharePoint, but they could at least be isolated and protected against modification or deletion, for retention, control and compliance purposes.
Microsoft Exchange Online Archiving is a Microsoft 365 cloud-based, enterprise-class archiving solution for organizations that have deployed Microsoft Exchange Server 2019, Microsoft Exchange Server 2016, Microsoft Exchange Server 2013, Microsoft Exchange Server 2010 (SP2 and later), or subscribe to certain Exchange Online or Microsoft365 plans. Exchange Online Archiving assists these organizations with their archiving, compliance, regulatory, and eDiscovery challenges while simplifying on-premises infrastructure, and thereby reducing costs and easing IT burdens.
The new ‘in place’ model
A newer form of in-place records management has appeared with Microsoft 365.
Essentially, the new model simply means leaving records where they were created or captured – in Exchange mailboxes, SharePoint sites, OneDrive or Teams (and so on). and applying additional controls where it is required.
The newer model of in place records management is based on the assumptions that:
It will never be possible to accurately or consistently identify and/or declare or manage every record that might exist across the Microsoft 365 ecosystem. For example, it is not possible to declare Teams chats or Yammer messages.
Only some and mostly high value or permanent records, will be subject to specific additional controls, including records declaration and label-based retention.
The authenticity, integrity and reliability of a some records may be based more on system information (event metadata) about its history, than by locking a point-in-time version.
Microsoft appear to support this dual in place model with their information governance (broader controls) and records management (specific controls, including declaration) approach to the management of content and records across Microsoft 365, as described in the Microsoft guidance ‘Information Governance in Microsoft 365‘, which includes the graphic below, modified to show the relationship between the two in place concepts.
The relationship between Microsoft’s ‘in place’ focussed records management, and managing everything (including records) in place.
In place co-existence
Can both in place models exist? Yes. There is nothing to prevent both in place models existing in the same environment, in which some records may need to be better managed or controlled than others, but it is important to understand the distinction between the two when it comes to applying controls.
Image: Quarantine Building, Portsea, Victoria Australia. Andrew Warland 2021
One of the first things to understand, when managing records in Microsoft 365, is where the records are stored.
Generally speaking, most records will be stored in either (a) Exchange mailboxes (which also store the ‘compliance copies’ of Teams chats) or (b) SharePoint sites (including sites linked with MS Teams). Some emails may be copied to SharePoint and some records may also end up in OneDrive accounts.
Records should provide evidence of business activities and may also be associated with metadata. But while records may be evidence of such activities, compliance requirements may differ. For example:
Some records must be retained permanently and transferred to archival institutions as a public record, while others may be of temporary or transient value.
The evidentiary value of records may depend on the context and content of the record. For example, records detailing a high value contract decision versus the minutes of a low-level team meeting.
Decision tree
With the above differences in mind, the following is a suggested decision tree for the management of records, in a controlled Microsoft 365 environment. This decision tree will not apply in organisations where end-users can create their own Teams and SharePoint sites.
The decision tree below starts with the simple question to the end-user or business area wanting a new Team or SharePoint site:
The answer to this question should help to identify what type of SharePoint site, or Team (with a SharePoint site), is required to manage the records. Note that ALL of the boxes should be considered BEFORE a site (or Team) is created.
Note: the box outline colours represent the four different types of site that are likely to be created.
How will the content be accessed?
It is important to understand how end-users will create, capture, manage and access the content that is stored in SharePoint.
Microsoft have made it clear that they expect ‘most’ end-users will access SharePoint via Teams.
They may also sync the SharePoint document library to File Explorer. If that option is not removed, then it is important to ensure that end-users know how to access the Recycle Bin and version history by going to the Files tab in Teams and clicking on ‘Open in SharePoint’.
If end-users are likely to mostly use the Teams Files tab or the synced version in File Explorer, it may be useful to suggest that they save the SharePoint site as a favourite in their browser.
It is important not to lose site of the Microsoft 365 Group-based option which gives the Group an email account that can be accessed via Outlook (the SharePoint site for the Group can also be accessed directly in Outlook Online).
What type of SharePoint site will be required?
Once the two points above are clear, it is then possible to understand what type of site will need to be created, and how the records will (or will need to) be managed. For example, if Content Types and/or additional metadata is required.
As a general rule, the SharePoint sites linked with Teams are more likely to remain simple, folder-based libraries because, while additional Content Types and metadata can be used, end-users may not react favourably to additional document saving requirements, especially if they access the library via File Explorer.
Organisations with more complex recordkeeping requirements might consider establishing SharePoint sites dedicated for that purpose and accessed via the browser. These may and may not be linked with Microsoft 365 Groups.
Creation method
The following shows how the new site will be created (if it’s not scripted). Microsoft recently announced that it will add the ‘Created From’ column to the list of active sites in the SharePoint admin portal.
Security Controls
Different site types are likely to have different security controls. Team/Microsoft 365 Group-based sites use the Group Owners and Members to restrict access to the SharePoint site (the Owners and Members Group are added (respectively) to the Owners and Members permission group of the SharePoint sites), whereas a SharePoint site not linked with an M365 Group is more likely to use Security Groups.
Communication sites are likely to have very few Members and ‘Everyone except external users’ added to the Visitors permission group.
Retention
Finally, a decision needs to be made about what type of retention policy will be applied to the SharePoint site or the content stored in it. My suggestion is to apply a broad ‘safety net’ retention policy to all SharePoint sites and then use label-based policies as necessary.
As much as possible, the content of sites should ‘map’ to business function and the retention classes in that business function. For example, try to avoid storing legal records with HR records and property records on the same site. Retention management will get complicated.
Label-based policies might have no disposal outcome (‘Do nothing’) but they ensure that the records cannot be destroyed and are labelled accordingly. For example, records that are of permanent value could be labelled with a ‘Do nothing’ retention label in dedicated document libraries where edit rights have been removed, thereby locking them from disposal.
Provisioning
Once the site has been created, it should be provisioned with three key elements shown below.
Records managers should be assigned to a Security Group that is added to the Site Collection Admin area of every SharePoint site. This will allow records managers to manage the content of those sites.
Document IDs should always be enabled and configured (remembering they have a 12 character prefix limit)
The SharePoint Viewers feature should be enabled on every site to provide visibility of who viewed the content.
The decision tree should help to guide the create of new sites and Teams (which have a SharePoint site) for the storage and management of records.
SharePoint has a long, twenty-year legacy. Over that period, organisations and third-party developers have developed and implemented a range of solutions and options across SharePoint. Many of these have been in place for a long time.
The arrival of the new look, ‘modern’ SharePoint Online (SPO) in the last five years has created a dilemma for organisations – migrate to SPO and lose the older functionality, or retain some of the functionality by creating ‘classic’ sites in SPO?
This post was inspired from discussions with a number of organisations that were either creating or using classic site templates in SPO – sometimes as the default option – or indicated a resistance to moving to SPO because of the ‘loss of functionality’.
What are modern sites in SPO?
The default options to create in SharePoint Online are either (a) modern team sites linked with a Microsoft 365 (M365) Group (most common) or (b) communication sites.
The ‘Create a site’ options in the SharePoint admin portal. These are just two of several ways to create new sites.
Modern team sites linked with M365 Groups are created:
From the SP Admin portal (first option ‘Team site’ – above)
By end users from their SharePoint portal (if that feature is not disabled)
When a new Team is created via MS Teams or the MS Teams admin portal
When a new Group is created from Outlook or from the Microsoft 365 admin portal
When a new ‘shared folder’ is created from OneDrive
Modern team and communication sites are almost identical in terms of core functionality. The primary visible difference is that: (a) team sites have the navigation bar at the left and are usually used to store documents and other digital objects; whereas (b) communication sites have the navigation bar along the top (which can be made into a mega menu easily) and are usually used to store information on pages. They may also store documents in document libraries.
The default page web parts are the same on both; for more information about web parts, see this Microsoft guidance ‘Using web parts on SharePoint pages‘.
The PowerShell script to create a new Group-based site uses the command ‘New-SPOSiteGroup’, which creates both the Group, the SharePoint site, and a Group-based mailbox in Exchange Online.
Note that M365 Group-based sites use the template ‘GROUP#0’.
What are classic sites in SPO?
Other, ‘classic’ SharePoint site templates can be created from the ‘Other Options’ section of the SP Admin portal.
‘Other options’ section of the Create Site section of the SP admin portal
The options, as grouped, are as follows. Note that for all of them, the SharePoint 2013 experience (look and feel) is used.
Collaboration: Team site (classic experience), Developer Site, Project Site, Community Site
Enterprise: Document Center, eDiscovery Center, Records Center, Team Site – SharePoint Online configuration, Business Intelligence Center, Compliance Policy Center, Enterprise Search Center, My Site Host, Community Portal, Basic Search Center, Visio Process Repository
Publishing: Publishing Portal, Enterprise Wiki
Custom: (if a custom template exists)
SharePoint sites, including using classic templates, can also be created using PowerShell and the ‘New-SPOSite’ command (with additional configuration details, see this Microsoft docs page). The PowerShell command will needs to include the template to be used:
SharePoint site templates
The PowerShell script to create a new classic site is slightly different from a modern Group-based siste, using ‘New-SPOSite’:
Organisations may have genuine technical reasons to want (or need) to create and continue to create some SharePoint Online sites using one of the classic templates. For example:
They may have many sites that were created in the legacy on-premise environment. Upgrading these sites will take time, so it may be easier simply to migrate from one to another ‘as is’.
A site or sites may have integration points with other systems that require the older functionality.
Classic sites should not normally be created simply because of loss of (or changed) legacy functionality (e.g., a preference for older style web parts), or simply a preference to continue to use older functionality.
As much as possible, organisations should aim to create modern sites. In some – or many – cases, this may require the organisation to adopt the new modern options.
Classic to modern functionality changes
Examples of broader-level functionality that has largely been replaced with new functionality, or may be deprecated in the future, include:
Publishing sites were often used for customised intranets, often with multiple levels of sub-sites. These have been replaced by communication sites and hub sites.
Publishing features, including features sometimes used for navigation purposes and linked with terms in the Term Store were a common ‘classic’ option. These have been replaced by new mega menu functionality.
Older web parts on pages have been replaced by modern web parts.
The Document Center and Records Center templates are more or less now superseded by in-place records management functionality, leaving records where they were created or captured.
The Business Intelligence Center has mostly been replaced by PowerBI or similar tools and interfaces.
Community Sites and Community Portals have been replaced by communication sites, Teams and Yammer.
The need for sub-sites has been replaced by creating one or more sites as hub sites and linking other sites to the hub.
Many of these classic options have been around for at least a decade, making it sometimes hard to move to the newer options.
Functionality differences between classic and modern sites
Possibly the most common of the classic sites that continue to be created, however, are classic team sites (‘STS#0’). In some cases, these site templates are the default option. It is not clear why these continue to be created when the STS03 (non M365 Group) template has the same basic functionality:
The same Site Settings.
The same ways to add a page, library or list (from Site Contents or from the gear icon – Add an app).
The same library and list settings (including the ‘Information management policy settings’).
One key difference is the look and feel, something that may become less of an issue with Teams becoming the primary UI to SharePoint.
It is still possible on classic (STS#0) sites to create classic style pages using classic web parts, as can be seen in the screenshot below from a classic (STS#0) site:
What’s the problem with creating new classic sites?
As noted above, however, just because it remains possible to create and use classic functionality isn’t a reason to keep it in face of newer and often better options – especially when end-users won’t even see the site.
From my own experience, anything that remains in an older version of any technology becomes (or is likely to become) technology debt.
It may be appropriate – temporarily – to migrate some older on-premise sites to the classic template until they can be upgraded, but there is no valid reason in my opinion to create new classic sites as a default option for all new sites.
What do you think? Do you continue to create classic sites?
Photographs (or, for that matter, any visual work) can be more powerful as a record of something than a written record. Some of the iconic events in history are only known from the visual record.
Photos (and videos) are ubiquitous in the digital age. They are captured in multiple ways and stored on a range of media including computers, portable drives, mobile devices and the cloud (including online storage and social media).
Digital photos can be easily changed or ‘photoshopped’, to use the more common term.
But records management requires us to ensure the authenticity, reliability and integrity of records. How can we ensure this for photographic records?
This post examines how SharePoint can be used to manage (some) photos as records.
What is a digital photo?
For the purpose of this post, a digital photo is any image that is captured and stored electronically as bits in a recognised image format. It excludes digital photos (including scanned or digitised paper records) that are embedded in PDF files.
To quote from the Wikipedia article titled ‘Digital Image‘, a digital photo is:
‘… an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively. Depending on whether the image resolution is fixed, it may be of vector or raster type. By itself, the term ‘digital image’ usually refers to raster images or bitmapped images (as opposed to vector images).’
According to the definition, a digital photo is a collection of bits set out in a map that a computer interprets to display it on a screen. For more detail on bitmaps, see this Microsoft page ‘Bitmaps‘.
When to use SharePoint to store digital photos
Almost any digital object that can be stored electronically can be captured in SharePoint. But, just because they can doesn’t mean they should.
It is important to keep in mind that SharePoint may be not be the most suitable option for storing digital photographs in general. The volume, size and intended usage of the photos are all likely to influence the decision to use SharePoint. Organisations may need to consider other ways to store and manage digital photos (and other digital media) such as dedicated digital asset management systems.
However, SharePoint may be a good option to store digital photos as records if those photos:
Need to be kept as a record of something.
Support or relate to other content in the same document library. For example, photos that relate to or support building plans (which themselves may be scanned images in PDF form) or construction.
Are about a specific subject. For example, photos of damage to a physical object.
Are relatively low in volume and file size.
Factors to consider before storing digital photos in SharePoint
The following points should be considered before storing any digital photo (or digital media) in SharePoint.
File names
Digital photos are usually stored on devices that capture them with meaningless, device-generated names such as ‘20210423_123192321’, ‘DSC_0330’, or ‘n594015825_1706121_4959’. These types of names provide no clue to the content when the image is saved to SharePoint, as shown below.
Ideally, the device-generated name should be replaced with a more meaningful name as shown below.
We can see from the unique Document ID that it is the same record. The version history also tells us that something was changed but the file size hasn’t changed.
Keep in mind that every change to the file name or any other metadata added to the library will create a new copy (‘version’) of the image. In the version history above, there are now four versions each 3 MB in size.
The Activity section of the information panel to the right (which also provides a preview of the image) also provides some key information to support the version history information:
Does changing the name potentially compromise a key element of metadata? Who should change the name, and when?
Organisations should consider establishing naming conventions for photographs to help with findability and context.
Format
Almost any type of digital object can be stored in SharePoint. SharePoint also supports the ability to view almost every type of contemporary digital photo format (as described in this Microsoft page) so format should not be a problem when it comes to viewing the file.
However, if the organisation plans to store digital photos in older or less common formats, it will need to ensure that these can be accessed for as long as required. This will generally mean ensuring that the appropriate software application is available to anyone who needs to access (open/view) the photos.
Alternatively, consideration should be given to saving the photos in different, more common formats.
Size/Resolution
The size of a digital photograph may affect its usability as a record. The two photos below are of the same scene but the first photo is very low resolution (= small size, 63 KB), while the second is a section of a much large photo (= large size, 3.7 MB). The second version is clearly a more accurate record.
Digital photos used as records should provide sufficient detail to be useful as records. Accordingly, in most instances, the original photo, not a re-sized version, should be saved.
Reliability as a record
As noted above, photos can be easily modified, sometimes making it difficult to know if it accurately reflects the image it purports to capture. This is also an issue for any form of visual record, including drawings.
Metadata created when the photo was taken and stored in the photo’s properties (including EXIF metadata) can provide evidence of the reliability of the photo as a record, even when the record was stored in SharePoint. The following are the key metadata properties that are usually created and stored with a digital photo:
Size.
(Date) Created. The date and time when the photograph was created as a photo.
Date taken. This is, for most digital photos, the same date and time as the ‘created’ date.
(Date) Modified. This should be very close to the original date and time created on the original photograph, but may be several seconds different. If the photograph has been modified (including simple photo adjustments or ‘photoshopped’), it will show a different date and time which might give a clue to its reliability as a record.
Image dimensions, width, height, horizontal and vertical resolution, bit depth
Camera details including F-stop, exposure time, ISO speed, flash mode
Storing digital photos in SharePoint Online
Note that SharePoint previously had the option to create a dedicated ‘Picture Library’. This is no longer necessary as any document library can be used to store any digital object, and SharePoint Online document libraries now have additional options to view the photos.
Any digital photo can now be saved to a standard SharePoint document library, including via the ‘Files’ tab in MS Teams.
Metadata
Unlike some EDRM systems, SharePoint does not ‘capture’ or extract the details of digital objects when they are saved to a document library. Instead, these remain with the original digital object.
If digital photos are to be saved to a SharePoint Online document library, consideration should be given to using existing or additional metadata columns to describe the image, for example, the ‘Image’ column (small version preview that is stored separately in the Site Assets library >’Lists’ folder > GUID-named folder).
The following screenshot shows a preview image of the main photo.
Viewing options
SharePoint includes three options to view the content in a document library list view: List, Compact List, or Tiles.
The following is an example of a single digital photo stored in a document library (same as the one in the other options earlier) set to the ‘Tiles’ view:
Reliability as a record
Once saved to SharePoint, the photo’s reliability as a record can be determined from version history.
Additional protections may include access/permission controls, retention and/or information security labels.
Summary
SharePoint can be used to store (some) digital photos as records, but it should not generally be used as a general storage location for digital photos. Other dedicated digital asset management systems may be more suitable for that purpose.
Before storing digital photos in SharePoint, organisations should establish procedural rules or principles including (re-) naming conventions, format and file size requirements.
SharePoint does not extract and store the metadata from digital objects, which means that digital photos should retain metadata that shows when they were created or modified, and other details about the photograph which will provide evidence to support the authenticity of the photo as a record. Some consideration should be given to adding additional metadata to help describe digital photos as records.
A combination of version history, access controls, information protection and retention labels should provide sufficient controls to ensure the reliability, integrity and authenticity of records – or at the very least provide the evidence of changes that may be made.
There are two sides to the question of authenticity, reliability and integrity:
Knowing if the photo that has been uploaded is a correct record of what it purports to be.
Preventing the photo from modification or deletion or tracking any modifications that may occur.
It might seem impossible to know if a photograph is what is purports to be, but its metadata payload may provide the detail required.
An electronic document management system (EDMS)supported day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
An electronic records management system (ERMS) was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted.
It could be argued that SharePoint began its life in 2001 as an EDMS and began to include ERMS functionality when SharePoint 2010 was released. So, how is it not an EDRMS?
In simple terms, unlike a traditional EDRMS model that was intended to be the repository for all business records, SharePoint is only one of several repositories that are used to store and manage business records. Instead of centralising the storage and management of records, systems such as Microsoft 365 provide centralised management of records wherever they are stored.
The EDRMS model
Almost all EDRM systems since the mid-1990s have had the following characteristics:
Compliance with a standard. For example, DOD 5015.2 (1997), AS 4390 (1996)/ISO 15489 (2001), ISO 16175 (2010), VERS (1999), MoReq (2001)/MoReq2, etc.
A relational database (for the various functions, metadata and controls (see diagram below) and a separate file share (for the actual content/records), accessed via a user interface.
An expectation that most or all (digital) records would be copied from other systems used to create or capture them to the EDRMS for ongoing storage and protection. This outcome might be achieved through integration with other systems, for example to copy emails to the EDRMS.
The integrated EDM/ERM system model was described in the following diagram in 2008 by the International Standards Organisation committee, ISO/TC171/SC2, in its document ‘Document management applications’:
There are two key problems with the EDRMS model:
They do not (and cannot) realistically capture, store or classify all business records. Emails have remained a problem in this regard for at least two decades. Additionally, there is now a much wider range of born-digital records that remain uncaptured.
Many of the born-digital records that were copied to the EDRMS continue to exist where they were first created or captured.
Any organisation that has been subject to a legal subpoena or Freedom of Information (FOI) request for records will know the limited extent to which EDRM systems provide evidence of all business activities or decision making.
And yet, almost every organisation has a requirement to manage records. There is clearly a disconnect between the various requirements and standards to keep records and the ability to keep them.
But SharePoint (on its own) is not the answer.
Why isn’t SharePoint an EDRMS?
SharePoint was never promoted or marketed as an EDRM system. It is not even a relational database (see this 2019 post by Sergey Golubenko ‘Why SharePoint is no good as a database‘ for details).
To paraphrase from Tony Redmond’s recent (21 March 2021) post ‘SharePoint Hits 20: Memories and Reflections‘, SharePoint was originally designed to allow content (including page-based content in the form of intranets) to be accessed via the web, thereby replacing network file shares.
In practice, most organisations retained their network file shares, built their intranet using SharePoint, and otherwise used SharePoint to manage only some digital content.
It could be argued SharePoint started out as an EDM system and became more like an ERMS when more recordkeeping capability was introduced in SharePoint 2011. But that doesn’t make it an EDRMS in the traditional sense of being a central repository of business records.
The reality is that records are, have been, and will continue to be created or captured in many places:
Email systems. In Microsoft 365, Exchange Online mailboxes are also used to store the ‘compliance copy’ of Teams chats messages.
Network file shares.
SharePoint, including OneDrive.
Other online document management systems, including Google Drive.
Text, chat and instant messages often created in third-party systems, often completely inaccessible or encrypted.
The type and format of a record can vary considerably. For example:
An emoji.
A calendar entry.
A photograph or video recording (including CCTV recording).
The recording of a meeting, in video or transcript form, or both.
Virtual reality simulations.
Social media posts.
3D and digital drawings (e.g., via digital whiteboards).
And, of course, all the data in line of business systems.
Instead of trying to save records into a single EDRM system, Microsoft 365 provides the ability to apply controls over the management of most records where they were created or captured – in email (including archived social media and other records), Teams chat, SharePoint, or OneDrive, and Yammer.
Is there a need for an EDRMS?
There is nothing stopping organisations acquiring and implementing traditional EDRM systems, or even setting up some SharePoint sites, to manage certain high value or permanent records, including some records that can be copied from other create/capture systems (such as email).
But there is also a need to address the management of all other records that remain in the systems where they were created or captured.
In most modern organisations, this requires broader controls such as applying minimum retention periods at the backend, monitoring usage and activity, and the proactive management of disposals. Not trying to copy them all to SharePoint.
One of the most confusing aspects of Teams and SharePoint in Microsoft 365 is the relationship between permission groups used to control access to both of these resources. This is especially the case as every Team in MS Teams has an associated SharePoint site (the ‘Files’ tab).
This post explains how permission groups work between MS Teams, Microsoft 365 Groups and SharePoint.
SharePoint permission groups
Before discussing how Teams permissions relate to SharePoint, here is a brief reminder of how SharePoint permissions work.
SharePoint has always had three default permission groups, prefixed by the URL name of the site, as shown in the screenshot below (the name of the site always prefixes the words Owners, Members and Visitors).
Site Owners
People (including in a Group, see below) added to the Owners permission group have full access (full control) to all parts of the site and are usually responsible for managing the SharePoint site. There would normally be two or three site owners.
Site Members
People (including in a Group, see below) added to the Members permission group have add/edit (contribute) rights.
Site Visitors
People added to the Visitors permission group have read-only (view) rights.
These permissions are set at the site level and inherited on everything in the site, unless that inheritance is broken and unique permission are applied. Additional permission groups can be created as necessary but most SharePoint sites only use the default Owners, Members and Visitors groups.
Microsoft 365 Groups
Microsoft 365 Groups were introduced in 2017 and control access to resources, like Security Groups.
However, unlike Security Groups, which usually provide access to individual resources (such as a single SharePoint site, or Line of Business (LOB) system), Microsoft 365 Groups control access to multiple linked Microsoft 365 resources.
Microsoft 365 groups, distribution lists, mail-enabled security groups, and security groups (collectively referred to as Active Directory (AD) groups, are all created in ‘Groups’ area of the Microsoft 365 Admin portal.
When a new group is created, the following options appear.
As noted above, Microsoft 365 groups are recommended. It is important to understand the relationship between Microsoft 365 groups, Teams and SharePoint.
A new group has a visible mailbox and a Team is created by default
When a new Microsoft 365 group is created (from the dialogue above), it creates:
At least one Owner must be specified. The Owner/s are responsible for managing the Members group.
An Exchange mailbox with the same email @ name as the Microsoft 365 group. The mailbox is visible in Outlook to the members of the Group.
A SharePoint site with the same URL name as the Microsoft 365 group.
By default (unless the checkbox is unchecked), a new Team is also created in MS Teams.
When a new Team is created from MS Teams, or a new SharePoint Team site is created, it creates:
A Microsoft 365 Group with an Exchange mailbox and a SharePoint site (‘Files’ tab).
The name of the Team becomes the name of the Group and the SharePoint site.
The mailbox is not visible in Outlook and is only used for calendaring and for the storage of Teams chats (in a hidden folder).
Importantly, when a new Microsoft 365 group or Team is created (which creates a Microsoft 365 group), the Group Owners: (a) are the same as the Team Owners and (b) are added to the SharePoint Owners permission group, as explained below. .
Group/Team Owners and Members
In other words, the Microsoft 365 group owners (group) is added to the SharePoint site owners permission group – a ‘group within a group’.
That is, the Microsoft 365 group controls access to the Team and the SharePoint site as shown in the diagram below. Security Groups may also be added to the Microsoft 365 Group site, but this does not provide access to the Team.
The relationship between Microsoft 365 Groups, Teams and SharePoint
This ‘group within a group’ model is visible from the ‘Site Permissions’ section of the gear/cog icon as shown below (the name of the Microsoft 365 Group/Team/SharePoint site is ‘SharePoint Admin’). The SharePoint Admin Group Owners (group) is in the SharePoint site owners group, and the SharePoint Admin Group Members (group) is in the Site members group.
If a mouse hovers over the Group ‘icon’ (in the above example, GO or GM), it is possible to view the members of the Group and, for Owners, to modify that list. Confusingly, the ‘GM’ in the SharePoint site permissions group becomes ‘SG’ in the drop down list.
You can also see the ‘group within group’ model from the back-end ‘Advanced permissions’ section of the SharePoint site, but you cannot manage the Microsoft 365 Group members here.
Implementing the model
As with Security Groups, the members of Microsoft 365 Groups will usually be a logical group of people who require access to something, in this case access to the SharePoint site or the Team (for chat, files, or other resources).
The main thing to remember is that membership of the (backend) Microsoft 365 Group provides access to BOTH the Team and the Team’s SharePoint site (the ‘Files’ tab in a Team).
Every Team in MS Teams will usually consist of the members of a logical group with a common interest – a business unit, project team, or with some other work relationship, for example, the members of a committee. The Team Owners are responsible for managing the Team Members.
The Team Owners are the SharePoint site owners and are responsible for managing the site if they decide to access it directly. The Team Members are the SharePoint site members and have the ability to add or edit content, usually via the ‘Files’ tab in Teams.
Note: Security Groups with the same members as Microsoft 365 Groups (and Teams) may already exist. There is no need to add a Security Group if it has the same members as a Microsoft 365 Group.
As noted earlier, a Group/Team does not have visitors with read-only rights. Every Member of the Team has add/edit access to both the Team and its associated SharePoint site.
If there is a requirement to give specific other people either add/edit or read-only access to the SharePoint site, that outcome is achieved by adding people by name, or a Security Group, to either the SharePoint Members or Visitors group.
If there is a requirement to give everyone in the organisation either add/edit rights, or read only access, to the SharePoint site, that outcome is achieved by adding ‘Everyone except external users’ to either the SharePoint Members or Visitors group.
External guests may also be added to the Team and the Team’s SharePoint site.
SharePoint Online (SPO) is the primary location to store digital objects and documents in Microsoft 365.
In this sense, it replaces on premise network file shares and drives as a location to store information although a bit ironically, it can also be accessed from File Explorer.
In Microsoft Teams channels, SPO sits behind the scenes via the ‘Files’ tab. This tab presents the content of the folder from the default Documents library that has the same name as the channel (General channel = General folder in the Documents library).
SPO can also be accessed via the SPO app on a mobile device and even directly from Outlook Online.
So, which one is the best way to access your information stored in SPO? The answer is – it depends on what you need to do. You could use all five options.
This post describes five ways to access content stored in SPO, with positives and negatives.
1- For day to day use – synced via File Explorer
Most people use File Explorer to store, organise and access their content, and generally only work from a few folder locations They don’t want to have to open a browser or other application.
The good news is that they can sync SPO document libraries to File Explorer and work directly from there. They can sync a document library from the SPO site or via Teams.
Syncing a library from SPOSyncing the same library in Teams
So, for most users, the main change will be a different location to access their content from File Explorer. It looks a bit like the image below. The first words after the folder are the site name, then the hyphen, then the library name. To make it as easy as possible, the library should ideally be the same as the old top level folder on the network file share.
Syncing downloads the metadata about the content that is stored the SPO library. The content is not downloaded to the location device (C: drive usually) until it is opened. From that point on, there is a ‘local’ copy. If that local copy is modified, the changes are synced back to the SPO site.
The content of a synced library (example)
End users can share directly from File Explorer. The dialogue box is exactly the same as the SPO/Teams option. This makes it much easier to share rather than attach a document to email.
End users can co-author an Office document opened from File Explorer provided they have the most recent version of Office installed. Other end-users may be accessing the same document at the same time via the online versions of Office in Teams or SPO.
The main negatives about the sync option are as follows:
Only basic File Explorer metadata is visible – Name, Date modified, etc. If end-users need to add or see added metadata columns, they will have to access this via the Teams Files tab or the SPO library.
More restrictive, granular-level permissions cannot be set (e.g., on a folder or a document). These have to be set from SPO.
End-users cannot access the version history or Recycle Bin (to restore deleted items)
A folder added at the same level as a Teams channel-mapped folder, will not create a new channel. However, folders created under the Teams channel-mapped folder will be visible.
2 – To collaborate – access via MS Teams
Microsoft have positioned Teams to be an ‘all in one’ collaboration application, allowing end-users to chat, upload and store files, have video calls and more.
In the 1:1 chat area of Teams, the ‘Files’ tab presents documents shared from the OneDrive account of a participant in the chat. OneDrive is, of course, a SharePoint service.
In the Teams area of Teams, the ‘Files’ tab displays, for each channel, a folder with the same name in the Documents library of the Team’s SPO site (every Team has a SPO site). End-users can create, capture and manage content in the channel’s Files tab, as shown in the example below.
Files and folders in the ‘General’ folder of the SPO Documents library
There are several positives of accessing SPO via Teams:
Teams includes additional collaboration options, including the ability to chat at the same time as a document is viewed or edited. Content (and folders) can be shared easily.
If this library is synced to File Explorer, any changes made in either location will be automatically updated in the other.
Any metadata columns added to the SPO document library will be visible here.
The SPO site can be accessed directly from a link on the menu bar.
The main negatives of accessing content via the Teams ‘Files’ tab are as follows:
End-users have to open and use an unfamiliar interface (although it has become more common)
The three dot ‘ellipsis’ menu is limited compared with the full SPO version. For example, it does not include versions. See the screenshot below.
The Recycle Bin is not accessible – you have to click on ‘Open in SharePoint’ to access it there.
Ellipsis menu in the Teams Files tab
3 – To see Group emails and files – access via Outlook
Microsoft 365 Groups are a key element in Microsoft 365 and provide a range of functionality that can replace or supplement existing access control and collaboration purposes.
Every Microsoft 365 Group has an Exchange mailbox and a SPO site, and can be linked to a new Team.
If the Microsoft 365 Group is created first, the Exchange mailbox is visible to the Group members in their Outlook.
If the Team is created first, a Microsoft 365 Group with a SPO site and mailbox are created, but the Exchange mailbox is not visible via Outlook. It is there only to store the compliance copies of chats and for calendaring purposes.
Microsoft 365 Groups can be used to replace shared mailboxes or to give business areas the ability to access both email and SPO-stored content from the same location (as well as via File Explorer and Teams).
In the first screenshot below from Outlook Online, you can see a square ‘documents’ icon to the right of the words ‘Send email’. This square icon opens the Group’s SPO documents library (next screenshot). In the installed version of Outlook, clicking this link opens the SPO site in the browser window.
The Group mailbox in Outlook Online
In the screenshot below, you can see the Group’s files from the Documents library, General folder in Outlook Online.
The Group’s files from Outlook Online
The main positives of accessing SPO content from Outlook Online is that it is relatively easy to move between the Group’s emails and document stored in SPO. End-users can open and documents directly from Outlook, although this (currently) opens Word Online.
The main negative of accessing SPO content from Outlook Online is the limited functionality available from the ellipsis menu, including the inability to see previous versions or access the Recycle Bin. It is also not possible to modify the view (display columns). However, any changes made to the view in either Teams or SPO will be visible in the Outlook Online view.
The ellipsis menu in Outlook
4 – Anywhere, anytime – via mobile devices
Both Google and Apple provide the SharePoint mobile app, as well as apps for OneDrive, MS Teams, Outlook and Microsoft Office.
This means that mobile users can access their SharePoint content directly from a mobile device. They can also use the SharePoint app to search for any content they have access to.
The document library from a mobile device
The main negative with accessing SharePoint from a mobile device is the functionality is very limited. End-users can access and edit the content (if they have the relevant app installed), and can share the documents, but that’s all.
On the other hand, they can access the content anywhere, any time. That makes it very useful.
5 – From the browser – the full SharePoint experience
Of course, the end-user may also access SharePoint from the browser and it is usually a good idea to let them know they can do this for the reasons below.
They can access the browser version in multiple ways:
By clicking on ‘Open in SharePoint’ from the Files tab in Teams.
By saving the site as a favorite in their browser.
By clicking on the files option in a Group’s email inbox area in Outlook installed on the desktop.
The main or common reasons they might want to access the browser version of SharePoint are:
To recover a file they deleted (from any of the other locations, including File Explorer), from the Recycle Bin. This option is available for 93 days after the file was deleted. After that point, unless a retention policy has been applied (in which case the document will be in the Preservation Hold library, accessible to admins), the file is gone forever.
To see who has been working on a file, from the version history.
To see who has viewed a file (when this feature is enabled).
To seek approval for, or see who has approved which version of, a document. This functionality comes with every SharePoint library and list.
To add additional metadata to the content.
To use the full functionality of document sets. Note that these appear as normal folders in a synced document library.
To copy or move documents.
To check out a document.
To search for content and to view content in multiple ways through views.
And more.
Summing up
Access to SharePoint has never been easier, but it is a good idea to let end-users know that they can access their SharePoint content in multiple ways.
Some users may rarely access it via Teams (unless they are interested in collaborating more effectively than attaching documents to emails), and even less so via the full SharePoint browser interface. In summary:
Option
Best for
File Explorer
Day to day use where no additional metadata or labels are needed. Sharing (instead of attaching)
MS Teams
Collaboration.
Outlook
For Groups that also use the Group mailbox.
Mobile device
Anywhere, anytime access
Browser
For the full set of functionality, including the Recycle Bin, versioning history, viewing, usage information, searching and more.
It is important to let end users know the functionality that they can access in the different areas, especially the version history and Recycle Bin in the browser version of SPO. These alone can be ‘life savers’.
Sources for this information are listed where this is known.
1973 – Plato Notes
A history of ERM and EDM systems must include reference to Lotus Notes.
Lotus Notes began its life in 1973 as Plato Notes, developed by the Computer-based Education Research Laboratory (CERL) at the University of Illinois in 1973. Elements of the Plato Notes system would be developed for PC by Ray Ozzie during the late 1970s. This was picked up by Lotus Development Corporation and in 1984 became Lotus Notes.
An early version of Lotus Notes was released (under contract to Lotus) in 1984. The original vision included on-line discussion, email, phone books and document databases. Eventually the product fell into the ‘groupware’ category. The capability of the product continued to grow and some organisations only used Notes.
Lotus acquired all the rights to Lotus Notes in 1987 and version 1.0 was released on 7 December 1989.
1974 – Compulink Management Center/Laserfiche founded
Compulink Management Center was founded in the US in 1974. It created Laserfiche, the first DOS-based document imaging system, in 1987.
1976 – Micro Focus founded
Micro Focus was founded in the UK in 1976. Its first software product was CIS COBOL, a solution for micro computers. It entered the EDRM market in 2017, see below.
1981 – Enterprise Informatics founded
Enterprise Informatics, a privately-held software company, was founded in 1981 by early pioneers of the document management industry. (Source: LinkedIn company profile) It would later be acquired by Spescom, a South African company.
1982 – FileNet founded
FileNet was founded in 1982 by Ted Smith, formerly of Basic 4. FileNet’s original focus of attention was the storage and management of scanned images but it also developed a workflow software. (Source: Wikipedia article on FileNet)
1983 – GMB/DocFind founded
GMB (named after the original founders, Gillett, Frank McKenna, and Bachmann) was formed in Australia in 1983. In 1984, GMB released DocFind 1.0. DocFind was renamed RecFind in 1986.
Tower Software was founded by Brand Hoff in Canberra in 1985 as a software development company. The company provided and supported enterprise content management software, notably its TRIM (Tower Records and Information Management) product line for electronic records management.
The ‘Tower’ in the company name derives from the telecommunications tower on top of Black Mountain (technically a hill, 812 m high) overlooking Canberra. A graphic of the tower was used in the TRIM logo until the company was acquired by HP’s Software Division in 2008 (see also below).
1986 – Autonomy founded (UK)
Autonomy was founded by Michael Lynch, David Tabizel and Richard Gaunt in Cambridge, UK in 1986 ‘as a spin-off from Cambridge Neurodynamics, a firm specializing in computer-based finger print recognition’.
Before 1987 – Saros Corp
Saros Corp was established in Washington by Mike Kennewick (a former Microsoft employee) before 1987. Saros Corp produced Saros Mezzanine, a client-server document management engine. In 1993, released Saros Document Manager.
1989 – Ymijs (later Valid Information Systems) founded – R/KYV (UK)
Ymijs was founded in the UK in 1989. It sold the R/KYV software initially as a basic document imaging processing system. The company name was changed to Valid Information Systems and R/KYV was further developed as a compliance and records management system ‘… that is widely used by major corporations as well as central and local government authorities and related governmental agencies’ (in the UK).
Valid Information Systems was acquired by Hummingbird in 2003. (Source: ‘Legal Technology Insider‘, 23 July 2003.
1989 – Provenance Systems (later TrueArc) founded (Canada)
Bruce Miller, sometimes noted as ‘the inventor of modern electronic recordkeeping software’, founded Provenance Systems in 1989 where he created ForeMost. The company name was changed to TrueArc. Bruce would go on to found Tarian Software as well in 1999.
TrueArc ForeMost RM would be acquired in 2002 by Documentum (which which it had a long-standing technology partnership).
1990 – Documentum founded (US)
According to this Wikipedia article, Documentum was founded in June 1990 by Howard Shao and John Newton who had previously worked at Ingres (a relational database vendor). They sought to solve the problem of unstructured information.
The first Documentum EDMS was released in 1993. According to the Wikipedia article, ‘This product managed access to unstructured information stored within a shared repository, running on a central server. End users connected to the repository through PC, Macintosh, and Unix Motif desktop client applications.’
1992 – Altris Software (UK)
Altris, established in 1992 (Source: Rob Liddell\’s LinkedIn profile. Rob was one of the co-founders of Altris), developed document management systems, including (according to this South African ITWeb post of 26 October 2001), eB, a ‘configuration management’ application.
This article titled ‘The Case for 11g‘ (referring to Oracle’s product, see below) noted that Optika’s original software development focus was Image and Process Management (IPM).
An undated (but likely mid to late 1990s) webpage on the Property and Casuality website titled ‘Optika and Xerox Package FilePower with Document Centre‘ noted that ‘Optika Imaging Systems, Inc. and Xerox announced that the two companies will jointly work to integrate Optika’s FilePower with Document Centre digital systems products from Xerox. The combination of the Document Centre and FilePower will provide a complete solution for capturing, managing and distributing large volumes of documents, increasing users’ productivity and significantly reducing labor and capital costs. Optika’s integrated product suite — FilePower — combines imaging, workflow and COLD technology into a unified software package. The Xerox Document Centre 220ST and 230ST combine network scanning, printing, faxing and copying into one hardware device.’
1993 – Workflow Management Coalition formed
The Workflow Management Coalition (WfMC), ‘a consortium formed to define standards for the interoperability of workflow management systems’, was founded in May 1993. Original members included IBM, Hewlett-Packard, Fujitsu, ICL, Staffware and approximately 300 software and services firms in the business software sector.
The WfMC’s Workflow Reference Model was published first in 1995 and still forms the basis of most BPM and workflow software systems in use today. (Source: Undated Gutenberg article)
1993 – Kainos Meridio (UK)
Meridio was developed in 1993 by Kainos (a Northern Ireland company and joint venture between Fujitsu and The Queens University in Belfast) as an electronic document and records management (EDRM) system based on Microsoft products. It would be acquired by HP Autonomy in 2007.
1993 – Saros (US) Document Manager
Saros Corp released Saros Document Manager in mid 1993. The product was said ‘to act as a front-end to the Bellevue, Washington-based firm’s client-server document management engine, Saros Mezzanine’. (Source: Computer Business Review article ‘Saros Sets Document Manager‘ )
ERM before the mid 1990s
Before the arrival of personal computers in offices in the early 1990s, computer mainframes and databases were the regarded by some observers as the only places where electronic ‘records’ (in the form of data in tables) were stored and managed.
A report by the United States General Account Office in July 1999 (GAO/GGD-99-94) titled ‘Preserving Electronic Records in an Era of Rapidly Changing Technology’) stated that, historically (as far back as 1972), NARA’s Electronic Records Management (ERM) guidance (GRS 20) was geared towards mainframes and databases, not personal computers.
The GAO report noted that until at least the late 1990s, there was a general expectation that all other electronic records not created or captured in ERM systems would be printed and placed on a paper file or another system. The original (electronic) records could then be destroyed.
Some early ERM (database) systems, such as TRIM from Tower Software in Australia, were originally developed in the mid 1980s to manage paper files and boxes. Similar systems were developed to manage library catalogues and the old card catalogues started to disappear.
But, although some of it was printed and filed, the volume of electronic records in email systems and stored across network file shares continued to grow. Several vendors released systems that could be used to manage electronic documents (EDM) more effectively than network drives but there was no agreed standard for managing that content as records.
1994 – The DLM Forum and MoReq
From the early 1990s, the European Council sought to promote greater cooperation between European governments on the management of archives. One of the outcomes of a meeting in 1996 was the creation of the DLM Forum. DLM is the acronym of the French term ‘Données Lisibles par Machine’, or ‘machine-readable data’.
One of the ten action points arising from the June 1994 DLM meeting was the creation of ‘Model Requirements for the Management of Electronic Records’, or MoReq, first published in 2001 (see below).
According to its website, ‘the InterPARES Project was borne out of previous research carried out at the University of British Columbia’s School of Library, Archival and Information Studies. “The Preservation of the Integrity of Electronic Records” (a.k.a. “The UBC Project”) defined the requirements for creating, handling and preserving reliable and authentic electronic records in active recordkeeping systems.’
‘The UBC Project researchers, Dr. Luciana Duranti and Professor Terry Eastwood, worked in close collaboration with the U.S. Department of Defense Records Management Task Force to identify requirements for Records Management Applications (RMA).
The work of the UBC team influenced the development of DOD 5015.2 published in 1997 (see below) and the subsequent development of a range of electronic document and records management (EDRM) systems.
Australia – intervention in business applications model
In 1996, the University of Pittsburgh published the ‘Functional Requirements for Evidence in Recordkeeping Project’, led by David Bearman. This work would influence the development of both MoReq2010 and the ICA standards that became ISO 16175-2010, both of which attempted to define a minimum set of functional requirements for a business application to be able to manage its own records. (Lappin)
1995 – IBM Acquires Notes
Lotus Notes was acquired by IBM in July 1995. By December 1996 it had 20 million users. By the end of 1999, Lotus Notes had extensive capability including ERM and EDM.
Lotus Notes continued to retain a strong presence in the market but its dominance began to be reduced by the arrival of Microsoft’s broader capabilities and other EDM solutions.
1995 – Alpharel (US) acquires Trimco (UK)
According to this Computer Business Review article of 23 November 1995, Alpharel Inc, San Diego was expected to acquire Trimco Group Plc of Ealing, London, a supplier of enterprise-wide document management systems.
1995 – FileNet acquires Saros
FileNet acquired Saros Corporation in 1995 to acquire its electronic document management capability. It was said to have pioneered ‘integrated document management’ (IDM), through a suite that offered document imaging, electronic document management, COLD and workflow. (Source: Wikipedia article on FileNet)
1996 – Australian Standard AS 4390
In February 1996 Australia issued the world’s first national records management standard, AS4390 ‘Records Management – General‘. The standard provided guidance for the implementation of records management strategies, procedures and practices.
Tower Software, the Canberra-based developers of TRIM, contributed to the development of the standard (according to its Wikipedia entry) although the standard did not prescribe requirements for the management of electronic records.
AS 4390 would become internationalised through ISO 15489 in 2002.
1996 – OpenText Corporation (US) – Livelink
OpenText Corporation was founded in 1991 from OpenText Systems. It released Livelink in 1996.
1996 – EDM solutions (UK listing)
The following is a list of EDM systems taken from the Document Management Resource Guide, 1995/96 Edition, kindly provided by Reynold Lemming in 2021. (^ = Original software author entry, all others are system resellers)
QStar: Axxess / Server / Worksgroup / Enterprise ^
1996 – Various EDM solutions
The March 1996 edition of Engineering Data Management included a number of updates on electronic document management solutions in the market at that time. Note that Trimco and Alpharel are listed separately; this may because Alpharel’s acquisition of Trimco had not been completed by that time.
Alpharel (San Diego, CA): Document Management solutions – Enabler, FlexFolder, RIPS, Toolkit API. Wisdom, a product that facilitated internet access to participating electronic document vaults.
Auto-trol Technology (Denver, CO): CENTRA 2000, document management, workflow, PDM, change management and messaging.
Cimage Enterprise Systems (Bracknell, UK): Document Manager for Windows.
Documentum, Inc. (Pleasanton, CA): Documentum Accelera for the World Wide Web and Documentum UnaLink for Lotus Notes.
Interleaf (Waltham, MA): Interleaft 6 SGML, a solution for publishing SGML doocuments. Intellecte/BusinessWeb, a document management solution that allowed organisations to access enterprise document repositories from the internet.
Trimco (Ealing, UK): Document management systems.
Alpharel changed its name to Altris Software (US) in October 1996, according to this Telecompaper article published the same month.
From 1996 – Germany’s DOMEA project
In 1996, the Coordinating and Advising Agency of the Federal Government for Information Technology in the Federal Administration (KBSt) introduced a pilot project named Document Management and Electronic Archiving in computer-assisted business processes (DOMEA).
Under the framework of DOMEA, a project group was set up in 1998 to find solutions for the disposition and archiving of electronic records. The goal was to find a suitable and efficient way for the disposition of electronic records created and maintained in office systems. Its “Concept for the Disposition and Archiving of Electronic Records in Federal Agencies,” containing recommendations for managing electronic records was published in September 1998. (Source: The Free Library article)
Late 1990s – EDMS vs ERMS
Electronic document management systems (EDMS) and electronic records management systems (ERMS) were regarded as separate types of system from the late 1990s until at least 2008.
According to Philip Bantin in August 2002:
An EDMS was said to support day-to-day use of documents for ongoing business. Among other things, this meant that the records stored in the system could continue to be modified and exist in several versions. Records could also be deleted.
An ERMS was designed to provide a secure repository for authentic and reliable business records. Although it contained the same or similar document management functionality as an EDMS, a key difference was that records stored in an ERMS could not be modified or deleted. (The concept of ‘declaring a record’ may be related to this point).
(Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001)
The difference between the two types of system endured for at least a decade. By the end of the 1990s, four main EDRMS options had emerged:
Extending an existing EDM product capability to include ERM.
Extending an existing ERM capability to include EDM.
Creating new ERM products (technically also with some EDM capability).
Integrating separate EDM and ERM products.
1997 – DOD 5015.2
According to the 1999 GAO report quoted above, for several years prior to 1997, NARA worked with the US Department of Defense, considered ‘one of the agencies that is most advanced in its ERM efforts’.
The outcome of this work was the release in November 1997 of the DOD standard titled ‘Design Criteria Standard for Electronic Records Management Software Applications’ usually known by its authority number – DOD Directive 5015.2, Department of Defense Records Management Program, 11 April 1997.
The GAO report stated that ‘ERM information systems that were in place before the approval of this standard must comply with the standard by November 1999’.
It added that US agencies ‘were confronted with many ERM challenges’ from the ever-increasing volume of digital records, including the ability to preserve and access those records over time. The ‘Year 2000 problem’ was drawing attention away from the issue.
Nevertheless, by 2 June 1999, nine companies were certified as compliant with the DOD standard. Some, it noted, were standalone ERM software, while others were an integrated solution.
An interesting small note on page 11 of the GAO report noted that ‘it is important that ERMS software requires users to make no more than two or three extra keystrokes, and that users realize there is a benefit to this additional ‘burden’.
From 1997 – SER eGovernment (Germany)
SER eGovernment was developed for the German/Austrian market following the release of the German eGovernment standard, DOMEA in 1997.
1998 – Documentum goes online
In 1998, Documentum released its Web Application Environment, a set of internet extensions for EDMS, offering web access to documents stored within an EDMS repository. Various additional products were acquired and their functionality added to the Documentum system.
1998 – Optika eMedia released
Optika released eMedia, ‘a software and methodology product designed to manage business transactions within an organization, across extranets, and throughout the supply chain’, in late 1998. (Source ‘Optika Delivers App to Manage Business Transactions‘) Optika eMedia was said to be ‘a workflow enabled replacement for an imaging solution named FilePower’.
1998 – FileNet Panagon suite released
In 1998, FileNet released its Panagon suite of products. This included Panagon Content Services that was previous Saros Mezzanine. (Source: Wikipedia article on FileNet)
1999 – International differences
The 1999 GAO report noted differences between the US, UK, Australia and Canada on their approach to ‘common ERM challenges’.
Australia was said to have ‘strong central authority (including for compliance audits) and decentralised custody’ (except when the records are transferred to permanent retention).
Canada had ‘vision statements rather than specific policies’ and also had decentralised custody, but agencies could transfer records at any time to the archives.
The UK had broad guidelines put into practice by individual agencies.
1999 – the UK PRO standard released
The UK Public Records Office (PRO, later The National Archives, TNA) released a standard in 1999 designed ‘to provide a tool for benchmarking the ability of government departments to support electronic records management’. This standard would be replaced by TNA 2003. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)
End of the 1900s – XML
By the end of the 20th century it was becoming clear (to some) that XML would likely play a strong role in the standardisation of electronic record formats and their management over time.
XML-based record structures meant that electronic records could contain their own ‘metadata payloads’ rather than being independent objects defined in a separate system (like a library catalogue describes books on shelves).
The establishment of XML-based formats would (after about 20 years) begin to change the way in which records would be managed, although paper records and the paradigm of managing records in pre-defined containers would continue to persist, largely because of the standards developed to manage electronic records – in particular DOD 5015.2.
1999 – EDM/early ERM products
The following is a collated list of EDM (and related) products collated in November 1999:
Autonomy Portal in a Box
CompuTechnics (1990 to 1999)/Objective (from 1999)
Hummingbird (from 1999 with acquisition of PC DOCS)
Insight Technologies Knowledge Server (IKS) / Document Management System (DMS)
Intraspect Knowledge Server (IKS) (KM)
Lotus Notes
Onyx Enterprise Portal, with integration to various EDM applications
Open Text Livelink
Pitney Bowes Digital Document Delivery (D3)
Oracle
Osyssey (ISYS)
PC DOCS (acquired by Hummingbird in early 1999)
ReadSoft (OCR processing)
RecordManager
Tower Software / TRIM Captura
Tower Technology
Verity (KM)
1999 – Tarian Software founded
Tarian Software was founded in Canada in 1999 by Bruce Miller, the founder of Provenance Systems (later TrueArc) and creator of ForeMost. Tarian developed the Tarian eRecordsEngine, an embedded electronic recordkeeping technology for business application software. Tarian was the first e-Records technology in the world to be certified against the revised 5015.2 June 2002 standard. Tarian was acquired by IBM in 2002.
1999 – The Victorian Electronic Records Strategy (VERS)
The (Australian) Victorian government’s Public Records Office (PROV) published a standard for the management of electronic records in 1999, Standard 99/007 ‘Standard for the Management of Electronic Records’. The standard, usually known as VERS, defined the (XML-based) format required for the transfer of permanent records to the PROV.
The Standard noted that:
Records must be self-documenting. It is possible to interpret and understand the content of the record without needing to refer to documentation about the system in which it was produced
Records must be self-contained. All the information about the record is contained within the record itself
The record structure must be extensible. It must be possible to extend the structure of the record to add new metadata or new record types without affecting the interoperability of the basic structure.
Several EDRMS vendors developed the capability to create VERS encapsulated objects (VEOs) as required by the standard.
2000 – Spescom (South Africa) acquires Altris (UK)
The South African company Spescom acquired the UK firm Altris Software in 2000, as noted in this (South Africa) ITWeb article of 3 May 2000. Altris was described in the article as ‘a global leader in integrated electronic document management software, with well established channels to international markets’. As a result of this acquisition, Altris UK was renamed Spescom Ltd (UK).
The same journal announced in 2001 that Spescom KMS was ‘the UK operation of Spescom Limited’s US based subsidiary, Altris Software Inc, which specialises in the provision of asset information management software to markets including transportation, utilities and telcos’.
From 2000 – Microsoft adopts XML for Office documents
In 2000, Microsoft released an initial version of an XML-based format for Microsoft Excel, which was incorporated in Office XP.
In 2002, a new file format for Microsoft Word followed. The Excel and Word formats, known as the Microsoft Office XML formats (with an ‘x’ on the end of the document extension), were later incorporated into the 2003 release of Microsoft Office.
Microsoft’s XML formats, known as Open Office XML, later became ECMA 376 in 2006 and later ISO 29500 in 2008 ‘amid some controversy’ over the need for another XML format (see below).
Before 2001 – Intranet Solutions (later Stellent)
Intranet Solutions had developed software called’IntraDoc!’. The product was briefly renamed Xpedio! before the company and product were renamed Stellent in 2001. (Source: ‘Wikipedia article on Oracle Acquisitions‘)
2001 – EDM systems with RM functionality
The following is a list of ‘EDM systems with records management’ functionality available by early 2001:
TRIM (Tower Software, Australia) – integrated ERM and EDM.
Hummingbird RM Family 4.0 (Hummingbird, Canada) – ERMS (later acquired by Open Text)
FileSurf 7.0 (MDY Advanced Technologies) – ERMS
R2M (Relativity Records Manager v2 (Relativity Inc) – EDMS with added records management functionality.
eManage 2000 (ByteQuest Technologies) – knowledge management (KM) integrated with records management.
FileNET IDM Content Services 5.1.1/ForeMost Enterprise 2.0 – EDMS with integrated records management.
IBM e-Records Solution (IeRS) v1.0 – IBM’s content management (CM) solution integrated with Tarian eRecords v1.0.
Work Manager/ForeMost (Eastman Software) – combined the DM, imaging and workflow of Work manager with the RM capabilities of ForeMost.
(Source: Presentation by Philip Bantin, University Archivist at the University of Indiana, dated 18 April 2001).
Many of the products above were sold internationally on the basis of their compliance with DOD 5015.2 even though that was a US-specific standard.
2001 – Birth of SharePoint
Microsoft released SharePoint in 2001. It was born out of a combination of Office Server and Tahoe.
Office Server provided the simplest of network administration actions
Tahoe expanded on the email, calendaring, contacts, and to-do list technology contained within Exchange.
However, it was not until the release of Office 2003 that SharePoint (rebranded as Windows SharePoint Services) provided collaboration and other functionality.
In addition to the EDMS/ERMS differences, organisations were also seeking solutions for knowledge management (KM) and content management (CM).
CM solutions were usually portal-based options that mostly became some form of intranet.
Some of the options in the early 2000s included:
Hummingbird’s PowerDOCS for DM and CyberDOCS as the web client for the DM solution, along with Hummingbird’s (formerly Fulcrum) Knowledge Server for KM and PD Accord for web-based collaboration, with the Hummingbird Enterprise Information Portal (EIP) as the portal solution. Plumtree Corporate Portal could also be used as an Enterprise Portal.
iManage’s DeskSite for DM and WorkTeam for collaboration. For KM, WorkKnowledge Server and Concept Search (based on the Autonomy Server). The portal to link all of these was called WorkPortal.
Open Text’s Livelink for DM, KM and collaboration.
Elite’s Encompass, built on Microsoft’s new SharePoint Portal Server (SPS).
Autonomy Server for KM, with Plumtree Corporate Portal as the portal.
Documentum’s DM and CM product coupled with Plumtree Corporate Portal.
Digital Asset Management (DAM) systems, used to manage other types of digital content such as photographs, also appeared around this time.
Information Technology Decisions published a paper on DOD 5015.2 certified products in November 2001 (original source/location has been lost). It noted that there were two types of products:
Products that started life as electronic document management (EDM) systems. Examples included Documentum, Livelink, and DOCS Open.
Products that started life as electronic recordkeeping (ERK/ERM) systems. Examples given included Tower Software’s TRIM, Foremost, iRIMS, Cuadra Star.
The presentation noted that DOD 5015.2 certification was based on alternative options:
Standalone. For example, True Arc Foremost, TRIM, iRIMS, Cuadra Associates Star, Relativity Records Manager, Hummingbird RM 4.0, Tarian eRecords, MDY/FileSurf, Cimage and Access Systems, Highland Technologies Highview-RM, Open Text, Livelink
Partnership. For example, Saperion with e-Manage 2000, Impact Systems eRecords Manager, FileNet with Foremost.
The report included three interesting points:
Both EDMS and ERKS required an enterprise view of information.
An EDMS is driven by business process requirements.
An ERKS (ERMS) is driven by enterprise requirements for the long-term preservation of information.
2001 – The first MoReq
The first version of MoReq was published in 2001. Volume 1 was 500 pages long.
MoReq emphasised the central importance of an electronic records management system, or ERMS. Its stated purpose was:
To provide guidance to organisations wishing to acquire ERMS.
As a tool to audit or check an existing ERMS.
As a reference document for use in training or teaching.
To guide product development by ERMS suppliers and developers.
To help define the nature of outsourced records management solutions.
Few, if any, products were certified against this version of MoReq.
2002 – Optika Acorde
Optika eMedia was rebranded to Optika Acorde in 2002, according to this website ‘The Case for 11g‘.
A June 2002 Gartner report titled ‘Optika Acorde Document Imaging, Workflow and Collaboration Suite‘ noted that Optika Acorde was an ‘integrated software family for managing the content associated with business transactions’ leveraging ‘Optika’s core strengths in document imaging, workflow and enterprise report management.’
2002 – FileNet BrightSpire, later P8 ECM
FileNet released BrightSpire in 2002. This product ‘leveraged the experience gained from integrated document management, web content management and workflow into what became ECM. (Source: Wikipedia article on FileNet)
By 2002 – Enterprise Content Management (ECM)
The term ‘Enterprise Content Management’ (ECM) began to appear more frequently by 2002. The Wikipedia post on ECM noted that ECM technologies descended from ‘electronic Document Management Systems (DMS) of the late 1980s and early 1990s’.
ECM promised:
The integration of records management (RM) with business practices.
The capability for integration between RM products, EDM, various other digital products (such as OCR/character recognition technologies), and web publishing products.
Incorporation of Knowledge Management (KM) concepts.
The key word here was ‘integration’ with EDM and other systems, rather than standalone systems. Web-based access became increasingly essential. IBM’s acquisition of Tarian, Documentum’s acquisition of TrueArc’s Foremost were examples of these integrations. (see below)
According to the Wikipedia article on ECM: ‘Before 2003, the ECM market was dominated by medium-sized independent vendors which fell into two categories: those who originated as document management companies (Laserfiche, Saros, Documentum, docStar, and OpenText) and began adding the management of other business content, and those who started as web content management providers (Interwoven, Vignette, and Stellent) tried to branch out into managing business documents and rich media’.
The emergence of ECM quite possibly created the first challenge to centralised ERM through the integration of multiple elements, some of which created, captured or stored records in ever increasing formats.
2002 – OpenDocument XML format
According to the Wikipedia article on the OpenDocument standard, the OpenDocument standard was developed by a Technical Committee (TC) under the Organization for the Advancement of Structured Information Standards (OASIS) industry consortium. Sun and IBM apparently had a large voting influence but the standardization process involved the developers of many office suites or related document systems. The first ODF-TC meeting was held in December 2002.
2002 – An updated GAO report into electronic records
The US General Accounting Office (GAO) released a new report in June 2002 titled ‘Information Management: Challenges in Managing and Preserving Electronic Records’ (GAO-02-586). This report, which was more detailed than the earlier 1999 one, noted among other things that:
The DOD had by March 2002 certified 31 applications against standard 5015.2.
Progress had been made on the development of the Open Archival Information System (OAIS) model which, while initially developed by NASA for archiving the large volume of data produced by space missions, could be applied to ‘any archive, digital library or repository’. XML-based solutions were considered the most likely to be accepted.
From that date, IBM released the IBM Records Manager Version 2.0 (IRM), previously known as the Tarian eRecords Engine (TeRe). Tarian’s e-Records management technology was integrated into IBM’s software offerings, including IBM Content Manager, DB2 database and Lotus software. (Source: IBM press release)
2002 – Documentum 5 and TrueArc Foremost acquisition
Documentum released Documentum 4i, its first Web-native platform, in 2000. In 2002, it launched Documentum 5 as ‘a unified enterprise content management (ECM) platform for storing a virtually unlimited range of content types within a shared repository’.
Documentum acquired TruArc’s Foremost product in October 2002. The Documentum Wikipage above noted that this acquisition ‘added records management capabilities and augmented Documentum’s offerings for compliance solutions.’ The press release cited in this paragraph noted that ‘Documentum and TrueArc are existing technology partners and have worked together to provide an integration for TrueArc’s enterprise-scalable records management solution with the Documentum ECM platform.’
2003 – TNA 2003
The National Archives (TNA) released an updated version of its PRO standard in 2003, known as TNA 2003. This standard would be superseded by MoReq2. (Source: @ZenInformation on Twitter, 12 February 2021).
In October 2003, Open Text acquired the (German) DOMEA-certified SER eGovernment Deutschland GmbH, based in Berlin, Germany as well as SER Solutions Software GmbH, based in Salzburg, Austria. (Source: Open Text Acquires SER eGovernment)
From 2003 – CNIPA (Italy)
The Italian Centro Nazionale per l’Informatica nella Pubblica Amministrazione (CNIPA) published a protocol for the management of electronic records, Protocollo Informatico in 2003.
CNIPA was renamed DigitPA in 2009 and Agenzia per I’Italia digitale (AGID) in 2012. AGID is responsible for defining standards for the management of electronic records in Italian government agencies. (Source: Protocollo Informatico)
Mid 2003 – The challenges of Enterprise Records Management
In Industry Trend Reports of May 2003, Bruce Silver (of Bruce Silver Associates) made the case for Enterprise Records Management in the wake of various ‘scandals’ involving the management of records at the time, including Enron/Anderson.
Silver argued that EDM, email archive, and back-up solutions did not meet the ‘new statutory and regulatory records management requirements’ – DOD 5015.2, SEC Rules 17a-3 and 17a-4, NASD Rules 2210, 3010, and 3110, NYSE Rules 342 and 440, ISO 15489 and MoReq.
Silver also noted that an effective (‘total’) ERM solution would ‘be implemented as an extension of the company’s ECM infrastructure’, providing for a single interface for all records stored in multiple locations ‘including third-party document management repositories in addition to the email system and network file system’.
2003 – Key integrated EDM/RM vendors
The following is a list of ‘key vendors in the (Integrated Document Management) IDM Market Space’ in October 2003. The list is believed to have come from a Butler Group report:
Stellent acquired Optika in early 2004. At this stage, Stellent’s primary product was Universal Content Management (UCM). (Source: Wikipedia article on Oracle Acquisitions)
The report of the sale in The Register stated that Stellent’s CEO said that ‘customers are looking to consolidate their content management needs, including imaging, business process management, web content management and record management with one vendor.’ The new product line was named Stellent Imaging and Business Procss Management (IBPM). The article also noted that Oracle would probably acquire Stellent following this acquisition (see 2006, below).
2004 – ReMANO (Netherlands)
In 2004, the Netherlands government established a catalogue of software specifications for ERM systems (ReMANO) used by Dutch government bodies. (Source: ‘ERM System Requirements’, published in INFuture, 4-6 November 2009, by, Marko Lukicic, Ericsson)
ReMANO was replaced by NEN2082 – Eisen voor Functionaliteit van Informatie- en Archiefmanagement in programmatuur” in 2008. (NEN 2082:2008 nl)
2005 – C6 (France) builds D2 on top of EMC Documentum
The French ECM company C6 built a solution named D2, ‘a fully configurable web application for creating, managing, storing and delivering any type of information’, on top of EMC’s Documentum. (Source: C6 website ‘Company’ tab).
2005 – ELAK (Austria)
A project to introduce electronic filing (ELAK) commenced in Austria late 2001 and was completed in 2005. The solution was based on standard software products (e.g. Fabasoft e-Gov-Suite, MS-Office). (Source (ELAK – The e-filing system of the Austrian Federal Ministries)
2006 – The National Archives of Australia ERMS standard
The National Archives of Australia (NAA) released its ‘Functional Specifications for Electronic Records Management Systems Software in February 2006. (ISBN 1 920807 34 9). The introduction noted that:
(The document) provided Australian Government agencies with a set of generic requirements for ensuring adequate recordkeeping functionality within electronic records management systems (ERMS) software.
Agencies were encouraged to make use of the ERMS specifications when designing or purchasing new, or upgrading existing, ERMS software. They could also be used when auditing, assessing or reviewing an agency’s existing ERMS software.
The requirements were not intended to be a complete specification, but rather provide a template of key functional requirements that agencies may incorporate into their tender documentation when preparing to select and purchase new ERMS software. Agencies were expected to assess and amend the functional requirements, and select requirements that best suit their own business and technical requirements and constraints.
Very few products met the specific requirements of the ERMS specifications which led to some suggestion at the time that it limited choice.
2006 – Rival XML Office document standards
The OpenDocument (ODF) standard was published as ISO/IEC 26300 in 2006.
Microsoft submitted initial material to the Ecma International Technical Committee TC45, where it was standardized to become ECMA-376, approved in December 2006. It was released as ISO 29500 in 2008.
According to the Wikipedia article on Open Office XML (OOXML), ‘The ISO standardization of Office Open XML was controversial and embittered’, as it seemed unnecessary to have two rival XML standards.
2006 – The world of collaboration
The Butler Group published a paper titled ‘Document Collaboration – Linking People, Process and Content’ in December 2006. The report noted that EDM systems had helped improve internal efficiency but there was now a need to ‘extend these systems to partners and stakeholders’ and deliver ‘sophisticated collaborative experiences’.
The paper listed the following EDM products:
Adobe Acrobat family
EMC eRoom
IBM Notes/Domino, Workplace collaboration services, QuickPlace
Microsoft Office 2007
Open Text Livelink ECM – eDOCS (incorporating the former Hummingbird product suite acquired by Open Text in 2006)
Oracle Collaboration Suite, Content DB and Records DB
Stellent Collaboration Management
Vignette
2006 – Spescom Software Inc
A US SEC submission in January 2006 noted that Spescom Software Inc, a San Diego-based provider of computer integrated systems was the successor to Alpharel Inc and Altris Software Inc.
2006 – Oracle acquires Stellent
A 2006 Oracle press release titled ‘Oracle Buys Stellent‘ stated that Stellent was a global provider of enterprise content management (ECM) software solutions that included Document and Records Management, Web Content Management, Digital Asset Management, Imaging and Business Process Management, and Risk and Compliance. It also noted that the acquisition would ‘complement and extend Oracle’s existing content management solution portfolio’. Despite the acquisition, the ‘Stellent’ name persisted.
2006 – Google enters the online EDM productivity and collaboration market
In 2006, Google launched Google Apps for Your Domain, a collection of cloud computing, productivity and collaboration tools, software and products. Various apps and elements were acquired and/or added over the years but a key one from an EDM point of view was Google Docs (Wikipedia article). However, Google Docs had no RM capability.
A ZDNet article in June 2007 noted that Google Apps offered a tool for switching from Exchange Server and Lotus Notes, making Google a real alternative to Microsoft and IBM. Google Apps would later be rebranded G-Suite in 2016.
2007 – Spescom exits the EDM market / Enterprise Informatics
In 2007, Spescom exiting the enterprise software sector with the sale of its US operation Enterprise Informatics. (Source – Wikipedia article on Spescom, original reference no longer accessible).
Enterprise Informatics, originally founded in 1981, continued in existence as a subsidiary of Bentley Systems, Incorporated. It continued to market a suite of integrated document, configuration, and records management software products, mostly under the name eB.
2007 – Zoho enters the online EDM collaboration market
The India-based Zoho Corporation, known as AdventNet Inc from 1996 to 2009, released Zoho Docs in 2007.
2007 – HP Autonomy acquires Meridio
Meridio was acquired by HP Autonomy (a company that had had a long business partnership with Kainos) in 2007. The parent company Kainos continued to work with SharePoint-based solutions.
2007 – EDRMS vendors
Forrester released a report into electronic records management vendors in early 2007. The products that it evaluated were as follows:
CA MDY FileSurf v7.5 ^
EMC Records Manager 5.3
IBM FileNet P8 Records Manager v3.7 ^
IBM Records Manager v4.1.3 #
Interwoven Records Manager v 5.1 *
Meridio Document and Records Manager v4.4 *
Open Text Livelink ECM – Records Management v3.8 ^
Open Text Livelink – eDOCS RM (formerly Hummingbird) v6.0.1
Oracle (formerly Stellent) Universal Records Management v7.1 #
Oracle Records DB v1.0 ~
Tower Software TRIM Context v6.0 *
Vignette Records & Documents v 7.0.5
(Forrester assessment: ^ = leaders, # = close behind leaders, * = have hurdles to remain competitive’, ~ = basic functionality only)
2008 – NEN 2082
The Dutch government replaced ReMANO with NEN 2082 ‘Eisen voor functionaliteit van informatie- en archiefmanagement in programmatuur’ (‘Requirements for functionality of information and archive management in software’) in 2008 (NEN 2082:2008 nl). NEN 2082 was derived from MoReq, DOD 5015.2 and Australian standards. (See Eric Burger’s blog post ‘Nee, NEN 2082 is geen wettelijke verplichting‘ about its legal standing)
2008 – MoReq2
MoReq2 was published in 2008. It included new sections to support the testing of ERMS software for compliance with the standard. MoReq2 included the following vendors on its panel (Acknowledgements section):
Docuware, Germany
EDRM Solutions, USA
EMC, Canada
ErgoGroup AS, Norway
Fabasoft, UK
FileNet, UK
Fujitsu, UK
Getronics, UK
Haessler Information, Germany
IBM, UK
ICZ, Czech Republic
Lockheed Martin, USA
Meridio, UK
Objective Corporation, UK
Open Text Corporation, UK
SAPERION, Germany
SER Solutions Deutschland, Germany
Tower Software, UK
Both MoReq and MoReq2 were based on the premise of a central ERMS being acquired and implemented by organisations to manage unstructured records, the types of records that are stored across network drives and in email systems. MoReq2 specifically clearly excluded the management of ‘structured data … stored under the management of a data processing application’. (Source: MoReq2, section 1.2 ‘Emphasis and Limitations of this Specification’, page 12.)
The first software product certified against MoReq2 was Fabasoft Folio. It was the only certified product until June 2014.
2008 – EDMS and ERMS
In 2008, the International Standards Organisation, under ISO/TC171/SC2 ‘Document management applications’ proposed a framework for the integration of EDM and ERM systems. The definitions contained in that framework document noted that:
An EDMS was used to manage, control, locate and retrieve information in an electronic system.
An ERMS was used to manage electronic and non-electronic records according to accepted principles and practices of records management.
An integrated EDRMS would combine both capabilities.
Section 6 of the report described general (but fairly detailed) functional requirements for an integrated EDMS/ERMS, outlined in the following diagram:
2009 – Autonomy acquires Interwoven
In 2009, HP Autonomy acquired Interwoven, a niche provider of enterprise content management software mostly to the legal industry. It primarily competed with Documentum in this space. Interwoven became Autonomy Interwoven and Autonomy iManage.
2010 – MoReq2010
MoReq was completely revised and published as MoReq2010 in 2010. There were key differences with its predecessor versions.
It de-emphasised, but did not remove, the idea of an ERMS being the central or sole recordkeeping system or repository for organisations.
It emphasised the need for line of business systems to incorporate a minimum, defined level of recordkeeping functionality.
It brought a degree of practicality about the management of records in other systems.
It provided for interoperability between all MoReq compliant systems, based on a common XML language.
MoReq2010 established ‘… a definition of a common set of core services that are shared by many different types of records systems’. It provided a set of modules that could be incorporated into any software solution, including line of business applications, so they can be ‘MoReq compliant records systems’ (MCRS).
2010 – Google’s DM capability enhanced
In March 2010, Google acquired DocVerse, an online document collaboration company. DocVerse allowed multiple user online collaboration on Microsoft Word documents, as well as other Microsoft Office formats, such as Excel and PowerPoint. (Source – Wikipedia article on Google Docs)
‘Microsoft SharePoint 2010 is a software product with a range of uses, including website development, content management and collaboration. SharePoint allows users to collaborate on the creation, review and approval of various types of content, including documents, lists, discussions, wiki pages, web pages and blog posts. SharePoint is not a recordkeeping system (i.e. a system purposely designed to capture, maintain and provide access to records over time). When implemented ‘out of the box’, SharePoint has limited capacities for capturing and keeping records in a way that supports their ability to function as authentic evidence of business.’
Adam Harmetz, the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft said in a recent online interview about Records in SharePoint 2010, “We constantly get questions from around the world about how to deal with local government and industry standards for information management. Let me throw just a few at you… MOREQ2, VERS, ISO 15489, DOMEA, TNA, ERKS, the list goes on. Some of these standards are loosely based on one another and some have contradictory elements. Rather than focus our engineering efforts on addressing each of these standards in turn, we made the choice to deliver the usability and innovation required to make records management deployments successful and allow our partner ecosystem to build out the SharePoint platform to deal with specific requirements for those customers that are mandated to adhere to a specific standard.”
Despite these comments, SharePoint 2010 was assessed by at least one consultant (Wise Technology Solutions) to meet 88% of the requirements of the then ICA Standard that became ISO 16175 Part 2.
On 16 December 2011, State Records NSW published a blog post titled ‘Initial advice on implementing recordkeeping in SharePoint 2010‘. The post noted that the Wise report had concluded that ‘SharePoint is 88% compliant with the ICA requirements’. It added that the areas where full compliance could not be achieved relate to:
ease of email capture
native security classification and access control
physical and hybrid records management
The report states that third party providers are able to offer products that plug SharePoint’s gaps in these areas.
The blog posted also stated that ‘… the report very clearly makes the point that ‘we note that the achievement of these results is reliant on appropriate design and governance of implementation, configuration and set up to ensure consistency with desired records management outcomes’.’
Early 2010 – Microsoft launches Office 365
Microsoft launched Office 365 on 28 June 2011. Office 365 was designed to be a successor to Microsoft’s Business Productivity Online Suite (BPOS). (Source: Wikipedia article on Office 365). It would not be until the mid 2010s that Office 365 would become an effective counter-solution to the G Suite.
2011 – HP acquires Autonomy
In 2011, Hewlett-Packard acquired Autonomy, a deal that resulted in some interesting subsequent legal issues reading the value of the company.
As noted in the scope section of the standard, ‘(The) specification provides models for software services to support management activities for electronic records’. Further, ‘… models are provided that describe the platform independent model (PIM) that defines the business domain of Records Management and the RM services to be provided’. Three technology-specific implementations are specified:
PSM-1 – Web Services definition for Records Management Services in Web Service Description Language (WSDL). This is actually supplied as ten WSDL files; one for each Records Management Service.
PSM-2 – A Records Management Service XSD. The XSD is for use in creating XML files for import/export of Managed Records from compliant environments and to use as a basis for forming XQuery/XPath statements for the query service.
PSM-3 – An Attribute Profile XSD. The XSD is for capturing and communicating attribute profiles to permit flexible attribution of certain types of Records Management Objects.
2011 – The death of ERM systems?
In a May 2011 blog post on MoReq2010, James Lappin suggested that traditional systems used to manage electronic documents and records, while not being entirely dead in the water, had ‘lost momentum’.
James proposed two specific reasons for this situation:
The global financial crisis (GFC) from 2008 that limited the ability of organisations to acquire and implement hugely expensive ERMS solutions.
The rise of Microsoft SharePoint and particularly SharePoint 2010. In some ways, Sharepoint 2010 had the potential to take – and may have already taken – the ERMS wind from the records managers sails.
He also noted that a series of interrelated user-environment issues may have also played a part in the loss of momentum.
Usability and take up rates of the ERMS. These solutions are sometimes seen as ‘yet another system’ to manage the same records, using a classification structure that doesn’t make sense to most end users and is different from the way end users see and categorise their world.
The ongoing availability of and access to alternative places to store information, including network drives and email folders, and cloud-based storage and email solutions.
The rise and general availability of social networking tools and mobile applications used to create and share new forms of information content, and collaborate and communicate, including wikis, blogs, Twitter, Facebook, and similar solutions, often in an almost parallel ‘personal’ world to the official record.
The inability of ERMS solutions to manage structured data or to maintain and reproduce easily the diverse range of content created and stored in products like SharePoint. Indeed, one reasonably well known product has been described as an archive for SharePoint, even though the latter can quite easily manage its own archives.
The rise of search as a tool to find relevant information in context, and the related change from unstructured to structured in XML-based documents generated by products such as Microsoft Office 2007 and 2010.
From 2013 – GEVER (Switzerland)
From 2013, the Swiss Federal Chancellery was responsible for managing all activities relating to electronic records and process management (Elektronische Geschäftsverwaltung), or GEVER. GEVER consisted of a collection of five standards for the management of electronic records. (Source with current update: Gever Bund)
2015 – Hewlett Packard separates
In October 2015, the software products previously under the Autonomy banner were divided between HP Inc and Hewlett Packard Enterprise (HPE). HP Inc was assigned Autonomy’s content management software components including TeamSite, Qfiniti, Qfiniti Managed Services, MediaBin, Optimost, and Explore.
2015 – EDRMS vendors
Despite the alleged death of ERMS products in around 2010, many continued to thrive and grow. Some were acquired by others.
The following is a list of EDRMS vendors in December 2015 taken from a Gartner report diagram titled ‘Product or Service Scores for Trusted System of Record’ (with the scores included). Many of these products also appeared in the October 2016 ‘Magic Quadrant’ for Enterprise Content Management Systems as indicated)
Alfresco (2.47) (also ECM)
Open Text (4.07) (also ECM)
EMC Documentum (3.94) (as Dell EMC)(Acquired by Open Text)
IBM (3.91) (also ECM)
HPE (3.85)
Oracle (3.45) (also ECM)
Laserfiche (2.45) (also ECM)
Microsoft (SharePoint) (2.38) (also ECM)
Hyland OnBase (2.37) (also ECM)
Lexmark (2.32) (also ECM)
Newgen (2.18) (also ECM)
Objective (2/10) (also ECM)
By 2015 – Oracle departing the scene?
In a July 2015 article titled ‘Looking for an Oracle IPM replacement‘ in the blog softwaredevelopmentforECM, it was noted that Oracle was ‘clearly, and publically, going in a different direction and moving away from traditional enterprise imaging and transactional content management’.
2016 – OpenText, Micro Focus
In May 2016, OpenText acquired HP TeamSite, HP MediaBin, HP Qfiniti, HP Explore, HP Aurasma, and HP Optimost from HP Inc.
The following is a list of products identified by the Victorian Public Records Office (PROV) in 2020. These products were all certified against the VERS standard, that required organisations to be able to create XML-based VERS Encapsulated Objects (VEOs) for long-term preservation.
Alfresco
Altus ECM
AvePoint RevIM, Records, Cloud Records
Bluepoint Content Manager
Canon Therefore 2012
Docbureau
ELOprofessional / ELOenterprise
HP Records Manager
HP TRIM
IBM Enterprise Records
IBM FileNet P8 Records Manager
Info-Organiser
Laserfiche
MAGIQ Documents
MicroFocus Content Manager
Objective
Objective ECM
OpenText eDOCS RM
OpenText Records Management
Oracle WebCentre Content
RecFind 6
RecordPoint
Technology One ECM
Technosoft OfficeTech
2021 – EDRMS vendors
The following is a list of dedicated vendors that offered EDRMS solutions (and more in most cases) by early 2021. Many of these vendors have a long history not necessarily reflected in the above text. Most of these vendors provide Enterprise Content Management (ECM) services, including EDM and ERM capabilities.
Alfresco ECM (alfresco.com)
Hyland OnBase (hyland.com)
IBM ECM (ibm.com)
Knowledgeone RecFind EDRM (knowledgeonecorp.com)
Laserfiche RME (laserfiche.com)
Lexmark RIM (lexmark.com)
Micro Focus Content Manager (microfocus.com)
Microsoft 365 (microsoft.com)
Newgen RMS (newgensoft.com)
Objective ECM (objective.com)
Open Text ECM (opentext.com)
Oracle ECM (docs.oracle.com)
TechnologyOne ECM (technologyonecorp.com)
2021 – Dedicated EDMS vendors
EDM vendors never went away, but many – like Google Drive, DropBox and Box – were built in and for the cloud. This Capterra website has a fairly detailed listing of current EDMS vendors.
2021 – NEN 2082 withdrawn
In January 2021, the Dutch government withdraw NEN 2082. (NEN 2082:2008 nl)
The future of standards-based ERM/EDRM/ECM systems
Although the definition of a record has remained largely intact for the past two decades – ‘evidence of business activity’ (ISO 15489) – the form of records has evolved and continues to do so.
The ever-expanded world of digital content has made it increasingly difficult to accurately and consistently identify, capture and manage records in all forms, a challenge to the notion that all records can be stored in a single system.
The ‘in place’ approach to managing electronic records – wherever they are stored – has strong appeal. But where will we be in another 20 years? Some thoughts:
Electronic databases, whether on-premise or cloud-based (including subscription based), will be the primary method of capturing and storing a wide range of digital content rather than network file shares.
Metadata will be automatically captured or auto-generated for all digital content based on the content itself.
Artificial Intelligence (AI) will continue to grow in maturity, allowing records to be identified from all other digital content, classified, aggregated, and managed through to disposal/disposition or transfer to archives.
Email will, slowly, disappear as the current workforce transitions to chat- and video-based communication methods.
During 2020, many organisations rolled out Microsoft Teams to support the need for employees to work from home (WFH) without paying much attention to the way Teams were named.
A reminder that when a new Team is created, it creates a Microsoft 365 (M365) Group. Every M365 Group has a SharePoint site (visible from the ‘Files’ tab in the Team channels) and an Exchange mailbox (used for calendaring and to store the ‘compliance copy’ of chat messages).
The name given to the Team becomes the display name for the M365 Group and the SharePoint site. For example, ‘Testing multi word Team name’.
The same name, less any spaces between words, becomes the URL name and the email address. For example, ‘/sites/TestingmultiwordTeamname’ and ‘TestingmultiwordTeamname@tenantname.onmicrosoft.com’.
What happens if you want to change the name of the Team, M365 Group or SharePoint site? And what are the potential implications of changing names?
Changing the name of the Team or Group
To change the name of the Team, click on the three dot menu to the right of the name, then ‘Edit Team’ and change the name in the dialog box that appears.
Click ‘Edit team’ to change the name
The new Team name will appear immediately. The name of the M365 Group will change soon after.
The name of the M365 Group (and its email address) can also be changed by admins via the Groups section of the Microsoft 365 admin portal. The name of the Team will change soon after.
Click ‘Edit name and description’ to change the name
The display name on the SharePoint site may take a little longer to change.
If you need to change the SharePoint URL name, go to the SharePoint Admin portal, click on the site name and, in the General tab area, click on ‘Edit’ under the URL name.
Changing the URL name
As long as you can access this section and the new URL name is available, it can be changed:
Keep in mind that, if you change the site URL name, the Team (initially at least) may throw an error:
But if you click ‘Open in SharePoint’, it will re-connect the site to the Team/Group and become visible again.
Implications of changing names
Generally there are no implications in changing the display name of a Team or a SharePoint site as described above.
However, ideally, there should be some correlation between the name of the Team/Group, the display name of the associated SharePoint site, and the URL name. It is not uncommon to see Teams or SharePoint site display names that bear little resemblance to the site URL name.
The main implication of changing a site URL name is that it may break any links, either shared or embedded in documents. For example, the example below is the URL of a link with the URL name highlighted in bold. If the URL is changed, the link will no longer work:
SharePoint is a core foundational element in Microsoft 365. It is primarily used for the storage of digital objects (including pages) in document libraries and rows and columns of data in lists. It is ubiquitous and almost impossible to remove from a Microsoft 365 licence because it ‘powers’ so many different things.
While the idea that anyone can easily create a SharePoint site seems a good idea in some ways, from a recordkeeping of view this starts to look like network file shares all over again.
Microsoft’s response to the default ‘free for all’ ability to create SharePoint sites is to use the so-called ‘records management’ functionality (via the more expensive E5 licence) to auto-classify content and auto-apply retention labels. The problem is that those (more expensive options) provide limited functionality, including inadequate metadata details to make decisions on disposal, and similarly inadequate metadata (for records subject to disposition review labels only) as ‘proof of disposition’.
So, records managers are more often than not left with a network file share-like sprawl of uncontrolled content.
Unfortunately, the ability to create a new SharePoint site is fairly easy, almost as easy as creating a folder on a … network file share.
The following is a list of the main ways a person can create a SharePoint site. Have I missed any?
This option also allows the administrator to provision new SharePoint sites.
2. Via the SharePoint Admin portal (+ Create)
This option allows the creation of three main types of sites: modern team sites (Team site), communication sites, and non-Microsoft 365 Group-linked sites (Other options).
3. By creating a Microsoft 365 Group
Microsoft 365 Groups are created in the Microsoft 365 Admin portal, in the Groups section, Add a group > Microsoft 365. This is also where Security Groups and Distribution Lists (both collectively known as ‘AD Groups’) are created.
Every new Microsoft 365 Group creates both a SharePoint site and an Exchange mailbox that is visible in the Outlook application (under ‘Groups’) of everyone who is an Owner or a Member of the Group.
The new Group creation process allows the Group email address to be created (it really should be the same as the Group name), the Group to be made public or private, and a new Team to be created.
Because the Microsoft 365 Group name becomes the SharePoint site (URL) name, it is a good idea to consider naming conventions.
4. By an end-user creating a new Team in MS Teams
Unless the creation of Microsoft 365 Groups is not restricted, an end-user can create a new SharePoint site (possibly without realising it) by creating a new Team in MS Teams. There is nothing in the creation process to indicate that (a) they will create a SharePoint site or a Microsoft 365 Group, or (b) that they will be the Owner of the Team, Group and SharePoint site – and therefore have responsibility for managing the Team/Group membership.
Every new Team creates a Microsoft 365 Group which always has a SharePoint site and an Exchange Online mailbox that is not visible in Outlook.
5. By creating a Private Channel in MS Teams
If the option is not disabled in the MS Teams admin portal under Teams > Teams Policies, end users will be able to create private channel in a Teams channel. Every private channel creates a new SharePoint site with a name that is an extension of the ‘parent’ Team site name.
For example, if the parent site name is ‘Finance’ and the private channel is named ‘Invoice chat’, the new SharePoint site will be ‘Finance-Invoicechat’. These new site is not connected with the ‘parent’ site and is not visible in the list of Active Sites from the SharePoint admin portal (and so the SharePoint Admin won’t know it exists). It is only visible in the list of Sites under the Resources section of the Microsoft 365 Admin portal.
A private channel does not create a new Microsoft 365 Group. A ‘compliance copy’ of the chats in the private channel are stored in the Exchange Online mailboxes of individual participants in the chat.
6. By the Teams Admin creating a new Team
The MS Teams admin area includes the ability for the Teams admin to go to Manage Teams, click +Add and create a new Team.
As with the end-user creation process, a new Team creates a Microsoft 365 Group that has an Exchange mailbox and a SharePoint site.
7. From the end-user SharePoint portal (+ Create site)
This process creates a Microsoft 365 Group that has a SharePoint site and an Exchange mailbox. It also creates a new Team with the same name.
It is recommended that the ability for end-users to create new sites this way is disabled, at least initially. This is done from the SharePoint admin portal under Settings > Site Creation.
8. From OneDrive for Business as a ‘shared library’
This option is relatively new. When the end-user opens their OneDrive for Business, they will see ‘Create shared library’ directly under a list of sites they have access to under a heading ‘Shared libraries’ (they are actually SharePoint sites; when you click on the site name, it (confusingly) displays the document libraries as … folders.
9. When a new Plan is created in Planner
If end-users open the Planner app, they will see ‘New Plan’ on the top left. This opens a dialogue to create a New Plan or add one to an existing Microsoft 365 Group. The process of creating a new Plan creates a new Microsoft 365 Group with a SharePoint site.
10. When a new Yammer community is created
End users with access to Yammer can click on ‘Create a Community’ from Yammer.
To quote from the Microsoft 365 documentation ‘Join and create a community in Yammer‘: ‘When a new Office 365 connected Yammer community is created, it gets a new SharePoint site, SharePoint document library, OneNote notebook, plan in Microsoft Planner, and shows up in the Global Address Book.’
Why have Microsoft allowed this?
It’s a smarter way to manage access.
Some years back, Microsoft moved away from the idea of having Security Groups that give access to individual IT resources, to having individual Microsoft 365 Groups that provide access to multiple IT resources, in this case resources across Microsoft 365. One Microsoft 365 Group controls access to a SharePoint site, an Exchange mailbox, a Team, a Plan, and a Yammer Community. Security Groups don’t have that sort of functionality.
The trade off is that you get all of these options with a Microsoft 365 Group, whether you like it or not.
But, some of the decisions don’t seem to make sense.
Why allow end-users to create a private channel in Teams when they can simply use the 1:1 chat area?
Why allow the creation of a so-called ‘Shared Library’ from OneDrive, limited to and controlled by the person who created it, when a SharePoint site provides that functionality.
Why does an end-user need an Exchange mailbox (for the Microsoft 365 Group) when they create a new site from the ‘Create site’ option in SharePoint?
And why does a new Plan create a SharePoint site? For what purpose?
Perhaps there is a reason for it. It’s just not clear.
