A recent ‘functional and efficiency’ review into the National Archives of Australia (also known as the ‘Tune Review’, published on 30 January 2021) noted the ‘rapid and ever-evolving challenges of the digital world’.
It stated that ‘the definition of a ‘record’ needs to reflect current international standards, be more directly applied to digital technologies, and more clearly provide for direct capture of records that are susceptible to deletion, such as emails, texts or online messages’.
The review also highlighted the difficulties associated with ingesting digital records ‘via manual intensive activities (due to lack of interoperable systems)’ and proposed a new model based on the ‘continuous automated appraisal of [Agency] digital records that would require a combination of artificial intelligence and skilled archivists’.
The review underlined the challenges of identifying and managing born-digital records, and the need for better solutions.
This post explores the challenges of accurately and identifying born-digital records in order to manage them.
Identifying and protecting records
Records usually provide evidence of something that happened – an action, an activity or process, a decision, or a current state (including a photograph or video record). They may have or be associated with descriptive metadata used to provide context to the records and guide or determine retention.
Like all other types of evidence, the authenticity, integrity and reliability should be protected for as long as they must be kept.
In the paper world, this outcome was achieved by storing physical records (including the printed version of born-digital records) on paper files or in physical storage spaces.
For the past twenty years or so, this outcome was achieved for (some) digital records by (mostly manually) copying them from a network drive or email system (or via a connector) to a dedicated electronic records management (ERM) system and then ‘locking’ them in that system to prevent unauthorised change or deletion. Most ERM systems consisted of a database for the metadata and an associated network drive file store for the objects.
The main problem with this centralised storage model – however good it might be at protecting copies of records stored in it – was that the original versions, along with all the other records that were not identified or could not be copied to the ERMS, remained where they were created or captured.
And the records stored ‘in’ the ERMS were actually stored on a network file share on a server that was (a) accessible to IT, and (b) almost always backed up. So, yet more copies existed.
The challenge of born-digital records
There are several key challenges with born-digital records:
- Consistently and accurately identifying (or ‘declaring’) all records in all formats created or captured in all locations. For too long, the focus has primarily been on emails and anything that can be saved to a network drive with the onus of identifying a record on end-users.
- Ensuring their authenticity, reliability and integrity over time. For records stored in the ERMS, this has usually involved locking them from edit, including through the ‘declaration’ process, or preventing deletion. But in almost all cases, the original version (in email, on the network drives), could continue to be modified. Other records that were not identified or stored in an ERMS may be deleted.
- Ensuring that born-digital records will remain accessible for as long as they are required.
It is not possible to consistently and accurately manually (or even automatically) identify every born-digital record that an organisation creates or captures to ensure their authenticity, reliability, integrity or accessibility over time. Only a small percentage of born-digital records are copied to an ERMS.
Records remain hidden in personal mailboxes, personal drives and third-party (often unauthorised) systems. Records may exist in multiple forms and formats, sometimes created or stored in ‘private’ systems or on social media platforms. They may take the form of text or instant messages or social networking posts and threads. They may be drawings, images, voice or video recordings.
Even if a record is identified, it is not always possible to save it to an ERMS. Text or instant messages on mobile devices are a case in point that has been a problem for at least two decades. More recent examples include chat messages, reactions (emojis, comments), and recordings of online meetings.
And even if a high percentage of born-digital records could be stored in the ERMS, the original versions will almost always remain where they were created or captured.
A different approach is needed.
Triaging records?
One approach to the problem would be to accept that not all records have equal value. That is, not all records need to be managed the same way.
To some degree, this way of thinking is already reflected in classes in the structure of records retention schedules and the attention paid to each:
- Records that have permanent or archival value and need to be transferred to archival institutions.
- Specific types of records that must be created or kept by the organisation for a minimum periods (sometimes quite long but not ‘forever’), for legal, compliance or auditing purposes.
- Records that are not subject to legal or compliance requirements but which the organisation decides to keep for a minimum period of time.
- Everything else.
Triaging records means that they can be managed as required at each level, but nothing is missed. It requires a risk management approach.
For records of permanent value, or are subject to legal or compliance requirements, it means that ensuring that these records receive the most attention and every effort it made to ensure that they are and can be identified (declared) and managed accordingly. This would include ensuring that it is possible to identify and capture these records in the systems used to create or capture them, for example, key emails.
A similar approach would be taken to records that need to be kept for legal, compliance or auditing purposes but with an understanding that some of these records (e.g., emails) may remain in the original system where they were created or captured. Technological solutions may be used to identify or tag these records. The destruction of these records should be subject to some form of review and a record kept of the approval and what was destroyed.
For all other records would remain stored wherever they were created or captured and subject to minimum retention periods after which they can be destroyed without review – but a record kept of the basic metadata of each record (including original storage location).
Protecting – or proving – the authenticity, integrity and reliability of records
The assumption behind the protection of records is that they should not be changed or deleted.
The reality, with digital records, is that they may change at any time through new threads, new revisions, new chats, or even through photoshopping.
A more realistic approach may be to use information about what was changed, by whom, and when – not to protect the record but to provide an evidentiary trail to prove what it is or was. The ‘smoking gun’ evidence for most born-digital records is the metadata that is recorded when it was captured or modified, not (necessarily) the added descriptive metadata.
For example:
- Someone may author a document (metadata records each revision, and each revision can be viewed).
- The document may be approved electronically (recorded in metadata).
- Someone then modifies the approved version.
- All of the above is recorded in the ‘modified’, ‘modified by’ and approval metadata.
- The record should (or may) also recorded who viewed the record, and when.
EXIF metadata stored on images provides a similar form of evidence (and may even include GPS information).
Which record is more likely to be accepted as evidence:
- A record stored in an EDRMS, versions or revisions of which may exist in multiple other places, including on network file shares, email system and even backup tapes
- A record stored in a system that shows the full set of metadata about access and changes, or the most recent thread of an email discussion?
Conclusions
At the end of the day, it should be possible to confirm the authenticity, reliability and integrity of records based on information/metadata that forms part of the born-digital record: who and when it was created, the context in which it was created and its relationship with other records.
Perhaps, instead of focussing on trying to identify and capture all born-digital objects that might be records and ‘protecting’ a version of that record, it may be more practical and easier to leave most records where they were created or captured (and retained by retention policies) and use change or revision metadata to provide evidence of authenticity.
This may, in the end, be a much easier way to protect the authenticity of records than having to rely on manual identification or declaration.
Records about the world 
One thought on “The challenge of identifying born-digital records”