Managing Physical Records in SharePoint 2010

It is sometimes said by records managers that one of the drawbacks of SharePoint 2010 (and SharePoint 2013) is that it doesn’t manage physical records, or at least not very well.

This article describes how to manage physical records in SharePoint 2010/2013 using a list that includes barcodes.

Up front and out of the box, the main limitations are the ability to: (a) print labels for paper files and boxes (including labels with barcodes) (b) attach a portable scanner to scan barcodes and (c) maintain a chronological history of file movements. These limitations may not be a problem for many organisations seeking a low-cost way to manage mostly inactive files and boxes, especially if they currently use an Access database or Excel spreadsheet to do this now.

Options to address each of these limitations are discussed in this article.

Metadata options

There are almost no limitations on the type of metadata you can use to describe physical records. I would suggest having the following metadata:

• Title (e.g., name on the file or box)
• Record Type (drop down list – file, box, etc)
• Reference ID (manually entered unless you use the default ‘ID’ option, possibly with a separate ‘year’ column)
• Date created (date first registered, default system date)
• Start date (start of date range)
• End date (end of date range)
• Business Owner
• Current location (can be a custom list of pre-defined locations)
• Date at current location
• Record status (drop down list: active, inactive, missing, destroyed etc)
• Internal barcode (if one exists)
• External barcode (usually the one provided by a storage provider)
• Records description (description of contents, usually of a box)
• Box number
• Box barcode (may be the same as the Internal or External Barcode)
• Retention schedule (can be from a pre-defined list)
• Disposal action (can be from a pre-defined list)
• Date to be destroyed (manually entered based on retention schedule and disposal action)
• Destroyed? (checkbox)
• Date destroyed (manually entered date)
• Legal hold? (checkbox)

If physical files are registered in the system, then only details of the box and disposal action would need to be entered. When the box is ready to be sent offsite, details from the offsite storage provider’s label (often provided in advance) can be entered in the external barcode and other metadata (including current location) updated.

An optional barcode can be obtained by checking ‘Barcode’ in the Information Management Policy settings for the list item. Once checked, the barcode appears when the item is displayed in view but not when it is edited or exported.

Printing labels

All list metadata (except the barcode) can be exported to an Excel spreadsheet, which can then be used to print labels. Microsoft provide details how to create and print labels for a list in Excel here:

Scanning

Using a portable (or attached) scanner to scan barcodes and update your records register, especially with a new location (or during a physical records audit) saves a great deal of time instead of entering details manually. It is possible to scan directly to an Excel spreadsheet however several steps may be required to match or validate the barcode with the details in the SharePoint list.

For example, if series of files are being scanned to a box (or location), the usual practice is to scan the box (or location) barcode first, then scan each individual file. If scanning directly to an Excel spreadsheet, you would do it the other way around – have the list of existing files in the spreadsheet, then scan the box or location barcode to the cell next to the file number. This detail can then be copied and pasted back to the SharePoint list.

Chronological movement history

The only option to maintain a chronological movement history, out of the box, would be to add versioning to the list. This means that each time the item is edited, a new version is created, keeping the previous details. The movement history will not display in a single view in the list but clicking on ‘Version History’ will show any changes made in single view. Only ‘major’ version numbers are supported in lists (i.e., 1, 2, 3, 4).

Advertisements

Implementing SharePoint 2010 as a Document and Records Management System

There are several things that are worth asking or considering as a records manager (or equivalent) in a company that is using, or plans to use, SharePoint 2010. This can significantly affect how things occur.

Governance

If your organisation uses SharePoint there is a good chance you will have a qualified SharePoint Administrator in the IT department. This is usually the person (or persons) who ‘own’ the entire SharePoint farm. They need to be your best friends, otherwise you’ll never see all the things you need to see and know about.

You should also consider organising regular meetings with the SharePoint Administrator, Web Admin person and anyone else you think needs to be part of this process.

Existing/legacy SharePoint environment

SharePoint has been around for a decade. If you have a legacy environment, you need to consider either how to migrate or what to do with the legacy environment, because they work differently from an RM point of view.

Also, many IT people haven’t yet worked out how to implement RM in SharePoint 2010 and tend to approach its implementation or configuration with SharePoint 2007 in mind.

Training and knowledge

Records managers need to have a really good understanding of SharePoint administration, at least for Site Collections. You will almost certainly not get access to Central Admin (the domain of the SP Administrator). But you need to know what happens where, and what are the implications of your decisions.

SharePoint domains

You need to understand the relationship and differences between the (a) publishing, intranet/internet environments, (b) the collaboration, DRM, team site environments (where most of your focus will be), and (c) application sites. Generally speaking, (a) belongs to the Web team, and they know how to differentiate between published documents and records. (b) is likely to be mostly uncustomised, out of the box. (c) is likely to be a customised and ‘localised’ solution for specific types of documents.

SQL

It’s worth having some understanding of how SharePoint stores and manages documents (as Base64 encoded data) in SQL tables. It will put you in good stead with a Database Administrator (DBA).

What happens to network shares?

It depends. Shared drives may continue to be used where users need to store working versions. Consider locking them down or migrating the content and then deleting them from the drive.

E-mails

See my post on Microsoft’s Messaging Records Management (MRM) functionality built into Exchange Server 2010. While best practice RM is to store all records together in context, in practice this is not easy with email unless you have a third party tool. There are several on the market including OnePlaceMail (an Australian company).

You can ’email-enable’ document libraries (so the emails go there instead of the email client) but this would only really work for specific types of business process. The real problem are the emails that users receive and don’t do anything with. This is where, possibly, MRM can help. Not an easy answer here because, even if you buy a third party tool, the onus will still be on the end users to do the right thing.

Records declaration

Unlike in the US, which uses the concept of declaring something a record (a feature including with SharePoint 2010), in the Australian legal environment any business related document (which has a broad definition) can be a record of a business process, regardless of whether it was stored in a recordkeeping system or subject to some other RK controls. There is no case law here that focusses on whether a record is a record because it was stored in a recordkeeping system. Instead, there are other factors which mostly relate to the admissibility of something as evidence.

Of course, records are ‘better’ (in terms of integrity, reliability and authenticity) if they are stored in a recordkeeping system that has unique IDs, metadata, audit trails etc. These features are not critical to admissibility as evidence (in the Australian legal context) however they may help.

It might be more useful to take a practical approach and say, let’s not try to distinguish between them, let’s assume anything can be considered a record of a business process or activity (or the smoking gun of something nefarious!). Users generally don’t care unless they are required to keep specific records for compliance reasons.

In-place records management vs the Records Centre – which is best?

It depends. One approach is to leave records with a retention period of less than 5 years on site (controlled via the Content Type IM Policies). At the end of the retention period, these records will be reviewed via a Retention Workflow built into the Content Type IM policies.

Anything with a longer retention period would, after (say) 3 years, be transferred to the Records Centre for longer term storage.

Note that enabling the Records Declaration option in the Site Collection is not essential to in-place records management. You can set it up without having to declare something a record. That said, you might want to consider using the feature to differentiate between records and non-records.

Also, the records declaration feature can also be used to ‘convert’ a ‘working document’ with a short-term retention to a longer-term retention (via a multi-stage retention policy – i.e., keep for x years after declared a record)

How should you structure the way information is stored?

An important thing to remember is that Document ID prefixes are set at the Site Collection level (top site in a collaboration team site environment). So, when you are creating your collaboration site environment, I suggest two things:

• create a ‘mud map’ of the proposed sites (and how the document ID prefix might ‘flow down’ to all sites in the collection, and
• structure your team sites according to a logical organisational structure, no deeper than 2 sites below the Site collection. E.g.:

Site collection (e.g., Division)
> Site (e.g., Section)
> > Sub Site (e.g., Business Unit).

Most documents are likely to be stored at the lowest level of the Site Collection (but some may be stored at levels above).

So, in a sub-site, you will find document libraries which should be a logical business activity that is undertaken. You can ‘group’ libraries, if required, using a group name that makes sense to users.

Document libraries may contain documents, or may contain document sets (which are roughly equivalent to paper files). It’s a good idea to disable the use of folders completely as they can make life very difficult for users to navigate and find information (not to mention potentially exceeding the limit for addresses).

Libraries used to store records should use and have customised document Content Types (created and published from the Content Type Hub) enabled, and the default document Content Type disabled.

Metadata on the Content Type can then used to help ‘view’ the information; you can have hierarchical metadata sets in the Term Store that can (and probably should) be applied to Content Types. You can then enable a view of the metadata, which enables the end user to navigate as they would on a drive, using this metadata.

Note that libraries in the Records Centre can use folders that are based on the File Plan, and those folders will have IM policies overriding Content Type IM policies when the documents are stored there.

Mapping Content Types to the classification and retention schedule

A senior Microsoft solution architect suggested to me in early 2012 that organisations should aim to have fewer rather than more content types, even as few as 30. Microsoft recommend trying to minimise your document Content Types to less than 30 if possible. If you have a lot of retention classes, you can use metadata applied to the Content Type to help route documents to the correct location in the File Plan/Retention Schedule based folder structure of the Records Centre.

For example, the IM Policies for these Content Types would be transferred, after 3 years, to the Records Centre; the Content Organiser uses the metadata to establish where to put the document in the the correct location in the folder-based document library.

Be careful how you configure your Content Types, and use the multi-stage retention to minimise the need to have hundreds of Content Types. And, remember that the metadata you use can also be used to display documents in a view.

Key messages

The key points to a successful implementation are:

• extensive communication and collaboration with all stakeholders
• plan your architecture carefully and have an approved ‘reference’ document (and stick with it!)
• iterative implementation (which is not how you would normally roll out an EDRMS, for example).

How to sell it to end users

One word – carefully. I use the Facebook analogy regularly – 1 billion people use the product and NONE of them needed training (as far as I know).

Sell the benefits, not the system, and if it’s logical and intuitive (and gives value), users will just start to use it. Make it complicated or an inconsistent user experience, tell them they have to be trained, or make it hard to find information (e.g., via folders) and they will just get turned off.

Not referring to it as an RM system in any way helps. This is a business solution that helps them work more efficiently. Most users are happy to know that RM is happening in the background.

Applying recordkeeping policies to email – Microsoft Messaging Records Management (MRM)

The problem

The problem of managing emails as records is summed up in the following statements:

“Many organizations have yet to define an email retention policy. More than one‐quarter of organizations have not yet established any sort of email retention policy despite the fact that there are a growing body of statutory requirements and legal obligations to preserve business records, including those stored in email. Among the nearly three‐quarters of organizations that have established an email retention policy, only two‐thirds of these organizations indicate that their users are fully aware of the policy.” Michael Osterman, “Messaging Archiving and Document Management Markets Trends, 2009-20112”, dated May 2009.

‘Over 40 years after the invention of email, relatively few institutions have developed policies, implementation strategies, procedures, tools and services that support the longterm preservation of records generated via this transformative communication mechanism.’  Christopher J Prom, ‘Preserving Email’, DPC Technology Watch Report 11-01 Decemer 2011. www.dpconline.org/component/docman/doc_download/739-dpctw11-01.pdf

Storing business records in context

Traditional records management theory recommends that there should be a clear relationship between records about a particular subject or issue, regardless of format, and the business context that originated it. (AS ISO 15489-2002: 9.3 Records Capture)

In the paper world, this was achieved by the co-location of related records in a physical file.

In the electronic world, this is usually achieved through the application of metadata. Business classification and naming systems applied to electronic folders generally achieve this; as well, electronic systems also allow for a range of cross-subject metadata that allows records to be organised in different contexts.

Additional, business context-specific metadata can be applied to emails (including from integrated business applications – for example, an email saved to TRIM will show the TRIM record number in its email metadata properties).  However, this ability (as with Properties in Office documents) is rarely enabled or used.

Instead, and as with Office documents, we tend to let users ‘categorise’ their emails (and documents on network shares) through folders – although not all users do this.  (Interestingly, online email systems like Google’s gmail use tags instead of folders).

Are emails documents?

The short answer is yes (in the Australian legal evidence context), but they are documents that, in a way like xml-based Office documents like docx, are made up of structured data that displays as a single ‘document’.

Part of the problem with emails as records is the perception (on the part of users who have never had to face court) that they are not documents, but messages.  The ability to use the system to send or receive ‘private’ messages exacerbates this perception.

The problem of storing emails as records

Emails have been a constant problem and challenge for records managers and recordkeeping since they first appeared in the early 1990s.

The three main approaches to keeping emails have been to (a) print to paper, (b) save to a recordkeeping system, and (c) save to a drive.

Print to paper, while relatively common in many organisations even now, is probably the poorest (and some might say ‘silliest’) option in the digital world as (a) it is dependent on users, (b) emails usually lose their message headers, (c) emails are unsearchable in their electronic form, (d) emails remain on the Exchange system and are discoverable.

Saving emails to a recordkeeping system, while better than printing, is an inadequate option because (a) it is usually dependent on users to do it, (b) the email still remains in the Exchange system, and (c) it can sometimes result in the email being saved in a different format that is not necessarily suitable for long-term preservation (e.g., TRIM’s .vmbx).  There is also the problem of users saving ‘dumb’ emails with (valuable) attachments, which can make the attachment harder to find, identify or access.  Some systems (such as SharePoint 2010) include email-enabled storage locations.

Chris Prom, in a blog posting titled ‘Practical E-Records ‘Facilitating the Generation of Archives in the Facebook Age’, notes that:

‘…the formal recordkeeping systems previously used by many organizations for electronic records have died or have one foot firmly in the grave.  At the same time, the habits that individuals use in producing, consuming, storing, filing, searching, and interpreting records are themselves undergoing constant change.  People adopt new communication technologies at an ever-quickening pace.   Divergent personal practices, rather than the centralized electronic systems, are the harsh reality that confronts our profession’.

Saving to a drive is also a poor option, and is usually based on user preferences to want to ‘keep’ emails.  Emails saved to drives (a) will still remain in the Exchange system, (b) may lose their header information, and (c) are not necessarily saved in appropriate or accessible formats.

In relation to the last point, Outlook does not make it easy for an end user to decide, with usually five options to choose from – which is the right one?  Users will usually choose whatever is the default (.msg), but this isn’t necessarily the best long term option (which is MIME or EML – the latter described by the National Archives of Australia (NAA) as ‘an acceptable open file format for long term storage).

In all cases, keeping these emails in the business context to which they relate has been a constant problem for records managers.  As a consequence, there is a tendency on the part of almost all businesses to leave and manage emails where they are (i.e., in Exchange).

Microsoft Exchange 2010 – Messaging Records Management

To try to address this problem, Microsoft introduced ‘Mailbox Manager Policies’ in Exchange Server 2003.

This was followed by ‘Message Records Management’ with Managed Folders in Exchange Server 2007 (a feature that remains in Exchange 2010).

Exchange Server 2010 includes a new model of managing emails as records, called ‘Messaging Records Management’.  Microsoft describe it as follows:

‘Messaging records management (MRM) is the records management technology in Microsoft Exchange Server 2010 that helps organizations reduce the legal risks associated with e-mail. MRM makes it easier to keep the messages needed to comply with company policy, government regulations, or legal needs, and to remove content that has no legal or business value. This is accomplished through the use of retention policies or managed folders’. (Source: http://technet.microsoft.com/en-us/library/dd335093)

As Microsoft notes, however (on the same page), MRM does not prevent users from deleting messaging; it is really only designed to remove them at the end of a given period.  Microsoft recommend ‘journaling’ emails where there are specific business reasons to keep them for longer (such as legal proceedings or the need to ensure specific email is kept), or applying the Legal Holds functionality.

The key elements of MRM are Retention Policy Tags (RPTs) and Retention Policies.

There are three types of Retention Tags: (1) Default Policy Tags (DPT), (2) Retention Policy Tags, and (3) Personal  Tags (which are an ‘opt-in’ on the email client).

• Retention Policy Tags (RPTs) are used on default folders (e.g., inbox, junk mail, sent, deleted). Users cannot change the RPT but can override it with a Personal Tag.
• Default Policy Tags can be applied by users to untagged items.  A Retention Policy can contain only one default policy tag.
• Personal Tags can be applied by users to their own custom folders or individual emails.

In most cases, users make the decision, and the retention applies on where the email is located.  If there is actual or anticipated litigation, a Retention Hold can be applied to the user’s mailbox; however, this does not prevent users deleting emails, it only overrides any retention policies.  The Legal Hold option should be applied to prevent deletion.  Once this option is applied, Legal Hold ‘captures any deleted or edited items into a special folder that’s neither accessible nor changeable by the user’.

All retention tags include: a Tag Name, a Tag Type, an age limit (in days) with an action to take, and comments.

The actions available are:

• Delete And Allow Recovery – This action will perform a hard delete, sending the message to the dumpster. The user will be able to recover the item using the Recover Deleted Items dialog box in Outlook 2010 or Outlook Web App.
• Mark As Past Retention Limit – This action will mark an item as past the retention limit, displaying the message using strikethrough text in Outlook 2007, 2010 or Outlook Web App.
• Move To Archive – This action moves the message to the users archive mailbox.(see below)
• Move To Deleted Items – This action will move the message to the Deleted Items folder.
• Permanently Delete – This action will permanently delete the message and cannot be restored using the Recover Deleted Items dialog box.

Once the tags are created, they can be added to a Retention Policy and this policy, in turn, is then applied to specific mailboxes – one policy per mailbox.

The ‘auto-tagging’ feature, once 500 items have been tagged, will automatically tag items in a user’s mailox based on their past tagging activities.

So, is MRM the answer to managing emails as records? 

Yes and no.  From a recordkeeping perspective, MRM:

• Does nothing to ensure that records are kept in the business activity or functional context to which they relate, unless (of course) the emails are the only form of record that exists for the business activity.
• Does not stop users from deleting emails.

On a positive note, MRM:

• Attempts to address the problem of email retention.
• Allows the application of a retention policy to emails that might be stored in a business context Outlook mailbox or fold As well, Exchange features like Legal Hold and Journaling allow further controls to be implemented.

Archiving

Exchange 2010 now includes a ‘personal archives feature’, which allows users to save emails to their own archive instead of saving emails to drives or using Personal Storage (.pst). A good article on this subject can be found at this location: http://mohamedridha.com/2011/11/07/exchange-2010-online-archiving-and-retention-tagspolicies-a-practical-example/

Sources (all retrieved 1 June 2012)

• http://www.ostermanresearch.com/execsum/or_arch2009execsum.pdf
• e-records.chrisprom.com
• http://www.dpconline.org/component/docman/doc_download/739-dpctw11-01.pdf
• technet.microsoft.com/en-us/library/dd335093
• blogs.technet.com/b/ianhamer/archive/2009/12/04/curious-about-exchange-2010-retention-policies.aspx
• sysadmin-talk.org/2010/04/manage-exchange-retention-policies-before-they-manage-you/
• techdom.nl/microsoft/exchange-2010-sp1-archiving-part-4-creating-retention-policy-tags-default-folders/

What records managers need to know about Sharepoint 2010

The following is a revised version of an article that appeared in the July 2011 edition of Informaa Quarterly, IQ, the quarterly journal of the Records and Information Management Professionals Australasia (RIMPA).

Sharepoint 2010, is starting to be used by more and more organisations. And it is being used to create and manage records.

The following are the things that records managers need to know about this product.

Check the version

First, Sharepoint 2010 is not Sharepoint 2007. Sharepoint 2010 has recordkeeping functionality built in that its predecessor lacked.

If someone is talking about a Sharepoint implementation, records managers need to know which version – 2007 or 2010.

Know what it does – and doesn’t. 

Microsoft has spent a lot of effort building recordkeeping functionality into Sharepoint 2010.

Records managers need to understand this functionality and how it works, and why they need to be part of the Sharepoint 2010 implementation process at the beginning of the project. They need to know and understand what it can do, and what it can’t.

For example, it doesn’t manage physical records out of the box. It doesn’t integrate well with Outlook. And it doesn’t do security classifications.*

It is very easy overlook or ignore the recordkeeping functionality in Sharepoint 2010 when it is being implemented. The product looks and behaves very much like Sharepoint 2007. So much so, it would be (and probably has been) easy for IT to implement it without any reference at all to records managers.

The first time records managers might find out about Sharepoint is when they find the intranet has been built using it and they are given a team site used to manage records.

The bits that matter

The following is a quick summary of the key recordkeeping elements Sharepoint 2010.

Sites

Sites and team sites in Sharepoint are more or less the same concept as a web site, grouped within an overall Site Collection.

Just like a web site, Sharepoint 2010 sites and team sites are locations accessed via an URL. But, unless access to the site or team site is open to everyone (such as the organisation’s intranet), access will be controlled (often by a site administrator) through a series of permissions. Records managers need to know and understand what this means for the management of records.

Sites may be made up of a range of different content types (see below) used by end users for all sorts of records and ‘non-records’.

One of the main reasons given by IT not to involve records managers in a Sharepoint 2010 implementation is likely to be that ‘there are no records stored on the sites’. This is a bit like saying that there are no records stored in network shares or in Outlook folders.

Of course, it depends on what industry sector you work in and the litigation risks associated with that sector and/or the records created by the organisation.

Without good planning and the involvement of records managers, users may create records on sites without any recordkeeping functionality at all.

Records managers may not know anything about these sites until well after they have been created and content added. By this time, the cat is out of the bag and it could be quite difficult (but not impossible) to retrofit the recordkeeping functionality that could have been applied at the beginning.

Lesson for records managers: Learn about the recordkeeping functionality of Sharepoint 2010 and get involved early with any implementation project.

Managed Metadata

Good metadata should be second nature to records managers. We know that metadata is a good way to describe records that will allow the organisation to find and make better use of records, including disposition decisions.

Sharepoint 2010 is a dream product for metadata lovers. The opportunities to apply metadata to – and in – records are almost limitless.

The close integration between Sharepoint 2010 and Office 2010 allows recordkeeping metadata to be be embedded in documents and then re-used on sites in metadata columns.

A key feature in Sharepoint 2010 is the Managed Metadata Service (MMS). The MMS allows the creation of multiple hierarchical descriptions of information that can then be applied to records across sites.

End users can also tag records with their own metadata which is then surfaced in the MMS, allowing it to be added to the rest of the organisation’s metadata as required. This is particularly useful when a new subject appears in the organisation.

In some ways, the MMS resembles folder names in Outlook or network shares. It is the way end users see their world.

Lesson for records managers: Understand how end users describe their world and how Sharepoint 2010 can capture that information. Recordkeeping metadata sometimes doesn’t make sense to end users.

Content Types

Content types are the main recordkeeping ‘currency’ in Sharepoint 2010.

Sites consist almost entirely of content types, including when a user clicks on ‘New – (Content Type)’ in a Library or List, or creates content in the form of a wiki or event.

Records managers who are familiar with TRIM record types will see a certain similarity with Sharepoint 2010 content types. But the resemblance is superficial and it would be a mistake to try to make content types behave like TRIM record types.

TRIM record types are generally few in number and mostly consist of documents, files (containers) and boxes. Retention information is generally applied at the file (container) level, and documents inherit that information. Documents are generally single, digital objects of some sort.

In the Sharepoint 2010 model, content can be literally anything. One way of thinking about content types is that each one can ‘map’ to a disposal class in a retention schedule.

Some content types can (and should) be created by customising the top level content type. This is because each content type can then have its own specific metadata and retention information.

Other content types, such as wikis and calendars, can be left ‘as is’ as they are likely to be managed and retained the same way across all sites. For example, calendars could all have a retention period of 2 years after last entry.

Records managers need to understand that *every* content type can be assigned two key elements. These are: (a)a range of recordkeeping metadata that is mostly invisible to end users, but can include end-user defined metadata, and, (b) Information management (IM) policies.

IM policies, and in particular retention policies, are one of the *key* differences between Sharepoint 2010 and 2007. This functionality is not overly obvious if you don’t know where it is or what it does.  It should not be confused with ‘expiry dates’.

The importance of getting IM policies right becomes more obvious when they are not configured correctly. When this happens, (such as by configuring the policies on the parent, generic content type), it can become impossible to do anything with them, and all records using the same content type could have either no recordkeeping rules applied, or it will be same regardless of the type of record it is.

This may not be a bad thing for generic types of records, such as calendars, but could be disastrous if every single document saved into the site collection has the same retention policy.

A *key* information management policy element is Retention. This check-box functionality is easily overlooked but it is a critical piece of recordkeeping as it defines how long the record will be kept. As noted already, each content type can (and perhaps should be) mapped to a class in a retention schedule (or to Normal Administrative Practice (NAP), for Australian government agencies).

Retention policies allow for multi-stage disposition changes to records. They also allow records either to be disposed of on a site after a given period, or sent to the Records Centre for review and/or disposal or further retention.

Sharepoint 2010 includes the US-style concept of ‘declaring’ something as a record. While this functionality is alien to most Australian records managers, it has potential use for allowing changes to the status of a record.

Records stored on network shares or in email folders provide a useful analogy. If the organisation has an EDRMS, end users must transfer the record from the drive or folder to the EDRMS. In most cases this process will leave the document or email in its original location.

The records declaration function in Sharepoint 2010 would allow for a document that is created or saved to a site as a ‘non-record’ (eg, a ‘working document’) to be ‘declared’ as a record on the site, resulting in a change to retention requirements for the same record, rather than expecting an end user to save it to a different system.

Information management policies also include the ‘auditing’ function, allowing auditing on a range of activities.

Lesson for records managers: Learn about Content Types and understand how Information Management policies work and can be configured.

Where are the containers?

Potentially one of the most confusing things about Sharepoint 2010 for records managers seeing the product for the first time is the apparent absence of containers.

Collections of document-type records can be stored in libraries. Libraries may contain folders (another content type) defined by users, or document sets which resemble containers.

It is important to keep an open mind about the concept of a container when learning about Sharepoint 2010.

The product offers multiple ways to organise records in a given context, including through libraries, folders and document sets, but also (and perhaps most importantly) through metadata.

The way end users organise their records is likely to depend on their (or their business unit’s) context. Different users, and records managers, may see the world in different ways and organise information accordingly.

Sharepoint 2010 provides the flexibility to meet these sometimes competing interests, including by allowing end users to organise information that makes sense to them, and then send it to the Records Centre at or after a given time, to be organised in a way that makes sense to records managers.

Lesson for records managers: Look outside the square at how records might be organised. Pre-defined containers are not necessarily the only option.

The Content Organiser

As noted above, information management policies in content types in Sharepoint 2010 can be used to define retention rules for all types of content.

In the same way that rules that can be applied in email applications to route emails to specific folders, the content organiser in Sharepoint 2010 can be used to route any content type to a specific, pre-defined location in the records centre.

This is done by using the feature to send the content type to a different location such as the records centre after a given period of time.

Lesson for records managers: Learn how content types work. If you are involved at the beginning of a Sharepoint 2010 implementation you have a better chance of setting up and configuring content types correctly so they will eventually be managed in the records centre. This can include the ability for records managers to review new, undefined content that is sent to a ‘drop off’ area in the Records Centre.

The Records Centre

The Records Centre in Sharepoint 2010 is, in effect, a separate site in the site collection (or farm) to store corporate records. An organisation may have more than one records centre.

The centre should be owned and managed by records managers. There are various ways to configure a records centre.

One option would be to create a library (or libraries) for records, each of which contains a pre-defined folder structure mapped to the organisation’s Business Classification Scheme (BCS).

As these folders can have IM policies applied to them, they can be mapped to specific classes in the organisation’s retention schedule. In this way, they start to look very much like containers!

As Sharepoint 2010 allows for retention rules to be applied to either libraries and folder OR content types, each of these folders can have retention rules applied to them.

Lesson for records managers: Own the records centre!

A Semantic Office

Abstract

The basic concepts and theory behind the idea of a ‘semantic web’ are already available in XML-based documents produced in Word 2007 and Open Office. This has potential implications for the way records could be managed within organisations, leading to the possibility of a ‘semantic office’.

Background

The conceptual basis for a ‘semantic web’ was first defined by Tim Berners-Lee and Mark Fischetti in their 1999 book ‘Weaving the Web’ (HarperSanFrancisco. chapter 12. ISBN 9780062515872.).

At its core, the concept behind the Semantic Web is creating the ability for computers to understand the meaning (‘semantics’) of information in a web page, by presenting that information in a machine understandable format.  In very simple terms, this can be achieved through the use of pre-defined, agreed terms to describe informational elements contained within a web page as individual elements of data.  This has led to the suggestion of a ‘World Wide Database’ (Nova Spivack, 2005).

Until recently, the primary way to present information that could be readable in a browser was to define it using hypertext mark-up language, or HTML (current standard version HTML 4), which (like all mark up languages derives from the original Standard Generalised Markup Language (SGML).

While HTML is an effective way of presenting and formatting information read by a browser, it (and its successors including XHTML and more recently HTML 5) does not allow for the definition of all individual informational elements contained within the web page.  Extensible mark-up language (XML, released in 1998), Web Ontology Language (OWL) and Resource Description Frameworks (RDF, released in 1999), on the other hand, allow individual informational elements to be defined.  The framework for defining information in this way is described below.

The concepts behind the description of individual elements within a web page has its origins in object oriented programming languages of the late 1980s and early 1990s.  It can be argued that its origins can be traced further back to the very early 1980s and the development of relational databases that could link, analyse and present data.

In 1988, B Pernici from the Politecnico di Milano presented an article to the Conference on Office Information System in Palo Alto.  The article, ‘Supporting OIS design through semantic queries’ , proposed a semantic query language to assist in the retrieval of information from conceptual schemas in office systems.

This was, possibly, the first reference to the Semantic Office.

Developments within the field of semantic representation of knowledge appear to have focussed on two main areas from the mid 1980s – the management of data within database systems, and the management of informational or data elements within web based information.

In the early to mid 2000s, possibly with this background and other drivers, both Microsoft and Open Office began to work on new XML-based document file format standards.  Microsoft’s version was called ‘Office Open XML’ and was published in November 2008 as ISO/IEC 29500:2008 (also released as ECMA-376 Office Open XML File Formats – 2nd edition, in December 2008); OpenOffice.org released ‘OpenOffice.org XML’, published in November 2006 as ISO/IEC 26300:2006 Open Document Format for Office Applications (OpenDocument) v1.0.

Interestingly, Office 2007 is not apparently entirely in compliance with the ISO/IEC 29500:2008, but Microsoft expects this will be the case with Office 2010. (http://www.microsoft.com/interop/letters/ChrisCapOpenLetter.mspx)

Defining Semantics

The broad concept behind the Semantic Web is that a document (in this case a web page) can contain elements of information (or data) that are described within pre-defined ‘categories’, therefore allowing other data described in the same category but in other documents (web pages, or documents on a server accessible through the same web page) to be found, retrieved but more importantly used in potentially completely different contexts.

In a wide sense, it should allow accessible information in any part of the internet to be retrieved and used in this way.  It turns what was unstructured information into structured information.

As described on the main page of http://semanticweb.org/wiki/Main_Page, ‘The Semantic Web is the extension of the World Wide Web that enables people to share content beyond the boundaries of applications and websites.’

At the core of the Semantic Web is agreement on how information should be described, in the form of metadata (‘information about information’).  One of the most well known set of metadata is the Dublin Core, which consists of 15 simple elements including : Title, Creator, Subject, Description, Date.

Metadata sets are often conceptually the same thing as ontologies, taxonomies, and even (to hark back to its origins) data dictionaries, often when metadata sets become quite complex with relationships with other sets and all terms are used interchangeably, often depending on the context of the person.  Web Ontology Language (OWL) is a technology for developing ontologies.

Agreed sets are known as schema; the schema used in a Semantic Web web page are defined at the beginning as XML name spaces or XMLNS (for example ‘xmlns:dc=http://purl.org/dc/terms/’).  The presence of this at the top of a web page means that the web page contains metadata elements drawn from Dublin Core within it somewhere. (For example: <span property=”dc:title”>How to Publish Linked Data on the Web</span>)

One ontology that has become very common on the net in recent years is ‘Friend of a Friend’, or FOAF.  FOAF is a way to describe people and their relationships in an agreed format.   FOAF includes metadata elements such as:  Person, name, nick, homepage, weblog, knows, interest, plan, based_near, age, OnlineAccount, Group, member, and so on. This would appear in a web page like this: <h1 property=”foaf:name”>Andrew Warland</h1>

These agreed metadata sets, ontologies or schema are presented on web pages in XML format as shown above using the agreed Resource Description Framework (RDF) for describing information.  RFD in its simplest form consists of ‘triples’ based on: subject, predicate, object.

For example: ‘The title of the book is the Semantic Office’.  Here, ‘Book’ is the subject, ‘title’ is the predicate’, and ‘The Semantic Office’ is the object. To define this in a web page, we would first have to choose an appropriate schema. In this case, Dublin Core seems appropriate:

xmlns:dc=”http://purl.org/dc/elements/1.1/”&gt;

Within the body of the web page, we would then include:

<dc:title>The Semantic Office</dc:title>

The relationship to Office Documents

To see the XML contents of a Microsoft docx or xlsx document, simply rename the format from docx or xlsx to zip.  You can then open the zipped package and explore the contents.

The following shows the XML-based content embedded within a recent sample docx document.  As you can see, this way of presenting information is identical to the way it is presented in Semantic Web formatted documents.

Within the folder ‘docProps’:

-xmlns:dc=http://purl.org/dc/elements/1.1/

-<dc:title>Test document</dc:title>

Within the folder ‘word’:

-xmlns:ve=http://schemas.openxmlformats.org/markup-compatibility/2006

-xmlns:o=”urn:schemas-microsoft-com:office:office”

-xmlns:r=”http://schemas.openxmlformats.org/officeDocument/2006/relationships&#8221;

-xmlns:m=”http://schemas.openxmlformats.org/officeDocument/2006/math&#8221;

-xmlns:v=”urn:schemas-microsoft-com:vml”

-xmlns:wp=”http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing&#8221;

-xmlns:w10=”urn:schemas-microsoft-com:office:word”

-xmlns:w=”http://schemas.openxmlformats.org/wordprocessingml/2006/main&#8221;

-xmlns:wne=”http://schemas.microsoft.com/office/word/2006/wordml&#8221;

and within the document a range of pre-defined information built around the schemas listed above.

What are we seeing here?  Microsoft Word documents presented in almost the same way that web pages in the Semantic Web are formed.

If we included recordkeeping metadata in the same structure, we are achieving, potentially, an internal electronic office environment, accessible via a browser interface, that allows information to be stored, accessed and used in the same way information in the semantic web is used.

So how do we achieve a Semantic Office?

According to Microsoft, this type of additional information is a customised extension of the document information properties, and can only be added using InfoPath.  But, once it’s there, it can then be utilised again and again in standardised ways.

What does this mean? Potentially, it means that end users can create documents with all the required recordkeeping metadata embedded within the document.

Of course, some of this information is already there in the form of basic document properties.  But, the ability to extend this basic set has ramifications for the way information is created, stored, found, retrieved and used that has very close similarities with the way information is being presented in the Semantic Web.

When we consider how almost every application to manage documents and records is now browser based, the ability to apply Semantic Web and Web 2.0/3.0 tools to this information within the enterprise means that records and the information content of those records could be used in ways that were never previously considered possible.  Some products, including EMC’s Documentum, now include an XML store in addition to the traditional file store and relational database.  (See http://www.emc.com/products/detail/software/xml-store.htm).

For example, instead of consigning documents to pre-defined containers or folders within a file plan, it might instead be possible to define the container, title and classification, and place organisational information about the author and her/his organisational context within the document metadata automatically, as part of the recordkeeping metadata schema.  This, in a sense, is an encapsulated object, and is not new, but the ability to do it through the original document (eg Word) is new.

Copyright – Andrew Warland 2010

References

Konsynski, B.R., Bracket, L.C., and Bracket, W.E., ‘A model for specification of office communications’, IEEE Trans. on Comm., Vol. COM-30, N. 1, Jan. 1982.

Nutt, G.J. and Ricci, P.A., ‘Quinault: an office modeling system’, Computer, May 1981.

David W. Shipman, The functional data model and the data languages DAPLEX, ACM Transactions on Database Systems (TODS), v.6 n.1, p.140-173, March 1981

Pernici, B., Barbic, F., Fugini, M.G., Maiocchi, R., Rames, J.R., and Rolland, C., ‘C-TODOS: An automatic tool for office system conceptual design’, Politecnico di Milano, Electronics Dept., Rep. N. 87-15, 1987.

Li Ding, Lina Zhou, Tim Finin, and Anupam Joshi, How the Semantic Web is Being Used:An Analysis of FOAF, Proceedings of the 38th International Conference on System Sciences, January 2005.

Spivack, Nova. ‘Towards a world wide database’.  Blog post 27 October 2005.  http://novaspivack.typepad.com/nova_spivacks_weblog/2005/10/towards_a_world.html accessed 23 January 2010.

Capossella, Chris ‘An Open Letter from Chris Caposella, Senior Vice President, Microsoft Office’ . http://www.microsoft.com/interop/letters/ChrisCapOpenLetter.mspx

http://xmlns.com/foaf/spec/

Integrity

Integrity is a key element in recordkeeping.  According to AS ISO 15489, integrity of documents means that they are complete and unaltered.

These words mirror s.11(3) of the Electronic Transactions Act 2000 (NSW) which notes that the integrity of information contained in a document is maintained if, and only if, the information has remained complete and unaltered, apart from:

(a) the addition of any endorsement, or

(b) any immaterial change, which arises in the normal course of communication, storage or display.

Processes, machines and devices

s.146 of the Evidence Act notes that:

• Where a document or thing is produced wholly or partly by a device or process, AND
• the device or process is one that, or is of a kind that, if properly used, ordinarily produces that outcome, THEN
• it is presumed (unless evidence sufficient to raise doubt about the presumption is adduced) that, in producing the document or thing on the occasion in question, the device or process produced that outcome.

The balance of probabilities

s. 142 (Admissibility of evidence: standard of proof) of the Evidence Act 2005 (NSW) notes that evidence will be usually admitted if the Court is satisfied that the facts have been proved on the balance of probabilities.  The matters that the court must take into account include (summarised – see legislation for full version): (a) the importance of the evidence in the proceeding, and (b) the gravity of the matters alleged in relation to the question.

David Hamer, in his very interesting and informative article ‘Chance Would Be a Fine Thing: Proof of Causation and Quantum in an Unpredictable World’, for the Melbourne University Law Review in 1999, noted that

It is well established that, while the civil standard is the balance of probabilities, the precise meaning of this expression depends upon the circumstances of the case and, in particular, what is at stake for each of the parties.  … Where the plaintiff’s allegations have moral overtones, as in fraud or adultery cases, the civil standard may be higher than where mere negligence or breach of contract is alleged.

The need for authenticity

In NAB v Rusu, Justice Bryson said that:

At its simplest, the authenticity of a document may be proved by the evidence of the person who made it or one of the persons who made it, or a person who was present when it was made, or in the case of a business record, a person who participates in the conduct of the business and compiled the document, or found it among the business’ records, or can recognise it as one of the records of the business. NAB v Rusu, at 17, ASIC v Rich at 98

Therefore, as noted by Justice Austin in ASIC v Rich

Authentication is about showing that the document is what it is claimed to be, not about assessing, at the point of the adducing of the evidence, whether the document proves what the tendering party claims it proves. ASIC v Rich at 118

All records management professionals should already be very familiar with the definitions of authenticity, reliability and integrity in AS ISO 15489.  In my mind, ensuring that documents are kept in line with this standard is a very good first step in ensuring that your documents will stand up as evidence in court.

The following are the definitions of authenticity, reliability and integrity from AS ISO 15489:

• To ensure authenticity, organisations should implement and document policies and procedures which control the creation, receipt, transmission, maintenance and disposition of records to ensure that records creators are authorised and identified and that records are protected against unauthorised addition, deletion, alteration, use and concealment.
• A reliable record is one whose contents can be trusted as a full and accurate representation of the transactions, activities or facts to which they attest and can be depended upon in the course of subsequent transactions or activities.
• Integrity – complete and unaltered.

The Uniform Civil Procedures Rules (UCPR) that govern court processes state (in summary) that, if a party receiving documents as part of discovery does not dispute their authenticity, then the authenticity of the documents will be taken to be accepted.  However, the admitting party may withdraw such admission with the leave of the court.

Judgments as to the authenticity of scanned documents may be influenced by:

• The balance of probabilities
• Inferences that can be drawn
• Proof of their contents
• How they were produced – for example by processes, machines and devices
• Integrity

Sources:

• ASIC v Rich [2005] NSWSC 417 (5 May 2007)
• Albrighton v Royal Prince Alfred Hospital [1980] NSWLR 542
• Daw v Toyworld [2001] NSWCA 25
• Nab v  Rusu  [1999] NSWSC 539 (4 June 1999)

Admissibility

Justice Bryon commented on the admitting in evidence of documents in NAB v Rusu.  He said that:

Documents are not ordinarily taken to prove themselves … Before a business record or any other document is admitted in evidence it is obviously necessary that there should be an evidentiary basis for finding that it is what is purports to be. NAB v Rusu at 17, also quoted in ASIC v Rich at 98

This is often know as the ‘evidentiary basis’ for documents, sometimes also known as ‘provenance evidence’.

Sources:

• ASIC v Rich [2005] NSWSC 417 (5 May 2007)
• Albrighton v Royal Prince Alfred Hospital [1980] NSWLR 542
• Daw v Toyworld [2001] NSWCA 25
• Nab v  Rusu  [1999] NSWSC 539 (4 June 1999)