Sentencing and disposing of records in SharePoint 2010

This article applies only to on-premises installations of SharePoint 2010. 

SharePoint’s ‘Records Centre’ was, in theory, a place to send records from various sites for long term storage, sentencing and disposal. The idea was that you could automatically, via Content Type rules, or directly, via the ‘Send to Records Centre’ option, transfer records out of team (and other) sites into the Records Centre. 

While the theory made sense, in practice there were several problems with the model, not the least being that ‘transferred’ records were not actually transferred but copied and effectively re-registered as new records in the Records Centre. They also lost previous versions, and so on. 

We needed to find another way to manage the sentencing and disposal of records. SharePoint 2013 has the (information management) option to apply a sentence on an entire site, which is great if you want to do that, but in most cases the requirement is to sentence parts of a site, including whole libraries or lists. 

As our document-based records are stored in document libraries, and these libraries generally (but not always) have names that make sense (if you can stop people using the generic ‘Shared Documents’ default library), it seemed a good idea to focus on how we could apply retention and disposal policies to document libraries. 

The first problem is visibility of all those libraries, created across multiple site collections. The only way to see all of them was to be the SharePoint Administrator or have Site Collection Administration privileges across all sites. But it was cumbersome to have to open every site to see what was stored in them. 

I put this problem to our SharePoint Administrator and developer (Eric Fang – blog here: http://fangdahai.blogspot.com.au/) . Using PowerShell scripts, Eric developed a method to display all document libraries across all Sharepoint sites in a list. The list updates on a regular basis.

NOTE: You cannot create this type of list ‘out of the box’, it requires PowerShell scripts, and that will need to be maintained over time. This is not a skill that is normally found with most SharePoint Administrators. 

The list displays:

• The library GUID
• The library name and URL
• The site collection and sites, including URLs
• The number of items stored in each library
• The date the library was created and who created it
• The date any item in the library was last modified

The first, immediate, benefit we could see from this method of displaying libraries was the number of libraries that had been created but not used. We could immediately see the ability to reduce the number of libraries, especially if these had not been used for a given period (say, one year).

The next benefit was the ability to group libraries by site collection. As many of our site collections map to business functions, we could start to see the volume of content that was stored for each function, by library – many of which were created to store documents created through various activities.

For example, a common document library is ‘Meetings’. We can now filter the view to see all libraries that contain documents relating to meetings. We can also see types of libraries that have specific retention and disposal requirements.

While the list of all libraries has provided an excellent way to view all our libraries, we are now working on a method to apply retention and disposal actions to these libraries. One way to do this would be to add an extra column in the list for retention and disposal information (class number, class decription, disposal action, expected disposal date, approved by, etc).

Once a disposal action had been applied to the list, we can then review it when the disposal action became due to determine if the library could in fact be destroyed. If it can be destroyed, it would be possible to export the original library metadata columns to a spreadsheet to keep a record of what was destroyed, and when.

SharePoint 2010 – Send to Records Center / Centre

SharePoint 2010 includes the option to send a document from the site where it was stored to the Records Center (Records Centre in Australia), and other locations. The option is set at the Central Administration Web site under General Applications – Configure Send to Connections. See http://technet.microsoft.com/en-us/library/ee424395(v=office.14).aspx for the details of how to configure this option.

The ‘Send to action’ list offers three options when a document is sent to the Records Centre (options which are then available across the farm):

• Copy the document (which simply makes a copy)
• Move the document (which appears to remove the document entirely. Microsoft state that ‘users will no longer be able to access the document from its original location’, suggesting it is not actually deleted from the original location)
• Move and leave a link (see below)

Microsoft state (in the link above) that this option will ‘… delete the document from its current location, move it to the destination repository, and leave a link at the current location indicating that the document has been moved’. If Document IDs have been enabled, the Document ID will disappear next to the document icon, which then shows a hyperlink ‘hook’ to the document in the Records Centre. A user clicking on that link will be advised that the document has been moved.

The impact of versioning, and metadata, on ‘Send to’ moves

There are a number of issues, and potential problems, with the ‘Send to ‘ move option that need to be understood. Some of these might be sufficient to decide against using the Send to Records Centre option (or even using the Records Centre).

• Versioning. If versioning has been applied to the document library, and versions exist, only the most current version will appear in the Records Centre. The Document ID will be the same as the current version. However, any previous versions remain in the original library. In one sense, it is good to show previous versions, however any of these previous versions can be restored. When this happens, the version that is restored now appears, with the original document number (and a ‘hook’ still on the icon). The logic of this appears to be that, if a previous version is restored, the version that has been sent to the Records Centre is no longer the current version. However, it remains – with the same Document ID – in the Records Centre.
• Metadata. Any additional, local metadata that has been added to the Content Type in the original Site Library will disappear when the document is moved. Metadata in Content Types stored in the Content Type Hub will remain.
• Created date. The ‘Date Created’ field in SharePoint is the date the document was created in SharePoint. When the document is sent to the Records Centre, it is assigned a new ‘Date Created’. The date created field on the original document remains unchanged.
• Permissions. Any permissions that may have been applied to the document, via inheritance from the Library or Site, are removed and the document inherits the permissions of the relevant library in the Records Centre.

The implications for ‘Send to Records Centre’

The issues documented above could affect planning and decision making regarding use of the Records Centre and the ‘Send to’ functionality. Some issues to consider include:

• Loss of versions when the document is moved
• Ability to restore a previous version (and confusion about which is the current version)
• Loss of metadata applied locally to content types
• Changes to access permissions

If any of these are concerns for the organisation, consideration might be given to leaving documents in place where they were stored and managing them in that way. The Record Centre could instead be used to archive and apply retention policies to content from network drives.

Implementing SharePoint 2010 as a Document and Records Management System

There are several things that are worth asking or considering as a records manager (or equivalent) in a company that is using, or plans to use, SharePoint 2010. This can significantly affect how things occur.

Governance

If your organisation uses SharePoint there is a good chance you will have a qualified SharePoint Administrator in the IT department. This is usually the person (or persons) who ‘own’ the entire SharePoint farm. They need to be your best friends, otherwise you’ll never see all the things you need to see and know about.

You should also consider organising regular meetings with the SharePoint Administrator, Web Admin person and anyone else you think needs to be part of this process.

Existing/legacy SharePoint environment

SharePoint has been around for a decade. If you have a legacy environment, you need to consider either how to migrate or what to do with the legacy environment, because they work differently from an RM point of view.

Also, many IT people haven’t yet worked out how to implement RM in SharePoint 2010 and tend to approach its implementation or configuration with SharePoint 2007 in mind.

Training and knowledge

Records managers need to have a really good understanding of SharePoint administration, at least for Site Collections. You will almost certainly not get access to Central Admin (the domain of the SP Administrator). But you need to know what happens where, and what are the implications of your decisions.

SharePoint domains

You need to understand the relationship and differences between the (a) publishing, intranet/internet environments, (b) the collaboration, DRM, team site environments (where most of your focus will be), and (c) application sites. Generally speaking, (a) belongs to the Web team, and they know how to differentiate between published documents and records. (b) is likely to be mostly uncustomised, out of the box. (c) is likely to be a customised and ‘localised’ solution for specific types of documents.

SQL

It’s worth having some understanding of how SharePoint stores and manages documents (as Base64 encoded data) in SQL tables. It will put you in good stead with a Database Administrator (DBA).

What happens to network shares?

It depends. Shared drives may continue to be used where users need to store working versions. Consider locking them down or migrating the content and then deleting them from the drive.

E-mails

See my post on Microsoft’s Messaging Records Management (MRM) functionality built into Exchange Server 2010. While best practice RM is to store all records together in context, in practice this is not easy with email unless you have a third party tool. There are several on the market including OnePlaceMail (an Australian company).

You can ’email-enable’ document libraries (so the emails go there instead of the email client) but this would only really work for specific types of business process. The real problem are the emails that users receive and don’t do anything with. This is where, possibly, MRM can help. Not an easy answer here because, even if you buy a third party tool, the onus will still be on the end users to do the right thing.

Records declaration

Unlike in the US, which uses the concept of declaring something a record (a feature including with SharePoint 2010), in the Australian legal environment any business related document (which has a broad definition) can be a record of a business process, regardless of whether it was stored in a recordkeeping system or subject to some other RK controls. There is no case law here that focusses on whether a record is a record because it was stored in a recordkeeping system. Instead, there are other factors which mostly relate to the admissibility of something as evidence.

Of course, records are ‘better’ (in terms of integrity, reliability and authenticity) if they are stored in a recordkeeping system that has unique IDs, metadata, audit trails etc. These features are not critical to admissibility as evidence (in the Australian legal context) however they may help.

It might be more useful to take a practical approach and say, let’s not try to distinguish between them, let’s assume anything can be considered a record of a business process or activity (or the smoking gun of something nefarious!). Users generally don’t care unless they are required to keep specific records for compliance reasons.

In-place records management vs the Records Centre – which is best?

It depends. One approach is to leave records with a retention period of less than 5 years on site (controlled via the Content Type IM Policies). At the end of the retention period, these records will be reviewed via a Retention Workflow built into the Content Type IM policies.

Anything with a longer retention period would, after (say) 3 years, be transferred to the Records Centre for longer term storage.

Note that enabling the Records Declaration option in the Site Collection is not essential to in-place records management. You can set it up without having to declare something a record. That said, you might want to consider using the feature to differentiate between records and non-records.

Also, the records declaration feature can also be used to ‘convert’ a ‘working document’ with a short-term retention to a longer-term retention (via a multi-stage retention policy – i.e., keep for x years after declared a record)

How should you structure the way information is stored?

An important thing to remember is that Document ID prefixes are set at the Site Collection level (top site in a collaboration team site environment). So, when you are creating your collaboration site environment, I suggest two things:

• create a ‘mud map’ of the proposed sites (and how the document ID prefix might ‘flow down’ to all sites in the collection, and
• structure your team sites according to a logical organisational structure, no deeper than 2 sites below the Site collection. E.g.:

Site collection (e.g., Division)
> Site (e.g., Section)
> > Sub Site (e.g., Business Unit).

Most documents are likely to be stored at the lowest level of the Site Collection (but some may be stored at levels above).

So, in a sub-site, you will find document libraries which should be a logical business activity that is undertaken. You can ‘group’ libraries, if required, using a group name that makes sense to users.

Document libraries may contain documents, or may contain document sets (which are roughly equivalent to paper files). It’s a good idea to disable the use of folders completely as they can make life very difficult for users to navigate and find information (not to mention potentially exceeding the limit for addresses).

Libraries used to store records should use and have customised document Content Types (created and published from the Content Type Hub) enabled, and the default document Content Type disabled.

Metadata on the Content Type can then used to help ‘view’ the information; you can have hierarchical metadata sets in the Term Store that can (and probably should) be applied to Content Types. You can then enable a view of the metadata, which enables the end user to navigate as they would on a drive, using this metadata.

Note that libraries in the Records Centre can use folders that are based on the File Plan, and those folders will have IM policies overriding Content Type IM policies when the documents are stored there.

Mapping Content Types to the classification and retention schedule

A senior Microsoft solution architect suggested to me in early 2012 that organisations should aim to have fewer rather than more content types, even as few as 30. Microsoft recommend trying to minimise your document Content Types to less than 30 if possible. If you have a lot of retention classes, you can use metadata applied to the Content Type to help route documents to the correct location in the File Plan/Retention Schedule based folder structure of the Records Centre.

For example, the IM Policies for these Content Types would be transferred, after 3 years, to the Records Centre; the Content Organiser uses the metadata to establish where to put the document in the the correct location in the folder-based document library.

Be careful how you configure your Content Types, and use the multi-stage retention to minimise the need to have hundreds of Content Types. And, remember that the metadata you use can also be used to display documents in a view.

Key messages

The key points to a successful implementation are:

• extensive communication and collaboration with all stakeholders
• plan your architecture carefully and have an approved ‘reference’ document (and stick with it!)
• iterative implementation (which is not how you would normally roll out an EDRMS, for example).

How to sell it to end users

One word – carefully. I use the Facebook analogy regularly – 1 billion people use the product and NONE of them needed training (as far as I know).

Sell the benefits, not the system, and if it’s logical and intuitive (and gives value), users will just start to use it. Make it complicated or an inconsistent user experience, tell them they have to be trained, or make it hard to find information (e.g., via folders) and they will just get turned off.

Not referring to it as an RM system in any way helps. This is a business solution that helps them work more efficiently. Most users are happy to know that RM is happening in the background.

What records managers need to know about Sharepoint 2010

The following is a revised version of an article that appeared in the July 2011 edition of Informaa Quarterly, IQ, the quarterly journal of the Records and Information Management Professionals Australasia (RIMPA).

Sharepoint 2010, is starting to be used by more and more organisations. And it is being used to create and manage records.

The following are the things that records managers need to know about this product.

Check the version

First, Sharepoint 2010 is not Sharepoint 2007. Sharepoint 2010 has recordkeeping functionality built in that its predecessor lacked.

If someone is talking about a Sharepoint implementation, records managers need to know which version – 2007 or 2010.

Know what it does – and doesn’t. 

Microsoft has spent a lot of effort building recordkeeping functionality into Sharepoint 2010.

Records managers need to understand this functionality and how it works, and why they need to be part of the Sharepoint 2010 implementation process at the beginning of the project. They need to know and understand what it can do, and what it can’t.

For example, it doesn’t manage physical records out of the box. It doesn’t integrate well with Outlook. And it doesn’t do security classifications.*

It is very easy overlook or ignore the recordkeeping functionality in Sharepoint 2010 when it is being implemented. The product looks and behaves very much like Sharepoint 2007. So much so, it would be (and probably has been) easy for IT to implement it without any reference at all to records managers.

The first time records managers might find out about Sharepoint is when they find the intranet has been built using it and they are given a team site used to manage records.

The bits that matter

The following is a quick summary of the key recordkeeping elements Sharepoint 2010.

Sites

Sites and team sites in Sharepoint are more or less the same concept as a web site, grouped within an overall Site Collection.

Just like a web site, Sharepoint 2010 sites and team sites are locations accessed via an URL. But, unless access to the site or team site is open to everyone (such as the organisation’s intranet), access will be controlled (often by a site administrator) through a series of permissions. Records managers need to know and understand what this means for the management of records.

Sites may be made up of a range of different content types (see below) used by end users for all sorts of records and ‘non-records’.

One of the main reasons given by IT not to involve records managers in a Sharepoint 2010 implementation is likely to be that ‘there are no records stored on the sites’. This is a bit like saying that there are no records stored in network shares or in Outlook folders.

Of course, it depends on what industry sector you work in and the litigation risks associated with that sector and/or the records created by the organisation.

Without good planning and the involvement of records managers, users may create records on sites without any recordkeeping functionality at all.

Records managers may not know anything about these sites until well after they have been created and content added. By this time, the cat is out of the bag and it could be quite difficult (but not impossible) to retrofit the recordkeeping functionality that could have been applied at the beginning.

Lesson for records managers: Learn about the recordkeeping functionality of Sharepoint 2010 and get involved early with any implementation project.

Managed Metadata

Good metadata should be second nature to records managers. We know that metadata is a good way to describe records that will allow the organisation to find and make better use of records, including disposition decisions.

Sharepoint 2010 is a dream product for metadata lovers. The opportunities to apply metadata to – and in – records are almost limitless.

The close integration between Sharepoint 2010 and Office 2010 allows recordkeeping metadata to be be embedded in documents and then re-used on sites in metadata columns.

A key feature in Sharepoint 2010 is the Managed Metadata Service (MMS). The MMS allows the creation of multiple hierarchical descriptions of information that can then be applied to records across sites.

End users can also tag records with their own metadata which is then surfaced in the MMS, allowing it to be added to the rest of the organisation’s metadata as required. This is particularly useful when a new subject appears in the organisation.

In some ways, the MMS resembles folder names in Outlook or network shares. It is the way end users see their world.

Lesson for records managers: Understand how end users describe their world and how Sharepoint 2010 can capture that information. Recordkeeping metadata sometimes doesn’t make sense to end users.

Content Types

Content types are the main recordkeeping ‘currency’ in Sharepoint 2010.

Sites consist almost entirely of content types, including when a user clicks on ‘New – (Content Type)’ in a Library or List, or creates content in the form of a wiki or event.

Records managers who are familiar with TRIM record types will see a certain similarity with Sharepoint 2010 content types. But the resemblance is superficial and it would be a mistake to try to make content types behave like TRIM record types.

TRIM record types are generally few in number and mostly consist of documents, files (containers) and boxes. Retention information is generally applied at the file (container) level, and documents inherit that information. Documents are generally single, digital objects of some sort.

In the Sharepoint 2010 model, content can be literally anything. One way of thinking about content types is that each one can ‘map’ to a disposal class in a retention schedule.

Some content types can (and should) be created by customising the top level content type. This is because each content type can then have its own specific metadata and retention information.

Other content types, such as wikis and calendars, can be left ‘as is’ as they are likely to be managed and retained the same way across all sites. For example, calendars could all have a retention period of 2 years after last entry.

Records managers need to understand that *every* content type can be assigned two key elements. These are: (a)a range of recordkeeping metadata that is mostly invisible to end users, but can include end-user defined metadata, and, (b) Information management (IM) policies.

IM policies, and in particular retention policies, are one of the *key* differences between Sharepoint 2010 and 2007. This functionality is not overly obvious if you don’t know where it is or what it does.  It should not be confused with ‘expiry dates’.

The importance of getting IM policies right becomes more obvious when they are not configured correctly. When this happens, (such as by configuring the policies on the parent, generic content type), it can become impossible to do anything with them, and all records using the same content type could have either no recordkeeping rules applied, or it will be same regardless of the type of record it is.

This may not be a bad thing for generic types of records, such as calendars, but could be disastrous if every single document saved into the site collection has the same retention policy.

A *key* information management policy element is Retention. This check-box functionality is easily overlooked but it is a critical piece of recordkeeping as it defines how long the record will be kept. As noted already, each content type can (and perhaps should be) mapped to a class in a retention schedule (or to Normal Administrative Practice (NAP), for Australian government agencies).

Retention policies allow for multi-stage disposition changes to records. They also allow records either to be disposed of on a site after a given period, or sent to the Records Centre for review and/or disposal or further retention.

Sharepoint 2010 includes the US-style concept of ‘declaring’ something as a record. While this functionality is alien to most Australian records managers, it has potential use for allowing changes to the status of a record.

Records stored on network shares or in email folders provide a useful analogy. If the organisation has an EDRMS, end users must transfer the record from the drive or folder to the EDRMS. In most cases this process will leave the document or email in its original location.

The records declaration function in Sharepoint 2010 would allow for a document that is created or saved to a site as a ‘non-record’ (eg, a ‘working document’) to be ‘declared’ as a record on the site, resulting in a change to retention requirements for the same record, rather than expecting an end user to save it to a different system.

Information management policies also include the ‘auditing’ function, allowing auditing on a range of activities.

Lesson for records managers: Learn about Content Types and understand how Information Management policies work and can be configured.

Where are the containers?

Potentially one of the most confusing things about Sharepoint 2010 for records managers seeing the product for the first time is the apparent absence of containers.

Collections of document-type records can be stored in libraries. Libraries may contain folders (another content type) defined by users, or document sets which resemble containers.

It is important to keep an open mind about the concept of a container when learning about Sharepoint 2010.

The product offers multiple ways to organise records in a given context, including through libraries, folders and document sets, but also (and perhaps most importantly) through metadata.

The way end users organise their records is likely to depend on their (or their business unit’s) context. Different users, and records managers, may see the world in different ways and organise information accordingly.

Sharepoint 2010 provides the flexibility to meet these sometimes competing interests, including by allowing end users to organise information that makes sense to them, and then send it to the Records Centre at or after a given time, to be organised in a way that makes sense to records managers.

Lesson for records managers: Look outside the square at how records might be organised. Pre-defined containers are not necessarily the only option.

The Content Organiser

As noted above, information management policies in content types in Sharepoint 2010 can be used to define retention rules for all types of content.

In the same way that rules that can be applied in email applications to route emails to specific folders, the content organiser in Sharepoint 2010 can be used to route any content type to a specific, pre-defined location in the records centre.

This is done by using the feature to send the content type to a different location such as the records centre after a given period of time.

Lesson for records managers: Learn how content types work. If you are involved at the beginning of a Sharepoint 2010 implementation you have a better chance of setting up and configuring content types correctly so they will eventually be managed in the records centre. This can include the ability for records managers to review new, undefined content that is sent to a ‘drop off’ area in the Records Centre.

The Records Centre

The Records Centre in Sharepoint 2010 is, in effect, a separate site in the site collection (or farm) to store corporate records. An organisation may have more than one records centre.

The centre should be owned and managed by records managers. There are various ways to configure a records centre.

One option would be to create a library (or libraries) for records, each of which contains a pre-defined folder structure mapped to the organisation’s Business Classification Scheme (BCS).

As these folders can have IM policies applied to them, they can be mapped to specific classes in the organisation’s retention schedule. In this way, they start to look very much like containers!

As Sharepoint 2010 allows for retention rules to be applied to either libraries and folder OR content types, each of these folders can have retention rules applied to them.

Lesson for records managers: Own the records centre!