Alternatives to folders in SharePoint 2010

Introduction

The content of this post is my take on an issue that has been discussed before.  The question is really to (a) use folders and deal with the potential confusion and difficulty of using them, or (b) use metadata and deal with the potential confusion from users who don’t ‘get’ it.

Personally I like the idea of metadata instead of folders, but (a) using metadata that is similar to what users use already to describe their folders, and (b) making use of site hierarchies, document libraries, and documents to replicate 80% of what users want.

Two great blog articles on this subject are:

http://www.glynblogs.com/2010/10/folders-in-sharepoint-are-not-evil.html

and

https://www.nothingbutsharepoint.com/sites/eusp/Pages/sharepoint-convert-folder-structures-to-metadata.aspx

The problem: Folders are deeply embedded in user culture

Folders to store digital objects like documents and photographs (and emails) have existed since computers were introduced. Folders are technically known as directories in most operating systems.

Given the history of folders on drives, users have a natural inclination to want to have something similar when they are first exposed to SharePoint. However, Windows folders and SharePoint folders are not the same thing.

To quote one Microsoft blogger, folders are ‘a byproduct of a file-based metaphor that users are familiar with’ (Pisaraek, Michael. 21 April 2011. ‘Document Sets vs Folders in SharePoint 2010’. http://sharepoint.microsoft.com/Blogs/GetThePoint/Lists/Posts/Post.aspx?ID=462)

What are Windows Folders, then?

A Windows NTFS file system stores information about the directory, not about the files stored within the directory. Information about each directory is recorded as an entry in the Master File Table (MFT), which is the main repository of information for the directory or file. Each directory and file entry in the MFT includes specific attributes, including things like date/time stamps, name and security permissions.

(See http://www.pcguide.com/ref/hdd/file/ntfs/filesDir-c.html for more information).

What is different about a SharePoint Folder?

A SharePoint folder is not the same thing as a Windows folder, despite its similar appearance. The following neatly explains how SharePoint stores information:

‘SharePoint is physically implemented as an ISAPI filter in IIS. It intercepts all HTTP requests to the web site and maps them to a SharePoint virtual site. All of the myriad sites and sub-sites in a SharePoint implementation do not exist in the file system of the server. Everything, including the files in a document library, is stored in a Microsoft SQL Server database. The SharePoint ISAPI filter interprets the URL path, then creates and returns the correct pages to the web browser.’

(Source: Felknor, Chris. 2006. ‘SharePoint PREP Technical Documentation’, MIT iCampus. Page 5)

SharePoint stores information in sites in ‘Lists’, usually known as Libraries when used to store documents. Document Libraries are a type of ‘folder’ construct that store binary objects (usually documents). These Libraries may contain a range of metadata and controls that can be inherited by objects stored within them.

By default, document libraries have folder creation enabled. Folders are a visual representation allowing information to be grouped together. However, unlike Windows Explorer, SharePoint does not display a hierarchical representation of the folder structure by default, making it difficult to navigate and maintain.  A common user reaction is ‘where is the +?’

Folders in document libraries are not recommended and should be disabled for end users.

So what are the alternatives?

The three main alternatives to folders are:

• Site collection/site/sub-site hierarchy
• Document sets
• Metadata

Generally, users create or seek folder structures in network drives to create a hierarchical view of their information. Commonly, the higher level of folders in drives are based on the organisational hierarchy, while lower levels are a combination of business and user ways of defining or classifying information.

Site Collections, sites, and sub-sites used for collaboration purposes (based on Team Sites) can and probably should mimic the organisational hierarchy and be no more than three sites ‘deep’. Sub-sites (or the lowest level sites, usually for a team of less than 50 people, or even less than 20) should have document libraries that ‘map’ to the business activities the team undertakes. Documents may be stored in these sites.

If there is a requirement to break down a document libraries into more categories, Document Sets are a way to aggregation information and in many ways replicate the paradigm of a physical file with a range of pre-defined metadata properties that can be inherited by the objects stored within it.

This combination of sites/sub-sites/document libraries and document sets will generally meet the requirements of most business users to store aggregations of information that were previously stored in network folders.

From a recordkeeping perspective, the most common use cases for using document sets will be for where there is a requirement to maintain a relatively ‘static’ or unchanging collection or set of documents about a given subject, where the collection (or ‘aggregation’) has a specific retention/disposal class or policy. Policy documents, or documents about a specific project, would be examples.

Metadata instead of folders

Metadata in SharePoint provides a very powerful alternative to hierarchical categorisation described above, particularly where there is less of a requirement to maintain a static collection.

The downside is that users don’t understand, or take a long time to understand, this paradigm.

Metadata can be created for a site collection or site within a collection and applied to Content Types, or it can be set centrally in the Term Store and applied to Content Types stored in the Content Type Hub, which are then made available in document libraries.

In either case, users can then apply metadata to the objects; this metadata can then be used to sort and categorise the objects in any way that is permitted. Additionally, a view of the hierarchical metadata ‘set’ can be displayed in the left hand side navigation for the site, mimicing (in a way) the hierarchical view previously achieved by using network folders.

Two examples:

• A common scenario on network drives is to store records of meetings or committees by year and month, with folders used to separate ‘agendas/related documents’ and ‘minutes’. Using a combination of system-generated and user-defined metadata, all the information can be stored in a single document library; customised views can then be created to display related information by year, month, and by document type (and any other metadata that has been applied).
• An organisation may receive and need to manage thousands of documents per year for a service they licence or run. There may only be a few documents, per year, for each licence. Instead of trying to categorise each licence into document libraries or document sets by licence number (for example), the organisation can include the licence number in the metadata for the document Content Type and store all documents for a single year in a single document library. By using search or views, all documents with the given licence number will be displayed.

In these two examples, the retention/disposal policy would be placed on the individual document Content Type.

Organisations thinking of using SharePoint 2010 to manage documents as records need to understand carefully the implications of using folders in document libraries. As noted earlier, there is a natural (and not surprising) inclination to user ‘folders’; they are available by default. However, there is more negative than positive in using and maintaining them and they should be disabled.

But folders aren’t completely bad (or evil!)

One effective use of folders is in the Records Centre, a type of long-term, inactive document archive. In the Records Centre, folders can be mapped to the organisation’s file plan or business classification scheme. Information Management Policies in the Records Centre document libraries can be changed to ‘Libraries and folders’ which overrides IM policies set on Content Types.

Implementing SharePoint 2010 as a Document and Records Management System

There are several things that are worth asking or considering as a records manager (or equivalent) in a company that is using, or plans to use, SharePoint 2010. This can significantly affect how things occur.

Governance

If your organisation uses SharePoint there is a good chance you will have a qualified SharePoint Administrator in the IT department. This is usually the person (or persons) who ‘own’ the entire SharePoint farm. They need to be your best friends, otherwise you’ll never see all the things you need to see and know about.

You should also consider organising regular meetings with the SharePoint Administrator, Web Admin person and anyone else you think needs to be part of this process.

Existing/legacy SharePoint environment

SharePoint has been around for a decade. If you have a legacy environment, you need to consider either how to migrate or what to do with the legacy environment, because they work differently from an RM point of view.

Also, many IT people haven’t yet worked out how to implement RM in SharePoint 2010 and tend to approach its implementation or configuration with SharePoint 2007 in mind.

Training and knowledge

Records managers need to have a really good understanding of SharePoint administration, at least for Site Collections. You will almost certainly not get access to Central Admin (the domain of the SP Administrator). But you need to know what happens where, and what are the implications of your decisions.

SharePoint domains

You need to understand the relationship and differences between the (a) publishing, intranet/internet environments, (b) the collaboration, DRM, team site environments (where most of your focus will be), and (c) application sites. Generally speaking, (a) belongs to the Web team, and they know how to differentiate between published documents and records. (b) is likely to be mostly uncustomised, out of the box. (c) is likely to be a customised and ‘localised’ solution for specific types of documents.

SQL

It’s worth having some understanding of how SharePoint stores and manages documents (as Base64 encoded data) in SQL tables. It will put you in good stead with a Database Administrator (DBA).

What happens to network shares?

It depends. Shared drives may continue to be used where users need to store working versions. Consider locking them down or migrating the content and then deleting them from the drive.

E-mails

See my post on Microsoft’s Messaging Records Management (MRM) functionality built into Exchange Server 2010. While best practice RM is to store all records together in context, in practice this is not easy with email unless you have a third party tool. There are several on the market including OnePlaceMail (an Australian company).

You can ’email-enable’ document libraries (so the emails go there instead of the email client) but this would only really work for specific types of business process. The real problem are the emails that users receive and don’t do anything with. This is where, possibly, MRM can help. Not an easy answer here because, even if you buy a third party tool, the onus will still be on the end users to do the right thing.

Records declaration

Unlike in the US, which uses the concept of declaring something a record (a feature including with SharePoint 2010), in the Australian legal environment any business related document (which has a broad definition) can be a record of a business process, regardless of whether it was stored in a recordkeeping system or subject to some other RK controls. There is no case law here that focusses on whether a record is a record because it was stored in a recordkeeping system. Instead, there are other factors which mostly relate to the admissibility of something as evidence.

Of course, records are ‘better’ (in terms of integrity, reliability and authenticity) if they are stored in a recordkeeping system that has unique IDs, metadata, audit trails etc. These features are not critical to admissibility as evidence (in the Australian legal context) however they may help.

It might be more useful to take a practical approach and say, let’s not try to distinguish between them, let’s assume anything can be considered a record of a business process or activity (or the smoking gun of something nefarious!). Users generally don’t care unless they are required to keep specific records for compliance reasons.

In-place records management vs the Records Centre – which is best?

It depends. One approach is to leave records with a retention period of less than 5 years on site (controlled via the Content Type IM Policies). At the end of the retention period, these records will be reviewed via a Retention Workflow built into the Content Type IM policies.

Anything with a longer retention period would, after (say) 3 years, be transferred to the Records Centre for longer term storage.

Note that enabling the Records Declaration option in the Site Collection is not essential to in-place records management. You can set it up without having to declare something a record. That said, you might want to consider using the feature to differentiate between records and non-records.

Also, the records declaration feature can also be used to ‘convert’ a ‘working document’ with a short-term retention to a longer-term retention (via a multi-stage retention policy – i.e., keep for x years after declared a record)

How should you structure the way information is stored?

An important thing to remember is that Document ID prefixes are set at the Site Collection level (top site in a collaboration team site environment). So, when you are creating your collaboration site environment, I suggest two things:

• create a ‘mud map’ of the proposed sites (and how the document ID prefix might ‘flow down’ to all sites in the collection, and
• structure your team sites according to a logical organisational structure, no deeper than 2 sites below the Site collection. E.g.:

Site collection (e.g., Division)
> Site (e.g., Section)
> > Sub Site (e.g., Business Unit).

Most documents are likely to be stored at the lowest level of the Site Collection (but some may be stored at levels above).

So, in a sub-site, you will find document libraries which should be a logical business activity that is undertaken. You can ‘group’ libraries, if required, using a group name that makes sense to users.

Document libraries may contain documents, or may contain document sets (which are roughly equivalent to paper files). It’s a good idea to disable the use of folders completely as they can make life very difficult for users to navigate and find information (not to mention potentially exceeding the limit for addresses).

Libraries used to store records should use and have customised document Content Types (created and published from the Content Type Hub) enabled, and the default document Content Type disabled.

Metadata on the Content Type can then used to help ‘view’ the information; you can have hierarchical metadata sets in the Term Store that can (and probably should) be applied to Content Types. You can then enable a view of the metadata, which enables the end user to navigate as they would on a drive, using this metadata.

Note that libraries in the Records Centre can use folders that are based on the File Plan, and those folders will have IM policies overriding Content Type IM policies when the documents are stored there.

Mapping Content Types to the classification and retention schedule

A senior Microsoft solution architect suggested to me in early 2012 that organisations should aim to have fewer rather than more content types, even as few as 30. Microsoft recommend trying to minimise your document Content Types to less than 30 if possible. If you have a lot of retention classes, you can use metadata applied to the Content Type to help route documents to the correct location in the File Plan/Retention Schedule based folder structure of the Records Centre.

For example, the IM Policies for these Content Types would be transferred, after 3 years, to the Records Centre; the Content Organiser uses the metadata to establish where to put the document in the the correct location in the folder-based document library.

Be careful how you configure your Content Types, and use the multi-stage retention to minimise the need to have hundreds of Content Types. And, remember that the metadata you use can also be used to display documents in a view.

Key messages

The key points to a successful implementation are:

• extensive communication and collaboration with all stakeholders
• plan your architecture carefully and have an approved ‘reference’ document (and stick with it!)
• iterative implementation (which is not how you would normally roll out an EDRMS, for example).

How to sell it to end users

One word – carefully. I use the Facebook analogy regularly – 1 billion people use the product and NONE of them needed training (as far as I know).

Sell the benefits, not the system, and if it’s logical and intuitive (and gives value), users will just start to use it. Make it complicated or an inconsistent user experience, tell them they have to be trained, or make it hard to find information (e.g., via folders) and they will just get turned off.

Not referring to it as an RM system in any way helps. This is a business solution that helps them work more efficiently. Most users are happy to know that RM is happening in the background.

Applying recordkeeping policies to email – Microsoft Messaging Records Management (MRM)

The problem

The problem of managing emails as records is summed up in the following statements:

“Many organizations have yet to define an email retention policy. More than one‐quarter of organizations have not yet established any sort of email retention policy despite the fact that there are a growing body of statutory requirements and legal obligations to preserve business records, including those stored in email. Among the nearly three‐quarters of organizations that have established an email retention policy, only two‐thirds of these organizations indicate that their users are fully aware of the policy.” Michael Osterman, “Messaging Archiving and Document Management Markets Trends, 2009-20112”, dated May 2009.

‘Over 40 years after the invention of email, relatively few institutions have developed policies, implementation strategies, procedures, tools and services that support the longterm preservation of records generated via this transformative communication mechanism.’  Christopher J Prom, ‘Preserving Email’, DPC Technology Watch Report 11-01 Decemer 2011. www.dpconline.org/component/docman/doc_download/739-dpctw11-01.pdf

Storing business records in context

Traditional records management theory recommends that there should be a clear relationship between records about a particular subject or issue, regardless of format, and the business context that originated it. (AS ISO 15489-2002: 9.3 Records Capture)

In the paper world, this was achieved by the co-location of related records in a physical file.

In the electronic world, this is usually achieved through the application of metadata. Business classification and naming systems applied to electronic folders generally achieve this; as well, electronic systems also allow for a range of cross-subject metadata that allows records to be organised in different contexts.

Additional, business context-specific metadata can be applied to emails (including from integrated business applications – for example, an email saved to TRIM will show the TRIM record number in its email metadata properties).  However, this ability (as with Properties in Office documents) is rarely enabled or used.

Instead, and as with Office documents, we tend to let users ‘categorise’ their emails (and documents on network shares) through folders – although not all users do this.  (Interestingly, online email systems like Google’s gmail use tags instead of folders).

Are emails documents?

The short answer is yes (in the Australian legal evidence context), but they are documents that, in a way like xml-based Office documents like docx, are made up of structured data that displays as a single ‘document’.

Part of the problem with emails as records is the perception (on the part of users who have never had to face court) that they are not documents, but messages.  The ability to use the system to send or receive ‘private’ messages exacerbates this perception.

The problem of storing emails as records

Emails have been a constant problem and challenge for records managers and recordkeeping since they first appeared in the early 1990s.

The three main approaches to keeping emails have been to (a) print to paper, (b) save to a recordkeeping system, and (c) save to a drive.

Print to paper, while relatively common in many organisations even now, is probably the poorest (and some might say ‘silliest’) option in the digital world as (a) it is dependent on users, (b) emails usually lose their message headers, (c) emails are unsearchable in their electronic form, (d) emails remain on the Exchange system and are discoverable.

Saving emails to a recordkeeping system, while better than printing, is an inadequate option because (a) it is usually dependent on users to do it, (b) the email still remains in the Exchange system, and (c) it can sometimes result in the email being saved in a different format that is not necessarily suitable for long-term preservation (e.g., TRIM’s .vmbx).  There is also the problem of users saving ‘dumb’ emails with (valuable) attachments, which can make the attachment harder to find, identify or access.  Some systems (such as SharePoint 2010) include email-enabled storage locations.

Chris Prom, in a blog posting titled ‘Practical E-Records ‘Facilitating the Generation of Archives in the Facebook Age’, notes that:

‘…the formal recordkeeping systems previously used by many organizations for electronic records have died or have one foot firmly in the grave.  At the same time, the habits that individuals use in producing, consuming, storing, filing, searching, and interpreting records are themselves undergoing constant change.  People adopt new communication technologies at an ever-quickening pace.   Divergent personal practices, rather than the centralized electronic systems, are the harsh reality that confronts our profession’.

Saving to a drive is also a poor option, and is usually based on user preferences to want to ‘keep’ emails.  Emails saved to drives (a) will still remain in the Exchange system, (b) may lose their header information, and (c) are not necessarily saved in appropriate or accessible formats.

In relation to the last point, Outlook does not make it easy for an end user to decide, with usually five options to choose from – which is the right one?  Users will usually choose whatever is the default (.msg), but this isn’t necessarily the best long term option (which is MIME or EML – the latter described by the National Archives of Australia (NAA) as ‘an acceptable open file format for long term storage).

In all cases, keeping these emails in the business context to which they relate has been a constant problem for records managers.  As a consequence, there is a tendency on the part of almost all businesses to leave and manage emails where they are (i.e., in Exchange).

Microsoft Exchange 2010 – Messaging Records Management

To try to address this problem, Microsoft introduced ‘Mailbox Manager Policies’ in Exchange Server 2003.

This was followed by ‘Message Records Management’ with Managed Folders in Exchange Server 2007 (a feature that remains in Exchange 2010).

Exchange Server 2010 includes a new model of managing emails as records, called ‘Messaging Records Management’.  Microsoft describe it as follows:

‘Messaging records management (MRM) is the records management technology in Microsoft Exchange Server 2010 that helps organizations reduce the legal risks associated with e-mail. MRM makes it easier to keep the messages needed to comply with company policy, government regulations, or legal needs, and to remove content that has no legal or business value. This is accomplished through the use of retention policies or managed folders’. (Source: http://technet.microsoft.com/en-us/library/dd335093)

As Microsoft notes, however (on the same page), MRM does not prevent users from deleting messaging; it is really only designed to remove them at the end of a given period.  Microsoft recommend ‘journaling’ emails where there are specific business reasons to keep them for longer (such as legal proceedings or the need to ensure specific email is kept), or applying the Legal Holds functionality.

The key elements of MRM are Retention Policy Tags (RPTs) and Retention Policies.

There are three types of Retention Tags: (1) Default Policy Tags (DPT), (2) Retention Policy Tags, and (3) Personal  Tags (which are an ‘opt-in’ on the email client).

• Retention Policy Tags (RPTs) are used on default folders (e.g., inbox, junk mail, sent, deleted). Users cannot change the RPT but can override it with a Personal Tag.
• Default Policy Tags can be applied by users to untagged items.  A Retention Policy can contain only one default policy tag.
• Personal Tags can be applied by users to their own custom folders or individual emails.

In most cases, users make the decision, and the retention applies on where the email is located.  If there is actual or anticipated litigation, a Retention Hold can be applied to the user’s mailbox; however, this does not prevent users deleting emails, it only overrides any retention policies.  The Legal Hold option should be applied to prevent deletion.  Once this option is applied, Legal Hold ‘captures any deleted or edited items into a special folder that’s neither accessible nor changeable by the user’.

All retention tags include: a Tag Name, a Tag Type, an age limit (in days) with an action to take, and comments.

The actions available are:

• Delete And Allow Recovery – This action will perform a hard delete, sending the message to the dumpster. The user will be able to recover the item using the Recover Deleted Items dialog box in Outlook 2010 or Outlook Web App.
• Mark As Past Retention Limit – This action will mark an item as past the retention limit, displaying the message using strikethrough text in Outlook 2007, 2010 or Outlook Web App.
• Move To Archive – This action moves the message to the users archive mailbox.(see below)
• Move To Deleted Items – This action will move the message to the Deleted Items folder.
• Permanently Delete – This action will permanently delete the message and cannot be restored using the Recover Deleted Items dialog box.

Once the tags are created, they can be added to a Retention Policy and this policy, in turn, is then applied to specific mailboxes – one policy per mailbox.

The ‘auto-tagging’ feature, once 500 items have been tagged, will automatically tag items in a user’s mailox based on their past tagging activities.

So, is MRM the answer to managing emails as records? 

Yes and no.  From a recordkeeping perspective, MRM:

• Does nothing to ensure that records are kept in the business activity or functional context to which they relate, unless (of course) the emails are the only form of record that exists for the business activity.
• Does not stop users from deleting emails.

On a positive note, MRM:

• Attempts to address the problem of email retention.
• Allows the application of a retention policy to emails that might be stored in a business context Outlook mailbox or fold As well, Exchange features like Legal Hold and Journaling allow further controls to be implemented.

Archiving

Exchange 2010 now includes a ‘personal archives feature’, which allows users to save emails to their own archive instead of saving emails to drives or using Personal Storage (.pst). A good article on this subject can be found at this location: http://mohamedridha.com/2011/11/07/exchange-2010-online-archiving-and-retention-tagspolicies-a-practical-example/

Sources (all retrieved 1 June 2012)

• http://www.ostermanresearch.com/execsum/or_arch2009execsum.pdf
• e-records.chrisprom.com
• http://www.dpconline.org/component/docman/doc_download/739-dpctw11-01.pdf
• technet.microsoft.com/en-us/library/dd335093
• blogs.technet.com/b/ianhamer/archive/2009/12/04/curious-about-exchange-2010-retention-policies.aspx
• sysadmin-talk.org/2010/04/manage-exchange-retention-policies-before-they-manage-you/
• techdom.nl/microsoft/exchange-2010-sp1-archiving-part-4-creating-retention-policy-tags-default-folders/