Alternatives to folders in SharePoint 2010

Introduction

The content of this post is my take on an issue that has been discussed before.  The question is really to (a) use folders and deal with the potential confusion and difficulty of using them, or (b) use metadata and deal with the potential confusion from users who don’t ‘get’ it.

Personally I like the idea of metadata instead of folders, but (a) using metadata that is similar to what users use already to describe their folders, and (b) making use of site hierarchies, document libraries, and documents to replicate 80% of what users want.

Two great blog articles on this subject are:

http://www.glynblogs.com/2010/10/folders-in-sharepoint-are-not-evil.html

and

https://www.nothingbutsharepoint.com/sites/eusp/Pages/sharepoint-convert-folder-structures-to-metadata.aspx

The problem: Folders are deeply embedded in user culture

Folders to store digital objects like documents and photographs (and emails) have existed since computers were introduced. Folders are technically known as directories in most operating systems.

Given the history of folders on drives, users have a natural inclination to want to have something similar when they are first exposed to SharePoint. However, Windows folders and SharePoint folders are not the same thing.

To quote one Microsoft blogger, folders are ‘a byproduct of a file-based metaphor that users are familiar with’ (Pisaraek, Michael. 21 April 2011. ‘Document Sets vs Folders in SharePoint 2010’. http://sharepoint.microsoft.com/Blogs/GetThePoint/Lists/Posts/Post.aspx?ID=462)

What are Windows Folders, then?

A Windows NTFS file system stores information about the directory, not about the files stored within the directory. Information about each directory is recorded as an entry in the Master File Table (MFT), which is the main repository of information for the directory or file. Each directory and file entry in the MFT includes specific attributes, including things like date/time stamps, name and security permissions.

(See http://www.pcguide.com/ref/hdd/file/ntfs/filesDir-c.html for more information).

What is different about a SharePoint Folder?

A SharePoint folder is not the same thing as a Windows folder, despite its similar appearance. The following neatly explains how SharePoint stores information:

‘SharePoint is physically implemented as an ISAPI filter in IIS. It intercepts all HTTP requests to the web site and maps them to a SharePoint virtual site. All of the myriad sites and sub-sites in a SharePoint implementation do not exist in the file system of the server. Everything, including the files in a document library, is stored in a Microsoft SQL Server database. The SharePoint ISAPI filter interprets the URL path, then creates and returns the correct pages to the web browser.’

(Source: Felknor, Chris. 2006. ‘SharePoint PREP Technical Documentation’, MIT iCampus. Page 5)

SharePoint stores information in sites in ‘Lists’, usually known as Libraries when used to store documents. Document Libraries are a type of ‘folder’ construct that store binary objects (usually documents). These Libraries may contain a range of metadata and controls that can be inherited by objects stored within them.

By default, document libraries have folder creation enabled. Folders are a visual representation allowing information to be grouped together. However, unlike Windows Explorer, SharePoint does not display a hierarchical representation of the folder structure by default, making it difficult to navigate and maintain.  A common user reaction is ‘where is the +?’

Folders in document libraries are not recommended and should be disabled for end users.

So what are the alternatives?

The three main alternatives to folders are:

• Site collection/site/sub-site hierarchy
• Document sets
• Metadata

Generally, users create or seek folder structures in network drives to create a hierarchical view of their information. Commonly, the higher level of folders in drives are based on the organisational hierarchy, while lower levels are a combination of business and user ways of defining or classifying information.

Site Collections, sites, and sub-sites used for collaboration purposes (based on Team Sites) can and probably should mimic the organisational hierarchy and be no more than three sites ‘deep’. Sub-sites (or the lowest level sites, usually for a team of less than 50 people, or even less than 20) should have document libraries that ‘map’ to the business activities the team undertakes. Documents may be stored in these sites.

If there is a requirement to break down a document libraries into more categories, Document Sets are a way to aggregation information and in many ways replicate the paradigm of a physical file with a range of pre-defined metadata properties that can be inherited by the objects stored within it.

This combination of sites/sub-sites/document libraries and document sets will generally meet the requirements of most business users to store aggregations of information that were previously stored in network folders.

From a recordkeeping perspective, the most common use cases for using document sets will be for where there is a requirement to maintain a relatively ‘static’ or unchanging collection or set of documents about a given subject, where the collection (or ‘aggregation’) has a specific retention/disposal class or policy. Policy documents, or documents about a specific project, would be examples.

Metadata instead of folders

Metadata in SharePoint provides a very powerful alternative to hierarchical categorisation described above, particularly where there is less of a requirement to maintain a static collection.

The downside is that users don’t understand, or take a long time to understand, this paradigm.

Metadata can be created for a site collection or site within a collection and applied to Content Types, or it can be set centrally in the Term Store and applied to Content Types stored in the Content Type Hub, which are then made available in document libraries.

In either case, users can then apply metadata to the objects; this metadata can then be used to sort and categorise the objects in any way that is permitted. Additionally, a view of the hierarchical metadata ‘set’ can be displayed in the left hand side navigation for the site, mimicing (in a way) the hierarchical view previously achieved by using network folders.

Two examples:

• A common scenario on network drives is to store records of meetings or committees by year and month, with folders used to separate ‘agendas/related documents’ and ‘minutes’. Using a combination of system-generated and user-defined metadata, all the information can be stored in a single document library; customised views can then be created to display related information by year, month, and by document type (and any other metadata that has been applied).
• An organisation may receive and need to manage thousands of documents per year for a service they licence or run. There may only be a few documents, per year, for each licence. Instead of trying to categorise each licence into document libraries or document sets by licence number (for example), the organisation can include the licence number in the metadata for the document Content Type and store all documents for a single year in a single document library. By using search or views, all documents with the given licence number will be displayed.

In these two examples, the retention/disposal policy would be placed on the individual document Content Type.

Organisations thinking of using SharePoint 2010 to manage documents as records need to understand carefully the implications of using folders in document libraries. As noted earlier, there is a natural (and not surprising) inclination to user ‘folders’; they are available by default. However, there is more negative than positive in using and maintaining them and they should be disabled.

But folders aren’t completely bad (or evil!)

One effective use of folders is in the Records Centre, a type of long-term, inactive document archive. In the Records Centre, folders can be mapped to the organisation’s file plan or business classification scheme. Information Management Policies in the Records Centre document libraries can be changed to ‘Libraries and folders’ which overrides IM policies set on Content Types.

Implementing SharePoint 2010 as a Document and Records Management System

There are several things that are worth asking or considering as a records manager (or equivalent) in a company that is using, or plans to use, SharePoint 2010. This can significantly affect how things occur.

Governance

If your organisation uses SharePoint there is a good chance you will have a qualified SharePoint Administrator in the IT department. This is usually the person (or persons) who ‘own’ the entire SharePoint farm. They need to be your best friends, otherwise you’ll never see all the things you need to see and know about.

You should also consider organising regular meetings with the SharePoint Administrator, Web Admin person and anyone else you think needs to be part of this process.

Existing/legacy SharePoint environment

SharePoint has been around for a decade. If you have a legacy environment, you need to consider either how to migrate or what to do with the legacy environment, because they work differently from an RM point of view.

Also, many IT people haven’t yet worked out how to implement RM in SharePoint 2010 and tend to approach its implementation or configuration with SharePoint 2007 in mind.

Training and knowledge

Records managers need to have a really good understanding of SharePoint administration, at least for Site Collections. You will almost certainly not get access to Central Admin (the domain of the SP Administrator). But you need to know what happens where, and what are the implications of your decisions.

SharePoint domains

You need to understand the relationship and differences between the (a) publishing, intranet/internet environments, (b) the collaboration, DRM, team site environments (where most of your focus will be), and (c) application sites. Generally speaking, (a) belongs to the Web team, and they know how to differentiate between published documents and records. (b) is likely to be mostly uncustomised, out of the box. (c) is likely to be a customised and ‘localised’ solution for specific types of documents.

SQL

It’s worth having some understanding of how SharePoint stores and manages documents (as Base64 encoded data) in SQL tables. It will put you in good stead with a Database Administrator (DBA).

What happens to network shares?

It depends. Shared drives may continue to be used where users need to store working versions. Consider locking them down or migrating the content and then deleting them from the drive.

E-mails

See my post on Microsoft’s Messaging Records Management (MRM) functionality built into Exchange Server 2010. While best practice RM is to store all records together in context, in practice this is not easy with email unless you have a third party tool. There are several on the market including OnePlaceMail (an Australian company).

You can ’email-enable’ document libraries (so the emails go there instead of the email client) but this would only really work for specific types of business process. The real problem are the emails that users receive and don’t do anything with. This is where, possibly, MRM can help. Not an easy answer here because, even if you buy a third party tool, the onus will still be on the end users to do the right thing.

Records declaration

Unlike in the US, which uses the concept of declaring something a record (a feature including with SharePoint 2010), in the Australian legal environment any business related document (which has a broad definition) can be a record of a business process, regardless of whether it was stored in a recordkeeping system or subject to some other RK controls. There is no case law here that focusses on whether a record is a record because it was stored in a recordkeeping system. Instead, there are other factors which mostly relate to the admissibility of something as evidence.

Of course, records are ‘better’ (in terms of integrity, reliability and authenticity) if they are stored in a recordkeeping system that has unique IDs, metadata, audit trails etc. These features are not critical to admissibility as evidence (in the Australian legal context) however they may help.

It might be more useful to take a practical approach and say, let’s not try to distinguish between them, let’s assume anything can be considered a record of a business process or activity (or the smoking gun of something nefarious!). Users generally don’t care unless they are required to keep specific records for compliance reasons.

In-place records management vs the Records Centre – which is best?

It depends. One approach is to leave records with a retention period of less than 5 years on site (controlled via the Content Type IM Policies). At the end of the retention period, these records will be reviewed via a Retention Workflow built into the Content Type IM policies.

Anything with a longer retention period would, after (say) 3 years, be transferred to the Records Centre for longer term storage.

Note that enabling the Records Declaration option in the Site Collection is not essential to in-place records management. You can set it up without having to declare something a record. That said, you might want to consider using the feature to differentiate between records and non-records.

Also, the records declaration feature can also be used to ‘convert’ a ‘working document’ with a short-term retention to a longer-term retention (via a multi-stage retention policy – i.e., keep for x years after declared a record)

How should you structure the way information is stored?

An important thing to remember is that Document ID prefixes are set at the Site Collection level (top site in a collaboration team site environment). So, when you are creating your collaboration site environment, I suggest two things:

• create a ‘mud map’ of the proposed sites (and how the document ID prefix might ‘flow down’ to all sites in the collection, and
• structure your team sites according to a logical organisational structure, no deeper than 2 sites below the Site collection. E.g.:

Site collection (e.g., Division)
> Site (e.g., Section)
> > Sub Site (e.g., Business Unit).

Most documents are likely to be stored at the lowest level of the Site Collection (but some may be stored at levels above).

So, in a sub-site, you will find document libraries which should be a logical business activity that is undertaken. You can ‘group’ libraries, if required, using a group name that makes sense to users.

Document libraries may contain documents, or may contain document sets (which are roughly equivalent to paper files). It’s a good idea to disable the use of folders completely as they can make life very difficult for users to navigate and find information (not to mention potentially exceeding the limit for addresses).

Libraries used to store records should use and have customised document Content Types (created and published from the Content Type Hub) enabled, and the default document Content Type disabled.

Metadata on the Content Type can then used to help ‘view’ the information; you can have hierarchical metadata sets in the Term Store that can (and probably should) be applied to Content Types. You can then enable a view of the metadata, which enables the end user to navigate as they would on a drive, using this metadata.

Note that libraries in the Records Centre can use folders that are based on the File Plan, and those folders will have IM policies overriding Content Type IM policies when the documents are stored there.

Mapping Content Types to the classification and retention schedule

A senior Microsoft solution architect suggested to me in early 2012 that organisations should aim to have fewer rather than more content types, even as few as 30. Microsoft recommend trying to minimise your document Content Types to less than 30 if possible. If you have a lot of retention classes, you can use metadata applied to the Content Type to help route documents to the correct location in the File Plan/Retention Schedule based folder structure of the Records Centre.

For example, the IM Policies for these Content Types would be transferred, after 3 years, to the Records Centre; the Content Organiser uses the metadata to establish where to put the document in the the correct location in the folder-based document library.

Be careful how you configure your Content Types, and use the multi-stage retention to minimise the need to have hundreds of Content Types. And, remember that the metadata you use can also be used to display documents in a view.

Key messages

The key points to a successful implementation are:

• extensive communication and collaboration with all stakeholders
• plan your architecture carefully and have an approved ‘reference’ document (and stick with it!)
• iterative implementation (which is not how you would normally roll out an EDRMS, for example).

How to sell it to end users

One word – carefully. I use the Facebook analogy regularly – 1 billion people use the product and NONE of them needed training (as far as I know).

Sell the benefits, not the system, and if it’s logical and intuitive (and gives value), users will just start to use it. Make it complicated or an inconsistent user experience, tell them they have to be trained, or make it hard to find information (e.g., via folders) and they will just get turned off.

Not referring to it as an RM system in any way helps. This is a business solution that helps them work more efficiently. Most users are happy to know that RM is happening in the background.