Archive for the ‘SharePoint 2013’ Category

Migrating to SharePoint Online – Part 1 (Planning)

August 25, 2018

We implemented SharePoint 2010 in early 2012 and then upgraded to SharePoint 2013 in early 2015. After acquiring Office 365 enterprise licences in April 2016 we began to play for the migration of our existing on-premise environment to SharePoint Online. After testing the migration process with inactive sites, we started to migrate active sites from early 2018. We expect to complete all the migrations by 31 December 2018.

This post, the first of three, outlines the factors that influenced and guided how we approached the migration. Our approach may not be the same as your approach, but many of the basic principles may be similar.

Overview of our SharePoint environment pre-migration

A key principle for our SharePoint environment since 2012 was to avoid customisation and dependencies, and use the product ‘out of the box’ (OOTB) as much as possible.

  • Customisation would almost always require some degree of development and ongoing maintenance. It also meant that upgrades could be more complex and expensive.
  • Dependencies of any sort – be they integration components or third-party add-ons – could also make upgrades more complex and expensive.

Governance model

We also implemented a ‘balanced’ controlled environment, following the technical design models for SharePoint 2010 described by Microsoft (extract in image above), which recommended that organisations strike balance across three key governance elements:

SharePoint2010GovernanceBalance

Source: https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-server-2010/cc303422(v%3doffice.14)

  • IT Governance. Centrally managed or locally managed?
  • Information Management. Tightly managed or loosely managed?
  • Application Management. Strictly managed or loosely managed development?

In our environment, the ability to create new SharePoint sites and sub-sites required the completion of a (SharePoint) online form and was restricted to the SharePoint Administrators. This enabled us to prevent uncontrolled growth in the environment and to ensure that all new sites were created within a pre-defined – but not overly strict – architecture design model.

Upgrade to SharePoint 2013 in early 2015

Our SharePoint site collections were created across five web applications: team (approximately 120 sites), project (approx. 120 sites), publication, apps, and intranet. Most of the corporate records were stored in team or project sites, as well as a single ‘apps’ site. (Our apps sites (< 10) were set up to address small business problems that in the past might have been addressed by using Microsoft Access).

Thanks to our OOTB model, we were able to upgrade to SharePoint 2013 over a weekend, with almost no errors. The only site we could not upgrade was the intranet which remains (as at August 2018) in ‘compatibility mode’.

Note: It is not possible to migrate directly from SharePoint 2010 to SharePoint Online. It must be upgraded to SharePoint 2013 or SharePoint 2016 first.

The situation in 2016

In May 2016 we changed our Microsoft Enterprise Agreement to an Office 365 subscription model. Our reasons for going to Office 365 were driven by multiple factors, including the need for mobile access to information.

It is important to remember that SharePoint Online is only one element among many others in Office 365. That is, while it is technically possible to do it, SharePoint would not normally be migrated on its own to SharePoint Online. Any migration must take in account a range of considerations relating to the broader Office 365 environment, including (but not limited to):

  • Office 365 licences (and what this meant for our users with Office installed on existing computers which were being upgraded to new Windows 10-based devices as part of a separate project)
  • Active Directory syncing so users can access the environment.
  • Exchange mailbox migrations so SharePoint-based, email-linked Flow workflows can work.
  • OneDrive for Business, as a SharePoint service to replace ‘personal’ drives on network file shares.
  • Security controls and records retention policies, set from the Office 365 Security and Compliance admin portal, as well as audit logs in that same portal.
  • Office 365 Groups with associated SharePoint sites, Yammer groups (which can be linked with Office 365 Groups) and Microsoft Teams (which can also be linked with Office 365 Groups).
  • ‘Classic’ and modern team sites, Office 365 Group-based sites, and communication sites.
  • The SharePoint user portal.
  • The mobile app, and how sub-sites are accessed.
  • The ever-changing SharePoint Online environment in which anything described as ‘classic’ is likely to be deprecated at some point, and new features appear.

Migrating multiple web applications to one

We needed to plan our migration process, moving away from our five web applications to a new model. We new that, with the exception of our customised intranet, we would probably be able to migrate almost all of our sites relatively easily because we had always kept to the OOTB model.

Fortunately, Microsoft produced a very useful 12-page document which provided a good overview describing how it ran its own SharePoint migration, and good advice for how we might do our own migration.

SharePoint_to_the_cloud_MSpaper.JPG

Learn how Microsoft ran its own migration

We had a range of factors to take into account.

  • One of our initial decisions was not to migrate any active site until all Exchange mailboxes were migrated (and preferably, end-users had new Windows 10 devices). As it turned out, the decision to migrate mailboxes was delayed and as a result we would end up migrating most sites first.
  • We need to work out how to migrate our content as it was no longer possible to do a ‘lift and shift’. We investigated the market and made the decision to acquire a migration tool, ShareGate, to do the migrations ourselves. We would later find the same tool useful to migrate personal drives to OneDrive for Business.
  • We identified the likelihood that we would create new SharePoint Online sites in parallel with the migration of on-premise sites; this was partially because some existing on-premise sites with multiple sub-sites would be split into separate sites instead, but also because the new SharePoint was so much more versatile and would likely be popular.

The new architecture model

An important point to note is that the new SharePoint Online architecture model provided the opportunity to re-think our SharePoint model and, to some extent, clean up or leave unwanted SharePoint content behind. To quote the Microsoft site above, ‘the best migration is no migration’.

As noted above, we had five primary web applications in our SharePoint 2013 environment. These had to be migrated (or re-created, in the case of publication sites) under one of two paths (only – /teams or /sites) to one of three site option:

  • ‘Classic’ sites (the default for all team and project sites)
  • Office 365 Group-based team sites
  • Communication sites (re-created page-based content)

That is:

  • Migrated team and project sites would become classic team sites under either (a) /teams/sitename path or (b) /teams/prj_sitename path, respectively. There were some exceptions:
    • Some sites with multiple sub-sites would be split up into multiple independent sites (including using the new ‘hub’ sites).
    • A couple of team sites would become communication sites.
    • Team sites that crossed multiple organisational business areas would be created as classic team sites under the /sites/sitename path.
  • Most publication sites that used the publishing features would need to be re-created as communication sites under the /sites/sitename path. There were some exceptions:
    • Some publication sites would become team sites instead.
    • The intranet would be managed separately as, at the very least, it would need to be re-created in SharePoint Online. It could not be migrated ‘as is’.
  • Application sites would become team sites.
  • Some existing sites or sub-sites might be migrated to SharePoint sites linked to Office 365 Groups, with the naming prefix of either GRP_ or PRJ_.

The above ‘mapping’ model was an early decision that did not change.

Preparatory work

We also commenced work on the following elements of work:

  • Reviewing all existing sites to determine which sites would be migrated or discarded – see below.
  • Re-developing our SharePoint Architecture documentation for the Online version.
  • Investigating and documenting all Office 365 admin and Office 365 Security and Compliance admin configuration settings, and determining roles. This process, which required Global Admin access, included establishing records retention policies (from mid 2018) in the Security and Compliance admin portal.
  • Re-developing our existing SharePoint admin documentation for the Online version, including all the configuration settings. We included the OneDrive for Business config settings in this same document as it is a SharePoint service.
  • Understanding how the new environment worked, and would work.
  • Re-establishing our SharePoint Admin and SharePoint User Group sites in SharePoint Online.
  • We also created a range of ‘test’ sites to better understand the new environment.
  • Creating an initial schedule for the migration of sites, targeting inactive sites first.
  • Assigning the initial batches of Office 365 licences.
  • Developing a repeatable process to migrate sites using ShareGate. In our environment steps involved:
    • Identify need to migrate site
    • Register a new site request in our SharePoint Admin portal.
    • Register the task in our Jira task management system.
    • Create the SharePoint Online site (via a script linked to the request).
    • Migrate the on-premise site, make it read only with a re-direct notice on the front page (and a three month deletion notice*).
    • Prepare the migrated site, including swapping the classic default home page to a modern home page.
    • Hand over the site to the business owners and close the task

* In practice many of these sites still remained after 6 months.

As part of our review process, we identified around a dozen sites that had one or all of the following elements, that would mean we had to devote more time to their migration (‘custom workload’ in the Microsoft document above):

  • Complex workflows which would need to be re-created.
  • Integration with other systems (mostly via BizTalk).
  • Links with ETL processes.

We also identified around 50 sites that would not be migrated:

  • Sites that were unused or had no content of value (often because the original was still on a drive).
  • Sites that did not need to be migrated, for example if their content had been migrated to a different business system.
  • Test sites.

Sites that were no longer used but contained records that needed to be kept were to be migrated with the word ‘Archive’ to the end of the site URL name, assigned a site retention policy, and then made read only.

By August 2017, we had identified that 250 site collections would be migrated to SharePoint Online. We acquired ShareGate in September 2017 and were ready to start migrating.

In Part 2 of this series of posts I will describe the migration process and the lessons we learned along the way.

Advertisements

Can ‘traditional’ records management survive the digital future

December 23, 2015

At a conference last year I listened with interest to a panel discussion on the subject of ‘paper versus digital’. Perhaps the debate was only meant to be light-hearted but it was clear that quite a few records managers continue to manage paper records and see them as their primary focus. In almost all cases these records are the printed version of a digital original. In most cases those digital originals remain stored on a network drive (or a person’s USB), or attached to an email, or – if you are lucky – copied to the organisation’s electronic document management (EDM) system to become the ‘official’ version.

Only a month ago I saw a colleague, whose position title clearly indicates she is responsible for digital recordkeeping, asking about KPIs for the creation of paper files. Perhaps it was part of the digital recordkeeping strategy to examine this subject, but it underlined for me the persistence of ‘traditional’ recordkeeping concepts, that documents belong in containers, that may be files or groups of files (or series etc). This is not just an issue for records managers, but for end users as well who continue to ‘think paper’.

It concerns me that records managers persist with the concept that digital records somehow ‘belong’ in digital files, often directly connected with the descriptive ‘business classification’ system of function and activity (in the sense that the file title or metadata may include the function and activity).

This way of thinking reinforces the concept that these records belong nowhere else, or might have no context outside the container, which of course is not necessarily true. It is, of course, possible to cross-link documents to a different container, but at what point is this a manual exercise? And where does it stop?

Part of the problem has been that digital systems mimic in their appearance and functionality the concept of a file or folder. We see them on network drives, in email, and in the containers of EDM systems. These folders or containers provide a sense of surety, that a document has been ‘stored’ or saved in a specific file. The file, however, is no more than a visual construct; the document’s file/folder/container metadata is what causes the document to appear that way.

This is all the more obvious in some EDM systems that store documents in a file store (often folder-based) and the metadata about the documents in a separate database. Aside from the folder structure, these documents are more or less a bunch of uncontrolled documents with metadata links to the ‘file’ construct in the database.

Of course, keeping records in some form of business context is important both for the management of those records (short and long term) and for retention and disposal purposes. But that doesn’t necessarily mean that documents must be associated with ‘files’ that, according to some records managers (and paper filing theory), should contain no more than 300 documents.

All of this is unnecessarily restrictive and, for end users, results in additional work. It’s no wonder that EDM systems have lagged behind the more ‘traditional’ way that users store documents, in network drives.

We should, I believe, think of digital records as being self-contained and dynamic objects with their own metadata payload that may be relevant in multiple contexts, not a fixed object stored in another fixed object.

Enter Microsoft Groups

In 2015 Microsoft announced the concept of Groups, accessed primarily from Outlook. Conceptually, you could think of a Group as being more similar to an Active Directory Distribution List rather than a shared mailbox. It is also similar to Yammer groups, but with much more functionality.

From Outlook, a user can create a Group to work collaboratively. Group members can initiate conversations with other members of the group (including Skype discussions) and – importantly – share documents.

So, Microsoft is taking us into the world of Group-centred information collaboration. This is not an experiment on the part of Microsoft, it is part of the world of Office 365 which includes SharePoint Online and OneDrive for Business, Delve, and more, supported by any device to enable users to create, access and share content anywhere, anytime.

Now, not every organisation uses Microsoft products, but an awful lot do. And for those who do, Groups in Office 365 is the future. The impact on traditional recordkeeping practices is likely to be large.

By 2020 or earlier, users will create Groups via Outlook to collaborate. Within the Group they will store digital content (not just documents); documents are stored in SharePoint Online site collections which are in addition to any other Site Collections. That’s right – in addition to your controlled Site Collections, there will be a multitude of Group-specific Site Collections. Conversations that take place in the group are stored as items in the inbox of the Group mailbox. In other words, all the content (and perhaps context) will be based around the Group ‘subject’.

To access all this rich content users will have three main options: (a) be an active member of the group; (b) SharePoint search; and (c) Delve. Business intelligence and data analytics, or e-Discovery type products, may also provide a fourth line of access.

Likely impact on recordkeeping

The recordkeeping practices that I believe will be most affected are: (a) metadata, including the application of BCS terms; (b) container-based storage, and (c) retention and disposal. It may also impact the careers of records managers. There will be no control over the creation of new groups (there can be, but why?), so their information content (in Outlook and in the associated SharePoint Online site collection) will become, in many respects, the new containers for records, in addition to other site collections.

On a positive note, I believe users will adapt and adopt this new model of working rapidly. Users, many of whom remain glued to Outlook as a business ‘collaboration’ tool, will find that Groups provide them with the ability to store and share documents via the Outlook interface. They may need to get used to the idea of ‘working out loud’ in Groups, but I think that will happen fairly quickly.

One thing is sure – change is inevitable.

Folders uninterrupted – the enduring problem of folders

November 19, 2015

In September 2014 I presented to the Records and Information Management Professionals Australasia (RIMPA)’s annual conference in Adelaide, South Australia, on the subject of folders and how it is possible to manage digital content differently using products like SharePoint.

Last week a couple of people in completely different parts of the company I work for demonstrated clearly the problem of folders on Outlook and network drives. The first had at least 100 folders and sub-folders in her Outlook where she said she ‘filed’ everything so she could find it again. The other person showed me a newly created section of the network drives with hundreds of new folders and sub-folders set up by someone who didn’t like or get the existing agreed folder structure that has been in place for many years.

In both cases I asked the same question – how do you find anything? The answer, in most cases, is that the person clicks on folder names that they created or are familiar with, until they find what they are looking for. Search was usually relegated to last ditch attempt to find things but was not considered to be very helpful or accurate. Both individuals lamented the duplication of information in folders, duplication they often didn’t know about until the inevitable clash of versions that occurs regularly.

At a workshop/presentation I did for another organisation recently, I was told that the organisation relied heavily on what are called ‘titling conventions’ to enable the information to be found. That is, the names of files, folders and documents follow a pre-defined model.

In reply to my comment that such titling was surely less valuable when you could search for the content in their document management system, I was advised that they didn’t yet have the ability to search by the content of documents (a resource issue). They could only search for content based on titles, once again reinforcing the need for folders (with unambiguous and consistent titles) to find that information.

Breaking the habit

I have said many times, folders are a very hard habit to break. Given the generally poor ability to search and find content in email and network drives (and in some EDRM systems – still), search is relegated to second place and folders and pre-defined titling conventions continue to dominate where content searching is not available.

For at least fifteen years, working across a range of organisations, I have had the ability to search within the content of digital records. This includes being able to search for the content of scanned documents that have been subject to OCR or optical character recognition. In this digital age I’d be very disappointed if the only way I could find content was by navigating labyrinthine folder structures on email or network drives.

I have fewer than five sub-folders under my work email inbox. I use those additional folders to route specific content such as junk mail, SharePoint notifications or personal email. I don’t save anything to a network drive and only use SharePoint to store and retrieve records and other corporate information. Why? Because search is really good.

I know this is not common practice around the organisation. Common practice is to create and maintain multiple folders and squirrel useful content away in network drive folders. It’s been that way for 30 years.

I don’t think folders are going to vanish from the landscape anytime soon – they are no more as unlikely to vanish in the next 10 years as email, although both are slowly morphing over time into something different, particularly as ‘collaboration’ tools, search, and the presentation of information via analytics engines are introduced.

Content finds you

Facebook and Google are successful for a reason – they both present content (including advertising) relevant to the user based on her or his digital content. A simple analogy is to compare the corner store where you know you can go to get milk with the ability of digital devices to remind a user they need to buy a specific type of milk because (a) their internet linked refrigerator has noticed that milk is running low and (b) they regularly post to social media (or even in a single email) about their preference for this type of milk. It’s creepy, but it works .

Many contemporary office environments are not much different – users know they find their digital content in the folders (the little corner stores) they created or have learned about. But they don’t know how much other interesting or directly relevant content is there and they don’t bother looking because they know it will be a poor and time-wasting experience.

They don’t know what they don’t know.

My experience over the past five years with SharePoint (and with similar products before it) is that users are resistant to ‘having’ to store documents in a different location to what they are used to (i.e. network drives).

Users worry their content will be lost, it’s ‘yet another place’ to store and find content, and the user experience (compared with folders) is alien. This doesn’t stop them consigning endless information to social media and being the recipients of targeted advertising and suggested friends based on their content. The difference between the two is so extraordinarily different it is sometimes hard to believe that users do this, checking in on Facebook’s content-driven feed while ‘navigating’ through endless folders, many with names that are about as vague as you can get.

Where is this heading

In the last weeks Microsoft has outlined what is sees as the future of the office and their Office. Three key elements of that future are:

  • SharePoint (on-premises and online) and OneDrive. These two will not replace folders completely but will give users a much better (and more importantly, more mobile) experience with storing and finding content.
  • Office 365 Groups. Groups of people who have a common interest and can communicate across platforms, whether it be via email or other collaboration tools including SharePoint or Yammer.
  • Analytics as the way we will harness and access our information, via interfaces like Delve that serves up relevant information based on the user’s digital content. Just like Facebook.

Does Delve deliver on the ‘Semantic Office’?

Many years ago I wrote on this blog about what I called at the time the ‘Semantic Office’. In that article I asked whether we will one day experience a time when we can truly harness the digital content in our business systems to link and present relevant information to users based on the same methods found on the internet at that time (for example in eBay, Amazon etc).

I think Microsoft has finally figured this out with Delve. My guess is that, by 2020, Delve or a version of it will be the most common ways users first access their content.

So where does this leave folders?

Folders aren’t going to go away anytime soon. I’d like to think that Microsoft took a close look at how they could deprecate folders in email and drives and realised that this was akin to giving up on email – it wasn’t going to happen.

Instead, Microsoft decided to introduce new tools that can capture and – more importantly, perhaps – present information in completely different ways that digital natives expect and even take for granted. As someone said recently, when a large group of young people were asked to send an email, many responded with ‘what’s email, why can’t I Inbox you?’.

Change is happening, slowly, along with the culture shift that must accompany it. For every older ‘paper native’ worker that retires, a younger digital native commences work in the workforce, bringing with them new expectations both in terms of work experience and tools they use.

Just like the clacking of manual typewriters in offices was eventually replaced by the slightly smoother sound of electronic typewriters, and then the click-clack of keyboards hooked to the back of computers, the silent touch screens of the future will eventually dominate and then themselves be replaced by something else.

Habits are hard to break, but change happens.

Managing documents in SharePoint 2013 and SharePoint Online – what’s changed?

September 20, 2015

We are in the process of upgrading our extensive ‘out of the box’ SharePoint 2010 environment to SharePoint 2013 and, at the same time, setting up a limited SharePoint Online presence. So, what’s changed in relation to managing documents as records?

In SharePoint 2013, the short answer is ‘not much’. However, there are some new things that will change some parts of our technical design model. The things that have remain unchanged include:

  • Document libraries as the primary ‘container’ to store documents, using folders, document sets, or metadata-based categorisation, to ‘group’ documents.
  • Document IDs, set in the Site Collection Administration – Document ID settings section.
  • Document versioning (major/minor, major, or none), set in the Library Settings – Versioning Settings. Other settings in this section, which we generally do not use, have also not changed: Content Approval, Draft Item Security, and Require Check Out.
  • Use of Content Types (and disabling of Folders), enabled via Library Settings – Advanced Settings, and then added in the general Library Settings – Content Types area. All other options in this section, which again we rarely use, are still there: Custom Send to Destination, Search visibility, Offline Client Availability, Site Assets Library, and Dialogs. The old familiar ‘Datasheet view’, which we use to ‘bulk upload’ or update metadata has been renamed ‘Quick Edit’.
  • Almost unlimited metadata options via pre-defined Content Types or library-specific columns, both of which can point to the centrally controlled metadata in the Managed Metadata Service or local ‘look-up’ lists.
  • Multiple list views, each with their own linkable URL.
  • The ability to copy documents, via drag and drop or copy/paste, using ‘Open with Explorer’. This, coupled with the ‘Quick Edit’ option, allows documents to be copied to SharePoint document libraries in bulk and metadata added easily.
  • Access controls that can be applied right down to the document level.
  • The ability, from a document-specific drop down menu, to view or edit the properties of a document, check it out or in, view the version history (and restore versions), run workflows, download a copy, share (the former ‘manage permissions’ – see below), and delete (where enabled).
  • Out of the box simple workflows for Review and Approval.
  • Site collection audit trails, accessed via the Site Collection Administration area. Unlike some other products, SharePoint audit trails are not ‘attached’ to individual documents, but are centralised in one place.

So, all in all, not much change really, except for the ‘Share’ option. In many respects, the way the simple ‘Share’ has been designed is a more intuitive process than ‘Manage Permissions’. When you add a user you decide if that person should have Edit or Read privileges. If you maintain the default Owner, Member and Visitor groups, then those with Edit rights are added to the Member group, those with Read rights are added to the Visitor group.

For more complex permission management, including to stop inheriting the default permissions (or to add them back, which is now called ‘Delete unique permissions’), or creating new groups, you need to select the ‘Shared With’ option from the drop down menu and then select Advanced.

What about disposal/disposition management?

Again, not much has changed in relation to document libraries or lists. Out of the box, your options are as follows:

  • Using centrally defined, document-based, Content Types, using Information management policy settings. Not a bad idea if (a) you have a way to ensure that these Content Types are always added to libraries, and (b) you are happy to manage the disposal of documents one by one.
  • Changing the default Information management policy settings option in document libraries from ‘By Content Types’ to ‘By Libraries and Folders’ and then applying retention policies on the folders you create. The main negatives of this option are that it means you have to use folders, and you have to manage disposal by document.
  • Leaving documents in document libraries, and having a way to review these, across the farm, in a centralised manner. This requires some kind of script to be written to (a) list all libraries across the farm and (b) work on the basis that the ‘Last Modified Date’ is the last action on any document in the library, but it seems the more logical and simplest way to achieve the outcome you seek, and keeps all the documents in the same container.

It remains surprising to me why Microsoft does not provide the option to set a disposal period on an entire document library.

Of course, SharePoint 2013 now allows you to set a disposal period on a Site, but this isn’t likely to work for sites that contain a range of diverse content that may be useful over a long period of time.

So, what about SharePoint Online?

The first thing to remember about SharePoint Online (SPO) is that it’s not SharePoint On-Premises. Seems obvious, but the natural instinct is to wonder if or how the two environments can be connected. In most cases they can’t, so it’s not worth thinking along those lines. SPO is a way to manage content in the cloud in addition to, or instead of, on-premises.

SPO has all the same document management features you find in SharePoint On-Premises, described above – document IDs, versioning, content types, metadata, multiple list views, open with Explorer and Quick Edit, access controls, document-specific menu options, simple workflows and audit trails.

O365_SPO_LibraryRibbon_DocMenu

The options for disposal/disposition management using SPO ‘out of the box’ (should that be ‘out of the cloud’?) is the same as for the on-premises version.

You didn’t mention Records Centers …

Records Centers (or in Australia, Centres) were in many respects designed to be the ‘send to’ archival repository for other sites. Great idea if it works in your world, it doesn’t work in ours. The main drawbacks are that documents ‘sent’ to a Records Centre are in fact copied. Custom metadata is lost, versions are lost, audit trails are lost. And, you can retrieve the document.

But Records Centres (or in the US, Centers) can be useful on their own as a repository of specific types of records that aren’t accessed too much. We have several Records Centres and we use them in a separate web application for specific purposes, including to store scanned documents. We don’t use them as they were originally intended for the reasons stated above.

And yes, you can create Record Centers in SPO, too!

Implementing SharePoint to manage documents – changing user behaviours

September 10, 2013

From around 25 years ago, computers began to replace paper-based processes for most office-based workers. Users grumbled and some (mostly at senior levels) managed to avoid using a computer altogether. The majority of office workers (who became known as ‘users’) just adapted to and eventually adopted the new work practices, often with the promise of greater productivity.

Over the past 25 years, if no other system was mandated for the storage of digital records, users mostly:

(a) created new content using desktop applications and stored it on largely uncontrolled and poorly managed network drives; and
(b) sent or received emails (often with attachments from stored on drives) and used email folders to store those emails

… in both cases with scant regard to business or corporate recordkeeping requirements.

The end-result of 25 years of these practices is network drives and email systems full of digital ‘stuff’, little of which relates to each other in any meaningful way (unless it all happens to be stored in a single folder, or folder-based structure, which is not common).

Many electronic document management (EDM) products have been released over the years, ostensibly to improve the management of electronic records, but organisations still grapple with getting users to use them. I suspect that one of the reasons for the poor take-up of EDM ‘solutions’ is that users are so familiar with using network drives and emails folders; it is just so heavily ingrained into the way we work, not to mention the way we manage our own personal records and other digital content on home PCs and laptops.

Interestingly, some of the EDM vendors have developed interfaces that look almost identical with network drive folders. My own personal experience with these products is that users often don’t understand the difference, or the value proposition, that comes with use of these systems.

Along comes SharePoint

SharePoint has been around for about 10 years now, but best practice document management in the product did not really emerge until SharePoint 2010 was released (see my other posts for further information).

What is, for me, interesting about SharePoint is the design of the user interface presented in standard team sites via a browser. Microsoft clearly did a bit of research (and understands) how users work, with document libraries on the left (looking a bit like network drives), as well as a number of other list-based options such as tasks, discussion boards, announcements and calendars.

Not infrequently, the initial user reaction to team sites is one of uncertainty and some degree of confusion – is it the intranet?. Depending on how an organisation delivers team sites, user reaction can be either enthusiastic (so much better than drives/folders) or negative (prefer to use drives/folders).

SharePoint team sites as communities of information

For most of the past 25 years or so, ‘collaboration’ has meant exchanging emails with attachments and/or accessing the same part of a network drive, structured in a way that ‘makes sense’ to the user or the team.

SharePoint also allows users to ‘collaborate’, but quite differently from how users have done it to date. In some respects, SharePoint is more in tune with social media sharing (open access to all, or to a trusted group), than with traditional ‘closed’ methods (i.e., via limited email distribution, or limited access to parts of drives).

SharePoint team site can be used to store documents (and photographs) in document libraries (with unique document IDs, version controls, check in/out controls, out of the box workflows, audit trails and custom metadata). It can also use the rows and columns of data in spreadsheets and small databases and store and present that data instead in lists (including discussion boards, announcements, calendars, task lists and custom lists). As the volume of information stored in the site increases, search becomes an additional valuable resource that brings a range of information together. That is, instead of a list of documents, users can see a range of other contextual data.

A single SharePoint team site starts to becomes a community of information.

Changing behaviours

In my view, the key to getting users on board with SharePoint team sites is selling its benefits, not the technology, and helping users understand how to ‘map’ current practices to the new environment.

One of the best ways to demonstrate the benefits of using SharePoint is to show how it is being used (differently) by other teams, and to emphasise the ability for site owners (not a controlling authority such as IT) to use standard features (not customise*) in team sites to meet the needs of the team. That is, the team identifies and implements ways to use it that makes sense to the team, within a few broad constraints to ensure consistency in use across the organisation.

But, despite the obvious benefits, deeply ingrained user behaviours can be exceedingly difficult to change unless that change is driven by users (preferable) or mandated (may have limited success). The SharePoint team site paradigm *is* a major change from using network drives to store documents and emails (with attachments) to collaborate.

‘Mapping’ current practices to SharePoint team sites in many ways underlines the weaknesses of current practices, and can include things like:

  • Demonstrating the relationship between network drives and document libraries, using document sets or document categorisation to replace folders.
  • Describing convincingly why folders are not a good thing to use (despite the apparently similarities with network drive folders). For example, they can be very hard to navigate, do not have unique IDs (like Document Sets), can be replaced by Document Sets with longer names (including the concatenation of network folder names), can results in very long URLs and so on.
  • Showing how document metadata and views can completely replace the need for folders.
  • Showing how a team calendar (or calendars) can replace white boards or similar.
  • Demonstrating how an Excel spreadsheet can become a much more useful list of information, so that users can edit their own items rather than wait for someone else to stop editing the spreadsheet.
  • Showing how a team discussion can not only replace email but also keep a record of otherwise inaccessible email trails.
  • Demonstrating audit trails and site web analytics to show (and know) ‘who did or saw what’.
  • Demonstrating the use of hyperlinks instead of emails with attachments to share information.
  • Demonstrating features in document libraries that are absent from network drives (including unique IDs, versioning, audit trails).
  • Demonstrating out of the box workflow that remains with the document.
  • Demonstrating search, and in particular searches that return metadata-rich documents.
  • Showing how the Recycle Bin keeps deleted items for three months (which, coupled with the audit trails, help to retrieve those items much more easily than requesting the document back from a back-up tape).

and so on.

*By customisation, I mean changing the overall look and feel of a team site by modifying the theme, creating new types of pages, adding documents to pages instead of to libraries, using folders instead of document sets, and so on. Customisation can lead to confusion which in turn makes users less likely to use the site.

Is your SharePoint team site a ghost town?

May 24, 2013

One of the best ‘unknown’ (to end users, anyway) features of SharePoint 2010 and SharePoint 2013 is ‘Site Web Analytics’, hidden away under Site Settings. This, out of the box, feature provides useful information such as the average number of site views per day, the average number of unique visitors per day, trends, as well as a listing by user names of the top visitors and the number of their page views. It’s a great way to see if a site is being used.

The default date range for Site Web Analytics is the past 30 days but the settings can be changed to the last day, week, month, quarter, half-year or year. Custom dates can also be set, and these reports can be exported to a spreadsheet for further analysis, including through scheduled reporting sent by email.

So, what does this have to do with the subject – is your SharePoint team site a ghost town?

Simply, Site Web Analytics tells you if your team site is being used – by your team. If it’s not used, then it’s a ghost town – and you need to understand why and consider changing or deleting it.

In many cases, the reason a SharePoint site becomes a ghost town is because users simply neither want nor know how to use it.

What makes a document management system popular?

My experience implementing and using many other document management systems (TRIM, Documentum, iManage, Alfresco) over the years, along with considerable anecdotal evidence, suggests that, unless there are legal or regulatory requirements mandating it, end users tend to baulk at using a document (and records) management system, especially if it:

  • Requires more than 20 minutes to learn. Some EDMS systems require hours of training – for end users!
  • Duplicates what they do already with network drives. More often than not it requires ‘saving’ an existing born-digital record to the recordkeeping system (AND leaving the original where it was stored).
  • Stores records of a business activity away from other, potentially rich, related records that are not documents.
  • Provides no obvious or little tangible business value for the user – it’s a ‘compliance overhead’.

A popular document management system is one that:

  • Users can start using the system with very little training. It should be intuitive and make sense at first glance. I often use the example of Facebook – no-one ever got training to use it and yet a billion people registered to use it.
  • Reduces (not necessarily eliminates) duplication of effort. Network drives can still exist to create and manage ‘working documents’ instead of final versions.
  • Can keep records in context with other related records such as calendars, lists, tasks, announcements, social networking messages, and so on.
  • Provides tangible business value for the user, with minimal effort. It’s hard to break the network drive habit but simple things like unique document IDs, versioning, hyperlinks, and the ability to email a link instead of a document help to break that habit.

What can turn a SharePoint team site into a ‘ghost town’?

Microsoft clearly put a lot of human-computer interaction (HCI) effort into the user interface design of SharePoint team sites (along with all their other products). Team sites in SharePoint 2013 are a design improvement over SharePoint 2010 but still maintain the same essential look and feel, which is:

  • Document libraries and lists on the left hand side navigation. It’s no coincidence that this is exactly where network drives and email folders are accessed from.
  • An editable main page that can be modifed as required to present additional information or options, including via webparts.
  • Default and modifiable views of documents and lists.
  • A simple search option on the top right.

Over 25 years of network drive use has conditioned users to navigating for documents from the left hand side, searching from the top right hand side, and seeing information presented in the middle of the page. Change this paradigm and you start to lose users.

I believe there are two main reasons why users don’t or stop using team sites (described further below):

  • The ‘plain vanilla’ out of the box functionality in a brand new team site.
  • Site pages that have been customised to the point where end users cannot work out how to navigate or access their information.

Out of the box generic functionality

Out of the box team sites include the two features which, if not addressed or re-configured (a 2 minute job), can negatively affect take up rates:

  • ‘Shared documents’ under ‘Libraries’. This to me is a ‘bookmark’ library that should be removed almost immediately and replaced with a library that has a name that makes sense, ideally named after a business activity. Otherwise, ‘shared’ is like ‘general’ in a network drive. If users have only one, generically-named library to choose from, they are unlikely to use it (compared to the rich tapestry of their network drives).
  • Folders. Unfortunately, while SharePoint folders look like network drive folders they don’t work the same way. They create unnecessarily long URLs and are often impossible to navigate. It’s not uncommon to hear users say they cannot find documents in a site because they cannot work out how to navigate folders (as they don’t include a + to guide navigation). My recommendation is to turn them off under Advanced Settings after the library is created. However, users inevitably want a way to categorise or group documents in a document library. Document Sets generally make sense to users and get quick take up but you can only have one level. A good site has a reasonable number of smartly-named document libraries – perhaps no more than 10, grouped as required by Headings – using Document Sets instead of folders. Another form of categorisation is using categories in the document metadata and then creating a view that groups the documents by those categories.

Customisation

Too much customisation of site pages can also turn users off. As noted above, it’s worth keeping in mind that Microsoft spent a lot of effort (and probably dollars) on designing their standard team site user interface. Some of the common modifications I have seen on ‘ghost town’ sites include:

  • Using themes that make the site look completely different from all other sites. It’s important to maintain consistency in themes.
  • Using unnecessarily large font sizes on the page.
  • Using too many images or graphics on the front page.
  • Assuming the text or content on the page is actually of interest to the target audience. I usually suggest that Site Owners (note, there should never be just one) agree among themselves; if this is not possible, test the proposed page design on a user who has never seen it.

Avoid ghost town sites – keep users coming back

Remember, most users are familiar with using network drives and folders. Use document libraries with sensible names that are obvious to users, don’t use folders, and keep the front page content simple and easy to view, and you are more likely to attract users back again and again.

SharePoint 2013 Site Disposal Policies

May 18, 2013

SharePoint 2013 includes the option to set a disposal date on site collections. This article describes how to configure a SharePoint 2013 site collection to include a site disposal policy.

Default settings

A site cannot be deleted (either manually or automatically) unless a Site Policy has been set up (exception – the SharePoint Administrator has permissions to do this).

Without a Site Policy, the default settings under the Site Closure and Deletion option (see below) are as follows:

  • Site Closure – ‘Close this site now’ click box default: greyed out.
  • Site Deletion – ‘This site will be deleted on:’ Default: ‘Never’.
  • Site Policy – Default:  ‘No Site Policy’.

Setting up a Site Policy

New site policies are created under Site SettingsSite Collection AdministrationSite Policies. Once created, the policy is applied under Site SettingsSite AdministrationSite Closure and Deletion. While you can create multiple policies, only one policy can be selected at a time under the Site Closure and Deletion option.

There are no default policies; the first time Site Policies is opened, the Site Policies section provides only one option – ‘Create’. Each policy must have a Name and may have a Description. The name and description can be the class description from a records retention schedule, using ‘after date created’ or ‘after date closed’ as the triggers (see below).

Site Closure and Deletion options

There are three options under Site Closure and Deletion:

  • Do not close or delete site automatically. The default option.
  • Delete sites automatically. This option deletes a site on a pre-defined date after it was created or closed.
  • Close and delete sites automatically. This option first closes the site and then deletes it on pre-defined dates.

In addition there is a check box ‘Site Collection Closure’ that allows the site collection to be made read only when it is closed.

Delete sites automatically

When this option is selected the following appears:

  • Set Deletion Event. The two options provided are ‘Site closed date’ and ‘Site created date’, plus n days, months, or years.
  • (Check box) ‘Send an email notification to site owners this far in advance of deletion:’ (i.e., to warn them of the pending deletion) – n days, months or years. Default setting is 3 months.
  • (Check box) ‘Send follow-up notifications every:’ (i.e., to remind site owners of the pending deletion) – n days, months, or years. Default setting is 14 days.
  • (Check box) ‘Owners can postpone imminent deletion for:’ (i.e., to postpone the proposed deletion) – n days, months or years. Default setting is 1 month.

Close and delete sites automatically

This option is identical to Delete Sites Automatically except that it also includes a date when the site can be closed – after which a deletion event date is set followed by the same three options above.

Site Closure and Deletion

As noted above, a Site Policy must exist before a site can be closed and deleted using these options. The Site Policy must be selected otherwise the default options (see above) apply.

  • If the Site Policy is based on the Delete Sites Automatically option, the option to ‘Close this site now’ becomes available. If the option ‘Site Closed Date’ was selected, the site will not be deleted (at the pre-defined time) until this option is selected. If the option ‘Site Created Date’ was selected there is no requirement to ‘manually’ close the site.
  • If the policy is based on the Close and Delete Sites Automatically option, the option to ‘Close this site now’ becomes available. This allows the site to be closed earlier, otherwise the deletion date will be automatically calculated from the site policy setting and displayed next to the Site Closure and the Site Deletion options.
  • If no policy is selected, the default settings will apply; this means that the site cannot be closed.

Further reading

Overview of site policies in SharePoint 2013 (Microsoft).

Managing Physical Records in SharePoint 2010

May 7, 2013

It is sometimes said by records managers that one of the drawbacks of SharePoint 2010 (and SharePoint 2013) is that it doesn’t manage physical records, or at least not very well.

This article describes how to manage physical records in SharePoint 2010/2013 using a list that includes barcodes.

Up front and out of the box, the main limitations are the ability to: (a) print labels for paper files and boxes (including labels with barcodes) (b) attach a portable scanner to scan barcodes and (c) maintain a chronological history of file movements. These limitations may not be a problem for many organisations seeking a low-cost way to manage mostly inactive files and boxes, especially if they currently use an Access database or Excel spreadsheet to do this now.

Options to address each of these limitations are discussed in this article.

Metadata options

There are almost no limitations on the type of metadata you can use to describe physical records. I would suggest having the following metadata:

  • Title (e.g., name on the file or box)
  • Record Type (drop down list – file, box, etc)
  • Reference ID (manually entered unless you use the default ‘ID’ option, possibly with a separate ‘year’ column)
  • Date created (date first registered, default system date)
  • Start date (start of date range)
  • End date (end of date range)
  • Business Owner
  • Current location (can be a custom list of pre-defined locations)
  • Date at current location
  • Record status (drop down list: active, inactive, missing, destroyed etc)
  • Internal barcode (if one exists)
  • External barcode (usually the one provided by a storage provider)
  • Records description (description of contents, usually of a box)
  • Box number
  • Box barcode (may be the same as the Internal or External Barcode)
  • Retention schedule (can be from a pre-defined list)
  • Disposal action (can be from a pre-defined list)
  • Date to be destroyed (manually entered based on retention schedule and disposal action)
  • Destroyed? (checkbox)
  • Date destroyed (manually entered date)
  • Legal hold? (checkbox)

If physical files are registered in the system, then only details of the box and disposal action would need to be entered. When the box is ready to be sent offsite, details from the offsite storage provider’s label (often provided in advance) can be entered in the external barcode and other metadata (including current location) updated.

An optional barcode can be obtained by checking ‘Barcode’ in the Information Management Policy settings for the list item. Once checked, the barcode appears when the item is displayed in view but not when it is edited or exported.

Printing labels

All list metadata (except the barcode) can be exported to an Excel spreadsheet, which can then be used to print labels. Microsoft provide details how to create and print labels for a list in Excel here:

Scanning

Using a portable (or attached) scanner to scan barcodes and update your records register, especially with a new location (or during a physical records audit) saves a great deal of time instead of entering details manually. It is possible to scan directly to an Excel spreadsheet however several steps may be required to match or validate the barcode with the details in the SharePoint list.

For example, if series of files are being scanned to a box (or location), the usual practice is to scan the box (or location) barcode first, then scan each individual file. If scanning directly to an Excel spreadsheet, you would do it the other way around – have the list of existing files in the spreadsheet, then scan the box or location barcode to the cell next to the file number. This detail can then be copied and pasted back to the SharePoint list.

Chronological movement history

The only option to maintain a chronological movement history, out of the box, would be to add versioning to the list. This means that each time the item is edited, a new version is created, keeping the previous details. The movement history will not display in a single view in the list but clicking on ‘Version History’ will show any changes made in single view. Only ‘major’ version numbers are supported in lists (i.e., 1, 2, 3, 4).

SharePoint 2010 – Send to Records Center / Centre

March 27, 2013

SharePoint 2010 includes the option to send a document from the site where it was stored to the Records Center (Records Centre in Australia), and other locations. The option is set at the Central Administration Web site under General Applications – Configure Send to Connections. See http://technet.microsoft.com/en-us/library/ee424395(v=office.14).aspx for the details of how to configure this option.

The ‘Send to action’ list offers three options when a document is sent to the Records Centre (options which are then available across the farm):

  • Copy the document (which simply makes a copy)
  • Move the document (which appears to remove the document entirely. Microsoft state that ‘users will no longer be able to access the document from its original location’, suggesting it is not actually deleted from the original location)
  • Move and leave a link (see below)

Microsoft state (in the link above) that this option will ‘… delete the document from its current location, move it to the destination repository, and leave a link at the current location indicating that the document has been moved’. If Document IDs have been enabled, the Document ID will disappear next to the document icon, which then shows a hyperlink ‘hook’ to the document in the Records Centre. A user clicking on that link will be advised that the document has been moved.

The impact of versioning, and metadata, on ‘Send to’ moves

There are a number of issues, and potential problems, with the ‘Send to ‘ move option that need to be understood. Some of these might be sufficient to decide against using the Send to Records Centre option (or even using the Records Centre).

  • Versioning. If versioning has been applied to the document library, and versions exist, only the most current version will appear in the Records Centre. The Document ID will be the same as the current version. However, any previous versions remain in the original library. In one sense, it is good to show previous versions, however any of these previous versions can be restored. When this happens, the version that is restored now appears, with the original document number (and a ‘hook’ still on the icon). The logic of this appears to be that, if a previous version is restored, the version that has been sent to the Records Centre is no longer the current version. However, it remains – with the same Document ID – in the Records Centre.
  • Metadata. Any additional, local metadata that has been added to the Content Type in the original Site Library will disappear when the document is moved. Metadata in Content Types stored in the Content Type Hub will remain.
  • Created date. The ‘Date Created’ field in SharePoint is the date the document was created in SharePoint. When the document is sent to the Records Centre, it is assigned a new ‘Date Created’. The date created field on the original document remains unchanged.
  • Permissions. Any permissions that may have been applied to the document, via inheritance from the Library or Site, are removed and the document inherits the permissions of the relevant library in the Records Centre.

The implications for ‘Send to Records Centre’

The issues documented above could affect planning and decision making regarding use of the Records Centre and the ‘Send to’ functionality. Some issues to consider include:

  • Loss of versions when the document is moved
  • Ability to restore a previous version (and confusion about which is the current version)
  • Loss of metadata applied locally to content types
  • Changes to access permissions

If any of these are concerns for the organisation, consideration might be given to leaving documents in place where they were stored and managing them in that way. The Record Centre could instead be used to archive and apply retention policies to content from network drives.

Recordkeeping functionality in SharePoint 2013 – What’s New?

September 9, 2012

By all accounts so far, SharePoint 2013 does not have a lot of new records management functionality, but the new functionality it now has is an important step forward.  It includes:

  • Records management in the cloud, but a parallel universe to existing on-premises deployments of things like the Records Centre, Content Type Hub, and Managed Metadata.
  • The ability to apply retention policies to entire sites.
  • Better integration with Outlook, including site mailboxes.  All other email will still need to be managed.
  • Better support for eDiscovery.

Records management features in Office 365

It will now be possible to do most on-premises records management activities online in Office 365 (i.e., in SharePoint 2013 online). This includes the ability to create and use Records Centres, Content Type Hubs, Managed Metadata, and in-place records management. (See Don Lueders’ interview with Adam Harmetz from Microsoft here for more information.)

However, as Mike Alsup from AIIM points out here, on-premises and cloud-based environments will remain completely separate. It will not be possible to send records from Office 365 to the on-premises Records Center, and vice versa.

Retention policies for entire sites

This is a welcome feature as the only realistic alternative in SharePoint 2012 was to make a site read only and then ‘archive’ it.

See Microsoft’s introduction and overview to this issue for more information.

Better integration with Outlook

This is a very interesting move by Microsoft who have made it very clear that they believe content should be kept where it belongs – email in Exchange and documents in SharePoint.

In SharePoint 2013, teams can have a site mailbox that is visible from Outlook as well as the site. Documents attached to emails can be saved easily to a SharePoint document library. With retention applied to an entire site, this means that the emails in the site mailbox will be kept – in its business context – along with the rest of the relevant content.

This leaves ‘Messaging Records Management’, a feature that has not changed in Exchange 2013, to cater for all other emails.

Better discovery

SharePoint 2013 includes new eDiscovery sites. Some might say this is not a records management feature but a feature for lawyers. In any cases, both are likely to find this new feature compelling.