Posted in Products and applications, Sharepoint 2010, SharePoint 2013, SharePoint Online

Using color-coded calendars in SharePoint to manage resources

Color-coded calendars in SharePoint are a simple to create, visually appealing, popular and easy way to manage resources that, for whatever reason, cannot be managed using Outlook calendars.

A common use case for this option is to manage pool cars in ‘local’ business areas (as opposed to the entire organisation). This avoids having to add tens of pool cars as resources in Exchange and allows business areas to manage the resources independently.

What is a color-coded calendar?

A color-coded calendar is a SharePoint calendar that allows end users to view calendar entries visually. When configured (usually takes <30 minutes for 10 resources), color-coded calendars look something like the view below.

O365_SPO_ColorCodeCalendar1.JPG

Why not use Outlook calendars?

As noted above, resources can be created in Exchange so they appear in Outlook (and can also be used in SharePoint calendars). This may be fine for a small business, but for larger companies (we had 9,000 geographically dispersed staff), adding and managing (a lot of) vehicles for ‘local’ business areas was considered too much.

One of our business areas used color-coded calendars for multiple purposes:

  • To book meeting rooms. Each meeting room (which had its own calendar) used color-coding to indicate if the meeting was for internal or external staff.
  • To book and manage pool cars.
  • To book and manage other pool vehicles (such as minibuses).

How to create a color-coded calendar

The three steps to create color-coding in a SharePoint calendar are described below. If you plan to create these types of calendars, it is recommended that you create a dedicated calendar for the purpose, e.g., ‘Pool Car Bookings’. In summary the steps are:

  • Create multiple choices in a SharePoint calendar choice column.
  • For each choice, create a view.
  • From the master calendar view, create a calendar overlay for each view, changing the color for each.

Keep in mind that these calendar can only have ten calendar overlays, each of which maps to a choice in a drop down list.

Step 1 – Create the choice options

  • Go to the calendar list settings and scroll down to the column settings.
  • Change the name of the choice column ‘Category’ to the name that you want to use – e.g., ‘Pool Car’.
  • Change the choice options to the options you want the users to select from, for example ‘XYZ123 Red Toyota Corolla’. Try to keep these with a consistent naming structure. Record each of the options in Notepad or some other text editor as we will come back to these shortly to add the list view URLs.
  • Remove the option in this column for end-users to select their own value.
  • Click OK.

While you are in this section, you might consider changing the default ‘Title’ column to ‘Driver’, and the default ‘Location’ column to another value. See below for how to change the order in which these options will appear.

Step 2 – Create the list views

Create a list view for each of the choices in the list column in Step 1. There are a couple of ways to do this, I suggest this one for anyone doing this for the first time. Another way to do it is via the ribbon menu.

  • Go to the calendar list settings and scroll down to Views. The default options are ‘All Events’ (which shows the calendar items in a list), ‘Calendar’, and ‘Current Events’.
  • Click on ‘Create view’ beneath the existing options and repeat this for each new view.
  • Click on ‘Calendar View’.
  • Give the new view a name. As this view is generally not going to be used, the name can be a short version of the choice option, without spaces, e.g., ‘YYZ123RedCorolla’. This will ensure that you get a clean URL without any spaces. URLs with spaces can be the cause of problems if the URL is copied but it doesn’t copy past the first space.
  • Scroll down to the Filters section and change the filter to sort by the Pool Car (column) to equal the exact name in the choice column – ‘XYZ123 Red Toyota Corolla’. (The example below shows a different choice option, for demonstration only).

O365_SPO_ColorCodeCalendarFilter

Step 3 – Create the calendar overlays

Now return to your calendar main view making sure that you have ‘calendar.aspx’ in the URL.

https://tenantname.sharepoint.com/teams/sitename/Lists/PoolCars/calendar.aspx

IMPORTANT note – It is quite easy to accidentally open one of the views you just created and create calendar overlays there. If you do this, you may create the calendar overlays ‘under’ that view instead of the main calendar view. If you do this, you will need to go back to the calendar view and delete all those calendar overlays.

Assuming the URL now has ‘calendar.aspx’, following the next steps.

  • Click on CALENDAR in the ribbon menu. (Note, you can create the views above here as well). The ‘Current View’ should show as ‘Calendar’.

O365_SPO_ColorCodeCalendarMenu.JPG

  • Click on ‘Calendars Overlay’. In the next screen, click on ‘New calendar’. You will repeat this process for each of the calendars you create.

O365_SPO_ColorCodeCalendarOverlay

  • For ‘Calendar Name’, copy the same text as the choice. e.g., XYZ123 Red Toyota Corolla. Leave the option as ‘SharePoint’.
  • In the Calendar Overlay Settings section:
    • Enter description if required.
    • Choose a color. You have 10 colors to choose from and these cannot be changed.
    • In the ‘Web URL’, delete the existing URL and copy the URL of the view that matches the name. Click ‘Resolve’ and wait a second or two.
    • The ‘List’ drop down should now show the Pool Car calendar name (e.g., PoolCars’); if it doesn’t, click the down down and select it.
    • In the ‘List View’ drop down, choose the List view (which should be the same as in the URL you pasted in Web URL).

After you have created each calendar overlay, the main calendar should now show each choice in a color box.

One last step – very easy to forget!

After you have created each calendar overlay, the main calendar is still ‘visible’. This means that any time you add a choice option, the calendar will show both the ‘master’ calendar entry as well as the colored-calendar entry.

To remove the master calendar from view, go to the primary ‘calendar.aspx’ view, click on CALENDAR, and click ‘Modify View’ in the ribbon menu.

Scroll down to the Filters section and change the choice to show only if the ‘PoolCar’ column is equal to ” ” (blank). As only completed items will be displayed, this removes any pool car from the master calendar view.

How to change or remove a choice option

Removing a choice option is the exact reverse of the process described below:

  • Go to the calendar overlays, click on the overlay and click on ‘Delete’.
  • Go to the calendar list settings, scroll down to the list views, click the list view, and click on ‘Delete’.
  • Go to the calendar list settings, scroll down to the columns and click on the choice column. Remove the choice option.

Changing a choice option is possible but you must:

  • Change the choice option. (Keep a note of the choice)
  • Change the view name and filter in the view. (Keep a note of the view URL)
  • Change the name of the calendar overlay, delete the existing view URL and replace it with the new one, click ‘Resolve’, and fix the List and List view options.

It is VERY easy to break these calendars if you don’t follow the exact steps. On the other hand, these are very easy to troubleshoot. The main ‘problems’ that arise are when a Site Owner or end-user with contribute permissions:

  • Doesn’t create the overlays from the main ‘calendar.aspx’ view.
  • Changes or adds a choice column without changing or creating a new view and calendar overlay.
  • Changes or adds a view with or without changing the choice column.
  • Changes a calendar overlay but forgets to click Resolve or doesn’t select the correct options.
  • Forgets to change the main ‘master’ calendar view to filter by ‘pool cars’ is equal to ‘blank’.

Four important additional points to note:

  • Removing or changing a choice option does NOT remove the option retrospectively. This means that all previous history of that vehicle
  • If you enable versioning on the list (via list settings), you will have a record of changes made to the list.
  • You can display (and edit) the complete listing via the ‘All events’ view.
  • Train your users to:
    • NOT use the ‘All Day Event’ option as this carries the booking to the next day. Instead, remind them to choose ONLY the timeframes they need.
    • NOT use the ‘Recurrence’ option. However, if you decide to use this, train users to not to change the option from ‘No end date’ to an actual date.

It is possible to add a simple formula to a calendar column to prevent bookings being made more than x days in the future.

How to change the order of the options that appear

The default calendar data entry fields are listed below. To change the order, click on the calendar’s List Settings, scroll down to ‘Content Types’ and click on ‘Event’.

The ‘Column order’ option appears at the bottom beneath the list of columns. Note that you cannot remove the ‘All Day Event’ or ‘Recurrence’ option.

O365_SPO_DefaultCalendarEntry

Enjoy color-coding, or for my British and Australian readers, colour-coding.

Posted in Access controls, Electronic records, Office 365, Office 365 Groups, OneDrive for Business, Products and applications, SharePoint 2013, SharePoint Online

Migrating to SharePoint Online Part 2 (implementation)

In my previous post I described what we did to prepare for the migration of our SharePoint 2013 (SP2013) environment to SharePoint Online (SPO). In this post I describe the process we undertook and the lessons we learned along the way.

By August 2017 we had around 245 SP2013 sites across seven web applications: apps (13 ‘purpose-built’ sites); intranet (1 site); ipform (an old site that was closed several years back); projects; publication (30 sites); sptest (used for testing sites); and team.

The bulk of our sites (around 210 sites split almost equally) containing most of our corporate records were in either the teams and projects web applications.

The details of our root sites were recorded in several key artefacts:

  • A SharePoint Online list in our SharePoint Admin site, used for new site requests. This was always our ‘master’ listing of sites and included a range of additional metadata, including the business owner and a ‘yes/no’ if the site had been migrated or not. This was one of the first sites we migrated.
  • Another SharePoint list that recorded the details of site collections and subsites.
  • An Excel spreadsheet used to ‘map’ of all our root sites (one per cell_ grouped under business areas. This map provided a simple, printable visual map of all our SP2013 sites grouped by business area. We used colours to indicate when sites were migrated to SPO, providing an immediate visual reference.

Configuring and learning about Office 365 Admin and SharePoint Online (and OneDrive) admin

By August 2017 we had access to our Office 365 tenant admin environment, access to which is required to get to the SharePoint Admin portal initially (subsequently, the SharePoint Service Administrator could access it directly).

After setting up and configuring the Office 365 elements and SharePoint Online (SPO) environment, we created some initial test sites (via the Admin portal) to understand the new environment.

One of the early SPO sites was a re-created SharePoint User Group site, used to store general training and other useful information about the new environment. This site has remained our primary go-to point for all SharePoint users.

Our SharePoint developer also re-created various scripts, including scripts to automatically create and configure new sites from a request in a SharePoint Online list.

Monitoring changes in Office 365 and SharePoint Online

We learned the importance of monitoring – daily – the Office 365 message centre and also the Microsoft techcommunity site (which had been moved off a Yammer platform a year or so earlier), as well as the Microsoft Office 365 roadmap to ensure we were aware of any likely changes – many of which were introduced during the time we were migrating.

We quickly learned a few things:

  • Customisation was not a friend of migration. Fortunately, almost none of our sites were customised. However, page-based content would need to be re-created.
  • Any complex workflows, integration or data extraction (e.g., ETL for business intelligence purposes which needed to be re-linked) could delay migrations. As it turned out, these sites ended up at the very end of the migration process and a couple were still waiting for migration as at the date of this post.
  • New ‘modern’ sites based on Office 365 Groups needed careful planning to get them right early on. We decided that any request for an Office 365 Group would go through the same request process as SharePoint requests.

Towards the end of 2018, when they were introduced, we also learned that hub sites were preferred over subsites.

Project management

While the migration of SharePoint sites was included as part of an internal IT project, that project was focussed for most of the first year on a range of other core networking elements including the broader network architecture model and high level designs required for our new cloud environment.

It would only later start work on the migration of Exchange mailboxes to Exchange Online and personal drives to OneDrive.

SharePoint migrations continued through the life of the project.

Migration tool

There were a number of options to migrate our SP2013 sites to SPO. We decided against going with an external provider for a number of reasons and instead – after reviewing the market – acquired the ShareGate migration tool in September 2017.

Final architecture

By September 2017 we had finalised the architecture for our SPO environment. As we had been using web applications in our on-premise environment we needed to ‘map’ this to the new environment.

The new model was based on the following:

  • All team and project sites would be created under the ‘/teams’ path. Project sites would have the prefix ‘PRJ’ to identify them. (Some would also be created as Office 365 Group-based sites, with ‘O365_PRJ’ as the prefix). ‘Team’ sites had to be based on a logical business unit or team.
  • All other sites, including communication sites and sites that crossed over multiple business areas would be created under ‘/sites/’.

SharePoint migrations were ready to go.

First batch

Our earlier analysis indicated that around 50 of our SP2013 project sites were inactive (because the projects had since closed). As no-one was accessing any of these sites we decided to use the ShareGate tool to test the migration process and learn from the experience.

We migrated 51 project sites in October 2017. The migrations initially took place during the day but we then changed to an early morning migration (before 9 AM usually) to avoid any network traffic issues.

First lessons learned

The ShareGate tool worked as expected, and proved to be a very useful tool for other reasons too, such as moving libraries and lists between sites.

The early batch of SP2013 sites were migrated ‘as is’ in terms of their look and feel. They looked exactly the same but were now in SPO. That is, they did not get the new ‘modern’ page look and were not mobile friendly. This didn’t matter too much as the sites were rarely accessed.

After the first batch, we did the following for all new sites:

  • Added a new page (named ‘home2’) and swapped over the old ‘classic’ page (renamed to ‘homex’) with the new one.
  • Edited the replacement home page and add any text/images from the old site home page.
  • Fixed up the left hand navigation; indented libraries and lists on old sites were now under a heading with a drop down menu option. In some cases we left them ‘as is’, in other cases we promoted the indented libraries via the left hand ‘Edit’ option.

For any sites that had the publishing feature enabled on the site collection and site settings, we disabled this on the SPO site post-migration as there was generally no need to have these settings enabled.

Some sites had Active Directory security groups in their SharePoint permission groups. As these were not migrated to our Office 365 environment (for multiple reasons, including the complexity of this legacy environment), these had to be added back in a different way to provide the same level of access. In almost all cases, existing SP permission groups (Owner, Member, Visitor) were sufficient. The primary one we had to re-create was the AD Group for ‘everyone’; this was replaced by the ‘Everyone except external users’ option.

The other factor we had to consider were Office 365 licences. By October 2017 few people had these licences. Anyone who needed to access SharePoint would need a licence, but these might not be issued to everyone until mid 2018. This limited the number of sites that we could migrate. By mid 2018, more or less anyone who accessed SharePoint 2013 before could now access the SPO environment.

Next batch – to end June 2018

From November 2017 to end June 2018 we migrated another 57 sites, including a further 16 inactive project sites in December 2017. The primary reason for the low number was (as noted above) the need for staff to have Office 365 licences, which were not allocated more broadly until mid 2018.

At the same time, however, we were also starting to create a range of new SPO sites, including Office 365 Group-based sites and new communication sites.

Publication sites re-created as communication sites

Almost none of our publication sites could be migrated ‘as is’ to SharePoint Online because the page-based content, while it could be migrated, was not in modern pages or mobile friendly.

Accordingly, it was decided to re-create all the publication sites as SPO communication sites. In almost all cases this was a relatively simple process of creating pages and copying content from the old to the new.

Our intranet was the only except to this process and, as of end 2018, remains as a SP2013 site because of heavy customisation.

Other project impacts

From July 2018 two key projects impacted on the SharePoint migrations.

The first was the roll-out of new Windows 10 devices. While a Windows 10 device was not required to access SharePoint, some of our older Windows 7 devices had Internet Explorer 9 that had issues with SPO. These users were asked to use Chrome instead.

The second related project work (which was part of the overall Office 365 project that included SharePoint migrations) was the migration of Exchange mailboxes and personal drives to Exchange Online and OneDrive respectively. This part of the project encountered a problem with Windows 7 devices and as a result that part of the project activity was delayed.

Final migrations – from July 2018

From July to the end of 2018 we migrated all except around 10 of the remaining sites, at an average rate of around 20 per month. Many of these were simple migrations.

For each migration we followed the same process:

  • Engaged with the business area to provide awareness of, and where required, training in, the new environment. (By the end of 2018, this training included information about Office 365 and MS Teams to help site owners understand that SharePoint is not an ‘isolated’ product as it was before, but part of a much larger ecosystem)
  • Identified a suitable date and time to migrate the site (most of these were completed before 8 AM on a working day, but some were done over a weekend)
  • Alerted our Service Desk to the proposed change
  • Migrated the site
  • Made minor changes to the site (home page swaps mostly)
  • Made the old site read only with a pointer to the new site
  • ‘Released’ the migrated site to the business area, usually before 9 AM.
  • Provide post-migration support to the business area. In most cases the business area was able to use the new site immediately as the new site was very similar to the old one in look and layout.

The last sites to migrate included sites with complex workflows, integration or ETIL elements and several large, complex and sensitive sites.

New site request process

While we had always had a SharePoint-based site request form, the new environment meant that we needed an updated form. The (SPO) online form changed several times during 2018 as we learned from experience what worked and what didn’t.

The current form captures the following (not all columns/fields are listed):

  • Site Acronym (up to 12 characters – becomes the DocID prefix)
  • Site Type: (a) team or project site (no Office 365 Group), (b) team or project site (with Office 365 Group, (c) communication site
  • Approver
  • Business area owner
  • Owner/s
  • Member/s
  • Sensitivity (information security)
  • Suggested site URL

Each of these requests was reviewed by the SharePoint administrators who, if the site request is OK, then run a workflow for non Office 365 Group-based sites only. For Office 365 Group based sites, these were created by creating the Office 365 Group.

By the end of 2018 we had approximately the same number of migrated sites as new sites and our SPO environment was ‘live’ and active.

Almost all the old SP2013 sites were made read only. We expect that these will be deleted by June 2019.

Posted in Information Management, Information Security, Products and applications, Records management, SharePoint 2013, SharePoint Online

Migrating to SharePoint Online – Part 1 (Planning)

We implemented SharePoint 2010 in early 2012 and then upgraded to SharePoint 2013 in early 2015. After acquiring Office 365 enterprise licences in April 2016 we began to play for the migration of our existing on-premise environment to SharePoint Online. After testing the migration process with inactive sites, we started to migrate active sites from early 2018. We expect to complete all the migrations by 31 December 2018.

This post, the first of three, outlines the factors that influenced and guided how we approached the migration. Our approach may not be the same as your approach, but many of the basic principles may be similar.

Overview of our SharePoint environment pre-migration

A key principle for our SharePoint environment since 2012 was to avoid customisation and dependencies, and use the product ‘out of the box’ (OOTB) as much as possible.

  • Customisation would almost always require some degree of development and ongoing maintenance. It also meant that upgrades could be more complex and expensive.
  • Dependencies of any sort – be they integration components or third-party add-ons – could also make upgrades more complex and expensive.

Governance model

We also implemented a ‘balanced’ controlled environment, following the technical design models for SharePoint 2010 described by Microsoft (extract in image above), which recommended that organisations strike balance across three key governance elements:

SharePoint2010GovernanceBalance

Source: https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-server-2010/cc303422(v%3doffice.14)

  • IT Governance. Centrally managed or locally managed?
  • Information Management. Tightly managed or loosely managed?
  • Application Management. Strictly managed or loosely managed development?

In our environment, the ability to create new SharePoint sites and sub-sites required the completion of a (SharePoint) online form and was restricted to the SharePoint Administrators. This enabled us to prevent uncontrolled growth in the environment and to ensure that all new sites were created within a pre-defined – but not overly strict – architecture design model.

Upgrade to SharePoint 2013 in early 2015

Our SharePoint site collections were created across five web applications: team (approximately 120 sites), project (approx. 120 sites), publication, apps, and intranet. Most of the corporate records were stored in team or project sites, as well as a single ‘apps’ site. (Our apps sites (< 10) were set up to address small business problems that in the past might have been addressed by using Microsoft Access).

Thanks to our OOTB model, we were able to upgrade to SharePoint 2013 over a weekend, with almost no errors. The only site we could not upgrade was the intranet which remains (as at August 2018) in ‘compatibility mode’.

Note: It is not possible to migrate directly from SharePoint 2010 to SharePoint Online. It must be upgraded to SharePoint 2013 or SharePoint 2016 first.

The situation in 2016

In May 2016 we changed our Microsoft Enterprise Agreement to an Office 365 subscription model. Our reasons for going to Office 365 were driven by multiple factors, including the need for mobile access to information.

It is important to remember that SharePoint Online is only one element among many others in Office 365. That is, while it is technically possible to do it, SharePoint would not normally be migrated on its own to SharePoint Online. Any migration must take in account a range of considerations relating to the broader Office 365 environment, including (but not limited to):

  • Office 365 licences (and what this meant for our users with Office installed on existing computers which were being upgraded to new Windows 10-based devices as part of a separate project)
  • Active Directory syncing so users can access the environment.
  • Exchange mailbox migrations so SharePoint-based, email-linked Flow workflows can work.
  • OneDrive for Business, as a SharePoint service to replace ‘personal’ drives on network file shares.
  • Security controls and records retention policies, set from the Office 365 Security and Compliance admin portal, as well as audit logs in that same portal.
  • Office 365 Groups with associated SharePoint sites, Yammer groups (which can be linked with Office 365 Groups) and Microsoft Teams (which can also be linked with Office 365 Groups).
  • ‘Classic’ and modern team sites, Office 365 Group-based sites, and communication sites.
  • The SharePoint user portal.
  • The mobile app, and how sub-sites are accessed.
  • The ever-changing SharePoint Online environment in which anything described as ‘classic’ is likely to be deprecated at some point, and new features appear.

Migrating multiple web applications to one

We needed to plan our migration process, moving away from our five web applications to a new model. We new that, with the exception of our customised intranet, we would probably be able to migrate almost all of our sites relatively easily because we had always kept to the OOTB model.

Fortunately, Microsoft produced a very useful 12-page document which provided a good overview describing how it ran its own SharePoint migration, and good advice for how we might do our own migration.

SharePoint_to_the_cloud_MSpaper.JPG

Learn how Microsoft ran its own migration

We had a range of factors to take into account.

  • One of our initial decisions was not to migrate any active site until all Exchange mailboxes were migrated (and preferably, end-users had new Windows 10 devices). As it turned out, the decision to migrate mailboxes was delayed and as a result we would end up migrating most sites first.
  • We need to work out how to migrate our content as it was no longer possible to do a ‘lift and shift’. We investigated the market and made the decision to acquire a migration tool, ShareGate, to do the migrations ourselves. We would later find the same tool useful to migrate personal drives to OneDrive for Business.
  • We identified the likelihood that we would create new SharePoint Online sites in parallel with the migration of on-premise sites; this was partially because some existing on-premise sites with multiple sub-sites would be split into separate sites instead, but also because the new SharePoint was so much more versatile and would likely be popular.

The new architecture model

An important point to note is that the new SharePoint Online architecture model provided the opportunity to re-think our SharePoint model and, to some extent, clean up or leave unwanted SharePoint content behind. To quote the Microsoft site above, ‘the best migration is no migration’.

As noted above, we had five primary web applications in our SharePoint 2013 environment. These had to be migrated (or re-created, in the case of publication sites) under one of two paths (only – /teams or /sites) to one of three site option:

  • ‘Classic’ sites (the default for all team and project sites)
  • Office 365 Group-based team sites
  • Communication sites (re-created page-based content)

That is:

  • Migrated team and project sites would become classic team sites under either (a) /teams/sitename path or (b) /teams/prj_sitename path, respectively. There were some exceptions:
    • Some sites with multiple sub-sites would be split up into multiple independent sites (including using the new ‘hub’ sites).
    • A couple of team sites would become communication sites.
    • Team sites that crossed multiple organisational business areas would be created as classic team sites under the /sites/sitename path.
  • Most publication sites that used the publishing features would need to be re-created as communication sites under the /sites/sitename path. There were some exceptions:
    • Some publication sites would become team sites instead.
    • The intranet would be managed separately as, at the very least, it would need to be re-created in SharePoint Online. It could not be migrated ‘as is’.
  • Application sites would become team sites.
  • Some existing sites or sub-sites might be migrated to SharePoint sites linked to Office 365 Groups, with the naming prefix of either GRP_ or PRJ_.

The above ‘mapping’ model was an early decision that did not change.

Preparatory work

We also commenced work on the following elements of work:

  • Reviewing all existing sites to determine which sites would be migrated or discarded – see below.
  • Re-developing our SharePoint Architecture documentation for the Online version.
  • Investigating and documenting all Office 365 admin and Office 365 Security and Compliance admin configuration settings, and determining roles. This process, which required Global Admin access, included establishing records retention policies (from mid 2018) in the Security and Compliance admin portal.
  • Re-developing our existing SharePoint admin documentation for the Online version, including all the configuration settings. We included the OneDrive for Business config settings in this same document as it is a SharePoint service.
  • Understanding how the new environment worked, and would work.
  • Re-establishing our SharePoint Admin and SharePoint User Group sites in SharePoint Online.
  • We also created a range of ‘test’ sites to better understand the new environment.
  • Creating an initial schedule for the migration of sites, targeting inactive sites first.
  • Assigning the initial batches of Office 365 licences.
  • Developing a repeatable process to migrate sites using ShareGate. In our environment steps involved:
    • Identify need to migrate site
    • Register a new site request in our SharePoint Admin portal.
    • Register the task in our Jira task management system.
    • Create the SharePoint Online site (via a script linked to the request).
    • Migrate the on-premise site, make it read only with a re-direct notice on the front page (and a three month deletion notice*).
    • Prepare the migrated site, including swapping the classic default home page to a modern home page.
    • Hand over the site to the business owners and close the task

* In practice many of these sites still remained after 6 months.

As part of our review process, we identified around a dozen sites that had one or all of the following elements, that would mean we had to devote more time to their migration (‘custom workload’ in the Microsoft document above):

  • Complex workflows which would need to be re-created.
  • Integration with other systems (mostly via BizTalk).
  • Links with ETL processes.

We also identified around 50 sites that would not be migrated:

  • Sites that were unused or had no content of value (often because the original was still on a drive).
  • Sites that did not need to be migrated, for example if their content had been migrated to a different business system.
  • Test sites.

Sites that were no longer used but contained records that needed to be kept were to be migrated with the word ‘Archive’ to the end of the site URL name, assigned a site retention policy, and then made read only.

By August 2017, we had identified that 250 site collections would be migrated to SharePoint Online. We acquired ShareGate in September 2017 and were ready to start migrating.

In Part 2 of this series of posts I will describe the migration process and the lessons we learned along the way.

Posted in Classification, Delve, Electronic records, Office 365, Records management, Sharepoint 2010, SharePoint 2013, Yammer

Can ‘traditional’ records management survive the digital future

At a conference last year I listened with interest to a panel discussion on the subject of ‘paper versus digital’. Perhaps the debate was only meant to be light-hearted but it was clear that quite a few records managers continue to manage paper records and see them as their primary focus. In almost all cases these records are the printed version of a digital original. In most cases those digital originals remain stored on a network drive (or a person’s USB), or attached to an email, or – if you are lucky – copied to the organisation’s electronic document management (EDM) system to become the ‘official’ version.

Only a month ago I saw a colleague, whose position title clearly indicates she is responsible for digital recordkeeping, asking about KPIs for the creation of paper files. Perhaps it was part of the digital recordkeeping strategy to examine this subject, but it underlined for me the persistence of ‘traditional’ recordkeeping concepts, that documents belong in containers, that may be files or groups of files (or series etc). This is not just an issue for records managers, but for end users as well who continue to ‘think paper’.

It concerns me that records managers persist with the concept that digital records somehow ‘belong’ in digital files, often directly connected with the descriptive ‘business classification’ system of function and activity (in the sense that the file title or metadata may include the function and activity).

This way of thinking reinforces the concept that these records belong nowhere else, or might have no context outside the container, which of course is not necessarily true. It is, of course, possible to cross-link documents to a different container, but at what point is this a manual exercise? And where does it stop?

Part of the problem has been that digital systems mimic in their appearance and functionality the concept of a file or folder. We see them on network drives, in email, and in the containers of EDM systems. These folders or containers provide a sense of surety, that a document has been ‘stored’ or saved in a specific file. The file, however, is no more than a visual construct; the document’s file/folder/container metadata is what causes the document to appear that way.

This is all the more obvious in some EDM systems that store documents in a file store (often folder-based) and the metadata about the documents in a separate database. Aside from the folder structure, these documents are more or less a bunch of uncontrolled documents with metadata links to the ‘file’ construct in the database.

Of course, keeping records in some form of business context is important both for the management of those records (short and long term) and for retention and disposal purposes. But that doesn’t necessarily mean that documents must be associated with ‘files’ that, according to some records managers (and paper filing theory), should contain no more than 300 documents.

All of this is unnecessarily restrictive and, for end users, results in additional work. It’s no wonder that EDM systems have lagged behind the more ‘traditional’ way that users store documents, in network drives.

We should, I believe, think of digital records as being self-contained and dynamic objects with their own metadata payload that may be relevant in multiple contexts, not a fixed object stored in another fixed object.

Enter Microsoft Groups

In 2015 Microsoft announced the concept of Groups, accessed primarily from Outlook. Conceptually, you could think of a Group as being more similar to an Active Directory Distribution List rather than a shared mailbox. It is also similar to Yammer groups, but with much more functionality.

From Outlook, a user can create a Group to work collaboratively. Group members can initiate conversations with other members of the group (including Skype discussions) and – importantly – share documents.

So, Microsoft is taking us into the world of Group-centred information collaboration. This is not an experiment on the part of Microsoft, it is part of the world of Office 365 which includes SharePoint Online and OneDrive for Business, Delve, and more, supported by any device to enable users to create, access and share content anywhere, anytime.

Now, not every organisation uses Microsoft products, but an awful lot do. And for those who do, Groups in Office 365 is the future. The impact on traditional recordkeeping practices is likely to be large.

By 2020 or earlier, users will create Groups via Outlook to collaborate. Within the Group they will store digital content (not just documents); documents are stored in SharePoint Online site collections which are in addition to any other Site Collections. That’s right – in addition to your controlled Site Collections, there will be a multitude of Group-specific Site Collections. Conversations that take place in the group are stored as items in the inbox of the Group mailbox. In other words, all the content (and perhaps context) will be based around the Group ‘subject’.

To access all this rich content users will have three main options: (a) be an active member of the group; (b) SharePoint search; and (c) Delve. Business intelligence and data analytics, or e-Discovery type products, may also provide a fourth line of access.

Likely impact on recordkeeping

The recordkeeping practices that I believe will be most affected are: (a) metadata, including the application of BCS terms; (b) container-based storage, and (c) retention and disposal. It may also impact the careers of records managers. There will be no control over the creation of new groups (there can be, but why?), so their information content (in Outlook and in the associated SharePoint Online site collection) will become, in many respects, the new containers for records, in addition to other site collections.

On a positive note, I believe users will adapt and adopt this new model of working rapidly. Users, many of whom remain glued to Outlook as a business ‘collaboration’ tool, will find that Groups provide them with the ability to store and share documents via the Outlook interface. They may need to get used to the idea of ‘working out loud’ in Groups, but I think that will happen fairly quickly.

One thing is sure – change is inevitable.

Posted in Electronic records, Information Management, Semantic Office, Sharepoint 2010, SharePoint 2013

Folders uninterrupted – the enduring problem of folders

In September 2014 I presented to the Records and Information Management Professionals Australasia (RIMPA)’s annual conference in Adelaide, South Australia, on the subject of folders and how it is possible to manage digital content differently using products like SharePoint.

Last week a couple of people in completely different parts of the company I work for demonstrated clearly the problem of folders on Outlook and network drives. The first had at least 100 folders and sub-folders in her Outlook where she said she ‘filed’ everything so she could find it again. The other person showed me a newly created section of the network drives with hundreds of new folders and sub-folders set up by someone who didn’t like or get the existing agreed folder structure that has been in place for many years.

In both cases I asked the same question – how do you find anything? The answer, in most cases, is that the person clicks on folder names that they created or are familiar with, until they find what they are looking for. Search was usually relegated to last ditch attempt to find things but was not considered to be very helpful or accurate. Both individuals lamented the duplication of information in folders, duplication they often didn’t know about until the inevitable clash of versions that occurs regularly.

At a workshop/presentation I did for another organisation recently, I was told that the organisation relied heavily on what are called ‘titling conventions’ to enable the information to be found. That is, the names of files, folders and documents follow a pre-defined model.

In reply to my comment that such titling was surely less valuable when you could search for the content in their document management system, I was advised that they didn’t yet have the ability to search by the content of documents (a resource issue). They could only search for content based on titles, once again reinforcing the need for folders (with unambiguous and consistent titles) to find that information.

Breaking the habit

I have said many times, folders are a very hard habit to break. Given the generally poor ability to search and find content in email and network drives (and in some EDRM systems – still), search is relegated to second place and folders and pre-defined titling conventions continue to dominate where content searching is not available.

For at least fifteen years, working across a range of organisations, I have had the ability to search within the content of digital records. This includes being able to search for the content of scanned documents that have been subject to OCR or optical character recognition. In this digital age I’d be very disappointed if the only way I could find content was by navigating labyrinthine folder structures on email or network drives.

I have fewer than five sub-folders under my work email inbox. I use those additional folders to route specific content such as junk mail, SharePoint notifications or personal email. I don’t save anything to a network drive and only use SharePoint to store and retrieve records and other corporate information. Why? Because search is really good.

I know this is not common practice around the organisation. Common practice is to create and maintain multiple folders and squirrel useful content away in network drive folders. It’s been that way for 30 years.

I don’t think folders are going to vanish from the landscape anytime soon – they are no more as unlikely to vanish in the next 10 years as email, although both are slowly morphing over time into something different, particularly as ‘collaboration’ tools, search, and the presentation of information via analytics engines are introduced.

Content finds you

Facebook and Google are successful for a reason – they both present content (including advertising) relevant to the user based on her or his digital content. A simple analogy is to compare the corner store where you know you can go to get milk with the ability of digital devices to remind a user they need to buy a specific type of milk because (a) their internet linked refrigerator has noticed that milk is running low and (b) they regularly post to social media (or even in a single email) about their preference for this type of milk. It’s creepy, but it works .

Many contemporary office environments are not much different – users know they find their digital content in the folders (the little corner stores) they created or have learned about. But they don’t know how much other interesting or directly relevant content is there and they don’t bother looking because they know it will be a poor and time-wasting experience.

They don’t know what they don’t know.

My experience over the past five years with SharePoint (and with similar products before it) is that users are resistant to ‘having’ to store documents in a different location to what they are used to (i.e. network drives).

Users worry their content will be lost, it’s ‘yet another place’ to store and find content, and the user experience (compared with folders) is alien. This doesn’t stop them consigning endless information to social media and being the recipients of targeted advertising and suggested friends based on their content. The difference between the two is so extraordinarily different it is sometimes hard to believe that users do this, checking in on Facebook’s content-driven feed while ‘navigating’ through endless folders, many with names that are about as vague as you can get.

Where is this heading

In the last weeks Microsoft has outlined what is sees as the future of the office and their Office. Three key elements of that future are:

  • SharePoint (on-premises and online) and OneDrive. These two will not replace folders completely but will give users a much better (and more importantly, more mobile) experience with storing and finding content.
  • Office 365 Groups. Groups of people who have a common interest and can communicate across platforms, whether it be via email or other collaboration tools including SharePoint or Yammer.
  • Analytics as the way we will harness and access our information, via interfaces like Delve that serves up relevant information based on the user’s digital content. Just like Facebook.

Does Delve deliver on the ‘Semantic Office’?

Many years ago I wrote on this blog about what I called at the time the ‘Semantic Office’. In that article I asked whether we will one day experience a time when we can truly harness the digital content in our business systems to link and present relevant information to users based on the same methods found on the internet at that time (for example in eBay, Amazon etc).

I think Microsoft has finally figured this out with Delve. My guess is that, by 2020, Delve or a version of it will be the most common ways users first access their content.

So where does this leave folders?

Folders aren’t going to go away anytime soon. I’d like to think that Microsoft took a close look at how they could deprecate folders in email and drives and realised that this was akin to giving up on email – it wasn’t going to happen.

Instead, Microsoft decided to introduce new tools that can capture and – more importantly, perhaps – present information in completely different ways that digital natives expect and even take for granted. As someone said recently, when a large group of young people were asked to send an email, many responded with ‘what’s email, why can’t I Inbox you?’.

Change is happening, slowly, along with the culture shift that must accompany it. For every older ‘paper native’ worker that retires, a younger digital native commences work in the workforce, bringing with them new expectations both in terms of work experience and tools they use.

Just like the clacking of manual typewriters in offices was eventually replaced by the slightly smoother sound of electronic typewriters, and then the click-clack of keyboards hooked to the back of computers, the silent touch screens of the future will eventually dominate and then themselves be replaced by something else.

Habits are hard to break, but change happens.

Posted in Electronic records, Governance, Information Management, Office 365, Records management, Sharepoint 2010, SharePoint 2013

Managing documents in SharePoint 2013 and SharePoint Online – what’s changed?

We are in the process of upgrading our extensive ‘out of the box’ SharePoint 2010 environment to SharePoint 2013 and, at the same time, setting up a limited SharePoint Online presence. So, what’s changed in relation to managing documents as records?

In SharePoint 2013, the short answer is ‘not much’. However, there are some new things that will change some parts of our technical design model. The things that have remain unchanged include:

  • Document libraries as the primary ‘container’ to store documents, using folders, document sets, or metadata-based categorisation, to ‘group’ documents.
  • Document IDs, set in the Site Collection Administration – Document ID settings section.
  • Document versioning (major/minor, major, or none), set in the Library Settings – Versioning Settings. Other settings in this section, which we generally do not use, have also not changed: Content Approval, Draft Item Security, and Require Check Out.
  • Use of Content Types (and disabling of Folders), enabled via Library Settings – Advanced Settings, and then added in the general Library Settings – Content Types area. All other options in this section, which again we rarely use, are still there: Custom Send to Destination, Search visibility, Offline Client Availability, Site Assets Library, and Dialogs. The old familiar ‘Datasheet view’, which we use to ‘bulk upload’ or update metadata has been renamed ‘Quick Edit’.
  • Almost unlimited metadata options via pre-defined Content Types or library-specific columns, both of which can point to the centrally controlled metadata in the Managed Metadata Service or local ‘look-up’ lists.
  • Multiple list views, each with their own linkable URL.
  • The ability to copy documents, via drag and drop or copy/paste, using ‘Open with Explorer’. This, coupled with the ‘Quick Edit’ option, allows documents to be copied to SharePoint document libraries in bulk and metadata added easily.
  • Access controls that can be applied right down to the document level.
  • The ability, from a document-specific drop down menu, to view or edit the properties of a document, check it out or in, view the version history (and restore versions), run workflows, download a copy, share (the former ‘manage permissions’ – see below), and delete (where enabled).
  • Out of the box simple workflows for Review and Approval.
  • Site collection audit trails, accessed via the Site Collection Administration area. Unlike some other products, SharePoint audit trails are not ‘attached’ to individual documents, but are centralised in one place.

So, all in all, not much change really, except for the ‘Share’ option. In many respects, the way the simple ‘Share’ has been designed is a more intuitive process than ‘Manage Permissions’. When you add a user you decide if that person should have Edit or Read privileges. If you maintain the default Owner, Member and Visitor groups, then those with Edit rights are added to the Member group, those with Read rights are added to the Visitor group.

For more complex permission management, including to stop inheriting the default permissions (or to add them back, which is now called ‘Delete unique permissions’), or creating new groups, you need to select the ‘Shared With’ option from the drop down menu and then select Advanced.

What about disposal/disposition management?

Again, not much has changed in relation to document libraries or lists. Out of the box, your options are as follows:

  • Using centrally defined, document-based, Content Types, using Information management policy settings. Not a bad idea if (a) you have a way to ensure that these Content Types are always added to libraries, and (b) you are happy to manage the disposal of documents one by one.
  • Changing the default Information management policy settings option in document libraries from ‘By Content Types’ to ‘By Libraries and Folders’ and then applying retention policies on the folders you create. The main negatives of this option are that it means you have to use folders, and you have to manage disposal by document.
  • Leaving documents in document libraries, and having a way to review these, across the farm, in a centralised manner. This requires some kind of script to be written to (a) list all libraries across the farm and (b) work on the basis that the ‘Last Modified Date’ is the last action on any document in the library, but it seems the more logical and simplest way to achieve the outcome you seek, and keeps all the documents in the same container.

It remains surprising to me why Microsoft does not provide the option to set a disposal period on an entire document library.

Of course, SharePoint 2013 now allows you to set a disposal period on a Site, but this isn’t likely to work for sites that contain a range of diverse content that may be useful over a long period of time.

So, what about SharePoint Online?

The first thing to remember about SharePoint Online (SPO) is that it’s not SharePoint On-Premises. Seems obvious, but the natural instinct is to wonder if or how the two environments can be connected. In most cases they can’t, so it’s not worth thinking along those lines. SPO is a way to manage content in the cloud in addition to, or instead of, on-premises.

SPO has all the same document management features you find in SharePoint On-Premises, described above – document IDs, versioning, content types, metadata, multiple list views, open with Explorer and Quick Edit, access controls, document-specific menu options, simple workflows and audit trails.

O365_SPO_LibraryRibbon_DocMenu

The options for disposal/disposition management using SPO ‘out of the box’ (should that be ‘out of the cloud’?) is the same as for the on-premises version.

You didn’t mention Records Centers …

Records Centers (or in Australia, Centres) were in many respects designed to be the ‘send to’ archival repository for other sites. Great idea if it works in your world, it doesn’t work in ours. The main drawbacks are that documents ‘sent’ to a Records Centre are in fact copied. Custom metadata is lost, versions are lost, audit trails are lost. And, you can retrieve the document.

But Records Centres (or in the US, Centers) can be useful on their own as a repository of specific types of records that aren’t accessed too much. We have several Records Centres and we use them in a separate web application for specific purposes, including to store scanned documents. We don’t use them as they were originally intended for the reasons stated above.

And yes, you can create Record Centers in SPO, too!

Posted in Products and applications, Records management, Sharepoint 2010, SharePoint 2013, Training and education

Implementing SharePoint to manage documents – changing user behaviours

From around 25 years ago, computers began to replace paper-based processes for most office-based workers. Users grumbled and some (mostly at senior levels) managed to avoid using a computer altogether. The majority of office workers (who became known as ‘users’) just adapted to and eventually adopted the new work practices, often with the promise of greater productivity.

Over the past 25 years, if no other system was mandated for the storage of digital records, users mostly:

(a) created new content using desktop applications and stored it on largely uncontrolled and poorly managed network drives; and
(b) sent or received emails (often with attachments from stored on drives) and used email folders to store those emails

… in both cases with scant regard to business or corporate recordkeeping requirements.

The end-result of 25 years of these practices is network drives and email systems full of digital ‘stuff’, little of which relates to each other in any meaningful way (unless it all happens to be stored in a single folder, or folder-based structure, which is not common).

Many electronic document management (EDM) products have been released over the years, ostensibly to improve the management of electronic records, but organisations still grapple with getting users to use them. I suspect that one of the reasons for the poor take-up of EDM ‘solutions’ is that users are so familiar with using network drives and emails folders; it is just so heavily ingrained into the way we work, not to mention the way we manage our own personal records and other digital content on home PCs and laptops.

Interestingly, some of the EDM vendors have developed interfaces that look almost identical with network drive folders. My own personal experience with these products is that users often don’t understand the difference, or the value proposition, that comes with use of these systems.

Along comes SharePoint

SharePoint has been around for about 10 years now, but best practice document management in the product did not really emerge until SharePoint 2010 was released (see my other posts for further information).

What is, for me, interesting about SharePoint is the design of the user interface presented in standard team sites via a browser. Microsoft clearly did a bit of research (and understands) how users work, with document libraries on the left (looking a bit like network drives), as well as a number of other list-based options such as tasks, discussion boards, announcements and calendars.

Not infrequently, the initial user reaction to team sites is one of uncertainty and some degree of confusion – is it the intranet?. Depending on how an organisation delivers team sites, user reaction can be either enthusiastic (so much better than drives/folders) or negative (prefer to use drives/folders).

SharePoint team sites as communities of information

For most of the past 25 years or so, ‘collaboration’ has meant exchanging emails with attachments and/or accessing the same part of a network drive, structured in a way that ‘makes sense’ to the user or the team.

SharePoint also allows users to ‘collaborate’, but quite differently from how users have done it to date. In some respects, SharePoint is more in tune with social media sharing (open access to all, or to a trusted group), than with traditional ‘closed’ methods (i.e., via limited email distribution, or limited access to parts of drives).

SharePoint team site can be used to store documents (and photographs) in document libraries (with unique document IDs, version controls, check in/out controls, out of the box workflows, audit trails and custom metadata). It can also use the rows and columns of data in spreadsheets and small databases and store and present that data instead in lists (including discussion boards, announcements, calendars, task lists and custom lists). As the volume of information stored in the site increases, search becomes an additional valuable resource that brings a range of information together. That is, instead of a list of documents, users can see a range of other contextual data.

A single SharePoint team site starts to becomes a community of information.

Changing behaviours

In my view, the key to getting users on board with SharePoint team sites is selling its benefits, not the technology, and helping users understand how to ‘map’ current practices to the new environment.

One of the best ways to demonstrate the benefits of using SharePoint is to show how it is being used (differently) by other teams, and to emphasise the ability for site owners (not a controlling authority such as IT) to use standard features (not customise*) in team sites to meet the needs of the team. That is, the team identifies and implements ways to use it that makes sense to the team, within a few broad constraints to ensure consistency in use across the organisation.

But, despite the obvious benefits, deeply ingrained user behaviours can be exceedingly difficult to change unless that change is driven by users (preferable) or mandated (may have limited success). The SharePoint team site paradigm *is* a major change from using network drives to store documents and emails (with attachments) to collaborate.

‘Mapping’ current practices to SharePoint team sites in many ways underlines the weaknesses of current practices, and can include things like:

  • Demonstrating the relationship between network drives and document libraries, using document sets or document categorisation to replace folders.
  • Describing convincingly why folders are not a good thing to use (despite the apparently similarities with network drive folders). For example, they can be very hard to navigate, do not have unique IDs (like Document Sets), can be replaced by Document Sets with longer names (including the concatenation of network folder names), can results in very long URLs and so on.
  • Showing how document metadata and views can completely replace the need for folders.
  • Showing how a team calendar (or calendars) can replace white boards or similar.
  • Demonstrating how an Excel spreadsheet can become a much more useful list of information, so that users can edit their own items rather than wait for someone else to stop editing the spreadsheet.
  • Showing how a team discussion can not only replace email but also keep a record of otherwise inaccessible email trails.
  • Demonstrating audit trails and site web analytics to show (and know) ‘who did or saw what’.
  • Demonstrating the use of hyperlinks instead of emails with attachments to share information.
  • Demonstrating features in document libraries that are absent from network drives (including unique IDs, versioning, audit trails).
  • Demonstrating out of the box workflow that remains with the document.
  • Demonstrating search, and in particular searches that return metadata-rich documents.
  • Showing how the Recycle Bin keeps deleted items for three months (which, coupled with the audit trails, help to retrieve those items much more easily than requesting the document back from a back-up tape).

and so on.

*By customisation, I mean changing the overall look and feel of a team site by modifying the theme, creating new types of pages, adding documents to pages instead of to libraries, using folders instead of document sets, and so on. Customisation can lead to confusion which in turn makes users less likely to use the site.