Migrating to SharePoint Online – Part 1 (Planning)

We implemented SharePoint 2010 in early 2012 and then upgraded to SharePoint 2013 in early 2015. After acquiring Office 365 enterprise licences in April 2016 we began to play for the migration of our existing on-premise environment to SharePoint Online. After testing the migration process with inactive sites, we started to migrate active sites from early 2018. We expect to complete all the migrations by 31 December 2018.

This post, the first of three, outlines the factors that influenced and guided how we approached the migration. Our approach may not be the same as your approach, but many of the basic principles may be similar.

Overview of our SharePoint environment pre-migration

A key principle for our SharePoint environment since 2012 was to avoid customisation and dependencies, and use the product ‘out of the box’ (OOTB) as much as possible.

• Customisation would almost always require some degree of development and ongoing maintenance. It also meant that upgrades could be more complex and expensive.
• Dependencies of any sort – be they integration components or third-party add-ons – could also make upgrades more complex and expensive.

Governance model

We also implemented a ‘balanced’ controlled environment, following the technical design models for SharePoint 2010 described by Microsoft (extract in image above), which recommended that organisations strike balance across three key governance elements:

Source: https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-server-2010/cc303422(v%3doffice.14)

• IT Governance. Centrally managed or locally managed?
• Information Management. Tightly managed or loosely managed?
• Application Management. Strictly managed or loosely managed development?

In our environment, the ability to create new SharePoint sites and sub-sites required the completion of a (SharePoint) online form and was restricted to the SharePoint Administrators. This enabled us to prevent uncontrolled growth in the environment and to ensure that all new sites were created within a pre-defined – but not overly strict – architecture design model.

Upgrade to SharePoint 2013 in early 2015

Our SharePoint site collections were created across five web applications: team (approximately 120 sites), project (approx. 120 sites), publication, apps, and intranet. Most of the corporate records were stored in team or project sites, as well as a single ‘apps’ site. (Our apps sites (< 10) were set up to address small business problems that in the past might have been addressed by using Microsoft Access).

Thanks to our OOTB model, we were able to upgrade to SharePoint 2013 over a weekend, with almost no errors. The only site we could not upgrade was the intranet which remains (as at August 2018) in ‘compatibility mode’.

Note: It is not possible to migrate directly from SharePoint 2010 to SharePoint Online. It must be upgraded to SharePoint 2013 or SharePoint 2016 first.

The situation in 2016

In May 2016 we changed our Microsoft Enterprise Agreement to an Office 365 subscription model. Our reasons for going to Office 365 were driven by multiple factors, including the need for mobile access to information.

It is important to remember that SharePoint Online is only one element among many others in Office 365. That is, while it is technically possible to do it, SharePoint would not normally be migrated on its own to SharePoint Online. Any migration must take in account a range of considerations relating to the broader Office 365 environment, including (but not limited to):

• Office 365 licences (and what this meant for our users with Office installed on existing computers which were being upgraded to new Windows 10-based devices as part of a separate project)
• Active Directory syncing so users can access the environment.
• Exchange mailbox migrations so SharePoint-based, email-linked Flow workflows can work.
• OneDrive for Business, as a SharePoint service to replace ‘personal’ drives on network file shares.
• Security controls and records retention policies, set from the Office 365 Security and Compliance admin portal, as well as audit logs in that same portal.
• Office 365 Groups with associated SharePoint sites, Yammer groups (which can be linked with Office 365 Groups) and Microsoft Teams (which can also be linked with Office 365 Groups).
• ‘Classic’ and modern team sites, Office 365 Group-based sites, and communication sites.
• The SharePoint user portal.
• The mobile app, and how sub-sites are accessed.
• The ever-changing SharePoint Online environment in which anything described as ‘classic’ is likely to be deprecated at some point, and new features appear.

Migrating multiple web applications to one

We needed to plan our migration process, moving away from our five web applications to a new model. We new that, with the exception of our customised intranet, we would probably be able to migrate almost all of our sites relatively easily because we had always kept to the OOTB model.

Fortunately, Microsoft produced a very useful 12-page document which provided a good overview describing how it ran its own SharePoint migration, and good advice for how we might do our own migration.

Learn how Microsoft ran its own migration

We had a range of factors to take into account.

• One of our initial decisions was not to migrate any active site until all Exchange mailboxes were migrated (and preferably, end-users had new Windows 10 devices). As it turned out, the decision to migrate mailboxes was delayed and as a result we would end up migrating most sites first.
• We need to work out how to migrate our content as it was no longer possible to do a ‘lift and shift’. We investigated the market and made the decision to acquire a migration tool, ShareGate, to do the migrations ourselves. We would later find the same tool useful to migrate personal drives to OneDrive for Business.
• We identified the likelihood that we would create new SharePoint Online sites in parallel with the migration of on-premise sites; this was partially because some existing on-premise sites with multiple sub-sites would be split into separate sites instead, but also because the new SharePoint was so much more versatile and would likely be popular.

The new architecture model

An important point to note is that the new SharePoint Online architecture model provided the opportunity to re-think our SharePoint model and, to some extent, clean up or leave unwanted SharePoint content behind. To quote the Microsoft site above, ‘the best migration is no migration’.

As noted above, we had five primary web applications in our SharePoint 2013 environment. These had to be migrated (or re-created, in the case of publication sites) under one of two paths (only – /teams or /sites) to one of three site option:

• ‘Classic’ sites (the default for all team and project sites)
• Office 365 Group-based team sites
• Communication sites (re-created page-based content)

That is:

• Migrated team and project sites would become classic team sites under either (a) /teams/sitename path or (b) /teams/prj_sitename path, respectively. There were some exceptions:
  • Some sites with multiple sub-sites would be split up into multiple independent sites (including using the new ‘hub’ sites).
  • A couple of team sites would become communication sites.
  • Team sites that crossed multiple organisational business areas would be created as classic team sites under the /sites/sitename path.
• Most publication sites that used the publishing features would need to be re-created as communication sites under the /sites/sitename path. There were some exceptions:
  • Some publication sites would become team sites instead.
  • The intranet would be managed separately as, at the very least, it would need to be re-created in SharePoint Online. It could not be migrated ‘as is’.
• Application sites would become team sites.
• Some existing sites or sub-sites might be migrated to SharePoint sites linked to Office 365 Groups, with the naming prefix of either GRP_ or PRJ_.

The above ‘mapping’ model was an early decision that did not change.

Preparatory work

We also commenced work on the following elements of work:

• Reviewing all existing sites to determine which sites would be migrated or discarded – see below.
• Re-developing our SharePoint Architecture documentation for the Online version.
• Investigating and documenting all Office 365 admin and Office 365 Security and Compliance admin configuration settings, and determining roles. This process, which required Global Admin access, included establishing records retention policies (from mid 2018) in the Security and Compliance admin portal.
• Re-developing our existing SharePoint admin documentation for the Online version, including all the configuration settings. We included the OneDrive for Business config settings in this same document as it is a SharePoint service.
• Understanding how the new environment worked, and would work.
• Re-establishing our SharePoint Admin and SharePoint User Group sites in SharePoint Online.
• We also created a range of ‘test’ sites to better understand the new environment.
• Creating an initial schedule for the migration of sites, targeting inactive sites first.
• Assigning the initial batches of Office 365 licences.
• Developing a repeatable process to migrate sites using ShareGate. In our environment steps involved:
  • Identify need to migrate site
  • Register a new site request in our SharePoint Admin portal.
  • Register the task in our Jira task management system.
  • Create the SharePoint Online site (via a script linked to the request).
  • Migrate the on-premise site, make it read only with a re-direct notice on the front page (and a three month deletion notice*).
  • Prepare the migrated site, including swapping the classic default home page to a modern home page.
  • Hand over the site to the business owners and close the task

* In practice many of these sites still remained after 6 months.

As part of our review process, we identified around a dozen sites that had one or all of the following elements, that would mean we had to devote more time to their migration (‘custom workload’ in the Microsoft document above):

• Complex workflows which would need to be re-created.
• Integration with other systems (mostly via BizTalk).
• Links with ETL processes.

We also identified around 50 sites that would not be migrated:

• Sites that were unused or had no content of value (often because the original was still on a drive).
• Sites that did not need to be migrated, for example if their content had been migrated to a different business system.
• Test sites.

Sites that were no longer used but contained records that needed to be kept were to be migrated with the word ‘Archive’ to the end of the site URL name, assigned a site retention policy, and then made read only.

By August 2017, we had identified that 250 site collections would be migrated to SharePoint Online. We acquired ShareGate in September 2017 and were ready to start migrating.

In Part 2 of this series of posts I will describe the migration process and the lessons we learned along the way.

Advertisements

Office 365 – Applying retention periods to SharePoint document libraries and disposal/disposition actions

Records retention policies are created in the Security and Compliance Admin portal, Classifications section of Office 365, as noted in my previous post of 9 March 2018 on the subject.

This post describes how these are applied to document libraries and what happens when the records reach their disposal/disposition period.

Note: In Australia we refer to the disposal of records. In the US this is called disposition.

Setting up retention policies

Organisations may have complex or quite simple records retention policies. An important point to keep in mind in Office 365 is how many policies should be displayed to the end user to choose from.

Ideally, there should be fewer than a dozen classes so they are easy to choose from (see below). There is nothing stopping you creating 100 or 500 policies, but all of them will appear in the drop down list to choose from. Microsoft say they are working on ‘grouping’ policies, so this may help to fix the issue.

For some organisations, it may be useful to distill or group retention policies down to a smaller number.

• For example, specific retention policies for certain types of records, and one (or two) for ‘all other’ records. The key, as we will see below, is naming them so they are obvious and easy to apply.

Viewing available retention policies

Retention policies that have been created appear in the Security and Compliance Admin portal, under Classifications > Labels.

Note: Labels must be published before they become visible to end users.

When you click on Labels, you can then see all the retention policies that have been created (but not necessarily published).

The screenshot below shows just the very top policy (a test/demonstration policy with a 7 day retention period) in a list of policies.

Note: Policies can be auto-applied, provided the policy has sufficient ability to identify what records they should be applied to.

Published policies appear in the Data Governance, Dispositions section:

The Dispositions section displays policies that have been published and are visible to end users in the Office 365 areas selected when the policy was created (e.g., Exchange, SharePoint, OneDrive etc).

Applying the policy in a SharePoint document library

To apply the policy to a SharePoint document library, go to the document library, library settings, and you will see the option to add the retention policy: ‘Apply label to items in this list or library’.

The ‘Apply Label’ dialogue shows the option to apply the label to existing items (recommended) and a drop down which shows all the published retention policies.

In this example below, there are four policies including the test policy.

The policy now applies to all records stored in that document library.

Managing disposal/disposition

When the records reach the end of the retention period configured in the policy, the person designated to be informed about the retention will receive an email notifying them of the need to review the dispositions.

Note, the person (or mailbox) receiving this email MUST be assigned to the Records Management role in the Security and Compliance Admin portal, Permissions section. No-one else will see the records due for disposal otherwise (not even the Global Admins, unless they have also been delegated to that role).

The records person clicks on the link ‘Go there now’ and it opens the following section in the Office 365, Security and Compliance Admin portal, showing the documents that are pending disposition. A number of options are available to sort by Type, to search, and to filter by several options.

 

The following options appear if a single document is selected. Note the option to extend the retention period or apply a different label, as well as the ability to delete the item permanently.

Filtering options are displayed below.

Finally, the records manager can choose all the documents in the list and complete three bulk actions as shown.

Positives and negatives

The positives of this method of disposing of documents are that all records from any location will appear in a single view that can be filtered and actions taken as required.

The negatives are that potentially thousands of documents might appear in this listing every single day making it difficult to decide what can deleted or not.

However, as it’s possible to filter by the retention policy, that at least should make it relatively easy to identify what can be destroyed. The more fine-grained the policies, the fewer records should appear.

Organisations that have function-based disposal classes should find that all records relating to the same function appear for disposal under that function.

Another potential negative is that records may not always appear in the same context, whether it be subject- or function-based. For example, a collection of documents (often known as a ‘file’) may not appear in the disposition listing as a collection but as a set of records that are only connected by the disposal policy name. Does this matter?

Recording disposal actions

A key requirement for most organisations is keeping a record of what was destroyed.

At the moment the only apparent option to do this is to apply filters and export the list, using the handy ‘Export’ option to keep a record of what was destroyed. That csv file can then be stored in a control library to ensure a record is kept. This type of action requires a degree of control to ensure it happens every time.

It may also be possible to identify what was destroyed – and by whom – in the audit logs. This is being investigated.

 

Is Microsoft Teams the future for office communications?

At a recent presentation on Office 365, the presenter started with Microsoft Teams and spent the next half hour or so demonstrating how it, not Outlook, had become the centre of his daily life. He didn’t mention the connection with Office 365 Groups until asked.

Is Microsoft Teams the future of office communications, replacing Outlook?

Teams was introduced to the Office 365 environment in late 2016. (See this video). At the time, it was described as ‘a true chat-based hub for teamwork and give customers the opportunity to create a more open, fluid, and digital environment.’ (https://docs.microsoft.com/en-us/microsoftteams/teams-overview)

Many early reviews suggested that Teams was Microsoft’s response to Slack, but this comparison is simplistic. Teams has much more functionality than Slack.

How do Teams link into the Office 365 environment?

Teams is not an isolated application in the Office 365 (O365) environment. It has direct links with O365 Groups.

This means that, unless your organisation controls the creation of O365 Groups, every new Team will create a new O365 Group – which in turn creates a Group mailbox and calendar, a SharePoint site, and a Planner.

If your organisation controls Group creation (which is not a bad idea), a Team cannot be created by users using the ‘Create Team’ option.

Instead, whoever controls the creation of Groups (ideally a defined Admin role) can create a Team through the ‘Create Team’ option or, preferably, by linking an existing Group to a new Team. That is, a Team is created (from the Teams interface) with the same name as the O365 Group.

The linkage with O365 Groups is important to understand. Both the Exchange and/or SharePoint Administrators should have a role in the creation of both O365 Groups, SharePoint sites and Teams in environments where this is controlled.

Where Group, Team and SharePoint site creation is not controlled, there is potential for their proliferation. There is some debate as to which is the best option but my own recommendation is to maintain controls, at least as the new Office 365 environment is being rolled out. Otherwise, the SharePoint Admin may have to deal with a plethora of similarly or poorly named SharePoint sites, and the Exchange Admin will also have a job on their hands.

The Outlook paradigm – 30 years of poorly managed records

Almost every office worker for the past 30 years has used Outlook as the primary communication medium, using folders to categorise content. Distribution Lists (DLs) helped to provide a way to communicate (in a single direction) with a known group of users.

The primary way to share a document in the Outlook environment was been to attach it to a new email. Email attachments may be left in Outlook and/or saved to a drive somewhere. Multiple copies probably exist.

Organisations that have deployed SharePoint over the last decade have learned that links in emails to documents are a much more effective way of controlling document versions and reducing copies, but this is a hard change for many users to accept.

The idea that there can be one version of a document in a globally accessible location seems counter-intuitive to users who prefer to squirrel information away in ‘personal’ email or network drive folders.

The rise of social networking and messaging

A range of social network applications, including MySpace, began to appear from the early 2000s (Facebook was open generally from September 2006). Originally browser-based, the general popularity of these applications took off once smartphones included those apps.

It wasn’t long before messaging apps such as Yahoo Messenger started to replace SMS messaging as the default way to communicate with others via phones.

Social networking and messaging apps began to change the way we communicated and connected and began to move personal communications away from email. Instead of emailing each other photos, we could now share them in a single location for all of our friends to view, like and comment.

Email has persisted, however, as the primary ‘formal’ way to communicate.

Probably the main reason for this was its recognition and persistence as a ‘record’ – many document and records management systems integrated with email systems, allowing emails to be captured as records.

Instant messaging, on the other hand, remained largely (and artificially) outside the formally accepted recordkeeping world despite the efforts of records managers to try to capture all this ephemeral content.

Enter Microsoft Teams

Microsoft Teams is an interesting technology from a social change point of view, and one that Microsoft seems to believe will be a game changer for business communications.

To understand Teams, it is important to understand what it’s not. It’s not ‘just’ an alternative to Slack. It’s not ‘just’ a replacement for Skype for Business. It’s not ‘just’ a messaging app. It’s a new way to connect, communicate, and collaborate any device.

Teams:

• Is accessible on almost any device or browser.
• Includes 1:1 messaging and group messaging.
• Includes a range of emojis and gifs.
• Includes voice and video calling.
• Has its own Office 365 Group (which has its own mailbox in Outlook).
• Has an email address for anyone who still prefers to use email to connect.
• Has its own dedicated (O365 Group) SharePoint site.
• Allows (and in fact encourages) users to share and work on a document at the same time in the Teams interface (rather than attaching it).
• Allows a team to communicate in multiple channels.
• Has cool ‘toast’ notifications.
• Includes a range of connectors to other services.
• Allows a user to see where other people fit into the organisation.
• Saves all the chat content to a hidden folder in the associated Group’s mailbox.
• Allows external (guest) access.

The Teams interface is, in fact, so useful, that some users might find it more useful than Outlook. If you use it for long enough, you may soon find yourself checking Teams instead of Outlook. In fact, Outlook looks a bit dated by comparison.

Is the end near for email?

I don’t think so, at least not for a few years.

Email is a heavily ingrained way of communicating for many people and is still seen as the ‘official’ communication medium for many organisations (having replaced the old paper Memo or Minute).

But, just as Facebook and Instragram (and other applications) replaced email because they were a more effecient and effective way for people to keep in touch (despite all the security issues), Teams – or its natural successor – has the potential to move a lot of communication traffic (and attachments) away from Outlook.

This change has already happened in part. Many (if not most) people – including government officials (allegedly) – already use a range of ‘unofficial’ applications such as Whatsapp, Facebook Messenger, Signal and so on, for both personal and professional use. The use of email is, slowly, being eroded in favour of more instant ways to communicate and share information.

Why? Because it’s faster and easier to use and meets the new paradigm of limited attention spans and interest in reading long sentences (TL;DR).

Is Microsoft really the game changer?

Perhaps, but it may not be the only one.

It is a relatively new app, and one that will probably get a lot of traction with lots of marketing by Microsoft, its inclusion in O365 licences, and the very recent ability to connect with external ‘guests’.

Whether users will use its full, Team-based collaboration functionality or remain more a Skype-replacement will remain to be seen. But for now, Outlook is looking like an ‘old’ person’s way to communicate.

Learn more here:

https://www.microsoft.com/en-us/education/products/teams/default.aspx

SharePoint Online and OneDrive for Business – Preventing external sharing of data

A recent (September 2017) article suggested that OneDrive for Business (ODfB) (and by extension SharePoint Online (SPO); ODfB is a SharePoint-based service), a key application in Office 365 was a potential source of data leaks and/or target for hacking attacks.

I don’t disagree that, if not configured correctly, any online document management system – not just ODfB/SPO – could be the source of leaks or the target of external attacks. Especially if these systems, and the security controls that can protect the data in them, are not properly configured, governed, administered, and monitored.

But, I would ask, what controls do most organisations have in place now for documents stored in file shares and personal file folders, not to mention USB sticks, and the ability to send document via Bluetooth to mobile devices or upload corporate data to third-party document storage systems? Probably not many, because users have no other way to access the data out of the office.

As we will see, the controls available in Office 365 are likely to be more than sufficient to allow users to access to their documents out of the office, while at the same time reducing (if not eliminating) the sharing of documents with unauthorised users.

How to stop or minimise sharing from OneDrive for Business and SharePoint Online

There is one simple way to prevent the sharing of data stored in SPO and ODfB with external people – don’t allow it.

There are several ways to control what can be shared, each allowing the user a bit more capability. All these options should be based on business requirements and information security risk assessments, and Office 365 configured accordingly.

In this article I will start with no sharing allowed, and then show how the controls can be reduced as necessary.

External sharing – on or off

This is the primary setting, found in the main Office 365 Admin centre under Settings > Services & add-ins > Sites. If you turn this off, no-one can share anything stored in SPO or ODfB.

The option is shown below:

If you do allow sharing, you need to decide (as shown above) if sharing will be with:

• Only existing external users
• New and existing external users [Recommended]
• Anyone, including anonymous users

The second option is recommended because it doesn’t restrict the ability to share with new users. The last option is unlikely to be used in most organisations and comes with some risks.

The next place to set these options are in the SPO and ODfB Admin centres.

OneDrive admin center

If the previous option is enabled, the following options are available for ODfB. Note that BOTH SharePoint and OneDrive are included here because the latter is a part of the SharePoint environment.

• Let users share SharePoint content with external users: ON or OFF.
  • NOTE: If this option is turned OFF, all the following options disappear.
• If sharing with external users is enabled, the following three options are offered:
  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users
• Let users share OneDrive content with external users: ON or OFF
  • This setting must be at least as restrictive as the SharePoint setting.
• If sharing with external users is enabled, the following three options are offered
  • Only existing external users
  • New and existing external users [Recommended]
  • Anyone, including anonymous users

If sharing is allowed, there are three sharing link options:

• Direct – only people who already have permission [Recommended]
• Internal – only people in the organisation
• Anonymous access – anyone with the link

You can limit external sharing by domain, by allowing or blocking sharing with people on selected domains.

External users have two options:

• External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]
• Let external users share items they don’t own. [This should normally be disabled]

A final ‘Share recipients’ checkbox allow the owners to see who viewed their files.

SharePoint admin center

The SPO admin center (to be upgraded in late 2017) has two options for sharing.

The first option is under the ‘sharing’ section which currently has the following options:

Sharing outside your organization

Control how users share content with people outside your organization.

• Don’t allow sharing outside your organization
• Allow sharing only with the external users that already exist in your organization’s directory
• Allow users to invite and share with authenticated external users [Recommended]
• Allow sharing to authenticated external users and using anonymous access links

Who can share outside your organization

• [Checkbox] Let only users in selected security groups share with authenticated external users

Default link type

Choose the type of link that is created by default when users get links.

• Direct – only people who have permission [Recommended, same as above]
• Internal – people in the organization only
• Anonymous Access – anyone with the link

Default link permission

Choose the default permission that is selected when users share. This applies to anonymous access, internal and direct links.

• View [Recommended]
• Edit

Additional settings (Checkboxes)

• Limit external sharing using domains (applies to all future sharing invitations). Separate multiple domains with spaces.
• Prevent external users from sharing files, folders, and sites that they don’t own [Recommended]
• External users must accept sharing invitations using the same account that the invitations were sent to [Recommended]

Notifications (Checkboxes)

E-mail OneDrive for Business owners when

• Other users invite additional external users to shared files [Recommended]
• External users accept invitations to access files [Recommended]
• An anonymous access link is created or changed [Recommended]

Sharing via the Site Collections option

In addition to the options above, sharing options for each SharePoint site are set in the ‘site collections’ section as follows. Note that the default is ‘no sharing allowed’. A conscious decision must be taken to allow sharing, and what type of sharing.

When a site collection name is checked, the following options are displayed.

Sharing outside your company

Control how users invite people outside your organisation to access content

• Don’t allowing sharing outside your organisation (default)
• Allow sharing only with the external users that already exist in your organization’s directory
• Allow external users who accept sharing invitations and sign in as authenticated users
• Allow sharing with all external users, and by using anonymous access links

If anonymous access is not permitted (setting above), a message in red is displayed:

Anonymous access links aren’t allowed in your organization

SharePoint Sharing option

The SharePoint Admin Centre has an additional ‘Sharing’ section with the same settings as shown above for ODfB. It is expected that these multiple options will be merged in the new SharePoint Admin Centre due for release in late 2017.

Additional security controls

In addition to all the above settings, there are a range of additional controls available:

• All user activities related to SPO and ODfB, including who accessed, viewed, edited, deleted, or shared files is accessible in the audit logs.
• SPO and ODfB content may be picked up by Data Loss Prevention (DLP) policies and users prevented from sending them externally. This is of course subject to the DLP policies being able to identify the content correctly.
• SPO and ODfB content may be subject to records retention policies set by preservation policies. These may impact on the ability to send documents externally.
• SPO and ODfB content may be subject to an eDiscovery case.
• Administrators can be notified when users perform specific activities in both SPO and ODfB.
• Sharing (and access to the documents once shared) may be subject to security controls enforced through Microsoft Information Protection.

Conclusion

In summary, the settings above allow an organisation to strongly control what can be shared. If sharing is allowed, certain additional controls determine whether the sharing is for internal users or for users external to the organisation. If the latter is chosen, there are further controls on what external users can do. Audit controls and policies may also control how users can share information externally.

The key takeaway is that organisations should ensure that the sharing options available in Office 365 are based on the organisation’s business requirements and security risk framework.

Office 365 – new data governance and records retention management features

At the September 2017 Ignite conference in Orlando, Florida, Microsoft announced a range of new features coming soon to data governance in Office 365.

These new features build on the options already available in the Security and Compliance section of the Office 365 Admin portal. You can watch the video of the slide presentation here.

Both information technology and records management professionals working in organisations that have Office 365 need to work together to understand these new features and how they will be implemented.

Some of the key catch-phrases to come out of the presentation included ‘keep information in place’, ‘don’t horde everything’, ‘no more moving everything to one bucket’, ‘three-zone policy’, and ‘defensible deletion process’. The last one is probably the most important.

How do you manage the retention of digital content?

If your organisation is like most others, you will have no effective records retention policy or process for emails or content stored across network file shares and in ‘personal’ drives.

If you have an old-style EDRM system you may have acquired a third-party product and/or tried to encourage users (with some success, perhaps) to store emails in that system, in ‘containers’ set up by records managers.

The problem with most of these traditional methods is that it assumes there should be one place to store records relating to a given subject. In reality, attempts to get all related records in the one place conjures up the ‘herding cats’ problem. It’s not easy.

What is Microsoft’s take on this?

For many years now, Microsoft have adopted an alternative approach, one that is not dissimilar to the view taken by eDiscovery vendors such as Recommind. Instead of trying to force users to put records in a single location, it makes more sense to use powerful search and tagging tools to find and manage the retention of records wherever they are stored.

Office 365 already comes with powerful eDiscovery capability, allowing the organisation to search for and put on hold records relating to a given subject, or ‘case’. But it also now has very powerful records retention tools that are about to get even better.

This post extends my previous posting ‘Applying New Retention Policies to Office 365 Content‘, and won’t repeat all of it as a result.

Where do you start?

A standard starting point for the management of the retention and disposal of records is a records retention schedule. These are also known in the Australian recordkeeping context as disposal authorities, general disposal authorities, and records authorities. They may be very granular and contain hundreds of classes, or ‘big bucket’ (for example, Australian Federal government RAs).

Records retention schedules usually describe types of records (sometimes grouped by ‘function’ and ‘activity’, or by business area) and how long they must be retained before they can be disposed of, unless they must be kept for a very long time as archival records.

The classes contained in records retention schedules or similar documents become retention policies in Office 365.

Records retention in Office 365

It is really important to understand that records retention management in Office 365 covers the entire environment – Exchange (EXO), SharePoint (SPO), OneDrive for Business (OD), Office 365 Groups (O365G), Skype for Business. Coverage for Microsoft Teams and OneNote is coming soon. Yammer will not be included until at least the second half of 2018.

That is, records retention is not just about documents stored in SharePoint. It’s everything except as noted.

Records managers working in organisations that have implemented (or are implementing) Office 365 need to be on top of this, to understand this way of approaching and managing the records retention process.

Retention policies in Office 365 are set up in the Security and Compliance Admin Centre, a part of the Office 365 Admin portal. Ideally, records managers should be allocated a role to allow them to access this area.

There are two retention policy subsections:

• Data Governance > Retention > Policy
• Classification > Labels > Policy

The settings in both are almost identical but have slightly different settings and purposes. However, note all retention policies that are set up are visible in both locations.

The difference between the two options is that:

• Retention-based policies are (according to Microsoft) meant for IT to be used more for ‘global’ policies. For example, a global policy for the retention of emails not subject to any other retention policy.
• Label-based policies map to the individual classes in a retention schedule or disposal authority.

Note: Organisations that have many hundreds or even thousands of records retention classes will need to create them using Powershell.

Creating a retention-based policy

Retention-based policies have the following options:

Directly underneath this are two options:

• Find specific types of records based on keyword searches [COMING > also label-based]
• Find Data Loss Prevention (DLP) sensitive information types. [COMING > label-based DLP-related polices can be auto-applied]

A decision must then be made as to where this policy will be applied – see below.

Creating a label-based policy

To create a classification label manually, click on ‘Create a label’.

Note:

• Labels are not available until they are published.
• Labels can be auto-applied

The screenshot below shows the options for creating a new label.

Label- based policies have the following settings:

• Retain the content for n days/months/years
• Based on Created or Last Modified [COMING > when labelled, an event*]
• Then three options: (a) delete it after n days/months/years (b) subject it to a disposition review process (labels only), or (c) don’t delete.

* Such as when certain actions take place on the system.

 

Applying the policies

Once a policy has been created it can then be applied to the entire Office 365 environment or to only specific elements, for example EXO, SPO, OD, O365G.

• IT may want to establish a specific global policy
• Most other policies will be based on the organisation’s records retention schedule

Once they have been published, labels may then be applied automatically or users can have the option to apply them manually.

In EXO, a user may create a folder and apply the policy there. All emails dragged into that folder will be subject to the same policy.

In SPO, retention policies may be applied to a document library and can be applied automatically as the default setting to all new documents. [COMING > also to a folder and a document set]. Adding a label-based policy to a library also creates a new column so the user can easily see what policy the documents are subject to.

Note: Individual documents stored in the library will be subject to disposal, not the library. 

What about Content Types?

Organisations that have used content types to manage groups of records including for retention management will be able to continue to do so, but Microsoft appears to take the view (in the presentation above) that this method should probably replaced by labelling. This points needs further consideration as content types are usually used as a way to apply metadata to records.

Note: If the ability to delete content (emails, documents) is enabled, any deleted content subject to a retention policy will be retained in a hidden location. The option also exists when a label-based policy is created to ‘declare’ records based on the application of a label. 

What happens when records are due for disposal?

Once the records reach the end of their retention period, they will be:

• Deleted
• Subject to a new disposition review process [COMING in 2017 – see below]
• Remain in place (i.e., nothing happens)

In relation to the second option above, a new ‘Disposition’ section under Data Governance will allow the records manager or other authorised person to review records (tagged for Disposition Review) that have become due for disposal.

This is an important point – only records that had a label with the option ‘Disposition Review’ checked will be subject to review. All other records will be destroyed. Therefore, if the organisation needs to keep a record of what was destroyed, then the classification label must have ‘Disposition Review’ selected.

Records that are reviewed and approved to be destroyed are marked as ‘Completed’. This means there is a record of everything (subject to disposition review) that has been destroyed, a key requirement for records managers.

Other new or coming features

A number of other new features demonstrated at the Ignite conference, are coming.

• Labels will have a new ‘Advanced’ check box. This option will allow records marked with that label to have any of the following: watermark, header/footer, subject line suffix, colour.
• Data Governance > Records Management Dashboard. The dashboard will provide an overview of all disposition activity.
• Data Governance > Access Governance. This dashboard, which supports data leakage controls, will show any items that (a) appear to contain sensitive content and (b) can be accessed by ‘too many’ people.
• Auto-suggested records retention policies. The system may identify groups of records that do not seem to be subject to a suitable retention policy and make a recommendation to create one.
• For those parts of the world who need it, new General Data Protection Regulations (GDPR) controls
• Microsoft Information Protection, to replace Azure Information Protection and provide a single set of controls over all of Microsoft’s platforms.

SharePoint On-Premise to SharePoint Online – New Page options

If you are planning to move to SharePoint Online and have customised or allowed users to customise site pages (especially in ‘publishing’ sites), you need to be aware of the new page options in SharePoint Online.

If you are completely familiar with the new SharePoint Framework (SPFx), you don’t need to read any further. This post is aimed at Site Administrators and Site Owners who have edited or ‘customised’ their SharePoint site pages – and organisations that have customised pages.

Background

Microsoft released the new ‘modern’ SharePoint site pages in 2017. These pages are based around HTML5 and provide different page editing options. To put it simply, existing site pages do not ‘work’ in modern pages; the only way to view them is using the ‘classic’ SharePoint experience.

What has changed

The most significant change, from an editing point of view, is the removal of the ribbon menu. Yes, that’s right, no more ribbon, what you see below no longer exists but has been replaced by completely new functionality described below.

 

Instead of a ribbon menu, in SharePoint Online, pages are made up of a set of web parts each with their own options. To add one of these new web parts, you edit the page, then (a) accept the default or choose a layout per section (see below), then click on the + in the middle top of that section to add the web part you want to add.

These options described below.

Why is this important

In SharePoint 2010 and SharePoint 2013 you could copy an entire page content and paste it to another page. You can’t do that with SharePoint Online modern pages.

This means that all pages will have to be ‘re-built’ unless you plan to keep users on the ‘classic’ look for a while.

Creating a new page

Even creating a new page is new. To create a new page in team sites, you click the gear/cog icon, then ‘Add a page’. For non-Communication sites, you can choose a layout from the new web parts – see below.

If you create a page from a Communications site, you get the option to choose which template you’d like to use:

The first thing you then have to do is give the new page a name. (Hint – use hyphens between multi-word page names, then go back later, edit the page, and remove them. It makes for a cleaner URL).

Once added, the only page ribbon options are shown below::

 

Adding web parts

On every new page, choose the layout or web parts is based on clicking the + sign as you can see above directly under the page naming section, or the image below

As noted above, SharePoint Online ‘modern’ pages are made up of multiple web parts that can be placed in five different page layouts.

The page layouts are:

• One column
• Two columns
• Three columns
• One third left column
• One third right column

These are – more or less, similar to the options in ‘Text Layout’ in the old ribbon menu, although there are now fewer options.

While a page can have multiple layouts, you should look at how a mobile device will render that view. To do that, simply reduce the page size layout on the screen to the same size as the mobile device. The page will render accordingly.

The web parts that can be placed within the page layouts, and their relationship with the old ribbon menu, are as follows:

• Text
  • Similar to a INSERT – Web Part – Content Editor Web Part but with the default black font text only.
  • Formatting options are Headings 1 – 4, Normal text, Pull quote
  • Bold, italic, underline
  • Dot points, numbered points, left, centered, right aligned text
  • Hyperlink (similar to INSERT – Link)
  • No font or size or colour options
  • No subscript or superscript or strikethrough
  • No highlighting
  • No images (this means that images are somewhat unnaturally separated from the text)
  • No tables. If you wish to add a table, create it in Word and copy and paste it in.
• Image
  • Similar to INSERT – Picture.
  • Displays the image with a name, it is not possible to add a hyperlink to the image.
  • If you want a hyperlink with an image, create a link using either the Link web part or the Quick Links web part and add the image.
• Document
  • New feature
  • Displays a document on the page.
• Link
  • Similar to INSERT – Link
  • Places a link on the page.
  • The link can include an image.
• Embed
  • Similar to INSERT – Embed Code
  • Embeds code on the page.
  • Use to embed a YouTube or other video that will play on the page.
  • No Java Script (as you could in SP2010/2013 – you will need to do some research to see what’s possible and what’s not).
• Highlighted Content
  • Similar to a library or list web part
  • Displays a range of content types from the site (‘most recent’ by default), the site collection, a document library, or all sites.
  • Allows a range of searches, filters, sort and layout options
• Bing maps
  • New feature but similar to embedding a Google or Bing map using embed.
  • Embeds a Bing map.
• Document library
  • Similar to a library web part
  • Displays content from a document library.
• Events
  • New feature.
  • Displays events. Not good for detailed calendars with multiple items per day.
• Hero
  • New feature
  • Displays content with links in 1 – 5 titles, or 1 – 5 layers.
• Image gallery
  • Similar to Picture Library Slideshow Web part
  • Use to display a set of images that you select, not necessarily from an image library
• List
  • Similar to a list web part
  • Displays content from a list.
• News
  • Similar to a blank page.
  • Display news (separate pages).
• Office 365 Video
  • Deprecated
  • To be deprecated in favour of Stream)
• Stream
  • New feature
  • Displays embeded videos stored in Stream
• People
  • New feature but similar to Site User web part
  • Displays clickable links to site users
• Power BI
  • New feature
  • Displays Power BI content.
• Quick Chart
  • New feature
  • Displays a very simple column or pie chart, based on figures entered from the page
• Quick Links
  • Similar to a links list web part
  • Displays links to other content. Links can include (small) images
• Site activity
  • New feature
  • Displays a list, by last modified date, of content created on the site.
• Yammer feed
  • Similar to Yammer feed embed
  • Displays a Yammer group feed.
• Group calendar
  • New feature
  • Displays items from an Office 365 Group calendar

Things you cannot do any more (or maybe shouldn’t)

There are a number of things you can no longer do any more unless possibly via the SharePoint Framework (SPFx). But, the fact that you need that to re-create those things using SPFx suggests you may want to consider whether those items are still relevant or useful. My advise is – don’t assume, but keep an open mind.

The things to keep in mind are:

• Most of the Format Text options from the old ribbon are now contained in the options contained in the Text or Layout web parts.
• There is no web part that allows you to create tables. These will have to be manually inserted. This means you cannot edit them on the page, it’s display only.
• ‘Upload file’ is no longer available. Instead you would use a link or quick links, or perhaps even display the document on the page.
• Search is now a single search box at the top left. There really isn’t a need for additional search boxes, but some may need them.
• Many of the existing web parts have gone or been replaced by other options including Highlighted Content.

Summary

All of the new page editing options are a massive improvement on most legacy web parts and options. However, many organisations are likely to have built quite complex pages based on these old options.

Accordingly, some thought needs to be put into how the content of existing SP2010 and SP2013 pages will be migrated to the new environment, especially to take advantage of new mobile device access.

 

The Recordkeeping World of Office 365 – More than just SharePoint

I’m often asked (and sometimes challenged to confirm) if SharePoint can manage records. The question is often based on a unspoken second element – like ‘xyz’ system?

It might be (and sometimes is) argued that any system can manage records if the records stored in the system provide evidence of business activity and are considered information assets. But recordkeeping is more than just keeping any records in any system.

The international standard for records management, ISO 15489-1-2016 states: ‘They (records) can be distinguished from other information assets by their role as evidence in the transaction of business and by their reliance on metadata. Metadata for records is used to indicate and preserve context and apply appropriate rules for managing records.’

So, records are not just distinguished by their evidentiary role, but also by their reliance on metadata.

Record types

In my opinion, it is a mistake to focus solely on SharePoint when the question is asked – can it manage records? The question assumes that one application will be used to store all the records – or at least the so-called ‘unstructured’ records – created by the organisation. Any discussion of SharePoint must include and take account of SharePoint Online as part of the Office 365 ecosystem.

The term ‘unstructured’ in this sense generally means email and any kind of digital record that can be saved to network file shares. The latter generally includes documents (in multiple formats), images, and other digital record types.

However, ‘what gets saved on a network files share’ generally overlooks the fast-increasing volumes of information that would not normally be ever stored in a file share. Simple examples of records that are never (or may never be) saved in network file shares (or dedicated recordkeeping systems) include tweets on Twitter, messages sent by personal messaging apps, and social network type information.

I’m always impressed (albeit a bit sceptical) when I hear that organisations state they are capturing all these types of information in their recordkeeping system. That’s pretty impressive.

Records in Office 365

Many organisations have moved (or are moving) their Microsoft enterprise licencing to Office 365, Microsoft’s subscription-based service. Office 365 includes a range of applications that create or store records including:

• Exchange (email, calendars, Groups, Planner)
• Office (used to create the content)
• SharePoint / OneDrive for Business* (document libraries and lists)
• Microsoft Teams
• Sway
• Skype for Business
• Stream (video)

*OneDrive for Business is a SharePoint-based service.

SharePoint is only one part of this information rich ecosystem, and really shouldn’t be thought of as a single destination for the storage of records. Yes, it can be used to store and manage records, but you need to stand back a little to appreciate the full picture.

How Microsoft (may) have approached recordkeeping in Office 365

Until recently, and in the on-premise world, records were stored and managed separately in Exchange and SharePoint, each with their own recordkeeping capabilities, quite independent of each other.

Over the past two years, Microsoft developed a unified strategy for recordkeeping in both systems, presumably based on the likelihood that most corporate records would continue to be stored separately. The requirement for additional recordkeeping metadata in either system would remain optional – see below.

In mid 2017, Microsoft introduced a centralised way to create, manage and apply retention policies to content stored across (no longer separately in) Exchange, SharePoint and OneDrive for Business. These new policies are created as labels in the Office 365 – Security and Compliance Portal under the slightly misleading section called ‘Classifications’.

These new retention policies can be applied across Exchange, SharePoint and OneDrive for Business. In SharePoint, they can be applied to a site, a document library (preferred), list, or individual documents. They remove the requirement to manage retention separately in Exchange and SharePoint.

But what about the metadata?

As noted above, the international standard for recordkeeping states that metadata ‘is used to indicate and preserve context and apply appropriate rules for managing records’.

So why or how is metadata optional in Office 365?  I think this is for two reasons:

• Making any form of metadata mandatory will turn users off using the system.
• Metadata may not be the ‘be all and end all’ for context-based discovery.

‘Context’ essentially means that records relating to a given context (e.g., ‘noise complaints’) can be identified, retrieved and managed in that context. For example, emails relating to a meeting; the meeting agenda and minutes may be stored in one location but more often than not the emails remain on the Exchange server. Another example might be emails relating to the development of a new policy; again, these are more often than not stored separately from the system used to store and manage documents.

Regardless of where they may be stored, metadata should provide and indicate the context of any records that may be created. Years of EDRMS use suggests that users generally don’t like to add additional metadata to records.

So how does Office 365 do this?

In most organisations, the only ways to apply recordkeeping metadata to an email is to save it to an EDRMS or in SharePoint. Most organisations will rarely configure Exchange to include the capture of metadata.

As with an EDRMS, the metadata options in SharePoint are more or less unlimited but careful thought needs to go into what metadata should be applied, and how. For example, metadata can be set:

• In the Managed Metadata Service (MMS)/Term Store, including with hierarchical models
• As site columns
• As library columns

Regardless of the option selected, metadata may be set as a default on each SharePoint document library column. That is, when a record (including an email) is saved to a specific library, it can be assigned specific metadata that is to be assigned to all documents in that library.

Applying metadata in this way, especially as site columns, means that information can be retrieved in context.

It should also be kept in mind that Microsoft Office documents saved to a SharePoint document library also retain their metadata in the document properties, even when the document is exported, a kind of ‘metadata payload’.

Is metadata still relevant?

In the mid 1990s, Yahoo introduced a new portal that allowed users to browse the nascent internet based on pre-defined categories. That is, a form of metadata tagging was applied to all content that allowed the user to browse to where they wanted to go to.

The problem with this idea was that it assumed everybody would understand the categories. Google’s response to this was to provide a single search box allowing users to retrieve whatever they were looking for – subject, of course, to the way the algorithm presented the information to the user based on their understood context.

Adding metadata to indicate the context of a record works as long as the context is still valid – both for the content and the user. Or, to put it another way, the way in which I might describe a record with metadata may be different from the way you want to access that record, because your context is not the same as mine. There may be a range of information that I want to find that hasn’t necessarily been recorded in the context in which I am looking for it.

Some years ago I was curious why users in one business area could not find many records relating to a specific subject – noise complaints in a city area well known for its nightlife. In most cases, they were searching for records containing complaints about noise in that specific area, recorded in the title or metadata of the record.

When we asked them to ignore the metadata and search by the content of the records they found thousands of records, all described in different contexts – building approvals and inspections, delivery of services, police liaison, visitor numbers and public feedback. All these contexts were quite valid, but they were not the context of the user searching for the records.

The lesson learned was simple – my context is not necessarily your context. Records, especially digital records, could relate to any context including future and unpredictable contexts.

Context-based information and eDiscovery

For some users, one of the most ‘startling’ features of Office 365 is Delve and the related Discover option in the user’s OneDrive for Business. Both are based on the underlying Office Graph that learns a user’s context based on their interactions (or ‘signals’) across the Office 365 environment and presents potentially relevant content (to which they have access) from SharePoint or another user’s OneDrive.

I used the term ‘startling’ because, for most users, the idea that you can find out what others are working on seems intuitively to be some kind of breach of privacy (even though they have access to that content already). And yet, what it is doing is letting a user know, based on her or his context, what may be of interest from potentially quite different contexts. It does this based on the interactions between users.

Office 365 also includes a powerful eDiscovery capability that allows the user (if licenced to do so) to find all information across Exchange and SharePoint relating to a specific context regardless of where it is stored, and quarantine that information as required in a case file. While metadata may assist in the process, it is not essential.

But what about all the other records?

So far I have not said anything about the records produced by and stored in the other Office 365 applications such as Teams, Planner, Skype for Business and so on. Or about the management of records produced in third-party social media or messaging applications.

The Office Graph already takes into account the interactions between users to present potentially relevant information stored in SharePoint or OneDrive for Business. At some point in the future, Microsoft may include the information in the various other Office 365 applications.

As for social media, the preferred model may be to capture the feed of that information in an Office 365 service – Microsoft Teams, for example, can receive a feed from third-party applications including Twitter. The answer to the use of third-party messaging applications is to use applications that have at least the same or, preferably, better functionality. Teams and Skype for Business are in this space.

Summary

If you have got to the end of this article, thank you for reading.

In summary, my main point is that when thinking about SharePoint for recordkeeping it is a good idea to consider it in the context of the broader Office 365 ecosystem and its recordkeeping capabilities, not as an isolated application capable of storing and managing records.

Knowledge Management in Office 365

A few articles in the past few weeks, and some internal discussions, prompted some thinking around how Office 365 can support knowledge management (KM) – however that may be defined.

What is Knowledge Management?

According to many knowledge management sources online, knowledge management appeared around 1990, and paralleled the rise of document management. Both appear to have arisen as computers appeared (from the mid 1980s) and digital ways of capturing and managing information took hold, and records management was still primarily focused on the management of paper records.

An early (1994) definition for the term ‘knowledge management’ suggested that it was ‘… the process of capturing, distributing, and effectively using knowledge’ (Davenport, 1994. Koenig, 2012)

Bryant Duhon expanded on this somewhat imprecise definition in his 1998 article ‘It’s All in our Heads’ (my emphasis):

‘Knowledge management is a discipline that promotes an integrated approach to identifying, capturing, evaluating, retrieving, and sharing all of an enterprise’s information assets. These assets may include databases, documents, policies, procedures, and previously un-captured expertise and experience in individual workers.’ (Duhon, 1998)

A key element was capturing the knowledge acquired by individuals.

Koenig (2012) noted that ‘Perhaps the most central thrust in KM is to capture and make available, so it can be used by others in the organization, the information and knowledge that is in people’s heads as it were, and that has never been explicitly set down.’

Explicit/implicit versus tacit knowledge

Generally speaking, there is a difference between explicit and implicit knowledge, the information that is recorded, and ‘the information and knowledge that is in people’s heads’ (and walks out doors when people leave).

The latter is defined generally as tacit knowledge. That is, information that is ‘understood or implied, without being stated’, from the Latin tacitus, the past participle of tacere ‘be silent’. (https://en.oxforddictionaries.com/definition/tacit)

I have worked with the issue of how to access and capture the knowledge in the heads of departing employees since around 1984, when I was first made aware that the departure of some very senior and/or long-term staff meant that we would lose access to the information they knew, gained not only from learned knowledge but also in many cases from many decades of personal experience.

At the time it was not my responsibility to worry about it, but I saw attempts to conduct interviews and document procedures and processes with departing (or already departed) employees.

This pre-digital era activity stuck in my head – was interviewing the departed employees the only way to get this information out of their heads?

(As a side note I learned that it was important to interview and talk to my ageing parents and their siblings about their memories and experiences before those memories were lost forever).

Enter the computer age

I consider myself lucky to have been witness over a generation to the change in working practices from paper to digital.

The start of the digital era from the mid 1980s and ubiquitous access to computers on desktops, person to person emails, network file shares and personal folders created another related dilemma – even if the information was created (or captured) by a user, how could it be accessed?

Users were encouraged to put this information in repositories – mostly document management systems – but the fact that email and information on file shares were stored in different servers meant that unless users would actively move emails to a document management system, that information remained hidden away.

What was needed was a way for users to create and store information – emails, documents – wherever they wanted to put it, and for that information to be accessible, restricted only by relevant security controls.

The only systems that seemed to really do this effectively were eDiscovery tools. Perhaps this was not surprising, as the survival (and financial viability) of a company might depend on the ability to find the information that was required.

The rise of smart phones and ubiquitous, always-on, digital communication within the past 10 years has only added to the types of knowledge available and the methods used to capture it.

In my opinion, traditional recordkeeping practices have not kept up and often remain rooted in the idea that knowledge can be stored in a single location or container. How does one capture instant messages sent via encrypted messaging services in a records container?

Microsoft Graph

Microsoft introduced the Microsoft Graph in 2015. The image below demonstrates how the Graph connects content created and stored through the Office 365 (and connected) environment/s.

The image above should resonate with most people who work in an office. We send emails, create documents or data, set tasks, make appointments, attend and record meetings, have digital conversations, send messages, connect with colleagues, maintaining personal profiles.

The Microsoft Graph collects and analyses this information and presents it to users based on their context. According to Microsoft:

‘Microsoft Graph is made up of resources connected by relationships. For example, a user can be connected to a group through a member of relationship, and to another user through a manager relationship. (The Graph) can traverse these relationships to access these connected resources and perform actions on them through the API. You can also get valuable insights and intelligence about the data from Microsoft Graph. For example, you can get the popular files trending around a particular user, or get the most relevant people around a user.’

(Source for image and text: https://developer.microsoft.com/en-us/graph/docs)

According to Tony Redmond, Microsoft Graph’s REST-based APIs provide ‘… a common access approach to all manner of Office 365 data from Exchange and SharePoint to Teams and Planner’. The Graph Explorer, a newly introduced user interface, extends the ability to access information, wherever it lives. (https://developer.microsoft.com/en-us/graph/graph-explorer)

How does a person access this knowledge?

In my opinion, two key points about tacit knowledge are that:

• It can be captured easily, just as other digital applications capture information about us, including by what we click on or search for.
• It can be accessed without a person necessarily having to search for it.

Most of us by now are familiar with the way Facebook, LinkedIn, eBay, Amazon and so on capture information about our interests and present suggestions for what we might like to do next. It does this by understanding our context

Organisational knowledge management should be the same. Users should go about their business using the various digital applications available to them and other users should be able to see that information or knowledge because they have an interest in the same subject matter, or need to know it to do their work.

Users should be presented with information (subject to any security restrictions) because it relates to their work context or interests. They should not have to go looking for knowledge (although that is an option, just as finding a friend in Facebook is an option), knowledge should come to them.

How does Office 365 do this?

Most Office 365 enterprise or business users will have one or two ways to access this information:

• Delve (may require a higher licence such as E3 for enterprise clients)
• The One Drive for Business ‘Discover’ option.

The ‘Discover’ option allows a user to explore further, to see what others are working on. The response I get to Discover is both positive and slightly startled – the latter because it will be possible to know what others are actually doing.

Why is this important?

The ability to access and ‘harness’ collective knowledge in this way is essential to modern day workplaces.

To quote Microsoft:

‘As the pace of work accelerates, it’s more important than ever that you tap into the collective knowledge of your organisation to find answers, inform decision making, re-purpose successes and learn from lessons of the past’. (Moneypenny, 2017)

Serendipitous discovery

In his 2007 book ‘Everything Is Miscellaneous: The Power of the New Digital Disorder’, David Weinberger spoke about three types of order:

• The first order is the order of physical things, like how books are lined up on shelves in a library.
• The second order is the catalogue order. A catalogue typically refers to a physical order; it is still physical, but one can make several catalogs of the same physical order. Weinberger’s prime example is the card catalog of libraries.
• The third order of order is the digital order, where there is no limit to the number of possible orderings. The digital order frees itself from physical reality, and in it, everything can be connected and related to everything else: Everything is miscellaneous.

The phrase ‘herding cats’ always comes to mind in relation to digital information. It resists order or compartmentalisation.

Further, your order is not my order, my way of browsing or searching may not correspond with your logic for storing or describing it (especially on network file shares!).

The internet pioneered serendipitous discovery. It is now completely taken for granted when, as noted above, we are are offered suggested friends in Facebook, jobs in LinkedIn, purchases on eBay and so on. We are presented this information because the application has collected information about what we clicked on, what jobs we do (or did), who our friends are, and what we like to search for.

The idea that our work environment can do the same thing and present information automatically based on our context (information finds us) is sometimes surprising for people used to the second order of things.

 

Davenport, Thomas H. (1994), Saving IT’s Soul: Human Centered Information Management.  Harvard Business Review,  March-April, 72 (2)pp. 119-131. Duhon, Bryant (1998), It’s All in our Heads. Inform, September, 12 (8). Quoted in Koenig (2012).

Duhon, Bryant (1998), It’s All in our Heads. Inform, September, 12 (8), pp. 8-13.

Koenig, Michael (4 May 2012), What is KM? Knowledge Management Explained, http://www.kmworld.com/Articles/Editorial/What-Is-…/What-is-KM-Knowledge-Management-Explained-82405.aspx, accessed 21 July 2017

Naomi Moneypenny (17 May 2017), Harnessing Collective Knowledge with SharePoint and Yammer, https://techcommunity.microsoft.com/t5/SharePoint-Blog/Harnessing-Collective-Knowledge-with-SharePoint-and-Yammer/ba-p/70164, accessed 21 July 2017

Redmond, Tony (20 July 2017), Exploring Office 365 with the Graph Explorer, https://www.petri.com/exploring-office-365-graph-explorer, accessed 21 July 2017

Weinberger, David, (2007) ‘Everything Is Miscellaneous: The Power of the New Digital Disorder’

Migrating to SharePoint Online – Early Learning with Modern and Communication sites

We have had a ‘controlled’ on-premise SharePoint environment since early 2012, starting with SharePoint 2010 and moving to SharePoint 2013 two and a half years ago.

‘Controlled’ in this sense means that users cannot create their own sites or sub-sites and site owners are responsible for managing their sites, including creating libraries and lists and managing page content.

Governance model

Our governance model, originally based on a Microsoft governance model, provided a good balance between (a) the need for excessive IT control and effort (there’s only two of us managing the whole environment), and (b) the potential for a feral environment when site creation gets out of hand.

An early decision was made to use multiple web applications for teams, projects, publishing sites, the intranet, and ‘apps’ (a handful of ‘purpose-built’ sites).

Another key governance decision made in 2012 was to keep the environment as much as possible ‘out of the box’ (OOTB) and avoid customization. By doing this we aimed to ensure that upgrades would be relatively straightforward. This didn’t prevent site owners from being fairly creative with their sites, especially site pages.

Preparing for SharePoint Online

If you are planning to move to Office 365 and SharePoint Online (SPO), you should understand how existing sites will migrate to the new platform, especially with the release of new ‘modern’ SharePoint sites and more recently ‘communication’ sites.

One of the first considerations is the architecture of the new SPO sites. These use only name-based paths – ‘/sites’ or ‘/teams’. If you have (like we did) multiple web applications or complex hierarchies of sites, you will need to consider how these will map to the new architecture.

For example:

• Sites in multiple web applications will need to be mapped to either /teams or /sites. For example, one of our web applications was /projects; these will be migrated to /teams and all new project sites will be Office 365 Group based, with a ‘PRJ’ prefix.
• Sites in complex hierarchies can, potentially, continue in SPO, but the SPO model is more suited to multiple, separate sites at the same level. A hierarchy or organisational structure may change and this could cause problems for moving content between sites. Having said that, all SharePoint sites site under the top level https://(organisation name).sharepoint.com ‘root’ site, followed by either /sites or /teams – e.g., https://(organisation name).sharepoint.com/sites/example.

Migrating site content

Most SharePoint site content consists of a combination of pages, libraries and lists, and the data stored in each.

Each has a new counterpart in SPO and you need to understood these in advance of migrating. Note however that Microsoft have continued the ‘classic’ look in SPO so that the pages look the same (for the time being); libraries and lists on the other hand are converted immediately to the new ‘modern’ style on migration.

Libraries and Lists

The most visible change to libraries and lists is the removal of the familiar ribbon menu and its replacement with a much simpler and user-friendly version, one that is almost identical with the new ‘ribbon’ that appears in OneDrive for Business.

The main library ribbon is as follows:

The ribbon changes when a document is selected, in this case a Word document:

The new ‘ribbon’ was designed to make it as easy as possible for users to add, edit and access content, including on mobile devices, focusing on the primary actions users need to perform:

• Add new content (including creating a new Office document from within the library, or a new folder or link)
• Edit content (including by using Office Online applications)
• Move and copy content
• Share content

The ribbon is minimalistic and expands with additional options with a document is selected. The following options are accessed by clicking the three-dot ‘ellipsis’ to the far right of the ribbon menu, or clicking on the ellipsis to the right of the document name:

• Copy to
• Rename
• Version history
• Alert me
• Manage by Alerts
• Check Out/In

‘Flow’ is a new option in both libraries and lists, replacing the older style library or list workflows (and possibly some simple SharePoint Designer workflows).

The primary consideration when moving to modern libraries and lists is change management. On a positive note, users who found the old ribbon menu just a bit too complex should find the new ribbon simple to use.

Library Settings and List Settings still remain and have the same look and feel; this option is now accessed from the gear/cog icon.

A new (or rather slightly modified) option for SPO users on the ribbon is the ability to synchronise (‘sync’) the SPO library selected with File Explorer. This option allows users to access SPO content from the familiar File Explorer view, although various library options such as check out/in are not available; the documents in File Explorer are copies.

• Note: Migrating to SPO provides the opportunity to ‘clean up’ libraries and lists, especially libraries without content.

Site pages

Perhaps one of the most challenging changes for SharePoint administrators and site owners or users will be the introduction of new ‘modern’ pages. This may be a challenge for organisations that have implemented or allowed site page customizations.

SharePoint Administrators need to make themselves familiar with the structure and layout of modern site pages well in advance of any planned migration, especially to understand how existing pages will migrate.

The main changes to site pages are the absence of the ribbon and completely new web parts. Instead of a ribbon, each new web part includes various editing options, outlined below.

The introduction of ‘communication’ sites in late June 2017 added to both the site type potential as well as the options for constructing a page. All of these changes make the new site pages mobile friendly.

Another key point to consider, in terms of site design, is whether sub-sites are really required.

New site page web parts

The new web parts are visible when any modern page is placed in edit mode; when you click on the page you will see the + option that allows you to add the required web part. This replaces the ‘App Part’ and ‘Web Part’ options under the SP2013 ribbon ‘INSERT’ option.

The new web parts are presented in three groups.

The first section offers the following web parts.

• Text. Allows formatted text to be insert in a defined area on the page. Similar in a way to the FORMAT TEXT options on the ribbon menu in SP2013, and also presenting text in a Content Editor Web Part. However, it only includes rich text (headings, formatting, but no tables or images).
• Image. Allows an image to be placed on the page, similar to SP2013 INSERT – Picture. No text can be added, and so if you need to place text and images together, you may end up with multiple text boxes with an image above or below.
• Document. Displays the first page of a document within a defined area. This may used as alternative to a table.
• Link. Allows a direct link to be provided to any other content. Similar to INSERT – Link in SP2013.
• Embed. Almost the same as the ‘Embed Code’ option in SP2013 INSERT ribbon menu, but note there are some limitations.
• Highlighted Content. Allows different types of content from the site or other locations to be displayed on the page. The content can be filtered and sorted, and various layout options are available. Type options are: Documents, Pages, News, Videos, Images, Events, Issues, Tasks, Links, Contacts, or All. As at the date of writing this post, the option to display the content from a List is still not available – but see below.

The next section offers various page layout options, similar to the Text Layout option under FORMAT TEXT.

• One column
• Two columns
• Three columns
• One-third left column
• One-third right column

The last section offers the following web parts.

• Bing maps. Displays a Bing map.
• Document library (preview). Presents an editable list view of documents.
• Events. Displays items created in the events list.
• Hero. Provides a way to highlight and link to content using two different designs: ‘topic’, which presents 1 – 5 tiles; ‘showcase’ which presents 1 – 5 layers. The tiles or layers both include the ability to add a photograph and a link to other content.
• Image gallery. Displays photographs from an image library.
• List (preview). Presents an editable list view of a list.
• News. Displays news that is created as news pages.
• Office 365 Video. To be deprecated in favour of Stream (see below). Presents a link to a video.
• People. Shows people from Active Directory.
• Power BI (preview).
• Quick chart. Displays a chart.
• Quick links. Displays links to other content.
• Site activity. Presents a tiled list of content that has been created recently on the site.
• Stream (preview). This will replace the option under SP2013 INSERT – Video.
• Yammer feed. Displays a Yammer group feed.

For more details on the new page options, see:
https://techcommunity.microsoft.com/t5/SharePoint-Blog/Reach-your-audience-via-SharePoint-communication-sites-in-Office/ba-p/70079

Considerations using the new modern pages

Aside from the overall page layout using the new web parts in modern pages, the key issues we have identified so far with migrating old site pages have been the following, none of which are possible in the OOTB modern site pages without (possibly) using the SharePoint Framework (see below):

• Content presented in tables, including images.
• Images with links, including image maps.
• Multicoloured text.
• Images embedded next to text.

If you have allowed extensive page editing or customisations, you may need to consider how to move away from this model.

Why are the page options now limited?

In a word – consistency, but also flexibility using the new SharePoint Framework (SPFx). Site Owners (and others) may have been able to create a range of page content in SP2013 or SP2010. Without central control, this could result in a range of user experiences which may in turn affect user take up. Consistency across SharePoint sites provides users with a familiar navigation model.

The need to access SharePoint on mobile devices also likely drove the requirement for consistency of content.

What are the other options?

The new SharePoint Framework (SPFx) offers the ability to create your own custom SharePoint web parts.

However, rather than use SPFx to re-create the web parts or options that no longer exist, it may be worth considering whether these ways of presenting information are still valid – for example, presenting information in a table on a page was a popular option, but was it the best way to present that content?

Office 365 – SharePoint Communication Sites

Microsoft released the new ‘Communication Sites’ into the SharePoint environment for First Release customers in late July 2017. The release of these new and eagerly anticipated site types underlined the need for a good SharePoint architecture, especially when moving from on-premise to online in Office 365.

What are Communication Sites?

To quote Microsoft, Commmunication Sites ‘… are perfect for internal cross-company campaigns, weekly and monthly reports or status updates, product launches, events and more.’ (Source: https://blogs.office.com/en-us/2017/06/27/sharepoint-communication-sites-begin-rollout-to-office-365-customers/)

But what are they and how do they fit into your SharePoint architecture? What the relationship between Communication Sites and other sites using the publishing features of SharePoint?

Communication sites are, essentially, a new type of online-only site with three different top-level site page designs:

• Topic. Use when you have ‘a lot of information to share, such as news, events and other content’.
• Showcase. Use when you want ‘to feature a product, team or event using photos or images’.
• Blank. Build your own.

Depending on the architecture of your current SharePoint environment, topic-based SharPoint sites have the ability to replace the top-level site of a publishing-based intranet site. The default layout of topic-based sites makes use of the ‘hero’ web part that presents information in several ’tiles’ on the screen as well as other web parts such as ‘news’, ‘events’, ‘documents’ and ‘contacts’. Multiple columns can be displayed on the page and various other options are possible, including by using the SharePoint Framework.

Showcase-based sites, on the other hand, allow you to promote and showcase parts of the organisation, events or products. The default layout also uses the hero web part that allows content to be displayed in one to five layers.

The blank design allows you to create your own site structure.

To quote Microsoft on the link above (which includes lots of screenshots), ‘When you create a page on a communication site, you can embed documents and video, and dynamically pull in real-time data from across Office 365, including documents from SharePoint, Power BI reports, Microsoft Stream videos and Yammer discussions. The resulting page is a rich and dynamic communication’.

How do you create Communication Sites?

Communication sites are created in the same (new) way as Office 365 Group-based sites, by clicking on the ‘Create Site’ option in the SharePoint portal (https://(your company).sharepoint.com/_layouts/15/sharepoint.aspx).

Clicking this option presents two options as shown above: (a) team sites and (b) communication sites. Only authorised users who can create O365 Groups can create a Group-based team site or a Communication site.

Creating a new Communication site using this option does not create an O365 Group, unlike a Group-based team site.

Note: The path for both new Group-based and Communication sites is set in the SharePoint Admin portal. In our experience most Group-based sites need to be created in the /teams name path, while Communication sites should be created in the /sites name path. It can take a little while (we found up to 20 minutes) for the changed option to appear in the SharePoint portal, ‘create site’ option.

When the ‘Communication Site’ option is selected, the authorised user must (a) select which design (topic, showcase, or blank) and (b) give the site a name (which becomes the URL address). We found it was very easy for a use not to select the correct site design because it appears on the left, whereas all the other options including the name appear on the right of the site creation process. The new site is created quickly after ‘Finish’ is selected – in a matter of minutes.

Note: The new sign designs are only available at the top level of the site. New sub-sites are standard sub-sites which, depending on your set up, are probably going to be ‘classic’ site pages with modern libraries and lists. The site pages can of course be easily swapped over for a new modern page, but these pages do not include (or do not seem to inherit) the same design options as on the top level topic and showcase based sites. There may be an architecture or design reason for this – see below.

Using Communication Sites

As noted above, Communication sites have two primary potential uses:

• Replacement for top level intranet sites that are usually built on sites with publishing features enabled
• New ‘showcase’ sites, that may also already exist as publishing sites

The meaning of ‘intranet’ in this context may vary, but in our context the intranet is a standard top-level site, with multiple sub-sites, with publishing features enabled and common organisation-wide centralised information such as news, organisational structure and information, and policies and forms. It may also include extensive customisation. Other types of ‘intranet’ might include:

• The top level in a hierarchy of team and publishing sites, all known as the ‘intranet’.
• Any other SharePoint site that is known as the ‘intranet’. This might include team sites.

Considerations when using Communication Sites

As noted above, the ‘topic’ and ‘showcase’ design elements of Communication sites are restricted to the top level site only. However, many ‘intranet’ sites include at least one level of sub-site. Therefore, careful consideration needs to be given to the architecture of the proposed ‘intranet’ if a decision is made to use Commmunication sites instead of traditional publishing sites for this purpose.

Communication sites include the following default elements:

• Top level site page, using the ‘hero’ web part that provides links to other information.
• Site pages (includes the top level page and any news pages)
• News (pages)
• Events (calendar)
• Documents (library)

Other apps that can be added to these sites include:

• Custom list
• Site mailbox

Organisations may also make use of the SharePoint Framework to add other types of content on the pages.

Clearly, this may limit the potential to use a Communication site to completely replace an existing multi-sub-site intranet.

The lesson that may be drawn from this is that Communication sites using the ‘topic’ design are not intended to be a complete replacement for a multi-sub-site intranet. The inference is that replacement intranets may actually be made up of multiple different sites.

A possible structure (based on a typical intranet site) might be made up of the following elements:

• Organisation ‘home site’ using the ‘topic’ design. This would typically be the first ‘go-to’ place for users to learn more about how the organisation works, the latest news, and policies and forms. It may also include multiple links to other applications or content. ‘Hero’ web part links may point to content within the site, or to other Communication sites (topic or showcase).
• A dedicated sub-site for policies and forms.
• News pages
• Multiple ‘showcase’ design sites for each organisational area or event, to promote their work, instead of using sub-sites from the main site to do this.
• Multiple sites under the ‘/teams’ (includes Group-based sites) and ‘/sites’ name paths.

How do you find anything?

A possible concern to separating elements of existing SharePoint sites into completely separate sites is finding the content; if the information forms part of the same site, it should be possible to find it relatively easily.

The simple answer to this is that the ‘Search’ option in SharePoint Online no longer points to the same site by default, and instead searches across all SharePoint content, regardless of its location.

Conclusions

Organisations that continue to host their SharePoint sites in on-premise servers will need to consider and plan how to migrate their sites, including their intranet, into the new SharePoint Online environment, with the following options:

• Team, publishing and other ‘traditional’ site types created via the SharePoint Admin portal, under the ‘/sites’ or ‘/teams’ paths.
• Office 365-Group based sites, created from the SharePoint Portal, which also creates a Group and all associated elements. Alternatively, O365 Groups created in the ‘Groups’ section of the Office 365 Admin portal, that create O365-linked SharePoint sites. The latter option is preferred to maintain naming conventions and restrict uncontrolled growth and inconsistent naming of both Groups and SharePoint sites.
• Communication sites, created from the SharePoint portal.

Traditional, multi-level intranets will almost certainly need to be discarded in favour of multi-site based intranet content, unless the organisation is prepared to use standard sub-site (modern) page layouts to present information to users.

Organisations that continue to want to have complex intranet sites may need to explore the SharePoint Framework and engage third-party vendors who can support this model.

Whichever option is selected, an important element not to lose sight of is the ability to access (and if necessary, add to or edit) content via a mobile device. The more complex the site, the harder it will be (without considerable extra cost) to present it on a mobile device.