Posted in Governance, Information Management, Information Security, Records management, SharePoint Online

Tips to reduce complexity when implementing SharePoint

Over a period of almost 10 years of providing support for SharePoint (2010, 2013, Online), one of my key learnings has been to avoid complexity in its implementation. The more complicated the implementation is, the harder it will be to support, fix and maintain.

This post contains several tips for implementing SharePoint to minimise complexity and therefore the need for support.

Before the tips, it is important to understand the original SharePoint architecture model as this model (a) likely continues to exist in many organisations and (b) can influence decisions – not necessarily in a positive way – about how to implement SharePoint Online.

Understand the (historic) dual nature of SharePoint

SharePoint was first released in 2001. There continues to be a lot of legacy SharePoint implementations and approaches to SharePoint Online that reflect these older implementation models.

SharePoint has always had two sides – the publishing functionality (usually in the form of an intranet) and the document management functionality (where documents are stored). Sometimes these two are conflated into the single term – ‘intranet’ when in fact they are two quite different things in terms of architecture and usage.

Older (pre Online) on-premise versions of SharePoint typically consisted of multiple web applications in several app pools, as the diagram below shows.

(Source: Microsoft ‘SPS 2010 Design Sample Corporate Portal with Classic Authentication architecture’ model)

Within each of these web applications, organisations would have one or more site collections, often with sub-sites. Almost none of this architecture model remains in SharePoint Online – there are now only sites, which are still technically site collections, but preferably without any subsites – see below. This is sometimes described as a ‘flat’ implementation as every site (formerly a site collection) is on the same ‘level’.

The publishing functionality, enabled through the publishing features, was typically used for intranets. Most information was presented on pages, typically across multiple subsites, with some use of the document libraries (e.g., to store common forms or policies).

But the publishing User Interface (UI) was poor; most organisations engaged a third-party to create more engaging (and usually heavily customised) intranets. The outcome has often been that organisations have had to maintain old legacy SharePoint servers to keep the ‘old’ intranet running. In most cases, these older SharePoint sites cannot be migrated ‘as is’ to SharePoint Online.

The former publishing functionality was replaced in SharePoint Online with communication sites.

The document management functionality was (and continues to remain) the default ‘out of the box’ (OOTB) option for all new team sites. Most older SharePoint sites had to be accessed via a browser and the older UI was often seen as not very interesting or engaging. Business areas would sometimes customise the site pages, sometimes to the point of making them appear to be ‘local’ (business area) intranets.

Although SharePoint Online has the same dual nature, it does not use web applications. Instead, all site templates (communication or team) must be created under one of two paths: /sites/ and /teams/. Organisations may default to just one of these two paths for all sites, or could decide to make use of them to differentiate between, e.g., enterprise sites (cross organisational, e.g., the intranet) and team sites (the name of the business area/team comes after /teams/).

Tip 1 – Try to avoid or minimise customisation

As noted above, customisation was common in on-premise versions of SharePoint. Microsoft have done a lot to improve the UI for both communication and team sites.

Microsoft have released a range of pre-canned templates for communication sites that can be downloaded from the ‘lookbook‘ site. (These templates will soon be available as a choice when new sites are created as well.) These templates and new web parts allow organisations to set up a nice-looking and engaging intranet, with no customisation, (other than via the web parts on each page), that is easy to support and manage.

Organisations that require more complex intranets can of course continue to customise their communication sites and/or use a wide range of third-party tools. These are likely to require a higher level of support to maintain and troubleshoot issues, including from third-party vendors.

The customisation of SharePoint team sites used primarily to store and manage document-type content (e.g., to replace network file share storage) is now a different matter. The rapid adoption of MS Teams as the primary UI to access this content over the past 18 months and the ability to sync document libraries to File Explorer and access the content on a mobile device, has in most cases made browser-based access redundant.

There is no longer much point in customising the browser (pages) view of SharePoint team sites used to store documents. Customisation in these sites is more likely to take the form of the site structure, content types and columns, permissions, or the types of information stored in them. Any customisation of team site pages should be restricted to the out of the box options.

Tip 2 – Don’t overcomplicate the structure

Older versions of SharePoint could have quite complicated structures – from the web application through to the site collection, sub-sites, libraries, content types and columns.

These structures would inevitably cause issues over time as organisations re-structured or were renamed, or wanted content moved to different parts of a site or a different site. The easiest sites to manage were always the simple stand-alone sites which is, fortunately, the preferred model in SharePoint Online.

To avoid complexity with all SharePoint Online sites, don’t create sub-sites unless absolutely necessary. Microsoft don’t recommend sub-sites any more – see ‘Planning Your SharePoint Hub sites‘. To quote from that page:

Subsites don’t give any room for flexibility and change. Since subsites are a physical construct reflected in the URL for content, if you reorganize your business relationships, you break all the intranet relationships in your content. Subsites can also create challenges when it comes to governance because many features (including policy features like retention and classification) in SharePoint apply to all sites within the site collection, whether you want them to or not. This means that you must frequently enable a feature for the entire site collection, even if it’s only applicable to one subsite.

Instead of using sub-sites in communication sites, create multiple pages linked via the out of the box mega menu or links.

Avoid using sub-sites at all for team sites unless there is a genuine requirement to restrict access to part of the site. The same outcome can usually be achieved by changing access controls on a document library.

Tip 3 – Use Content Types and columns selectively

SharePoint is built on content types, including the common ‘item’, ‘document’ and ‘folder’ types. Older SharePoint implementation models sometimes made extensive use of Content Types and site (or library) columns to define content added to document libraries. This meant that end-users would have to select a Content Type and/or add or choose metadata when content was saved, a process that was seen as off-putting for most.

To minimise complexity, deploy custom Content Types and additional site or library columns only where really necessary, not as a default approach. Don’t use Content Types to differentiate between (ironically) content or document types but consider using site or library choice columns with default values instead. For example, if a document library is used to store records about meetings or committees, a drop down choice value for ‘document type’ is more likely to be used (and useful) than Content Types.

Tip 4 – Avoid complex permissions

Possibly up to 70% of all support queries over a decade related to permissions or access restrictions relating to a team site and usually started with ‘Why can’t I access …?’ or ‘Why can (so and so) see (or not see) this content?’ They rarely related to permissions on a publishing or communication site, which tend to have simple permission structures.

Almost all of the queries about team site permissions could be tracked back to someone changing the default inherited permissions. In one case, on a site with very complex permissions, a site owner deleted the site owners permission group, locking them out not only of the site but on every single part of the site that had unique permissions. It took a couple of weeks to restore these permissions.

Try to avoid changing the way permission inheritance works. The permissions set at the site level (example shown below) flow down to all the content. It is all too easy to lose sight of (and then find) what permissions have been changed.

If there is a genuine requirement to restrict access, consider moving that specific content to a separate library or even a separate site, rather than restricting access on items within a library, especially one that has a complex folder-based structure.

Also, keep in mind that over the past decade, the access model for SharePoint content has changed from (a) restrict access everywhere and only grant access to those who can access it, to (b) open access everywhere and only restrict access to what needs to be restricted. This is a much easier model to support.

The easiest way to give ‘everyone’ internally access to all SharePoint content is to add ‘Everyone except external users’ to the Visitors group of every SharePoint site, including sites linked with MS Teams. This access does NOT give them access to the Team channel posts in MS Teams.

Whether or not external users should be allowed access via external sharing will depend on each organisation, but it is a good way to reduce the volume of duplication that happens when content is attached to emails. It can also be monitored.

Avoiding access control complexity should also help to improve information security, by ensuring that controls are more easily understood.

Tip 5 – Avoid storing too much and/or unrelated content together

SharePoint sites (including those linked with Teams) are a form of logical aggregation or container. And you can store a LOT of content on those sites. According to the Microsoft site ‘SharePoint Online limits‘:

A library can have up to 30 million files and folders. When a list, library, or folder contains more than 100,000 items, you can’t break permissions inheritance on the list, library, or folder.

But, as the saying goes, just because you can doesn’t mean you should.

It is all too easy to see the ‘Documents’ library of every SharePoint site (especially Teams-linked sites) as the equivalent of a top level folder in a network file share with multiple folder hierarchies. This might seen quite appealing and simple to use at first, but after several years, the ‘Documents’ library can end up a lot of content, not all of which is related to the site’s original intended purpose.

After 10 years, the content is likely to resemble an uncontrolled network file share, with redundant, trivial and outdated content all over the place. With potentially up to 30 million items, this will make managing the content so much harder.

As much as possible, try to ensure that SharePoint sites are based on logical business functions and the libraries in those sites mapped to business activities.

This is not always easy; for example, many smaller organisations have a ‘corporate services’ type function that includes a range of business activities, including managing property, general admin, legal, HR/personnel, fleet and corporate meetings etc. While all of this content could be stored in a single SharePoint site, there is likely to be a requirement to restrict access to content (see previous point) which will only result in increasingly complex permission controls and support issues over time. Mixing unrelated content could also become a nightmare for compliance and records management.

A better, more sustainable, option would be to create multiple SharePoint sites (or Teams) with multiple libraries, linked via a hub site as necessary. This approach will also make it easier to manage records in the longer-term.

Feature image – original source unknown

Posted in Access controls, Information Management, Information Security, Microsoft 365, Microsoft Teams, Office 365 Groups, SharePoint Online

Understanding permission groups in Teams and SharePoint

One of the most confusing aspects of Teams and SharePoint in Microsoft 365 is the relationship between permission groups used to control access to both of these resources. This is especially the case as every Team in MS Teams has an associated SharePoint site (the ‘Files’ tab).

This post explains how permission groups work between MS Teams, Microsoft 365 Groups and SharePoint.

SharePoint permission groups

Before discussing how Teams permissions relate to SharePoint, here is a brief reminder of how SharePoint permissions work.

SharePoint has always had three default permission groups, prefixed by the URL name of the site, as shown in the screenshot below (the name of the site always prefixes the words Owners, Members and Visitors).

Site Owners

  • People (including in a Group, see below) added to the Owners permission group have full access (full control) to all parts of the site and are usually responsible for managing the SharePoint site. There would normally be two or three site owners.

Site Members

  • People (including in a Group, see below) added to the Members permission group have add/edit (contribute) rights.

Site Visitors

  • People added to the Visitors permission group have read-only (view) rights.

These permissions are set at the site level and inherited on everything in the site, unless that inheritance is broken and unique permission are applied. Additional permission groups can be created as necessary but most SharePoint sites only use the default Owners, Members and Visitors groups.

Microsoft 365 Groups

Microsoft 365 Groups were introduced in 2017 and control access to resources, like Security Groups.

However, unlike Security Groups, which usually provide access to individual resources (such as a single SharePoint site, or Line of Business (LOB) system), Microsoft 365 Groups control access to multiple linked Microsoft 365 resources.

Microsoft 365 groups, distribution lists, mail-enabled security groups, and security groups (collectively referred to as Active Directory (AD) groups, are all created in ‘Groups’ area of the Microsoft 365 Admin portal.

When a new group is created, the following options appear.

As noted above, Microsoft 365 groups are recommended. It is important to understand the relationship between Microsoft 365 groups, Teams and SharePoint.

A new group has a visible mailbox and a Team is created by default

When a new Microsoft 365 group is created (from the dialogue above), it creates:

  • At least one Owner must be specified. The Owner/s are responsible for managing the Members group.
  • An Exchange mailbox with the same email @ name as the Microsoft 365 group. The mailbox is visible in Outlook to the members of the Group.
  • A SharePoint site with the same URL name as the Microsoft 365 group.
  • By default (unless the checkbox is unchecked), a new Team is also created in MS Teams.

When a new Team is created from MS Teams, or a new SharePoint Team site is created, it creates:

  • A Microsoft 365 Group with an Exchange mailbox and a SharePoint site (‘Files’ tab).
  • The name of the Team becomes the name of the Group and the SharePoint site.
  • The mailbox is not visible in Outlook and is only used for calendaring and for the storage of Teams chats (in a hidden folder).

Importantly, when a new Microsoft 365 group or Team is created (which creates a Microsoft 365 group), the Group Owners: (a) are the same as the Team Owners and (b) are added to the SharePoint Owners permission group, as explained below. .

Group/Team Owners and Members

In other words, the Microsoft 365 group owners (group) is added to the SharePoint site owners permission group – a ‘group within a group’.

That is, the Microsoft 365 group controls access to the Team and the SharePoint site as shown in the diagram below. Security Groups may also be added to the Microsoft 365 Group site, but this does not provide access to the Team.

The relationship between Microsoft 365 Groups, Teams and SharePoint

This ‘group within a group’ model is visible from the ‘Site Permissions’ section of the gear/cog icon as shown below (the name of the Microsoft 365 Group/Team/SharePoint site is ‘SharePoint Admin’). The SharePoint Admin Group Owners (group) is in the SharePoint site owners group, and the SharePoint Admin Group Members (group) is in the Site members group.

If a mouse hovers over the Group ‘icon’ (in the above example, GO or GM), it is possible to view the members of the Group and, for Owners, to modify that list. Confusingly, the ‘GM’ in the SharePoint site permissions group becomes ‘SG’ in the drop down list.

You can also see the ‘group within group’ model from the back-end ‘Advanced permissions’ section of the SharePoint site, but you cannot manage the Microsoft 365 Group members here.

Implementing the model

As with Security Groups, the members of Microsoft 365 Groups will usually be a logical group of people who require access to something, in this case access to the SharePoint site or the Team (for chat, files, or other resources).

The main thing to remember is that membership of the (backend) Microsoft 365 Group provides access to BOTH the Team and the Team’s SharePoint site (the ‘Files’ tab in a Team).

  • Every Team in MS Teams will usually consist of the members of a logical group with a common interest – a business unit, project team, or with some other work relationship, for example, the members of a committee. The Team Owners are responsible for managing the Team Members.
  • The Team Owners are the SharePoint site owners and are responsible for managing the site if they decide to access it directly. The Team Members are the SharePoint site members and have the ability to add or edit content, usually via the ‘Files’ tab in Teams.

Note: Security Groups with the same members as Microsoft 365 Groups (and Teams) may already exist. There is no need to add a Security Group if it has the same members as a Microsoft 365 Group.

As noted earlier, a Group/Team does not have visitors with read-only rights. Every Member of the Team has add/edit access to both the Team and its associated SharePoint site.

  • If there is a requirement to give specific other people either add/edit or read-only access to the SharePoint site, that outcome is achieved by adding people by name, or a Security Group, to either the SharePoint Members or Visitors group.
  • If there is a requirement to give everyone in the organisation either add/edit rights, or read only access, to the SharePoint site, that outcome is achieved by adding ‘Everyone except external users’ to either the SharePoint Members or Visitors group.

External guests may also be added to the Team and the Team’s SharePoint site.

Posted in Compliance, Data Loss Prevention - DLP, Electronic records, Governance, Information Classification, Information Management, Information Security, Legal, Microsoft 365, Records management, Retention and disposal, SharePoint Online

What happens if you mix label-based retention policies and non-label retention policies on the same SharePoint site?

Two types of retention policy can be created in Microsoft 365:

  • Label-based retention policies, where the label is used to define the retention and retention outcomes. Labels must be published in a retention policy, a process that includes determining where the labels will be applied and appear (‘explicit’) to end users.
  • Non-label-based retention policies, where the policy includes the retention details and the outcomes. As part of the policy creation, these policies are then applied to specific Microsoft 365 workloads where they are mostly invisible to end-users (except in Exchange mailboxes). In SharePoint and OneDrive for Business, these policies create a Preservation Hold library that is only visible to Site Collection Admins and above.

It is possible to apply both a label-based retention policy and a non-label retention policy to the same SharePoint site. In theory, this would allow for (a) everything on the site to be covered by an overarching retention policy and (b) specific libraries or lists to be covered by a label-based policy.

In practice, it gets a little complicated, as described in this post.

Creating the two labels

For the purpose of this post, I will apply the two types of policy to a SharePoint site (‘FinanceAP’) that contains specific types of financial information that needs to be kept for 7 years, but I want to allow other content on the site to be destroyed after 5 years.

Label-based policy

Retention labels are created in the Information Governance section of the Compliance admin portal in Microsoft 365. I created a label titled ‘Financial records’ with a retention period of 7 years. I then published that label to a retention policy named ‘Financial Records – 7 years’ and applied it only to the FinanceAP site.

More than one label can be published in the same policy, making this a useful option if your SharePoint architecture ‘maps to your file plan or Business Classification Scheme (BCS) and your records retention classes are based on either. It also allows you to create and add the same retention class for types of records that occur in multiple functions where the classes have the same retention – for example, ‘Meetings – 7 years’ or ‘Policy – 10 years’.

Once the policy has been published to a site or sites, the option (in Library Settings) to ‘Apply label to items in this list or library’ can be used to choose which label will apply to the content in the library, as shown below.

If the column ‘Retention label’ is checked, the retention label name appears in that column.

Non-label retention policy

Non-label retention policies are also created in the Information Governance section of the Compliance admin portal which also (a little confusingly) lists all the label-based policies as well.

The process of creating these policies includes the retention (e.g, 5 years) and retention outcome (delete) definitions, as well as the location where the policy will be applied.

For the purpose of this post I created a retention label named ‘Financial Working Records – 5 years’ and applied it to the same site (only) as the label-based policy.

I should expect now to find a Preservation Hold library (via Site Contents as a SharePoint admin) when something is deleted.

At this point, I have two retention policies, (a) one label-based and applied to the site, and (b) one that applies to the whole site.

What happens now?

In the document library where the label-based policy has been selected, I can see that the retention label (Financial Records) that has been applied to items in this library.

This means that I cannot delete this document unless (as an end-user with edit rights or admins) the retention label is removed. However, as we will see below, another policy is working behind the scenes.

In a document library where no label-based policy has been applied, I can see that no label appears under the Retention label policy. From an end-user point of view, it appears that the record can be deleted – or is it?

As this site is the subject of an ‘implicit’ or invisible retention policy that has been applied to the entire site, any attempt to delete anything will be captured by the back-end Preservation Hold library seen below via Site Contents (visible to Admins only).

Interestingly, any attempt to delete a document from a library where a label-based retention policy has been applied, which is ‘denied’ in the actual library, is recorded in the Preservation Hold library, although the document remains in the original library.

If anyone with access to the Preservation Hold library tries to delete that item there, they will receive this message:

The only way to remove this item is to remove the policy.

Posted in Compliance, Conservation and preservation, Electronic records, Governance, Information Management, Information Security, Legal, Records management, Retention and disposal, Security

Destroying digital records – are they really destroyed?

Most people should be aware that pressing the ‘delete’ option for a file stored on a computer doesn’t actually delete the item, it only makes the file ‘invisible’. The actual file is still accessible on the disk and can be retrieved relatively easily or using forensic tools until the space it was stored on is overwritten.

Traditional legacy electronic document and records management (EDRM) systems have two components:

  • A database (e.g., SQL, Oracle) where the metadata about the records are stored
  • A linked file share where the actual objects are stored, most of which are copies of emails or network file share files that remain in their original location.

In most on-premise systems, email mailboxes, network file shares, and the EDRMS database and linked file share are likely to be backed up.

When a digital record comes to the end of its retention and is subject to a ‘destruction’ process, how do you know if the record has actually been destroyed? And even if it is, how can you be sure that the original isn’t still stored in a mailbox, network file share, or a back up?

This post examines what actually happens when a file is ‘deleted’ from a Windows NT File System (NTFS), and questions whether digital records stored in an EDRMS are really destroyed at the end of the retention period.

The Windows NTFS Master File Table (MFT)

Details of every file stored on a computer drive will be found in the NTFS Master File Table (MFT).

In some ways, the MFT operates like a traditional electronic document management system – it is a kind of database that it records metadata about the attributes of the digital objects stored on the drive. These attributes include the following:

attriblist

As noted in the diagram above, the details stored by the MFT include the $File_Name and $Data attributes.

  • The $File_Name attributes include the actual name of the file as well as when it was created and modified, and its size.  This is the information that can be seen via File Explorer and is often copied to the EDRMS metadata.
  • The $Data attribute contains details of where the actual data in the file is stored on the disk (in 0s and 1s) or the complete data if the file is small enough to fit in the MFT record.

If the MFT record has many attributes or the file data is stored in multiple fragments on a disk (for example as a file is being edited), additional MFT ‘extension’ records may be created.

When a file is deleted, the MFT records the deletion.

  • If the file is simply deleted, the record will remain on the disk and can be recovered from the Recycle Bin.
  • If the file is deleted through SHIFT-DEL or emptying the Recycle Bin, the MFT will be updated to the ‘Deleted’ state and update the cluster bitmap section to set the file’s cluster (where the data is stored) as being free for reuse. The MFT record remains until it is re-used or the data clusters are allocated in whole or part to another file.

So, in summary, ‘deleting’ a file does not actually delete it. It may either:

  • Store the file in the Recycle Bin, making it relatively easy to recover, or
  • Change the MFT record to show the file as being deleted but leave the file data on the desk until it is overwritten.

How does an EDRMS store and manage files?

The following summary relates to a well-known Electronic Document and Records Management System (EDRMS). Other systems may work differently but the point is that records managers should understand exactly how they work and what happens when electronic files are destroyed at the end of a retention period.

Most EDRM systems are made up of two parts:

  • A database (SQL, Oracle etc) to store the metadata about the record.
  • An attached file store that stores the actual digital objects.

When EDRM systems are used to register paper or physical records (files and boxes), only the database is used.

When digital records are uploaded to the EDRMS:

  • The metadata in the original file, including the file type, original file name, date created, date modified and author are ‘captured’ by the system and recorded in the new database record.
  • Additional metadata may be added, including a content or record ‘type’.
  • The record will usually be associated with a ‘container’ (e.g., ‘file’). This containment makes the record appear to be ‘contained’ within that container, whereas in fact it is simply a metadata record of an object stored elsewhere.
  • The original record filename is changed to random characters (to make it harder to find, in theory) and then stored on the attached (usually Windows NTFS) file store, often in a series of folders.
  • A link is made between the database record and the record object stored in the file store (the MFT record).

When the end-user opens the EDRMS, they can search for or navigate to containers/files and see what appears to be the digital objects ‘stored’ in that container/file. In reality, they are seeing a link to the object stored (randomly) in the file store.

What happens when an EDRMS record is destroyed?

If there is no requirement to extend their retention, or keep them on a legal hold, records may be destroyed at the conclusion of a retention period.

For physical records, this usually means destroying the physical objects so they cannot be recovered, a process that may include bulk shredding or pulping.

For digital records, however, there may be less certainty about the outcome of the destruction. While the EDRMS may flag the record as being ‘destroyed’ it is not completely clear if the destruction process has actually destroyed the records and overwritten the digital records in a way that ensures its destruction to the same level as destroyed paper files. 

Also:

  • If the original associated NTFS file share becomes full and a new one is used, the original is likely to be made read only.
  • There is likely to be a backup of the EDRMS.
  • The original records uploaded to the EDRMS probably continue to exist on network files shares, in email, or in back up tapes.
  • Digital forensics can be used to recover ‘deleted’ files from the associated file share.

Consider this scenario:

  • An email containing evidence of something is saved to a container in an EDRMS.
  • The container of records is ‘destroyed’ after the retention period expires.
  • A legal case arises after the container is ‘destroyed’
  • A subpoena is made for all records, including those specific records.
  • Has the record actually been destroyed, or could it still be recoverable, including from backups or the digital originals?

Is it really possible to destroy digital records, and does it matter?

Yes, records can be destroyed by overwriting the cluster where the record is kept, and some EDRM systems may offer this option.

But:

  • Do EDRM systems overwrite the cluster when a digital record is destroyed in line with your records retention and disposal authorities, or simply mark the record as being deleted, when it is still technically recoverable?
  • Could the record still exist in the network file shares or email, or in backups of these or the EDRMS?
  • Might it be possible to recover the record with digital forensics tools?
  • Does it matter?

It might be worth asking IT and your EDRMS vendor.

References:

 

 

Posted in Disasters, Electronic records, Information Management, Information Security, Legal, Office 365, Records management, Retention and disposal, SharePoint Online, Training and education

Why is it so hard to ‘go digital’?

I visited a local fast-food outlet recently and could not help but notice the ‘Lever Arch’ binders in the small office behind the counter. A small two-drawer filing cabinet was also located below the desk.

20191002_125518

It made me wonder – in this day and age when pretty much everyone has access to the internet including via their smart phone, why are there any paper records?

And, why is it so hard to ‘go digital’, when so many better and safer digital options are available?

Reasons for not going digital

People probably want to keep paper records in this digital age for a few fairly common reasons, all of which I’ve encountered over the years.

  • Ease of access. It is much ‘easier’ to access a record if it’s in the folder with an obvious name, like ‘Rosters’.
  • Speed of access. You can access a paper record in a couple of seconds. Accessing the same record on a computer means logging on then searching or navigating to where it is stored (potentially including on personal removable storage devices).
  • Easier to archive. At the end of a given period the records can ‘simply’ be placed in an archive box and sent off for archiving.
  • Keeping digital records is too ‘hard’.
  • The company doesn’t offer any other option.
  • ‘Computers are hard’.
  • No obvious or pressing business reason to go digital.
  • A preference for paper, or belief that paper records must be kept.

Which of the above have you encountered? Let me know via this anonymous Form:

Or click this link:

https://forms.office.com/Pages/ResponsePage.aspx?id=DQSIkWdsW0yxEjajBLZtrQAAAAAAAAAAAAN__td1WRVUM0hJM0g2Q1NCWFdLS0JYM0k5QUlOUVUxRC4u

Keeping paper records can be risky

Keeping paper records can be all well and good, unless this sort of thing happens:

burger-king-fire-hed-2017-1260x840
Source: https://finance.yahoo.com/news/burger-king-used-photos-real-105654804.html

If you keep paper records when better digital options exist, you are taking a calculated risk that doing so is ‘OK’.

Of course, not all businesses (a) store the only copy of their physical records locally or (b) burn down (including by being constructed in fire-prone areas). However, these are not the only risks. Other risks include:

  • Flooding, from burst pipes, storms, or floodwaters. Water-damaged records are not easy to recover.
  • Damage from falling objects, including trees or other objects falling from the sky.
  • Theft or vandalism.
  • Business closure and leaving records behind in the abandoned building.
  • Any combination of the above.

What’s the back up for physical records?

What’s the back up for these paper records when disaster strikes?

Generally, unless the physical records have been transferred off-site, or they are the printed version of a digital original that can still be accessed, there isn’t one.

Is there a better, digital way?

Yes.

Printed records are likely to fall into several broad categories, each of which can be managed in their own way. For example, in the business above:

  • Policies and procedures, including ‘operating manuals’ and similar types of instructions are likely to be the printed version of digital originals. They can be made available on the company intranet or, if one doesn’t exist, sent via email.
  • Financial records (e.g., invoices). Again, these are likely to be the printed version of a digital original. If they were in printed form when received (e.g., by mail, with a delivery), the company should (a) ask for digital copies to be sent by email, or (b) scan them and store them digitally.
  • Rosters and general documents relating to groups of employees (as opposed to individual staff ‘files’). Rosters could still be printed for display purposes, but the original should be kept in digital form.
  • Staff files. The format of these may depend on the organisation, but there should be no reason for ‘local’ staff files to be kept in an organisation that has a centralised HR system.
  • Other types of business documents. If necessary, these could be scanned and kept in digital form.

And, of course, all of these could be kept in Office 365, including SharePoint for document storage and MS Teams for teams chat, including for front line workers.

Additional training and support may be required to help these areas ‘go digital’.

 

 

Posted in Classification, Compliance, Information Classification, Information Management, Office 365, Products and applications, Records management, Retention and disposal, Security

Office 365 Security and Compliance – classification label changes

Microsoft have improved the Classification section in the Office 365 Security and Compliance centre. The change will help to reduce confusion and make it easier for records managers and security administrators to focus on their individual needs.

Previous user interface

The primary change is to the menu interface. The previous menu options, shown in the screenshot below, showed only ‘Labels’ and ‘Label policies’.

O365_Classifications_Labels

When the previous ‘Labels’ option was selected, a new screen with two tabs ‘Sensitivity’ (default) and ‘Retention’ was displayed, as shown below.

O365_Classifications_Labels

The sensitivity or retention tab had to be selected to create or publish a new label. The user interface was unclear and the difference between creating and publishing a label was not obvious.

New user interface

The sensitivity and retention elements have now been separated and placed under the primary ‘Classification’ menu option as shown below.

O365_Compliance_ClassificationLabels23Aug19.JPG

Now, ‘Labels’ and ‘Label policies’ are two tabs under the relevant section as can be seen below.

 

O365_Compliance_ClassificationLabelsRetention23Aug19

O365_Compliance_ClassificationLabelsSensitivity23Aug19

The options to create and publish labels remain the same.

Posted in Information Management, Information Security, Products and applications, Records management, SharePoint 2013, SharePoint Online

Migrating to SharePoint Online – Part 1 (Planning)

We implemented SharePoint 2010 in early 2012 and then upgraded to SharePoint 2013 in early 2015. After acquiring Office 365 enterprise licences in April 2016 we began to play for the migration of our existing on-premise environment to SharePoint Online. After testing the migration process with inactive sites, we started to migrate active sites from early 2018. We expect to complete all the migrations by 31 December 2018.

This post, the first of three, outlines the factors that influenced and guided how we approached the migration. Our approach may not be the same as your approach, but many of the basic principles may be similar.

Overview of our SharePoint environment pre-migration

A key principle for our SharePoint environment since 2012 was to avoid customisation and dependencies, and use the product ‘out of the box’ (OOTB) as much as possible.

  • Customisation would almost always require some degree of development and ongoing maintenance. It also meant that upgrades could be more complex and expensive.
  • Dependencies of any sort – be they integration components or third-party add-ons – could also make upgrades more complex and expensive.

Governance model

We also implemented a ‘balanced’ controlled environment, following the technical design models for SharePoint 2010 described by Microsoft (extract in image above), which recommended that organisations strike balance across three key governance elements:

SharePoint2010GovernanceBalance

Source: https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-server-2010/cc303422(v%3doffice.14)

  • IT Governance. Centrally managed or locally managed?
  • Information Management. Tightly managed or loosely managed?
  • Application Management. Strictly managed or loosely managed development?

In our environment, the ability to create new SharePoint sites and sub-sites required the completion of a (SharePoint) online form and was restricted to the SharePoint Administrators. This enabled us to prevent uncontrolled growth in the environment and to ensure that all new sites were created within a pre-defined – but not overly strict – architecture design model.

Upgrade to SharePoint 2013 in early 2015

Our SharePoint site collections were created across five web applications: team (approximately 120 sites), project (approx. 120 sites), publication, apps, and intranet. Most of the corporate records were stored in team or project sites, as well as a single ‘apps’ site. (Our apps sites (< 10) were set up to address small business problems that in the past might have been addressed by using Microsoft Access).

Thanks to our OOTB model, we were able to upgrade to SharePoint 2013 over a weekend, with almost no errors. The only site we could not upgrade was the intranet which remains (as at August 2018) in ‘compatibility mode’.

Note: It is not possible to migrate directly from SharePoint 2010 to SharePoint Online. It must be upgraded to SharePoint 2013 or SharePoint 2016 first.

The situation in 2016

In May 2016 we changed our Microsoft Enterprise Agreement to an Office 365 subscription model. Our reasons for going to Office 365 were driven by multiple factors, including the need for mobile access to information.

It is important to remember that SharePoint Online is only one element among many others in Office 365. That is, while it is technically possible to do it, SharePoint would not normally be migrated on its own to SharePoint Online. Any migration must take in account a range of considerations relating to the broader Office 365 environment, including (but not limited to):

  • Office 365 licences (and what this meant for our users with Office installed on existing computers which were being upgraded to new Windows 10-based devices as part of a separate project)
  • Active Directory syncing so users can access the environment.
  • Exchange mailbox migrations so SharePoint-based, email-linked Flow workflows can work.
  • OneDrive for Business, as a SharePoint service to replace ‘personal’ drives on network file shares.
  • Security controls and records retention policies, set from the Office 365 Security and Compliance admin portal, as well as audit logs in that same portal.
  • Office 365 Groups with associated SharePoint sites, Yammer groups (which can be linked with Office 365 Groups) and Microsoft Teams (which can also be linked with Office 365 Groups).
  • ‘Classic’ and modern team sites, Office 365 Group-based sites, and communication sites.
  • The SharePoint user portal.
  • The mobile app, and how sub-sites are accessed.
  • The ever-changing SharePoint Online environment in which anything described as ‘classic’ is likely to be deprecated at some point, and new features appear.

Migrating multiple web applications to one

We needed to plan our migration process, moving away from our five web applications to a new model. We new that, with the exception of our customised intranet, we would probably be able to migrate almost all of our sites relatively easily because we had always kept to the OOTB model.

Fortunately, Microsoft produced a very useful 12-page document which provided a good overview describing how it ran its own SharePoint migration, and good advice for how we might do our own migration.

SharePoint_to_the_cloud_MSpaper.JPG

Learn how Microsoft ran its own migration

We had a range of factors to take into account.

  • One of our initial decisions was not to migrate any active site until all Exchange mailboxes were migrated (and preferably, end-users had new Windows 10 devices). As it turned out, the decision to migrate mailboxes was delayed and as a result we would end up migrating most sites first.
  • We need to work out how to migrate our content as it was no longer possible to do a ‘lift and shift’. We investigated the market and made the decision to acquire a migration tool, ShareGate, to do the migrations ourselves. We would later find the same tool useful to migrate personal drives to OneDrive for Business.
  • We identified the likelihood that we would create new SharePoint Online sites in parallel with the migration of on-premise sites; this was partially because some existing on-premise sites with multiple sub-sites would be split into separate sites instead, but also because the new SharePoint was so much more versatile and would likely be popular.

The new architecture model

An important point to note is that the new SharePoint Online architecture model provided the opportunity to re-think our SharePoint model and, to some extent, clean up or leave unwanted SharePoint content behind. To quote the Microsoft site above, ‘the best migration is no migration’.

As noted above, we had five primary web applications in our SharePoint 2013 environment. These had to be migrated (or re-created, in the case of publication sites) under one of two paths (only – /teams or /sites) to one of three site option:

  • ‘Classic’ sites (the default for all team and project sites)
  • Office 365 Group-based team sites
  • Communication sites (re-created page-based content)

That is:

  • Migrated team and project sites would become classic team sites under either (a) /teams/sitename path or (b) /teams/prj_sitename path, respectively. There were some exceptions:
    • Some sites with multiple sub-sites would be split up into multiple independent sites (including using the new ‘hub’ sites).
    • A couple of team sites would become communication sites.
    • Team sites that crossed multiple organisational business areas would be created as classic team sites under the /sites/sitename path.
  • Most publication sites that used the publishing features would need to be re-created as communication sites under the /sites/sitename path. There were some exceptions:
    • Some publication sites would become team sites instead.
    • The intranet would be managed separately as, at the very least, it would need to be re-created in SharePoint Online. It could not be migrated ‘as is’.
  • Application sites would become team sites.
  • Some existing sites or sub-sites might be migrated to SharePoint sites linked to Office 365 Groups, with the naming prefix of either GRP_ or PRJ_.

The above ‘mapping’ model was an early decision that did not change.

Preparatory work

We also commenced work on the following elements of work:

  • Reviewing all existing sites to determine which sites would be migrated or discarded – see below.
  • Re-developing our SharePoint Architecture documentation for the Online version.
  • Investigating and documenting all Office 365 admin and Office 365 Security and Compliance admin configuration settings, and determining roles. This process, which required Global Admin access, included establishing records retention policies (from mid 2018) in the Security and Compliance admin portal.
  • Re-developing our existing SharePoint admin documentation for the Online version, including all the configuration settings. We included the OneDrive for Business config settings in this same document as it is a SharePoint service.
  • Understanding how the new environment worked, and would work.
  • Re-establishing our SharePoint Admin and SharePoint User Group sites in SharePoint Online.
  • We also created a range of ‘test’ sites to better understand the new environment.
  • Creating an initial schedule for the migration of sites, targeting inactive sites first.
  • Assigning the initial batches of Office 365 licences.
  • Developing a repeatable process to migrate sites using ShareGate. In our environment steps involved:
    • Identify need to migrate site
    • Register a new site request in our SharePoint Admin portal.
    • Register the task in our Jira task management system.
    • Create the SharePoint Online site (via a script linked to the request).
    • Migrate the on-premise site, make it read only with a re-direct notice on the front page (and a three month deletion notice*).
    • Prepare the migrated site, including swapping the classic default home page to a modern home page.
    • Hand over the site to the business owners and close the task

* In practice many of these sites still remained after 6 months.

As part of our review process, we identified around a dozen sites that had one or all of the following elements, that would mean we had to devote more time to their migration (‘custom workload’ in the Microsoft document above):

  • Complex workflows which would need to be re-created.
  • Integration with other systems (mostly via BizTalk).
  • Links with ETL processes.

We also identified around 50 sites that would not be migrated:

  • Sites that were unused or had no content of value (often because the original was still on a drive).
  • Sites that did not need to be migrated, for example if their content had been migrated to a different business system.
  • Test sites.

Sites that were no longer used but contained records that needed to be kept were to be migrated with the word ‘Archive’ to the end of the site URL name, assigned a site retention policy, and then made read only.

By August 2017, we had identified that 250 site collections would be migrated to SharePoint Online. We acquired ShareGate in September 2017 and were ready to start migrating.

In Part 2 of this series of posts I will describe the migration process and the lessons we learned along the way.

Posted in Classification, Compliance, Digital preservation, Electronic records, Governance, Information Management, Information Security, Office 365, Records management, Retention and disposal, SharePoint Online

Office 365 – Applying retention periods to SharePoint document libraries and disposal/disposition actions

Records retention policies are created in the Security and Compliance Admin portal, Classifications section of Office 365, as noted in my previous post of 9 March 2018 on the subject.

This post describes how these are applied to document libraries and what happens when the records reach their disposal/disposition period.

Note: In Australia we refer to the disposal of records. In the US this is called disposition.

Setting up retention policies

Organisations may have complex or quite simple records retention policies. An important point to keep in mind in Office 365 is how many policies should be displayed to the end user to choose from.

Ideally, there should be fewer than a dozen classes so they are easy to choose from (see below). There is nothing stopping you creating 100 or 500 policies, but all of them will appear in the drop down list to choose from. Microsoft say they are working on ‘grouping’ policies, so this may help to fix the issue.

For some organisations, it may be useful to distill or group retention policies down to a smaller number.

  • For example, specific retention policies for certain types of records, and one (or two) for ‘all other’ records. The key, as we will see below, is naming them so they are obvious and easy to apply.

Viewing available retention policies

Retention policies that have been created appear in the Security and Compliance Admin portal, under Classifications > Labels.

O365_Classifications_Labels

Note: Labels must be published before they become visible to end users.

When you click on Labels, you can then see all the retention policies that have been created (but not necessarily published).

The screenshot below shows just the very top policy (a test/demonstration policy with a 7 day retention period) in a list of policies.

O365_Classifications_Labels_List.png

Note: Policies can be auto-applied, provided the policy has sufficient ability to identify what records they should be applied to.

Published policies appear in the Data Governance, Dispositions section:

O365_DataGovernance_Dispositions.png

The Dispositions section displays policies that have been published and are visible to end users in the Office 365 areas selected when the policy was created (e.g., Exchange, SharePoint, OneDrive etc).

O365_DataGovernance_Dispositions_List.png

Applying the policy in a SharePoint document library

To apply the policy to a SharePoint document library, go to the document library, library settings, and you will see the option to add the retention policy: ‘Apply label to items in this list or library’.

O365_RetentionPolicy_LibrarySet1.PNG

The ‘Apply Label’ dialogue shows the option to apply the label to existing items (recommended) and a drop down which shows all the published retention policies.

O365_RetentionPolicy_LibrarySet2.PNG

In this example below, there are four policies including the test policy.

O365_RetentionPolicy_LibrarySet3

The policy now applies to all records stored in that document library.

Managing disposal/disposition

When the records reach the end of the retention period configured in the policy, the person designated to be informed about the retention will receive an email notifying them of the need to review the dispositions.

O365_Dispositions_EmailNotification.pngNote, the person (or mailbox) receiving this email MUST be assigned to the Records Management role in the Security and Compliance Admin portal, Permissions section. No-one else will see the records due for disposal otherwise (not even the Global Admins, unless they have also been delegated to that role).

The records person clicks on the link ‘Go there now’ and it opens the following section in the Office 365, Security and Compliance Admin portal, showing the documents that are pending disposition. A number of options are available to sort by Type, to search, and to filter by several options.

 

O365_Dispositions_DocListing

The following options appear if a single document is selected. Note the option to extend the retention period or apply a different label, as well as the ability to delete the item permanently.

O365_Dispositions_Doc_OneDocument

Filtering options are displayed below.

O365_DataGovernance_Dispositions_Filters

Finally, the records manager can choose all the documents in the list and complete three bulk actions as shown.

O365_DataGovernance_Dispositions_BulkActions.png

Positives and negatives

The positives of this method of disposing of documents are that all records from any location will appear in a single view that can be filtered and actions taken as required.

The negatives are that potentially thousands of documents might appear in this listing every single day making it difficult to decide what can deleted or not.

However, as it’s possible to filter by the retention policy, that at least should make it relatively easy to identify what can be destroyed. The more fine-grained the policies, the fewer records should appear.

Organisations that have function-based disposal classes should find that all records relating to the same function appear for disposal under that function.

Another potential negative is that records may not always appear in the same context, whether it be subject- or function-based. For example, a collection of documents (often known as a ‘file’) may not appear in the disposition listing as a collection but as a set of records that are only connected by the disposal policy name. Does this matter?

Recording disposal actions

A key requirement for most organisations is keeping a record of what was destroyed.

At the moment the only apparent option to do this is to apply filters and export the list, using the handy ‘Export’ option to keep a record of what was destroyed. That csv file can then be stored in a control library to ensure a record is kept. This type of action requires a degree of control to ensure it happens every time.

It may also be possible to identify what was destroyed – and by whom – in the audit logs. This is being investigated.