Archive for the ‘Electronic records’ Category

The Recordkeeping World of Office 365 – More than just SharePoint

August 28, 2017

I’m often asked (and sometimes challenged to confirm) if SharePoint can manage records. The question is often based on a unspoken second element – like ‘xyz’ system?

It might be (and sometimes is) argued that any system can manage records if the records stored in the system provide evidence of business activity and are considered information assets. But recordkeeping is more than just keeping any records in any system.

The international standard for records management, ISO 15489-1-2016 states: ‘They (records) can be distinguished from other information assets by their role as evidence in the transaction of business and by their reliance on metadata. Metadata for records is used to indicate and preserve context and apply appropriate rules for managing records.’

So, records are not just distinguished by their evidentiary role, but also by their reliance on metadata.

Record types

In my opinion, it is a mistake to focus solely on SharePoint when the question is asked – can it manage records? The question assumes that one application will be used to store all the records – or at least the so-called ‘unstructured’ records – created by the organisation. Any discussion of SharePoint must include and take account of SharePoint Online as part of the Office 365 ecosystem.

The term ‘unstructured’ in this sense generally means email and any kind of digital record that can be saved to network file shares. The latter generally includes documents (in multiple formats), images, and other digital record types.

However, ‘what gets saved on a network files share’ generally overlooks the fast-increasing volumes of information that would not normally be ever stored in a file share. Simple examples of records that are never (or may never be) saved in network file shares (or dedicated recordkeeping systems) include tweets on Twitter, messages sent by personal messaging apps, and social network type information.

I’m always impressed (albeit a bit sceptical) when I hear that organisations state they are capturing all these types of information in their recordkeeping system. That’s pretty impressive.

Records in Office 365

Many organisations have moved (or are moving) their Microsoft enterprise licencing to Office 365, Microsoft’s subscription-based service. Office 365 includes a range of applications that create or store records including:

  • Exchange (email, calendars, Groups, Planner)
  • Office (used to create the content)
  • SharePoint / OneDrive for Business* (document libraries and lists)
  • Microsoft Teams
  • Sway
  • Skype for Business
  • Stream (video)

*OneDrive for Business is a SharePoint-based service.

SharePoint is only one part of this information rich ecosystem, and really shouldn’t be thought of as a single destination for the storage of records. Yes, it can be used to store and manage records, but you need to stand back a little to appreciate the full picture.

How Microsoft (may) have approached recordkeeping in Office 365

Until recently, and in the on-premise world, records were stored and managed separately in Exchange and SharePoint, each with their own recordkeeping capabilities, quite independent of each other.

Over the past two years, Microsoft developed a unified strategy for recordkeeping in both systems, presumably based on the likelihood that most corporate records would continue to be stored separately. The requirement for additional recordkeeping metadata in either system would remain optional – see below.

In mid 2017, Microsoft introduced a centralised way to create, manage and apply retention policies to content stored across (no longer separately in) Exchange, SharePoint and OneDrive for Business. These new policies are created as labels in the Office 365 – Security and Compliance Portal under the slightly misleading section called ‘Classifications’.

These new retention policies can be applied across Exchange, SharePoint and OneDrive for Business. In SharePoint, they can be applied to a site, a document library (preferred), list, or individual documents. They remove the requirement to manage retention separately in Exchange and SharePoint.

But what about the metadata?

As noted above, the international standard for recordkeeping states that metadata ‘is used to indicate and preserve context and apply appropriate rules for managing records’.

So why or how is metadata optional in Office 365?  I think this is for two reasons:

  • Making any form of metadata mandatory will turn users off using the system.
  • Metadata may not be the ‘be all and end all’ for context-based discovery.

‘Context’ essentially means that records relating to a given context (e.g., ‘noise complaints’) can be identified, retrieved and managed in that context. For example, emails relating to a meeting; the meeting agenda and minutes may be stored in one location but more often than not the emails remain on the Exchange server. Another example might be emails relating to the development of a new policy; again, these are more often than not stored separately from the system used to store and manage documents.

Regardless of where they may be stored, metadata should provide and indicate the context of any records that may be created. Years of EDRMS use suggests that users generally don’t like to add additional metadata to records.

So how does Office 365 do this?

In most organisations, the only ways to apply recordkeeping metadata to an email is to save it to an EDRMS or in SharePoint. Most organisations will rarely configure Exchange to include the capture of metadata.

As with an EDRMS, the metadata options in SharePoint are more or less unlimited but careful thought needs to go into what metadata should be applied, and how. For example, metadata can be set:

  • In the Managed Metadata Service (MMS)/Term Store, including with hierarchical models
  • As site columns
  • As library columns

Regardless of the option selected, metadata may be set as a default on each SharePoint document library column. That is, when a record (including an email) is saved to a specific library, it can be assigned specific metadata that is to be assigned to all documents in that library.

Applying metadata in this way, especially as site columns, means that information can be retrieved in context.

It should also be kept in mind that Microsoft Office documents saved to a SharePoint document library also retain their metadata in the document properties, even when the document is exported, a kind of ‘metadata payload’.

Is metadata still relevant?

In the mid 1990s, Yahoo introduced a new portal that allowed users to browse the nascent internet based on pre-defined categories. That is, a form of metadata tagging was applied to all content that allowed the user to browse to where they wanted to go to.

The problem with this idea was that it assumed everybody would understand the categories. Google’s response to this was to provide a single search box allowing users to retrieve whatever they were looking for – subject, of course, to the way the algorithm presented the information to the user based on their understood context.

Adding metadata to indicate the context of a record works as long as the context is still valid – both for the content and the user. Or, to put it another way, the way in which I might describe a record with metadata may be different from the way you want to access that record, because your context is not the same as mine. There may be a range of information that I want to find that hasn’t necessarily been recorded in the context in which I am looking for it.

Some years ago I was curious why users in one business area could not find many records relating to a specific subject – noise complaints in a city area well known for its nightlife. In most cases, they were searching for records containing complaints about noise in that specific area, recorded in the title or metadata of the record.

When we asked them to ignore the metadata and search by the content of the records they found thousands of records, all described in different contexts – building approvals and inspections, delivery of services, police liaison, visitor numbers and public feedback. All these contexts were quite valid, but they were not the context of the user searching for the records.

The lesson learned was simple – my context is not necessarily your context. Records, especially digital records, could relate to any context including future and unpredictable contexts.

Context-based information and eDiscovery

For some users, one of the most ‘startling’ features of Office 365 is Delve and the related Discover option in the user’s OneDrive for Business. Both are based on the underlying Office Graph that learns a user’s context based on their interactions (or ‘signals’) across the Office 365 environment and presents potentially relevant content (to which they have access) from SharePoint or another user’s OneDrive.

I used the term ‘startling’ because, for most users, the idea that you can find out what others are working on seems intuitively to be some kind of breach of privacy (even though they have access to that content already). And yet, what it is doing is letting a user know, based on her or his context, what may be of interest from potentially quite different contexts. It does this based on the interactions between users.

Office 365 also includes a powerful eDiscovery capability that allows the user (if licenced to do so) to find all information across Exchange and SharePoint relating to a specific context regardless of where it is stored, and quarantine that information as required in a case file. While metadata may assist in the process, it is not essential.

But what about all the other records?

So far I have not said anything about the records produced by and stored in the other Office 365 applications such as Teams, Planner, Skype for Business and so on. Or about the management of records produced in third-party social media or messaging applications.

The Office Graph already takes into account the interactions between users to present potentially relevant information stored in SharePoint or OneDrive for Business. At some point in the future, Microsoft may include the information in the various other Office 365 applications.

As for social media, the preferred model may be to capture the feed of that information in an Office 365 service – Microsoft Teams, for example, can receive a feed from third-party applications including Twitter. The answer to the use of third-party messaging applications is to use applications that have at least the same or, preferably, better functionality. Teams and Skype for Business are in this space.

Summary

If you have got to the end of this article, thank you for reading.

In summary, my main point is that when thinking about SharePoint for recordkeeping it is a good idea to consider it in the context of the broader Office 365 ecosystem and its recordkeeping capabilities, not as an isolated application capable of storing and managing records.

Advertisements

Applying (new) Retention Policies to Office 365 Content

April 30, 2017

From time to time I’m asked about the way records retention policies ‘work’ in SharePoint. A common criticism has been that SharePoint’s retention model is based on applying retention policies to individual records (e.g., documents in a library or individual emails) rather than to aggregations of records, the most obvious of which is a document library.

The idea of storing and managing related records together in a single aggregation derives from the management of paper records – in files, boxes, and series. This model (of aggregations containing all records relating to a given subject) was largely replicated in electronic document management systems (EDMS – many of which were used to register paper files and boxes previously) when they appeared or were modified to manage digital records in the late 1990s.

In fact, many EDM systems did not actually manage records in an aggregation; the actual digital records were stored in a secure network file stored, and presented in the EDMS user interface though a common ‘file number’ (or similar) ID.

In any case, the ability to store all digital records on the same subject together in the one system (e.g., EDMS) was always hampered by the fact that (a) email and documents were created by different systems, (b) stored in different locations (servers), and (c) use of network file shares continued more or less unabated.

The increasing complexity and types of digital records underlines the difficulty of ever storing, let alone managing or applying retention and disposal actions, to them in a single aggregation.

Until recently, Microsoft’s retention and disposal options reflected the fact that applications used to create digital records stored them in different locations (servers) – Exchange and SharePoint. Retention policies targeted individual records stored in those applications, rather than aggregations.

In March 2017, Microsoft introduced a new, single central way to create and apply retention and disposal policies to most Office 365 content, wherever it was stored – Exchange, SharePoint, OneDrive for Business, Office 365 Groups, and Skype for Business.

This post:

  • Summarizes the existing ‘out of the box’ retention and disposal options in SharePoint, but not Exchange (see my earlier post on this subject).
  • Discusses issues with existing retention and disposal options in SharePoint.
  • Describes how the new centrally-managed retention policies and labels can be applied to most content in Office 365.
  • Discusses why applying retention policies to individual records rather than aggregations may be a better option in the digital world.

Records managers working in organisations that use Office 365 to manage records should familiarize themselves with the way these new retention policies work.

Note: The details in this post are based on the Australian recordkeeping context, which may be different from your specific location.

SharePoint out of the box (OOTB) retention and disposal options

Until recently, the only available OOTB options to apply retention and disposal actions to SharePoint were to:

  • Apply an information management policy to an entire site via the Site Collection Settings. This option is suitable for short-lived sites such as project or closed, archived sites, but less suitable for long-lived team sites which might have a range of different content.
  • Create a retention policy using the information management policy settings in Content Types. This option applies the policy to individual records. Content Types also include the ability to ‘transfer’ (actually copy) records after a defined period to another location, such as a Records Center.
  • Use a folder-based information management policy. This option requires the default Content Type-based policy on a document library to be changed via Library Settings – Information Management Policy Settings, to Library and Folders.

Another option was to adopt a form of ‘retention in place’ and regard each library as a logical aggregation of records, the equivalent of a ‘file’, and manage retention and disposal manually or using PowerShell scripts to identify libraries for potential disposal based on the last modified date of the records. Some vendors have developed a similar model to manage retention policies on libraries using a central ‘console’.

Applying retention and disposal actions to individual records

Both the Content Type and folder-based options noted above apply the retention policy to individual records in the library, not the library (aggregation/container) as a whole.

That is, disposal was based on a time period after which each individual record was created, modified, or declared a record. The logic behind this model appears to be that a document library may store multiple record types each with different retention requirements. This may not be true for all document libraries, but it usually is for many.

Applying automated disposal actions on individual records (rather than an aggregation of records) is probably counter-intuitive for most records managers. The main concerns, from a recordkeeping (and possibly also archival) point of view are the absence of (a) a documented review and approval process before the records are destroyed, and (b) a metadata record of what was destroyed. That is, the records simple disappear from the document library, removing records that may would be relevant to the context of the original aggregation. This, of course, assumes that all records relating to the subject were stored in a single aggregation which, as noted above, may not always be the case.

Global Retention Policies and Labels in Office 365

In March 2017, Microsoft introduced two new ‘global’ retention options – retention policies and labels – to Office 365. The two options allow organisations to apply centrally set and apply retention policies to the same type of record, in whatever form and wherever they are stored – emails in Exchange, documents and lists in SharePoint, conversations (in Office 365 Groups and Skype)..

Examples of ‘types’ of information could include:

  • Corporate records that must be kept for the life of the company.
  • Financial records that need to be kept for 7 years.
  • ‘Working records’ that could be deleted after a minimum period of time.
  • Personnel records or staff files that had to be kept indefinitely.

As Tony Redmond noted in this recent article, these new retention policies build on the type of retention policies first released in Exchange 2010 using folder, system, personal and default tags. The article suggests that organisations that have applied Exchange retention policies may need to consider the impact of these new types of policies. In particular, the ability to move email to archive mailboxes is lost, replaced with a retention policy.

How Retention Policies work

Retention policies in Office 365 are created by authorized users (ideally, records managers) in the Retention section of the Security and Compliance Center.

Creating a new retention policy

Each policy has the following options: Name, Settings, Locations and Preservation Lock.

Name

The name of the retention policy should reflect the class name or number in the records retention schedules so that it can be easily identified and applied to content wherever it can be applied in Office 365 (see below for ‘Locations’).

Settings

The two Settings options are based on two questions:

  • Do you want to retain the content? 
    • If ‘Yes, I want to retain it’ is selected, the choices are either ‘Forever’ or a configurable ‘n days/months/years’ (e.g. 7 years). The administrator must then decide if, once it reaches that point, the record should be deleted or not. If ‘Yes’ is selected, the content will be deleted from where it is currently stored as described in the next two points.
    • >>For SharePoint content there are two options when the retention period expires. (1) If the record has not been modified or deleted it will be deleted from the original library where it was stored, and then remain in the two-stage Recycle Bin for up to 90 days. (2) If the content has been modified or deleted, it is transferred to the hidden Preservation Hold library that is created when the retention policy is applied to a SharePoint site and deleted from that library. In this case, the administrator has only 7 days to recover the content before it is deleted permanently.
    • >>For Exchange content there are also two options. (1) If the item is modified or permanently deleted by the user during the retention period, the item is copied (if modified) or moved (if deleted) to the Recoverable Items folder. The retention policy process identifies and deletes items whose retention period has expired within 14 to 30 (configurable) days of the end of the retention period.  (2) If the item is not modified or deleted during the retention period, the same process runs on all folders in the mailbox and identifies items whose retention period has expired. These items are also permanently deleted within 14 to 30 days of the end of the retention period. (Note: If a user leaves the organization, and their mailbox is included in a retention policy, the mailbox becomes an inactive mailbox. ‘The contents of an inactive mailbox are still subject to any retention policy that was placed on the mailbox before it was made inactive.)
    • If ‘No’ is selected, the content will be left in place and must be manually deleted at some point.
  • No, just delete the content that’s older than … The options are to delete: (a) after ‘n days/months/years’, and (b) based on when it was created or modified.

The (subtle) difference between these two options is that the first option (Yes) ensures that records are not permanently deleted before the end of the retention period, while the second option (No) just deletes records permanently at the end of the retention period.

Advanced retention settings are also available these allow the administrator to create a search query with specific words phrases, or link the policy with the same sensitive information options found under DLP policies, e.g., financial, medical and health, privacy, and custom.

Locations

The Locations section sets where the policy will be applied. By default this is all locations across Office 365, including content in Exchange, SharePoint, OneDrive, Office 365 Groups and Skype for Business.

  • Office 365 has a limit of 10 organisation-wide policies and entire-location policies combined per tenant. Therefore, careful consideration should be given to what specific types of record need a global policy, especially given that not all types of records will be found globally across the organisation.

The alternative option is to apply the policy only to specific locations or users. In most cases this is likely to be Exchange and SharePoint where the majority of key records are created and stored.

  • A retention policy that includes or excludes over 1,000 specific users can contain no more than 1,000 mailboxes and 100 sites. A tenant can contain no more than 1,000 such retention policies. According to Microsoft ‘… you can get over these limits by applying either an org-wide policy or a policy that applies to entire locations’.

Retention policies applied to a SharePoint site or OneDrive account result in the creation of a hidden Preservation Hold library as noted above.

Retention policies applied to Exchange user mailboxes apply the policy to the mailbox. For public folders, the retention policy is applied at the folder level.

Preservation Lock

Finally, the administrator has the option to apply a Preservation Lock, which prevents anyone from changing or deleting the policy after it is turned on. This option should only be applied in specific circumstances as it cannot be turned off or made less restricted (by anyone, including the administrator) after it has been applied. .

Review and save

Finally, the new retention policy should be reviewed, may be saved for later, or published.

Labels

A separate option for managing retention and disposal is to use (retention) labels, which should not be confused with security labels. This option is designed to replace the following:

  • Exchange Online retention tags and retention policies, also known as messaging records management (MRM).
  • In SharePoint Online and OneDrive for Business: (a) in-place records management, (b) the Records Center, and (c) information management policies.

Labels are used to manage retention policies for specific types of content across the Office 365 environment. Labels can be applied automatically to content if it matches certain conditions or keywords (E5 licence only), or manually by users to emails, documents, or Office 365 Group conversations.

See below for the relationship and priority between retention policies and labels.

Who can create labels

Labels are created by individuals (ideally records managers or similar) assigned to a compliance role in the Security and Compliance Admin portal in Office 365.

Creating Labels

Labels are created in the Security and Compliance Admin Portal under ‘Classifications’. Labels may also be created without having an associated retention policy; that is, a label can be created and applied to content as no more than a visual ‘tag’. A policy can be added to it at a later stage.

If the ‘Retention’ option is enabled for labels (on/off switch), a new section appears titled ‘When users apply this label to content’. This section is where the retention policy is defined with two options:

  • Retain the content. The choices are either ‘Forever’ or ‘n days/months/years’ (e.g., 7 years). The administrator must decide if, once it reaches that point, the labelled record should be deleted or not. The ‘Yes’ and ‘No’ options are the same as for retention policies, described above.
    • If ‘Yes’ is selected, the record will be deleted from where it is stored. Administrators have 93 days to recover records that have not been edited or deleted, or 7 days to records that have been edited or deleted (and moved to the Preservation Hold library).
    • If ‘No’ is selected, the content will be left in place and must be manually deleted.
  • Don’t retain the content. The choices are to delete (a) after ‘n days/months/years’, and (b) based on when the record was created, modified, or labelled.

If the first option (‘Retain the content’) above is selected a check box option allows the administrator to use the label to classify content as a record. If the content is classified as a record, users are unable to change or delete the content or change or remove the label. They may still, however, edit the metadata.

The final step in the process is to review the settings. Once created, the administrator is returned to the main Labels screen which displays the label that has been created, allowing the administrator to then publish it.

Label limitations when used on a SharePoint document library

There are some limitations to applying a default label to a SharePoint document library:

  • It applies the label to all records except those that already have a label and those contained in document sets.
  • If the default label is removed, it removes the label from all records except those that have a label and those contained in document sets.
  • Labels cannot be applied to folders in SharePoint or OneDrive (but can be applied to folders in Exchange).
  • If the record is moved to a different library that has a different default label, it will inherit that label. Conversely, if it is moved to a library with no label, the existing label will be removed.

Note: When labels are published to an Office 365 group, the labels appear in both the group site and group mailbox in Outlook on the web. The experience of applying a label to content is identical to that shown above for email and documents.

What about legal holds?

eDiscovery in Office 365 is based around the creation of ‘cases’ in a SharePoint eDiscovery site. Cases are generally established in response to litigation (or potential litigation) and can be used to search across a range of sources. Once found, the information that forms part of the case can then be placed on hold, overriding any retention policy. However, once the hold is released, retention policies on records continue.

For more information on this subject, see:

https://support.office.com/en-gb/article/Add-content-to-a-case-and-place-sources-on-hold-in-the-eDiscovery-Center-54d70de9-1ec2-4325-84f3-aeb588554479?ui=en-US&rs=en-GB&ad=GB

What’s the relationship between retention policies and labels?

Retention policies and labels do the same thing but the former is more likely to be set centrally, while the latter is set by the end user. This means that a record could have more than one retention policy applied to it.

According to Microsoft’s documentation (link below), records will be retained until the end of the longest retention period applied to it, regardless of whether that policy was based on the retention policy or the label.

Are retention policies and labels better than previous retention options?

One of the primary benefits of the new retention policy regime in Office 365 is that it enables organisations to apply retention policies centrally rather than do this separately for each application (e.g., Exchange, SharePoint) as was the case until recently. It also allows end users to apply retention policies via labels.

Retention and disposal continues to be based on the individual record, or type of record (as defined by the policy or label), not logical aggregations or containers of records such as a document library.

As noted above, the concept of an aggregation that contains all the records on a given subject is ill-suited to the digital world. The reality is that records may be created using different applications (e.g., email in Exchange, document, list item or page in SharePoint, conversation in Groups, discussions in Skype etc) and stored in multiple application locations (e.g. in Exchange folders, SharePoint libraries, etc).

The dilemma for many records managers using Office 365 is how to store or manage records together in context, including based on the organisation’s File Plan or Business Classification Scheme (BCS) terms. The need to keep records together has been the driver behind the integration of EDRM systems with email applications, allowing email to be ‘captured’ in the EDRM along with other types of documents. This has rarely been successful in practice and, in most cases, emails are duplicated and remain stored in the email server.

The new Office 365 retention policies, including those applied as labels to specific types of content, may well be the answer to this dilemma. Rather than try to capture all types of records (e.g, document email, list item, conversation) in a single aggregation or container, Office 365 allows the option for them to be stored wherever the user prefers, with the same retention policy applied.

If necessary, all records with the same label can then be found using a content search in the ‘Search and Investigation’ section of Office 365.

In my view, there are still some shortcomings in basing retention policies on individual record types:

  • Individual documents, rather than logical aggregations of documents, will be continue to be subject to disposal actions.
  • Records that may provide context to other records (including those stored in different locations) may be destroyed.
  • Appraisal options may be limited and appropriate review and approval steps before disposal may not be possible.
  • Disposal actions may be automatic and unrecoverable.
  • There may be no record kept, including the metadata, of the individual records that were destroyed.
  • It is not known how courts might view the automatic disposal of records without prior review and approval.

Final thoughts

The new Office 365 records retention policy and label options centralise the management of retention and disposal for most types of records across Office 365, reducing complexity.

Retention and disposal continues to be based on individual records rather than aggregations, but this may be better suited to the digital world in which aggregations of records may not always be achievable.

Records managers working in organisations using Office 365 need to understand and provide guidance to IT on how records retention schedules can be applied as retention policies, and how they can be directly involved in decisions regarding the new options.

For more information: –

https://support.office.com/en-us/article/Overview-of-retention-policies-5e377752-700d-4870-9b6d-12bfc12d2423

https://support.office.com/en-us/article/Overview-of-labels-af398293-c69d-465e-a249-d74561552d30

 

How Office 365 challenges traditional records management practices

September 27, 2016

If your organisation is using SharePoint on-premise now, or just starting out with Office 365, it is important to understand how the Office 365 ecosystem will challenge traditional ways of managing records practices while at the same time delivering a transformational all-digital experience for end users.

SharePoint On-Premises

When configured well, SharePoint on-premises (e.g. versions up to SharePoint 2016) allowed organisations to manage unstructured (i.e., document-based) content through a hierarchy of site collections – sites/sub-sites – document libraries – (folders/document sets) – documents.

In on-premise SharePoint environments, document libraries could be used to store and manage records, thereby becoming the logical containers or aggregations of records, similar to ‘files’ in traditional EDRM systems.

The Office 365 ecosystem

Office 365 changes and challenges the on-premises model of SharePoint by adding new ways of working to standard SharePoint team and publishing sites. These new ways of working include:

  • Office 365 Groups, each of which has a dedicated SharePoint site
  • OneDrive for Business, a personal version of SharePoint
  • Yammer
  • Skype for Business
  • Delve
  • Planner
  • Sway

Why is this important? 

SharePoint has been clearly positioned as Microsoft’s online document management engine. SharePoint, not network file shares, is the document management future. And so, by extension, it becomes the future location for the management of digital records for any organisation that subscribes to Office 365.

From both the business and end-user points of view, SharePoint provides easy-to-use and more efficient content management and collaboration capabilities allowing users to access and use a range of content anywhere, anytime, on any device. Coupled with collaboration options such as Office 365 Groups, Yammer, and Skype for Business, information is now available across a number of different applications within the same single ecosystem.

From a records management point of view, this new way of working challenges the idea that information can be stored in the context of a single function, activity or transaction that created it. Instead, it supports the concept that digital information cannot truly be assigned to a single function or context; its context may also depend on the context of the person seeking to access it.

That is, how one person stores information is not necessarily how others may expect to find or use it. Think of the parallels with eBay, Facebook, LinkedIn and similar products – algorithms present information to you, often in a ‘feed’, based on what the application knows about you, not how other people store that information.

‘Modern’ Team Sites

The most striking change with ‘modern’ team sites in SharePoint Online (compared with SharePoint 2013 and earlier) is the disappearance of the ribbon menu and the simplification of the user-experience to be more or less identical with OneDrive for Business.

When any library is selected (and before a document is selected), the user is presented with the common options: New (Folder, Word, Excel, PowerPoint, OneNote, Link), Upload, Quick Edit, and Sync.

o365sp1

When a document is selected, the user is presented with a context-specific menu offering again commonly used options: Open, Share, Get a link, Download, Delete, Pin to top, Move to, Copy to, Rename, Version History, Alert me, and Check out.

O365SP2.JPG

O365SP3.JPG

The familiar Library Settings, previously located on the ribbon menu, are now found via the Office 365 settings ‘cog’.

O365SP4.JPG

Microsoft have also changed the look of SharePoint Online sites and provided a new ‘SharePoint’ landing page to help users access all the sites they are following, and also present suggestions for sites to follow. In other words, the system understands the user’s context and presents content suggestions, the same way Facebook users are invited to befriend people.

From a records management point of view, little has changed with document libraries in team sites. SharePoint Online continues to offer all the same features as before:

  • Almost unlimited metadata options allowing multiple metadata-based views to be set up
  • Unique, persistent document IDs
  • Folders and document sets (although the latter are even harder to set up than they were)
  • Versioning (and more efficient storage of versions)
  • Popularity trends and per-document views
  • Detailed audit trails
  • Access/permission controls
  • Legal compliance/retention and disposal
  • Powerful search
  • Full integration with Office but now allowing users to save directly by default to SharePoint and OneDrive by default.
  • Hyperlinkable documents
  • Easy sharing

While it is still possible in SharePoint Online to manage records out the box, the other elements that make up the Office 365 ecosystem provide a much broader and complex environment for the storage and management of records. SharePoint Online is just one component of this environment.

Office 365 Groups

Office 365 Groups provide a way for a group of people within the organisation – as well as external users – to discuss and share information.

  • They are similar to Active Directory (AD) Distribution Groups in the sense that they are a pre-defined organisational group designed to receive information.
  • They are different in that, instead of being just the recipient of information, users (and people who join the group at a later date) can see all discussions that have been sent to all members and access any Group documents.

Office 365 Groups are made up of two main content elements: ‘Conversations’ email-based threads and ‘Files’.

O365SP5.JPG

  • Conversation threads are based on simple email exchanges presented in Outlook – currently it is not possible to create folders in the group.
  • The Files option in Office 365 Groups is a SharePoint site that allows the group to store, share and collaborate on any unstructured content.

Groups also include a calendar and a group Notebook (which opens OneNote Online in the Group SharePoint site).

Office 365 Groups content is stored either within the context of the Group’s email-based conversations or in unstructured content stored in an associated SharePoint site.

Office 365 Groups SharePoint sites are visible in the user’s list of SharePoint sites, making it easy to get back to the Group’s site or its conversations.

OneDrive for Business

OneDrive for Business is built on the SharePoint engine. The consumer version of OneDrive has been around for a few years and is a direct competitor to the likes of Google Drive, iDrive, DropBox, Box and so on.

OneDrive for Business, the online replacement for ‘personal’ network drives, allows users to store, synchronise and share ‘personal’ work information through an interface that in Office 365 is now almost identical with modern team sites (less the Library Settings).

As with personal drives on network drives, content stored by users on OneDrive for Business is inaccessible unless shared with others. Organisations have only 30 days by default to do something about the user’s OneDrive for Business content when they cease to be an employee, before the content is deleted.

Options to manage the otherwise hidden content of a departed user’s OneDrive for Business account include allowing the user’s manager to review and if necessary move or delete it, allowing an authorised person in IT to review it, and/or backing it up to other storage so it is not deleted.

Yammer

While the long-term future of Yammer is unclear in the face of Office 365 Groups, Yammer may still exist and capture information and records for a time to come.

Skype for Business

In addition to Yammer and the conversation options provided through Office 365 Groups, Skype for Business provides yet another option to discuss and share information including via voice and/or video calls.

Delve

All the options described above provide a function-rich environment to store and manage unstructured content and collaborate with other people both within and external to the organisation. But how to make sense of all this information?

Depending on licensing, Delve provides a way to find content that may be relevant to the user.

O365SP6.JPG

Delve suggests a range of content that may be of interest (based on the user’s profile, connections and content created or accessed), and provides an analysis of the user’s activity as recorded in Outlook, the calendar and other actions.

Challenges with managing records in Office 365

While Office 365 provides a transformative digital experience for end users, managing the records created and stored in various parts of Office 365 presents new challenges for records managers.

For example, there is far less ability to control the way content is stored or described in specific, pre-defined and/or metadata-driven aggregations and contexts. Users are likely to use whatever application is the most appropriate or convenient. For example, they may use OneDrive for Business to create and store large volumes of content, hidden away from corporate view. They may even share content from this application, including with external users.

The default settings in SharePoint, if not disabled, provide end-users with considerable latitude to create new SharePoint sites and Office 365 Groups, in addition to their personal OneDrive for Business sites, to store, manage and share rich digital content including with external users. In reality, these settings probably need to be disabled to prevent uncontrolled growth in the environment.

Even if records managers (as Site Collection Administrators) have oversight and control of the creation of SharePoint Online team sites, some questions arise:

  • How will they extend this control to SharePoint sites created to support Office 365 Groups, or the conversations that take place within those groups?
  • What about content stored in and shared from OneDrive for Business?
  • How will it be possible in the future to bring together all information about a given function/activity for disposal or disposition actions, especially if it’s not all stored in the one aggregation?

Good SharePoint (and Office 365) governance requires a good balance of control. Too much control and users will be put off using and benefiting from the ecosystem. Too little and the ecosystem may become uncontrollable but possibly very ‘lively’ in terms of content profusion.

Ideally, users should feel that they have the ability to manage their information within a lightly controlled environment – for example, SharePoint site owners cannot create new Sites (to prevent the massive proliferation of sites) but they can create document libraries (thereby reducing IT administrative controls).

Can analytics help with managing records?

Analytics via the Office Graph may provide a way to bring together information and records in context, a context (or contexts) which may be unforeseen by the person who created the content in the first instance. For example, a user may store information in a document library, unaware of its relevance or similarity to others in the organisation. Analytics may be able to connect the two, or the different people doing similar things.

At this stage, Analytics does not seem to provide the ability to bring together all information about a given subject. The model, instead, appears to be about presenting or making information accessible in any context at any time to users depending on their context at the time.

eDiscovery?

eDiscovery, a feature available from SharePoint 2013, has the potential ability to bring together all information about a given subject from across the Office 365 ecosystem. However, the primary purpose of eDiscovery is to support legal processes, not records management.

New ways of thinking are necessary

Records managers need to think differently about how they will approach the management of all types of digital records and other content (conversations, discussions, photographs, videos, Sway presentations) created and stored by users across the complex ecosystems that is Office 365.

It will no longer be possible to assume that all records relating to a given function/activity pair, subject, or context can or will be stored in the same aggregation of records. Instead, records managers need to find other ways to manage digital content, including to manage disposition activities.

Artificial Intelligence (AI) may provide the clue to this. Microsoft CEO Satya Nadella made this very clear in a keynote presentation to the Microsoft Ignite conference on 26 September where he noted that AI would be able to: “… to reason over large amounts of data and convert that into intelligence”. He also noted Microsoft’s ambition is to create an intelligent assistant that “… can take text input, can take speech input, that knows your deeply. It knows your context, your family, your work. It knows about the world.”

Nadella also noted that: ” The most profound shift is in the fact that the data underneath the applications of Office 365 is exposed in a graph structure. And in a trusted, private-preserving way, we can reason over this data and create intelligence. That’s really the profound shift in Office 365.” (Source: https://techcrunch.com/2016/09/26/microsoft-ceo-satya-nadella-on-how-ai-will-transform-his-company/)

(Note, the last two paragraphs were added on 29 September to include comments made by Satya Nadella about Microsoft’s AI ambitions).

 

 

Information Security in SharePoint Online

May 24, 2016

Until now, the security of information stored in SharePoint on-premise implementations was largely based on access control groups that gave or restricted access to the content on the site. Access to the content, and ability to do anything with it (e.g., edit, read) depending on what group you belonged to. The main five access control groups are:

  • SharePoint Administrator/s: Access to everything.
  • Site Collection Administrator: (Usually) access to everything, but this can be disabled.
  • Site Owners: ‘Full Control’ access to everything (except for the Site Collection Administration elements in Site Settings).
  • Site Members: ‘Contribute’ or add/edit access.
  • Site Visitors: Read only.

Other groups such as Designer and Reader existed for specific purposes.

At any point from the top level Site Collection downwards through all the content, these inherited permissions could be stopped and unique permissions – including for both individuals and new access groups – could be created and applied to control access to content.

Audit logs supplemented access controls by providing details of who did (including changing security permissions) or accessed what, and when. While the SharePoint Administrator and Site Collection Administrator’s names are not visible to Site Owners, Members or Visitors, they appear in the audit logs if any activity is recorded. System account activity is also recorded in the logs.

New Security Controls in SharePoint Online

SharePoint Online brings a range of new options to protect the security of information, in addition to access controls. These options, some of which are included with SharePoint 2013 an onwards, are:

  • Information security classifications
  • Data Loss Prevention (DLP)
  • Audited sharing
  • Information Rights Management (IRM)
  • Shredded storage (new from SP 2013)

Two of these options can be seen in the following Microsoft diagram:

mt718319.001.png

Source: ‘Monitoring and protecting sensitive data in Office 365’ https://msdn.microsoft.com/en-us/library/mt718319.aspx

Information Security Classifications

According to a number of online sources, from at least March 2011, Microsoft has classified its own information into three categories: High Business Impact (HBI), Moderate Business Impact (MBI), and Low Business Impact (LBI).

  • High Business Impact (HBI): Authentication / authorization credentials (i.e., usernames and passwords, private cryptography keys, PIN’s, and hardware or software tokens), and highly sensitive personally identifiable information (PII) including government provided credentials (i.e. passport, social security, or driver’s license numbers), financial data such as credit card information, credit reports, or personal income statements, and medical information such as records and biometric identifiers.
  • Moderate Business Impact (MBI): Includes all personally identifiable information (PII) that is not classified as HBI such as: Information that can be used to contact an individual such as name, address, e-mail address, fax number, phone number, IP address, etc; Information regarding an individual’s race, ethnic origin, political opinions, religious beliefs, trade union membership, physical or mental health, sexual orientation, commission or alleged commission of offenses and court proceedings.
  • Low Business Impact (LBI): Includes all other information that does not fall into the HBI or MBI categories.

Source: ‘Microsoft Vendor Data Privacy – Part 1’ (March 2011) https://www.auditwest.com/microsoft-vendor-data-privacy/

Microsoft released code (via Github) to apply these classifications to SharePoint on-premise deployments in 2014.

Source: https://github.com/OfficeDev/PnP/tree/master/Solutions/Governance.TimerJobs

In 2016 Microsoft released a Technical Case Study highlighting how it migrated all its SharePoint content to SharePoint Online – and how information classification formed part of that process.

Source: ‘SharePoint to the Cloud – Learn how Microsoft ran its own migration’ (Case Study – 2016)  https://msdn.microsoft.com/en-us/library/mt668814.aspx

In May 2016, Microsoft announced that this form of classification would be added to new SharePoint Online site collections during 2016.

The application of security classifications to SharePoint Online sites has two elements:

  • Security and compliance policies, set by the SharePoint Administrator via either the ‘Security policies’ or ‘Data management’ section of the Office 365 Security & Compliance Center. [As of 23 May 2016 the only policies are ‘Device management’ and ‘Data Loss Prevention’. While the DLP policies appear to allow the inclusion of security classifications, it is expected that Microsoft will add more options to support the application of security classifications during 2016. See below for more information on DLP.]
  • A new drop-down, three choice (LBI, MBI, HBI) option in the ‘Start a new site’ dialogue box under the question ‘How sensitive is your data?’ The choice of classification invokes the relevant security and compliance policies.

Microsoft provides examples of the types of information that would be covered by each of these at this interactive site: https://www.microsoft.com/security/data/

The application of these policies will enable organisations to control what happens to information stored in sites assigned these classifications. Among other things, this can prevent users from sending (or trying to send) MBI or HBI classified information to people not allowed to receive or view it, including through DLP policies discussed in the next section.

Data Loss Prevention (DLP)

Data Loss Prevention policies allow organisations to:

  • Identify sensitive information across both SharePoint Online and OneDrive for Business sites (and in Exchange, through the same settings).
  • Prevent the accidental sharing of sensitive information, including information classified MBI or HBI.
  • Monitor and protect sensitive information in the desktop versions of Word, Excel and Powerpoint 2016.
  • Help users learn how to stay compliant by providing DLP tips.
  • View reporting on compliance with policies.

 

DLP Conditions

DLP works by giving Site Administrators the ability to create and apply DLP policies in the Security & Compliance Center for SharePoint (which includes OneDrive for Business; there is a separate Center for Exchange). In the Center, the Administrator navigates from ‘Security policies’ to ‘Data loss prevention’.

The DLP policy area includes a range of ‘ready-to-use’, financial, medical and privacy templates for a number of countries including the US, UK and Australia. Examples of pre-defined Australian sensitive information types include: bank account numbers, driver’s licence numbers, medical account numbers, passport numbers, and tax file numbers.

You may also create a custom DLP policy.

Sources: https://technet.microsoft.com/en-us/library/ms.o365.cc.newpolicyfromtemplate.aspx  https://support.office.com/en-gb/article/Send-notifications-and-show-policy-tips-for-DLP-policies-87496bc5-9601-4473-8021-cb05c71369c1

DLP Actions

Specific actions must be set for every DLP policy; that is, what happens if the policy conditions are met. The default actions are:

  • Block access to content (for everyone except its owner, the person who last modified the content, and the owner of the site where the content is stored AND send a notification by email.
  • Suggest a Policy Tip to users. Options are (a) Use the default Policy Tip or (b) Customise the Policy Tip.
  • Allow override options. There is one main checkable option (‘Allow people who receive this notification to override the actions in this rule’) and two sub options:
    • A business justification is required to override this rule, and
    • A false positive can override this rule.

In addition to these actions, where the DLP policy identifies sensitive content in a document stored in SharePoint Online or OneDrive for Business it displays a small warning ‘stop’ sign icon on the document icon. Hovering over the item displays information about the DLP policy and options to resolve it.

DLP Incident Reports

Incident reports are designed to alert a compliance officer to details of events triggered by the DLP conditions, and provide reporting on those events.

Sources:

https://technet.microsoft.com/en-US/library/ms.o365.cc.DLPLandingPage.aspx

Audited Sharing

Information sharing is a common activity in SharePoint and in SharePoint 2016 and SharePoint Online it is actively encouraged through a new Share option.

In addition to other existing audit options, sharing activity can now be audited in SharePoint Online. The audit logs for Office 365 (which must be enabled) are accessed through the Office 365 Admin Center > Security & Compliance Center > Search & investigation > Audit log search.

Source: https://support.office.com/en-us/article/Use-sharing-auditing-in-the-Office-365-audit-log-50bbf89f-7870-4c2a-ae14-42635e0cfc01?ui=en-US&rs=en-US&ad=US]

Information Rights Management (IRM)

Microsoft’s Information Rights Management capability provides an additional layer of protection for a number of document types at the list and library level in SharePoint Online sites.

Supported document types include PDF, the 97-2003 file formats for Word, Excel and PowerPoint (e.g., Office documents without the ‘x’ at the end of the file extension – ‘word.doc’, the Office Open XML formats for Word, Excel, and PowerPoint (e.g. with the ‘x’ at the end – ‘word.docx’), the XML Paper Specification (XPS) format.

According to Microsoft, IRM:

‘… enables you to limit the actions that users can take on files that have been downloaded from lists or libraries. IRM encrypts the downloaded files and limits the set of users and programs that are allowed to decrypt these files. IRM can also limit the rights of the users who are allowed to read files, so that they cannot take actions such as print copies of the files or copy text from them.’

IRM is enabled via the Office 365 Admin Center > Admin > SharePoint > Settings > Information Rights Management > ‘Use the IRM service specific in your configuration’ and then ‘Refresh IRM Settings’.

Microsoft_IRM

Image source: ‘Apply IRM to a List or Library’ https://support.office.com/en-us/article/Apply-Information-Rights-Management-to-a-list-or-library-3bdb5c4e-94fc-4741-b02f-4e7cc3c54aa1

 

When IRM is activated on a library, any file that is downloaded is encrypted so that only authorised people can view them. Again, according to Microsoft:

‘Each rights-managed file also contains an issuance license that imposes restrictions on the people who view the file. Typical restrictions include making a file read-only, disabling the copying of text, preventing people from saving a local copy, and preventing people from printing the file. Client programs that can read IRM-supported file types use the issuance license within the rights-managed file to enforce these restrictions. This is how a rights-managed file retains its protection even after it is downloaded.’

Source:

https://support.office.com/en-us/article/Set-up-Information-Rights-Management-IRM-in-SharePoint-admin-center-239ce6eb-4e81-42db-bf86-a01362fed65c

Shredded storage

Shredded storage, as the name suggests, describes the way documents are stored in SharePoint, starting from SharePoint 2013. Instead of storing a document as a single blob, documents are stored in multiple blobs.

This is a more efficient – and possibly more secure – way to manage documents when they are updated by only updating the element/s that were changed. According to a Microsoft presentation on 4 May 2016:

‘… every file stored in SharePoint is broken down into multiple chunks that are individually encrypted. And, the keys are stored separately to keep the data safe. In the future, we would like to give you the ability to manage and bring your own encryption keys that are used to encrypt your data stored in SharePoint. If you want, you can revoke our access to the keys. And we will not be able to access your data in the service’.

Source:

https://blogs.technet.microsoft.com/wbaer/2012/11/12/introduction-to-shredded-storage-in-sharepoint-2013-rtm-update/

Other Information Security related options

The Microsoft website ‘Monitoring and protecting sensitive data in Office 365’ provides further information about other Information Security options in Office 365, including reporting options to support auditing of activity in the tenant.

Source: https://msdn.microsoft.com/en-us/library/mt718319.aspx

 

Can ‘traditional’ records management survive the digital future

December 23, 2015

At a conference last year I listened with interest to a panel discussion on the subject of ‘paper versus digital’. Perhaps the debate was only meant to be light-hearted but it was clear that quite a few records managers continue to manage paper records and see them as their primary focus. In almost all cases these records are the printed version of a digital original. In most cases those digital originals remain stored on a network drive (or a person’s USB), or attached to an email, or – if you are lucky – copied to the organisation’s electronic document management (EDM) system to become the ‘official’ version.

Only a month ago I saw a colleague, whose position title clearly indicates she is responsible for digital recordkeeping, asking about KPIs for the creation of paper files. Perhaps it was part of the digital recordkeeping strategy to examine this subject, but it underlined for me the persistence of ‘traditional’ recordkeeping concepts, that documents belong in containers, that may be files or groups of files (or series etc). This is not just an issue for records managers, but for end users as well who continue to ‘think paper’.

It concerns me that records managers persist with the concept that digital records somehow ‘belong’ in digital files, often directly connected with the descriptive ‘business classification’ system of function and activity (in the sense that the file title or metadata may include the function and activity).

This way of thinking reinforces the concept that these records belong nowhere else, or might have no context outside the container, which of course is not necessarily true. It is, of course, possible to cross-link documents to a different container, but at what point is this a manual exercise? And where does it stop?

Part of the problem has been that digital systems mimic in their appearance and functionality the concept of a file or folder. We see them on network drives, in email, and in the containers of EDM systems. These folders or containers provide a sense of surety, that a document has been ‘stored’ or saved in a specific file. The file, however, is no more than a visual construct; the document’s file/folder/container metadata is what causes the document to appear that way.

This is all the more obvious in some EDM systems that store documents in a file store (often folder-based) and the metadata about the documents in a separate database. Aside from the folder structure, these documents are more or less a bunch of uncontrolled documents with metadata links to the ‘file’ construct in the database.

Of course, keeping records in some form of business context is important both for the management of those records (short and long term) and for retention and disposal purposes. But that doesn’t necessarily mean that documents must be associated with ‘files’ that, according to some records managers (and paper filing theory), should contain no more than 300 documents.

All of this is unnecessarily restrictive and, for end users, results in additional work. It’s no wonder that EDM systems have lagged behind the more ‘traditional’ way that users store documents, in network drives.

We should, I believe, think of digital records as being self-contained and dynamic objects with their own metadata payload that may be relevant in multiple contexts, not a fixed object stored in another fixed object.

Enter Microsoft Groups

In 2015 Microsoft announced the concept of Groups, accessed primarily from Outlook. Conceptually, you could think of a Group as being more similar to an Active Directory Distribution List rather than a shared mailbox. It is also similar to Yammer groups, but with much more functionality.

From Outlook, a user can create a Group to work collaboratively. Group members can initiate conversations with other members of the group (including Skype discussions) and – importantly – share documents.

So, Microsoft is taking us into the world of Group-centred information collaboration. This is not an experiment on the part of Microsoft, it is part of the world of Office 365 which includes SharePoint Online and OneDrive for Business, Delve, and more, supported by any device to enable users to create, access and share content anywhere, anytime.

Now, not every organisation uses Microsoft products, but an awful lot do. And for those who do, Groups in Office 365 is the future. The impact on traditional recordkeeping practices is likely to be large.

By 2020 or earlier, users will create Groups via Outlook to collaborate. Within the Group they will store digital content (not just documents); documents are stored in SharePoint Online site collections which are in addition to any other Site Collections. That’s right – in addition to your controlled Site Collections, there will be a multitude of Group-specific Site Collections. Conversations that take place in the group are stored as items in the inbox of the Group mailbox. In other words, all the content (and perhaps context) will be based around the Group ‘subject’.

To access all this rich content users will have three main options: (a) be an active member of the group; (b) SharePoint search; and (c) Delve. Business intelligence and data analytics, or e-Discovery type products, may also provide a fourth line of access.

Likely impact on recordkeeping

The recordkeeping practices that I believe will be most affected are: (a) metadata, including the application of BCS terms; (b) container-based storage, and (c) retention and disposal. It may also impact the careers of records managers. There will be no control over the creation of new groups (there can be, but why?), so their information content (in Outlook and in the associated SharePoint Online site collection) will become, in many respects, the new containers for records, in addition to other site collections.

On a positive note, I believe users will adapt and adopt this new model of working rapidly. Users, many of whom remain glued to Outlook as a business ‘collaboration’ tool, will find that Groups provide them with the ability to store and share documents via the Outlook interface. They may need to get used to the idea of ‘working out loud’ in Groups, but I think that will happen fairly quickly.

One thing is sure – change is inevitable.

Folders uninterrupted – the enduring problem of folders

November 19, 2015

In September 2014 I presented to the Records and Information Management Professionals Australasia (RIMPA)’s annual conference in Adelaide, South Australia, on the subject of folders and how it is possible to manage digital content differently using products like SharePoint.

Last week a couple of people in completely different parts of the company I work for demonstrated clearly the problem of folders on Outlook and network drives. The first had at least 100 folders and sub-folders in her Outlook where she said she ‘filed’ everything so she could find it again. The other person showed me a newly created section of the network drives with hundreds of new folders and sub-folders set up by someone who didn’t like or get the existing agreed folder structure that has been in place for many years.

In both cases I asked the same question – how do you find anything? The answer, in most cases, is that the person clicks on folder names that they created or are familiar with, until they find what they are looking for. Search was usually relegated to last ditch attempt to find things but was not considered to be very helpful or accurate. Both individuals lamented the duplication of information in folders, duplication they often didn’t know about until the inevitable clash of versions that occurs regularly.

At a workshop/presentation I did for another organisation recently, I was told that the organisation relied heavily on what are called ‘titling conventions’ to enable the information to be found. That is, the names of files, folders and documents follow a pre-defined model.

In reply to my comment that such titling was surely less valuable when you could search for the content in their document management system, I was advised that they didn’t yet have the ability to search by the content of documents (a resource issue). They could only search for content based on titles, once again reinforcing the need for folders (with unambiguous and consistent titles) to find that information.

Breaking the habit

I have said many times, folders are a very hard habit to break. Given the generally poor ability to search and find content in email and network drives (and in some EDRM systems – still), search is relegated to second place and folders and pre-defined titling conventions continue to dominate where content searching is not available.

For at least fifteen years, working across a range of organisations, I have had the ability to search within the content of digital records. This includes being able to search for the content of scanned documents that have been subject to OCR or optical character recognition. In this digital age I’d be very disappointed if the only way I could find content was by navigating labyrinthine folder structures on email or network drives.

I have fewer than five sub-folders under my work email inbox. I use those additional folders to route specific content such as junk mail, SharePoint notifications or personal email. I don’t save anything to a network drive and only use SharePoint to store and retrieve records and other corporate information. Why? Because search is really good.

I know this is not common practice around the organisation. Common practice is to create and maintain multiple folders and squirrel useful content away in network drive folders. It’s been that way for 30 years.

I don’t think folders are going to vanish from the landscape anytime soon – they are no more as unlikely to vanish in the next 10 years as email, although both are slowly morphing over time into something different, particularly as ‘collaboration’ tools, search, and the presentation of information via analytics engines are introduced.

Content finds you

Facebook and Google are successful for a reason – they both present content (including advertising) relevant to the user based on her or his digital content. A simple analogy is to compare the corner store where you know you can go to get milk with the ability of digital devices to remind a user they need to buy a specific type of milk because (a) their internet linked refrigerator has noticed that milk is running low and (b) they regularly post to social media (or even in a single email) about their preference for this type of milk. It’s creepy, but it works .

Many contemporary office environments are not much different – users know they find their digital content in the folders (the little corner stores) they created or have learned about. But they don’t know how much other interesting or directly relevant content is there and they don’t bother looking because they know it will be a poor and time-wasting experience.

They don’t know what they don’t know.

My experience over the past five years with SharePoint (and with similar products before it) is that users are resistant to ‘having’ to store documents in a different location to what they are used to (i.e. network drives).

Users worry their content will be lost, it’s ‘yet another place’ to store and find content, and the user experience (compared with folders) is alien. This doesn’t stop them consigning endless information to social media and being the recipients of targeted advertising and suggested friends based on their content. The difference between the two is so extraordinarily different it is sometimes hard to believe that users do this, checking in on Facebook’s content-driven feed while ‘navigating’ through endless folders, many with names that are about as vague as you can get.

Where is this heading

In the last weeks Microsoft has outlined what is sees as the future of the office and their Office. Three key elements of that future are:

  • SharePoint (on-premises and online) and OneDrive. These two will not replace folders completely but will give users a much better (and more importantly, more mobile) experience with storing and finding content.
  • Office 365 Groups. Groups of people who have a common interest and can communicate across platforms, whether it be via email or other collaboration tools including SharePoint or Yammer.
  • Analytics as the way we will harness and access our information, via interfaces like Delve that serves up relevant information based on the user’s digital content. Just like Facebook.

Does Delve deliver on the ‘Semantic Office’?

Many years ago I wrote on this blog about what I called at the time the ‘Semantic Office’. In that article I asked whether we will one day experience a time when we can truly harness the digital content in our business systems to link and present relevant information to users based on the same methods found on the internet at that time (for example in eBay, Amazon etc).

I think Microsoft has finally figured this out with Delve. My guess is that, by 2020, Delve or a version of it will be the most common ways users first access their content.

So where does this leave folders?

Folders aren’t going to go away anytime soon. I’d like to think that Microsoft took a close look at how they could deprecate folders in email and drives and realised that this was akin to giving up on email – it wasn’t going to happen.

Instead, Microsoft decided to introduce new tools that can capture and – more importantly, perhaps – present information in completely different ways that digital natives expect and even take for granted. As someone said recently, when a large group of young people were asked to send an email, many responded with ‘what’s email, why can’t I Inbox you?’.

Change is happening, slowly, along with the culture shift that must accompany it. For every older ‘paper native’ worker that retires, a younger digital native commences work in the workforce, bringing with them new expectations both in terms of work experience and tools they use.

Just like the clacking of manual typewriters in offices was eventually replaced by the slightly smoother sound of electronic typewriters, and then the click-clack of keyboards hooked to the back of computers, the silent touch screens of the future will eventually dominate and then themselves be replaced by something else.

Habits are hard to break, but change happens.

Managing documents in SharePoint 2013 and SharePoint Online – what’s changed?

September 20, 2015

We are in the process of upgrading our extensive ‘out of the box’ SharePoint 2010 environment to SharePoint 2013 and, at the same time, setting up a limited SharePoint Online presence. So, what’s changed in relation to managing documents as records?

In SharePoint 2013, the short answer is ‘not much’. However, there are some new things that will change some parts of our technical design model. The things that have remain unchanged include:

  • Document libraries as the primary ‘container’ to store documents, using folders, document sets, or metadata-based categorisation, to ‘group’ documents.
  • Document IDs, set in the Site Collection Administration – Document ID settings section.
  • Document versioning (major/minor, major, or none), set in the Library Settings – Versioning Settings. Other settings in this section, which we generally do not use, have also not changed: Content Approval, Draft Item Security, and Require Check Out.
  • Use of Content Types (and disabling of Folders), enabled via Library Settings – Advanced Settings, and then added in the general Library Settings – Content Types area. All other options in this section, which again we rarely use, are still there: Custom Send to Destination, Search visibility, Offline Client Availability, Site Assets Library, and Dialogs. The old familiar ‘Datasheet view’, which we use to ‘bulk upload’ or update metadata has been renamed ‘Quick Edit’.
  • Almost unlimited metadata options via pre-defined Content Types or library-specific columns, both of which can point to the centrally controlled metadata in the Managed Metadata Service or local ‘look-up’ lists.
  • Multiple list views, each with their own linkable URL.
  • The ability to copy documents, via drag and drop or copy/paste, using ‘Open with Explorer’. This, coupled with the ‘Quick Edit’ option, allows documents to be copied to SharePoint document libraries in bulk and metadata added easily.
  • Access controls that can be applied right down to the document level.
  • The ability, from a document-specific drop down menu, to view or edit the properties of a document, check it out or in, view the version history (and restore versions), run workflows, download a copy, share (the former ‘manage permissions’ – see below), and delete (where enabled).
  • Out of the box simple workflows for Review and Approval.
  • Site collection audit trails, accessed via the Site Collection Administration area. Unlike some other products, SharePoint audit trails are not ‘attached’ to individual documents, but are centralised in one place.

So, all in all, not much change really, except for the ‘Share’ option. In many respects, the way the simple ‘Share’ has been designed is a more intuitive process than ‘Manage Permissions’. When you add a user you decide if that person should have Edit or Read privileges. If you maintain the default Owner, Member and Visitor groups, then those with Edit rights are added to the Member group, those with Read rights are added to the Visitor group.

For more complex permission management, including to stop inheriting the default permissions (or to add them back, which is now called ‘Delete unique permissions’), or creating new groups, you need to select the ‘Shared With’ option from the drop down menu and then select Advanced.

What about disposal/disposition management?

Again, not much has changed in relation to document libraries or lists. Out of the box, your options are as follows:

  • Using centrally defined, document-based, Content Types, using Information management policy settings. Not a bad idea if (a) you have a way to ensure that these Content Types are always added to libraries, and (b) you are happy to manage the disposal of documents one by one.
  • Changing the default Information management policy settings option in document libraries from ‘By Content Types’ to ‘By Libraries and Folders’ and then applying retention policies on the folders you create. The main negatives of this option are that it means you have to use folders, and you have to manage disposal by document.
  • Leaving documents in document libraries, and having a way to review these, across the farm, in a centralised manner. This requires some kind of script to be written to (a) list all libraries across the farm and (b) work on the basis that the ‘Last Modified Date’ is the last action on any document in the library, but it seems the more logical and simplest way to achieve the outcome you seek, and keeps all the documents in the same container.

It remains surprising to me why Microsoft does not provide the option to set a disposal period on an entire document library.

Of course, SharePoint 2013 now allows you to set a disposal period on a Site, but this isn’t likely to work for sites that contain a range of diverse content that may be useful over a long period of time.

So, what about SharePoint Online?

The first thing to remember about SharePoint Online (SPO) is that it’s not SharePoint On-Premises. Seems obvious, but the natural instinct is to wonder if or how the two environments can be connected. In most cases they can’t, so it’s not worth thinking along those lines. SPO is a way to manage content in the cloud in addition to, or instead of, on-premises.

SPO has all the same document management features you find in SharePoint On-Premises, described above – document IDs, versioning, content types, metadata, multiple list views, open with Explorer and Quick Edit, access controls, document-specific menu options, simple workflows and audit trails.

O365_SPO_LibraryRibbon_DocMenu

The options for disposal/disposition management using SPO ‘out of the box’ (should that be ‘out of the cloud’?) is the same as for the on-premises version.

You didn’t mention Records Centers …

Records Centers (or in Australia, Centres) were in many respects designed to be the ‘send to’ archival repository for other sites. Great idea if it works in your world, it doesn’t work in ours. The main drawbacks are that documents ‘sent’ to a Records Centre are in fact copied. Custom metadata is lost, versions are lost, audit trails are lost. And, you can retrieve the document.

But Records Centres (or in the US, Centers) can be useful on their own as a repository of specific types of records that aren’t accessed too much. We have several Records Centres and we use them in a separate web application for specific purposes, including to store scanned documents. We don’t use them as they were originally intended for the reasons stated above.

And yes, you can create Record Centers in SPO, too!

Sentencing and disposing of records in SharePoint 2010

August 28, 2014

This article applies only to on-premises installations of SharePoint 2010. 

SharePoint’s ‘Records Centre’ was, in theory, a place to send records from various sites for long term storage, sentencing and disposal. The idea was that you could automatically, via Content Type rules, or directly, via the ‘Send to Records Centre’ option, transfer records out of team (and other) sites into the Records Centre. 

While the theory made sense, in practice there were several problems with the model, not the least being that ‘transferred’ records were not actually transferred but copied and effectively re-registered as new records in the Records Centre. They also lost previous versions, and so on. 

We needed to find another way to manage the sentencing and disposal of records. SharePoint 2013 has the (information management) option to apply a sentence on an entire site, which is great if you want to do that, but in most cases the requirement is to sentence parts of a site, including whole libraries or lists. 

As our document-based records are stored in document libraries, and these libraries generally (but not always) have names that make sense (if you can stop people using the generic ‘Shared Documents’ default library), it seemed a good idea to focus on how we could apply retention and disposal policies to document libraries. 

The first problem is visibility of all those libraries, created across multiple site collections. The only way to see all of them was to be the SharePoint Administrator or have Site Collection Administration privileges across all sites. But it was cumbersome to have to open every site to see what was stored in them. 

I put this problem to our SharePoint Administrator and developer (Eric Fang – blog here: http://fangdahai.blogspot.com.au/) . Using PowerShell scripts, Eric developed a method to display all document libraries across all Sharepoint sites in a list. The list updates on a regular basis.

NOTE: You cannot create this type of list ‘out of the box’, it requires PowerShell scripts, and that will need to be maintained over time. This is not a skill that is normally found with most SharePoint Administrators. 

The list displays:

  • The library GUID
  • The library name and URL
  • The site collection and sites, including URLs
  • The number of items stored in each library
  • The date the library was created and who created it
  • The date any item in the library was last modified

The first, immediate, benefit we could see from this method of displaying libraries was the number of libraries that had been created but not used. We could immediately see the ability to reduce the number of libraries, especially if these had not been used for a given period (say, one year).

The next benefit was the ability to group libraries by site collection. As many of our site collections map to business functions, we could start to see the volume of content that was stored for each function, by library – many of which were created to store documents created through various activities.

For example, a common document library is ‘Meetings’. We can now filter the view to see all libraries that contain documents relating to meetings. We can also see types of libraries that have specific retention and disposal requirements.

While the list of all libraries has provided an excellent way to view all our libraries, we are now working on a method to apply retention and disposal actions to these libraries. One way to do this would be to add an extra column in the list for retention and disposal information (class number, class decription, disposal action, expected disposal date, approved by, etc).

Once a disposal action had been applied to the list, we can then review it when the disposal action became due to determine if the library could in fact be destroyed. If it can be destroyed, it would be possible to export the original library metadata columns to a spreadsheet to keep a record of what was destroyed, and when.

Planning access controls in SharePoint 2010 team sites

October 15, 2013

A key factor that needs to be considered when planning for the creation of team sites for the management of records is access control. Getting this right (or as close to right as possible) early on for team sites with sub-sites will save a lot of potential pain later on.

Team sites will normally have three levels of access permission:

  • Site owners. Full control. One to three for each site collection.
  • Site members. Contribute rights, i.e., add/edit/delete.
  • Site visitors. Read only access.

There are generally three ways to manage access controls in team sites that have sub-sites:

  • Inheritance, broken as required. Top level site owners control all sub-sites, site members and visitors can be the same.
  • Independent, within a site collection. Different site owners/members/visitors between the top site and sub-sites – although the site owners could be the same.
  • Independent, separate sites but linked. Different site owners/members/visitors for each site.

Inheritance model

This is the default access model for team sites with sub-sites and is enabled when a Site Administrator creates a new sub-site without choosing ‘More options’ on the ‘Create’ Team Site dialogue page. Site owners control the top and sub-sites, site members and visitors are the same unless inheritance is broken on any part of the site (sub-sites, libraries or lists, or documents).

  • The benefits of this model is that site owners control all the site collection and both members and visitors can be the same.
  • The negatives relate to the effort involved in restricting access and giving unique access to sub-sites, libraries/lists, or documents. For example, to provide access to a specific area, a person who is not part of the default members or visitors group must be either added to one of those groups, or added individually. When they are added individually by user name, their user name appears across the entire site collection; they will be able to navigate ‘up’ (and down) but they won’t see any content in any page, library or list. Another negative is that, once selected, it is not possible to revert to the independent model (unique permissions); it is only possible to break the inheritance model and create new security groups which is a bit messy.

Independent model, within a site collection

The option to select this model must be selected when the new sub-site is created by clicking on ‘More Options’ after the Title and URL name are entered. A new dialogue box opens with a section ‘Permissions’. This notes ‘You can give permissions to access your new site to the same users who have access to this parent site, or you can give permission to a unique set of users’. It adds, ‘If you select “Use same permissions as parent site”, one set of user permissions is shared by both sites. Consequently, you cannot change user permissions on your new site unless you are an administrator of this parent site’.

There are two options:

  • Use unique permissions
  • Use same permissions as parent site (the ‘inheritance’ model above).

Again, there are positives and negatives to this model:

  • The benefits of this model are that you can have unique access permissions on sub-sites, removing the requirement to break inheritance or worry about what users with access can see on the other parts of the site collection. Each sub-group within a team can have their own site that cannot be accessed by the other parts of the team – although you might consider giving other parts of the team visitor access, with the sub-group having ‘contribute’ rights.
  • The negatives of this model are in the requirement to manage multiple access permissions for the top level site and sub-sites. Consider a common team environment – normally only a couple of people will be the site owner. If the site collection has multiple sub-sites with unique permissions, each of those sub-sites will have their own site owners. Of course, you can assign the same person to the site owner of each sub-site, but it is still a degree of overhead you don’t get with the inheritance model.

Independent model, separate site collections

This model simply consists of either of the first two options, with separate site collections added as links either on the global (top) or current (left hand side) navigation. End users don’t know that these links are completely separate sites unless they look at the URLs.

  • The benefits of this model are the same as the benefits for the second option above (independent model). Each site collection has its own unique permissions. It also allows you, if you restrict team sites to one sub-site level, to have sub-sites on the linked sites, to get around that restriction. Separate site collections could have either ‘open’ access to everyone, or ‘closed’ access.
  • The negatives of this model are also the same as the negatives for the second option above. Generally, however, there will usually be a good or compelling reason for having a completely separate site collection linked to the primary team site, and this often relates to the proposed site audience. For example, members of the team may be involved in a project; the (separate) project site can be linked to the main team site, but only those members of the team site with access will be able to see it.

In summary, it is important to consider access controls carefully, understanding both the benefits and the negatives of each option, then plan and implement accordingly. Otherwise, if there is a requirement to change the access type, this could be difficult to implement later on and the sub-site may have to be re-created.

Is your SharePoint team site a ghost town?

May 24, 2013

One of the best ‘unknown’ (to end users, anyway) features of SharePoint 2010 and SharePoint 2013 is ‘Site Web Analytics’, hidden away under Site Settings. This, out of the box, feature provides useful information such as the average number of site views per day, the average number of unique visitors per day, trends, as well as a listing by user names of the top visitors and the number of their page views. It’s a great way to see if a site is being used.

The default date range for Site Web Analytics is the past 30 days but the settings can be changed to the last day, week, month, quarter, half-year or year. Custom dates can also be set, and these reports can be exported to a spreadsheet for further analysis, including through scheduled reporting sent by email.

So, what does this have to do with the subject – is your SharePoint team site a ghost town?

Simply, Site Web Analytics tells you if your team site is being used – by your team. If it’s not used, then it’s a ghost town – and you need to understand why and consider changing or deleting it.

In many cases, the reason a SharePoint site becomes a ghost town is because users simply neither want nor know how to use it.

What makes a document management system popular?

My experience implementing and using many other document management systems (TRIM, Documentum, iManage, Alfresco) over the years, along with considerable anecdotal evidence, suggests that, unless there are legal or regulatory requirements mandating it, end users tend to baulk at using a document (and records) management system, especially if it:

  • Requires more than 20 minutes to learn. Some EDMS systems require hours of training – for end users!
  • Duplicates what they do already with network drives. More often than not it requires ‘saving’ an existing born-digital record to the recordkeeping system (AND leaving the original where it was stored).
  • Stores records of a business activity away from other, potentially rich, related records that are not documents.
  • Provides no obvious or little tangible business value for the user – it’s a ‘compliance overhead’.

A popular document management system is one that:

  • Users can start using the system with very little training. It should be intuitive and make sense at first glance. I often use the example of Facebook – no-one ever got training to use it and yet a billion people registered to use it.
  • Reduces (not necessarily eliminates) duplication of effort. Network drives can still exist to create and manage ‘working documents’ instead of final versions.
  • Can keep records in context with other related records such as calendars, lists, tasks, announcements, social networking messages, and so on.
  • Provides tangible business value for the user, with minimal effort. It’s hard to break the network drive habit but simple things like unique document IDs, versioning, hyperlinks, and the ability to email a link instead of a document help to break that habit.

What can turn a SharePoint team site into a ‘ghost town’?

Microsoft clearly put a lot of human-computer interaction (HCI) effort into the user interface design of SharePoint team sites (along with all their other products). Team sites in SharePoint 2013 are a design improvement over SharePoint 2010 but still maintain the same essential look and feel, which is:

  • Document libraries and lists on the left hand side navigation. It’s no coincidence that this is exactly where network drives and email folders are accessed from.
  • An editable main page that can be modifed as required to present additional information or options, including via webparts.
  • Default and modifiable views of documents and lists.
  • A simple search option on the top right.

Over 25 years of network drive use has conditioned users to navigating for documents from the left hand side, searching from the top right hand side, and seeing information presented in the middle of the page. Change this paradigm and you start to lose users.

I believe there are two main reasons why users don’t or stop using team sites (described further below):

  • The ‘plain vanilla’ out of the box functionality in a brand new team site.
  • Site pages that have been customised to the point where end users cannot work out how to navigate or access their information.

Out of the box generic functionality

Out of the box team sites include the two features which, if not addressed or re-configured (a 2 minute job), can negatively affect take up rates:

  • ‘Shared documents’ under ‘Libraries’. This to me is a ‘bookmark’ library that should be removed almost immediately and replaced with a library that has a name that makes sense, ideally named after a business activity. Otherwise, ‘shared’ is like ‘general’ in a network drive. If users have only one, generically-named library to choose from, they are unlikely to use it (compared to the rich tapestry of their network drives).
  • Folders. Unfortunately, while SharePoint folders look like network drive folders they don’t work the same way. They create unnecessarily long URLs and are often impossible to navigate. It’s not uncommon to hear users say they cannot find documents in a site because they cannot work out how to navigate folders (as they don’t include a + to guide navigation). My recommendation is to turn them off under Advanced Settings after the library is created. However, users inevitably want a way to categorise or group documents in a document library. Document Sets generally make sense to users and get quick take up but you can only have one level. A good site has a reasonable number of smartly-named document libraries – perhaps no more than 10, grouped as required by Headings – using Document Sets instead of folders. Another form of categorisation is using categories in the document metadata and then creating a view that groups the documents by those categories.

Customisation

Too much customisation of site pages can also turn users off. As noted above, it’s worth keeping in mind that Microsoft spent a lot of effort (and probably dollars) on designing their standard team site user interface. Some of the common modifications I have seen on ‘ghost town’ sites include:

  • Using themes that make the site look completely different from all other sites. It’s important to maintain consistency in themes.
  • Using unnecessarily large font sizes on the page.
  • Using too many images or graphics on the front page.
  • Assuming the text or content on the page is actually of interest to the target audience. I usually suggest that Site Owners (note, there should never be just one) agree among themselves; if this is not possible, test the proposed page design on a user who has never seen it.

Avoid ghost town sites – keep users coming back

Remember, most users are familiar with using network drives and folders. Use document libraries with sensible names that are obvious to users, don’t use folders, and keep the front page content simple and easy to view, and you are more likely to attract users back again and again.