Using PnP PowerShell to move folders and document sets in SharePoint Online

In a previous post ‘Observations on ‘Move To’ in SharePoint Online‘, I noted that the ‘Move To’ functionality was generally suitable for moving 1000s of files between sites in the same tenant subject to a number of conditions.

But what if the ‘Move To’ function does not show the destination location? There are several suggestions to make the destination appear including by adding content there, following the site and so on. If none of these work, the next option may be to use PnP PowerShell.

This post describes how to use a remarkably simple PnP PowerShell ‘script’, either for one-by-one folder moves or multiple folder moves, to move folders (including document sets) that contain other folders or files between libraries in different sites (or between libraries in the same site) in the same tenant in SharePoint Online. It can also be used for moving individual files/documents but it’s a lot easier to do it when they are stored in folders or document sets.

Despite its simplicity, it can be easy to make a mistake. Fortunately, PowerShell provides reasonably detailed error messages.

Starting point – access rights

Using PnP PowerShell to move folders requires elevated privileges. These privileges are usually assigned to Admins via Role Based Access Controls (RBAC). The SharePoint Admin role can usually run commands that relate to SharePoint but note that some (such as those sites connected with Microsoft 365 Groups or Teams) may require other roles.

Running PowerShell commands assumes that the person running the script

• Knows at least the basics of how to run PowerShell (although it’s not hard to learn!)
• Has the required PowerShell and PnP installed on their machine
• Has the required access to content in both the source and destination sites.

To learn more about managing various parts of Microsoft 365 with PowerShell, see this Microsoft page ‘Manage Microsoft 365 with PowerShell‘.

See below for the steps followed to run the script.

First step – document the source library

For the purpose of this post, SharePoint library moves/migrations involve moving content from a library in one SharePoint site to another library in the same site or a different site, in the same tenant.

It is important to: (a) know what content and metadata exists in the source library; (b) be able to confirm that all the content, including metadata was moved and if not, why not; and (b) ensure the integrity of the migrated content is maintained – to the extent that it is possible (see below).

So, as a starting point, the source library (or libraries) should be documented. This involves the following:

• Exporting a full set of metadata that shows the complete detail of the library. See below.
• Documenting all permissions (to the extent that it is possible, libraries with multiple and complex folder-based permissions may be very hard to document) and deciding if these are to be retained or lost in the destination.
• Documenting all library columns. Take a note of every column and the column type and any additional elements such as character limits or formulas. This information is required in the second step below.
  • For each choice column, note what what the choices are.
  • For each managed metadata column, note all the managed metadata values that have been applied and if they still exist.
  • For any lookup column, note what list the column looks up and what impact this may have if the library is migrated. How will it be replaced?
  • For each date and time column, note if the time is included or just the date.

It is important to export a full set of metadata for the library as a CSV file, from the ‘Export’ option in any library menu. ‘Full set’ means:

• Metadata in the form of the folder or document set names (which will be used again, see below)
• Any document set metadata that needs to be retained to show what was applied before.
• It may also be useful to create a ‘no folder’ view and export the full set of metadata for all the documents.

Note: As indicated already, pay particular attention to the values in choice and managed metadata columns. It is not uncommon for choices or managed metadata to be changed or removed, leaving legacy values that no longer exist. These will NOT migrate if the destination library doesn’t have them.

Take a note of the source library URL, usually ‘sites/sitename/sourcelibraryname’ (‘/sites’ could be ‘/teams’) as this will be required later.

Second step – set up the destination library

The PowerShell script described in this post does not create a new library as part of the move. The new library will need to be created and configured in advance, based on the details from the source library documented in the previous step. Note that the moving of content provides an opportunity to rename the library.

Using the information obtained in the first step above:

• Create all the required site or library columns (but not any system columns such as Created, Created By, Modified, Modified By).
• Populate any choice columns with the relevant choices. Note that values that were applied in the source library, then changed or removed, will not migrate if they don’t exist in the destination library.
• If there is any lookup column, make sure this is in place.
• Cross check with the documentation from the first step.

Take a note of the destination library URL, usually ‘/sites/sitename/destinationlibraryname’ (‘/sites’ could be ‘/teams’).

Third step – familiarise yourself with the PnPOnline options

All the details below assume the reader or the person who will run the PowerShell script has the requisite permissions as noted earlier.

Review this GitHub web page ‘Move-PnPFile | PnP PowerShell‘ and in particular note the options as some of these will be used in the script below.

• -AllowSchemaMismatch (will move even if metadata mismatches)
• -AllowSmallerVersionLimitOnDestination (helps to reduce the version history)
• -Connection (unlikely to be required)
• -Force (no confirmation will be requested)
• -IgnoreVersionHistory (only latest version will be moved)
• -Overwrite (overwrites whatever is in the destination)
• -SourceUrl (used in the script)
• -TargetUrl (used in the script)
• -NoWait (could be useful but suggest not to use)

Starting PowerShell and connecting to PnP

On a local machine, start Windows PowerShell ISE, as administrator. Do not run Windows PowerShell as it probably won’t work.

You should see the following:

PS C:\WINDOWS\system32>

If not already installed, run the following command to install PnP PowerShell:

PS C:\WINDOWS\system32>Install-Module -Name "PnP.PowerShell"

(You will be asked to confirm, say ‘Yes to all’).

Now connect to PnP:

PS C:\WINDOWS\system32> Connect-PnPOnline -Interactive

This command – if you have everything installed and the relevant access – will request a URL. In this case, since we are moving from one site to another in the same tenant, enter the root site URL similar to the example shown below.

You will be asked to authenticate. If authentication fails, check with your Global Admins. Otherwise, you will be returned to the command line:

PS C:\WINDOWS\system32>

Moving folders one by one

To move a single folder to a destination SharePoint site, run the following command, substituting the various elements as required. It may be a good idea to copy this line to Notepad and modify it there.

PS C:\WINDOWS\system32>Move-PnPFile -SourceUrl "/sites/sourcesitename/sourcelibraryname/foldername" -TargetUrl "/sites/destinationsitename/destinationlibraryname" -AllowSchemaMismatch

Note

• The above code will request confirmation for every move. This is not a bad idea to leave enabled, especially if there are issues.
• Do NOT include the ‘/’ after the source or destination library name. This will throw a ‘not found’ error.
• Content can be moved between both /sites or /teams.
• Note that ‘-AllowSchemaMismatch’ is enabled. This is because, after a thorough review of all the source metadata, the script will allow migration to happen even if there is a mismatch with some of the metadata. If this option is not included, there may be many error messages.
• When moving document sets, sometimes the path (/sites or /teams) in the source library is not always required. If there is a ‘not found’ error, try removing the source site path (/teams or /sites).

See below for issues that can arise with this option and the next.

Moving folders in bulk

Moving folders in bulk is a variation of the above but involves the creation of a PowerShell (ps1) file, which is very simple.

It is a good idea to test this script with a few folders at first.

Step 1 – Create a master CSV file

Open the original CSV file containing all the metadata for the library and extract the list of folder or document set names as a new CSV or Excel file. Keep this aside as your master version as you may need to refer to it during the migration.

Step 2 – Create the CSV file for the migration

Create a version of the CSV file with only a few folders, for initial testing. The CSV file should have one column with the word ‘Folder’ at the top (cell A1). Each subsequent row (A2, A3, etc) should be in the form below.

• /sourcesitename/sourcelibraryname/foldername (or docsetname)

Tip: If you have a list of folder names, put these in column B, then put the preceding elements in column A. Concatenate (e.g., =CONCATENATE(A2,B2) the two columns in column C, then copy the values back to column A (and remove the rest of the columns).

Save this CSV file somewhere where you can access it again. You will need to put the location of the file into the script below.

Step 3 – Prepare the ps1 file

Copy the text below to Notepad, modify it as required and save it as a ps1 file (save as – example.ps1). You will be able to continue editing the file via Notepad. Take a note of where you saved this file as you will refer to it again when you run the script.

$folders = Import-csv -Path "c:\users\username\downloads\SourceFolders.csv" | select -ExpandProperty Folder


Foreach($Folder in $folders)
{
      Move-PnPFile -SourceUrl $Folder -TargetUrl "/sites/destinationsitename/destinationlibraryname" -AllowSchemaMismatch		
}

Step 4 – Run the script

To run the above script, in the PowerShell window, either put the full path where the ps1 file is located, or navigate to the location.

• To navigate to the location, go to the ‘root’ C: drive (enter ‘cd\’ or ‘cd ..’), then ‘dir’ to see the content then ‘cd users\username’ etc to get to the location of your ps1 file.
• Once you have reached the location, enter ‘.\’ and you should see all the files available including your ps1 file.
• Select your ps1 file and press enter (similar to the example below).

PS C:\users\username\documents\powershell> .\MoveFoldersBetweenSites.ps1

Issues to be aware of

Generally, either option works well for moving folders between libraries. There were three main issues identified:

• Cannot find the file specified. This comes up as an error in the PowerShell window itself, usually with the preceding words ‘Failed to verify the existence of source object at …’. In every case, this error was caused by leaving the ‘/’ character at the end of the source string (e.g., sitename/libraryname/foldername/). A similar error is thrown when the same character is left at the end of the destination string.
• Folder and document path length. In a few instances, this required re-creating the source folder/sub-folder path in the destination and trying again with individual folder moves. Where this failed, the only option was to download the content and re-upload it. This is not ideal from a file metadata integrity point of view but hopefully will be relatively rare.
• Files that were still checked out. These files did not move. In these cases, the only option (to retain the integrity) was to (a) discard the check out and (b) run the ‘one by one’ option above using the folder/file path and name.
• Minor versions. This issue was surprisingly common. The files that did not move all had a version 0.1, then 1, and sometimes 1.1. Deleting version 0.1 seemed to fix the issue. This issue also occurs with other migration tools.

Advertisement

• Migration
• Moving Content
• PowerShell
• Products and applications
• SharePoint Online