Part 4 - Migrating a SharePoint 2007 farm to SharePoint 2013

Getting There

In Part 3, we showed how we created scripts for our data migration from SharePoint 2007 to a SharePoint 2013 mirror (our methodology explained in Part 1) using SharePoint 2007 inventory data (covered in Part 2).  In our final part, we'll discuss how that all came together; to the point where we could migrate 60 sites with 1.5 TB of data from SharePoint 2007 to SharePoint 2013 in just six hours.

Prepping the Production Farm

We already had a DEV farm which held our mirror.  The mirror was an exact copy of our SharePoint 2007 content in a SharePoint 2013 farm.  Since it needed to be maintained as an exact copy for our methodology it work, that meant it couldn't be used for developer testing or user acceptance testing (UAT).  The solution for that quandary turned out to be fairly easy - we set up our production farm, which we called PROD, in the production configuration we wanted to use when we went live.  That included creating the Web Applications, content databases, and services.  The Web Application was created to respond to the production hostnames (which, as we stated in Part 1, needed to the same as was being used in the SharePoint 2007 farm).  We then Extended the Web Application to respond to a second hostname, which we called TEST.  This achieved multiple things:

• Allowed us to create a TEST environment for users to manipulate data and break things while leaving the mirror pristine
• Allowed us to test Production equipment and configuration during UAT, since users were testing on what would be the Production environment
• Allowed us to test and validate our step-by-step migration procedure and scripting, since the steps would be exactly the same when updating the TEST instance or the PROD instance
• Eliminated discrepancies that could exist between UAT and production, since even identically configured environments can have subtle differences that could cause problems when migrating to PROD that didn't exist when migrating to TEST
• Allowed us to test the Production environment's response to production hostnames before going live by modifying the host file on the servers

By configuring our Production farm in this manner and using it as our Test environment, by the time the day came for migration, we had effectively already done the migration dozens of times - making the final migration (the one that mattered) go off without a hitch.

Putting the Mirror Into Production

Whether we were copying the mirror into our TEST environment, or actually conducting the final migration, the steps were exactly the same each time:

1. Run the mirror synchronization script
2. Import the script output files into our inventory database
3. Review script result reports to identify issues that may have occurred during mirror synchronization
4. Fix synchronization issues
5. Backup the mirror
6. Restore the mirror to the Production farm

Since we scripted most of these actions, and we practiced/executed this process so many times, we found the whole thing could actually be accomplished mostly hands-off in as little as 4 hours:

• 2 and half hours to synchronize the mirror
• 30 minutes to review and fix synchronization issues
• 1 hour to back-up the mirror and restore it to the Production farm

We only did one thing differently when putting the mirror into final Production - after running the mirror synchronization script, the SharePoint 2007 farm was placed into read-only mode.


Once we put the mirror into Production for the final time, only a few steps remained:

• Remove the TEST host header from the Web Application
• Change DNS entries for the hostnames
• Change host files in the old SharePoint 2007 farm, allowing us to still access SharePoint 2007 using the same URL as long as we were on one of the old servers

Conclusion

From a user perspective, they went home on a Friday using SharePoint 2007 and came in on Monday using SharePoint 2013, and it was business as usual.


We couldn't have asked for or envisioned a better outcome, but that's exactly what happened.  Sure, there were issues here and there (what migration goes off with zero hitches?) - but nearly all the issues that were brought up were anticipated, and most were fixed very quickly.


We had gone from not even knowing how we could do a migration from SharePoint 2007 to SharePoint 2013 to successfully doing the migration so often to TEST we could do it in our sleep - or from another perspective - scared out of our minds to confident and optimistic. 


Being organized and methodical, taking and keeping inventory in a database, and using the right tools (ShareGate and PowerShell) all contributed to our success.  Those of you reading this who have done migrations before may have noted I've left a lot of things out when I talk about how we did all this - and you are right.  There were many other factors and things we did that were key to our success, but they are the same ones that apply to any migration - engaged project sponsorship, frequent engagement of site owners, requirements gathering, documentation, training, and testing - just to name a few.  We wanted to highlight the way we handled the technical issues of migrating a huge amount of content from SharePoint 2007 to SharePoint 2013.  Hopefully we've done that - and if one day this story helps someone else complete a successful SharePoint migration - even better.

< Part 3 - Using the Tools to Get There

Part 3 - Migrating a SharePoint 2007 farm to SharePoint 2013

Using the Tools to Get There

We've discussed the methodology we used to migrate from SharePoint 2007 to SharePoint 2013 (using a mirror), and how we took stock of what we had to migrated (using various tools), so we'll move on to how we actually built our mirror.

Setting Up The Mirror

The first SharePoint 2013 farm we stood up as part of our migration process was called DEV.  It was not very robust, and our intent was to use the DEV farm for development after our migration to SharePoint 2013 was complete.  In the meantime, its primary function was to house our mirror - an exact replica of our SharePoint 2007 content, but housed in SharePoint 2013.  Many services weren't even configured in this farm - just a few that we wanted to test or validate - such as Search and SQL Server Reporting Services (SSRS).  We setup the Web Applications such that our current URL structure would be maintained between SharePoint 2007 and SharePoint 2013, but with a different host name.  We also setup the database files just like we wanted them to be in our final product, since the actual migration would be accomplished with a database backup and restore between our two SharePoint 2013 instances (DEV and PROD).

Filling the Mirror

Creating the mirror was the easy part - filling the mirror with content is what consumed 80% of our schedule.  Recall from Part 1 that one of our success criteria was to make improvements where we could - this meant that each library and list was reviewed to see what, if any improvements could be made.  One of the biggest areas of improvements we wanted to focus on was Search - we wanted our Search to be more usable - which meant trying to create improved content types and site columns for better search results.  Ultimately, we came up with a procedure that was repeated for each site:

1. Create the site using SharePoint 2013
2. Enable/disable required site features
3. Migrate all site pages, lists and libraries (including workflows and public views) without content using ShareGate
4. Cleanup/adjust permissions from what was migrated (part of our 'making improvements where we could' effort)
5. Make desired improvements to content types, lists, or libraries
6. Create the script for content in each list and library using ShareGate PowerShell
7. Validate and test the site migration script
8. Add the site script to the farm script

The farm script is what we ran twice a week to synchronize all content from SharePoint 2007 to SharePoint 2013.  The majority of the work for each site was step 5: creating the script for content in each list and library.

Scripting For the Mirror

Even with ShareGate and PowerShell at our disposal, the prospect of creating scripts for hundreds of lists and libraries was daunting.  In Part 1 I mentioned that the migration wasn't all that bad if one is organized and methodical, and in Part 2 I mentioned that having our SharePoint 2007 inventory in a database format made using our tools easier than we could have imagined - and here is where those two things come together.


An example of the ShareGate script to migrate a single list looks like this:

For the most part, the script structure doesn't change from list to list.  Most of the string values in the script were stored in the database we used for inventory.  This meant we could create a function that built the script for us, with only a small part of the script itself being variable.  We added a column to our inventory tables that contained the part of the script that would vary from list to list, which is lines 27 - 37:

This section of the script block controls, for a particular list: how the version history is handled; how duplicate records are handled (skip, overwrite, or incremental update); and how content, column, and user mappings are handled.


This gave us the capability to build a migration script for an entire site using a single function that output the script to a single file.  By building a farm-level script, we could update the mirror with a single script that called all the site-level scripts.

Tracking Script Output

The last bit of tool-making we had to do revolved around the scripts' output.  Since ShareGate can export the results of a migration session to an Excel file, we consolidated all the script output to a single folder where we could then process them using SQL Server Integration Services, import them into our database, and create reports using SQL Server Reporting Services.  This allowed us to review an entire script execution session (which could take over two hours) at a glance, allowing us to quickly focus and find migration errors so we could fix them.

Conclusion

By being organized and methodical, and using a database to store the architecture of our SharePoint 2007 farm as an inventory, we could automate our script-building - allowing us to rapidly create typo-free scripts to simplify the update our SharePoint 2013 mirror of our SharePoint 2007 environment.  Which leads us to our last part...

< Part 2 - What Have We Got? | Part 4 - Getting There >

Part 2 - Migrating a SharePoint 2007 farm to SharePoint 2013

What Have We Got?

In the previous article, we discussed the methodology we used to get from SharePoint 2007 to SharePoint 2013, which was using a mirror.  The mirror became an exact copy of our SharePoint 2007 site, but in a SharePoint 2013 farm.  The content in the mirror was continuously updated so that once we were ready to go live, we backed up the DEV instance of SharePoint and restored it to our Production instance of SharePoint.  We also highlighted the fact that ShareGate was integral to our process and success.


Knowing the methodology was only the first step; before we could take any others, we needed to know what we actually had to migrate.

Taking Stock

A SharePoint farm with 60 sites also means a lot of lists and libraries.  We knew that it would take months to get the mirror ready; all the while, the site would still be in use - meaning the inventory was dynamic.  Lists could come and go, columns could be added to a library, views could change, and workflows could be added.  It quickly became evident that a spreadsheet was not the way to go for taking inventory and tracking progress.


Being a DBA, and being a huge fan of SQL Server Reporting Services (SSRS) and SQL Server Integration Services (SSIS), it seemed only logical that creating some tools for our own use using technology we were already familiar with would be a tremendous benefit.

PowerShell is Your Friend

With 60 sites worth of lists and libraries, we didn't want to manually gather and review all that information.  Though PowerShell support for SharePoint 2007 doesn't come close to what SharePoint 2013 offers, it's good enough to create scripts that extract all the information you need to create a site inventory - so we created scripts to give us the following for all of our site collections:

• Sites
• Site permissions
• Groups and group members
• Libraries and lists
• Library and list content types
• Library and list columns
• Library and list pages
• Library and list permissions
• Library and list event receivers
• Library and list workflows
• Library and list sizes

The scripts output their results to .csv files, which makes them very easy to process with an SSIS package.  We created a custom set of tables to store the data, and custom SSRS reports to get the data back out in a easy-to-use format.  We could use these reports to not only take stock of what we had, but we could use them in our meetings with Site Owners when discussing the migration as it relates to their site, and we could also use them for status reporting.  Having the information on-hand for our own status reporting allowed us to report actual metrics and progress, rather than estimations on how far along we thought we were.


Since we ran the PowerShell inventory scripts once a week, a key part of making this method of inventorying effective was the ability to know when things changed; if our site inventory determined a list changed after it was added to the mirror, we had to review our migration script to ensure the changes were properly captured.  Given the size of our farm, the small pool of users permitted to change list definitions, and the fact that our SharePoint 207 site architecture didn't change that frequently this was a fairly minor effort.

What to Inventory

Aside from the data we gathered via PowerShell, we also had to gather information related to the following:

• SSRS Reports
• Custom code (custom event receivers and webparts)
• InfoPath forms
• 3rd-party solutions
• SSIS packages

Each of these items had various considerations, and we had to ensure they were migrated properly.  For SSRS Reports and custom code, Team Foundation Server was invaluable with its code branching and merging capability.  This allowed us to maintain code branches for both SharePoint 2007 and 2013 during the migration process with little effort, which was all the more impressive when you consider the package structure is so completely different between SharePoint 2013 and SharePoint 2007 that our solutions for SharePoint 2013 had to be created from scratch. 


While ShareGate can migrate InfoPath forms, the forms themselves still require some attention - even if the SharePoint 2007 and 2013 library URL's are the same, InfoPath relies on list GUIDS - so an InfoPath form will not migrate directly without modification.  We read a lot of articles and postings related to InfoPath and SharePoint trying to answer the question "can I use InfoPath 2007 forms on SharePoint 2013?"  We tried for ourselves and learned the answer is "yes, but."  We were tied to the SharePoint 2007 forms client both before and after the migration (and were unable to migrate any of our forms to InfoPath Forms Services due to using digital signatures), and we learned that:

• When creating a data connection to a SharePoint 2013 list or library in a form that will open in InfoPath 2007 client, the connection must be modified in InfoPath 2007
• Editing anything other than a data connection in a form that will open in InfoPath 2007 client can be accomplished in InfoPath 2007 or InfoPath 2013

Though ShareGate is quite powerful, it doesn't operate at the farm level, so any 3rd-party solutions needed to be manually installed and configured.  SSIS packages, while not in SharePoint, connect to it; so all packages needed reviewed to ensure they would function properly post-migration.

Conclusion

Creating our own tools - PowerShell scripts, a SQL database, SSIS packages, and SSRS reports - comprised about a week of our total migration schedule, but the time and hassle ultimately saved was worth the effort.  It allowed us to get an excellent picture of our SharePoint 2007 inventory so we knew exactly what needed migrated and could track our progress.  As it turned out, having all this information in a database format made using our migration tools even easier than we could have imagined.  But that's a topic for the next article...

< Part 1 - How Do We Get There? | Part 3 - Using the Tools to Get There