Part 4 - Migrating a SharePoint 2007 farm to SharePoint 2013

Getting There

In Part 3, we showed how we created scripts for our data migration from SharePoint 2007 to a SharePoint 2013 mirror (our methodology explained in Part 1) using SharePoint 2007 inventory data (covered in Part 2).  In our final part, we'll discuss how that all came together; to the point where we could migrate 60 sites with 1.5 TB of data from SharePoint 2007 to SharePoint 2013 in just six hours.

Prepping the Production Farm

We already had a DEV farm which held our mirror.  The mirror was an exact copy of our SharePoint 2007 content in a SharePoint 2013 farm.  Since it needed to be maintained as an exact copy for our methodology it work, that meant it couldn't be used for developer testing or user acceptance testing (UAT).  The solution for that quandary turned out to be fairly easy - we set up our production farm, which we called PROD, in the production configuration we wanted to use when we went live.  That included creating the Web Applications, content databases, and services.  The Web Application was created to respond to the production hostnames (which, as we stated in Part 1, needed to the same as was being used in the SharePoint 2007 farm).  We then Extended the Web Application to respond to a second hostname, which we called TEST.  This achieved multiple things:

• Allowed us to create a TEST environment for users to manipulate data and break things while leaving the mirror pristine
• Allowed us to test Production equipment and configuration during UAT, since users were testing on what would be the Production environment
• Allowed us to test and validate our step-by-step migration procedure and scripting, since the steps would be exactly the same when updating the TEST instance or the PROD instance
• Eliminated discrepancies that could exist between UAT and production, since even identically configured environments can have subtle differences that could cause problems when migrating to PROD that didn't exist when migrating to TEST
• Allowed us to test the Production environment's response to production hostnames before going live by modifying the host file on the servers

By configuring our Production farm in this manner and using it as our Test environment, by the time the day came for migration, we had effectively already done the migration dozens of times - making the final migration (the one that mattered) go off without a hitch.

Putting the Mirror Into Production

Whether we were copying the mirror into our TEST environment, or actually conducting the final migration, the steps were exactly the same each time:

1. Run the mirror synchronization script
2. Import the script output files into our inventory database
3. Review script result reports to identify issues that may have occurred during mirror synchronization
4. Fix synchronization issues
5. Backup the mirror
6. Restore the mirror to the Production farm

Since we scripted most of these actions, and we practiced/executed this process so many times, we found the whole thing could actually be accomplished mostly hands-off in as little as 4 hours:

• 2 and half hours to synchronize the mirror
• 30 minutes to review and fix synchronization issues
• 1 hour to back-up the mirror and restore it to the Production farm

We only did one thing differently when putting the mirror into final Production - after running the mirror synchronization script, the SharePoint 2007 farm was placed into read-only mode.


Once we put the mirror into Production for the final time, only a few steps remained:

• Remove the TEST host header from the Web Application
• Change DNS entries for the hostnames
• Change host files in the old SharePoint 2007 farm, allowing us to still access SharePoint 2007 using the same URL as long as we were on one of the old servers

Conclusion

From a user perspective, they went home on a Friday using SharePoint 2007 and came in on Monday using SharePoint 2013, and it was business as usual.


We couldn't have asked for or envisioned a better outcome, but that's exactly what happened.  Sure, there were issues here and there (what migration goes off with zero hitches?) - but nearly all the issues that were brought up were anticipated, and most were fixed very quickly.


We had gone from not even knowing how we could do a migration from SharePoint 2007 to SharePoint 2013 to successfully doing the migration so often to TEST we could do it in our sleep - or from another perspective - scared out of our minds to confident and optimistic. 


Being organized and methodical, taking and keeping inventory in a database, and using the right tools (ShareGate and PowerShell) all contributed to our success.  Those of you reading this who have done migrations before may have noted I've left a lot of things out when I talk about how we did all this - and you are right.  There were many other factors and things we did that were key to our success, but they are the same ones that apply to any migration - engaged project sponsorship, frequent engagement of site owners, requirements gathering, documentation, training, and testing - just to name a few.  We wanted to highlight the way we handled the technical issues of migrating a huge amount of content from SharePoint 2007 to SharePoint 2013.  Hopefully we've done that - and if one day this story helps someone else complete a successful SharePoint migration - even better.

< Part 3 - Using the Tools to Get There

Part 3 - Migrating a SharePoint 2007 farm to SharePoint 2013

Using the Tools to Get There

We've discussed the methodology we used to migrate from SharePoint 2007 to SharePoint 2013 (using a mirror), and how we took stock of what we had to migrated (using various tools), so we'll move on to how we actually built our mirror.

Setting Up The Mirror

The first SharePoint 2013 farm we stood up as part of our migration process was called DEV.  It was not very robust, and our intent was to use the DEV farm for development after our migration to SharePoint 2013 was complete.  In the meantime, its primary function was to house our mirror - an exact replica of our SharePoint 2007 content, but housed in SharePoint 2013.  Many services weren't even configured in this farm - just a few that we wanted to test or validate - such as Search and SQL Server Reporting Services (SSRS).  We setup the Web Applications such that our current URL structure would be maintained between SharePoint 2007 and SharePoint 2013, but with a different host name.  We also setup the database files just like we wanted them to be in our final product, since the actual migration would be accomplished with a database backup and restore between our two SharePoint 2013 instances (DEV and PROD).

Filling the Mirror

Creating the mirror was the easy part - filling the mirror with content is what consumed 80% of our schedule.  Recall from Part 1 that one of our success criteria was to make improvements where we could - this meant that each library and list was reviewed to see what, if any improvements could be made.  One of the biggest areas of improvements we wanted to focus on was Search - we wanted our Search to be more usable - which meant trying to create improved content types and site columns for better search results.  Ultimately, we came up with a procedure that was repeated for each site:

1. Create the site using SharePoint 2013
2. Enable/disable required site features
3. Migrate all site pages, lists and libraries (including workflows and public views) without content using ShareGate
4. Cleanup/adjust permissions from what was migrated (part of our 'making improvements where we could' effort)
5. Make desired improvements to content types, lists, or libraries
6. Create the script for content in each list and library using ShareGate PowerShell
7. Validate and test the site migration script
8. Add the site script to the farm script

The farm script is what we ran twice a week to synchronize all content from SharePoint 2007 to SharePoint 2013.  The majority of the work for each site was step 5: creating the script for content in each list and library.

Scripting For the Mirror

Even with ShareGate and PowerShell at our disposal, the prospect of creating scripts for hundreds of lists and libraries was daunting.  In Part 1 I mentioned that the migration wasn't all that bad if one is organized and methodical, and in Part 2 I mentioned that having our SharePoint 2007 inventory in a database format made using our tools easier than we could have imagined - and here is where those two things come together.


An example of the ShareGate script to migrate a single list looks like this:

For the most part, the script structure doesn't change from list to list.  Most of the string values in the script were stored in the database we used for inventory.  This meant we could create a function that built the script for us, with only a small part of the script itself being variable.  We added a column to our inventory tables that contained the part of the script that would vary from list to list, which is lines 27 - 37:

This section of the script block controls, for a particular list: how the version history is handled; how duplicate records are handled (skip, overwrite, or incremental update); and how content, column, and user mappings are handled.


This gave us the capability to build a migration script for an entire site using a single function that output the script to a single file.  By building a farm-level script, we could update the mirror with a single script that called all the site-level scripts.

Tracking Script Output

The last bit of tool-making we had to do revolved around the scripts' output.  Since ShareGate can export the results of a migration session to an Excel file, we consolidated all the script output to a single folder where we could then process them using SQL Server Integration Services, import them into our database, and create reports using SQL Server Reporting Services.  This allowed us to review an entire script execution session (which could take over two hours) at a glance, allowing us to quickly focus and find migration errors so we could fix them.

Conclusion

By being organized and methodical, and using a database to store the architecture of our SharePoint 2007 farm as an inventory, we could automate our script-building - allowing us to rapidly create typo-free scripts to simplify the update our SharePoint 2013 mirror of our SharePoint 2007 environment.  Which leads us to our last part...

< Part 2 - What Have We Got? | Part 4 - Getting There >

Part 2 - Migrating a SharePoint 2007 farm to SharePoint 2013

What Have We Got?

In the previous article, we discussed the methodology we used to get from SharePoint 2007 to SharePoint 2013, which was using a mirror.  The mirror became an exact copy of our SharePoint 2007 site, but in a SharePoint 2013 farm.  The content in the mirror was continuously updated so that once we were ready to go live, we backed up the DEV instance of SharePoint and restored it to our Production instance of SharePoint.  We also highlighted the fact that ShareGate was integral to our process and success.


Knowing the methodology was only the first step; before we could take any others, we needed to know what we actually had to migrate.

Taking Stock

A SharePoint farm with 60 sites also means a lot of lists and libraries.  We knew that it would take months to get the mirror ready; all the while, the site would still be in use - meaning the inventory was dynamic.  Lists could come and go, columns could be added to a library, views could change, and workflows could be added.  It quickly became evident that a spreadsheet was not the way to go for taking inventory and tracking progress.


Being a DBA, and being a huge fan of SQL Server Reporting Services (SSRS) and SQL Server Integration Services (SSIS), it seemed only logical that creating some tools for our own use using technology we were already familiar with would be a tremendous benefit.

PowerShell is Your Friend

With 60 sites worth of lists and libraries, we didn't want to manually gather and review all that information.  Though PowerShell support for SharePoint 2007 doesn't come close to what SharePoint 2013 offers, it's good enough to create scripts that extract all the information you need to create a site inventory - so we created scripts to give us the following for all of our site collections:

• Sites
• Site permissions
• Groups and group members
• Libraries and lists
• Library and list content types
• Library and list columns
• Library and list pages
• Library and list permissions
• Library and list event receivers
• Library and list workflows
• Library and list sizes

The scripts output their results to .csv files, which makes them very easy to process with an SSIS package.  We created a custom set of tables to store the data, and custom SSRS reports to get the data back out in a easy-to-use format.  We could use these reports to not only take stock of what we had, but we could use them in our meetings with Site Owners when discussing the migration as it relates to their site, and we could also use them for status reporting.  Having the information on-hand for our own status reporting allowed us to report actual metrics and progress, rather than estimations on how far along we thought we were.


Since we ran the PowerShell inventory scripts once a week, a key part of making this method of inventorying effective was the ability to know when things changed; if our site inventory determined a list changed after it was added to the mirror, we had to review our migration script to ensure the changes were properly captured.  Given the size of our farm, the small pool of users permitted to change list definitions, and the fact that our SharePoint 207 site architecture didn't change that frequently this was a fairly minor effort.

What to Inventory

Aside from the data we gathered via PowerShell, we also had to gather information related to the following:

• SSRS Reports
• Custom code (custom event receivers and webparts)
• InfoPath forms
• 3rd-party solutions
• SSIS packages

Each of these items had various considerations, and we had to ensure they were migrated properly.  For SSRS Reports and custom code, Team Foundation Server was invaluable with its code branching and merging capability.  This allowed us to maintain code branches for both SharePoint 2007 and 2013 during the migration process with little effort, which was all the more impressive when you consider the package structure is so completely different between SharePoint 2013 and SharePoint 2007 that our solutions for SharePoint 2013 had to be created from scratch. 


While ShareGate can migrate InfoPath forms, the forms themselves still require some attention - even if the SharePoint 2007 and 2013 library URL's are the same, InfoPath relies on list GUIDS - so an InfoPath form will not migrate directly without modification.  We read a lot of articles and postings related to InfoPath and SharePoint trying to answer the question "can I use InfoPath 2007 forms on SharePoint 2013?"  We tried for ourselves and learned the answer is "yes, but."  We were tied to the SharePoint 2007 forms client both before and after the migration (and were unable to migrate any of our forms to InfoPath Forms Services due to using digital signatures), and we learned that:

• When creating a data connection to a SharePoint 2013 list or library in a form that will open in InfoPath 2007 client, the connection must be modified in InfoPath 2007
• Editing anything other than a data connection in a form that will open in InfoPath 2007 client can be accomplished in InfoPath 2007 or InfoPath 2013

Though ShareGate is quite powerful, it doesn't operate at the farm level, so any 3rd-party solutions needed to be manually installed and configured.  SSIS packages, while not in SharePoint, connect to it; so all packages needed reviewed to ensure they would function properly post-migration.

Conclusion

Creating our own tools - PowerShell scripts, a SQL database, SSIS packages, and SSRS reports - comprised about a week of our total migration schedule, but the time and hassle ultimately saved was worth the effort.  It allowed us to get an excellent picture of our SharePoint 2007 inventory so we knew exactly what needed migrated and could track our progress.  As it turned out, having all this information in a database format made using our migration tools even easier than we could have imagined.  But that's a topic for the next article...

< Part 1 - How Do We Get There? | Part 3 - Using the Tools to Get There

Part 1 - Migrating a SharePoint 2007 farm to SharePoint 2013

How Do We Get There?

Our task was pretty straightforward - migrate an entire SharePoint 2007 farm for 300 users, with 3 Site Collections totaling 60 sites and 1.5 TB of data with minimal downtime.  Except when you consider that ensconced in those 60 sites were over 200 SQL Server Reporting Services (SSRS) Reports, many affected SQL Server Integration Services (SSIS) packages, thousands of lines of custom event receiver and webpart code, multiple 3rd-party add-ins (both commercial and CodePlex), InfoPath forms, and many workflows it sounded more like a nightmare.
Except that when you get right down to it, it really wasn't all that bad.


The key is to be organized and methodical - but more on that in a later article.


Our first challenge was to figure out How the Heck Do We Get There?  How could we conceivably get all that content from one farm to another in a minimal amount of time?  Just trying to imagine how we would do that was daunting.  Even just considering the amount of data to move and the speed of our network, we thought we were in a losing position from the get-go.

What Methodology Do We Use?

We did decide on a few things early on, which could be considered success criteria:

• We wanted to keep the site address the same when we went live - we wanted to break as few links, and as few user bookmarks, as possible
• We wanted to minimize confusion - we knew this would be a traumatic experience for our users just with the look-and-feel difference alone, so we wanted to reduce the pain to our users as much as we could
• We wanted to make improvements where we could - SharePoint search was pretty much universally despised by our users, there was a lot of unused 'junk' out there, and if we're spending all this money we should get something more out of it than just a different look
• We wanted to minimize downtime during the migration - without knowing how we would get there, we didn't know how to quantify 'minimal,' but we knew we had to strive for the least amount of downtime we could
• We wanted everything to work - kind of goes without saying, but with all the custom reports and code we had, we had to ensure our users would be reassured

With that decided, we could finally start brainstorming (remember, no idea is the wrong one when it comes to brainstorming - at least that's what they say).  Some of the ideas we considered:

Let's migrate all the sites at one time

"Let's just set that off to the side for later, shall we?"  We know that was a non-starter as soon as the suggestion was put forth.  The of amount of data to move vs. the speed of our network was enough of an obstacle we didn't need any other arguments toward setting this one aside.

Let's migrate one site at a time

Recall I said we had 60 sites, but only 3 site collections. Two of those site collections only had one site each, meaning our main site collection had all the rest.  Our main site collection was the star of the show - each department in our project had their own site, and many had more than one.  Each site had its own set of owners, but everyone on the project had some sort of access to every site in the site collection.  Our main site collection was huge - nearly 1 TB.


Let me step back for just one minute.  I'm sure those of you reading this who are SharePoint admins are raising your eyebrows right about now.  I am a SharePoint admin by circumstance - I didn't design our farm; I just inherited it.  Despite the fact we blew past the recommend site collection size years ago, the SharePoint farm's performance was still really quite good.


With that out of the way, let's return to the topic at hand.  This had a number of concerns that were readily apparent.  Confusion would reign  - users would have to go to two different websites to get their content (with completely different look-and-feel), broken links would abound, we'd be forever fixing links, and we'd be confused trying to keep track of it all.  We would pretty much fail all of our success criteria.

Let's do some reorganization and move the important functionality first

While this would meet the 'make improvements' success criteria, it has all the downfalls of the 'migrate one site at a time method,' while creating even more confusion amongst our users.

Let's pay someone else to do it

After a probe into this solution a year prior to this effort left a bad taste in our mouth, we decided no one would give our site more love than us, so we wanted to do it.

Well, I'm out...

There were some other suggestions that were just variations on the methods already considered,   but about this time we realized something (which may have already become apparent to other SharePoint admins reading this) - we didn't even know what 'migrate' really meant!  Obviously, we knew the general definition of the word, but didn't know how we could even migrate the data!  Since we already knew Microsoft doesn't support upgrading from SharePoint 2007 to SharePoint 2013, we had absolutely no idea what mechanism we could use to actually move the content.  We decided it would probably be wise to figure out what tool(s) we could use - as the tool we chose would probably drive our methodology.  Turns out we were right.

ShareGate Saves The Day

Let's spare you the pain of our migration tool selection process, and skip right to the final result.  We weren't halfway through the ShareGate demo when we figured out a migration methodology that would meet all of our success criteria.


If any of you out there are considering a SharePoint migration, or just have a need to move content between farms or sites, stop looking at all the other tools out there and look at this one.  I'm not getting paid to promote this small Canadian company, but I cannot praise them enough.  We had vendors calling us, "hey, I hear you are looking to migrate to SharePoint 2013; we'd like to help."  They would try to tell us how they were better than the other vendors we were considering, but when we told them we had chosen ShareGate, they hung up and never called back.  ShareGate is reasonably priced, does its job extremely well, and has great support.  But two main things really sold us on it:

• Pricing is not based on the size of the content you are moving (as most solutions seemed to be)
• It has use after migration is complete

ShareGate has some very nice SharePoint management tools, allowing you to manage your sites, lists, and users from within the ShareGate GUI.  It has the ability to create and save reports based on your own criteria that allow you to manage your farm with much greater ease than the SharePoint GUI alone.  All that was icing on the cake, compared to the functionality we really needed.


ShareGate allows you to migrate ANYTHING in your SharePoint site to another SharePoint site and works with 2007, 2010, and 2013 (the only limitation is the destination site can't be SharePoint 2007).  You can migrate at various levels - every list, library and page in a site (and anything below that site); just a library; just a workflow on a library; all SharePoint groups; or just one SharePoint group.  It supports user, group, content type, and column mapping - so your source and destination lists don't have to match exactly (which , in some cases, was a key in meeting the success criteria of making improvements - i.e. some list columns were replaced with site columns with new names that better reflected their meaning).  And the absolutely glorious feature we didn't even know was there until after we bought it - it's fully PowerShell capable.  That feature became one of the key factors in our success.  Another key factor is ShareGate's ability to migrate changes only  - when migrating content using Sharegate, you have to tell it what to do when it encounters the same record in the source and the destination - overwrite it, skip it, or update it with what's changed.  What constitutes 'same record' can be defined by the operator (with Document Libraries, it's always the document name - but with List Items, since IDs are not retained upon migration, you have to create a pseudo-relationship between source and destination records).


There are three main drawbacks to ShareGate (to be fair, these same drawbacks exist in any tool used in a migration):

• Item IDs are not retained upon migration - so an items' ID at the destination will likely not match the ID at the source.  In many cases, this is a non-issue.  In some cases, this is a deal-breaker.  A workaround that worked really well for us - adding a field called ID_Migrated at the destination allowed us to link the source and destination records.
• Person Fields at the destination can only contain users who still exist in Active Directory - just like IDs, sometimes this is an issue, sometimes it's not.  ShareGate does give you the ability to substitute a default user if a user can't be found, but if you do need to retain Person data, creating a text field at the destination to hold the person data will allow you to retain the data (ShareGate gives you the ability to send a single column to multiple destination columns).
• Workflow states cannot be migrated - ShareGate migrates workflow definitions just fine (Nintex workflows too), but it can't migrate the state of a running workflow, so all your workflows should be stopped prior to migration (it doesn't hurt anything, however, to migrate a record that has a workflow running against it).  Note the ability to migrate workflows makes ShareGate very useful to move workflows from a Development environment to a Production environment.

So What Methodology Did We Use?

Once we had a tool in hand, we knew what we could do (and not do).  Ultimately, we decided on a course of action that was never considered in our brainstorming session - we could migrate one list at a time to what we called a mirror, while keeping the same site hierarchy - and once the mirror was complete, we could copy it to Production.  The mirror became an exact copy of our SharePoint 2007 site, but in a SharePoint 2013 farm (which we called DEV).  The content in the mirror was updated twice a week with a single script.  Once we were ready to go live, we simply backed up the DEV instance of SharePoint and restored it to our Production instance of SharePoint.  It took us nine months to complete the mirror.  Once all sites, lists, and libraries existed in the mirror, the mirror could be updated in two and half hours.  The process of backing up DEV and copying to PROD could be accomplished in one hour.  This meant that the definition of 'minimal downtime' for us became 'one business day'.

Conclusion

At the outset, none of us would have believed it if someone said we could migrate 60 SharePoint sites carrying 1.5 TB of data in just six hours.  But using the mirror methodology, we could (and did) achieve all of our success criteria in migrating from SharePoint 2007 to SharePoint 2013, with ShareGate being a critical and integral part of that success.


But wait - there's more!  The next article in this series discusses the next challenge we faced - how do we know what we have to ensure it all get's migrated?

< Migrating a SharePoint 2007 farm to SharePoint 2013 in a Single Weekend | Part 2 - What Have We Got? >

My Task - Migrating a SharePoint 2007 farm to SharePoint 2013 in a Single Weekend

A little over a year ago, my boss called me into her office and asked me to be the Project Manager to migrate our SharePoint 2007 farm to SharePoint 2013.  I asked her, "you mean you want me to DO the migration?"  She just smiled at me with her "thanks for understanding what I'm not saying" smile, so I slunk back to my desk wondering how the heck I was going to migrate an entire SharePoint 2007 farm for 300 users, with 3 Site Collections totaling 60 sites and 1.5 TB of data with minimal downtime.
We utilize SQL Server Reporting Services (over 200 reports), SQL Server Integration Services, custom web parts and event receivers, and 3rd party add-ins (both commercial and CodePlex).  I'm a SQL Server Admin, MCSE, and .NET developer - aside from using SharePoint, I knew very little about how a SharePoint farm worked on any version.  I knew PowerShell existed, but had never used it.
What follows is a multi-part series on how we went about the migration process - high-level planning and preparation, strategy, tools, and processes that went into what was ultimately a very successful migration.  I'm not going to focus on technical aspects such as farm architecture, setting up farm services and service associations, service account configuration, features and things of that nature - articles on those topics have been done to death.  I wish I had kept better track of some of the blogs and sites that were instrumental to our success in order to give them credit, but I only saved a few that helped tremendously along the way:

Thanks to the following, who have no idea who I am, but who also have no idea how much they helped us:
For teaching me that using the Farm Configuration Wizard is not for DBAs who like to control their database names: Mike Robbins' Blog.


For helping me not go crazy after the Farm Configuration Wizard was used on our DEV site: Multiple TechNet Articles.


For a great series of articles regarding installation of Workflow Manager for SharePoint 2013 in a multi-server farm using SSL: Spence Harbar by way of Andrew Connell.


For a helpful article regarding the uninstallation of Workflow Manager for SharePoint 2013 - since it took a lot of trial and error in DEV getting things right in the first place: TechNet Again.


For a supremely helpful series of articles on scaling Enterprise Search in SharePoint 2013: SteveMann's Path


For providing the answer to the question "Why can't I search my external content types?!": Sivarajan's Blog.


For providing multiple answers to the single question, "How do I customize my search results?"
SharePoint IT Pro Blog, Abel Solutions, SharePoint 24x7, and Heather Solomon (I wish I could find it, but I swear the first article I read regarding SharePoint Search that was also the most helpful initially, came from her).

Part 1 - How Do We Get There?