Merging Team Projects: planning to move 120k+ work items

Here we are: Work Items, my friends, you have been the trickiest thing I had to deal with during this unification operation of 14 Team Projects into one big TP.

The picture

The bad news with work items is that you *can’t* move one item from a project to another. This would have been too simple, change the owner TP of each work item. Alas, no, what we needed was actually to copy the work items from all source TPs to the new one.

On the paper, it is quite simple :

image

  • Study the Work Items definitions and see which one to keep and how to adapt the others (do not plan anything too fancy)
  • Use existing tooling (the very helpful TFS migrating toolkit) to copy the work items from each project into the new one
    • Hopefully, field mappings will be straightforward, we just wanted to transpose the Areas
    • The tricky thing will be to reproduce links between work items in the new Team Project

Volume considerations

Copying this whole bunch of work items will make your TFS database grow a bit. At worst, work item tables in SQL Server will double in size. I think this should not be considered as a problem : work item tables were far behind our build history in volume even after the migration, and anyway you can still write a tool that will destroy all your work items to save up space.

Finding the common work item types (WITs)

No magic here, I exported them all, and compared them all textually. It’s a pity, that depending on the tool you are using, XML nodes may be moved across the definition document which does not facilitate comparison at all. Especially if you happen to edit the WIT within the Visual Studio graphic editor, be prepared for a surprise. I just wish my XML organization were kept intact.

Most of the projects had been created from CMMI process template (this template seems more popular in Europe Winking smile ), so that was not the nightmare I was afraid of.

When I ended up with my selection of work item types, I was ready to import them in the new Team Project.

I prepared a batch file with a few variables in order to speed up the export of all Work Item types of every source Team Project and organize in a clean folder structure. You can download the file and adapt it for your needs, it’s faster to edit lines in a batch for mass export rather than waiting the result of each command line. It is based on the following structure :

rem use this to export all wi types from a single TP

set TEAMPROJECT=SourceTP1
md “%TEAMPROJECT%”

set WITYPE=Bug
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

set WITYPE=Shared Steps
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

set WITYPE=Task
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

After having created the target Team Project I used another script in order to import the final Work item types. Since I had to test the Work item migration multiple times, those scripts were very handful :

rem use this to overwrite the wi types of the target TP
set TEAMPROJECT=TargetTP8
set TPCURL=
http://mytfstestserver:8080/tfs/DefaultCollection

witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Bug.xml”
witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Change Request.xml”
witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Configuration.xml”

Note : since I’m using CMMI, but dropped some of its aspects, the script also deletes the unwanted Work item types from this base template. 

What about a small transform and refactor of all WI types on the go ? Well, I kept this principle in mind : what was important for migration was to keep all the data, refactoring and the like would be much easier to perform *afterwards* and is actually another matter.

Next time I’ll talk about how to use the TFS migration tools to perform the actual migration.

Merging Team Projects: moving the source code

This post is part of my merging team projects posts series.

Today, I’ll write about how we managed to move our source code into the new team project source code structure.

All your Main are belong to us

Before writing too much and getting everyone asleep, let’s consider something visual of what we achieved :

 

image

For input, we have 14 team projects, each one with a few branches representing a common branching model :

  • Dev
  • Main
  • Releases (aka candidate)
  • Production
  • Maintenance (this is because many fixes are finally not integration)

About the target branching model

I think my client’s projects organization was a quite specific : many of the projects involved were actually linked one to another, there are code dependencies and cross-references, there are even folders that you need to map within another project’s code sub-folder (baaaaad, I know…). This is to explain why we chose to use one single big Main branch at the root level of the target project instead of keeping a Main branch for every project : anytime we would branch anything from Main, the whole code base would be taken. This looks like a loss of flexibility, because no more individual project can branched independently, but it was an actual productivity booster for everyone. Things were much simpler for developers and integrators, because all in all it ends up into a few very big deliverables. *AND* of course heavy refactoring among projects is now possible without much less efforts (moves, renames, etc.).

By the way, having a common code structure does not prevent from proper componentization and versioning.

Choices had to be made…

To be honest, this operation was not challenging because we chose to keep only the Main branches. That means we threw away all the other branches, yes, you read well. All the dev teams had been involved in the process, and were aware that they had to integrate a maximum of the Dev code into the Main branch. On the other hand, any code that could not make it to the Main branch had to be manually reported. The use of shelvesets has been encouraged for developers to keep their changes ready for the future Dev branch after the migration.

Tip of the day : How to get your shelveset back to another branch than the one your code originates from ? (The Visual Studio unshelve wizard prevents you from doing it because the target folder or branch is not the same…)

Answer : you need to get the latest TFS Power Tools and use the following command :

tfpt unshelve /migrate /source:$/MyTP1/BranchA /target:$/MyTP2/BranchB

What if we wanted to keep the Dev branches in the process ? Technically speaking, it would have been easy, simply move Dev branches as well as Main branches, no difference in the process. But then, what about your first integrations from newly obtained Dev branch to the new Main ? Looks very risky to me, as I don’t know how you can move two branches in TFS and keep the exact status of what has been merged from one branch to another. Because TFS knows this exactly and keeps track about which changeset has been merged between linked branches, you lose that history in the process anyway. I think we avoided a hard problem by choosing to create a new Dev branch from Main, starting over with a top clean status for merges history.

Reporting the code manually took two 1 ~ 2 days from each developer. That was quite costly, I have to say. But from the client’s point of view, there were 2 actual operations :

  • Team projects unification
  • Clean up of the Dev branch

There was finally no big problem to solve, keeping only the Main branches made this whole operation quite straightforward. This is why I claim there was no challenge, my client had been wanting to clean its “messy” (not me saying it! Winking smile ) Dev branch.

On the other hand, production branches and the like were not an issue, since they are typically re-branched for each major release. However, we had to plan this operation just before the next major release branch phase.

How did we do this

Create a “Main” folder in the target TP, and convert it into a branch.

Then, for each source Main branch to move, convert the branch back into a common folder, and “move” it into the new Main branch.

Check in for each moved branch.

What about history ?

Short answer : history is conserved after the move. But, history is less readable than before due to the move operation.

In  the history windows, you now have to unfold the changeset of the “move” checkin to retrieve all your old history Sad smile A flatten option for at least one or two levels would be nice…

Detailed procedure

Here is the procedure I wrote for all the moves :

Create a new workspace and map the target team project root folder, assuming the Main branch has been created.

Then for each team project :

  • map the whole project root folder at the root of your workspace, but avoid getting sources to avoid wasting time (click no when asked for it)
  • Upon the Main branch of the source project : File –> Source Control –> Branching and Merging -> Convert to Folder
  • Manually cloak every subfolder under Main (you can do it even there has been no get sources). Alternatively, you can consider skipping this step if sources are not to long to get or if there are too many subfolders.
  • Get the latest sources of the mapped Main (should be very fast)
  • Right-click on it and move it under the Main branch of the target TP
  • You may be facing locks, then :
    • Retry with the command line to get details
      • tf rename Main ..\NewProject\Main
    • Then, try the regular unlock command :
      • tf lock $/SourceTP/Main/* /recursive /lock:none /workspace:MACHINE534;theusername
    • If needed, change the project source code options to multiple check-out
    • You may have to destroy the user workspace (but her informed the user informed she should keep her source code locally)
  • You may be facing access denied despite you’re the Uber admin (frustrating I know)
    • Go hunting for the faulty file or folder by keeping repeating moves on various folders and cancelling them, then remove the overriding permission entry that masks your almighty permission set locally, eg :
    • tf perm /remove:Read “/group:[DefaultCollection]\Project Collection Service Accounts” FaultyFileName.ext
  • One the move is successful in your workspace, check-in with a nice comment.

Remarks :

  • The check-in should only contain one single rename operation
  • If you happen to test this from a recent backup and restore of your TFS databases (highly recommended), you’ll figure out the locks and denied accesses in advance, this makes it much faster to perform all this on D-day.

 

That’s all for today, in conclusion, I don’t how you can do all this moves with TFS without starting up a new branch structure. Hopefully, you don’t do that everyday…