Merging Team projects: numbers and final thoughts

I will now conclude and share my thoughts about this whole operation. But before that let’s have a look at a few numbers.

Numbers

Volume

  • 13 branches moved (only the Main branches were moved), approx 800Mb of source code in total
  • 250+ builds moved with their msbuild scripts (they were legacy, highly customized builds from TFS 2008)
  • 134.000+ work items moved (I announced 120.000, but this has raised faster than I had planned)
  • Average WI migration rate: 900 work items per hour, that is 5 full days, but a bit more in practice, because I split the workload in chunks
  • 162.000+ WI links restored, covering 134.000+ Work items
  • Average WI link restoration rate: 400 WI processed per minute

 

Investment

  • 28 days to plan, conceive, communicate, set up and execute the migration
  • 4 days of external help
  • a few meetings
  • 50 days of development into total to bring back the evolutions into the newly created dev branch (story here)

Benefits

  • Increased productivity for everyone, having multiple team projects for the same final product was a bit confusing
    • Developers
    • Code integratorsimage
  • The starting point to have a unified documentation of all internal processes (simplified by the way), and the starting point of many other ALM improvements
  • Morale of troops: this move was the proof that managers cared about their dev infrastructure, and new comers would not find a mess

Conclusion

Given the costs and benefits, I think my client did the right choice. I was so much convinced it was the road to follow. I admit I spent more time on it than initially planned, but people around me were really motivated into doing this, it was the first time something big was done for the sake of industrialization without carefully calculated benefits (just like agile processes). This was also the starting point for other work streams about ALM improvements at my client’s.

If I had the chance to find the kind of articles I’ve just posted, things would have been a bit easier. I’ve given:Package by Anonymous - Package icon by Fr�d�ric Moser. From old OCAL website.

  • An overall approach and methodology for every aspect
  • A procedure and work-arounds for moving the branches
  • Tools and technical info for moving the Work Items
  • A sample tool for moving the build definitions
  • Concrete numbers to help planning

I finally encourage the readers that are interested to merge their team projects if they feel that the projects are part of a single big product (from an external point of view), it’s just like an agile practice, an investment to make things more cohesive and, with time, avoid wasting time and money, because technical things are just reflecting reality. I just hope you’ll find these pieces of information useful, feel free to ask any question, and good luck with your migrations!

Batch copying build definitions in TFS 2010

[get the source code of this sample utility]

This post is part my “merging team projects series” but can be considered independently. I’ll explain and publish a code template that will help you to batch copy build definitions from a team project to another. This is all possible thanks to the TFS API.

More than just raw copying

I’m not the first to blog about this, and you can find various small pieces of code here, or here. Why am I bothering then ? Because what I intend to do is more than just raw copying, we need also to transpose Workspaces, change build templates location, edit build process parameters, edit build templates in the source controller and check them in…

Features:

  • Regex selection of build names on command line
    • I recommend using a batch file with every possible filters in there are many
  • 3 log files in append mode (support for batch file with many calls)
  • Workspace transformations (customizable in the code)
    • Easy to report paths that are non-standard according to your own rules
  • Build process parameters transformations (customizable in the code)
  • Copy, transform and check-in files in the source controller (customizable in the code)

Limitations:

  • Single TFS Server (cannot migrate to another server)
  • Need to customize the C# code to get exactly what you want

Prerequisistes

My starting point what Jim Lamb’s piece of code.

You’ll need to reference a few classic TFS assemblies, including a private one: Microsoft.TeamFoundation.Build.Workflow.dll, you’ll find it in C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies. Note that this reference will force you to change the targeted framework to .NET Framework 4.0 instead of the client profile, but that doesn’t matter too much for that kind of utility.

image

I’ve also included an easy to use class for processing command line parameters C#/.NET Command Line Arguments Parser, many thanks to GriffonRL.

I wrote a small utility class to deal with workspaces, you’ll find it in this project.

Code Highlights

A few pieces of code that can be of interest:

// clone the build into the target team project
IBuildDefinition newDefinition = Utilities.CloneBuildDefinition(_bs, buildDef, targetTeamProject);

As I mentioned earlier, this basically calls Jim Lamb’s piece of code in order to get the build object properly duplicated.

// accessing the build process parameters stored in TFS in a serialized format
IDictionary<String, Object> processParameters = WorkflowHelpers.DeserializeProcessParameters(buildDef.ProcessParameters);

Continue reading

Merging team projects: Restoring the Work Item links

This has been the biggest issue of the migration: I knew how to migrate the Work Items, but I knew also that links between work items would only be *partially* migrated. I’ve explained the problem in this post. The only solution I could find was to write a tool myself to recreate the work item links in the destination Team Project.

What links exactly are migrated by default ?

Hyperlinks

Those links are URLs, they are static and they are migrated by default.

Changesets

Changeset links are implemented in TFS with “External links”, there is no much to worry about them as long as you migrate onto the same TFS server, and that was my case.

Related links

These are links between work items themselves. Here is a schema that shows the partial links migration:

image

The TFS Integration Platform will only migrate links that are “contained” in the source Work Item query, and for each link, both concerned work items need to be included in the query.

Migration by range of work items

Because it was not possible in my case to perform the migration in one single query, I adopted a *chunk* strategy based on time, or, more exactly, based on IDs, which is actually the same.

The source query was like : get all work items from all the source team projects where the ID is ranging from XXX to YYY

An important thing is that the TFS Integration tools are idempotent: you may launch the same migration twice, it would not duplicate what’s already been migrated, but it detects and takes anything that is new. It just works, cool!

Time considerations

There was more than 120.000 work items to migrate, and I needed to plan the migration timing right, that is why starting the real migration several weeks before the dead line had a great value :

  • I got real performance feedback that I could use to calculate the global migration time needed
  • I was sure about what I was getting, limiting the risks as time was passing, because the job was getting done from week to week

A tool to restore links after the migration

What I like very much with TFS is its API. You can do magic with it. Yeah sometimes I complain that a particular feature is missing in the base product, yet to realize that it can be easily done by using the API. It’s just .NET programming, and with a few lines of code you can access Work Items, Source control, builds, and more.Though, the community around this API is not very big, and even if support forums are great, I find it lacks documentation, and this is where you’ll be happy to find cool guys like Shai Raiten and his TFS API blog posts series, great place to get started.

Continue reading

TFS Integration tools configuration for work items

Hi, today I’m detailing the configuration I used for the migration. At first I thought this step would have been quite complex. I have to say I was pleasantly surprised, the TFS Integration tools are powerful indeed and I ended up with readable configuration file.

Avoiding validation errors

I cannot afford being delayed by invalid work item fields, such as a missing user account that blocks the copy of a work item. I needed to bypass any kind of control on work item fields. Invalid stuff is not the subject of the migration.

I changed the work item types and added the ALLOWEXISTINGVALUE in many fields! Typically in custom fields that use global lists because the list have been modified since then. Also, in many fields bearing user accounts (System.ChangedBy, System.AssignedTo, Microsoft.VSTS.Common.ActivatedBy, Microsoft.VSTS.Common.ResolvedBy, Microsoft.VSTS.Common.ClosedBy).

<FIELD name="Changed By" refname="System.ChangedBy" type="String" syncnamechanges="true" reportable="dimension">

<!—VLA remove this after migration –>
  <ALLOWEXISTINGVALUE />
  <VALIDUSER />
</FIELD>

 

Enable bypass rule data submission

This is a migration option I really needed, when enabled, the work items are not created with regular TFS API, but with an XML construction that is submitted. This actually bypasses a few controls on the work item fields. This also it will avoid too many errors when you actually raw copies without too much control.

Here are interesting details about this option.

Each time I created a new migration in the TFS integration tools I would copy paste this block in the actual configuration.

  <Settings>
    <Addins />
    <UserIdentityLookup />
    <DefaultUserIdProperty UserIdPropertyName="DisplayName" />
  </Settings>
         <CustomSettings>
           <CustomSetting SettingKey="EnableBypassRuleDataSubmission" SettingValue="true" />
         </CustomSettings>
  <StoredCredential />

Areas transposition

I previously said I wanted no transformation during the migration, but we had in mind to restructure completely Areas and Iterations. The root node for Areas is the Team Project name, then you start having major nodes such as : Front office, Main portal, Back office, Client, BI, Databases, and the like. In this unification of team projects, we wanted to design a single and cohesive Area structure.

image

Continue reading

Using the TFS integration tools to migrate Work Items from multiple projects

I’ll just be continuing my post series on Team project merging and write about the TFS migration toolkit.

Using the migration toolkit

The core of the migration has been performed by this migration toolkit engine and this where I have spent most of my time. The toolkit is decently documented, maybe not as much as I may have wanted, but there are samples, a cool training kit, a very good documentation on how different scenarios can be solved, what kind of migrations are available. Add on top many blog posts by Willy Peter-Schaub and a cool support on a forum, I really had the feeling I had my bang for my buck…

Work Item migration basics

You launch the TFS integration tools, and create a so called “migration” between a source TP to a destination TP. You’ll be able to choose multiple modes and I’ll forward you to the manual part which explains which one to use according to your business scenario. In my case, the one way migration was all indicated. I needed a one shot migration, I didn’t need to keep work items in sync between projects or any bi-directional fancy stuff. And above all, I needed performance.

Note : even one way migration keeps track of source WIs and dest WIs Ids, you may start the same migration twice, changes in the source Work Items (or any new stuff) would be reflected, that is just magic !

Next, you may specify a Query to filter the bunch of work items to be migrated. In my first experiments I tried to move the WIs project by project, but I quickly realized that links among work items were not reproduced in the destination project.

When are links not preserved ?

You lose work items links when you perform multiple Migrations with the TFS integration tools. The only way to preserve all links to use a “large query”, the opposite of what Willy Peter-Schaub calls a narrow query. His posts explain the problem well.

In other words, the TFS integration engine has been implemented in order to preserve the WI links of your current migration, but does not take into account anything existing before your current migration.

image_thumb

Why do I bother about links if have many different projects ? Well, actually, there are really many links between work items of my different Team Projects! Bugs from A are linked to tasks from B, and the tasks trees we use span over many projects.

Yes, although this works and is supported, it was a bit messy…

Unsuccessful first attempts

Ok let’s use very large queries then…But wait, oh no, big surprise, it was not possible to use queries that span over multiple Team Projects : the tool adds a hard coded filter to the queries ! First bad news for my big migration Sad smile

So, why not after all, let’s do it: let’s modify the tool code and recompile it. I changed the following lines :

// VLA : original code
//var c = new StringBuilder("[System.TeamProject]=@project");
//if (!string.IsNullOrEmpty(m_core.Config.Filter))
//{
//    c.AppendFormat(CultureInfo.InvariantCulture, " AND ({0})", m_core.Config.Filter);
//}

// VLA : select WIs from the filter if defined, otherwise from the team project
var c = new StringBuilder();
if (!string.IsNullOrEmpty(m_core.Config.Filter))
{
    c.AppendFormat(CultureInfo.InvariantCulture, "({0})", m_core.Config.Filter);
}
else
{
    c.Append("[System.TeamProject]=@project");
}

Now I was ready to launch very big queries. Sadly, when I started testing the migrations, I ended up with strange error messages :

[19/01/2012 11:54:57] TfsMigrationShell.exe  Information: 0 : WorkItemTracking: Area ‘Mycompany\ADS\Mycompany\ADSWebShop’ does not exist in the TFS work item store ‘ads-tfs\DefaultCollection (Newtp)’ or access is denied.

[19/01/2012 11:55:04] TfsMigrationShell.exe Information: 0 : WorkItemTracking: Wake up from 100 millisec sleep for polling CSS node Id

Not sure why I had those, I apologize I may not be able to explain this clearly since myself I could really understand what was happening. The good news is that after a couple of unsuccessful attempts, I ended up recreating the plan, along with the destination Team Projects and the errors finally disappeared. In order to test my migrations, I had to often create a new destination TPs…

One big query is not the solution

In order to make sure I miss no link in the migration process, I’ll just have to run one single very very big that selects *all* of the work items I need to migrate… in one single query… How long would it take ? And if it fails, I would have to restart the whole query ? Is is the right way ?

Surely not, this operation would take weeks! The TFS integration tools engine is powerful, but relies on TFS API, and mass operations are costly in time. Remember now the constraints for this operation : the final migration should not take more than a Week-End…

That means I’ll have to pre-migrate the vast majority of work items, and I start feeling I’ll have to write a tool myself to reflect links because I won’t be able to do all this in one shot. Even with two shots, I would potentially miss links…

I’ll leave this for the next posts… stay tuned!

Merging Team Projects: planning to move 120k+ work items

Here we are: Work Items, my friends, you have been the trickiest thing I had to deal with during this unification operation of 14 Team Projects into one big TP.

The picture

The bad news with work items is that you *can’t* move one item from a project to another. This would have been too simple, change the owner TP of each work item. Alas, no, what we needed was actually to copy the work items from all source TPs to the new one.

On the paper, it is quite simple :

image

  • Study the Work Items definitions and see which one to keep and how to adapt the others (do not plan anything too fancy)
  • Use existing tooling (the very helpful TFS migrating toolkit) to copy the work items from each project into the new one
    • Hopefully, field mappings will be straightforward, we just wanted to transpose the Areas
    • The tricky thing will be to reproduce links between work items in the new Team Project

Volume considerations

Copying this whole bunch of work items will make your TFS database grow a bit. At worst, work item tables in SQL Server will double in size. I think this should not be considered as a problem : work item tables were far behind our build history in volume even after the migration, and anyway you can still write a tool that will destroy all your work items to save up space.

Finding the common work item types (WITs)

No magic here, I exported them all, and compared them all textually. It’s a pity, that depending on the tool you are using, XML nodes may be moved across the definition document which does not facilitate comparison at all. Especially if you happen to edit the WIT within the Visual Studio graphic editor, be prepared for a surprise. I just wish my XML organization were kept intact.

Most of the projects had been created from CMMI process template (this template seems more popular in Europe Winking smile ), so that was not the nightmare I was afraid of.

When I ended up with my selection of work item types, I was ready to import them in the new Team Project.

I prepared a batch file with a few variables in order to speed up the export of all Work Item types of every source Team Project and organize in a clean folder structure. You can download the file and adapt it for your needs, it’s faster to edit lines in a batch for mass export rather than waiting the result of each command line. It is based on the following structure :

rem use this to export all wi types from a single TP

set TEAMPROJECT=SourceTP1
md “%TEAMPROJECT%”

set WITYPE=Bug
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

set WITYPE=Shared Steps
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

set WITYPE=Task
witadmin exportwitd /collection:
http://mytfsserver:8080/tfs “/p:%TEAMPROJECT%” “/n:%WITYPE%” “/f:%TEAMPROJECT%\%WITYPE%.wit”

After having created the target Team Project I used another script in order to import the final Work item types. Since I had to test the Work item migration multiple times, those scripts were very handful :

rem use this to overwrite the wi types of the target TP
set TEAMPROJECT=TargetTP8
set TPCURL=
http://mytfstestserver:8080/tfs/DefaultCollection

witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Bug.xml”
witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Change Request.xml”
witadmin importwitd /collection:%TPCURL% “/p:%TEAMPROJECT%” “/f:Configuration.xml”

Note : since I’m using CMMI, but dropped some of its aspects, the script also deletes the unwanted Work item types from this base template. 

What about a small transform and refactor of all WI types on the go ? Well, I kept this principle in mind : what was important for migration was to keep all the data, refactoring and the like would be much easier to perform *afterwards* and is actually another matter.

Next time I’ll talk about how to use the TFS migration tools to perform the actual migration.

Merging Team Projects: moving the source code

This post is part of my merging team projects posts series.

Today, I’ll write about how we managed to move our source code into the new team project source code structure.

All your Main are belong to us

Before writing too much and getting everyone asleep, let’s consider something visual of what we achieved :

 

image

For input, we have 14 team projects, each one with a few branches representing a common branching model :

  • Dev
  • Main
  • Releases (aka candidate)
  • Production
  • Maintenance (this is because many fixes are finally not integration)

About the target branching model

I think my client’s projects organization was a quite specific : many of the projects involved were actually linked one to another, there are code dependencies and cross-references, there are even folders that you need to map within another project’s code sub-folder (baaaaad, I know…). This is to explain why we chose to use one single big Main branch at the root level of the target project instead of keeping a Main branch for every project : anytime we would branch anything from Main, the whole code base would be taken. This looks like a loss of flexibility, because no more individual project can branched independently, but it was an actual productivity booster for everyone. Things were much simpler for developers and integrators, because all in all it ends up into a few very big deliverables. *AND* of course heavy refactoring among projects is now possible without much less efforts (moves, renames, etc.).

By the way, having a common code structure does not prevent from proper componentization and versioning.

Choices had to be made…

To be honest, this operation was not challenging because we chose to keep only the Main branches. That means we threw away all the other branches, yes, you read well. All the dev teams had been involved in the process, and were aware that they had to integrate a maximum of the Dev code into the Main branch. On the other hand, any code that could not make it to the Main branch had to be manually reported. The use of shelvesets has been encouraged for developers to keep their changes ready for the future Dev branch after the migration.

Tip of the day : How to get your shelveset back to another branch than the one your code originates from ? (The Visual Studio unshelve wizard prevents you from doing it because the target folder or branch is not the same…)

Answer : you need to get the latest TFS Power Tools and use the following command :

tfpt unshelve /migrate /source:$/MyTP1/BranchA /target:$/MyTP2/BranchB

What if we wanted to keep the Dev branches in the process ? Technically speaking, it would have been easy, simply move Dev branches as well as Main branches, no difference in the process. But then, what about your first integrations from newly obtained Dev branch to the new Main ? Looks very risky to me, as I don’t know how you can move two branches in TFS and keep the exact status of what has been merged from one branch to another. Because TFS knows this exactly and keeps track about which changeset has been merged between linked branches, you lose that history in the process anyway. I think we avoided a hard problem by choosing to create a new Dev branch from Main, starting over with a top clean status for merges history.

Reporting the code manually took two 1 ~ 2 days from each developer. That was quite costly, I have to say. But from the client’s point of view, there were 2 actual operations :

  • Team projects unification
  • Clean up of the Dev branch

There was finally no big problem to solve, keeping only the Main branches made this whole operation quite straightforward. This is why I claim there was no challenge, my client had been wanting to clean its “messy” (not me saying it! Winking smile ) Dev branch.

On the other hand, production branches and the like were not an issue, since they are typically re-branched for each major release. However, we had to plan this operation just before the next major release branch phase.

How did we do this

Create a “Main” folder in the target TP, and convert it into a branch.

Then, for each source Main branch to move, convert the branch back into a common folder, and “move” it into the new Main branch.

Check in for each moved branch.

What about history ?

Short answer : history is conserved after the move. But, history is less readable than before due to the move operation.

In  the history windows, you now have to unfold the changeset of the “move” checkin to retrieve all your old history Sad smile A flatten option for at least one or two levels would be nice…

Detailed procedure

Here is the procedure I wrote for all the moves :

Create a new workspace and map the target team project root folder, assuming the Main branch has been created.

Then for each team project :

  • map the whole project root folder at the root of your workspace, but avoid getting sources to avoid wasting time (click no when asked for it)
  • Upon the Main branch of the source project : File –> Source Control –> Branching and Merging -> Convert to Folder
  • Manually cloak every subfolder under Main (you can do it even there has been no get sources). Alternatively, you can consider skipping this step if sources are not to long to get or if there are too many subfolders.
  • Get the latest sources of the mapped Main (should be very fast)
  • Right-click on it and move it under the Main branch of the target TP
  • You may be facing locks, then :
    • Retry with the command line to get details
      • tf rename Main ..\NewProject\Main
    • Then, try the regular unlock command :
      • tf lock $/SourceTP/Main/* /recursive /lock:none /workspace:MACHINE534;theusername
    • If needed, change the project source code options to multiple check-out
    • You may have to destroy the user workspace (but her informed the user informed she should keep her source code locally)
  • You may be facing access denied despite you’re the Uber admin (frustrating I know)
    • Go hunting for the faulty file or folder by keeping repeating moves on various folders and cancelling them, then remove the overriding permission entry that masks your almighty permission set locally, eg :
    • tf perm /remove:Read “/group:[DefaultCollection]\Project Collection Service Accounts” FaultyFileName.ext
  • One the move is successful in your workspace, check-in with a nice comment.

Remarks :

  • The check-in should only contain one single rename operation
  • If you happen to test this from a recent backup and restore of your TFS databases (highly recommended), you’ll figure out the locks and denied accesses in advance, this makes it much faster to perform all this on D-day.

 

That’s all for today, in conclusion, I don’t how you can do all this moves with TFS without starting up a new branch structure. Hopefully, you don’t do that everyday…

Merging Team Projects in TFS: general approach

In my last post I talked about why I had to plan a big “unification” of many team projects. Now we’ll start having a look on what we need in a more detailed way.

In short : the plan is to create a clean new team project, and move (or copy) everything we need into it. It’s kind of a big migration.

What is the scope of the migration ?

A team project encompasses the following aspects :

  • Work Items and their definitions
  • Source control folders and their branches
    • This means all Workspaces will have to be recreated!
  • Build definitions and their history
  • Reports definitions
  • Alerts subscriptions
  • Work item queries, personal and team queries

Overall goal and constraints

With more than 150 people using TFS every day, it was out of question to interrupt the server during day work time, so all the final migration steps had to be done during the week end.

That means everything had to be carefully prepared and tested. Most of the work items have been already copied and ready before the final week end. We’ll see that the volume of work item to migrate is important and requires several days to migrate.

The source control folders were also an issue, since there were many branches to migrate, copying or moving branches would lose all of the merges history (between each pair of branches). Well, this was *the* occasion to reset all branches history and recreate all of them from the Main branch. I’ll talk about it later.

To summarize :

  • No interruption of service
  • The migration was the occasion start over with new clean branch structure (which simplifies migration a lot)
  • Complete migration of work items history
  • Builds have to be recreated in the new project, it is ok to lose their history in the process
  • No special reporting needs

What is the global strategy for each of those elements ?

Work Items

This is, with the source controller, the biggest part, and by far, the most time consuming for the TFS admins.

First I had to rationalize work item definitions, then plan the migration of more than 120.000 Work Items. This took me weeks… I’ll post about this specific topic in the near future.

The deal was also to assign correct Areas to all imported work items.

Branches and sources

Luckily, we could negotiate to lose any merge history between the branches. So we could import every Main branch in one single big Main branch, then “branch” the other branches from there. Any evolution in the Dev branch had to be manually redone from there. Again, I’ll develop this point later.

Builds

We had to write a small utility to recreate all the builds and transpose the paths.

Reports

This was easy since we did not customize any report, and all the process templates were based on Microsoft CMMI, there was nothing really special to perform.

Alerts, Workspaces, WI Queries…

Well, we had plan to actively assist people with those, we actually ended up in writing a very small guide with proper steps to get their Alerts back, to recreate their Queries (with examples) and more importantly, the new bible on how to create a proper Workspace.

So all those aspects were covered manually.

A word on security…

Security has also to be updated in a bunch of places.

In TFS you have to configure security in :

  • The source control folders
  • The work Item areas (to define accessibility of work items within one single team project)
  • SharePoint and Reports (but we did not bother since we have basic use of those)

You’ll want to use local groups (groups defined in TFS), and put in these groups AD groups. On that point my client was mature, and security was easier to manage than I first expected.

 

Now, the overall strategy is set, we’ll see the issues with work items, sources and builds in the next posts. Stay tuned Smile