Testing your TFS 2012 build workflow activities

I recently posted about setting up a solution for editing build workflows (with TFS 2012), I’m going to write directly today about testing your custom activities, because I think good guides about writing them are already out there, and there (even though they talk about TFS 2010, the logic is quite the same). Today, we’ll have a look at how I we test activities without actually starting a real build.

As I’m currently writing an activity for launching Sonar during TFS builds (you can learn about Sonar with TFS here). Although I’m not finished writing the whole stuff, I can still share a few tips I learned for testing activities.

Classic testing approach

The approach is very classic : set up a test context, exercise the test, and check the results (and clean up).

The tests rely on the ability of given by Workflow Foundation to host one’s own workflow engine. So we’ll need mainly two things :

  • Create the right workflow context to emulate the behavior of build (only what’s needed for the test of course)
  • Create an instance of the activity, passing necessary parameters and stubs to fulfill the test

During our test, the workflow engine will run the activity. The activity will use objects and value from its parameters and interact with the context. Then we can add checks about what has actually happened. Ready?

Workflow activities testing tips

Creating the activity and executing it

You can instantiate a Worklow activity with a classic new statement. Literal parameters can be passed in the constructor or affected (literals here are value types and Strings). All other objects must be passed in a Dictionnary<String, Object> structure at invocation time.

  1. // constants (literals)
  2. var activity = new Sonar
  3. {
  4.     // this is a String (a literal)
  5.     SonarRunnerPath = SonarRunnerPath,
  6.     // this is a boolean (a literal as well)
  7.     FailBuildOnError = FailBuildOnError,
  8.     GeneratePropertiesIfMissing = GeneratePropertiesIfMissing,
  9.     SonarPropertiesTemplatePath = TemplatePropertiesPath,
  10.     FailBuildOnAlert = FailBuildOnAlert,
  11.     // StringList is not a workflow literal
  12.     // the following line will cause an exception at run time
  13.     ProjectsToAnalyze = new StringList("dummy.sln")
  14. };

 

 

Here all values are booleans or strings, except one StringList that will cause an error at run time (so we must remove it). Here’s how to invoke the activity (actually a workflow composed of one activity) and pass the StringList as an argument:

  1. // object variables
  2. var parameters = new Dictionary<string, object>
  3. {
  4.     { "ProjectsToAnalyze", new StringList("dummy.sln") }
  5. };
  6.  
  7. // the workflow invoker, our workflow is composed of only one activity!
  8. WorkflowInvoker invoker = new WorkflowInvoker(activity);
  9. // executes the activity
  10. invoker.Invoke(parameters);

 

Tracking build messages

You may want to check what your activity is logging, you know, when you call the TrackBuildMessage method or use the WriteBuildMessage (or Warning or Error) activity. To do this you need to set up a recorder, or more exactly TrackingParticipant. Here is a TrackingParticipant derived class that is specialized to recording build messages:

A build message tracking class
  1. namespace MyBuilds.BuildProcess.Tests
  2. {
  3.     using System;
  4.     using System.Collections.Generic;
  5.     using System.Linq;
  6.     using System.Text;
  7.     using System.Activities.Tracking;
  8.     using Microsoft.TeamFoundation.Build.Workflow.Tracking;
  9.     using Microsoft.TeamFoundation.Build.Workflow.Activities;
  10.  
  11.     /// <summary>
  12.     /// BuildMessageTrackingParticipant that logs build messages during build workflow activities
  13.     /// </summary>
  14.     public class BuildMessageTrackingParticipant : TrackingParticipant
  15.     {
  16.         private StringBuilder _sb = new StringBuilder(4096);
  17.  
  18.         public override string ToString()
  19.         {
  20.             return _sb.ToString();
  21.         }
  22.         protected override void Track(TrackingRecord record, TimeSpan timeout)
  23.         {
  24.             var buildMessage = record as BuildInformationRecord<BuildMessage>;
  25.             if (buildMessage != null && buildMessage.Value != null)
  26.             {
  27.                 _sb.AppendLine(buildMessage.Value.Message);
  28.             }
  29.  
  30.             var buildWarning = record as BuildInformationRecord<BuildWarning>;
  31.             if (buildWarning != null && buildWarning.Value != null)
  32.             {
  33.                 _sb.AppendLine(buildWarning.Value.Message);
  34.             }
  35.  
  36.             var buildError = record as BuildInformationRecord<BuildError>;
  37.             if (buildError != null && buildError.Value != null)
  38.             {
  39.                 _sb.AppendLine(buildError.Value.Message);
  40.             }
  41.         }
  42.     }
  43. }

 
To use it, all you need is to instantiate it and pass the instance to the workflow invoker:
 
  1. var workflowLogger = new BuildMessageTrackingParticipant();
  2. invoker.Extensions.Add(workflowLogger);

 
After the test, you can get the build “log” by calling the .ToString() method on the worfklowLogger instance.
 

Setting up a custom IBuildDetail instance

 
During builds, activities regularly get the “build details”, an IBuildDetail instance that contains lots of useful contextual data. This instance comes from the worfklow context, and activities get it by using code that looks like the following:
 
  1. IBuildDetail build = this.ActivityContext.GetExtension<IBuildDetail>();

 
Thankfully, it is an interface, so it is very easy to stub. I like to use the Moq mocking framework because it is very easy (yet not very powerful, but it is perfect for classic needs). Now we need to create a stub out of the IBuildDetail interface, customizing it for your needs, and to inject it in the workflow “context”. I’ll actually assemble multiple stubs together because I also need to set up the name of the build definition for my activity (yes, the activity uses the current build definition name!):
 
  1. // the build definition stub that holds its name
  2. var buildDefinition = new Mock<IBuildDefinition>();
  3. buildDefinition.Setup(d => d.Name).Returns("My Dummy Build");
  4.  
  5. // a build detail stub with the buildnumber, build definition stub and log location
  6. var buildDetail = new Mock<IBuildDetail>();
  7. buildDetail.Setup(b => b.BuildNumber).Returns("My Dummy Build_20130612.4");
  8. buildDetail.Setup(b => b.BuildDefinition).Returns(buildDefinition.Object);
  9. buildDetail.Setup(b => b.LogLocation).Returns(Path.Combine(TestContext.TestDeploymentDir, "build.log"));
  10.  
  11. // pass the stub to the invoker extensions
  12. invoker.Extensions.Add(BuildDetail.Object);

 
Now, the activity “thinks” it is using real “build details” from the build, but during tests we are using a fake object, with just the necessary values for the test to pass. So this is actually a pure classic stubbing scenario, no more.
 

Passing TFS complex objects

 
Unfortunately, not all of the objects and classes we need in build activities are interfaces, or pure virtual classes. Those are easy to stub. In the case of objects such as a Workspace, a VersionControlServer, a WorkItemStore, or a WorkItemType, you have to use more powerful stubbing frameworks such as Microsoft Fakes or Typemock.
Let’s use Fakes since it is available the Visual Studio Premium edition.
First, locate the assembly that our target type belongs to. The Workspace class belongs to the Microsoft.TeamFoundation.VersionControl.Client assembly. Right-click it in the References of your project and add a Fakes assembly:
 
image
 
Fakes processes all types and members of this assembly and dynamically generates a new reference which contain “empty” objects, with all overridable properties and members, and compatible with the original types. All types are prefixed by Shim or Stub, and methods include types of their signatures in their names. Here is an example that illustrates how to set up a Workspace “Shim”. When we call the GetLocalItemForServerItem method, it will return the value we want, that is LocalSolutionPath:
 
  1. // our workspace stub
  2. ShimWorkspace workpace = new ShimWorkspace()
  3. {
  4.     // we override String GetLocalItemForServerItem()
  5.     // and have it return a value of our own for the test
  6.     GetLocalItemForServerItemString = (s) => LocalSolutionPath
  7. };

 
To pass the actual Workspace compatible object to our activity as a parameter, use its .Instance property. Since it is not a workflow literal, let’s use the Dictionnary like we did before:
 
  1. // object variables
  2. var parameters = new Dictionary<string, object>
  3. {
  4.     { "BuildWorkspace", workpace.Instance },
  5.     { "ProjectsToAnalyze", new StringList("dummy.sln") }
  6. };

 
 

Ok, we covered a few techniques that should allow you to test most activities now. When I’m satisfied with the tests I’m currently writing, I’ll publish them in the Community TFS Build Extensions project. So keep an eye on them if you’re are interested for a full running piece of code, sorry to make you wait!

Migrating Coded UI Tests to VS 2012: small issue with project dependencies

I’ve upgraded my Visual Studio 2010 Coded UI tests to Visual Studio 2012, and I faced a small issue. The migration considerations are documented in MSDN here. For me, it did not work out of the box, my tests were not found by the Test Explorer, and as soon as I could fix this, tests would not run either, spawning strange exceptions I never had before. I’ll describe below how I finally fixed my Visual Studio solution. The problem is covered by this blog post by the ALM team, but I’m providing the symptoms, and the resolution process in a detailed way. In case people face similar situations, I’m including the intermediate steps, but the real answer to my problem is in the middle of the post (emphasized with bold red font).

How my solution is structured

SNAGHTML68ec91

A smooth migration

When opening the solution with VS 2012, the projects containing Coded UI tests are “repaired”: some magic is performed in the project files in order to keep the compatibility with VS 2010 SP1. This cross-compatibility between VS 2010 SP1 and VS 2012 is quite cool, both versions can concurrently develop on the same project, unusual but true!

The “migration” log reports the following message:

FrontSiteUiTests.csproj: Visual Studio has made non-functional changes to this project in order to enable the project to open in this version and Visual Studio 2010 SP1 without impacting project behavior.
 

But where are my Coded UI Tests ?

In VS 2012, there is no more Test View, but a Test Explorer. The Test Explorer has a discovery process which asks you to compile the solution if no tests are found. Despite hammering CTRL+SHIFT+B on my keyboard, no test showed up in the box.

image

What to do then ? Well, there is a new Visual Studio Output for the Testing tools, you’ll find there discovery errors messages. Head to the Output Window, next to “Show output from:”, select “Tests”. The following error message is displayed multiple times:

Error loading C:\Sources\Platform Tests\Company\CodedUITests\FrontSiteUiTests\bin\Debug\Company.Testing.Ui.FrontSiteUiTests.dll: Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

Accusing inheritance among Coded UI tests

I first thought the error was coming from the structure of my coded UI tests. I have a base class for all my Coded UI tests, this class is decorated by a CodedUITestAttribute, and moreover, there is an intermediate class in the inheritance tree. This one also has a CodedUITestAttribute on it.

I’ve always thought this was a small trick, but VS 2010 seemed support it, though I’m not sure about future versions… So I tried different combinations of attributes on the base and intermediate classes, varying from no attribute, to TestClass, to CodedUITest. I thought I was in the right direction since tests would now appear in the Test Explorer (restarting VS helped Winking smile ).

SNAGHTML6dd51b

The details are no real matter here, but I faced various exception messages while playing with attributes, for reference sake I’ll post them here:

At compile time with no attribute in the base class, so TestClass or CodedUITest is at least required:

UTA005: Illegal use of attributes on Company.Testing.Ui.Core.CdsUITestBase.MyTestInitialize.The TestInitializeAttribute can be defined only inside a class marked with the TestClass attribute.
UTA006: Illegal use of attributes on Company.Testing.Ui.Core.CdsUITestBase.MyTestCleanup. The TestCleanupAttribute can be defined only inside a class marked with the TestClass attribute.

At run time, using TestClass instead of CodedUITest :

Initialization method Cdiscount.Testing.Ui.OrderProcess.Check_Customer.Customer_Login.CustomerLogin.MyTestInitialize threw exception. System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.VisualStudio.TestTools.UITesting, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.

This message (similar to the one reported by the discovery process) pointed out the real problem: VS 2012 should use version 11.0.xxx references and not 10.0.xxx

The problem is actually pretty simple, the wizard has upgraded my Coded UI Test project but not the project that contains the base and utility classes. There is no messing around with attributes involved.

So all we need is to reproduce the “magic” on our referenced projects.

Additionally, beware of external dependencies directly built upon version 10.0.xxx of the testing tools assembly set, you’ll have to rebuild them upon v11.0.xxx.

The problem was also reported and solved here.

Upgrading (or reparing) projects manually

Disclaimer: although I’m exposing here as a commodity the implementation of this “magic” performed by VS 2012 update 1, I advise to watch closely at what is done to your own Coded UI test projects as a base. Simply compare the changes before checkin-in your proj file. That said, the following should work for you.

In the first PropertyGroup with most global properties:

<VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>
<VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)</VSToolsPath>
<ReferencePath>$(ProgramFiles)\Common Files\microsoft shared\VSTT\$(VisualStudioVersion)\UITestExtensionPackages</ReferencePath>
<IsCodedUITest>True</IsCodedUITest>
<FileUpgradeFlags>
</FileUpgradeFlags>
<OldToolsVersion>4.0</OldToolsVersion>
<UpgradeBackupLocation />

After the last ItemGroup and before the last Import:

<Choose>
  <When Condition="'$(VisualStudioVersion)' == '10.0' And '$(IsCodedUITest)' == 'True'">
    <ItemGroup>
      <Reference Include="Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
      <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Common, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
      <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Extension, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
      <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Extension.Firefox, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
      <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Extension.Silverlight, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
      <Reference Include="Microsoft.VisualStudio.TestTools.UITesting, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
        <Private>False</Private>
      </Reference>
    </ItemGroup>
  </When>
</Choose>
<Import Project="$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets" Condition="Exists('$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets')" />

Next, in the <Reference … /> node under <ItemGroup>, delete all Reference to assemblies like Microsoft.VisualStudio.QualityTools.* and Microsoft.VisualStudio.TestTools.*, except for UnitTestFramework, so leave this entry:

<Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />

These evolutions actually make the ReferencePath depend on the version of Visual Studio. Cool.

Now all the references, including those from the utility project are mapped to the correct version, that is v11.0.0.0:

image

So in the end everything’s fine, tests are properly discovered, and run fine as they just ran with VS 2010.

Finalizing the migration

There have been deep evolutions in the testing tools between VS 2010 and VS 2012, for example vsmdi files, which used to contain test lists, are no longer supported. This is all well documented in MSDN. This may be qualified as a regression in functionality, but personally I absolutely don’t feel angry at all or even surprised, those lists needed to evolve, I’ve never been keen on them and I’m quite happy the way it is today. Now, the test tagging system is the only rightful way to categorize tests and define running lists: just as it has always been with other testing frameworks. Less ambiguity, better tools, more productivity.

You’ll surely want to add multiple categories to single test methods, it is good practice:

[TestMethod, TestCategory("Daily"), TestCategory("Nightly"), Priority(1), Description("Check the customer login")]
public void CodedUITestMethod1()

eg: some database tests may be eligible for your nightly builds.

image

Just in case

As a final word, just in case you have some mess in your class attributes for your Coded UI Tests, TestClass is not suitable as base class for Coded UI Tests, as you will have the following message at run time:

Result Message:    Initialization method Cdiscount.Testing.Ui.OrderProcess.Check_Customer.Customer_Login.CustomerLogin.MyTestInitialize threw exception. Microsoft.VisualStudio.TestTools.UITest.Extension.TechnologyNotSupportedException: Microsoft.VisualStudio.TestTools.UITest.Extension.TechnologyNotSupportedException: The browser  is currently not supported..