How to install VSTS deployment group agents on Azure VMs

I recently got to work on an Azure migration project where we took the lift & shift approach as a first step. This means that the solution, while running in Azure, was still making use of virtual machines.

We decided to create two separate release pipelines:

  • the one that would provision the infrastructure in Azure — this one would be run only once for each environment as we don’t plan on tearing down/bringing up the resources for each application deployment; and
  • the application deployment one, which would update the applications bits on the virtual machines created in the first step — this one would be run much more frequently

The second one, that deploys the applications to the virtual machines, runs from a cloud-hosted agent provided by VSTS and uses WinRM to connect to the VMs to perform all the necessary steps, like copy scripts and packages over, configure IIS, deploy the packages, etc…

When I presented that solution to a few colleagues, one of them asked:

Why didn’t you install VSTS agents on the VMs? It’s more secure since it uses a pull model (instead of a push one), meaning you wouldn’t need to punch holes in the firewall for the cloud agent to connect to the virtual machines.

They have a very good point! I might add that another benefit of running the release directly from the VMs would likely speed up the process, as the artifacts would be downloaded automatically on the VM at the start of the release, and each and every step in the release wouldn’t need to set up a WinRM connection to the VM.

So I started looking for a way to do exactly this. We are using the built-in Azure Resource Group Deployment task, and one of the arguments called Enable Prerequisites allows to install the VSTS deployment group agent on all the VMs declared in your ARM template.

What’s this deployment group agent?

VSTS introduced some time ago the concept of deployment group, which is a bunch of target machines that all have an agent installed and can be assigned tags. I find it’s similar to the way Octopus Deploy works. When using deployment groups, the release pipeline is made of deployment group phases, where each phase runs on servers with specific tags. This means you could execute different tasks on your database servers and on your web servers, or you could decide to split them based on which application they run. If you’re more interested in this, I suggest you read the official documentation.

Going back to the VSTS task, here’s the property that allows you to install the agent on the virtual machines:

Install the VSTS deployment group agent on VMs
The setting that drives the installation of the deployment group agent on VMs

After selecting that option, we’re prompted to fill in a few additional properties:

  • a VSTS service endpoint;
  • a team project within the previously selected VSTS instance;
  • a deployment group that belongs to the selected team project;
  • whether we want to copy the tags from each VM to the associated agent; and finally
  • whether we want to run the VSTS agent service as a different user than the default one
Settings required to install the deployment group agent on VMs
The settings required to install the deployment group agent

This all worked out as expected, and going back to my deployment group after the privisionning of the VMs, I could see as many agents as VMs that were created. The next task was to modify the application deployment pipeline to adapt it to the fact that the process would now run directly on the virtual machines, and remove the rules that allowed inbound traffic for WinRM. It’s also worth noting that the process now needs to contain deployment group phases as opposed to agent phases.

Using this approach has several benefits:

  • increased security, as no inbound traffic is required to the VMs;
  • a quicker release process as there’s no need for WinRM connections for each step;
  • it also handles potential changes in the infrastructure: if we decide to increase the number of VMs for an application for increased reliability, the fact that the application deployment pipeline is based on VM tags means this will be transparent

Going deeper

While the main goal was achieved, I had a few questions in my mind:

  • how does the VSTS task install the VSTS agent on all the VMs?
  • why does the task require a VSTS service endpoint if the agent is to be connected to the same VSTS instance as the one where the release runs?

As all the VSTS tasks are open-source — if you didn’t know, you can find the source code in the Microsoft/vsts-tasks repository on GitHub — I decided to take a look under the hood.

The code for the Azure Resource Group Deployment task is in the Tasks/AzureResourceGroupDeploymentV2 folder.

The task.json file contains metadata about the task, like its name, the different input properties — and the rules around conditional visibility, like show setting B only when setting A has this value — and the execution entry point to invoke when the task need to run.

After finding the Enable prerequisites property, I traced the execution flow of the task until I landed on the DeploymentGroupExtensionHelper.ts which handles all things related to the installation of the deployment group agent on VMs.

And surprise! The VSTS task delegates the installation to the TeamServicesAgent Azure VM extension, as these two functions show. This answers the second question I had: the VSTS task needs a VSTS service endpoint to generate a PAT to register the agent as the underlying Azure VM extension rquires one.

The good thing about the fact that the agent installation is handled with an Azure VM extension is that we can easily reduce the coupling to this task by deploying the extension ourselves in the ARM template. This means that if we decide to move away from the VSTS task and do the deployment with either PowerShell scripts or the Azure CLI, we won’t be losing anything.

How to integrate Autofac in ASP.NET Core generic hosts

ASP.NET Core 2.1 brought a new feature that is generic hosts. They allow to write apps that rely on ASP.NET Core concepts like logging, configuration and built-in DI but that are not web applications.

I was playing with them yesterday and wanted to see if I could easily integrate the Autofac IoC container with it. After looking at the ASP.NET Core integration page in the Autofac docs, I came up with code that looks like the following:

using System.Threading.Tasks;
using Autofac;
using Autofac.Extensions.DependencyInjection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;

internal class Program
{
    public static async Task Main(string[] args)
    {
        await new HostBuilder()
            .ConfigureServices(services => services.AddAutofac())
            .ConfigureContainer<ContainerBuilder>(builder =>
            {
                // registering services in the Autofac ContainerBuilder
            })
            .UseConsoleLifetime()
            .Build()
            .RunAsync();
    }
}

This all looks pretty straightforward and follows the docs, but at runtime the application threw an exception with the following error message:

System.InvalidCastException: 'Unable to cast object of type 'Microsoft.Extensions.DependencyInjection.ServiceCollection' to type 'Autofac.ContainerBuilder'.'

That’s interesting, given:

  • services.AddAutofac() registers an AutofacServiceProviderFactory instance as IServiceProviderFactory as we can see here; and
  • the code tells us that the CreateBuilder method of AutofacServiceProviderFactory returns an instance of ContainerBuilder

So we’re all good, right?! What’s wrong?! Interestingly, I also read Andrew Lock’s post about the differences between web host and generic host yesterday, and thought maybe something was fooling us into thinking we were doing the right thing.

So I cloned the aspnet/Hosting repo, checked out the 2.1.1 tag, opened the solution in Visual Studio, and started readong through the HostBuilder.cs file.

And there it was: the HostBuilder class uses a ServiceProviderAdapter that wraps the IServiceProviderFactory. This means that registering an IServiceProviderFactory like services.AddAutofac() does conveys no meaning to a HostBuilder.

Luckily, while going through the code, I also found the UseServiceProviderFactory method on the HostBuilder class. The difference is that this one wraps the provided factory within the adapter.

The code then became:

using System.Threading.Tasks;
using Autofac;
using Autofac.Extensions.DependencyInjection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;

internal class Program
{
    public static async Task Main(string[] args)
    {
        await new HostBuilder()
            .UseServiceProviderFactory(new AutofacServiceProviderFactory())
            .ConfigureContainer<ContainerBuilder>(builder =>
            {
                // registering services in the Autofac ContainerBuilder
            })
            .UseConsoleLifetime()
            .Build()
            .RunAsync();
    }
}

And it worked!

I don’t know why the generic host uses an adapter around the service provider factory — I asked the question on Twitter, time will tell if we get the answer.

The morale here is very close to the one in Andrew’s post: don’t assume everything you know about web host is true or will work with generic host.

Why, when and how to reinstall NuGet packages after upgrading a project

I was working a codebase this week and noticed a few build warnings that looked like this:

Some NuGet packages were installed using a target framework different from the current target framework and may need to be reinstalled.
Visit http://docs.nuget.org/docs/workflows/reinstalling-packages for more information.
Packages affected: <name-of-nuget-package>

The docs page is really helpful in understanding in which situations this can happen, but we’ll focus on the one situation mentioned in the warning, that is upgrading a project to target a different framework.

How it looks like before upgrading the project

Let’s imagine we have an exiting project targeting .NET 4.5.2 and the Serilog NuGet package is installed. If we’re using packages.config and the old .NET project system, our .csproj file will contain something that looks like the following:

<Reference Include="Serilog, Version=2.0.0.0, Culture=neutral, PublicKeyToken=24c2f752a8e58a10, processorArchitecture=MSIL">
  <HintPath>..\packages\Serilog.2.7.1\lib\net45\Serilog.dll</HintPath>
</Reference>

The above snippet shows that the assembly that is being used by the project is the one living in the net45 folder of the NuGet package, which makes sense since we’re targeting .NET 4.5.2.

Upgrading the project

We then decide to upgrade the project to target .NET 4.7.1 through Visual Studio.

Immediately after doing this, we get a build error with the message shown at the beginning of this post. On subsequent builds, though, the error goes away and we get a warning, which is consistent with what’s documented in item #4 of the docs page.

But why?!

Why do we get those warnings?

NuGet analysed all the installed packages and found out that there are more appropriate assemblies for the new target framework than the ones we’re referencing. This is because a NuGet package can contain different assemblies for different target frameworks.

Let’s inspect the content of the lib directory of the Serilog package:

└─lib
  ├─net45
  │   Serilog.dll
  │   Serilog.pdb
  │   Serilog.xml
  │
  ├─net46
  │   Serilog.dll
  │   Serilog.pdb
  │   Serilog.xml
  │
  ├─netstandard1.0
  │   Serilog.dll
  │   Serilog.pdb
  │   Serilog.xml
  │
  └─netstandard1.3
      Serilog.dll
      Serilog.pdb
      Serilog.xml

We can see different assemblies for 4 different target frameworks.

My guess is that those warnings are driven by the requireReinstallation attribute that is added for those packages in the packages.config file:

<packages>
  <package id="Serilog" version="2.7.1" targetFramework="net452" requireReinstallation="true" />
</packages>

How to fix this?

The way I find easiest to do this is by using the Package Manager Console in Visual Studio by running this command:

Update-Package <name-of-nuget-package> -Reinstall -ProjectName <name-of-project>

The most important parameter here is -Reinstall as it instructs NuGet to remove the specified NuGet package and reinstall the same version. This gives NuGet a chance to determine which assembly is most appropriate for the current framework targeted by the project.

Running this command in our sample project would change the .csproj:

<Reference Include="Serilog, Version=2.0.0.0, Culture=neutral, PublicKeyToken=24c2f752a8e58a10, processorArchitecture=MSIL">
  <HintPath>..\packages\Serilog.2.7.1\lib\net46\Serilog.dll</HintPath>
</Reference>

And also the packages.config file:

<packages>
  <package id="Serilog" version="2.7.1" targetFramework="net471" />
</packages>

The project now references the .NET 4.6 assembly of the package, and the build warning is gone. I don’t know how NuGet internally determines which set of assemblies is best suited for a target framework, though. There might be a matrix somewhere that shows this.

We can run the command for every package that is flagged by NuGet to make sure we reference the correct assemblies. Alternatively, if too many packages are

Conclusion

We saw that it’s easy to get rid of the warnings that can occur when a project is upgraded to target a different framework.

Do you see those warnings when you build a solution? Does a solution-wide search for requireReinstallation fetch some results? You’re only a few commands away to being in a cleaner state! Fire away!

Asynchronous initialisation and cleanup operations with xUnit

Last week I was writing integration tests and I wanted to reset the underlying database to a known state before each test. I/O-bound operations are a great use case of asynchronous tasks, so I was wondering how xUnit would help me support this.

Lifecycle events

Every .NET test framework supports some lifecycle events. They allow you to execute operations at different times, usually on 3 different levels.

The first one, the test level, is also the most fine-grained one as it lets you run code before and after every test in a class. I tend to use this lifecycle event when I need a clean state for every test. In this case, I would create a new instance of the SUT.

The second level is the class or fixture level. Like its name implies, this gives you a chance to execute operations before the first and after the last test of a specific test class. This is useful when you want to have all the tests of a single test class to share some common state. While I don’t use it as often as the two others, a good use case for this one would be when I test a class that doesn’t hold any state, so creating new instances for each test wouldn’t add any value.

The last one, I’ll call the suite level; this one allows you to run some code before the first test and after the last test of the whole suite. This comes in handy when you have some initialisation code that needs to run only once. Usually, it will match what you do at the very start of your application. I use this one to configure my AutoMapper mappings or, if I write integration tests, to run migrations on the underlying database.

How to use lifecycle events with xUnit

xUnit supports all these options, and you can read about how to use them on the official documentation page.

One thing you’ll notice is that initialisation and cleanup mechanics fit the .NET semantics; the former is done in the constructor of the class, the latter by optionally implementing the IDisposable interface.

Unfortunately, at the time of writing, neither do constructors nor IDisposable support asynchronous, Task-based operations without somehow blocking the running thread.

IAsyncLifetime to the rescue

Fortunately, xUnit has us covered with a special interface. Its declaration is very simple (see the official one):

public interface IAsyncLifetime
{
    Task InitializeAsync();
    Task DisposeAsync();
}

If either your test class, class fixture or collection fixture implement this interface, xUnit will execute the appropriate methods at the right time. InitializeAsync will be called right after the constructor is invoked, and DisposeAsync just before Dispose - if it exists - is called.

How I used it

To have each of my integration tests run on top of a known state, I wanted clean up the SQL database before each test. To do this I used yet another open-source library created by Jimmy Bogard called Respawn.

He decided, after porting it to .NET Standard 2.0, to make the Reset method async.

This was the perfect opportunity to use IAsyncLifetime, and because a picture some code is worth a thousand words:

public class TestClass : IAsyncLifetime
{
    private readonly string _connectionString;
    private readonly Checkpoint _checkpoint;

    public TestClass()
    {
        _connectionString = GetItFromConfigurationFile();
        _checkpoint = new Checkpoint();
    }

    public Task InitializeAsync() => _checkpoint.Reset(_connectionString);

    [Fact]
    public async Task TestOne()
    {
        // Asynchronous operations again
    }

    [Fact]
    public async Task TestTwo()
    {
        // Asynchronous operations again
    }

    public Task DisposeAsync => Task.CompletedTask;
}

Playing with the VSTS Linux agent and Docker

Microsoft recently added Linux agents to the VSTS hosted pool in preview. These agents are Docker containers and have the specifity of reusing the host Docker instance from which they were created.

We’ll see in this post how we can take advantage of such capability.

Use case

At Readify, we’ve been working on an internal application that uses OrientDB, a graph database engine. People working on this project agreed to use the OrientDB Docker container as opposed to having to install Java and a local instance of OrientDB on their development machines.

While this works great for development, we couldn’t apply it to the VSTS CI build to run integration tests as the agents were Windows-based and didn’t have Docker installed. The workaround was to spin up a local instance of OrientDB if an existing one couldn’t be found. This means that developers could still run tests as they’d have an existing, reachable instance of OrientDB running in Docker, while during the CI build a local instance would be created.

This worked fine, but required more code to create the new OrientDB instance, and also meant we didn’t have consistency between the dev and CI setups. When the Linux Hosted pool showed up on the Readify tenant of VSTS, we decided to give it a go and try to run OrientDB in Docker.

Running a Docker container as part of the CI build

Out of the box, VSTS doesn’t provide tasks to run Docker workloads. Fortunately, an extension backed by Microsoft exists on the Visual Studio Marketplace. The code is hosted in the vsts-docker GitHub repo for the curious ones who want to have a look at it.

The extension brings 3 new tasks:

  • Docker: to build, push or run Docker images. It can also run custom Docker commands
  • Docker Compose: to build, push or run multi-container Docker applications
  • Docker Deploy: this one allows use to deploy single or multi-container Docker applications to Azure Container Services

We are here only interested in running a Docker container, so the first task will suffice our needs.

Run an OrientDB container

As you can see, it takes the parameters you would expect:

  • The name of the image
  • The name of the container
  • Port mapping
  • Volume mapping
  • Environment variables

Remember that if this doesn’t suit your needs, you can always fall back to running a custom docker command that would allow you to specify all the parameters yourself.

Connecting to the Docker container

There was an issue where the tests couldn’t connect to the OrientDB database when running on the CI build. The error message was Connection refused 127.0.0.1:2424. It took a while to figure out, but since the VSTS agent is not the Docker host, it’s normal that the OrientDB container is not reachable through localhost. This means we need to figure out the IP address of the OrientDB container and connecting using this IP.

Getting the IP of the OrientDB container

That was easy. I mean, the first Google search pointed to a serverfault question that explains how. docker inspect returns information about the container, and the --format option allows you to specify what portion you want back. To only get the IP address, you can run:

docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_id

Have the tests use this IP

The tests get the OrientDB connection information through the configuration model of ASP.NET Core to get information about how to connect to OrientDB. It’s a provider-based model, which means you can add several sources of different types to the configuration. Providers added last can override configuration values added by previous providers; in other words, the last provider has the highest priority.

During development, all the settings are read from a JSON file. This can’t work for CI since the IP address of the OrientDB container could change between runs - even though a few tries showed it wasn’t the case. One option to override the hostname specified in the JSON file is to create an environment variable using the same name as the config setting. Environment variables are added last so they’ll take precedence. The configuration code then looks like:

var configuration = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("appsettings.json")
    .AddEnvironmentVariables()
    .Build();

and here’s the JSON file:

{
  "OrientDB": {
    "Port": "2424",
    "Username": "root",
    "Password": "verysecurepassword",
    "Host": "localhost"
  }
}

The goal is then to create an environment variable with the name OrientDB:Host and set it to the IP address of the OrientDB container. We saw that getting the IP was easy, but how do we create an environment variable as part of the build? VSTS has the concept of logging commands. Emitting specific text through the standard output means VSTS will parse that output and react appropriately. One of these commands can create an environment variable, the format is the following:

##vso[task.setvariable variable=<variable-name>;]<variable-value>

The solution was then to include a bash script task in the CI build - remember, this is a Linux agent we’re running on - to get the IP and output the appropriate logging command. Overall it looks something like:

#!/bin/bash

IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' orientdb`
echo "##vso[task.setvariable variable=OrientDB:Host;]$IP"

The tests now happily pick up the container’s IP address from the environment variable via the ASP.NET Core configuration model.

Other challenges when running in Linux

A few other things had to be changed for the build to run smoothly. The first one was folder separators in the various build tasks, as Linux uses /. We also had some code where backslashes were used in strings to represent paths; switching over to Path.Combine fixed this.

The file system on Linux is also case-sensitive, so myproject.tests and MyProject.Tests are two different things. We had a few occurences of this that needed updating.

Finally, we have some scripts in the git repository to setup dummy data before running each integration test. They are prefixed with numbers to indicate you need to run them sequentially for them to work properly. We found out that, on Linux, Directory.EnumerateFiles doesn’t return files in alphabetical order, while it does on Windows. We had to sort them manually before iterating over them.

Overall, it took some time to get this working, and at some point a lot of trial & error when running the builds to figure out what was happening, but we now have a CI build that is consistent with what happens during development.