Creating reusable build scripts with NUKE components

When your application landscape expands, doing CI/CD becomes a challenge. Applications need to be built, tested, and deployed, and then tested again. Typically, each application is a bit of a snowflake in some way; it’s built in a slightly different way, or some tests need to be excluded, or publishing is more involved than with others. Maintaining CI/CD pipelines becomes a chore, and problems that arise tend to eat up a lot of time.

In this situation, it can be beneficial to simplify things a bit. In our case (that case being Coolblue), almost all applications have a three-stage build pipeline:

Pull Request

This is executed each time a pull request is made or updated; it builds the application and runs unit and integration tests.
Build Release

This is executed each time a change is merged (or pushed) to the main branch; it also builds the applications and runs unit and integration tests, but it also prepares the application to be deployed–this means packaging the application, uploading it to our deployment system, and associating some metadata with it.
Release

This is executed after the previous step completes successfully, and deploys the application to one of the environments. If the environment is the last one before production, it also runs automated acceptance tests.

For each stage of the pipeline, there’s only one step: execute a PowerShell script called build.ps1, which is always located in the root of the application’s repository, and pass it some strictly defined arguments. These argument always include the ‘target’, or ‘what to do’, and, depending on the stage, arguments like ‘which environment’ and so forth.

If an application needs to do something special, it can almost always be solved by just modifying this PowerShell script. The CI/CD pipelines become static and can be created at the click of a button.

Cake

Of course, only the most simplistic applications can get away with a build script as simple as dotnet publish and dotnet test. Deploying an application already involves using external tooling to be able to communicate with the deployment system. In addition, our build scripts do things beside building and testing, like:

Adding build metadata, like when an application was built, and from which Git commit.
Setting up secrets to be able to run integration tests against a live database, or to be able to send authenticated requests to a deployed application.
Notifying the team when the application fails to build, tests fail, or when a deployment fails.

While this can be done in PowerShell, it’s not the easiest language to use, and not everyone is well versed in it.

Enter Cake.

For those not familiar with it, Cake is a build automation system with a C#-based DSL. It is based on the concept of tasks, which are small units of work, like restoring packages, building, or testing, which can have dependencies on, or trigger, other tasks.

For example:

Task("PullRequest")
    .IsDependentOn("Build")
    .IsDependentOn("Test");

Task("Build")
    .Does(() =>
          {
              DotNetCoreBuild("MyApplication.sln");
          });

Task("Test")
    .Does(() =>
          {
              DotNetCoreTest("test/MyApplication.Tests/MyApplication.Tests.csproj");
          });

RunTarget("PullRequest");

This will run the PullRequest task, which is dependent on Build and Test (in that order), so it will run dotnet build MyApplication.sln and dotnet test test/MyApplication.Tests/MyApplication.Tests.csproj.

So where does build.ps1 come in? Well, you can’t just run a Cake script; you need a runner. The build.ps1 script’s job is to bootstrap this. It downloads the Cake runner from NuGet and invokes the runner, which will then compile and execute the Cake script.

Bad Cake

Cake is a very nice system, but there are some problems when using it at scale, as we do.

Because Cake is not ‘pure’ C#, but rather a C#-based DSL, enhanced with some additional features, there’s no good editor support. Some editors, like Visual Studio Code, support syntax highlighting, but IntelliSense is limited to the default ‘hey I’ve seen this symbol before but it might not be appropriate in this context’ version¹. Because of this, the feedback loop is very cumbersome: edit a file, run build.ps1, get a syntax error, fix it, rinse and repeat.

It does not help that Cake’s primary building block, the task, is ‘stringly typed’. Task names are always expressed as strings. That means that if you mistype the name of a task somewhere, you won’t know until you run the build. Navigating to a particular target is also notoriously cumbersome.

Cake’s API for dealing with file system paths is well-meant, but because it is not consistently applied, it frequently means you have to convert to and from strings, and between ‘convertable paths’ and ‘actual paths’. This is a major source of compilation errors, again because there’s no proper editor support.

Finally, one of the biggest issues is that reusing code is not made very easy. You can write NuGet packages to extend Cake, but as far as I’m aware those cannot define new tasks. You can also package one or more Cake scripts into a NuGet package, but this is again cumbersome (as you have to mess about with .nuspec files), and as far as I’m aware, this also doesn’t let you define tasks.

NUKE

All of these issues were encountered relatively recently when I had to make changes to a Cake script. The way Cake scripts have evolved within our organization is that some people have invested some time into creating a single set of Cake scripts that does almost everything we want. That set of scripts was then massively copied and pasted into other projects. Over time, some scripts gained new abilities, but because there was no proper way of sharing them back to the origin or all of its clones, each build script becomes a snowflake. The result is that the mental load of modifying Cake scripts is massive, because you need to account for a specific script’s peculiarities and limitations.

It was at that point I started searching for an alternative. I quickly ran into another build automation system that I’d heard of before, but never really used: NUKE. Dennis Doomen (also known as ‘the guy who wrote FluentAssertions’) had written a very nice article outlining why he thought it was a great replacement. He outlines more reasons, but NUKE specifically addresses all of the issues I was having with Cake.

It is ‘normal’ C# in a normal project in your solution, which means you get the benefits of IntelliSense, or Smart Completion, or whatever your editor calls it.
Targets (the NUKE equivalent of Cake’s tasks) are regular C# symbols (either fields or properties), which means the compiler and your editor are aware of when you mistype a name and can point this out to you.
The file system and path API is excellent, although it still looks weird when you see the ‘division’ operator being overloaded to construct paths (Root / "src" / "MyApplication" / "MyApplication.csproj").
You can reuse code by using NuGet packages, just like in regular projects.

The Cake example from before looks like this in NUKE:

class Build: NukeBuild
{
  public static int Main() => Execute<Build>(x => x.PullRequest);

  [Solution]
  Solution Solution;

  Target PullRequest =>
    _ => _.Triggers(Build)
          .Triggers(Test);

  Target Build =>
    _ => _.Executes(
      () => DotNetTasks.DotNetBuild(o => o.SetProjectFile(Solution))
    );

  Target Test =>
    _ => _.Executes(
      () => DotNetTasks.DotNetTest(o => o.SetProjectFile(Solution))
    );
}

Reusability

So how does one reuse code in a NUKE build script? Let’s say we want to reuse a target, like publishing in Release mode for the win10-x64 runtime while treating warnings as errors.

Since the build class needs to contain all the supported targets, composition is not an option. The only option left, then, is to introduce a base class.

abstract class MyBuild: NukeBuild
{
  Target Publish =>
    _ => _.Executes(() => { /* publish code goes here */ });
}

class Build: MyBuild
{
  Target BuildRelease =>
    _ => _.Triggers(Publish);
}

Because of inheritance, the Build class now exposes two targets: Publish and BuildRelease. Job done, right? Well, not so fast. What about optional features? For example, not all of our projects have automated acceptance tests. If every build exposed a target for running acceptance tests, you’d have no idea whether that target is there because of the base class or because it actually does something useful.

Of course, we could define a base class for ‘build without acceptance tests’ and ‘build with acceptance tests’, but that only works for the one feature. If you have multiple optional features that are not mutually exclusive, then you end up having to create many different subclasses for all the different combinations. If you have n number of optional features, the number of subclasses you have to create (and maintain) is 2ⁿ. For two features, that’s 4 classes; for three features, 8 classes, and for four features it adds up to 16 classes. You’ll also have a lot of code duplication, because you can’t inherit from anything (otherwise you’d end up with the same problem you were trying to solve). You can see why this isn’t a great idea.

Default interface implementations

Fortunately, NUKE’s got you covered. The solution is... (drum roll) multiple inheritance! Great, but C# doesn’t have multiple inheritance. Well, since C# 8.0, it kind of does.

C# has technically always supported multiple inheritance, using interfaces. Originally, this meant that you could only ‘inherit’ multiple contracts, but not actual behavior. That’s where default interface implementations come in. Introduced in C# 8.0, it allows any interface member to have a default implementation. An implementing class (or struct) does not need to provide an implementation for that member, although it is allowed to, in order to override the default behavior.

One of the examples of this I like, which was also used in a video where Mads Torgersen explains C# 8.0’s new features, was that of a logger. A typical logger interface has members like LogDebug, LogInformation, LogError, and so forth. It’ll usually also have something like a Log or Write method, which has the same parameters, preceded by a ‘level’ parameter. The implementation of LogDebug and similar methods is usually to call that Log method with the respective log level.

enum LogLevel
{
  Debug,
  Information,
  Warning,
  // etc.
}

interface ILog
{
  void Log(LogLevel level, string message);
  void LogDebug(string message);
  void LogInformation(string message);
  // etc.
}

class Log: ILog
{
  public void Log(LogLevel level, string message) { /* omitted for brevity */ }

  public void LogDebug(string message) => Log(LogLevel.Debug, message);
}

Every implementation of ILog needs to implement LogDebug and friends, even though that implementation will likely be exactly the same as the existing ones.

With default interface implementations, the default implementation of these methods can be provided in the interface:

interface ILog
{
  void Log(LogLevel level, string message);
  void LogDebug(string message) => Log(LogLevel.Debug, message);
  void LogInformation(string message) => Log(LogLevel.Information, message);
  // etc.
}

Now the only thing an implementation of ILog needs to implement is the Log method. This also means that you can add new methods to the interface without breaking existing implementations.

NUKE build components

So how does all of this help us with the problem of supporting optional features in our NUKE build script? NUKE calls this concept ‘build components’. It ‘allow us to organize parameters and targets in reusable build components, and freely compose our build out of them’.

In a nutshell, your Build class can ‘inherit’ from multiple interfaces, where each interface represents a ‘component’, which can have its own targets and parameters. Each interface is ultimately derived from a well-known interface (INukeBuild). NUKE understands this mechanism of defining targets, and will find targets defined on all ‘inherited’ interfaces.

Our earlier example with acceptance tests can now look like this:

interface IDefaultBuild: INukeBuild
{
  Target Publish => _ => _.Executes(() => /* publish code goes here */);
}

interface IBuildWithAcceptanceTests: INukeBuild
{
  Target RunAcceptanceTests => _ => _.Executes(() => /* code to run acceptance tests goes here */);
}

class Build: NukeBuild, IDefaultBuild, IBuildWithAcceptanceTests
{
  public static int Main() => Execute<Build>();
}

By just implementing the IBuildWithAcceptanceTests interface, the build gets a whole new set of features, whereas if you choose to not implement it, the features are not in your way.

But it gets better. In our pipeline, we run the acceptance tests as part of the Release target — first we release the application into an environment, and then we run acceptance tests. How do we combine this with the fact that acceptance tests are optional?

We can’t use Triggers on the Release target, because that would fail if the acceptance test component hasn’t been implemented. We could use TriggeredBy from the acceptance test target, but that will fail if the release component hasn’t been implemented. We don’t want to force people to implement particular components; they’re there if they are useful to you, and nothing more.

Instead, we can use TryTriggeredBy<T>. This lets you express ‘if the build is a T, this target is triggered by these targets’. This is just what we need. There are similar methods for DependsOn, DependentOn, Triggers, and even After and Before.

interface IDefaultBuild: INukeBuild
{
  Target Release => _ => _.Executes(() => /* omitted for brevity */);
}

interface IBuildWithAcceptanceTests: INukeBuild
{
  Target RunAcceptanceTests =>
    _ => _.TryTriggeredBy<IDefaultBuild>(b => b.Release)
          .Executes(() => /* omitted for brevity */);
}

Now, if your build implements IDefaultBuild and IBuildWithAcceptanceTests, the RunAcceptanceTests will automatically be triggered by the Release target. I call that pretty sweet.

Combining all of this with the excellent value-binding mechanism and all the other benefits provided by NUKE has made me switch to NUKE for good.