Pulumi Multiple projects with Custom Backends

I’ve been working with Pulumi to create a reference architecture for a client that provides individual team autonomy while providing some shared resources. In addition, the client is also wanting to utilise a custom backend in Azure Blob storage. This presented some issues as the documentation around projects, stacks and stack references for custom backends. Hopefully this post should clear up some of those patterns.

What is a StackReference?

A stack reference in pulumi is a way of your current project accessing the outputs of another project. This is really useful when you’re sharing things like a Load Balancer, API Gateway, AppService Plan, etc, between multiple services.

In Terraform this would be a “Remote State” that you’d bring in as a datasource.

More information on Stack References: https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences

Why Multiple Projects?

When you’re working with multiple teams of people, you will inevitably hit a situation where you’ll want to have multiple, independently versioned and iterated infrastructure projects. Some of the time, there will be outputs from these projects that other projects depend on.

What is a Custom Backend?

Pulumi offers an awesome service for managing the state at https://app.pulumi.com. This provides an intuitive UI, along with some cool features like Deployment Histories, and integration into CI/CD pipelines etc. This is my go to, and if you can justify the cost, it’s something I would highly recommend. However, this isn’t mandatory as you can also host the backend yourself as you would with terraform.

You can choose to store that recorded state, however, in your own storage engine like S3 or Azure Blob Storage. Full information of the storage options are here:https://www.pulumi.com/docs/intro/concepts/state/#logging-into-a-self-managed-backend

Projects and Custom Backends

When using app.pulumi.com for hosting you backend, the providers automatically take into account your project name, as well as the stack name. Therefore if you run this:

pulumi stack init dev

In a project called demoyou’ll have stack calledmy-awesome-org/demo/dev. However, when it comes to using a custom backend, the default syntax doesn’t provide this separation.

When using Azure Blob Storage, that syntax will result in a stack simply called dev, therefore you will need to namespace the stack manually.

pulumi stack init demo.dev

You’ll note that I’ve not added slashes, and instead used periods. This is because the CLI doesn’t allow slashes for stack names. I’ve also not added the organisation name as that’s specific to the pulumi service.

Referencing using a StackReference

Now we have 2 stacks, we’ll need to reference the first in the second. We can do this in C# using the following code, referencing the now qualifed stack reference.

var stackReference = new StackReference("shared", new StackReferenceArgs { Name = "demo.dev" });

Limitations

This doesn’t provide the ability to reference a stack in a separate blob storage container, this isn’t something that’s supported right now as far as I can tell.