15
Build Monorepos, not Monoliths
This article was written as part of the C# Advent 2021 event. Find more articles and information here: https://csadvent.christmas/
C# has a long history with monoliths. They have their own MSDN page, which describes them as below.
A monolithic app has all or most of its functionality within a single process or container and it's componentized in internal layers or libraries
On MSDN, some downsides are immediately noted.
- Scaling the individual components is either hard, or impossible in some cases. As an example, you may be able to scale up the database as a whole, but you would not be able to easily bump up the authentication layer of your application.
- Changing a single layer of the application requires the whole app to be rebuilt and tested.
- The whole app must be deployed at the same time.
In response to these issues, many developers and architects have been building microservices. In a microservices architecture, each layer or vertical slice of the application is written and deployed as its own executable or container. Each service communicates through protocols like gRPC or HTTP, rather than directly calling functions.
This eliminates many of the scaling concerns, as each individual host for the services can be scaled independently. Testing can also be slimmed down. As long as the public API for a service is not changed, or the changes are backwards compatible, only that service must be tested.
However, some benefits from monoliths are lost. Sharing code between microservices is much harder. If you have a common "microservice-template" repository, updating it becomes a chore. Existing microservices may not get the benefits from the update, or you may have to manually copy and paste the functionality to each microservice. You may decide to publish a company internal package to nuget, or NPM to alleviate some of that pain, but now testing your changes across the services can result in going back and forth between solutions to check if your changes work.
Additionally, layers of complexity quickly stack up and progress can slow down to a halt if the architecture begins to slip.
Some big cons from microservices that are split across multiple repositories include:
- Implementing one feature may have a developer submitting pull requests to several repositories. These PR's may require each other for integration tests to pass.
- Deployment and CI/CD work is much more complicated, as integration tests should test the whole system but are harder to orchestrate
The front-end community has began to adopt the concept of monorepos, and many top companies have been using them for years.
In its most basic form, a C# monorepo may be a git repository that contains all of the microservices inside a solution. Even at this basic level (which I refer to as colocation), you start getting some benefits.
- A single pull request represents the entire change set for a feature.
- System level integration tests and CI checks can be implemented much easier, since you are only dealing with one source repository.
Introducing something like @nrwl_io's Nx build tooling allows for many more benefits to come through, with advanced task orchestration to maximize developer productivity and minimize CI times.
There are some similar tools available, but I'm most familiar with Nx so that is the example I will use here. For a comparison with other tools, see monorepo.tools.
After adopting Nx, and using its commands for your builds you get several things, but we will focus on a few of them.
- Local and Remote caching for build / test targets
- Project Graph Analysis
- Distributed Task Execution
When using Nx, commands such as build and test have their outputs (both on disk and terminal) cached during execution. What this means, is that when running the same command with the same source files you don't waste computation or developer time.
With the help of Nx Cloud, this time saved gets extended on your CI/CD agents as well. Outputs get cached to the cloud servers, and if an agent runs a command which has already been ran it retrieves it from the cache instead of running the command.
This remote cache can also help developers, as when checking out a PR branch locally the builds will pull from the remote cache that was created during PR checks or on the other developers machine.
With the help of nx-dotnet, Nx is able to understand how the projects in your repository depend on each other. Nx is then able to look at what has changed on your branch and only build and retest what has been affected by your changes. When using microservices you may not have ProjectReferences in your .csproj files. Nx makes it easy to add an "implicit dependency" between projects to inform the tooling that there is a hidden link there.
Taking advantage of nx affected in CI / CD means that you are not wasting cloud agent time to retest things that haven't changed, but you are not missing tests that may fail in a dependent project.
Nx can intelligently run targets across a network of CI agents, such that dependent jobs are ran with the inputs they need as soon as they are ready. This can speed up CI/CD times by a large factor, as jobs are parallelized and computational time is not wasted redoing jobs that have already been completed.
There is an existing article describing how this process works from @nrwl_io.
15