47
Build your own turborepo remote cache
Photo by Clark Van Der Beken on Unsplash
I have been looking at turborepo, the new fast build tool for javascript/typescript monorepos.
Read on for a brief overview of the space, some tools that are also vying for your attention, and a short code example for creating your own remote-cache for turborepo.
Turborepo is a high-performance build system for JavaScript and TypeScript code bases. https://turborepo.org/
It's a Golang devtool that uses directed acyclic graph this module actually, to accelerate builds with clever caching strategies, and non sequential dependency management.
This DAG space is hotting up, you might know it from GitLab Aug 2019 release, and its about ready to burst onto the scene in a big way with dagger bringing it to the masses.
Turborepo packages this graph technology up nicely, and succeeds in lowering the barrier to entry, targeting Javascript/Typescript mono-repos and their maintainers. Its been so successful that vercel scooped it up this month.
There are still more mature tools in this space, https://nx.dev/ is 4 years old, so there is definitely stiff competition.
So will turborepo blow up in 2022, absolutely it will, it's a smooth developer experience, with some big name backers.
Remote caching is just taking some folders stored locally in ./node_modules/.cache/turbo; creating a zipped tarball (a group of files collected together as one) of those files, and sending it over the wire with a deterministic cache key.
Turborepo itself warns of some dangers when doing this:
Remote Caching is a ... Make sure you are caching correctly first and double check handling of environment variables. Please also remember Turborepo treats logs as artifacts, so be aware of what you are printing to the console. https://turborepo.org/docs/features/remote-caching
We all can be vulnerable to these kinds of mistakes, like accidentally committing an SSH key or API Key to GitHub etc. The stakes are higher when the tools are less transparent, and turborepo understand this, hence the warning.
Vercel are offering free remote caching at the moment. However, turborepo welcomes us to build our own remote caches, and this can be more secure, as you are controlling the location of your cache, and its encryption.
So a quick remote cache can be built with some serverless technologies but there are other implementations here:
-
https://github.com/Tapico/tapico-turborepo-remote-cache
- Docker hosted, with Google Cloud Storage or AWS S3
-
https://github.com/cometkim/turbocache
- A cloudflare workers implementation, this one is really quite neat, and will be super cost effective
AWS is a big cloud, and since it has a unified auth plane (IAM) it can be a secure place to build your dev-tools. We can then use AWS IAM to enable access to the remote cache(s) locally, or in CI, without having a long lived token. But for the time being that is not an option, so consider a custom authorizer, or a JWT authorizer.
I have used simple Auth using a plaintext header, and for simplicity there is no authorizer being used. The source code is here https://github.com/simonireilly/turbo-remote
- Tools implementing DAG's are becoming more common, and easier to interface with as a developer.
- Turborepo is one of these tools, and it does a good job of lowering the barrier to entry.
47