Exploring the Monorepo #1: Can't we just make project-folders?

The Good
The Bad
- Why don't we script that?
- What about Yarn?

Let's get the simple solution out of the way first: Can't we just move everything into different projects within the same repository?

To test that out let's extract web and api into two separate apps, and make a libs folder for the shared dependencies. By moving the files around we end up with this structure:

webby
├── apps
│  ├── api/
│  └── web/
├── libs
│  ├── analytics/
│  ├── logging/
│  └── types/
└── tsconfig-base.json

And if we look at web's package.json we see a small list of dependencies that are used entirely by web:

"express": "^4.17.1",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-router-dom": "^5.2.0",
    "types": "file:../../libs/types"

ℹ️ As before I've prepared this project on GitHub1s if you'd like to familiarize yourself with the code. But note that now each app and lib are their own projects: You can open them up individually to avoid "drowning" in all the other code (but to do that you'll have to clone the repository and check out branch attempt-can't-we-just because GitHub1s doesn't support opening subfolders).

The Good

The overview has improved greatly! The high-level architecture is now easily readable: We have two apps, and some libraries, so as a new hire I can quickly get a feel for what large-scale projects we work on.

And if we dive into web we see its package.json references the local dependency ../../libs/types, which makes it simple to understand at-a-glance that if I work on web I only need to understand libs/types code to get my work done. How amazing!

It is worth acknowledging though that, yes, there are now more files. Where analytics was one file before, it is now a whole project which means it has its own package.json, tsconfig.json, + other scaffolding files. This looks quite bad with our example because our libraries are so anemic, but keep in mind we're pretending our extracted projects are those we agree are complex enough to warrant extraction. If each project actually had dozens of files and a non-trivial amount of dependencies then the high-level clarity would outweigh the added number of files. But it is a tradeoff: Clarity in the overview causes more project-bootstrapping files to appear. It's simpler, not necessarily easier, and only you can decide on your own balance.

The Bad

Unfortunately this solution doesn't work 😱😅. It's a frustrating conclusion because it can appear to work, but actually it breaks in various subtle ways

ℹ️ BTW, throughout this if you'd like to get back to a clean checkout state you can run this command: git clean -dxi .

If we begin from a clean checkout and start the web app we immediately hit an error:

$ cd apps/web
$ npm ci
$ npm start
../../libs/types/src/index.ts(1,23): error TS2307: Cannot find module 'type-fest' or its corresponding type declarations.

What happened? This has to do with how npm installs local dependencies:

When we run npm ci (or npm install, it's the same problem either way) local dependencies are handled in a special way: A local dependency is symlinked into the node_modules folder. In this case web depends on libs/types and we can see how it's just a symlink by looking in web's node_modules folder:

$ ls -a node_modules | grep types
types -> ../../../libs/types

But it is just a symlink, npm didn't install the dependencies of libs/types for us like it does for normal dependencies, and so we get the Cannot find module 'type-fest' error because the dependency tree of libs/types hasn't been resolved.

Does that mean if we manually install dependencies for libs/types then web will start working?

$ cd ../../libs/types/
$ npm ci
$ cd ../../apps/web
$ npm start
> Started on port 3000

Voila! But hang on, this is a brittle and time-wasting workflow because we have to manually install each of our own dependencies… that's what npm is supposed to do for us!

Why don't we script that?

Maybe we can script our way out of this? Here's a quick way to install all dependencies at once:

ℹ️ BTW I'm on macOS so I'll use its built-in tooling just to keep things simple, but of course a more mature, cross-platform solution can be made if needed.

$ cd ../..
$ for p in ./*/*; do; (cd "${p}" && npm ci > /dev/null && echo "Installed ${p}"); done
Installed ./apps/api
Installed ./apps/web
Installed ./libs/analytics
Installed ./libs/logging
Installed ./libs/types

And now everything works, right?

Mm, not quite, web does work but api doesn't:

$ cd apps/api
$ npm start
../../libs/analytics/src/index.ts(8,3): error TS2564: Property 'uninitializedProperty' has no initializer and is not definitely assigned in the constructor.

Oh boy… what's wrong now?

Well, this is a case that I've purposefully put in to mimic a real-world scenario I've seen: libs/analytics is not valid strict Typescript, it only works with the Typescript setting strict:false. As its own project that's fine, which can be demonstrated by running libs/analytics's test-suite:

$ cd ../../libs/analytics
$ npm test
Ran all test suites.

And if we look at its tsconfig.json file we see it correctly specifies the strict:false option:

$ cat tsconfig.json
  "compilerOptions": {
    "strict": false
  },

But apps/api does work with strict, and so specifies strict:true for itself, but when it runs it pulls in the analytics code via api's TypeScript configuration… How annoying.

I'm not sure how to fix this. Is this what Typescript references are meant for? Do I need to build each sub-project and only use the build-output? Please comment with your ideas and suggestions!

What about Yarn?

Maybe it's just npm that's the problem? Let's give Yarn a try.

First let's reset the repo and install yarn:

$ cd ../..
$ git clean -dxi .
$ npm install --global yarn

And now we can start web:

$ cd apps/web
$ yarn install
$ yarn start
> Started on port 3000

Hey that worked! Yarn actually fully installs local dependencies, including resolving their transient dependencies. So it avoids the "type-test" error 🎉

But this has a problem too: The dependency isn't "live", meaning changes to libs/types aren't reflected in apps/web until it re-installs its dependencies. That's not a good workflow!, we want to just change code and have it all working together, not worry about what state each project's node_modules folders is in.

And besides, apps/api has a problem too:

$ cd ../api
$ yarn install
$ yarn start
SyntaxError: Cannot use import statement outside a module

Is there a solution for this? Some Yarn or Typescript setting to use that will make it all work?

It feels a lot like we're chasing down problems that I've created for myself. We can't be the first ones to tackle this problem, right? I hope I've just missed a chunk of documentation that will set us right, if you got any suggestions I'm all ears!