19
Thoughts for a Major React Refactor
Hurray, I have a holiday! Hurray, I have time to write! Hurray, why the **** I spend my holiday thinking about work related things!?
One of the bigger challenges in the world of programming is choosing the right tool for the job. And it can go wrong. So here is some background and thoughts on how SSG (server side generated) and TypeScript can go a bit wrong and make things harder to deal with, while contributing some thoughts on how things could be improved.
I can't go to too much (business) detail on what I'm talking about, but in short we're dealing with a site that has two main usages: a checkout for selling products, and content side with the focus on maintainable content via Contentful. The technology stack was chosen to be based on React, with Gatsby being chosen as the framework.
So far things sound pretty sensible, since Gatsby is built for serving static content. The way it works is that it gets all the content and then generates all the pages which you can then just serve.
The checkout side of things was integrated straight in with the rest of the application. This is also fine, it is nice to have things shared and thus a bit less repeated code as you can re-use some components and overall design.
Here we get to the uglier side. The checkout side is a complex beast and depends on a Redux store. And one thing about Redux is that it is hard to do code splitting with it. You can do it, but it is hard and would need a lot of thought and time spent into it.
The end result is that all the checkout logic is always shipped on every page load. This means lots of JavaScript that is much meaningless for most of the content pages since there are only a few specific locations where you can enter the checkout from the content side of things.
So that's one clear thing to fix: separate the checkout from the content and be happy. But it isn't the only problem!
Content side has also grown substantially so that generating the content pages takes a lot of time. This also makes it slow to boot the app for development.
One of the things that isn't really helping is that the front-end application has also been written with TypeScript which has it's own quirks mixed in. Since it isn't ideal to just type everything as any all the content stuff needs to have their types generated. Due to reasons this means megabytes worth of types.
Things even used to be worse, the project went over 100 MB worth of content types which meant my 16 GB RAM work laptop could take as long as 1½ hours to run Jest test suite. And before you say it, "just add more RAM" is not a solution. After one major refactor the types became saner and now the tests always run in less than 1½ minutes.
Another TypeScript issue is that the discipline for types has not been maintained throughout the project's lifespan. One reason for this is just the number of people that have been working on the project. For example when I joined in there were roughly 10 developers working on the project, and many more had had their hands on the thing.
The lack of discipline on typing means that types generated from our BFF (Backend For Frontend) are not always correct. Some shortcuts have been taken, and some incorrect typings have been done just to make some random tool happy. A simple example: a value in object is always a number, but types tell it is a string | undefined, so you need to write extra code to make the number a number. My current goto for known integers is ~~(value || 0), but really, the types need to be fixed.
The main cause for all the issues is lack of maintenance work. When I joined the team I immediately started updating npm dependencies, because I know keeping the dependencies outdated only means even worse issues further down the road.
However there is still a constant push for more features to be added as, despite the checkout being a complex thing already, even more specialized products and use-cases need to be there. These are existing ones that are being supported by an older app, but the desire to get rid of the old app is very high.
And then we get to this state where old developers leave the project to work on something else, and at the same time hiring more developers becomes harder. We're now essentially down to architect/lead, two senior full-stack, senior front-end, and a junior front-end.
This means we just don't have much of a luxury for maintenance at the moment. Throughout spring npm packages have mostly not been updated.
There is no way around it: for us to keep going fast with features we must slow down to do maintenance and refactoring. Otherwise we keep hitting our head to a wall where not enough gets done, because we need to workaround issues over other workarounds.
There are many ways to go with the refactors.
We could spend time to just fixing the most obvious issues: locate and fix the types, cleanup the existing app structure, do some big brain thinking and work on how to split the checkout better. Essentially stuff that keeps the existing application structure intact, but just make it more solid and robust.
There is another path to take though. One possibly less obvious would be to quit serving the existing JavaScript bundle entirely on content pages, and instead write some separate front-end only code to deal with functionality on the static content pages. This would greatly speed up serving of the content pages since you would ensure none of the checkout logic gets loaded on pages where it is irrelevant.
However there is a thing I don't yet know: can this be achieved with Gatsby? In case after investigation it seems like a thing that would take a lot of time there might be a bigger refactor to be done: to abandon Gatsby entirely, and consider switching away from SSG to something like Remix.
In my previous job I actually wrote a universal JavaScript app that is (or was) very much like Remix, just not as obvious and of course more of a DIY. It did many Remix-like things like ensured serving correct status codes, embraced correct cache headers, and used fetch on both server and client side code. The project started way before there was anything like Gatsby, Next.js, or Remix available (Remix is still in beta).
With the experience I know the Remix way would be great. However it would be a major change as instead of a pure static generated site we would run a front-end app with some static assets.
However we would also get some major benefits! For the current SSG means that each time we want to publish new content we also need to re-generate all the static content. You could make it only generate what is changed, but that also would mean some extra work on that front, especially ensuring that it won't break over time. So with an app that is actually running all the time we would simply always get the latest content and serve that. HTTP caching would ensure we won't overload the app.
The switch to Remix or Remix-like pattern wouldn't be a magical solution to everything, but it could mean we would get rid of some issues, or have a much better opportunity to get rid of them. Such as some generated types for content.
As usual this is a piece written pretty much as a brain dump. Sorry about that! I hope you do find the contents interesting regardless of me being a bit lazy with the work.
I have other major changes in thinking, such as abandoning Styled Components / CSS-in-JS and instead embrace CSS Modules or some other technique that makes CSS stand as a solution instead of relying on JS execution to do styles. Maybe a topic for another day.
As a conclusion the app itself isn't really that bad despite all the problems. It works pretty great and customers like it. Most of the focus here is on how to make further development great again so that the team could keep going fast with features without being slowed down by spaghetti.
19