24
Reduce WIP by practicing trunk-based development, rather than pull requests
Poorly managed work in progress (WIP) is a common cause for low performance in development teams. When you have a team juggling too much work, it engages in frequent, expensive context switching, which reduces quality and slows delivery. Your team is working hard, but not delivering.
The reduction in quality has a compounding effect on WIP as you will get unexpected bugs, re-work, and outages; which in itself causes more WIP. Unplanned work is poison for productivity and predictability.
Managing WIP involves looking at how work flows through your system, trying to make it go quickly, smoothly and predictably. Identifying bottlenecks & constraints allows you to manage them more effectively and focus your efforts on improving them and making sure only the highest priority work goes to them. Any efforts to improve efficiency outside these constraints is waste because it’s the bottlenecks in your system that dictate pace. With a focus on quality you can reduce the amount of unplanned work which helps you manage WIP.
There is no one true way of working, the best method depends on your team and the context they’re in. Managing WIP, and having a smooth flow of work, is not a negotiation if you are serious about managing an effective team.
Pretend you are the lead of a 4-person development team. How your team approaches creating value obviously has a big impact on its ability to deliver it, yet many teams copy other team’s rigid models, rather than thinking about the context they are in and tailoring the process to it.
This is an often-cited, popular way for teams to work.
- Each developer picks up a ticket/story/task
- Developer writes code on a branch. This allows them to work isolated, focused, and not affect other team members with their work which may not be completed.
- When the developer believes the work is finished, they raise a pull request. When another developer is free, they can take a look at the work and then initiate a feedback cycle with the person who raised the PR until it is of sufficient quality.
- The code gets merged and deployed to customers
Often the reasons people adopt this model is it allows a degree of gatekeeping and socialising knowledge. It’s argued that it can help prevent mistakes from being shipped and maintain code quality.
For a team of 4 developers, having at least 4 tickets in play at a time feels like high WIP. Chasing 100% “resource” allocation is not a desirable management goal because all systems need a level of slack to cope with unexpected work. If a team is running at full-capacity it is inflexible, and inflexible things tend to break under pressure.
If you raise a PR and none of your colleagues are free, you may be tempted to pick up another ticket. The WIP of the team has now increased to 5. You can easily imagine, and have probably experienced, scenarios where this is a slippery slope to climbing WIP.
How many times have you heard:
It’s done, it just needs to be reviewed
In software development, the word “just” is often doing a lot of heavy lifting. The work is not “done” until it is actually delivering value.
The feedback cycle in the PR can take some time with asynchronous back-and-forth, so you may end up having to switch context between multiple tickets at a time. In lean terms, the process has caused a bottleneck.
Even with a team of only 4 you can easily end up in a situation where they’re context switching between the story they’re assigned to, but also reviewing lots of other developers of code. The personal WIP for each team member can climb steeply depending on how back-and-forth the review process is. As the WIP increases for each person, the ability to get code reviewed diminishes.
PRs are remote-friendly due to their asynchronous nature. This is true, but you have to accept then you will have a lot of unfinished work in play, which will raise your team’s WIP.
If I were to be managing a team working like this, I think it would be prudent to ensure that you only have 3 developers working on stories at a time, giving your system some slack. The other developer can help reviews happen quicker, find ways to improve flow, increase quality in other ways (improve tests, logging, metrics, e.t.c.), and you’ll have more slack to deal with unplanned work.
The longer your code lives on a branch, the less integrated you are with the rest of your team. Whilst the 4 of you are described as a team, you are all working on different versions of the code, as individuals.
If you find it hard to get your colleagues to review your changes, this slow integration compounds, and teams that adopt this system often have to deal with challenging merge conflicts. In the previous example this way of working can lead to multiple PRs that need merging which can result in multiple merge conflicts having to be resolved. This is waste.
The review back-and-forth can last minutes, hours, days, or even weeks. In the worst cases, it involves a lot of re-work. Once the developer falls in to the trap of picking up another story whilst their other is in review, WIP can quickly spiral out of control.
Plenty of successful teams ship software like this, it’s not unworkable, but I’m not convinced it’s simple or efficient. I think even when it is managed extremely well, it still causes a lot of waste.
What is the most obvious way to reduce WIP?
- Do less work concurrently
- Concentrate on finishing work
- Increase quality to reduce unexpected re-work
So instead of having 4 developers work on 4 tickets, let's instead have 2 developers pairing per ticket.
By having 2 heads work on a problem, it liberates us from having to gate-keep the main branch. We should trust that 2 developers on our team can take ownership of a problem from idea to production. Therefore, they can commit their code straight to main. This simplifies the flow of work. We no-longer have a wasteful bottleneck to shipping value.
How many then? 3, 4? Or worse, can it only be put into the main branch if it is blessed by the lead? If you’re a lead making yourself a bottleneck, you’re not doing a good job. You’re not spreading your knowledge, you’re not coaching well enough, you’re increasing bus factor, and you’re hurting the productivity of your team.
By integrating frequently through the day, the chance of merge conflict is low and even when it happens it is typically trivial to fix. Far easier than having to merge a week’s worth of changes back into main.
This is true continuous integration, the team can consistently see what everyone is working on, which helps collaboration. No longer will you hear excuses like:
I’d like to refactor the blah blah, but I can’t because I know Alice is working on a PR related to it.
By removing the barriers to changing code and having a tight integration with your team (rather than disparate branches) you can now fearlessly refactor.
It’s an assumption, but not an unfair one, that the quality of code should be higher if 2 people are working on a change. High internal-quality makes changing our system simpler and should reduce the chance of unexpected re-work.
If you practice continuous deployment, it means you’ll release smaller changes frequently throughout the day. This reduces risk and makes recovery easier compared to diagnosing a pull request which may have been worked on for a number of days (or weeks!) rather than hours.
The idea of code-review and socialising ideas is now constant and easily available. Developers don’t have to write large amounts of code and then inconvenience another developer, forcing them to context switch to give feedback on your ideas. Instead, you have a colleague that you can easily bounce ideas with.
To make sure the socialising of code is spread amongst all developers, we ensure that we rotate the pairs frequently. We’ll also do a “code review” session every week for a couple of hours when we can talk about our code in broader strokes, in a more holistic way; rather than focusing on an isolated change in a PR.
If the idea of pushing directly to main is uncomfortable for you, it’s likely you’re working on an unsafe system. For TBD to work, you need to engineer your system so that it is safe to work on. You’ll need things like:
- Good alerting
- Easy or automatic rollbacks
- An excellent test-suite
- Observability
- “Test in prod”
— Charity Majors
Gatekeeping around PRs and releasing is often reliability-theatre. Nothing can save you from mistakes, they’re inevitable. What you need to focus on is your mean time to recovery.
To commit to main directly, safely & frequently you need an excellent, fast test-suite and a well configured CI-pipeline. Developers need to be disciplined and only commit code they’re happy to put on the main branch.
Every developer is touching mainline, so all features grow in the mainline… which acts as a communication point. With CI, the mainline must always be healthy, so in theory (and often in practice) you can safely release after any commit.
— Martin Fowler
To live with the constraints of wanting to continuously integrate but not make “bad” commits, you need to find ways to break your work down into small, achievable pieces. This discipline makes work more achievable and will reduce risk. By working like this, I’d expect our 2 pairs of engineers to push several times per day.
If you do decide to try TBD, be prepared for some friction and growing-pains if you have a team unfamiliar working in this way. A few years back, I wrote about how I introduced TBD to a team that was previously familiar with a PR-based approach.It will be a journey of uncovering issues in quality, bottlenecks and collaborating to find ways to improve them.
TBD may expose some uncomfortable shortcomings in your system, and the quality of your code and architecture. I’ve noticed that often a PR flow masks problems like:
- Developers finding it hard to break problems down in to smaller, less risky changes. They have not learned how to work in small batches.
- Low internal quality. If your system suffers from a lot of inappropriate tight coupling, you’ll tend to trip over each other’s toes when working together on the main branch.
- Poor/inadequate tests. Lots of teams use the “CI server” as a crutch for poorly written tests that they run slowly and infrequently. Every so often, the system is designed in such a way that important tests are impossible to run locally.
- If you’re pushing every hour, you need to be able to test locally and be confident your change is shippable and won’t affect your colleagues negatively.
This is a journey that many factories (real factories!) have gone through. To work leanly, with excellent flow, managing WIP requires you to look at how you do work and commit to increase quality from start to finish as a team.
TBD is not niche or unusual. Plenty of big businesses, in well-regulated industries, work with pairs of developers committing many times a day to do the main branch. Chromium releases every day and does not use branches.
Usually, these high-performing teams ship these changes extremely frequently to customers, so they can get real feedback quickly and build superior products.
Whilst pairing may seem like an expensive frivolity, the workflow is simpler, more focused and of higher quality. You’ll find that this approach can out-perform the same team working as individuals on branches. There is plenty of evidence to back up this assertion.
— Daniel Terhorst-North
PRs optimise for individual resource allocation, but introduce bottlenecks in your system which drive up WIP, and make the flow of work less smooth than it could be. High WIP diminishes a team’s productivity and is a management dysfunction, making work harder than it needs to be.
To those unfamiliar with TBD, it may seem risky and even unprofessional, but it forces you to work with more discipline, adopt best practices which will make your system more resilient and the work simpler.
— Continuous Delivery: Jez Humble & David Farley
Many arguments that dismiss TBD as impractical for their team often are centred around fear, and a lack of trust. If you’re a leader of a team, don’t dodge these issues with walls and process. Deal with it.
Pull-requests work excellently in the open-source, where you can't trust external contributors. For your closed-source, day job though?
— 2017 State of DevOps Report
Think about your context, examine how work flows through your system, set a WIP limit and stick to it. Find ways to improve the flow and your team’s performance will improve.
24