29
How to deal with complex projects
This article was originally published on my blog, where I write about being a technical advisor to early- and mid-stage startups.
As a founder, one of the underrated skills is to know how to deal with complex projects and build complicated stuff. Whenever you start a project, you usually have a good idea of how difficult it will get. If there are challenges to be solved, you can see possible places to look for hidden complexity – this comes from experience.
Over the last ten years, I’ve built projects of different complexity – mobile banking apps, finance tools, ordering apps, running apps, you name it. But even with such experience, some projects still fail. I would never agree to handle a big project alone. It’s always the effort of many people that keep the project from falling apart – it’s not a one-person show.
We know why projects fail – there are multitudes of books/articles/studies on why software projects fail. We know how to prevent their failure – so why do they still fail?
I don’t know why projects fail in general. That’s the simple answer. Every startup is building its megaproject, and it is unique and has its flaws, and you learn how to adapt and make sure that it’s done on time and in the correct way.
I’ve split this article into several sections:
- As an early-stage founder, what necessary soft skills do you need to cultivate.
- Things that I do day-to-day to move a product forward.
- Most common issues that you will face.
- General tips for indie hackers or early-stage startups on managing your product development.
The easiest way to build a complex project is to reduce complexity. That’s even more true if the timeline is tight.
Before we dive deep into the nits and bits, I’d like to clarify what exactly I consider as a complex project – these are projects that require cross-product integrations with aggressive deadlines and multiple Agents involved. They don’t necessarily need to have huge codebases, as that’s not what makes it complex – it’s more about everything around the project that makes it difficult – difficult people, complex processes, consultants all around. An early-stage startup that is solving a single problem can be considered a complex project.
As many will tell you – building software products is less about coding and more about everything that revolves around it. These three character traits can have a profound effect on success.
Try to be as flexible as possible – get ready for unexpected things to happen. When planning a timeline – assume people will get sick or leave the company, consider that the freelancers will not deliver on time.
There are multitudes of factors that are out of your control when you’re building a product:
- People get sick
- People leave you
- Freelancers overpromise and underdeliver
- Contractors fail to meet the deadlines or goes entirely out of business.
- The complexity of the project increases with new requirements.
- Of course, you can say – but I have it in the contract that these things shouldn’t happen. A contract can help you when you sue them six months later, but here and now, it has zero influence.
These are all improbable events, but I hadn’t had a single project where at least one of the above things didn’t happen. So a good rule of thumb – trust Murphy’s Law – “Anything that can go wrong will go wrong.” Managing a project is the same as managing life – you can’t have everything under control – mostly, you ride that chaos wave and hope it’s going in the right direction.
Failure is the rule, not the exception, in large-scale IT projects. I’ve read a study some time ago that says that 45% of the projects run over budget, 7% of the projects run over time, and 56% of the projects deliver less value than planned.
A more “developer-friendly” example: Your team wants to build a payment integration. Sounds straightforward as you were planning on using Stripe, which has excellent documentation. But then, during the development, you find out that Stripe doesn’t allow selling weed-based products through its platform. So you search for a payment provider that does allow it. But this provider doesn’t have a REST API – it only has SOAP. The whole processing of payments is different, leading to additional handling of currencies and orders, resulting in more chaos in the project.
When something unexpected happens, it’s crucial to make decisions and to make them quickly. You’re accountable for all the choices that you make and also those that you do not. Being indecisive is a one-way ticket to a failed project.
On the one side, you need to gather as much data as possible to make an informed choice. On the other side – you don’t have the luxury of spending too much time on it. Any decision is better than no decision at all.
My thought process during some unexpected crisis is as follow:
- You get hit by something unexpected. It happens, no way around it—time to squeeze the lemons that life threw at you.
- Don’t look for excuses or someone to blame, no time for reflection on WHY it went wrong either – you can do this after you have the situation under control.
- Start asking yourself the essential questions – How does it affect our endgame? What are my next steps? Who can help me decide what my alternatives are?
- Start planning the measures that can solve the issue. As I said – make a plan A and then a few more just in case.
- Make it a priority and discuss it with the key people involved.
- After solving the problem – start documenting why it happened and what you can do not to allow this to happen in the future.
- Time to have some rest.
So as you see from the two points above – communication is one of those skills that you’re going to have to master to be a good manager or just, in general, to lead product development.
You’re going to have many discussions of the underlying problems without pointing fingers and blaming someone. It’s a skill to keep the conversation civil and focused on the task. Too many times have I witnessed the passive-aggressive pushing of blame to avoid responsibility instead of accepting the failure and moving forward.
One thing to mention, from experience, when dealing with big corporations – remember that they will be playing politics to get maximum value from collaboration with you. There’s no way around it. Just try to learn enough of it to navigate it, but don’t get sucked into it.
Any big project always requires a lot of work. Here are some things that can make your life easier when dealing with the chaos that complexity brings.
Before you start setting up the team, the responsibilities and tasks define the business problem you’re trying to solve and outline the value in solving it. It doesn’t have to be too detailed, just the high-level stuff. Talk about the vision of the end product – discuss it with your stakeholders, make sure you don’t forget anything. When making a mobile application – build a persona, do scoping canvas, talk about features, prioritize them, think about user flows. Everything you do know will make the work later easier.
“If you don’t know where you are going. How can you expect to get there?”
A good practice is to conduct interviews with your audience as soon as you defined some form of the product. Many tools on the web allow you to select people based on criteria that suit your product persona. Talk to your target audience, whether internal or external, so you fully understand the problem space and gather information on it. Find out edge-cases, things that people expect, ways to build them.
It’s good practice to start from the release date and go backward. Discuss the interdependencies of the sub-projects. Does something need to be finished before the next phase starts?
For example:
- the content needs to be completed before the development starts.
- The designers need to finish the UX/UI before development kicks off.
- The UX/UI needs to be greenlighted by the stakeholders.
- The functionality needs to be signed off before the UX starts etc.
Usually, there are quite a few dependencies, and not everything can be done in parallel, even in big teams.
You always need your best people working on complex projects. Make sure a suitable team member covers every part of the project. Empower them to be the decision-makers and take responsibility for their tiny island. There’s no need to micro-manage every aspect, as long as you have the big picture view (and of course, you need to trust your colleagues).
Make sure everyone knows who’s responsible for what, who can make what decisions. Here’s it’s also vital to define communication channels and intervals. For example, you say that you will have a meeting every week on Thursday to go through the latest activities from different departments and everything that went wrong. People need to be aware of the problems- to deal with the unexpected as soon as possible.
Start iterating. If you’re building something in an agile way, then the product is never finished; you finish one of the iterations. Get ready to build and rebuild based on feedback.
No amount of testing can prove a software right, yet a single test can prove a software wrong.
All the assumptions that you made, the rules that you tried to skip to deliver faster – all of that will surface during the alpha-release.
To make this period less painful, you need to schedule some time before the release for beta testing, a week, maybe even a month, depending on the release cycle and the size of the Software. You will still get a considerable bug list regardless of how well you test. Testing is not meaningless, though – the more you test, the more bugs you’ll see and the more bugs you can fix.
I can summarize the possible issues that you will face into three categories:
- Business issues – This includes the time-to-market and limited resources.
- Management issues – This includes things like lack of oversight and over-optimism.
- Product Issues – Bad technology choices and flawed estimations.
The business side always wants to have the highest quality in the shortest amount of time. As a person from the technical side, you always need to push back on that, as you know – the faster you go, the worse it gets. Shortcuts in implementation, skipping testing, forgetting documentation – the usual consequences of tight deadlines.
You either cut the scope and keep the deadline or keep the scope and push the deadline.
There are always limited resources available. Of course, we all want to work on multi-billion projects with unlimited budgets. Sadly, that’s not how it works, as the budget is usually the one thing that investors tightly control. Everything money-related will need to be justified, and the decisions will be under scrutiny.
If there are consultants on the project – they will try to minimize the costs to show that they are squeezing the maximum efficiency out of the teams.
Even though the resources you will have available are limited – the business will think that anything is possible. That’s a common misconception in IT. Example from High-Frequency Trading – if there’s a requirement to keep the latency below 20ms, but you are sitting in Europe, and the Server is in China, things get tricky. Push back on things that don’t make sense from the technical perspective.
Delegation is cool. Letting people run wild isn’t. If you give people the chance to slack off, they will take that chance. This doesn’t mean you should micromanage everyone, but you should instead define clear goals and make sure that people get there without causing too much trouble.
For example, suppose one of the stakeholders takes too much time with some critical decision blocking your development teams. In that case, it’s essential to step in and remind them that there are deadlines and the project needs to be moving forward.
Speaking of deadlines – be realistic or even pessimistic when planning. The problem with being optimistic is that if you think that each task will be finished at exactly the time allotted, then a single hiccup somewhere in the chain will result in everyone missing deadlines. Each mistake accumulates as an avalanche effect:
- Someone forgot to send an email – people don’t know what to do.
- A decision is unmade.
- A manager is afraid of taking responsibility,
- and bam, you’re behind your deadline by a week.
The complex projects are usually multi-year contracts. This means people tend to relax during the first few months. The closer it gets to release, the more they start running around like bees. So my rule of thumb is – act as if the release is next month and solve problems with that state of mind.
One of the most common product issues is that no one knows what the project’s vision is. Or worse – some stakeholders have conflicting interests – both want the product to succeed but in different ways. This is when politics start. Everyone will be pushing their agenda. Again, when it comes to politics, I try to stay away from it.
Nobody would force a builder to build the basement after having finished the roof, but in the software industry, that is standard practice.
The second most common thing is that a change request later in the project requires rewriting the core functionality. Of course, with agile development, you rarely have the full specification of the product when you start developing. Otherwise, it would be waterfall development and not Agile.
Other than that – unless it’s an entirely new project with no dependencies, you will have to work with some legacy system that lay dormant for ages. Sometimes you find relevant parts with zero documentation, and no one in the company knows how it works. It’s better to evaluate the legacy system before the development starts and calculate the time to build/migrate/extend the legacy code.
I’ve gathered a few valuable tidbits of advice over the years. I think you will find them helpful also:
It’s crucial to have everything in writing – all the information regarding who does what, who’s responsible for what, who built what, how the Software works, how it should work, or why it doesn’t. As long as you have it on paper, you can always come back and reflect on what you did and why.
If you’ve successfully finished some part of your Software – review how the assumptions you made earlier changed. Make sure no new issues have arisen going forward from this milestone. Talk to your team. With the new knowledge that they have – do they see any problems now that some part of the project is complete?
Set up regular reports with the key stakeholders so that you can tackle issues and manage expectations proactively. These could be written or through any channel that you defined.
You don’t know from the start in which direction the project will evolve. So it’s impossible to prepare the “core” for all possible deviations from the initial requirement. Prepare to be flexible, but don’t start over-optimizing too early.
A megaproject is a combination of predictable, standardized, and repetitive tasks performed many times before and novel and innovative procedures being applied for the first time.
You may be building something you’ve never built before, so it’s essential to do research and talk with someone more knowledgeable. When performing risk assessment, where are the pitfalls/shortcomings? Find out all the ins and outs of the technology that you’re using. How were similar projects built? Are there any case studies? This is especially true for software engineers that need to get their estimates right – the more you know, the more accurately you will know what to expect.
Multiple agents (subcontractors) always increase the complexity of the project. The best-case scenario: your company can do the whole project alone, which sadly rarely ever happens on megaprojects.
If you can’t reduce the number of agents involved – strictly control all the communication. Otherwise, it will get out of hand quickly. Clear structures help everyone know where they sit in the hierarchy and who they need to talk to to get things done, e.g., change approvals, resources release.
The main difference is the number of changes during the execution of the project. For example, if you’re building an airplane, nobody will tell you that they want it to go to space in the middle of the project. But in the software industry, this is possible. This goes hand-in-hand with the logic of business that anything is possible in IT. If you built something that flies, then it should fly anywhere we want it to.
The task of estimation is a crucial one, so take your time doing it. This is where good technical spec can help you a lot.
First of all, never estimate a software project by yourself. It’s good practice to have at least several teams assess the project and then take the average number of hours for reference.
Second, split the project into modules and modules into tasks, estimate the tasks, and sum the person-hours needed for the functions and modules. The rule of thumb is people only count the raw development time, forgetting all the small things that add up in between, so add another 20-30% for unexpected bugs. Communication will also take a few hours per week. Then there’s the Scalability requirements, Security requirements, Performance, etc.
This is a tricky one. If a developer estimates the task will take 30 hours, can you trust him to deliver finished work in time and up to the standards? To be honest, you shouldn’t. If a developer says he will do this task in X amount of hours, you plan it as X + 20% hours internally.
The reason is that the developer may not see the whole complexity from the start and will only notice how hard it is after starting. I can not remember how often I created a task that seemed easy initially but was quite complex after the second glance. Consider a job to build a currentTime library that returns the current time. It seems easy enough, but then time zones come into play, then suddenly some countries decide they’re not doing Daylight Savings anymore, and the spiral continues.
As your startup grows, the project becomes less chaotic and more defined. Processes get added, procedures get documented – you have either adapted to the chaos, built that first version of the product, or you failed. Still, even then, you have gained valuable knowledge that will allow you to build your next endeavor better and more efficiently.
In conclusion, I want to say – never say NO to building something extraordinary. It’s always a win-win situation. If you fail – you learn. If you succeed – even better.
Follow me on Twitter as I build my consulting business in public, or let me know if you need a technical advisor – I might be able to help you out.
29