Spacecrafts With Explosive Engines: How to Survive a Fight With Technical Debt

How do you survive a fight with technical debt? What should you do if you have a severe legacy? In this article, I’ll share three cases with advice on how to organize working processes with technical debt and which engineering approaches to use.

My name is Denis, and I’m the backend team lead at Wrike. I’m responsible for delivery in my team and for developers growth in several teams. I gained almost all of my working experience in financial technology. I also worked for two large banks before working with Wrike. At banks you learn how to work with systems in which reliability and fault tolerance are important. I’ve experienced a large amount of legacy as a developer and used that experience to help others as a team lead.

Wrike is a SaaS solution for cooperative teamwork and developing organizations that we sell to our clients.

Wrike is developed by 30 Scrum teams. Service is available 24/7, so decisions that we make must be reliable. If something goes wrong, it’ll affect the work of almost every team.

Why spacecrafts?

I use spacecrafts as a metaphor to describe what programmers work with. Application starts with several building blocks, and then it grows: It carries a load, microservices, and communication between the application, integrations, and external APIs, and clients are involved. It becomes a large, interconnected ecosystem. The bigger and older an application gets, the more interconnected the ecosystem becomes. And it’s more important to support the relevant nodes in a current, reliable status. It’s crucial for the most important nodes to meet modern requirements and don’t break down.

Your spacecraft won’t warp jump if it operates on 95 octane gasoline or travel through four galaxies if the central computer is Intel Celeron.

What’s a technical debt?

Imagine that you have a feature in which bugs are constantly registered. A tester comes to you and says: “This week we’ve found four new bugs.” Is it a technical debt or not? What if this feature’s existence blocks other stories you need to create? What if the solution is hard to work with and you have to refactor it every time? What if users complain of this feature? What if it doesn’t meet present-day requirements or offends the developers’ feelings?

I’ll explain the features, solutions, or approaches that get in the way of further product development. All those problems are the consequences of technical debt. Here are specific reasons why technical debt exists and what to do with it.

Algorithm of work with technical debt

While organizing effective work with technical debt you have to ask yourself three main questions:

1. Why? Why do we need to do this work? There must be a clear understanding of what benefits we can get from solving a problem. From the technical debt term itself we can understand that it blocks product development. By working with it we can obtain new capabilities and opportunities.
2. What? We must clearly understand where the technical debt starts, where it ends, and the amount of effort to overcome it.
3. How? We need to consider the actions to take to get rid of technical debt.

There’s an easy four-step iteration algorithm to answer the last question. The first step is to select technical debt and understand its boundaries. The next step is to separate it from everything else: Encapsulate it and initiate a work contract between the solution you selected and the other system. Then you can create a new solution next to it, replace it, and with this action remove a part of technical debt from the application. By repeating this iteration several times, you’ll get a complete solution.

Now let’s see how this algorithm works in reality via examples of several cases.

Case #1

There’s an application that handles a client’s orders: the order management system. It was written in 2010 and has been working in production for the past 10 years, but now management understands that there’s a need to occupy new markets and develop the system. But it’s still important to save the data and update new capabilities of the system.

So there are outdated technologies along with the data we can’t lose. Not all possibilities can be developed in an application based on old technology.

To be absolutely honest the problem seems to not be in the old frameworks but in the task itself. The application isn’t supported, and it’s nearly impossible to find developers who can work with 10-year-old frameworks. Something has to be done.

Let’s launch our algorithm. We can select several parts of the technical debt and organize the process iteratively. Firstly, we’ll handle the frontend. We can launch a new frontend using the old backend. We can expand and adapt the new frontend to updated technology so that it meets our goals. We can either fully base it on the old backend or improve it to work with the new frontend. The next step is encapsulation, and the new architecture will help us. A contract with the backend will be an encapsulation point. After launching the new frontend, we can remove its old parts. Now our whole application will become greener and greener.

On the next stage, we’ll work with the backend. Here, the layer of working on the database will become an encapsulation point. So the architecture will again perform encapsulation for us. And we can create a new solution nearby that works with the same data and switch the frontend to it. Now that we’ve completely rejected the old solution, we can throw it away. This allows us to achieve the goal we stated for this project.

Case #2

Let’s have a look at a more difficult case. There’s an application with a specific feature that’s responsible for saving currency pairs into a database (e.g., rouble-dollar, dollar-yen, etc.). The information is kept in a table in the database. To brighten it up, we’ll add a few dependencies: a consumer who receives data directly from the database and a data provider who can also directly supply it to the database.

We don’t like the data layout or how the data is supplied to the database. We need to fix it because there are many dependencies.

To do that we select a specific part: data. We need to encapsulate it with the help of intermediate layers, of which the consumer and provider won’t notice any changes. That’s the point of encapsulation. Now we can alter storage structure because it won’t affect our external dependencies. After that we frame the new solution, which will record data in a new layout. Lastly, we transfer old data to the new layout and get what we want from our project — the data is in a new layout and the old logic can be deleted from the application.

This process ensures that data consumers won’t notice the changes, which means we did everything completely safely and maintained backward compatibility. If it’s necessary for the project, we’ll be able to persuade the consumer and data provider to switch to the new layout too.

Case #3

To exaggerate the scale and understand how it works in global projects, we can imagine that there’s a large project, big code base, and some key capability which spreads on each application point. It’s used by other backend parts and has access to public API so the data is leaking somewhere. The feature is used in frontend and external systems and is also transferred directly from base to analytics. To brighten this case up, we’ll add a touch of legacy here.

2 fun facts about legacy:

1. It definitely works.
2. Nobody knows how it exactly works. That’s why it’s called legacy.

While handling such cases with many entry points and many unknown factors, we need to understand what we’re really working with, what the solution looks like, and what opportunities it has. It’s important to know how the solution we want to redesign or to get rid of interacts with other applications. We need to find each entry point and understand how it operates and its contracts to suggest something else.

Here are some approaches that can help, especially if the damage in amount of code is large enough:

• Mark out the code. In Java we can use annotations to mark out the code that shows how this solution is used. We’re then able to do automatic reports and see what the big picture looks like and what to do with it.
• Draw graphs. We can draw any graphs that show how these solutions operate and interact. Specific graphs that you put together or draw will help you better understand a complicated system. It’ll be useful not only for you but for the whole team.
• Put the project in a task tracker.

If we find entry points in the code we can use them and transform the solution with encapsulation points. We upload new contracts on our feature’s interaction with the application.

At this point there’s a lesser amount of legacy because we already started to add to the application what we want.

Then we need to take the first step to new solutions: Run some tests. This will solve fun fact #2 of legacy I mentioned above. Tests will show exactly how your solution works. Apart from checking key flow we need to make sure that it operates as expected.

Sometimes the solution for some business goals has been used at a 100% level now only meets the goals at a rate of 30%, and 70% stay behind. In today’s context, the 70% isn’t that important. The tests you compose will let you identify the 30%. By using a coverage test, you can understand which part of the code isn’t used at all, delete it, and lower the connectivity and complexity of your solution.

If we understand how a solution works after testing, we can enter it into an encapsulated field and displace everything old and unneeded, as well as remove legacy. We can then switch over to the new solution that meets our requirements.

The new solution should be easy, clear, solve your current problem, and meet subsequent goals. There’s no need to over-engineer since we fulfilled today’s objective.

This is where you have to stop and think: “’Why are we doing it?’’ At this stage you’ll obtain enough information on how the solution operates, what it has, and what it doesn’t have. You composed tests, marked up the code, drew a graph, and gained a greater understanding. This is probably where you should consider if it’s possible to solve a much larger problem.

How to organize work

We practice an approach when we try to divide a new solution into iterations. These are its specific parts that can be put into production and change something in it.

To understand what an iteration is and what pile of tasks it contains, we adopted the Scrum term definition of done. It’s a set of criteria that should be satisfied before a story can be considered finished.

Here I use this term differently. My interpretation of the term is that it’s a description of what will change in an application when a certain iteration is put into production.

Let’s remember our first case with the order management system. A user can create a new order by using a new UI and old backend. This is a type of iteration — a specific part of capability that will work when we can put into production. Or, a user can be authorized by a new access permission model, which is also a quality change. Similarly, we can describe what each iteration contributes to your crusade against technical debt.

When you divide a solution into iterations, many tasks may appear. There will be dependencies between them, forming a graph. There will be several ways to achieve your final goal in this graph. Back scheduling might be used as a tool to help reduce time of work. We start from the final point of the project and ask ourselves: “What do we need to do to achieve this goal?” We understand that we have to make the previous step for our goal achievement. So we move from the end to the beginning, and on each intermediate step we answer this question running along the critical path. This approach once saved us a quarter of development.

A tool called Gantt chart is very convenient to visualize your work. It’s a graph that shows dependencies between tasks, how long a task is being worked on, and what the project looks like. You can see a screenshot from Wrike in the picture. We consistently use this tool for our projects.

If you work on a broad solution, there’s a possibility you’ll encounter a situation where someone changes the code you’ve been refactoring and adapting. These changes might add obstacles in your path to fight technical debt.

You can defend yourself from such changes:

• The easiest way is to put everything into production as soon as possible. So, the other developer who needs to change something in your code or use your solution will work with the most relevant version of the code, which can minimize mistakes.
• We can use an automatic system to monitor all changes. We can use a git hook, embed a verification at the moment of git commit, or git push. You can compose a unit test to check if there are changes in any building blocks or classes and give off a warning for you to consult your coworker or to not apply it at all.

You can adjust an external monitoring system for your code. We use PMD at Wrike. The system launches and checks each new codeline’s accordance with different rules. There’s an example from a build log in the picture: The rule “public method results must be immutable” was broken. PMD notifies us and shows which line — here, it’s the “wrong method.” Below is a hint for how to fix it. That’s why we always know where the rule is broken and how to fix it.

How to beat the final boss

An important thing we didn’t discuss yet is the final boss you have to beat while working on technical debt. He has many names: business owner, product owner, ordering party. He can get in the way of putting our engineering initiatives into production.

We all know cases where a product owner pushed solutions that weren’t the best from an engineering point of view. But even if your product owner has a heavy outlook you can come to an agreement with him. While working with technical debt you open new possibilities that a product owner needs for his product development.

You can agree on a time quota. For example, 10% of a developer’s time will be spent working on technical debt. Such a quota won’t get rid of technical debt but also won’t expand it.

It’s obviously hard to talk about technical debt if you don’t understand the context. That’s why you must have a technical backlog in your team that includes estimated tasks according to priority by developers.

However, these tools aren’t enough to manage large-scale projects. In this case it’s important to try to explain the benefits to your ordering party using all the three cases mentioned above (technical debt, tracking projects, critical path in a graph, etc.). The more detailed the story, the easier it’ll be for others to understand. Every business has goals that you help to achieve. You have to be aware of what’s not only happening right now but of plans for future quarters. That’s why you need to communicate with business owners, understand what they want, how to help achieve their goals, and use this information to unite engineering and business goals.

The objectives might not coincide right now but rather in a quarter or even a year later. But they’ll allow you to understand when the business is ready for change.

If you’re successful with synchronizing the goals, you’ll receive all bonuses, priorities, resources, and work organization. A product owner can help you to do that. The main task is to come to an agreement with them.

Summary

When we discuss changes in product key points, an easy four-step algorithm becomes more complicated.

Two more steps are added. The first one is understanding what we’re working with. The second is understanding how the solution works (after encapsulation) and protecting your code from unwanted changes.

Using this algorithm and my recommendations on organizing the process, you’ll be able to work with any technical debt.