It is known that the competency of a CTO is assessed only during the second tenure in this role. Because working for several years within a company, evolving alongside it, and gradually taking on more responsibility in the same cultural context is one thing, while stepping directly into the position of a technical director in a company burdened with legacy issues and a multitude of problems is quite another.

In this sense, the experience of Leon Fire that he shared at DevOpsConf, is not exactly unique, but combined with the years of experience and the various roles he has taken on over the past 20 years, it is very useful. Below is a timeline of events from the past 90 days and many anecdotes that are amusing to hear when they happen to someone else but not so enjoyable when faced personally.

Leon narrates very vividly in Russian, so if you have 35-40 minutes, I recommend watching the video. The text version is provided below for time-saving purposes.

The first version of the report was a well-structured description of working with people and processes, containing useful recommendations. However, it did not convey all the surprises encountered along the way. Therefore, I changed the format and laid out the issues that popped up in the new company like a genie from a lamp, along with the methods of solving them in chronological order.

A month before

Like many good stories, this one began with alcohol. We were sitting with acquaintances in a bar, and as is customary among techies, everyone was sharing their problems. One of them had just changed jobs and was discussing his issues with both the technologies and the people, as well as with the team. The longer I listened, the more I realized that he simply needed to hire me, as I had been solving exactly such problems for the last 15 years. I told him so, and the next day we met in a work environment. The company was called Teaching Strategies.

Teaching Strategies leads the market in educational programs for very young children — from birth to three years old. The traditional 'paper' company has been around for about 40 years, while the digital SaaS version of the platform is 10 years old. Relatively recently, the process of adapting digital technology to the company's standards began. The 'new' version launched in 2017 and was almost like the old one, only it worked worse.

The most interesting thing is that the traffic for this company is very predictable — day by day, year by year, it is possible to forecast very clearly how many people will come and when. For example, between 1 PM and 3 PM, all children in kindergartens go to sleep, and teachers start entering data. And this happens every day except weekends, because almost no one works on weekends.

To jump ahead a bit, I'll mention that I started my work during the peak of the annual traffic, which is interesting for various reasons.

The platform, which seemed to be only 2 years old, had a peculiar stack: ColdFusion & SQL Server 2008. ColdFusion, if you don't know — and you probably don't — is like enterprise PHP that came out in the mid-90s, and I hadn't heard about it since then. There were also Ruby, MySQL, PostgreSQL, Java, Go, and Python. But the core monolith ran on ColdFusion and SQL Server.

Issues

The more I talked to the company's employees about work and the issues they faced, the more I understood that the problems were not just technical in nature. Okay, the technology is old — I've worked with worse, but there were issues with the team and processes, and the company was beginning to understand that.

Traditionally, the tech guys were sitting in a corner, doing their own work. But more and more business started flowing through the digital version. Therefore, in the year prior to my joining, new roles emerged in the company: a board of directors, CTO, CPO, and QA director. In other words, the company began investing in the tech sector.

The traces of a heavy legacy were not only in the systems. The company had legacy processes, legacy people, and legacy culture. All of this needed to change. I thought it would definitely not be boring, so I decided to give it a try.

Two days before

Two days before starting my new job, I arrived at the office, filled out the last paperwork, met the team, and discovered that the team was currently struggling with an issue. It turned out that the average page load time had jumped to 4 seconds, which is twice as long.

According to the graph, something clearly went wrong, but it was unclear what. It turned out that the problem was with network latency in the data center: 5 ms latency in the data center transformed into 2 seconds for users. I didn't know why this happened, but in any case, it became known that the problem was in the data center.

Day one

Two days passed, and on my first working day, I discovered that the problem had not gone away.

For two days, users experienced average page load times of 4 seconds. I asked if they found out what the problem was.

— Yes, we opened a ticket.
— And?
— Well, they haven't replied to us yet.

At this point, I realized that everything I had been told before was just the tip of the iceberg that needed to be addressed.

There's a good quote that fits this situation perfectly:

“Sometimes, to change the technology, you need to change the organization.”

But since I started working at the busiest time of the year, I had to consider both quick fixes and long-term solutions. I needed to start with what was critical right now.

Day three

So, the load time is 4 seconds, with the biggest spikes occurring between 1 PM and 3 PM.

On the third day, during this time frame, the loading speed looked like this:

From my perspective, nothing was working at all. From everyone else's perspective, it was working a bit slower than usual. But it can't just be that—it’s a serious issue.

I tried to convince the team, but they responded that we simply needed more servers. That is, of course, a solution, but far from the only and most effective one. I asked why we were short on servers and what the traffic volume was. I extrapolated the data and found we had about 150 requests per second, which is generally within reasonable limits.

But we mustn't forget that before getting the right answer, we need to ask the right question. My next question was: how many frontend servers do we have? The answer "slightly puzzled" me—we had 17 frontend servers!

— I hesitate to ask, 150 divided by 17 is about 8, right? Are you saying that each server handles 8 requests per second, and if tomorrow we have 160 requests per second, we need 2 more servers?

Of course, we didn't need additional servers. The solution was in the code itself, right on the surface:

var currentClass = classes.getCurrentClass();
return currentClass;

There was a function getCurrentClass(), because everything on the site operates within the context of the class—that's correct. And for this one function on each page, there were 200+ requests.

The solution was very simple, there was no need to rewrite anything: just do not request the same information again.

if ( !isDefined("REQUEST.currentClass") ) {
    var classes = new api.private.classes.base();
   REQUEST.currentClass = classes.getCurrentClass();
}
return REQUEST.currentClass;

I was very happy because I thought that on the third day I had found the main problem. How naive I was, it was just one of many issues.

But solving this first problem lowered the schedule significantly.

At the same time, we were working on other optimizations. There was a lot on the surface that could be fixed. For example, on that same third day, I discovered that there was indeed caching in the system (at first, I thought that all requests were coming straight from the database). When I think of caching, I imagine standard Redis or Memcached. But that was only my thought, because MongoDB and SQL Server were being used for caching in that system—the same one from which we had just read the data.

Day ten

In the first week, I was dealing with issues that needed to be resolved immediately. Somewhere in the second week, I attended the standup for the first time to talk with the team, see what was happening, and how the entire process was going.

Once again, interesting findings emerged. The team consisted of: 18 developers; 8 testers; 3 managers; 2 architects. And they all participated in the common rituals, meaning that over 30 people showed up for the standup every morning and talked about what they had been doing. It was clear that the meeting did not last 5 or even 15 minutes. Nobody listened to each other because everyone was working on different systems. In that format, getting 2-3 tickets done in an hour during grooming session was already a good result.

The first thing we did was to split the team into several lines by products. We formed separate teams for different sections and systems that included developers, testers, product managers, and business analysts.

As a result, we achieved:

• Shorter standups and meetings.
• Subject matter knowledge of the product.
• A sense of ownership. Previously, when people constantly rotated through systems, they knew that someone else would likely have to deal with their bugs, but not themselves.
• Collaboration between teams. It's no secret that QA hadn't interacted much with developers before; the product team was doing its own thing, and so on. Now they have a common point of responsibility.

We primarily focused on efficiency, performance, and quality — these were the issues we aimed to solve through team transformation.

Day eleven

During the process of changing the team structure, I discovered how they calculate StoryPoints. 1 SP was equivalent to one day, and each ticket contained SP for both development and QA, meaning at least 2 SP.

How did I find this out?

We found a bug: in one of the reports where the start and end dates for the reporting period are input, the last day wasn’t included. This means somewhere in the query it was not <= but just <. I was told this was three Story Points, which equals 3 days.

After this we:

• Revised the Story Points estimation system. Now, fixing minor bugs that can quickly pass through the system reaches the user faster.
• Started to merge related tickets for development and testing. Previously, each ticket, each bug was a closed ecosystem, unconnected to anything else. Changing three buttons on one page could be three different tickets with three different QA processes instead of one automated test for the page.
• Began working with developers on the approach to estimating labor costs. Three days to change one button — that’s not funny.

Day twenty

By the middle of the first month, the situation had stabilized a bit; I figured out what was mainly happening and began to look to the future, thinking about long-term solutions.

Long-term goals:

• Managed platform. Hundreds of requests on each page — it's not serious.
• Predictable trends. There were periodic traffic spikes that, at first glance, did not correlate with other metrics — it was necessary to understand why this happened and learn to predict.
• Platform expansion. The business is constantly growing, more users are coming in, and traffic is increasing.

In the past, it was often said: "Let's rewrite everything in [language/framework]; everything will work better!"

In most cases, it doesn't work, and it's a good thing if the rewritten version works at all. Therefore, we needed to create a roadmap—a specific strategy illustrating step by step how the business goals will be achieved (what we will do and why), that:

• reflects the mission and goals of the project;
• prioritizes key goals;
• contains a timeline for their achievement.

Until now, no one had spoken to the team about the purpose of any changes being made. The right success metrics are needed for this. For the first time in the company's history, we set KPIs for the technical team, tying these metrics to the organization.

That is, organizational KPIs are supported by teams, and team KPIs are supported by individuals. Otherwise, if technological KPIs do not align with organizational ones, everyone pulls the blanket toward themselves.

For example, one of the organizational KPIs is increasing market share through new products.

What can support the goal of having more new products?

• Firstly, we want to spend more time developing new products instead of fixing defects. This is a logical solution that is easy to measure.
• Secondly, we want to support an increase in transaction volumes because the larger the market share, the more users there are, and consequently, more traffic.

Then, the individual KPIs that can be executed within the group will, for example, focus on where the main defects come from. If we focus specifically on this section, we can significantly reduce defects, thereby increasing the time available for developing new products and again supporting organizational KPIs.

Thus, every decision, including rewriting code, must support the specific goals set by the company (organizational growth, new features, staffing).

During this process, an interesting thing emerged, which became news not only for the tech team but for the whole company: all tickets must be oriented toward at least one KPI. That is, if the product manager says they want to create a new feature, the first question should be: "Which KPI does this feature support?" If there isn't one, then sorry—it seems this feature is unnecessary.

The thirtieth day

At the end of the month, I discovered another detail: none of my Ops team had ever seen the contracts we sign with clients. You might ask why it's important to see the contracts.

• First of all, because the SLAs are outlined in the contracts.
• Secondly, all SLAs are different. Each client comes with their own requirements, and the sales department signed without looking.

Another interesting detail is that in the contract with one of our largest clients, it states that all software versions supported by the platform must be n-1, meaning not the latest version, but the one before it.

It's clear how far we were from n-1 when the platform was running on ColdFusion and SQL Server from 2008, which stopped being supported back in July.

Day forty-five

Somewhere around the middle of the second month, I found enough time to sit down and work valuestreamIn the server response, I've added the field values of the indexed document in comments: fully on the entire process. These are the necessary steps that need to be taken, from creating the product to delivering it to the consumer, with a detailed breakdown.

You break the process into small pieces and look at what takes too much time, what can be optimized, improved, etc. For example, how much time it takes for a request from the product to go through grooming until it becomes a ticket that a developer can take, QA, etc. You examine each individual step in detail and consider what can be optimized.

When I did this, two things stood out:

• a high percentage of tickets being returned from QA back to developers;
• pull request reviews took too much time.

The problem was that these were conclusions like: it seems to take a lot of time, but we're not sure exactly how much.

"You can't improve what you can't measure."

How do you justify how serious the problem is? Does it waste days or hours?

To measure this, we added a couple of steps to the Jira process: "ready for dev" and "ready for QA" to track how long each ticket waits and how many times it returns to a specific step.

We also added "in review" to know how long tickets are in review on average, and build from that. We had system metrics, now we've added new metrics, and started measuring:

• Process efficiency: productivity and planned/delivered.
• Process quality: number of defects, defects from QA.

This really helps to understand what is going well and what is not.

The fiftieth day

All of this is, of course, good and interesting, but towards the end of the second month, something happened that was, in principle, predictable, though I did not expect it to be on such a scale. People started leaving because the leadership changed. New people came into management, who began to make changes, and the old ones were let go. In a company that has been around for a few years, everyone usually knows each other and is friends.

This was expected, but the scale of the layoffs was unexpected. For example, two team leads submitted their resignations simultaneously in one week. So, I had to not only forget about other problems but also focus on building the team. This is a long and challenging issue to solve, but it needed to be addressed because I wanted to retain the people who remained (or most of them). We had to find a way to respond to the departures to maintain morale in the team.

In theory, this is good: a new person comes in, who has a clean slate and can assess the skills of the team and make replacements. In reality, you can't just bring in new people for many reasons. Balance is always needed.

• Old and new. It is necessary to keep the old employees, who can adapt and support the mission. But at the same time, new blood needs to be brought in; we'll talk about that a bit later.
• Experience. I spoke a lot with good juniors who were eager and wanted to join us. But I couldn't hire them because there weren't enough seniors to support the juniors and act as their mentors. We needed to build the upper level first and only then bring in the youth.
• Carrot and stick.

I don't have a good answer to the question of what the right balance is, how to maintain it, how many people to keep, and how much pressure to apply. It's purely an individual process.

The fifty-first day

I started to observe the team to understand who I have, and once again recalled:

Most problems are people problems.

I found that there are three major issues in the team, both among developers and Ops:

• Satisfaction with the current state of affairs.
• Lack of accountability. — because no one has ever linked the results of the performers' work to its impact on the business.
• Fear of change.

Changes always push people out of their comfort zone, and the younger people are, the more they dislike change, because they don’t understand why it’s necessary and how it works. The most common response I’ve heard is, "We’ve never done it like this before." It reached a point of absurdity — even the slightest changes would provoke outrage from someone. And it didn’t matter how the changes affected their work; people would say, "No, why? That won’t work."

But you cannot improve without making changes.

I had an absolutely absurd conversation with an employee; I was sharing my ideas for optimization, to which he replied:
— Oh, you didn’t see what we had last year!
"So what?"
— It’s much better now than it was.
— So, it couldn’t be even better?
— Why should it be?

Good question — why? As if the fact that things are better now means everything is good enough. This leads to a lack of accountability, which seems perfectly normal. As I said, the tech team was a bit on the sidelines. The company believed they should be involved, but no one ever set standards. In tech support, we never saw an SLA, so for the team, it was entirely "acceptable" (and this shocked me the most):

• 12 seconds loading;
• 5-10 minutes of downtime with each release;
• resolving critical issues takes days and weeks;
• no 24/7 on-call support.

No one ever tried to ask why we couldn’t do it better, and no one understood that it shouldn’t be like this.

As a bonus, there was another problem: a lack of experience.Seniors left, and the remaining young team grew up under the old regime and was poisoned by it.

On top of all this, people were also afraid of failing, of appearing incompetent. This is expressed in the fact that they, first of all, under no circumstances asked for help.How many times have we talked in a group and individually, and I said, 'Ask a question if you don't know how to do something'? I have confidence in myself and know that I can solve any problem, but it takes time. So if I can ask someone who knows how to solve it in 10 minutes, I will ask. The less experience you have, the more afraid you are to ask, because you think that you'll be considered incompetent.

This fear of asking a question manifests in interesting ways. For example, you ask, 'How's it going with this task?' — 'Just a couple of hours left, I'm wrapping it up.' The next day you ask again and get the answer that everything is fine, but there's one little problem, it will definitely be ready by the end of the day. Another day passes, and unless you corner them and force them to talk to someone, it all just continues. A person wants to solve the task themselves; they think if they don't solve it, it will be a huge failure.

That is why developers overestimate tasks. It was quite a joke when we were discussing a specific task, and I was given such a figure that I was very surprised. To which I was told that in estimates developers include the time it will take for the ticket to come back from QA because they will find errors, and the time PR will take, and the time while people who are supposed to review it will be busy — that is, everything possible.

Secondly, people who are afraid of appearing incompetent, overanalyze. When you say what exactly needs to be done, it starts: 'No, but what if we think about it here?' In this sense, our company is not unique; this is a standard problem among the youth.

In response, I implemented the following practices:

• The 30-Minute Rule. If you can't solve a problem in half an hour, ask someone for help. This works with varying success because people still don’t ask, but at least the process has started.
• Exclude everything but the essence, in estimating task completion time, meaning only count how long it will take to write the code.
• Continuous learning for those who overanalyze. It's just constant work with people.

Day sixty

While I was dealing with all this, it was time to look into the budget. Of course, I found a lot of interesting things about where we were spending money. For instance, we had an entire rack in a separate data center with one FTP server that was being used by a single client. It turned out that '... we were moving, and it just stayed there, we didn't change it.' That was two years ago.

The cloud service bill was particularly interesting. I am convinced that the main reason for the high cloud service bill is that developers have unlimited access to servers for the first time in their lives. They don’t need to ask: 'Can I have a test server, please?' — they can take one themselves. On top of that, developers always want to build such a cool system that Facebook and Netflix would envy.

However, developers lack experience in procuring servers and the skill to determine the right server size because they haven’t needed it before. And usually, they don’t quite understand the difference between scalability and performance.

Inventory Results:

• We exited one data center.
• We terminated contracts with 3 log services. Because we had 5 — every developer who started playing with something took a new one.
• We shut down 7 AWS systems. Again, no one stopped the dead projects; they just kept running.
• We reduced software expenses by 6 times.

Day seventy-five

Time passed, and after two and a half months, I had to meet with the board of directors. Our board of directors is not better or worse than others; like all boards, they want to know everything. People invest money and want to understand how what we are doing fits into the set KPIs.

The board receives a lot of information every month: user numbers, growth, what services they use and how, performance and productivity, and finally, the average loading speed of the page.

The only problem is that I believe averages are pure evil. But it is very difficult to explain this to the board. They are used to operating with aggregated numbers instead of, for example, the variance in loading time in seconds.

In this regard, there were interesting moments. For example, I said that we need to distribute traffic among separate web servers depending on the type of content.

This means ColdFusion runs through Jetty and nginx, serving the pages. Images, JS, and CSS are handled by a separate nginx with its own configurations. This is quite a standard practice, which I written mentioned a couple of years ago. As a result, images load much faster, and the average loading speed has increased by 200 ms.

This happened because the graph is based on data coming from Jetty. That is, fast content is not included in the calculation—the average value jumped up. We understood this, we laughed, but how to explain to the board why something we did made it worse by 12%?

Day eighty-five

By the end of the third month, I realized one thing I hadn’t anticipated was time. Everything I talked about takes time.

This is my actual calendar for the week—just a regular work week, not too busy. There’s not enough time for everything. Therefore, again, we need to hire people to help manage the issues.

Conclusion

This is not all. In this story, I haven’t even reached how we worked with the product and tried to attune ourselves to a common vibe, or how we integrated tech support, or how we resolved other technical issues. For example, I accidentally found out that on the largest tables in the database, we don’t use SEQUENCE. We have a custom function nextID, and it is not used in the transaction.

There were a million similar things to discuss. But the most important thing to mention is culture.

It is precisely culture, or the lack of it, that leads to all other problems. We are trying to build a culture where people:

• are not afraid of failure;
• learn from mistakes;
• collaborate with other teams;
• show initiative;
• take responsibility;
• embrace results as a goal;
• celebrate success.

With this, everything else will follow.

Leon Fire on Twitter, facebook and on medium.

Regarding legacy, there are two strategies: to avoid working with it at all costs, or to bravely tackle the accompanying challenges. We are taking the second path, changing processes and approaches. Join us in DevOpsConf the Telegram channel youtube, , and let’s together foster a DevOps culture. and It is known that the competency of a CTO is only tested during the second execution of this role.🥇Inheriting legacy systems and processes or The First 90 Days in the CTO Role | ProHoster

Source: habr.com