A retrospective on our hurdles. How a custom solution turned out to be better than a paid one.

Hello! My name is Alexey Pyankov, and I am the lead programmer at the Sportmaster company. Let me clarify that 'lead' does not mean 'the most important among all programmers'; no, it's just the title, a charming translation for 'Senior+'

I have been working at Sportmaster since 2012, and during this time, our development team has created many interesting solutions from a technical standpoint. However, today I would like to discuss our work, focusing more on how we reasoned in certain ambiguous situations.

This article will not present specific technical solutions (or anything technical) that should be grabbed and applied to your projects. Rather, it will be a reflection on the work we’ve done. There were unique moments that impacted us as a team — bonded us, forged us, and tested us for resilience. I will try to share about these moments, the atmosphere of teamwork, our stumbling blocks, and a series of psychological traps into which we sometimes fall ourselves.

And I will start precisely from 2012.

I joined in 2012 with the main goal at that time — working on our flagship website. At that time, it was a 'Frankenstein monster': part of the team worked with our old system, which didn't handle loads very well (Bitrix), while another part of the team (myself included) was attempting to implement a new system, selected based on the criterion 'Since this is the most expensive e-commerce platform in the world, let's use it.' We were 'attempting to implement' because the system was fiercely resistant, and for every issue we figured out, a 'surprise' would inevitably arise in response. We worked hard but progressed at a snail's pace.

For me, the last straw was getting acquainted with the code of a certain method in this "most expensive e-commerce in the world", when several hours of focused work on a complicated bug led to the discovery of the cause hidden somewhere in a custom tag that processes HTML generation in JSP. The purpose of this custom tag is to display the sum of certain values. That's fine; that's what custom tags are meant for. But the surprise was that during this process, some data in the database was being altered, which affected the behavior on subsequent pages, and if you pressed F5, the call repeated, violating data consistency. Moreover, it did so in a way that only manifested after several steps, on the 3rd page of the sequence. No, I'm not against having such a "ninja master" on the team keeping the attention of colleagues sharp with his code. But to have that happen in the library of the most expensive system!

It was Friday. My colleague and I spent Saturday and Sunday in the office aiming to figure out what tasks the business is setting for the system right now and what tasks might arise in a year. Accordingly, how would we solve them if we weren't constrained by the use of this most expensive and infuriating system?

Said and done. We created a pilot project that laid the foundation for the development of the new Sportmaster website. Many of these ideas took root and their continuation is currently actively running on the site.

Pilot stages and timelines

2 days. We created a micro-prototype — over the weekend, we migrated our database to ElasticSearch and implemented faceted search. Voila! In that very purchased system, such a configuration took 2 weeks. Here, it took literally a couple of hours! And it works faster. In fact, significantly faster.

2 weeks. We are "building" the prototype, adding functionality for adequate personalized output.

For example, if a user has several discounts and promotions applicable specifically to them — then the search results for products need to display the price that can be obtained by applying all available perks in the most advantageous way.

Promotions aren't straightforward. For instance, you buy skis and now get a 40% discount on a hat, but the welcome discount of 10% on the entire order is canceled. Yes, this is a real case 🙂 To set up such a promotion in the purchasing system, three consultations with the supplier were paid for, resulting in many examples of how to create different other promotions. Very diplomatically done, and considering the cost of consultations — quite economically viable.

We showed the business a detailed demo. They promised to quickly assemble a pilot project and immediately got to work.

2 months. The pilot project — we’re creating it as a live site with a catalog search. The search features faceted navigation, and the search results include personalized discounts, resembling a pilot as if it's the Sportmaster site, with the same products uploaded as well. It's fantastic!

We added "Eloquent Speech:100" from our department head, and the business presentation went exceptionally well! We received a green light to develop the eCommerce platform ourselves.

And that means, hold on, guys, team, hold on, guys, budget. Isn’t that great!

2 years. Launching the site into production. Yes, it took a long time. Everything we knew at the time, we only tried at the prototype scale. Two people can easily form a cohesive team. The tasks we ‘checked off’ were mostly small tweaks of ‘Hello World’ in new technologies. We easily generated new hypotheses, quickly tested them, didn’t get attached, and thus without regrets, we ‘eliminated’ them. When we became a team of ten, we mistakenly extrapolated our work speed onto everyone else. We promised timelines for task completion that equated our ideal vision multiplied by our enthusiasm.

Is this a familiar situation? 🙂

So, do you already know what will happen next?

Trap #1. “The Extrapolator’s Trap”

It's clear that new technologies look very impressive in presentations and perform excellently in ‘Hello World’ applications. But reality is usually a bit different.

So here’s the deal. We take a library, write a ton of application code. We consider unit tests a burden (we're cool and working at hypersonic speed, with modern code and all that). Constantly changing and improving the API on the fly — come on, what tests, seriously? And all this under the banner of ‘we’ve coolly optimized the development process’ (yeah, it’s scary to even describe it now).

And then it all becomes quite obvious.

We're rolling out a new build on UAT. The business team is eagerly testing everything and clicking buttons. Sometimes they get quite creative with their clicks—something breaks. At this point, it would be good to find out what was done about it. But on the other side of the monitor, there isn't a meticulous tester ready to provide environment specifications accounting for regional weather; it's a client from the business. For them, it simply 'isn't working.' And that means they're unhappy. Ask them, and they'll be terrified unhappy!

Then, to reproduce the bug, you have to go in and click through everything methodically. Of course, we didn't ignore any complaints and fixed everything. We dropped scheduled tasks and ‘extinguished the fire.’

Thus, we dug ourselves another pit.

Trap #2. 'Stakhanovite'

An unpleasant bug lands in your lap. You start investigating. No luck—frustration—another attempt to figure it out—another dead end—you clarify everything you can—still not right—you think about how you're getting older while everyone else has kids and mortgages—you try again—still not it. Several cups of coffee later, and everything repeats. 12-14 hours of solid work is almost the norm. Just when everyone is on edge—bam, an epiphany!

From outside, the assessment of such a day’s efficiency might seem clear and accurate. However, from the inside, it can feel quite different.

In my case, the impression of such work included, 'I did great, I'm awesome, I sorted it out.' Not always consciously, but unconsciously—always!

You get hooked on that, no joke. It turns out that internal metrics of success shift from results to the amount of effort put in and the level of heroic feats you accomplished, how much you suffered trying to solve the problem.

Probably, this is the most terrifying trap.

It will get easier and more fun from here 🙂

Trap #3. 'The Power of Hello World'

Our tech stack at that time included: ElasticSearch, Hazelcast, Pentaho, Freemarker (along with established technologies like Java, Spring, Tomcat, nginx). Freemarker was not very informative in terms of error messages. However, both ElasticSearch and Hazelcast required patching several times—we skillfully identified cases where they didn't function as per the documentation.

A quick start and rapid rewards from new technology are great, but they can create euphoria and lower vigilance. Because new technology comes with bugs, it always does. And if they haven't been written about yet, rejoice—you will inevitably be the pathfinder who digs up something crooked and heads to Google or Stack Overflow. Of course, 'crooked' can be found in proven products, but in new ones, it's much easier.

Despite all the challenges, we made it into production. Yes, with lags. Yes, not very stable. But overall—without disasters.

Once again, I'll highlight the traps that distort a healthy perception of the work process.

1. "The Super Extrapolator". Under the influence of our current successes, we joyfully extrapolate the speed of development onto upcoming projects.
2. "The Stakhanovite". We work tirelessly, satisfied with ourselves, but fail to notice that the issues we solve are a result of our personal mistakes/oversights/neglect. Works not to be done.
3. "The Power of Hello World". We rush to implement the newest and most interesting things into production.

Why it all worked out

Of course, I haven't listed all the mistakes we've made during this time, but the most common ones, likely to occur in any project. Such a documentation of mistakes helps avoid them in the future.

A little about how we managed to create such a mini-startup within the company and convince the business to move from an already purchased system to something custom-made.

Condition #0. A healthy climate within the company. This is not only about the 'sparkling eyes' of employees and communication under stress of cookie production, no. It’s about all interactions.

Condition #1. Believing in what you do. Seriously, I don't think we would have had any chance if we had embarked on the pilot without thoroughly understanding the purchased system 'down to the bolts'—that is, stepping back while subconsciously knowing that this system was cooler and would outdo us.

What we did: 1) we understood the purchased system, using it to address key business requests 2) we compiled a list of tasks that are not only relevant now but will also be in the foreseeable future 3) we identified a solution that fits better. And then, our assessment of the solution was an expert assessment.

Would they give us anything if we just showed up and said, 'Guys, this is all nonsense, we don't want to deal with it and decided to start from scratch'? Unlikely. The answer would be delivered in such a way that it would be well remembered 🙂

Condition #2. We start small. We generate the first hypothesis and test it. You can spend your personal time on this. If you don’t want to waste your time, then you really shouldn’t take on such a task. And if you don’t want to test a small hypothesis, but instead wish to create something impressive and polished—stay away from such people!

We were lucky, and the very first hypothesis worked. But that doesn't always happen. For example, in one of the subsequent projects, when we promoted the admin panel as part of a similar pilot, we only succeeded with the 18th variant. The first 17 attempts were in vain. By the way, in the story of creating the admin panel, the plot twists were at the level of Brazilian soap operas because the team consisted of guys who were already veterans at that point, true 'seasoned players'.

Condition #3. We create an MVP and look for pain points with the decision-maker. Of course, he may show horror on his face just from the fact that you are bringing him some initiative for the thirtieth time. But all the same. And we must show exactly how we solve his problems with our product.

Condition #4. We quickly create a pilot that looks approximately like the final result. Making everything absolutely great is tempting, but you can get caught up in perfectionism, which results in wanting to present a pilot version of an already ideal product. And that doesn’t exist. So just make it at least from sticks.

Condition #5. The product. The project grows, secures funding, and specialists with solid experience join it.
And if you're a classic startup founder, this is the moment when you need to exit with a bang. Because the easy flights at the very top and the overall sense of bliss quickly dissipate.

Going to production is a confrontation with real loads, integrating with dozens of systems, and while creating new functionality, you simultaneously work on supporting the older versions. All of this presents challenges that are much more serious than just coming up with an idea and resolving, albeit well, just one client’s problem.

These challenges and skill growth occur at this stage.

Thank you for reading. Happy New Code!

Source: habr.com