Damn it, Google, I didn't want to write a blog again. I have so much to do. Blogging takes time, energy, and creativity that I could use elsewhere: my books, , my game, and so on. But you've annoyed me enough, so I have to write this.
So let's get this over with.
I'll start with a small but instructive story from the time when I first began working at Google. I know I've said a lot of bad things about Google lately, but it frustrates me when my own company regularly makes incompetent business decisions. That said, I must give credit where it's due: Google's internal infrastructure is truly extraordinary; one could confidently say that there's nothing better today. The founders of Google were much better engineers than I will ever be, and this story only confirms that fact.
First, a little background: Google has a data storage technology called . It was a remarkable technical achievement, one of the first (if not the first) 'infinitely scalable' key-value stores: essentially, the beginning of NoSQL. These days, Bigtable is still doing well in a rather crowded K/V storage space, but back then (in 2005) it was incredibly cool.
One amusing detail about Bigtable is that they had internal control plane objects (as part of the implementation) called tablet servers, with large indexes, and at one point they became a bottleneck for scaling the system. Bigtable engineers were scratching their heads over how to achieve scalability, and suddenly realized they could replace tablet servers with other Bigtable storages. So, Bigtable is part of the implementation of Bigtable. These storages are present at all levels.
Another interesting detail is that for a while, Bigtable became popular and ubiquitous within Google, with each team having its own storage. So during one of the Friday meetings, Larry Page casually asked, "Why do we have more than one Bigtable? Why not just stick with one?" Theoretically, one storage should suffice for all Googleās storage needs. Of course, they never switched to just one for practical development reasons (such as potential failure implications), but the theory was intriguing. One storage for the entire Universe (By the way, does anyone know if Amazon did something similar with its Sable?)
In any case, hereās my story.
At that time, I had been working at Google for just over two years, and one day I received an email from the Bigtable engineering team with roughly this content:
Dear Steve,
Hello from the Bigtable team. We want to inform you that in the [data center name] data center, you are using a very, very old Bigtable binary file. This version is no longer supported, and we want to help you transition to the latest version.
Please let us know if you can schedule some time to work together on this issue.
Best wishes,
The Bigtable Team
At Google, you receive a lot of emails, so at first glance, I read it something like this:
Dear Recipient,
Hello from some team. We want to inform you that blah blah blah blah blah. Blah blah blah blah blah blah, and blah blah blah immediately.
Please let us know if you can schedule some of your precious time for blah blah blah.
Best wishes,
Some Team
I almost deleted it right away, but at the edge of my consciousness, I felt a nagging, dull sense that this didnāt quite sound like a formal email, although it was obviousthat they had mistakenly addressed it to me since I hadnāt used Bigtable.
But it was strange.
The rest of the day, I alternated between thinking about work and contemplating which type of shark meat to try in the micro-kitchen, of which at least three were close enough to hit with a well-aimed throw of a biscuit from my spot, but the thought of the email stayed with me, accompanied by a growing sense of mild anxiety.
They clearly named my name. And the letter was sent to my email address, not someone else's, and it's not cc: or bcc:. The tone is very personal and clear. Could this be some kind of mistake?
Finally, curiosity got the better of me and I went to take a look at the Borg console in the data center they mentioned.
And of course, I had a BigTable storage under my management. What?! I looked at its contents, and ā wow! It was from the Codelab incubator, where I spent my first week at Google in June 2005. Codelab made you fire up Bigtable to record some values, and I apparently never shut down the storage after that. It's still running, even though more than two years have passed.
There are several notable aspects to this story. Firstly, the operation of Bigtable was so insignificant in the scale of Google that only after two years did someone notice the excess storage, and only because the binary version was outdated. For comparison, I once considered using for my online game. At that time, this service cost about $16,000 a year for an empty Bigtable on GCP. Iām not saying theyāre conning you, but in my personal opinion, thatās a lot of money for a completely useless database.
Another notable aspect is that the storage was still running after two years.. WTF? Data centers come and go; they experience outages, undergo scheduled maintenance, and theyāre constantly changing. Hardware gets upgraded, switches get replaced, everything is constantly being improved. How on earth did they manage to keep my program running for two years considering all these changes? It may seem like a modest achievement in 2020, but from 2005 to 2007, it was quite impressive.
And the most remarkable aspect is that an external engineering team in some other state reached out to me, the owner of some tiny, practically empty instance of Bigtable, which had zero traffic for the past two years ā and offered help to upgrade it.
I thanked them, deleted the storage, and life went on. But thirteen years later, I still think about that letter. Because sometimes I receive similar letters from Google Cloud. They look like this:
Dear Google Cloud user,
We would like to remind you that we are discontinuing support for the service [important service you are using] starting August 2020, after which you will not be able to update your instances. We recommend switching to the latest version that is currently in beta testing, which has no documentation, migration path, and has already been deprecated with our kind assistance.
We strive to minimize the impact of this change on all users of the Google Cloud platform.
Best friends forever,
Google Cloud Platform
But I almost never read such letters because they actually say the following:
Dear Recipient,
Go to hell. Go away, go away, go away. Forget everything you are doing because it doesnāt matter. What matters is our time. We are wasting time and money maintaining our crap, and we are tired of it, so we will no longer support it. So throw away your damn plans and start sifting through our lousy documentation, begging for scraps on forums, and by the way, our new crap is completely different from the old crap because we messed up this design pretty badly, haha, but thatās your problem, not ours.
We are still making efforts to render all your developments unusable within one year.
Please go away,
Google Cloud Platform
And the thing is, I get such letters about once a month. It happens so frequently and consistently that they inevitably repelled me from GCP to the camp of cloud opponents. I no longer agree to rely on their proprietary developments because itās actually easier for a DevOps to maintain a system with open-source on a bare virtual machine than to try to keep up with Googleās policy of shutting down "deprecated" products.
Before coming back to Google Cloud because I am not even close he didn't stop criticizing them; let's take a look at the company's work in some other areas. Google engineers take pride in their software development discipline, and this is actually causing problems. Pride is a trap for the reckless; it has led many Google employees to believe that their decisions are always correct and that correctness (by some vague undefined measure) is more important than caring for customers.
I will provide a few arbitrary examples from other large projects outside of Google, but I hope you will see this pattern everywhere. It boils down to the following: backward compatibility supports the longevity and relevance of systems for decades.
Backward compatibility is the design goal of all successful systems intended for open use, meaning implemented with open source and/or open standards. I feel like I'm stating something too obvious that it's uncomfortable for everyone, but no. This is a political issue, so examples are needed.
The first system I will choose is the oldest: GNU Emacs, which is somewhat of a hybrid between Windows Notepad, the OS kernel, and the International Space Station. It's a bit complicated to explain, but in short, Emacs is a platform created in 1976 (yes, nearly half a century ago) for programming to enhance your productivity, though it masquerades as a text editor.
I use Emacs every single day. Yes, I also use IntelliJ every day; it has already transformed into a powerful tool platform itself. But writing extensions for IntelliJ is a far more ambitious and complex task than writing extensions for Emacs. And more importantly, everything written for Emacs is preserved forever.
I still use software that I wrote for Emacs back in 1995. And I am confident that some modules written for Emacs in the mid-80s, if not earlier, are still in use. Occasionally they might require minor adjustments, but that's really quite rare. I don't know of anything that I've ever written for Emacs (and I've written a lot) that would require a complete architectural overhaul.
Emacs has a function called make-obsolete for deprecated entities. Emacs terminology for fundamental computer concepts (like what a 'window' is) often differs from industry conventions because Emacs introduced them a long time ago. This is a typical danger for those who were ahead of their time: all your terms are incorrect. But Emacs does have a concept of deprecation, which in their jargon is called obsolescence.
But in the world of Emacs, it seems to have a different working definition. A different foundational philosophy, if you will.
In the world of Emacs (and in many other areas we'll discuss below), the status of deprecated APIs mainly means: "You really shouldn't use this approach because, while it works, it suffers from various drawbacks that we'll outline here. But ultimately, it's your choice."
In the world of Google, the status of a deprecated product means: "We are failing our obligations to you." This is indeed the case. Here's what it essentially means. It means they will make you regularly do some work, possibly a lot of work, as a punishment for believing in their : we have the best software. The fastest! You follow all the instructions, run your application or service, and thenābam, a year or two later it breaks.
It's like selling a used car that is guaranteed to break down after 1500 km.
These are two completely different philosophical definitions of 'deprecation'. The Google definition smells of . I don't believe that's actually planned obsolescence in the same way as Apple. But Google definitely plans to break your programs in a roundabout way. I know this because I worked there as a software engineer for over 12 years. They have vague internal guidelines on how much backward compatibility should be observed, but ultimately it depends on each individual team or service. There are no corporate or engineering level recommendations, and the boldest recommendation in terms of deprecation cycles is to "try to give customers 6-12 months to upgrade before breaking their entire system."
The problem is much more serious than they think, and it will persist for many years to come because customer care is not in their DNA. More on this below.
At this point, Iām going to make a bold statement that Emacs is significantly successful and even mostly because they take backward compatibility so seriously. In fact, this is the thesis of our article. Successful long-lived open systems owe their success to micro-communities that have existed for decades around extensions/plugins.This is the ecosystem. I have already pondered the essence of platforms and how important they are, along with the fact that Google has never understood what goes into creating a successful open platform throughout its corporate history, except for Android or Chrome.
Actually, I should briefly mention Android because youāve probably thought about it.
First of all, Android is not Google.They have almost nothing in common. Android is a company that was bought by Google in July 2005, and that company was allowed to operate more or less autonomously and has remained largely untouched over the years. Android is a notorious tech stack and equally notorious prickly organization. As one Googler said, "You canāt just waltz into Android."
In one of my previous articles, I already discussed how bad some of the early design decisions of Android were. Damn, when I wrote that article, they were deploying the crap called "instant apps" which are now (surprise!) , and I sympathize if you were foolish enough to listen to Google and move your content into those instant apps.
But there is a difference, a significant difference, which lies in the fact that the people from Android truly understand how important platforms are; they work hard to maintain the functionality of old Android applications. In fact, their efforts to preserve backward compatibility are so extreme that even during my brief time in the Android division a few years ago, I found myself trying to persuade them to drop support for some of the oldest devices and APIs (I was wrong, as I was in many other things from the past and present. Sorry, Android folks! Now that I've been to Indonesia, I understand why they are important to us).
The people from Android support backward compatibility to almost unimaginable extremes, which piles up a massive amount of technical debt in their systems and toolchains. Oh my, you should see some of the crazy things they have to do in their build system, all in the name of compatibility.
For this, I award Android the coveted title of 'You Are Not Google.' They genuinely do not want to become Google, which is unable to create long-lasting platforms, while Android knows, how to do it. And so Google behaves very wisely in one regard: it allows the people in Android to do things their way.
However, instant applications for Android were quite a foolish idea. And do you know why? Because they required to rewrite and redesign your application! As if people would just go ahead and rewrite two million applications. I assume that the instant apps idea was conceived by some Googler.
But here is the difference. Backward compatibility comes with high costs. Android itself bears the burden of those costs, while Google insists that this burden be borne by you, the paying customer.
You can see Android's commitment to backward compatibility in its APIs. When you have four or five different subsystems performing literally the same task, itās a sure sign that a commitment to backward compatibility is at play. In the world of platforms, this is synonymous with commitment to your clients and your market.
The main problem with Google here is their pride in their engineering hygiene. They dislike having many different ways to achieve the same end, with old, less desirable methods sitting alongside newer, more quirky ones. This increases the learning curve for newcomers to the system, escalates the burden of supporting deprecated APIs, slows down the speed of new features, and the biggest sināit's just not pretty. Google is like the Mad Hatter from Tim Burton's 'Alice in Wonderland':
Mad Hatter:
ā Alice, do you know what I fear the most?
ā The decline of aristocracy?
ā I feared I would have ugly grandchildren..
To understand the compromise between beauty and practicality, letās take a look at the third successful platform (after Emacs and Android) and see how it works: Java itself.
Java has a multitude of deprecated APIs. Deprecation is very popular among Java developers, even more so than in most programming languages. Within Java, the main language and libraries are constantly undergoing API deprecation.
If we take just one of thousands of examples, is considered deprecated. It has been deprecated since the release of Java 1.2 in December 1998. It has been 22 years since it became outdated.
But my actual production code still kills threads every day.Is this a bad thing? Absolutely! I mean, of course, if I were to rewrite the code today, I would implement it differently. But the code for my game, which has made hundreds of thousands of people happy over the past two decades, is written with the thread-closing function that hangs around for too long, and I have never had to change it.I know my system better than anyone, I literally have 25 years of experience working with it in production, and I can say for sure: in my case, closing these specific worker threads is completely harmless.It's not worth the time and effort to rewrite this code, and praise be to Larry Ellison (probably), that Oracle didn't force me to rewrite it.
Perhaps Oracle knows a thing or two about platforms as well. Who knows.
You can find evidence throughout all key Java APIs that are permeated with obsolescence, much like the lines of a glacier in a canyon. In the Java Swing library, you can easily find five or six different keyboard navigation managers (KeyboardFocusManager). It's actually hard to find a Java API that isn't outdated. But they still work! I believe the Java team will only truly remove an API if its interface poses a glaring security issue.
Here's the deal, folks: we software developers are all very busy, and in every software domain, we face competing alternatives. At any given time, programmers using language X are considering language Y as a potential replacement. Oh, you don't believe me? Want to mention Swift? Saying that everyone is migrating to Swift and no one is dropping it, right? Wow, how little you know. Companies are weighing the costs of dual mobile development teams (iOS and Android) ā and they're starting to realize that these cross-platform development systems with funny names like Flutter and React Native actually work, allowing them to cut the size of their mobile teams in half or, conversely, double their productivity. Real money is at stake. Yes, there are trade-offs, but on the other hand, it's about the money.
Let's hypothetically assume that Apple foolishly took a page from Guido van Rossum and announced that Swift 6.0 was not backward compatible with Swift 5.0, much like Python 3 is incompatible with Python 2.
I may have told this story ten years ago, but about fifteen years ago, I went to OāReilly's Foo Camp with Guido, sitting in a tent with Paul Graham and a bunch of big shots. We were sitting in excruciating heat, waiting for Larry Page to arrive on his personal helicopter, while Guido monotonously droned on about 'Python 3000,' which he estimated would take everyone years to migrate to. We kept asking him why he was breaking compatibility, and he replied: 'Unicode.' And we asked if we would have to rewrite our code, what other benefits we would see? And he replied, 'Yoooooooooooooouuuuuuuniiiiiiicoooooooode.'
If you install the Google Cloud Platform SDK ('gcloud'), you will receive the following notification:
Dear Recipient,
Weād like to remind you that support for Python 2 has ended, so go ahead and move on.
ā¦ and so on. The circle of life.
But the fact is that every developer has a choice. And if you force them to rewrite code often enough, they might consider other options. They are not your prisoners, no matter how much you might want that. They are your guests. Python is still a very popular programming language, but, damn it, Python 3(000) has created such a mess within its communities and among its users that the consequences have been stuck for fifteen years. How many Python programs have been rewritten in Go (or Ruby, or some other alternative) due to this backward incompatibility? How much new software has been written in something other than Python, even though it
could have been written in Python if Guido hadnāt burned the whole village down? It's hard to say, but Python has clearly suffered. Itās a huge mess, and everyone loses. So, suppose Apple takes a cue from Guido and breaks compatibility. What do you think will happen next? Well, maybe 80-90% of developers will rewrite their software if they can. In other words, 10-20% of the user base will automatically switch to some competing language, like Flutter.
Do this a few times, and you will lose half of your user base. Just like in sports, in the programming world, current form matters
a lot. . Anyone who loses half their users in five years will be considered a Big Fat Loser. You have to stay on trend in the world of platforms. But this is where the end of support for old versions will eventually be your downfall. Because every time you get rid of some developers, you (a) lose them forever, as they are angry with you for breaking the contract, and (b) hand them over to your competitors.Ironically, I also helped Google become such a diva that ignores backward compatibility when I created Grok, a system for analyzing and understanding source code that makes automation and tooling based on the code easierāitās like an IDE, but here the cloud service stores materialized views of all the billions of lines of Googleās source code in a large data repository.
Ironically, I also helped Google transform into that diva that disregards backwards compatibility when I created Grok, a system for analyzing and understanding source code that facilitates automation and tooling based on the code ā similar to an IDE, but here the cloud service stores materialized views of all billions of lines of Google's source code in a large data warehouse.
Grok has provided Google with a powerful foundation for automated refactoring across the entire codebase (literally throughout Google). The system calculates not only your upstream dependencies (the ones you depend on) but also downstream (those that depend on you), so when you change an API, you know everyone you are breaking! This way, when making changes, you can check that every consumer of your API has updated to the new version, and in reality, often with the tool Rosie they wrote, you can fully automate the process.
This allows Google's codebase to be almost supernaturally 'clean' internally, as these robotic helpers scurry around the house and automatically tidy everything up if they have renamed SomeDespicablyLongFunctionName to SomeDespicablyLongMethodName because someone decided it was an ugly grandchild and needed to be put to sleep.
And, honestly, this works quite well for Googleā¦ internally. I mean, yes, the Go community at Google really does good-naturedly tease the Java community at Google for their continuous refactoring habits. If you are restarting something N times, it means you not only broke it N-1 times, but after a while, it becomes clear that you likely broke it on the N-th attempt as well. But overall, they rise above this frenzy and keep their code 'clean.'
The problems start when they try to impose such an attitude on their cloud clients and users of other APIs.
Iāve introduced you a bit to Emacs, Android, and Java; letās look at the last successful long-lived platform: the Web itself. Can you imagine how many iterations HTTP has gone through since 1995 when we used blinking tags and 'In Development' badges on web pages?
But it still works! And those pages still function! Yes, folks, browsers are world champions of backward compatibility. Chrome is another example of a rare Google platform that has its heads screwed on right, and as you guessed, Chrome effectively operates as an isolated company separate from the rest of Google.
I also want to thank our friends among the operating system developers: Windows, Linux, NOT APPLE GET LOST APPLE, FreeBSD, and so on, for the tremendous work they've done on backward compatibility on their successful platforms (Apple gets at best a C with a minus, as they constantly break everything for no good reason, but somehow the community copes with this in every release, and so far, containers with OS X are not yet fully obsoleteā¦ for now).
But wait, you might say. Aren't we comparing apples to orangesāautonomous software systems on one machine, like Emacs/JDK/Android/Chrome, with multi-server systems and APIs as found in cloud services?
Well, I tweeted about this yesterday in the style of Larry Wall (the creator of the Perl programming languageāed.) based on the principle of "sucks/rules" as I searched for the word deprecated on the developer sites of Google and Amazon. And even though AWS has hundreds of times more service offerings than GCP, Google's developer documentation mentions deprecation about seven times more often.
If someone from Google is reading this, they are surely ready to pull out charts showcasing, like Donald Trumpās style, that they are actually doing everything right, and that I shouldnāt make unfair comparisons like āthe number of mentions of the word deprecated in relation to the number of services.ā
But after so many years, Google Cloud still remains the #3 service (I still haven't written the article about the failed attempt to become #2), but if insiders are to be believed, there are some concerns that they may soon drop to #4.
I have no strong arguments to 'prove' my thesis. All I have are colorful examples I've accumulated over 30 years as a developer. Iāve already mentioned the deeply philosophical nature of this issue; in a sense, itās politicized in developer communities. Some believe that platform creators should care about compatibility, while others believe it's the responsibility users (of the developers themselves). One or the other. And isnāt it indeed a political question when we decide who should bear the costs for common issues?
So this is politics. And there will surely be angry responses to my statement.
How user As a user of AWS for two years while working at Grab, and having experience with Google Cloud, I can say there is a significant difference between the philosophies of Amazon and Google when it comes to priorities. I don't actively develop on AWS, so I'm not very familiar with how often they deprecate old APIs. However, I suspect it's not nearly as frequent as it is with Google. I genuinely believe that this source of ongoing disputes and frustrations in GCP is one of the biggest factors hindering the platform's development.
I realize I haven't named specific GCP systems that have been deprecated. I can say that nearly everything I used, from networking (from the oldest to VPC) to storage (Cloud SQL v1-v2), Firebase (now Firestore with a completely different API), App Engine (let's not even start), Cloud Endpoint, and... I don't know, absolutely all of this forced me to rewrite code at most every 2-3 years, and they never automated the migration for you, and often . It's almost as if it's just meant to be.
And every time I look at AWS, I ask myself, what the hell am I still doing on GCP? They clearly don't want clients. They want buyers. Do you understand the difference? Let me explain.
Google Cloud has a , where people offer their software solutions, and to avoid an empty restaurant effect, they had to fill it with some offerings, so they contracted with Bitnami to create a bunch of solutions that deploy with "one click," or I have to create the "solutions" myself because they don't solve anything. They just exist as checkboxes, as marketing fluff, and Google never cared whether any of the tools actually work. I know product managers who were at the helm, and I can assure you, these people don't care.
Take, for example, the supposedly "one-click" deployment solution . I got so fed up with Google Cloud SQL, that I started considering creating my own Percona cluster as an alternative. And this time, Google seemed to do a good job; they were going to save me some time and effort with a single button!
Well, great, let's go. We'll follow the link and click this button. We choose 'Yes' to agree to all the default parameters and deploy a cluster in our Google Cloud project. Ha-ha, it doesn't work. None of this crap works. The tool was never tested and it started to rot from the first minute, and I wouldn't be surprised if more than half of the 'one-click deployment solutions' (now we understand why the quotes) are useless. at all work. It's an absolute dead end that is better to avoid.
But Google directly invites you to use them. They want you to buy. For them, itās a transaction. They don't want anything support. This is not part of Google's DNA. Yes, engineers support each other, evidenced by my story with Bigtable. But in products and services for regular people, they have been recommendations only ruthless in , that doesn't meet profitability standards, even if it has millions of users.
And this poses a real problem for GCP because this DNA is behind all cloud offerings. They are not inclined to support anything; it's well known that they refuse to host (as a managed service) any third-party software until, AWS does the same and builds a successful business around it, and when customers literally demand the same. However, it takes some effort to get Google to support anything.
This lack of a support culture, combined with the principle of 'let's break things to make them beautiful', alienates developers from them.
And that's not very good if you want to build a long-lasting platform.
Google, wake up, damn it. It's 2020. You're still losing. It's time to take a hard look in the mirror and answer if you really want to stay in the cloud business.
If you want to stay, then stop breaking everything.Guys, you are rich. We, the developers, are not. So when it comes to who will bear the burden of compatibility, you need to take this on. Not us.
Because there are at least three really good clouds out there. They are luring us in.
And now I will go back to fixing all my broken systems. Sigh.
Until next time!
P. S. Update after reading some discussions on this article (the discussions are excellent, by the way). Firebase support hasn't ended, and there are no plans I am aware of. However, there is a frustrating streaming bug that causes the Java client to stop in App Engine. One of their engineers helped me deal with this issue, when I worked at Google, but they never actually fixed the bug, so I have a lousy workaround where I have to restart the GAE application every day. It's been like this for four years! Now they have Firestore. It will take a lot of work to migrate to it since it's a completely different system, and the Firebase bug will never be fixed. What conclusion can be drawn? You can get help, if you work at a company. Iām probably the only one using Firebase on GAE, because I am recording fewer than 100 keys in a 100% native app, and it stops working every few days due to the known bug. What can be said here, but to use it at your own risk? I am switching to Redis.
I have also seen some more experienced AWS users mention that AWS generally never ends support for any services, and SimpleDB is a great example. My assumptions that AWS doesnāt have that disease of discontinuation like Google seems to be valid.
Additionally, I noticed that 20 days ago, the Google App Engine team broke the hosting of a critical Go library, shutting the GAE application out from one of the core Go developers. It really turned out to be silly.
Finally, I've heard that Googlers are already discussing this issue and generally agree with me (love you guys!). But it seems they consider the problem unsolvable because there has never been a proper incentive structure in Googleās culture. I think it would be nice to carve out some time to talk about the absolutely amazing experience of working with AWS engineers when I was at Grab. Hopefully in the future!
Yes, in 2005, they really had different types of shark meat at the massive buffet in building 43, and I liked the meat of the hammerhead sharks the most. However, by 2006, Larry and Sergey got rid of all the unhealthy snacks. So during the Bigtable story in 2007, there were indeed no sharks, and I tricked you shamefully.
When I looked at Google Bigtable four years ago (give or take), the cost was just like that. It seems to have decreased a bit now, but it's still an outrageous amount for an empty data store, especially considering that my first story shows how insignificant an empty big table is at their scale.
I apologize for offending the Apple community and for not saying anything nice about Microsoft, etc. You are all right; I really appreciate all the discussions this article sparked! But sometimes you need to stir the waters a bit to start a conversation, you know?
Thank you for reading.
Update 2, 08/19/2020. Stripe !
Update 3, 08/31/2020. I was contacted by an engineer from Google in the Cloud Marketplace who turned out to be an old friend of mine. He wanted to find out why C2D wasn't working, and eventually, we figured it out: the issue was that I created my network several years ago, and C2D does not work in outdated networks due to the missing subnet parameter in their templates. I think potential GCP users should ensure they have enough knowledgeable engineers at Google...
Source: habr.com
