Note. transl.: This article, which became a hit on Medium, is an overview of key (2010-2019) changes in the world of programming languages and the associated technology ecosystem (with a special focus on Docker and Kubernetes). Its original author is Cindy Sridharan, who specializes in developer tools and distributed systems - in particular, she wrote the book “Distributed Systems Observability” - and is quite popular in the Internet space among IT specialists, especially interested in the topic of cloud native.
As 2019 comes to a close, I wanted to share my thoughts on some of the most important technological advances and innovations of the past decade. In addition, I will try to look a little into the future and outline the main problems and opportunities of the coming decade.
I want to make it clear that in this article I do not cover changes in areas such as data science (data science), artificial intelligence, frontend engineering, etc., since I personally do not have sufficient experience in them.
Typification Strikes Back
One of the most positive trends of the 2010s was the revival of statically typed languages. However, such languages never disappeared (C++ and Java are in demand today; they dominated ten years ago), but dynamically typed languages (dynamics) experienced a significant increase in popularity after the emergence of the Ruby on Rails movement in 2005. This growth peaked in 2009 with the open source of Node.js, which made Javascript-on-the-server a reality.
Over time, dynamic languages have lost some of their appeal in the field of creating server software. The Go language, popularized during the container revolution, seemed better suited to creating high-performance, resource-efficient servers with parallel processing (with which the creator of Node.js himself).
Rust, introduced in 2010, included advances in in an attempt to become a safe and typed language. In the first half of the decade, the industry's reception of Rust was rather lukewarm, but its popularity increased significantly in the second half. Notable use cases for Rust include its use for , (we talked about it in - approx. transl.), an early WebAssembly compiler from Fastly (now part of bytecodealliance), etc. With Microsoft considering the possibility of rewriting some parts of the Windows OS in Rust, it is safe to say that this language has a bright future in the 2020s.
Even dynamic languages got new features like (optional types). They were first implemented in TypeScript, a language that allows you to create typed code and compile it into JavaScript. PHP, Ruby and Python have their own optional typing systems (, ), which are successfully used in .
Returning SQL to NoSQL
NoSQL is another technology that was much more popular at the beginning of the decade than at the end. I think there are two reasons for this.
First, the NoSQL model, with its lack of schema, transactions, and weaker consistency guarantees, turned out to be more difficult to implement than the SQL model. IN with the title "Why you should prefer strong consistency whenever possible" (Why you should pick strong consistency, whenever possible) Google writes:
One of the things we've learned at Google is that application code is simpler and development time is shorter when engineers can rely on existing storage to handle complex transactions and keep data in order. To quote the original Spanner documentation, “We believe that it is better for programmers to deal with application performance problems due to transaction abuse as bottlenecks arise, rather than to constantly keep the absence of transactions in mind.”
The second reason is due to the rise of "scale-out" distributed SQL databases (such as и ) in the public cloud space, as well as Open Source alternatives like CockroachDB (we're talking about her too - approx. transl.), which solve many of the technical problems that caused traditional SQL databases to “not scale.” Even MongoDB, once the epitome of the NoSQL movement, is now distributed transactions.
For situations that require atomic reads and writes across multiple documents (across one or more collections), MongoDB supports multi-document transactions. In the case of distributed transactions, transactions can be used across multiple operations, collections, databases, documents, and shards.
Total streaming
Apache Kafka is without a doubt one of the most important inventions of the past decade. Its source code was opened in January 2011, and over the years, Kafka has revolutionized the way businesses work with data. Kafka has been used in every company I've worked for, from startups to large corporations. The guarantees and use cases it provides (pub-sub, streams, event-driven architectures) are used in a variety of tasks, from data storage to monitoring and streaming analytics, in demand in many areas such as finance, healthcare, public sector, retail and etc.
Continuous Integration (and to a lesser extent Continuous Deployment)
Continuous Integration did not appear in the last 10 years, but over the past decade it has spread to such an extent, which became part of the standard workflow (run tests on all pull requests). Establishing GitHub as a platform for developing and storing code and, more importantly, developing a workflow based on means that running tests before accepting a pull request to master is only workflow in development, familiar to engineers who started their careers in the last ten years.
Continuous Deployment (deploying each commit as and when it hits master) is not as widespread as continuous integration. However, with the plethora of different cloud APIs for deployment, the growing popularity of platforms like Kubernetes (which provide a standardized API for deployments), and the emergence of multi-platform, multi-cloud tools like Spinnaker (built on top of those standardized APIs), deployment processes have become more automated, streamlined, and , in general, more secure.
Containers
Containers are perhaps the most hyped, discussed, advertised and misunderstood technology of the 2010s. On the other hand, it is one of the most important innovations of the previous decade. Part of the reason for all this cacophony lies in the mixed signals we were receiving from almost everywhere. Now that the hype has died down a bit, some things have come into sharper focus.
Containers have become popular not because they are the best way to run an application that satisfies the needs of the global developer community. Containers became popular because they successfully fit into a marketing request for a certain tool that solves a completely different problem. Docker turned out to be fantastic a development tool that solves the pressing compatibility issue (“works on my machine”).
More precisely, the revolution was made Docker image, because it solved the problem of parity between environments and provided true portability not only of the application file, but also of all its software and operating dependencies. The fact that this tool somehow spurred the popularity of “containers,” which are essentially a very low-level implementation detail, remains to me perhaps the main mystery of the past decade.
Serverless
I'd wager that the advent of "serverless" computing is even more important than containers because it truly makes the dream of on-demand computing a reality (On-demand). Over the past five years, I've seen the serverless approach gradually expand in scope by adding support for new languages and runtimes. The emergence of products such as Azure Durable Functions seems to be the right step towards the implementation of stateful functions (at the same time a decisive related to FaaS limitations). I will watch with interest how this new paradigm develops in the coming years.
Automation
Perhaps the biggest beneficiary of this trend is the operations engineering community, as it has enabled concepts like infrastructure as code (IaC) to become a reality. Additionally, the passion for automation has coincided with the rise of the “SRE culture,” which aims to take a more software-centric approach to operations.
Universal API-fication
Another interesting feature of the past decade has been the API-fication of various development tasks. Good, flexible APIs allow the developer to create innovative workflows and tools, which in turn help with maintenance and improve the user experience.
In addition, API-fication is the first step towards SaaS-fication of some functionality or tool. This trend also coincided with the rise in popularity of microservices: SaaS has become just another service that can be accessed via API. There are now many SaaS and FOSS tools available in areas such as monitoring, payments, load balancing, continuous integration, alerts, feature switching (feature flagging), CDN, traffic engineering (e.g. DNS), etc., which have flourished in the past decade.
observability
It is worth noting that today we have access to much more advanced tools to monitor and diagnose application behavior than ever before. The Prometheus monitoring system, which received Open Source status in 2015, can perhaps be called лучшей monitoring system from those with which I have worked. It's not perfect, but a significant number of things are implemented in exactly the right way (for example, support for measurements [dimensionality] in the case of metrics).
Distributed tracing was another technology that entered the mainstream in the 2010s, thanks to initiatives such as OpenTracing (and its successor OpenTelemetry). Although tracing is still quite difficult to apply, some of the latest developments give hope that we will unlock its true potential in the 2020s. (Note: Read also in our blog the translation of the article “"by the same author.)
Looking to the future
Unfortunately, there are many pain points that await resolution in the coming decade. Here are my thoughts on them and some potential ideas on how to get rid of them.
Solving the Moore's Law Problem
The end of Dennard's scaling law and the lag behind Moore's law require new innovations. John Hennessy in explains why problem addicts (domain specific) architectures like TPU may be one of the solutions to the problem of lagging behind Moore's Law. Toolkits like from Google already seem to be a good step forward in this direction:
Compilers must support new applications, be easily ported to new hardware, link multiple layers of abstraction ranging from dynamic, managed languages to vector accelerators and software-controlled storage devices, while providing high-level switches for auto-tuning, providing just-in functionality -time, diagnostics, and distributing debugging information about the functioning and performance of systems throughout the stack, while in most cases providing performance that is reasonably close to hand-written assembler. We intend to share our vision, progress, and plans for the development and public availability of such a compilation infrastructure.
CI / CD
While the rise of CI has become one of the biggest trends of the 2010s, Jenkins is still the gold standard for CI.
This space is in dire need of innovation in the following areas:
- user interface (DSL for encoding test specifications);
- implementation details that will make it truly scalable and fast;
- integration with various environments (staging, prod, etc.) to implement more advanced forms of testing;
- continuous testing and deployment.
Developer Tools
As an industry, we have begun to create increasingly complex and impressive software. However, when it comes to our own tools, the situation could be much better.
Collaborative and remote (via ssh) editing gained some popularity, but never became the new standard way of development. If you, like me, reject the very idea of need a permanent connection to the Internet just to be able to do programming, then working via ssh on a remote machine is unlikely to suit you.
Local development environments, especially for engineers working on large service-oriented architectures, are still a challenge. Some projects are trying to solve this, and I'd be interested to know what the most ergonomic UX would look like for a given use case.
It would also be interesting to extend the concept of "portable environments" to other areas of development such as bug reproduction (or ) that occur under certain conditions or settings.
I'd also like to see more innovation in areas like semantic and context-sensitive code search, tools to correlate production incidents with specific parts of the codebase, etc.
Computing (the future of PaaS)
Following the hype around containers and serverless in the 2010s, the range of solutions in the public cloud space has expanded significantly in the last few years.
This raises several interesting questions. First of all, the list of available options in the public cloud is constantly growing. Cloud service providers have the staff and resources to easily keep up with the latest developments in the Open Source world and release products like "serverless pods" (I suspect simply by making their own FaaS runtimes OCI compliant) or others similar fancy things.
One can only envy those who use these cloud solutions. In theory, Kubernetes cloud offerings (GKE, EKS, EKS on Fargate, etc.) provide cloud provider-independent APIs for running workloads. If you use similar products (ECS, Fargate, Google Cloud Run, etc.), you are probably already making the most of the most interesting features offered by the service provider. Additionally, as new products or computing paradigms emerge, migration is likely to be simple and stress-free.
Considering how quickly the range of such solutions is evolving (I will be very surprised if a couple of new options do not appear in the near future), small “platform” teams (teams associated with infrastructure and responsible for creating on-premise platforms for running workloads companies) will be incredibly difficult to compete in terms of functionality, ease of use and overall reliability. The 2010s have seen Kubernetes as a tool for building PaaS (platform-as-a-service), so it seems completely pointless to me to build an internal platform on top of Kubernetes that offers the same choice, simplicity and freedom available in the public cloud space. Framing container-based PaaS as a “Kubernetes strategy” is tantamount to deliberately avoiding the cloud's most innovative capabilities.
If you look at the available today computing capabilities, it becomes obvious that creating your own PaaS based solely on Kubernetes is tantamount to painting yourself into a corner (not a very forward-thinking approach, huh?). Even if someone decides to build a containerized PaaS on Kubernetes today, in a couple of years it will look outdated compared to cloud capabilities. Although Kubernetes started out as an open source project, its ancestor and inspiration is an internal Google tool. However, it was originally developed in the early/mid 2000s when the computing landscape was completely different.
Also, in a very broad sense, companies don't have to become experts at running a Kubernetes cluster, nor do they build and maintain their own data centers. Providing a reliable computing foundation is a core challenge cloud service providers.
Finally, I feel like we've regressed a bit as an industry in terms of interaction experience (). Heroku was launched in 2007 and is still one of the most easy to use platforms. There's no denying that Kubernetes is much more powerful, extensible, and programmable, but I miss how easy it is to get started and deploy to Heroku. To use this platform, you only need to know Git.
All this leads me to the following conclusion: we need better, higher-level abstractions to work (this is especially true for highest level abstractions).
The right API at the highest level
Docker is a great example of the need for better separation of concerns at the same time correct implementation of the highest level API.
The problem with Docker is that (at least) initially the project had too broad goals: all for the sake of solving the compatibility problem (“works on my machine”) using container technology. Docker was an image format, a runtime with its own virtual network, a CLI tool, a daemon running as root, and much more. In any case, the exchange of messages was more confusing, not to mention "lightweight VMs", cgroups, namespaces, numerous security issues and features mixed with the marketing call to "build, deliver, run any application anywhere".
As with all good abstractions, it takes time (and experience and pain) to break down various problems into logical layers that can be combined with each other. Unfortunately, before Docker could reach similar maturity, Kubernetes entered the fray. It monopolized the hype cycle so much that now everyone was trying to keep up with changes in the Kubernetes ecosystem, and the container ecosystem took on a secondary status.
Kubernetes shares many of the same problems as Docker. For all the talk about cool and composable abstraction, separating different tasks into layers not very well encapsulated. At its core, it is a container orchestrator that runs containers on a cluster of different machines. This is a fairly low-level task, applicable only to engineers operating the cluster. On the other hand, Kubernetes is also abstraction of the highest level, a CLI tool that users interact with via YAML.
Docker was (and still is) cool development tool, despite all its shortcomings. In an attempt to keep up with all the “hares” at once, its developers managed to correctly implement abstraction at the highest level. By abstraction at the highest level I mean subset functionality that the target audience (in this case, developers who spent most of their time in their local development environments) were really interested in and that worked great out of the box.
Dockerfile and CLI utility docker should be an example of how to build a good “highest level user experience”. An ordinary developer can start working with Docker without knowing anything about the intricacies implementations that contribute to operational experiencesuch as namespaces, cgroups, memory and CPU limits, etc. Ultimately, writing a Dockerfile is not much different from writing a shell script.
Kubernetes is intended for different target groups:
- cluster administrators;
- software engineers working on infrastructure issues, expanding the capabilities of Kubernetes and creating platforms based on it;
- end users interacting with Kubernetes via
kubectl.
Kubernetes's "one API fits all" approach presents an insufficiently encapsulated "mountain of complexity" with no guidance on how to scale it. All this leads to an unjustifiably protracted learning trajectory. How Adam Jacob, “Docker brought a transformative user experience that has never been surpassed. Ask anyone who uses a K8s if they wish it worked like their first one docker run. The answer will be yes":
I would argue that most infrastructure technology today is too low-level (and therefore considered "too complex"). Kubernetes is implemented at a fairly low level. Distributed tracing in its (a lot of spans stitched together to form a traceview) is also implemented at too low a level. Developer tools that implement the “highest level abstractions” tend to be the most successful. This conclusion holds true in a surprising number of cases (if the technology is too complex or difficult to use, then the “highest level API/UI” for that technology has yet to be discovered).
Right now, the cloud native ecosystem is confusing due to its low-level focus. As an industry, we must innovate, experiment, and educate on what the right level of “maximum, highest abstraction” looks like.
Retail
In the 2010s, the digital retail experience remained largely unchanged. On the one hand, the ease of online shopping should have hit traditional retail stores, on the other hand, online shopping has fundamentally remained almost unchanged in a decade.
While I have no specific thoughts on how this industry will evolve over the next decade, I would be very disappointed if we shop in 2030 the same way we do in 2020.
Journalism
I am increasingly disillusioned with the state of global journalism. It is becoming increasingly difficult to find unbiased news sources that report objectively and meticulously. Very often the line between the news itself and opinions about it is blurred. As a rule, information is presented in a biased manner. This is especially true in some countries where historically there has been no separation between news and opinion. In a recent article published after the last UK general election, Alan Rusbridger, former editor of The Guardian, :
The main point is that for many years I looked at American newspapers and felt sorry for my colleagues there, who were solely responsible for the news, leaving the commentary to completely different people. However, over time, pity turned into envy. I now think that all British national newspapers should separate their responsibility for news from their responsibility for commentary. Unfortunately, it is too difficult for the average reader—especially online readers—to discern the difference.
Given Silicon Valley's rather dubious reputation when it comes to ethics, I would never trust technology to "revolutionize" journalism. That being said, I (and many of my friends) would be glad if there was an impartial, disinterested and trustworthy news source. While I have no idea what such a platform might look like, I am confident that in an era where the truth is becoming increasingly difficult to discern, the need for honest journalism is greater than ever.
Social Networks
Social media and community news platforms are the primary source of information for many people around the world, and the lack of accuracy and reluctance of some platforms to do even basic fact-checking has led to disastrous consequences such as genocide, election interference, and more.
Social media is also the most powerful media tool that has ever existed. They radically changed political practice. They changed the advertising. They changed pop culture (for example, the main contribution to the development of the so-called cancel culture [cultures of ostracism - approx. transl.] social networks contribute). Critics argue that social media has proven to be fertile ground for rapid and capricious changes in moral values, but it has also provided members of marginalized groups with the opportunity to organize in ways they never had before. In essence, social media has changed the way people communicate and express themselves in the 21st century.
However, I also believe that social media brings out the worst human impulses. Consideration and thoughtfulness are often neglected in favor of popularity, and it becomes almost impossible to express reasoned disagreement with certain opinions and positions. Polarization often gets out of control, resulting in the public simply not hearing individual opinions while absolutists control issues of online etiquette and acceptability.
I wonder if it is possible to create a “better” platform that promotes better quality discussions? After all, it is what drives “engagement” that often brings the main profit to these platforms. How Kara Swisher in the New York Times:
It is possible to develop digital interactions without provoking hatred and intolerance. The reason most social media sites seem so toxic is because they were built for speed, virality, and attention, rather than content and accuracy.
It would be truly unfortunate if, in a couple of decades, the only legacy of social media was the erosion of nuance and appropriateness in public discourse.
PS from translator
Read also on our blog:
- «";
- «";
- «";
- «».
Source: habr.com