The faster the development process, the faster the technology company grows.
Unfortunately, modern applications work against us — our systems need to update in real-time without causing interruptions or downtime. Deployment in such systems becomes a complex task and requires sophisticated continuous delivery pipelines even in small teams.
These pipelines are typically narrowly focused, slow, and unreliable. Developers often need to create them manually first and then manage them, which leads companies to hire entire DevOps teams for this purpose.
The speed of these pipelines directly impacts development speed. In the best teams, deployment takes 5 to 10 minutes, but usually, it takes much longer, with each deployment taking several hours.
At Dark, it takes 50 ms. Fifty. Milliseconds.. Dark — , specifically created for continuous delivery, and every aspect of Dark, including the language itself, is built for safe instantaneous deployment.
Why are continuous delivery pipelines so slow?
Let's say we have a Python web application and we've already created a fantastic and modern continuous delivery pipeline. For a developer who works on this project daily, deploying a single minor change will look something like this:
Making changes
- Creating a new branch in git
- Making changes behind the feature toggle
- Modular testing to verify changes with and without the feature toggle
Pull request
- Committing changes
- Pushing changes to the remote repository on GitHub
- Pull request
- The CI build runs automatically in the background
- Code review
- A few more reviews if necessary
- Merging changes with the master branch in git.
CI runs on the master branch
- Installing frontend dependencies via npm
- Building and optimizing HTML+CSS+JS assets
- Running modular and functional tests in the frontend
- Installing Python dependencies from PyPI
- Running modular and functional tests in the backend
- Integration testing on both ends
- Sending frontend assets to the CDN
- Building a container for the Python program
- Sending the container to the registry
- Updating the Kubernetes manifest
Replacing the old code with new
- Kubernetes launches several instances of the new container
- Kubernetes waits for instances to become operational
- Kubernetes adds instances to the HTTP load balancer
- Kubernetes waits for old instances to stop being used
- Kubernetes stops old instances
- Kubernetes repeats these operations until new instances replace all old ones
Enabling the new feature toggle
- New code is enabled only for itself to ensure everything is functioning properly
- New code is enabled for 10% of users, operational and business metrics are tracked
- New code is enabled for 50% of users, operational and business metrics are tracked
- New code is enabled for 100% of users, operational and business metrics are tracked
- Finally, you repeat the entire procedure to remove the old code and toggle
The process depends on the tools, language, and service-oriented architectures used, but generally, it looks like this. I haven't mentioned deployments with database migration because that requires careful planning, but below I'll explain how Dark handles it.
There are many components, and many of them can easily slow down, crash, create temporary contention, or bring down a working system.
And since these pipelines are almost always created for a specific case, they are hard to rely on. Many days occur when code cannot be deployed due to issues in the Dockerfile, a failure in one of the dozens of services, or the required specialist being on vacation.
Worse still, many of these steps do nothing useful at all. They were needed back when we deployed code directly to users, but now we have toggles for new code, and these processes have diverged. As a result, the step where code is deployed (the old is replaced with the new) has now become just an unnecessary risk.
Of course, this is a very well-thought-out pipeline. The team that created it did not spare time and money for a rapid deployment. Typically, deployment pipelines are much slower and less reliable.
Implementing continuous delivery in Dark
Continuous delivery is so important to Dark that we aimed for sub-second times from the very beginning. We examined every step of the pipeline to eliminate unnecessary parts, refining the rest. Here's how we removed the steps.
Jessie Frazelle () coined the term deployless at the Future of Software Development conference in Reykjavik.
We immediately decided that Dark would be based on the concept of 'deployless' (thanks to for the neologism). Deployless means that any code is instantly deployed and ready for use in production. Of course, we won't skip faulty or incomplete code (I will outline the safety principles below).
During Dark demonstrations, we were often asked how we managed to accelerate deployment so much. An odd question. People probably think we invented some sort of super-technology that compares code, compiles it, packages it into a container, launches a virtual machine, cold-starts the container, and everything like that — all in 50 ms. That's unlikely to be possible. But we created a special deployment engine that doesn't require any of that.
Dark launches interpreters in the cloud. Let's say you write code in a function or an HTTP or event handler. We send a diff to the abstract syntax tree (the implementation of the code that our editor and servers internally use) on our servers, and then we run that code when requests come in. So deployment looks just like a simple database record — instantaneous and straightforward. Deployment happens so quickly because it involves the bare minimum.
In the future, we plan to make Dark an infrastructure compiler that will create and run the perfect infrastructure for high performance and reliability of applications. Instant deployment, of course, will remain.
Safe Deployment
Structured Editor
Code in Dark is written in the Dark editor. The structured editor does not allow syntax errors. Essentially, there isn't even a parser in Dark. As you type text, we work directly with the abstract syntax tree (AST), like , , , and .
Any unfinished code in Dark has valid execution semantics, similar to For example, if you change a function call, we keep the old function until the new one is ready to use.
Each program in Dark has its own purpose, so incomplete code doesn’t interfere with the completed code.
Editing Modes
You write code in Dark in two cases. First: you’re writing new code and are the sole user. For instance, it's in REPL, and other users will never access it, or it's a new HTTP route that you don't reference anywhere. Here, you can work without any precautions, and currently, you’re doing just that in the development environment.
The second situation: code is already in use. If traffic runs through the code (functions, event handlers, databases, etc.), caution is necessary. To address this, we block all used code and require the use of more structured tools for editing it. I'll discuss structural tools below: function switches for HTTP and event handlers, a powerful migration platform for databases, and a new version control method for functions and types.
Function Switches
One way in Dark is to solve multiple problems with one solution. Function switches perform many different tasks: replacing the local development environment, Git branches, deploying code, and of course, the traditional slow and controlled release of new code.
Creating and deploying a function switch is done in our editor in one operation. It creates an empty space for new code and provides access controls to both the old and new code, along with buttons and commands for gradually transitioning to the new code or excluding it.
Function switches are built into the Dark language, and even unfinished switches perform their task — if the condition in the switch is not met, the old blocked code will execute.
Development environment
Feature toggles replace the local development environment. Today, teams struggle to ensure everyone uses the same versions of tools and libraries (code formatters, linters, package managers, compilers, preprocessors, testing tools, etc.). With Dark, there's no need to install dependencies locally, manage a local Docker installation, or take other measures to achieve some semblance of parity between development and production environments. , we won't even pretend that we are striving for it.
Instead of creating a cloned local environment, Dark’s toggles create a new sandbox in production that replaces the development environment. In the future, we also plan to create sandboxes for other parts of the application (such as instant database clones), although this doesn't seem that important right now.
Branches and Deployments
There are currently several ways to introduce new code into systems: git branches, deployment stages, and feature toggles. They address one problem in different parts of the workflow: git at the pre-deployment stages, deployment at the moment of switching from old code to new, and feature toggles for controlled releases of new code.
The most effective method is feature toggles (also the simplest to understand and use). They allow complete abandonment of the other two methods. It's particularly useful to eliminate deployments—if we are already using feature toggles to enable code, then the step of migrating servers to new code only introduces unnecessary risks.
Git is hard to use, especially for newcomers, which significantly limits its effectiveness, but it does have convenient branches. We’ve smoothed out many of git’s shortcomings. Dark supports real-time editing and offers collaboration features similar to Google Docs, so you don’t have to send code around and can do rebasing and merging less frequently.
Feature toggles are the foundation of safe deployments. Together with rapid deployments, they allow for quickly testing concepts in small, low-risk fragments instead of applying one large change that could crash the system.
Versioning
To change functions and types, we use versioning. If you want to change a function, Dark creates a new version of that function. You can then call that version using a toggle in the HTTP or event handler. (If this function is deep in the call graph, a new version of each function is created throughout the process. It may seem excessive, but functions aren't affected if you're not using them, so you won't even notice.)
For the same reasons, we version types as well. We talked extensively about our type system. .
Thanks to versioning of functions and types, you can make incremental changes to the application. You can check that each individual handler works with the new version, without having to implement all changes in the applications at once (though we have tools to quickly do so if you want).
This is much safer than deploying everything at once, as is done currently.
New versions of packages and the standard library
When you update a package in Dark, we don't immediately replace the usage of every function or type in the entire codebase. This is unsafe. The code continues to use the same version it was using, and you update the usage of functions and types to the new version for each individual case using toggles.
Screenshot of part of the automation process in Dark, showing two versions of the function Dict::get. Dict::get_v0 returned the type Any (which we are deprecating), while Dict::get_v1 returns the type Option.
We often provide new functions in the standard library and deprecate older versions. Users with older versions in their code will retain access to them, but new users will not be able to obtain them. We plan to provide tools to transition users from old versions to new ones in one step, again using feature toggles.
Dark also provides a unique opportunity: since we run your working code, we can test new versions ourselves by comparing the output of old and new requests to inform you of changes. As a result, package updates, which often occur blindly (or require thorough safety testing), present much less risk and can happen automatically.
New Versions of Dark
The transition from Python 2 to Python 3 stretched over a decade and remains a challenge. Since we are developing Dark for continuous delivery, these language changes must be taken into account.
When we make small changes to the language, we create a new version of Dark. The old code remains in the old version of Dark, while the new code is used in the new version. You can use switches or feature versions to transition to the new version of Dark.
This is particularly useful considering that Dark has only recently emerged. Many changes to the language or library can turn out to be unsuccessful. Incremental versioning of the language allows us to make minor updates, meaning we can take our time and postpone many language decisions until we have more users, and therefore more information.
Database Migrations
There is a standard formula for safe database migration :
- Transform all data into the new format
- Remove old data access methods
- As a result, database migration becomes prolonged and resource-intensive. We accumulate outdated schemas since even simple tasks, like renaming a table or column, do not justify the effort involved.
Dark has an efficient database migration platform that (we hope) simplifies the process so much that you will cease to fear it. All data stores in Dark (key-value pair stores or persistent hash tables) have a type. To migrate a data store, you simply assign it a new type and provide rollback and rollforward functions to convert values between the two types.
Dark has an efficient database migration platform that (we hope) will simplify the process to the point where you no longer fear it. All data stores in Dark (key-value pairs or persistent hash tables) have a type. To migrate a data store, you simply assign it a new type and add rollback and rollback functions for converting values between the two types.
Access to data stores in Dark is done through versioned variable names. For example, the Users data store will initially be named Users-v0. When a new version with a different type is created, the name changes to Users-v1. If data is saved through Users-v0 and you access it via Users-v1, a rollback function is applied. If data is saved through Users-v1 and you access it via Users-v0, a fallback function is applied.
The database migration screen shows the field names of the old database, rollback and fallback expressions, and instructions for enabling the migration.
Use feature switches to route calls to Users-v0 in the Users-v1 version. This can be done one HTTP handler at a time to reduce risks, and the switches work for individual users, so you can check that everything is working as expected. When there are no more Users-v0 users, Dark will automatically convert all remaining data from the old format to the new one in the background. You won’t even notice it.
Testing
Dark is and immutable values, which means its testing surface is significantly smaller compared to object-oriented languages with dynamic typing. However, testing is still necessary.
In Dark, the editor automatically runs unit tests in the background for the edited code and by default executes these tests for all feature switches. In the future, we aim to use static types to automatically fuzz the code to identify bugs.
Additionally, Dark runs your infrastructure in production, which opens new possibilities. We automatically save HTTP requests made in the Dark infrastructure (currently, we save all requests, but plan to switch to sampling later). We test new code against them and conduct unit tests, and if desired, you can easily convert interesting requests into unit tests.
What we have eliminated
Since we don't have deployments but do have feature switches, about 60% of the deployment pipeline is left out. We don't need git branches or pull requests, building backend resources and containers, pushing resources and containers to registries, or deployment steps in Kubernetes.
Comparison of the standard continuous delivery pipeline (on the left) and the Dark continuous delivery (on the right). In Dark, delivery consists of 6 steps and one cycle, while the traditional version includes 35 steps and 3 cycles.
In Dark, there are only 6 steps and 1 cycle in deployment (steps that repeat multiple times), whereas a modern continuous delivery pipeline consists of 35 steps and 3 cycles. In Dark, tests are run automatically, and you don’t even notice it; dependencies are installed automatically; everything related to git or Github is no longer needed; building, testing, and deploying Docker containers is unnecessary; and deployment in Kubernetes is no longer needed.
Even the remaining steps in Dark have become simpler. Since feature switches can be controlled with a single action, there’s no need to go through the entire deployment pipeline again to remove old code.
We have simplified code delivery as much as possible, reducing the time and risks related to continuous delivery. Furthermore, we have significantly simplified package updates, database migrations, testing, version management, dependency installation, equality between development and production environments, and quick and safe language version upgrades.
I answer questions about this on .
To learn more about the Dark device, read the , (or on ) or . If you’re going to StrangeLoop in September, .
Source: habr.com
