Article Translation: Envoy threading model — https://blog.envoyproxy.io/envoy-threading-model-a8d44b922310

I found this article quite interesting, and since Envoy is most commonly used as part of 'istio' or simply as an 'ingress controller' for Kubernetes, most people do not have as direct an interaction with it as they do with standard installations like Nginx or Haproxy. However, if something breaks, it would be good to understand how it works internally. I've tried to translate as much text as possible into English, including technical terms, and for those who find such things difficult to look at, I've left the originals in parentheses. Welcome under the cut.

Low-level technical documentation on the Envoy codebase is currently quite scarce. To remedy this, I plan to create a series of blog articles about the various subsystems of Envoy. Since this is the first article, please let me know what you think and what might interest you in the following articles.

One of the most common technical questions I receive about Envoy is a request for a low-level description of the threading model used. In this post, I will describe how Envoy maps connections to threads, as well as an overview of the Thread Local Storage system used internally to make the code more parallel and high-performance.

Threading Overview

Envoy uses three different types of threads:

Main: This thread manages the startup and shutdown of the process, all XDS (xDiscovery Service) API handling, including DNS, health checking, overall cluster management and service runtime, statistic resets, administration, and general process management — Linux signals, hot restart, etc. Everything that happens in this thread is asynchronous and 'non-blocking'. Overall, the main thread coordinates all critical functionality processes that do not require a lot of CPU. This allows much of the management code to be written as if it were single-threaded.
Worker: By default, Envoy creates a worker thread for each hardware thread in the system, which can be controlled using the option --concurrency. Each worker thread runs a 'non-blocking' event loop, which is responsible for listening to each listener. As of the writing of this article (July 29, 2017), there is no segmentation of listeners, handling new connections, creating an instance of the filter stack for the connection, and processing all input-output (IO) operations for the duration of the connection. Again, this allows most of the connection handling code to be written as if it were single-threaded.
File flusher: Each file that Envoy writes, primarily access logs, currently has an independent blocking thread. This is because writing to files cached by the filesystem can sometimes block, even when using O_NONBLOCK . When worker threads need to write to a file, the data is actually moved to a memory buffer, where it is ultimately flushed through the file flush. This is one area of the code where technically all worker threads can block the same lock while trying to fill the memory buffer.

Connection handling

As briefly discussed above, all worker threads listen to all listeners without any segmentation. Thus, the kernel is used to efficiently pass accepted sockets to worker threads. Modern kernels are generally very good at this; they use features like I/O prioritization to attempt to fill a thread with work before starting to use other threads that also listen on the same socket, and they avoid using spinlocks to handle each request.
Once a connection is accepted on a worker thread, it never leaves that thread. All further connection processing is handled entirely within the worker thread, including any forwarding behavior.

This has several important implications:

All connection pools in Envoy belong to the worker thread. Thus, although HTTP/2 connection pools maintain only one connection to each upstream host at a time, if there are four worker threads, there will be four HTTP/2 connections to the upstream host in a steady state.
The reason Envoy operates this way is that by keeping everything in one worker thread, almost all of the code can be written without locks and as if it were single-threaded. This design simplifies the writing of a lot of code and scales incredibly well for almost unlimited numbers of worker threads.
However, one of the key takeaways is that, in terms of efficiency, configuring the parameter is actually very important. --concurrencyHaving more worker threads than necessary will result in memory waste, the creation of more idle connections, and a decrease in connection pool hit rate. At Lyft, our Envoy sidecar containers operate with very low parallelism, so performance roughly matches the services they sit next to. We run Envoy as an edge proxy only under maximum concurrency.

What non-blocking means

The term 'non-blocking' has been used several times in discussions about how the main and worker threads operate. All code is written under the assumption that nothing will ever be blocked. However, this is not entirely accurate.

Envoy uses several long process locks:

As mentioned, when logging access, all worker threads acquire the same lock before filling the log buffer in memory. The lock hold time should be very low, but it is possible that this lock will be contended under high parallelism and high throughput.
Envoy uses a very complex system for processing statistics that is local to the stream. This will be the topic of a separate post. However, I will briefly mention that as part of the local processing of stream statistics, it may sometimes be necessary to acquire a lock for the central 'statistics repository.' This lock should never be needed.
The main thread periodically needs to coordinate with all worker threads. This is done by 'publishing' from the main thread to the worker threads, and sometimes from the worker threads back to the main thread. A lock is required for sending so that the published message can be queued for later delivery. These locks should never be subject to serious contention, but they can technically still be blocked.
When Envoy writes logs to the system error stream (standard error), it acquires a lock for the entire process. Generally, local logging in Envoy is considered terrible in terms of performance, so not much effort is dedicated to improving it.
There are a few other incidental locks, but none of them are critical for performance and should never be contested.

Thread local storage

Due to the way Envoy separates the responsibilities of the main thread from those of the worker threads, there is a requirement that complex processing can be performed in the main thread and then provided to each worker thread with a high degree of parallelism. This section describes the Envoy Thread Local Storage (TLS) system at a high level. In the next section, I will describe how it is used for cluster management.

As already described, the main thread handles virtually all management functions and control plane functionality within the Envoy process. The control plane here is somewhat overloaded, but when considering it within the context of the Envoy process itself and comparing it to the forwarding done by the worker threads, it seems reasonable. Generally, the main thread does some work and then needs to update each worker thread according to the results of that work. At the same time, the worker thread does not need to acquire a lock on each access..

The TLS (Thread Local Storage) system in Envoy works as follows:

Code executing in the main thread can allocate a TLS slot for the entire process. Although this is abstracted, in practice, it is an index in a vector that provides O(1) access.
The main thread can store arbitrary data in its slot. Once this is done, the data is published in each worker thread as a regular event in the event loop.
Worker threads can read from their TLS slot and retrieve any thread-local data available there.

While this is a very simple and incredibly powerful paradigm, it is very similar to the concept of RCU (Read-Copy-Update) locking. Essentially, worker threads never see any changes to data in TLS slots while executing work. Changes only occur during idle periods between worker events.

Envoy uses this in two different ways:

By maintaining different data in each worker thread, access to this data is accomplished without any locking.
By maintaining a shared pointer to global data in read-only mode in each worker thread. This way, each worker thread has a reference count to the data that cannot be reduced during work execution. Only when all workers are idle and load new shared data will the old data be destroyed. This is identical to RCU.

Cluster Update Threading

In this section, I will describe how TLS (Thread Local Storage) is used for cluster management. Cluster management includes handling xDS API and/or DNS, as well as health checking.

Cluster thread management includes the following components and stages:

The cluster manager is a component within Envoy that manages all known upstreams in the cluster, the CDS (Cluster Discovery Service) API, the SDS (Secret Discovery Service) interfaces, and the EDS (Endpoint Discovery Service), DNS, and active external health checks. It is responsible for creating an 'eventually consistent' view of each upstream in the cluster, which includes discovered hosts as well as their health status.
The health checker performs active checks and reports changes in status to the cluster manager.
CDS (Cluster Discovery Service) / SDS (Secret Discovery Service) / EDS (Endpoint Discovery Service) / DNS are executed to determine cluster membership. State changes are returned to the cluster manager.
Each worker thread continuously executes an event processing loop.
When the cluster manager detects that the state for the cluster has changed, it creates a new read-only snapshot of the cluster state and sends it to each worker thread.
During the next idle period, the worker thread will update the snapshot in the dedicated TLS slot.
During an I/O event that must determine the host for load balancing, the load balancer will request the TLS (Thread local storage) slot for host information. No locking is required for this. It should also be noted that TLS may initiate events during updates, allowing load balancing subsystems and other components to recalculate caches, data structures, etc. This is beyond the scope of this post, but is utilized in various parts of the code.

Using the aforementioned procedure, Envoy can handle each request without any locks (except for those previously described). Aside from the complexity of the TLS code itself, most of the code does not need to understand how multithreading works, and can be written in a single-threaded manner. This simplifies writing most of the code in addition to excellent performance.

Other subsystems that make use of TLS

TLS (Thread local storage) and RCU (Read Copy Update) are widely used in Envoy.

Examples of usage:

Mechanism for changing functionality at runtime: The current list of enabled functionality is computed in the main thread. Each worker thread is then provided with a read-only snapshot using RCU semantics.
Route table replacementFor the route tables provided by RDS (Route Discovery Service), the route tables are created in the main thread. A read-only snapshot will then be provided to each worker thread using RCU (Read Copy Update) semantics. This makes modifying route tables atomically efficient.
HTTP Header Caching: It turns out that calculating the HTTP header for each request (when executing ~25K+ RPS per core) is quite costly. Envoy centrally computes the header about every half second and provides it to each worker via TLS and RCU.

There are other scenarios, but the previous examples should give a good understanding of why TLS is used.

Known Performance Pitfalls

Although Envoy generally performs well, there are several known areas that need attention when it is used with very high parallelism and throughput:

As already described in this article, currently all worker threads are blocked when writing to the access log memory buffer. With high parallelism and throughput, it will be necessary to batch access logs for each worker thread at the cost of unordered delivery when writing to the final file. As an alternative, a separate access log can be created for each worker thread.
Although the statistics are highly optimized, at very high parallelism and throughput, there will likely be atomic contention on individual statistics. The solution to this problem is to use counters for one worker thread with periodic resets of central counters. This will be discussed in a subsequent post.
The existing architecture will not work well if Envoy is deployed in a scenario with very few connections requiring significant resources to handle. There is no guarantee that connections will be evenly distributed among worker threads. This can be addressed by implementing connection load balancing, which would enable connection sharing between worker threads.

Conclusion

The Envoy threading model is designed to ensure ease of programming and massive parallelism through potentially excessive memory and connection usage if not configured properly. This model allows it to perform exceptionally well at very high thread counts and throughput.
As I briefly mentioned on Twitter, the design can also operate on top of a fully functional user-space network stack, like DPDK (Data Plane Development Kit), enabling regular servers to process millions of requests per second with full L7 processing. It will be very interesting to see what gets built in the coming years.
One last quick note: I've been asked many times why we chose C++ for Envoy. The reason remains that it is still the only widely used industry-grade language for building the architecture described in this post. C++ is definitely not suitable for everyone or even for many projects, but for certain use cases, it is still the only tool to get the job done.