Hello Habr, my name is Ilya, and I work in the platform team at Exness. We develop and implement foundational infrastructure components that are utilized by our product development teams.
In this article, I would like to share the experience of implementing encrypted SNI (ESNI) technology in the infrastructure of public websites.
Utilizing this technology will enhance security levels when operating public websites and align with the internal security standards adopted by the Company.
First of all, I want to highlight that this technology is not standardized and is still in draft form; however, CloudFlare and Mozilla already support it (in ). This motivated us to undertake such an experiment.
A Bit of Theory
ESNI is an extension to the TLS 1.3 protocol that allows encrypting the SNI in the 'Client Hello' message of the TLS handshake. Here’s how a Client Hello with ESNI support looks (instead of the usual SNI, we see ESNI):
To use ESNI, three components are required:
- DNS;
- Client support;
- Server support.
DNS
Two DNS records need to be added - A, and TXT (The TXT record contains the public key that the client can use to encrypt the SNI) – see below. Additionally, there must be support for DoH (DNS over HTTPS), as the available clients (see below) do not activate ESNI support without DoH. This makes sense, as ESNI implies encrypting the resource name we are accessing, which means it is pointless to query DNS via UDP. Moreover, using helps protect against 'cache poisoning' attacks in this scenario.
Currently, several DoH providers are available OpenDNS
CloudFlare record:
A curl 'https://dns.google.com/resolve?name=www.cloudflare.com&type=A' -s -H 'accept: application/dns+json' { "Status": 0, "TC": false, "RD": true, "RA": true, "AD": true, "CD": false, "Question": [ { "name": "www.cloudflare.com.", "type": 1 } ], "Answer": [ { "name": "www.cloudflare.com.", "type": 1, "TTL": 257, "data": "104.17.210.9" }, { "name": "www.cloudflare.com.", "type": 1, "TTL": 257, "data": "104.17.209.9" } ] }
record, the request is formed according to the template
TXT _esni.FQDN _esni.FQDN:
curl 'https://dns.google.com/resolve?name=_esni.www.cloudflare.com&type=TXT'
-s -H 'accept: application/dns+json'
{
"Status": 0,
"TC": false,
"RD": true,
"RA": true,
"AD": true,
"CD": false,
"Question": [
{
"name": "_esni.www.cloudflare.com.",
"type": 16
}
],
"Answer": [
{
"name": "_esni.www.cloudflare.com.",
"type": 16,
"TTL": 1799,
"data": ""/wEUgUKlACQAHQAg9SiAYQ9aUseUZr47HYHvF5jkt3aZ5802eAMJPhRz1QgAAhMBAQQAAAAAXtUmAAAAAABe3Q8AAAA=""
}
],
"Comment": "Response from 2400:cb00:2049:1::a29f:209."
}So, from a DNS perspective, we need to use DoH (preferably with DNSSEC) and add two records.
Client support
If we're talking about browsers, as of now, . Here's how to enable ESNI and DoH support in FireFox. Once the browser is configured, we should see something like this:
to check the browser.
Of course, TLS 1.3 must be used for ESNI support, since ESNI is an extension to TLS 1.3.
For testing the backend with ESNI support, we've implemented a client on go, but more on that later.
Server support
Currently, ESNI is not supported by web servers like nginx/apache, etc., as they work with TLS via OpenSSL/BoringSSL, in which ESNI is not officially supported.
Therefore, we decided to create our front-end component (ESNI reverse proxy) that would support TLS 1.3 termination with ESNI and HTTP(S) traffic proxying to upstreams that do not support ESNI. This allows the technology to be applied in an existing infrastructure without changing the core components – that is, to use current web servers that do not support ESNI.
For clarity, let's provide a diagram:
I should note that the proxy was designed with the ability to terminate TLS connections without ESNI, to support clients without ESNI. Also, the communication protocol with the upstream can be either HTTP or HTTPS with a TLS version lower than 1.3 (if the upstream does not support 1.3). This setup provides maximum flexibility.
The implementation of ESNI support on go we borrowed from . I should note that the implementation itself is quite non-trivial, as it involves changes in the standard library crypto/tls and therefore requires 'patching' GOROOT before building.
To generate ESNI keys, we used (also a product of CloudFlare). These keys are used for SNI encryption/decryption.
We tested the build using go 1.13 on Linux (Debian, Alpine) and MacOS.
A few words about operational features
The ESNI reverse proxy provides metrics in Prometheus format, such as rps, upstream latency & response codes, failed/successful TLS handshakes & TLS handshake duration. At first glance, this seemed sufficient to assess how the proxy handles traffic.
We also conducted load testing before use. The results are below:
wrk -t50 -c1000 -d360s 'https://esni-rev-proxy.npw:443' --timeout 15s
Running 6m test @ https://esni-rev-proxy.npw:443
50 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.77s 1.21s 7.20s 65.43%
Req/Sec 13.78 8.84 140.00 83.70%
206357 requests in 6.00m, 6.08GB read
Requests/sec: 573.07
Transfer/sec: 17.28MB We conducted purely qualitative load testing to compare the configuration using the ESNI reverse proxy with one without. We 'poured' traffic locally to eliminate 'interference' from intermediate components.
So, with ESNI support and upstream HTTP proxying, we achieved around ~ 550 rps from a single instance, with average CPU/RAM consumption of the ESNI reverse proxy:
- 80% CPU Usage (4 vCPU, 4 GB RAM hosts, Linux)
- 130 MB Mem RSS
For comparison, the RPS for the same upstream nginx without TLS termination (HTTP protocol) is ~ 1100:
wrk -t50 -c1000 -d360s 'http://lb.npw:80' --timeout 15s
Running 6m test @ http://lb.npw:80
50 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.11s 2.30s 15.00s 90.94%
Req/Sec 23.25 13.55 282.00 79.25%
393093 requests in 6.00m, 11.35GB read
Socket errors: connect 0, read 0, write 0, timeout 9555
Non-2xx or 3xx responses: 8111
Requests/sec: 1091.62
Transfer/sec: 32.27MB The presence of timeouts indicates there is a resource shortage (we used 4 vCPU, 4 GB RAM hosts, Linux), and the potential RPS is higher (we received numbers up to 2700 RPS on more powerful resources).
In conclusion, I would like to note that the ESNI technology looks quite promising. There are still many open questions, such as storing the public ESNI key in DNS and rotating ESNI keys – these issues are actively discussed, and the latest draft version (as of this writing) of ESNI is already .
Source: habr.com
