The QUIC protocol is extremely interesting to observe, which is why we love writing about it. However, while previous publications on QUIC focused more on its historical and technical aspects, today we are pleased to publish a different type of translation – it will discuss the real-world application of the protocol in 2019. And this isn’t about a small infrastructure based in a hypothetical garage, but about Uber, which operates in almost every corner of the world. How the company's engineers came to the decision to use QUIC in production, how they conducted tests, and what they observed after deploying it in production – is discussed below.
The images are clickable. Happy reading!
Uber operates on a global scale, specifically in 600 cities, each relying on wireless internet from over 4500 mobile operators. Users expect the application to function not just quickly, but in real time – to ensure this, the Uber application requires low latency and a highly reliable connection. Unfortunately, the stack does not perform well in dynamic and loss-prone wireless networks. We realized that, in this case, low performance is directly related to TCP implementations in operating system kernels.
To address the issue, we implemented , a modern protocol with channel multiplexing that gives us more control over transport protocol performance. Currently, the working group is standardizing QUIC as .
After thorough testing, we concluded that implementing QUIC in our application would reduce tail latencies compared to TCP. We observed a reduction in the range of 10-30% for HTTPS traffic in the case of both driver and passenger applications. Additionally, QUIC provided us with end-to-end control over user packets.
In this article, we share our experience optimizing TCP for Uber applications using a stack that supports QUIC.
The latest in technology: TCP
Today, TCP is the most widely used transport protocol for delivering HTTPS traffic over the Internet. TCP ensures a reliable stream of bytes, thus handling network congestion and link-layer losses. The widespread use of TCP for HTTPS traffic is explained by its ubiquity (almost every OS contains TCP), availability on most infrastructure (for instance, on load balancers, HTTPS proxies, and CDNs), and the out-of-the-box functionality that is accessible on nearly all platforms and networks.
Most users utilize our application on the go, and the 'tail' latencies of TCP have been far from the requirements of our real-time HTTPS traffic. Simply put, users around the world have encountered this issue – Figure 1 reflects the latencies in major cities:
Figure 1. The extent of 'tail' latencies varies across major cities where Uber operates.
Although latencies in Indian and Brazilian networks were higher than those in the USA and the UK, tail latencies are significantly greater than average latencies. This holds true even for the USA and the UK.
TCP Performance Over Wireless
TCP was designed for wired networks, focusing on well-predictable links. However, wireless networks have their own characteristics and challenges. Firstly, wireless networks are prone to losses due to interference and signal attenuation. For instance, Wi-Fi networks are sensitive to microwaves, Bluetooth, and other radio signals. Cellular networks suffer from signal loss () due to reflection/absorption of the signal by objects and buildings, as well as from from neighboring This leads to more significant (4-10 times) and varied and packet losses compared to a wired connection.
To combat fluctuations in bandwidth and losses, cellular networks typically utilize large buffers to handle traffic spikes. This can lead to excessive queuing, which means longer delays. Very often, TCP interprets such queuing as losses due to an increased timeout, causing TCP to retransmit and thereby filling the buffer. This issue is known as (), and it is a very the modern internet.
Finally, the performance of the cellular network varies depending on the carrier, region, and time. In Figure 2, we have gathered median HTTPS latency data for cells within a range of 2 kilometers. The data was collected for two major cellular operators in Delhi, India. As can be seen, performance varies from cell to cell. Additionally, the performance of one operator differs from that of the second. Factors such as time and location entry patterns, user mobility, and the network infrastructure concerning tower density and network type ratio (LTE, 3G, etc.) play a role in this.
Figure 2. Latencies in a 2-kilometer radius. Delhi, India.
The performance of cellular networks also changes over time. In Figure 3, we show the median latency by days of the week. We also observed differences on a smaller scale—within a single day and hour.
Figure 3. Tail latencies can vary significantly on different days, even for the same operator.
All of the above leads to inefficiencies in TCP performance over wireless networks. However, before seeking alternatives to TCP, we wanted to develop a precise understanding of the following points:
- Is TCP the main culprit of tail latencies in our applications?
- Do modern networks experience significant and diverse round-trip delays (RTT)?
- What is the impact of RTT and losses on TCP performance?
TCP performance analysis
To understand how we analyzed TCP performance, let's briefly recall how TCP transmits data from the sender to the receiver. Initially, the sender establishes a TCP connection by performing a three-way If a packet or ACK is lost, the sender retransmits it after a timeout (RTO,
retransmission timeout ). RTO is calculated dynamically based on various factors, such as the expected RTT latency between sender and receiver.
Figure 4. Packet exchange over TCP/TLS involves retransmission mechanisms.
To determine how TCP functioned in our applications, we tracked TCP packets using over a week on live traffic coming from Indian border servers. We then analyzed TCP connections using . Additionally, we created an Android application that sends emulated traffic to a test server, closely mimicking real traffic. Smartphones with this application were distributed to several employees who collected logs over several days.
The results of both experiments were consistent with each other. We observed high RTT delays; tail values were nearly 6 times the median; the mean delay exceeded 1 second. Many connections experienced losses, causing TCP to retransmit 3.5% of all packets. In congested areas, such as airports and train stations, we noted a 7% loss rate. These findings challenge the common belief that the advanced retransmission schemes used in cellular networks Network metrics
RTT, milliseconds [50%, 75%, 95%, 99%]
Values
RTT deviation, seconds
[350, 425, 725, 2300]
On average ~1.2 s
Packet loss in unstable connections
On average ~3.5% (7% in congested areas)
Almost half of these connections experienced at least one packet loss, primarily involving SYN and SYN-ACK packets. Most TCP implementations use an RTO value of 1 second for SYN packets, which increases exponentially for subsequent losses. Application loading times can increase because TCP requires more time to establish connections.
In the case of data packets, high RTO values significantly reduce the useful network utilization in the presence of temporary losses in wireless networks. We found that the average retransmission time is around 1 second with a tail delay of nearly 30 seconds. Such high TCP-level delays caused HTTPS timeouts and retries, further increasing latency and network inefficiency.
In the case of data packets, high RTO values significantly reduce the effective network utilization in the presence of temporary losses in wireless networks. We found that the average retransmission time is about 1 second with a tail delay of nearly 30 seconds. Such high TCP-level delays caused HTTPS timeouts and retransmissions, further increasing latency and network inefficiency.
While the 75th percentile of measured RTT was around 425 ms, the 75th percentile for TCP was nearly 3 seconds. This suggests that losses caused TCP to make 7-10 passes to successfully transmit data. This may be due to ineffective RTO calculation or TCP's inability to quickly respond to packet loss. in the window and the inefficiency of the congestion control algorithm, which does not distinguish between wireless losses and losses due to network congestion. Below are the test results for TCP losses:
TCP Packet Loss Statistics
Value
Percentage of connections with at least 1 packet loss
45%
Percentage of connections with losses during the connection establishment
30%
Percentage of connections with losses during data exchange
76%
Delay distribution in retransmission, seconds [50%, 75%, 95%, 99%]
[1, 2.8, 15, 28]
Distribution of the number of retransmissions for a single packet or TCP segment
[1,3,6,7]
Application of QUIC
Originally designed by Google, QUIC is a modern multistreaming transport protocol that operates over UDP. Currently, QUIC is (we have already mentioned that there are actually two versions of QUIC, the curious – translator's note). As shown in Figure 5, QUIC is positioned under HTTP/3 (essentially, HTTP/2 over QUIC is HTTP/3, which is currently undergoing extensive standardization). It partially replaces the HTTPS and TCP layers, using UDP for packet formation. QUIC supports only secure data transmission, as TLS is fully integrated into QUIC.
Figure 5: QUIC operates under HTTP/3, replacing TLS, which previously operated under HTTP/2.
Below we present the reasons that convinced us to use QUIC to enhance TCP:
- 0-RTT connection establishment. QUIC allows the reuse of authorizations from previous connections, reducing the number of security handshakes. In the future, will support 0-RTT, however, the three-way TCP handshake will still be mandatory.
- overcoming HoL-blocking. HTTP/2 uses a single TCP connection for each client to improve performance, but this can lead to HoL (head-of-line) blocking. QUIC simplifies multiplexing and delivers requests to the application independently of each other.
- congestion control. QUIC operates at the application level, making it easier to update the main transport algorithm that manages sending based on network parameters (packet loss or RTT). Most TCP implementations use the , which is not optimal for latency-sensitive traffic. Newly developed algorithms like model the network more accurately and optimize delays. QUIC allows the use of BBR and the updating of this algorithm as it .
- loss recovery. QUIC triggers two TLP () before RTO activates – even when losses are quite noticeable. This differs from TCP implementations. TLP primarily retransmits the last packet (or a new one if available) to initiate fast recovery. Handling tail loss is particularly beneficial for how Uber interacts with the network, specifically for short, sporadic, and delay-sensitive data transmissions.
- optimized ACK. Since each packet has a unique sequence number, there is no issue with packets during retransmission. ACK packets also include the time taken to process the packet and generate the ACK on the client side. These features ensure that QUIC calculates RTT more accurately. ACK in QUIC supports up to 256 ranges of , helping the sender be more resilient to packet reordering and use fewer bytes in the process. Selective ACK () in TCP does not resolve this issue in all cases.
- connection migration. QUIC connections are identified by a 64-bit ID, so if a client changes IP addresses, they can continue using the ID of the old connection on the new IP address without interruptions. This is a very common practice for mobile applications when a user switches between Wi-Fi and cellular connections.
Alternatives to QUIC
We considered alternative approaches to solving the problem before selecting QUIC.
First, we attempted to deploy TPC PoPs (Points of Presence) to terminate TCP connections closer to users. Essentially, PoPs terminate the TCP connection with the mobile device nearer to the cellular network and proxy the traffic to the original infrastructure. By terminating TCP closer, we can potentially reduce RTT and ensure that TCP responds more actively to the dynamic wireless environment. However, our experiments showed that for the most part, RTT and losses stem from cellular networks, and using PoPs does not provide a significant performance improvement.
We also explored tuning TCP parameters. Configuring the TCP stack on our heterogeneous edge servers was challenging, as TCP has inconsistent implementations across different OS versions. It was difficult to implement and test various network configurations. Directly tuning TCP on mobile devices was impossible due to lack of permissions. More importantly, features like 0-RTT connections and improved RTT prediction are critically important to the protocol architecture, which means substantial advantages cannot be gained simply by tuning TCP.
Finally, we evaluated several UDP-based protocols that address video streaming issues – we wanted to see if these protocols would help in our case. Unfortunately, they significantly lacked many security settings and required an additional TCP connection for metadata and control information.
Our research indicated that QUIC is arguably the only protocol that can help with Internet traffic issues while also considering both security and performance.
Integrating QUIC into the platform
To successfully integrate QUIC and enhance application performance under poor connectivity conditions, we replaced the old stack (HTTP/2 over TLS/TCP) with the QUIC protocol. We utilized the network library from , which contains the original Google version of the protocol – gQUIC. This implementation is also continuously improved to follow the latest IETF specification.
First, we integrated Cronet into our Android applications to add QUIC support. The integration was designed to minimize migration costs. Instead of fully replacing the old network stack that used the library , we integrated Cronet UNDER the OkHttp API framework. By performing the integration this way, we avoided changes to our network calls (which use ) at the API level.
Similarly to our approach with Android devices, we implemented Cronet in Uber's iOS applications by intercepting HTTP traffic using , using . This abstraction, provided by the iOS Foundation, handles protocol-specific URL data and ensures that we can integrate Cronet into our iOS applications without significant migration costs.
QUIC termination on Google Cloud load balancers
On the backend, QUIC termination is provided by Google Cloud Load balancing infrastructure, which utilizes headers in responses to support QUIC. Generally, for each HTTP request, the load balancer adds an alt-svc header which validates QUIC support for the domain. When the Cronet client receives an HTTP response with such a header, it uses QUIC for subsequent HTTP requests to that domain. Once the load balancer terminates QUIC, our infrastructure explicitly sends this action via HTTP2/TCP to our data centers.
Performance: results
Output performance is the main reason for our search for a better protocol. To start, we created a stand with , to see how QUIC behaves under different network profiles. To test QUIC in real networks, we conducted experiments driving around New Delhi, using emulated network traffic that closely resembled HTTP calls in the passenger app.
Experiment 1
Inventory for the experiment:
- Android test devices with OkHttp and Cronet stacks to ensure we direct HTTPS traffic over TCP and QUIC respectively;
- a Java-based emulation server that sends consistent HTTPS headers in responses and loads client devices to receive requests from them;
- cloud proxies that are physically located close to India to terminate TCP and QUIC connections. For TCP termination, we used a reverse proxy on , it was difficult to find an open-source reverse proxy for QUIC. We built a reverse proxy for QUIC ourselves, using the basic QUIC stack from Chromium and its implementation in Chromium as open-source.
Figure 6. The testbed for TCP vs QUIC consisted of Android devices with OkHttp and Cronet, cloud proxies for connection termination, and an emulation server.
Experiment 2
When Google made QUIC available through , we used the same inventory but with one modification: instead of NGINX, we used Google’s load balancers for terminating TCP and QUIC connections from devices, as well as routing HTTPS traffic to the emulation server. The load balancers are distributed globally but utilize the nearest PoP server to the device (thanks to geolocation).
Figure 7. In the second experiment, we wanted to compare the latency of TCP and QUIC termination: using Google Cloud and our cloud proxy.
Ultimately, we were faced with several revelations:
- termination through PoP improved TCP performance. As the load balancers terminate TCP connections closer to users and are highly optimized, it results in lower RTT, which enhances TCP performance. And while QUIC was less affected by this, it still outperformed TCP in terms of tail latency reduction (by 10-30 percent).
- tail latencies are influenced by . Although our QUIC proxy was farther from the devices (with a latency around 50 ms higher) than Google’s load balancers, it delivered similar performance – a 15% reduction in latencies compared to a 20% reduction in the 99th percentile for TCP. This indicates that the last-mile transition is a bottleneck in network performance.
Figure 8. The results of the two experiments show that QUIC significantly outperforms TCP.
Production traffic
Inspired by the experiments, we implemented QUIC support in our Android and iOS applications. We conducted A/B testing to determine the impact of QUIC in cities where Uber operates. Overall, we observed a significant reduction in tail latencies across both regions and mobile operators and types of networks.
The graphs below illustrate the percentage improvements in tails (95th and 99th percentiles) across macro-regions and different types of networks – LTE, 3G, 2G.
Figure 9. In live tests, QUIC outperformed TCP in terms of latency.
Onward
Perhaps this is just the beginning – the rollout of QUIC into production has provided incredible opportunities to enhance application performance in both stable and unstable networks, namely:
Increased coverage
Analyzing the performance of the protocol on real traffic, we found that approximately 80% of sessions successfully utilized QUIC for all requests, while 15% of sessions used a combination of QUIC and TCP. We suspect this combination arose because the Cronet library switches back to TCP on timeout, as it cannot distinguish between real UDP failures and poor network conditions. We are currently seeking a solution to this issue as we work on further implementing QUIC.
QUIC Optimization
Traffic from mobile applications is sensitive to delays, but not to bandwidth. Additionally, our applications are primarily used in cellular networks. Based on experiments, tail delays are still significant, even with proxies terminating TCP and QUIC close to users. We are actively exploring ways to improve congestion management and enhance the effectiveness of QUIC loss recovery algorithms.
With these and several other improvements, we plan to enhance the user experience regardless of network and region, making convenient and seamless packet transport more accessible globally.
Source: habr.com
