Brendan Gregg, one of the developers of DTrace, currently developing performance analysis tools based on BPF in the Linux kernel, summarized the experience gained from troubleshooting performance issues that Netflix encountered when migrating the Cassandra database from CentOS to Ubuntu in Amazon EC2 cloud environments running on Xen. After the migration, CPU load increased by 30%, and similar increases in latency were observed during write operations. It turned out that the performance of applications that heavily query time information is highly dependent on the chosen source of precise time in the system.
Initially, the cause of the performance drop was not obvious, and diagnostics began with tracking the potential impact of continuously running or periodically started resource-intensive system processes using the top and execsnoop utilities. However, everything pointed to increased resource consumption specifically within the Cassandra database, which is written in Java. A comparison of profiling metrics of two Cassandra processes, running in parallel on CentOS and Ubuntu and handling the same queries, showed that about 32% of the total time was spent calling os::javaTimeMillis(), used to obtain information about the current time.
Subsequently, an experiment was conducted in which a simple Java application was written, calling the System.currentTimeMillis() method one hundred million times in a loop. The application run showed that on CentOS, it took 13 seconds to execute, while on Ubuntu it took about 68 seconds, meaning it was 5 times slower. A similar program written in C called the gettimeofday() function one hundred million times, but similar results were obtained during its execution.
As it became clear that the source of the problem was the function returning the current time, attention shifted to changing the parameters when selecting different time sources in the system. According to the contents of "/sys/devices/system/clocksource/clocksource0/current_clocksource", by default, the "xen" timer was used when starting Linux in the guest system. After changing the time source to "tsc", the execution time of the test application in Ubuntu decreased from 68 to 3.3 seconds, making it twenty times faster. Additionally, a performance test of the kvm-clock time source was conducted, which showed an increase in latency by 20% compared to TSC. $ cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen tsc hpet acpi_pm $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen $ time java TimeBench real 1m8.300s user 0m38.337s sys 0m29.875s $ echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource $ time java TimeBench real 0m3.370s user 0m3.353s sys 0m0.026s
To obtain time when selecting the TSC source, the CPU instruction RDTSC is used, the execution of which does not require a system call (the instruction does not require elevated privileges and returns a value from the time counter built into the CPU). By default, TSC is not activated since in the past this source did not eliminate gradual time drift, which in other handlers is corrected programmatically to achieve more accurate readings. According to an engineer specializing in CPU design, concerns about time shifts when using TSC are no longer valid and modern processors this source can provide stable readings for years.
The transition of workloads servers in Netflix to the TSC source led to a 43% reduction in latency when writing and achieved results in Ubuntu that are four times superior to those of configurations using CentOS with the "xen" time source. The results of the study were passed on to Amazon, which officially recommended using the TSC time source by default in AWS EC2 environments based on the Xen hypervisor (in environments based on the Nitro hypervisor, kvm-clock remains recommended).
Source: opennet.ru
