Brendan Gregg, one of the developers behind DTrace, who is now developing BPF-based performance analysis tools in the Linux kernel, summarized the experience gained during the analysis of the performance problems that Netflix encountered when migrating the Cassandra database from CentOS to Ubuntu in environments running in the Amazon EC2 cloud powered by Xen. After the migration, the load on the CPU increased by 30% and the latency during write operations increased by about the same amount. As it turned out, the performance of applications that intensively request time information is very dependent on the exact time source chosen in the system.
At first, the reason for the performance degradation was not obvious, and the diagnosis began with monitoring the possible impact of constantly running or periodically running resource-intensive system processes using the top and execsnoop utilities. But everything pointed to the fact that resource consumption increased precisely in the Cassandra DBMS, written in Java. Comparing the profiling scores of two Cassandra processes running in parallel on CentOS and Ubuntu and processing the same requests showed that about 32% of the total time is spent on the os::javaTimeMillis() call, which is used to obtain information about the current time.
After that, an experiment was conducted, during which a simple Java application was written, in a loop calling the System.currentTimeMillis() method a hundred million times. Running the application showed that on CentOS it took 13 seconds to complete, and on Ubuntu it took about 68 seconds, i.e. 5 times slower. A similar program was written in C that called the gettimeofday() function a hundred million times, but when it was run, the same results were obtained.
Since it became clear that the current time return function was the source of the problem, attention turned to the change in indicators when choosing different sources of exact time in the system. Judging by the contents of "/sys/devices/system/clocksource/clocksource0/current_clocksource", by default, when starting Linux in the guest system, the "xen" timer was used. After changing the time source to "tsc", the test application execution time in Ubuntu decreased from 68 to 3.3 seconds, i.e. became 20 times faster. Additionally, a performance test of the kvm-clock time source was carried out, which showed an increase in delays by 20% compared to TSC. $ cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen tsc hpet acpi_pm $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen $ time java TimeBench real 1m8.300s user 0m38.337s sys 0m29.875s $ echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource $ time java TimeBench real 0m3.370s user 0m3.353s sys 0m0.026s
To get the time when selecting the TSC source, the RDTSC processor instruction is used, the execution of which does not require a system call (the instruction does not require elevated privileges and issues a value from the time counter built into the CPU). By default, TSC is not activated, since in the old days this source did not exclude the gradual drift of time, which in other processors is corrected by software to achieve more accurate readings. According to an engineer specializing in processor development, fears about time shifts when using TSC have long been untrue, and in modern processors this source can give stable readings for years.
Switching production servers in Netflix to a TSC source resulted in a 43% reduction in write latency and achieved results using Ubuntu that were 4 times better than CentOS configurations with a "xen" time source. The results of the study were submitted to Amazon, which officially recommended that the default time source TSC be used in AWS EC2 environments based on the Xen hypervisor (kvm-clock remains recommended in environments based on the Nitro hypervisor).
Source: opennet.ru