Brendan Gregg, one of the developers of DTrace, currently developing performance analysis tools based on BPF in the Linux kernel, summarized the experience gained from troubleshooting performance issues that Netflix encountered when migrating the Cassandra database from CentOS to Ubuntu in Amazon EC2 cloud environments running on Xen. After the migration, CPU load increased by 30%, and similar increases in latency were observed during write operations. It turned out that the performance of applications that heavily query time information is highly dependent on the chosen source of precise time in the system. <\/p>\n
Initially, the cause of the performance drop was not obvious, and diagnostics began with tracking the potential impact of continuously running or periodically started resource-intensive system processes using the top and execsnoop utilities. However, everything pointed to increased resource consumption specifically within the Cassandra database, which is written in Java. A comparison of profiling metrics of two Cassandra processes, running in parallel on CentOS and Ubuntu and handling the same queries, showed that about 32% of the total time was spent calling os::javaTimeMillis(), used to obtain information about the current time. <\/p>\n
Following this, an experiment was conducted where a simple application was written in Java that called the method System.currentTimeMillis() one hundred million times in a loop. Running the application showed that it took 13 seconds to execute on CentOS, while on Ubuntu it took about 68 seconds, which means it was 5 times slower. A similar program was written in C, calling the function gettimeofday() one hundred million times, but similar results were obtained during its execution. <\/p>\n
As it became clear that the source of the problem was the function returning the current time, attention shifted to altering metrics by selecting different precise time sources in the system. According to the contents of '\/sys\/devices\/system\/clocksource\/clocksource0\/current_clocksource', the timer 'xen' was used by default when Linux was launched in the guest system. After switching the time source to 'tsc', the execution time of the test application in Ubuntu decreased from 68 to 3.3 seconds, making it 20 times faster. Additionally, a performance test of the time source kvm-clock was conducted, which showed a 20% increase in latencies compared to TSC. $ cat \/sys\/devices\/system\/clocksource\/clocksource0\/available_clocksource xen tsc hpet acpi_pm $ cat \/sys\/devices\/system\/clocksource\/clocksource0\/current_clocksource xen $ time java TimeBench real 1m8.300s user 0m38.337s sys 0m29.875s $ echo tsc > \/sys\/devices\/system\/clocksource\/clocksource0\/current_clocksource $ time java TimeBench real 0m3.370s user 0m3.353s sys 0m0.026s <\/p>\n
To obtain time when selecting the TSC source, the CPU instruction RDTSC is used, the execution of which does not require a system call (the instruction does not require elevated privileges and returns a value from the time counter built into the CPU). By default, TSC is not activated since in the past this source did not eliminate gradual time drift, which in other handlers is corrected programmatically to achieve more accurate readings. According to an engineer specializing in CPU design, concerns about time shifts when using TSC are no longer valid and modern processors<\/a> this source can provide stable readings for years. <\/p>\n The transition of workloads servers<\/a> In Netflix, switching to the TSC source led to a 43% reduction in write latencies and achieved results in Ubuntu that were 4 times better than the configuration with CentOS using the time source 'xen'. The results of the study were forwarded to Amazon, which officially recommended using the TSC time source by default in AWS EC2 environments based on the Xen hypervisor (kvm-clock remains recommended in Nitro hypervisor-based environments). \u0411\u0440\u0435\u043d\u0434\u0430\u043d \u0413\u0440\u0435\u0433 (Brendan Gregg), \u043e\u0434\u0438\u043d \u0438\u0437 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u043e\u0432 DTrace, \u043d\u044b\u043d\u0435 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0449\u0438\u0439 \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0430\u043d\u0430\u043b\u0438\u0437\u0430 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043d\u0430 \u0431\u0430\u0437\u0435 BPF \u0432 \u044f\u0434\u0440\u0435 Linux, \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0439 \u0432 \u0445\u043e\u0434\u0435 \u0440\u0430\u0437\u0431\u043e\u0440\u0430 \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u0441 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c\u044e, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c\u0438 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f Netflix \u0441\u0442\u043e\u043b\u043a\u043d\u0443\u043b\u0430\u0441\u044c \u043f\u0440\u0438 \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u0435 \u0421\u0423\u0411\u0414 Cassandra c CentOS \u043d\u0430 Ubuntu \u0432 \u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u044f\u0445, \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u043c\u044b\u0445 \u0432 \u043e\u0431\u043b\u0430\u043a\u0435 Amazon EC2 \u043d\u0430 \u0431\u0430\u0437\u0435 Xen. \u041f\u043e\u0441\u043b\u0435 \u043c\u0438\u0433\u0440\u0430\u0446\u0438\u0438 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0430 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-101444","post","type-post","status-publish","format-standard","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Analysis of the performance impact of the selected time source in the system | ProHoster","description":"Brendan Gregg, one of the developers of DTrace, is now developing performance analysis tools based on BPF in the Linux kernel. He summarized the experience gained from troubleshooting performance issues.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/news\/analiz-vliyaniya-na-proizvoditelnost-vybrannogo-v-sisteme-istochnika-vremeni","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u043d\u0430\u043b\u0438\u0437 \u0432\u043b\u0438\u044f\u043d\u0438\u044f \u043d\u0430 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u0432\u044b\u0431\u0440\u0430\u043d\u043d\u043e\u0433\u043e \u0432 \u0441\u0438\u0441\u0442\u0435\u043c\u0435 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u0430 \u0432\u0440\u0435\u043c\u0435\u043d\u0438 | ProHoster","og:description":"\u0411\u0440\u0435\u043d\u0434\u0430\u043d \u0413\u0440\u0435\u0433 (Brendan Gregg), \u043e\u0434\u0438\u043d \u0438\u0437 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u043e\u0432 DTrace, \u043d\u044b\u043d\u0435 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0449\u0438\u0439 \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0430\u043d\u0430\u043b\u0438\u0437\u0430 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043d\u0430 \u0431\u0430\u0437\u0435 BPF \u0432 \u044f\u0434\u0440\u0435 Linux, \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0439 \u0432 \u0445\u043e\u0434\u0435 \u0440\u0430\u0437\u0431\u043e\u0440\u0430 \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u0441 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c\u044e, \u0441.","og:url":"https:\/\/prohoster.info\/en\/blog\/news\/analiz-vliyaniya-na-proizvoditelnost-vybrannogo-v-sisteme-istochnika-vremeni","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2021-09-28T14:22:58+00:00","article:modified_time":"2026-02-08T16:56:03+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"101444","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-09-28 14:23:42","updated":"2026-02-09 16:51:02","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/101444","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=101444"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/101444\/revisions"}],"predecessor-version":[{"id":158854,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/101444\/revisions\/158854"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=101444"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=101444"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=101444"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n
Source: opennet.ru<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"