How to ensure that time itself does not lie when you have millions of large and small devices interacting via TCP/IP? Each of them has a clock, and the time must be accurate across all of them. This problem cannot be solved without NTP.
Imagine for a moment that one segment of the industrial IT infrastructure is facing difficulties with time synchronization among services. The enterprise software cluster stack immediately starts to fail, domains break down, and master and standby nodes are unsuccessfully striving to restore the status quo.
There is also the possibility that a malicious actor deliberately attempts to disrupt time through a MiTM or DDoS attack. In such a scenario, anything can happen:
- user account passwords may expire;
- X.509 certificate validity may lapse;
- TOTP two-factor authentication may stop functioning;
- backups may become 'outdated' and the system may delete them;
- DNSSEC may break.
It is clear that every IT department is concerned about the reliable operation of time synchronization services, and it would be beneficial if they were both reliable and secure in industrial deployment.
Breaking NTP in 25 minutes
Network protocols — millennials have one particularity; they have long been and are no longer suitable, but replacing them is not so easy even when a critical mass of enthusiasts and funding is gathered.
The main complaint against classic NTP is the lack of reliable protection mechanisms against attacks from malicious actors. Various attempts have been made to solve this problem. Initially, a pre-shared key (PSK) mechanism was implemented for symmetric key exchange.
Unfortunately, this method did not prove effective due to a simple reason — it does not scale well. Manual configuration is required on the client side depending on the server. This means that you can't simply add another client like that. If something changes on the NTP server, all clients need to be reconfigured.
Then AutoKey was invented, but a number of serious vulnerabilities were immediately discovered in the design of the algorithm, leading to its abandonment. The issue is that the initial number (seed) contains only 32 bits, which is too small and does not provide enough computational complexity for a brute force attack.
- Key ID — a symmetric 32-bit key;
- MAC (message authentication code) — checksum of the NTP packet;
Autokey is calculated as follows.
Autokey=H(Sender-IP||Receiver-IP||KeyID||Cookie)Where H() — is a cryptographic hash function.
The same function is used to calculate the checksum for packets.
MAC=H(Autokey||NTP packet)Thus, the integrity of packet checks relies on the authenticity of the cookies. By taking them, one can restore the autokey and then forge the MAC. However, the NTP server uses a seed during their generation. This is where the trick lies.
Cookie=MSB_32(H(Client IP||Server IP||0||Server Seed))The MSB_32 function cuts off the 32 most significant bits from the result of the MD5 hash calculation. The client cookie does not change as long as the server parameters remain unchanged. The attacker only needs to restore the seed to be able to generate cookies independently.
First, one must connect to the NTP server as a client and obtain cookies. After that, by brute force, the attacker recovers the seed using a simple algorithm.
Brute force algorithm for calculating the seed.
for i=0:2^32 − 1 do
Ci=H(Server-IP||Client-IP||0||i)
if Ci=Cookie then
return i
end if
end forThe IP addresses are known, so one only needs to create 2^32 hashes until the created cookie matches the one obtained from the NTP server. On a regular home station with an Intel Core i5, this will take about 25 minutes.
NTS — new Autokey
It was impossible to tolerate such security holes in Autokey, and in 2012 a new protocol was introduced. To address the compromised name, rebranding was done, and Autokey v.2 was renamed to Network Time Security.
The NTS protocol is an extension of NTP security and currently supports only unicast mode. It provides reliable cryptographic protection against packet tampering, prevents tracking, scales well, is resilient to packet loss, and results in minimal accuracy loss during connection protection.
An NTS connection consists of two stages, utilizing lower-level protocols. At the first stage, the client and server agree on various connection parameters and exchange cookies containing keys along with all the accompanying data. At the second stage, the actual secure NTS session takes place between the client and the NTP server.
NTS consists of two lower-level protocols: Network Time Security Key Exchange (NTS-KE), which initializes a secure connection over TLS, and NTPv4—the latest incarnation of the NTP protocol. More details on this are below.
The first stage is NTS KE
At this stage, the NTP client initiates a TLS 1.2/1.3 session over a separate TCP connection with the NTS KE server. During this session, the following occurs.
- The parties define the parameters of the algorithm for the second stage.
- The parties define the second lower-level protocol, but currently, only NTPv4 is supported.
- The parties define the IP address and port of the NTP server.
- The NTS KE server issues cookies for NTPv4.
- The parties extract from the cookie material a pair of symmetric keys (C2S and S2C).
This approach has a significant advantage in that the entire load of transmitting the secret information of the connection parameters rests on a tested and reliable TLS protocol. Thus, there is no need to reinvent the wheel for secure NTP handshakes.
The second stage is NTP under NTS protection
At the second stage, the client securely synchronizes time with the NTP server. For this purpose, it sends four special extensions (extension fields) in the NTPv4 packet structure.
- The Unique Identifier Extension contains a random nonce to prevent replay attacks.
- The NTS Cookie Extension contains one of the NTP cookies that the client has. Since only the client possesses the symmetric AAED keys C2S and S2C, the NTP server must extract them from the cookie material.
- The NTS Cookie Placeholder Extension is a means for the client to request additional cookies from the server. This extension is necessary to ensure that the response from the NTP server is not significantly longer than the request, thus preventing amplification attacks.
- The NTS Authenticator and Encrypted Extension Fields Extension contains the AAED encryption algorithm with the C2S key, the NTP header, timestamps, and the aforementioned EF as ancillary data. Without this extension, timestamp forgery is possible.
Upon receiving a request from the client, the server verifies the authenticity of the NTP packet. To do this, it must decrypt the cookie, extract the AAED algorithm, and the keys. After successfully validating the NTP packet, the server responds to the client in the following format.
- The Unique Identifier Extension is a mirror of the client’s request, a measure against replay attacks.
- The NTS Cookie Extension issues more cookies to continue the session.
- NTS Authenticator and Encrypted Extension Fields Extension uses AEAD with an S2C key.
The second handshake can be repeated many times, bypassing the first stage, as each request and response gives the client additional cookies. This has the advantage of distributing relatively resource-intensive TLS operations for computation and transmission of PKI data across multiple repeat requests. This is particularly convenient for specialized FPGA timekeepers, where the entire core functionality can be packed into a few symmetric cryptography functions, delegating the entire TLS stack to another device.
NTPSec
What is special about NTP? Despite the efforts of the project creator Dave Mills to document his code thoroughly, few programmers can navigate the intricate time synchronization algorithms dating back 35 years. Some of the code was written before the POSIX era, and the Unix API at that time differed significantly from what is used today. Additionally, statistical knowledge is needed to filter signals from noise on unreliable lines.
NTS was not the first attempt to fix NTP. After attackers learned to exploit vulnerabilities in NTP for DDoS amplification, it became clear that radical changes were needed. While drafts for NTS were being prepared and refined, the National Science Foundation in the USA urgently allocated a grant at the end of 2014 to modernize NTP.
The working group was led by none other than — one of the founders and pillars of the Open Source community and the author of the book . First, Eric and his colleagues attempted to migrate the NTP code from the BitKeeper platform to git, but that didn’t pan out. The project leader Harlan Stenn opposed this decision, and negotiations reached an impasse. It was then decided to fork the project’s code, leading to the creation of NTPSec.
With solid experience, including work on GPSD, a mathematical background, and a magical ability to read ancient code — Eric Raymond was exactly the hacker who could pull off such a project. The team included a migration expert, and in just 10 weeks NTP on GitLab. Work began in earnest.
Eric Raymond’s team tackled the project like Auguste Rodin working on a block of stone. By removing 175 KLOC of old code, they significantly reduced the attack surface, closing numerous security holes.
Here is a partial list of the vulnerabilities addressed:
- Undocumented, outdated, deprecated, or broken refclock.
- Unused ICS library.
- libopts/autogen.
- Old code for Windows.
- ntpdc.
- Autokey.
- C code of ntpq rewritten in Python.
- C code of sntp/ntpdig rewritten in Python.
In addition to code cleanup, the project had other goals as well. Here is an incomplete list of accomplishments:
- Significantly enhanced code protection against buffer overflows. To prevent buffer overflows, all unsafe string functions (strcpy / strcat / strtok / sprintf / vsprintf / gets) were replaced with safe versions that implement buffer size limits.
- NTS support added.
- Achieved a tenfold increase in time step accuracy by binding to physical hardware. This is due to modern computer clocks being much more accurate than those available at the inception of NTP. GPSDO and dedicated time stations benefited the most from this improvement.
- The number of programming languages has been reduced to two. Instead of using Perl, awk, and even S scripts, there is now solely Python. This allows for greater code reuse opportunities.
- Instead of a mishmash of scripts, autotools, the project now uses a software build system. .
- Updated and reorganized the project's documentation. From a contradictory and sometimes archaic collection of documents, we created a quite decent documentation set. Every command-line key and every configuration entity now has a single source of truth. Additionally, man pages and web documentation are now generated from the same core files.
NTPSec is available for a range of Linux distributions. Currently, the latest stable version is 1.1.8, while for Gentoo Linux it's the second-to-last version.
(1:696)$ sudo emerge -av ntpsec
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] net-misc/ntpsec-1.1.7-r1::gentoo USE="samba seccomp -debug -doc -early -gdb -heat -libbsd -nist -ntpviz -rclock_arbiter -rclock_generic -rclock_gpsd -rclock_hpgps -rclock_jjy -rclock_local -rclock_modem -rclock_neoclock -rclock_nmea -rclock_oncore -rclock_pps -rclock_shm -rclock_spectracom -rclock_trimble -rclock_truetime -rclock_zyfer -smear -tests" PYTHON_TARGETS="python3_6" 0 KiB
Total: 1 package (1 reinstall), Size of downloads: 0 KiB
Would you like to merge these packages? [Yes/No]
Chrony
There was another attempt to replace the old NTP with a more secure alternative. Chrony, unlike NTPSec, is written from scratch and designed for reliable operation in a wide range of conditions, including unstable network connections, partial availability or network overloads, and temperature changes. In addition, Chrony has several other advantages:
- Chrony can synchronize system clocks faster and with greater accuracy;
- Chrony is smaller, uses less memory, and accesses the CPU only when needed. This is a significant advantage for resource and energy savings;
- Chrony supports hardware timestamping on Linux, which provides extremely accurate synchronization in local networks.
However, Chrony lacks some features of the old NTP, such as broadcast and multicast client/server support. Additionally, the classic NTP supports a wider range of operating systems and platforms.
To disable the server functionality and NTP requests to the chronyd process, simply specify port 0 in the chrony.conf file. This is done in cases where there is no need to serve time for NTP clients or peer nodes. Starting from version 2.0, the NTP server port is only open when access is allowed by the allow directive or corresponding command, or if a peer NTP node is configured, or the broadcast directive is used.
The program consists of two modules.
- chronyd is a background service that gets information about the time difference between system clocks and an external time server and adjusts the local time. It also implements the NTP protocol and can act as either a client or a server.
- chronyc is a command-line utility for monitoring and controlling the program. It is used to fine-tune various parameters of the service, such as adding or removing NTP servers while chronyd continues to run.
Starting from the 7th version of RedHat Linux Chrony serves as a time synchronization service. The package is also available for other Linux distributions. The latest stable version is 3.5, with version 4.0 pending release.
(1:712)$ sudo emerge -av chrony
These are the packages that would be merged, in order:
Calculating dependencies... done!
[binary N ] net-misc/chrony-3.5-r2::gentoo USE="adns caps cmdmon ipv6 ntp phc readline refclock rtc seccomp (-html) -libedit -pps (-selinux)" 246 KiB
Total: 1 package (1 new, 1 binary), Size of downloads: 246 KiB
Would you like to merge these packages? [Yes/No]
How to set up your own remote chrony server on the internet for time synchronization in the office network. Below is an example of configuration on a VPS.
Example of Chrony setup on RHEL / CentOS on a VPS
Now let's practice a bit and set up our own NTP server on a VPS. It's very simple, just choose a suitable plan on the RuVDS website, get a ready server, and run a handful of straightforward commands. This option will work perfectly for our needs.
Let's move on to configuring the service, and first we install the chrony package.
[root@server ~]$ yum install chronyRHEL 8 / CentOS 8 use a different package manager.
[root@server ~]$ dnf install chronyAfter installing chrony, you need to start and enable the service.
[root@server ~]$ systemctl enable chrony --nowIf desired, you can edit /etc/chrony.conf, replacing the NTP servers with the nearest local ones to reduce response time.
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.ru.pool.ntp.org iburst
server 1.ru.pool.ntp.org iburst
server 2.ru.pool.ntp.org iburst
server 3.ru.pool.ntp.org iburst
Next, we configure the NTP server synchronization with the nodes from the specified pool.
[root@server ~]$ timedatectl set-ntp true
[root@server ~]$ systemctl restart chronyd.service
You also need to open the NTP port externally, otherwise the firewall will block incoming connections from client nodes.
[root@server ~]$ firewall-cmd --add-service=ntp --permanent
[root@server ~]$ firewall-cmd --reload
On the client side, it's enough to set the correct time zone.
[root@client ~]$ timedatectl set-timezone Europe/MoscowIn the /etc/chrony.conf file, specify the IP or hostname of our VPS server where the NTP server chrony is running.
server my.vps.serverAnd finally, start the time synchronization on the client.
[root@client ~]$ systemctl enable --now chronyd
[root@client ~]$ timedatectl set-ntp true
Next time, I will tell you about options for time synchronization without the internet.
Source: habr.com
