IEEE P802.3ba, the standard for data transmission over 100-gigabit Ethernet channels (100GbE), was developed from 2007 to 2010 [3], but it only gained widespread adoption in 2018 [5]. Why specifically in 2018, and not earlier? And why all at once? There are at least five reasons for this...
The IEEE P802.3ba was primarily developed to meet the needs of data centers and internet traffic exchange points (between independent operators); it was also aimed at ensuring the uninterrupted operation of resource-intensive web services, such as portals with a large amount of video content (e.g., YouTube); and for performing high-performance computing. [3] Regular Internet users are also contributing to the changing bandwidth requirements: many have digital cameras, and people want to upload the content they shoot online. Thus, the volume of circulating content on the Internet is continuously increasing over time, both at professional and consumer levels. In all these cases, when transmitting data from one domain to another, the total bandwidth of key network nodes has long exceeded the capabilities of 10GbE ports. [1] This is what necessitated the emergence of the new standard: 100GbE.
Large data centers and cloud service providers are already actively using 100GbE and plan to gradually transition to 200GbE and 400GbE over the next couple of years. They are also looking into speeds that exceed a terabit. [6] Although among large providers, there are those who only transitioned to 100GbE last year (e.g., Microsoft Azure). Data centers performing high-performance computing for financial services, government platforms, oil and gas platforms, and utilities have also started transitioning to 100GbE. [5]
In corporate data centers, the demand for bandwidth is somewhat lower: only recently has 10GbE become a necessity rather than a luxury. However, as traffic consumption rates continue to grow rapidly, it is doubtful that 10GbE will last in corporate data centers for even 10 or 5 years. Instead, we will see a swift transition to 25GbE and an even faster move to 100GbE. [6] This is because, as Intel analysts point out, traffic intensity within data centers is increasing by 25% each year. [5]
Analysts at Dell and Hewlett Packard state [4] that 2018 is the year of 100GbE for data centers. Back in August 2018, shipments of 100GbE equipment doubled those of the entire year of 2017. And the pace of shipments continues to grow as data centers have begun to move away from 40GbE in large numbers. It is expected that by 2022, 19.4 million 100GbE ports will be shipped annually (in 2017, for comparison, this figure was 4.6 million). [4] As for costs, in 2017, $7 billion was spent on 100GbE ports, while in 2020, it is forecasted that approximately $20 billion will be spent (see Fig. 1). [1]
Figure 1. Statistics and forecasts of demand for network equipment
Why now? 100GbE is not such a new technology, so why is there such a buzz around it now?
1) Because this technology has matured and become cheaper. It was in 2018 that we crossed the threshold where the use of data center platforms with 100-gigabit ports became more cost-effective than 'stacking' several 10-gigabit platforms. For example, the Ciena 5170 (see Fig. 2) is a compact platform providing a total capacity of 800GbE (4x100GbE, 40x10GbE). If several 10-gigabit ports are required to meet necessary throughput, the costs of additional hardware, additional space occupied, excessive power consumption, ongoing maintenance, extra parts, and additional cooling systems add up to a considerable amount. [1] For instance, specialists at Hewlett Packard, analyzing the potential benefits of transitioning from 10GbE to 100GbE, arrived at the following figures: better performance (by 56%), lower total costs (by 27%), reduced energy consumption (by 31%), and simplified cabling connections (by 38%). [5]
Figure 2. Ciena 5170: an example of a platform with 100-gigabit ports
2) Juniper and Cisco have finally created their own ASIC chips for 100GbE switches. [5] This is a telling confirmation that the 100GbE technology has truly matured. The fact is that ASIC chips can only be produced cost-effectively when, firstly, the logic implemented on them does not require changes in the foreseeable future, and secondly, when a large number of identical chips are produced. Juniper and Cisco would not start manufacturing these ASICs unless they were confident in the maturity of 100GbE.
3) Because Broadcom, Cavium, and Mellanox Technologies have started producing 100GbE-supported processors, and these processors are already being used in switches from manufacturers such as Dell, Hewlett Packard, Huawei Technologies, Lenovo Group, and others. [5]
4) Because rack-mounted servers are increasingly equipped with the latest Intel network adapters (see Fig. 3), featuring two 25-gigabit ports, and sometimes even converged network adapters with two 40-gigabit ports (XXV710 and XL710). {Figure 3. The latest Intel network adapters: XXV710 and XL710}
5) Because 100GbE equipment is backward compatible, making deployment easier: existing cables can be reused (just connect a new transceiver to them).
Additionally, the availability of 100GbE prepares us for new technologies, such as ‘NVMe over Fabrics’ (e.g., Samsung Evo Pro 256 GB NVMe PCIe SSD; see Fig. 4) [8, 10], ‘Storage Area Network’ (SAN) / ‘Software Defined Storage’ (see Fig. 5) [7], RDMA [11], which could not fully realize their potential without 100GbE.
Figure 4. Samsung Evo Pro 256 GB NVMe PCIe SSD
Figure 5. ‘Storage Area Network’ (SAN) / ‘Software Defined Storage’
Finally, as an exotic example of the practical demand for 100GbE and related high-speed technologies, we can cite the scientific cloud of the University of Cambridge (see Fig. 6), which is built on 100GbE (Spectrum SN2700 Ethernet switches), to efficiently support the operation of the distributed storage system NexentaEdge SDS, which can easily overload 10/40GbE networks. [2] Such high-performance scientific clouds are deployed to solve a wide variety of practical scientific tasks [9, 12]. For instance, medical researchers utilize such clouds for decoding the human genome, while 100GbE channels are used to transmit information between university research teams.
Figure 6. Fragment of the University of Cambridge scientific cloud
Bibliography
- John Hawkins. // 2017.
- Amit Katz. // 2016.
- Margaret Rouse. .
- David Graves. // 2018.
- Mary Branscombe. // 2018.
- Jarred Baker. // 2017.
- Tom Clark. Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel and IP SANs. 2003. 572p.
- James O’Reilly. Network Storage: Tools and Technologies for Storing Your Company’s Data // 2017. 280p.
- James Sullivan. Student cluster competition 2017, Team University of Texas at Austin/Texas State University: Reproducing vectorization of the Tersoff multi-body potential on the Intel Skylake and NVIDIA V100 architectures // Parallel Computing. v.79, 2018. pp. 30-35.
- Manolis Katevenis. The next Generation of Exascale-class Systems: the ExaNeSt Project // Microprocessors and Microsystems. v.61, 2018. pp. 58-71.
- Hari Subramoni. RDMA over Ethernet: A Preliminary Study // Proceedings of the Workshop on High Performance Interconnects for Distributed Computing. 2009.
- Chris Broekema. Energy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA // Future Generation Computer Systems. v.79, 2018. pp. 215-224.
PS. The article was originally published in .
Only registered users can participate in the survey. , please.
Why have large data centers begun to transition to 100GbE en masse?
Actually, no one has started transitioning yet...
Because this technology has matured and become more affordable
Because Juniper and Cisco created ASICs for 100GbE switches
Because Broadcom, Cavium, and Mellanox Technologies added support for 100GbE
Because servers are now available with 25- and 40-gigabit ports
Your own version (write in the comments)
12 users voted. 15 users abstained.
Source: habr.com
