🥇Fundamentals of reliable data transfer

To those who seeks to understand networks and protocols, is dedicated.

Briefly

The article discusses the basics of reliable data transmission, implements examples on Go, including UDP and TCP. Based on time, two, three and the book "Computer networks. Top-down approach", otherwise everyone discusses only Tannenbaum and Olifer.

Transport layer protocol

Provides a logical connection between application processes running on different hosts. A logical connection from the point of view of applications looks like a channel that directly connects processes.

Transport layer protocols are supported by end systems, but not by network routers (except − DPI). On the sender side, the transport layer converts the application layer data it receives from the sending application process into transport layer packets called segments.

This is done by splitting (if necessary) the application layer messages into fragments and adding a transport layer header to each of them.

The transport layer then passes the segment to the sender's network layer, where the segment is encapsulated in a network layer packet (datagram) and sent out. On the receiving side, the network layer extracts the transport layer segment from the datagram and passes it up to the transport layer. Next, the transport layer processes the received segment so that its data becomes available to the receiving application.

Principles of reliable data transmission

Reliable data transmission over a completely reliable channel

The simplest case. The sender simply receives the data from the top layer, creates a packet containing it, and sends it to the channel.

Server

package main

import (
    "log"
    "net"
)

func main() {
    // IP-адрес сервера и порт
    serverAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:12000")
    if err != nil {
        log.Fatal(err)
    }

    // создаем сокет с портом
    serverConn, err := net.ListenUDP("udp", serverAddr)
    if err != nil {
        log.Fatal(err)
    }
    // отложенное закрытие соединения
    defer serverConn.Close()

    // создаем буфер для данных
    buf := make([]byte, 1024)

    // ждем соединение
    for {
        // читаем запрос
        n, addr, err := serverConn.ReadFromUDP(buf)
        // передаем данные в ВЕРХНИЙ уровень: в нашем случае stdout
        println(string(buf[0:n]), " form ", addr.IP.String())
        if err != nil {
            log.Fatal(err)
        }
        // ответа нет, т.к. это UDP + надежный канал
    }
}

Customer

package main

import (
    "fmt"
    "log"
    "net"
    "time"
)

func main() {
    // IP-адрес сервера и порт
    serverAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:12000")
    if err != nil {
        log.Fatal(err)
    }
    // локальный IP-адрес и порт
    localAddr, err := net.ResolveUDPAddr("udp", "127.0.0.1:0")
    if err != nil {
        log.Fatal(err)
    }
    // установка соединения
    conn, err := net.DialUDP("udp", localAddr, serverAddr)
    if err != nil {
        log.Fatal(err)
    }
    // отложенное закрытие соединения
    defer conn.Close()

    for {
        // получение данных от ВЕРХНЕГО уровня
        fmt.Print("Введите строчное предложение > ")
        var msg string
        _, err := fmt.Scanf("%s", &msg)
        if err != nil {
            log.Fatal(err)
        }
        // передается поток байт, а не строка
        buf := []byte(msg)
        // запись (передача) в соединение
        _, err = conn.Write(buf)
        if err != nil {
            log.Fatal(err)
        }
        // 1 секундочку
        time.Sleep(time.Second * 1)
    }
}

Reliable data transmission over the channel with possible errors

The next step is to assume that all transmitted packets are received in the order in which they were sent, but the bits in them may be corrupted due to the fact that the channel sometimes transmits data with distortion.

In this case, the following mechanisms apply:

error detection;
feedback;
retransmission.

Reliable data transfer protocols that have similar retransmission mechanisms are called Automatic Repeat reQuest (ARQ) protocols.
Additionally, it is worth considering the possibility of errors in the receipts, when the receiving party does not receive any information about the results of the transmission of the last packet.
The solution to this problem, which is also used in TCP, is to add a new field to the data packet containing the sequence number of the packet.

Reliable data transmission over an unreliable channel that is susceptible to corruption and packet loss

Along with the distortion, unfortunately, there is packet loss in the network.
And to solve this problem, mechanisms are required:

determination of the fact of packet loss;
re-delivery of lost packets to the receiving party.

Additionally, in addition to the loss of the package, it is necessary to provide for the possibility of losing the receipt or, if nothing is lost, its delivery with a significant delay. In all cases, the same thing is done: retransmission of the packet. To control the time in this mechanism, a countdown timer is used, which allows you to determine the end of the waiting interval. So in the package NET TCPKeepAlive is set to 15 seconds by default:

// defaultTCPKeepAlive is a default constant value for TCPKeepAlive times
// See golang.org/issue/31510
const (
    defaultTCPKeepAlive = 15 * time.Second
)

The transmitting side needs to start the timer every time a packet is transmitted (both the first and the second), process interrupts from the timer and stop it.

So, we got acquainted with the key concepts of reliable data transfer protocols:

checksums;
sequence numbers of packages;
timers;
positive and negative receipts.

But that is not all!

Reliable data transfer protocol with pipelining

In the variant that we have already considered, the reliable delivery protocol is very inefficient. It begins to “slow down” the transmission provided by the communication channel as the RTT increases. Pipelining is used to improve its efficiency and better utilize the bandwidth of the communication channel.

The use of pipelining leads to:

increasing the range of sequence numbers, since all sent packets (with the exception of retransmissions) must be uniquely identifiable;
the need to increase the buffers on the transmitting and receiving sides.

The sequence number range and buffer size requirements depend on the actions taken by the protocol in response to packet corruption, loss, and delay. In the case of pipelining, there are two methods for correcting errors:

going back N packets back;
selective repetition.

Backward N Packets - Sliding Window Protocol

The sender must support three types of events:

call by a higher level protocol. When the send data function is called "from above", the transmitting side first checks the degree of filling the window (that is, the presence of N sent messages waiting to receive receipts). If the window is empty, a new packet is formed and transmitted, and the values of the variables are updated. Otherwise, the transmitting side returns the data to the upper layer, and this is an implicit indication that the window is full. Typically, the upper layer will retry the data transfer after some time. In a real application, the sender would most likely either buffer the data (instead of sending it immediately) or have a synchronization mechanism (such as a semaphore or flag) that would allow the higher layer to call the send function only when the window is empty.
receiving confirmation. In the protocol, a general acknowledgment is issued for a packet with sequence number N, indicating that all packets with sequence numbers preceding N were successfully received.
timeout expiration. To determine the facts of losses and delays of packets and receipts, the protocol uses a timer. If the timeout interval expires, the transmitting side resends all unacknowledged packets sent.

Selective repetition

When the window size and throughput times propagation delay are large, a large number of packets can be in the pipeline. In such a case, a single packet error may cause a large number of packets to be retransmitted, most of which were not required.

Example

Best theoretical practices collected in practical implementation TCP. And if someone knows how better - welcome.

Server

package main

import (
    "bufio"
    "fmt"
    "log"
    "net"
    "strings"
)

func main() {
    // создаем сокет с портом 
    ln, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Fatalln(err)
    }
    // ожидание вызова
    conn, _ := ln.Accept()

    for {
        // считывание данных
        msg, err := bufio.NewReader(conn).ReadString('n')
        if err != nil {
            log.Fatalln(err)
        }
        // вывод сообщения в stdout
        fmt.Print("Message Received:", string(msg))
        // перевод строки в верхний регистр
        newMsg := strings.ToUpper(msg)
        // отправка данных
        conn.Write([]byte(newMsg + "n"))
    }
}

Customer

package main

import (
    "bufio"
    "fmt"
    "log"
    "net"
    "os"
)

func main() {
    // установка соединения
    conn, err := net.Dial("tcp", "127.0.0.1:8081")
    if err != nil {
        log.Fatalln(err)
    }

    for {
        // считывание данных с stdin
        reader := bufio.NewReader(os.Stdin)
        fmt.Print("Text to send: ")
        // построчно
        text, err := reader.ReadString('n')
        if err != nil {
            log.Fatalln(err)
        }
        // отправка
        fmt.Fprintf(conn, text+"n")
        // прием
        msg, err := bufio.NewReader(conn).ReadString('n')
        if err != nil {
            log.Fatalln(err)
        }
        // вывод полученного ответа
        fmt.Print("Msg from Server: " + msg)
    }
}

Hack and predictor Aviator

Mechanisms to ensure reliable data transmission and use

Movement
Application, comment

Check sum
Used to detect bit errors in a transmitted packet.

Timer
The countdown of the waiting interval and an indication of its expiration. The latter means that with a high degree of probability the package or its receipt is lost during transmission. If the packet is delivered with a delay, but is not lost (premature expiration of the waiting interval), or if the receipt is lost, retransmission leads to duplication of the packet on the receiving side

Serial number
Used for sequential numbering of data packets transmitted from the sender to the recipient. Breaks in the sequence numbers of received packets allow the receiver to detect packet loss. The same packet sequence numbers mean that the packets are duplicates of each other.

Confirmation
Generated by the receiving side and indicating to the transmitting side that the corresponding packet or group of packets was successfully received. Typically, an acknowledgment contains sequence numbers of successfully received packets. Depending on the protocol, individual and group confirmations are distinguished.

Negative confirmation
Used by the receiver to inform the sender that the packet was received incorrectly. The negative acknowledgment usually includes the sequence number of the packet that was not correctly received.

Window, pipelining
Limit the range of sequence numbers that can be used to transmit packets. Multicast and handshake can significantly increase the throughput of protocols compared to the acknowledgment waiting mode. As we will see, the window size can be calculated based on the reception and buffering capabilities of the receiving side, as well as the network load level.

Other examples of using Go for networking

В repositories.

Source: habr.com

Fundamentals of Reliable Data Transfer