ProHoster > Blog > Pulega > Fa'aaogaina o le TSDuck e Mata'itū ai Vaitafe IP(TS).

Fa'aaogaina o le TSDuck e Mata'itū ai Vaitafe IP(TS).

I aso nei, o loʻo i ai ni fofo saunia (faʻapitoa) mo le mataʻituina o vaitafe IP (TS), mo se faʻataʻitaʻiga VB и iQ, o lo'o i ai a latou fa'aputuga mauoa o galuega ma e masani lava o fa'alapotopotoga tetele o lo'o feagai ma auaunaga TV e iai fa'afitauli fa'apea. O lenei tusiga o loʻo faʻamatalaina ai se fofo e faʻavae i luga o se galuega faʻavae tatala TSDuck, fuafuaina mo le pulea itiiti o le IP(TS) vaitafe e CC(continuity counter) counter ma bitrate. O se talosaga e mafai o le pulea lea o le leiloa o pepa po'o le tafe atoa e ala i le lisi L2 alalaupapa (lea e le mafai ona mata'ituina masani, mo se fa'ata'ita'iga, e ala i le faitauina o fa'amaumauga leiloa i laina).

Fa'apuupuu e uiga i TSDuck

O le TSDuck o se puna tatala (2-Clause BSD license) software (se seti o mea aoga faʻamafanafana ma se faletusi mo le atinaʻeina o mea faʻapitoa poʻo plugins) mo le faʻaogaina o vaitafe TS. I le avea ai o se faʻaoga, e mafai ona galue i le IP (multicast / unicast), http, hls, dvb tuners, dektec dvb-asi demodulator, o loʻo i ai se afi TS-stream generator ma faitau mai faila. O le gaioiga e mafai ona tusitusi i se faila, IP (multicast/unicast), hls, dektec dvb-asi ma HiDes modulators, tagata taʻalo (mplayer, vlc, xine) ma pa'ū. E mafai ona fa'aaofia le tele o fa'agaioiga fe'avea'i i le va o fa'aoga ma mea e fai, mo se fa'ata'ita'iga, toe fa'avasegaina o le PID, fa'asolo / fa'asalalau, su'esu'ega fa'atau CC, fa'atusatusaga o le bitrate, ma isi gaioiga masani mo vaitafe TS.

I totonu o lenei tusiga, o le IP streams (multicast) o le a faʻaaogaina e fai ma faʻaoga, o le processors bitrate_monitor (mai le igoa e manino le mea) ma le faʻaauau (suʻesuʻega o CC counters) faʻaaogaina. E faigofie ona e suia le IP multicast i se isi ituaiga fa'aoga e lagolagoina e TSDuck.

Avanoa fau aloaia/fa'aputu TSDuck mo le tele o faiga fa'aoga o lo'o iai nei. E le o avanoa mo Debian, ae na mafai ona matou fausia i lalo o le debian 8 ma le debian 10 e aunoa ma ni faʻafitauli.

O le isi, faʻaaogaina le TSDuck 3.19-1520, faʻaaogaina Linux o le OS (debian 10 na faʻaaogaina e saunia ai le fofo, CentOS 7 na faʻaaogaina mo le faʻaoga moni)

Saunia TSDuck ma OS

Aʻo leʻi mataʻituina le tafe moni, e tatau ona e mautinoa o loʻo galue lelei le TSDuck ma e leai ni pa'ū i luga o le kata fesoʻotaʻiga poʻo le OS (socket) level. E manaʻomia lenei mea ina ia aua neʻi mateina mulimuli ane le mea na tupu ai le pa'ū - i luga o le upega tafailagi poʻo "i totonu o le server". E mafai ona e siaki pa'ū i le tulaga kata network ma le ethtool -S ethX poloaiga, tuning e faia e le ethtool tutusa (e masani lava, e te manaʻomia le faʻatuputeleina o le RX paʻu (-G) ma o nisi taimi e faʻamalo ai nisi offloads (-K)). I le avea ai o se fautuaga lautele, e mafai ona fautuaina e faʻaaoga se isi taulaga mo le mauaina o le auʻiliʻiliga fefaʻatauaʻiga, pe a mafai, e faʻaitiitia ai faʻamatalaga sese e fesoʻotaʻi ma le mea moni o le pa'ū na tupu tonu lava i luga o le vaʻavaʻa ona o le i ai o isi fefaʻatauaiga. Afai e le mafai (o se mini-komepiuta / NUC ma le tasi uafu e faʻaaogaina), ona matua manaʻomia ai lea o le faʻatulagaina o le faʻamuamua o fefaʻatauaʻiga suʻesuʻe e fesoʻotaʻi ma isi mea i luga o le masini e fesoʻotaʻi ai le suʻega. E tusa ai ma siosiomaga faʻapitoa, iinei e tatau ona e faʻaeteete ma mafai ona e suʻeina paʻu paʻu e amata mai i se taulaga faʻapitoa ma faʻaiʻu i se talosaga i totonu o se masini masini.

Fausia ma le mauaina o se vaitafe i totonu o le talimalo

I le avea ai o se laasaga muamua i le saunia o le TSDuck, o le a matou gaosia ma maua fefaʻatauaiga i totonu o se tasi talimalo e faʻaaoga ai upega.

Saunia le siosiomaga:

ip netns add P #создаём netns P, в нём будет происходить анализ трафика
ip link add type veth #создаём veth-пару - veth0 оставляем в netns по умолчанию (в этот интерфейс будет генерироваться трафик)
ip link set dev veth1 netns P #veth1 - помещаем в netns P (на этом интерфейсе будет приём трафика)
ip netns exec P ifconfig veth1 192.0.2.1/30 up #поднимаем IP на veth1, не имеет значения какой именно
ip netns exec P ip ro add default via 192.0.2.2 #настраиваем маршрут по умолчанию внутри nents P
sysctl net.ipv6.conf.veth0.disable_ipv6=1 #отключаем IPv6 на veth0 - это делается для того, чтобы в счётчик TX не попадал посторонний мусор
ifconfig veth0 up #поднимаем интерфейс veth0
ip route add 239.0.0.1 dev veth0 #создаём маршрут, чтобы ОС направляла трафик к 239.0.0.1 в сторону veth0

Ua sauni le siosiomaga. Matou te amataina le suʻesuʻega o feoaiga:

ip netns exec P tsp --realtime -t 
 -I ip 239.0.0.1:1234 
 -P continuity 
 -P bitrate_monitor -p 1 -t 1 
 -O drop

lea "-p 1 -t 1" o lona uiga e te manaʻomia le faʻatatauina o le bitrate i sekone taʻitasi ma faʻaalia faʻamatalaga e uiga i le bitrate i sekone taʻitasi
Matou te amataina le afi afi ma le saoasaoa o le 10Mbps:

tsp -I craft 
 -P regulate -b 10000000 
 -O ip -p 7 -e --local-port 6000 239.0.0.1:1234

pe a "-p 7 -e" o lona uiga e te manaʻomia le faʻapipiʻiina o 7 TS pepa i totonu o le 1 IP packet ma fai faʻamalosi (-e), i.e. fa'atali pea 7 TS packets mai le processor mulimuli ae le'i tu'uina atu se pusa IP.

O le su'esu'e e amata ona fa'aalia fe'au fa'amoemoeina:

* 2020/01/03 14:55:44 - bitrate_monitor: 2020/01/03 14:55:44, TS bitrate: 9,970,016 bits/s
* 2020/01/03 14:55:45 - bitrate_monitor: 2020/01/03 14:55:45, TS bitrate: 10,022,656 bits/s
* 2020/01/03 14:55:46 - bitrate_monitor: 2020/01/03 14:55:46, TS bitrate: 9,980,544 bits/s

Faaopoopo nei i ai ni mataua:

ip netns exec P iptables -I INPUT -d 239.0.0.1 -m statistic --mode random --probability 0.001 -j DROP

ma o fe'au fa'apenei e aliali mai:

* 2020/01/03 14:57:11 - continuity: packet index: 80,745, PID: 0x0000, missing 7 packets
* 2020/01/03 14:57:11 - continuity: packet index: 83,342, PID: 0x0000, missing 7 packets

lea ua faamoemoeina. Fa'agata pepa gau (ip netns exec P iptables -F) ma taumafai e fa'ateleina le bitrate o le afi i le 100Mbps. O loʻo lipotia e le tagata suʻesuʻe le tele o mea sese CC ma e uiga i le 75 Mbps nai lo le 100. O loʻo matou taumafai e iloa poʻo ai e tuʻuaʻia - e leai se taimi o le afi pe leai foi le faʻafitauli, ona o lenei mea matou te amata fatuina se numera tumau o afifi (700000 TS packets = 100000 IP packets):

# ifconfig veth0 | grep TX
       TX packets 151825460  bytes 205725459268 (191.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
# tsp -I craft -c 700000 -P regulate -b 100000000 -P count -O ip -p 7 -e --local-port 6000 239.0.0.1:1234
* count: PID    0 (0x0000):    700,000 packets
# ifconfig veth0 | grep TX
        TX packets 151925460  bytes 205861259268 (191.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

E pei ona e vaʻaia, saʻo 100000 IP packets na gaosia (151925460-151825460). O lea seʻi o tatou vaʻai i le mea o loʻo tupu i le suʻega, o le mea lea matou te siakiina ma le RX counter i luga o le veth1, e matua tutusa lava ma le TX counter i luga o le veth0, ona tatou vaʻavaʻai lea i le mea e tupu i le socket level:

# ip netns exec P cat /proc/net/udp                                                                                                           
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops             
  133: 010000EF:04D2 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 72338 2 00000000e0a441df 24355

O iinei e mafai ona e vaʻai i le numera o mataua = 24355. I totonu o TS packets, o le 170485 poʻo le 24.36% o le 700000, o lea matou te vaʻai ai o ia lava 25% o le bitrate leiloa o pa'ū i le socket udp. Pa'u i totonu o le UDP socket e masani lava ona tupu ona o le leai o se paʻu, vaʻai i le lapoʻa faʻamaufaʻailoga faʻaletonu ma le maualuga o le paʻu pa puipui:

# sysctl net.core.rmem_default
net.core.rmem_default = 212992
# sysctl net.core.rmem_max
net.core.rmem_max = 212992

O le mea lea, afai e le faʻatonuina e talosaga se lapoʻa faʻapipiʻi, e faia sockets ma se paʻu o le 208 KB, ae afai latou te talosagaina atili, latou te le mauaina lava le mea na talosagaina. Talu ai e mafai ona e setiina le tele o le paʻu i le tsp mo le faʻaogaina o le IP (-buffer-size), matou te le paʻi atu i le lapoʻa faʻaletonu, ae naʻo le setiina o le maualuga o le paʻu paʻu ma faʻamaonia manino le tele o le pa e ala i finauga tsp:

sysctl net.core.rmem_max=8388608
ip netns exec P tsp --realtime -t -I ip 239.0.0.1:1234 -b 8388608 -P continuity -P bitrate_monitor -p 1 -t 1 -O drop

Faatasi ai ma lenei faʻalogoina o le socket buffer, o lea ua lipotia mai le bitrate e tusa ma le 100Mbps, e leai ni mea sese CC.

E tusa ai ma le faʻaaogaina o le CPU o le tsp application lava ia. Fa'atatau ile tasi core i5-4260U CPU @ 1.40GHz, 10Mbps su'esu'ega tafe ole a mana'omia 3-4% CPU, 100Mbps - 25%, 200Mbps - 46%. Pe a setiina le % Packet Loss, o le uta i luga o le PPU e toetoe lava a le faʻateleina (ae ono faʻaititia).

I luga o meafaigaluega sili atu ona aoga, na mafai ona gaosia ma auʻiliʻili vaitafe e sili atu i le 1Gb / s e aunoa ma ni faʻafitauli.

Su'ega i kata feso'ota'iga moni

A maeʻa faʻataʻitaʻiga i luga o se veth pair, e tatau ona e ave ni 'au se lua poʻo ni ports se lua o le tasi talimalo, faʻafesoʻotaʻi ports i le tasi ma le isi, amata le generator i le tasi, ma le suʻega i le lona lua. E leai se mea e ofo ai iinei, ae o le mea moni e faalagolago uma lava i le uʻamea, o le vaivai, o le sili atu ona manaia o le ai ai iinei.

Fa'aaogāina o fa'amaumauga na maua e le faiga mata'ituina (Zabbix)

tsp e leai se API e mafai ona faitau masini pei ole SNMP po'o se mea faapena. O fe'au CC e tatau ona tu'ufa'atasia mo le itiiti ifo i le 1 sekone (fa'atasi ai ma se pasene maualuga o le pa'u o le pa'u, e mafai ona i ai le selau/afe/sefulu afe i le sekone, e fa'atatau i le bitrate).

O le mea lea, ina ia mafai ona faʻasaoina faʻamatalaga uma ma tusi kalafi mo CC sese ma bitrate ma faia nisi ituaiga o faʻalavelave, atonu o loʻo i ai filifiliga nei:

  1. Fa'avasega ma fa'aputu (e le CC) le gaosiga o le tsp, i.e. liliu i le fomu e mana'omia.
  2. Fa'auma tsp lava ma/po'o fa'apipi'i fa'apipi'i bitrate_monitor ma fa'aauau ina ia tu'uina atu le fa'ai'uga i se fomu e mafai ona faitau masini e talafeagai mo le faiga mata'ituina.
  3. Tusi lau talosaga i luga o le faletusi tsduck.

E manino lava, o le filifiliga 1 e sili ona faigofie i tulaga o taumafaiga, aemaise lava le manatu o le tsduck lava ia o loʻo tusia i se gagana maualalo (i tulaga faʻaonapo nei) (C ++)

O se faʻataʻitaʻiga faigofie o le bash parser + aggregator na faʻaalia i luga o le 10Mbps stream ma le 50% packet loss (sili ona leaga), o le faʻaogaina o le bash 3-4 taimi sili atu PPU nai lo le tsp process lava ia. E le taliaina lea tulaga. O le mea moni o se vaega o lenei prototype i lalo

Saimini i luga

#!/usr/bin/env bash

missingPackets=0
ccErrorSeconds=0
regexMissPackets='^* (.+) - continuity:.*missing ([0-9]+) packets$'
missingPacketsTime=""

ip netns exec P tsp --realtime -t -I ip -b 8388608 "239.0.0.1:1234" -O drop -P bitrate_monitor -p 1 -t 1  -P continuity 2>&1 | 
while read i
do
    #line example:* 2019/12/28 23:41:14 - continuity: packet index: 6,078, PID: 0x0100, missing 5 packets
    #line example 2: * 2019/12/28 23:55:11 - bitrate_monitor: 2019/12/28 23:55:11, TS bitrate: 4,272,864 bits/s
    if [[ "$i" == *continuity:* ]] 
    then
        if [[ "$i" =~ $regexMissPackets ]]
        then
            missingPacketsTimeNew="${BASH_REMATCH[1]}" #timestamp (seconds)
            if [[ "$missingPacketsTime" != "$missingPacketsTimeNew" ]] #new second with CC error
            then
                ((ccErrorSeconds += 1))
            fi
            missingPacketsTime=$missingPacketsTimeNew
            packets=${BASH_REMATCH[2]} #TS missing packets
            ((missingPackets += packets))
        fi
    elif [[ "$i" == *bitrate_monitor:* ]]
    then
        : #...
    fi
done

I le faaopoopo atu i le le taliaina lemu, e leai ni filo masani i le bash, o galuega bash o ni faiga eseese ma e tatau ona ou tusia le tau o missingPackets tasi i le sekone i luga o le itu aʻafiaga (pe a maua feʻau bitrate e oʻo mai i sekone uma). O le i'uga, na tu'u ai na'o le bash ma filifili ai e tusi se afifi (parser + aggregator) i golang. O le faʻaaogaina o le PPU o le code golang tutusa e 4-5 taimi e itiiti ifo nai lo le tsp process lava ia. O le saoasaoa o le afifi ona o le suia o le bash ma golang na foliga mai e tusa ma le 16 taimi ma i se tulaga lautele o le taunuuga e talia (CPU i luga ole ulu e 25% i le tulaga sili ona leaga). O lo'o iai le faila puna o le golang iinei.

Tafe ta'avale

Ina ia amata le afifi, na faia le mamanu auaunaga sili ona faigofie mo systemd (iinei). O le afifi lava ia e tatau ona tuʻufaʻatasia i se faila binary (alu fausia tsduck-stat.go) o loʻo i totonu /opt/tsduck-stat/. E fa'apea o lo'o e fa'aogaina le golang ma le lagolago mo le uati monotoni (>=1.9).

Ina ia fatuina se faʻataʻitaʻiga o le tautua, e tatau ona e faʻatautaia le systemctl enable command [imeli puipuia]:1234 ona tamoe lea ma systemctl amata [imeli puipuia]: 1234.

Mauaina mai Zabbix

Ina ia mafai e zabbix ona maua le tautua o loʻo taʻavale, ua maeʻa generator lisi vaega (discovery.sh), i le faatulagaga e manaʻomia mo le mauaina o Zabbix, e manatu o loʻo tu i le nofoaga e tasi - i /opt/tsduck-stat. Ina ia faʻatautaia suʻesuʻega e ala i le zabbix-agent, e tatau ona e faʻaopopo .conf faila i le zabbix-agent configuration directory e faʻaopoopo ai le faʻaogaina o le tagata faʻaoga.

Fa'ata'ita'iga Zabbix

Fausia mamanu (tsduck_stat_template.xml) o lo'o i ai le tulafono o le su'esu'e otometi, mea fa'ata'ita'i, kalafi, ma fa'aoso.

Lisi siaki puupuu (ia, faʻafefea pe a filifili se tasi e faʻaaogaina)

  1. Ia mautinoa o le tsp e le tuʻuina pepa i lalo o tulaga "lelei" (generator ma suʻesuʻega e fesoʻotaʻi saʻo), pe a iai ni mataua, tagaʻi i le parakalafa 2 poʻo le tusitusiga o le tusiga i lenei mataupu.
  2. Fai le fa'alogoina o le pito maualuga o le pa'u (net.core.rmem_max=8388608).
  3. Fa'aopoopo le tsduck-stat.go (alu fausia tsduck-stat.go).
  4. Tuu le faʻataʻitaʻiga o le tautua i totonu /lib/systemd/system.
  5. Amata auaunaga ile systemctl, siaki ua amata ona aliali mai fa'atau (grep "" /dev/shm/tsduck-stat/*). Ole numera o au'aunaga ile numera ole fa'asalalau tele. O iinei atonu e te manaʻomia le fatuina o se auala i le multicast kulupu, masalo faʻamalo le rp_filter pe fatuina se auala i le puna ip.
  6. Tafe le discovery.sh, ia mautinoa e maua ai le json.
  7. Faʻaopoopo le zabbix agent config, toe amata le zabbix agent.
  8. Tuʻu le faʻataʻitaʻiga i le zabbix, faʻaoga i le talimalo o loʻo mataʻituina ma faʻapipiʻi le zabbix-agent, faʻatali pe a ma le 5 minute, vaʻai pe iai ni mea fou, kalafi ma faʻaoso.

iʻuga

Fa'aaogaina o le TSDuck e Mata'itū ai Vaitafe IP(TS).

Mo le galuega o le suʻeina o paʻu gau, e toetoe lava a lava, o le mea sili e sili atu nai lo le leai o se mataʻituina.

O le mea moni, e mafai ona tupu le "toilalo" CC pe a tuʻufaʻatasia vaega o ata vitio (e pei ona ou iloa, o le auala lea e faʻapipiʻi ai i nofoaga autu o TV i totonu o Rusia, e aunoa ma le toe faʻatulagaina o le CC counter), e tatau ona manatua lenei mea. O vaifofo fa'apitoa e fa'ato'a soloia ai lenei fa'afitauli e ala i le su'eina o fa'ailoga SCTE-35 (pe a fa'aopoopoina e le afi afi).

I tulaga o le mataituina lelei o felauaiga, o loʻo i ai le le lava o le mataʻituina o le jitter (IAT). Meafaigaluega TV (pe o modulators po'o masini fa'ai'uga) ei ai mana'oga mo lenei parakalafa ma e le mafai i taimi uma ona fa'atuputeleina le jitbuffer i le le gata. Ma e mafai ona opeopea le jitter pe a faʻaogaina meafaigaluega e tele paʻu i femalagaiga ma QoS e leʻo faʻapipiʻiina pe le lelei foʻi le faʻaogaina e faʻafeiloaʻi ai ia fefaʻatauaiga i taimi moni.

puna: www.habr.com

Faaopoopo i ai se faamatalaga