(Pafupifupi) kusakatula kopanda phindu kuchokera pa msakatuli. Gawo 2. WebRTC

Mwanjira ina mkati одной kuyambira zolemba zakale komanso zosiyidwa kale, ndidalemba za momwe mungaulutsire makanema mosavuta komanso mwachilengedwe kuchokera pansalu kudzera pa ma websockets. Munkhaniyi, ndidalankhula mwachidule za momwe mungajambulire kanema kuchokera pa kamera ndi mawu kuchokera pa maikolofoni pogwiritsa ntchito MediaStream API, momwe mungasinthire mtsinje wolandiridwa ndikutumiza kudzera pa ma websockets ku seva. Komabe, kwenikweni samachita izi, chifukwa chowulutsa amagwiritsa ntchito mapulogalamu apadera omwe amafunika kukhazikitsidwa ndikukonzedwa: offhand, izi zitha kukhala. Tsegulani Mapulogalamu Otsatsa, kapena amagwiritsa ntchito WebRTC, yomwe imagwira ntchito kunja kwa bokosi, ndiye kuti, sikutanthauza kukhazikitsa mapulagini aliwonse a la flash player, omwe adzadulidwa mu msakatuli wa Chromium kale mu December.

Lero tikambirana za WebRTC.

Web Real-Time Communication (WebRTC) si protocol imodzi, ndi gulu lonse la miyezo, ma protocol ndi JavaScript API zomwe pamodzi zimapereka kulumikizana kwapanthawi yeniyeni kwamavidiyo ndi audio, komanso kutha kugwiritsidwa ntchito kusamutsa chilichonse. data binary. Nthawi zambiri asakatuli amakhala ngati anzawo, koma amathanso kukhala pulogalamu yam'manja, mwachitsanzo. Kuti mukonzekere kulumikizana kwa p2p pakati pa makasitomala, chithandizo cha msakatuli chamitundu yosiyanasiyana yamavidiyo ndi ma encoding amawu chimafunika, kuthandizira ma protocol osiyanasiyana a netiweki, kuwonetsetsa kuyanjana kwa Hardware ndi osatsegula (kupyolera mu zigawo za OS): ma webukamu, makadi amawu. Matekinoloje onsewa amabisika kuseri kwa JavaScript API abstraction kuti wopanga athandizire.

Zonse zimatengera ma API atatu:

• MediaStream API - Anathyoledwa nthawi yotsiriza, lero ndilemba zambiri za iye. Imatumikira kuti ilandire mavidiyo / ma audio kuchokera pa hardware

• RTCPeerConnection - imapereka kulumikizana pakati pa makasitomala awiri (p2p)

• RTCDataChannel - imathandizira kusamutsa deta yosagwirizana pakati pa makasitomala awiri

Kukonzekera ma audio ndi makanema kuti atumizidwe

Zonse zimayamba ndi "kujambula" ma webukamu ndi maikolofoni media mitsinje. Zachidziwikire, mitsinje yaiwisi si yoyenera kukonza ma teleconference, mtsinje uliwonse uyenera kukonzedwa: kusintha mtundu, kulunzanitsa mawu ndi kanema, ikani zizindikiro zolumikizirana pamayendedwe a kanema, ndikuwonetsetsa kuti bitrate ikugwirizana ndi bandwidth yomwe ikusintha nthawi zonse. . Msakatuli amasamalira zonsezi, wopanga sakhala ndi nkhawa kuti apereke encoding ya media media. Mkati mwa msakatuli wamakono, muli kale zigawo za mapulogalamu ojambulira, kuwongolera khalidwe (chotsani echo ndi phokoso pamawu, sinthani chithunzi), mavidiyo ndi ma encoding. Dongosolo lachidule likuwonetsedwa mkuyu. 1:

Mpunga. 1. Zigawo za audio ndi kanema processing mu msakatuli

Kukonza konse kumachitika mwachindunji mu msakatuli wokha, palibe zowonjezera. palibe mapulagini chofunika. Komabe, zinthu sizili bwino mu 2020. Pali asakatuli omwe sanagwirizanebe mokwanira MediaStream API, mutha kutsata ulalo ndikuwona tebulo lofananira pansi kwambiri. IE makamaka ndikukhumudwitsanso.

Mutha kuchita zinthu zosangalatsa kwambiri ndi mitsinje yomwe mwalandira: mutha kufananiza, kusintha mawonekedwe a kanema, kusintha mtundu wamawu, mutha kutenga ndi "kulumikiza" mtsinje wa Media Stream kuti. tag ndikudziwonera nokha patsamba la html. Kapena mutha kujambula mtsinje pa canvas, ndikuyika WebGL kapena CSS3, ndikugwiritsa ntchito zosefera zosiyanasiyana pavidiyo, kujambula kanema wokonzedwa kuchokera pansalu ndikutumiza pa netiweki ku seva, transcode ndikusindikiza kwa aliyense (hello bigo live, twitch ndi ena). Apa sindisanthula momwe zinthu zotere zimachitikira, ndipereka zitsanzo zingapo zopezeka pa intaneti:

https://jeeliz.com/ - anyamata akuchita realtime CV mu Javascript. Iwo ali ndi zonse zida malaibulale osiyanasiyana a js ogwirira ntchito ndi mtsinje wamavidiyo pansalu: kuzindikira nkhope, zinthu, kugwiritsa ntchito zosefera (masks, monga mu Instagram), ndi zina. Chitsanzo chabwino kwambiri cha momwe mungasinthire kanema wanthawi yeniyeni mwachindunji mumsakatuli popanda mapulagini owonjezera.

Canvas CaptureStream API - Zolemba za API zojambulitsa kanema kuchokera ku canvas. Zathandizidwa kale mu Chrome, Opera ndi Firefox

RTCPeerConnection

Kotero ife tinafika pa mfundo, koma momwe kwenikweni kusamutsa kanema kwa wosuta wina? Imafika patsogolo RTCPeerConnection. Mwachidule, pafupifupi sitepe iyi muyenera kupanga RTCPeerConnection chinthu:

const peerConnection = new RTCPeerConnection({
  iceServers: [{
    urls: 'stun:stun.l.google.com:19302'
  }]
});

Timatchula iceServers ngati imodzi mwazosankha - iyi ndi seva yomwe imathandizira kupereka kulumikizana pakati pa asakatuli awiri kumbuyo kwa NAT'om. Ndiye kuti, vutoli likuthetsedwa apa: momwe mungapezere ip ya interlocutor ngati ali kumbuyo kwa NAT wa wothandizira wake? Protocol ya ICE imabwera kudzapulumutsa, kwenikweni, ICE sigwira ntchito ku WebRTC konse, koma zambiri pambuyo pake.

M'mbuyomu, tinali ndi mitsinje ya Usermedia:

navigator.mediaDevices.getUserMedia({ video: true, audio: true }).then(stream => {
  // Usermedia-потоки, обычно это видео и аудио 
  const tracks = stream.getTracks();

   for (const track of tracks) {
     // каждый трек присоединяем к peerConnection
     peerConnection.addTrack(track);
   }
}).catch(console.error);

Chotsatira, chochitika chomwe chikufunika chikuwotcha pa peerConnection, mu chothandizira chomwe tiyenera kupanga chopereka (molingana ndi SDP - Session Description Protocol) ndikuchipereka kwa peerConnection kudzera pa setLocalDescript njira. Za SDP - zomwe zili komanso mawonekedwe operekera ndi mayankho - tikambirana zambiri.

Pambuyo popereka LocalDescript peerConnection, msakatuli "amasonkhanitsa" okonda ayezi, ndiko kuti, amapeza njira zosiyanasiyana zolankhulirana kudzera mu NAT. Chochitika cha onicegatheringstatechange chikuyaka. Mu onicegatheringstatechange handler, timalola kuti kulumikizana ndi seva ya webrtc-signaling-server kusinthanitsa Mafotokozedwe a Gawo pakati pa anzathu:

peerConnection.oniceconnectionstatechange = (event) => {
      console.log('Connection state: ', peerConnection.iceConnectionState);

      if (peerConnection.iceConnectionState === 'connected') {
        // Можем активировать кнопку Start broadcast
        setBroadcasting(true);
        setBroadcastingBtnActive(true);
      }
    };

// Событие срабатывает сразу, как только добавился медаиапоток в peerConnection
peerConnection.onnegotiationneeded = (event) => {
      // Создаем и назначаем SDP offer
      peerConnection.createOffer().
        then((offer) => peerConnection.setLocalDescription(offer)).
        catch(console.error);
    };

// Событие срабатывает каждый раз, как появляется ICE кандидат
peerConnection.onicegatheringstatechange = (ev) => {
      let connection = ev.target;

      // Now we can activate broadcast button
      if (connection.iceGatheringState === 'complete') {
        let delay = 50;
        let tries = 0;
        let maxTries = 3;

        let timerId = setTimeout(function allowStreaming() {
          if (isOnline) {
            setBroadcastingBtnActive(true);
            return;
          }

          if (tries < maxTries) {
            tries += 1;
            delay *= 2;
            timerId = setTimeout(allowStreaming, delay);
          } else {
            // TODO: show user notification
            console.error("Can't connect to server");

            alert("Can't connect to server");
          }
        }, delay);
      }
    };

Webrtc-signaling-server ndi seva yofunikira kusinthanitsa kufotokozera gawoli pakati pa anzawo awiri, ikhoza kukhala websocket yosavuta kapena xhr-server pa PL iliyonse. Ntchito yake ndi yosavuta: kuvomereza kufotokozera kwa gawo kuchokera kwa anzawo ndikusamutsira wina.

Pambuyo pa kusinthana kwa mafotokozedwe a Gawoli, mbali zonse ziwiri zakonzeka kufalitsa ndi kulandira mavidiyo, kumbali yomwe imalandira mtsinje wa kanema, chochitika cha ontrack chimayambika pa peerConnection, mwa wothandizira omwe, mayendedwe omwe adalandira angaperekedwe. ndipo yang'anani pa interlocutor yomwe mumakonda. Chiphunzitso china ndi tsatanetsatane.

Maulalo ndi zolemba:

https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection - zolemba RTCPeerConnection

https://github.com/pion/webrtc - kukhazikitsa ma protocol a WebRTC popita

https://webrtcforthecurious.com/ - buku lochokera kwa omwe adalenga pion

https://hpbn.co/ - Book High Performance Browser Networking. Nkhani zowonetsetsa kuti ntchito zapaintaneti zikuyenda bwino zimakambidwa mwatsatanetsatane. Pamapeto pake, WebRTC ikufotokozedwa. Bukulo ndi lakale (2013), koma silitaya kufunikira kwake.

Mu gawo lotsatira, ndikufuna kupereka gawo lochulukirapo la chiphunzitso komanso kuchitapo kanthu kusanthula kulandila ndi kukonza mavidiyo pa seva pogwiritsa ntchito pion, transcoding ku HLS kudzera pa ffmpeg kuti iwulutsidwe kwa owonera mu msakatuli.

Kwa osapirira: chithunzi changa choyipa kwambiri chowulutsa kanema kuchokera pa webukamu kuti ndichite kudzera pa seva yochokera ku pion mu twitch (uku ndi kuyesa chabe).

Source: www.habr.com