🥇(Almost) useless webcam streaming from the browser. Media Stream and Websockets

In this article, I want to share my attempts to stream video via websockets without using third-party browser plugins like Adobe Flash Player. What came of it, read on.

Adobe Flash, formerly Macromedia Flash, is a platform for creating applications that run in a web browser. Before the introduction of the Media Stream API, it was practically the only platform for streaming video and audio from a webcam, as well as for creating various kinds of conferences and chats in the browser. The protocol for transmitting media information, RTMP (Real Time Messaging Protocol), was effectively closed for a long time, which meant that if you wanted to set up your streaming service, you had to use software from Adobe itself—Adobe Media Server (AMS).

After some time in 2012, Adobe "gave up and spit out" to the public specification the RTMP protocol, which contained errors and was essentially incomplete. By that time, developers began to create their own implementations of this protocol, which led to the emergence of the Wowza server. In 2011, Adobe filed a lawsuit against Wowza for the illegal use of patents related to RTMP, and after 4 years, the conflict was resolved peacefully.

Adobe Flash has been around for over 20 years, during which many critical vulnerabilities have been discovered, and support were promised is set to end by 2020, so there aren't many alternatives for streaming services left.

For my project, I immediately decided to completely abandon the use of Flash in the browser. The main reason I mentioned above; also, Flash is not supported on mobile platforms, and I really didn't want to set up Adobe Flash for development on Windows (using the Wine emulator). Therefore, I set out to write a client in JavaScript. This will just be a prototype, as I later learned that streaming can be done much more efficiently based on p2p; it will be peer — server — peers, but more on that another time, as it's not ready yet.

To begin working, we actually need our own websockets server. I created the simplest one based on the Go package melody:

Server-side code

package main

import (
	"errors"
	"github.com/go-chi/chi"
	"gopkg.in/olahol/melody.v1"
	"log"
	"net/http"
	"time"
)

func main() {
	r := chi.NewRouter()
	m := melody.New()

	m.Config.MaxMessageSize = 204800

	r.Get("/", func(w http.ResponseWriter, r *http.Request) {
		http.ServeFile(w, r, "public/index.html")
	})
	r.Get("/ws", func(w http.ResponseWriter, r *http.Request) {
		m.HandleRequest(w, r)
	})

		// Broadcasting video stream 
	m.HandleMessageBinary(func(s *melody.Session, msg []byte) {
		m.BroadcastBinary(msg)
	})

	log.Println("Starting server...")

	http.ListenAndServe(":3000", r)
}

On the client side (the transmitting side), access to the camera must first be obtained. This is done through MediaStream API.

Access (permission) to the camera/microphone via Media Devices API. This API provides the method MediaDevices.getUserMedia(), which shows a popup window asking the user for permission to access the camera and/or microphone. I would like to note that all experiments were conducted in Google Chrome, but I believe it will work similarly in Firefox.

Next, getUserMedia() returns a Promise, which passes a MediaStream object — a stream of audio-video data. This object is assigned to the src property of the video element. Code:

Transmitting side

<style>
  #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; }
</style>
</head>
<body>
<!-- Здесь в этом "окошечке" клиент будет видеть себя -->
<video autoplay id="videoObjectHtml5ApiServer"></video>

<script type="application/javascript">
  var
        video = document.getElementById('videoObjectHtml5ApiServer');

// если доступен MediaDevices API, пытаемся получить доступ к камере (можно еще и к микрофону)
// getUserMedia вернет обещание, на которое подписываемся и полученный видеопоток в колбеке направляем в video объект на странице

if (navigator.mediaDevices.getUserMedia) {
        navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) {
          // видео поток привязываем к video тегу, чтобы клиент мог видеть себя и контролировать 
          video.srcObject = stream;
        });
}
</script>

To transmit a video stream over sockets, it must be encoded, buffered, and sent in parts. A raw video stream cannot be sent via websockets. This is where MediaRecorder API. This API allows encoding and splitting the stream into chunks. I encode it to compress the video stream to send fewer bytes over the network. After splitting into chunks, each chunk can be sent via websocket. Code:

We encode the video stream, splitting it into parts

<style>
  #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; }
</style>
</head>
<body>
<!-- Здесь в этом "окошечке" клиент будет видеть себя -->
<video autoplay id="videoObjectHtml5ApiServer"></video>

<script type="application/javascript">
  var
        video = document.getElementById('videoObjectHtml5ApiServer');

// если доступен MediaDevices API, пытаемся получить доступ к камере (можно еще и к микрофону)
// getUserMedia вернет обещание, на которое подписываемся и полученный видеопоток в колбеке направляем в video объект на странице

if (navigator.mediaDevices.getUserMedia) {
        navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) {
          // видео поток привязываем к video тегу, чтобы клиент мог видеть себя и контролировать 
          video.srcObject = s;
          var
            recorderOptions = {
                mimeType: 'video/webm; codecs=vp8' // будем кодировать видеопоток в формат webm кодеком vp8
              },
              mediaRecorder = new MediaRecorder(s, recorderOptions ); // объект MediaRecorder

               mediaRecorder.ondataavailable = function(e) {
                if (e.data && e.data.size > 0) {
                  // получаем кусочек видеопотока в e.data
                }
            }

            mediaRecorder.start(100); // делит поток на кусочки по 100 мс каждый

        });
}
</script>

Now let's add transmission via websockets. Surprisingly, all you need is the object WebSocket. It has only two methods, send and close. The names speak for themselves. The updated code:

We transmit the video stream to the server

<style>
  #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; }
</style>
</head>
<body>
<!-- Здесь в этом "окошечке" клиент будет видеть себя -->
<video autoplay id="videoObjectHtml5ApiServer"></video>

<script type="application/javascript">
  var
        video = document.getElementById('videoObjectHtml5ApiServer');

// если доступен MediaDevices API, пытаемся получить доступ к камере (можно еще и к микрофону)
// getUserMedia вернет обещание, на которое подписываемся и полученный видеопоток в колбеке направляем в video объект на странице

if (navigator.mediaDevices.getUserMedia) {
        navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) {
          // видео поток привязываем к video тегу, чтобы клиент мог видеть себя и контролировать 
          video.srcObject = s;
          var
            recorderOptions = {
                mimeType: 'video/webm; codecs=vp8' // будем кодировать видеопоток в формат webm кодеком vp8
              },
              mediaRecorder = new MediaRecorder(s, recorderOptions ), // объект MediaRecorder
              socket = new WebSocket('ws://127.0.0.1:3000/ws');

               mediaRecorder.ondataavailable = function(e) {
                if (e.data && e.data.size > 0) {
                  // получаем кусочек видеопотока в e.data
                 socket.send(e.data);
                }
            }

            mediaRecorder.start(100); // делит поток на кусочки по 100 мс каждый

        }).catch(function (err) { console.log(err); });
}
</script>

The transmitting side is ready! Now let's try to receive the video stream and display it on the client. What do we need for this? First of all, a socket connection, of course. We attach a listener to the WebSocket object and subscribe to the 'message' event. Upon receiving a piece of binary data, our server broadcasts it to its subscribers, meaning clients. On the client side, a callback function associated with the 'message' event listener is triggered, receiving the object itself — a piece of the video stream encoded in vp8.

Receiving the video stream

<style>
  #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; }
</style>
</head>
<body>
<!-- Здесь в этом "окошечке" клиент будет видеть тебя -->
<video autoplay id="videoObjectHtml5ApiServer"></video>

<script type="application/javascript">
  var
        video = document.getElementById('videoObjectHtml5ApiServer'),
         socket = new WebSocket('ws://127.0.0.1:3000/ws'), 
         arrayOfBlobs = [];

         socket.addEventListener('message', function (event) {
                // "кладем" полученный кусочек в массив 
                arrayOfBlobs.push(event.data);
                // здесь будем читать кусочки
                readChunk();
            });
</script>

I spent a long time trying to understand why it was impossible to send the received chunks directly for playback to the video element, but it turned out that this couldn't be done; first, you need to place the chunk in a special buffer associated with the video element, and only then will the video stream start playing. For this, you will need MediaSource API and FileReader API.

MediaSource acts as an intermediary between the media playback object and the source of the media stream. The MediaSource object contains a buffer connected to the video/audio stream source. One feature is that the buffer can only contain data of type Uint8, so creating such a buffer will require FileReader. Look at the code, and it will become clearer:

Playing video stream

<style>
  #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; }
</style>
</head>
<body>
<!-- Здесь в этом "окошечке" клиент будет видеть тебя -->
<video autoplay id="videoObjectHtml5ApiServer"></video>

<script type="application/javascript">
  var
        video = document.getElementById('videoObjectHtml5ApiServer'),
         socket = new WebSocket('ws://127.0.0.1:3000/ws'),
        mediaSource = new MediaSource(), // объект MediaSource
        vid2url = URL.createObjectURL(mediaSource), // создаем объект URL для связывания видеопотока с проигрывателем
        arrayOfBlobs = [],
        sourceBuffer = null; // буфер, пока нуль-объект

         socket.addEventListener('message', function (event) {
                // "кладем" полученный кусочек в массив 
                arrayOfBlobs.push(event.data);
                // здесь будем читать кусочки
                readChunk();
            });

         // как только MediaSource будет оповещен , что источник готов отдавать кусочки 
        // видео/аудио потока
        // создаем буфер , следует обратить внимание, что буфер должен знать в каком формате 
        // каким кодеком был закодирован поток, чтобы тем же способом прочитать видеопоток
         mediaSource.addEventListener('sourceopen', function() {
            var mediaSource = this;
            sourceBuffer = mediaSource.addSourceBuffer("video/webm; codecs="vp8"");
        });

      function readChunk() {
        var reader = new FileReader();
        reader.onload = function(e) { 
          // как только FileReader будет готов, и загрузит себе кусочек видеопотока
          // мы "прицепляем" перекодированный в Uint8Array (был Blob) кусочек в буфер, связанный
          // с проигрывателем, и проигрыватель начинает воспроизводить полученный кусочек видео/аудио
          sourceBuffer.appendBuffer(new Uint8Array(e.target.result));

          reader.onload = null;
        }
        reader.readAsArrayBuffer(arrayOfBlobs.shift());
      }
</script>

The prototype of the streaming service is ready. The main downside is that the video playback will lag by 100 ms from the transmitting side, which we set ourselves when slicing the video stream before sending it to the server. Moreover, when I checked it on my laptop, I gradually noticed a lag accumulating between the transmitting and receiving sides, which was very noticeable. I started looking for ways to overcome this issue and… stumbled upon RTCPeerConnection API, which allows transmitting the video stream without tricks like breaking the stream into chunks. The accumulating lag, I think, is due to the fact that each chunk is re-encoded in the webm format in the browser before transmission. I didn’t dig deeper and started studying WebRTC; I think I’ll write a separate article about the results of my research if I find it interesting for the community.

Source: habr.com