Exploring the Mediastreamer2 VoIP engine. Part 1

The material of the article is taken from my zen channel.

Introduction

This article is the beginning of a series of articles about real-time media processing using the Mediastreamer2 engine. The presentation will involve the minimum skills of working in the Linux terminal and programming in the C language.

Mediastreamer2 is the VoIP engine behind the popular open-source software voip phone project. Linphone. In Linphone Mediastreamer2 implements all functions related to sound and video. A detailed list of engine features can be seen on this Mediastreamer page. The source code is here: GitLab.

Further in the text, for convenience, instead of the word Mediastreamer2 we will use its Russian notation: “media streamer”.

The history of its creation is not entirely clear, but judging by its source code, it previously used the library glib, which, as it were, hints at a possible distant relationship with GStreamer. In comparison with which the media streamer looks more lightweight. The first version of Linphone appeared in 2001, so at the moment the media streamer exists and develops for almost 20 years.

At the heart of the media streamer is an architecture called "Data flow" (data flow). An example of such an architecture is shown in the figure below.

In this architecture, the data processing algorithm is specified not by a program code, but by a scheme (graph) for connecting functions that can be arranged in any order. These functions are called filters.

This architecture makes it possible to implement the media processing functionality in the form of a set of filters connected to the VoIP phone RTP traffic processing and transmission scheme.

The ability to combine filters into arbitrary schemes, the simple development of new filters, the implementation of the media streamer as an independent separate library, allow it to be used in other projects. Moreover, the project can be in the field of VoIP, since it is possible to add filters made by oneself.

The filter library supplied by default is quite rich and, as already mentioned, can be extended with filters of our own design. But first, let's describe the ready-made filters that come with the media streamer. Here is their list:

Sound filters

Audio capture and playback

• Alsa (Linux): MS_ALSA_WRITE, MS_ALSA_READ
• Android native sound (libmedia): MS_ANDROID_SOUND_WRITE, MS_ANDROID_SOUND_READ
• Audio Queue Service (Mac OS X): MS_AQ_WRITE, MS_AQ_READ
• Audio Unit Service (Mac OS X)
• Arts (Linux): MS_ARTS_WRITE, MS_ARTS_READ
• DirectSound (Windows): MS_WINSNDDS_WRITE, MS_WINSNDDS_READ
• File player (raw/wav/pcap files) (Linux): MS_FILE_PLAYER
• File player (raw/wav files) (Windows): MS_WINSND_READ
• Write to file (wav files) (Linux): MS_FILE_REC
• Write to file (wav files) (Windows): MS_WINSND_WRITE
• Mac Audio Unit (Mac OS X)
• MME (Windows)
• OSS (Linux): MS_OSS_WRITE, MS_OSS_READ
• PortAudio (Mac OS X)
• PulseAudio (Linux): MS_PULSE_WRITE, MS_PULSE_READ
• Windows Sound (Windows)

Audio encoding/decoding

• G.711 a-law: MS_ALAW_DEC, MS_ALAW_ENC
• G.711 µ-law: MS_ULAW_DEC, MS_ULAW_ENC
• G.722: MS_G722_DEC, MS_G722_ENC
• G.726: MS_G726_32_ENC, MS_G726_24_ENC, MS_G726_16_ENC
• GSM: MS_GSM_DEC, MS_GSM_ENC
• Linear PCM: MS_L16_ENC, MS_L16_DEC
• Speex: MS_SPEEX_ENC, MS_SPEEX_DEC

Sound processing

• Channel conversion (mono->stereo, stereo->mono): MS_CHANNEL_ADAPTER
• Conference: MS_CONF
• DTMF Generator: MS_DTMF_GEN
• Echo cancellation (speex): MS_SPEEX_EC
• Equalizer: MS_EQUALIZER
• Mixer: MS_MIXER
• Packet Loss Compensator (PLC): MS_GENERIC_PLC
• Resampler: MS_RESAMPLE
• Tone detector: MS_TONE_DETECTOR
• Volume control and signal level measurement: MS_VOLUME

Video filters

Video capture and playback

• android capture
• android playback
• AV Foundation capture (iOS)
• AV Foundation playback (iOS)
• DirectShow Capture (Windows)
• DrawDib playback (Windows)
• External playback - Sending video to the top layer
• GLX playback (Linux): MS_GLXVIDEO
• Mire - Synthetic moving picture: MS_MIRE
• OpenGL playback (Mac OS X)
• OpenGL ES2 playback (Android)
• Quicktime Capture (Mac OS X)
• SDL playback: MS_SDL_OUT
• Static image output: MS_STATIC_IMAGE
• Video For Linux (V4L) capture (Linux): MS_V4L
• Video For Linux 2 (V4L2) capture (Linux): MS_V4L2_CAPTURE
• Video4windows (DirectShow) capture (Windows)
• Video4windows (DirectShow) capture (Windows CE)
• Video For Windows (vfw) capture (Windows)
• XV playback (Linux)

Video encoding/decoding

• H.263, H.263-1998, MP4V-ES, JPEG, MJPEG, Snow: MS_MJPEG_DEC, MS_H263_ENC, MS_H263_DEC
• H.264 (decoder only): MS_H264_DEC
• Theora: MS_THEORA_ENC, MS_THEORA_DEC
• VP8: MS_VP8_ENC, MS_VP8_DEC

Video processing

• jpeg snapshot
• Pixel format converter: MS_PIX_CONV
• Resizer
• Other filters
• Exchange of data blocks between threads: MS_ITC_SOURCE, MS_ITC_SINK
• Collecting blocks of data from multiple inputs to a single output: MS_JOIN
• RTP receive/transmit: MS_RTP_SEND, MS_RTP_RECV
• Copying input data to multiple outputs: MS_TEE
• Terminated load: MS_VOID_SINK
• Silence Source: MS_VOID_SOURCE

Plugins

Sound filters

• AMR-NB encoder/decoder
• G.729 encoder/decoder
• iLBC encoder/decoder
• SILK encoder/decoder
  

  Video filters

• H.264 software encoder
• H.264 V4L2 hardware accelerated encoder/decoder

After a short description of the filter, the name of the type is shown, which is used when creating a new instance of this filter. In what follows, we will refer to this list.

Installation under Linux Ubuntu

Now we will install the media streamer on the computer and build our first application with it.

Installing Mediastremer2 on a computer or virtual machine running Ubuntu does not require any special skills. Here and below, the symbol "$" will denote the shell prompt for entering commands. Those. if in the listing you see this symbol at the beginning of the line, then this is the line in which commands are shown to be executed in the terminal.

It is assumed that during the steps in this article, your computer has access to the Internet.

Installing the libmediastremer-dev package

Launch the terminal and type the command:

$ sudo apt-get update

You will be asked for a password to make changes, enter it and the package manager will update its databases. After that, you need to run:

$ sudo apt-get install libmediastreamer-dev

The necessary dependency packages and the media streamer library itself will be automatically downloaded and installed.

The total size of the downloaded dependency deb packages will be approximately 35 MB. Details about the installed package can be found with the command:

$ dpkg -s libmediastreamer-dev

Answer example:

Package: libmediastreamer-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 244
Maintainer: Ubuntu Developers <[email protected]>
Architecture: amd64
Source: linphone
Version: 3.6.1-2.5
Depends: libmediastreamer-base3 (= 3.6.1-2.5), libortp-dev
Description: Linphone web phone's media library - development files
Linphone is an audio and video internet phone using the SIP protocol. It
has a GTK+ and console interface, includes a large variety of audio and video
codecs, and provides IM features.
.
This package contains the development libraries for handling media operations.
Original-Maintainer: Debian VoIP Team <[email protected]>
Homepage: http://www.linphone.org/

Installing development tools

Install the C compiler and its accompanying tools:

$ sudo apt-get install gcc

We check the result by requesting the compiler version:

$ gcc --version

The answer should be something like this:

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Building and Running the Trial Application

We create in home folder for our tutorial projects, let's call it mstutorial:

$ mkdir ~/mstutorial

Use your favorite text editor and create a C program file called mstest.c with the following content:

#include "stdio.h"
#include <mediastreamer2/mscommon.h>
int main()
{
  ms_init();
  printf ("Mediastreamer is ready.n");
}

It initializes the media streamer, prints a greeting, and exits.

Save the file and compile the test application with the command:

$ gcc mstest.c -o mstest `pkg-config mediastreamer --libs --cflags`

Note that the line

`pkg-config mediastreamer --libs --cflags`

enclosed in quotation marks, which are located on the keyboard in the same place as the letter "Ё".

If the file does not contain errors, then after compilation a file will appear in the directory mstest. We start the program:

$ ./mstest

The result will be like this:

ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL default:0
ortp-warning-Could not attach mixer to card: Invalid argument
ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default:0
ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default:0
ortp-warning-Strange, sound card HDA Intel PCH does not seems to be capable of anything, retrying with plughw...
Mediastreamer is ready.

In this listing, we see the error messages that the ALSA library displays, it is used to control the sound card. The developers of the media streamer themselves believe that this is normal. In this case, we reluctantly agree with them.

Now we are all set to work with the media streamer. We have installed the media streamer library, the compilation tool, and using a trial application, verified that the tools are configured and the media streamer initializes successfully.

Next article we will create an application that will assemble and run the processing of an audio signal in a chain of several filters.

Source: habr.com