🥇Exploring the VoIP engine Mediastreamer2. Part 1

The article material is taken from my Zen channel.

Introduction

This article begins a series on real-time media data processing using the Mediastreamer2 engine. It will require basic Linux terminal skills and some programming knowledge in C.

Mediastreamer2 is a VoIP engine that underpins the popular open-source VoIP phone project. Linphone. In Linphone, Mediastreamer2 implements all functions related to audio and video. A detailed list of the engine's capabilities can be seen on this Mediastreamer page. The source code is available here: GitLab.

For convenience, we will use the Russian term "медиастример" instead of Mediastreamer2 throughout the text.

The history of its creation is not entirely clear, but judging by its source code, it previously used the library Glib, which hints at a possible distant relationship with GStreamer. Compared to which, Mediastreamer appears to be more lightweight. The first version of Linphone was released in 2001, so Mediastreamer has been around and evolving for nearly 20 years.

The core of Mediastreamer is based on an architecture called "Data flow." An example of such an architecture is depicted in the illustration below.

In this architecture, the data processing algorithm is defined not by program code but by a diagram (graph) of the connection of functions, which can be arranged in any order. These functions are called filters.

This architecture allows implementing media processing functionality as a set of filters connected in a scheme for processing and transmitting RTP traffic of a VoIP phone.

The ability to connect filters in arbitrary schemes, the ease of developing new filters, and the implementation of Mediastreamer as a standalone library enable its use in other projects. Moreover, the project can be in the VoIP field, as there is an option to add custom-made filters.

The default library of filters is quite rich and, as mentioned earlier, can be extended with filters of custom development. But first, let's describe the ready-made filters that come with Mediastreamer. Here's the list:

Audio Filters

Capture and playback of audio

Alsa (Linux): MS_ALSA_WRITE, MS_ALSA_READ
Android native sound (libmedia): MS_ANDROID_SOUND_WRITE, MS_ANDROID_SOUND_READ
Audio Queue Service (Mac OS X): MS_AQ_WRITE, MS_AQ_READ
Audio Unit Service (Mac OS X)
Arts (Linux): MS_ARTS_WRITE, MS_ARTS_READ
DirectSound (Windows): MS_WINSNDDS_WRITE, MS_WINSNDDS_READ
File Player (raw/wav/pcap files) (Linux): MS_FILE_PLAYER
File Player (raw/wav files) (Windows): MS_WINSND_READ
File Recording (wav files) (Linux): MS_FILE_REC
File Recording (wav files) (Windows): MS_WINSND_WRITE
Mac Audio Unit (Mac OS X)
MME (Windows)
OSS (Linux): MS_OSS_WRITE, MS_OSS_READ
PortAudio (Mac OS X)
PulseAudio (Linux): MS_PULSE_WRITE, MS_PULSE_READ
Windows Sound (Windows)

Audio Encoding/Decoding

G.711 a-law: MS_ALAW_DEC, MS_ALAW_ENC
G.711 µ-law: MS_ULAW_DEC, MS_ULAW_ENC
G.722: MS_G722_DEC, MS_G722_ENC
G.726: MS_G726_32_ENC, MS_G726_24_ENC, MS_G726_16_ENC
GSM: MS_GSM_DEC, MS_GSM_ENC
Linear PCM: MS_L16_ENC, MS_L16_DEC
Speex: MS_SPEEX_ENC, MS_SPEEX_DEC

Audio Processing

Channel Conversion (mono->stereo, stereo->mono): MS_CHANNEL_ADAPTER
Conference: MS_CONF
DTMF Generator: MS_DTMF_GEN
Echo Suppression (speex): MS_SPEEX_EC
Equalizer: MS_EQUALIZER
Mixer: MS_MIXER
Packet Loss Concealment (PLC): MS_GENERIC_PLC
Resampler: MS_RESAMPLE
Tone Detector: MS_TONE_DETECTOR
Volume Control and Signal Level Measurement: MS_VOLUME

Video Filters

Video Capture and Playback

Android Capture
Android Playback
AV Foundation Capture (iOS)
AV Foundation Playback (iOS)
DirectShow Capture (Windows)
DrawDib Playback (Windows)
External Playback — Sending Video to the Top Layer
GLX Playback (Linux): MS_GLXVIDEO
Mire — Synthetic Moving Picture: MS_MIRE
OpenGL Playback (Mac OS X)
OpenGL ES2 Playback (Android)
Quicktime Capture (Mac OS X)
SDL Playback: MS_SDL_OUT
Static Image Output: MS_STATIC_IMAGE
Video For Linux (V4L) Capture (Linux): MS_V4L
Video For Linux 2 (V4L2) Capture (Linux): MS_V4L2_CAPTURE
Video4Windows (DirectShow) Capture (Windows)
Video4Windows (DirectShow) Capture (Windows CE)
Video For Windows (vfw) Capture (Windows)
XV Playback (Linux)

Video encoding/decoding

H.263, H.263-1998, MP4V-ES, JPEG, MJPEG, Snow: MS_MJPEG_DEC, MS_H263_ENC, MS_H263_DEC
H.264 (decoder only): MS_H264_DEC
Theora: MS_THEORA_ENC, MS_THEORA_DEC
VP8: MS_VP8_ENC, MS_VP8_DEC

Video Processing

JPEG Snapshot
Pixel Format Converter: MS_PIX_CONV
Resizer
Other Filters
Data Block Exchange Between Threads: MS_ITC_SOURCE, MS_ITC_SINK
Collecting Data Blocks from Multiple Inputs into One Output: MS_JOIN
RTP Receive/Send: MS_RTP_SEND, MS_RTP_RECV
Copying Input Data to Multiple Outputs: MS_TEE
Coherent Load: MS_VOID_SINK
Silence Source: MS_VOID_SOURCE

Plugins

Audio Filters

AMR-NB Encoder/Decoder
G.729 Encoder/Decoder
iLBC Encoder/Decoder
SILK Encoder/Decoder
Video Filters
H.264 Software Encoder
H.264 V4L2 Encoder/Decoder with Hardware Acceleration

The type name used for creating a new instance of this filter is shown after the brief filter description. In the subsequent text, we will refer to this list.

Installation under Linux Ubuntu

Now we will install the media streamer on the computer and build our first application with it.

Installing Mediastremer2 on a computer or virtual machine running Ubuntu does not require special skills. Here and thereafter, the symbol "$" will denote the shell prompt for inputting commands. That is, if you see this symbol at the beginning of a line in the listing, it means that this line shows the commands to be executed in the terminal.

It is assumed that during the execution of the actions described in this article, your computer has access to the Internet.

Installing the libmediastremer-dev package

We open the terminal and type the command:

$ sudo apt-get update

You will be prompted to enter a password for making changes; enter it and the package manager will update its databases. After that, you need to execute:

$ sudo apt-get install libmediastreamer-dev

The necessary dependency packages and the media streamer library will be automatically downloaded and installed.

The total size of the downloaded deb dependency packages will be approximately 35 MB. Details about the installed package can be obtained using the command:

$ dpkg -s libmediastreamer-dev

Example response:

Package: libmediastreamer-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 244
Maintainer: Ubuntu Developers 
Architecture: amd64
Source: linphone
Version: 3.6.1-2.5
Depends: libmediastreamer-base3 (= 3.6.1-2.5), libortp-dev
Description: Linphone web phone's media library - development files
Linphone is an audio and video internet phone using the SIP protocol. It
has a GTK+ and console interface, includes a large variety of audio and video
codecs, and provides IM features.
.
This package contains the development libraries for handling media operations.
Original-Maintainer: Debian VoIP Team 
Homepage: http://www.linphone.org/

Installing development tools

We install the C compiler and its related tools:

$ sudo apt-get install gcc

Check the result by querying the version of the compiler:

$ gcc --version

The response should be approximately as follows:

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Building and running a test application

We create in home a folder for our training projects, let's call it mstutorial:

$ mkdir ~/mstutorial

Use your favorite text editor to create a C program file named mstest.c with the following content:

#include "stdio.h"
#include <mediastreamer2/mscommon.h>
int main()
{
  ms_init();
  printf ("Mediastreamer is ready.n");
}

It initializes the media streamer, prints a greeting, and ends execution.

Save the file and compile the test application with the command:

$ gcc mstest.c -o mstest `pkg-config mediastreamer --libs --cflags`

Note that the line

`pkg-config mediastreamer --libs --cflags`

is enclosed in quotation marks, which are located on the keyboard where the letter «Ё» is.

If the file has no errors, then after compilation a file will appear in the directory mstest. Let's run the program:

$ .\/mstest

The result will be as follows:

ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL default:0
ortp-warning-Could not attach mixer to card: Invalid argument
ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default:0
ALSA lib conf.c:4738:(snd_config_expand) Unknown parameters 0
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM default:0
ortp-warning-Strange, sound card HDA Intel PCH does not seem to be capable of anything, retrying with plughw...
Mediastreamer is ready.

In this listing, we see error messages produced by the ALSA library, which is used for managing the sound card. The mediastreamer developers consider this to be normal. In this case, we reluctantly agree with them.

Now we are all set to work with the mediastreamer. We have installed the mediastreamer library, the compilation tool, and verified through a test application that the tools are configured, and the mediastreamer initializes successfully.

In the next article we will create an application that will collect and process sound signals in a chain of several filters.

Source: habr.com