Hi all!

It is probably no secret to anyone that cloud video surveillance services have been gaining popularity in recent years. And it’s understandable why this happens, video is “heavy” content, which requires infrastructure and large amounts of disk storage to store. Using a local video surveillance system requires funds for operation and support, both in the case of an organization using hundreds of surveillance cameras, and in the case of an individual user with several cameras.

Cloud video surveillance systems solve this problem by providing customers with an existing video storage and processing infrastructure. A cloud video surveillance client simply needs to connect the camera to the Internet and link it to their account in the cloud.

There are several technological ways to connect cameras to the cloud. Undoubtedly, the most convenient and cheapest way - the camera directly connects and works with the cloud, without the participation of additional equipment such as a server or registrar.

To do this, it is necessary that the software module working with the cloud is installed on the camera. However, if we talk about cheap cameras, then they have very limited hardware resources, which are almost 100% occupied by the native firmware of the camera vendor, but there are no resources necessary for the cloud plugin. Developers from ivideon dedicated to this problem Article, which says why they can't install the plugin on cheap cameras. As a result, the minimum price of the camera is 5000r ($80) and millions of money spent on equipment.

We have successfully solved this problem. If you are interested in how - wellcome under the cat

A bit of history

In 2016, we started developing a cloud video surveillance platform for Rostelecom.

In terms of camera software, at the first stage we went the “standard” way for such tasks: we developed our own plugin, which is installed in the firmware of the vendor’s camera and works with our cloud. However, it is worth noting that when designing, we used the most lightweight and efficient solutions (for example, plain C implementation of protobuf, libev, mbedtls and completely abandoned convenient but heavy libraries like boost)

Now there are no universal integration solutions on the IP camera market: each vendor has its own way of installing the plug-in, its own set of APIs for firmware operation and a unique update mechanism.

This means that for each camera vendor it is necessary to individually develop a volumetric layer of integration software. And at the time of the start of development, it is advisable to work with only 1 vendor in order to focus the team's efforts on developing the logic of working with the cloud.

Hikvision was chosen as the first vendor, one of the world leaders in the camera market, providing a well-documented API and competent engineering technical support.

Using Hikvision cameras, we launched our first pilot project - cloud video surveillance Videocomfort.

Almost immediately after the launch, our users began to ask questions about the possibility of connecting cheaper cameras from other manufacturers to the service.

I discarded the option of implementing an integration layer for each vendor almost immediately - as it is poorly scalable and imposes serious technical requirements on the camera hardware. The cost of a camera that meets these requirements at the entrance: ~ 60-70 $

Therefore, I decided to dig deeper - to make my own firmware for cameras of any vendors. This approach significantly reduces the requirements for camera hardware resources. the cloud layer is an order of magnitude more effectively integrated with the video application, and there is no extra unused fat in the firmware.

And what is important, when working with a camera at a low level, it is possible to use hardware AES, which encrypts data without creating additional load on a low-power CPU.

At that moment we had nothing at all. Nothing at all.

Almost all vendors were not ready to work with us at such a low level. There is no information about circuitry and components, there is no official chipset SDK and sensor documentation.
There is also no technical support.

Answers to all questions had to be obtained by reverse engineering - by trial and error. But we did it.

Xiaomi Yi Ants, Hikvision, Dahua, Spezvision, D-Link cameras and a few super cheap unnamed Chinese cameras were the first camera models on which we stuffed bumps.

Technique

Cameras based on the Hisilicon 3518E chipset. The hardware characteristics of the cameras are as follows:

Xiaomi Yi Ants
noname

SoC
Hisilicon 3518E
Hisilicon 3518E

RAM
64MB
64MB

FLASH
16MB
8MB

WiFi
mt7601/bcm43143
—

Sensor
ov9732 (720p)
ov9712 (720p)

Ethernet
—
+

MicroSD
+
+

Microphone
+
+

Speaker
+
+

IRLed
+
+

IRCut
+
+

We started with them.

We currently support Hisilicon 3516/3518 chipsets, as well as Ambarella S2L/S2LM. The number of camera models is dozens.

The composition of the firmware

submarine

uboot is the bootloader, after power on it boots first, initializes the hardware and loads the linux kernel.

The camera boot script is quite trivial:

bootargs=mem=38M console=ttyAMA0,115200 rootfstype=ramfs mtdparts=hi_sfc:256K(boot),64K(tech),4096K(kernel),8192K(app),-(config) hw_type=101
bootcmd=sf probe 0; sf read 0x82000000 0x50000 0x400000; bootm 0x82000000; setenv bootargs $(bootargs) bkp=1; sf read 0x82000000 0x450000 0x400000; bootm 0x82000000

Of the features - it is called twice bootm, more about this a little later, when we get to the update subsystem.

Pay attention to the line mem=38M. Yes, yes, this is not a typo - the Linux kernel and all-all-all applications have only 38 megabytes of RAM available.

Also next to uboot is a special block called reg_info, which contains a low-level DDR initialization script and a number of SoC system registers. Content reg_info depends on the camera model, and if it is not correct, then the camera will not even be able to load uboot, but will hang at the very early stage of loading.

At first, when we worked without vendor support, we simply copied this block from the original camera firmware.

Linux kernel and rootfs

The cameras use the Linux kernel, which is part of the SDK of the chip, usually these are not the latest kernels from the 3.x branch, so we often have to deal with the fact that additional equipment drivers are not compatible with the kernel used, and we have to back-port them to the kernel cameras.

Another problem is the kernel size. When the FLASH size is only 8MB, then every byte counts, and our task is to carefully disable all unused kernel functions in order to reduce the size to a minimum.

Rootfs is the base file system. It includes busybox, wifi module drivers, a set of standard system libraries, such as libld и libc, as well as software of our own development, which is responsible for the LED control logic, network connection management and firmware updates.

The root file system is connected to the kernel as an initramfs and as a result of the assembly we get one file uImage, which has both kernel and rootfs.

video application

The most complex and resource-intensive part of the firmware is an application that provides video-audio capture, video encoding, configures image parameters, implements video analytics, such as motion or sound detectors, controls PTZ and is responsible for switching day and night modes.

An important, I would even say a key feature is how the video application interacts with the cloud plugin.

In traditional 'firmware + cloud plugin' solutions that cannot work on cheap hardware, the video inside the camera is transmitted via RTSP protocol - and this is a huge overhead: copying and transmitting data via socket, extra syscalls.

At this point, we use the shared memory mechanism - the video is not copied or sent via a socket between the camera software components, thereby optimally and carefully using the camera's modest hardware capabilities.

Update Subsystem

The subject of separate pride is the fault-tolerant subsystem for online firmware updates.

Let me explain the problem. Updating the firmware is technically not an atomic operation, and if there is a power failure in the middle of the update, there will be a part of the “underwritten” new firmware on the flash memory. If you do not take special measures, then the camera will then become a "brick" that must be carried to the service center.

We have dealt with this problem as well. Even if the camera is turned off at the time of the update, it will automatically and without user intervention download the firmware from the cloud and restore operation.

Let's analyze the technique in more detail:

The most vulnerable point is overwriting the partition with the Linux kernel and the root file system. If one of these components turns out to be damaged, then the camera will not boot at all beyond the uboot bootloader, which cannot download firmware from the cloud.

This means that we need to ensure that the camera has a working kernel and rootfs at any time during the update process. It would seem that the simplest solution would be to permanently store two copies of the kernel with rootfs on flash memory and, in case of damage to the main kernel, load it from the backup copy.

A good solution - however, the kernel with rootfs takes about 3.5MB and 3.5MB needs to be allocated for a permanent backup. On the cheapest cameras, there is simply not so much free space for backup kernels.

Therefore, for the backup kernel during the firmware update, we use the application partition.
And to select the desired partition with the kernel, two commands are used bootm in uboot - at the beginning we try to load the main kernel and if it is damaged, then the backup one.

This ensures that at any given time the camera has the correct kernel with rootfs, so that it can boot and restore the firmware.

CI / CD system for building and deploying firmware

To build the firmware, we use gitlab CI, in which firmware for all supported camera models is automatically collected, after the firmware is built, it is automatically deployed to the camera software update service.

From the service, firmware updates are delivered to the test cameras of our QA, and upon completion of all testing stages, to the users' cameras.

Information Security

It's no secret that in our time, information security is the most important aspect of any IoT device, including cameras. Botnets like Mirai roam the Internet, attacking millions of cameras with standard firmware from vendors. With all due respect to camera vendors, I cannot help but note that the standard firmware contains a lot of functionality that is not required for working with the cloud, but contains many vulnerabilities that are used by botnets.

Therefore, all unused functionality in our firmware is disabled, all tcp / udp ports are closed, and when updating the firmware, the digital signature of the software is checked.

And besides this, the firmware is regularly tested in the information security laboratory.

Conclusion

Now our firmware is actively used in video surveillance projects. Perhaps the most ambitious of them is the broadcast of voting on the day of the election of the President of the Russian Federation.
The project involved more than 70 thousand cameras with our firmware, which were installed in the polling stations of our country.

Having solved a number of complex, and in some places, even at that time almost impossible tasks, we, of course, received great satisfaction as engineers, but besides this, we saved millions of dollars on the purchase of cameras. And in this case, savings are not only words and theoretical calculations, but the results of a tender for the purchase of equipment that has already happened. Accordingly, if we talk about cloud video surveillance: there are two approaches - to strategically rely on low-level expertise and development, resulting in huge savings on equipment, or to use expensive equipment, which, if you look specifically at consumer characteristics, is practically no different from similar cheap ones.

Why is it strategically important to make a decision regarding the choice of approach to the integration method as early as possible? When developing a plug-in, developers rely on certain technologies (libraries, protocols, standards). And if a set of technologies is chosen only for expensive equipment, then in the future an attempt to switch to cheap cameras will most likely take an insanely long time or even fail and there will be a return to expensive equipment.

Source: habr.com