release of spam filtering system , which provides tools for evaluating messages according to various criteria, including rules, statistical methods, and blacklists, on the basis of which the final weight of the message is formed, which is used to decide whether to block. Rspamd supports almost all the features implemented in SpamAssassin and has a number of features that allow you to filter mail on average 10 times faster than SpamAssassin, as well as provide better filtering quality. The system code is written in C language and licensed under Apache 2.0.
Rspamd is built using an event-driven architecture and was originally designed for use in high-load systems, allowing it to process hundreds of messages per second. The rules for detecting spam signs are highly flexible and can contain regular expressions in the simplest form, and in more complex situations they can be written in the Lua language. Extending the functionality and adding new types of checks is implemented through modules that can be created in the C and Lua languages. For example, modules are available for verifying the sender using SPF, verifying the sender's domain via DKIM, and querying DNSBL lists. An administrative web interface is provided to simplify configuration, create rules, and track statistics.
Main innovations:
- Transition to the new numbering scheme of releases is carried out. Since the first number in the version number has not changed for several years, and the real indicator of the version is the second number, it was decided to switch to the "yz" format instead of the "xyz" scheme;
- For the event loop, instead of library involved , which removes some of the limitations of libevent and allows for better performance. Usage
libev made it possible to simplify code, improve signal and timeout handling, and unify file change tracking using the inotify mechanism (not all releases of libevent shipped for supported platforms could work with inotify); - The message classification module that uses the Torch deep learning library has been discontinued. The reason is the excessive complexity of Torch and the high complexity of maintaining it up to date. A completely rewritten module is proposed as a replacement for classification using machine learning methods. , in which the library is used to ensure the operation of the neural network , which includes only 4000 lines of C code. In the new implementation, many problems with the occurrence of deadlocks during training have been solved;
- Module replaced the SURBL and Emails modules, which made it possible to unify the processing of all blacklist checks. RBL's capabilities have been extended with support for additional types such as selectors and tools to easily extend existing rules. Email blocking rules based on map lists instead of DNS RBL are no longer supported, it is recommended to use multimap with selectors instead;
- To determine file types by content, the new Lua Magic library is used, using Lua and Hyperscan instead of libmagic.
The reason given for creating our own library is to achieve better performance, get rid of crashes when detecting docx files, get a better API, and add new kinds of heuristics that are not limited by hard and fast rules; - Improved module for storing data in DBMS . Added LowCardinality fields and significantly optimized memory consumption;
- Extended module capabilities , which has support ΠΈ comparisons;
- In the Maillist module, the definition of mailing lists has been improved;
- Worker processes now have the ability to send heartbeat messages to the main process confirming normal operation. In the absence of such messages for a certain time, the main process can forcefully terminate the worker process. By default, this mode is currently disabled;
- Added a series of new Lua scanners. For example, modules for scanning messages have been added to the anti-virus packages of Kaspersky ScanEngine, Trend Micro IWSVA (via icap ) and
F-Secure Internet Gatekeeper (via icap), as well as external scanners for Razor, oletools and P0F; - Added the ability to change messages through the Lua API. To make changes to MIME blocks, a module is proposed ;
- Separate processing of settings set via "Settings-Id:" is provided, for example, now you can bind rules only to certain settings identifiers;
- Optimized the performance of the Lua engine, base64 decoding and language detection for text. Added support for caching complex mappings (maps). Implemented support
HTTP keep-alive.
Source: opennet.ru