spam filtering system release , providing tools for assessing messages based on various criteria, including rules, statistical methods, and blacklists, which determine the final weight of the message used to decide whether blocking is necessary. Rspamd supports nearly all features implemented in SpamAssassin and has a number of characteristics that allow filtering emails on average 10 times faster than SpamAssassin, while also ensuring better quality of filtering. The system's code is written in C and under the Apache 2.0 license.
Rspamd is built using an event-driven architecture and is originally designed for use in high-load systems, allowing for the processing of hundreds of messages per second. Spam detection rules are highly flexible and can include regular expressions in their simplest form, while more complex scenarios can be implemented using Lua. Functionality extensions and new types of checks can be added through modules that can be created in C and Lua. For example, modules are available for sender verification using SPF, domain verification via DKIM, and forming queries to DNSBL lists. An administrative web interface is provided to simplify configuration, rule creation, and statistics tracking.
Key innovations:
- A transition to a new version numbering scheme has been made. Since the first digit of the version number has not changed for several years, and the real version indicator is the second digit, it has been decided to switch from the "x.y.z" scheme to the "y.z" format;
- For the event processing cycle instead of the library , which is free of some limitations of libevent and allows for higher performance. The use of
libev has simplified the code, improved signal and timeout handling, and unified file change tracking using the inotify mechanism (not all releases of libevent available for supported platforms could work with inotify); - Support for the message classification module using the deep learning library Torch has been discontinued. The reason given is the excessive complexity of Torch and the significant effort required to keep it up-to-date. A completely rewritten module has been proposed as a replacement for classification using machine learning methods. , in which the library is used to ensure the operation of the neural network. , contains only 4000 lines of C code. The new implementation resolves many issues related to deadlocks during training;
- Module has replaced the SURBL and Emails modules, allowing for the unification of processing all checks against blacklists. RBL's capabilities have been expanded to support additional types, such as selectors, and tools have been provided for simplifying the extension of existing rules. Email blocking rules based on map-lists instead of DNS RBL are no longer supported; instead, it is recommended to use multimap with selectors.
- To determine file types based on content, a new library Lua Magic has been employed, utilizing Lua and Hyperscan instead of libmagic.
The creation of a custom library is motivated by the desire to achieve higher performance, eliminate errors in detecting docx files, obtain a more suitable API, and add new types of heuristics that are not constrained by rigid rules. - The module for data storage in the DBMS has been improved. New LowCardinality fields have been added and memory consumption has been significantly optimized.
- The capabilities of the module , which now supports and mappings;
- The Maillist module has improved the detection of mailing lists.
- In workflows, the ability to send heartbeat messages to the main process has been introduced, confirming normal operation. If such messages are absent for a certain period, the main process can forcibly terminate the workflow. By default, this mode is currently disabled;
- A series of new scanners written in Lua has been added. For example, modules for scanning messages in Kaspersky ScanEngine, Trend Micro IWSVA (via icap), and
F-Secure Internet Gatekeeper (via icap) have been introduced, as well as external scanners for Razor, oletools, and P0F. - The ability to modify messages through the Lua API has been added. A module has been proposed for making changes to MIME blocks. ;
- Separate processing of settings set via 'Settings-Id:' has been ensured, for example, it is now possible to bind rules only to specific setting identifiers;
- Performance optimizations of the Lua engine, base64 decoding, and language detection for text have been conducted. Support for caching complex mappings (maps) has been added. Implementation support.
HTTP keep-alive.
Source: opennet.ru
