AOL Inc. a system for capturing, storing, and indexing network packets , providing tools for visually assessing traffic streams and finding information related to network activity. The code is written in C (with an interface in Node.js/JavaScript) and is licensed under Apache 2.0. It supports operation in Linux and FreeBSD. Pre-packaged are prepared for various versions of CentOS and Ubuntu.
The project was created in 2012 with the goal of developing an open replacement for commercial packet processing platforms, capable of scaling to the traffic levels of AOL. The implementation of the new system at AOL enabled full control over the infrastructure by deploying it on their own servers and significantly reduced costs—the use of Moloch for complete traffic capture across all AOL networks cost about the same as what was previously spent that was exclusively capturing traffic in a single network. The system can scale to handle traffic at speeds of tens of gigabits per second. The volume of stored data is limited only by the size of the available storage array.
Session metadata is indexed in a cluster based on the engine .
Moloch includes tools for capturing and indexing traffic in standard PCAP format, as well as for quick access to indexed data. For analyzing the accumulated information, a web interface is provided, allowing for navigation, search, and export of samples. It also offers , which enables the transfer of data about captured packets in PCAP format and parsed sessions in JSON format to third-party applications. The use of the PCAP format significantly simplifies integration with existing traffic analyzers such as Wireshark.
Moloch consists of three basic components:
- The traffic capture system is a multithreaded application in C for traffic monitoring, recording dumps in PCAP format to disk, parsing captured packets, and sending metadata about sessions (SPI, Stateful Packet Inspection) and protocols to the Elasticsearch cluster. PCAP files can be stored in encrypted form.
- Web interface based on the Node.js platform, which runs on each traffic capture server and processes requests related to access to indexed data and transmission of PCAP files through .
- Metadata storage based on Elasticsearch.
The web interface provides several viewing modes — from overall statistics, connection maps, and visual graphs showing changes in network activity to tools for examining individual sessions, analyzing activity by the protocols used, and parsing data from PCAP dumps.
In :
- Transitioned to using a type-free format for indexing in Elasticsearch.
- Examples of traffic capture filters in Lua have been added.
- Support for the 46-draft version of the QUIC protocol has been implemented.
- The code for parsing protocols has been revamped, allowing for the creation of parsers for Ethernet and IP layer protocols.
- New parsers for arp, bgp, igmp, isis, lldp, ospf, and pim protocols have been proposed, as well as parsers for unknown protocols unkEthernet and unkIpProtocol.
- An option for selectively disabling parsers (disableParsers) has been added.
- The web interface now includes the ability to display any integer field on graphs, set on the settings page.
- Graphs and headers can now be pinned and will not shift when scrolling the page.
- Most navigation panels are hidden or collapsed by default.
Source: opennet.ru
