AOL Company release of a system for capturing, storing and indexing network packets , which provides tools for visually assessing traffic flows and searching for information related to network activity. The code is written in C language (interface in Node.js/JavaScript) and licensed under Apache 2.0. Work in Linux and FreeBSD is supported. Finished prepared for different versions of CentOS and Ubuntu.
The project was created in 2012 with the goal of creating an open source replacement for a commercial network packet processing platform that can scale to AOL's traffic volumes. The introduction of the new system at AOL allowed them to achieve complete control over the infrastructure by deploying it on their servers and significantly reduce costs - using Moloch to completely capture traffic in all AOL networks cost a similar amount to that when using used to be spent on capturing traffic in only one network. The system can scale to process traffic at speeds of tens of gigabits per second. The amount of stored data is limited only by the size of the available disk array.
Session metadata is indexed in the engine-based cluster .
Moloch includes tools for capturing and indexing traffic in native PCAP format, as well as for quick access to indexed data. To analyze the accumulated information, a web-interface is proposed that allows you to navigate, search and export samples. Also provided , which allows you to transfer data about captured packets in PCAP format and parsed sessions in JSON format to third-party applications. Using the PCAP format greatly simplifies integration with existing traffic analyzers such as Wireshark.
Moloch consists of three basic components:
- The traffic capture system is a multi-threaded C application for monitoring traffic, writing PCAP dumps to disk, parsing captured packets, and sending stateful packet inspection (SPI) and protocol metadata to an Elasticsearch cluster. It is possible to store PCAP files in encrypted form.
- A Node.js-based web interface that runs on each traffic capture server and processes requests related to accessing indexed data and transferring PCAP files through .
- Metadata storage based on Elasticsearch.
The web interface provides several viewing modes - from general statistics, connection maps and visual graphs with data on changes in network activity to tools for studying individual sessions, analyzing activity in the context of the protocols used, and parsing data from PCAP dumps.
Π :
- The transition to the use of a typeless format for indexing in Elasticsearch has been made.
- Added examples of traffic capture filters in Lua.
- Implemented support for the 46-draft edition of the QUIC protocol.
- The code for parsing protocols has been redesigned, it has become possible to write parsers for Ethernet and IP protocols.
- New parsers for arp, bgp, igmp, isis, lldp, ospf and pim protocols, as well as parsers for unknown protocols unkEthernet and unkIpProtocol are proposed.
- Added an option to selectively disable parsers (disableParsers).
- The ability to display any integer field on charts, set on the settings page, has been added to the web interface.
- Graphs and titles can now be docked and not move when scrolling the page.
- Most navigation bars are hidden or collapsed by default.
Source: opennet.ru