significant release of the distributed event processing system , notable for its transition to a new architecture implemented in Java, instead of the previously used Clojure language.
The project enables guaranteed processing of various events in real time. For example, Storm can be used for real-time data stream analysis, performing machine learning tasks, organizing continuous computations, implementing RPC, ETL, etc. The system supports clustering, creation of fault-tolerant configurations, guaranteed data processing mode, and has high performance, sufficient to handle over a million requests per second on a single cluster node.
Integration with various queuing systems and database technologies is supported. The Storm architecture implies the reception and processing of unstructured continuously updating data streams using arbitrary complex processors with the ability to partition between different stages of computation. The project was handed over to the Apache community after Twitter acquired BackType, the company that originally developed the framework. In practice, Storm was used in BackType for analyzing event reflections in microblogs by matching newly posted tweets and the links they contained in real time (for example, evaluating how external links or Twitter announcements are retweeted by other participants).
The functionality of Storm is compared to the Hadoop platform, with the key difference being that data is not stored but comes from external sources and is processed in real time. Storm does not have a built-in layer for organizing storage, and analytical queries are applied to incoming data as long as they are not canceled (while Hadoop uses time-consuming MapReduce jobs, Storm implements the idea of continuously running 'topologies'). The execution of processors can be distributed across multiple servers — Storm automatically parallelizes stream processing across different cluster nodes.
Initially, the system was written in Clojure and runs within the JVM. The Apache Foundation launched an initiative to port Storm to a new core written in Java, the results of which are offered in the Apache Storm 2.0 release. All core components of the platform have been rewritten in Java. Support for writing handlers in Clojure remains, but is now offered in the form of bindings. Storm 2.0.0 requires Java 8. The multithreading model has been completely redesigned, allowing for performance gains (for some topologies, latencies have decreased by 50-80%).
The new version also introduces a new typed Streams API, allowing the specification of handlers using functional programming-style operations. The new API is implemented on top of the standard base API and supports automatic operation merging for optimization purposes. The Windowing API for window operations has added support for state saving and recovery in the backend.
The handler launch scheduler has been updated to take additional resources into account when making decisions, extending beyond
CPU and memory, such as network and GPU parameters. Numerous improvements have been made to enhance integration with the platform. The access control system has been expanded, allowing the creation of administrator groups and delegation of tokens. Enhancements related to SQL and metrics support have been added. The administrator interface has introduced new commands for debugging the cluster state.
Applications of Storm:
- Processing streams of new data or database updates in real time;
- Continuous computations: Storm can execute continuous queries and process continuous streams, delivering results to the client in real time.
- Distributed Remote Procedure Call (RPC): Storm can be utilized to ensure parallel execution of resource-intensive requests. A job ("topology") in Storm represents a distributed function across nodes that awaits incoming messages to process. Upon receiving a message, the function processes it in a local context and returns the result. An example of using distributed RPC could be the parallel processing of search queries or executing operations over a large set of datasets.
Features of Storm:
- Simple programming model that significantly simplifies real-time data processing;
- Support for any programming language. There are modules for Java, Ruby, and Python, and adapting for other languages is straightforward due to a very simple communication protocol, which requires about 100 lines of code for implementation;
- Fault tolerance: to run a data processing job, a jar file containing the code needs to be generated. Storm will automatically distribute this jar file across the cluster nodes, connect the related processors, and organize monitoring. Upon job completion, the code will be automatically disabled on all nodes;
- Horizontal scalability. All computations are performed in parallel; as load increases, it is sufficient to simply connect additional nodes to the cluster;
- Reliability. Storm guarantees that each incoming message will be fully processed at least once. A message will only be processed once if there are no errors during the passage through all processors; if issues arise, failed processing attempts will be retried.
- Speed. Storm's code is written with high performance in mind and uses a system for fast asynchronous messaging. .
Source: opennet.ru
