The conference will be held in St. Petersburg on July 11-12 , dedicated to the development of parallel and distributed systems. The highlight of Hydra is that it brings together top scientists (who are usually only found at foreign scientific conferences) and well-known practicing engineers into one comprehensive program at the intersection of science and practice.
Hydra is one of the most important conferences we've had in the last few years. It was preceded by extensive preparation, including the selection of speakers and presentations. Last week, an with the director of JUG.ru Group, Alexey Fedorov ().
We ) about three key figures, pioneers of distributed systems theory—Leslie Lamport, Maurice Herlihy, and Michael Scott. It’s time to discuss the entire program in more detail!
Motivation
If you work in programming, you inevitably deal with multithreading and distributed computing. Specialists in these areas work with them directly, but distribution is implicitly present all around us: in any multi-core computer or distributed service, there is something that performs computations in parallel.
There are many conferences that explore various aspects of applied programming. On the other side of the spectrum, we have specialized scientific schools that reveal vast amounts of complex theory through lectures. For example, alongside Hydra in St. Petersburg, there's the . . At the Hydra conference, we aimed to bring together rigorous practice, science, and everything at their intersection.
Consider this: we live in an amazing time when you can meet the founders of the fields of science and engineering we are involved in. Physicists won’t meet Newton or Einstein—the train has left the station. But among us are still those who laid the foundations of distributed systems theory, invented popular programming languages, and first implemented all this into working prototypes. These people haven’t abandoned their work midway; they are currently tackling relevant challenges at world-renowned universities and companies, and they are today’s greatest sources of knowledge and experience.
On the other hand, the opportunity to meet them usually remains purely theoretical: few of us can continuously monitor public events at any Rochester university to then dash to the USA and back for a lecture by Michael Scott. Visiting all the participants of Hydra would cost quite a bit, not to mention the abyss of time spent (though it does sound like an interesting quest).
On the other hand, we have many top engineers who are currently working on pressing issues in distributed systems, and they definitely have stories to tell. But here's the problem — they they are working, and their time is valuable. Yes, if you work for Microsoft, Google, or JetBrains, the odds of encountering one of the well-known speakers at an internal event increase dramatically, but overall — no, it doesn’t happen every day.
Thus, the Hydra conference fulfills an important role that most of us cannot manage alone — it gathers people in one place at one time, whose ideas or interactions could change your life. I admit that not everyone needs distributed systems or some complex fundamental concepts. One can program CRUD applications in PHP their whole life and still be quite happy. But for those who do need it — this is your chance.
A considerable amount of time has passed since the first announcement of the Hydra conference on Habr. During this time, immense work has been done — and now we have a list of almost all the talks. No sluggish single-threaded algorithms, just pure distributed hardcore! Let's finish with the generalities and see what we have on our hands.
Keynotes
Keynotes start and end the conference days. Typically, the purpose of the opening keynote is to establish the overall spirit and direction of the conference. The closing keynote wraps things up and explains how we should live with the knowledge and skills acquired during the conference days. The beginning and end: what will be remembered best and holds greater significance.
Cliff Click —
Cliff is a legend in the Java world. In the late 1990s, for his PhD thesis, he wrote a paper titled , which later became the foundation for the HotSpot JVM Server Compiler. Two years later, he was already working at Sun Microsystems on the JVM and demonstrated to the world that JIT has a rightful place. This whole story about Java being one of the fastest modern runtimes with the smartest and fastest optimizations began with Cliff Click. Initially, it was believed that if something was available to a static compiler, there was no need to even try JIT compilation. Thanks to the work of Cliff and his team, all new languages started to be developed with the idea of JIT compilation by default. Undoubtedly, this was not the work of one person, but Cliff played a very important role in it.
In the opening keynote, Cliff will talk about another of his ventures — , an in-memory platform for distributed and scalable machine learning for industrial applications. More precisely — about the distributed key-value store within it. This is a very fast storage solution with a lot of interesting features (the exact list is in ), which allow such solutions to be used in the mathematics of big data streaming.
Another presentation that Cliff will give — . Another part of his biography includes ten years , where he upgraded and improved many aspects of the Azul hardware and technology stack: JIT compilers, runtime, threading model, error handling, stack operations, hardware interrupts, class loading, and so on — you get the idea.
The most interesting part began when they created hardware for big business — a supercomputer for running Java. This was quite an innovative thing, specifically tailored for Java, which has special requirements — memory read barriers for low-pause garbage collection, boundary-checked arrays, virtual calls… One of the coolest technologies was hardware transactional memory. The entire L1 of any of the 864 cores could participate in transactional writes, which is particularly important for handling locks in Java (synchronized blocks can operate in parallel, as long as there's no actual conflict over memory). But the beautiful idea crashed against harsh reality — and in this presentation, Cliff will explain why HTM and STM are not very suitable for the practical needs of multithreaded computations.
Michael Scott —
— professor of Computer Science at the University of Rochester, whom fate has linked for , and at his alma mater, Wisconsin–Madison, he was dean for five years. He conducts research in parallel and distributed programming and language design, teaching students in these areas.
The whole world knows Michael for his textbook , the latest edition of which was released relatively recently — in 2015. His work has received as one of the most well-known in the field of distributed computing and in the online library of the University of Rochester. You might also know him as the author of the famous Michael-Scott algorithm from .
In the Java world, the case is special: along with Doug Lea, he developed the non-blocking algorithms and synchronous queues on which Java libraries operate. This is what the keynote "Dual data structures" will be about — the implementation of these structures in Java SE 6 allowed for a tenfold improvement in java.util.concurrent.ThreadPoolExecutor. If you're curious about what these "Dual data structures" are, there is .
Maurice Herlihy —
— a recipient of two Dijkstra prizes. The first was for his work at (Brown University), and the second, more recent one — (Virginia Tech University). The Dijkstra prize is awarded for works whose significance and impact have been noticeable for at least ten years, and it’s clear that Maurice is one of the most recognized experts in the field. Currently, he is a professor at Brown University and has numerous accomplishments that would fill a paragraph.
In this closing keynote, Maurice will discuss the theory and practice of blockchain distributed systems from the perspective of classic distributed computing and how it simplifies many related problems. This talk is strictly relevant to the conference topic — not about mining hype, but rather about how our knowledge can be surprisingly efficiently and appropriately used to address various tasks.
In July 2017, Maurice already visited Russia at the SPTDC school, participated in the JUG.ru meetup, and you can watch the recording on YouTube:
Main program
Next, there will be a brief overview of the presentations included in the program. Some presentations are described in detail, while others are shorter. The lengthy descriptions are mainly for English-language reports that require references to scientific papers, terms on Wikipedia, and so on. A complete list can be The list on the website will be updated and supplemented.
Leslie Lamport —
Leslie Lamport is the author of fundamental works in distributed computing. stands for “Lamport TeX.” He introduced the concept for the first time in 1979. , and his article received the Dijkstra Prize.
This is the most unusually formatted part of the program, as it is not even a lecture, but a question-and-answer session. When a significant portion of the audience is already familiar with (or can become familiar with) various works based on the “Lamport theory,” including his own articles and presentations, it is crucial to spend all available time on direct communication.
The idea is simple — you watch two lectures on YouTube: and and prepare at least one question, which Leslie answers.
We have already If you don't have an hour to watch the video, you can quickly read all this in text form.
Note: there are many more videos featuring Leslie Lamport on YouTube. For instance, there is a great one. The offline version of this entire course is on , and on YouTube, it has been made available for easier viewing on mobile devices.
Martin Kleppmann —
Martin Kleppmann is a researcher at the University of Cambridge working on CRDT and formal verification of algorithms. Martin's book , published in 2017, turned out to be very successful and made it to the bestseller lists in the field of storage and processing of data. Kevin Scott, CTO of Microsoft, : “This book should be mandatory for software engineers. It is a rare resource that combines theory and practice, helping developers design and implement data processing infrastructure and systems more intelligently.” Something similar was said by the creator of Kafka and CTO of Confluent, Jay Kreps.
Before embarking on academic research, Martin worked in the industry and co-founded two successful startups:
- Rapportive, dedicated to displaying social profiles of contacts from your email, which was acquired by LinkedIn in 2012;
- Go Test It, a service for automatic website testing across different browsers, which was acquired by RedGate in 2009.
Overall, Martin may be less known than our keynote speakers, but he has already made some contributions to the development of distributed computing and the industry.
In this talk, Martin will discuss a topic closer to his academic research. In Google Docs and similar collaborative editing software, 'collaborative editing' refers to the task of replication: each user has their own replica of a shared document, which they then modify, and all changes are sent across the network to the other participants. Changes to documents offline lead to temporary inconsistency of the document with respect to the other participants, and re-synchronization requires conflict resolution. This is where (CRDT), essentially a relatively new concept that was only defined in 2011, comes into play. This talk discusses what has happened since then in the world of CRDT, what the latest advancements are, the approach to creating local-first applications in general, and the use of the open-source library specifically.
Next week, we will publish a big interview with Martin on Habr, which will be interesting.
Pedro Ramalhete —
Pedro works at Cisco and has been developing parallel algorithms for the last ten years, including synchronization mechanisms, lock-free and wait-free data structures, and everything you can imagine on this topic. His current research and engineering interests are focused on Universal Constructions, Software Transactional Memory, Persistent Memory, and similar technologies that allow the implementation of correct, scalable, and fault-tolerant applications. He is also the author of a widely known blog in niche circles called .
Most multi-threaded applications now operate on parallel data structures, ranging from message queues between actors to indexed data structures in key-value stores. They have been successfully functioning in Java JDK for many years, and are gradually being added in C++.
The simplest way to implement a parallel data structure is through a sequential (single-threaded) implementation, where methods are protected by mutexes. This is accessible to any junior developer, but has obvious scaling and performance issues. At the same time, lock-free and wait-free data structures not only handle errors more effectively but also have a better performance profile—however, their development requires deep expertise and adaptation for specific use cases. One wrong line of code is enough to break everything.
How can we ensure that even non-experts can design and implement such data structures? It is known that any sequential algorithm can be made thread-safe either through , or through transactional memory. Both can lower the barrier to entry for solving this problem. However, both solutions generally lead to inefficient implementations. Pedro will discuss how they managed to make these constructs more efficient and how they can be leveraged in your algorithms.
Heidi Howard —
Heidi Howard, like Martin, is a researcher in distributed systems at the University of Cambridge. Her specialization includes consistency, fault tolerance, performance, and distributed consensus. She is best known for generalizing the Paxos algorithm, called .
Recall that — a family of protocols for solving the consensus problem in a network of unreliable computators, based on the work of Leslie Lamport. Thus, some of our speakers are working on problems originally proposed by other speakers— and that’s wonderful.
The ability to reach a consensus among multiple hosts—for addressing, leader selection, blocking, or coordination—is a fundamental question in modern distributed systems. Paxos is currently the primary method for solving consensus problems, and a lot of research is being conducted to extend and optimize the algorithm for various practical needs.
In this paper, we will review the theoretical foundation of Paxos, relaxing the initial requirements and generalizing the algorithm. We will see that Paxos, in essence, is just one option among a vast array of approaches to consensus, and that other points on the spectrum are also quite useful for building robust distributed systems.
Alex Petrov —
Alex is a database and storage systems specialist, and most importantly for us, a committer at . Together with O'Reilly, he is currently working on a book titled Database Internals.
For systems with eventual consistency, after a node failure or network partition, one must resolve the following dilemma: either continue processing requests at the expense of consistency or deny their execution at the cost of availability. In such a system, quorums—overlapping subsets of nodes that ensure that at least one node holds the most recent value—can serve as a good compromise solution. One can withstand failures and loss of connectivity to some nodes while continuing to respond with the most current values.
However, everything comes at a cost. The quorum replication scheme implies an increased storage cost: it requires storing redundant data on multiple nodes simultaneously to ensure a sufficient number of accessible copies during a problem. It turns out that it is not necessary to store all data on all replicas. Storage load can be reduced by keeping data only on a subset of nodes and using specialized nodes (Transient Replica) for failure handling scenarios.
Throughout the paper, we will discuss , a replication scheme used in and , and the implementation of this concept in Apache Cassandra under the names .
Dmitry V Yukov —
Dmitry is a developer at Google, working on dynamic testing for C/C++ and Go — Address/Memory/ThreadSanitizer, and similar tools for the Linux kernel. He contributed a scalable goroutine scheduler, network poller, and a parallel garbage collector to Go. He is an expert in multithreading, the author of a dozen new non-blocking algorithms, and holds Intel.
Now let's talk a bit about the report itself. The Go language has native support for multithreading in the form of goroutines (lightweight threads) and channels (FIFO queues). Thanks to these mechanisms, users can easily and enjoyably write modern multithreaded applications, and it feels like magic. As we understand, there is no magic here. In this report, Dmitry will delve into the intricacies of Go's scheduler and reveal the secrets of implementing this "magic". He will start with an overview of the main components of the scheduler, explaining how it works. Then we will closely examine specific aspects such as the parking/unparking strategy and handling blocking system calls. Finally, Dmitry will briefly discuss potential improvements in the scheduler.
Dmitry Bugaychenko is
Dmitry has worked in outsourcing for almost 9 years while maintaining contact with the university and the academic environment. The analysis of big data at Odnoklassniki provided him a unique opportunity to combine theoretical training and scientific foundation with the development of real, in-demand products.
Distributed graph analysis has been and remains a challenging task: when there is a need to obtain information about the connections of a neighboring vertex, data often needs to be transferred between machines, leading to increased execution time and network infrastructure load. In this report, we will see how significant acceleration can be achieved using probabilistic data structures or facts such as the symmetry of the friendship graph in social networks. All this will be illustrated with code examples in Apache Spark.
Denis Ryscov is
Denis is a developer , an expert in consistency model verification, consensus algorithms, and distributed transactions. He is currently working at Microsoft and previously worked on distributed systems at Amazon and Yandex.
In this presentation, we will explore the distributed transaction protocols developed over the last few years that can be implemented on the client side above any data store supporting conditional updates (compare and set). The essence is that life does not end with a two-phase commit; transactions can be added on top of any databases—at the application level—but different protocols (2PC, Percolator, RAMP) have different trade-offs and are not free.
Alexey Zinovyev —
Alexey () — our long-time speaker and a member of program committees at other conferences. A practicing trainer at EPAM Systems, he has been working with Hadoop/Spark and other big data technologies since 2012.
In this presentation, Alexey will discuss the challenges of adapting classical machine learning algorithms for execution in a distributed mode based on his experience with Apache Spark ML, Apache Mahout, Apache Flink ML, and his experience creating Apache Ignite ML. Alexey will also cover the implementation of distributed ML algorithms in these frameworks.
And finally — two presentations from Yandex about Yandex Database.
Vladislav Kuznetsov —
Vladislav is a developer at Yandex in the distributed platform group. Yandex Database is a horizontally scalable, geo-distributed, fault-tolerant DBMS that withstands the failure of disks, servers, racks, and data centers without compromising consistency. To ensure fault tolerance, a proprietary distributed consensus algorithm is employed, along with a number of technical solutions that are discussed in detail in the presentation. The talk may be of interest to both DBMS developers and developers of application solutions based on the DBMS.
Semyon Checherynda —
Semyon is a developer in the distributed platform group at Yandex, working on the multi-tenancy capability of YDB installations.
Yandex Database is designed for OLTP queries and meets ACID requirements for transactional systems. In this report, we will discuss the transaction scheduling algorithm underlying the YDB transactional system. We will examine the entities involved in transactions, who assigns global order to transactions, how transaction atomicity, reliability, and strict isolation levels are achieved. Using a common task as an example, we will explore implementations of transactions using two-phase commit and deterministic transactions. We will discuss their differences.
What's next?
The conference program continues to be filled with new reports. In particular, we expect a report from Nikita Kovalyov () from JetBrains and Oleg Anastasyev () from Odnoklassniki. Nikita works on algorithms for coroutines in the Kotlin team, while Oleg develops architecture and solutions for high-load systems on the Odnoklassniki platform. Additionally, there is one more conditionally empty slot, with candidates currently being worked on by the program committee.
The Hydra conference will be held on July 11-12 in Saint Petersburg. Tickets can be . Please note the availability of Online tickets — if, for some reason, you cannot physically make it to Petersburg on these days.
See you at Hydra!
Source: habr.com
