Recently there were two events at once on the development of multithreaded and distributed systems: conference (July 11-12) and school (July 8-12). People who are close to this topic understand that coming to Russia Leslie Lamport, Maurice Herlihy и Michael Scott - the most important event. But there were other questions:
- What to expect from the conference: "academic" or "production"?
- How do school and conference compare? Who is this and that for?
- Why do they overlap in dates?
- Will they be useful to those who have not devoted their entire lives to distributed systems?
All this is well known to the person who made Hydra come into existence: our director Alexey Fedorov (). He answered all questions.
Format
- An introductory question for those who are far from distributed systems: what are both events about?
— The global problem is that there are services around us with large volumes of operations and complex computational tasks that cannot be done on a single computer. So there must be several cars. And then questions arise related to how to correctly synchronize their work and what to do in conditions of not the highest reliability (because the equipment breaks down and the network falls off).
The more machines, the more points of failure. What if different machines produce different results for the same calculations? What to do if the network disappeared for some time and part of the calculations became isolated, how then to combine all this? In general, a million related problems. New solutions, new problems.
In this area, there are completely applied areas, but there are more scientific ones - something that has not yet become mainstream. I would like to talk about what is happening both in practice and in science, and most importantly, at their intersection. This is what the first Hydra conference will be about.
- I would like to understand that there is a conference, and there is a summer school. How do they compare? If there is a discount for attending the conference for school participants, then why do they overlap in dates, so that it is impossible to attend all at once without loss?
“The school is a chamber event for 100-150 people, where leading experts from all over the world come and give lectures for five days. And a situation arises when world-class luminaries gather in St. Petersburg for five days, ready to tell something. And in this case, the decision suggests itself to organize not only a chamber school, but also a larger conference.
You can hold such a school only in the summer, in July, because among these specialists there are acting university professors, and they are simply not ready at other times: they have students, diplomas, lectures, and so on. The school format is five weekdays. It is known that people like to go somewhere on weekends in the summer. This means that we cannot make a conference either on the weekend before school or on the weekend after school.
And if you extend it for a couple more days before or after the weekend, then magically five days of the specialists' stay in St. Petersburg turn into nine. And they are not ready for it.
Therefore, the only solution we found was to simply hold the conference in parallel with the school. Yes, this creates some problems. There are people who want both the school and the conference, and they will have to skip part of the lectures there or there. The good news is that all this will take place in neighboring halls, it will be possible to run back and forth. And another good side is the availability of video recordings, on which you can later safely watch what you missed.
- When two events are held in parallel, people have the question "which one I need more." What exactly is expected of each, and what are the differences?
— The school is a purely academic event, a classical scientific school for several days. Those who themselves were engaged in science and had something to do with graduate school, imagine what an academic school is.
Usually such academic events are not very well organized due to the lack of event expertise of the people who do it. And we are still quite experienced guys, so we can do everything quite competently. I think that from an organizational point of view, SPTDC will be head and shoulders above any scientific or scientific-practical school that you have ever seen.
SPTDC school - this is a format where each large lecture is read in two pairs: "an hour and a half - a break - an hour and a half." It must be understood that for the first time it can be difficult for a participant: when this school was held for the first time two years ago, it was unusual for me myself, I switched off several times around the middle of a double lecture, and then it was already difficult to understand what was happening. But it really depends on the lecturer: a good lecturer talks very interestingly all three hours.
Conference Hydra (Hydra) - more practical. There will be several luminaries of science who have come to lecture at the School: from , whose work underlies the very theory of multithreaded and distributed systems, to , one of the authors of the famous concurrency textbook "The Art of Multiprocessor Programming". But at the conference, we will try to talk about how certain algorithms are implemented in reality, what problems engineers face in practice, who succeeds and fails, why some algorithms are used in practice, while others are not. And of course, let's talk about the future of the development of multi-threaded and distributed systems. That is, we will give such a cutting edge: what the world science is talking about now, what the thoughts of leading engineers revolve around, and how it all fits together.
- Since the conference is more applied, will there be not only academic luminaries, but also speakers "from production"?
- Definitely. We try to look at all the "big ones": Google, Netflix, Yandex, Odnoklassniki, Facebook. There are specific funny problems. For example, everyone says: “Netflix is distributed systems, almost half of the US traffic, very cool,” and when you start looking at their real reports, articles and publications, a slight disappointment sets in. Because while it's definitely world class and there's cutting egde, there's less of it than meets the eye.
An interesting dilemma arises: you can call representatives of large eminent companies, or you can call someone already known to us. In reality, there is expertise both there and there. And we are rather trying to pull out not “people from very large brands”, but very large specialists, specific people.
For example, there will be Martin Kleppman, who at one time caused a stir on LinkedIn, and also released - perhaps one of the basic books in the field of distributed systems.
- If a person does not work at Netflix, but at a simpler company, he may think: “I should go to such a conference, or are there all sorts of Netflix talking to each other, but I have nothing to do?”
- I will say this: when I worked at Oracle for a little over three years, I heard the most amazing and interesting things in the kitchen and in smoking rooms, when colleagues gathered there, making certain pieces of the Java platform. These could be people from the virtual machine, or from the testing department, or from performance concurrency - for example, Lyosha Shipilev and Seryozha Kuksenko.
When they started to discuss something among themselves, I usually just listened with my mouth open. For me, these were amazing and unexpected things that I had not even thought about. Naturally, at first I did not understand 90% of what they were talking about. Then it became 80% incomprehensible. And after I did my homework and read a few books, that number dropped to 70%. I still don't understand a lot of what they say to each other. But, sitting in the corner with a cup of coffee and eavesdropping, I began to understand a little what was going on.
Therefore, when Google, Netflix, LinkedIn, Odnoklassniki and Yandex talk to each other, this does not mean that this is something incomprehensible and uninteresting. On the contrary, we must listen carefully, because this is our future.
Of course, there are people who do not need all this. If you do not want to develop in this topic, do not go to this conference, you will simply lose time there. But if the topic is interesting, but you don’t understand anything about it or are just looking closely, then you should come, because you won’t find anything like it anywhere. And I think that not only in Russia, but also in the world. We are trying to make a conference that will not just be the leader on this topic in Russia, but in general number one in the world.
This is not an easy task, but when we have such an amazing chance to gather strong speakers from all over the world, then I am ready to give a lot to make it work. Of course, some of those whom we invited to the first Hydra will not be able to come. But I will say this: we have never started a new conference with such a strong lineup. Except, perhaps, the very first JPoint six years ago.
- I would like to develop the words “this is our future”: will the topic later affect those who do not think about it today?
— Yes, I'm sure of it. Therefore, it seems to me very correct to start discussing it as soon as possible. For example, the theory of multithreading appeared quite a long time ago (in the 70s, work was already being published with might and main), but for a long time they were the lot of narrow specialists, until the first user dual-core computer appeared at the beginning of the 10s. And now we all have multi-core servers, laptops and even phones, and this is the mainstream. It took about XNUMX years for this to become widespread, for people to understand that this discourse is not the lot of a narrow circle of specialists.
And about the same thing we are now seeing with distributed systems. Because basic solutions like load balancing, failover, and the like have been made for quite some time, but very few people know what, for example, distributed consensus or Paxos.
One of the most important tasks that I set for this event is to immerse engineers more and more in this discussion. It should be understood that at conferences not only some topics and solutions are discussed, but also a thesaurus appears - a single conceptual apparatus.
I see it as my task to create a platform where everyone can discuss all this, share experiences and opinions. So that you and I have a common understanding of what one algorithm does, what the other does, which one is better under what conditions, how they are related to each other, and so on.
A very interesting thing is connected with the same multithreading. When our friends from Oracle (first of all, Lyosha Shipilev and Sergey Kuksenko) began to actively talk about performance and, in particular, about multithreading, literally two or three years later these questions began to be asked at interviews in companies, people began to discuss this in smoking rooms. That is, a thing that was the lot of narrow specialists suddenly became mainstream.
And this is very correct. It seems to me that we helped these guys to popularize all these issues, which are really important, useful and interesting. If earlier no one thought about how the Java server processes requests in parallel, now people have at least some level of understanding of how it all works. And that is great.
The task that I see now is to do about the same with distributed systems. So that everyone roughly understands what it is, where the legs grow from, what tasks and problems there are, so that it also becomes mainstream.
Companies have a huge demand for people who understand something about this, but there are few such people. The more we create around this content and the opportunity to learn about it, the more we give people opportunities to ask questions that are in the air, the more chances we have to somehow move in this direction.
prehistory
- The conference is held for the first time, but the school is not the first time. How did it all arise and develop?
— It's an interesting story. Two years ago, in May 2017, we were sitting in Kyiv with Nikita Koval (), an expert in the field of multithreading. And he told me that in St. Petersburg there will be .
The topic of multithreaded programming has been fantastically interesting in the last three years of my engineering career. And then it turned out that very, very famous people come to St. Petersburg in the summer, the same Maurice Herlihy and Nir Shavit, according to which I studied. And many of my friends were related to this - for example, Roma Elizarov (). I realized that I just can not miss such an event.
When it became clear that the school program in 2017 would be great, the idea arose that lectures should be recorded on video. We at the JUG.ru Group had a complete understanding of how such lectures should be recorded. And we fit into SPTCC as the guys who made the video for the school. As a result, all the lectures of the school on our YouTube channel.
I began to communicate with Peter Kuznetsov, who was the main ideologist and organizer of this school, and with Vitaly Aksyonov, who helped organize all this in St. Petersburg. I realized that it's fantastically cool and interesting and, probably, it's very bad that only 100 participants can touch the beauty.
When Peter thought about what he needed to do school again (in 2018 there was no time and energy, so he decided to do it in 2019), it became clear that we could help him by simply removing all organizational things from him. This is what is happening now, Peter is engaged in content, and we are in charge of everything else. And this seems to be the right scheme: probably, Peter is more interested in the program than "where and when everyone will have lunch." And we are good at working with halls, venues, and so on.
This time, instead of SPTCC, the school is called SPTDC, not "concurrent computing", but "distributed computing". Accordingly, this is approximately the difference: last time at school they did not talk about distributed systems, but this time we will actively talk about them.
- Since the school is not held for the first time, it is already possible to draw some conclusions from the past. What happened last time?
- When the first school was being created two years ago, it was expected that there would be an academic event, primarily of interest to students. Moreover, students from all over the world, because the school is only in English, and it was thought that a significant number of foreign students would come.
In fact, it turned out that a lot of engineers came from large Russian companies like Yandex. There was Andrey Pangin () from Odnoklassniki, there were guys from JetBrains who are actively involved in this topic. In general, there were many familiar faces from our partner companies. I'm not surprised at all, I understand perfectly why they came there.
Actually, the organizers had expectations that there would be academic people at the School, but suddenly people from the industry came, and then it became clear to me that there was a demand in the industry.
If an event, which was almost never promoted anywhere, gathered an audience of adults from the first time at the click of a finger, it means that there is actually an interest. It seems to me that the request for this topic is overdue.
Maurice Herlihy at the JUG.ru meeting
- In addition to school, Maurice Herlihy spoke in St. Petersburg at a meeting of JUG.ru in 2017, about transactional memory, and this is a little closer to the conference format. Who then came - the same people who usually come to JUG.ru meetups, or a different audience?
— It was interesting, because we understood that Maurice will have a general report, and not specific to Java, and made a slightly wider announcement than we usually do for our JUG news subscribers.
A lot of people I know came from communities that are not about Java at all: from the .NET party, from the JavaScript party. Because the topic of transactional memory does not apply to a specific development technology. When a world-class specialist arrives to talk about transactional memory, missing the opportunity to listen to such a person, ask him questions is simply a crime. It just makes a powerful impression when the person whose book you study from comes to you and tells you something. Just fantastic.
— And what was the feedback on the results? Has the approach turned out to be too academic and incomprehensible for people from the industry?
— Feedback on Herlihy's report was good. People wrote that he was very simple and clear, which was not expected from an academic professor. But we must understand that we did not just call him, he is a world-famous specialist with extensive speaking experience and a background from a bunch of books and articles. And, probably, he became famous in many respects due to his ability to convey material to people. Therefore, this is not surprising.
He has normal understandable English, and he, of course, is very good at what he talks about. That is, you can ask him absolutely any questions. Basically, the people complained that we gave Maurice too little time to report: two hours is not enough for such a thing, at least two more are needed. Well, what we managed in two hours, we did it.
Motivation
- Usually JUG.ru Group is engaged in large-scale events, and this topic looks more highly specialized. Why was it decided to take it? Is there a willingness to hold a smaller event, or can a lot of spectators gather on such a topic?
— Indeed, when you do some kind of event and set a certain level of discussion, the question always arises of how massive this discussion is. How many people - ten, a hundred or a thousand - is interesting. There is a trade-off between mass and depth. This is a completely normal question, and everyone solves it differently.
In this case, I want to make the event “for myself”. In multithreading, I still understand something (I gave lectures on this topic at conferences, told students something several times), but in distributed systems I am a layman: I read some articles and saw several lectures, but not even a single full-fledged book read.
We have a program committee that is made up of experts in the field and is able to assess the accuracy of the reports. And I, for my part, am trying to make this event the one that I would like to go to with my lack of expertise. Whether it will be of interest to the mass public, I do not know. Probably, at this stage, this is not the main task of this event. Now it is more important to form the strongest program in the shortest possible time.
Probably, now I am setting the team not the task of “gathering a thousand people the first time”, but “make the conference appear”. This may sound not very businesslike and somewhat naive, although I am not at all an altruist. But I can sometimes allow myself some liberties.
There are things more important than money and beyond money. We already do a large number of cool large-scale events for a thousand people or more. Our Java conferences have long exceeded a thousand people, and now other events are jumping over this bar. That is, the question that we have become experienced and well-known organizers is no longer worth it. And, probably, what we earn at these events gives us the opportunity to reinvest in what is interesting to ourselves, and in this case, to me personally.
By doing this event, I go against some of the principles of our organization. For example, we usually try to prepare conferences very much in advance, but now we have a very tight deadline, and we finalize the program just a month before the event itself.
And this event will be 70-80% English-speaking. Here, too, there is always a discussion about whether it is necessary to be closer to the people (which is more understandable when most of the reports are in Russian) or to the whole world (because the technical world is English-speaking). Usually we try to make a lot of presentations in Russian. But not at this time.
Moreover, we will also ask some of our Russian-speaking speakers to speak in English. In a sense, this is an absolutely anti-user and inhumane approach. But one must understand that there is no Russian-language literature on this topic now, and any person who is generally interested in this is forced to read in English. So, somehow he is able to deal with English. If in the case of JavaScript, Java or .NET there are a lot of people who do not know English very well, but at the same time they can program well, then, probably, distributed systems are an area in which there is simply no other way to learn now.
I really want to conduct this experiment: how the public in Russia will perceive an event that is 70-80% English-speaking. Will it go or not? We do not know this in advance, because we have never done this before. But why not do it? Let's put it this way: this is one big experiment that I just can't help doing.
The SPTDC school program is already entirely, and in the case of Hydra already a noticeable part, and soon we will publish an analysis of the entire conference program.
Source: habr.com