Materials from the Kafka meetup: CDC connectors, growth problems, Kubernetes

Hello! Recently, a meetup on Kafka was held in our office. Full-time places on him scattered at the speed of light. As one of the speakers said: "Kafka is sexy."

With colleagues from Booking.com, Confluent, and Avito, we discussed sometimes awkward integration and support for Kafka, the consequences of its crossing with Kubernetes, as well as well-known and personally written connectors for PostgreSQL

We edited videos of reports, collected presentations from speakers and selected beautiful photos from the meetup. If you are interested in watching and listening - go under the cat.

Video recordings of reports

Connectors: why are there so many of you - Anatoly Soldatov, Avito

Anatoly told what connectors are, what advantages and disadvantages JDBC and Debezium have, and how we in Avito came up with the idea of ​​writing our own CDC connector.

00:00 - Speaker introduction
04:09 — What is Kafka Connect, what connectors exist
18:10 What is Connect commonly used for?
23:47 - Avito's first case: data-bus
28:20 - Avito's second case: how we made our CDC connector
41:28 — Answers to listeners' questions

→ Anatoly's presentation

How to tame Apache Kafka? The Booking.com experience — Alexander Mironov, Booking.com

Alexander shared stories about whom, what difficulties his team had while working with Kafka, and how they were solved.

00:00 - Speaker introduction
06:22 - What problems appeared with the increase in the number of use cases using Kafka
08:58 — Creation of abstractions for users
14:15 — Authentication and security clusters
23:53 — Control plane, which connects everything
27:41 — Funny and not so stories from practice
34:00 — Answers to listeners' questions

→ Alexander's presentation

Kafka on Kubernetes: is it a sombrero according to Juan? — Viktor Gamov, Confluent

Victor answered a question about whether to run Apache Kafka and threading applications on Kubernetes.

00:00 - Speaker introduction
05:19 — Problems of running stateful applications on Kubernates
17:52 - Kubernetes pod
22:33 — Working with storage
28:24 - Custom Kubernetes controllers
48:58 — Rolling upgrade
58:05 — Answers to listeners' questions

→ Victor's presentation

Playlist with all videos can be found on our youtube channel.

Photos

We posted photos from the meetup on facebook и VK. See how it went and tag yourself and your friends if you were at the meeting.

We will be back soon with announcements of new events in Avito's Moscow office. See you soon!

Source: habr.com