ืืื ืืืื ืึทืจืืขื, ืืื ืึธืคื ืืจืขืคื ื ืืึทืข ืืขืื ืืฉ ืกืึทืืืฉืึทื ื / ืืืืืืืืืจื ืคึผืจืึธืืืงืื, ืืื ืคึฟืึธืจืืึทืฆืืข ืืืขืื ืืืึธืก ืืื ืืึทื ืฅ ืงื ืึทืคึผ ืืืืฃ ืื ืจืืกืืฉ-ืฉืคึผืจืึทื ืืื ืืขืจื ืขื. ืืื ืืขื ืึทืจืืืงื ืืื ืืืขื ืคึผืจืืืืจื ืฆื ืคึผืืึธืืืืจื ืืืื ืึทืืึท ืจืืก ืืื ืึท ืืืืฉืคึผืื ืคืื ืืืื ืืขืฆืืข ืคืืจ, ืืืขื ืืื ืืืจืฃ ืฆื ืงืึทื ืคืืืืขืจ ืฉืืงื CDC events ืคืื ืฆืืืื ืคืึธืืงืก DBMSs (PostgreSQL ืืื MongoDB) ืฆื ืึท Kafka ืงื ืืื ื ืืฆื Debezium. ืืื ืืึธืคึฟื ืึทื ืืขืจ ืึธืคึผืฉืึทืฆืื ื ืึทืจืืืงื, ืืืึธืก ืืื ืืจืืืก ืืื ืึท ืจืขืืืืืึทื ืคืื ืื ืืืจืืืขืงืึธืื ืึทืจืืขื, ืืืขื ืืืื ื ืืฆืืง ืคึฟืึทืจ ืื ืืขืจืข.
ืืืึธืก ืืื ืืขืืขืืืื ืืื CDC ืืื ืึทืืืขืืืื?
- ืคืืจืฉืืืืขืจ ืคืื ืื CDC ืืืืืืืืืจื ืงืึทืืขืืึธืจืืข (), ืึธืืขืจ ืืขืจ ืืขื ืื, ืขืก ืืื ืึท ืืึทื ื ืคืื ืงืึทื ืขืงืืขืจื ืคึฟืึทืจ ืคืึทืจืฉืืื ืืืืก ืงืึทืืคึผืึทืืึทืืึทื ืืื ืื Apache Kafka Connect ืคืจืืืืืืขืจืง.
ืขืก ืืืืกืึทื ืกื ืืื ืืขืจ ืื Apache License v2.0 ืืื ืืืฆืืืืข ืืืจื Red Hat. ืึทื ืืืืืงืืื ื ืืื ืึธื ืืึธืื ื ืืื ื 2016 ืืื ืืขืจืืืืึทื ืขืก ืืื ืืึทืึทืืืขืจ ืฉืืืฆื ืคึฟืึทืจ ืื ืคืืืืขื ืืข ืืืืก: MySQL, PostgreSQL, MongoDB, SQL Server. ืขืก ืืขื ืขื ืืืื ืงืึทื ืขืงืืขืจื ืคึฟืึทืจ Cassandra ืืื Oracle, ืึธืืขืจ ืืื ืืขื ืืึธืืขื ื ืืื ืืขื ืขื ืืื "ืคืจื ืึทืงืกืขืก" ืกืืึทืืืก, ืืื ื ืืึท ืจืืืืกืื ืืึธื ื ืื ืืึทืจืึทื ืืืจื ืฆืืจืืง ืงืึทืืคึผืึทืืึทืืืืึทืื.
ืืืื ืืืจ ืคืึทืจืืืืึทืื CDC ืืื ืื ืืจืืืืฆืืื ืขืื ืฆืืืึทื ื (ืืืขื ืื ืึทืคึผืืึทืงืืืฉืึทื ืืืืขื ื ืืึทืื ืคืื ืื DBMS ืืืืึทื), ืื ืืืืคึผื ืึทืืืืึทื ืืืืืฉืื ืึทืจืืึทื ื ืขืืขื ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืกืืจืืืื ื ืคืื ืืึทืื ืขื ืืขืจืื ืืขื ืืืืฃ ืื ืจืืืขืจื ืืืจืื ืืื ื ืืืขืจืืง ืืืืืึทื ืกื, ืืืื ืจืืืืืึทืืืืึทืื ืืื ืึทืืืืืืึทืืืืึทืื. ืื ืืขืฆืืข ืฆืืืื ืคืื ืงืื ืืขื ืขื ืึทืืฉืืืื ืืืจื ื ืืฆื ืึท ืงืึทืคืงืึท ืงื ืืื ืืื ืึท ืจืืคึผืึทืืึทืืึธืจื ืคึฟืึทืจ CDC events.
ืื ืื ืืขืจ ืืืึทืืข ืืื ืืขืจ ืคืึทืงื ืึทื ืึท ืืืื ืืึธืืขื ืืื ืืขื ืืฆื ืฆื ืงืจืึธื ืืขืฉืขืขื ืืฉื, ืึทืืื ืื ืกืืฃ ืึทืคึผืืึทืงืืืฉืึทื ืืื ื ืืฉื ืืึธืื ืฆื ืืึธืจื ืืืขืื ืื ื ืืึทื ืกืื ืคืื ืึทืคึผืขืจืืืืื ื ืคืึทืจืฉืืืขื ืข ืืืืก.
ืฆืื ืกืืฃ, ื ืืฆื ืึท ืึธื ืืึธื ืืขืงืืขืจ ืึทืืึทืื ืึทืคึผืืึทืงืืืฉืึทื ื ืึทื ืืึธื ืืืึธืจ ืขื ืืขืจืื ืืขื ืืื ืืึทืื ืฆื ืืืึธื ืืืืก ืืึธืจืืืึทื ืืึทืื. ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืื ืคึผืจืึทื ืืืืฃ ืื ืืึทืื ืืงืืจ ืืื ืืื ืึทืืืืื, ืืืืึทื ืื ืืึทืื ืืขื ืขื ืืืงืืืขื ื ืืฉื ืืืืึทื ืคึฟืื ืื DBMS, ืึธืืขืจ ืคึฟืื ืื ืงืึทืคืงืึท ืงื ืืื.
ืืืขืื ืื ืืขืืขืืืื ืึทืจืงืึทืืขืงืืฉืขืจ
ื ืืฆื Debezium ืงืืื ืึทืจืึธืคึผ ืฆื ืืขื ืคึผืฉืื ืกืืขืืข:
DBMS (ืืื ืึท ืืึทืื ืืงืืจ) โ ืงืึทื ืขืงืืขืจ ืืื Kafka Connect โ Apache Kafka โ ืงืึทื ืกืืืขืจ
ืืื ืึท ืืขืืขื, ืืึธ ืืื ืึท ืืืึทืืจืึทืืข ืคืื โโืื ืคึผืจืืืขืงื ืืืขืืืืืื:
ืึธืืขืจ, ืืื ืืึธื ื ืื ืืึทืงืข ืืื ืืขื ืกืืขืืข, ืืืืึทื ืขืก ืืืื ื ืึทื ืืืืื ืื ื ืืฆื ืคืื ืึท ืืื ืงืขื ืงืึทื ืขืงืืขืจ ืืื ืืขืืืขื.
ืืื ืคืึทืงื, ืื ืกืืืืึทืฆืืข ืืื ืึทื ืืขืจืฉ: ืคืืืื ื ืืืื ืืึทืืึท ืืืืง (ืืขืฆืืข ืืื ืง ืืื ืื ืืืึทืืจืึทืืข ืืืืื) ืืึธืก ืืื ื ืืฉื ืืขืจ ืืืืื ืืืขื ืฆื ื ืืฆื Debezium. ืืขืฉืขืขื ืืฉื ืืขืฉืืงื ืฆื Apache Kafka ืงืขื ืขื ืืืื ืืขืืืืื ื ืืืจื ืืืื ืึทืคึผืืึทืงืืืฉืึทื ื ืฆื ืืึทื ืืืขื ืืื ืคืึทืจืฉืืื ืกืืืืึทืืืึธื ืก. ืืืฉื:
- ืจืืืืืืื ื ืืจืขืืึทืืืึทื ื ืืึทืื ืคืื ืื ืงืึทืฉ;
- ืฉืืงื ื ืึธืืืึทืคืึทืงืืืฉืึทื ื;
- ืืืื ืืื ืืขืงืก ืืขืจืืืึทื ืืืงืื ืืขื;
- ืขืืืขืืข ืกืึธืจื ืคืื ืงืึธื ืืจืึธืืืจื ืืึธืืก;
- ...
ืืืื ืืืจ ืืึธื ืึท Java ืึทืคึผืืึทืงืืืฉืึทื ืืื ืขืก ืืื ื ืื ื ืืืืืง / ืืขืืืขืืงืืื ืฆื ื ืืฆื ืึท Kafka ืงื ืืื, ืขืก ืืื ืืืื ืื ืืขืืืขืืงืืื ืฆื ืึทืจืืขืื ืืืจื . ืื ืงืืึธืจ ืืื ืืขืจ ืืึธื ืืืึทืืข ืืื ืึทื ืขืก ืืืืืึทื ืืืฅ ืื ื ืืื ืคึฟืึทืจ ื ืึธื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ (ืืื ืื ืคืึธืจืขื ืคืื ืึท ืงืึทื ืขืงืืขืจ ืืื ืงืึทืคืงืึท). ืึธืืขืจ, ืื ืืืืืื ื ืืื ืืืคึผืจืืฉืืืืืื ืืื ื ืืืขืจืกืืข 1.1 ืืื ืืื ื ืื ืืขืจ ืจืขืงืึทืืขื ืืื ืคึฟืึทืจ ื ืืฆื (ืฉืืืฆื ืคึฟืึทืจ ืขืก ืงืขื ืืืื ืึทืืืขืงืืขื ืืืขื ืืื ืฆืืงืื ืคึฟื ืจืืืืกืื).
ืืขืจ ืึทืจืืืงื ืืืขื ืืืกืงืืืืจื ืื ืึทืจืงืึทืืขืงืืฉืขืจ ืจืขืงืึทืืขื ืืื ืืืจื ืืขืืืขืืึธืคึผืขืจืก, ืืืึธืก ืืื ืฉืืื ืืึธืืขืจืึทื ืฅ ืืื ืกืงืึทืืึทืืืืืื.
ืงืึทื ืขืงืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื
ืืื ืกืืจ ืฆื ืึธื ืืืืื ืืจืึทืงืื ื ืขื ืืขืจืื ืืขื ืืื ืื ืืขืจืกื ืืืืืืืง ืืืขืจื - ืืึทืื - ืืืจ ืืึทืจืคึฟื:
- ืืึทืื ืืงืืจ, ืืืึธืก ืงืขื ืขื ืืืื MySQL ืกืืึทืจืืื ื ืคึฟืื ืืืขืจืกืืข 5.7, PostgreSQL 9.6+, MongoDB 3.2+ ();
- ืึทืคึผืึทืืฉื ืงืึทืคืงืึท ืงื ืืื;
- ืงืึทืคืงืึท ืงืึธื ื ืขืงื ืืืึทืฉืคึผืื (ืืืขืจืกืืขืก 1.ืงืก, 2.ืงืก);
- ืงืึทื ืคืืืืขืจื ืืขืืขืืืื ืงืึทื ืขืงืืขืจ.
ืึทืจืืขื ืืืืฃ ืื ืขืจืฉืืข ืฆืืืื ืคืื ืงืื, ื.ื. ืืขืจ ืืึทื ืืึธื ืืืจืื ื ืคึผืจืึธืฆืขืก ืคืื ืื DBMS ืืื Apache Kafka ืืื ืืืืึทืืขืจ ืคืื ืื ืคืึทืจื ืขื ืคืื ืืขื ืึทืจืืืงื. ืึธืืขืจ, ืคึฟืึทืจ ืื ืืืืก ืืืืื ืฆื ืฆืขืืืืงืืขื ืึทืืฅ ืืื ืื ืืึทืืืงืึทืกืื, ืืขืจ ืืึทืึทืืืขืจ ืจืืคึผืึทืืึทืืึธืจื ืืื ืืืืฉืคืืื ืืื ืึท ืคืึทืจืืืง .
ืืืืฃ ืื ืืขืฆืืข ืฆืืืื ื ืงืืืืช ืืืขืื ืืืจ ืืื ืืขืจ ืืขืืึทืื ืึธืคึผืฉืืขืื.
0. ืงืึทืคืงืึท ืงืึธื ื ืขืงื
ืืึธ ืืื ืืืืึทืืขืจ ืืื ืืขื ืึทืจืืืงื, ืึทืืข ืงืึทื ืคืืืืขืจืืืฉืึทื ืืืืฉืคืืื ืืขื ืขื ืืืกืงืึทืกื ืืื ืืขื ืงืึธื ืืขืงืกื ืคืื ืื ืืึธืงืงืขืจ ืืืื ืคืื ืื ืืขืจืืขืืืืื ืืืจื ืื ืืขืืขืืืื ืืขืืืขืืึธืคึผืขืจืก. ืขืก ืึผืืื ืึทืืข ืื ื ืืืืืง ืคึผืืืืื ืืขืงืขืก (ืงืึทื ืขืงืืขืจื) ืืื ืืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืคืื Kafka Connect ื ืืฆื ืกืืืืืืข ืืืขืจืืึทืืึทืื.
ืืืื ืืืจ ืืืขื ืฆื ื ืืฆื Kafka Connect ืคึฟืื ืงืึธื ืคืืืขื ื, ืืืจ ืืืขื ืืึทืจืคึฟื ืฆื ืื ืืืคึผืขื ืืึทื ืืื ืืืืื ืื ืคึผืืืืื ืก ืคืื ืื ื ืืืืืง ืงืึทื ืขืงืืขืจื ืฆื ืื ืืืขืืืืืึทืืขืจ ืกืคึผืขืกืืคืืขื ืืื plugin.path ืึธืืขืจ ืฉืืขืื ืืืจื ืึท ืกืืืืืืข ืืืึทืืขืืืืืง CLASSPATH. ืกืขืืืื ืืก ืคึฟืึทืจ ืื Kafka Connect ืึทืจืืขืืขืจ ืืื ืงืึทื ืขืงืืขืจื ืืขื ืขื ืืืฉืืืกื ืืืจื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืขืก ืืืึธืก ืืขื ืขื ืืืจืืืขืืื ืืขื ืืื ืึทืจืืืืขื ืื ืฆื ืื ืึทืจืืขืืขืจ ืงืึทืืขืจ ืืึทืคึฟืขื. ืคึฟืึทืจ ืคืจืืื ืืขื .
ืืขืจ ืืื ืฆืขืจ ืคึผืจืึธืฆืขืก ืคืื ืืึทืฉืืขืืืงื Debeizum ืืื ืื ืงืึทื ืขืงืืขืจ ืืืขืจืกืืข ืืื ืืืจืืืขืงืึธืื ืืื ืฆืืืื ืกืืึทืืขืก. ืืื ืก ืงืืง ืืืึท ืืขืืขืจ ืคืื ืืื:
1. ืืึทืฉืืขืืืงื ืื Kafka Connect ืคืจืืืืืืขืจืง
ืฆื ืกืืจืืืื ื ืืึทืื ืฆื ืื Apache Kafka ืงื ืืื, ืกืคึผืขืฆืืคืืฉ ืคึผืึทืจืึทืืขืืขืจืก ืืขื ืขื ืืึทืฉืืืื ืืื ืื Kafka Connect ืคืจืืืืืืขืจืง, ืึทืืึท ืืื:
- ืคึผืึทืจืึทืืขืืขืจืก ืคึฟืึทืจ ืงืึทื ืขืงืืื ื ืฆื ืื ืงื ืืื,
- ื ืขืืขื ืคืื ืืขืืขืก ืืื ืืืึธืก ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืคืื ืื ืงืึทื ืขืงืืขืจ ืืื ืืืขื ืืืื ืกืืึธืจื ืืืืึทื,
- ืื ื ืึธืืขื ืคืื ืืขืจ ืืจืืคึผืข ืืื ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ืืื ืคืืืกื ืืืง (ืืืื ืืืกืืจืืืืืืื ืืึธืืข ืืื ืืขื ืืฆื).
ืืขืจ ืืึทืึทืืืขืจ ืืึธืงืงืขืจ ืืืื ืคืื ืื ืคึผืจืืืขืงื ืฉืืืฆื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ื ืืฆื ืกืืืืืืข ืืืขืจืืึทืืึทืื - ืืึธืก ืืื ืืืึธืก ืืืจ ืืืขืื ื ืืฆื. ืึทืืื, ืืจืืคืงืืคืืข ืื ืืืื:
docker pull debezium/connectืื ืืื ืืืื ืืึทื ื ืคืื ืื ืืืืืจืึทื ืืขื ืึทื ืืืขืจืืึทืืึทืื ืคืืจืืื ืื ืฆื ืืืืคื ืื ืงืึทื ืขืงืืขืจ ืืื ืืื ืืืื:
-
BOOTSTRAP_SERVERS=kafka-1:9092,kafka-2:9092,kafka-3:9092- ืขืจืฉื ืจืฉืืื ืคืื Kafka ืงื ืืื ืกืขืจืืืขืจืก ืฆื ืืึทืงืืืขื ืึท ืืึทื ืฅ ืจืฉืืื ืคืื ืงื ืืื ืืืืืืืืขืจ; -
OFFSET_STORAGE_TOPIC=connector-offsets- ืึท ืืขืืข ืคึฟืึทืจ ืกืืึธืจืื ื ืฉืืขืืขืก ืืื ืื ืงืึทื ืขืงืืขืจ ืืื ืืขืจืืืืึทื ืืืื; -
CONNECT_STATUS_STORAGE_TOPIC=connector-status- ืืขืืข ืคึฟืึทืจ ืกืืึธืจืื ื ืื ืกืืึทืืืก ืคืื ืื ืงืึทื ืขืงืืขืจ ืืื ืืืึทื ืืึทืกืงืก; -
CONFIG_STORAGE_TOPIC=connector-config- ืืขืืข ืคึฟืึทืจ ืกืืึธืจืื ื ืงืึทื ืขืงืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืืึทืื ืืื ืืืึทื ืืึทืกืงืก; -
GROUP_ID=1- ืืืืขื ืืืคึฟืืฆืืจืื ื ืคืื ืืขืจ ืืจืืคึผืข ืคืื โโืืืขืจืก ืืืืฃ ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ืึทืจืืขื ืงืขื ืขื ืืืื ืขืงืกืึทืงืืืืึทื; ื ืืืืืง ืืืขื ื ืืฆื ืคืื ืื ืืขืจืืขืืืืื (ืคืึทืจืฉืคึผืจืืืื) ืจืขืืฉืื.
ืืืจ ืงืึทืืขืจ ืืขื ืงืึทื ืืืื ืขืจ ืืื ืื ืืืขืจืืึทืืึทืื:
docker run
-e BOOTSTRAP_SERVERS='kafka-1:9092,kafka-2:9092,kafka-3:9092'
-e GROUP_ID=1
-e CONFIG_STORAGE_TOPIC=my_connect_configs
-e OFFSET_STORAGE_TOPIC=my_connect_offsets
-e STATUS_STORAGE_TOPIC=my_connect_statuses debezium/connect:1.2ืืึทืืขืจืงืื ื ืืืขืื ืึทืืืจืึธ
ืืืจื ืคืขืืืงืืึทื, Debezium ืฉืจืืืื ืืึทืื ืืื JSON ืคึฟืึธืจืืึทื, ืืืึธืก ืืื ืคึผืึทืกืืง ืคึฟืึทืจ ืืึทืืืืึธืงืกืขืก ืืื ืงืืืื ืึทืืึทืื ืฅ ืคืื ืืึทืื, ืึธืืขืจ ืงืขื ืขื ืืืขืจื ืึท ืคึผืจืึธืืืขื ืืื ืืขืืกื ืืึธืืืื ืืึทืืึทืืืืกืื. ืึทื ืึธืืืขืจื ืึทืืืื ืฆื ืึท JSON ืงืึทื ืืืขืจืืขืจ ืืื ืฆื ืกืืจืืึทืืืืืจื ืึทืจืืืงืืขื ื ืืฆื ืฆื ืึท ืืืื ืขืจื ืคึฟืึธืจืืึทื, ืืืึธืก ืจืึทืืืกืึทื ืื ืืึทืกืข ืืืืฃ ืื I / O ืกืึทืืกืืกืืขื ืืื Apache Kafka.
ืฆื ื ืืฆื Avro ืืืจ ืืึทืจืคึฟื ืฆื ืฆืขืืืืงืืขื ืึท ืืึทืืื ืืขืจ (ืคึฟืึทืจ ืกืืึธืจืื ื ืืืืึทืืจืึทืื). ืื ืืืขืจืืึทืืึทืื ืคึฟืึทืจ ืื ืงืึทื ืืืขืจืืขืจ ืืืขื ืงืืงื ืืื ืืึธืก:
name: CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
value: http://kafka-registry-01:8081/
name: CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL
value: http://kafka-registry-01:8081/
name: VALUE_CONVERTER
value: io.confluent.connect.avro.AvroConverterืืขืืึทืืืก ืืืขืื ื ืืฆื Avro ืืื ืืึทืฉืืขืืืงื ืื ืจืขืืืกืืจื ืคึฟืึทืจ ืขืก ืืขื ืขื ืืืืึทืืขืจ ืคืื ืืขื ืคืึทืจื ืขื ืคืื ืืขื ืึทืจืืืงื - ืืืืึทืืขืจ, ืคึฟืึทืจ ืงืืขืจืืื, ืืืจ ืืืขืื ื ืืฆื JSON.
2. ืงืึทื ืคืืืืขืจ ืื ืงืึทื ืขืงืืขืจ ืืื
ืืืฆื ืืืจ ืงืขื ืขื ืืืื ืืืืึทื ืฆื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืคืื ืื ืงืึทื ืขืงืืขืจ ืืื, ืืืึธืก ืืืขื ืืืืขื ืขื ืื ืืึทืื ืคึฟืื ืื ืืงืืจ.
ืืึธืืืจ ืงืืงื ืืื ืืขื ืืืืฉืคึผืื ืคืื ืงืึทื ืขืงืืขืจื ืคึฟืึทืจ ืฆืืืื ืืืืก: PostgreSQL ืืื MongoDB, ืืื ืืืึธืก ืืื ืืึธืื ืืขืจืคืึทืจืื ื ืืื ืืื ืืืึธืก ืขืก ืืขื ืขื ืืืคืขืจืึทื ืกืื (ืึธืืืืื ืงืืืื, ืึธืืขืจ ืืื ืขืืืขืืข ืงืึทืกืขืก ืืึทืืืึทืืืง!).
ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืื ืืืกืงืจืืืื ืืื JSON ื ืึธืืืืืฉืึทื ืืื ืืคึผืืึธืึทืืขื ืฆื Kafka Connect ืืื ืึท POST ืืขืื.
2.1. PostgreSQL
ืืืึทืฉืคึผืื ืงืึทื ืขืงืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืคึฟืึทืจ PostgreSQL:
{
"name": "pg-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"database.hostname": "127.0.0.1",
"database.port": "5432",
"database.user": "debezium",
"database.password": "definitelynotpassword",
"database.dbname" : "dbname",
"database.server.name": "pg-dev",
"table.include.list": "public.(.*)",
"heartbeat.interval.ms": "5000",
"slot.name": "dbname_debezium",
"publication.name": "dbname_publication",
"transforms": "AddPrefix",
"transforms.AddPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.AddPrefix.regex": "pg-dev.public.(.*)",
"transforms.AddPrefix.replacement": "data.cdc.dbname"
}
}ืืขืจ ืคึผืจืื ืฆืืคึผ ืคืื ืึธืคึผืขืจืึทืฆืืข ืคืื โโืื ืงืึทื ืขืงืืขืจ ื ืึธื ืืขื ืกืขืืึทืคึผ ืืื ืืึทื ืฅ ืคึผืฉืื:
- ืืืขื ืืึธื ืืฉื ืคึฟืึทืจ ืื ืขืจืฉืืขืจ ืืึธื, ืขืก ืงืึทื ืขืงืฅ ืฆื ืื ืืึทืืึทืืืืก ืกืคึผืขืกืืคืืขื ืืื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืื ืกืืึทืจืฅ ืืื ืืึธืืข ืขืจืฉื ืืึธืืขื ืืืืื, ืฉืืงื ืฆื Kafka ืื ืขืจืฉื ืกืืื ืคืื ืืึทืื ืืืงืืืขื ื ืืฆื ืื ืงืึทื ืืืฉืึทื ืึทื
SELECT * FROM table_name. - ื ืึธื ืื ืืืืึทืืืืึทืืืึธื ืืื ืืึทื ืฅ, ืื ืงืึทื ืขืงืืขืจ ืืืื ืืื ืืึธืืข ืฆื ืืืืขื ืขื ืขื ืืขืจืื ืืขื ืคืื PostgreSQL WAL ืืขืงืขืก.
ืืืขืื ืื ืึธืคึผืฆืืขืก ืืขื ืืฆื:
-
name- ืื ื ืึธืืขื ืคืื ืื ืงืึทื ืขืงืืขืจ ืคึฟืึทืจ ืืืึธืก ืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืืกืงืจืืืื ืืื ืื ืืื ืืขื ืืฆื; ืืื ืืขืจ ืฆืืงืื ืคึฟื, ืืขื ื ืึธืืขื ืืื ืืขื ืืฆื ืฆื ืึทืจืืขืื ืืื ืื ืงืึทื ืขืงืืขืจ (ื"ื ืงืืง ืื ืกืืึทืืืก / ืจืืกืืึทืจื / ืืขืจืืืึทื ืืืงื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื) ืืืจื ืื Kafka Connect REST API; -
connector.class- DBMS ืงืึทื ืขืงืืขืจ ืงืืึทืก ืืืึธืก ืืืขื ืืืื ืืขืืืืื ื ืืืจื ืื ืงืึทื ืคืืืืขืจื ืงืึทื ืขืงืืขืจ; -
plugin.name- ืื ื ืึธืืขื ืคืื ืื ืคึผืืืืื ืคึฟืึทืจ ืืึทืืืฉืืงืึทื ืืืงืึธืืืื ื ืคืื ืืึทืื ืคึฟืื WAL ืืขืงืขืก. ืื ืืืฆื ืฆื ืงืืืึทืื ืคืืwal2json,decoderbuffsะธpgoutput. ืื ืขืจืฉืืข ืฆืืืื ืืึทืจืคื ืื ืกืืึทืืืจืื ื ืคืื ืื ืฆืื ืขืืขื ืืงืกืืขื ืฉืึทื ื ืืื ืื DBMS, ืืืpgoutputืคึฟืึทืจ PostgreSQL ืืืขืจืกืืข 10 ืืื ืืขืืขืจ ืืื ื ืืฉื ืืึทืจืคื ื ืึธื ืืึทื ืืคึผืืึทืืืืฉืึทื ื; -
database.*- ืึธืคึผืฆืืขืก ืคึฟืึทืจ ืงืึทื ืขืงืืื ื ืฆื ืื ืืึทืืึทืืืืก, ืืืdatabase.server.name- PostgreSQL ืืืึทืฉืคึผืื ื ืึธืืขื ืืขื ืืฆื ืฆื ืคืึธืจืขื ืื ืืขืืข ื ืึธืืขื ืืื ืื Kafka ืงื ืืื; -
table.include.list- ืึท ืจืฉืืื ืคืื ืืืฉื ืืื ืืืึธืก ืืืจ ืืืืื ืฆื ืฉืคึผืืจ ืขื ืืขืจืื ืืขื; ืกืคึผืขืกืืคืืขื ืืื ืืขื ืคึฟืึธืจืืึทืschema.table_name; ืงืขื ืขื ื ืื ืืืื ืืขืืืืื ื ืฆืืืึทืืขื ืืืtable.exclude.list; -
heartbeat.interval.ms- ืืขืืึทืืขื (ืืื ืืืืืกืขืงืึทื ืื) ืืื ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ืกืขื ืื ืืึทืจืืืื ืึทืจืืืงืืขื ืฆื ืึท ืกืคึผืขืฆืืขื ืืขืืข; -
heartbeat.action.query- ืึท ืืงืฉื ืืืึธืก ืืืขื ืืืื ืขืงืกืึทืงืืืืึทื ืืืขื ืืืจ ืฉืืงื ืืขืืขืจ ืืึทืจืืืื ืึธื ืืึธื (ืื ืึธืคึผืฆืืข ืืื ืืจืืืก ืืื ืืืขืจืกืืข 1.1); -
slot.name- ืื ื ืึธืืขื ืคืื ืื ืจืขืคึผืืึทืงืืืฉืึทื ืฉืคึผืขืืื ืืืึธืก ืืืขื ืืืื ืืขืืืืื ื ืืืจื ืื ืงืึทื ืขืงืืขืจ; publication.name- ื ืึธืืขื ืืื PostgreSQL, ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ื ืืฆื. ืืืื ืขืก ืืื ื ืืฉื ืขืงืกืืกืืืจื, ืืขืืขืืืื ืืืขื ืคึผืจืืืืจื ืฆื ืืึทืื ืขืก. ืืืื ืืขืจ ืืึทื ืืฆืขืจ ืืื ืืขืจ ืืืขืืขื ืื ืงืฉืจ ืืื ืืขืืืื ืืื ื ืืฉื ืืขื ืื ืจืขืื ืคึฟืึทืจ ืืขื ืงืึทืืฃ, ืืขืจ ืงืึทื ืขืงืืขืจ ืืืขื ืคืึทืจืขื ืืืงื ืืื ืึท ืืขืืช;-
transformsืืืืขืจืืึทื ื ืคึผืื ืงื ืืื ืฆื ืืืืฉื ืื ื ืึธืืขื ืคืื ืื ืฆืื ืืขืืข:-
transforms.AddPrefix.typeืื ืืืงืืืฅ ืึทื ืืืจ ืืืขืื ื ืืฆื ืจืขืืืืขืจ ืืืืกืืจืืงื; -
transforms.AddPrefix.regex- ืึท ืืึทืกืงืข ืืืึธืก ืจืืืืคืื ื ืื ื ืึธืืขื ืคืื ืื ืฆืื ืืขืืข; -
transforms.AddPrefix.replacement- ืืืืื ืืืึธืก ืืืจ ืจืืืืคืืื ืื ื.
-
ืืขืจ ืืืขืื ืืึทืจืืืื ืืื ืืจืึทื ืกืคืึธืจืืึทืฆืืข
ืืืจื ืคืขืืืงืืึทื, ืื ืงืึทื ืขืงืืขืจ ืกืขื ืื ืืึทืื ืฆื Kafka ืคึฟืึทืจ ืืขืืขืจ ืืืืื ืืขื ืืจืึทื ืกืึทืงืืืึธื, ืืื ืืืื LSN (ืืึธื ืกืืงืืืึทื ืก ื ืืืขืจ) ืืื ืจืขืงืึธืจืืขื ืืื ืื ืกืขืจืืืืก ืืขืืข offset. ืึธืืขืจ ืืืึธืก ืืึทืคึผืึทื ื ืืืื ืื ืงืึทื ืขืงืืขืจ ืืื ืงืึทื ืคืืืืขืจื ืฆื ืืืืขื ืขื ื ืืฉื ืื ืืื ืฆืข ืืึทืืึทืืืืก, ืึธืืขืจ ืืืืื ืืืื ืคืื ืืืึทื ืืืฉื (ืืื ืืืึธืก ืืึทืื ืืขืจืืืึทื ืืืงืื ืืขื ืืึธื ื ืื ืคืึทืื ืึธืคื)?
- ืืขืจ ืงืึทื ืขืงืืขืจ ืืืขื ืืืืขื ืขื WAL ืืขืงืขืก ืืื ืืืขื ื ืืฉื ืืขืืขืงื ืงืืื ืืจืึทื ืกืึทืงืืืึธื ืงืึทืืืฅ ืฆื ืื ืืืฉื ืขืก ืืื ืืึธื ืืืึธืจืื ื.
- ืืขืจืืืขืจ, ืขืก ืืืขื ื ืืฉื ืืขืจืืืึทื ืืืงื ืืืื ืงืจืึทื ื ืฉืืขืืข ืืื ืืขืจ ืืขืืข ืึธืืขืจ ืืื ืื ืจืขืคึผืืึทืงืืืฉืึทื ืฉืคึผืขืืื.
- ืืึธืก, ืืื ืงืขืจ, ืืืขื ืคืืจื ืฆื WAL ืืขืงืขืก ืืืืฃ ืืืกืง ืืื ืืืกืืึธืืข ืืืืคื ืืืืก ืคืื ืืืกืง ืคึผืืึทืฅ.
ืืื ืืึธืก ืืื ืืื ืึธืคึผืฆืืขืก ืงืืืขื ืฆื ืจืึทืืขืืืขื. heartbeat.interval.ms ะธ heartbeat.action.query. ื ืืฆื ืื ืึธืคึผืฆืืขืก ืืื ืคึผืขืจื ืืืื ืขืก ืืขืืืขื ืฆื ืืืจืืคืืจื ืึท ืืงืฉื ืฆื ืืืืฉื ืืึทืื ืืื ืึท ืืึทืืื ืืขืจ ืืืฉ ืืขืืขืจ ืืึธื ืึท ืืขืึทืจืืืขืึทื ืึธื ืืึธื ืืื ืืขืฉืืงื. ืืืื, ืื LSN ืืืืฃ ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ืืื ืืืฆื ืืืื (ืืื ืื ืจืขืคึผืืึทืงืืืฉืึทื ืฉืคึผืขืืื) ืืื ืงืขืกืืืืขืจ ืืขืจืืืึทื ืืืงื. ืืึธืก ืึทืืึทืื ืื DBMS ืฆื ืืึทืืืึทืืืงื WAL ืืขืงืขืก ืืืึธืก ืืขื ืขื ื ืื ืืขืจ ืืืจืฃ. ืืืจ ืงืขื ื ืืขืจื ืขื ืืขืจ ืืืขืื ืืื ืื ืึธืคึผืฆืืขืก ืึทืจืืขื ืืื .
ืื ืื ืืขืจ ืึธืคึผืฆืืข ืืืขืจื ืคืื ื ืขืขื ืืขืจ ืืคืืขืจืงืืึทืืงืืึทื ืืื transforms. ืืึธืืฉ ืขืก ืืื ืืขืจ ืืืขืื ืงืึทื ืืืื ืืึทื ืก ืืื ืฉืืื ืงืืื ...
ืืืจื ืคืขืืืงืืึทื, Debezium ืงืจืืืืฅ ืืขืืขืก ื ืืฆื ืื ืคืืืืขื ืืข ื ืึทืืื ื ืคึผืึธืืืืืง: serverName.schemaName.tableName. ืืึธืก ืงืขื ื ืืฉื ืฉืืขื ืืืง ืืืื ืืึทืงืืืขื. ืึธืคึผืฆืืขืก transforms ืืืจ ืงืขื ืขื ื ืืฆื ืจืขืืืืขืจ ืืืืกืืจืืงื ืฆื ืืขืคืื ืืจื ืึท ืจืฉืืื ืคืื ืืืฉื, ืืขืฉืขืขื ืืฉื ืคืื ืืืึธืก ืืืจ ืืึทืจืคึฟื ืฆื ืืืื ืจืึทืืืื ืฆื ืึท ืืขืืข ืืื ืึท ืกืคึผืขืฆืืคืืฉ ื ืึธืืขื.
ืืื ืืื ืืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืืึทื ืงืขื transforms ืื ืคืืืืขื ืืข ืืึทืคึผืึทื ื: ืึทืืข CDC events ืคืื ืื ืืึธื ืืืึธืจืขื ืืึทืืึทืืืืก ืืืขื ืืืื ืฆื ืึท ืืขืืข ืืื ืืขื ื ืึธืืขื data.cdc.dbname. ืึทื ืืขืจืฉ (ืึธื ืื ืกืขืืืื ืืก), Debezium ืืืึธืื ืืืจื ืคืขืืืงืืึทื ืฉืึทืคึฟื ืึท ืืขืืข ืคึฟืึทืจ ืืขืืขืจ ืืืฉ ืืื: pg-dev.public.<table_name>.
ืงืึทื ืขืงืืขืจ ืืืืืืืืฉืึทื ื
ืฆื ืคืึทืจืขื ืืืงื ืื ืืึทืฉืจืืึทืืื ื ืคืื ืื ืงืึทื ืขืงืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืคึฟืึทืจ PostgreSQL, ืขืก ืืื ืืืขืจื ืฆื ืจืขืื ืืืขืื ืื ืคืืืืขื ืืข ืคึฟืขืึดืงืืืื / ืืืืืืืืฉืึทื ื ืคืื ืืืึทื ืึธืคึผืขืจืึทืฆืืข:
- ืื ืคืึทื ืืงืฉืึทื ืึทืืืื ืคืื ืื ืงืึทื ืขืงืืขืจ ืคึฟืึทืจ PostgreSQL ืจืืืืื ืืืืฃ ืืขืจ ืืึทืืจืืฃ ืคืื ืืึทืืืฉืืงืึทื ืืืงืึธืืืื ื. ืืขืจืืืขืจ ืขืจ ืืื ื ืืฉื ืฉืคึผืืจ ืจืืงืืืขืก ืฆื ืืืืฉื ืื ืืึทืืึทืืืืก ืกืืจืืงืืืจ (DDL) - ืึทืงืึธืจืืื ืืื, ืื ืืึทืื ืืืขื ื ืืฉื ืืืื ืืื ืื ืืขืืขืก.
- ืืื ื ืจืขืคึผืืึทืงืืืฉืึทื ืกืืึธืฅ ืืขื ืขื ืืขื ืืฆื, ืงืึทื ืขืงืืื ื ืึท ืงืึทื ืขืงืืขืจ ืืื ืืขืืืขื ืืืืื ืฆื ืื ืืืืื ื DBMS ืืืึทืฉืคึผืื.
- ืืืื ืืขืจ ืืึทื ืืฆืขืจ ืืื ืืขืจ ืืืขืืขื ืื ืงืึทื ืขืงืืขืจ ืงืึทื ืขืงืฅ ืฆื ืื ืืึทืืึทืืืืก ืืื ืืขืืขืื ืืืืื ืืืืขื ืขื ืจืขืื, ืืืืืขืจ ืืขืจ ืขืจืฉืืขืจ ืงืึทืืขืจ ืืืจ ืืึทืจืคึฟื ืฆื ืืึทื ืืืึทืื ืฉืึทืคึฟื ืึท ืจืขืคึผืืึทืงืืืฉืึทื ืฉืคึผืขืืื ืืื ืึทืจืืืกืืขืื ืฆื ืื ืืึทืืึทืืืืก.
ืึทืคึผืืืืื ื ืึท ืงืึธื ืคืืืืจืึทืืืึธื
ืึทืืื, ืืึธืื ืืื ืื ืืึธืื ืืื ืืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืืื ืื ืงืึทื ืขืงืืขืจ:
curl -i -X POST -H "Accept:application/json"
-H "Content-Type:application/json" http://localhost:8083/connectors/
-d @pg-con.jsonืืืจ ืงืึธื ืืจืึธืืืจื ืึทื ืื ืืจืืคืงืืคืืข ืืื ืืขืืืขื ืืขืจืึธืื ืืื ืื ืงืึทื ืขืงืืขืจ ืกืืึทืจืืขื:
$ curl -i http://localhost:8083/connectors/pg-connector/status
HTTP/1.1 200 OK
Date: Thu, 17 Sep 2020 20:19:40 GMT
Content-Type: application/json
Content-Length: 175
Server: Jetty(9.4.20.v20190813)
{"name":"pg-connector","connector":{"state":"RUNNING","worker_id":"172.24.0.5:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.24.0.5:8083"}],"type":"source"}ืืจืืืก: ืขืก ืืื ืกืขืืึทืคึผ ืืื ืืจืืื ืฆื ืืืื. ืืืฆื ืืึธืื ืืื ืื ืคืึทืจืืืื ืฆื ืืืื ืึท ืงืึธื ืกืืืขืจ ืืื ืคืึทืจืืื ืื ืฆื Kafka, ื ืึธื ืืืึธืก ืืืจ ืืืขืื ืืืืื ืืื ืืืืฉื ืึท ืคึผืึธืืืฆืืข ืืื ืื ืืืฉ:
$ kafka/bin/kafka-console-consumer.sh
--bootstrap-server kafka:9092
--from-beginning
--property print.key=true
--topic data.cdc.dbname
postgres=# insert into customers (id, first_name, last_name, email) values (1005, 'foo', 'bar', '[email protected]');
INSERT 0 1
postgres=# update customers set first_name = 'egg' where id = 1005;
UPDATE 1ืืื ืืื ืืืขืจ ืืขืืข ืขืก ืืืขื ืืืื ืืขืืืืื ืืื ืืืื:
ืืืืขืจ ืืึทื ื JSON ืืื ืืื ืืืขืจ ืขื ืืขืจืื ืืขื
{
"schema":{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
}
],
"optional":false,
"name":"data.cdc.dbname.Key"
},
"payload":{
"id":1005
}
}{
"schema":{
"type":"struct",
"fields":[
{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
},
{
"type":"string",
"optional":false,
"field":"first_name"
},
{
"type":"string",
"optional":false,
"field":"last_name"
},
{
"type":"string",
"optional":false,
"field":"email"
}
],
"optional":true,
"name":"data.cdc.dbname.Value",
"field":"before"
},
{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
},
{
"type":"string",
"optional":false,
"field":"first_name"
},
{
"type":"string",
"optional":false,
"field":"last_name"
},
{
"type":"string",
"optional":false,
"field":"email"
}
],
"optional":true,
"name":"data.cdc.dbname.Value",
"field":"after"
},
{
"type":"struct",
"fields":[
{
"type":"string",
"optional":false,
"field":"version"
},
{
"type":"string",
"optional":false,
"field":"connector"
},
{
"type":"string",
"optional":false,
"field":"name"
},
{
"type":"int64",
"optional":false,
"field":"ts_ms"
},
{
"type":"string",
"optional":true,
"name":"io.debezium.data.Enum",
"version":1,
"parameters":{
"allowed":"true,last,false"
},
"default":"false",
"field":"snapshot"
},
{
"type":"string",
"optional":false,
"field":"db"
},
{
"type":"string",
"optional":false,
"field":"schema"
},
{
"type":"string",
"optional":false,
"field":"table"
},
{
"type":"int64",
"optional":true,
"field":"txId"
},
{
"type":"int64",
"optional":true,
"field":"lsn"
},
{
"type":"int64",
"optional":true,
"field":"xmin"
}
],
"optional":false,
"name":"io.debezium.connector.postgresql.Source",
"field":"source"
},
{
"type":"string",
"optional":false,
"field":"op"
},
{
"type":"int64",
"optional":true,
"field":"ts_ms"
},
{
"type":"struct",
"fields":[
{
"type":"string",
"optional":false,
"field":"id"
},
{
"type":"int64",
"optional":false,
"field":"total_order"
},
{
"type":"int64",
"optional":false,
"field":"data_collection_order"
}
],
"optional":true,
"field":"transaction"
}
],
"optional":false,
"name":"data.cdc.dbname.Envelope"
},
"payload":{
"before":null,
"after":{
"id":1005,
"first_name":"foo",
"last_name":"bar",
"email":"[email protected]"
},
"source":{
"version":"1.2.3.Final",
"connector":"postgresql",
"name":"dbserver1",
"ts_ms":1600374991648,
"snapshot":"false",
"db":"postgres",
"schema":"public",
"table":"customers",
"txId":602,
"lsn":34088472,
"xmin":null
},
"op":"c",
"ts_ms":1600374991762,
"transaction":null
}
}{
"schema":{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
}
],
"optional":false,
"name":"data.cdc.dbname.Key"
},
"payload":{
"id":1005
}
}{
"schema":{
"type":"struct",
"fields":[
{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
},
{
"type":"string",
"optional":false,
"field":"first_name"
},
{
"type":"string",
"optional":false,
"field":"last_name"
},
{
"type":"string",
"optional":false,
"field":"email"
}
],
"optional":true,
"name":"data.cdc.dbname.Value",
"field":"before"
},
{
"type":"struct",
"fields":[
{
"type":"int32",
"optional":false,
"field":"id"
},
{
"type":"string",
"optional":false,
"field":"first_name"
},
{
"type":"string",
"optional":false,
"field":"last_name"
},
{
"type":"string",
"optional":false,
"field":"email"
}
],
"optional":true,
"name":"data.cdc.dbname.Value",
"field":"after"
},
{
"type":"struct",
"fields":[
{
"type":"string",
"optional":false,
"field":"version"
},
{
"type":"string",
"optional":false,
"field":"connector"
},
{
"type":"string",
"optional":false,
"field":"name"
},
{
"type":"int64",
"optional":false,
"field":"ts_ms"
},
{
"type":"string",
"optional":true,
"name":"io.debezium.data.Enum",
"version":1,
"parameters":{
"allowed":"true,last,false"
},
"default":"false",
"field":"snapshot"
},
{
"type":"string",
"optional":false,
"field":"db"
},
{
"type":"string",
"optional":false,
"field":"schema"
},
{
"type":"string",
"optional":false,
"field":"table"
},
{
"type":"int64",
"optional":true,
"field":"txId"
},
{
"type":"int64",
"optional":true,
"field":"lsn"
},
{
"type":"int64",
"optional":true,
"field":"xmin"
}
],
"optional":false,
"name":"io.debezium.connector.postgresql.Source",
"field":"source"
},
{
"type":"string",
"optional":false,
"field":"op"
},
{
"type":"int64",
"optional":true,
"field":"ts_ms"
},
{
"type":"struct",
"fields":[
{
"type":"string",
"optional":false,
"field":"id"
},
{
"type":"int64",
"optional":false,
"field":"total_order"
},
{
"type":"int64",
"optional":false,
"field":"data_collection_order"
}
],
"optional":true,
"field":"transaction"
}
],
"optional":false,
"name":"data.cdc.dbname.Envelope"
},
"payload":{
"before":{
"id":1005,
"first_name":"foo",
"last_name":"bar",
"email":"[email protected]"
},
"after":{
"id":1005,
"first_name":"egg",
"last_name":"bar",
"email":"[email protected]"
},
"source":{
"version":"1.2.3.Final",
"connector":"postgresql",
"name":"dbserver1",
"ts_ms":1600375609365,
"snapshot":"false",
"db":"postgres",
"schema":"public",
"table":"customers",
"txId":603,
"lsn":34089688,
"xmin":null
},
"op":"u",
"ts_ms":1600375609778,
"transaction":null
}
}ืืื ืืืืืข ืงืึทืกืขืก, ืจืขืงืึธืจืืก ืฆืื ืืืคืฉืืขืื ืืื ืคืื ืื ืฉืืืกื (PK) ืคืื ืื ืจืขืงืึธืจื ืืืึธืก ืืื ืืขืืืขื ืืฉืืื ืืืฉื, ืืื ืื ืืืืขืจ ืขืกืึทื ืก ืคืื ืื ืขื ืืขืจืื ืืขื: ืืืึธืก ืื ืจืขืงืึธืจื ืืื ืืขืืืขื ืคืจืืขืจ ืืื ืืืึธืก ืขืก ืืื ืืขืืืืจื ื ืึธื.
- ืืื ืืขื ืคืึทื
INSERT: ืืืขืจื ืคืจืืขืจ (before) ืืืืืnull, ืืื ื ืึธื - ืื ืฉืืจื ืืืึธืก ืืื ืืขืืืขื ืื ืกืขืจืืึทื. - ืืื ืืขื ืคืึทื
UPDATE: ืืืpayload.beforeืื ืคืจืืขืจืืืงืข ืฉืืึทื ืคืื ืื ืฉืืจื ืืื ืืขืืืืื, ืืื ืืืpayload.afterโ ื ืฒืขื ืืื ืืขื ืืืืช ืคืื ืขื ืืขืจืื ืืขื.
2.2 MongoDB
ืืขืจ ืงืึทื ืขืงืืขืจ ื ืืฆื ืื ื ืึธืจืืึทื MongoDB ืจืขืคึผืืึทืงืืืฉืึทื ืืขืงืึทื ืืืึทื, ืืืืขื ืขื ืืื ืคึฟืึธืจืืึทืฆืืข ืคึฟืื ืื ืึธืคึผืืึธื ืคืื ืื ืขืจืฉืืืง DBMS ื ืึธืืข.
ืขื ืืขื ืฆื ืื ืฉืืื ืืืกืงืจืืืื ืงืึทื ืขืงืืขืจ ืคึฟืึทืจ PgSQL, ืืึธ ืืืื, ืืื ืืขืจ ืขืจืฉืืขืจ ืึธื ืืืื, ืื ืขืจืฉืืืง ืืึทืื ืืึธืืขื ืืืืื ืืื ืืขื ืืืขื, ื ืึธื ืืืึธืก ืื ืงืึทื ืขืงืืขืจ ืกืืืืืฉืื ืฆื ืึธืคึผืืึธื ืืืืขื ืขื ืืึธืืข.
ืงืึธื ืคืืืืจืึทืืืึธื ืืืึทืฉืคึผืื:
{
"name": "mp-k8s-mongo-connector",
"config": {
"connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
"tasks.max": "1",
"mongodb.hosts": "MainRepSet/mongo:27017",
"mongodb.name": "mongo",
"mongodb.user": "debezium",
"mongodb.password": "dbname",
"database.whitelist": "db_1,db_2",
"transforms": "AddPrefix",
"transforms.AddPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.AddPrefix.regex": "mongo.([a-zA-Z_0-9]*).([a-zA-Z_0-9]*)",
"transforms.AddPrefix.replacement": "data.cdc.mongo_$1"
}
}ืืื ืืืจ ืงืขื ืขื ืืขื, ืขืก ืืขื ืขื ืงืืื ื ืืึทืข ืึธืคึผืฆืืขืก ืืึธ ืงืึทืืคึผืขืจื ืืื ืื ืคืจืืขืจืืืงืข ืืืึทืฉืคึผืื, ืึธืืขืจ ืืืืื ืื ื ืืืขืจ ืคืื ืึธืคึผืฆืืขืก ืคืึทืจืึทื ืืืืึธืจืืืขื ืคึฟืึทืจ ืงืึทื ืขืงืืื ื ืฆื ืื ืืึทืืึทืืืืก ืืื ืืืืขืจ ืคึผืจืขืคืืงืก ืืื ืจืืืืกื.
ืกืขืืืื ืืก transforms ืืึธืก ืืึธื ืืื ืืึธื ืื ืคืืืืขื ืืข: ืืื ืืืขืจืืึทืื ืื ื ืึธืืขื ืคืื ืื ืฆืื ืืขืืข ืคึฟืื ืื ืกืืฉืขืืึท <server_name>.<db_name>.<collection_name> ะฒ data.cdc.mongo_<db_name>.
ืฉืืื ืืึธืืขืจืึทื ืฅ
ืื ืึทืจืืืกืืขืื ืคืื ืฉืืื ืืึธืืขืจืึทื ืฅ ืืื ืืืื ืึทืืืืืืึทืืืืึทืื ืืื ืืื ืืืขืจ ืฆืืื ืืื ืืขืจ ืึทืงืืืข ืืื ืืืฅ - ืกืคึผืขืฆืืขื ืืืขื ืืืจ ืจืขืื ืืืขืื ืืึทืื ืืื ืืจืึทื ืืึทืงืฉืึทื ื, ืืื ืืจืึทืงืื ื ืืึทืื ืขื ืืขืจืื ืืขื ืืื ื ืืฉื ืืึทืืื ืืขืจ ืืื ืืขื ืึทืจืืืกืืขืื. ืืึธืืืจ ืงืืงื ืืื ืืืึธืก ืืื ืคึผืจืื ืฆืืคึผ ืงืขื ืขื ืืืื ืคืึทืืฉ ืืื ืืืึธืก ืืืขื ืคึผืึทืกืืจื ืืื ืืขืืขืืืื ืืื ืืขืืขืจ ืคืึทื.
ืขืก ืืขื ืขื ืืจืื ืึธืคึผื-ืืืืก ืึธืคึผืฆืืขืก:
- ืงืึทืคืงืึท ืงืึธื ื ืขืงื ืืืจืืคืึทื. ืืืื ืงืึธื ื ืขืงื ืืื ืงืึทื ืคืืืืขืจื ืฆื ืึทืจืืขืื ืืื ืคืื ืื ืืขืจืืขืืืืื ืืึธืืข, ืืึธืก ืจืืงืืืืืขืจื ืงืืืคื ืืืขืจืก ืฆื ืฉืืขืื ืื ืืขืืืข group.id. ืืขืจื ืึธื, ืืืื ืืืื ืขืจ ืคืื ืืื ืคืืืื, ืื ืงืึทื ืขืงืืขืจ ืืืขื ืืืื ืจืืกืืึทืจืืื ืืืืฃ ืื ืื ืืขืจ ืึทืจืืขืืขืจ ืืื ืคืึธืจืืขืฆื ืืืืขื ืขื ืคืื ืื ืืขืฆืืข ืงืึทืืืืึทื ืฉืืขืืข ืืื ืืขืจ ืืขืืข ืืื ืงืึทืคืงืึท.
- ืึธื ืืืขืจ ืคืื ืงืึทื ืขืงืืืืืืื ืืื ืื ืงืึทืคืงืึท ืงื ืืื. ืืขืจ ืงืึทื ืขืงืืขืจ ืืืขื ืคืฉืื ืืึทืืื ืืืืขื ืขื ืืื ืืขืจ ืฉืืขืืข ืืืึธืก ืืื ื ืื ืึทื ืืขืจืฉ ืฆื ืฉืืงื ืฆื ืงืึทืคืงืึท, ืืื ืืืขื ืคึผืืจืืึทืืืงืื ืคึผืจืืืืจื ืฆื ืฉืืงื ืขืก ืืืืืขืจ ืืื ืื ืคึผืจืืืื ืืื ืืขืจืึธืื.
- ืึทื ืึทืืืืืืึทืืืืืื ืคืื ืืึทืื ืืงืืจ. ืืขืจ ืงืึทื ืขืงืืขืจ ืืืขื ืคึผืจืืืื ืฆื ืจืืงืึทื ืขืงื ืฆื ืื ืืงืืจ ืืื ืงืึทื ืคืืืืขืจื. ืื ืคืขืืืงืืึทื ืืื 16 ืคืจืืืื ื ืืฆื . ื ืึธื ืื 16 ื ืื ืืขืจืึธืื ืคึผืจืืืื, ืื ืึทืจืืขื ืืืขื ืืืื ืื ืืขืฆืืืื ื ืืื ื ืื ืึทื ืืขืจืฉ ืืื ืืืจ ืืืขื ืืึทืจืคึฟื ืฆื ืืึทื ืืืึทืื ืจืืกืืึทืจื ืขืก ืืืจื ืื Kafka Connect REST ืฆืืืื ื.
- ืืื ืืขื ืคืึทื ืคึผืึธืกืืืจืขืกืงื ืื ืืึทืื ืืืขื ื ืื ืืืื ืคืึทืจืคืึทืื, ืืืืึทื ื ืืฆื ืจืขืคึผืืึทืงืืืฉืึทื ืกืืึธืฅ ืืืขื ืคืึทืจืืืึทืื ืืืจ ืฆื ืืืกืืขืงื WAL ืืขืงืขืก ืืืึธืก ืืขื ืขื ื ืืฉื ืืืืขื ืขื ืืืจื ืื ืงืึทื ืขืงืืขืจ. ืืื ืืขื ืคืึทื, ืขืก ืืื ืืืื ืึท ืืึทืื ืกืืื ืฆื ืื ืืึทืืืืืข: ืืืื ืื ื ืขืฅ ืงืึทื ืขืงืืืืืืื ืฆืืืืฉื ืื ืงืึทื ืขืงืืขืจ ืืื ืื DBMS ืืื ืืืกืจืึทืคึผืืื ืคึฟืึทืจ ืึท ืืึทื ื ืฆืืึทื, ืขืก ืืื ืึท ืืขืืืขืืงืืื ืึทื ืื ืืืกืง ืคึผืืึทืฅ ืืืขื ืืืืคื ืืืืก, ืืื ืืึธืก ืงืขื ืคืืจื ืฆื ืึท ืืืจืืคืึทื ืคืื ืื ืืึทืืืืืข. ืื ืืื ืฆืข DBMS.
- ืืื ืืขื ืคืึทื ืืืกืงื binlog ืืขืงืขืก ืงืขื ืขื ืืืื ืจืึธืืืืืืื ืืืจื ืื DBMS ืืื ืืืืืขืจ ืงืึทื ืขืงืืืืืืื ืืื ืืขืืื ื. ืืึธืก ืืืขื ืคืึทืจืฉืึทืคื ืื ืงืึทื ืขืงืืขืจ ืืื ืื ื ืื ืึทื ืืขืจืฉ ืฉืืึทื, ืืื ืฆื ืืืงืขืจื ื ืึธืจืืึทื ืึธืคึผืขืจืึทืฆืืข, ืืืจ ืืึทืจืคึฟื ืฆื ืจืืกืืึทืจื ืืื ืขืจืฉื ืืึธืืขื ืืืืื ืืึธืืข ืฆื ืคืึธืจืืขืฆื ืืืืขื ืขื ืคึฟืื ืืื ืืึธืืก.
- ืืืืฃ ืืึธื ืืึธืื. ืื ืืึทืงืืืืขื ืืืืฉืึทื ืืืื: ืื ื ืึทืืืจ ืคืื ืื ืงืึทื ืขืงืืขืจ ืืื ืื ืืขืฉืขืขื ืืฉ ืึทื ืงืืึธืฅ / ืึธืคึผืืึธื ืืขืงืขืก ืืขื ืขื ืืืืกืืขืืขืงื ืืื ืืขืจ ืงืึทื ืขืงืืขืจ ืงืขื ื ืืฉื ืคืึธืจืืขืฆื ืืืืขื ืขื ืคึฟืื ืื ืฉืืขืืข ืืื ืขืก ืืื ืืื ืงืก ืืื ืื ืืขืืืข ืคึฟืึทืจ ืึทืืข ืืืืก. ืขืก ืืืื ืึทื ืื ืงืึทื ืขืงืืขืจ ืืืขื ืืืื ืืื ืื ืฉืืึทื ื ืื ืึทื ืืขืจืฉ ืืื ืืืขื ืืึทืจืคื ืจืืกืืึทืจืืื ื ืืื ืืึธืืข ืขืจืฉื ืืึธืืขื ืืืืื.
ืึธืืขืจ, ืขืก ืืขื ืขื ืืืืกื ืขืืขื. ืืืื ืื ืงืึทื ืขืงืืขืจ ืืื ืืขืืืขื ืืืกืงืึทื ืขืงืืื ืคึฟืึทืจ ืึท ืืึทื ื ืฆืืื (ืึธืืขืจ ืงืขื ื ืืฉื ืืขืจืืจืืืื ืืขื MongoDB ืืืึทืฉืคึผืื), ืืื ืื ืึธืคึผืืึธื ืืื ืืืจืืืขืืื ืืขื ืจืึธืืืืืฉืึทื ืืขืฉืึทืก ืืขื ืฆืืื, ืืืขื ืื ืงืฉืจ ืืื ืืขืืื ื, ืื ืงืึทื ืขืงืืขืจ ืืืขื ืจืืืง ืคืึธืจืืขืฆื ืฆื ืืืืขื ืขื ืืึทืื ืคึฟืื ืืขืจ ืขืจืฉืืขืจ ืคืึทืจืึทื ืขื ืฉืืขืืข, ืืืึธืก ืืื ืืืึธืก ืขืืืขืืข ืคืื โโืื ืืึทืื ืืื Kafka ืงืืื ืืืขื ืฉืืึธืื.
ืกืึธืฃ
ืืขืืขืืืื ืืื ืืืื ืขืจืฉืืขืจ ืืขืจืคืึทืจืื ื ืืื CDC ืกืืกืืขืืขื ืืื ืงืืืืขืืืืง ืืืืขืจ positive. ืื ืคึผืจืืืขืงื ืืืึทื ืืืืขืจ ืืื ืืืื ืฉืืืฆื ืคึฟืึทืจ ืืืืคึผื DBMSs, ืื ืคืื ืงืึทื ืคืืืืขืจืืืฉืึทื, ืงืืึทืกืืขืจืื ื ืฉืืืฆื ืืื ืึทืงืืืื ืงืื. ืคึฟืึทืจ ืืขื ืข ืืื ืืขืจืขืกืืจื ืืื ืคืืจ, ืืื ืจืขืงืึธืืขื ืืืจื ืืืจ ืืืืขื ืขื ืื ืคืืจืขืจ ืคึฟืึทืจ ะธ .
ืงืึทืืคึผืขืจื ืืื ืื JDBC ืงืึทื ืขืงืืขืจ ืคึฟืึทืจ Kafka Connect, ืื ืืืืคึผื ืืืึทืืข ืคืื โโDebezium ืืื ืึทื ืขื ืืขืจืื ืืขื ืืขื ืขื ืืืืขื ืขื ืคึฟืื ืื DBMS ืืึธืืก, ืืืึธืก ืึทืืึทืื ืืึทืื ืฆื ืืืื ืืืงืืืขื ืืื ืืื ืืืึทื ืืืืืึทื ืกื. ืื JDBC ืงืึทื ืขืงืืขืจ (ืคึฟืื Kafka Connect) ืคึฟืจืืื ืื ืืึธื ืืืึธืจืขื ืืืฉ ืืื ืึท ืคืึทืจืคืขืกืืืงื ืืขืืึทืืขื ืืื (ืคึฟืึทืจ ืืขืจ ืืขืืืืงืขืจ ืกืืื) ืืืฉืขื ืขืจืืื ื ืืฉื ืึทืจืืืงืืขื ืืืขื ืืึทืื ืืขื ืขื ืืืืกืืขืืขืงื (ืืื ืงืขื ืขื ืืืจ ืึธื ืคืจืขื ืืึทืื ืืืึธืก ืืึธื ื ืื ืขืงืกืืกืืืจื?).
ืฆื ืกืึธืืืืข ืขื ืืขื ืคึผืจืึธืืืขืืก, ืืืจ ืงืขื ืขื ืืึทืฆืึธืื ืืคืืขืจืงืืึทืืงืืึทื ืฆื ืื ืคืืืืขื ืืข ืกืึทืืืฉืึทื ื (ืืื ืืขืจืฆื ืฆื ืืขืืขืืืื):
- ืขืืืขืืข ืกืึทืืืฉืึทื ื ืืืืื ืคึฟืึทืจ MySQL:
- , ืึธืืขืจ ืืึธืก ืืื ืึท ืืึธืจ ืึทื ืืขืจืฉ "ืืืึธื ืงืึทืืขืืึธืจืืข".
ืคึผืก
ืืืืขื ืขื ืืืื ืืืืฃ ืืื ืืืขืจ ืืืึธื:
- ยซ";
- ยซ";
- ยซ'.
ืืงืืจ: www.habr.com