ããã«ã¡ã¯ãããã«ïŒ
ãã€ãŠãç§ãã¡ã¯ãã®ãããã¯ããã·ã¢åžå Žã«åããŠçŽ¹ä»ããŸããã
å°å ¥
Kubernetes ã¯ã¹ããŒãã¬ã¹ãªã¯ãŒã¯ããŒããåŠçããããã«èšèšãããŠããŸãã éåžžããã®ãããªã¯ãŒã¯ããŒãã¯ãã€ã¯ããµãŒãã¹ ã¢ãŒããã¯ãã£ã®åœ¢åŒã§æ瀺ããã軜éã§æ°Žå¹³æ¹åã«æ¡åŒµæ§ãé«ãã12 èŠçŽ ã¢ããªã±ãŒã·ã§ã³ã®ååã«åŸãããµãŒããã ãã¬ãŒã«ãŒãã«ãªã¹ ã¢ã³ããŒãšé£æºã§ããŸãã
äžæ¹ãKafka ã¯åºæ¬çã«åæ£ããŒã¿ããŒã¹ãšããŠæ©èœããŸãã ãããã£ãŠãåäœãããšãã«ç¶æ ãåŠçããå¿ èŠãããããã€ã¯ããµãŒãã¹ãããã¯ããã«éãã§ãã Kubernetes ã¯ã¹ããŒããã« ããŒãããµããŒãããŠããŸãããKelsey Hightower ã XNUMX ã€ã®ãã€ãŒãã§ææããŠããããã«ãã¹ããŒããã« ããŒãã¯æ³šæããŠæ±ãå¿ èŠããããŸãã
Kubernetes ãã¹ããŒããã« ã¯ãŒã¯ããŒãã«çµã¿èŸŒããšãRDS ã«å¹æµãããã«ãããŒãžã ããŒã¿ããŒã¹ã«ãªããšèãã人ãããŸãã ããã¯ééã£ãŠããŸãã ãããããååã«åªåããŠã³ã³ããŒãã³ããè¿œå ããSRE ãšã³ãžãã¢ã®ããŒã ãåŒãä»ããã°ãKubernetes äžã« RDS ãæ§ç¯ã§ããã§ãããã
ç§ã¯åžžã«ãKubernetes ã§ã¹ããŒããã« ã¯ãŒã¯ããŒããå®è¡ãããšãã¯çŽ°å¿ã®æ³šæãæãããšããå§ãããŸãã ãKubernetes ã§ã¹ããŒããã« ã¯ãŒã¯ããŒããå®è¡ã§ããŸãã?ããšå°ããã»ãšãã©ã®äººã¯ãKubernetes ã«ã€ããŠååãªçµéšããªããå€ãã®å Žåã質åããŠããã¯ãŒã¯ããŒãã«ã€ããŠãååãªçµéšããããŸããã
ã§ã¯ãKubernetes äžã§ Kafka ãå®è¡ããå¿ èŠãããã§ãããã? é質å: Kafka 㯠Kubernetes ãªãã§ããé©åã«åäœããŸãã? ãã®ããããã®èšäºã§ã¯ãKafka ãš Kubernetes ãã©ã®ããã«çžäºè£å®ããã®ãããŸããããããçµã¿åãããããšã§ã©ã®ãããªèœãšãç©Žãçããå¯èœæ§ãããã®ãââã匷調ããããšæããŸãã
å®æææ
åºæ¬çãªããšãã€ãŸãã©ã³ã¿ã€ã ç°å¢èªäœã«ã€ããŠè©±ããŸããã
ããã»ã¹
Kafka ãããŒã«ãŒã¯ CPU ã«åªããã§ãã TLS ã§ã¯ãªãŒããŒããããçºçããå¯èœæ§ããããŸãã ãã ããKafka ã¯ã©ã€ã¢ã³ãã¯æå·åã䜿çšãããš CPU ã®äœ¿çšéãå¢ããå¯èœæ§ããããŸãããããã¯ãããŒã«ãŒã«ã¯åœ±é¿ããŸããã
ã¡ã¢ãª
Kafka ãããŒã«ãŒã¯ã¡ã¢ãªãæ¶è²»ããŸãã JVM ããŒã ãµã€ãºã¯éåžž 4 ïœ 5 GB ã«å¶éãããŠããŸãããKafka ã¯ããŒãž ãã£ãã·ã¥ãéåžžã«é »ç¹ã«äœ¿çšããããã倧éã®ã·ã¹ãã ã¡ã¢ãªãå¿ èŠã«ãªããŸãã Kubernetes ã§ã¯ãã³ã³ãããŒã®ãªãœãŒã¹ãšãªã¯ãšã¹ãã®å¶éãããã«å¿ããŠèšå®ããŸãã
ããŒã¿ã¹ãã¢
ã³ã³ããå
ã®ããŒã¿ ã¹ãã¬ãŒãžã¯äžæçãªãã®ã§ãããåèµ·åãããšããŒã¿ã倱ãããŸãã Kafka ããŒã¿ã®å Žåã¯ããªã¥ãŒã ã䜿çšã§ããŸã emptyDir
ãšåæ§ã®å¹æãåŸãããŸããå®äºåŸã«ãããŒã«ãŒã®ããŒã¿ã¯å€±ãããŸãã ã¡ãã»ãŒãžã¯åŒãç¶ãä»ã®ãããŒã«ãŒã«ã¬ããªã«ãšããŠä¿åã§ããŸãã ãããã£ãŠãåèµ·ååŸãé害ãçºçãããããŒã«ãŒã¯ãŸããã¹ãŠã®ããŒã¿ãã¬ããªã±ãŒãããå¿
èŠãããããã®ããã»ã¹ã«ã¯æéããããããšããããŸãã
ãã®ãããé·æããŒã¿ ã¹ãã¬ãŒãžã䜿çšããå¿
èŠããããŸãã XFS ãã¡ã€ã« ã·ã¹ãã ãããæ£ç¢ºã«ã¯ ext4 ã䜿çšããéããŒã«ã«ã®é·æã¹ãã¬ãŒãžã«ããŸãã NFS ã¯äœ¿çšããªãã§ãã ããã ç§ã¯ããªãã«èŠåããŸããã NFS ããŒãžã§ã³ v3 ãŸã㯠v4 ã¯æ©èœããŸããã ã€ãŸããNFS ã®ãæããªååå€æŽãåé¡ã«ããããŒã¿ ãã£ã¬ã¯ããªãåé€ã§ããªãå ŽåãKafka ãããŒã«ãŒã¯ã¯ã©ãã·ã¥ããŸãã ãŸã ããªããçŽåŸãããŠããªãå Žåã¯ãæ
éã«
СеÑÑ
ã»ãšãã©ã®åæ£ã·ã¹ãã ãšåæ§ã«ãKafka ã®ããã©ãŒãã³ã¹ã¯ããããã¯ãŒã¯é 延ãæå°éã«æãã垯åå¹ ãæ倧éã«ä¿ã€ããšã«å€§ããäŸåããŸãã å¯çšæ§ãäœäžãããããåãããŒãäžã§ãã¹ãŠã®ãããŒã«ãŒããã¹ãããããšããªãã§ãã ããã Kubernetes ããŒãã«é害ãçºçãããšãKafka ã¯ã©ã¹ã¿ãŒå šäœã«é害ãçºçããŸãã ãŸããKafka ã¯ã©ã¹ã¿ãŒãããŒã¿ã»ã³ã¿ãŒå šäœã«åæ£ãããªãã§ãã ããã Kubernetes ã¯ã©ã¹ã¿ãŒã«ã€ããŠãåæ§ã§ãã ãã®å Žåã®é©åãªåŠ¥åçã¯ãå¥ã®ã¢ãã€ã©ããªã㣠ãŸãŒã³ãéžæããããšã§ãã
èšå®
å®æãããã§ã¹ã
Kubernetes Web ãµã€ãã«ã¯ã
- äžã«: ãããã¯ãKubernetes ã§ãããã€å¯èœãªæå°åäœã§ãã ãããã«ã¯ã¯ãŒã¯ããŒããå«ãŸããŠããããããèªäœã¯ã¯ã©ã¹ã¿ãŒå ã®ããã»ã¹ã«å¯Ÿå¿ããŸãã ãããã«ã¯ XNUMX ã€ä»¥äžã®ã³ã³ãããå«ãŸããŸãã ã¢ã³ãµã³ãã«å ã®å ZooKeeper ãµãŒããŒãš Kafka ã¯ã©ã¹ã¿ãŒå ã®åãããŒã«ãŒã¯ãå¥ã®ãããã§å®è¡ãããŸãã
- ã¹ããŒããã«ã»ãã: StatefulSet ã¯ãè€æ°ã®ã¹ããŒããã« ã¯ãŒã¯ããŒããåŠçãã Kubernetes ãªããžã§ã¯ãã§ããããã®ãããªã¯ãŒã¯ããŒãã«ã¯èª¿æŽãå¿ èŠã§ãã StatefulSet ã¯ããããã®é åºãšãã®äžææ§ã«é¢ããä¿èšŒãæäŸããŸãã
- ãããã¬ã¹ãµãŒãã¹: ãµãŒãã¹ã䜿çšãããšãè«çåã䜿çšããŠã¯ã©ã€ã¢ã³ãããããããåãé¢ãããšãã§ããŸãã ãã®å ŽåãKubernetes ã¯è² è·åæ£ãæ åœããŸãã ãã ããZooKeeper ã Kafka ãªã©ã®ã¹ããŒããã« ã¯ãŒã¯ããŒããæäœããå Žåãã¯ã©ã€ã¢ã³ãã¯ç¹å®ã®ã€ã³ã¹ã¿ã³ã¹ãšéä¿¡ããå¿ èŠããããŸãã ããã§ãããã¬ã¹ ãµãŒãã¹ã圹ã«ç«ã¡ãŸãããã®å Žåãã¯ã©ã€ã¢ã³ãã«ã¯è«çåãæ®ããŸããããããã«çŽæ¥æ¥ç¶ããå¿ èŠã¯ãããŸããã
- é·æä¿ç®¡é: ãããã®ããªã¥ãŒã ã¯ãåè¿°ã®éããŒã«ã« ãããã¯æ°žç¶ã¹ãã¬ãŒãžãæ§æããããã«å¿ èŠã§ãã
Ðа
ãã«ã ãã£ãŒã
Helm 㯠Kubernetes ã®ããã±ãŒãž ãããŒãžã£ãŒã§ãããyumãaptãHomebrewãChocolatey ãªã©ã® OS ããã±ãŒãž ãããŒãžã£ãŒãšæ¯èŒã§ããŸãã ããã«ãããHelm ãã£ãŒãã«èšè¿°ãããäºåå®çŸ©ããããœãããŠã§ã¢ ããã±ãŒãžãç°¡åã«ã€ã³ã¹ããŒã«ã§ããŸãã é©åã«éžæããã Helm ãã£ãŒãã«ãããKubernetes 㧠Kafka ã䜿çšããããã«ãã¹ãŠã®ãã©ã¡ãŒã¿ãŒãé©åã«æ§æããæ¹æ³ãšããé£ããã¿ã¹ã¯ãç°¡åã«ãªããŸãã Kafka ã®å³ã¯ããã€ããããŸããå
¬åŒã®ãã®ã¯æ¬¡ã®å Žæã«ãããŸãã
æŒç®å
Helm ã«ã¯ç¹å®ã®æ¬ ç¹ããããããå¥ã®ããŒã«ã§ãã Kubernetes ãªãã¬ãŒã¿ãŒãããªãã®äººæ°ãéããŠããŸãã ãªãã¬ãŒã¿ãŒã¯ãKubernetes çšã®ãœãããŠã§ã¢ãããã±ãŒãžåããã ãã§ãªãããã®ãããªãœãããŠã§ã¢ã®ãããã€ãšç®¡çãå¯èœã«ããŸãã
ÐÑпОÑке
ÐÑПОзвПЎОÑелÑМПÑÑÑ
Kafka ã€ã³ã¹ã¿ã³ã¹ã®ãã³ãããŒã¯ãè¡ã£ãŠããã©ãŒãã³ã¹ããã¹ãããããšãéèŠã§ãã ãã®ãããªãã¹ãã¯ãåé¡ãçºçããåã«æœåšçãªããã«ããã¯ãèŠã€ããã®ã«åœ¹ç«ã¡ãŸãã 幞ããªããšã«ãKafka ã¯ãã§ã« XNUMX ã€ã®ããã©ãŒãã³ã¹ ãã¹ã ããŒã«ãæäŸããŠããŸãã kafka-producer-perf-test.sh
О kafka-consumer-perf-test.sh
ã ç©æ¥µçã«æŽ»çšããŠãã ããã åèãŸã§ã«ãã«èšèŒãããŠããçµæãåç
§ããŠãã ããã
ãªãã¬ãŒã·ã§ã³
ç£èŠ
ã·ã¹ãã ã®éææ§ã¯éåžžã«éèŠã§ããããã§ãªããšãã·ã¹ãã å ã§äœãèµ·ãã£ãŠããã®ãç解ã§ããªããªããŸãã çŸåšãã¯ã©ãŠã ãã€ãã£ã ã¹ã¿ã€ã«ã§ã¡ããªã¯ã¹ããŒã¹ã®ç£èŠãæäŸããå ç¢ãªããŒã«ããããååšããŸãã ãã®ç®çã§ãã䜿çšããã XNUMX ã€ã®ããŒã«ã¯ãPrometheus ãš Grafana ã§ãã Prometheus ã¯ãJMX ãšã¯ã¹ããŒã¿ãŒã䜿çšããŠããã¹ãŠã® Java ããã»ã¹ (KafkaãZookeeperãKafka Connect) ããæãç°¡åãªæ¹æ³ã§ã¡ããªã¯ã¹ãåéã§ããŸãã cAdvisor ã¡ããªã¯ã¹ãè¿œå ãããšãKubernetes ã§ãªãœãŒã¹ãã©ã®ããã«äœ¿çšãããããããå®å šã«ç解ã§ããŸãã
Strimzi ã«ã¯ãKafka çšã® Grafana ããã·ã¥ããŒãã®éåžžã«äŸ¿å©ãªäŸããããŸãã ããšãã°ãè€è£œãäžååãªã»ã¯ã¿ãŒããªãã©ã€ã³ã®ã»ã¯ã¿ãŒã«é¢ããäž»èŠãªã¡ããªã¯ã¹ãèŠèŠåããŸãã ããã§ã¯ãã¹ãŠãéåžžã«æ確ã§ãã ãããã®ã¡ããªã¯ã¹ã¯ããªãœãŒã¹äœ¿çšçãšããã©ãŒãã³ã¹æ å ±ãããã³å®å®æ§ææšã«ãã£ãŠè£å®ãããŸãã ãããã£ãŠãåºæ¬ç㪠Kafka ã¯ã©ã¹ã¿ãŒç£èŠãç¡æã§å©çšã§ããŸãã
åºæïŒ
ããããã¹ãŠããã¯ã©ã€ã¢ã³ãç£èŠ (ã³ã³ã·ã¥ãŒããšãããã¥ãŒãµãŒã«é¢ããã¡ããªã¯ã¹) ããã³ã¬ã€ãã³ã·ç£èŠ (ããã«ã€ããŠã¯ã
ãã®ã³ã°
ãã®ã³ã°ãéèŠãªã¿ã¹ã¯ã§ãã Kafka ã€ã³ã¹ããŒã«å
ã®ãã¹ãŠã®ã³ã³ããããã°ã«èšé²ãããŠããããšã確èªããŠãã ãã stdout
О stderr
ãŸããKubernetes ã¯ã©ã¹ã¿ãŒããã¹ãŠã®ãã°ãäžå€®ã®ãã° ã€ã³ãã©ã¹ãã©ã¯ãã£ã«éçŽããŠããããšã確èªããŸãã
æ©èœãã¹ã
Kubernetes ã¯ãliveness ãããŒããš readiness ãããŒãã䜿çšããŠãããããæ£åžžã«å®è¡ãããŠãããã©ããã確èªããŸãã 掻æ§ãã§ãã¯ã倱æããå ŽåãKubernetes ã¯ãã®ã³ã³ãããåæ¢ããåèµ·åããªã·ãŒãé©åã«èšå®ãããŠããã°èªåçã«åèµ·åããŸãã æºåç¶æ³ãã§ãã¯ã倱æããå ŽåãKubernetes ã¯ãããããªã¯ãšã¹ãã®åŠçããéé¢ããŸãã ãããã£ãŠããã®ãããªå Žåãæåã«ããä»å ¥ã¯ãŸã£ããå¿ èŠãªããªããããã¯å€§ããªå©ç¹ãšãªããŸãã
ã¢ããããŒãã®ããŒã«ã¢ãŠã
StatefulSet ã¯èªåæŽæ°ããµããŒãããŠããŸããRollingUpdate æŠç¥ãéžæãããšãKafka ã®äžã®ãããããé çªã«æŽæ°ãããŸãã ãã®ããã«ããŠãããŠã³ã¿ã€ã ããŒãã«æžããããšãã§ããŸãã
ã¹ã±ãŒãªã³ã°
Kafka ã¯ã©ã¹ã¿ãŒã®ã¹ã±ãŒãªã³ã°ã¯ç°¡åãªäœæ¥ã§ã¯ãããŸããã ãã ããKubernetes ã䜿çšãããšãããããç¹å®ã®æ°ã®ã¬ããªã«ã«æ¡åŒµããããšãéåžžã«ç°¡åã«ãªããŸããã€ãŸããå¿ èŠãªæ°ã® Kafka ãããŒã«ãŒã宣èšçã«å®çŸ©ã§ããããšã«ãªããŸãã ãã®å Žåãæãé£ããã®ã¯ãã¹ã±ãŒã«ã¢ããåŸãŸãã¯ã¹ã±ãŒã«ããŠã³åã«ã»ã¯ã¿ãŒãåå²ãåœãŠããããšã§ãã ç¹°ãè¿ãã«ãªããŸãããKubernetes ããã®ã¿ã¹ã¯ãæ¯æŽããŸãã
è¡æ¿
ãããã¯ã®äœæãã»ã¯ã¿ãŒã®åå²ãåœãŠãªã©ãKafka ã¯ã©ã¹ã¿ãŒã®ç®¡çã«é¢é£ããã¿ã¹ã¯ã¯ããããã§ã³ãã³ã ã©ã€ã³ ã€ã³ã¿ãŒãã§ã€ã¹ãéããŠæ¢åã®ã·ã§ã« ã¹ã¯ãªããã䜿çšããŠå®è¡ã§ããŸãã ãã ãããã®è§£æ±ºçã¯ããŸãçŸãããããŸããã Strimzi ã¯ãå¥ã®æŒç®åã䜿çšãããããã¯ã®ç®¡çããµããŒãããŠããŸãã ããã«ã¯æ¹åã®äœå°ããããŸãã
ããã¯ã¢ãããšãªã«ããª
çŸåšãKafka ã®å¯çšæ§ã¯ Kubernetes ã®å¯çšæ§ã«ãäŸåããŸãã Kubernetes ã¯ã©ã¹ã¿ãŒã倱æãããšãææªã®å ŽåãKafka ã¯ã©ã¹ã¿ãŒã倱æããŸãã ããŒãã£ãŒã®æ³åã«ããã°ãããã¯å¿
ãèµ·ãããããŒã¿ã¯å€±ãããŸãã ãã®çš®ã®ãªã¹ã¯ã軜æžããã«ã¯ãé©åãªããã¯ã¢ããã®æŠå¿µãçšæããŠãã ããã MirrorMaker ã䜿çšããããšãã§ããŸãããå¥ã®ãªãã·ã§ã³ãšããŠããã®ããã« S3 ã䜿çšããããšãã§ããŸããããã«ã€ããŠã¯ããã®èšäºã§èª¬æããŠããŸãã
ãŸãšã
äžå°èŠæš¡ã® Kafka ã¯ã©ã¹ã¿ãŒãæäœããå Žåãæè»æ§ãåäžãããªãã¬ãŒã¿ãŒã®ãšã¯ã¹ããªãšã³ã¹ãç°¡çŽ åããããããKubernetes ã䜿çšãã䟡å€ã¯ééããªããããŸãã æ©èœä»¥å€ã®é
延ãã¹ã«ãŒãããã®èŠä»¶ãéåžžã«å€§ããå Žåã¯ãä»ã®å±éãªãã·ã§ã³ãæ€èšããæ¹ãããå ŽåããããŸãã
åºæïŒ habr.com