å°ããªæ¬ã®ç¿»èš³ã®ç¶ã:
ã¡ãã»ãŒãžãããŒã«ãŒãç解ãã
èè
ïŒJakub Korabãåºç瀟ïŒO'Reilly Media, Inc.ãçºè¡æ¥ïŒ2017 幎 9781492049296 æãISBNïŒXNUMXã
以åã«ç¿»èš³ãããéšå:
CHAPTER 3
ã«ãã«
Kafka ã¯ãåŸæ¥ã®ã¡ãã»ãŒãž ãããŒã«ãŒã®å¶éã®äžéšãåé¿ããããŸããŸãªãã€ã³ãããŒãã€ã³ã ã€ã³ã¿ã©ã¯ã·ã§ã³çšã«è€æ°ã®ã¡ãã»ãŒãž ãããŒã«ãŒãã»ããã¢ããããå¿ èŠãé¿ããããã« LinkedIn ã«ãã£ãŠéçºãããŸãããããã«ã€ããŠã¯ãæ¬æžã® 28 ããŒãžã®ãã¹ã±ãŒã«ã¢ãããšã¹ã±ãŒã«ã¢ãŠããã§èª¬æãããŠããŸãããŠãŒã¹ã±ãŒã¹ LinkedIn ã¯ãããŒãžã®ã¯ãªãã¯ãã¢ã¯ã»ã¹ ãã°ãªã©ã®éåžžã«å€§éã®ããŒã¿ã®äžæ¹åã®åã蟌ã¿ã«äž»ã«äŸåããªãããããããã¥ãŒãµãŒãä»ã®æ¶è²»è ã®çç£æ§ã«åœ±é¿ãäžããããšãªãããã®ããŒã¿ãè€æ°ã®ã·ã¹ãã ã§äœ¿çšã§ããããã«ããŠããŸãã å®éãKafka ãååšããçç±ã¯ãUniversal Data Pipeline ã§èª¬æãããŠããçš®é¡ã®ã¡ãã»ãŒãžã³ã° ã¢ãŒããã¯ãã£ãååŸããããã§ãã
ãã®æçµç®æšãèãããšãåœç¶ä»ã®èŠä»¶ãçããŸãã ã«ãã«ã¯æ¬¡ã®ããšãè¡ãå¿ èŠããããŸãã
- 極ããŠéãè¡åãã
- ã¡ãã»ãŒãžãåŠçãããšãã«ããå€ãã®åž¯åå¹ ãæäŸããŸã
- ãããªãã·ã£ãŒ/ãµãã¹ã¯ã©ã€ããŒããã³ãã€ã³ãããŒãã€ã³ã ã¢ãã«ã®ãµããŒã
- æ¶è²»è ã®è¿œå ãé ãããªãã§ãã ããã ããšãã°ãå®å ã®ã³ã³ã·ã¥ãŒãã®æ°ãå¢ãããšãActiveMQ ã®ãã¥ãŒãšãããã¯ã®äž¡æ¹ã®ããã©ãŒãã³ã¹ãäœäžããŸãã
- æ°Žå¹³æ¹åã«ã¹ã±ãŒã©ãã«ã§ããããšã ã¡ãã»ãŒãžãä¿æãã XNUMX ã€ã®ãããŒã«ãŒãæ倧ãã£ã¹ã¯é床ã§ã®ã¿ã¡ãã»ãŒãžãä¿æã§ããå Žåãããã©ãŒãã³ã¹ãåäžãããããã«åäžã®ãããŒã«ãŒ ã€ã³ã¹ã¿ã³ã¹ãè¶ ããŠã¡ãã»ãŒãžãä¿æããããšã¯çã«ããªã£ãŠããŸãã
- ã¡ãã»ãŒãžã®ä¿åãšåååŸãžã®ã¢ã¯ã»ã¹ãå¶éãã
ããããã¹ãŠãéæããããã«ãKafka ã¯ã¯ã©ã€ã¢ã³ããšã¡ãã»ãŒãžã³ã° ãããŒã«ãŒã®åœ¹å²ãšè²¬ä»»ãåå®çŸ©ããã¢ãŒããã¯ãã£ãæ¡çšããŸããã JMS ã¢ãã«ã¯éåžžã«ãããŒã«ãŒæåã§ããããããŒã«ãŒãã¡ãã»ãŒãžã®é
åžãæ
åœããã¯ã©ã€ã¢ã³ãã¯ã¡ãã»ãŒãžã®éåä¿¡ã®ã¿ãæ°ã«ããå¿
èŠããããŸãã äžæ¹ãKafka ã¯ã¯ã©ã€ã¢ã³ãäžå¿ã§ãããã¯ã©ã€ã¢ã³ãã¯ãéåžžã«é«éã§ã¹ã±ãŒã©ãã«ãªãããŒã«ãŒãšåŒãæãã«ãæ¶è²»è
ãžã®é¢é£ã¡ãã»ãŒãžã®å
¬å¹³ãªé
ä¿¡ãªã©ãåŸæ¥ã®ãããŒã«ãŒã®æ©èœã®å€ããåŒãåããŸãã åŸæ¥ã®ã¡ãã»ãŒãžã³ã° ã·ã¹ãã ã䜿çšããŠãã人ã«ãšã£ãŠãKafka ã䜿çšããã«ã¯æ ¹æ¬çãªèãæ¹ã®å€æŽãå¿
èŠã§ãã
ãã®ãšã³ãžãã¢ãªã³ã°ã®æ¹åæ§ã«ãããåŸæ¥ã®ãããŒã«ãŒãšæ¯èŒããŠã¹ã«ãŒããããäœæ¡ãåäžã§ããã¡ãã»ãŒãžã³ã° ã€ã³ãã©ã¹ãã©ã¯ãã£ã®äœæãå¯èœã«ãªããŸããã åŸã§èª¬æããããã«ããã®ã¢ãããŒãã«ã¯ãã¬ãŒããªãã䌎ããŸããã€ãŸããKafka ã¯ç¹å®ã®çš®é¡ã®ã¯ãŒã¯ããŒããã€ã³ã¹ããŒã«ãããŠãããœãããŠã§ã¢ã«ã¯é©ããŠããŸããã
çµ±åå®å ã¢ãã«
äžèšã®èŠä»¶ãæºããããã«ãKafka ã¯ãXNUMX çš®é¡ã®å®å ã®äžã§ãããªãã·ã¥/ãµãã¹ã¯ã©ã€ããšãã€ã³ãããŒãã€ã³ã ã¡ãã»ãŒãžã³ã°ãçµã¿åãããŸãã- ãããã¯ã ããã¯ãã¡ãã»ãŒãžã³ã° ã·ã¹ãã ã䜿çšããããšããã人ã ã«ãšã£ãŠã¯æ··ä¹±ãæããŸããããããã¯ããšããèšèã¯ã(ãããã¯ããã®) èªã¿åããæ°žç¶çã§ã¯ãªããããŒããã£ã¹ã ã¡ã«ããºã ãæããŸãã æ¬æžã®åºæã§å®çŸ©ãããŠããããã«ãKafka ãããã¯ã¯ãã€ããªããå®å ã¿ã€ããšã¿ãªãããå¿ èŠããããŸãã
ãã®ç« ã®æ®ãã®éšåã§ã¯ãç¹ã«æèšããªãéããããããã¯ããšããçšèªã¯ Kafka ãããã¯ãæããŸãã
ãããã¯ãã©ã®ããã«åäœãããããŸããããã¯ãã©ã®ãããªä¿èšŒãæäŸããããå®å
šã«ç解ããã«ã¯ããŸããããã¯ã Kafka ã§ã©ã®ããã«å®è£
ãããŠãããã確èªããå¿
èŠããããŸãã
Kafka ã®åãããã¯ã«ã¯ç¬èªã®ãã°ããããŸãã
Kafka ã«ã¡ãã»ãŒãžãéä¿¡ãããããã¥ãŒãµãŒã¯ãã®ãã°ã«æžã蟌ã¿ãã³ã³ã·ã¥ãŒããŒã¯åžžã«åé²ãããã€ã³ã¿ãŒã䜿çšããŠãã°ããèªã¿åããŸãã Kafka ã¯ããã®éšåã®ã¡ãã»ãŒãžãèªãŸãããã©ããã«é¢ä¿ãªãããã°ã®æãå€ãéšåãå®æçã«åé€ããŸãã Kafka ã®èšèšã®äžå¿çãªéšåã¯ããããŒã«ãŒã¯ã¡ãã»ãŒãžãèªãŸãããã©ãããæ°ã«ããªããšããããšã§ããããã¯ã¯ã©ã€ã¢ã³ãã®è²¬ä»»ã§ãã
ããã°ããšããã€ã³ã¿ããšããçšèªã¯ã
Kafka ããã¥ã¡ã³ã ã ããã§ã¯ãç解ãå©ããããã«ãããã®ããç¥ãããçšèªã䜿çšããŠããŸãã
ãã®ã¢ãã«ã¯ããã¹ãŠã®ãã¥ãŒããã®ã¡ãã»ãŒãžãåããã°ã«ä¿åããããããŒã«ãŒãã¡ãã»ãŒãžãèªã¿åã£ãåŸã«åé€æžã¿ãšããŠããŒã¯ãã ActiveMQ ãšã¯ãŸã£ããç°ãªããŸãã
ããã§ãããå°ãæ·±ãæãäžããŠãããã㯠ãã°ãããã«è©³ããèŠãŠã¿ãŸãããã
Kafka ãã°ã¯è€æ°ã®ããŒãã£ã·ã§ã³ã§æ§æãããŸã (
å³ 3-1ã Kafka ããŒãã£ã·ã§ã³
ãããã¥ãŒãµã Kafka ãããã¯ã«ã¡ãã»ãŒãžãéä¿¡ãããšããã¡ãã»ãŒãžãã©ã®ããŒãã£ã·ã§ã³ã«éä¿¡ãããã決å®ããŸãã ããã«ã€ããŠã¯åŸã»ã©è©³ããèŠãŠãããŸãã
ã¡ãã»ãŒãžãèªã
ã¡ãã»ãŒãžãèªã¿ããã¯ã©ã€ã¢ã³ãã¯ããšåŒã°ããååä»ããã€ã³ã¿ã管çããŸãã æ¶è²»è å£äœãæããŸãã ãªãã»ãã ããŒãã£ã·ã§ã³å ã®ã¡ãã»ãŒãžã ãªãã»ããã¯ãããŒãã£ã·ã§ã³ã®å é ã® 0 ããå§ãŸãå¢åäœçœ®ã§ãã ãã®ã³ã³ã·ã¥ãŒã ã°ã«ãŒãã¯ãAPI ã§ãŠãŒã¶ãŒå®çŸ©ã® group_id ãä»ããŠåç §ããã以äžã«å¯Ÿå¿ããŸãã XNUMX ã€ã®è«çã³ã³ã·ã¥ãŒããŸãã¯ã·ã¹ãã .
ã»ãšãã©ã®ã¡ãã»ãŒãžã³ã° ã·ã¹ãã ã¯ãè€æ°ã®ã€ã³ã¹ã¿ã³ã¹ãšã¹ã¬ããã䜿çšããŠå®å ããããŒã¿ãèªã¿åããã¡ãã»ãŒãžã䞊ååŠçããŸãã ãããã£ãŠãéåžžã¯ãåãã³ã³ã·ã¥ãŒã ã°ã«ãŒããå ±æããå€ãã®ã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ãååšããŸãã
èªè§£ã®åé¡ã¯æ¬¡ã®ããã«è¡šãããšãã§ããŸãã
- ãããã¯ã«ã¯è€æ°ã®ããŒãã£ã·ã§ã³ããããŸã
- ã³ã³ã·ã¥ãŒãã®è€æ°ã®ã°ã«ãŒããåæã«ãããã¯ã䜿çšã§ãã
- ã³ã³ã·ã¥ãŒãã®ã°ã«ãŒãã¯è€æ°ã®åå¥ã®ã€ã³ã¹ã¿ã³ã¹ãæã€ããšãã§ããŸã
ããã¯ãèªæã§ã¯ãªãå€å¯Ÿå€ã®åé¡ã§ãã Kafka ãã³ã³ã·ã¥ãŒã ã°ã«ãŒããã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ãããŒãã£ã·ã§ã³éã®é¢ä¿ãã©ã®ããã«åŠçããããç解ããããã«ãåŸã ã«è€éã«ãªãäžé£ã®èªã¿åãã·ããªãªãèŠãŠã¿ãŸãããã
æ¶è²»è ããã³æ¶è²»è å£äœ
éå§ç¹ãšããŠãXNUMX ã€ã®ããŒãã£ã·ã§ã³ (
å³ 3-2ã ã³ã³ã·ã¥ãŒããããŒãã£ã·ã§ã³ããèªã¿åã
ã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ãç¬èªã® group_id ã䜿çšããŠãã®ãããã¯ã«æ¥ç¶ãããšãèªã¿åãããŒãã£ã·ã§ã³ãšãã®ããŒãã£ã·ã§ã³å
ã®ãªãã»ãããå²ãåœãŠãããŸãã ãã®ãªãã»ããã®äœçœ®ã¯ãææ°ã®äœçœ® (ææ°ã®ã¡ãã»ãŒãž) ãŸãã¯æãå€ãäœçœ® (æãå€ãã¡ãã»ãŒãž) ãžã®ãã€ã³ã¿ãŒãšããŠã¯ã©ã€ã¢ã³ãã§æ§æãããŸãã ã³ã³ã·ã¥ãŒãã¯ãããã¯ããã¡ãã»ãŒãžããªã¯ãšã¹ã (ããŒãªã³ã°) ããããã«ããã¡ãã»ãŒãžããã°ããé çªã«èªã¿åãããŸãã
ãªãã»ããäœçœ®ã¯å®æçã« Kafka ã«ã³ããããããå
éšãããã¯ã«ã¡ãã»ãŒãžãšããŠä¿åãããŸãã _consumer_offsetsã éåžžã®ãããŒã«ãŒãšã¯ç°ãªããèªã¿åãã¡ãã»ãŒãžã¯åé€ããããã¯ã©ã€ã¢ã³ãã¯ãªãã»ãããå·»ãæ»ããŠããã§ã«è¡šç€ºãããã¡ãã»ãŒãžãååŠçã§ããŸãã
XNUMX çªç®ã®è«çã³ã³ã·ã¥ãŒããå¥ã® group_id ã䜿çšããŠæ¥ç¶ãããšãæåã®ãã€ã³ã¿ããç¬ç«ãã XNUMX çªç®ã®ãã€ã³ã¿ã管çããŸã (
å³3-3ã ç°ãªãã³ã³ã·ã¥ãŒã ã°ã«ãŒãå
ã® XNUMX ã€ã®ã³ã³ã·ã¥ãŒããåãããŒãã£ã·ã§ã³ããèªã¿åã
æ¶è²»è ã°ã«ãŒãã®æ¶è²»è
XNUMX ã€ã®ã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ãããŒãã£ã·ã§ã³ããããŒã¿ãèªã¿åããšãããã€ã³ã¿ãå®å
šã«å¶åŸ¡ããåã®ã»ã¯ã·ã§ã³ã§èª¬æããããã«ã¡ãã»ãŒãžãåŠçããŸãã
ã³ã³ã·ã¥ãŒããŒã®è€æ°ã®ã€ã³ã¹ã¿ã³ã¹ãåã group_id 㧠XNUMX ã€ã®ããŒãã£ã·ã§ã³ãæã€ãããã¯ã«æ¥ç¶ãããŠããå ŽåãæåŸã«æ¥ç¶ããã€ã³ã¹ã¿ã³ã¹ã«ãã€ã³ã¿ãŒã®å¶åŸ¡ãäžãããããã®ç¬éãããã¹ãŠã®ã¡ãã»ãŒãžãåä¿¡ããŸã (
å³3-4ã åãã³ã³ã·ã¥ãŒã ã°ã«ãŒãå
ã® XNUMX ã€ã®ã³ã³ã·ã¥ãŒããåãããŒãã£ã·ã§ã³ããèªã¿åã
ã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ã®æ°ãããŒãã£ã·ã§ã³ã®æ°ãè¶ ãããã®åŠçã¢ãŒãã¯ãäžçš®ã®æä»çã³ã³ã·ã¥ãŒããšèããããšãã§ããŸãã ããã¯ãã³ã³ã·ã¥ãŒã ã€ã³ã¹ã¿ã³ã¹ã®ãã¢ã¯ãã£ã-ããã·ãã(ãŸãã¯ãããã-ãŠã©ãŒã ã) ã¯ã©ã¹ã¿ãªã³ã°ãå¿ èŠãªå Žåã«åœ¹ç«ã¡ãŸãããè€æ°ã®ã³ã³ã·ã¥ãŒãã䞊è¡ããŠå®è¡ãã (ãã¢ã¯ãã£ã-ã¢ã¯ãã£ãããŸãã¯ãããã-ãããã) ããšã®æ¹ãã¯ããã«äžè¬çã§ããæ¶è²»è ã¯ã¹ã¿ã³ãã€ç¶æ ã§ãã
äžã§èª¬æãããã®ã¡ãã»ãŒãžé åžåäœã¯ãéåžžã® JMS ãã¥ãŒã®åäœãšæ¯èŒãããšé©ãã¹ããã®ã«ãªãå¯èœæ§ããããŸãã ãã®ã¢ãã«ã§ã¯ããã¥ãŒã«éä¿¡ãããã¡ãã»ãŒãžã¯ XNUMX ã€ã®ã³ã³ã·ã¥ãŒããŒéã§åçã«åæ£ãããŸãã
ã»ãšãã©ã®å Žåãã³ã³ã·ã¥ãŒããŒã®è€æ°ã®ã€ã³ã¹ã¿ã³ã¹ãäœæãããšãã¯ãã¡ãã»ãŒãžã䞊è¡ããŠåŠçããããèªã¿åãé床ãåäžãããããèªã¿åãããã»ã¹ã®å®å®æ§ãé«ããããã«ãããè¡ããŸãã ããŒãã£ã·ã§ã³ããããŒã¿ãèªã¿åãããšãã§ããã®ã¯äžåºŠã« XNUMX ã€ã®ã³ã³ã·ã¥ãŒã㌠ã€ã³ã¹ã¿ã³ã¹ã ãã§ãããKafka ã§ã¯ãããã©ã®ããã«å®çŸããã®ã§ãããã?
ãããè¡ã XNUMX ã€ã®æ¹æ³ã¯ãåäžã®ã³ã³ã·ã¥ãŒã㌠ã€ã³ã¹ã¿ã³ã¹ã䜿çšããŠãã¹ãŠã®ã¡ãã»ãŒãžãèªã¿åããã¹ã¬ãã ããŒã«ã«æž¡ãããšã§ãã ãã®ã¢ãããŒãã¯åŠçã¹ã«ãŒããããåäžãããŸãããã³ã³ã·ã¥ãŒã ããžãã¯ã®è€éæ§ã¯å¢å€§ããèªã¿åãã·ã¹ãã ã®å ç¢æ§ãåäžãããããšã¯ã§ããŸããã åé»ãŸãã¯åæ§ã®ã€ãã³ãã«ããã³ã³ã·ã¥ãŒãã® XNUMX ã€ã®ã³ããŒãããŠã³ããå Žåãæžç®ã¯åæ¢ããŸãã
Kafka ã§ãã®åé¡ã解決ããæšæºçãªæ¹æ³ã¯ã b ã䜿çšããããšã§ããÐããå€ãã®ããŒãã£ã·ã§ã³ã
ããŒãã£ã·ã§ãã³ã°
ããŒãã£ã·ã§ã³ã¯ãåäžã®ãããŒã«ãŒ ã€ã³ã¹ã¿ã³ã¹ã®åž¯åå¹
ãè¶
ããŠãããã¯ã®èªã¿åããšã¹ã±ãŒãªã³ã°ã䞊ååããããã®äž»èŠãªã¡ã«ããºã ã§ãã ãããããããç解ããããã«ãXNUMX ã€ã®ããŒãã£ã·ã§ã³ãæã€ãããã¯ããããXNUMX 人ã®ã³ã³ã·ã¥ãŒããŒããã®ãããã¯ããµãã¹ã¯ã©ã€ãããç¶æ³ãèããŠã¿ãŸããã (
å³3-5ã XNUMX 人ã®ã³ã³ã·ã¥ãŒããŒãè€æ°ã®ããŒãã£ã·ã§ã³ããèªã¿åããŸã
ãã®ã·ããªãªã§ã¯ãã³ã³ã·ã¥ãŒããŒã¯äž¡æ¹ã®ããŒãã£ã·ã§ã³ã® group_id ã«å¯Ÿå¿ãããã€ã³ã¿ãŒã«å¯Ÿããå¶åŸ¡ãäžããããäž¡æ¹ã®ããŒãã£ã·ã§ã³ããã¡ãã»ãŒãžã®èªã¿åããéå§ããŸãã
åã group_id ã®è¿œå ã®ã³ã³ã·ã¥ãŒããŒããã®ãããã¯ã«è¿œå ããããšãKafka ã¯ããŒãã£ã·ã§ã³ã® XNUMX ã€ãæåã®ã³ã³ã·ã¥ãŒããŒãã XNUMX çªç®ã®ã³ã³ã·ã¥ãŒããŒã«åå²ãåœãŠããŸãã ãã®åŸãã³ã³ã·ã¥ãŒãã®åã€ã³ã¹ã¿ã³ã¹ã¯ãããã¯ã® XNUMX ã€ã®ããŒãã£ã·ã§ã³ããèªã¿åããŸã (
ã¡ãã»ãŒãžã 20 ã¹ã¬ããã§äžŠååŠçãããããã«ããã«ã¯ãå°ãªããšã 20 ã®ããŒãã£ã·ã§ã³ãå¿ èŠã§ãã ããŒãã£ã·ã§ã³ã®æ°ãå°ãªãå Žåã¯ãæä»çã³ã³ã·ã¥ãŒãã®èª¬æã§åè¿°ããããã«ãäœãåŠçããå¿ èŠã®ãªãã³ã³ã·ã¥ãŒããæ®ãããŸãã
å³3-6ã åãã³ã³ã·ã¥ãŒã ã°ã«ãŒãå
ã® XNUMX ã€ã®ã³ã³ã·ã¥ãŒããç°ãªãããŒãã£ã·ã§ã³ããèªã¿åããŸã
ãã®ã¹ããŒã ã«ãããJMS ãã¥ãŒãç¶æããããã«å¿ èŠãªã¡ãã»ãŒãžé åžãšæ¯èŒããŠãKafka ãããŒã«ãŒã®è€éããå€§å¹ ã«è»œæžãããŸãã ããã§ã¯ã次ã®ç¹ãå¿é ããå¿ èŠã¯ãããŸããã
- ã©ãŠã³ãããã³å²ãåœãŠãããªãã§ãã ãããã¡ã®çŸåšã®å®¹éããŸãã¯åã®ã¡ãã»ãŒãž (JMS ã¡ãã»ãŒãž ã°ã«ãŒãã®å Žå) ã«åºã¥ããŠãã©ã®ã³ã³ã·ã¥ãŒãã次ã®ã¡ãã»ãŒãžãåä¿¡ããå¿ èŠããããã
- ã©ã®ã¡ãã»ãŒãžãã©ã®ã³ã³ã·ã¥ãŒãã«éä¿¡ããããããŸã倱æããå Žåã«ã¯ã¡ãã»ãŒãžãåé ä¿¡ããå¿ èŠããããã©ããã
Kafka ãããŒã«ãŒãè¡ãå¿ èŠãããã®ã¯ãã³ã³ã·ã¥ãŒããã¡ãã»ãŒãžãèŠæ±ãããšãã«ãã³ã³ã·ã¥ãŒãã«ã¡ãã»ãŒãžãé çªã«æž¡ãããšã ãã§ãã
ãã ããæ ¡æ£ãšå€±æããã¡ãã»ãŒãžã®åéä¿¡ã䞊ååããèŠä»¶ã¯ãªããªããŸããããããã®è²¬ä»»ã¯åã«ãããŒã«ãŒããã¯ã©ã€ã¢ã³ãã«ç§»ãããã ãã§ãã ããã¯ãã³ãŒãå ã§ããããèæ ®ããå¿ èŠãããããšãæå³ããŸãã
ã¡ãã»ãŒãžã®éä¿¡
ã©ã®ããŒãã£ã·ã§ã³ã«ã¡ãã»ãŒãžãéä¿¡ãããã決å®ããã®ã¯ããã®ã¡ãã»ãŒãžã®äœæè ã®è²¬ä»»ã§ãã ãããè¡ãããã¡ã«ããºã ãç解ããã«ã¯ããŸãå®éã«äœãéä¿¡ããŠããã®ããèããå¿ èŠããããŸãã
JMS ã§ã¯ã¡ã¿ããŒã¿ (ããããŒãšããããã£) ãšãã€ããŒã (ãã€ããŒã) ãå«ãæ¬æãå«ãã¡ãã»ãŒãžæ§é ã䜿çšããŸãããKafka ã§ã¯ã¡ãã»ãŒãžã¯æ¬¡ã®ããã«ãªããŸãã ãããŒãšå€ãã®ãã¢ã ã¡ãã»ãŒãž ãã€ããŒãã¯å€ãšããŠéä¿¡ãããŸãã äžæ¹ãããŒã¯äž»ã«ããŒãã£ã·ã§ã³åã«äœ¿çšããã次ã®ãã®ãå«ãŸããŠããå¿ èŠããããŸãã ããžãã¹ããžãã¯åºæã®ããŒé¢é£ããã¡ãã»ãŒãžãåãããŒãã£ã·ã§ã³ã«é 眮ããŸãã
第 2 ç« ã§ã¯ãé¢é£ã€ãã³ãã XNUMX 人ã®æ¶è²»è ãé çªã«åŠçããå¿ èŠããããªã³ã©ã€ã³ ãããã£ã³ã° ã·ããªãªã«ã€ããŠèª¬æããŸããã
- ãŠãŒã¶ãŒã¢ã«ãŠã³ããèšå®ãããŸããã
- ãéãã¢ã«ãŠã³ãã«å ¥éãããŸãã
- ã¢ã«ãŠã³ããããéãåŒãåºãè³ããè¡ãããŸãã
åã€ãã³ãããããã¯ã«æçš¿ãããã¡ãã»ãŒãžã§ããå Žåãèªç¶ããŒã¯ã¢ã«ãŠã³ã ID ã«ãªããŸãã
Kafka ãããã¥ãŒãµãŒ API ã䜿çšããŠã¡ãã»ãŒãžãéä¿¡ããããšããã®ã¡ãã»ãŒãžã¯ããŒãã£ã·ã§ã³é¢æ°ã«æž¡ãããã¡ãã»ãŒãžãš Kafka ã¯ã©ã¹ã¿ãŒã®çŸåšã®ç¶æ
ã«å¿ããŠãã¡ãã»ãŒãžã®éä¿¡å
ãšãªãããŒãã£ã·ã§ã³ã® ID ãè¿ãããŸãã ãã®æ©èœã¯ãPartitioner ã€ã³ã¿ãŒãã§ã€ã¹ãä»ã㊠Java ã«å®è£
ãããŸãã
ãã®ã€ã³ã¿ãŒãã§ãŒã¹ã¯æ¬¡ã®ããã«ãªããŸãã
interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
}
ããŒãã£ã·ã§ããŒã®å®è£ ã§ã¯ãããŒã«å¯ŸããŠããã©ã«ãã®æ±çšããã·ã¥ ã¢ã«ãŽãªãºã ã䜿çšããŠããŒãã£ã·ã§ã³ã決å®ããããããŒãæå®ãããŠããªãå Žåã¯ã©ãŠã³ãããã³ã䜿çšããŸãã ã»ãšãã©ã®å Žåããã®ããã©ã«ãå€ãé©åã«æ©èœããŸãã ãã ããå°æ¥çã«ã¯ãç¬èªã®ã³ãŒããäœæããããšã«ãªãã§ãããã
ç¬èªã®ããŒãã£ã·ã§ãã³ã°æŠç¥ãäœæãã
ã¡ãã»ãŒãž ãã€ããŒããšãšãã«ã¡ã¿ããŒã¿ãéä¿¡ããäŸãèŠãŠã¿ãŸãããã ãã®äŸã®ãã€ããŒãã¯ãã²ãŒã ã¢ã«ãŠã³ãã«å
¥éããããã®åœä»€ã§ãã åœä»€ã¯ãéä¿¡æã«å€æŽãããªãããšãä¿èšŒããããã®ã§ãããä¿¡é Œã§ããäžæµã·ã¹ãã ã®ã¿ããã®åœä»€ãéå§ã§ããããšã確èªããããã®ã§ãã ãã®å Žåãéä¿¡ã·ã¹ãã ãšåä¿¡ã·ã¹ãã ã¯ãã¡ãã»ãŒãžãèªèšŒããããã«çœ²åã䜿çšããããšã«åæããŸãã
éåžžã® JMS ã§ã¯ãåã«ãã¡ãã»ãŒãžçœ²åãããããã£ãå®çŸ©ãããããã¡ãã»ãŒãžã«è¿œå ããŸãã ãã ããKafka ã¯ã¡ã¿ããŒã¿ãæž¡ãããã®ã¡ã«ããºã ãæäŸãããããŒãšå€ã®ã¿ãæäŸããŸãã
å€ã¯æŽåæ§ãä¿æãããéè¡æ¯èŸŒãã€ããŒãã§ãããããããŒã§äœ¿çšããããŒã¿æ§é ãå®çŸ©ãã以å€ã«éžæè¢ã¯ãããŸããã ã¢ã«ãŠã³ãã«é¢é£ãããã¹ãŠã®ã¡ãã»ãŒãžãé çªã«åŠçããå¿ èŠããããããããŒãã£ã·ã§ãã³ã°ã«ã¢ã«ãŠã³ã ID ãå¿ èŠã§ãããšä»®å®ãããšã次㮠JSON æ§é ãèããããŸãã
{
"signature": "541661622185851c248b41bf0cea7ad0",
"accountId": "10007865234"
}
眲åã®å€ã¯ãã€ããŒãã«å¿ããŠå€åãããããããŒãã£ã·ã§ã㌠ã€ã³ã¿ãŒãã§ã€ã¹ã®ããã©ã«ãã®ããã·ã¥æŠç¥ã§ã¯é¢é£ã¡ãã»ãŒãžã確å®ã«ã°ã«ãŒãåã§ããŸããã ãããã£ãŠããã®ããŒã解æããaccountId å€ãåå²ããç¬èªã®æŠç¥ãäœæããå¿ èŠããããŸãã
Kafka ã«ã¯ãã¹ãã¢å ã®ã¡ãã»ãŒãžã®ç Žæãæ€åºããããã®ãã§ãã¯ãµã ãå«ãŸããŠãããã»ãã¥ãªãã£æ©èœã®å®å šãªã»ãããåãã£ãŠããŸãã ããã§ããäžèšã®ãããªæ¥çåºæã®èŠä»¶ãçºçããããšããããŸãã
ãŠãŒã¶ãŒã®ããŒãã£ã·ã§ã³åæŠç¥ã§ã¯ãé¢é£ãããã¹ãŠã®ã¡ãã»ãŒãžãåãããŒãã£ã·ã§ã³ã«åãŸãããã«ããå¿ èŠããããŸãã ããã¯åçŽããã«èŠããŸãããé¢é£ããã¡ãã»ãŒãžã®é åºä»ãã®éèŠæ§ãšãããã¯å ã®ããŒãã£ã·ã§ã³æ°ãã©ã®ããã«åºå®ãããŠãããã«ãããèŠä»¶ãè€éã«ãªãå ŽåããããŸãã
ãããã¯å ã®ããŒãã£ã·ã§ã³ã®æ°ã¯ããã©ãã£ãã¯ãåœåã®äºæ³ãè¶ ããå Žåã«è¿œå ãããå¯èœæ§ããããããæéã®çµéãšãšãã«å€åããå¯èœæ§ããããŸãã ãããã£ãŠãã¡ãã»ãŒãž ããŒã¯ãæåã«éä¿¡ãããããŒãã£ã·ã§ã³ã«é¢é£ä»ããããšãã§ãããããã¥ãŒãµãŒ ã€ã³ã¹ã¿ã³ã¹éã§å ±æãããç¶æ ã®äžéšãæå³ããŸãã
èæ
®ãã¹ããã XNUMX ã€ã®èŠçŽ ã¯ãããŒãã£ã·ã§ã³éã§ã®ã¡ãã»ãŒãžã®åçãªåæ£ã§ãã éåžžãããŒã¯ã¡ãã»ãŒãžå
šäœã«åçã«åæ£ããããããã·ã¥é¢æ°ã¯å°æ°ã®ããŒã®ã»ããã«å¯ŸããŠã¡ãã»ãŒãžãå
¬å¹³ã«åæ£ãããããšãä¿èšŒããŸããã
ã©ã®ãããªã¡ãã»ãŒãžã®åå²ãéžæããŠããåºåãæåèªäœãåå©çšããå¿
èŠãããå Žåãããããšã«æ³šæããããšãéèŠã§ãã
å°ççã«ç°ãªãå Žæã«ãã Kafka ã¯ã©ã¹ã¿ãŒéã§ããŒã¿ãã¬ããªã±ãŒãããèŠä»¶ãèæ ®ããŠãã ããã ãã®ç®çã®ããã«ãKafka ã«ã¯ MirrorMaker ãšåŒã°ããã³ãã³ã ã©ã€ã³ ããŒã«ãä»å±ããŠããŸããããã¯ãããã¯ã©ã¹ã¿ãŒããã¡ãã»ãŒãžãèªã¿åããå¥ã®ã¯ã©ã¹ã¿ãŒã«è»¢éããããã«äœ¿çšãããŸãã
MirrorMaker ã¯ãã¯ã©ã¹ã¿ãŒéã§ã¬ããªã±ãŒããããšãã«ã¡ãã»ãŒãžéã®çžå¯Ÿçãªé åºãç¶æããããã«ãã¬ããªã±ãŒãããããããã¯ã®ããŒãç解ããå¿ èŠããããŸããããã¯ããã®ãããã¯ã®ããŒãã£ã·ã§ã³ã®æ°ã XNUMX ã€ã®ã¯ã©ã¹ã¿ãŒã§åãã§ã¯ãªãå¯èœæ§ãããããã§ãã
ããã©ã«ãã®ããã·ã¥ãŸãã¯ã©ãŠã³ã ããã³ãã»ãšãã©ã®ã·ããªãªã§é©åã«æ©èœãããããã«ã¹ã¿ã ããŒãã£ã·ã§ãã³ã°æŠç¥ã¯æ¯èŒçãŸãã§ãã ãã ãã匷åãªé åºä»ãã®ä¿èšŒãå¿ èŠãªå ŽåããŸãã¯ãã€ããŒãããã¡ã¿ããŒã¿ãæœåºããå¿ èŠãããå Žåã¯ãããŒãã£ã·ã§ãã³ã°ã詳ããæ€èšããå¿ èŠããããŸãã
Kafka ã®ã¹ã±ãŒã©ããªãã£ãšããã©ãŒãã³ã¹ã®å©ç¹ã¯ãåŸæ¥ã®ãããŒã«ãŒã®è²¬ä»»ã®äžéšãã¯ã©ã€ã¢ã³ãã«ç§»ãããšã«ãã£ãŠããããããŸãã ãã®å Žåã䞊è¡ããŠåäœããè€æ°ã®ã³ã³ã·ã¥ãŒãéã§æœåšçã«é¢é£ããã¡ãã»ãŒãžãé åžãã決å®ãè¡ãããŸãã
JMS ãããŒã«ãŒããã®ãããªèŠä»¶ã«å¯ŸåŠããå¿ èŠããããŸãã èå³æ·±ãããšã«ãJMS ã¡ãã»ãŒãž ã°ã«ãŒã (ã¹ãã£ãã㌠ããŒã ãã©ã³ã·ã³ã° (SLB) æŠç¥ã®ããªãšãŒã·ã§ã³) ãéããŠå®è£ ããããé¢é£ããã¡ãã»ãŒãžãåãã³ã³ã·ã¥ãŒãã«éä¿¡ããã¡ã«ããºã ã§ããéä¿¡è ãã¡ãã»ãŒãžãé¢é£ãããã®ãšããŠããŒã¯ããå¿ èŠããããŸãã JMS ã®å ŽåããããŒã«ãŒã¯ããã®é¢é£ã¡ãã»ãŒãžã®ã°ã«ãŒããå€æ°ã®ã³ã³ã·ã¥ãŒããŒã®äžãã XNUMX ã€ã®ã³ã³ã·ã¥ãŒããŒã«éä¿¡ããã³ã³ã·ã¥ãŒããŒãèœã¡ãå Žåã«ã¯ã°ã«ãŒãã®æææš©ãè²æž¡ãã責任ããããŸãã
ãããã¥ãŒãµãŒå¥çŽ
ã¡ãã»ãŒãžãéä¿¡ãããšãã«èæ ®ãã¹ãããšã¯ããŒãã£ã·ã§ãã³ã°ã ãã§ã¯ãããŸããã Java API ã®ãããã¥ãŒãµãŒ ã¯ã©ã¹ã® send() ã¡ãœãããèŠãŠã¿ãŸãããã
Future < RecordMetadata > send(ProducerRecord < K, V > record);
Future < RecordMetadata > send(ProducerRecord < K, V > record, Callback callback);
ã©ã¡ãã®ã¡ãœããã Future ãè¿ãããšã«æ³šæããŠãã ãããããã¯ãéä¿¡æäœãããã«å®è¡ãããªãããšã瀺ããŸãã ãã®çµæãã¡ãã»ãŒãž (ProducerRecord) ãåã¢ã¯ãã£ã ããŒãã£ã·ã§ã³ã®éä¿¡ãããã¡ã«æžã蟌ãŸããKafka ã¯ã©ã€ã¢ã³ã ã©ã€ãã©ãªã®ããã¯ã°ã©ãŠã³ã ã¹ã¬ãããšããŠãããŒã«ãŒã«éä¿¡ãããŸãã ããã«ããåŠçãé©ãã»ã©é«éã«ãªããŸãããçµéšã®æµ ãã¢ããªã±ãŒã·ã§ã³ã§ã¯ããã»ã¹ãåæ¢ãããšã¡ãã»ãŒãžã倱ãããå¯èœæ§ããããŸãã
ãã€ãã®ããã«ãããã©ãŒãã³ã¹ãç ç²ã«ããŠéä¿¡æäœã®ä¿¡é Œæ§ãé«ããæ¹æ³ããããŸãã ãã®ãããã¡ã®ãµã€ãºã¯ 0 ã«èšå®ã§ãã次ã®ããã«ãéä¿¡ã¢ããªã±ãŒã·ã§ã³ ã¹ã¬ããã¯ãããŒã«ãžã®ã¡ãã»ãŒãžè»¢éãå®äºãããŸã§åŸ æ©ããããã«ãªããŸãã
RecordMetadata metadata = producer.send(record).get();
ã¡ãã»ãŒãžã®èªã¿åãã«é¢ãã詳现
ã¡ãã»ãŒãžã®èªã¿åãã«ã¯ãããã«è€éãªç¹ããããããæšæž¬ããå¿ èŠããããŸãã ã¡ãã»ãŒãžã«å¿çããŠã¡ãã»ãŒãž ãªã¹ããŒãå®è¡ã§ãã JMS API ãšã¯ç°ãªãã æ¶è²»è²¡ Kafka ã¯ããŒãªã³ã°ã®ã¿ãè¡ããŸãã æ¹æ³ã詳ããèŠãŠã¿ãŸããã ããŒãªã³ã°()ãã®ç®çã®ããã«äœ¿çšãããŸã:
ConsumerRecords < K, V > poll(long timeout);
ã¡ãœããã®æ»ãå€ã¯ãè€æ°ã®ãªããžã§ã¯ããå«ãã³ã³ããæ§é ã§ãã æ¶è²»è èšé² å Žåã«ãã£ãŠã¯è€æ°ã®ããŒãã£ã·ã§ã³ããã æ¶è²»è èšé² ããèªäœã¯ã掟çå ã®ããŒãã£ã·ã§ã³ãªã©ãé¢é£ããã¡ã¿ããŒã¿ãå«ãããŒãšå€ã®ãã¢ã®ãã«ã㌠ãªããžã§ã¯ãã§ãã
第 2 ç« ã§èª¬æããããã«ãã¯ã©ã€ã¢ã³ããã¡ãã»ãŒãžãåŠçã§ããªãå Žåãã¡ãã»ãŒãžãäžæ¢ãããå Žåãªã©ãã¡ãã»ãŒãžã®åŠçãæåãŸãã¯å€±æããåŸã«ã©ããªããã«çæããå¿
èŠããããŸãã JMS ã§ã¯ãããã¯ç¢ºèªå¿çã¢ãŒããéããŠåŠçãããŸããã ãããŒã«ãŒã¯ãæ£åžžã«åŠçãããã¡ãã»ãŒãžãåé€ããããçã®ã¡ãã»ãŒãžãŸãã¯åœã®ã¡ãã»ãŒãžãåé
ä¿¡ããŸã (ãã©ã³ã¶ã¯ã·ã§ã³ã䜿çšããããšä»®å®ããŸã)ã
Kafka ã®åäœã¯å€§ããç°ãªããŸãã ã¡ãã»ãŒãžã¯æ ¡æ£åŸã«ãããŒã«ãŒã§åé€ãããŸããã倱æããå Žåã«äœãèµ·ãããã¯æ ¡æ£ã³ãŒãèªäœã®è²¬ä»»ã§ãã
ãã§ã«è¿°ã¹ãããã«ãã³ã³ã·ã¥ãŒã ã°ã«ãŒãã¯ãã°å ã®ãªãã»ããã«é¢é£ä»ããããŠããŸãã ãã®ãªãã»ããã«é¢é£ä»ãããããã°äœçœ®ã¯ã次ã®ã¡ãã»ãŒãžã«å¿çããŠçºè¡ãããã¡ãã»ãŒãžã«å¯Ÿå¿ããŸãã ããŒãªã³ã°()ã ãã®ãªãã»ãããå¢å ããæç¹ãèªã¿åãã®æ±ºå®çã«ãªããŸãã
åã«èª¬æããèªã¿åãã¢ãã«ã«æ»ããšãã¡ãã»ãŒãžåŠç㯠XNUMX ã€ã®æ®µéã§æ§æãããŸãã
- èªãã¡ãã»ãŒãžãååŸããŸãã
- ã¡ãã»ãŒãžãåŠçããŸãã
- ã¡ãã»ãŒãžã確èªããŸãã
Kafka ã³ã³ã·ã¥ãŒããŒã«ã¯æ§æãªãã·ã§ã³ãä»å±ããŠããŸã èªåã³ããããæå¹ã«ããã ããã¯ããèªåããšããåèªãå«ãèšå®ãšåæ§ã«ãé »ç¹ã«äœ¿çšãããããã©ã«ãèšå®ã§ãã
Kafka 0.10 ããåã§ã¯ããã®ãªãã·ã§ã³ã䜿çšããã¯ã©ã€ã¢ã³ãã¯ã次ã®åŒã³åºãã§èªã¿åãããæåŸã®ã¡ãã»ãŒãžã®ãªãã»ãããéä¿¡ããŠããŸããã ããŒãªã³ã°() å å·¥åŸã ããã¯ãã¯ã©ã€ã¢ã³ãããã§ã«ãã§ããããã¡ãã»ãŒãžãåŠçããŠããŠããåŒã³åºãåã«äºæããç Žæ£ãããå Žåããããã®ã¡ãã»ãŒãžãååŠçã§ããããšãæå³ããŸãã ããŒãªã³ã°()ã ãããŒã«ãŒã¯ã¡ãã»ãŒãžãèªã¿åãããåæ°ã«é¢ããç¶æ ãä¿æããªãããã次ã«ãã®ã¡ãã»ãŒãžãååŸããã³ã³ã·ã¥ãŒããŒã¯ãäœãæªãããšãèµ·ãã£ãããšãç¥ããŸããã ãã®åäœã¯ç䌌ãã©ã³ã¶ã¯ã·ã§ã³ã§ããã ãªãã»ããã¯ã¡ãã»ãŒãžãæ£åžžã«åŠçãããå Žåã«ã®ã¿ã³ããããããŸãããã¯ã©ã€ã¢ã³ããäžæ¢ãããå ŽåããããŒã«ãŒã¯åãã¡ãã»ãŒãžãå¥ã®ã¯ã©ã€ã¢ã³ãã«å床éä¿¡ããŸãã ãã®åäœã¯ã¡ãã»ãŒãžé ä¿¡ã®ä¿èšŒãšäžèŽããŠããŸãããå°ãªããšãäžåºŠã¯"
Kafka 0.10 ã§ã¯ãèšå®ã«åŸã£ãŠã¯ã©ã€ã¢ã³ã ã©ã€ãã©ãªã«ãã£ãŠã³ããããå®æçã«ããªã¬ãŒãããããã«ã¯ã©ã€ã¢ã³ã ã³ãŒããå€æŽãããŸããã auto.commit.interval.msã ãã®åäœã¯ãJMS AUTO_ACKNOWLEDGE ã¢ãŒããš DUPS_OK_ACKNOWLEDGE ã¢ãŒãã®éã®ã©ããã«ãããŸãã èªåã³ãããã䜿çšãããšãã¡ãã»ãŒãžãå®éã«åŠçããããã©ããã«é¢ä¿ãªããã¡ãã»ãŒãžãã³ããããããå¯èœæ§ããããŸããããã¯ãé ãã³ã³ã·ã¥ãŒããŒã®å Žåã«çºçããå¯èœæ§ããããŸãã ã³ã³ã·ã¥ãŒããäžæ¢ãããå Žåãã¡ãã»ãŒãžã¯æ¬¡ã®ã³ã³ã·ã¥ãŒãã«ãã£ãŠã³ããããããäœçœ®ãããã§ããããããããã¡ãã»ãŒãžã倱ãããå¯èœæ§ããããŸãã ãã®å ŽåãKafka ã¯ã¡ãã»ãŒãžã倱ã£ãã®ã§ã¯ãªããèªã¿åãã³ãŒããã¡ãã»ãŒãžãåŠçããªãã£ãã ãã§ãã
ãã®ã¢ãŒãã«ã¯ããŒãžã§ã³ 0.9 ãšåãçŽæããããŸããã€ãŸããã¡ãã»ãŒãžã¯åŠçã§ããŸããã倱æãããšãªãã»ãããã³ãããããããé ä¿¡ã XNUMX åã«ãªãå¯èœæ§ããããŸãã å®è¡æã«ååŸããã¡ãã»ãŒãžãå¢ããã»ã© ããŒãªã³ã°()ããã®åé¡ã¯ããã«å€§ãããªããŸãã
21 ããŒãžã®ããã¥ãŒããã®ã¡ãã»ãŒãžã®èªã¿åããã§èª¬æããããã«ãé害ã¢ãŒããèæ ®ãããšãã¡ãã»ãŒãžã³ã° ã·ã¹ãã ã§ã¯ã¡ãã»ãŒãžã XNUMX åã ãé ä¿¡ããããšã¯ã§ããŸããã
Kafka ã§ã¯ããªãã»ãã (offset) ãã³ããã (ã³ããã) ããæ¹æ³ã¯ãèªåãšæåã® 0.9 ã€ããããŸãã ã©ã¡ãã®å Žåããã¡ãã»ãŒãžãåŠçããããã®ã®ã³ãããåã«å€±æããå Žåãã¡ãã»ãŒãžã¯è€æ°ååŠçãããå¯èœæ§ããããŸãã ã³ããããããã¯ã°ã©ãŠã³ãã§çºçããã³ãŒããåŠçãããåã«å®äºããŠããŸã£ãå Žå (ãããã Kafka XNUMX 以åã®å Žå)ãã¡ãã»ãŒãžããŸã£ããåŠçããªãããšãéžæã§ããŸãã
ãã©ã¡ãŒã¿ãèšå®ããããšã§ãKafka ã³ã³ã·ã¥ãŒã API ã§æåãªãã»ãã ã³ããã ããã»ã¹ãå¶åŸ¡ã§ããŸãã èªåã³ããããæå¹ã«ãã false ã«èšå®ãã次ã®ã¡ãœããã®ãããããæ瀺çã«åŒã³åºããŸãã
void commitSync();
void commitAsync();
ã¡ãã»ãŒãžããå°ãªããšã XNUMX åãåŠçãããå Žåã¯ããªãã»ãããæåã§ã³ãããããå¿ èŠããããŸãã commitSync()ã¡ãã»ãŒãžãåŠçããçŽåŸã«ãã®ã³ãã³ããå®è¡ããŸãã
ãããã®ã¡ãœããã§ã¯ãã¡ãã»ãŒãžãåŠçãããåã«ç¢ºèªå¿çããããšã¯ã§ããŸãããããã©ã³ã¶ã¯ã·ã§ã³ã§ãããã®ããã«èŠããªããæœåšçãªåŠçé 延ãæé€ããããšã¯ã§ããŸããã Kafka ã«ã¯ãã©ã³ã¶ã¯ã·ã§ã³ã¯ãããŸããã ã¯ã©ã€ã¢ã³ãã«ã¯æ¬¡ã®ããšãè¡ãæ©èœããããŸããã
- åœã®ã¡ãã»ãŒãžãèªåçã«ããŒã«ããã¯ããŸãã ã¡ãã»ãŒãžã®åé ä¿¡ããããŒã«ãŒã«äŸåã§ããªããããã³ã³ã·ã¥ãŒããŒèªèº«ããåé¡ã®ãããã€ããŒããããã¯ãšã³ãã®åæ¢ã«ãã£ãŠçããäŸå€ãåŠçããå¿ èŠããããŸãã
- 98 ã€ã®ã¢ãããã¯æäœã§è€æ°ã®ãããã¯ã«ã¡ãã»ãŒãžãéä¿¡ããŸãã ããã«èª¬æããããã«ãããŸããŸãªãããã¯ãããŒãã£ã·ã§ã³ã«å¯Ÿããå¶åŸ¡ã¯ãéä¿¡æã«ãã©ã³ã¶ã¯ã·ã§ã³ã調æŽããªã Kafka ã¯ã©ã¹ã¿ãŒå ã®ããŸããŸãªãã·ã³ã«åžžé§ããããšãã§ããŸãã ãã®èšäºã®å·çæç¹ã§ã¯ãKIP-XNUMX ã§ãããå¯èœã«ããããã«ããã€ãã®äœæ¥ãè¡ãããŠããŸãã
- ãããããã¯ããã® XNUMX ã€ã®ã¡ãã»ãŒãžã®èªã¿åãããå¥ã®ãããã¯ãžã®å¥ã®ã¡ãã»ãŒãžã®éä¿¡ãšé¢é£ä»ããŸãã ç¹°ãè¿ãã«ãªããŸãããKafka ã®ã¢ãŒããã¯ãã£ã¯ XNUMX ã€ã®ãã¹ãšããŠå®è¡ãããå€ãã®ç¬ç«ãããã·ã³ã«äŸåããŠããããããé èœããè©Šã¿ã¯è¡ãããŠããŸããã ããšãã°ããªã³ã¯ãå¯èœã«ãã API ã³ã³ããŒãã³ãã¯ãããŸããã æ¶è²»è О ãããã¥ãŒãµãŒ ãã©ã³ã¶ã¯ã·ã§ã³äžã JMS ã§ã¯ãããã¯ãªããžã§ã¯ãã«ãã£ãŠæäŸãããŸãã ã»ãã·ã§ã³ãéãããããäœæããã ã¡ãã»ãŒãžãããã¥ãŒãµãŒ О ã¡ãã»ãŒãžæ¶è²»è .
ãã©ã³ã¶ã¯ã·ã§ã³ã«äŸåã§ããªãå ŽåãåŸæ¥ã®ã¡ãã»ãŒãžã³ã° ã·ã¹ãã ãæäŸããã»ãã³ãã£ã¯ã¹ã«è¿ãã»ãã³ãã£ã¯ã¹ãæäŸããã«ã¯ã©ãããã°ããã§ãããã?
ã³ã³ã·ã¥ãŒãã®ã¯ã©ãã·ã¥æãªã©ãã¡ãã»ãŒãžãåŠçãããåã«ã³ã³ã·ã¥ãŒãã®ãªãã»ãããå¢å ããå¯èœæ§ãããå Žåãã³ã³ã·ã¥ãŒãã«ã¯ãããŒãã£ã·ã§ã³ãå²ãåœãŠãããŠãããšãã«ã³ã³ã·ã¥ãŒã ã°ã«ãŒããã¡ãã»ãŒãžãèŠéãããã©ãããç¥ãæ¹æ³ããããŸããã ãããã£ãŠãXNUMX ã€ã®æŠç¥ã¯ããªãã»ãããåã®äœçœ®ã«å·»ãæ»ãããšã§ãã Kafka ã³ã³ã·ã¥ãŒã㌠API ã¯ããã®ããã«æ¬¡ã®ã¡ãœãããæäŸããŸãã
void seek(TopicPartition partition, long offset);
void seekToBeginning(Collection < TopicPartition > partitions);
æ¹æ³ æ±ããïŒïŒ ã¡ãœããã§äœ¿çšã§ããŸã
offsetsForTimes(å°å³æ€çŽ¢ããã¿ã€ã ã¹ã¿ã³ã) éå»ã®ããç¹å®ã®æç¹ã®ç¶æ
ã«å·»ãæ»ãããšã
ãã®ã¢ãããŒãã䜿çšãããšãæé»çã«ã以åã«åŠçãããäžéšã®ã¡ãã»ãŒãžãå床èªã¿åãããŠåŠçãããå¯èœæ§ãéåžžã«é«ããªããŸãã ãããåé¿ããã«ã¯ã第 4 ç« ã§èª¬æããããã«ãåªçèªã¿åãã䜿çšããŠã以åã«è¡šç€ºããã¡ãã»ãŒãžã远跡ããéè€ãæé€ããŸãã
ãããã¯ãã¡ãã»ãŒãžã®æ倱ãéè€ã蚱容ãããéããã³ã³ã·ã¥ãŒã㌠ã³ãŒããåçŽã«ããããšãã§ããŸãã ãã°ã€ãã³ããã¡ããªã¯ã¹ãã¯ãªãã¯è¿œè·¡ãªã©ã®åŠçãªã©ãKafka ãäžè¬çã«äœ¿çšããããŠãŒã¹ã±ãŒã¹ãèŠããšãåã ã®ã¡ãã»ãŒãžã®æ倱ãåšå²ã®ã¢ããªã±ãŒã·ã§ã³ã«å€§ããªåœ±é¿ãäžããå¯èœæ§ã¯äœãããšãããããŸãã ãã®ãããªå Žåã¯ãããã©ã«ãå€ããã®ãŸãŸäœ¿çšã§ããŸãã äžæ¹ãã¢ããªã±ãŒã·ã§ã³ã§æ¯æããéä¿¡ããå¿ èŠãããå Žåã¯ãåã ã®ã¡ãã»ãŒãžã泚ææ·±ãåŠçããå¿ èŠããããŸãã ãã¹ãŠã¯ã³ã³ããã¹ãã«äŸåããŸãã
å人çãªèŠ³å¯ã«ãããšãã¡ãã»ãŒãžã®åŒ·åºŠãå¢ãã«ã€ããŠãåã ã®ã¡ãã»ãŒãžã®äŸ¡å€ã¯æžå°ããŸãã 倧ããªã¡ãã»ãŒãžã¯ãéçŽããã圢åŒã§è¡šç€ºãããšäŸ¡å€ãããåŸåããããŸãã
é«å¯çšæ§
é«å¯çšæ§ã«å¯Ÿãã Kafka ã®ã¢ãããŒãã¯ãActiveMQ ã®ã¢ãããŒããšã¯å€§ããç°ãªããŸãã Kafka ã¯ããã¹ãŠã®ãããŒã«ãŒ ã€ã³ã¹ã¿ã³ã¹ãåæã«ã¡ãã»ãŒãžãåä¿¡ããã³é åžããã¹ã±ãŒã«ã¢ãŠã ã¯ã©ã¹ã¿ãŒãäžå¿ã«èšèšãããŠããŸãã
Kafka ã¯ã©ã¹ã¿ãŒã¯ãç°ãªããµãŒããŒäžã§å®è¡ãããè€æ°ã®ãããŒã«ãŒ ã€ã³ã¹ã¿ã³ã¹ã§æ§æãããŸãã Kafka ã¯ãåããŒããç¬èªã®å°çšã¹ãã¬ãŒãžãæã€éåžžã®ã¹ã¿ã³ãã¢ãã³ ããŒããŠã§ã¢äžã§å®è¡ãããããã«èšèšãããŠããŸãã è€æ°ã®èšç®ããŒããæéã競ãå¯èœæ§ãããããããããã¯ãŒã¯æ¥ç¶ã¹ãã¬ãŒãž (SAN) ã®äœ¿çšã¯æšå¥šãããŸãããЫä¿ç®¡ééãé·ããªãã競åãçºçããŸãã
ã«ãã«ã¯ åžžã«ãªã³ ã·ã¹ãã ã å€ãã®å€§èŠæš¡ãª Kafka ãŠãŒã¶ãŒã¯ã¯ã©ã¹ã¿ãŒãã·ã£ããããŠã³ããããšã¯ãªãããœãããŠã§ã¢ã¯åžžã«é 次åèµ·åããŠæŽæ°ãããŸãã ããã¯ãã¡ãã»ãŒãžããã³ãããŒã«ãŒéã®å¯Ÿè©±ã«ã€ããŠã以åã®ããŒãžã§ã³ãšã®äºææ§ãä¿èšŒããããšã«ãã£ãŠå®çŸãããŸãã
ãµãŒããŒã¯ã©ã¹ã¿ãŒã«æ¥ç¶ããããããŒã«ãŒ
åºæ¬çãªã±ãŒã¹ã§ã¯ã次ã®ããããã£ãæã€ãããã¯ã Kafka ã¯ã©ã¹ã¿ãŒå ã«äœæãããŸãã
- ããŒãã£ã·ã§ã³ã®æ°ã åã«èª¬æããããã«ãããã§äœ¿çšãããæ£ç¢ºãªå€ã¯ã䞊åèªã¿åãã®å¿ èŠãªã¬ãã«ã«ãã£ãŠç°ãªããŸãã
- ã¬ããªã±ãŒã·ã§ã³ä¿æ° (ä¿æ°) ã¯ãã¯ã©ã¹ã¿ãŒå ã®ãããŒã«ãŒ ã€ã³ã¹ã¿ã³ã¹ã®æ°ããã®ããŒãã£ã·ã§ã³ã®ãã°ãå«ãå¿ èŠããããã決å®ããŸãã
ZooKeeper ã調æŽã«äœ¿çšããŠãKafka ã¯ã¯ã©ã¹ã¿ãŒå ã®ãããŒã«ãŒéã§æ°ããããŒãã£ã·ã§ã³ãå ¬å¹³ã«åæ£ããããšããŸãã ããã¯ãã³ã³ãããŒã©ãŒãšããŠæ©èœããåäžã®ã€ã³ã¹ã¿ã³ã¹ã«ãã£ãŠå®è¡ãããŸãã
å®è¡æ ãããã¯ããŒãã£ã·ã§ã³ããšã« ã³ã³ãããŒã©ãŒ ãããŒã«ãŒã«åœ¹å²ãå²ãåœãŠã ãªãŒã㌠(ãªãŒããŒããã¹ã¿ãŒããã¬ãŒã³ã¿ãŒ) ãã©ãã¯ãŒ ïŒåŸè ã奎é·ãéšäžïŒã ãã®ããŒãã£ã·ã§ã³ã®ãªãŒããŒãšããŠæ©èœãããããŒã«ãŒã¯ããããã¥ãŒãµããéä¿¡ããããã¹ãŠã®ã¡ãã»ãŒãžãåä¿¡ãããã®ã¡ãã»ãŒãžãã³ã³ã·ã¥ãŒãã«é åžãã責任ããããŸãã ã¡ãã»ãŒãžãããã㯠ããŒãã£ã·ã§ã³ã«éä¿¡ããããšããã®ããŒãã£ã·ã§ã³ã®ãã©ãã¯ãŒãšããŠæ©èœãããã¹ãŠã®ãããŒã«ãŒ ããŒãã«ã¡ãã»ãŒãžãè€è£œãããŸãã ããŒãã£ã·ã§ã³ã®ãã°ãå«ãåããŒãã¯æ¬¡ã®ããã«åŒã°ããŸãã ã¬ããªã«ã ãããŒã«ãŒã¯ãäžéšã®ããŒãã£ã·ã§ã³ã§ã¯ãªãŒããŒãšããŠæ©èœããä»ã®ããŒãã£ã·ã§ã³ã§ã¯ãã©ãã¯ãŒãšããŠæ©èœããŸãã
ãªãŒããŒãä¿æãããã¹ãŠã®ã¡ãã»ãŒãžãå«ããã©ãã¯ãŒãåŒã³åºãããŸãã åæãããã¬ããªã« (åæç¶æ ã«ããã¬ããªã«ãåæã¬ããªã«)ã ããŒãã£ã·ã§ã³ã®ãªãŒããŒãšããŠæ©èœãããããŒã«ãŒãããŠã³ããå Žåããã®ããŒãã£ã·ã§ã³ã«å¯ŸããŠææ°ã®ç¶æ ãŸãã¯åæãããŠãããããŒã«ãŒããªãŒããŒã®åœ¹å²ãåŒãç¶ãããšãã§ããŸãã ä¿¡ããããªãã»ã©æç¶å¯èœãªãã¶ã€ã³ã§ãã
ãããã¥ãŒãµãŒèšå®ã®äžéšã¯ãã©ã¡ãŒã¿ã§ãã ACKããã«ãããã¢ããªã±ãŒã·ã§ã³ ã¹ã¬ãããéä¿¡ãç¶è¡ããåã«ã¡ãã»ãŒãžã®åä¿¡ãç¢ºèª (æ¿èª) ããå¿ èŠãããã¬ããªã«ã®æ°ã決ãŸããŸã (0ã1ããŸãã¯ãã¹ãŠ)ã ã«èšå®ãããŠããå Žå ãã¡ãã»ãŒãžãåä¿¡ãããšããªãŒããŒã¯ããããã¯èšå®ã§å®çŸ©ãããè€æ°ã®ãã¥ãŒ (ããèªäœãå«ã) ããã¬ã³ãŒãã®ç¢ºèª (è¯å®å¿ç) ãåä¿¡ãããšããã«ããããã¥ãŒãµãŒã«ç¢ºèªãéãè¿ããŸãã min.insync.ã¬ããªã« (ããã©ã«ã㯠1)ã ã¡ãã»ãŒãžãæ£åžžã«è€è£œã§ããªãå Žåããããã¥ãŒãµã¯ã¢ããªã±ãŒã·ã§ã³äŸå€ãã¹ããŒããŸã (ã¬ããªã«ã足ããªã ãŸã㯠远å åŸã®ã¬ããªã«ãäžè¶³ããŠããŸã).
äžè¬çãªæ§æã§ã¯ãã¬ããªã±ãŒã·ã§ã³ä¿æ° 3 (ããŒãã£ã·ã§ã³ããšã« 1 ã€ã®ãªãŒããŒã2 ã€ã®ãã©ãã¯ãŒ) ãšãã©ã¡ãŒã¿ãŒã䜿çšããŠãããã¯ãäœæãããŸãã min.insync.ã¬ããªã« ãã®å Žåãã¯ã©ã¹ã¿ãŒã¯ãã¯ã©ã€ã¢ã³ã ã¢ããªã±ãŒã·ã§ã³ã«åœ±é¿ãäžããããšãªããããã㯠ããŒãã£ã·ã§ã³ã管çããŠãããããŒã«ãŒã® 2 ã€ãããŠã³ããããšãèš±å¯ããŸãã
ããã¯ãããã©ãŒãã³ã¹ãšä¿¡é Œæ§ã®éã®ãã§ã«ããªãã¿ã®ãã¬ãŒããªãã«æ»ããŸãã ã¬ããªã±ãŒã·ã§ã³ã¯ããã©ãã¯ãŒããã®ç¢ºèª (確èªå¿ç) ãåŸ æ©ããè¿œå ã®æéãç ç²ã«ããŠçºçããŸãã ãã ãã䞊åå®è¡ããããããå°ãªããšã XNUMX ã€ã®ããŒããžã®ã¬ããªã±ãŒã·ã§ã³ã¯ XNUMX ã€ã®ããŒããšåãããã©ãŒãã³ã¹ã«ãªããŸã (ãããã¯ãŒã¯åž¯åå¹ ã®äœ¿çšéã®å¢å ãç¡èŠããŸã)ã
ãã®ã¬ããªã±ãŒã·ã§ã³ ã¹ããŒã ã䜿çšããããšã«ãããKafka ã¯ã次ã®æäœã§åã¡ãã»ãŒãžãç©ççã«ãã£ã¹ã¯ã«æžã蟌ãå¿
èŠæ§ãå·§ã¿ã«åé¿ããŸãã åæ()ã ãããã¥ãŒãµã«ãã£ãŠéä¿¡ãããåã¡ãã»ãŒãžã¯ããŒãã£ã·ã§ã³ ãã°ã«æžã蟌ãŸããŸããã第 2 ç« ã§èª¬æããããã«ããã¡ã€ã«ãžã®æžã蟌ã¿ã¯æåã¯ãªãã¬ãŒãã£ã³ã° ã·ã¹ãã ã®ãããã¡å
ã§è¡ãããŸãã ãã®ã¡ãã»ãŒãžãå¥ã® Kafka ã€ã³ã¹ã¿ã³ã¹ã«ã¬ããªã±ãŒãããããã®ã¡ã¢ãªå
ã«ããå ŽåããªãŒããŒã®åªå€±ã¯ã¡ãã»ãŒãžèªäœã倱ãããããšãæå³ãããã®ã§ã¯ãªããåæãããã¬ããªã«ã«ãã£ãŠåŒãç¶ãããå¯èœæ§ããããŸãã
æè¡ã®æåŠ åæ() ã€ãŸããKafka ã¯ã¡ãã»ãŒãžãã¡ã¢ãªã«æžã蟌ãã®ãšåãéãã§ã¡ãã»ãŒãžãåä¿¡ã§ããŸãã éã«ãã¡ã¢ãªããã£ã¹ã¯ã«ãã©ãã·ã¥ããããšãé¿ããããšãã§ããæéãé·ããã°é·ãã»ã©ãããè¯ãããšã«ãªããŸãã ãã®ãããKafka ãããŒã«ãŒã« 64 GB 以äžã®ã¡ã¢ãªãå²ãåœãŠãããããšãçãããããŸããã ãã®ã¡ã¢ãªäœ¿çšéã¯ãåäžã® Kafka ã€ã³ã¹ã¿ã³ã¹ãåŸæ¥ã®ã¡ãã»ãŒãž ãããŒã«ãŒã®æ°ååã®é床ã§ç°¡åã«å®è¡ã§ããããšãæå³ããŸãã
Kafka ã¯æäœãé©çšããããã«æ§æããããšãã§ããŸã åæ() ã¡ãã»ãŒãžããã±ãŒãžã«ã Kafka ã®ãã¹ãŠã¯ããã±ãŒãžæåã§ãããããå®éã«ã¯å€ãã®ãŠãŒã¹ã±ãŒã¹ã§éåžžã«ããŸãæ©èœããéåžžã«åŒ·åãªä¿èšŒãå¿
èŠãšãããŠãŒã¶ãŒã«ãšã£ãŠäŸ¿å©ãªããŒã«ã§ãã Kafka ã®çŽç²ãªããã©ãŒãã³ã¹ã®å€ãã¯ããã±ãããšããŠãããŒã«ãŒã«éä¿¡ãããã¡ãã»ãŒãžããåŸããããããã®ã¡ãã»ãŒãžã¯ã次ã®ã³ãã³ãã䜿çšããŠé£ç¶ãããã¯ã§ãããŒã«ãŒããèªã¿åãããŸãã
ããã㯠ããŒãã£ã·ã§ã³ã¯å€ãã®åå¥ã®ãã·ã³ã«ã¹ã±ãŒã«ã¢ãŠãã§ãããããKafka ã¯ã©ã¹ã¿ãŒã§ã¯åäžã® Kafka ãããŒã«ãŒã䜿çšãããããã¯ããã«åªããããã©ãŒãã³ã¹ãå¯èœã§ãã
çµæ
ãã®ç« ã§ã¯ãKafka ã¢ãŒããã¯ãã£ãã¯ã©ã€ã¢ã³ããšãããŒã«ãŒéã®é¢ä¿ãã©ã®ããã«åèããŠãåŸæ¥ã®ã¡ãã»ãŒãž ãããŒã«ãŒã®äœåãã®ã¹ã«ãŒããããåããéåžžã«å ç¢ãªã¡ãã»ãŒãžã³ã° ãã€ãã©ã€ã³ãæäŸããæ¹æ³ã«ã€ããŠèª¬æããŸããã ãããå®çŸããããã«äœ¿çšãããæ©èœã«ã€ããŠèª¬æãããã®æ©èœãæäŸããã¢ããªã±ãŒã·ã§ã³ã®ã¢ãŒããã¯ãã£ã«ã€ããŠç°¡åã«èª¬æããŸããã 次ã®ç« ã§ã¯ãã¡ãã»ãŒãžã³ã° ããŒã¹ã®ã¢ããªã±ãŒã·ã§ã³ã解決ããå¿ èŠãããäžè¬çãªåé¡ãèŠãŠããããã«å¯ŸåŠããæŠç¥ã«ã€ããŠèª¬æããŸãã ãã®ç« ã®æåŸã§ã¯ãã¡ãã»ãŒãžã³ã° ãã¯ãããžäžè¬ã«ã€ããŠèª¬æããæ¹æ³ãæŠèª¬ããŠããŠãŒã¹ ã±ãŒã¹ã«å¯Ÿããã¡ãã»ãŒãžã³ã° ãã¯ãããžã®é©åæ§ãè©äŸ¡ã§ããããã«ããŸãã
以åã«ç¿»èš³ãããéšå:
翻蚳å®äº:
ç¶ç¶ããã«ã¯...
ç»é²ãŠãŒã¶ãŒã®ã¿ãã¢ã³ã±ãŒãã«åå ã§ããŸãã
ããªãã®çµç¹ã§ã¯ Kafka ã䜿çšãããŠããŸãã?
-
ã¯ã
-
ããŒ
-
以åã¯äœ¿çšãããŠããŸããããçŸåšã¯äœ¿çšãããŠããŸãã
-
䜿çšããäºå®ã§ã
38 人ã®ãŠãŒã¶ãŒãæ祚ããŸããã 8åã®ãŠãŒã¶ãŒãæ£æš©ããã
åºæïŒ habr.com