ä»ã«ã©ã®ãããªããã»ã¹ãèªååã§ãããã«ã€ããŠã®ã¢ã€ãã¢ãäŒè°ããããŸããŸãªèŠæš¡ã®äŒæ¥ã§æ¯æ¥çãŸããŠããŸãã ãã ããã¢ãã«ã®äœæã«å€ãã®æéãè²»ããããããšã«å ããŠãã¢ãã«ãè©äŸ¡ããåŸãããçµæãã©ã³ãã ã§ã¯ãªãããšã確èªããããšã«ãæéãè²»ããå¿ èŠããããŸãã å°å ¥åŸã¯ãã©ã®ã¢ãã«ãç£èŠããå®æçã«ãã§ãã¯ããå¿ èŠããããŸãã
ãããã¯ãèŠæš¡ã«é¢ä¿ãªããã©ã®äŒæ¥ã§ãå®äºããå¿ èŠããããã¹ãŠã®æ®µéã§ãã ãºãã«ãã³ã¯ã®èŠæš¡ãšã¬ã¬ã·ãŒã«ã€ããŠè©±ããŠããå Žåã埮調æŽã®æ°ã¯å€§å¹ ã«å¢å ããŸãã 2019 幎æ«ãŸã§ã«ãSber ã¯ãã§ã« 2000 ãè¶ ããã¢ãã«ã䜿çšããŠããŸããã åã«ã¢ãã«ãéçºããã ãã§ã¯ååã§ã¯ãªããç£æ¥ã·ã¹ãã ãšçµ±åããã¢ãã«ãæ§ç¯ããããã®ããŒã¿ ããŒããéçºããã¯ã©ã¹ã¿ãŒäžã§ã®åäœã確å®ã«å¶åŸ¡ããå¿ èŠããããŸãã
ç§ãã¡ã®ããŒã 㯠Sber.DS ãã©ãããã©ãŒã ãéçºããŠããŸãã ããã«ãããæ©æ¢°åŠç¿ã®åé¡ã解決ãã仮説ããã¹ãããããã»ã¹ãã¹ããŒãã¢ããããåççã«ã¯ã¢ãã«ã®éçºãšæ€èšŒã®ããã»ã¹ãç°¡çŽ åããPROM å ã®ã¢ãã«ã®çµæãå¶åŸ¡ããããšãã§ããŸãã
çããã®æåŸ ãè£åããªãããã«ããã®æçš¿ã¯å ¥éçãªãã®ã§ããããšããããããèšã£ãŠãããŸãããŸããåºæ¬çã« Sber.DS ãã©ãããã©ãŒã ã®å éšã«ãããã®ã«ã€ããŠèª¬æããŸãã ã¢ãã«ã®äœæããå®è£ ãŸã§ã®ã©ã€ããµã€ã¯ã«ã«ã€ããŠã¯ãå¥éã話ããŸãã
Sber.DS ã¯ããã€ãã®ã³ã³ããŒãã³ãã§æ§æãããŠãããäž»èŠãªãã®ã¯ã©ã€ãã©ãªãéçºã·ã¹ãã ãã¢ãã«å®è¡ã·ã¹ãã ã§ãã
ãã®ã©ã€ãã©ãªã¯ãã¢ãã«ãéçºãããšããã¢ã€ãã¢ãæµ®ããã ç¬éãããPROM ã§ã®å®è£ ãç£èŠãããã³å»æ¢ã«è³ããŸã§ãã¢ãã«ã®ã©ã€ã ãµã€ã¯ã«ãå¶åŸ¡ããŸãã ã©ã€ãã©ãªã®æ©èœã®å€ãã¯ããã¬ãŒãã³ã°ããã³æ€èšŒãµã³ãã«ã®ã¬ããŒãäœæãä¿åãªã©ãèŠå¶åœå±ã®èŠåã«ãã£ãŠæ±ºãŸããŸãã å®éãããã¯åœç€Ÿã®ãã¹ãŠã®ã¢ãã«ã®ç»é²ã§ãã
éçºã·ã¹ãã ã¯ãã¢ãã«ãšæ€èšŒææ³ãèŠèŠçã«éçºã§ããããã«èšèšãããŠããŸãã éçºãããã¢ãã«ã¯åææ€èšŒãåããããžãã¹æ©èœãå®è¡ããããã«å®è¡ã·ã¹ãã ã«æäŸãããŸãã ãŸããã©ã³ã¿ã€ã ã·ã¹ãã ã§ã¯ãã¢ãã«ã®åäœãç£èŠããããã®æ€èšŒææ³ãå®æçã«èµ·åããç®çã§ãã¢ãã«ãã¢ãã¿ãŒäžã«é 眮ããããšãã§ããŸãã
ã·ã¹ãã ã«ã¯ããã€ãã®ã¿ã€ãã®ããŒãããããŸãã ããŸããŸãªããŒã¿ ãœãŒã¹ã«æ¥ç¶ããããã«èšèšãããŠãããã®ãããã°ããœãŒã¹ ããŒã¿ãå€æããŠåŒ·å (ããŒã¯ã¢ãã) ããããã«èšèšãããŠãããã®ããããŸãã ããŸããŸãªã¢ãã«ãæ§ç¯ããããã®ããŒããšãããããæ€èšŒããããã®ããŒããå€æ°ãããŸãã éçºè ã¯ãä»»æã®ãœãŒã¹ããããŒã¿ãããŒãããäžéããŒã¿ãå€æããã£ã«ã¿ãªã³ã°ãèŠèŠåãããããéšåã«åå²ã§ããŸãã
ãã©ãããã©ãŒã ã«ã¯ãèšèšé åã«ãã©ãã° ã¢ã³ã ããããã§ããæ¢è£œã®ã¢ãžã¥ãŒã«ãå«ãŸããŠããŸãã ãã¹ãŠã®ã¢ã¯ã·ã§ã³ã¯ãèŠèŠåãããã€ã³ã¿ãŒãã§ã€ã¹ã䜿çšããŠå®è¡ãããŸãã å®éãã³ãŒãã XNUMX è¡ãæžããã«åé¡ã解決ã§ããŸãã
çµã¿èŸŒã¿ã®æ©èœãååã§ãªãå Žåãã·ã¹ãã ã¯ç¬èªã®ã¢ãžã¥ãŒã«ãè¿
éã«äœæããæ©èœãæäŸããŸãã ã«åºã¥ããŠçµ±åéçºã¢ãŒããäœæããŸããã
Sber.DS ã®ã¢ãŒããã¯ãã£ã¯ãã€ã¯ããµãŒãã¹ã«åºã¥ããŠæ§ç¯ãããŠããŸãã ãã€ã¯ããµãŒãã¹ãšã¯äœãã«ã€ããŠã¯å€ãã®æèŠããããŸãã ã¢ããªã·ãã¯ãªã³ãŒããããã€ãã®éšåã«åå²ããã ãã§ååã ãšèãã人ãããŸãããåæã«åãããŒã¿ããŒã¹ã«ã¢ã¯ã»ã¹ããŸãã ãã€ã¯ããµãŒãã¹ã¯ãREST API ãä»ããŠã®ã¿å¥ã®ãã€ã¯ããµãŒãã¹ãšéä¿¡ããå¿ èŠããããŸãã ããŒã¿ããŒã¹ã«çŽæ¥ã¢ã¯ã»ã¹ããåé¿çã¯ãããŸããã
ç§ãã¡ã¯ããµãŒãã¹ãéåžžã«å€§ãããªããæ±ãã«ãããªããªãããã«åªããŠããŸãã4 ã€ã®ã€ã³ã¹ã¿ã³ã¹ã 8 ïœ XNUMX GB ãè¶
ãã RAM ãæ¶è²»ããŠã¯ãªãããæ°ããã€ã³ã¹ã¿ã³ã¹ãèµ·åããŠãªã¯ãšã¹ããæ°Žå¹³ã«ã¹ã±ãŒãªã³ã°ããæ©èœãæäŸããå¿
èŠããããŸãã åãµãŒãã¹ã¯ REST API çµç±ã§ã®ã¿ä»ã®ãµãŒãã¹ãšéä¿¡ããŸã (
ã¢ããªã±ãŒã·ã§ã³ã®ã³ã¢ã¯ãSpring Framework ã䜿çšã㊠Java ã§äœæãããŸãã ãã®ãœãªã¥ãŒã·ã§ã³ã¯åœåãã¯ã©ãŠã ã€ã³ãã©ã¹ãã©ã¯ãã£ã«è¿
éã«å°å
¥ã§ããããã«èšèšãããŠãããããã¢ããªã±ãŒã·ã§ã³ã¯ã³ã³ããåã·ã¹ãã ã䜿çšããŠæ§ç¯ãããŸããã
ç§ãã¡ã®ãã©ãããã©ãŒã ã®ç¹åŸŽã® XNUMX ã€ã¯ãããžã¥ã¢ã« ã€ã³ã¿ãŒãã§ã€ã¹ã§éçºãããã³ãŒããä»»æã® Sberbank ã¢ãã«å®è¡ã·ã¹ãã äžã§å®è¡ã§ããããšã§ãã çŸåšããããã¯ãã§ã« XNUMX ã€ãããŸããXNUMX ã€ã¯ Hadoop äžã«ããã XNUMX ã€ã¯ OpenShift (Docker) äžã«ãããŸãã ç§ãã¡ã¯ããã«ãšã©ãŸããããªã³ãã¬ãã¹ãã¯ã©ãŠããå«ãããããã€ã³ãã©ã¹ãã©ã¯ãã£äžã§ã³ãŒããå®è¡ããããã®çµ±åã¢ãžã¥ãŒã«ãäœæããŸãã ãºãã«ãã³ã¯ã®ãšã³ã·ã¹ãã ãžã®å¹æçãªçµ±åã®å¯èœæ§ã«é¢ããŠã¯ãæ¢åã®å®è¡ç°å¢ãšã®é£æºããµããŒãããäºå®ã§ãã å°æ¥çã«ã¯ããã®ãœãªã¥ãŒã·ã§ã³ã¯ãããã«äœ¿ãããããããçµç¹ã®ããããç¶æ³ã«æè»ã«çµ±åã§ããããã«ãªããŸãã
PROM ã® Hadoop äžã§ Python ãå®è¡ãããœãªã¥ãŒã·ã§ã³ããµããŒãããããšããããšã®ãã人ã¯ãPython ãŠãŒã¶ãŒç°å¢ãæºåããŠåããŒã¿ããŒãã«é ä¿¡ããã ãã§ã¯ååã§ã¯ãªãããšãç¥ã£ãŠããŸãã Python ã¢ãžã¥ãŒã«ã䜿çšããæ©æ¢°åŠç¿çšã® C/C++ ã©ã€ãã©ãªã¯èšå€§ãªæ°ã«ãããããå®å¿ããããšã¯ã§ããŸããã æ°ããã©ã€ãã©ãªããµãŒããŒãè¿œå ãããšãã¯ãæ¢ã«å®è£ ãããŠããã¢ãã« ã³ãŒããšã®äžäœäºææ§ãç¶æããªãããå¿ããã«ããã±ãŒãžãæŽæ°ããå¿ èŠããããŸãã
ãããè¡ãæ¹æ³ã«ã¯ããã€ãã®æ¹æ³ããããŸãã äŸãã°ããã䜿ãã©ã€ãã©ãªããããããããã€ãçšæããŠPROMã«å®è£
ããŠãããŸãã Cloudera ã® Hadoop ãã£ã¹ããªãã¥ãŒã·ã§ã³ã§ã¯ãéåžžã
ãã®éè¡ã¯ããµãŒãããŒã㣠ã³ãŒããå®è¡ããéã®ã»ãã¥ãªãã£ãéåžžã«éèŠããŠãããããããã»ã¹ãåé¢ãããç°å¢ã§å®è¡ããã Linux ã«ãŒãã«ã®æ°æ©èœãæ倧éã«æŽ»çšããŠããŸãã
ä»å¹Žã¯ãHadoop äžã§ Python/R/Java ã§æžãããã¢ãã«ãèµ·åãããšãã MVP ãå®äºããäºå®ã§ãã ç§ãã¡ã¯ããã©ãããã©ãŒã ã®ãŠãŒã¶ãŒããããªã圢ã§ãå¶éããªãããã«ãHadoop äžã§ã«ã¹ã¿ã ç°å¢ãå®è¡ããæ¹æ³ãåŠã¶ãšããéå¿çãªèª²é¡ãèªãã«èª²ããŸããã
ããã«ãå€ãã® DS ã¹ãã·ã£ãªã¹ãã¯æ°åŠãšçµ±èšã«åªããã¯ãŒã«ãªã¢ãã«ãäœæããŸãããããã° ããŒã¿å€æã«ã¯ããŸã粟éããŠãããããã¬ãŒãã³ã° ãµã³ãã«ãæºåããããã«ããŒã¿ ãšã³ãžãã¢ã®å©ããå¿ èŠãšããŠããããšãå€æããŸããã ç§ãã¡ã¯ååãå©ããSpark ãšã³ãžã³äžã®ã¢ãã«ã®æšæºå€æãšæ©èœã®æºåã«äŸ¿å©ãªã¢ãžã¥ãŒã«ãäœæããããšã«ããŸããã ããã«ãããããŒã¿ ãšã³ãžãã¢ãæ°ããããŒã¿ã»ãããæºåããã®ãåŸ ããã«ãã¢ãã«ã®éçºã«ããå€ãã®æéãè²»ããããšãã§ããŸãã
åœç€Ÿã§ã¯ãLinux ãš DevOpsãHadoop ãš SparkãJava ãš SpringãScala ãš AkkaãOpenShift ãš Kubernetes ãªã©ãããŸããŸãªåéã®ç¥èãæã€äººæãéçšããŠããŸãã 次åã¯ãã¢ãã« ã©ã€ãã©ãªãã¢ãã«ã瀟å ã§ã©ã€ããµã€ã¯ã«ãã©ã®ããã«ééããããæ€èšŒãšå®è£ ãã©ã®ããã«è¡ããããã«ã€ããŠèª¬æããŸãã
åºæïŒ habr.com