Lub lim tiam dhau los kuv tau mus rau DUMP IT lub rooj sib tham (https://dump-ekb.ru/) hauv Yekaterinburg thiab kuv xav qhia koj txog dab tsi tau tham txog hauv Backend thiab Devops, thiab seb lub rooj sib tham hauv cheeb tsam IT puas tsim nyog mloog.
Nikolay Sverchkov los ntawm Evil Martians txog Serverless
Puas muaj dabtsi?
Nyob rau hauv tag nrho, lub rooj sib tham muaj 8 seem: Backend, Frontend, Mobile, Testing thiab QA, Devops, Tsim, Science thiab Management.
Cov chav loj tshaj plaws, los ntawm txoj kev, yog ntawm Science thiab Kev Tswj Xyuas)) Rau ~ 350 tus neeg txhua. Backend thiab Frontend tsis ntau me dua. Chav Devops yog qhov tsawg tshaj plaws, tab sis nquag.
Kuv tau mloog cov lus ceeb toom hauv ntu Devops thiab Backend thiab tham me ntsis nrog cov neeg hais lus. Kuv xav tham txog cov ncauj lus uas tau hais thiab tshuaj xyuas cov ntu no ntawm lub rooj sab laj.
Cov neeg sawv cev ntawm SKB-Kontur, DataArt, Evil Martians, Ekaterinburg web studio Flag, Miro (RealTimeBoard) tau hais hauv Devops thiab Backend seem. Cov ntsiab lus suav nrog CI / CD, ua haujlwm nrog cov kev pabcuam hauv kab, kev nkag; cov ntsiab lus Serverless thiab ua haujlwm nrog PostgreSQL hauv Go tau txais txiaj ntsig zoo.
Kuj tseem muaj cov lus ceeb toom los ntawm Avito, Tinkoff, Yandex, Jetstyle, Megafon, Ak Bars Bank, tab sis kuv tsis muaj sijhawm los koom nrog lawv lub cev (cov yeeb yaj kiab thiab cov slides ntawm cov ntawv ceeb toom tseem tsis tau muaj, lawv cog lus tias yuav tshaj tawm hauv 2 lub lis piam ntawm dump-ekb.ru).
Devops ntu
Dab tsi yog qhov xav tsis thoob yog tias ntu tau tuav hauv lub tsev me tshaj plaws, txog 50 lub rooj zaum. Cov neeg txawm sawv hauv txoj kev :) Kuv yuav qhia koj txog cov lus ceeb toom uas kuv tau mloog.
Elastic hnyav ib petabyte
Tshooj pib nrog tsab ntawv ceeb toom los ntawm Vladimir Lil (SKB-Kontur) txog Elasticsearch hauv Kontur. Lawv muaj qhov loj loj thiab thauj khoom Elastic (~ 800 TB ntawm cov ntaub ntawv, ~ 1.3 petabytes suav rau hauv tus account redundancy). Elasticsearch rau tag nrho cov kev pabcuam Kontur yog ib leeg, muaj 2 pawg (ntawm 7 thiab 9 servers), thiab tseem ceeb heev uas Kontur muaj tus kws tshaj lij Elasticsearch tshwj xeeb (qhov tseeb, Vladimir nws tus kheej).
Vladimir kuj tau qhia nws txoj kev xav txog cov txiaj ntsig ntawm Elasticsearch thiab cov teeb meem uas nws coj.
Cov txiaj ntsig:
- Tag nrho cov cav yog nyob rau hauv ib qho chaw, yooj yim rau lawv
- Khaws cov cav rau ib xyoos thiab yooj yim txheeb xyuas lawv
- Kev kub ceev ntawm kev ua haujlwm nrog cov cav
- Txias cov ntaub ntawv visualization tawm ntawm lub thawv
Teeb meem:
- lus broker yog ib qho yuav tsum muaj (rau Kontur nws lub luag haujlwm yog ua los ntawm Kafka)
- nta ntawm kev ua hauj lwm nrog Elasticsearch Curator (ib ntus tsim siab load los ntawm cov dej num tsis tu ncua hauv Curator)
- tsis muaj kev tso cai nyob rau hauv (tsuas yog rau kev sib cais, nyiaj loj heev, los yog qhib qhov plugins ntawm ntau qib ntawm kev npaj rau kev tsim khoom)
Tsuas muaj kev tshuaj xyuas zoo txog Qhib Distro rau Elasticsearch :) Tib qhov teeb meem ntawm kev tso cai tau raug daws nyob rau ntawd.
Petabyte tuaj qhov twg?Lawv cov nodes muaj cov servers nrog 12 * 8 Tb SATA + 2 * 2 Tb SSD. Txias cia ntawm SATA, SSD tsuas yog rau kub cache (kub cia).
7 + 9 servers, (7 + 9) * 12 * 8 = 1536 Tb.
Ib feem ntawm qhov chaw yog nyob rau hauv reserve, teem tseg rau redundancy, thiab lwm yam.
Cov ntaub ntawv los ntawm txog 90 daim ntawv thov raug xa mus rau Elasticsearch, suav nrog txhua qhov kev qhia qhia ntawm Kontur, Elba, thiab lwm yam.
Nta ntawm kev txhim kho ntawm Serverless
Tom ntej no yog tsab ntawv ceeb toom los ntawm Ruslan Serkin los ntawm DataArt txog Serverless.
Ruslan tham txog qhov kev txhim kho nrog Serverless txoj hauv kev yog qhov dav, thiab nws cov nta yog dab tsi.
Serverless yog ib txoj hauv kev rau kev txhim kho uas cov neeg tsim khoom tsis kov cov txheej txheem hauv txhua txoj kev. Piv txwv - AWS Lambda Serverless, Kubeless.io (Serverless hauv Kubernetes), Google Cloud Functions.
Ib daim ntawv thov Serverless zoo tagnrho yog ib qho haujlwm uas xa cov lus thov mus rau Serverless tus neeg muab kev pabcuam los ntawm API Gateway tshwj xeeb. Ib qho microservice zoo tagnrho, thaum AWS Lambda kuj txhawb nqa ntau hom lus niaj hnub programming. Tus nqi ntawm kev tswj hwm thiab kev siv cov cuab yeej cuab tam yuav raug xoom nyob rau hauv cov ntaub ntawv ntawm cov chaw muab huab, kev txhawb nqa cov ntawv thov me me kuj tseem pheej yig heev (AWS Lambda - $ 0.2 / 1 lab thov yooj yim).
Lub scalability ntawm xws li ib tug system yog yuav luag zoo tagnrho - cov huab muab saib xyuas ntawm no nws tus kheej, Kubeless scales txiav nyob rau hauv lub Kubernetes pawg.
Muaj qhov tsis zoo:
- tsim cov ntawv thov loj yog qhov nyuaj dua
- muaj teeb meem nrog cov ntawv thov profile (tsuas yog cov cav muaj rau koj, tab sis tsis yog profileing hauv kev xav ib txwm muaj)
- tsis muaj versioning
Ua ncaj ncees, Kuv tau hnov ββββtxog Serverless ob peb xyoos dhau los, tab sis tag nrho cov xyoo no nws tsis meej rau kuv siv nws kom raug. Tom qab Ruslan tsab ntawv ceeb toom, kev nkag siab tau tshwm sim, thiab tom qab tsab ntawv ceeb toom ntawm Nikolai Sverchkov (Evil Martians) los ntawm Backend seem, nws tau sib sau ua ke. Nws tsis yog qhov tsis muaj nuj nqis uas kuv tau mus rau lub rooj sib tham :)
CI yog rau cov neeg pluag, lossis nws puas tsim nyog sau koj tus kheej CI rau lub vev xaib studio?
Mikhail Radionov, tus thawj coj ntawm Flag web studio los ntawm Yekaterinburg, tau hais txog tus kheej sau CI / CD.
Nws lub studio tau ploj mus los ntawm "phau ntawv CI / CD" (nkag mus rau hauv lub server ntawm SSH, ua git rub, rov ua dua 100 zaug hauv ib hnub) rau Jenkins thiab rau tus kheej sau cov cuab yeej uas tso cai rau koj saib xyuas cov cai thiab ua cov ntawv tshaj tawm hu ua Pullkins .
Vim li cas Jenkins tsis ua haujlwm? Nws tsis tau muab kev hloov pauv txaus los ntawm lub neej ntawd thiab nyuaj dhau los kho.
"Flag" txhim kho hauv Laravel (PHP moj khaum). Thaum tsim lub CI / CD server, Mikhail thiab nws cov npoj yaig siv Laravel's built-in mechanisms hu ua Telescope thiab Envoy. Qhov tshwm sim yog ib tus neeg rau zaub mov hauv PHP (thov nco ntsoov) uas ua cov kev thov webhook tuaj, tuaj yeem tsim cov frontend thiab backend, xa mus rau cov servers sib txawv, thiab tshaj tawm rau Slack.
Tom qab ntawd, kom muaj peev xwm ua tau xiav / ntsuab xa mus thiab muaj cov teeb tsa tsis sib xws hauv dev-theem-prod ib puag ncig, lawv tau hloov mus rau Docker. Cov txiaj ntsig tseem zoo li qub, qhov muaj peev xwm ntawm homogenizing ib puag ncig thiab kev xa tawm seamless tau ntxiv, thiab qhov yuav tsum tau kawm Docker ua haujlwm nrog nws raug raug ntxiv.
Yuav ua li cas peb txo tus naj npawb ntawm cov neeg rau zaub mov tso tawm rollbacks los ntawm 99%
Daim ntawv tshaj tawm kawg hauv ntu Devops yog los ntawm Viktor Eremchenko, Lead devops engineer ntawm Miro.com (yav tas los RealTimeBoard).
RealTimeBoard, pab pawg Miro cov khoom lag luam tseem ceeb, yog raws li daim ntawv thov Java monolithic. Kev sau, sim thiab siv nws yam tsis muaj sijhawm poob yog txoj haujlwm nyuaj. Nyob rau hauv cov ntaub ntawv no, nws yog ib qho tseem ceeb rau deploy xws li ib tug version ntawm txoj cai kom nws tsis tas yuav dov rov qab (nws yog ib tug hnyav monolith).
Nyob rau hauv txoj kev los tsim ib lub kaw lus uas tso cai rau koj ua qhov no, Miro tau dhau los ntawm txoj hauv kev uas suav nrog kev ua haujlwm ntawm architecture, cov cuab yeej siv (Atlassian Bamboo, Ansible, thiab lwm yam), thiab ua haujlwm ntawm cov qauv ntawm pawg (lawv tam sim no muaj. pab pawg Devops nplooj siab + ntau pawg Scrum cais los ntawm cov neeg tsim khoom sib txawv).
Txoj kev tau hloov mus ua nyuaj thiab muaj pos, thiab Victor tau qhia txog qhov mob thiab kev cia siab uas tsis tau xaus rau ntawd.
Yeej ib phau ntawv nug cov lus nug
Backend seem
Kuv tau mus koom 2 tsab ntawv ceeb toom - los ntawm Nikolay Sverchkov (Evil Martians), kuj hais txog Serverless, thiab los ntawm Grigory Koshelev (Kontur tuam txhab) txog telemetry.
Serverless rau mere mortals
Yog tias Ruslan Sirkin tham txog dab tsi Serverless yog, Nikolay tau qhia cov ntawv thov yooj yim siv Serverless, thiab tham txog cov ntsiab lus uas cuam tshuam rau tus nqi thiab kev ceev ntawm cov ntawv thov hauv AWS Lambda.
Cov ntsiab lus nthuav dav: qhov tsawg kawg them lub caij yog 128 Mb ntawm lub cim xeeb thiab 100 ms CPU, nws raug nqi $ 0,000000208. Ntxiv mus, 1 lab qhov kev thov no hauv ib hlis yog pub dawb.
Qee qhov ntawm Nikolai txoj haujlwm feem ntau tshaj li 100 ms txwv (cov ntawv thov tseem ceeb tau sau hauv Ruby), yog li rov sau lawv hauv Go muab kev txuag nyiaj zoo.
Vostok Hercules - ua telemetry zoo dua!
Daim ntawv tshaj tawm tshiab ntawm Backend seem los ntawm Grigory Koshelev (Kontur tuam txhab) txog telemetry. Telemetry txhais tau tias cov cav, ntsuas, daim ntawv thov kab.
Rau lub hom phiaj no, Contour siv cov cuab yeej sau tus kheej tso rau ntawm Github. Cov cuab yeej los ntawm tsab ntawv ceeb toom - Hercules, , yog siv los xa cov ntaub ntawv telemetry.
Vladimir Lila tsab ntawv ceeb toom nyob rau hauv Devops seem tau tham txog kev khaws cia thiab ua cov ntaub ntawv hauv Elasticsearch, tab sis tseem muaj txoj haujlwm xa cov cav los ntawm ntau txhiab lub cuab yeej thiab cov ntawv thov, thiab cov cuab yeej zoo li Vostok Hercules daws lawv.
Lub voj voog ua raws txoj kev paub rau ntau tus - los ntawm RabbitMQ mus rau Apache Kafka, tab sis tsis yog txhua yam yooj yim heev)) Lawv yuav tsum ntxiv Zookeeper, Cassandra thiab Graphite rau hauv Circuit Court. Kuv yuav tsis nthuav tawm tag nrho cov ntaub ntawv ntawm daim ntawv tshaj tawm no (tsis yog kuv qhov profile), yog tias koj txaus siab, koj tuaj yeem tos cov slides thiab cov yeeb yaj kiab ntawm lub vev xaib sablaj.
Nws piv rau lwm lub rooj sab laj li cas?
Kuv tsis tuaj yeem piv nws nrog cov rooj sib tham hauv Moscow thiab St. Petersburg, Kuv tuaj yeem sib piv nrog lwm cov xwm txheej hauv Urals thiab nrog 404fest hauv Samara.
DAMP tau tuav hauv 8 ntu, qhov no yog cov ntaub ntawv rau Ural cov rooj sib tham. Loj heev Science thiab Management seem, qhov no kuj txawv. Cov neeg tuaj saib nyob rau hauv Yekaterinburg yog cov qauv zoo heev - lub nroog muaj cov chaw loj hlob loj rau Yandex, Kontur, Tinkoff, thiab qhov no tawm ntawm nws cov cim ntawm cov ntawv ceeb toom.
Lwm qhov nthuav qhia yog tias ntau lub tuam txhab muaj 3-4 tus neeg hais lus ntawm lub rooj sib tham ib zaug (qhov no yog qhov teeb meem nrog Kontur, Evil Martians, Tinkoff). Ntau tus ntawm lawv yog cov neeg txhawb nqa, tab sis cov ntawv tshaj tawm zoo sib xws nrog lwm tus, cov no tsis yog tshaj tawm tshaj tawm.
Mus los tsis mus? Yog tias koj nyob hauv Urals lossis nyob ze, koj muaj sijhawm thiab txaus siab rau cov ncauj lus - yog, tau kawg. Yog tias koj tab tom xav txog kev mus ncig ntev, kuv yuav saib cov ntsiab lus ntawm cov lus ceeb toom thiab cov ntawv tshaj tawm video los ntawm xyoo dhau los thiab txiav txim siab.
Lwm qhov zoo ntawm cov rooj sib tham hauv cheeb tsam, raws li txoj cai, yog qhov yooj yim rau kev sib txuas lus nrog tus neeg hais lus tom qab cov lus ceeb toom; tsuas muaj tsawg dua cov neeg thov kev sib txuas lus.
Ua tsaug rau Dump thiab Ekaterinburg! )
Tau qhov twg los: www.hab.com