ʻO ka pule i hala aku nei ua hele au i ka ʻaha kūkā DUMP IT (https://dump-ekb.ru/) ma Yekaterinburg a makemake wau e haʻi iā ʻoe i ka mea i kūkākūkā ʻia ma nā ʻāpana Backend a me Devops, a inā pono e nānā ʻia nā hālāwai IT kūloko.
Nikolay Sverchkov mai Evil Martians e pili ana i Serverless
He aha naʻe ma laila?
I ka huina, he 8 mau ʻāpana ka hālāwai: Backend, Frontend, Mobile, Testing and QA, Devops, Design, Science and Management.
ʻO nā keʻena nui loa, ma ke ala, aia ma Science and Management)) No ~ 350 poʻe i kēlā me kēia. ʻAʻole ʻoi aku ka liʻiliʻi o Backend a me Frontend. ʻO ka lumi Devops ka liʻiliʻi loa, akā ikaika.
Ua hoʻolohe au i nā hōʻike ma nā ʻāpana Devops a me Backend a kamaʻilio liʻiliʻi me nā mea haʻiʻōlelo. Makemake wau e kamaʻilio e pili ana i nā kumuhana i uhi ʻia a nānā i kēia mau ʻāpana ma ka ʻaha kūkā.
Ua ʻōlelo nā Lunamakaʻāinana o SKB-Kontur, DataArt, Evil Martians, Ekaterinburg web studio Flag, Miro (RealTimeBoard) ma nā ʻāpana Devops a me Backend. Ua uhi ʻia nā kumuhana iā CI/CD, hana pū me nā lawelawe queue, logging; Ua uhi maikaʻi ʻia nā kumuhana Serverless a me ka hana pū me PostgreSQL ma Go.
Aia kekahi mau hōʻike e Avito, Tinkoff, Yandex, Jetstyle, Megafon, Ak Bars Bank, akā ʻaʻohe oʻu manawa e hele kino ai iā lākou (ʻaʻole i loaʻa nā wikiō wikiō a me nā kiʻi paheʻe o nā hōʻike, hoʻohiki lākou e kau iā lākou i loko o 2 mau pule. ma ka dump-ekb.ru).
ʻāpana Devops
ʻO ka mea kupanaha, ua mālama ʻia ka ʻāpana ma ke keʻena liʻiliʻi loa, ma kahi o 50 mau noho. Ke kū nei ka poʻe ma nā aisles :) E haʻi wau iā ʻoe e pili ana i nā hōʻike aʻu i hoʻolohe ai.
Elastic ke kaupaona ʻana i kahi petabyte
Ua hoʻomaka ka ʻāpana me kahi hōʻike a Vladimir Lil (SKB-Kontur) e pili ana iā Elasticsearch ma Kontur. Loaʻa iā lākou kahi Elastic nui a hoʻouka ʻia (~ 800 TB o ka ʻikepili, ~ 1.3 petabytes e noʻonoʻo ana i ka redundancy). Hoʻokahi wale nō ʻo Elasticsearch no nā lawelawe Kontur a pau, aia nā pūʻulu 2 (ʻo 7 a me 9 mau kikowaena), a he mea nui loa i loaʻa iā Kontur kahi ʻenekini Elasticsearch kūikawā (ʻoiaʻiʻo, ʻo Vladimir ponoʻī).
Ua haʻi pū ʻo Vladimir i kona mau manaʻo i nā pono o Elasticsearch a me nā pilikia e lawe mai ai.
Pōmaikaʻi:
- Aia nā lāʻau a pau ma kahi hoʻokahi, hiki ke maʻalahi iā lākou
- Ka mālama ʻana i nā lāʻau no hoʻokahi makahiki a maʻalahi ka nānā ʻana iā lākou
- ʻO ka wikiwiki kiʻekiʻe o ka hana me nā lāʻau
- ʻIke ʻikepili maikaʻi ma waho o ka pahu
Nā pilikia:
- Pono e loaʻa i ka mea kūʻai leka uila (no Kontur kāna kuleana e hoʻokani ʻia e Kafka)
- nā hiʻohiʻona o ka hana ʻana me Elasticsearch Curator (i hana ʻia i nā manawa kiʻekiʻe mai nā hana maʻamau i Curator)
- ʻaʻohe ʻae i kūkulu ʻia (no ke kālā kaʻawale, ʻoi aku ka nui, a i ʻole ma ke ʻano he open source plugins o nā pae like ʻole o ka mākaukau no ka hana ʻana)
He mau manaʻo maikaʻi wale nō e pili ana i Open Distro no Elasticsearch :) Ua hoʻoholo ʻia ka manaʻo like ʻole ma laila.
No hea mai ka petabyte?ʻO kā lākou mau nodes he mau kikowaena me 12 * 8 Tb SATA + 2 * 2 Tb SSD. ʻO ka mālama anuanu ma SATA, SSD wale nō no ka hūnā wela (hoʻahu wela).
7+9 mau kikowaena, (7 + 9) * 12 * 8 = 1536 Tb.
Aia kekahi hapa o ka hakahaka, waiho ʻia no ka redundancy, etc.
Hoʻouna ʻia nā logs mai kahi o 90 mau noi i Elasticsearch, me nā lawelawe hōʻike a pau o Kontur, Elba, etc.
Nā hiʻohiʻona o ka hoʻomohala ʻana ma Serverless
ʻO ka mea aʻe kahi hōʻike e Ruslan Serkin mai DataArt e pili ana i Serverless.
Ua kamaʻilio ʻo Ruslan e pili ana i ke ʻano o ka hoʻomohala ʻana me ke ala Serverless ma ka laulā, a me kāna mau hiʻohiʻona.
ʻO Serverless kahi ala i ka hoʻomohala ʻana i ʻole e hoʻopā nā mea hoʻomohala i nā ʻōnaehana ma kekahi ʻano. Laʻana - AWS Lambda Serverless, Kubeless.io (Serverless inside Kubernetes), Google Cloud Functions.
ʻO kahi noi Serverless kūpono he hana wale nō ia e hoʻouna i kahi noi i kahi mea lawelawe Serverless ma o kahi API Gateway kūikawā. ʻO kahi microservice maikaʻi, ʻoiai ʻo AWS Lambda e kākoʻo ana i kahi helu nui o nā ʻōlelo papahana hou. ʻO ke kumukūʻai o ka mālama ʻana a me ka hoʻohana ʻana i nā ʻoihana e lilo i zero i ka hihia o nā mea hāʻawi kapuaʻi, ʻo ke kākoʻo ʻana i nā noi liʻiliʻi e uku nui ʻia (AWS Lambda - $0.2 / 1 miliona mau noi maʻalahi).
ʻO ka scalability o ia ʻōnaehana ʻaneʻane kūpono loa - na ka mea hāʻawi kapua e mālama iā ia iho, ʻo Kubeless unahi ʻakomi i loko o ka hui Kubernetes.
Aia nā hemahema:
- ʻoi aku ka paʻakikī o ka hoʻomohala ʻana i nā noi nui
- aia ka paʻakikī me nā noi profiling (ʻo nā lāʻau wale nō i loaʻa iā ʻoe, akā ʻaʻole ka profiling ma ke ʻano maʻamau)
- ʻaʻohe hoʻololi
ʻO kaʻoiaʻiʻo, ua lohe au e pili ana iā Serverless i kekahi mau makahiki i hala aku nei, akā i kēia mau makahiki āpau ʻaʻole maopopo iaʻu pehea e hoʻohana pono ai. Ma hope o ka hōʻike a Ruslan, ʻike ʻia ka ʻike, a ma hope o ka hōʻike a Nikolai Sverchkov (Evil Martians) mai ka ʻāpana Backend, ua hoʻohui ʻia. ʻAʻole makehewa koʻu hele ʻana i ka ʻaha kūkā :)
ʻO CI no ka poʻe ʻilihune, a i ʻole pono ke kākau ʻana i kāu CI ponoʻī no kahi studio pūnaewele?
Ua kamaʻilio ʻo Mikhail Radionov, ke poʻo o ka hale kiʻi pūnaewele Flag mai Yekaterinburg, e pili ana iā CI/CD i kākau ponoʻī.
Ua hele kāna studio mai "manual CI / CD" (e komo i loko o ka server ma o SSH, e hana i kahi git pull, e hana hou i 100 mau manawa i ka lā) iā Jenkins a i kahi mea hana ponoʻī e hiki ai iā ʻoe ke nānā i ke code a hana i nā hoʻokuʻu i kapa ʻia ʻo Pullkins .
No ke aha i hana ʻole ai ʻo Jenkins? ʻAʻole lawa ka maʻalahi ma ka paʻamau a paʻakikī loa ka hana ʻana.
Hoʻokumu ʻia ka "Hae" ma Laravel (pūnaewele PHP). I ka hoʻomohala ʻana i kahi kikowaena CI/CD, ua hoʻohana ʻo Mikhail a me kāna mau hoa hana i nā hana i kūkulu ʻia ʻo Laravel i kapa ʻia ʻo Telescope a me Envoy. ʻO ka hopena he kikowaena ma PHP (e ʻoluʻolu e hoʻomaopopo) e hana ana i nā noi webhook e hiki mai ana, hiki ke kūkulu i ka frontend a me ka hope, hoʻoili i nā kikowaena like ʻole, a hōʻike iā Slack.
A laila, i hiki ke hana i ka blue/green deploy a loaʻa nā hoʻonohonoho like ʻole i nā wahi dev-stage-prod, ua hoʻololi lākou iā Docker. Ua hoʻomau ʻia nā pōmaikaʻi, ua hoʻohui ʻia nā mea hiki ke homogenizing i ke kaiapuni a me ka hoʻopili ʻole ʻana, a ua hoʻohui ʻia ka pono e aʻo iā Docker e hana me ia.
Pehea mākou i hōʻemi ai i ka helu o ka hoʻokuʻu ʻana o ka server e 99%
ʻO ka hōʻike hope loa ma ka ʻāpana Devops mai Viktor Eremchenko, Lead devops engineer ma Miro.com (RealTimeBoard ma mua).
ʻO RealTimeBoard, ka huahana hae o ka hui Miro, ua hoʻokumu ʻia ma kahi noi Java monolithic. ʻO ka hōʻiliʻili, hoʻāʻo a me ka hoʻohana ʻana me ka ʻole o ka downtime he hana paʻakikī. I kēia hihia, he mea nui e kau i kēlā ʻano o ke code i ʻole e ʻōwili ʻia i hope (he monolith kaumaha).
Ma ke ala e kūkulu ai i kahi ʻōnaehana e hiki ai iā ʻoe ke hana i kēia, ua hele ʻo Miro ma ke ala e komo pū ana me ka hana ʻana i ka hale hoʻolālā, nā mea hana i hoʻohana ʻia (Atlassian Bamboo, Ansible, etc.), a me ka hana ʻana i ke ʻano o nā hui (loaʻa iā lākou i kēia manawa. he hui Devops i hoʻolaʻa ʻia + he nui nā hui Scrum kaʻawale mai nā mea hoʻomohala o nā ʻaoʻao like ʻole).
Ua paʻakikī a paʻakikī ke ala, a ua hui ʻo Victor i ka ʻeha a me ka manaʻo maikaʻi ʻaʻole i pau i laila.
Ua lanakila i puke no ka nīnau nīnau
ʻāpana hope
Ua hiki iaʻu ke hele i nā hōʻike 2 - mai Nikolay Sverchkov (Evil Martians), no Serverless, a mai Grigory Koshelev (Kontur hui) e pili ana i ka telemetry.
Serverless no na kanaka make
Inā kamaʻilio ʻo Ruslan Sirkin e pili ana i ka Serverless, ua hōʻike ʻo Nikolay i nā noi maʻalahi me ka hoʻohana ʻana i Serverless, a kamaʻilio e pili ana i nā kikoʻī e pili ana i ke kumukūʻai a me ka wikiwiki o nā noi ma AWS Lambda.
ʻO kahi kikoʻī hoihoi: ʻo ka hapa liʻiliʻi i uku ʻia ʻo 128 Mb o ka hoʻomanaʻo a me 100 ms CPU, he $0,000000208. Eia kekahi, ʻaʻole manuahi ka 1 miliona mau noi i kēlā me kēia mahina.
ʻO kekahi o nā hana a Nikolai i ʻoi aku ma mua o ka palena 100 ms (ua kākau ʻia ka palapala noi nui ma Ruby), no laila ke kākau hou ʻana iā lākou ma Go e hāʻawi i kahi kālā maikaʻi loa.
ʻO Vostok Hercules - hana hou i ka telemetry maikaʻi!
ʻO ka hōʻike hou loa o ka ʻāpana Backend mai Grigory Koshelev (hui Kontur) e pili ana i ka telemetry. Telemetry 'o ia ho'i nā logs, metrics, track traces.
No kēia kumu, hoʻohana ʻo Contour i nā mea hana ponoʻī i kau ʻia ma Github. Mea hana mai ka hōʻike - Hercules, , hoʻohana ʻia e hāʻawi i ka ʻikepili telemetry.
ʻO ka hōʻike a Vladimir Lila ma ka ʻāpana Devops i kūkākūkā i ka mālama ʻana a me ka hoʻoponopono ʻana i nā lāʻau ma Elasticsearch, akā aia nō ka hana o ka hāʻawi ʻana i nā lāʻau mai nā tausani o nā mea hana a me nā noi, a me nā mea hana e like me Vostok Hercules e hoʻoponopono iā lākou.
Ua hahai ke kaapuni i kahi ala i ʻike ʻia e nā mea he nui - mai RabbitMQ a Apache Kafka, akā ʻaʻole maʻalahi nā mea a pau)) Pono lākou e hoʻohui iā Zookeeper, Cassandra a me Graphite i ke kaapuni. ʻAʻole wau e hōʻike piha i ka ʻike ma kēia hōʻike (ʻaʻole koʻu ʻaoʻao), inā makemake ʻoe, hiki iā ʻoe ke kali i nā paheʻe a me nā wikiō ma ka pūnaewele kūkā.
Pehea e hoʻohālikelike ai i nā ʻaha kūkā ʻē aʻe?
ʻAʻole hiki iaʻu ke hoʻohālikelike me nā hālāwai kūkā ma Moscow a me St. Petersburg, hiki iaʻu ke hoʻohālikelike me nā hanana ʻē aʻe ma nā Urals a me 404fest ma Samara.
Hoʻopaʻa ʻia ʻo DAMP ma nā ʻāpana 8, he moʻolelo kēia no nā hālāwai Ural. ʻO nā ʻāpana ʻepekema a me Management nui loa, he mea maʻamau kēia. Hoʻonohonoho ʻia ka lehulehu ma Yekaterinburg - he mau keʻena hoʻomohala nui ke kūlanakauhale o Yandex, Kontur, Tinkoff, waiho kēia i kāna hōʻailona ma nā hōʻike.
ʻO kekahi mea hoihoi ʻo ia ka nui o nā hui he 3-4 mau ʻōlelo ma ka hālāwai kūkā i ka manawa hoʻokahi (ʻo ia ka hihia me Kontur, Evil Martians, Tinkoff). He poʻe kākoʻo ka nui o lākou, akā ua kūlike nā hōʻike me nā poʻe ʻē aʻe, ʻaʻole ia he hōʻike hoʻolaha.
E hele a ʻaʻole paha e hele? Inā noho ʻoe ma nā Urals a kokoke paha, loaʻa iā ʻoe ka manawa kūpono a makemake i nā kumuhana - ʻae, ʻoiaʻiʻo. Inā ʻoe e noʻonoʻo ana i kahi huakaʻi lōʻihi, e nānā wau i nā kumuhana o nā hōʻike a me nā hōʻike wikiō mai nā makahiki i hala a hoʻoholo.
ʻO kekahi pōmaikaʻi ʻē aʻe o nā ʻaha kūkā ma nā ʻāina, ma ke ʻano he kānāwai, he maʻalahi ke kamaʻilio me ka mea haʻiʻōlelo ma hope o nā hōʻike; ʻoi aku ka liʻiliʻi o nā mea noi no ia kamaʻilio.
Mahalo iā Dump a me Ekaterinburg! )
Source: www.habr.com