Aloha kākou, e nā hoa!
* Hoʻokumu ʻia kēia ʻatikala ma ka REBRAIN & Yandex.Cloud open workshop, inā makemake ʻoe e nānā i ke wikiō, hiki iā ʻoe ke loaʻa ma kēia loulou -
Ua loaʻa iā mākou ka manawa kūpono e hoʻāʻo iā Yandex.Cloud ola. No ka mea makemake mākou e ʻimi lōʻihi a paʻakikī, ua haʻalele koke mākou i ka manaʻo o ka hoʻolaha ʻana i kahi blog Wordpress maʻalahi me kahi kumu ao - ʻoluʻolu loa. Ma hope o ka noʻonoʻo ʻana, ua hoʻoholo mākou e kau i kahi mea like me kahi hoʻolālā lawelawe hana no ka loaʻa ʻana a me ka nānā ʻana i nā hanana ma kahi kokoke i ke ʻano manawa maoli.
ʻIke loa wau ʻo ka hapa nui o nā ʻoihana pūnaewele (a ʻaʻole wale) e hōʻiliʻili i kahi mauna o ka ʻike e pili ana i kā lākou mea hoʻohana a me kā lākou hana. Ma ka liʻiliʻi, pono kēia no ka hoʻoholo ʻana i kekahi mau hoʻoholo - no ka laʻana, inā hoʻokele ʻoe i kahi pāʻani pūnaewele, hiki iā ʻoe ke nānā i nā ʻikepili kahi e paʻa pinepine ai nā mea hoʻohana a holoi i kāu pāʻani. A i ʻole no ke aha e haʻalele ai nā mea hoʻohana i kāu pūnaewele me ke kūʻai ʻole ʻana i kekahi mea (hello, Yandex.Metrica).
No laila, ʻo kā mākou moʻolelo: pehea mākou i kākau ai i kahi noi ma golang, hoʻāʻo ʻo kafka vs rabbitmq vs yqs, kākau i ka kahe ʻana o ka ʻikepili ma kahi hui Clickhouse a nānā i ka ʻikepili me ka hoʻohana ʻana i nā yandex datalens. ʻO ka mea maʻamau, ua hoʻomaʻamaʻa ʻia kēia mau mea me nā mea leʻaleʻa i ke ʻano o docker, terraform, gitlab ci a, ʻoiaʻiʻo, prometheus. E hele kāua!
Makemake wau e hana koke i kahi hoʻopaʻa ʻaʻole hiki iā mākou ke hoʻonohonoho i nā mea āpau i kahi noho hoʻokahi - no kēia mea e pono ai mākou i kekahi mau ʻatikala ma ka moʻo. He wahi liʻiliʻi e pili ana i ka hale:
Mahele 1 (ke heluhelu nei ʻoe). E hoʻoholo mākou i nā kikoʻī a me ka hoʻolālā ʻana o ka hopena, a kākau pū i kahi noi ma golang.
Mahele 2. Hoʻokuʻu mākou i kā mākou noi i ka hana ʻana, hana i ka scalable a hoʻāʻo i ka ukana.
Mahele 3. E hoʻāʻo kākou e noʻonoʻo i ke kumu e pono ai mākou e mālama i nā memo i loko o kahi buffer a ʻaʻole i loko o nā faila, a hoʻohālikelike pū i ka lawelawe kafka, rabbitmq a me yandex queue.
Mahele 4 E kau mākou i kahi pūʻulu Clickhouse, e kākau i kahi lawelawe streaming e hoʻoili i ka ʻikepili mai ka buffer ma laila, a hoʻonohonoho i ka ʻike ma nā datalens.
Mahele 5 E hoʻokomo i ka ʻōnaehana holoʻokoʻa i ke ʻano kūpono - hoʻonohonoho i ka ci/cd me ka hoʻohana ʻana i ka gitlab ci, hoʻohui i ka nānā ʻana a me ka ʻike lawelawe me ka prometheus a me ke consul.
TK
ʻO ka mea mua, e hoʻokumu i nā huaʻōlelo o ke kuhikuhi - he aha ka mea a mākou e makemake ai e loaʻa i ka hopena.
- Makemake mākou e loaʻa kahi hopena e like me events.kis.im (kis.im ka kahua hoʻāʻo a mākou e hoʻohana ai ma nā ʻatikala āpau), pono e loaʻa nā hanana me ka hoʻohana ʻana i ka HTTPS.
- He json maʻalahi nā hanana e like me: {“event”: “view”, “os”: “linux”, “browser”: “chrome”}. I ka pae hope loa e hoʻohui mākou i nā māla liʻiliʻi, akā ʻaʻole e pāʻani nui kēia. Inā makemake ʻoe, hiki iā ʻoe ke hoʻololi i protobuf.
- Pono ka lawelawe e hana i nā hanana 10 i kēlā me kēia kekona.
- Hiki ke hoʻonui ʻia ma ka hoʻohui ʻana i nā mea hou i kā mākou hoʻonā. A maikaʻi inā hiki iā mākou ke neʻe i ka ʻaoʻao mua i nā geolocations ʻokoʻa e hōʻemi i ka latency no nā noi o nā mea kūʻai aku.
- Hoʻomanawanui hewa. Pono e kūpaʻa ka hopena a hiki ke ola i ka hāʻule ʻana o nā ʻāpana (a hiki i kahi helu, ʻoiaʻiʻo).
kuhikuhipuʻuone
Ma keʻano holoʻokoʻa, no kēia ʻano hana, ua hana ʻia nā kiʻi kahiko kahiko e hiki ai ke hoʻonui pono. Hōʻike ke kiʻi i kahi laʻana o kā mākou hoʻonā.
No laila he aha kā mākou:
1. Aia ma ka ʻaoʻao hema kā mākou mau mea hana e hoʻopuka ai i nā hanana like ʻole, ʻo ia nā mea pāʻani e hoʻopiha ana i kahi pae ma kahi pāʻani ma ke kelepona a i ʻole ke hana ʻana i kahi kauoha ma kahi hale kūʻai pūnaewele ma o ka polokalamu kele pūnaewele maʻamau. ʻO kahi hanana, e like me ka mea i ʻōlelo ʻia ma ka kikoʻī, he json maʻalahi i hoʻouna ʻia i kā mākou hopena - events.kis.im.
2. ʻO nā kikowaena mua ʻelua he mea kaulike maʻalahi, ʻo kā lākou hana nui:
- E loaʻa mau. No ka hana ʻana i kēia, hiki iā ʻoe ke hoʻohana, no ka laʻana, keepalived, ka mea e hoʻololi i ka IP virtual ma waena o nā nodes inā pilikia.
- Hoʻopau iā TLS. ʻAe, e hoʻopau mākou iā TLS ma luna o lākou. ʻO ka mea mua, no ka hoʻokō ʻana o kā mākou hopena i nā kikoʻī ʻenehana, a ʻo ka lua, i mea e hoʻomaha ai i ke kaumaha o ka hoʻokumu ʻana i kahi pilina i hoʻopili ʻia mai kā mākou mau kikowaena backend.
- E kaulike i nā noi e hiki mai ana i nā kikowaena hope loa. Hiki ke ʻike ʻia ka ʻōlelo nui ma aneʻi. Ma muli o kēia, hiki mākou i ka hoʻomaopopo ʻana e hiki i nā mea kau kaulike ke nānā i kā mākou mau kikowaena me nā noi a hoʻōki i ke kaupaona ʻana i nā kaʻa i nā nodes i hāʻule ʻole.
3. Ma hope o nā mea kaulike, loaʻa iā mākou nā kikowaena noi e holo ana i kahi noi maʻalahi. Hiki iā ia ke ʻae i nā noi e komo mai ana ma o HTTP, e hōʻoia i ka json i hoʻouna ʻia a hoʻokomo i ka ʻikepili i kahi buffer.
4. Hōʻike ke kiʻi i ka kafka ma ke ʻano he pale, ʻoiai, ʻoiaʻiʻo, hiki ke hoʻohana ʻia nā lawelawe like ʻole ma kēia pae. E hoʻohālikelike mākou iā Kafka, rabbitmq a me yqs ma ka ʻatikala ʻekolu.
5. ʻO ka helu penultimate o kā mākou hale hoʻolālā ʻo Clickhouse - kahi waihona kolamu e hiki ai iā ʻoe ke mālama a mālama i ka nui o ka ʻikepili. Ma kēia pae, pono mākou e hoʻololi i ka ʻikepili mai ka buffer i ka ʻōnaehana mālama ponoʻī (ʻoi aku ma kēia ma ka ʻatikala 4).
Hāʻawi kēia hoʻolālā iā mākou e hoʻonui i kēlā me kēia papa me ke kaʻawale. ʻAʻole hiki i nā server backend ke hoʻokō - e hoʻohui hou i hoʻokahi mea - ma hope o nā mea a pau, he mau palapala mokuʻāina lākou, a no laila, hiki ke hana maʻalahi. ʻAʻole hana ka mea hoʻopaʻa ʻano Kafka-e hoʻohui i nā kikowaena hou aʻe a hoʻoili i kekahi o nā ʻāpana o kā mākou kumuhana iā lākou. ʻAʻole hiki iā Clickhouse ke mālama - ʻaʻole hiki ʻole :) ʻOiaʻiʻo, e hoʻopili pū mākou i nā kikowaena a hoʻokaʻawale i ka ʻikepili.
Ma ke ala, inā makemake ʻoe e hoʻokō i ka ʻāpana koho o kā mākou kikoʻī ʻenehana a me ka pālākiō i nā geolocations like ʻole, a laila ʻaʻohe mea maʻalahi:
Ma kēlā me kēia geolocation mākou e kau nei i kahi mea hoʻohālikelike ukana me ka noi a me ka kafka. Ma keʻano laulā, ua lawa nā kikowaena noi 2, 3 kafka nodes a me kahi mea kaulike o ke ao, no ka laʻana, cloudflare, e nānā i ka loaʻa o nā nodes noi a me nā noi kaulike e ka geolocation e pili ana i ka helu IP kumu o ka mea kūʻai. No laila, e pae ka ʻikepili i hoʻouna ʻia e kahi mea kūʻai aku ʻAmelika ma nā kikowaena ʻAmelika. A aia ka ʻikepili mai ʻApelika ma ʻApelika.
A laila maʻalahi nā mea a pau - hoʻohana mākou i ka hāmeʻa aniani mai ka hoʻonohonoho Kafka a kope i nā ʻikepili āpau mai nā wahi āpau i kā mākou kikowaena data kikowaena i Rusia. Ma loko, hoʻopaʻa mākou i ka ʻikepili a hoʻopaʻa iā ia ma Clickhouse no ka ʻike ʻana ma hope.
No laila, ua hoʻokaʻawale mākou i ka hoʻolālā - e hoʻomaka kākou e lulu Yandex.Cloud!
Ke kākau ʻana i kahi noi
Ma mua o ka Cloud, pono ʻoe e hoʻomanawanui a kākau i kahi lawelawe maʻalahi e hoʻoponopono i nā hanana e hiki mai ana. E hoʻohana mākou i ka golang no ka mea ua hōʻoia ʻo ia iā ia iho he ʻōlelo no ke kākau ʻana i nā noi pūnaewele.
Ma hope o ka hoʻolimalima ʻana i hoʻokahi hola (ʻelua paha mau hola), loaʻa iā mākou kekahi mea e like me kēia: .
He aha nā manaʻo nui aʻu e makemake ai e hoʻomaopopo ma aneʻi:
1. I ka hoʻomaka ʻana i ka noi, hiki iā ʻoe ke kuhikuhi i nā hae ʻelua. ʻO kekahi ke kuleana o ke awa kahi e hoʻolohe ai mākou i nā noi http e hiki mai ana (-addr). ʻO ka lua no ka helu kikowaena kafka kahi e hoʻopaʻa ai mākou i kā mākou mau hanana (-kafka):
addr = flag.String("addr", ":8080", "TCP address to listen to")
kafka = flag.String("kafka", "127.0.0.1:9092", "Kafka endpoints”)2. Hoʻohana ka palapala noi i ka waihona sarama () e hoʻouna i nā leka i ka pūʻulu kafka. Hoʻonohonoho koke mākou i nā hoʻonohonoho e pili ana i ka wikiwiki o ka hoʻoili ʻana:
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForLocal
config.Producer.Compression = sarama.CompressionSnappy
config.Producer.Return.Successes = true3. Loaʻa i kā mākou noi kahi mea kūʻai prometheus i kūkulu ʻia, nāna e hōʻiliʻili i nā metric like ʻole, e like me:
- helu o nā noi i kā mākou noi;
- ka helu o nā hewa i ka wā e hoʻokō ai i ka noi (hiki ʻole ke heluhelu i ka noi pou, haki json, hiki ʻole ke kākau iā Kafka);
- ka manawa hana no hoʻokahi noi mai ka mea kūʻai aku, me ka manawa no ke kākau ʻana i leka iā Kafka.
4. ʻEkolu mau hopena i kā mākou noi noi:
- / kūlana - e hoʻihoʻi wale i ka maikaʻi e hōʻike i ko mākou ola. ʻOiai hiki iā ʻoe ke hoʻohui i kekahi mau loiloi, e like me ka loaʻa ʻana o ka pūʻulu Kafka.
- /metrics - e like me kēia url, e hoʻihoʻi ka mea kūʻai prometheus i nā ana i hōʻiliʻili ai.
- ʻO / post ka hopena nui kahi e hoʻouna ʻia ai nā noi POST me json i loko. Nānā kā mākou noi i ka json no ka pono a inā maikaʻi nā mea a pau, kākau ia i ka ʻikepili i ka hui Kafka.
E hoʻopaʻa wau ʻaʻole kūpono ke code - hiki iā ia (a pono!) No ka laʻana, hiki iā ʻoe ke hoʻōki i ka hoʻohana ʻana i ka ʻupena i kūkulu ʻia/http a hoʻololi i ka http wikiwiki. A i ʻole hiki iā ʻoe ke loaʻa ka manawa kaʻina hana a me nā kumuwaiwai cpu ma ka neʻe ʻana i ka nānā pono json i kahi pae hope - ke hoʻololi ʻia ka ʻikepili mai ka buffer i ka cluster clickhouse.
Ma waho aʻe o ka ʻaoʻao hoʻomohala o ka pilikia, ua noʻonoʻo koke mākou e pili ana i kā mākou ʻoihana e hiki mai ana a ua hoʻoholo mākou e kau i kā mākou noi ma o docker. ʻO ka Dockerfile hope loa no ke kūkulu ʻana i ka noi . Ma keʻano laulā, maʻalahi loa ia, ʻo ka mea wale nō aʻu e makemake ai e hoʻolohe i ka hui multistage, e hiki ai iā mākou ke hōʻemi i ke kiʻi hope o kā mākou pahu.
Nā ʻanuʻu mua i ke ao
ʻO ka mea mua, e kākau inoa ma . Ma hope o ka hoʻopihaʻana i nā māhele kūpono a pau, e hanaʻia mākou i kahi moʻokāki a hāʻawiʻia i kahi hāʻawi kālā no kekahi kālā, hiki ke hoʻohanaʻia e ho'āʻo i nā lawelawe kapua. Inā makemake ʻoe e hana hou i nā ʻanuʻu āpau mai kā mākou ʻatikala, pono e lawa kēia hāʻawi iā ʻoe.
Ma hope o ka hoʻopaʻa inoa ʻana, e hana ʻia kahi ao ʻokoʻa a me kahi papa kuhikuhi paʻamau no ʻoe, kahi e hiki ai iā ʻoe ke hoʻomaka e hana i nā kumuwaiwai kapua. Ma keʻano laulā, ma Yandex.Cloud, ka pilina o nā kumuwaiwai e like me kēia:
Hiki iā ʻoe ke hana i kekahi mau ao no hoʻokahi moʻokāki. A i loko o ke ao, e hana i nā papa kuhikuhi like ʻole no nā papahana ʻoihana like ʻole. Hiki iā ʻoe ke heluhelu hou aʻe e pili ana i kēia ma ka palapala - . Ma ke ala, e kuhikuhi pinepine au iā ia ma lalo o ka kikokikona. I koʻu hoʻonohonoho ʻana i ka ʻōnaehana holoʻokoʻa mai ka wā ʻōpala, ua kōkua ka palapala iaʻu ma mua o hoʻokahi manawa, no laila ke aʻo aku nei au iā ʻoe e aʻo.
No ka hoʻokele ʻana i ke ao, hiki iā ʻoe ke hoʻohana i ka pilina pūnaewele a me ka pono console - yc. Hana ʻia ka hoʻouka ʻana me hoʻokahi kauoha (no Linux a me Mac Os):
curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bashInā ʻo kāu loea palekana kūloko e huhū nei e pili ana i ka holo ʻana i nā palapala mai ka Pūnaewele, a laila, ʻo ka mea mua, hiki iā ʻoe ke wehe i ka palapala a heluhelu iā ia, a ʻo ka lua, holo mākou ma lalo o kā mākou mea hoʻohana - ʻaʻohe kuleana kumu.
Inā makemake ʻoe e hoʻokomo i kahi mea kūʻai aku no Windows, hiki iā ʻoe ke hoʻohana i nā kuhikuhi a laila e hoʻokō yc inite hoʻopilikino pono iā ia:
vozerov@mba:~ $ yc init
Welcome! This command will take you through the configuration process.
Please go to https://oauth.yandex.ru/authorize?response_type=token&client_id= in order to obtain OAuth token.
Please enter OAuth token:
Please select cloud to use:
[1] cloud-b1gv67ihgfu3bp (id = b1gv67ihgfu3bpt24o0q)
[2] fevlake-cloud (id = b1g6bvup3toribomnh30)
Please enter your numeric choice: 2
Your current cloud has been set to 'fevlake-cloud' (id = b1g6bvup3toribomnh30).
Please choose folder to use:
[1] default (id = b1g5r6h11knotfr8vjp7)
[2] Create a new folder
Please enter your numeric choice: 1
Your current folder has been set to 'default' (id = b1g5r6h11knotfr8vjp7).
Do you want to configure a default Compute zone? [Y/n]
Which zone do you want to use as a profile default?
[1] ru-central1-a
[2] ru-central1-b
[3] ru-central1-c
[4] Don't set default zone
Please enter your numeric choice: 1
Your profile default Compute zone has been set to 'ru-central1-a'.
vozerov@mba:~ $Ma ke kumu, maʻalahi ke kaʻina hana - pono ʻoe e kiʻi i kahi hōʻailona oauth e hoʻokele i ke ao, koho i ke ao a me ka waihona āu e hoʻohana ai.
Inā loaʻa iā ʻoe nā moʻokāki a i ʻole nā waihona i loko o ke ao like, hiki iā ʻoe ke hana i nā ʻaoʻao hou me nā hoʻonohonoho kaʻawale ma o ka yc config profile e hana a hoʻololi i waena o lākou.
Ma waho aʻe o nāʻano i luna, ua kākau ka hui Yandex.Cloud i kahi maikaʻi loa no ka mālama ʻana i nā kumuwaiwai ao. No kaʻu ʻāpana, ua hoʻomākaukau wau i kahi waihona git, kahi aʻu i wehewehe ai i nā kumuwaiwai āpau e hana ʻia ma ke ʻano o ka ʻatikala - . Makemake mākou i ka lālā kumu, e hoʻopaʻa i ka ʻāina:
vozerov@mba:~ $ git clone https://github.com/rebrainme/yandex-cloud-events/ events
Cloning into 'events'...
remote: Enumerating objects: 100, done.
remote: Counting objects: 100% (100/100), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 100 (delta 37), reused 89 (delta 26), pack-reused 0
Receiving objects: 100% (100/100), 25.65 KiB | 168.00 KiB/s, done.
Resolving deltas: 100% (37/37), done.
vozerov@mba:~ $ cd events/terraform/Ua kākau ʻia nā ʻano nui āpau i hoʻohana ʻia i ka terraform ma ka faila main.tf. No ka hoʻomaka ʻana, hana i kahi faila private.auto.tfvars i loko o ka waihona terraform me kēia ʻike:
# Yandex Cloud Oauth token
yc_token = ""
# Yandex Cloud ID
yc_cloud_id = ""
# Yandex Cloud folder ID
yc_folder_id = ""
# Default Yandex Cloud Region
yc_region = "ru-central1-a"
# Cloudflare email
cf_email = ""
# Cloudflare token
cf_token = ""
# Cloudflare zone id
cf_zone_id = ""Hiki ke lawe ʻia nā ʻano like ʻole mai ka papa inoa config yc, ʻoiai ua hoʻonohonoho mua mākou i ka pono console. Manaʻo wau iā ʻoe e hoʻohui koke iā private.auto.tfvars iā .gitignore, i ʻole e hoʻolaha i ka ʻikepili pilikino.
Ma private.auto.tfvars, ua kuhikuhi pū mākou i ka ʻikepili mai Cloudflare - e hana i nā moʻolelo DNS a me ke koho ʻana i ka domain nui events.kis.im i kā mākou mau kikowaena. Inā ʻaʻole ʻoe makemake e hoʻohana i ka cloudflare, a laila e wehe i ka hoʻomaka ʻana o ka mea hāʻawi cloudflare ma main.tf a me ka faila dns.tf, nona ke kuleana no ka hana ʻana i nā moʻolelo dns pono.
Ma kā mākou hana e hoʻohui mākou i nā ʻano ʻekolu - ke kikowaena pūnaewele, ka pono console, a me ka terraform.
Nā pūnaewele puni honua
ʻO kaʻoiaʻiʻo, hiki iā ʻoe ke lele i kēia ʻanuʻu, ʻoiai ke hana ʻoe i kahi ao hou, e loaʻa iā ʻoe kahi pūnaewele ʻokoʻa a hana ʻia nā subnets 3 - hoʻokahi no kēlā me kēia wahi loaʻa. Akā makemake mākou e hana i kahi pūnaewele ʻokoʻa no kā mākou papahana me kāna ʻōlelo ponoʻī. Hōʻike ʻia ke kiʻikuhi maʻamau o ka hana ʻana o ka pūnaewele ma Yandex.Cloud ma ke kiʻi ma lalo nei (i lawe pololei ʻia mai )
No laila, hana ʻoe i kahi pūnaewele maʻamau i hiki i nā kumuwaiwai ke kamaʻilio me kekahi. No kēlā me kēia ʻāpana loaʻa, hana ʻia kahi subnet me kāna ʻōlelo ponoʻī a pili i ka pūnaewele maʻamau. ʻO ka hopena, hiki ke kamaʻilio nā kumu ao āpau i loko, ʻoiai inā aia lākou i nā ʻāpana ʻokoʻa. Hiki ke ʻike ʻia nā kumuwaiwai i pili i nā ʻupena kapuaʻi ʻokoʻa ma o nā helu waho wale nō. Ma ke ala, pehea ka hana ʻana o kēia kupua i loko. .
Hōʻike ʻia ka hana ʻana o ka pūnaewele ma ka faila network.tf mai ka waihona. Ma laila mākou e hana ai i hoʻokahi pūnaewele pilikino maʻamau i loko a hoʻohui i ʻekolu subnets iā ia i nā ʻāpana ʻokoʻa - internal-a (172.16.1.0/24), internal-b (172.16.2.0/24), internal-c (172.16.3.0/24 ).
E hoʻomaka i ka terraform a hana i nā pūnaewele:
vozerov@mba:~/events/terraform (master) $ terraform init
... skipped ..
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_vpc_subnet.internal-a -target yandex_vpc_subnet.internal-b -target yandex_vpc_subnet.internal-c
... skipped ...
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
yandex_vpc_network.internal: Creating...
yandex_vpc_network.internal: Creation complete after 3s [id=enp2g2rhile7gbqlbrkr]
yandex_vpc_subnet.internal-a: Creating...
yandex_vpc_subnet.internal-b: Creating...
yandex_vpc_subnet.internal-c: Creating...
yandex_vpc_subnet.internal-a: Creation complete after 6s [id=e9b1dad6mgoj2v4funog]
yandex_vpc_subnet.internal-b: Creation complete after 7s [id=e2liv5i4amu52p64ac9p]
yandex_vpc_subnet.internal-c: Still creating... [10s elapsed]
yandex_vpc_subnet.internal-c: Creation complete after 10s [id=b0c2qhsj2vranoc9vhcq]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.Nui! Ua hana mākou i kā mākou pūnaewele a ua mākaukau mākou e hana i kā mākou lawelawe kūloko.
Ke hana nei i nā mīkini virtual
No ka hoʻāʻo ʻana i ka noi, pono mākou e hana i ʻelua mīkini virtual - pono mākou i ka mea mua e kūkulu a holo i ka noi, ʻo ka lua e holo i kafka, a mākou e hoʻohana ai no ka mālama ʻana i nā memo e hiki mai ana. A e hana mākou i kahi mīkini ʻē aʻe kahi e hoʻonohonoho ai mākou i ka prometheus e nānā i ka noi.
E hoʻonohonoho ʻia nā mīkini virtual me ka hoʻohana ʻana i ka ansible, no laila ma mua o ka hoʻomaka ʻana i ka terraform, e hōʻoia e loaʻa iā ʻoe kekahi o nā mana hou o ansible. A hoʻokomo i nā kuleana kūpono me ka galaxy ansible:
vozerov@mba:~/events/terraform (master) $ cd ../ansible/
vozerov@mba:~/events/ansible (master) $ ansible-galaxy install -r requirements.yml
- cloudalchemy-prometheus (master) is already installed, skipping.
- cloudalchemy-grafana (master) is already installed, skipping.
- sansible.kafka (master) is already installed, skipping.
- sansible.zookeeper (master) is already installed, skipping.
- geerlingguy.docker (master) is already installed, skipping.
vozerov@mba:~/events/ansible (master) $Aia i loko o ka waihona ansible kahi laʻana .ansible.cfg file hoʻonohonoho aʻu e hoʻohana ai. Maikaʻi paha ia.
Ma mua o ka hana ʻana i nā mīkini virtual, e hōʻoia i ka holo ʻana o ssh-agent a hoʻohui ʻia kahi kī ssh, inā ʻaʻole hiki ke hoʻopili ʻia ka terraform i nā mīkini i hana ʻia. ʻO wau, ʻoiaʻiʻo, ua loaʻa iaʻu kahi bug ma os x: . No ka pale ʻana i kēia, e hoʻohui i kahi ʻano liʻiliʻi i env ma mua o ka hoʻomaka ʻana iā Terraform:
vozerov@mba:~/events/terraform (master) $ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESMa ka waihona me ka terraform hana mākou i nā kumuwaiwai e pono ai:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_compute_instance.build -target yandex_compute_instance.monitoring -target yandex_compute_instance.kafka
yandex_vpc_network.internal: Refreshing state... [id=enp2g2rhile7gbqlbrkr]
data.yandex_compute_image.ubuntu_image: Refreshing state...
yandex_vpc_subnet.internal-a: Refreshing state... [id=e9b1dad6mgoj2v4funog]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
... skipped ...
Plan: 3 to add, 0 to change, 0 to destroy.
... skipped ...Inā hoʻopau maikaʻi nā mea āpau (a pono ia), a laila e loaʻa iā mākou ʻekolu mau mīkini virtual:
- kūkulu - he mīkini no ka hoʻāʻo ʻana a me ke kūkulu ʻana i kahi noi. Ua hoʻokomo ʻia ʻo Docker e Ansible.
- nānā - he mīkini nānā - prometheus & grafana i kau ʻia ma luna. Kau inoa / ʻōlelo huna maʻamau: admin / admin
- ʻO kafka kahi mīkini liʻiliʻi me ka kafka i hoʻokomo ʻia, hiki ke loaʻa ma ke awa 9092.
E hōʻoia mākou aia lākou a pau i kahi:
vozerov@mba:~/events (master) $ yc compute instance list
+----------------------+------------+---------------+---------+---------------+-------------+
| ID | NAME | ZONE ID | STATUS | EXTERNAL IP | INTERNAL IP |
+----------------------+------------+---------------+---------+---------------+-------------+
| fhm081u8bkbqf1pa5kgj | monitoring | ru-central1-a | RUNNING | 84.201.159.71 | 172.16.1.35 |
| fhmf37k03oobgu9jmd7p | kafka | ru-central1-a | RUNNING | 84.201.173.41 | 172.16.1.31 |
| fhmt9pl1i8sf7ga6flgp | build | ru-central1-a | RUNNING | 84.201.132.3 | 172.16.1.26 |
+----------------------+------------+---------------+---------+---------------+-------------+
Aia nā kumuwaiwai, a mai ʻaneʻi hiki iā mākou ke kiʻi i kā lākou IP address. Ma nā mea a pau e hoʻohana wau i nā leka uila IP e hoʻopili ma o ssh a hoʻāʻo i ka noi. Inā loaʻa iā ʻoe kahi moʻokāki cloudflare e pili ana i ka terraform, e ʻoluʻolu e hoʻohana i nā inoa DNS hou.
Ma ke ala, i ka hana ʻana i kahi mīkini virtual, hāʻawi ʻia kahi IP kūloko a me kahi inoa DNS kūloko, no laila hiki iā ʻoe ke komo i nā kikowaena i loko o ka pūnaewele ma ka inoa:
ubuntu@build:~$ ping kafka.ru-central1.internal
PING kafka.ru-central1.internal (172.16.1.31) 56(84) bytes of data.
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=1 ttl=63 time=1.23 ms
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=2 ttl=63 time=0.625 ms
^C
--- kafka.ru-central1.internal ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.625/0.931/1.238/0.308 msHe mea pono kēia no mākou e hōʻike i ka noi i ka hopena me kafk.
ʻO ka hui ʻana i ka noi
Nui, aia nā kikowaena, aia kahi noi - ʻo ke koena wale nō e hōʻuluʻulu a hoʻolaha. No ke kūkulu ʻana e hoʻohana mākou i ka hana docker maʻamau, akā ma ke ʻano he mālama kiʻi e hoʻohana mākou i kahi lawelawe mai Yandex - ka hoʻopaʻa inoa pahu. Akā ʻo ka mea mua.
Hoʻopili mākou i ka noi i ka mīkini kūkulu, e komo ma o ssh a hōʻuluʻulu i ke kiʻi:
vozerov@mba:~/events/terraform (master) $ cd ..
vozerov@mba:~/events (master) $ rsync -av app/ [email protected]:app/
... skipped ...
sent 3849 bytes received 70 bytes 7838.00 bytes/sec
total size is 3644 speedup is 0.93
vozerov@mba:~/events (master) $ ssh 84.201.132.3 -l ubuntu
ubuntu@build:~$ cd app
ubuntu@build:~/app$ sudo docker build -t app .
Sending build context to Docker daemon 6.144kB
Step 1/9 : FROM golang:latest AS build
... skipped ...
Successfully built 9760afd8ef65
Successfully tagged app:latestUa pau ka hapalua o ke kaua - i kēia manawa hiki iā mākou ke nānā i ka hana o kā mākou noi ma ka hoʻokuʻu ʻana a hoʻouna ʻia i kafka:
ubuntu@build:~/app$ sudo docker run --name app -d -p 8080:8080 app /app/app -kafka=kafka.ru-central1.internal:9092</code>
С локальной машинки можно отправить тестовый event и посмотреть на ответ:
<code>vozerov@mba:~/events (master) $ curl -D - -s -X POST -d '{"key1":"data1"}' http://84.201.132.3:8080/post
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 13 Apr 2020 13:53:54 GMT
Content-Length: 41
{"status":"ok","partition":0,"Offset":0}
vozerov@mba:~/events (master) $Ua pane ka palapala noi me ka holomua o ka hoʻopaʻa ʻana a hōʻike i ka id o ka ʻāpana a me ka offset kahi i hoʻokomo ʻia ai ka memo. ʻO nā mea a pau i koe e hana i kahi hoʻopaʻa inoa ma Yandex.Cloud a hoʻouka i kā mākou kiʻi ma laila (pehea e hana ai i kēia me ka hoʻohana ʻana i nā laina ʻekolu i wehewehe ʻia ma ka file registry.tf). E hana i kahi waihona:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_container_registry.events
... skipped ...
Plan: 1 to add, 0 to change, 0 to destroy.
... skipped ...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Nui nā ala e hōʻoia ai i loko o ka waihona waihona - me ka hoʻohana ʻana i kahi hōʻailona oauth, iam token, a i ʻole kahi kī moʻokāki lawelawe. Hiki ke loaʻa nā kikoʻī hou aku e pili ana i kēia mau ʻano ma ka palapala. . E hoʻohana mākou i ke kī moʻokāki lawelawe, no laila hana mākou i kahi moʻokāki:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_iam_service_account.docker -target yandex_resourcemanager_folder_iam_binding.puller -target yandex_resourcemanager_folder_iam_binding.pusher
... skipped ...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.I kēia manawa, ʻo ka hana ʻana i kahi kī no ia mea:
vozerov@mba:~/events/terraform (master) $ yc iam key create --service-account-name docker -o key.json
id: ajej8a06kdfbehbrh91p
service_account_id: ajep6d38k895srp9osij
created_at: "2020-04-13T14:00:30Z"
key_algorithm: RSA_2048Loaʻa iā mākou ka ʻike e pili ana i ka id o kā mākou waihona, hoʻololi i ke kī a komo i:
vozerov@mba:~/events/terraform (master) $ scp key.json [email protected]:
key.json 100% 2392 215.1KB/s 00:00
vozerov@mba:~/events/terraform (master) $ ssh 84.201.132.3 -l ubuntu
ubuntu@build:~$ cat key.json | sudo docker login --username json_key --password-stdin cr.yandex
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
ubuntu@build:~$No ka hoʻouka ʻana i ke kiʻi i ka registry, pono mākou i ka ID registry container, lawe mākou iā ia mai ka pono yc:
vozerov@mba:~ $ yc container registry get events
id: crpdgj6c9umdhgaqjfmm
folder_id:
name: events
status: ACTIVE
created_at: "2020-04-13T13:56:41.914Z"Ma hope o kēlā, kau mākou i kā mākou kiʻi me kahi inoa hou a hoʻouka:
ubuntu@build:~$ sudo docker tag app cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
ubuntu@build:~$ sudo docker push cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
The push refers to repository [cr.yandex/crpdgj6c9umdhgaqjfmm/events]
8c286e154c6e: Pushed
477c318b05cb: Pushed
beee9f30bc1f: Pushed
v1: digest: sha256:1dd5aaa9dbdde2f60d833be0bed1c352724be3ea3158bcac3cdee41d47c5e380 size: 946Hiki iā mākou ke hōʻoia ua hoʻouka pono ʻia ke kiʻi:
vozerov@mba:~/events/terraform (master) $ yc container repository list
+----------------------+-----------------------------+
| ID | NAME |
+----------------------+-----------------------------+
| crpe8mqtrgmuq07accvn | crpdgj6c9umdhgaqjfmm/events |
+----------------------+-----------------------------+Ma ke ala, inā ʻoe e hoʻokomo i ka pono yc ma kahi mīkini Linux, hiki iā ʻoe ke hoʻohana i ke kauoha
yc container registry configure-dockere hoʻonohonoho i ka docker.
hopena
Ua hana mākou i nā hana nui a ʻo ka hopena:
- Ua hele mai mākou me ka hoʻolālā o kā mākou lawelawe e hiki mai ana.
- Ua kākau mākou i kahi noi ma golang e hoʻokō i kā mākou loiloi ʻoihana.
- Ua hōʻiliʻili mākou a ninini ʻia i loko o kahi waihona puke pilikino.
Ma ka ʻāpana aʻe, e neʻe mākou i nā mea hoihoi - e hoʻokuʻu mākou i kā mākou noi i ka hana a hoʻomaka i ka ukana ma luna. Mai hoʻololi!
Aia kēia mea i loko o ka hoʻopaʻa wikiō o ka hale hana wehe REBRAIN & Yandex.Cloud: Ua ʻae mākou i nā noi 10 i kēlā me kēia kekona ma Yandex Cloud -
Inā makemake ʻoe e hele i kēlā mau hanana ma ka pūnaewele a nīnau i nā nīnau i ka manawa maoli, e hoʻopili iā.
Makemake mākou e ʻōlelo i kahi mahalo kūikawā iā Yandex.Cloud no ka manawa kūpono e hoʻokipa i kēlā hanana. loulou iā lākou -
Inā pono ʻoe e neʻe i ke ao a i ʻole he nīnau e pili ana i kāu ʻōnaehana, .
PS Loaʻa iā mākou 2 mau loiloi manuahi i kēlā me kēia mahina, malia paha ʻo kāu papahana kekahi o lākou.
Source: www.habr.com