Nyob zoo sawv daws, cov phooj ywg!
* Cov kab lus no yog ua raws li REBRAIN & Yandex.Cloud qhib kev cob qhia, yog tias koj nyiam saib cov yeeb yaj kiab, koj tuaj yeem nrhiav nws ntawm qhov txuas no -
Peb nyuam qhuav muaj sijhawm los sim Yandex.Cloud nyob. Txij li thaum peb xav soj ntsuam ntev thiab nyuaj, peb tam sim ntawd tso tseg lub tswv yim ntawm kev nthuav tawm ib qho yooj yim Wordpress blog nrog lub hauv paus huab - nws yog qhov tho txawv heev. Tom qab qee qhov kev xav, peb tau txiav txim siab los siv qee yam zoo ib yam li kev pabcuam tsim khoom rau kev txais thiab tshuaj xyuas cov xwm txheej nyob ze rau lub sijhawm tiag tiag.
Kuv paub tseeb tias feem coob ntawm online (thiab tsis yog xwb) cov lag luam qee zaum sau ib lub roob ntawm cov ntaub ntawv hais txog lawv cov neeg siv thiab lawv cov kev ua. Qhov tsawg kawg nkaus, qhov no yog qhov tsim nyog rau kev txiav txim siab qee yam - piv txwv li, yog tias koj tswj hwm kev ua si hauv online, koj tuaj yeem saib cov txheeb cais ntawm qib twg cov neeg siv feem ntau tau daig thiab tshem tawm koj cov khoom ua si. Los yog vim li cas cov neeg siv tawm hauv koj lub xaib yam tsis muaj kev yuav khoom (nyob zoo, Yandex.Metrica).
Yog li, peb zaj dab neeg: yuav ua li cas peb tau sau ib daim ntawv thov hauv golang, kuaj kafka vs rabbitmq vs yqs, sau cov ntaub ntawv streaming hauv Clickhouse pawg thiab pom cov ntaub ntawv siv yandex datalens. Lawm, tag nrho cov no tau seasoned nrog infrastructure delights nyob rau hauv daim ntawv ntawm docker, terraform, gitlab ci thiab, ntawm chav kawm, prometheus. Wb mus!
Kuv xav tam sim ua qhov kev tshwj tseg uas peb yuav tsis tuaj yeem teeb tsa txhua yam hauv ib qho zaum - rau qhov no peb yuav xav tau ntau cov ntawv hauv koob. Ib me ntsis txog cov qauv:
Part 1 (koj nyeem nws). Peb yuav txiav txim siab txog cov kev qhia tshwj xeeb thiab cov qauv kev daws teeb meem, thiab tseem sau ib daim ntawv thov hauv golang.
Ntu 2. Peb tso peb daim ntawv thov mus rau hauv ntau lawm, ua kom nws scalable thiab kuaj lub load.
Ntu 3. Cia peb sim xyuas seb vim li cas peb yuav tsum khaws cov lus hauv qhov tsis thiab tsis nyob hauv cov ntaub ntawv, thiab tseem muab piv rau kafka, rabbitmq thiab yandex queue kev pabcuam.
Ntu 4 Peb yuav xa ib pab pawg Clickhouse, sau cov kev pabcuam streaming los hloov cov ntaub ntawv los ntawm qhov tsis nyob ntawd, thiab teeb tsa kev pom hauv datalens.
Ntu 5 Cia peb nqa tag nrho cov kev tsim kho kom zoo - teeb ci / cd siv gitlab ci, txuas kev soj ntsuam thiab kev pabcuam nrhiav pom siv prometheus thiab consul.
TK
Ua ntej, cia peb tsim cov ntsiab lus ntawm kev siv - dab tsi raws nraim peb xav tau raws li qhov tshwm sim.
- Peb xav kom muaj qhov kawg zoo li events.kis.im (kis.im yog qhov chaw xeem uas peb yuav siv thoob plaws hauv txhua tshooj), uas yuav tsum tau txais cov xwm txheej siv HTTPS.
- Cov xwm txheej yog json yooj yim xws li: {"event": "saib", "os": "linux", "browser": "chrome"}. Nyob rau theem kawg peb yuav ntxiv me ntsis ntxiv, tab sis qhov no yuav tsis ua lub luag haujlwm loj. Yog tias koj xav tau, koj tuaj yeem hloov mus rau protobuf.
- Cov kev pabcuam yuav tsum muaj peev xwm ua tiav 10 txheej xwm ib ob.
- Nws yuav tsum muaj peev xwm ntsuas kab rov tav los ntawm tsuas yog ntxiv cov xwm txheej tshiab rau peb qhov kev daws teeb meem. Thiab nws yuav zoo yog tias peb tuaj yeem txav sab pem hauv ntej mus rau qhov chaw sib txawv kom txo qis latency rau cov neeg thov.
- Ua txhaum cai. Cov tshuaj yuav tsum ruaj khov txaus thiab muaj peev xwm ciaj sia lub caij nplooj zeeg ntawm txhua qhov chaw (txog rau ib tus lej, tau kawg).
architecture
Feem ntau, rau hom hauj lwm no, classical architectures tau ntev lawm invented uas tso cai rau scaling npaum. Daim duab qhia ib qho piv txwv ntawm peb qhov kev daws teeb meem.
Yog li peb muaj:
1. Ntawm sab laug yog peb cov khoom siv uas tsim ntau yam xwm txheej, yog nws cov players ua tiav qib hauv cov khoom ua si ntawm lub xov tooj smartphone lossis tsim kev txiav txim hauv khw hauv online los ntawm qhov browser tsis tu ncua. Ib qho kev tshwm sim, raws li tau teev tseg hauv qhov tshwj xeeb, yog json yooj yim uas xa mus rau peb qhov kawg - events.kis.im.
2. Thawj ob lub servers yog cov sib npaug yooj yim, lawv cov haujlwm tseem ceeb yog:
- Yuav muaj nyob tas li. Ua li no, koj tuaj yeem siv, piv txwv li, khaws cia, uas yuav hloov pauv tus IP virtual ntawm cov nodes thaum muaj teeb meem.
- Terminate TLS. Yog lawm, peb yuav txiav TLS rau lawv. Ua ntej, kom peb cov kev daws teeb meem ua raws li cov kev qhia tshwj xeeb, thiab qhov thib ob, txhawm rau txo lub nra ntawm kev tsim kom muaj kev sib txuas encrypted los ntawm peb cov backend servers.
- Sib npaug cov lus thov tuaj rau cov servers backend muaj. Lo lus tseem ceeb ntawm no yog siv tau. Raws li qhov no, peb tuaj rau kev nkag siab tias load balancers yuav tsum tuaj yeem saib xyuas peb cov servers nrog cov ntawv thov thiab tso tseg kev sib npaug ntawm cov tsheb mus rau cov nodes ua tsis tiav.
3. Tom qab cov balancers, peb muaj daim ntawv thov servers khiav ib daim ntawv thov yooj yim. Nws yuav tsum muaj peev xwm lees txais cov lus thov tuaj ntawm HTTP, lees paub qhov xa json thiab muab cov ntaub ntawv tso rau hauv qhov tsis muaj.
4. Daim duab qhia tau tias kafka yog qhov tsis muaj, txawm hais tias, tau kawg, lwm yam kev pabcuam zoo sib xws tuaj yeem siv rau qib no. Peb yuav muab piv Kafka, rabbitmq thiab yqs nyob rau hauv peb tsab xov xwm.
5. Lub ntsiab lus kawg ntawm peb cov architecture yog Clickhouse - columnar database uas tso cai rau koj khaws thiab ua cov ntaub ntawv loj heev. Nyob rau theem no, peb yuav tsum tau hloov cov ntaub ntawv los ntawm qhov tsis mus rau qhov system cia nws tus kheej (ntxiv rau qhov no hauv kab lus 4).
Qhov kev tsim no tso cai rau peb ntsuas txhua txheej ntawm nws tus kheej horizontally. Backend servers tsis tuaj yeem tiv taus - cia peb ntxiv ib qho ntxiv - tom qab tag nrho, lawv yog cov ntawv thov tsis muaj neeg nyob, thiab yog li ntawd, qhov no tuaj yeem ua tiav txawm tias tau txais. Kafka-style tsis ua haujlwm - cia peb ntxiv cov servers ntxiv thiab hloov qee qhov kev faib ntawm peb cov ncauj lus rau lawv. Clickhouse tsis tuaj yeem tuav nws - nws tsis yooj yim sua :) Qhov tseeb, peb tseem yuav txuas cov servers thiab shard cov ntaub ntawv.
Los ntawm txoj kev, yog tias koj xav siv qhov kev xaiv ntawm peb cov kev specifications thiab teev nyob rau hauv txawv geolocations, ces tsis muaj dab tsi yooj yim dua:
Nyob rau hauv txhua geolocation peb deploy ib tug load balancer nrog daim ntawv thov thiab kafka. Feem ntau, 2 daim ntawv thov servers, 3 kafka nodes thiab huab balancer, piv txwv li, cloudflare, yog txaus, uas yuav kuaj xyuas qhov muaj ntawm daim ntawv thov nodes thiab sib npaug thov los ntawm geolocation raws li tus neeg siv khoom qhov chaw nyob IP. Yog li, cov ntaub ntawv xa los ntawm tus neeg siv khoom Asmeskas yuav tsaws ntawm Asmeskas servers. Thiab cov ntaub ntawv los ntawm Africa yog nyob rau hauv African.
Tom qab ntawd txhua yam yog qhov yooj yim heev - peb siv cov cuab yeej iav los ntawm Kafka teeb thiab luam tag nrho cov ntaub ntawv los ntawm txhua qhov chaw mus rau peb lub chaw nruab nrab cov ntaub ntawv nyob hauv Russia. Sab hauv, peb txheeb xyuas cov ntaub ntawv thiab sau nws hauv Clickhouse rau kev pom tom ntej.
Yog li, peb tau txheeb xyuas cov architecture - cia peb pib co Yandex.Cloud!
Sau daim ntawv thov
Ua ntej Huab, koj tseem yuav tsum ua tus neeg mob me ntsis thiab sau cov kev pabcuam yooj yim los ua cov txheej xwm tuaj. Peb yuav siv golang vim tias nws tau ua pov thawj nws tus kheej zoo heev uas yog ib hom lus rau kev sau ntawv hauv network.
Tom qab siv sijhawm ib teev (tej zaum ob peb teev), peb tau txais qee yam zoo li no: .
Dab tsi yog cov ntsiab lus tseem ceeb uas kuv xav sau tseg ntawm no:
1. Thaum pib daim ntawv thov, koj tuaj yeem teev ob tus chij. Ib qho yog lub luag haujlwm rau qhov chaw nres nkoj uas peb yuav mloog cov lus thov los ntawm http (-addr). Qhov thib ob yog rau kafka server chaw nyob qhov twg peb yuav sau peb cov xwm txheej (-kafka):
addr = flag.String("addr", ":8080", "TCP address to listen to")
kafka = flag.String("kafka", "127.0.0.1:9092", "Kafka endpointsβ)2. Daim ntawv thov siv lub tsev qiv ntawv sarama () xa lus mus rau pawg kafka. Peb tam sim ntawd teeb tsa lub hom phiaj ntawm kev ua haujlwm siab tshaj plaws:
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForLocal
config.Producer.Compression = sarama.CompressionSnappy
config.Producer.Return.Successes = true3. Peb daim ntawv thov kuj muaj cov neeg siv khoom tsim hauv prometheus, uas sau ntau yam kev ntsuas, xws li:
- tus naj npawb ntawm kev thov rau peb daim ntawv thov;
- tus lej yuam kev thaum ua tiav qhov kev thov (tsis tuaj yeem nyeem cov ntawv thov, json tawg, tsis tuaj yeem sau rau Kafka);
- Lub sijhawm ua haujlwm rau ib qho kev thov los ntawm tus neeg siv khoom, suav nrog lub sijhawm sau ntawv rau Kafka.
4. Peb qhov kawg uas peb cov txheej txheem thov:
- / xwm txheej - tsuas yog rov qab los qhia tias peb tseem muaj sia nyob. Txawm hais tias koj tuaj yeem ntxiv qee qhov kev kuaj xyuas, xws li qhov muaj ntawm Kafka pawg.
- /metrics - raws li qhov url no, tus neeg siv khoom prometheus yuav rov qab cov metrics nws tau sau.
- /post yog lub ntsiab kawg uas POST thov nrog json sab hauv yuav raug xa. Peb daim ntawv thov tshuaj xyuas json kom siv tau thiab yog tias txhua yam zoo, nws sau cov ntaub ntawv mus rau Kafka pawg.
Kuv yuav ua qhov kev tshwj tseg tias cov cai tsis zoo tag nrho - nws tuaj yeem (thiab yuav tsum!) ua tiav. Piv txwv li, koj tuaj yeem tso tseg tsis siv lub net / http thiab hloov mus rau qhov nrawm http. Lossis koj tuaj yeem tau txais lub sijhawm ua haujlwm thiab cov peev txheej cpu los ntawm kev txav lub json validity check mus rau theem tom ntej - thaum cov ntaub ntawv raug xa los ntawm qhov tsis mus rau pawg clickhouse.
Ntxiv rau qhov kev txhim kho ntawm qhov teeb meem, peb tam sim ntawd xav txog peb cov kev tsim kho yav tom ntej thiab txiav txim siab los xa peb daim ntawv thov ntawm docker. Qhov kawg Dockerfile rau kev tsim daim ntawv thov yog . Feem ntau, nws yog qhov yooj yim heev, tib lub ntsiab lus uas kuv xav tau them sai sai rau yog lub rooj sib txoos multistage, uas tso cai rau peb txo qis cov duab kawg ntawm peb lub thawv.
Thawj kauj ruam hauv huab
Ua ntej tshaj plaws, sau npe rau . Tom qab ua tiav tag nrho cov haujlwm tsim nyog, peb yuav tsim ib tus as-qhauj thiab muab nyiaj pub dawb rau qee qhov nyiaj, uas tuaj yeem siv los ntsuas cov kev pabcuam huab. Yog tias koj xav rov ua txhua kauj ruam ntawm peb tsab xov xwm, qhov nyiaj pab no yuav tsum txaus rau koj.
Tom qab sau npe, ib qho huab cais thiab ib daim ntawv teev npe yuav raug tsim rau koj, uas koj tuaj yeem pib tsim cov peev txheej huab. Feem ntau, hauv Yandex.Cloud, kev sib raug zoo ntawm cov peev txheej zoo li no:
Koj tuaj yeem tsim ntau huab rau ib tus account. Thiab hauv huab, ua cov npe sib txawv rau cov tuam txhab sib txawv. Koj tuaj yeem nyeem ntxiv txog qhov no hauv cov ntaub ntawv - . Los ntawm txoj kev, kuv yuav feem ntau xa mus rau nws hauv qab no hauv cov ntawv nyeem. Thaum kuv teeb tsa tag nrho cov txheej txheem los ntawm kos, cov ntaub ntawv tau pab kuv tawm ntau dua ib zaug, yog li kuv qhia koj kom kawm nws.
Txhawm rau tswj hwm huab, koj tuaj yeem siv ob qho tib si hauv web interface thiab console utility - yc. Kev teeb tsa tau ua nrog ib qho lus txib (rau Linux thiab Mac Os):
curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bashYog tias koj tus kws tshaj lij kev ruaj ntseg sab hauv tab tom npau taws txog kev khiav cov ntawv sau los ntawm Is Taws Nem, tom qab ntawd, thawj zaug, koj tuaj yeem qhib tsab ntawv thiab nyeem nws, thiab thib ob, peb khiav nws hauv qab peb cov neeg siv - tsis muaj cai hauv paus.
Yog tias koj xav txhim kho tus neeg siv khoom rau Windows, koj tuaj yeem siv cov lus qhia thiab ces execute yc initkom tag nrho customize nws:
vozerov@mba:~ $ yc init
Welcome! This command will take you through the configuration process.
Please go to https://oauth.yandex.ru/authorize?response_type=token&client_id= in order to obtain OAuth token.
Please enter OAuth token:
Please select cloud to use:
[1] cloud-b1gv67ihgfu3bp (id = b1gv67ihgfu3bpt24o0q)
[2] fevlake-cloud (id = b1g6bvup3toribomnh30)
Please enter your numeric choice: 2
Your current cloud has been set to 'fevlake-cloud' (id = b1g6bvup3toribomnh30).
Please choose folder to use:
[1] default (id = b1g5r6h11knotfr8vjp7)
[2] Create a new folder
Please enter your numeric choice: 1
Your current folder has been set to 'default' (id = b1g5r6h11knotfr8vjp7).
Do you want to configure a default Compute zone? [Y/n]
Which zone do you want to use as a profile default?
[1] ru-central1-a
[2] ru-central1-b
[3] ru-central1-c
[4] Don't set default zone
Please enter your numeric choice: 1
Your profile default Compute zone has been set to 'ru-central1-a'.
vozerov@mba:~ $Hauv txoj ntsiab cai, cov txheej txheem yog yooj yim - ua ntej koj yuav tsum tau txais ib qho oauth token los tswj huab, xaiv huab thiab cov ntawv tais ceev tseg koj yuav siv.
Yog tias koj muaj ntau tus account lossis folders hauv tib huab, koj tuaj yeem tsim cov profiles ntxiv nrog cov chaw sib cais ntawm yc config profile tsim thiab hloov ntawm lawv.
Ntxiv nrog rau cov txheej txheem saum toj no, pab pawg Yandex.Cloud sau tau zoo heev rau kev tswj huab kev pab cuam. Rau kuv ib feem, kuv tau npaj git repository, qhov twg kuv tau piav qhia tag nrho cov peev txheej uas yuav raug tsim los ua ib feem ntawm tsab xov xwm - . Peb txaus siab rau tus tswv ceg, cia peb clone nws hauv zos:
vozerov@mba:~ $ git clone https://github.com/rebrainme/yandex-cloud-events/ events
Cloning into 'events'...
remote: Enumerating objects: 100, done.
remote: Counting objects: 100% (100/100), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 100 (delta 37), reused 89 (delta 26), pack-reused 0
Receiving objects: 100% (100/100), 25.65 KiB | 168.00 KiB/s, done.
Resolving deltas: 100% (37/37), done.
vozerov@mba:~ $ cd events/terraform/Txhua qhov hloov pauv tseem ceeb uas siv hauv terraform yog sau rau hauv cov ntaub ntawv main.tf. Txhawm rau pib, tsim cov ntaub ntawv ntiag tug.auto.tfvars hauv daim nplaub tshev terraform nrog cov ntsiab lus hauv qab no:
# Yandex Cloud Oauth token
yc_token = ""
# Yandex Cloud ID
yc_cloud_id = ""
# Yandex Cloud folder ID
yc_folder_id = ""
# Default Yandex Cloud Region
yc_region = "ru-central1-a"
# Cloudflare email
cf_email = ""
# Cloudflare token
cf_token = ""
# Cloudflare zone id
cf_zone_id = ""Txhua qhov hloov pauv tuaj yeem raug coj los ntawm yc config daim ntawv teev npe, txij li peb twb tau teeb tsa lub console utility. Kuv qhia koj tam sim ntawd ntxiv private.auto.tfvars rau .gitignore, thiaj li tsis txhob yuam kev luam tawm cov ntaub ntawv ntiag tug.
Nyob rau hauv private.auto.tfvars peb kuj tau teev cov ntaub ntawv los ntawm Cloudflare - los tsim DNS cov ntaub ntawv thiab tso npe rau lub ntsiab domain events.kis.im rau peb cov servers. Yog tias koj tsis xav siv cloudflare, ces tshem tawm qhov pib ntawm cloudflare tus muab kev pabcuam hauv main.tf thiab cov ntaub ntawv dns.tf, uas yog lub luag haujlwm tsim cov ntaub ntawv dns tsim nyog.
Hauv peb txoj haujlwm peb yuav muab tag nrho peb txoj hauv kev - lub web interface, console utility, thiab terraform.
Virtual networks
Ua kom ncaj ncees, koj tuaj yeem hla cov kauj ruam no, txij li thaum koj tsim huab tshiab, koj yuav cia li muaj ib lub network sib cais thiab 3 subnets tsim - ib qho rau txhua thaj chaw muaj. Tab sis peb tseem xav ua kom sib cais network rau peb qhov project nrog nws tus kheej qhov chaw nyob. Daim duab dav dav ntawm yuav ua li cas lub network ua haujlwm hauv Yandex.Cloud yog qhia hauv daim duab hauv qab no (ua ncaj ncees coj los ntawm )
Yog li, koj tsim ib qho kev sib koom ua ke nyob rau hauv uas cov peev txheej tuaj yeem sib txuas lus nrog ib leeg. Rau txhua qhov chaw muaj, ib lub subnet tsim nrog nws tus kheej qhov chaw nyob thiab txuas nrog lub network dav dav. Raws li qhov tshwm sim, txhua qhov kev pabcuam huab hauv nws tuaj yeem sib txuas lus, txawm tias lawv nyob hauv thaj chaw muaj sib txawv. Cov peev txheej txuas nrog cov huab sib txawv tuaj yeem pom ib leeg los ntawm qhov chaw nyob sab nraud. Los ntawm txoj kev, qhov khawv koob no ua haujlwm li cas hauv, .
Network creation yog piav nyob rau hauv lub network.tf cov ntaub ntawv los ntawm lub repository. Nyob ntawd peb tsim ib qho kev sib koom tes sib koom nrog hauv thiab txuas peb lub subnets rau nws hauv thaj chaw muaj nyob sib txawv - internal-a (172.16.1.0/24), internal-b (172.16.2.0/24), internal-c (172.16.3.0/24). ).
Pib ua terraform thiab tsim networks:
vozerov@mba:~/events/terraform (master) $ terraform init
... skipped ..
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_vpc_subnet.internal-a -target yandex_vpc_subnet.internal-b -target yandex_vpc_subnet.internal-c
... skipped ...
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
yandex_vpc_network.internal: Creating...
yandex_vpc_network.internal: Creation complete after 3s [id=enp2g2rhile7gbqlbrkr]
yandex_vpc_subnet.internal-a: Creating...
yandex_vpc_subnet.internal-b: Creating...
yandex_vpc_subnet.internal-c: Creating...
yandex_vpc_subnet.internal-a: Creation complete after 6s [id=e9b1dad6mgoj2v4funog]
yandex_vpc_subnet.internal-b: Creation complete after 7s [id=e2liv5i4amu52p64ac9p]
yandex_vpc_subnet.internal-c: Still creating... [10s elapsed]
yandex_vpc_subnet.internal-c: Creation complete after 10s [id=b0c2qhsj2vranoc9vhcq]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.Zoo heev! Peb tau tsim peb lub network thiab tam sim no npaj los tsim peb cov kev pabcuam sab hauv.
Tsim cov tshuab virtual
Txhawm rau kuaj daim ntawv thov, peb tsuas yog yuav tsum tsim ob lub tshuab virtual - peb yuav xav tau thawj zaug los tsim thiab khiav daim ntawv thov, qhov thib ob los khiav kafka, uas peb yuav siv los khaws cov lus xa tuaj. Thiab peb yuav tsim lwm lub tshuab uas peb yuav teeb tsa prometheus los saib xyuas daim ntawv thov.
Lub tshuab virtual yuav raug teeb tsa siv ansible, yog li ua ntej pib terraform, nco ntsoov tias koj muaj ib qho ntawm cov qauv tshiab ntawm ansible. Thiab nruab lub luag haujlwm tsim nyog nrog ansible galaxy:
vozerov@mba:~/events/terraform (master) $ cd ../ansible/
vozerov@mba:~/events/ansible (master) $ ansible-galaxy install -r requirements.yml
- cloudalchemy-prometheus (master) is already installed, skipping.
- cloudalchemy-grafana (master) is already installed, skipping.
- sansible.kafka (master) is already installed, skipping.
- sansible.zookeeper (master) is already installed, skipping.
- geerlingguy.docker (master) is already installed, skipping.
vozerov@mba:~/events/ansible (master) $Hauv daim nplaub tshev ansible muaj ib qho piv txwv .ansible.cfg configuration file uas kuv siv. Nws tuaj yeem ua ke.
Ua ntej tsim cov tshuab virtual, xyuas kom tseeb tias koj muaj ssh-tus neeg sawv cev khiav thiab qhov tseem ceeb ssh ntxiv, txwv tsis pub terraform yuav tsis tuaj yeem txuas mus rau cov tshuab tsim. Kuv, tau kawg, tuaj hla ib kab hauv os x: . Txhawm rau tiv thaiv qhov no tsis tshwm sim dua, ntxiv qhov hloov pauv me me rau env ua ntej tso Terraform:
vozerov@mba:~/events/terraform (master) $ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESHauv daim nplaub tshev nrog terraform peb tsim cov peev txheej tsim nyog:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_compute_instance.build -target yandex_compute_instance.monitoring -target yandex_compute_instance.kafka
yandex_vpc_network.internal: Refreshing state... [id=enp2g2rhile7gbqlbrkr]
data.yandex_compute_image.ubuntu_image: Refreshing state...
yandex_vpc_subnet.internal-a: Refreshing state... [id=e9b1dad6mgoj2v4funog]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
... skipped ...
Plan: 3 to add, 0 to change, 0 to destroy.
... skipped ...Yog tias txhua yam ua tiav tiav (thiab nws yuav tsum yog), ces peb yuav muaj peb lub tshuab virtual:
- tsim - lub tshuab rau kev sim thiab tsim daim ntawv thov. Docker raug teeb tsa los ntawm Ansible.
- saib xyuas - lub tshuab saib xyuas - prometheus & grafana ntsia rau ntawm nws. ID nkag mus / password tus qauv: admin / admin
- kafka yog lub tshuab me me nrog kafka ntsia, nkag tau rau ntawm chaw nres nkoj 9092.
Cia peb paub tseeb tias lawv txhua tus nyob hauv qhov chaw:
vozerov@mba:~/events (master) $ yc compute instance list
+----------------------+------------+---------------+---------+---------------+-------------+
| ID | NAME | ZONE ID | STATUS | EXTERNAL IP | INTERNAL IP |
+----------------------+------------+---------------+---------+---------------+-------------+
| fhm081u8bkbqf1pa5kgj | monitoring | ru-central1-a | RUNNING | 84.201.159.71 | 172.16.1.35 |
| fhmf37k03oobgu9jmd7p | kafka | ru-central1-a | RUNNING | 84.201.173.41 | 172.16.1.31 |
| fhmt9pl1i8sf7ga6flgp | build | ru-central1-a | RUNNING | 84.201.132.3 | 172.16.1.26 |
+----------------------+------------+---------------+---------+---------------+-------------+
Cov peev txheej yog nyob rau hauv qhov chaw, thiab los ntawm no peb tuaj yeem tau txais lawv qhov chaw nyob IP. Thoob plaws hauv qab no kuv yuav siv IP chaw txuas los ntawm ssh thiab sim daim ntawv thov. Yog tias koj muaj cloudflare tus account txuas nrog terraform, xav siv cov npe DNS tshiab uas tau tsim tshiab.
Los ntawm txoj kev, thaum tsim lub tshuab virtual, tus IP sab hauv thiab lub npe DNS sab hauv tau muab, yog li koj tuaj yeem nkag mus rau cov servers hauv lub network los ntawm lub npe:
ubuntu@build:~$ ping kafka.ru-central1.internal
PING kafka.ru-central1.internal (172.16.1.31) 56(84) bytes of data.
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=1 ttl=63 time=1.23 ms
64 bytes from kafka.ru-central1.internal (172.16.1.31): icmp_seq=2 ttl=63 time=0.625 ms
^C
--- kafka.ru-central1.internal ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.625/0.931/1.238/0.308 msQhov no yuav muaj txiaj ntsig rau peb los qhia rau daim ntawv thov qhov kawg nrog kafk.
Sib sau ua ke daim ntawv thov
Zoo heev, muaj cov servers, muaj ib daim ntawv thov - txhua yam uas tseem tshuav yog sib sau ua ke thiab tshaj tawm nws. Rau kev tsim peb yuav siv cov docker ib txwm ua, tab sis raws li cov duab khaws cia peb yuav siv cov kev pabcuam los ntawm Yandex - thawv sau npe. Tab sis thawj yam ua ntej.
Peb luam daim ntawv thov mus rau lub tshuab tsim, nkag mus ntawm ssh thiab sib sau cov duab:
vozerov@mba:~/events/terraform (master) $ cd ..
vozerov@mba:~/events (master) $ rsync -av app/ [email protected]:app/
... skipped ...
sent 3849 bytes received 70 bytes 7838.00 bytes/sec
total size is 3644 speedup is 0.93
vozerov@mba:~/events (master) $ ssh 84.201.132.3 -l ubuntu
ubuntu@build:~$ cd app
ubuntu@build:~/app$ sudo docker build -t app .
Sending build context to Docker daemon 6.144kB
Step 1/9 : FROM golang:latest AS build
... skipped ...
Successfully built 9760afd8ef65
Successfully tagged app:latestIb nrab ntawm kev sib ntaus sib tua ua tiav - tam sim no peb tuaj yeem tshawb xyuas qhov ua haujlwm ntawm peb daim ntawv thov los ntawm kev tso nws thiab xa mus rau kafka:
ubuntu@build:~/app$ sudo docker run --name app -d -p 8080:8080 app /app/app -kafka=kafka.ru-central1.internal:9092</code>
Π‘ Π»ΠΎΠΊΠ°Π»ΡΠ½ΠΎΠΉ ΠΌΠ°ΡΠΈΠ½ΠΊΠΈ ΠΌΠΎΠΆΠ½ΠΎ ΠΎΡΠΏΡΠ°Π²ΠΈΡΡ ΡΠ΅ΡΡΠΎΠ²ΡΠΉ event ΠΈ ΠΏΠΎΡΠΌΠΎΡΡΠ΅ΡΡ Π½Π° ΠΎΡΠ²Π΅Ρ:
<code>vozerov@mba:~/events (master) $ curl -D - -s -X POST -d '{"key1":"data1"}' http://84.201.132.3:8080/post
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 13 Apr 2020 13:53:54 GMT
Content-Length: 41
{"status":"ok","partition":0,"Offset":0}
vozerov@mba:~/events (master) $Daim ntawv thov teb nrog kev ua tiav ntawm kev kaw lus thiab qhia tus ID ntawm qhov muab faib thiab offset uas cov lus tau suav nrog. Txhua yam uas tseem tshuav yog tsim ib daim ntawv teev npe hauv Yandex.Cloud thiab xa peb cov duab nyob ntawd (yuav ua li cas siv peb kab yog piav qhia hauv cov ntaub ntawv sau npe.tf). Tsim ib qho chaw cia:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_container_registry.events
... skipped ...
Plan: 1 to add, 0 to change, 0 to destroy.
... skipped ...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Muaj ntau ntau txoj hauv kev los ua pov thawj hauv lub thawv ntim khoom - siv lub cim oauth, ib qho token iam, lossis tus lej nyiaj pabcuam. Cov ntsiab lus ntxiv txog cov txheej txheem no tuaj yeem pom hauv cov ntaub ntawv. . Peb yuav siv tus lej nyiaj pabcuam, yog li peb tsim ib tus account:
vozerov@mba:~/events/terraform (master) $ terraform apply -target yandex_iam_service_account.docker -target yandex_resourcemanager_folder_iam_binding.puller -target yandex_resourcemanager_folder_iam_binding.pusher
... skipped ...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.Tam sim no txhua yam uas tseem tshuav yog los ua tus yuam sij rau nws:
vozerov@mba:~/events/terraform (master) $ yc iam key create --service-account-name docker -o key.json
id: ajej8a06kdfbehbrh91p
service_account_id: ajep6d38k895srp9osij
created_at: "2020-04-13T14:00:30Z"
key_algorithm: RSA_2048Peb tau txais cov ntaub ntawv hais txog tus ID ntawm peb qhov chaw cia, hloov tus yuam sij thiab nkag rau hauv:
vozerov@mba:~/events/terraform (master) $ scp key.json [email protected]:
key.json 100% 2392 215.1KB/s 00:00
vozerov@mba:~/events/terraform (master) $ ssh 84.201.132.3 -l ubuntu
ubuntu@build:~$ cat key.json | sudo docker login --username json_key --password-stdin cr.yandex
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
ubuntu@build:~$Txhawm rau muab cov duab tso rau hauv daim ntawv teev npe, peb xav tau lub thawv sau npe ID, peb muab nws los ntawm yc utility:
vozerov@mba:~ $ yc container registry get events
id: crpdgj6c9umdhgaqjfmm
folder_id:
name: events
status: ACTIVE
created_at: "2020-04-13T13:56:41.914Z"Tom qab ntawd, peb tag peb cov duab nrog lub npe tshiab thiab upload:
ubuntu@build:~$ sudo docker tag app cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
ubuntu@build:~$ sudo docker push cr.yandex/crpdgj6c9umdhgaqjfmm/events:v1
The push refers to repository [cr.yandex/crpdgj6c9umdhgaqjfmm/events]
8c286e154c6e: Pushed
477c318b05cb: Pushed
beee9f30bc1f: Pushed
v1: digest: sha256:1dd5aaa9dbdde2f60d833be0bed1c352724be3ea3158bcac3cdee41d47c5e380 size: 946Peb tuaj yeem txheeb xyuas tias cov duab tau teeb tsa tiav:
vozerov@mba:~/events/terraform (master) $ yc container repository list
+----------------------+-----------------------------+
| ID | NAME |
+----------------------+-----------------------------+
| crpe8mqtrgmuq07accvn | crpdgj6c9umdhgaqjfmm/events |
+----------------------+-----------------------------+Los ntawm txoj kev, yog tias koj nruab yc nqi hluav taws xob ntawm lub tshuab Linux, koj tuaj yeem siv cov lus txib
yc container registry configure-dockerlos configure docker.
xaus
Peb tau ua haujlwm hnyav heev thiab yog li ntawd:
- Peb tuaj nrog cov architecture ntawm peb cov kev pabcuam yav tom ntej.
- Peb tau sau ib daim ntawv thov nyob rau hauv golang uas siv peb lub lag luam logic.
- Peb sau nws thiab muab tso rau hauv ib lub thawv ntim khoom ntiag tug.
Hauv ntu tom ntej, peb yuav txav mus rau cov khoom nthuav dav - peb yuav tso peb daim ntawv thov rau hauv kev tsim khoom thiab thaum kawg tso lub nra ntawm nws. Tsis txhob hloov!
Cov ntaub ntawv no yog nyob rau hauv cov ntaubntawv povthawj siv video qhib REBRAIN & Yandex.Cloud: Peb txais 10 thov ib ob ntawm Yandex Cloud -
Yog tias koj xav koom nrog cov xwm txheej zoo li no hauv online thiab nug cov lus nug hauv lub sijhawm, txuas mus rau.
Peb xav hais ua tsaug tshwj xeeb rau Yandex.Cloud rau lub sijhawm los tuav cov xwm txheej zoo li no. Txuas rau lawv -
Yog tias koj xav tau txav mus rau huab lossis muaj lus nug txog koj qhov kev tsim kho vaj tse, .
PS Peb muaj 2 kev tshuaj xyuas dawb hauv ib hlis, tej zaum koj qhov project yuav yog ib qho ntawm lawv.
Tau qhov twg los: www.hab.com