4 mainjiniya, 7000 maseva uye imwe yepasi rose denda

Mhoro, Habr! Ndinokupa kutariswa kweshanduro yechinyorwa "Injiniya ina, 4 Servers, uye Imwe Global Pandemic" naAdib Daw.

Kana iwo musoro wenyaya usingatumire kubvunda kudiki pasi nemuzongoza wako, unofanirwa kusvetukira kundima inotevera kana kushanyira peji redu rakatsaurirwa basa mukambani - tinoda kutaura.

Isu tiri ani

Isu tiri timu ye4 penguin vanoda kunyora kodhi uye kushanda nehardware. Munguva yedu yakasununguka, isu tine basa rekutumira, kuchengetedza nekushandisa boka rezvikepe zvinopfuura 7000 zvemuviri maseva anomhanya Linux, akagoverwa munzvimbo nhatu dzakasiyana dze data kuUnited States.

Isu takavawo nemukana wekuita izvi 10 km kubva kunzvimbo, kubva kunyaradzo yehofisi yedu pachedu, iyo iri chinhambwe chipfupi kubva kumhenderekedzo yegungwa paGungwa reMediterranean.

Matambudziko ekuyera

Kunyangwe zvine musoro kuti yekutanga itange nekutambira zvivakwa zvayo mugore nekuda kwekudyara kwakadzikira kwekutanga, isu kuOutbrain takasarudza kushandisa maseva edu. Isu takaita izvi nekuti mitengo yezvivakwa zvegore inodarika mutengo wekushandisa yedu midziyo iri munzvimbo dzedata mushure mekuvandudzwa kune imwe nhanho. Uye zvakare, sevha yako inopa yakanyanya dhigirii rekutonga uye kugadzirisa matambudziko.

Sezvo isu tichikura, matambudziko anogara ari pedyo. Uyezve, vanowanzouya vari mapoka. Server lifecycle management inoda kugara ichizvinatsiridza kuitira kuti ikwanise kushanda zvakanaka mumamiriro ekukurumidza kuwedzera kwenhamba yemaseva. Nzira dzeSoftware dzekutarisira mapoka emaseva munzvimbo dzedata dzinokurumidza kushata. Kuona, kugadzirisa matambudziko, uye kudzikisira kutadza paunenge uchisangana neQoS zviyero inova nyaya yekujairira zvakasiyana-siyana zvakasiyana-siyana zvehardware, mabasa akasiyana-siyana, kusimudzira nguva dzekupedzisira, uye zvimwe zvinhu zvakanaka izvo hapana anoda kunetseka nazvo.

Tora maDomains ako

Kuti tigadzirise akawanda ematambudziko aya, takaputsa sevha yehupenyu muOutbrain kuita zvikamu zvayo zvikuru uye tikazvidaidza kuti madomasi. Semuenzaniso, imwe dura inovhara zvinodiwa zvemidziyo, imwe inovhara zvinhu zvine chekuita nehupenyu hwekutenderera, uye chetatu chinovhara kutaurirana nevashandi vemumunda. Pane imwe ine chekuita nekuonekwa kwehardware, asi isu hatisi kuzotsanangura ese mapoinzi. Chinangwa chedu chaive chekudzidza nekutsanangura madomasi kuitira kuti akwanise kutorwa pachishandiswa kodhi. Kana imwe nguva yekushanda yakagadziriswa, inoendeswa kune imwe nzira yekunyora iyo inotumirwa, yakaedzwa, uye yakanatswa. Chekupedzisira, iyo domain inogadziriswa kuti ibatanidze nemamwe madomasi kuburikidza neAPIs, ichigadzira yakazara, ine simba, uye inogara ichichinja hardware lifecycle system iyo inogona kutumirwa, inoyedzwa, uye inoonekwa. Kungofanana nemamwe masystem edu ese ekugadzira.

Kutora nzira iyi kwakatibvumira kugadzirisa matambudziko mazhinji nemazvo - nekugadzira maturusi uye otomatiki.

Need Domain

Kunyangwe maemail nemaspredishiti yaive nzira inoshanda yekusangana nezvinodiwa mumazuva ekutanga, yanga isiri mhinduro yakabudirira, kunyanya kana nhamba yemaseva uye huwandu hwezvikumbiro zvinouya zvasvika pane imwe nhanho. Kuronga zvirinani uye kuisa pamberi zvikumbiro zvinouya takatarisana nekuwedzera nekukurumidza, taifanira kushandisa matiketi system yaigona kupa:

  • Kugona kugadzirisa maonero endima chete dzakakodzera (dzakareruka)
  • Vhura maAPI (akawedzerwa)
  • Zvinozivikanwa kuchikwata chedu (zvinonzwisiswa)
  • Kubatanidzwa neyedu iripo workflows (yakabatana)

Sezvo isu tinoshandisa Jira kubata sprints yedu uye mabasa emukati, takafunga kugadzira imwe chirongwa chaizobatsira vatengi vedu kuendesa matikiti uye kuteedzera mhedzisiro yavo. Kushandisa Jira kune zvikumbiro zvinouya uye zvekugadzirisa mabasa emukati zvakatibvumira kugadzira imwe Kanban bhodhi iyo yaitibvumira kutarisa maitiro ose pamwe chete. "Matengi" edu emukati aingoona zvikumbiro zvemidziyo, pasina kuongorora mune zvisinganyanyi kukosha zvemamwe mabasa (akadai sekuvandudza maturusi, kugadzirisa tsikidzi).

4 mainjiniya, 7000 maseva uye imwe yepasi rose denda
Kanban board muJira

Sebhonasi, chokwadi chekuti mitsetse nezvaikosha zvaive zvave kuonekwa kumunhu wese zvakaita kuti zvikwanise kunzwisisa "papi pamutsetse" chimwe chikumbiro chaive uye chii chakatangira. Izvi zvaibvumira varidzi kuti vatarise zvakare zvikumbiro zvavo pasina kutibata. Zvikweve uye ndizvozvo. Izvo zvakatibvumidzawo kutarisa uye kuongorora maSLA edu zvichienderana nemhando dzekukumbira zvichienderana nemametric anogadzirwa muJira.

Equipment Lifecycle Domain

Edza kufungidzira kuoma kwekugadzirisa hardware inoshandiswa mune imwe neimwe server rack. Chakatonyanya kuipa ndechekuti zvidimbu zvakawanda zvehardware (RAM, ROM) zvinogona kutamiswa kubva mudura kuenda kukamuri yeseva uye kumashure. Ivo zvakare vanokundikana kana kunyorerwa uye kuchinjwa uye kudzoserwa kune mutengesi kuti vatsive / kugadzirisa. Zvese izvi zvinofanirwa kutaurirwa kune vashandi vebasa recolocation vanobatanidzwa mukuchengetedza kwemuviri kwemidziyo. Kugadzirisa matambudziko aya, takagadzira chishandiso chemukati chinonzi Floppy. Basa rake ndere:

  • Kutungamira kwekutaurirana nevashandi vemumunda, kuunganidzwa kweruzivo rwese;
  • Kugadziridza iyo "warehouse" data mushure meimwe neimwe yapera uye yakasimbiswa yekugadzira michina basa.

Imba yekuchengetera zvinhu, inoonekwa tichishandisa Grafana, yatinoshandisa kuronga metrics edu ese. Nekudaro, isu tinoshandisa chishandiso chimwechete chekutarisa chengetedzo uye kune zvimwe zvinodiwa zvekugadzira.

4 mainjiniya, 7000 maseva uye imwe yepasi rose dendaWarehouse equipment control panel muGrafana

Kune maseva zvishandiso zviri pasi pewaranti, tinoshandisa chimwe chishandiso chatinodaidza kuti Dispatcher. Iye:

  • Inounganidza matanda ehurongwa;
  • Inogadzira mishumo muchimiro chinodiwa nemutengesi;
  • Inogadzira chikumbiro kubva kumutengesi kuburikidza neAPI;
  • Inogamuchira uye inochengeta iyo identifier yekushandisa kuti iwedzere kutarisisa mafambiro ayo.

Kamwe kana kudai kwedu kuchigamuchirwa (kazhinji mukati memaawa ebhizinesi), iyo spare part inotumirwa kune yakakodzera data data uye inogamuchirwa nevashandi.

4 mainjiniya, 7000 maseva uye imwe yepasi rose denda
Jenkins console yakabuda

Communication Domain

Kuti tienderane nekukura nekukurumidza kwebhizinesi redu, izvo zvinoda kugara uchiwedzera, taifanira kugadzirisa nzira yatinoshanda nayo nenyanzvi dzehunyanzvi munzvimbo dze data data. Kana pakutanga kukwira kwaireva kutenga maseva matsva, zvino mushure mekubatanidza chirongwa (zvichienderana neshanduko kuenda Kubernetes) yakava chimwe chinhu chakasiyana zvachose. Shanduko yedu kubva pa "kuwedzera rack" kune "repurposing maseva."

Kushandisa nzira itsva kwaidawo maturusi matsva aiita kuti zvikwanise kutaurirana zvakasununguka nevashandi vepa data. Zvishandiso izvi zvaidiwa kuti:

  • Simplicity;
  • Autonomy;
  • Kubudirira;
  • Kuvimbika.

Isu taifanira kuzvibvisa isu kubva kucheni uye kuronga basa kuitira kuti matekiniki ashande zvakananga nemidziyo yeseva. Pasina kupindira kwedu uye tisina kugara tichisimudza zvese izvi zvine chekuita nebasa, maawa ekushanda, kuwanikwa kwemidziyo, nezvimwe.

Kuti tiite izvi, takaisa iPads munzvimbo yese yedata. Mushure mekubatanidza kune server, zvinotevera zvichaitika:

  • Chishandiso chinosimbisa kuti sevha iyi inoda rimwe basa;
  • Zvikumbiro zvinomhanya pane sevha zvakavharwa (kana zvichidikanwa);
  • Seti yemirairo yebasa inotumirwa paSlack chiteshi ichitsanangura matanho anodiwa;
  • Pakupera kwebasa, chigadziro chinotarisa kurongeka kwemamiriro ekupedzisira evhavha;
  • Inotangazve maapplication kana zvichidikanwa.

Pamusoro pezvo, isu takagadzirirawo Slack bot kubatsira nyanzvi. Kutenda kune huwandu hwakasiyana hwekugona (taigara tichiwedzera mashandiro), iyo bot yakaita kuti basa ravo rive nyore, uye rakaita kuti hupenyu hwedu huve nyore. Nenzira iyi isu takakwenenzvera yakawanda yemaitiro ekudzorera nekuchengetedza maseva, tichizvibvisa pachedu pakufambiswa kwebasa.

4 mainjiniya, 7000 maseva uye imwe yepasi rose denda
iPad mune imwe yedu data nzvimbo

Hardware Domain

Nekuvimbika kuyera nzvimbo yedu yedata data inoda kuoneka kwakanaka muchikamu chega chega, semuenzaniso:

  • Kuonekwa kwekutadza kwehardware
  • Server states (inoshanda, yakagarwa, zombie, nezvimwewo)
  • Power Consumption
  • Firmware version
  • Analytics yeiyi bhizinesi rese

Mhinduro dzedu dzinotibvumira kuita sarudzo pamusoro pekuti, kupi uye rini kwekutenga michina, dzimwe nguva kunyange isati yanyatsodiwa. Zvakare, nekuona chiyero chemutoro pamidziyo yakasiyana, takakwanisa kuwana yakagadziridzwa yekugovera zviwanikwa. Kunyanya, kushandiswa kwesimba. Isu tinogona ikozvino kuita sarudzo dzine ruzivo nezvekuiswa kwesevha isati yaiswa muraki uye yakabatana kune sosi yemagetsi, mukati mehupenyu hwayo hwose uye kusvika pakuzorega.

4 mainjiniya, 7000 maseva uye imwe yepasi rose denda
Energy Dashboard muGrafana

Uye ipapo COVID-19 yakaonekwa ...

Chikwata chedu chinogadzira matekinoroji anopa simba makambani emidhiya nevashambadziri pamhepo kubatsira vashanyi kuwana zvakakodzera zvemukati, zvigadzirwa uye masevhisi angangove anofarira kwavari. Zvivakwa zvedu zvakagadzirirwa kushandira traffic inogadzirwa kana dzimwe nhau dzinonakidza dzaburitswa.

Iyo yakanyanya kutepfenyura midhiya yakatenderedza COVID-19, pamwe nekuwedzera kwetraffic, zvaireva kuti isu taifanira kukurumidza kudzidza maitiro ekuita nedzvinyiriro idzi. Zvakare, zvese izvi zvaifanira kuitwa panguva yedambudziko repasi rose, apo cheni dzekugovera dzakakanganiswa uye vazhinji vevashandi vaive kumba.

Asi, sezvatakataura, modhi yedu inotofunga kuti:

  • Zvishandiso zviri munzvimbo dzedu dzedata, kazhinji, hazvigoneke kwatiri;
  •  Isu tinoita rinenge mabasa ese emuviri kure;
  • Basa rinoitwa asynchronously, autonomously uye pamwero mukuru;
  • Isu tinosangana nezvinodiwa zvemidziyo uchishandisa nzira ye "kuvaka kubva kune zvikamu" pane kutenga michina mitsva;
  • Tine imba yekuchengetera iyo inotibvumira kugadzira chimwe chinhu chitsva, uye kwete kungoita zvechinyakare kugadzirisa.

Saka, zvirambidzo zvepasi rose izvo zvaitadzisa makambani mazhinji kuwana mukana wenyama kunzvimbo yavo yedata hazvina zvazvinoita patiri.Uye kana zviri zvezvimedu nemaseva, hongu, takaedza kuve nechokwadi chekushanda kwakagadzikana kwemidziyo. Asi izvi zvakaitwa nechinangwa chekudzivirira zvinogona kuitika kana zvikangoerekana zvabuda kuti chimwe chikamu chehardware hachipo. Takava nechokwadi chekuti matura edu azadzwa pasina chinangwa chekuzadzisa zvaidiwa pari zvino.

Muchidimbu, ndinoda kutaura kuti nzira yedu yekushanda mubazi re data data inoratidza kuti zvinokwanisika kushandisa mazano ekugadzirwa kwekodhi yakanaka kune hutungamiri hwemuviri we data data. Uye zvichida uchazviwana zvichifadza.

Chokutanga: tyts

Source: www.habr.com

Voeg