Momwe ndidakhala sabata limodzi ngati injiniya wa SRE. Ntchito kudzera m'maso mwa wopanga mapulogalamu

SRE injiniya - wophunzira

Choyamba, ndiroleni ndidzidziwitse ndekha. Ine - @tristan.read, injiniya wakutsogolo pagulu Monitor::Thanzi GitLab. Sabata yatha ndidakhala ndi mwayi wolumikizana ndi m'modzi mwa mainjiniya athu a SRE. Cholinga chinali kuwona momwe wogwira ntchitoyo amachitira zochitika tsiku ndi tsiku ndikupeza zochitika zenizeni pa ntchito. Tikufuna mainjiniya athu amvetsetse zosowa za ogwiritsa ntchito ntchito Monitor::Thanzi.

Ndinayenera kutsatira injiniya wa SRE kulikonse kwa sabata. Ndiko kuti, ndinalipo popereka, kuyang'anira njira zochenjeza zomwezo ndikuyankha zochitika ngati zichitika.

Zochitika

Panali zochitika 2 mkati mwa sabata.

1. Cryptominer

GitLab.com idawona kulumpha mukugwiritsa ntchito Lachitatu GitLab Wothamanga'a, chifukwa choyesa kugwiritsa ntchito mphindi za wothamanga kukumba cryptocurrency. Chochitikacho chidagwiritsidwa ntchito pogwiritsa ntchito chida chathu chophwanya malamulo, chomwe chimayimitsa ntchito za wothamanga ndikuchotsa projekiti ndi akaunti yogwirizana nayo.

Ngati chochitikachi sichinazindikiridwe, chida chodzidzimutsa chikanachigwira, koma pamenepa, injiniya wa SRE adawona kuphwanya poyamba. Ntchito yochitika idapangidwa, koma chidziwitso chake chatsekedwa.

2. Kuwonongeka kwa magwiridwe antchito a Canary ndi Main application

Chochitikacho chidachitika chifukwa cha kuchepa kwapang'onopang'ono komanso kuchuluka kwa zolakwika mu canary ndi mawebusayiti akuluakulu pa Gitlab.com. Makhalidwe angapo a Apdex adaphwanyidwa.

Tsegulani zochitika: https://gitlab.com/gitlab-com/gl-infra/production/issues/1442

Zotsatira Zofunikira

Nazi zinthu zingapo zomwe ndinaphunzira mkati mwa sabata ndikugwira ntchito.

1. Оповещения наиболее полезны, когда засекают отклонения от нормы.

Zidziwitso zitha kugawidwa m'mitundu ingapo:

• Zidziwitso zochokera pamtengo wina, monga "zolakwika 10 5xx zidachitika pamphindikati."
• Zidziwitso zomwe malirewo ndi amtengo wapatali monga "kuchuluka kwa zolakwika za 5xx pa 10% ya voliyumu yonse ya zopempha panthawi yoperekedwa."
• Zidziwitso zochokera ku mbiri yakale monga "zolakwika 5xx pa 90th percentile".

Nthawi zambiri, mitundu 2 ndi 3 ndiyothandiza kwambiri kwa ma SRE omwe ali pantchito, chifukwa amawulula zopatuka kuchokera pazomwe zimachitika.

2. Zidziwitso zambiri sizifika pazochitika.

Mainjiniya a SR amakumana ndi zidziwitso zanthawi zonse, zambiri zomwe sizowopsa.

Ndiye bwanji osachepetsa zidziwitso zanu pazofunikira kwenikweni? Ndi njira iyi, komabe, simungazindikire zizindikiro zoyambirira za zomwe snowball zingakhale vuto lenileni lomwe likuwopseza kuwonongeka kwakukulu.

Ntchito ya pa-call SRE ndikuzindikira kuti ndi zidziwitso ziti zomwe zikuwonetsa chinthu chachikulu, komanso ngati ziyenera kuchulukitsidwa ndikuthana nazo. Ndikukayikira kuti izi zilinso chifukwa cha kusasinthika kwa zidziwitso: zingakhale bwino ngati pangakhale magawo angapo kapena njira "zanzeru" zosinthira machenjezo molingana ndi zomwe tafotokozazi.

Malingaliro: https://gitlab.com/gitlab-org/gitlab/issues/42633

3. Ma SRE athu omwe ali pantchito amagwiritsa ntchito zida zambiri.

Zamkati:

• Pulojekiti ya GitLab infra: ma runbooks amakhala pano, ntchito zosinthira / sabata, ntchito zoyankhira zochitika.
• Nkhani za GitLab: Kufufuza, kuwunika, ndi kukonza kumatsatiridwanso pazokhudza.
• Zolemba za GitLab: Ntchito zongopanga zokha zimayambitsidwa pogwiritsa ntchito zilembo zapadera, zomwe bots amagwiritsa ntchito kutsata ntchito.

Zakunja:

• PagerDuty: Zidziwitso
• Slack: Kuyenda kwa uthenga wa PagerDuty/AlertManager kumapita apa. Kuphatikiza ndi malamulo a slash kuti mugwire ntchito zosiyanasiyana, monga kutseka chenjezo kapena kukwera ku chochitika.
• Grafana: mawonekedwe a ma metric omwe amayang'ana kwambiri zomwe zimachitika nthawi yayitali.
• Kibana: Amapereka mawonekedwe / kusaka kwa chipika, kuthekera kukumba mozama muzochitika zinazake.
• Zoom: Pali "chipinda chopumira" chomwe chikuyenda nthawi zonse mu Zoom. Izi zimalola akatswiri a SRE kukambirana mwachangu zochitika popanda kuwononga nthawi yofunika kupanga chipinda ndikulumikiza otenga nawo mbali.

Ndi ena ambiri.

4. Kuwunika GitLab.com ndi GitLab ndi gawo limodzi lolephera

Ngati GitLab.com ikumana ndi vuto lalikulu, sitikufuna kuti izitikhudze pakuthana ndi vutoli. Itha kuyimitsidwa poyambitsa chitsanzo chachiwiri cha GitLab kuyang'anira GitLab.com. M'malo mwake, izi zimagwira ntchito kwa ife: https://ops.gitlab.net/.

5. Zina zingapo zomwe mungaganizire kuwonjezera ku GitLab

• Kusintha ntchito kwa ogwiritsa ntchito ambiri, zofanana ndi Google Docs. Izi zingathandize ndi ntchito pazochitika pazochitika, komanso ntchito zofotokozera. Muzochitika zonsezi, ambiri omwe atenga nawo mbali angafunike kuwonjezera zina mu nthawi yeniyeni.
• Ma webhooks ambiri a ntchito. Kutha kuyendetsa masitepe osiyanasiyana a GitLab kuchokera mkati kudzakuthandizani kuchepetsa kudalira kwanu pakuphatikiza kwa Slack. Mwachitsanzo, kuthekera kulola chenjezo mu PagerDuty kudzera pa lamulo la slash mu nkhani ya GitLab.
  Pomaliza

Akatswiri a SRE amavutika ndi zovuta zambiri. Zingakhale zabwino kuwona zinthu zambiri za GitLab zomwe zikuthandizira izi. Tikugwira ntchito kale pazowonjezera zina zomwe zingapangitse kuti ntchito zotchulidwa pamwambapa zikhale zosavuta. Zambiri zilipo pa Gawo la Ops Product Vision.

Tikukulitsa gulu mu 2020 kuti tibweretse zinthu zabwino zonsezi pamodzi. Ngati mukufuna, chonde onani ntchito, ndipo khalani omasuka kulumikizana ndi aliyense pagulu lathu ndi mafunso aliwonse.

Source: www.habr.com