🥇Momwe mungayang'anire ma disks okhala ndi fio kuti agwire ntchito mokwanira etcd

Zindikirani. transl.: Nkhaniyi ndi zotsatira za kafukufuku wochepa wopangidwa ndi IBM Cloud engineers pofunafuna njira yothetsera vuto lenileni lokhudzana ndi ntchito ya database ya etcd. Ntchito yofananayi inali yofunika kwa ife, komabe, kulingalira ndi zochita za olemba kungakhale kosangalatsa muzochitika zambiri.

Chidule chachidule cha nkhani yonse: fio ndi etcd

Kuchita kwa gulu la etcd kumadalira kwambiri kuthamanga kwa malo osungiramo. etcd imatumiza ma metric osiyanasiyana a Prometheus kuti aziwunika momwe ntchito ikuyendera. Mmodzi wa iwo ndi wal_fsync_duration_seconds. Muzolemba za etcd akutikuti kusungirako kumatha kuganiziridwa mwachangu ngati 99th percentile ya metric iyi sidutsa 10 ms…

Ngati mukuganiza zokhazikitsa gulu la etcd pamakina a Linux ndipo mukufuna kuyesa ngati ma drive (monga ma SSD) akuthamanga mokwanira, timalimbikitsa kugwiritsa ntchito choyesa chodziwika bwino cha I/O chotchedwa. fio. Ndikokwanira kuyendetsa lamulo lotsatirali (directory test-data iyenera kukhala pagawo lokhazikitsidwa la drive yoyesedwa):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Zimangoyang'ana zomwe zatuluka ndikuwona ngati 99th percentile ikukwanira fdatasync pa 10ms. Ngati ndi choncho, ndiye kuti galimoto yanu ikugwira ntchito mofulumira mokwanira. Nachi chitsanzo chotulutsa:

fsync/fdatasync/sync_file_range:
  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Zolemba zingapo:

Mu chitsanzo pamwambapa, tasintha magawo --size и --bs pa nkhani inayake. Kuti mupeze zotsatira zabwino kuchokera fio, tchulani zikhalidwe zoyenera pazogwiritsa ntchito. Momwe mungasankhire zidzakambidwa pansipa.
Panthawi yoyesedwa kokha fio imatsitsa disk subsystem. M'moyo weniweni, ndizotheka kuti njira zina (kupatula zomwe zimagwirizanitsidwa ndi wal_fsync_duration_seconds). Mtolo wowonjezerawu ukhoza kuwonjezeka wal_fsync_duration_seconds. Mwanjira ina, ngati 99th percentile kuchokera kuyezetsa ndi fio, kuchepera pang'ono kuposa 10 ms, pali mwayi woti kusungirako kusungidwe sikokwanira.
Kwa mayeso mudzafunika Baibulo fio osakwana 3.5, chifukwa zomasulira zakale siziphatikiza zotsatira fdatasync mu mawonekedwe a percentiles.
Mawu omaliza pamwambawa ndi gawo laling'ono chabe la mawu omaliza fio.

Zambiri za fio ndi etcd

Mawu ochepa okhudza WALs etcd

Nthawi zambiri, ma database amagwiritsa ntchito kudula mitengo mwachangu (lembani mitengo, WAL). etcd imakhudzidwanso. Kukambitsirana kwa WAL sikupitilira kuchuluka kwa nkhaniyi, koma pazolinga zathu, zomwe muyenera kudziwa ndikuti membala aliyense wa gulu la etcd amasunga WAL posungira mosalekeza. etcd imalemba zosungirako zamtengo wapatali (monga zosintha) ku WAL musanazichite. Node ikasokonekera ndikuyambiranso pakati pazithunzi, etcd idzatha kubweza zomwe zachitika kuyambira pachithunzi cham'mbuyomu kutengera zomwe zili mu WAL.

Choncho, nthawi iliyonse kasitomala akuwonjezera kiyi ku sitolo ya KV kapena kukonzanso mtengo wachinsinsi chomwe chilipo, etcd amawonjezera kufotokozera kwa ntchitoyo ku WAL, yomwe ndi fayilo yokhazikika mu sitolo yosalekeza. etcd AYENERA kukhala otsimikiza 100% kuti kulowa kwa WAL kwasungidwa musanayambe. Kuti mukwaniritse izi pa Linux, sikokwanira kugwiritsa ntchito kuyimba foni write, popeza ntchito yolemba yokha kuzinthu zakuthupi ikhoza kuchedwa. Mwachitsanzo, Linux ikhoza kusunga cholowa cha WAL mu kache ya kernel yokumbukira (monga pa cache yatsamba) kwakanthawi. Kuonetsetsa kuti deta yalembedwa kwa atolankhani, kuyitana kwadongosolo kuyenera kuyitanidwa pambuyo polemba fdatasync - izi ndi zomwe etcd amachita (monga mukuwonera pazotsatira zotsatirazi strace; Pano 8 - Wofotokozera mafayilo a WAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ".20210220361223255266632$1020103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Tsoka ilo, kulembera kusungirako kosalekeza kumatenga nthawi. Kuchita kwanthawi yayitali kwa mafoni a fdatasync kumatha kukhudza magwiridwe antchito a etcd. M'mabuku osungira anasonyeza, kuti pakuchita mokwanira ndikofunikira kuti 99 peresenti ya nthawi ya mafoni onse fdatasync polembera fayilo ya WAL inali yosakwana 10 ms. Palinso ma metric okhudzana ndi kusungirako, koma nkhaniyi ifotokoza za izi.

Kutengera kusungirako ndi fio

Mutha kuwunika ngati chosungirako ndi choyenera kugwiritsidwa ntchito ndi etcd pogwiritsa ntchito chidacho fio - Woyesa I/O wotchuka. Kumbukirani kuti disk I/O ikhoza kuchitika m'njira zosiyanasiyana: kulunzanitsa/async, makalasi ambiri a syscall, ndi zina zotero. Mbali ina ya ndalama ndi imeneyo fio zovuta kwambiri kugwiritsa ntchito. Zomwe zimagwiritsidwa ntchito zimakhala ndi magawo ambiri, ndipo kuphatikiza kosiyanasiyana kwazomwe zimatengera kumabweretsa zotsatira zosiyana. Kuti mupeze chiyerekezo choyenera cha etcd, muyenera kuwonetsetsa kuti zolemba zolembedwa ndi fio zili pafupi kwambiri ndi etcd's WAL file kulemba katundu:

Izi zikutanthauza kuti amapangidwa fio katunduyo ayenera kukhala mndandanda wa zolemba zotsatizana ku fayilo, pomwe kulemba kulikonse kumakhala ndi kuyimba kwadongosolo. writeotsatidwa ndi fdatasync.
Kuti muthe kulemba motsatizana, muyenera kufotokoza mbendera --rw=write.
kuti fio analemba pogwiritsa ntchito mafoni write (m'malo mwa mafoni ena - mwachitsanzo, pwrite), gwiritsani ntchito mbendera --ioengine=sync.
Pomaliza, mbendera --fdatasync=1 zimatsimikizira kuti aliyense write ayenera kukhala fdatasync.
Njira zina ziwiri mu chitsanzo chathu ndi: --size и --bs - zitha kusiyanasiyana kutengera momwe amagwiritsidwira ntchito. Gawo lotsatira lidzalongosola kasinthidwe kawo.

Chifukwa chiyani tinasankha fio ndi momwe taphunzirira momwe tingakhazikitsire

Cholemba ichi chikuchokera kuzochitika zenizeni zomwe tidakumana nazo. Tidali ndi gulu pa Kubernetes v1.13 ndikuwunika pa Prometheus. Ma SSD adagwiritsidwa ntchito ngati yosungirako etcd v3.2.24. Etcd ma metrics adawonetsa kuchedwa kwambiri fdatasync, ngakhale gululo linali lopanda ntchito. Kwa ife, ma metricswa adawoneka ngati okayikitsa kwambiri, ndipo sitinkadziwa chomwe amayimira. Kuphatikiza apo, gululi linali ndi makina owoneka bwino, kotero sikunali kotheka kunena ngati kuchedwako kudachitika chifukwa cha Virtualization kapena SSD ndiyomwe idayambitsa.

Kuphatikiza apo, tidawona kusintha kosiyanasiyana pamakonzedwe a hardware ndi mapulogalamu, kotero tidafunikira njira yowunikira. Zachidziwikire, zitha kukhala zotheka kuthamanga etcd pamasinthidwe aliwonse ndikuyang'ana ma metric ofananira a Prometheus, koma izi zingafune kuyesetsa kwakukulu. Zomwe tinkafunikira zinali njira yosavuta yowunikira masanjidwe enaake. Tinkafuna kuyesa kumvetsetsa kwathu ma metric a Prometheus akuchokera etcd.

Izi zinafuna kuthetsa mavuto awiri:

Choyamba, kodi katundu wa I / O wopangidwa ndi etcd polemba mafayilo a WAL amawoneka bwanji? Ndi mafoni amtundu wanji omwe amagwiritsidwa ntchito? Kodi kukula kwa mabuloko ojambulira ndi chiyani?
Kachiwiri, tinene kuti tili ndi mayankho a mafunso ali pamwambawa. Momwe mungapangirenso katundu wolingana ndi fio? Izi zili choncho fio - chida chosinthika kwambiri chokhala ndi magawo ambiri (izi ndizosavuta kutsimikizira, mwachitsanzo, apa - pafupifupi. transl.).

Tinathetsa mavuto onsewa ndi njira imodzi yotengera malamulo lsof и strace:

Ndi chithandizo cha lsof mutha kuwona zolemba zonse zamafayilo zomwe zimagwiritsidwa ntchito ndi ndondomekoyi, komanso mafayilo omwe amawatchula.
Ndi chithandizo cha strace mukhoza kusanthula ndondomeko yomwe ikuyenda kale kapena kuyendetsa ndondomeko ndikuwona. Lamulo likuwonetsa mafoni onse opangidwa ndi njirayi ndipo, mwachisawawa, mbadwa zake. Chotsatiracho ndi chofunikira pamachitidwe omwe akufota, ndipo etcd ndi imodzi mwazinthu zotere.

Chinthu choyamba chimene tinachita chinali kugwiritsa ntchito strace kuti muwone seva ya etcd mu gulu la Kubernetes pomwe inali yopanda ntchito.

Chifukwa chake zidapezeka kuti midadada ya WAL imakhala m'magulu ambiri, kukula kwaunyinji kunali mumitundu ya 2200-2400 byte. Ndiye chifukwa chake lamulo lomwe lili koyambirira kwa nkhaniyi likugwiritsa ntchito mbendera --bs=2300 (bs ndi kukula kwa mabayiti a chipika cholembera chilichonse fio).

Chonde dziwani kuti kukula kwa etcd kulemba midadada kungasiyane kutengera mtundu, kutumiza, mayendedwe, ndi zina. - zimakhudza nthawi fdatasync. Ngati muli ndi vuto lofananalo, santhulani ndi strace etcd njira zanu kuti mupeze zosintha zatsopano.

Kenako, kuti timvetsetse bwino komanso momveka bwino momwe etcd imagwirira ntchito ndi mafayilo amafayilo, timayiyendetsa kuchokera pansi. strace ndi mbendera -ffttT. Izi zidapangitsa kuti zitheke kujambula njira za ana ndikulemba zomwe zatulutsidwa ku fayilo yosiyana. Kuphatikiza apo, zambiri za nthawi yoyambira komanso nthawi ya kuyimba kulikonse kwadongosolo zidapezedwa.

Tinagwiritsanso ntchito lamulo lsofkuti mutsimikizire kuti mukumvetsa zomwe zatuluka strace momwe fayilo yofotokozera idagwiritsidwa ntchito pazifukwa zake. Ndapeza mfundo strace, zofanana ndi zomwe zili pamwambapa. Ziwerengero zokhala ndi nthawi yolumikizira zidatsimikizira kuti metric wal_fsync_duration_seconds kuchokera ku etcd machesi mafoni fdatasync ndi WAL mafayilo ofotokozera.

Kupanga ndi fio ntchito yofanana ndi ya etcd, zolemba zazomwe zimagwiritsidwa ntchito zidaphunziridwa ndipo magawo oyenerera ntchito yathu adasankhidwa. Tatsimikizira kuti ma foni olondola amachitidwe ali mkati ndikutsimikizira nthawi yawo poyendetsa fio kuchokera strace (monga momwe zidachitikira etcd).

Chisamaliro chapadera chinaperekedwa pozindikira mtengo wa parameter --size. Zimayimira kuchuluka kwa I / O komwe kumapangidwa ndi fio utility. Kwa ife, ichi ndi chiwerengero chonse cha ma byte olembedwa kwa atolankhani. Zimagwirizana mwachindunji ndi chiwerengero cha mafoni write (ndi fdatasync). Zachindunji bs nambala ya mafoni fdatasync zofanana size / bs.

Popeza tinali ndi chidwi ndi percentile, tinali ndi cholinga chowonetsetsa kuti chiwerengero cha zitsanzo chinali chachikulu mokwanira kuti chikhale chofunikira. Ndipo ndinaganiza zimenezo 10^4 (yomwe ikufanana ndi kukula kwa 22 MB) idzakwanira. Magawo ang'onoang'ono --size adapereka phokoso lodziwika bwino (mwachitsanzo, mafoni fdatasync, zomwe zimatenga nthawi yayitali kuposa nthawi zonse ndipo zimakhudza 99th percentile).

Zili ndi inu

Nkhaniyi ikuwonetsa momwe mungagwiritsire ntchito fio munthu akhoza kuwunika ngati zoulutsira nkhani zomwe zimafunidwa kuti zigwiritsidwe ntchito ndi etcd ndizothamanga mokwanira. Tsopano zili ndi inu! Mutha kuyang'ana makina enieni omwe ali ndi zosungirako za SSD muutumiki Mtambo wa IBM.

PS kuchokera kwa womasulira

Ndi milandu yogwiritsira ntchito okonzeka fio Kwa ntchito zina, onani zolemba kapena mwachindunji ku nkhokwe za polojekiti (pali zambiri za izo kuposa zomwe zatchulidwa muzolemba).

PPS kuchokera kwa womasulira

Werenganinso pa blog yathu:

Source: www.habr.com

Momwe mungayang'anire ma disks okhala ndi fio kuti agwire ntchito mokwanira etcd