🥇Ungawahlola kanjani amadiski ukusebenza okwanele kwe- etcd nge-fio

Qaphela. transl.: Lesi sihloko siwumphumela wocwaningo oluncane olwenziwa onjiniyela be-IBM Cloud befuna isixazululo senkinga yangempela ehlobene nokusebenza kwesizindalwazi njll. Umsebenzi ofanayo ububalulekile kithi, nokho, inkambo yokucabangisisa kanye nezenzo zababhali zingase zithakazelise esimweni esibanzi.

Isifinyezo esifushane sayo yonke i-athikili: fio kanye nokunye

Ukusebenza kweqoqo etcd kuncike kakhulu esivinini sesitoreji esingaphansi. njlld ithumela kwamanye amazwe amamethrikhi e-Prometheus ukuze igade ukusebenza. Omunye wabo wal_fsync_duration_seconds. Embhalweni we- etcd kusholeso sitoreji singabhekwa sishesha ngokwanele uma iphesenti elingu-99 lale metric lingeqi ku-10 ms...

Uma ucabanga ukusetha iqoqo njlld emishinini ye-Linux futhi ufuna ukuhlola ukuthi ingabe amadrayivu (afana nama-SSD) ayashesha ngokwanele yini, sincoma ukusebenzisa isihloli se-I/O esidumile esibizwa ngokuthi. fio. Kwanele ukusebenzisa umyalo olandelayo (directory test-data kufanele ibe sengxenyeni egingqiwe yedrayivu ehloliwe):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Kuhlala kuphela ukubheka okukhiphayo nokuhlola ukuthi i-percentile engu-99 iyalingana yini fdatasync kwe 10ms. Uma kunjalo, khona-ke idrayivu yakho isebenza ngokushesha ngokwanele. Nasi isibonelo sokukhiphayo:

fsync/fdatasync/sync_file_range:
  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Amanothi ambalwa:

Esibonelweni esingenhla, silungise amapharamitha --size и --bs ngecala elithile. Ukuze uthole umphumela ophusile ku fio, cacisa amanani afanele isimo sakho sokusebenzisa. Indlela yokukhetha kuzo kuzoxoxwa ngayo ngezansi.
Ngesikhathi sokuhlolwa kuphela fio ilayisha isistimu engaphansi yediski. Empilweni yangempela, kungenzeka ukuthi ezinye izinqubo zizobhalela kudiski (ngaphandle kwalezo ezihlobene ne wal_fsync_duration_seconds). Lo mthwalo owengeziwe ungakhula wal_fsync_duration_seconds. Ngamanye amazwi, uma i-percentile engu-99 kusukela ekuhloleni nge fio, ngaphansi kancane kuka-10 ms kuphela, kunethuba elihle lokuthi ukusebenza kwesitoreji akwanele.
Ukuze uthole ukuhlolwa uzodinga inguqulo fio hhayi ngaphansi kuka-3.5, ngoba izinguqulo ezindala azihlanganisi imiphumela fdatasync ngendlela yamaphesenti.
Isiphetho esingenhla siyingxenye encane yesiphetho esijwayelekile fio.

Okuningi mayelana ne-fio nokunye

Amagama ambalwa mayelana nama-WAL njll

Ngokuvamile, i-database isetshenziswa ukugawula okusebenzayo (bhala-phambili ukugawulwa kwemithi, WAL). etcd nayo iyathinteka. Ingxoxo ye-WAL ingaphezu kobubanzi balesi sihloko, kodwa ngezinhloso zethu, okudingeka ukwazi ukuthi ilungu ngalinye leqoqo le- etcd ligcina i-WAL endaweni yokugcina eqhubekayo. njlld ibhala imisebenzi ethile yokugcina yenani elingukhiye (njengezibuyekezo) ku-WAL ngaphambi kokuyiqalisa. Uma i-node iphahlazeka futhi iqala kabusha phakathi kwezifinyezo, njlld ingakwazi ukubuyisela okwenziwayo kusukela kusifinyezo sangaphambilini ngokusekelwe kokuqukethwe kwe-WAL.

Ngakho, isikhathi ngasinye iklayenti lengeza ukhiye esitolo se-KV noma libuyekeza inani lokhiye okhona, njlld lengeza incazelo yokusebenza ku-WAL, okuyifayela elivamile esitolo esiqhubekayo. etcd KUMELE uqiniseke ngo-100% ukuthi okufakiwe kwe-WAL kulondoloziwe ngempela ngaphambi kokuqhubeka. Ukufeza lokhu ku-Linux, akwanele ukusebenzisa ikholi yesistimu write, njengoba umsebenzi wokubhala ngokwawo emithonjeni yezokuxhumana ungase ubambezeleke. Isibonelo, i-Linux ingase igcine okufakiwe kwe-WAL kunqolobane ye-kernel yememori (isb, kunqolobane yekhasi) isikhathi esithile. Ukuqinisekisa ukuthi idatha ibhaliwe kwabezindaba, ucingo lwesistimu kufanele lufakwe ngemuva kokubhalwa fdatasync - yilokhu kanye i- etcd ekwenzayo (njengoba ungabona kokuphumayo okulandelayo strace; Lapha 8 - Isichazi sefayela le-WAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ".      20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Ngeshwa, ukubhalela isitoreji esiqhubekayo kuthatha isikhathi. Ukwenza isikhathi eside kwamakholi we-fdatasync kungase kuthinte ukusebenza kwe- etcd. Embhalweni wenqolobane kukhonjisiwe, ukuthi ukuze kusebenze okwanele kuyadingeka ukuthi iphesenti elingu-99 lobude besikhathi bazo zonke izingcingo fdatasync ngenkathi ubhalela ifayela le-WAL bekungaphansi kuka-10 ms. Akhona amanye amamethrikhi ahlobene nesitoreji, kodwa lesi sihloko sizogxila kulokho.

Ukulinganisa isitoreji nge-fio

Ungahlola ukuthi ingabe isitoreji esithile sifanelekile yini ukusetshenziswa ne-etcd usebenzisa insiza fio — umhloli we-I/O odumile. Khumbula ukuthi i-disk I/O ingenzeka ngezindlela eziningi ezahlukene: ukuvumelanisa/ukuvumelanisa, amakilasi amaningi e-syscall ahlukene, njalo njalo. Olunye uhlangothi lwemali yilokho fio kunzima kakhulu ukusebenzisa. Insiza inamapharamitha amaningi, futhi inhlanganisela ehlukene yamanani awo iholela emiphumeleni ehluke ngokuphelele. Ukuze uthole isilinganiso esifanele se-etcd, udinga ukwenza isiqiniseko sokuthi umthwalo wokubhala okhiqizwe i-fio useduze kakhulu nomthwalo wokubhala wefayela le-WAL njll:

Lokhu kusho ukuthi okukhiqizwayo fio umthwalo kufanele okungenani ube uchungechunge lokubhala okulandelanayo efayeleni, lapho ukubhala ngakunye kuqukethe ucingo lwesistimu. writeulandelwa ngu fdatasync.
Ukuze unike amandla ukubhala okulandelanayo, kufanele ucacise ifulegi --rw=write.
ukuthi fio wabhala esebenzisa izingcingo write (kunokuba ezinye izingcingo zesistimu - isibonelo, pwrite), sebenzisa ifulege --ioengine=sync.
Ekugcineni, ifulege --fdatasync=1 iqinisekisa ukuthi wonke write iyaqhubeka fdatasync.
Amanye amapharamitha amabili esibonelweni sethu yilezi: --size и --bs - zingahluka kuye ngesimo esithile sokusetshenziswa. Isigaba esilandelayo sizochaza ukucushwa kwazo.

Kungani sikhethe i-fio nokuthi sifunde kanjani ukuyimisa

Leli nothi livela endabeni yangempela esihlangabezane nayo. Sibe neqoqo ku-Kubernetes v1.13 ngokuqapha ku-Prometheus. Ama-SSD asetshenziswe njengendawo yokugcina izinto njlld v3.2.24. I-Etcd metrics ibonise ukubambezeleka okuphezulu kakhulu fdatasync, ngisho nalapho iqoqo lalingenzi lutho. Kithina, lawa mamethrikhi abonakala engabaza kakhulu, futhi besingaqiniseki ukuthi amele ini ngempela. Ngaphezu kwalokho, iqoqo lalihlanganisa imishini ebonakalayo, ngakho-ke kwakungenakwenzeka ukusho ukuthi ukubambezeleka kwakungenxa ye-virtualization noma i-SSD yayinecala.

Ngaphezu kwalokho, sicabangele izinguquko ezihlukahlukene ku-hardware ne-software ukucushwa, ngakho sidinga indlela yokuzihlola. Vele, kungenzeka ukusebenzisa njlld ekucushweni ngakunye futhi ubheke amamethrikhi e-Prometheus ahambisanayo, kodwa lokho kungadinga umzamo omkhulu. Esasikudinga kwakuyindlela elula yokuhlola ukucushwa okuthile. Besifuna ukuhlola ukuqonda kwethu amamethrikhi e-Prometheus avela njlld.

Lokhu kwakudinga ukuxazulula izinkinga ezimbili:

Okokuqala, ingabe umthwalo we-I/O okhiqizwe njlld lapho ubhalela amafayela e-WAL ubukeka kanjani? Yiziphi izingcingo zesistimu ezisetshenziswayo? Bungakanani usayizi wamabhulokhi werekhodi?
Okwesibili, ake sithi sinezimpendulo zemibuzo engenhla. Ungakhiqiza kanjani kabusha umthwalo ohambelana ne fio? Ngemva kwakho konke fio - Usizo oluguquguquka kakhulu olunenqwaba yamapharamitha (lokhu kulula ukukuqinisekisa, isibonelo, lapha - cishe. transl.).

Sixazulule zombili izinkinga ngendlela efanayo esekelwe kumyalo lsof и strace:

Ngosizo luka lsof ungabuka zonke izichazi zefayela ezisetshenziswe inqubo, kanye namafayela ababhekisela kuwo.
Ngosizo luka strace ungakwazi ukuhlaziya inqubo esebenzayo kakade noma wenze inqubo futhi uyibuke. Umyalo ubonisa zonke izingcingo zesistimu ezenziwe yile nqubo futhi, uma kunesidingo, inzalo yayo. Lesi sakamuva sibalulekile ezinqubweni eziyimfoloko, futhi njlld ingenye yenqubo enjalo.

Into yokuqala esiyenzile kwakuwukusebenzisa strace ukuhlola iseva etcd kuqoqo le-Kubernetes ngenkathi lingenzi lutho.

Ngakho-ke kwatholakala ukuthi amabhulokhi amarekhodi e-WAL ahlanganiswe kakhulu, usayizi weningi wawukububanzi obungamabhayithi angu-2200-2400. Kungakho umyalo osekuqaleni kwalesi sihloko usebenzisa ifulege --bs=2300 (bs ungusayizi wamabhayithi webhulokhi ngayinye yokubhala fio).

Sicela uqaphele ukuthi usayizi wamabhulokhi wokubhala njll ungahluka kuye ngenguqulo, ukusetshenziswa, amanani epharamitha, njll. - kuthinta ubude besikhathi fdatasync. Uma unecala lokusebenzisa elifanayo, hlaziya nge strace izinqubo zakho njll ukuze uthole amanani akamuva.

Bese, ukuze sithole umbono ocacile nophelele wokuthi njlld isebenza kanjani nesistimu yefayela, siyiqale ngaphansi strace namafulegi -ffttT. Lokhu kwenze kwaba nokwenzeka ukuthwebula izinqubo zengane nokubhala okukhiphayo ngakunye efayeleni elihlukile. Ngaphezu kwalokho, kutholwe ulwazi oluningiliziwe mayelana nesikhathi sokuqala kanye nobude bekholi yesistimu ngayinye.

Siphinde sasebenzisa umyalo lsofukuze uqinisekise ukuqonda kwakho okuphumayo strace ngokwendlela isichazi sefayela esasetshenziselwa yiphi inhloso. Ngithole isiphetho strace, efana nale engenhla. Ukukhohlisa kwezibalo ngezikhathi zokuvumelanisa kuqinisekisile ukuthi imethrikhi wal_fsync_duration_seconds kusuka kumakholi we-etcd fdatasync ngezichazi zefayela le-WAL.

Ukukhiqiza nge fio umthwalo wokusebenza ofana nalowo ovela njlld, imibhalo yensiza yafundwa futhi nemingcele elungele umsebenzi wethu yakhethwa. Siqinisekise ukuthi amakholi esistimu alungile ayaqhubeka futhi saqinisekisa ubude bawo ngokusebenza fio kusuka ku strace (njengoba kwenziwa uma kwenzeka njlld).

Ukunaka ngokukhethekile kukhokhwe ekunqumeni inani lepharamitha --size. Imele isamba somthwalo we-I/O okhiqizwe insiza ye-fio. Esimweni sethu, leli inani eliphelele lamabhayithi abhalelwe abezindaba. Ilingana ngqo nenani lezingcingo write (futhi fdatasync). Ukuze uthole okuthile bs inombolo yezingcingo fdatasync ngokulinganayo size / bs.

Njengoba besinentshisekelo kuphesenti, besifuna ukuthi inani lamasampuli libe likhulu ngokwanele ukuthi libaluleke ngokwezibalo. Futhi wanquma lokho 10^4 (okuhambisana nosayizi ongu-22 MB) kuzokwanela. Amanani epharamitha amancane --size inikeze umsindo ogqame kakhulu (ngokwesibonelo, izingcingo fdatasync, okuthatha isikhathi eside kunokuvamile futhi kuthinte i-99th percentile).

Kuphuma kuwe

Isihloko sibonisa indlela yokusebenzisa fio umuntu angakwazi ukwahlulela ukuthi imidiya ehloselwe ukusetshenziswa ne-etcd ishesha ngokwanele yini. Manje kukuwe! Ungakwazi ukuhlola imishini ebonakalayo enesitoreji esisekelwe ku-SSD kusevisi I-IBM Cloud.

I-PS evela kumhumushi

Ngamacala okusetshenziswa esenziwe ngomumo fio Ngeminye imisebenzi, bheka imibhalo noma ngqo ku amaqoqo amaphrojekthi (ziningi zazo kunalezo ezishiwo embhalweni).