🥇Ingabe isivinini sokugcina sifanele njlld? Asibuze fio

Indaba emfushane mayelana ne-fio nokunye

Ukusebenza kweqoqo njll kakhulu kuncike ekusebenzeni kwesitoreji sayo. etcd ithekelisa amanye amamethrikhi kuwo Prometheusukuhlinzeka ngolwazi olufiselekayo lokusebenza kwesitoreji. Isibonelo, i-wal_fsync_duration_seconds metric. Amadokhumenti e- etcd athi: Ukuze isitoreji sicatshangelwe ngokushesha okwanele, i-99th percentile yalesi silinganiso kufanele ibe ngaphansi kuka-10 ms. Uma uhlela ukusebenzisa i-etcd cluster emishinini Linux futhi ufuna ukuhlola ukuthi isitoreji sakho (njenge-SSD) sishesha ngokwanele, ungasebenzisa fio iyithuluzi elidumile lokuhlola imisebenzi ye-I/O. Qalisa umyalo olandelayo, lapho idatha yokuhlola iyinkomba ngaphansi kwendawo yokubeka isitoreji:

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Udinga nje ukubuka imiphumela futhi uhlole ukuthi amaphesenti angama-99 wesikhathi fdatasync ngaphansi kuka-10 ms. Uma kunjalo, unesitoreji esisheshayo ngokunengqondo. Nasi isibonelo semiphumela:

  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Amanothi

Senze ngokwezifiso izinketho ze--size kanye --bs zesimo sethu esithile. Ukuze uthole umphumela owusizo ku-fio, hlinzeka ngamavelu akho. Ungawatholaphi? Funda ukuthi sifunde kanjani ukumisa i-fio.
Ngesikhathi sokuhlolwa, wonke umthwalo we-I/O uvela ku-fio. Esimeni sempilo yangempela, kungase kube nezinye izicelo zokubhala ezifika esilondolozini ngaphandle kwalezo ezihlobene nama-wal_fsync_duration_seconds. Umthwalo owengeziwe uzokhuphula inani lama-wal_fsync_duration_seconds. Ngakho-ke uma i-percentile engu-99 iseduze no-10ms, isitoreji sakho siphelelwa isivinini.
Thatha inguqulo fio hhayi ngaphansi kuka-3.5 (ezandulele azibonisi ubude besikhathi be-fdatasync).
Ngenhla amazwibela nje emiphumela evela ku-fio.

Indaba ende mayelana ne-fio nokunye

Iyini i-WAL njll

Ngokuvamile ukusetshenziswa kolwazi bhala-phambili log; etcd iyayisebenzisa futhi. Ngeke sixoxe ngelogi yokubhala phambili (WAL) ngokuningiliziwe lapha. Kwanele ukuthi sazi ukuthi ilungu ngalinye leqoqo le- etcd liyigcina endaweni yokugcina eqhubekayo. njlld ibhala umsebenzi ngamunye wenani lokhiye (okufana nesibuyekezo) ku-WAL ngaphambi kokuwusebenzisa esitolo. Uma elinye lamalungu esitoreji liphahlazeka futhi liqala kabusha phakathi kwezifinyezo, lingakwazi ukubuyisela okwenziwayo endaweni kusukela esithombweni sokugcina sokuqukethwe kwe-WAL.

Uma iklayenti lengeza ukhiye esitolo senani lokhiye noma libuyekeza inani lokhiye okhona, i-etcd iqopha lo msebenzi ku-WAL, okuyifayela elijwayelekile eligcinwe njalo. Ngaphambi kokuqhubeka nokucubungula, i-etcd KUMELE iqiniseke ngokuphelele ukuthi ukubhala kwe-WAL kwenzeke ngempela. Linux ucingo olulodwa lwesistimu alwanele kulokhu bhala, njengoba ukubhala kwangempela endaweni yokugcina izinto kungase kubambezeleke. Isibonelo, Linux ingagcina irekhodi le-WAL ku-kernel memory cache (isb., ikhasi le-cache) isikhathi esithile. Ukuqinisekisa ukuthi idatha ibhalwe kwisitoreji esiqhubekayo, kudingeka ucingo lwesistimu ye-fdatasync ngemuva kokubhala, futhi i-etcd isebenzisa lokhu (njengoba kungabonakala kumphumela). intambo, lapho u-8 eyisichazi sefayela le-WAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Ngeshwa, ukubhalela isitoreji esiqhubekayo akwenzeki ngokushesha. Uma ikholi ye-fdatasync ihamba kancane, ukusebenza kwesistimu njll kuzophazamiseka. Amadokhumenti e- etcd athiukuthi isitoreji sibhekwa njengokushesha ngokwanele uma, kumaphesenti angama-99, izingcingo ze-fdatasync zithatha ngaphansi kuka-10ms ukubhala kufayela le-WAL. Akhona amanye amamethrikhi awusizo esitoreji, kodwa kulokhu okuthunyelwe sikhuluma ngale metric kuphela.

Ilinganisa isitoreji nge-fio

Uma udinga ukuhlola ukuthi isitoreji sakho sifanele njlld, sebenzisa i-fio, ithuluzi elidume kakhulu le-I/O lokuhlola ukulayisha. Kufanele kukhunjulwe ukuthi imisebenzi yediski ingahluka kakhulu: i-synchronous ne-asynchronous, amakilasi amaningi wezingcingo zesistimu, njll. Ngenxa yalokho, i-fio inzima kakhulu ukuyisebenzisa. Inamapharamitha amaningi, futhi inhlanganisela ehlukene yamanani azo ikhiqiza imithwalo yemisebenzi ye-I/O ehluke kakhulu. Ukuze uthole izibalo ezanele ze- etcd, kufanele uqiniseke ukuthi umthwalo wokubhala wokuhlola ovela ku-fio useduze ngangokunokwenzeka nomthwalo wangempela ovela njlld lapho ubhala amafayela we-WAL.

Ngakho-ke, i-fio kufanele, okungenani, idale umthwalo ngendlela yochungechunge lokubhala okulandelanayo efayeleni, ukubhala ngakunye kuzoba nocingo lwesistimu. bhalakulandelwe ikholi yesistimu ye-fdatasync. Okulandelanayo kubhala ku-fio kudinga inketho ethi --rw=write. Ukuze i-fio isebenzise ikholi yesistimu yokubhala lapho ibhala, kunokuba bhala, kufanele ucacise ipharamitha --ioengine=sync. Okokugcina, ukuze ushayele i-fdatasync ngemva kokubhala ngakunye, udinga ukwengeza --fdatasync=1 ipharamitha. Ezinye izinketho ezimbili kulesi sibonelo (--size kanye -bs) ziqondene nombhalo othile. Esigabeni esilandelayo, sizokukhombisa ukuthi ungawamisa kanjani.

Kungani ngempela i-fio nokuthi sifunde kanjani ukuyibeka

Kulokhu okuthunyelwe, sichaza icala langempela. Sineqoqo Kubernetes I-v1.13 esiyiqaphe nge-Prometheus. etcd v3.2.24 isingathwe ku-SSD. I-Etcd metrics ibonise ukubambezeleka kwe-fdatasync kuphezulu kakhulu, nanoma iqoqo lingenzi lutho. Amamethrikhi ayeyinqaba futhi sasingazi ngempela ukuthi ayesho ukuthini. Iqoqo lalihlanganisa imishini ebonakalayo, kwakudingeka ukuqonda ukuthi yayiyini inkinga: kuma-SSD angokwenyama noma isendlalelo se-virtualization. Ngaphezu kwalokho, sasivame ukwenza izinguquko ku-hardware ne-software ukucushwa, futhi sasidinga indlela yokuhlola imiphumela yabo. Singasebenzisa njlld kukho konke ukucushwa futhi sibheke amamethrikhi e-Prometheus, kodwa lokho kuyinkinga enkulu kakhulu. Besifuna indlela elula yokuhlola ukucushwa okuthile. Besifuna ukuhlola ukuthi siyawaqonda yini amamethrikhi e-Prometheus kusuka njlld ngendlela efanele.

Kodwa ngenxa yalokhu, izinkinga ezimbili kwakufanele zixazululwe. Okokuqala, ingabe umthwalo we-I/O odalwa njlld uma ubhalela ku-WAL ubukeka kanjani? Yiziphi izingcingo zesistimu ezisetshenziswayo? Bungakanani usayizi wamarekhodi? Okwesibili, uma siphendula le mibuzo, siwenza kanjani umsebenzi ofanayo nge-fio? Ungakhohlwa ukuthi i-fio iyithuluzi eliguquguqukayo kakhulu elinezinketho eziningi. Sixazulule zombili izinkinga ngendlela eyodwa - sisebenzisa imiyalo lsof и intambo. I-lsof ibala zonke izincazelo zefayela ezisetshenziswa inqubo kanye namafayela azo ahlobene. Futhi nge-strace, ungahlola inqubo esivele isebenza, noma uqale inqubo futhi uyihlole. i-strace iphrinta zonke izingcingo zesistimu kusukela kunqubo ehlolwayo (kanye nezinqubo zayo zengane). Lokhu kokugcina kubaluleke kakhulu, njengoba njlld ithatha indlela efanayo.

Siqale sasebenzisa i-strace ukuhlola iseva etcd ye-Kubernetes lapho kungekho mthwalo kuqoqo. Sibonile ukuthi cishe wonke amarekhodi e-WAL ayenosayizi ofanayo: 2200–2400 bytes. Ngakho-ke, emyalweni osekuqaleni kokuthunyelwe, sicacise ipharamitha -bs=2300 (bs isho usayizi wamabhayithi wokungena ngakunye kwe-fio). Qaphela ukuthi usayizi wokufakwayo njlld uncike enguqulweni njlld, ukusatshalaliswa, amanani epharamitha, njll., futhi kuthinta ubude besikhathi be-fdatasync. Uma unesimo esifanayo, hlola izinqubo zakho njll nge-strace ukuze uthole izinombolo eziqondile.

Bese, ukuze uthole umbono omuhle walokho okwenziwa uhlelo lwefayela njll, siluqale nge-strace kanye nezinketho ze -ffttT. Ngakho sizame ukuhlola izinqubo zengane futhi sirekhode okukhiphayo ngayinye yazo efayeleni elihlukile, futhi siphinde sithole imibiko enemininingwane mayelana nokuqala nobude bekholi yesistimu ngayinye. Sisebenzise i-lsof ukuze siqinisekise ukuhlaziya kwethu kokuphumayo kwe-strace futhi sibone ukuthi iyiphi isichazi sefayela ebesisetshenziselwa yiphi inhloso. Ngakho ngosizo lwe-strace, imiphumela eboniswe ngenhla yatholwa. Izibalo zesikhathi sokuvumelanisa ziqinisekisile ukuthi i-wal_fsync_duration_seconds esuka ku-etcd ihambisana namakholi we-fdatasync anezincazelo zefayela le-WAL.

Sidlule kumadokhumenti e-fio futhi sakhetha izinketho zeskripthi sethu ukuze i-fio ikhiqize umthwalo ofana no-ejjd. Siphinde sahlola amakholi wesistimu nobude bawo ngokusebenzisa i-fio kusuka ku-strace, efana ne- etcd.

Sikhethe ngokucophelela inani lepharamitha --size ukumela wonke umthwalo we-I/O ovela ku-fio. Esimweni sethu, leli inani eliphelele lamabhayithi abhalelwe isitoreji. Kuvele ukuthi ilingana ngqo nenombolo yezingcingo zohlelo (kanye ne-fdatasync). Ngenani elithile lika-bs, inombolo yezingcingo ze-fdatasync = usayizi/bs. Njengoba besinentshisekelo ku-percentile, bekufanele sibe namasampuli anele ukuze siqiniseke, futhi sabala ukuthi u-10^4 uzosanela (lokho kungama-mebibytes angama-22). Uma --usayizi mncane, abangaphandle bangase bavele (isibonelo, izingcingo ezimbalwa ze-fdatasync zithatha isikhathi eside kunokuvamile futhi zithinte i-99th percentile).

Zizame wena

Sikubonise ukuthi ungasebenzisa kanjani i-fio futhi ubone ukuthi isitoreji siyashesha ngokwanele yini ukuthi i- etcd isebenze kahle. Manje usungazama ngokwakho usebenzisa, isibonelo, imishini ebonakalayo enesitoreji se-SSD ngaphakathi I-IBM Cloud.

Source: www.habr.com