Qaphela. transl.: Lesi sihloko siwumphumela wocwaningo oluncane olwenziwa onjiniyela be-IBM Cloud befuna isixazululo senkinga yangempela ehlobene nokusebenza kwesizindalwazi njll. Umsebenzi ofanayo ububalulekile kithi, nokho, inkambo yokucabangisisa kanye nezenzo zababhali zingase zithakazelise esimweni esibanzi.
Isifinyezo esifushane sayo yonke i-athikili: fio kanye nokunye
Ukusebenza kweqoqo etcd kuncike kakhulu esivinini sesitoreji esingaphansi. njlld ithumela kwamanye amazwe amamethrikhi e-Prometheus ukuze igade ukusebenza. Omunye wabo wal_fsync_duration_seconds
. Embhalweni we- etcd
Uma ucabanga ukusetha iqoqo njlld emishinini ye-Linux futhi ufuna ukuhlola ukuthi ingabe amadrayivu (afana nama-SSD) ayashesha ngokwanele yini, sincoma ukusebenzisa isihloli se-I/O esidumile esibizwa ngokuthi. test-data
kufanele ibe sengxenyeni egingqiwe yedrayivu ehloliwe):
fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest
Kuhlala kuphela ukubheka okukhiphayo nokuhlola ukuthi i-percentile engu-99 iyalingana yini fdatasync
fsync/fdatasync/sync_file_range:
sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
sync percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
| 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
| 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
| 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
| 99.99th=[15795]
Amanothi ambalwa:
- Esibonelweni esingenhla, silungise amapharamitha
--size
ΠΈ--bs
ngecala elithile. Ukuze uthole umphumela ophusile kufio
, cacisa amanani afanele isimo sakho sokusebenzisa. Indlela yokukhetha kuzo kuzoxoxwa ngayo ngezansi. - Ngesikhathi sokuhlolwa kuphela
fio
ilayisha isistimu engaphansi yediski. Empilweni yangempela, kungenzeka ukuthi ezinye izinqubo zizobhalela kudiski (ngaphandle kwalezo ezihlobene newal_fsync_duration_seconds
). Lo mthwalo owengeziwe ungakhulawal_fsync_duration_seconds
. Ngamanye amazwi, uma i-percentile engu-99 kusukela ekuhloleni ngefio
, ngaphansi kancane kuka-10 ms kuphela, kunethuba elihle lokuthi ukusebenza kwesitoreji akwanele. - Ukuze uthole ukuhlolwa uzodinga inguqulo
fio
hhayi ngaphansi kuka-3.5, ngoba izinguqulo ezindala azihlanganisi imiphumelafdatasync
ngendlela yamaphesenti. - Isiphetho esingenhla siyingxenye encane yesiphetho esijwayelekile
fio
.
Okuningi mayelana ne-fio nokunye
Amagama ambalwa mayelana nama-WAL njll
Ngokuvamile, i-database isetshenziswa
Ngakho, isikhathi ngasinye iklayenti lengeza ukhiye esitolo se-KV noma libuyekeza inani lokhiye okhona, njlld lengeza incazelo yokusebenza ku-WAL, okuyifayela elivamile esitolo esiqhubekayo. etcd KUMELE uqiniseke ngo-100% ukuthi okufakiwe kwe-WAL kulondoloziwe ngempela ngaphambi kokuqhubeka. Ukufeza lokhu ku-Linux, akwanele ukusebenzisa ikholi yesistimu write
fdatasync
- yilokhu kanye i- etcd ekwenzayo (njengoba ungabona kokuphumayo okulandelayo strace
8
- Isichazi sefayela le-WAL):
21:23:09.894875 lseek(8, 0, SEEK_CUR) = 12808 <0.000012>
21:23:09.894911 write(8, ".20210220361223255266632$1020103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8) = 0 <0.008314>
Ngeshwa, ukubhalela isitoreji esiqhubekayo kuthatha isikhathi. Ukwenza isikhathi eside kwamakholi we-fdatasync kungase kuthinte ukusebenza kwe- etcd. Embhalweni wenqolobane fdatasync
ngenkathi ubhalela ifayela le-WAL bekungaphansi kuka-10 ms. Akhona amanye amamethrikhi ahlobene nesitoreji, kodwa lesi sihloko sizogxila kulokho.
Ukulinganisa isitoreji nge-fio
Ungahlola ukuthi ingabe isitoreji esithile sifanelekile yini ukusetshenziswa ne-etcd usebenzisa insiza fio
kunzima kakhulu ukusebenzisa. Insiza inamapharamitha amaningi, futhi inhlanganisela ehlukene yamanani awo iholela emiphumeleni ehluke ngokuphelele. Ukuze uthole isilinganiso esifanele se-etcd, udinga ukwenza isiqiniseko sokuthi umthwalo wokubhala okhiqizwe i-fio useduze kakhulu nomthwalo wokubhala wefayela le-WAL njll:
- Lokhu kusho ukuthi okukhiqizwayo
fio
umthwalo kufanele okungenani ube uchungechunge lokubhala okulandelanayo efayeleni, lapho ukubhala ngakunye kuqukethe ucingo lwesistimu. ulandelwa nguwrite
fdatasync
. - Ukuze unike amandla ukubhala okulandelanayo, kufanele ucacise ifulegi
--rw=write
. - ukuthi
fio
wabhala esebenzisa izingcingowrite
(kunokuba ezinye izingcingo zesistimu - isibonelo, ), sebenzisa ifulegepwrite
--ioengine=sync
. - Ekugcineni, ifulege
--fdatasync=1
iqinisekisa ukuthi wonkewrite
iyaqhubekafdatasync
. - Amanye amapharamitha amabili esibonelweni sethu yilezi:
--size
ΠΈ--bs
- zingahluka kuye ngesimo esithile sokusetshenziswa. Isigaba esilandelayo sizochaza ukucushwa kwazo.
Kungani sikhethe i-fio nokuthi sifunde kanjani ukuyimisa
Leli nothi livela endabeni yangempela esihlangabezane nayo. Sibe neqoqo ku-Kubernetes v1.13 ngokuqapha ku-Prometheus. Ama-SSD asetshenziswe njengendawo yokugcina izinto njlld v3.2.24. I-Etcd metrics ibonise ukubambezeleka okuphezulu kakhulu fdatasync
, ngisho nalapho iqoqo lalingenzi lutho. Kithina, lawa mamethrikhi abonakala engabaza kakhulu, futhi besingaqiniseki ukuthi amele ini ngempela. Ngaphezu kwalokho, iqoqo lalihlanganisa imishini ebonakalayo, ngakho-ke kwakungenakwenzeka ukusho ukuthi ukubambezeleka kwakungenxa ye-virtualization noma i-SSD yayinecala.
Ngaphezu kwalokho, sicabangele izinguquko ezihlukahlukene ku-hardware ne-software ukucushwa, ngakho sidinga indlela yokuzihlola. Vele, kungenzeka ukusebenzisa njlld ekucushweni ngakunye futhi ubheke amamethrikhi e-Prometheus ahambisanayo, kodwa lokho kungadinga umzamo omkhulu. Esasikudinga kwakuyindlela elula yokuhlola ukucushwa okuthile. Besifuna ukuhlola ukuqonda kwethu amamethrikhi e-Prometheus avela njlld.
Lokhu kwakudinga ukuxazulula izinkinga ezimbili:
- Okokuqala, ingabe umthwalo we-I/O okhiqizwe njlld lapho ubhalela amafayela e-WAL ubukeka kanjani? Yiziphi izingcingo zesistimu ezisetshenziswayo? Bungakanani usayizi wamabhulokhi werekhodi?
- Okwesibili, ake sithi sinezimpendulo zemibuzo engenhla. Ungakhiqiza kanjani kabusha umthwalo ohambelana ne
fio
? Ngemva kwakho konkefio
- Usizo oluguquguquka kakhulu olunenqwaba yamapharamitha (lokhu kulula ukukuqinisekisa, isibonelo,lapha - cishe. transl.).
Sixazulule zombili izinkinga ngendlela efanayo esekelwe kumyalo lsof
strace
- Ngosizo luka
lsof
ungabuka zonke izichazi zefayela ezisetshenziswe inqubo, kanye namafayela ababhekisela kuwo. - Ngosizo luka
strace
ungakwazi ukuhlaziya inqubo esebenzayo kakade noma wenze inqubo futhi uyibuke. Umyalo ubonisa zonke izingcingo zesistimu ezenziwe yile nqubo futhi, uma kunesidingo, inzalo yayo. Lesi sakamuva sibalulekile ezinqubweni eziyimfoloko, futhi njlld ingenye yenqubo enjalo.
Into yokuqala esiyenzile kwakuwukusebenzisa strace
ukuhlola iseva etcd kuqoqo le-Kubernetes ngenkathi lingenzi lutho.
Ngakho-ke kwatholakala ukuthi amabhulokhi amarekhodi e-WAL ahlanganiswe kakhulu, usayizi weningi wawukububanzi obungamabhayithi angu-2200-2400. Kungakho umyalo osekuqaleni kwalesi sihloko usebenzisa ifulege --bs=2300
(bs
ungusayizi wamabhayithi webhulokhi ngayinye yokubhala fio
).
Sicela uqaphele ukuthi usayizi wamabhulokhi wokubhala njll ungahluka kuye ngenguqulo, ukusetshenziswa, amanani epharamitha, njll. - kuthinta ubude besikhathi fdatasync
. Uma unecala lokusebenzisa elifanayo, hlaziya nge strace
izinqubo zakho njll ukuze uthole amanani akamuva.
Bese, ukuze sithole umbono ocacile nophelele wokuthi njlld isebenza kanjani nesistimu yefayela, siyiqale ngaphansi strace
namafulegi -ffttT
. Lokhu kwenze kwaba nokwenzeka ukuthwebula izinqubo zengane nokubhala okukhiphayo ngakunye efayeleni elihlukile. Ngaphezu kwalokho, kutholwe ulwazi oluningiliziwe mayelana nesikhathi sokuqala kanye nobude bekholi yesistimu ngayinye.
Siphinde sasebenzisa umyalo lsof
ukuze uqinisekise ukuqonda kwakho okuphumayo strace
ngokwendlela isichazi sefayela esasetshenziselwa yiphi inhloso. Ngithole isiphetho strace
, efana nale engenhla. Ukukhohlisa kwezibalo ngezikhathi zokuvumelanisa kuqinisekisile ukuthi imethrikhi wal_fsync_duration_seconds
kusuka kumakholi we-etcd fdatasync
ngezichazi zefayela le-WAL.
Ukukhiqiza nge fio
umthwalo wokusebenza ofana nalowo ovela njlld, imibhalo yensiza yafundwa futhi nemingcele elungele umsebenzi wethu yakhethwa. Siqinisekise ukuthi amakholi esistimu alungile ayaqhubeka futhi saqinisekisa ubude bawo ngokusebenza fio
kusuka ku strace
(njengoba kwenziwa uma kwenzeka njlld).
Ukunaka ngokukhethekile kukhokhwe ekunqumeni inani lepharamitha --size
. Imele isamba somthwalo we-I/O okhiqizwe insiza ye-fio. Esimweni sethu, leli inani eliphelele lamabhayithi abhalelwe abezindaba. Ilingana ngqo nenani lezingcingo write
(futhi fdatasync
). Ukuze uthole okuthile bs
inombolo yezingcingo fdatasync
ngokulinganayo size / bs
.
Njengoba besinentshisekelo kuphesenti, besifuna ukuthi inani lamasampuli libe likhulu ngokwanele ukuthi libaluleke ngokwezibalo. Futhi wanquma lokho 10^4
(okuhambisana nosayizi ongu-22 MB) kuzokwanela. Amanani epharamitha amancane --size
inikeze umsindo ogqame kakhulu (ngokwesibonelo, izingcingo fdatasync
, okuthatha isikhathi eside kunokuvamile futhi kuthinte i-99th percentile).
Kuphuma kuwe
Isihloko sibonisa indlela yokusebenzisa fio
umuntu angakwazi ukwahlulela ukuthi imidiya ehloselwe ukusetshenziswa ne-etcd ishesha ngokwanele yini. Manje kukuwe! Ungakwazi ukuhlola imishini ebonakalayo enesitoreji esisekelwe ku-SSD kusevisi
I-PS evela kumhumushi
Ngamacala okusetshenziswa esenziwe ngomumo fio
Ngeminye imisebenzi, bheka
I-PPS evela kumhumushi
Funda futhi kubhulogi yethu:
- Β«
njlld 3.4.3: ukuthembeka kwesitoreji kanye nesifundo sokuphepha "; - Β«
Umuzwa wethu ngedatha ku- etcd Kubernetes cluster ngokuqondile (ngaphandle kwe-K8s API) "; - Β«
Izimbungulu zesistimu yokuzijabulisa eyi-6 ekusebenzeni kwe-Kubernetes [nesixazululo sazo] ".
Source: www.habr.com