🥇Uzijonga njani iidiski ezinefio ukuba zisebenza ngokwaneleyo etcd

Phawula. transl.: Eli nqaku liyisiphumo sophando oluncinci olwenziwa ziinjineli ze-IBM Cloud ekufuneni isisombululo kwingxaki yangempela enxulumene nokusebenza kwe-database etcd. Umsebenzi ofanayo wawufanelekile kuthi, nangona kunjalo, ikhosi yokucinga kunye nezenzo zababhali zinokuba nomdla kumxholo obanzi.

Isishwankathelo esifutshane salo lonke inqaku: fio kunye etcd

Ukusebenza kweqela etcd kuxhomekeke kakhulu kwisantya sogcino olungaphantsi. etcd ithumela ngaphandle ezahlukeneyo Prometheus metrics ukujonga ukusebenza. Omnye wabo ngu wal_fsync_duration_seconds. Kumaxwebhu e- etcd itshoukuba ugcino lunokuqwalaselwa ngokukhawuleza ngokwaneleyo ukuba ipesenti ezingama-99 zale metric ayidluli kwi-10 ms...

Ukuba ucinga ngokuseta iqela le etcd koomatshini beLinux kwaye ufuna ukuvavanya ukuba ingaba iidrives (ezifana nee-SSD) zikhawuleza ngokwaneleyo, sicebisa ukuba usebenzise umvavanyi we-I/O odumileyo obizwa ngokuba fio. Kwanele ukwenza lo myalelo ulandelayo (uluhlu test-data kufuneka ibekwe kwindawo exhonyiweyo ye-drive evavanyiweyo):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Kuhlala kuphela ukujonga imveliso kwaye ukhangele ukuba i-99th percentile iyalingana fdatasync kwi10ms. Ukuba kunjalo, ke drive yakho isebenza ngokukhawuleza ngokwaneleyo. Nanku umzekelo wemveliso:

fsync/fdatasync/sync_file_range:
  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Amanqaku ambalwa:

Kulo mzekelo ungasentla, siye sahlengahlengisa iiparamitha --size и --bs kwityala elithile. Ukufumana isiphumo esinentsingiselo kwi fio, khankanya amaxabiso afanelekileyo kwimeko yakho yosetyenziso. Indlela yokuyikhetha iya kuxoxwa ngezantsi.
Ngexesha lovavanyo kuphela fio ilayisha inkqubo esezantsi yediski. Kubomi bokwenyani, kunokwenzeka ukuba ezinye iinkqubo ziya kubhala kwidiski (ngaphandle kwezo zinxulumene ne wal_fsync_duration_seconds). Lo mthwalo owongezelelweyo unokunyuka wal_fsync_duration_seconds. Ngamanye amazwi, ukuba ipesenti ezingama-99 ukusuka kuvavanyo nge fio, kuphela ngaphantsi kwe-10 ms, kukho ithuba elihle lokuba ukusebenza kokugcinwa akwanele.
Kuvavanyo uya kufuna inguqulelo fio hayi ngaphantsi kwama-3.5, kuba iinguqulelo ezindala azihlanganisi iziphumo fdatasync ngendlela yeepesenti.
Isiphetho esingentla sisicatshulwa nje esincinci kwisigqibo esiqhelekileyo fio.

Okunye malunga nefio kunye etcd

Amagama ambalwa malunga nee-WALs etcd

Ngokubanzi, kusetyenziswa oovimba beenkcukacha ukugawulwa ngokuqhubekayo (bhala ukugawulwa kwangaphambili, WAL). etcd nayo iyachaphazeleka. Ingxoxo ye-WAL ingaphaya kobubanzi beli nqaku, kodwa ngeenjongo zethu, into okufuneka uyazi kukuba ilungu ngalinye leqela le- etcd ligcina i-WAL kwindawo yokugcina eqhubekayo. njl. Ukuba i node iyantlitheka kwaye iphinda iqale phakathi kwesnapshots, etcd iyakukwazi ukubuyisela intengiselwano ukusukela kwisnapshot yangaphambili esekwe kwimixholo ye WAL.

Ngaloo ndlela, ixesha ngalinye umxhasi wongeza isitshixo kwivenkile ye-KV okanye ehlaziya ixabiso leqhosha elikhoyo, njl njl yongeza inkcazo yokusebenza kwi-WAL, eyifayile eqhelekileyo kwivenkile eqhubekayo. njl. Ukufezekisa oku kwiLinux, akwanelanga ukusebenzisa umnxeba wenkqubo write, ekubeni umsebenzi wokubhala ngokwawo kwimidiya ebonakalayo unokulibaziseka. Umzekelo, i-Linux inokugcina ungeno lwe-WAL kwi-cache ye-kernel ekwimemori (umzekelo, kwi-cache yephepha) ixesha elithile. Ukuqinisekisa ukuba idatha ibhaliwe kumajelo eendaba, kufuneka kufakwe ifowuni emva kokubhalwa kwayo fdatasync - yile nto kanye eyenziwa yi etcd (njengoko unokubona kwimveliso elandelayo strace; Apha 8 -Inkcazelo yefayile yeWAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ".20210220361223255266632$1020103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Ngelishwa, ukubhalela ukugcinwa okuzingisileyo kuthatha ixesha. Ukwenziwa kwexesha elide kweefowuni ze-fdatasync kunokuchaphazela ukusebenza kwe- etcd. Kumaxwebhu ogcino kubonisiwe, ukuba ukusebenza okwaneleyo kuyimfuneko ukuba ipesenti ye-99 yobude bazo zonke iifowuni fdatasync xa ubhalela ifayile ye WAL ibingaphantsi kwe 10 ms. Kukho ezinye iimetriki ezinxulumene nokugcinwa, kodwa eli nqaku liza kugxila kulo.

Ukuxabisa ugcino ngefio

Unokuvavanya ukuba ugcino oluthile lukulungele na ukusetyenziswa nge-tdd usebenzisa into eluncedo fio - umhloli we-I / O owaziwayo. Gcina ukhumbule ukuba i-disk I / O inokwenzeka ngeendlela ezininzi ezahlukeneyo: ukuvumelanisa / ukuvumelanisa, iiklasi ezininzi ze-syscall, njalo njalo. Elinye icala lengqekembe lelo fio kunzima kakhulu ukusebenzisa. Usetyenziso luneeparamitha ezininzi, kwaye indibaniselwano eyahlukileyo yamaxabiso abo ikhokelela kwiziphumo ezahluke ngokupheleleyo. Ukuze ufumane uqikelelo olufanelekileyo lwe- etcd, kufuneka uqinisekise ukuba umthwalo wokubhala owenziwe yi-fio ukufutshane kangangoko kumthwalo wokubhala wefayile ye-WAL ye etcd:

Oku kuthetha ukuba yenziwe fio umthwalo kufuneka noko ube luluhlu olulandelelanayo lokubhala kwifayile, apho ubhalo ngalunye luqulathe inkqubo yokufowuna. writeilandelwe ngu fdatasync.
Ukwenza ukubhala ngokulandelelanayo, kufuneka ucacise iflegi --rw=write.
ukuba fio wabhala esebenzisa iminxeba write (kunokuba ezinye iifowuni zenkqubo - umzekelo, pwrite), sebenzisa iflegi --ioengine=sync.
Ekugqibeleni, iflegi --fdatasync=1 iqinisekisa ukuba yonke into write kufanele fdatasync.
Ezinye iiparameters ezimbini kumzekelo wethu zezi: --size и --bs - inokwahluka ngokuxhomekeke kwimeko yosetyenziso oluthile. Icandelo elilandelayo liza kuchaza uqwalaselo lwabo.

Kutheni sikhethe i-fio kunye nendlela esifunde ngayo ukuseta

Eli nqaku livela kwimeko yokwenyani esidibene nayo. Sasineqela kwi-Kubernetes v1.13 ngokubeka iliso kwi-Prometheus. Ii-SSD zisetyenziswe njengendawo yokugcina i-etcd v3.2.24. I-Etcd metrics ibonise ukubambezeleka okuphezulu kakhulu fdatasync, naxa iqela lalingasebenzi. Kuthi, ezi metrics zazibonakala zithandabuzeka kakhulu, kwaye sasingaqinisekanga ukuba zimele ntoni kanye kanye. Ukongeza, iqela liqulethwe ngoomatshini benyani, ngoko ke kwakungenakwenzeka ukuthetha ukuba ukulibaziseka kwakungenxa ye-virtualization okanye i-SSD yayinetyala.

Ukongeza, siye saqwalasela utshintsho olwahlukeneyo kwi-hardware kunye noqwalaselo lwesoftware, ngoko ke sifuna indlela yokuyivavanya. Ewe kunjalo, kuyakwenzeka ukuqhuba etcd kuqwalaselo ngalunye kwaye ujonge iPrometheus metrics ehambelanayo, kodwa oko kuya kufuna umgudu omkhulu. Into ebesiyidinga yayiyindlela elula yokuvavanya uqwalaselo oluthile. Besifuna ukuvavanya ukuqonda kwethu i-Prometheus metrics evela kwi-etcd.

Oku kufuna ukusonjululwa kweengxaki ezimbini:

Okokuqala, yintoni i-I/O yomthwalo oveliswe yi- etcd xa ubhalela kwiifayile ze-WAL zibukeka njani? Zeziphi iifowuni zesistim ezisetyenziswayo? Yintoni ubungakanani beebhloko zerekhodi?
Okwesibini, masithi sineempendulo kule mibuzo ingasentla. Indlela yokuvelisa kwakhona umthwalo ohambelanayo kunye fio? Emveni kwakho konke fio - into eluncedo ebhetyebhetye ngokugqithisileyo enobuninzi beeparamitha (oku kulula ukukuqinisekisa, umzekelo, apha - malunga. guqulela.).

Sizisombulule zombini iingxaki ngendlela efanayo esekwe kumyalelo lsof и strace:

Ngo kunceda lsof ungajonga zonke iinkcazo zefayile ezisetyenziswe yinkqubo, kunye neefayile ezibhekisa kuzo.
Ngo kunceda strace ungahlalutya inkqubo esele iqhutywa okanye uqhube inkqubo kwaye uyibukele. Umyalelo ubonisa zonke iifowuni zenkqubo ezenziwe yile nkqubo kwaye, ukuba kuyimfuneko, inzala yayo. Le yokugqibela ibalulekile kwiinkqubo ezenza ifolokhwe, kwaye etcd yenye inkqubo enjalo.

Into yokuqala esiyenzileyo kukusebenzisa strace ukujonga i etcd umncedisi kwiqela le Kubernetes ngelixa ibingasebenzi.

Ngoko kwafunyaniswa ukuba iibhloko zerekhodi ze-WAL zidityaniswe kakhulu, ubukhulu bobuninzi babukuluhlu lwe-2200-2400 bytes. Yiyo loo nto umyalelo osekuqaleni kweli nqaku usebenzisa iflegi --bs=2300 (bs bubungakanani bee-bytes zebhloko nganye yokubhala fio).

Nceda uqaphele ukuba ubungakanani beebhloko zokubhala etcd zinokwahluka ngokuxhomekeke kuguqulelo, ukusasazwa, amaxabiso eparameter, njl. - ichaphazela ubude bexesha fdatasync. Ukuba unemeko efanayo yokusetyenziswa, hlalutya kunye strace Iinkqubo zakho ze etcd ukufumana amaxabiso ahlaziyiweyo.

Ke, ukuze sifumane umbono ocacileyo kunye obanzi wendlela etcd esebenza ngayo nenkqubo yefayile, siyiqale phantsi. strace ngeeflegi -ffttT. Oku kwenza ukuba kube nokwenzeka ukubamba iinkqubo zomntwana kunye nokubhala imveliso nganye kwifayile eyahlukileyo. Ukongeza, ulwazi oluthe kratya malunga nexesha lokuqalisa kunye nobude befowuni nganye yesistim yafunyanwa.

Sasebenzisa nomyalelo lsofukuqinisekisa ukuqonda kwakho isiphumo strace ngokwemigaqo yeyiphi inkcazo yefayile esetyenziselwe injongo. Ndifumene isigqibo strace, efana nale ingasentla. Ukuguqulwa kwamanani ngamaxesha ongqamaniso kuqinisekisile ukuba imetriki wal_fsync_duration_seconds ukusuka etcd imidlalo iminxeba fdatasync ngezichazi zefayile ye WAL.

Ukwenza nge fio umthwalo womsebenzi ofanayo nalowo uvela kwi-etcd, amaxwebhu oncedo afundwe kwaye iiparameters ezifanelekileyo kumsebenzi wethu zikhethiwe. Siye saqinisekisa ukuba iifowuni ezichanekileyo ziyaqhubeka kwaye zaqinisekisa ubude bazo ngokusebenza fio из strace (njengoko bekusenziwa kwimeko etcd).

Ingqwalasela ekhethekileyo ihlawulwe ekumiseleni ixabiso lepharamitha --size. Imele umthwalo opheleleyo we-I/O oveliswe yi-fio utility. Kwimeko yethu, eli linani elipheleleyo leebhayithi ezibhaliweyo kumajelo eendaba. Ilingana ngokuthe ngqo nenani leefowuni write (kunye fdatasync). Ngengcaciso ethile bs inani leefowuni fdatasync ngokulinganayo size / bs.

Kuba besinomdla kwipercentile, besifuna ukuba inani leisampulu libe likhulu ngokwaneleyo ukuba libaluleke ngokwezibalo. Kwaye ndagqiba ukuba 10^4 (ehambelana nobukhulu be-22 MB) iya kwanela. Amaxabiso amancinci eparameter --size inike ingxolo engaphezulu (umzekelo, iifowuni fdatasync, ezithatha ixesha elide kunesiqhelo kwaye zichaphazela ipesenti ezingama-99).

Ixhomekeke kuwe

Inqaku libonisa indlela yokusebenzisa fio umntu unokugweba ukuba ngaba imidiya ejoliswe ekusetyenzisweni kunye ne-etcd ikhawuleza ngokwaneleyo. Ngoku kukuwe! Unokuphonononga oomatshini ababonakalayo kunye nokugcinwa kwe-SSD esekwe kwinkonzo IBM Cloud.

PS evela kumguquleli

Ngeemeko zokusetyenziswa esele zenziwe fio Ngeminye imisebenzi, bona amaxwebhu okanye ngqo kwi iindawo zokugcina iiprojekthi (zininzi zazo kunokuba zikhankanywe kumaxwebhu).

PPS evela kumguquleli

Funda nakwibhlog yethu:

umthombo: www.habr.com

Uzijonga njani iidiski ngefio ukwenzela ukusebenza okwaneleyo kweetd