Isantya sogcino silungele etcd? Makhe sibuze fio

Ibali elifutshane malunga nefio kunye etcd

Ukusebenza kweqela njl ubukhulu becala kuxhomekeke ekusebenzeni kogcino lwayo. etcd ithumela ngaphandle ezinye iimetrikhi ukuya Prometheusukubonelela ngeenkcukacha zokusebenza kogcino olufunayo. Umzekelo, i-metric wal_fsync_duration_seconds. Amaxwebhu etcd athi: Ukuze ugcino luthathelwe ingqalelo ngokukhawuleza ngokwaneleyo, ipesenti ye-99 yale metric kufuneka ibe ngaphantsi kwe-10 ms. Ukuba uceba ukuqhuba iqela le etcd koomatshini beLinux kwaye ufuna ukuvavanya ukuba ugcino lwakho (olufana ne-SSD) lukhawuleza ngokwaneleyo, ungasebenzisa. fio sisixhobo esidumileyo sokuvavanya imisebenzi ye-I/O. Sebenzisa lo myalelo ulandelayo, apho uvavanyo-data luluhlu oluphantsi kwendawo yokugcina indawo:

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Kufuneka nje ujonge iziphumo kwaye ujonge ukuba ipesenti ezingama-99 zexesha fdatasync ngaphantsi kwe-10 ms. Ukuba ewe, unokugcinwa okukhawulezayo okwaneleyo. Nanku umzekelo weziphumo:

  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Amanqaku

• Senze ngokwezifiso amaxabiso e--ubungakanani kunye ne-bs parameters kwimeko yethu ethile. Ukufumana iziphumo eziluncedo kwifio, ngenisa amaxabiso akho. Ndingazifumana phi? Funda, indlela esifunde ngayo ukuqwalasela i-fio.
• Ngexesha lokuvavanya, yonke imithwalo ye-I / O ivela kwi-fio. Kwimeko yokwenyani, kuya kubakho ezinye izicelo ezibhaliweyo eziza kwindawo yokugcina kunezo zinxulunyaniswa ne-wal_fsync_duration_seconds. Umthwalo owongezelelweyo uya kwandisa ixabiso le-wal_fsync_duration_seconds. Ke ukuba i-99th percentile iphantse ibe yi-10ms, ugcino lwakho alukhawulezi ngokwaneleyo.
• Thatha inguqulelo fio hayi ngaphantsi kwama-3.5 (ezangaphambili azibonisi fdatasync ubude beepesenti).
• Entla sisiqwengana nje seziphumo ezivela kwifio.

Ibali elide malunga nefio kunye etcd

Yintoni i-WAL kwi etcd

Ngokuqhelekileyo kusetyenziswa oovimba beenkcukacha bhala-phambili log; etcd iyayisebenzisa. Asiyi kuxoxa ngelogi yokubhala ngaphambili (WAL) ngokweenkcukacha apha. Kwanele ukuba sazi ukuba ilungu ngalinye le-etd cluster liyigcina kwindawo yokugcina eqhubekayo. njl. Ukuba elinye lamalungu okugcina liyawa kwaye liqalise kwakhona phakathi kwe-snapshots, liyakwazi ukubuyisela utshintshiselwano ukusuka kwi-snapshot yokugqibela usebenzisa imixholo ye-WAL.

Xa umxhasi esongeza isitshixo kwi-key-value store okanye ehlaziya ixabiso lesitshixo esele sikhona, etcd ubhala irekhodi yalo msebenzi kwi-WAL, eyifayile eqhelekileyo kwisitoreji esiqhubekayo. njl. Kwi-Linux, enye inkqubo yokufowuna ayanele kule nto bhala, ekubeni ngokwenene ukubhala kwindawo yokugcina ebonakalayo kunokulibaziseka. Umzekelo, i-Linux inokugcina okwexeshana ungeno lwe-WAL kwindawo efihlakeleyo kwimemori yekernel (enje ngecache yephepha). Kwaye ukuze idatha ibhalwe ngokuchanekileyo kwisitoreji esiqhubekayo, ifowuni yenkqubo ye-fdatasync iyadingeka emva kokurekhoda, kwaye etcd isebenzise nje (njengoko kunokubonwa ngenxa yomsebenzi. umtya, apho u-8 yi-WAL ichaza ifayile):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Ngelishwa, ukubhalela ugcino oluzingisileyo akukhawulezi. Ukuba umnxeba we-fdatasync uyacotha, ukusebenza kwenkqubo ye- etcd kuya kuthotywa. Amaxwebhu etcd athiukuba ugcino luthathwa ngokukhawuleza ngokwaneleyo ukuba ipesenti ye-99 yeefowuni ze-fdatasync zithatha ngaphantsi kwe-10 ms ukubhala kwifayile ye-WAL. Kukho ezinye iimetriki eziluncedo zokugcina, kodwa le kuphela kwemetric esithetha ngayo kwesi sithuba.

Uvavanyo logcino usebenzisa fio

Ukuba ufuna ukuvavanya ukuba ugcino lwakho lufanelekile etcd, sebenzisa i-fio, isixhobo esithandwa kakhulu sokuvavanya umthwalo we-I/O. Kufuneka kukhunjulwe ukuba imisebenzi yediski inokwahluka kakhulu: i-synchronous kunye ne-asynchronous, iiklasi ezininzi zeefowuni zenkqubo, njl. Ngenxa yoko, i-fio inzima kakhulu ukuyisebenzisa. Ineeparamitha ezininzi, kwaye indibaniselwano eyahlukileyo yamaxabiso azo ivelisa imithwalo yemisebenzi ye-I/O eyahlukileyo kakhulu. Ukufumana amanani aneleyo e-etcd, kufuneka uqinisekise ukuba uvavanyo lokubhala umthwalo ovela kwi-fio usondele kakhulu kumthwalo wangempela ovela kwi-etcd xa ubhala iifayile ze-WAL.

Ngoko ke, i-fio kufuneka, ubuncinci, ivelise umthwalo wenani lolandelelwano ibhala kwifayile, ukubhala ngakunye okubandakanya umnxeba wenkqubo. bhalailandelwa yinkqubo yefdatasync umnxeba. Ukubhalwa kwe-fio ngokulandelelanayo, i--rw=bhala ukhetho luyafuneka. Ukuze fio isebenzisa inkqubo yokubhala umnxeba xa ubhala, kunokuba bhala, kufanelekile ukukhankanya ipharamitha -ioengine=sync. Okokugqibela, ukufowunela i-fdatasync emva kokubhala ngakunye, kufuneka udibanise i--fdatasync=1 ukhetho. Ezinye iinketho ezimbini kulo mzekelo (--ubungakanani kunye no -bs) ziyimeko ethile. Kwicandelo elilandelayo siza kukubonisa indlela yokuseta.

Kutheni i-fio kunye nendlela esifunde ukuyiqwalasela ngayo

Kule post sichaza imeko yokwenyani. Sasineqela Kubernetes I-v1.13, esiyibeke iliso sisebenzisa i-Prometheus. etcd v3.2.24 yabanjwa kwi-SSD. I-Etcd metrics ibonise ukubambezeleka okuphezulu kakhulu kwe-fdatasync, naxa iqela lingenzi nto. Iimetrics zazingaqhelekanga kwaye sasingazi ukuba zithetha ukuthini. Iqela laliquka oomatshini ababonakalayo, kwakuyimfuneko ukuqonda ukuba yintoni ingxaki: kwii-SSD ezibonakalayo okanye kuluhlu lwe-virtualization. Ukongeza, besihlala sisenza utshintsho kwihardware kunye nolungelelwaniso lwesoftware, kwaye sifuna indlela yokuvavanya iziphumo zabo. Singaqhuba etcd kuqwalaselo ngalunye kwaye sijonge iPrometheus metrics, kodwa oku kuxakeke kakhulu. Besijonge indlela elula yokuvavanya ubumbeko oluthile. Besifuna ukujonga ukuba siyaziqonda na iimetrics zePrometheus ngokuchanekileyo.

Kodwa ngenxa yoko kwakuyimfuneko ukusombulula iingxaki ezimbini. Okokuqala, yintoni umthwalo we-I/O odalwa yi- etcd xa ubhalela i-WAL? Zeziphi iifowuni zesistim ezisetyenziswayo? Zingakanani izithuba? Okwesibini, ukuba siphendula le mibuzo, sinokuphinda sivelise umsebenzi ofanayo kunye ne-fio? Ungalibali ukuba i-fio sisixhobo esibhetyebhetye kakhulu esinokhetho oluninzi. Sizisombulule zombini iingxaki ngendlela enye - sisebenzisa imiyalelo lsof и umtya. I-lsof ibonisa zonke izichazi zefayile ezisetyenziswe yinkqubo kunye neefayile ezihambelana nazo. Kwaye ngomtya unokufunda inkqubo esele iqhuba okanye uqalise inkqubo kwaye uyifunde. strace iprinta zonke iifowuni zenkqubo ukusuka kwinkqubo efundwayo (kunye neenkqubo zayo zomntwana). Le yokugqibela ibaluleke kakhulu, kuba etcd ithatha indlela efanayo.

Into yokuqala esiyenzileyo kukusebenzisa umtya ukufunda iseva etcd yeKubernetes xa kwakungekho mthwalo kwiqela. Sibonile ukuba phantse zonke iirekhodi ze-WAL zilingana nobukhulu obufanayo: 2200-2400 bytes. Ngoko ke, kumyalelo ekuqaleni kweposi, sichaze iparameter -bs=2300 (bs ithetha ubungakanani bebhayithi kwifio nganye yokungena). Qaphela ukuba ubungakanani bengeno etcd ixhomekeke kuguqulelo etcd, ngenqanawa, amaxabiso parameter, njalo njalo kwaye kuchaphazela ubude bexesha fdatasync. Ukuba unemeko efanayo, jonga iinkqubo zakho ze-etd usebenzisa umtya ukuze ufumane amanani achanekileyo.

Ke, ukufumana umbono olungileyo wokuba yintoni inkqubo yefayile ye- etcd, siyibalekisile ngomtya kunye neenketho ze -ffttT. Ngoko sizame ukufunda iinkqubo zomntwana kwaye sirekhode umphumo ngamnye wabo kwifayile eyahlukileyo, kwaye ufumane iingxelo ezicacileyo malunga nokuqala kunye nobude befowuni nganye yenkqubo. Sisebenzise i-lsof ukuqinisekisa uhlalutyo lwethu lwemveliso yomtya kwaye sibone ukuba yeyiphi inkcazo yefayile esetyenziselwe ziphi iinjongo. Ke, sisebenzisa umtya, sifumene iziphumo eziboniswe ngasentla. Izibalo zexesha longqamaniso ziqinisekisile ukuba i-wal_fsync_duration_seconds metric ukusuka etcd ihambelana neefowuni ze-fdatasync ezineenkcazo zefayile ye-WAL.

Sijonge amaxwebhu e-fio kwaye sakhetha iiparamitha zeskripthi sethu ukuze i-fio ivelise umthwalo ofana ne-etd. Siphinde sajonga iifowuni zesistim kunye nobude bazo ngokuqhuba i-fio ukusuka kumtya, ngokufana nokunye.

Sikhethe ngononophelo ixabiso le---size parameter, emele wonke umthwalo we-I/O we-fio. Kwimeko yethu, eli linani elipheleleyo leebhayithi ezibhaliweyo kwindawo yokugcina. Kuye kwabonakala kuhambelana ngqo nenani lokubhala (kunye ne-fdatasync) iifowuni zenkqubo. Ngexabiso elithile le-bs, inani leefowuni kwi-fdatasync = ubukhulu/bs. Ekubeni sasinomdla kwi-percentile, kufuneka sibe neesampuli ezaneleyo ukuze sithembeke, kwaye sibale ukuba i-10 ^ 4 iya kukwanela kuthi (yi-22 mebibytes). Ukuba --ubungakanani buncinci, ii-outliers zingenzeka (umzekelo, iifowuni ezininzi ze-fdatasync zithatha ixesha elide kunesiqhelo kwaye zichaphazela ipesenti ezingama-99).

Yizame ngokwakho

Sibonise indlela yokusebenzisa i-fio kwaye sifumanise ukuba ugcino lukhawuleza ngokwaneleyo ukuba etcd luqhube kakuhle. Ngoku unokuzama oku ngokuziqhelanisa ngokwakho, usebenzisa, umzekelo, oomatshini ababonakalayo abanokugcinwa kwe-SSD kwi IBM Cloud.

umthombo: www.habr.com