🥇Kumhanyisa kwekuchengetedza kwakakodzera here etcd? Ngatibvunzei fio

Nyaya pfupi nezve fio uye etcd

Cluster performance nezvimwewo zvakanyanya zvinoenderana nekuita kwekuchengetedza kwayo. etcd inotumira kunze mamwe metrics kune Prometheuskupa ruzivo rwaunoda kuchengetedza kuita. Semuenzaniso, iyo wal_fsync_duration_seconds metric. The documentation for etcd inoti: Kuti nzvimbo yekuchengetera zvinhu ionekwe nekukurumidza, chikamu che99th percentile chechiyero ichi chinofanira kunge chiri pasi pe10 ms. Kana uchironga kushandisa cluster ye etcd pamichina. Linux uye uchida kuongorora kana nzvimbo yako yekuchengetera (senge SSD) inokurumidza zvakakwana, unogona kushandisa fio chishandiso chakakurumbira chekuyedza mashandiro eI/O. Mhanya unotevera kuraira, uko test-data ndiyo dhairekitori pasi penzvimbo yekuchengetedza nzvimbo:

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Iwe unongoda kutarisa mhedzisiro uye tarisa kuti iyo 99th percentile yenguva fdatasync isingasviki 10 ms. Kana zvakadaro, une kukurumidza kukurumidza kuchengetedza. Heino muenzaniso wezvabuda:

  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Notes

Isu takagadzirisa iyo --saizi uye --bs sarudzo kune yedu chaiyo mamiriro. Kuti uwane mhedzisiro inobatsira kubva kufio, ipa yako tsika. Ndozviwanepi? Verenga kuti takadzidza sei kugadzirisa fio.
Panguva yekuyedzwa, yese I / O mutoro unobva ku fio. Mune mamiriro ehupenyu chaihwo, panogona kunge paine zvimwe zvikumbiro zvekunyora zvinouya mudura kunze kweizvo zvine chekuita newal_fsync_duration_seconds. Iyo yakawedzera mutoro ichawedzera kukosha kwe wal_fsync_duration_seconds. Saka kana iyo 99th percentile iri pedyo ne10ms, chengetedzo yako iri kupera nekumhanya.
Tora shanduro fio kwete pasi pe3.5 (iyo yapfuura hairatidze fdatasync duration percentiles).
Pamusoro ingori chidimbu chezvabuda kubva kufio.

Nyaya yakareba nezve fio uye etcd

Chii chinonzi WAL mune etcd

Kazhinji databases inoshandiswa kunyora-mberi log; etcd anoishandisa futi. Hatisi kuzokurukura nyora-mberi log (WAL) zvakadzama pano. Zvakakwana kuti isu tizive kuti nhengo yega yega yeetcd cluster inoichengeta mukuchengetedza kunoramba kuripo. etcd inonyora yega kiyi-kukosha mashandiro (senge yekuvandudza) kuWAL usati waisa kuchitoro. Kana imwe yenhengo dzekuchengetera ikaparara uye yotangazve pakati pezvimedu, inogona munharaunda kudzoreredza kutengeserana kubva pakupedzisira snapshot neWAL zvemukati.

Kana mutengi akawedzera kiyi kuchitoro chekiyi-value kana kugadzirisa kukosha kwekiyi iripo, etcd inorekodha mashandiro aya muWAL, inova faira renguva dzose riri mukuchengetedza nguva dzose. Usati waenderera mberi nekugadzirisa, etcd UNOFANIRA kuva nechokwadi chekuti WAL write yakaitika zvechokwadi. Linux kufona kamwe chete kwesystem hakuna kukwana pane izvi nyora, sezvo kunyora chaiko kuchinyorwa chekuchengetedza kunogona kunonoka. Semuenzaniso, Linux inogona kuchengeta rekodhi yeWAL mu kernel memory cache (semuenzaniso, peji cache) kwenguva yakati. Kuti ive nechokwadi chekuti data rakanyorwa kuti rirambe riripo, fdatasync system call inodiwa mushure mekunyora, uye etcd inoshandisa izvi (sezvinogona kuonekwa mumhedzisiro). tambo, apo 8 ndiyo WAL faira descriptor):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Sezvineiwo, kunyorera kunoramba kuchichengetera hakuitiki ipapo ipapo. Kana iyo fdatasync kufona ichinonoka, kuita kweiyo etcd system kuchatambura. The documentation for etcd inotikuti chengetedzo inoonekwa seyakakurumidza kana, mu99th percentile, fdatasync call inotora isingasviki 10ms kunyora kuWAL faira. Kune mamwe ma metric anobatsira ekuchengetedza, asi mune ino positi tiri kungotaura nezve iyi metric.

Kufungidzira kuchengetedza nefio

Kana iwe uchida kuongorora kana chengetedzo yako yakakodzera etcd, shandisa fio, inozivikanwa kwazvo I/O mutoro wekuyedza chishandiso. Izvo zvinofanirwa kuyeukwa kuti dhisiki mashandiro anogona kunge akasiyana zvakanyanya: synchronous uye asynchronous, akawanda makirasi ekufona system, etc. Somugumisiro, fio yakaoma chaizvo kushandisa. Iyo ine akawanda ma paramita, uye akasiyana masanganiswa ehukoshero hwawo anoburitsa akasiyana zvakanyanya I/O mabasa. Kuti uwane nhamba dzakakwana dze etcd, unofanirwa kuve nechokwadi chekuti bvunzo yekunyora mutoro kubva kufio iri padyo sezvinobvira kumutoro chaiwo kubva etcd paunenge uchinyora WAL mafaera.

Naizvozvo, fio inofanirwa, padiki, kugadzira mutoro muchimiro cheakateedzana anoteedzana anonyora kufaira, kunyora kwega kwega kuchange kuine kufona system. nyorainoteverwa nefdatasync system call. Sequential inonyora kune fio inoda iyo --rw=write sarudzo. Kuti fio ishandise iyo yekunyora system kufona paunenge uchinyora, kwete nyora, unofanira kutsanangura iyo --ioengine=sync parameter. Chekupedzisira, kuti ufonere fdatasync mushure mekunyora kwega kwega, unofanirwa kuwedzera iyo --fdatasync = 1 paramende. Dzimwe sarudzo mbiri mumuenzaniso uyu (--saizi uye -bs) ndeye script-chaiyo. Muchikamu chinotevera, tinokuratidza maitiro ekuamisa.

Nei chaizvo fio uye kuti takadzidza sei kumisikidza

Mune ino post, tinotsanangura chiitiko chaicho. Tine cluster Kubernetes v1.13 iyo yatakaongorora nePrometheus. etcd v3.2.24 yakagarwa pane SSD. Etcd metrics yakaratidza fdatasync latencies yakakwira zvakanyanya, kunyangwe iyo cluster yanga isingaite chinhu. Mametrics acho aishamisa uye isu takanga tisingazive zvazvaireva. Iro sumbu raive nemashini chaiwo, zvaive zvakafanira kuti unzwisise kuti dambudziko raive rei: mune zvemuviri SSDs kana mune virtualization layer. Pamusoro pezvo, isu taiwanzoita shanduko kune Hardware uye software kumisikidzwa, uye isu taida nzira yekuongorora mhedzisiro yavo. Isu taigona kumhanya etcd mukugadziriswa kwese uye kutarisa Prometheus metrics, asi izvo zvakawandisa zvekunetsa. Isu taitsvaga nzira iri nyore yekuongorora imwe gadziriso. Taida kutarisa kana tichinzwisisa Prometheus metrics kubva etcd nenzira kwayo.

Asi nokuda kweizvi, matambudziko maviri aifanira kugadziriswa. Kutanga, iyo I / O mutoro unogadzirwa etcd kana uchinyorera kuWAL unotaridzika sei? Ndeapi masystem mafoni anoshandiswa? Marekodhi akakura sei? Chechipiri, kana tikapindura mibvunzo iyi, tinoburitsa sei basa rakafanana nefio? Usakanganwa kuti fio chishandiso chinochinjika chine zvakawanda zvingasarudzwa. Isu takagadzirisa matambudziko ese nenzira imwe - tichishandisa mirairo lsof и tambo. lsof inonyora ese mafaera anotsanangura anoshandiswa nemaitiro uye anoenderana mafaera. Uye netambo, unogona kuongorora yatove kuita maitiro, kana kutanga maitiro uye woiongorora. strace inodhinda ese masystem ekufona kubva pakuita kuri kuongororwa (uye maitiro emwana). Iyo yekupedzisira yakakosha, sezvo etcd iri kungotora maitiro akafanana.

Takatanga kushandisa strace kuongorora iyo etcd server yeKubernetes pakange pasina mutoro pasumbu. Takaona kuti anenge ese maWAL marekodhi aive akaenzana saizi: 2200–2400 bytes. Naizvozvo, mukuraira pakutanga kweiyo positi, isu takatsanangura parameter -bs=2300 (bs zvinoreva saizi mumabhayiti ega ega ekupinda fio). Ziva kuti saizi yekupinda etcd inotsamira pane etcd vhezheni, kugovera, parameter kukosha, nezvimwewo, uye inokanganisa nguva yefdatasync. Kana iwe uine mamiriro akafanana, ongorora yako etcd maitiro netambo kuti uwane iwo chaiwo manhamba.

Zvadaro, kuti uwane pfungwa yakanaka yezviri kuitwa neetd file system, takaitanga ne strace uye -ffttT sarudzo. Saka takaedza kuongorora maitiro emwana uye kurekodha kuburitsa kweumwe neumwe wavo mune imwe faira rakasiyana, uye zvakare kuwana yakadzama mishumo nezvekutanga uye nguva yeimwe system yekufona. Isu takashandisa lsof kusimbisa ongororo yedu yezvakabuda uye kuona kuti ndeipi descriptor yefaira yaishandiswa nechinangwa chipi. Saka nerubatsiro rwetambo, zvigumisiro zvakaratidzwa pamusoro zvakawanikwa. Nhamba dzenguva yekuwiriranisa dzakasimbisa kuti wal_fsync_duration_seconds kubva etcd inopindirana nefdatasync mafoni ane WAL faira descriptors.

Isu takapfuura nemagwaro efio uye takasarudza sarudzo dzechinyorwa chedu kuti fio ibudise mutoro wakafanana ne etcd. Isu takatarisawo masystem ekufona uye nguva yavo nekumhanya fio kubva strace, yakafanana neetcd.

Isu takanyatsosarudza kukosha kwe --saizi paramende kumiririra iyo yese I / O mutoro kubva fio. Muchiitiko chedu, iyi ndiyo nhamba yose yebhaiti yakanyorwa kune yekuchengetedza. Zvakazove zvakaenzana nenhamba yekunyora (uye fdatasync) system mafoni. Kune imwe kukosha kwe bs, nhamba yefdatasync inofona = saizi/bs. Sezvo taifarira percentile, taifanira kuva nemasampuli akakwana kuti tive nechokwadi, uye takaverenga kuti 10 ^ 4 yaizotikwanira (ndiyo 22 mebibytes). Kana --saizi idiki, zvinobuda kunze zvinogona kuitika (semuenzaniso, akati wandei fdatasync mafoni anotora nguva yakareba kupfuura zvakajairwa uye anokanganisa 99th percentile).

Edza iwe pachako

Isu takakuratidza mashandisiro efio uye kuona kana chengetedzo ichikurumidza zvakakwana kuti etcd iite zvakanaka. Iye zvino unogona kuzviedza iwe pachako uchishandisa, semuenzaniso, chaiwo michina ine SSD yekuchengetedza mukati IBM Cloud.

Source: www.habr.com