Nyaya pfupi nezve fio uye etcd
Cluster performance zvakanyanya zvinoenderana nekuita kwekuchengetedza kwayo. etcd inotumira kunze mamwe metrics kune kupa ruzivo rwaunoda kuchengetedza kuita. Semuenzaniso, iyo wal_fsync_duration_seconds metric. : Kuti kuchengetedza kuonekwe nekukurumidza zvakakwana, iyo 99th percentile yeiyi metric inofanira kunge iri pasi pegumi. Kana uri kuronga kumhanyisa etcd cluster paLinux michina uye uchida kuongorora kana chengetedzo yako ichikurumidza zvakakwana (semuenzaniso SSD), unogona kushandisa chishandiso chakakurumbira chekuyedza mashandiro eI/O. Mhanya unotevera kuraira, uko test-data ndiyo dhairekitori pasi penzvimbo yekuchengetedza nzvimbo:
fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytestIwe unongoda kutarisa mhedzisiro uye tarisa kuti iyo 99th percentile yenguva isingasviki 10 ms. Kana zvakadaro, une kukurumidza kukurumidza kuchengetedza. Heino muenzaniso wezvabuda:
sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
sync percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
| 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
| 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
| 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
| 99.99th=[15795]Notes
- Isu takagadzirisa iyo --saizi uye --bs sarudzo kune yedu chaiyo mamiriro. Kuti uwane mhedzisiro inobatsira kubva kufio, ipa yako tsika. Ndozviwanepi? Verenga .
- Panguva yekuyedzwa, yese I / O mutoro unobva ku fio. Mune mamiriro ehupenyu chaihwo, panogona kunge paine zvimwe zvikumbiro zvekunyora zvinouya mudura kunze kweizvo zvine chekuita newal_fsync_duration_seconds. Iyo yakawedzera mutoro ichawedzera kukosha kwe wal_fsync_duration_seconds. Saka kana iyo 99th percentile iri pedyo ne10ms, chengetedzo yako iri kupera nekumhanya.
- Tora shanduro kwete pasi pe3.5 (iyo yapfuura hairatidze fdatasync duration percentiles).
- Pamusoro ingori chidimbu chezvabuda kubva kufio.
Nyaya yakareba nezve fio uye etcd
Chii chinonzi WAL mune etcd
Kazhinji databases inoshandiswa ; etcd anoishandisa futi. Hatisi kuzokurukura nyora-mberi log (WAL) zvakadzama pano. Zvakakwana kuti isu tizive kuti nhengo yega yega yeetcd cluster inoichengeta mukuchengetedza kunoramba kuripo. etcd inonyora yega kiyi-kukosha mashandiro (senge yekuvandudza) kuWAL usati waisa kuchitoro. Kana imwe yenhengo dzekuchengetera ikaparara uye yotangazve pakati pezvimedu, inogona munharaunda kudzoreredza kutengeserana kubva pakupedzisira snapshot neWAL zvemukati.
Kana mutengi akawedzera kiyi kune kiyi-kukosha chitoro kana kugadzirisa kukosha kwekiyi iripo, etcd inorekodha oparesheni muWAL, inova faira yenguva dzose mukuchengetedza kunoramba kuripo. etcd INOFANIRA kuve nechokwadi chakazara kuti iyo WAL yekupinda yakaitika isati yaenderera mberi nekugadzirisa. PaLinux, imwe system yekufona haina kukwana pane izvi. , sezvo kunyorera chaiko kukuchengetedza kwemuviri kunganonoka. Semuenzaniso, Linux inogona kuchengetedza WAL yekupinda mune cache mukernel memory (senge peji cache) kwenguva yakati. Uye kuitira kuti data inyorwe nemazvo kunoramba kuchengetwa, iyo fdatasync system call inodiwa mushure mekunyora, uye etcd inongoishandisa (sezvaunogona kuona mumhedzisiro yebasa. , apo 8 ndiyo WAL faira descriptor):
21:23:09.894875 lseek(8, 0, SEEK_CUR) = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8) = 0 <0.008314>Sezvineiwo, kunyorera kunoramba kuchichengetera hakuitiki ipapo ipapo. Kana iyo fdatasync kufona ichinonoka, kuita kweiyo etcd system kuchatambura. kuti chengetedzo inoonekwa seyakakurumidza kana, mu99th percentile, fdatasync call inotora isingasviki 10ms kunyora kuWAL faira. Kune mamwe ma metric anobatsira ekuchengetedza, asi mune ino positi tiri kungotaura nezve iyi metric.
Kufungidzira kuchengetedza nefio
Kana iwe uchida kuongorora kana chengetedzo yako yakakodzera etcd, shandisa fio, inozivikanwa kwazvo I/O mutoro wekuyedza chishandiso. Izvo zvinofanirwa kuyeukwa kuti dhisiki mashandiro anogona kunge akasiyana zvakanyanya: synchronous uye asynchronous, akawanda makirasi ekufona system, etc. Somugumisiro, fio yakaoma chaizvo kushandisa. Iyo ine akawanda ma paramita, uye akasiyana masanganiswa ehukoshero hwawo anoburitsa akasiyana zvakanyanya I/O mabasa. Kuti uwane nhamba dzakakwana dze etcd, unofanirwa kuve nechokwadi chekuti bvunzo yekunyora mutoro kubva kufio iri padyo sezvinobvira kumutoro chaiwo kubva etcd paunenge uchinyora WAL mafaera.
Naizvozvo, fio inofanirwa, padiki, kugadzira mutoro muchimiro cheakateedzana anoteedzana anonyora kufaira, kunyora kwega kwega kuchange kuine kufona system. inoteverwa nefdatasync system call. Sequential inonyora kune fio inoda iyo --rw=write sarudzo. Kuti fio ishandise iyo yekunyora system kufona paunenge uchinyora, kwete , unofanira kutsanangura iyo --ioengine=sync parameter. Chekupedzisira, kuti ufonere fdatasync mushure mekunyora kwega kwega, unofanirwa kuwedzera iyo --fdatasync = 1 paramende. Dzimwe sarudzo mbiri mumuenzaniso uyu (--saizi uye -bs) ndeye script-chaiyo. Muchikamu chinotevera, tinokuratidza maitiro ekuamisa.
Nei chaizvo fio uye kuti takadzidza sei kumisikidza
Mune ino post, tinotsanangura chiitiko chaicho. Tine cluster v1.13 iyo yatakaongorora nePrometheus. etcd v3.2.24 yakagarwa pane SSD. Etcd metrics yakaratidza fdatasync latencies yakakwira zvakanyanya, kunyangwe iyo cluster yanga isingaite chinhu. Mametrics acho aishamisa uye isu takanga tisingazive zvazvaireva. Iro sumbu raive nemashini chaiwo, zvaive zvakafanira kuti unzwisise kuti dambudziko raive rei: mune zvemuviri SSDs kana mune virtualization layer. Pamusoro pezvo, isu taiwanzoita shanduko kune Hardware uye software kumisikidzwa, uye isu taida nzira yekuongorora mhedzisiro yavo. Isu taigona kumhanya etcd mukugadziriswa kwese uye kutarisa Prometheus metrics, asi izvo zvakawandisa zvekunetsa. Isu taitsvaga nzira iri nyore yekuongorora imwe gadziriso. Taida kutarisa kana tichinzwisisa Prometheus metrics kubva etcd nenzira kwayo.
Asi nokuda kweizvi, matambudziko maviri aifanira kugadziriswa. Kutanga, iyo I / O mutoro unogadzirwa etcd kana uchinyorera kuWAL unotaridzika sei? Ndeapi masystem mafoni anoshandiswa? Marekodhi akakura sei? Chechipiri, kana tikapindura mibvunzo iyi, tinoburitsa sei basa rakafanana nefio? Usakanganwa kuti fio chishandiso chinochinjika chine zvakawanda zvingasarudzwa. Isu takagadzirisa matambudziko ese nenzira imwe - tichishandisa mirairo ΠΈ . lsof inonyora ese mafaera anotsanangura anoshandiswa nemaitiro uye anoenderana mafaera. Uye netambo, unogona kuongorora yatove kuita maitiro, kana kutanga maitiro uye woiongorora. strace inodhinda ese masystem ekufona kubva pakuita kuri kuongororwa (uye maitiro emwana). Iyo yekupedzisira yakakosha, sezvo etcd iri kungotora maitiro akafanana.
Takatanga kushandisa strace kuongorora iyo etcd server yeKubernetes pakange pasina mutoro pasumbu. Takaona kuti anenge ese maWAL marekodhi aive akaenzana saizi: 2200β2400 bytes. Naizvozvo, mukuraira pakutanga kweiyo positi, isu takatsanangura parameter -bs=2300 (bs zvinoreva saizi mumabhayiti ega ega ekupinda fio). Ziva kuti saizi yekupinda etcd inotsamira pane etcd vhezheni, kugovera, parameter kukosha, nezvimwewo, uye inokanganisa nguva yefdatasync. Kana iwe uine mamiriro akafanana, ongorora yako etcd maitiro netambo kuti uwane iwo chaiwo manhamba.
Zvadaro, kuti uwane pfungwa yakanaka yezviri kuitwa neetd file system, takaitanga ne strace uye -ffttT sarudzo. Saka takaedza kuongorora maitiro emwana uye kurekodha kuburitsa kweumwe neumwe wavo mune imwe faira rakasiyana, uye zvakare kuwana yakadzama mishumo nezvekutanga uye nguva yeimwe system yekufona. Isu takashandisa lsof kusimbisa ongororo yedu yezvakabuda uye kuona kuti ndeipi descriptor yefaira yaishandiswa nechinangwa chipi. Saka nerubatsiro rwetambo, zvigumisiro zvakaratidzwa pamusoro zvakawanikwa. Nhamba dzenguva yekuwiriranisa dzakasimbisa kuti wal_fsync_duration_seconds kubva etcd inopindirana nefdatasync mafoni ane WAL faira descriptors.
Isu takapfuura nemagwaro efio uye takasarudza sarudzo dzechinyorwa chedu kuti fio ibudise mutoro wakafanana ne etcd. Isu takatarisawo masystem ekufona uye nguva yavo nekumhanya fio kubva strace, yakafanana neetcd.
Isu takanyatsosarudza kukosha kwe --saizi paramende kumiririra iyo yese I / O mutoro kubva fio. Muchiitiko chedu, iyi ndiyo nhamba yose yebhaiti yakanyorwa kune yekuchengetedza. Zvakazove zvakaenzana nenhamba yekunyora (uye fdatasync) system mafoni. Kune imwe kukosha kwe bs, nhamba yefdatasync inofona = saizi/bs. Sezvo taifarira percentile, taifanira kuva nemasampuli akakwana kuti tive nechokwadi, uye takaverenga kuti 10 ^ 4 yaizotikwanira (ndiyo 22 mebibytes). Kana --saizi idiki, zvinobuda kunze zvinogona kuitika (semuenzaniso, akati wandei fdatasync mafoni anotora nguva yakareba kupfuura zvakajairwa uye anokanganisa 99th percentile).
Edza iwe pachako
Isu takakuratidza mashandisiro efio uye kuona kana chengetedzo ichikurumidza zvakakwana kuti etcd iite zvakanaka. Iye zvino unogona kuzviedza iwe pachako uchishandisa, semuenzaniso, chaiwo michina ine SSD yekuchengetedza mukati .
Source: www.habr.com