🥇Maitiro ekutarisa madhisiki ane fio kuita kwakakwana kwe etcd

Cherechedza. transl.: Chinyorwa ichi mhedzisiro yemini-tsvakurudzo yakaitwa neBMM Cloud mainjiniya mukutsvaga mhinduro kudambudziko chairo rine chekuita nekushanda kweiyo etcd database. Basa rakafanana raive rinokosha kwatiri, zvisinei, nzira yekufungisisa nezviito zvevanyori zvingave zvinonakidza mumamiriro ezvinhu akawanda.

Pfupiso pfupi yechinyorwa chose: fio uye etcd

Kuita kweiyo etcd cluster kunoenderana zvakanyanya nekumhanya kweiyo pasi pekuchengetedza. etcd inotumira kunze akasiyana Prometheus metrics yekutarisa mashandiro. Mumwe wavo ndiye wal_fsync_duration_seconds. Mune zvinyorwa zve etcd inotiiyo chengetedzo inogona kutariswa nekukurumidza zvakakwana kana iyo 99th percentile yeiyi metric isingapfuure gumi ms…

Kana uri kufunga kumisikidza etcd cluster paLinux michina uye uchida kutarisa kana madhiraivha (akadai seSSD) ari kukurumidza zvakakwana, tinokurudzira kushandisa yakakurumbira I/O tester inonzi. fio. Zvakakwana kumhanyisa unotevera kuraira (dhairekitori test-data inofanira kunge iri muchikamu chakaiswa cheiyo tested drive):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Izvo zvinongosara kutarisa zvakabuda uye tarisa kana iyo 99th percentile inokodzera fdatasync mu10 ms. Kana zvakadaro, saka drive yako iri kushanda nekukurumidza zvakakwana. Heino muenzaniso wekubuda:

fsync/fdatasync/sync_file_range:
  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Zvinyorwa zvishoma:

Mumuenzaniso wepamusoro, takagadzirisa maparameter --size и --bs kune imwe nyaya. Kuti uwane mhedzisiro ine musoro kubva fio, tsanangura kukosha kwakakodzera kune yako kesi yekushandisa. Nzira yekuvasarudza ichakurukurwa pasi apa.
Panguva yekuedzwa chete fio inotakura disk subsystem. Muhupenyu chaihwo, zvingangoita kuti mamwe maitiro achanyorera ku diski (kunze kweaya ane hukama ne wal_fsync_duration_seconds) Uyu mutoro wekuwedzera unogona kuwedzera wal_fsync_duration_seconds. Mune mamwe mazwi, kana iyo 99th percentile kubva pakuyedzwa ne fio, zvishoma zvishoma pane 10 ms, pane mukana wakanaka wekuti kuchengetedza kushanda hakuna kukwana.
Pakuedza iwe uchada shanduro fio kwete pasi pe3.5, nekuti shanduro dzekare hadziunganidzi mibairo fdatasync muchimiro chepercentiles.
Mhedziso iri pamusoro ingori chidimbu chidiki kubva kumhedziso huru fio.

Zvakawanda nezve fio uye etcd

Mazwi mashoma nezve WALs etcd

Kazhinji, databases inoshandiswa proactive matanda (nyora-kumberi kutema, WAL). etcd inobatwawo. Hurukuro yeWAL inodarika chiyero chechinyorwa ichi, asi nekuda kwezvinangwa zvedu, chaunoda kuziva ndechekuti imwe neimwe etcd cluster nhengo inochengeta WAL mukuchengetedza kunoramba kuripo. etcd inonyora mamwe makiyi-value ekuchengetedza mashandiro (senge zvigadziriso) kuWAL usati waaita. Kana node ikaputsika uye yotangazve pakati pezvipfukuto, etcd inozokwanisa kudzoreredza kutengeserana kubva pamufananidzo wapfuura zvichibva pane zviri muWAL.

Nokudaro, nguva imwe neimwe mutengi anowedzera kiyi kuchitoro cheKV kana kugadzirisa kukosha kwekiyi iripo, etcd inowedzera tsananguro yekushanda kuWAL, iyo inogara iri faira muchitoro chinoramba chiripo. etcd INOFANIRA kuve 100% chokwadi chekuti iyo WAL yekupinda yakachengetwa isati yaenderera. Kuti uite izvi paLinux, hazvina kukwana kushandisa system call write, sezvo basa rekunyora pacharo kune vezvenyama rinogona kunonoka. Semuyenzaniso, Linux inogona kuchengeta WAL yekupinda mune in-memory kernel cache (semuenzaniso, mune peji cache) kwenguva yakati. Kuve nechokwadi chekuti data rakanyorwa kune midhiya, kufona system kunofanirwa kukumbirwa mushure mekunyora fdatasync - izvi ndizvo chaizvo zvinoita etcd (sezvauri kuona mune inotevera kuburitsa strace; Here 8 - WAL faira descriptor):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ".20210220361223255266632$1020103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Nehurombo, kunyorera kune inoenderera kuchengetedza kunotora nguva. Kuitwa kwenguva refu kwefdatasync kufona kunogona kukanganisa kuita kwe etcd. Mune repository zvinyorwa yakaratidza, kuti kuita kwakakwana kunodiwa kuti 99th percentile yenguva yemafoni ese fdatasync pakunyorera WAL faira yaive isingasviki gumi ms. Kune mamwe ma metric ane chekuita nekuchengetedza, asi chinyorwa ichi chinotarisa pane icho.

Kukoshesa kuchengetedza nefio

Iwe unogona kuongorora kana imwe chengetedzo yakakodzera kushandiswa neetcd uchishandisa utility fio - muongorori weI/O ane mukurumbira. Ramba uchifunga kuti dhisiki I/O inogona kuitika nenzira dzakawanda dzakasiyana: sync/async, akawanda akasiyana syscall makirasi, zvichingodaro. Rimwe divi remari ndiro iroro fio yakaoma zvakanyanya kushandisa. Iyo yekushandisa ine akawanda ma paramita, uye akasiyana masanganiswa ehukoshi hwawo anotungamira kune akasiyana mhedzisiro. Kuti uwane fungidziro inonzwisisika ye etcd, iwe unofanirwa kuve nechokwadi chekuti kunyora mutoro unogadzirwa nefio uri padyo sezvinobvira kune etcd's WAL faira kunyora mutoro:

Izvi zvinoreva kuti yakagadzirwa fio mutoro wacho unofanirwa kunge uri mutsara weanoteedzana anonyora kune faira, apo imwe neimwe inonyora ine system yekufona. writeichiteverwa ne fdatasync.
Kugonesa kunyora kwakatevedzana, unofanirwa kutsanangura mureza --rw=write.
kuti fio akanyora achishandisa mafoni write (panzvimbo pedzimwe nharembozha - semuenzaniso, pwrite), shandisa mureza --ioengine=sync.
Pakupedzisira, mureza --fdatasync=1 inovimbisa kuti zvese write ndinofanira fdatasync.
Mamwe maparameter maviri mumuenzaniso wedu ndeaya: --size и --bs - inogona kusiyana zvichienderana neyakashandiswa kesi. Chikamu chinotevera chichatsanangura magadzirirwo avo.

Nei takasarudza fio uye kuti takadzidza sei kumisikidza

Chinyorwa ichi chinobva kunyaya chaiyo yatakasangana nayo. Isu takanga tine cluster paKubernetes v1.13 nekutarisa paPrometheus. SSD dzakashandiswa sekuchengetedza etcd v3.2.24. Etcd metrics yakaratidza kunonoka kwakanyanya fdatasync, kunyange apo sumbu racho rakanga risina basa. Kwatiri, ma metrics aya airatidzika kunge asina chokwadi, uye takanga tisina chokwadi kuti aimiririrei chaizvo. Pamusoro pezvo, sumbu racho raive nemakina chaiwo, saka zvakange zvisingaite kutaura kuti kunonoka kwacho kwakakonzerwa nekuona kana kuti SSD yaive mhosva.

Pamusoro pezvo, takafunga zvakasiyana-siyana zvehardware uye magadzirirwo esoftware, saka taida nzira yekuzviongorora. Ehe, zvinogoneka kumhanya etcd mune imwe neimwe gadziriso uye kutarisa inoenderana Prometheus metrics, asi izvo zvingada kushanda nesimba. Chataida yaive nzira iri nyore yekuongorora imwe gadziriso. Isu taida kuyedza kunzwisisa kwedu kwePrometheus metrics inouya kubva etcd.

Izvi zvaida kugadzirisa matambudziko maviri:

Kutanga, iyo I / O mutoro unogadzirwa ne etcd kana uchinyorera kune WAL mafaera anotaridzika sei? Ndeapi masystem mafoni anoshandiswa? Mabhuroko erekodhi akakura sei?
Chechipiri, ngatitii tine mhinduro kumibvunzo iri pamusoro. Nzira yekugadzirazve mutoro unoenderana ne fio? Shure kwezvose fio -Kunyanya kuchinjika utility ine kuwanda kwema parameter (izvi zviri nyore kuratidza, semuenzaniso, pano - approx. transl.).

Isu takagadzirisa matambudziko ese nenzira imwechete-yakavakirwa nzira lsof и strace:

Nekubatsirwa kwe lsof unogona kuona ese mafaera anotsanangura anoshandiswa nemaitiro, pamwe nemafaira avanoreva.
Nekubatsirwa kwe strace unogona kuongorora yatove kuita maitiro kana kumhanyisa maitiro uye woiona. Iwo murairo unoratidza ese masystem mafoni akaitwa neichi maitiro uye, kana zvichidikanwa, vazukuru vayo. Iyo yekupedzisira yakakosha kune maitiro ari foroking, uye etcd ndeimwe nzira yakadaro.

Chinhu chekutanga chatakaita kwaive kushandisa strace kuongorora iyo etcd server muKubernetes cluster ichiri isina basa.

Saka zvakaonekwa kuti WAL rekodhi mabhuroki akanyatso kuunganidzwa, saizi yeruzhinji yaive mumhando ye2200-2400 bytes. Ndosaka murairo uri kwekutanga kwechinyorwa chino uchishandisa mureza --bs=2300 (bs ukuru mumabhaiti ebhuroko rimwe nerimwe rekunyora mukati fio).

Ndokumbira utarise kuti saizi ye etcd yekunyora mabhuroko inogona kusiyana zvichienderana neshanduro, deployment, parameter values, nezvimwe. - inokanganisa nguva fdatasync. Kana iwe uine imwe kesi yekushandisa yakafanana, ongorora nayo strace yako etcd maitiro ekuwana-ku-ku-date tsika.

Zvino, kuti tiwane yakajeka uye yakazara pfungwa yekuti etcd inoshanda sei nefaira system, takaitanga kubva pasi. strace nemireza -ffttT. Izvi zvakaita kuti zvikwanise kutora maitiro evana uye kunyora zvakabuda zvega kune imwe faira rakasiyana. Pamusoro pezvo, ruzivo rwakadzama nezve nguva yekutanga uye nguva yekufona yega yega system yakawanikwa.

Takashandisawo murairo lsofkusimbisa kunzwisisa kwako kwezvakabuda strace maererano neiyo faira descriptor yakashandiswa nechinangwa chipi. Ndakawana mhedziso strace, yakafanana neiri pamusoro. Statistical manipulations nenguva dzekuenzanisa dzakasimbisa kuti metric wal_fsync_duration_seconds kubva etcd machisi mafoni fdatasync ine WAL faira descriptors.

Kugadzira ne fio basa rakafanana neiro kubva etcd, zvinyorwa zvekushandisa zvakadzidzwa uye maparameter akakodzera basa redu akasarudzwa. Isu takaona kuti iyo chaiyo system yekufona iri kuenderera uye yakasimbisa nguva yavo nekumhanya fio kubva strace (sezvazvaitwa munyaya ye etcd).

Kunyanya kutarisisa kwakabhadharwa pakusarudza kukosha kweparameter --size. Inomiririra iyo yakazara I / O mutoro unogadzirwa nefio utility. Muchiitiko chedu, iyi ndiyo nhamba yose yebhaiti yakanyorwa kune vezvenhau. Iyo yakanangana nenhamba yekufona write (uye fdatasync) Kune chaiyo bs nhamba dzekufona fdatasync zvakaenzana size / bs.

Sezvo isu taifarira mupercentile, taida kuti nhamba yemasamples ive yakakura zvekuti ive yakakosha. Uye akasarudza izvozvo 10^4 (iyo inoenderana nehukuru hwe22 MB) ichakwana. Diki parameter values --size yakapa ruzha rwakanyanya (semuenzaniso, mafoni fdatasync, izvo zvinotora nguva yakareba kupfuura nguva dzose uye zvinokanganisa 99th percentile).

Zviri kwauri

Chinyorwa chinoratidza nzira yekushandisa fio munhu anogona kutonga kana nhepfenyuro yakarongedzerwa kushandiswa neetcd inokurumidza zvakakwana. Zvino zvava kwauri! Iwe unogona kuongorora chaiwo michina ine SSD-yakavakirwa kuchengetedza musevhisi IBM Cloud.

PS kubva kumushanduri

Nemakesi ekushandisa akagadzirira fio Kuti uwane mamwe mabasa, ona zvinyorwa kana zvakananga ku project repositories (kune akawanda akawanda kupfuura ataurwa muzvinyorwa).

PPS kubva kumushanduri

Verenga zvakare pablog yedu:

Source: www.habr.com

Maitiro ekutarisa madhisiki ane fio ekuita kwakakwana kwe etcd