🥇Yadda ake duba faifai tare da fio don isassun aikin da sauransu

Lura. fassara: Wannan labarin ya samo asali ne sakamakon wani karamin bincike da injiniyoyin IBM Cloud suka yi don neman mafita ga wata matsala ta gaske da ke da alaka da aiki da rumbun adana bayanai da sauransu. Irin wannan ɗawainiya ya dace da mu, duk da haka, tsarin tunani da ayyukan marubuta na iya zama mai ban sha'awa a cikin yanayi mai faɗi.

Takaitaccen taƙaitaccen labarin duka: fio da sauransu

Ayyukan gungu na etcd ya dogara sosai akan saurin ma'ajiyar da ke ƙasa. etcd suna fitar da ma'aunin Prometheus daban-daban don saka idanu akan aiki. Daya daga cikinsu shine wal_fsync_duration_seconds. A cikin takardun don etcd yana cewaAna iya ɗaukar ajiyar ajiya cikin sauri idan kashi 99 na wannan ma'aunin bai wuce 10 ms ba…

Idan kuna la'akari da kafa gungu da dai sauransu akan injunan Linux kuma kuna son bincika idan abubuwan tafiyarwa (kamar SSDs) suna da saurin isa, muna ba da shawarar yin amfani da mashahurin gwajin I/O da ake kira. fio. Ya isa ya gudanar da umarni mai zuwa (directory test-data dole ne a kasance a cikin sashin da aka ɗora na injin ɗin da aka gwada):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Ya rage kawai don duba fitarwa kuma bincika idan kashi 99th ya dace fdatasync ku 10 ms. Idan haka ne, to motarku tana aiki da sauri sosai. Ga fitowar misali:

fsync/fdatasync/sync_file_range:
  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Bayanan kula:

A cikin misalin da ke sama, mun daidaita sigogi --size и --bs ga wani lamari na musamman. Don samun sakamako mai ma'ana daga fio, ƙididdige ƙimar da suka dace don yanayin amfani da ku. Yadda za a zabi su za a tattauna a kasa.
Lokacin gwaji kawai fio Loads da faifai subsystem. A cikin rayuwa ta ainihi, yana yiwuwa wasu matakai za su rubuta zuwa faifai (ban da waɗanda ke da alaƙa da wal_fsync_duration_seconds). Wannan ƙarin kaya na iya ƙaruwa wal_fsync_duration_seconds. A wasu kalmomi, idan kashi 99th daga gwaji tare da fio, kawai ɗan ƙasa da 10 ms, akwai kyakkyawar dama cewa aikin ajiya bai isa ba.
Don gwajin za ku buƙaci sigar fio ba kasa da 3.5 ba, saboda tsofaffin sigar baya tara sakamako fdatasync a cikin nau'i na percentiles.
Ƙarshen da ke sama kaɗan ne kawai daga ƙarshen ƙarshe fio.

Ƙarin bayani game da fio da sauransu

Kalmomi kaɗan game da WALs da sauransu

Yawanci, ana amfani da databases shigar da hankali (rubuta gaba-gaba, WAL). etcd kuma abin ya shafa. Tattaunawar WAL ta wuce iyakar wannan labarin, amma don dalilanmu, abin da kuke buƙatar sani shine kowane memba na ƙungiyar da sauransu yana adana WAL a cikin ma'auni na dindindin. etcd yana rubuta wasu ayyukan ma'ajin ƙima (kamar sabuntawa) zuwa WAL kafin aiwatar da su. Idan kumburi ya yi karo kuma ya sake farawa tsakanin hotuna, da sauransu na iya dawo da ma'amaloli tun daga hoton da ya gabata dangane da abubuwan da ke cikin WAL.

Don haka, duk lokacin da abokin ciniki ya ƙara maɓalli zuwa kantin sayar da KV ko sabunta ƙimar maɓalli da ke akwai, da sauransu suna ƙara bayanin aikin zuwa WAL, wanda shine fayil na yau da kullun a cikin kantin sayar da dagewa. etcd DOLE ya tabbata 100% cewa an adana shigarwar WAL kafin a ci gaba. Don cimma wannan akan Linux, bai isa ba don amfani da kiran tsarin write, tunda aikin rubuta kanta zuwa kafofin watsa labarai na zahiri na iya jinkirtawa. Misali, Linux na iya kiyaye shigarwar WAL a cikin ma'ajiyar kernel (misali, a cikin cache shafi) na wani lokaci. Don tabbatar da cewa an rubuta bayanan zuwa ga kafofin watsa labarai, dole ne a kira tsarin kira bayan rubutawa fdatasync - wannan shine ainihin abin da etcd ke yi (kamar yadda kuke gani a cikin fitarwa mai zuwa strace; nan 8 - Mai kwatanta fayil ɗin WAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ".20210220361223255266632$1020103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Abin takaici, rubuta zuwa ma'ajiya mai tsayi yana ɗaukar ɗan lokaci. Tsawaita aiwatar da kiran fdatasync na iya shafar aikin da sauransu. A cikin takardun ajiya nuna, cewa don isashen aiki ya zama dole cewa kashi 99 na tsawon lokacin duk kira fdatasync lokacin rubutawa zuwa fayil ɗin WAL bai wuce 10 ms ba. Akwai wasu ma'auni masu alaƙa da ajiya, amma wannan labarin zai mai da hankali kan wancan.

Ƙimar ajiya tare da fio

Kuna iya kimanta ko takamaiman ma'ajin ya dace don amfani tare da sauransu ta amfani da abin amfani fio - mashahurin gwajin I/O. Ka tuna cewa faifai I / O na iya faruwa ta hanyoyi daban-daban: sync / async, yawancin nau'ikan syscall daban-daban, da sauransu. Daya gefen tsabar kudin shine fio matuƙar wuya a yi amfani da. Mai amfani yana da sigogi da yawa, kuma haɗuwa daban-daban na ƙimar su suna haifar da sakamako daban-daban. Domin samun ma'auni mai ma'ana don etcd, kuna buƙatar tabbatar da cewa nauyin rubutun da fio ya haifar yana kusa da yiwuwar rubuta fayil na WAL na etcd:

Wannan yana nufin cewa generated fio nauyin ya kamata ya zama aƙalla jerin rubuce-rubuce a jere zuwa fayil ɗin, inda kowane rubutu ya ƙunshi tsarin kira. writebiye da fdatasync.
Don kunna rubutun jeri, dole ne ka saka tuta --rw=write.
cewa fio ya rubuta ta amfani da kira write (maimakon sauran tsarin kira - alal misali, pwrite), amfani da tuta --ioengine=sync.
A ƙarshe, tutar --fdatasync=1 tabbatar da cewa duk write dole ne fdatasync.
Sauran sigogi biyu a misalinmu sune: --size и --bs - na iya bambanta dangane da takamaiman yanayin amfani. Sashe na gaba zai bayyana tsarin su.

Dalilin da ya sa muka zaɓi fio da yadda muka koyi yadda ake saita shi

Wannan bayanin ya fito ne daga ainihin lamarin da muka ci karo da shi. Muna da gungu akan Kubernetes v1.13 tare da saka idanu akan Prometheus. An yi amfani da SSDs azaman ajiya don etcd v3.2.24. Da dai sauransu ma'auni sun nuna rashin jinkiri da yawa fdatasync, ko da a lokacin da cluster ba shi da aiki. A gare mu, waɗannan ma'aunin sun yi kama da shakku, kuma ba mu da tabbacin ainihin abin da suke wakilta. Bugu da kari, gungu ya ƙunshi na'urori masu kama-da-wane, don haka ba zai yiwu a ce ko jinkirin ya faru ne saboda haɓakawa ko kuma SSD ne ke da laifi.

Bugu da kari, mun yi la'akari daban-daban canje-canje a cikin hardware da software sanyi, don haka muna bukatar wata hanya don kimanta su. Tabbas, zai yiwu a gudanar da etcd a cikin kowane tsari kuma duba ma'aunin Prometheus masu dacewa, amma hakan zai buƙaci gagarumin ƙoƙari. Abin da muke buƙata shine hanya mai sauƙi don kimanta takamaiman tsari. Muna so mu gwada fahimtarmu game da ma'aunin Prometheus da ke fitowa daga etcd.

Wannan ya buƙaci magance matsalolin guda biyu:

Na farko, menene nauyin I/O da etcd ke samarwa yayin rubutawa zuwa fayilolin WAL yayi kama? Wane tsarin kira ake amfani da shi? Menene girman tubalan rikodin?
Na biyu, a ce muna da amsoshin tambayoyin da ke sama. Yadda ake sake haifar da nauyin da ya dace da shi fio? Bayan haka fio - musamman m mai amfani tare da yalwar sigogi (wannan yana da sauƙin tabbatarwa, misali, a nan - kimanin. fassara).

Mun warware matsalolin biyu tare da tsarin tushen umarni iri ɗaya lsof и strace:

Tare da taimakon lsof Kuna iya duba duk bayanan fayilolin da tsarin ke amfani da su, da fayilolin da suke magana akai.
Tare da taimakon strace za ku iya bincika tsarin da ke gudana ko gudanar da tsari kuma ku duba shi. Umurnin yana nuna duk kiran tsarin da wannan tsari yayi kuma, idan ya cancanta, zuriyarsa. Ƙarshen yana da mahimmanci ga hanyoyin da suke yin cokali mai yatsa, kuma da dai sauransu ɗaya ne irin wannan tsari.

Abu na farko da muka yi shi ne amfani strace don bincika uwar garken etcd a cikin gungun Kubernetes yayin da ba shi da aiki.

Don haka an gano cewa tubalan rikodin WAL suna tattare da yawa sosai, girman yawancin yana cikin kewayon 2200-2400 bytes. Shi ya sa umarnin da ke farkon wannan labarin ke amfani da tuta --bs=2300 (bs shine girman a cikin bytes na kowane tubalan rubutu a ciki fio).

Lura cewa girman tubalan rubuta etcd na iya bambanta dangane da sigar, turawa, ƙimar sigina, da sauransu. - yana rinjayar tsawon lokaci fdatasync. Idan kuna da irin wannan yanayin amfani, bincika tare da strace ayyukan ku etcd don samun sabbin ƙima.

Sa'an nan, domin samun wani bayyananne da kuma m ra'ayi na yadda etcd aiki tare da fayil tsarin, mun fara shi daga karkashin. strace tare da tutoci -ffttT. Wannan ya ba da damar ɗaukar matakan yara da rubuta abin da kowannensu ya fitar zuwa wani fayil daban. Bugu da ƙari, an sami cikakken bayani game da lokacin farawa da tsawon kowane kiran tsarin.

Mun kuma yi amfani da umarnin lsofdon tabbatar da fahimtar ku game da fitarwa strace dangane da abin da aka yi amfani da bayanin fayil don wane dalili. Na samu karshen strace, kama da wanda ke sama. Ma'auni na ƙididdiga tare da lokutan aiki tare sun tabbatar da cewa ma'aunin wal_fsync_duration_seconds daga etcd matches kira fdatasync tare da kwatancen fayilolin WAL.

Don samar da fio nauyin aiki mai kama da na daga etcd, an yi nazarin takardun kayan aiki kuma an zaɓi sigogi masu dacewa da aikinmu. Mun tabbatar da cewa madaidaicin kiran tsarin yana kan ci gaba kuma mun tabbatar da tsawon lokacin su ta hanyar gudu fio daga strace (kamar yadda aka yi a yanayin da sauransu).

An biya kulawa ta musamman don tantance ƙimar siga --size. Yana wakiltar jimlar nauyin I/O wanda mai amfani da fio ya samar. A wajenmu, wannan shine jimillar adadin bytes da aka rubuta wa kafafen yada labarai. Yana daidai da adadin kira kai tsaye write (kuma fdatasync). Don takamaiman bs yawan kira fdatasync daidai size / bs.

Tun da muna da sha'awar kashi, muna son adadin samfurori ya zama babba don ya zama mahimmancin ƙididdiga. Kuma ya yanke shawarar haka 10^4 (wanda yayi daidai da girman 22 MB) zai wadatar. Ƙananan ma'auni --size ya ba da ƙarin ƙarar amo (misali, kira fdatasync, wanda ke ɗaukar lokaci fiye da yadda aka saba kuma yana shafar kashi 99).

Ya rage naku

Labarin ya nuna yadda ake amfani da shi fio mutum zai iya yin hukunci ko kafofin watsa labaru da aka yi niyyar amfani da su da sauransu sun yi saurin isa. Yanzu ya rage naku! Kuna iya bincika injunan kama-da-wane tare da ajiyar tushen SSD a cikin sabis ɗin IBM Cloud.

PS daga mai fassara

Tare da shirye-shiryen amfani fio Don wasu ayyuka, duba takardun ko kai tsaye zuwa wuraren ajiyar aikin (akwai da yawa daga cikinsu fiye da aka ambata a cikin takardun).

PPS daga mai fassara

Karanta kuma a kan shafinmu:

source: www.habr.com

Yadda ake duba faifai tare da fio don isassun aiki don da dai sauransu