Kuthamanga kosungirako koyenera etcd? Tiyeni tifunse fio

Nkhani yachidule ya fio ndi etcd

Kuchita kwa Cluster etcd makamaka zimadalira kagwiridwe kake kakusungirako. etcd imatumiza ma metric ena ku Prometheuskuti mupereke chidziwitso cha momwe mungasungire chomwe mukufuna. Mwachitsanzo, wal_fsync_duration_seconds metric. Zolemba za etcd zimati: Kuti kusungirako kuganizidwe mwachangu mokwanira, 99th percentile ya metric iyi iyenera kukhala yosakwana 10ms. Ngati mukukonzekera kuyendetsa etcd cluster pamakina a Linux ndipo mukufuna kuyesa ngati chosungira chanu chikufulumira (monga SSD), mutha kugwiritsa ntchito fio ndi chida chodziwika bwino choyesera ntchito za I/O. Thamangani lamulo ili, pomwe test-data ndi chikwatu pansi pa malo osungira:

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

Mukungoyenera kuyang'ana zotsatira ndikuwona kuti 99 peresenti ya nthawiyo fdatasync zosakwana 10 ms. Ngati ndi choncho, muli ndi zosungira zofulumira. Nachi chitsanzo cha zotsatira:

  sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
  sync percentiles (usec):
   | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
   | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
   | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
   | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
   | 99.99th=[15795]

Mfundo

• Tasintha makonda --size ndi --bs zosankha pazochitika zathu. Kuti mupeze zotsatira zothandiza kuchokera ku fio, perekani zomwe mukufuna. Muwapeze kuti? Werengani momwe tinaphunzirira kukonza fio.
• Pakuyesa, katundu yense wa I/O amachokera ku fio. Muzochitika zenizeni, padzakhala zopempha zina zolembera zomwe zikubwera posungira kupatula zomwe zikugwirizana ndi wal_fsync_duration_seconds. Zowonjezera zidzakulitsa mtengo wa wal_fsync_duration_seconds. Chifukwa chake ngati 99th percentile ili pafupi ndi 10ms, kusungirako kwanu kukutha mwachangu.
• Tengani Baibulo fio osakwana 3.5 (zoyambazo sizikuwonetsa fdatasync kutalika kwa ma percentiles).
• Pamwambapa ndi kagawo kakang'ono ka zotsatira kuchokera ku fio.

Nkhani yayitali ya fio ndi etcd

WAL ndi chiyani etcd

Nthawi zambiri ma database amagwiritsa ntchito kulemba-patsogolo chipika; etcd amagwiritsanso ntchito. Sitidzakambirana za chipika cholembera kutsogolo (WAL) mwatsatanetsatane apa. Ndikokwanira kuti tidziwe kuti membala aliyense wa gulu la etcd amasunga posungira mosalekeza. etcd imalemba ntchito iliyonse yamtengo wapatali (monga zosintha) ku WAL musanayigwiritse ntchito ku sitolo. Ngati m'modzi mwa mamembala osungira aphwanyidwa ndikuyambiranso pakati pazithunzithunzi, imatha kubwezeretsanso zomwe zikuchitika kwanuko kuyambira chithunzi chomaliza cha WAL.

Pamene kasitomala akuwonjezera kiyi ku sitolo yamtengo wapatali kapena kusinthira mtengo wa kiyi yomwe ilipo, etcd imalemba ntchitoyo mu WAL, yomwe ndi fayilo yokhazikika yosungirako nthawi zonse. etcd MUYENERA kutsimikiza kuti kulowetsa kwa WAL kudachitikadi musanapitirize kukonza. Pa Linux, kuyimba kamodzi sikukwanira pa izi. kulemba, popeza zolemba zenizeni zosungirako zitha kuchedwa. Mwachitsanzo, Linux ikhoza kusunga cholowa cha WAL mu cache mu kernel memory (monga posungira tsamba) kwakanthawi. Ndipo kuti deta ilembedwe molondola kusungirako kosalekeza, kuyitana kwa dongosolo la fdatasync kumafunika pambuyo polemba, ndipo etcd amangogwiritsa ntchito (monga momwe mukuonera mu zotsatira za ntchitoyo. chingwe, pomwe 8 ndi WAL wofotokozera fayilo):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8)            = 0 <0.008314>

Tsoka ilo, kulembera kusungirako kosalekeza sikuchitika nthawi yomweyo. Ngati kuyimba kwa fdatasync kukuchedwa, magwiridwe antchito a etcd amavutika. Zolemba za etcd zimatikuti kusungirako kumayesedwa mofulumira ngati, mu 99th percentile, mafoni a fdatasync amatenga zosakwana 10ms kuti alembe ku fayilo ya WAL. Pali ma metric ena othandiza posungira, koma mu positi iyi tikungolankhula za metric iyi.

Kuyerekeza kusungirako ndi fio

Ngati mukufuna kuyesa ngati kusungirako kwanu kuli koyenera etcd, gwiritsani ntchito fio, chida chodziwika kwambiri cha I / O choyesa katundu. Tiyenera kukumbukira kuti ntchito za disk zingakhale zosiyana kwambiri: zogwirizanitsa ndi zosasinthika, magulu ambiri a mafoni a machitidwe, etc. Zotsatira zake, fio ndizovuta kugwiritsa ntchito. Ili ndi magawo ambiri, ndipo zophatikizika zosiyanasiyana zamakhalidwe awo zimatulutsa ntchito zambiri za I/O. Kuti mupeze ziwerengero zokwanira za etcd, muyenera kuwonetsetsa kuti mayeso alemba kuchokera ku fio ali pafupi kwambiri ndi katundu weniweni kuchokera etcd polemba mafayilo a WAL.

Chifukwa chake, fio iyenera, pang'ono, kupanga zolemba zingapo zotsatizana ndikulemba ku fayilo, kulemba kulikonse kudzakhala ndi kuyimba kwadongosolo. kulembakutsatiridwa ndi foni ya fdatasync system. Sequential imalemba ku fio imafuna --rw=write njira. Kuti fio igwiritse ntchito pulogalamu yolembera polemba, osati lembani, muyenera kufotokoza --ioengine=sync parameter. Pomaliza, kuti muyimbe fdatasync mukatha kulemba chilichonse, muyenera kuwonjezera --fdatasync=1 parameter. Zosankha zina ziwiri mu chitsanzo ichi (--size ndi -bs) ndizolemba zenizeni. Mu gawo lotsatira, tikuwonetsani momwe mungawakhazikitsire.

Chifukwa chiyani fio ndi momwe tinaphunzirira kuyiyika

Mu positi iyi, tikufotokoza nkhani yeniyeni. Tili ndi gulu Kubernetes v1.13 yomwe tidayang'anira ndi Prometheus. etcd v3.2.24 inachitikira pa SSD. Etcd metrics adawonetsa fdatasync latencies yokwera kwambiri, ngakhale gululi silikuchita kalikonse. Ma metrics anali odabwitsa ndipo sitinkadziwa zomwe amatanthauza. Gululi linali ndi makina enieni, kunali koyenera kumvetsetsa chomwe vuto linali: mu SSDs thupi kapena wosanjikiza virtualization. Kuonjezera apo, nthawi zambiri tinkasintha kusintha kwa hardware ndi mapulogalamu, ndipo timafunikira njira yowunikira zotsatira zawo. Titha kuthamanga etcd pamasinthidwe aliwonse ndikuyang'ana ma metric a Prometheus, koma ndizovutirapo kwambiri. Tinkafuna njira yosavuta yowonera masinthidwe enaake. Tinkafuna kuwona ngati tikumvetsetsa ma metric a Prometheus kuchokera etcd molondola.

Koma pa izi, mavuto awiri anayenera kuthetsedwa. Choyamba, kodi katundu wa I / O omwe etcd amapanga polemba ku WAL amawoneka bwanji? Ndi mafoni amtundu wanji omwe amagwiritsidwa ntchito? Kodi kukula kwa zolemba ndi chiyani? Chachiwiri, ngati tiyankha mafunsowa, timapanga bwanji ntchito yofanana ndi fio? Musaiwale kuti fio ndi chida chosinthika kwambiri chokhala ndi zosankha zambiri. Tinathetsa mavuto onsewa ndi njira imodzi - pogwiritsa ntchito malamulo lsof и chingwe. lsof imatchula zofotokozera zonse zamafayilo zomwe zimagwiritsidwa ntchito ndi ndondomekoyi ndi mafayilo ogwirizana nawo. Ndipo ndi zingwe, mutha kuyang'ana zomwe zikuchitika kale, kapena kuyambitsa ndondomeko ndikuyiyang'ana. strace imasindikiza mafoni onse pamachitidwe omwe akuwunikidwa (ndi njira zake zamwana). Izi ndizofunika kwambiri, chifukwa etcd ikungotenga njira yofanana.

Tidayamba kugwiritsa ntchito strace kufufuza etcd seva ya Kubernetes pomwe panalibe katundu pagulu. Tinawona kuti pafupifupi zolemba zonse za WAL zinali zofanana: 2200-2400 byte. Choncho, mu lamulo kumayambiriro kwa positi, tidatchula chizindikiro -bs = 2300 (bs zikutanthauza kukula kwa ma byte pa kulowa kwa fio). Dziwani kuti kukula kwa etcd kumadalira mtundu wa etcd, kugawa, ma parameter, ndi zina zotero, ndipo zimakhudza nthawi ya fdatasync. Ngati muli ndi mawonekedwe ofanana, yang'anani njira zanu za etcd ndi strace kuti mudziwe manambala enieni.

Kenako, kuti mukhale ndi lingaliro labwino la zomwe etcd file system ikuchita, tidayambitsa ndi strace ndi -ffttT zosankha. Chifukwa chake tidayesa kuyang'ana njira za mwana ndikulemba zomwe zatuluka mu fayilo yosiyana, komanso kupeza malipoti atsatanetsatane okhudza chiyambi ndi nthawi ya kuyimba kulikonse. Tinagwiritsa ntchito lsof kutsimikizira kusanthula kwathu kwa strace ndikuwona kuti ndi fayilo iti yomwe ikugwiritsidwa ntchito pazifukwa zake. Kotero mothandizidwa ndi strace, zotsatira zomwe zasonyezedwa pamwambapa zinapezedwa. Ziwerengero za nthawi yolumikizira zidatsimikizira kuti wal_fsync_duration_seconds kuchokera ku etcd imagwirizana ndi mafoni a fdatasync okhala ndi zofotokozera za fayilo ya WAL.

Tinadutsa zolemba za fio ndikusankha zosankha za script yathu kuti fio apange katundu wofanana ndi etcd. Tidayang'ananso mafoni amachitidwe ndi nthawi yake poyendetsa fio kuchokera ku strace, yofanana ndi etcd.

Tasankha mosamala mtengo wa --size parameter kuyimira katundu yense wa I/O kuchokera ku fio. Kwa ife, ichi ndi chiwerengero chonse cha ma byte olembedwa ku yosungirako. Zinapezeka kuti zimagwirizana mwachindunji ndi kuchuluka kwa mafoni olembera (ndi fdatasync). Pa mtengo wina wa bs, chiwerengero cha mafoni a fdatasync = size/bs. Popeza tinali ndi chidwi ndi percentile, tinayenera kukhala ndi zitsanzo zokwanira kuti titsimikize, ndipo tinawerengera kuti 10 ^ 4 ikanakhala yokwanira kwa ife (ndiyo 22 mebibytes). Ngati --size ndi yaying'ono, zotuluka zimatha kuchitika (mwachitsanzo, mafoni angapo a fdatasync amatenga nthawi yayitali kuposa masiku onse ndipo amakhudza 99th percentile).

Yesani nokha

Tinakuwonetsani momwe mungagwiritsire ntchito fio ndikuwona ngati kusungirako kuli ndi liwiro lokwanira pakuchita bwino etcd. Tsopano mutha kuyesa nokha pogwiritsa ntchito, mwachitsanzo, makina enieni okhala ndi SSD yosungirako Mtambo wa IBM.

Source: www.habr.com