He moʻolelo pōkole e pili ana i ka fio a me etcd
Hoʻokō pūʻulu pili nui i ka hana o kona waihona. etcd lawe aku i kekahi mau ana i e hāʻawi i ka ʻike hana hoʻopaʻa i makemake ʻia. No ka laʻana, ka wal_fsync_duration_seconds metric. : No ka mālama ʻana i ka wikiwiki, pono ka 99th percentile o kēia metric ma lalo o 10ms. Inā ʻoe e hoʻolālā e holo i kahi hui etcd ma nā mīkini Linux a makemake ʻoe e loiloi inā wikiwiki kāu mālama ʻana (e like me SSD), hiki iā ʻoe ke hoʻohana. He mea paahana kaulana ia no ka hoao ana i na hana I/O. E holo i ke kauoha aʻe, kahi i hōʻike ʻia ai ka papa kuhikuhi ma lalo o ka pae mauna mālama:
fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytestPono ʻoe e nānā i nā hopena a nānā i ka 99th percentile o ka lōʻihi emi iho malalo o 10 ms. Inā pēlā, loaʻa iā ʻoe kahi mālama wikiwiki. Eia kekahi laʻana o nā hopena:
sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
sync percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
| 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
| 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
| 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
| 99.99th=[15795]Nā memo
- Ua hana mākou i nā koho --size a me --bs no kā mākou hiʻohiʻona. No ka loaʻa ʻana o kahi hopena maikaʻi mai fio, hāʻawi i kāu mau waiwai ponoʻī. Ma hea e kiʻi ai iā lākou? Heluhelu .
- I ka hoʻāʻo ʻana, hele mai nā ukana I/O a pau mai fio. Ma kahi hiʻohiʻona ola maoli, aia paha nā noi kākau ʻē aʻe e hele mai i loko o ka waihona ma waho o nā mea e pili ana i wal_fsync_duration_seconds. E hoʻonui ka ukana ʻoi aku ka waiwai o wal_fsync_duration_seconds. No laila, inā kokoke ka 99th percentile i 10ms, ua pau ka wikiwiki o kāu waihona.
- Lawe i ka mana ʻaʻole ma lalo o 3.5 (ʻaʻole hōʻike nā mea mua i ka fdatasync lōʻihi percentiles).
- Aia ma luna kahi ʻāpana o nā hopena mai fio.
Moʻolelo lōʻihi e pili ana i ka fio a me etcd
He aha ka WAL ma etcd
Hoʻohana maʻamau nā waihona ; etcd hoʻohana pū kekahi. ʻAʻole mākou e kūkākūkā kikoʻī i ka log kākau mua (WAL) ma aneʻi. Ua lawa iā mākou ke ʻike i kēlā me kēia lālā o ka hui etcd e mālama iā ia i ka mālama mau. etcd kākau i kēlā me kēia hana waiwai kī (e like me ka hoʻohou) iā WAL ma mua o ka hoʻohana ʻana i ka hale kūʻai. Inā hāʻule kekahi o nā lālā mālama a hoʻomaka hou ma waena o nā kiʻi paʻi kiʻi, hiki iā ia ke hoʻihoʻi hou i nā kālepa mai ka paʻi hope ʻana e WAL maʻiʻo.
Ke hoʻohui ka mea kūʻai aku i kahi kī i ka hale kūʻai waiwai kī a i ʻole e hoʻonui i ka waiwai o kahi kī i loaʻa, etcd e hoʻopaʻa i ka hana ma WAL, he faila maʻamau i ka mālama mau. etcd Pono e hōʻoia ʻiʻo i ka komo ʻana o ka WAL ma mua o ka hoʻomau ʻana i ka hana. Ma Linux, ʻaʻole lawa kahi kelepona ʻōnaehana no kēia. , no ka mea hiki ke hoʻopaneʻe ʻia ke kākau maoli ʻana i ka waihona kino. No ka laʻana, hiki i Linux ke mālama i kahi komo WAL i loko o kahi huna i loko o ka hoʻomanaʻo kernel (e like me kahi huna ʻaoʻao) no kekahi manawa. A i mea e kākau pololei ʻia ai ka ʻikepili i ka mālama hoʻomau, pono ke kelepona ʻana i ka ʻōnaehana fdatasync ma hope o ke kākau ʻana, a hoʻohana wale ʻia etcd (e like me kāu e ʻike ai i ka hopena o ka hana. , ma kahi o 8 ka wehewehe faila WAL):
21:23:09.894875 lseek(8, 0, SEEK_CUR) = 12808 <0.000012>
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 <0.000130>
21:23:09.895041 fdatasync(8) = 0 <0.008314>ʻO ka mea pōʻino, ʻaʻole hiki koke ke kākau ʻana i kahi waihona hoʻomau. Inā lohi ke kelepona fdatasync, pilikia ka hana o ka ʻōnaehana etcd. e manaʻo wikiwiki ʻia ka mālama ʻana inā, ma ka 99th percentile, lawe ʻia nā kelepona fdatasync ma lalo o 10ms e kākau i ka faila WAL. Aia nā metric pono ʻē aʻe no ka mālama ʻana, akā ma kēia pou ke kamaʻilio wale nei mākou no kēia metric.
Ke kuhi nei i ka waiho ʻana me fio
Inā pono ʻoe e loiloi inā kūpono kāu waihona no etcd, e hoʻohana i ka fio, kahi mea hana hoʻāʻo I/O kaulana loa. Pono e hoʻomanaʻo ʻia hiki ke ʻokoʻa loa nā hana disk: synchronous a asynchronous, nui nā papa o nā kelepona ʻōnaehana, etc. ʻO ka hopena, paʻakikī loa ka hoʻohana ʻana i ka fio. Loaʻa iā ia nā ʻāpana he nui, a ʻo nā hui like ʻole o kā lākou mau waiwai e hana i nā mea hana I/O ʻokoʻa loa. No ka loaʻa ʻana o nā huahelu kūpono no etcd, pono ʻoe e hōʻoia ʻo ka hoʻāʻo kākau haʻawina mai fio e pili kokoke loa i ka ukana maoli mai etcd ke kākau ʻana i nā faila WAL.
No laila, pono, ma ka liʻiliʻi loa, e hana i kahi haʻawe o kahi moʻo o ka sequential kākau i ka faila, kēlā me kēia kākau e loaʻa i kahi kelepona ʻōnaehana. ukali ʻia e ke kelepona ʻōnaehana fdatasync. Kākau ʻo Sequential iā fio e koi i ke koho --rw=write. No ka fio e hoʻohana i ke kākau ʻōnaehana kelepona i ka wā kākau, ma mua o , pono ʻoe e kuhikuhi i ka --ioengine=sync parameter. ʻO ka hope, i mea e kāhea ai i ka fdatasync ma hope o kēlā me kēia kākau ʻana, pono ʻoe e hoʻohui i ka --fdatasync=1 parameter. ʻO nā koho ʻelua ʻē aʻe i kēia hiʻohiʻona (--size a me -bs) he kikoʻī kikoʻī. Ma ka ʻāpana aʻe, e hōʻike mākou iā ʻoe pehea e hoʻonohonoho ai iā lākou.
No ke aha fio a pehea mākou i aʻo ai e hoʻonohonoho
Ma kēia pou, wehewehe mākou i kahi hihia maoli. He pūʻulu kā mākou v1.13 a mākou i nānā ai me Prometheus. etcd v3.2.24 ua mālama ʻia ma kahi SSD. Hōʻike nā metric Etcd i nā latencies fdatasync kiʻekiʻe loa, ʻoiai inā ʻaʻohe hana o ka puʻupuʻu. He ʻano ʻē nā metric a ʻaʻole maopopo mākou i ke ʻano o ia mea. ʻO ka pūʻulu he mau mīkini virtual, pono e hoʻomaopopo i ke ʻano o ka pilikia: ma nā SSD kino a i ʻole ka papa virtualization. Eia hou, ua hana pinepine mākou i nā hoʻololi i ka hoʻonohonoho ʻenehana a me nā polokalamu, a pono mākou i kahi ala e loiloi ai i kā lākou hopena. Hiki iā mākou ke holo etcd i kēlā me kēia hoʻonohonoho ʻana a nānā i nā metric Prometheus, akā nui loa ka pilikia. Ua ʻimi mākou i kahi ala maʻalahi e loiloi i kahi hoʻonohonoho kikoʻī. Makemake mākou e nānā inā maopopo mākou i nā metric Prometheus mai etcd pololei.
Akā no kēia, pono e hoʻoponopono ʻia nā pilikia ʻelua. ʻO ka mea mua, he aha ke ʻano o ka ukana I/O a etcd i ke kākau ʻana iā WAL? He aha nā kelepona pūnaewele i hoʻohana ʻia? He aha ka nui o nā moʻolelo? ʻO ka lua, inā mākou e pane i kēia mau nīnau, pehea mākou e hana hou ai i kahi hana like me fio? Mai poina ʻo fio he mea hana maʻalahi loa me nā koho he nui. Hoʻoholo mākou i nā pilikia ʻelua me kahi ala hoʻokahi - me ka hoʻohana ʻana i nā kauoha и . lsof papa inoa i nā mea wehewehe waihona i hoʻohana ʻia e ke kaʻina hana a me kā lākou mau faila pili. A me ka strace, hiki iā ʻoe ke nānā i kahi kaʻina hana e holo nei, a i ʻole e hoʻomaka i kahi kaʻina a nānā iā ia. paʻi ʻo strace i nā kelepona ʻōnaehana āpau mai ke kaʻina hana e nānā ʻia nei (a me kāna mau hana keiki). He mea koʻikoʻi ka hope, no ka mea ke lawe wale nei ʻo etcd i kahi ala like.
Ua hoʻohana mua mākou i ka strace no ka ʻimi ʻana i ka server etcd no Kubernetes inā ʻaʻohe haʻahaʻa ma ka pūʻulu. Ua ʻike mākou ʻaneʻane like ka nui o nā moʻolelo WAL a pau: 2200–2400 bytes. No laila, ma ke kauoha i ka hoʻomaka ʻana o ka pou, ua kuhikuhi mākou i ka palena -bs=2300 (bs ʻo ia ka nui o nā bytes no kēlā me kēia komo fio). E hoʻomaopopo i ka nui o ke komo ʻana etcd e pili ana i ka etcd version, ka hāʻawi ʻana, nā koina parameter, etc., a pili i ka lōʻihi fdatasync. Inā loaʻa iā ʻoe kahi hiʻohiʻona like, e nānā i kāu kaʻina etcd me strace e ʻike i nā helu kikoʻī.
A laila, no ka loaʻa ʻana o ka manaʻo maikaʻi i ka hana a ka ʻōnaehana file etcd, hoʻomaka mākou me ka strace a me nā koho -ffttT. No laila ua hoʻāʻo mākou e nānā i nā kaʻina hana o ke keiki a hoʻopaʻa i ka hopena o kēlā me kēia o lākou i kahi faila ʻokoʻa, a loaʻa pū kekahi i nā hōʻike kikoʻī e pili ana i ka hoʻomaka a me ka lōʻihi o kēlā me kēia kelepona ʻōnaehana. Ua hoʻohana mākou i lsof e hōʻoia i kā mākou loiloi o ka strace output a ʻike i ka faila wehewehe i hoʻohana ʻia no ia kumu. No laila me ke kōkua o strace, ua loaʻa nā hopena i hōʻike ʻia ma luna. Ua hōʻoia ʻia nā helu helu manawa hoʻonohonoho ʻana ua kūlike ʻo wal_fsync_duration_seconds mai etcd me nā kelepona fdatasync me nā mea wehewehe faila WAL.
Ua hele mākou i nā palapala no fio a koho i nā koho no kā mākou palapala i hiki ai i ka fio ke hoʻopuka i kahi ukana e like me etcd. Ua nānā pū mākou i nā kelepona ʻōnaehana a me ko lākou lōʻihi ma ka holo ʻana o fio mai strace, e like me etcd.
Ua koho pono mākou i ka waiwai o ka --size parameter e hōʻike i ka ukana I/O holoʻokoʻa mai fio. I kā mākou hihia, ʻo ia ka huina o nā bytes i kākau ʻia i ka waihona. Ua hoʻololi pololei ia i ka helu o nā kelepona ʻōnaehana kākau (a me fdatasync). No kekahi waiwai o bs, ka helu o fdatasync kelepona = nui/bs. Ma muli o ko mākou hoihoi i ka pākēneka, pono mākou e lawa nā laʻana e maopopo ai, a ua helu mākou he 10^4 ka lawa no mākou (ʻo ia ka 22 mebibytes). Inā ʻoi aku ka liʻiliʻi o --size, hiki ke puka mai nā outliers (no ka laʻana, ʻoi aku ka lōʻihi o nā kelepona fdatasync ma mua o ka maʻamau a pili i ka 99th percentile).
E ho'āʻo iā ʻoe iho
Ua hōʻike mākou iā ʻoe pehea e hoʻohana ai i ka fio a ʻike inā lawa ka wikiwiki o ka waihona no ka hana kiʻekiʻe etcd. I kēia manawa hiki iā ʻoe ke hoʻāʻo iā ʻoe iho me ka hoʻohana ʻana, no ka laʻana, nā mīkini virtual me ka waiho SSD i loko .
Source: www.habr.com