"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Manaʻo wau e heluhelu ʻoe i ka transcript o ka haʻi'ōlelo "Hadoop. ZooKeeper" mai ka moʻo "Methods for distributed processing of large volumes of data in Hadoop"

He aha ka ZooKeeper, kona wahi i loko o ke kaiaola Hadoop. ʻO ka ʻoiaʻiʻo ʻole e pili ana i ka hoʻopili helu helu. Diagram of a standard distributed system. Paʻakikī i ka hoʻonohonoho ʻana i nā ʻōnaehana puʻunaue. Nā pilikia hoʻonohonoho maʻamau. ʻO nā loina ma hope o ka hoʻolālā ʻana o ZooKeeper. Hoʻohālike ʻikepili ZooKeeper. nā hae znode. Nā Kau. API mea kūʻai aku. Primitives (hoʻonohonoho, hui pūʻulu, laka maʻalahi, koho alakaʻi, laka me ka hopena ʻole o ka puaʻa). hale hana ZooKeeper. ZooKeeper DB. ZAB. Ka mea lawelawe noi.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

I kēia lā e kamaʻilio mākou e pili ana iā ZooKeeper. He mea pono loa keia. ʻO ia, e like me nā huahana Apache Hadoop, he logo. Hōʻike ia i ke kāne.

Ma mua o kēia, ua kamaʻilio nui mākou e pili ana i ka hana ʻana o ka ʻikepili ma laila, pehea e mālama ai, ʻo ia hoʻi, pehea e hoʻohana ai a hana pū me ia. A i kēia lā makemake wau e kamaʻilio liʻiliʻi e pili ana i ke kūkulu ʻana i nā palapala hoʻolaha. A ʻo ZooKeeper kekahi o nā mea e hiki ai iā ʻoe ke maʻalahi i kēia mea. He ʻano lawelawe kēia i manaʻo ʻia no kekahi ʻano hoʻonohonoho ʻana o ka launa pū ʻana o nā kaʻina hana i nā ʻōnaehana hoʻolaha, i nā noi hoʻolaha.

Ke ulu nui nei ka pono o ia mau noi i kēlā me kēia lā, ʻo ia ke ʻano o kā mākou papa. Ma kekahi ʻaoʻao, ʻo MapReduce a me kēia hoʻolālā i hoʻomākaukau ʻia e hiki ai iā ʻoe ke hoʻohaʻahaʻa i kēia paʻakikī a hoʻokuʻu i ka mea papahana mai ke kākau mua ʻana e like me ka pilina a me ka hoʻonohonoho ʻana o nā kaʻina hana. Akā ma kekahi ʻaoʻao, ʻaʻohe mea e hōʻoiaʻiʻo ʻaʻole pono e hana ʻia kēia. ʻO MapReduce a i ʻole nā ​​papa hana mākaukau ʻē aʻe ʻaʻole e hoʻololi mau i kekahi mau hihia i hiki ʻole ke hoʻokō ʻia me kēia. Hoʻohui pū me MapReduce iā ia iho a me kahi pūʻulu o nā papahana Apache ʻē aʻe; ʻoiaʻiʻo, ua hāʻawi pū ʻia lākou i nā noi. A i mea e maʻalahi ai ke kākau ʻana, ua kākau lākou iā ZooKeeper.

E like me nā noi āpau e pili ana iā Hadoop, ua hoʻomohala ʻia e Yahoo! I kēia manawa he noi Apache mana. ʻAʻole i hoʻomohala ʻia e like me ka HBase. Inā hele ʻoe i JIRA HBase, a laila i kēlā me kēia lā he pūʻulu o nā hōʻike bug, kahi pūʻulu o nā manaʻo e hoʻomaikaʻi i kekahi mea, ʻo ia hoʻi ke ola mau nei ka papahana. A ʻo ZooKeeper, ma kekahi ʻaoʻao, he huahana maʻalahi, a ma kekahi ʻaoʻao, hōʻoia kēia i kona hilinaʻi. A he mea maʻalahi loa ia e hoʻohana, ʻo ia ke kumu i lilo ai ia i mea maʻamau i nā noi i loko o ka kaiaola Hadoop. No laila, manaʻo wau he mea pono ke nānā ʻana iā ia e hoʻomaopopo i ka hana a pehea e hoʻohana ai.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

He kiʻi kēia mai kekahi haʻiʻōlelo i loaʻa iā mākou. Hiki iā mākou ke ʻōlelo he orthogonal i nā mea āpau a mākou i noʻonoʻo ai i kēia manawa. A ʻo nā mea a pau i hōʻike ʻia ma ʻaneʻi, i kekahi degere a i ʻole, hana pū me ZooKeeper, ʻo ia hoʻi, he lawelawe ia e hoʻohana nei i kēia mau huahana āpau. ʻAʻole kākau ʻo HDFS a i ʻole MapReduce i kā lākou mau lawelawe like ʻole e hana pono no lākou. No laila, hoʻohana ʻia ʻo ZooKeeper. A maʻalahi kēia i ka hoʻomohala ʻana a me kekahi mau mea e pili ana i nā hewa.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

No hea mai kēia mau mea a pau? Me he mea lā ua hoʻolauna mākou i ʻelua mau noi i ka like ʻole ma nā kamepiula ʻokoʻa, hoʻopili iā lākou me kahi kaula a i ʻole i ka mesh, a hana nā mea a pau. Akā ʻo ka pilikia, ʻaʻole hiki ke hilinaʻi ʻia ka Pūnaewele, a inā ʻoe i honi i ke kaʻa a nānā paha i nā mea e hana nei ma laila ma kahi haʻahaʻa, pehea ka launa ʻana o nā mea kūʻai aku ma ka Pūnaewele, hiki iā ʻoe ke ʻike pinepine ua nalowale a hoʻouna hou ʻia kekahi mau ʻeke. ʻAʻole ia no ka mea ʻole i hana ʻia nā protocol TCP, e ʻae iā ʻoe e hoʻokumu i kahi kau a hōʻoia i ka lawe ʻana i nā leka. Akā i kekahi hihia, ʻaʻole hiki iā TCP ke hoʻopakele iā ʻoe. Loaʻa i nā mea a pau kahi manawa. Hiki ke hāʻule wale ka pūnaewele no kekahi manawa. Hiki wale nō paha. A ke alakaʻi nei kēia i ka ʻoiaʻiʻo ʻaʻole hiki iā ʻoe ke hilinaʻi i ka Pūnaewele i hilinaʻi ʻia. ʻO kēia ka ʻokoʻa nui mai ke kākau ʻana i nā noi like e holo ana ma hoʻokahi kamepiula a i ʻole hoʻokahi supercomputer, kahi ʻaʻohe Pūnaewele, kahi i loaʻa kahi pahi hoʻololi data hilinaʻi i ka hoʻomanaʻo. A he ʻokoʻa kumu kēia.

Ma waena o nā mea ʻē aʻe, i ka wā e hoʻohana ai i ka Pūnaewele, aia mau kahi latency. Loaʻa iā ia ka disk, akā ʻoi aku ka nui o ka Pūnaewele. ʻO ka latency kekahi manawa hoʻopaneʻe, hiki ke liʻiliʻi a koʻikoʻi paha.

Ke loli nei ka topology pūnaewele. He aha ka topology - ʻo ia ke kau ʻana o kā mākou lako pūnaewele. Aia nā kikowaena ʻikepili, aia nā pā e kū ana ma laila, aia nā kukui. Hiki ke hoʻopili hou ʻia kēia mau mea a pau, hoʻoneʻe ʻia, a me nā mea ʻē aʻe. Hoʻololi nā inoa IP, hoʻololi ke alahele e hele ai kā mākou huakaʻi. Pono e noʻonoʻo ʻia kēia.

Hiki ke hoʻololi ʻia ka pūnaewele ma ke ʻano o nā lako. Mai ka hoʻomaʻamaʻa ʻana, hiki iaʻu ke ʻōlelo aku makemake nui kā mākou ʻenekini pūnaewele e hoʻonui i kekahi mea ma nā kukui. Ua puka koke mai kahi firmware hou a ʻaʻole lākou i makemake nui i kekahi hui Hadoop. Aia kā lākou hana pono'ī. No lākou, ʻo ka mea nui ka hana o ka Pūnaewele. No laila, makemake lākou e hoʻouka hou i kekahi mea ma laila, e hoʻolohilohi i kā lākou hāmeʻa, a hoʻololi hoʻi ka hāmeʻa i kēlā me kēia manawa. Pono e noʻonoʻo ʻia kēia mau mea a pau. Pili kēia mau mea a pau i kā mākou palapala hoʻolaha.

ʻO ka mea maʻamau ka poʻe e hoʻomaka e hana me ka nui o ka ʻikepili no kekahi kumu e manaʻoʻiʻo he palena ʻole ka Pūnaewele. Inā loaʻa kahi faila o nā terabytes ma laila, hiki iā ʻoe ke lawe i kāu kikowaena a i ʻole kamepiula a wehe iā ia me ka hoʻohana ʻana popoki a e kiai. Aia kekahi hewa i loko Vim e nana i na laau. Mai hana i kēia no ka mea ʻino. No ka mea, hoʻāʻo ʻo Vim e pale i nā mea āpau, e hoʻouka i nā mea āpau i ka hoʻomanaʻo, ʻoi aku ka wā e hoʻomaka ai mākou e neʻe i kēia log a ʻimi i kahi mea. He mau mea poina kēia, akā pono e noʻonoʻo.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻOi aku ka maʻalahi o ke kākau ʻana i hoʻokahi papahana e holo ana ma hoʻokahi kamepiula me hoʻokahi mea hana.

Ke ulu aʻe kā mākou ʻōnaehana, makemake mākou e hoʻohālikelike i nā mea a pau, a e hoʻohālikelike iā ia ʻaʻole wale ma ke kamepiula, akā ma kahi hui pū kekahi. Aia ka nīnau: pehea e hoʻonohonoho ai i kēia mea? ʻAʻole hiki i kā mākou mau noi ke launa pū kekahi me kekahi, akā ua holo mākou i nā kaʻina hana like ʻole ma kekahi mau kikowaena. A pehea e nānā ai e holo maikaʻi ana nā mea a pau iā lākou? No ka laʻana, hoʻouna lākou i kekahi mea ma ka Pūnaewele. Pono lākou e kākau e pili ana i ko lākou mokuʻāina ma kekahi wahi, no ka laʻana, i loko o kekahi ʻano waihona a i ʻole ka log, a laila e hōʻuluʻulu i kēia log a laila e kālailai ma kahi. Eia kekahi, pono mākou e noʻonoʻo i ka hana ʻana a me ka hana ʻana o ke kaʻina hana, ʻike koke ʻia kekahi hewa i loko o ia mea a hāʻule paha, a laila pehea ka wikiwiki e ʻike ai mākou e pili ana iā ia?

Ua maopopo e hiki ke nānā kokeʻia kēia mau mea a pau. He maikaʻi nō hoʻi kēia, akā he mea kaupalena ka nānā ʻana e hiki ai iā ʻoe ke nānā i kekahi mau mea ma ka pae kiʻekiʻe.

Ke makemake mākou i kā mākou kaʻina hana e hoʻomaka e launa pū me kekahi, no ka laʻana, e hoʻouna i kekahi i kekahi i kekahi mau ʻikepili, a laila e kū mai ka nīnau - pehea e hiki ai kēia? E loaʻa paha kekahi ʻano kūlana lāhui, e kākau paha lākou i kekahi i kekahi, e hōʻea pololei ka ʻikepili, e nalowale kekahi mea ma ke ala? Pono mākou e kūkulu i kekahi ʻano protocol, etc.

ʻO ka hoʻonohonoho ʻana i kēia mau hana a pau ʻaʻole ia he mea liʻiliʻi. A ke koi nei ia i ka mea hoʻomohala e iho i lalo i kahi pae haʻahaʻa, a kākau i nā ʻōnaehana mai ka ʻohi ʻana, a i ʻole mai ka ʻohi ʻana, akā ʻaʻole maʻalahi kēia.

Inā hele mai ʻoe me kahi algorithm cryptographic a hoʻokō paha iā ia, a laila e hoʻolei koke iā ia, no ka mea, ʻaʻole pono ia no ʻoe. Loaʻa paha ia i kahi hui hewa i poina iā ʻoe e hoʻolako. Mai hoʻohana iā ia no kekahi mea koʻikoʻi no ka mea e paʻa paha ia. No ka mea, ua hoʻāʻo ʻia nā algorithm a pau e ka manawa no ka manawa lōʻihi loa. Hoʻopilikia ʻia e ke kaiāulu. He kumuhana kaawale keia. A ua like ia maanei. Inā hiki ʻole ke hoʻokō i kekahi ʻano kaʻina hana synchronization iā ʻoe iho, a laila ʻoi aku ka maikaʻi o ka hana ʻole i kēia, no ka mea paʻakikī loa ia a alakaʻi iā ʻoe i ke ala haʻalulu o ka ʻimi mau ʻana i nā hewa.

I kēia lā ke kamaʻilio nei mākou no ZooKeeper. Ma kekahiʻaoʻao, he hoʻolālā ia, ma kekahiʻaoʻao, he lawelawe ia e maʻalahi ai ke ola no ka mea hoʻomohala a hoʻomāmā i ka hoʻokōʻana i ka loiloi a me ka hoʻohuiʻana i kā mākou mau kaʻina hana e like me ka hiki.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

E hoʻomanaʻo kākou i ke ʻano o ka ʻōnaehana puʻunaue maʻamau. ʻO kēia ka mea a mākou i kamaʻilio ai - HDFS, HBase. Aia kahi papa hana Master e hoʻokele i nā limahana a me nā hana kauā. ʻO ia ke kuleana no ka hoʻonohonoho ʻana a me ka hāʻawi ʻana i nā hana, ka hoʻomaka hou ʻana i nā limahana, ka hoʻokuʻu ʻana i nā mea hou, a me ka hāʻawi ʻana i ka ukana.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO kahi mea ʻoi aku ka holomua ʻo ka Coordination Service, ʻo ia hoʻi, e hoʻoneʻe i ka hana hoʻonohonoho ponoʻī iā ia iho i kahi kaʻina hana ʻokoʻa, a me ka holo ʻana i kekahi ʻano backup a i ʻole stanby Master i ka like, no ka mea hiki ʻole ke Kumu. A inā hāʻule ka Haku, ʻaʻole e holo kā mākou ʻōnaehana. Ke holo nei mākou i ke kākoʻo. Ua ʻōlelo kekahi e pono e hoʻihoʻi ʻia ka Master i ka backup. Hiki ke hāʻawi ʻia kēia i ka Coordination Service. Akā ma kēia kiʻikuhi, ʻo ka Haku ponoʻī ke kuleana no ka hoʻonohonoho ʻana i nā limahana; eia ka lawelawe e hoʻonohonoho i nā hana replication data.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO kahi koho kiʻekiʻe loa ke mālama ʻia nā hoʻonohonoho āpau e kā mākou lawelawe, e like me ka hana maʻamau. Lawe ʻo ia i ke kuleana no ka hōʻoia ʻana e holo pono nā mea a pau. A inā ʻaʻole hana kekahi mea, ʻike mākou e pili ana iā ia a hoʻāʻo e hoʻopuni i kēia kūlana. I kēlā me kēia hihia, waiho mākou me kahi Haku nāna e launa pū me nā kauā a hiki ke hoʻouna i ka ʻikepili, ka ʻike, nā leka, a me nā mea ʻē aʻe ma o kekahi lawelawe.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Aia kekahi papahana ʻoi aku ka holomua, ke loaʻa ʻole iā mākou kahi Haku, he mau kauā haku nā node a pau, ʻokoʻa kā lākou ʻano. Akā, pono lākou e launa pū kekahi me kekahi, no laila aia kekahi lawelawe i koe e hoʻonohonoho i kēia mau hana. Malia paha, ʻo Cassandra, ka mea e hana nei ma kēia kumumanaʻo, kūpono i kēia hoʻolālā.

He paʻakikī ke haʻi aku ʻo wai o kēia mau papahana ʻoi aku ka maikaʻi. Loaʻa i kēlā me kēia me kona pono ponoʻī.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

A ʻaʻohe pono e makaʻu i kekahi mau mea me ka Haku, no ka mea, e like me ka hōʻike ʻana o ka hana, ʻaʻole hiki iā ia ke lawelawe mau. ʻO ka mea nui ma ʻaneʻi ʻo ke koho ʻana i ka hopena kūpono no ka hoʻokipa ʻana i kēia lawelawe ma kahi puʻupuʻu ikaika ʻokoʻa, i lawa ka waiwai, i hiki ʻole i nā mea hoʻohana ke komo i laila, i ʻole lākou e pepehi i kēia kaʻina hana. Akā i ka manawa like, i loko o ia ʻano hoʻolālā ʻoi aku ka maʻalahi o ka hoʻokele ʻana i nā limahana mai ke kaʻina Master, ʻo ia hoʻi, ʻoi aku ka maʻalahi o kēia hoʻolālā mai ka manaʻo o ka hoʻokō.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

A ʻoi aku ka paʻakikī o kēia hoʻolālā (ma luna), akā ʻoi aku ka hilinaʻi.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO ka pilikia nui he hapa hapa. No ka laʻana, ke hoʻouna mākou i kahi leka ma luna o ka Pūnaewele, hiki mai kekahi ʻano pōʻino, a ʻo ka mea nāna i hoʻouna i ka leka ʻaʻole ʻike i ka loaʻa ʻana o kāna leka a me ka mea i hana ʻia ma ka ʻaoʻao o ka mea hoʻokipa, ʻaʻole ia e ʻike inā ua hana pololei ʻia ka leka. , ʻo ia hoʻi, ʻaʻole e loaʻa iā ia kahi hōʻoia.

No laila, pono mākou e hoʻoponopono i kēia kūlana. A ʻo ka mea maʻalahi ka hoʻouna hou ʻana i kēia leka a kali a loaʻa iā mākou kahi pane. I kēia hihia, ʻaʻole i manaʻo ʻia inā ua loli ke kūlana o ka mea hoʻokipa. Hiki iā mākou ke hoʻouna i kahi leka a hoʻohui i ka ʻikepili like ʻelua.

Hāʻawi ʻo ZooKeeper i nā ala e hoʻoponopono ai i kēlā mau hōʻole, e maʻalahi hoʻi i ko mākou ola.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

E like me ka mea i haʻi ʻia ma mua, ua like kēia me ke kākau ʻana i nā papahana multi-threaded, akā ʻo ka ʻokoʻa nui ʻo ia i nā noi i hoʻokaʻawale ʻia a mākou e kūkulu ai ma nā mīkini like ʻole, ʻo ke ala wale nō e kamaʻilio ai ʻo ia ka Pūnaewele. ʻO ka mea nui, he hoʻolālā like ʻole kēia. ʻO kēlā me kēia kaʻina hana a lawelawe paha e holo ana ma ka mīkini hoʻokahi he hoʻomanaʻo ponoʻī kona, kāna disk ponoʻī, kāna mea hana ponoʻī, ʻaʻole ia e kaʻana like me kekahi.

Inā mākou e kākau i kahi papahana multi-threaded ma hoʻokahi kamepiula, a laila hiki iā mākou ke hoʻohana i ka hoʻomanaʻo like e hoʻololi i ka ʻikepili. Loaʻa iā mākou kahi hoʻololi pōʻaiapili ma laila, hiki ke hoʻololi nā kaʻina hana. Pili kēia i ka hana. Ma kekahiʻaoʻao,ʻaʻohe mea i loko o ka papahana ma kahi hui, akā aia nā pilikia me ka Pūnaewele.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

No laila, ʻo nā pilikia nui e kū mai ana i ka wā e kākau ai i nā ʻōnaehana puʻupuʻu he hoʻonohonoho. Ke kākau nei mākou i kekahi ʻano noi. Inā he maʻalahi, a laila paʻakikī mākou i nā ʻano helu āpau i ke code, akā he mea kūpono ʻole kēia, no ka mea inā hoʻoholo mākou ma kahi o kahi manawa o ka hapalua kekona makemake mākou i kahi manawa o hoʻokahi kekona, a laila pono mākou e hoʻohui hou i ka noi a e owili hou i na mea a pau. Hoʻokahi mea i ka wā ma ka mīkini hoʻokahi, hiki iā ʻoe ke hoʻomaka hou, akā inā loaʻa iā mākou nā mīkini he nui, pono mākou e kope mau i nā mea āpau. Pono mākou e ho'āʻo e hoʻonohonoho i ka noi.

Eia mākou e kamaʻilio nei e pili ana i ka hoʻonohonoho static no nā kaʻina ʻōnaehana. ʻAʻole holoʻokoʻa kēia, mai ka manaʻo o ka ʻōnaehana hana, he hoʻonohonoho static paha ia no kā mākou kaʻina hana, ʻo ia hoʻi, he hoʻonohonoho ʻaʻole hiki ke lawe wale ʻia a hoʻonui ʻia.

Aia kekahi hoʻonohonoho dynamic. ʻO kēia nā ʻāpana a mākou e makemake ai e hoʻololi i ka lele i ʻohi ʻia lākou ma laila.

He aha ka pilikia ma ʻaneʻi? Ua hōʻano hou mākou i ka config, ʻōwili ʻia, no laila pehea? ʻO ka pilikia paha ma ka ʻaoʻao hoʻokahi ua ʻōwili mākou i ka config, akā poina i ka mea hou, ua noho ka config ma laila. ʻO ka lua, ʻoiai mākou e holo nei, ua hoʻonui ʻia ka hoʻonohonoho ʻana ma kekahi mau wahi, akā ʻaʻole ma nā wahi ʻē aʻe. A ua hoʻomaka hou kekahi mau kaʻina hana o kā mākou noi e holo ana ma ka mīkini hoʻokahi me kahi config hou, a ma kahi me kahi mea kahiko. Hiki i kēia ke hopena i kā mākou palapala hoʻolaha i kū ʻole mai kahi hiʻohiʻona hoʻonohonoho. He mea maʻamau kēia pilikia. No kahi hoʻonohonoho hoʻoikaika, ʻoi aku ka pili no ka mea e hōʻike ana hiki ke hoʻololi i ka lele.

ʻO kekahi pilikia ʻo ka hui pū ʻana. Loaʻa iā mākou kekahi mau limahana, makemake mau mākou e ʻike ʻo wai o lākou e ola nei, ʻo wai o lākou i make. Inā he Kumu, a laila pono ʻo ia e hoʻomaopopo i nā limahana hiki ke hoʻihoʻi ʻia i nā mea kūʻai aku i holo lākou i ka helu ʻana a i ʻole hana me ka ʻikepili, a ʻaʻole hiki. ʻO kahi pilikia e ulu mau nei, pono mākou e ʻike i ka mea e hana nei i kā mākou hui.

ʻO kekahi pilikia maʻamau ʻo ke koho alakaʻi, ke makemake mākou e ʻike i ka mea nāna e mālama. ʻO kekahi laʻana, ʻo ka hoʻopiʻi ʻana, ke loaʻa iā mākou kekahi kaʻina hana e loaʻa ai nā hana kākau a laila hoʻopili iā lākou i waena o nā kaʻina hana ʻē aʻe. ʻO ia ke alakaʻi, e hoʻolohe nā mea ʻē aʻe iā ia, e hahai iā ia. Pono e koho i kahi kaʻina hana i maopopo ʻole i nā mea a pau, i ʻole e ʻike ʻia nā alakaʻi ʻelua.

Loaʻa pū kekahi. ʻOi aku ka paʻakikī o ka pilikia ma ʻaneʻi. Aia kekahi mea e like me ka mutex, ke kākau ʻoe i nā papahana multi-threaded a makemake ʻoe e komo i kekahi kumuwaiwai, no ka laʻana, kahi cell memory, e kaupalena ʻia a hoʻokō ʻia e hoʻokahi wale nō kaula. Ma ʻaneʻi hiki ke lilo i mea ʻokoʻa. A ʻo nā noi ʻokoʻa mai nā node like ʻole o kā mākou Pūnaewele e loaʻa wale ke komo kūʻokoʻa i kahi kumuwaiwai i hāʻawi ʻia, ʻaʻole i hiki i nā mea a pau ke hoʻololi a kākau paha i kekahi mea ma laila. ʻO kēia nā mea i kapa ʻia nā loka.

Hāʻawi ʻo ZooKeeper iā ʻoe e hoʻoponopono i kēia mau pilikia āpau i kahi pae a i ʻole. A e hōʻike wau me nā hiʻohiʻona pehea e hiki ai iā ʻoe ke hana i kēia.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻAʻohe mea pale primitives. Ke hoʻomaka mākou e hoʻohana i kekahi mea, ʻaʻole e kali kēia primitive no kekahi hanana. ʻO ka mea maʻamau, e hana like kēia mea, no laila e ʻae i nā kaʻina hana ʻaʻole e kau i ka wā e kali ana lākou i kekahi mea. He mea pono loa keia.

Hoʻoponopono ʻia nā noi a pau o ka mea kūʻai aku ma ke ʻano o ka pila maʻamau.

A loaʻa i nā mea kūʻai ka manawa e loaʻa ai ka hoʻolaha e pili ana i nā loli i kekahi mokuʻāina, e pili ana i nā loli i ka ʻikepili, ma mua o ka ʻike ʻana o ka mea kūʻai aku i ka ʻikepili i hoʻololi ʻia iā lākou iho.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Hiki iā ZooKeeper ke hana i ʻelua mau ʻano. ʻO ka mua he kūʻokoʻa, ma hoʻokahi node. He kūpono kēia no ka hoʻāʻo ʻana. Hiki iā ia ke hana ma ke ʻano cluster ma kekahi helu o nā kikowaena. Inā loaʻa iā mākou kahi hui o 100 mau mīkini, a laila ʻaʻole pono ia e hana i nā mīkini 100. Ua lawa ke koho ʻana i nā mīkini kahi e hiki ai iā ʻoe ke holo ZooKeeper. A haʻi ia i ke kumu o ka loaʻa kiʻekiʻe. Ma kēlā me kēia hiʻohiʻona holo, mālama ʻo ZooKeeper i kahi kope holoʻokoʻa o ka ʻikepili. Ma hope e haʻi wau iā ʻoe pehea ʻo ia e hana ai. ʻAʻole ia e hoʻokaʻawale i ka ʻikepili a ʻokiʻoki paha. Ma kekahi ʻaoʻao, he hōʻemi ia ʻaʻole hiki iā mākou ke mālama nui, ma kekahi ʻaoʻao, ʻaʻohe pono e hana i kēia. ʻAʻole ia ka mea i hoʻolālā ʻia, ʻaʻole ia he waihona.

Hiki ke hūnā ʻia ka ʻikepili ma ka ʻaoʻao o ka mea kūʻai aku. He kumu maʻamau kēia i ʻole mākou e hoʻopau i ka lawelawe a ʻaʻole hoʻi e hoʻouka me nā noi like. ʻIke pinepine ka mea kūʻai akamai i kēia a hūnā iā ia.

Eia kekahi laʻana, ua loli kekahi mea ma ʻaneʻi. Aia kekahi ʻano noi. Ua koho ʻia kahi alakaʻi hou, nona ke kuleana, no ka laʻana, no ka hana ʻana i nā hana kākau. A makemake mākou e hana hou i ka ʻikepili. ʻO kahi hoʻonā e hoʻokomo i loko o kahi loop. A nīnau mau mākou i kā mākou lawelawe - ua loli paha kekahi mea? ʻOi aku ka maikaʻi o ka koho lua. He hana wati kēia e hiki ai iā ʻoe ke hoʻomaopopo i nā mea kūʻai aku ua loli kekahi mea. ʻO kēia kahi ala ʻoi aku ka uku ma ke ʻano o nā kumuwaiwai a ʻoi aku ka maʻalahi no nā mea kūʻai aku.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO ka mea hoʻohana ka mea hoʻohana iā ZooKeeper.

ʻO ke kikowaena ka hana ZooKeeper ponoʻī.

ʻO Znode ka mea nui ma ZooKeeper. Mālama ʻia nā znodes a pau i ka hoʻomanaʻo e ZooKeeper a ua hoʻonohonoho ʻia ma ke ʻano o kahi kiʻi hierarchical, ma ke ʻano o kahi lāʻau.

ʻElua ʻano hana. ʻO ka mea mua, ʻo ia ka hoʻohou/kākau, ke hoʻololi kekahi hana i ke kūlana o kā mākou kumulāʻau. He mea maʻamau ka lāʻau.

A ʻaʻole hiki i ka mea kūʻai ke hoʻopau i hoʻokahi noi a hoʻokaʻawale ʻia, akā hiki ke hoʻokumu i kahi hālāwai e launa pū ai me ZooKeeper.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Ua like ko ZooKeeper ikepili me ka waihona waihona. Aia kahi kumu maʻamau a laila hele mākou me he mea lā ma o nā papa kuhikuhi e hele mai ke kumu. A laila ka papa inoa o ka pae mua, pae ʻelua. ʻO kēia nā znodes a pau.

Hiki i kēlā me kēia znode ke mālama i kekahi mau ʻikepili, ʻaʻole nui loa, no ka laʻana, 10 kilobytes. A hiki i kēlā me kēia znode ke loaʻa kahi helu o nā keiki.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Hiki mai nā Znodes i nā ʻano like ʻole. Hiki ke hana ia lakou. A i ka hana ʻana i kahi znode, kuhikuhi mākou i ke ʻano e pono ai.

ʻElua ʻano. ʻO ka mua ka hae ephemeral. Noho ʻo Znode i loko o kahi kau. No ka laʻana, ua hoʻokumu ka mea kūʻai aku i kahi kau. A i ka wā e ola ana kēia kau, e mau ana. Pono kēia i ʻole e hana i kahi mea pono ʻole. He kūpono kēia no nā manawa he mea nui iā mākou ke mālama i nā ʻikepili primitives i loko o kahi kau.

ʻO ka lua o ka ʻano he sequential hae. Hoʻonui ia i ka counter ma ke ala i ka znode. No ka laʻana, loaʻa iā mākou kahi papa kuhikuhi me ka noi 1_5. A i ka wā i hana ai mākou i ka node mua, loaʻa iā ia ka p_1, ka lua - p_2. A ke kāhea mākou i kēia ʻano i kēlā me kēia manawa, hele mākou i ke ala piha, e hōʻike ana i kahi hapa o ke ala, a hoʻonui ʻia kēia helu no ka mea e hōʻike ana mākou i ke ʻano node - sequential.

Znode mau. E ola mau ʻo ia a loaʻa iā ia ka inoa a mākou e haʻi aku ai iā ia.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO kekahi mea pono ʻo ka hae kiaʻi. Inā mākou e hoʻokomo iā ia, a laila hiki i ka mea kūʻai ke kau inoa i kekahi mau hanana no kahi node kikoʻī. E hōʻike wau iā ʻoe ma hope me kahi laʻana pehea e hana ʻia ai kēia. Hoʻomaopopo ʻo ZooKeeper i ka mea kūʻai aku ua loli ka ʻikepili ma ka node. Eia naʻe, ʻaʻole hōʻoia ka hoʻolaha ʻana ua hiki mai kekahi mau ʻikepili hou. 'Ōlelo wale lākou ua loli kekahi mea, no laila pono ʻoe e hoʻohālikelike i ka ʻikepili ma hope me nā kelepona ʻokoʻa.

A e like me kaʻu i ʻōlelo ai, ua hoʻoholo ʻia ke ʻano o ka ʻikepili e nā kilobytes. ʻAʻole pono e mālama i ka ʻikepili kikokikona nui ma laila, no ka mea, ʻaʻole ia he waihona, he kikowaena hoʻonohonoho hana.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

E haʻi iki wau iā ʻoe e pili ana i nā kau. Inā loaʻa iā mākou nā kikowaena he nui, a laila hiki iā mākou ke neʻe pono mai kahi kikowaena i kahi kikowaena me ka hoʻohana ʻana i ka ʻike manawa. He kūpono loa.

Loaʻa i kēlā me kēia kau kekahi ʻano manawa pau. Ua wehewehe ʻia kahi kau ma ka hoʻouna ʻana o ka mea kūʻai aku i kekahi mea i ke kikowaena ma ia kau. Inā ʻaʻole ʻo ia i hoʻouna i kekahi mea i ka wā hoʻomaha, hāʻule ka hālāwai, a i ʻole hiki i ka mea kūʻai ke pani iā ​​ia iho.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻAʻole ia he nui nā hiʻohiʻona, akā hiki iā ʻoe ke hana i nā mea like ʻole me kēia API. ʻO kēlā kelepona a mākou i ʻike ai e hana i kahi znode a lawe i ʻekolu mau ʻāpana. ʻO kēia ke ala i ka znode, a pono e wehewehe piha ʻia mai ke kumu. A ʻo kēia kekahi mau ʻikepili a mākou e makemake ai e hoʻoili i laila. A me ke ʻano o ka hae. A ma hope o ka hana ʻana e hoʻihoʻi i ke ala i ka znode.

ʻO ka lua, hiki iā ʻoe ke holoi. ʻO ka hoʻopunipuni ma aneʻi ʻo ka lua o ka ʻāpana, me ka hoʻohui i ke ala i ka znode, hiki ke kuhikuhi i ka mana. No laila, e hoʻopau ʻia kēlā znode inā like kona mana a mākou i hoʻoili ai me ka mea i loaʻa maoli.

Inā ʻaʻole makemake mākou e nānā i kēia mana, a laila e hāʻawi wale mākou i ka hoʻopaʻapaʻa "-1".

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO ke kolu, nānā ia i ke ola o kahi znode. E hoʻi i ka ʻoiaʻiʻo inā loaʻa ka node, ʻaʻole hewa.

A laila ʻike ʻia ka wati hae, kahi e hiki ai iā ʻoe ke nānā i kēia node.

Hiki iā ʻoe ke hoʻonoho i kēia hae ma kahi node ʻaʻole i loaʻa a loaʻa ka leka hoʻomaopopo ke ʻike ʻia. He mea pono no hoi keia.

ʻElua mau pilikia getData. Ua maopopo hiki iā mākou ke loaʻa ka ʻikepili ma o znode. Hiki iā ʻoe ke hoʻohana i ka wati hae. I kēia hihia, ʻaʻole ia e hoʻokomo inā ʻaʻohe node. No laila, pono ʻoe e hoʻomaopopo aia aia, a laila loaʻa ka ʻikepili.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Aia kekahi SetData. Eia mākou e hala i ka mana. A inā mākou e hāʻawi i kēia, e hoʻonui ʻia ka ʻikepili ma ka znode o kekahi mana.

Hiki iā ʻoe ke kuhikuhi i ka "-1" no ka haʻalele ʻana i kēia māka.

ʻO kekahi ala kūpono loaaKeiki. Hiki iā mākou ke loaʻa i kahi papa inoa o nā znodes āpau nona ia. Hiki iā mākou ke nānā i kēia ma ka hoʻonohonoho ʻana i ka wati hae.

A me ke ʻano lōkahi hiki ke hoʻouna ʻia nā hoʻololi a pau i ka manawa hoʻokahi, no laila e hōʻoiaʻiʻo ʻia e mālama ʻia a hoʻololi ʻia nā ʻikepili āpau.

Inā mākou e kiʻi i nā analogies me ka papahana maʻamau, a laila ke hoʻohana ʻoe i nā ʻano e like me ke kākau ʻana, e kākau i kekahi mea i ka disk, a ma hope o ka hoʻihoʻi ʻana i kahi pane iā ʻoe, ʻaʻohe mea hōʻoia ua kākau ʻoe i ka ʻikepili i ka disk. A ʻoiai ke hilinaʻi nei ka ʻōnaehana hana ua kākau ʻia nā mea āpau, aia nā mīkini i loko o ka disk ponoʻī kahi e hele ai ke kaʻina hana ma nā papa o nā buffers, a ma hope wale nō e kau ʻia ka ʻikepili ma ka disk.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Hoʻohana pinepine ʻia nā kelepona asynchronous. ʻAe kēia i ka mea kūʻai aku e hana like me nā noi like ʻole. Hiki iā ʻoe ke hoʻohana i ke ala synchronous, akā ʻoi aku ka liʻiliʻi.

ʻO nā hana ʻelua a mākou i kamaʻilio ai ʻo ia ka hoʻopou/kākau, e hoʻololi i ka ʻikepili. ʻO kēia nā mea hana, setData, sync, holoi. A aia ka heluhelu, getData, getChildren.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

I kēia manawa he mau hiʻohiʻona pehea e hiki ai iā ʻoe ke hana i nā primitives no ka hana ʻana i kahi ʻōnaehana puʻupuʻu. No ka laʻana, pili i ka hoʻonohonoho ʻana o kekahi mea. Ua puka mai kekahi mea hana hou. Hoʻohui mākou i ka mīkini a hoʻomaka i ka hana. A eia nā nīnau ʻekolu. Pehea e nīnau ai iā ZooKeeper no ka hoʻonohonoho ʻana? A inā makemake mākou e hoʻololi i ka hoʻonohonoho, pehea mākou e hoʻololi ai? A ma hope o ko mākou hoʻololi ʻana, pehea e loaʻa ai i nā limahana i loaʻa iā mākou?

Hana maʻalahi kēia ZooKeeper. Eia kekahi laʻana, aia kā mākou lāʻau znode. Aia kahi node no kā mākou noi ma aneʻi, hana mākou i kahi node hou i loko, aia ka ʻikepili mai ka hoʻonohonoho. He mau palena kaawale paha keia. No ka mea liʻiliʻi ka nui, liʻiliʻi hoʻi ka nui hoʻonohonoho, no laila hiki ke mālama ʻia ma aneʻi.

Ke hoʻohana nei ʻoe i ke ʻano getData e kiʻi i ka hoʻonohonoho no ka mea hana mai ka node. Hoʻonoho i ka ʻoiaʻiʻo. Inā no kekahi kumu ʻaʻole i loaʻa kēia node, e hoʻomaopopo ʻia iā mākou i ka wā e ʻike ʻia ai, a i ʻole ke loli. Inā makemake mākou e ʻike ua loli kekahi mea, a laila hoʻonohonoho mākou i ka ʻoiaʻiʻo. A inā hoʻololi ka ʻikepili i kēia node, e ʻike mākou e pili ana.

SetData. Hoʻonohonoho mākou i ka ʻikepili, hoʻonohonoho i ka "-1", ʻo ia hoʻi, ʻaʻole mākou e nānā i ka mana, manaʻo mākou he hoʻokahi hoʻonohonoho mau mākou, ʻaʻole pono mākou e mālama i nā hoʻonohonoho he nui. Inā pono ʻoe e mālama i ka nui, pono ʻoe e hoʻohui i kahi pae ʻē aʻe. Maʻaneʻi mākou manaʻoʻiʻo hoʻokahi wale nō, no laila mākou e hōʻano hou i ka mea hou loa, no laila ʻaʻole mākou e nānā i ka mana. I kēia manawa, loaʻa i nā mea kūʻai aku a pau i kau inoa ma mua i ka hoʻolaha ʻana ua loli kekahi mea i kēia node. A ma hope o ka loaʻa ʻana, pono lākou e noi hou i ka ʻikepili. ʻO ka hoʻolaha ʻana ʻaʻole lākou e loaʻa i ka ʻikepili ponoʻī, akā hoʻolaha wale i nā loli. Ma hope o kēia, pono lākou e noi i nā ʻikepili hou.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO ka lua o ka koho no ka hoʻohana ʻana i ka primitive hui hui. Loaʻa iā mākou kahi noi i hoʻolaha ʻia, aia kahi pūʻulu o nā limahana a makemake mākou e hoʻomaopopo aia lākou a pau. No laila, pono lākou e hoʻopaʻa inoa iā lākou iho e hana lākou i kā mākou noi. A makemake pū mākou e ʻike, mai ke kaʻina Master a i ʻole ma kahi ʻē aʻe, e pili ana i nā limahana hana a pau i loaʻa iā mākou i kēia manawa.

Pehea mākou e hana ai i kēia? No ka noi, hana mākou i kahi node limahana a hoʻohui i kahi sublevel ma laila me ka hoʻohana ʻana i ke ʻano hana. Loaʻa iaʻu kahi hewa ma ka paheʻe. Maanei ʻoe e pono ai kaʻina kau kikoʻī, a laila e hana ʻia nā mea hana a pau. A ʻo ka noi, e noi ana i nā ʻikepili āpau e pili ana i nā keiki o kēia node, loaʻa i nā limahana hana āpau e noho nei.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

He hoʻokō weliweli kēia o ka hana ʻana i kēia ma ka code Java. E hoʻomaka kākou mai ka hopena, me ke ʻano nui. ʻO kā mākou papa kēia, e hana kākou i kona ʻano. E like me ka hoʻopaʻapaʻa mua mākou e hoʻohana ai i ka host, kahi a mākou e hoʻopili ai, ʻo ia hoʻi mākou i hoʻonohonoho ai i hoʻopaʻapaʻa. A ʻo ka hoʻopaʻapaʻa ʻelua ka inoa o ka hui.

Pehea ka pili ana? He laʻana maʻalahi kēia o ka API i hoʻohana ʻia. He mea maʻalahi nā mea a pau maʻaneʻi. Aia kahi ZooKeeper papa maʻamau. Hāʻawi mākou i nā mea hoʻokipa iā ia. A hoʻonohonoho i ka manawa hoʻopau, no ka laʻana, i 5 kekona. A loaʻa iā mākou kahi lālā i kapa ʻia ʻo connectedSignal. ʻO ka mea nui, hana mākou i kahi hui ma ke ala i hoʻouna ʻia. ʻAʻole mākou kākau ʻikepili ma laila, ʻoiai ua hiki ke kākau ʻia kekahi mea. A ʻo ka node ma ʻaneʻi o ke ʻano hoʻomau. ʻO ka mea nui, he node maʻamau maʻamau kēia e noho mau i nā manawa āpau. ʻO kēia kahi i hana ʻia ai ke kau. ʻO kēia ka hoʻokō ʻana o ka mea kūʻai aku ponoʻī. E hoʻouna kā mākou mea kūʻai aku i nā memo i kēlā me kēia manawa e hōʻike ana ke ola nei ke kau. A ke hoʻopau mākou i ke kau, kāhea mākou i kahi kokoke a ʻo ia, hāʻule ke kau. ʻO kēia inā hāʻule kekahi mea iā mākou, no laila e ʻike ʻo ZooKeeper e pili ana iā ia a ʻoki i ke kau.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Pehea e laka ai i kahi punawai? Maʻaneʻi ua paʻakikī iki nā mea a pau. Loaʻa iā mākou kahi poʻe limahana, aia kekahi mau kumuwaiwai a mākou e makemake ai e laka. No ka hana ʻana i kēia, hana mākou i kahi node ʻokoʻa, no ka laʻana, i kapa ʻia ʻo lock1. Inā hiki iā mākou ke hana, a laila loaʻa iā mākou kahi laka ma ʻaneʻi. A inā ʻaʻole hiki iā mākou ke hana, a laila hoʻāʻo ka mea hana e kiʻi iā getData mai ʻaneʻi, a ʻoiai ua hana ʻia ka node, a laila kau mākou i kahi kiaʻi ma ʻaneʻi a i ka manawa e loli ai ke kūlana o kēia node, e ʻike mākou e pili ana iā ia. A hiki iā mākou ke hoʻāʻo e loaʻa ka manawa e hana hou ai. Inā mākou i lawe i kēia node, lawe i kēia laka, a laila ma hope o kā mākou pono ʻole i ka laka, e haʻalele mākou iā ia, no ka mea aia wale nō ka node i loko o ke kau. No laila, e nalowale ana. A ʻo kekahi mea kūʻai aku, i loko o ke ʻano o kahi hālāwai ʻē aʻe, hiki iā ia ke lawe i ka laka ma kēia node, a i ʻole, e loaʻa iā ia kahi leka hoʻomaopopo ua loli kekahi mea a hiki iā ia ke hoʻāʻo e hana i ka manawa.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO kekahi hiʻohiʻona o ke koho ʻana i ke alakaʻi nui. ʻOi aku ka paʻakikī o kēia, akā maʻalahi hoʻi. He aha ka hana ma ʻaneʻi? Loaʻa kahi node nui e hōʻuluʻulu i nā mea hana a pau. Ke hoʻāʻo nei mākou e kiʻi i ka ʻikepili e pili ana i ke alakaʻi. Inā holomua kēia, ʻo ia hoʻi, ua loaʻa iā mākou kekahi ʻikepili, a laila hoʻomaka kā mākou limahana e hahai i kēia alakaʻi. Manaʻo ʻo ia aia kekahi alakaʻi.

Inā make ke alakaʻi no kekahi kumu, no ka laʻana, hāʻule, a laila hoʻāʻo mākou e hana i alakaʻi hou. A inā lanakila mākou, a laila lilo kā mākou limahana i alakaʻi. A inā hiki i kekahi i kēia manawa ke hana i alakaʻi hou, a laila e hoʻāʻo mākou e hoʻomaopopo ʻo wai ia a hahai iā ia.

Eia ka mea i kapaia herd effect, oia hoi ka herd effect, no ka mea, i ka make ana o ke alakai, e lilo ana ka mea mua i ka manawa i alakai.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

I ka hopu ʻana i kahi kumuwaiwai, hiki iā ʻoe ke hoʻāʻo e hoʻohana i kahi ala ʻokoʻa, ʻo ia hoʻi. No ka laʻana, makemake mākou e kiʻi i kahi laka, akā me ka ʻole o ka hopena hert. Loaʻa ia i ka ʻoiaʻiʻo e noi ana kā mākou noi i nā papa inoa o nā id node āpau no kahi node i loaʻa me kahi laka. A inā ma mua o ka node a mākou i hana ai i kahi laka, ʻo ia ka liʻiliʻi loa o ka hoʻonohonoho i loaʻa iā mākou, ʻo ia hoʻi, ua hopu mākou i ka laka. Nānā mākou ua loaʻa iā mākou kahi laka. Ma ke ʻano he hōʻoia, e loaʻa kahi kūlana e liʻiliʻi ka id i loaʻa iā mākou i ka wā e hana ai i kahi laka hou. A inā loaʻa iā mākou, a laila hana hou mākou.

Inā loaʻa kekahi id i ʻoi aku ka liʻiliʻi ma mua o kā mākou laka, a laila kau mākou i kahi kiaʻi ma kēia hanana a kali i ka hoʻolaha a hiki i ka hoʻololi ʻana o kekahi mea. ʻO ia hoʻi, ua loaʻa iā mākou kēia laka. A hiki i ka hāʻule ʻana, ʻaʻole mākou e lilo i ka id liʻiliʻi a ʻaʻole e loaʻa ka laka liʻiliʻi, a pēlā mākou e hiki ai ke komo. A inā ʻaʻole i hoʻokō ʻia kēia kūlana, hele koke mākou i ʻaneʻi a hoʻāʻo e kiʻi hou i kēia laka, no ka mea, ua loli paha kekahi mea i kēia manawa.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

He aha ka ZooKeeper? He 4 mau mea nui. ʻO kēia nā kaʻina hana - Noi. A me ZooKeeper Atomic Broadcast. Aia kahi Commit Log kahi i hoʻopaʻa ʻia ai nā hana a pau. A ʻo ka In-memory Replicated DB ponoʻī, ʻo ia hoʻi ka ʻikepili ponoʻī kahi i mālama ʻia ai kēia lāʻau holoʻokoʻa.

Pono e hoʻomaopopo ʻia e hele nā ​​hana kākau āpau ma o ka Process Processor. A hele pololei nā hana heluhelu i ka waihona In-memory.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Hoʻopili piha ʻia ka ʻikepili ponoʻī. ʻO nā manawa a pau o ZooKeeper e mālama i kahi kope piha o ka ʻikepili.

No ka hoʻihoʻi hou ʻana i ka waihona ma hope o kahi ulia, aia kahi log Commit. ʻO ka hana maʻamau, ʻo ia ma mua o ka komo ʻana o ka ʻikepili i ka hoʻomanaʻo, ua kākau ʻia ma laila inā e hāʻule, hiki ke hoʻokani ʻia kēia log a hiki ke hoʻihoʻi ʻia ke kūlana ʻōnaehana. A hoʻohana pū ʻia nā kiʻi paʻi kiʻi o ka waihona.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

ʻO ZooKeeper Atomic Broadcast kahi mea i hoʻohana ʻia no ka mālama ʻana i ka ʻikepili hou.

Koho ʻo ZAB i kahi alakaʻi mai ka manaʻo o ka node ZooKeeper. ʻO nā node ʻē aʻe e lilo i mau ukali a manaʻo i kekahi mau hana mai ia. Inā loaʻa iā lākou nā palapala, hoʻouna lākou iā lākou a pau i ke alakaʻi. Hana mua ʻo ia i kahi hana kākau a hoʻouna i kahi leka e pili ana i nā mea i hoʻololi ʻia i kāna poʻe hahai. ʻO kēia, ʻoiaʻiʻo, pono e hoʻokō ʻia ma ka atomically, ʻo ia hoʻi ka hana hoʻopaʻa ʻana a me ka hoʻolaha ʻana o ka mea holoʻokoʻa pono e hana ʻia ma ka atomically, a laila e hōʻoiaʻiʻo ai i ka kūlike o ka ʻikepili.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop" Hoʻoponopono wale ia i nā noi kākau. ʻO kāna hana nui ʻo ia ka hoʻololi ʻana i ka hana i kahi hoʻolaha transactional. He noi hana kūikawā kēia.

A eia he mea pono e hoʻomaopopo i ka idempotency o nā mea hou no ka hana like. He aha ia? ʻO kēia mea, inā e hoʻokō ʻia ʻelua, e loaʻa ka mokuʻāina like, ʻo ia hoʻi, ʻaʻole e loli ka noi ponoʻī. A pono e hana ʻia kēia i ka wā e hāʻule ai, hiki iā ʻoe ke hoʻomaka hou i ka hana, a laila e hoʻohuli i nā loli i hāʻule i kēia manawa. I kēia hihia, e lilo ka moku'āina o ka ʻōnaehana i mea like, ʻo ia hoʻi, ʻaʻole pono ke ʻano o ke ʻano o nā mea like, no ka laʻana, nā kaʻina hana hou, alakaʻi i nā mokuʻāina hope loa o ka ʻōnaehana.

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

"Hadoop. ZooKeeper" mai ka Mail.Ru Group Technostream series "Methods for distributed processing of large volume of data in Hadoop"

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka