Kutulutsidwa kwa nsanja yogawa data processing Apache Hadoop 3.3

Pambuyo pa chaka ndi theka la chitukuko, Apache Software Foundation losindikizidwa kumasula Apache Hadoop 3.3.0, nsanja yaulere yokonzekera kugawira kugawidwa kwa ma data ambiri pogwiritsa ntchito paradigm mapu/chepetsa, momwe ntchitoyi imagawidwa m'zidutswa zing'onozing'ono zosiyana, zomwe zingathe kukhazikitsidwa pamagulu osiyana. Kusungirako kochokera ku hadoop kumatha kupitilira masauzande ambiri ndipo kumakhala ndi ma exabytes a data.

Hadoop imaphatikizapo kukhazikitsidwa kwa Hadoop Distributed Filesystem (HDFS), yomwe imangopereka zosunga zobwezeretsera deta ndipo imakongoletsedwa ndi MapReduce. Kuti muchepetse mwayi wopezeka muzosungirako za Hadoop, database ya HBase ndi Nkhumba yofanana ndi SQL yapangidwa, yomwe ndi mtundu wa SQL ya MapReduce, mafunso omwe amatha kufananizidwa ndikusinthidwa ndi nsanja zingapo za Hadoop. Ntchitoyi imayesedwa kuti ndi yokhazikika komanso yokonzeka kugwira ntchito m'mafakitale. Hadoop imagwiritsidwa ntchito mwachangu m'mapulojekiti akuluakulu amakampani, kupereka mphamvu zofananira ndi nsanja ya Google Bigtable/GFS/MapReduce, pomwe Google idachita mwalamulo. operekedwa Hadoop ndi ma projekiti ena a Apache ali ndi ufulu wogwiritsa ntchito matekinoloje omwe ali ndi ma patent okhudzana ndi njira ya MapReduce.

Hadoop ndiye woyamba pakati pa nkhokwe za Apache potengera kuchuluka kwa zosintha zomwe zidachitika komanso lachisanu potengera kukula kwa codebase (pafupifupi mizere 4 miliyoni yamakhodi). Kukhazikitsa kwakukulu kwa Hadoop kumaphatikizapo Netflix (zopitilira mabiliyoni a 500 patsiku zimasungidwa), Twitter (gulu la node 10 zikwizikwi limasunga zambiri kuposa zettabyte ya data munthawi yeniyeni ndikuchita magawo opitilira 5 biliyoni patsiku), Facebook (gulu za 4 zikwi nodes zimasunga kuposa ma petabytes a 300 ndipo zikuwonjezeka tsiku ndi tsiku ndi 4 PB patsiku).

waukulu kusintha Apache Hadoop 3.3:

• Thandizo lowonjezera pamapulatifomu otengera kamangidwe ka ARM.
• Kukhazikitsa mawonekedwe Protobuf (Protocol buffers), yomwe imagwiritsidwa ntchito popanga deta yokhazikika, yasinthidwa kuti itulutse 3.7.1 chifukwa cha kutha kwa moyo wa nthambi ya protobuf-2.5.0.
• Kuthekera kwa cholumikizira cha S3A kwakulitsidwa: kuthandizira kutsimikizika pogwiritsa ntchito zizindikiro zawonjezedwa (Chizindikiro cha Delegation), kuthandizira bwino kwa mayankho a caching ndi code 404, kuwonjezeka kwa S3guard, ndi kudalirika kwa ntchito.
• Mavuto akusintha basi adathetsedwa mu fayilo ya ABFS.
• Thandizo lachilengedwe la Tencent Cloud COS fayilo yamafayilo kuti mupeze kusungirako zinthu za COS.
• Adawonjezera chithandizo chonse cha Java 11.
• Kukhazikitsa kwa HDFS RBF (Router-based Federation) kwakhazikika. Zowongolera zachitetezo zawonjezedwa ku HDFS Router.
• Onjezani ntchito ya DNS Resolution kuti kasitomala adziwe ma seva kudzera pa DNS ndi mayina olandila, zomwe zimakupatsani mwayi wochita osalemba mndandanda wamakasitomala onse.
• Thandizo lowonjezera loyambira zotengera mwayi kudzera mwa woyang'anira gwero lapakati (ResourceManager), kuphatikiza kuthekera kogawa zotengera poganizira kuchuluka kwa node iliyonse.
• Yowonjezera yosaka ya YARN (Yet Another Resource Negotiator) chikwatu.

Source: opennet.ru