Sakin dandamali don sarrafa bayanai da aka rarraba Apache Hadoop 3.3

Bayan shekara daya da rabi na ci gaba, Apache Software Foundation aka buga saki Apache Hadoop 3.3.0, dandali na kyauta don shirya rarraba rarraba na manyan kundin bayanai ta amfani da tsarin taswira/rage, wanda a cikinsa aka raba aikin zuwa ƙananan ɓangarorin daban-daban, kowannensu ana iya ƙaddamar da su akan kullin gungu daban-daban. Ma'ajiyar tushen Hadoop na iya wuce dubunnan nodes kuma ya ƙunshi bayanan exabyte.

Hadoop ya haɗa da aiwatar da Hadoop Distributed Filesystem (HDFS), wanda ke ba da ajiyar bayanai ta atomatik kuma an inganta shi don aikace-aikacen MapReduce. Don sauƙaƙa samun damar samun bayanai a cikin ma’adanar Hadoop, an ƙirƙiro bayanan HBase da kuma harshen Pig kamar SQL, wanda wani nau’in SQL ne don MapReduce, tambayoyin da za a iya daidaita su da sarrafa su ta hanyar dandamali na Hadoop da yawa. An kiyasta aikin a matsayin cikakken kwanciyar hankali kuma yana shirye don aikin masana'antu. Ana amfani da Hadoop sosai a cikin manyan ayyukan masana'antu, yana ba da damar kama da dandalin Google Bigtable/GFS/MapReduce, yayin da Google ke da hukuma a hukumance. wakilai Hadoop da sauran ayyukan Apache suna da hakkin yin amfani da fasahohin da ke tattare da haƙƙin mallaka masu alaƙa da hanyar MapReduce.

Hadoop yana matsayi na farko a cikin ma'ajiyar Apache dangane da adadin canje-canjen da aka yi kuma na biyar dangane da girman codebase (kimanin layukan lamba miliyan 4). Manyan Hadoop aiwatarwa sun hada da Netflix (fiye da abubuwan da suka faru biliyan 500 a kowace rana ana adana su), Twitter (gungu na nodes dubu 10 yana adana fiye da zettabyte na bayanai a cikin ainihin lokacin da aiwatar da fiye da zaman biliyan 5 a kowace rana), Facebook (gungu) na 4 dubu nodes Stores fiye da 300 petabytes kuma yana karuwa kowace rana ta 4 PB kowace rana).

Main canji a cikin Apache Hadoop 3.3:

• Ƙara goyon baya don dandamali dangane da gine-ginen ARM.
• Aiwatar da tsarin Protobuf (Masu buffers na yarjejeniya), da aka yi amfani da su don tsara bayanan da aka tsara, an sabunta su don sakin 3.7.1 saboda ƙarshen tsarin rayuwa na reshen protobuf-2.5.0.
• An faɗaɗa ƙarfin mai haɗin S3A: an ƙara goyan bayan tabbatarwa ta amfani da alamun (Alamar wakilci), ingantacciyar tallafi don amsawar caching tare da lambar 404, haɓaka aikin S3guard, da haɓaka amincin aiki.
• An warware matsalolin daidaitawa ta atomatik a cikin tsarin fayil na ABFS.
• Ƙara tallafin ɗan ƙasa don tsarin fayil na Tencent Cloud COS don samun dama ga ma'ajin abu na COS.
• Ƙara cikakken tallafi don Java 11.
• An daidaita aiwatar da HDFS RBF (Federation-based Federation). An ƙara sarrafa tsaro zuwa HDFS Router.
• Ƙara sabis na ƙuduri na DNS don abokin ciniki don ƙayyade sabobin ta hanyar DNS ta sunayen masu watsa shiri, yana ba ku damar yin ba tare da lissafin duk runduna a cikin saitunan ba.
• Ƙara tallafin jadawalin ƙaddamarwa kwantena masu dama ta hanyar mai sarrafa albarkatun ƙasa (ResourceManager), gami da ikon rarraba kwantena la'akari da nauyin kowane kumburi.
• Ƙara YARN (Har yanzu Wani Mai Neman Taimako) kundin adireshin aikace-aikacen.

source: budenet.ru