Mushure megore nehafu yebudiriro, iyo Apache Software Foundation kusunungura , chikuva chemahara chekuronga kugoverwa kugadziridzwa kwemavhoriyamu makuru e data uchishandisa paradigm , umo basa racho rakakamurwa kuita zvidimbu zvidiki zvakawanda zvakaparadzana, chimwe nechimwe chinogona kutangwa pane imwe nhambo yemasumbu. Hadoop-yakavakirwa kuchengetedza inogona kutenderera zviuru zvemanodhi uye ine exabytes yedata.
Hadoop inosanganisira kushandiswa kweHadoop Distributed Filesystem (HDFS), iyo inopa otomatiki kuchengetedza data uye inogadziridzwa kuMepuReduce application. Kurerutsa kuwana data muHadoop chengetedzo, iyo HBase dhatabhesi uye iyo SQL-yakafanana mutauro Nguruve yakagadziridzwa, inova mhando yeSQL yeMepuReduce, iyo mibvunzo inogona kufananidzwa uye kugadziridzwa nemapuratifomu akati wandei eHadoop. Iyo purojekiti inoongororwa seyakagadzikana zvachose uye yakagadzirira kushanda kwemaindasitiri. Hadoop inoshandiswa zvakanyanya mumapurojekiti makuru emaindasitiri, ichipa hunyanzvi hwakafanana neGoogle Bigtable/GFS/MapReduce chikuva, nepo Google iine zviri pamutemo. Hadoop nemamwe mapurojekiti eApache ane kodzero yekushandisa matekinoroji akafukidzwa nematendi ane hukama neMepuReduce nzira.
Hadoop inomira pekutanga pakati peApache repositori maererano nehuwandu hwekuchinja kwakaitwa uye yechishanu maererano nesaizi yecodebase (inenge 4 miriyoni mitsetse yekodhi). Kushandiswa kukuru kweHadoop kunosanganisira Netflix (zvinopfuura 500 bhiriyoni zviitiko pazuva zvakachengetwa), Twitter (boka rezviuru gumi node rinochengeta zvinopfuura zettabyte yedata munguva chaiyo uye inogadzirisa zvinopfuura 10 bhiriyoni zvikamu pazuva), Facebook (boka ye5 zviuru nodes inochengetedza kupfuura 4 petabytes uye iri kuwedzera zuva nezuva ne300 PB pazuva).
chikuru muApache Hadoop 3.3:
- Yakawedzera rutsigiro rwemapuratifomu akavakirwa paArM architecture.
- Kuitwa kwefomati (Protocol buffers), inoshandiswa pakurongedza data yakarongeka, yakagadziridzwa kuburitsa 3.7.1 nekuda kwekupera kwehupenyu hweiyo protobuf-2.5.0 bazi.
- Iko kugona kweS3A yekubatanidza kwakawedzerwa: rutsigiro rwekusimbisa uchishandisa tokens rwakawedzerwa (), yakagadziridzwa tsigiro yemhinduro dzecaching nekodhi 404, yakawedzera S3guard performance, uye yakawedzera kuvimbika kwekushanda.
- Matambudziko neotomatiki tuning akagadziriswa muABFS faira system.
- Yakawedzerwa yerudzi rutsigiro rweTencent Cloud COS faira system yekuwana COS chinhu chekuchengetedza.
- Yakawedzera rutsigiro rwakazara rweJava 11.
- Kuitwa kweHDFS RBF (Router-based Federation) kwakagadziriswa. Chengetedzo dzinodzora dzakawedzerwa kuHDFS Router.
- Yakawedzera iyo DNS Resolution sevhisi kuti mutengi atarise maseva kuburikidza neDNS nemazita ekugamuchira, zvichikubvumidza kuti uite pasina kunyora ese anotambira muzvirongwa.
- Yakawedzera hurongwa hwekuronga rutsigiro kuburikidza necentralized resource maneja (ResourceManager), kusanganisira kugona kugovera midziyo uchifunga nezvekuremerwa kweimwe node.
- Yakawedzera yekutsvaga YARN (Yet Another Resource Negotiator) application directory.
Source: opennet.ru