Malaha wax sir ah maaha in sannadkii hore uu ahaa sannad isbeddello waaweyn Apache Hadoop. Sannadkii hore, Cloudera iyo Hortonworks ayaa midoobay (asal ahaan, la wareegida kan dambe), iyo Mapr, dhibaatooyin dhaqaale oo daran dartood, waxaa laga iibiyay Hewlett Packard. Oo haddii dhowr sano ka hor, kiiska rakibaadda dhismaha, doorashada inta badan waxay ahayd in la sameeyo inta u dhaxaysa Cloudera iyo Hortonworks, maanta, hoogay, ma haysanno doorashadan. Arrinta kale ee la yaabka leh waxay ahayd xaqiiqda ah in Cloudera ay ku dhawaaqday bishii Febraayo ee sanadkan inay joojinayso sii deynta shirarka binary ee qaybinta kaydinta dadweynaha, oo hadda waxaa la heli karaa oo keliya iyada oo loo marayo isdiiwaangelinta lacag bixinta. Dabcan, weli waa suurtogal in la soo dejiyo noocyadii ugu dambeeyay ee CDH iyo HDP ee la sii daayay ka hor dhammaadka 2019, waxaana la filayaa in taageerada iyaga la siiyo hal ilaa laba sano. Laakiin maxaa la sameeyaa marka xigta? Kuwii hore u bixiyay is-diiwaangelinta, waxba iskamay beddelin. Iyo kuwa aan rabin inay u beddelaan qaybta lacagta ah ee qaybinta, laakiin isla mar ahaantaana raba inay awoodaan inay helaan noocyadii ugu dambeeyay ee qaybaha kooxda, iyo sidoo kale dhejisyo iyo cusbooneysiin kale, waxaanu diyaarinay maqaalkan. Halkaas waxaan kaga fiirsan doonaa fursadaha suurtagalka ah ee looga bixi karo xaaladdan.
Maqaalku waa dib u eegis badan. Kuma jiri doono isbarbardhigga qaybinta iyo falanqaynta faahfaahsan ee iyaga, mana jiri doonto cuntooyin lagu rakibo iyo habayntooda. Maxaa dhici doona? Waxaan si kooban uga hadli doonaa qaybinta noocan oo kale ah sida Arenadata Hadoop, taas oo si sax ah u mudan dareenkayaga sababtoo ah helitaankeeda, taas oo aad dhif u ah maanta. Ka dibna waxaan ka hadli doonaa Vanilla Hadoop, badiyaa ku saabsan sida loo "karin karo" iyadoo la isticmaalayo Apache Bigtop. Diyaar? Markaa ku soo dhawoow bisad.
Arenadata Hadoop
Kani waa qalab cusub oo ilaa hadda aan la aqoon oo qaybinta horumarinta gudaha. Nasiib darro, waqtigan la joogo HabrΓ© waxaa jira oo keliya .
Macluumaad dheeraad ah ayaa laga heli karaa rasmiga ah mashruuc. Noocyadii ugu dambeeyay ee qaybinta waxay ku saleysan yihiin Hadoop 3.1.2 ee nooca 3, iyo 2.8.5 ee nooca 2.
Macluumaadka ku saabsan khariidadda tubta waa la heli karaa .
Interface Maareeyaha Kooxda Arenadata
Arenadata wax soo saarkeeda xudunta u ah waa , kaas oo loo isticmaalo in lagu rakibo, habeeyo oo la socdo xalalka software ee shirkadda ee kala duwan. ADCM waxa loo qaybiyaa lacag la'aan, shaqadeedana waxa lagu kordhiyaa xidhmooyin, kuwaas oo ah buug-ciyaareed-ciyaareed oo la sugi karo. Xirmooyinka waxaa loo qaybiyaa laba nooc: ganacsi iyo bulsho. Kuwa dambe waxay diyaar u yihiin soo dejin bilaash ah oo laga helo degelka Arenadata. Waxa kale oo suurtagal ah in aad horumariso xidhmadaada oo aad ku xidho ADCM.
Soo dirida iyo maamulka Hadoop 3, nooca bulshada ee xidhmada ayaa lala bixiyaa ADCM, laakiin Hadoop 2 waxaa jira oo kaliya bedel ahaan. Xagga kaydinta baakadaha, waxay u furan yihiin gelitaanka dadweynaha, waa la soo dejin karaa oo lagu rakibi karaa habka caadiga ah ee dhammaan qaybaha kooxda. Guud ahaan, qaybinta ayaa u muuqata mid aad u xiiso badan. Waxaan hubaa inay jiri doonaan kuwa caadaystay xalalka sida Maamulaha Cloudera iyo Ambari, oo jeclaan doona ADCM laftiisa. Qaar ka mid ah, waxay sidoo kale u noqon doontaa wax weyn oo lagu daray qaybinta beddelka soo dejinta.
Haddii aan ka hadalno khasaarooyinka, waxay la mid noqon doonaan dhammaan qaybinta Hadoop kale. Kuwaas oo kala ah:
- Waxa loogu yeero "qufulka iibiyaha". Isticmaalka tusaalooyinka Cloudera iyo Hortonworks, waxaan horeyba u ogaanay in mar walba ay jirto khatar ah in la beddelo siyaasadda shirkadda.
- Dib u dhac weyn oo ka dambeeya Apache korka.
Vanilla Hadoop
Sidaad ogtahay, Hadoop maaha wax soo saar keli ah, laakiin, dhab ahaantii, adeeg galaxyo dhan oo ku wareegsan nidaamka faylka la qaybiyey ee HDFS. Dad yar ayaa heli doona ku filan hal koox oo fayl ah. Qaar waxay u baahan yihiin Hive, kuwa kale Presto, ka dibna waxaa jira HBase iyo Phoenix; Spark si sii kordheysa ayaa loo isticmaalaa. Wixii abaabulka iyo rarka xogta, Oozie, Sqoop iyo Flume ayaa mararka qaarkood la helaa. Oo haddii arrinta ammaanku soo baxdo, markaas Kerberos oo lala xiriirinayo Ranger isla markiiba maskaxda ku hay.
Noocyada binary ee qaybaha Hadoop ayaa laga heli karaa mareegaha mid kasta oo ka mid ah mashaariicda deegaanka ee qaabka daamurka. Waad soo dejisan kartaa oo bilaabi kartaa rakibaadda, laakiin hal shardi: marka lagu daro inaad si madax-bannaan u ururiso baakadaha "binary" cayriin, oo aad u badan tahay inaad rabto inaad sameyso, wax kalsooni ah kuma yeelan doontid waafaqid qaybaha la soo dejiyey ee qaybaha mid kasta kale. Doorashada la door biday waa in la dhiso iyadoo la isticmaalayo Apache Bigtop. Bigtop wuxuu kuu ogolaanayaa inaad ka dhisto kaydadka Apache maven, tijaabiyo oo aad dhisto baakado. Laakiin, waxa aadka muhiimka noogu ah, Bigtop wuxuu soo ururin doonaa noocyada qaybaha kuwaas oo midba midka kale ku socon doona. Waxaan uga hadli doonaa si faahfaahsan hoos.
Apache Bigtop
Apache Bigtop waa qalab lagu dhiso, baakadaha iyo tijaabinta tiro ka mid ah
Mashaariic il furan, sida Hadoop iyo Greenplum. Bigtop wax badan buu leeyahay
sii daayo. Waqtiga qorista, sii dayntii ugu dambeysay ee xasillooni waxay ahayd nooca 1.4,
Master-ka waxaa ku jiray 1.5. Noocyada kala duwan ee sii deynta waxay isticmaalaan noocyo kala duwan
qaybaha. Tusaale ahaan, qaybaha 1.4 Hadoop core waxay leeyihiin nooca 2.8.5, iyo Master
2.10.0. Halabuurka qaybaha la taageeray ayaa sidoo kale isbeddelaya. Wax duugoobay iyo
Midda aan la cusboonaysiin karin ayaa meesha ka baxda, oo meesheeda wax cusub ayaa ka soo baxa, oo baahi badan loo qabo, iyo
daruuri maaha wax ka yimid qoyska Apache laftiisa.
Intaa waxaa dheer, Bigtop wuxuu leeyahay qaar badan .
Markii aan bilownay inaan barano Bigtop, waxaan ugu horreyntii la yaabnay dhammaantiis, marka la barbardhigo mashaariicda kale ee Apache, baahsanaanta iyo caannimada, iyo sidoo kale bulsho aad u yar. Taas waxa ay raacdaa in ay jirto macluumaadka ugu yar ee alaabta, iyo raadinta xalalka dhibaatooyinka ka dhashay forumyada iyo liisaska boostada laga yaabo in aanay waxba soo saarin gabi ahaanba. Ugu horreyntii, waxay noo noqotay hawl adag si aan u dhamaystirno isku-dhafka dhamaystiran ee qaybinta sababtoo ah sifooyinka qalabka laftiisa, laakiin waxaan ka hadli doonaa arrintan wax yar ka dib.
Sida teaser, kuwa hal mar xiisaynayay mashaariicda noocan oo kale ah ee caalamka Linux sida Gentoo iyo LFS waxaa laga yaabaa inay u arkaan inay aad ugu faraxsan tahay inay la shaqeeyaan shaydaan oo ay xasuustaan ββββwakhtiyada "epic" markii aan raadinaynay (ama xitaa qorista) dhistay oo si joogto ah u dhisay Mozilla oo leh balastar cusub.
Faa'iidada ugu weyn ee Bigtop waa furfurnaanta iyo kala duwanaanta qalabka ay ku saleysan tahay. Waxay ku salaysan tahay Gradle iyo Apache Maven. Gradle waxa aad loogu yaqaanaa aaladda Google u adeegsato dhisidda Android. Waa dabacsanaan, iyo, sida ay yiraahdaan, "dagaal la tijaabiyey." Maven waa aalad caadi ah oo loogu talagalay dhismaha mashaariicda Apache lafteeda, iyo maadaama inta badan alaabteeda lagu sii daayo Maven, lama samayn karo la'aanteed halkan midkoodna. Waxaa habboon in fiiro gaar ah loo yeesho POM (qaabka shayga mashruuca) - faylka xml "asaasiga ah" oo qeexaya wax kasta oo lagama maarmaanka u ah Maven si uu ula shaqeeyo mashruucaaga, kaas oo dhammaan shaqada lagu dhisay. Dhab ahaan at
qaybo ka mid ah Maven waxaana jira caqabado qaar oo markii ugu horeysay isticmaalayaasha Bigtop caadi ahaan la kulmaan.
Tababarka
Hadaba halkee ka bilaabataa? Tag bogga soo dejinta oo soo deji nuqulkii ugu dambeeyay ee xasilloon sidii kayd ahaan. Waxa kale oo aad ka heli kartaa artifacts binary oo ay ururisay Bigtop halkaas. Jid ahaan, ka mid ah maareeyayaasha xirmooyinka guud, YUM iyo APT waa la taageeraa.
Haddii kale, waxaad si toos ah uga soo dejisan kartaa sii dayntii ugu dambaysay
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitIsku xidhka "weyn"...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΎΠ±ΡΠ΅ΠΊΡΠΎΠ²: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, Π³ΠΎΡΠΎΠ²ΠΎ.
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ: 100% (20503/20503), Π³ΠΎΡΠΎΠ²ΠΎ.
Updating files: 100% (1998/1998), Π³ΠΎΡΠΎΠ²ΠΎ.Hagaha ./bigtop natiijadu waxay u egtahay sidan:
./bigtop-bigpetstore - codsiyada demo, tusaaleyaal synthetic
./bigtop-ci - Qalabka CI, jenkins
./bigtop-data-generators - abuurista xogta, synthetics, baaritaanka qiiqa, iwm.
./bigtop-deploy - qalab geynta
./bigtop-packages - habayn, qoraallo, balastar kulan, qaybta ugu muhiimsan ee qalabka
./bigtop-test-framework - qaabka tijaabada
./bigtop-tests - imtixaanada laftooda, rarka iyo qiiqa
./bigtop_toolchain - bay'ada isu-ururinta, u diyaarinta deegaanka si uu qalabku u shaqeeyo
./build - dhis hagaha shaqada
./dl - tusaha ilaha la soo dejiyay
./docker - dhisitaanka sawirada docker, tijaabinta
./gradle - qaabeynta gradle
./output - tusaha halka dhismuhu ku socdo
./provisioner - bixinta
Waxa ugu xiisaha badan ee noo marxaladan waa habka ugu muhiimsan ./bigtop/bigtop.bom, kaas oo aan ku aragno dhammaan qaybaha la taageeray oo leh noocyo. Tani waa meesha aan ku qeexi karno nooc ka duwan badeecada (haddii aan si lama filaan ah u rabno inaan isku dayno inaan dhisno) ama qaab dhismeed (haddii, tusaale ahaan, aan ku darnay balastar muhiim ah).
Buug-hoosaadku sidoo kale waa mid xiiso weyn leh ./bigtop/bigtop-packages, kaas oo si toos ah ula xidhiidha habka isku-dubbaridka qaybaha iyo baakadaha iyaga.
Marka, waxaan soo dejinay kaydka, ka furnay ama ka samaynay clone github, ma bilaabi karnaa dhismo?
Maya, aynu marka hore deegaanka u diyaarino.
Diyaarinta Deegaanka
Oo halkan waxaan u baahanahay dib u gurasho yar. Si aad u dhisto wax kasta oo ka sii adag ama ka yar, waxaad u baahan tahay deegaan gaar ah - kiiskeena, tani waa JDK, maktabadaha la wadaago, faylasha madaxa, iwm, qalab, tusaale ahaan, quraanjada, ivy2 iyo wax ka badan. Mid ka mid ah fursadaha aad ku heli karto deegaanka aad uga baahan tahay Bigtop waa inaad ku rakibto qaybaha lagama maarmaanka ah ee dhismaha martida loo yahay. Waan ku khaldami karaa taariikhda, laakiin waxay u muuqataa in nooca 1.0 ay sidoo kale jirtay ikhtiyaar lagu dhisayo sawirada Docker ee horay loo habeeyey oo la heli karo, kaas oo laga heli karo halkan.
Sida diyaarinta deegaanka, waxaa jira kaaliyaha tan - Puppet.
Waxaad isticmaali kartaa amarrada soo socda, oo ka socda tusaha xididka
qalab, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Ama si toos ah iyada oo loo marayo puppet:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Nasiib darro, dhibaatooyin ayaa laga yaabaa inay soo baxaan mar hore marxaladdan. Talada guud ee halkan waa in la isticmaalo qaybin la taageeray, oo ku cusboonaatay dhisaha martida loo yahay, ama la tijaabiyo jidka xidhitaanka.
Golaha
Maxaan isku dayi karnaa inaan ururino? Jawaabta su'aashan waxaa lagu siin doonaa soo saarida amarka
./gradlew tasks
Qaybta Hawlaha Xidhmada waxa ku jira tiro badeecooyin ah kuwaas oo ah alaabtii ugu dambaysay ee Bigtop.
Waxaa lagu aqoonsan karaa daba-galka -rpm ama -pkg-ind (haddii dhismaha
in docker). Xaaladeena, kan ugu xiisaha badan waa Hadoop.
Aan isku dayno inaan ku dhisno deegaanka dhisme serverkeena:
./gradlew hadoop-rpmBigtop lafteedu waxay soo dejisan doontaa ilaha lagama maarmaanka u ah qayb gaar ah waxayna bilaabi doontaa isu imaatinka. Markaa, hawlgalka qalabku wuxuu ku xidhan yahay kaydadka Maven iyo ilo kale, taas oo ah, waxay u baahan tahay marin internet.
Inta lagu jiro hawlgalka, wax soo saarka caadiga ah ayaa la soo saaray. Mararka qaarkood farriimaha khaldan waxay kaa caawin karaan inaad fahamto waxa qaldamay. Mararka qaarkoodna waxaad u baahan tahay inaad hesho macluumaad dheeraad ah. Xaaladdan oo kale waxaa habboon in lagu daro doodaha --info ama --debug, iyo sidoo kale laga yaabaa inay faa'iido leh βstacktrace. Waxaa jira hab ku habboon oo lagu soo saari karo xogta loo dejiyay gelitaanka xiga ee liisaska boostada, furaha --scan.
Caawinteeda, bigtop ayaa ururin doonta dhammaan macluumaadka oo ku dhejin doonta gradle, ka dib waxay bixin doontaa isku xirka,
Marka la raaco taas, qof karti u leh wuxuu awoodi doonaa inuu fahmo sababta golaha u fashilmay.
Fadlan la soco in doorashadani ay soo bandhigi karto macluumaadka aadan rabin, sida isticmaalayaasha, noodhka, doorsoomayaasha deegaanka, iwm., markaa taxaddar.
Inta badan khaladaadku waa natiijada awood la'aanta in la helo wax kasta oo ka mid ah qaybaha lagama maarmaanka u ah isu-ururinta. Caadi ahaan, waxaad hagaajin kartaa dhibaatada adoo abuuraya balastar si aad u hagaajisid wax ilaha, tusaale ahaan, ciwaanada pom.xml ee tusaha xididka ilaha. Tan waxaa lagu sameeyaa iyadoo la abuurayo oo la gelinayo tusaha ku habboon ./bigtop/bigtop-packages/src/common/oozie/ balastar, tusaale ahaan, qaabka balastar2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
Waxay u badan tahay, waqtiga aad akhrinayso maqaalkan, uma baahnid inaad adigu hagaajiso tan kor ku xusan.
Markaad soo bandhigto balastar kasta iyo isbeddelada habka isu-ururinta, waxaa laga yaabaa inaad u baahato inaad "dib u dejiso" golaha adoo isticmaalaya amarka nadiifinta:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedHawlgalkan ayaa dib u soo celin doona dhammaan isbeddellada ku yimaadda isku dhafka qaybtan, ka dib golaha ayaa la samayn doonaa mar kale. Markan waxaan isku dayi doonaa inaan mashruuca ku dhisno sawir docker:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
ΠΌΠ½ΠΎΠ³ΠΎ Π²ΡΠ²ΠΎΠ΄Π°
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedDhismaha waxaa lagu sameeyay hoosta CentOS, laakiin sidoo kale waxaa lagu samayn karaa hoosta Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indMarka lagu daro dhismaha baakadaha qaybinta Linux ee kala duwan, qalabku wuxuu abuuri karaa bakhaar leh baakado la soo ururiyey, tusaale ahaan:
./gradlew yumWaxa kale oo aad xasuusan kartaa wax ku saabsan tijaabooyinka qiiqa iyo geynta Docker.
Samee koox ka kooban saddex noodood:
./gradlew -Pnum_instances=3 docker-provisionerKu samee tijaabooyinka qiiqa koox ka kooban saddex noodood:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerTirtir koox:
./gradlew docker-provisioner-destroyHel amarrada isku xirka weelasha docker-ka:
./gradlew docker-provisioner-sshMuuji heerka:
./gradlew docker-provisioner-statusWaxaad ka akhrisan kartaa wax badan oo ku saabsan hawlaha Dirista dukumeentiyada.
Haddii aan ka hadalno imtixaanada, waxaa jira tiro aad u badan oo ka mid ah, gaar ahaan qiiqa iyo isdhexgalka. Falanqeyntoodu waa mid ka baxsan baaxadda maqaalkan. Aan idhaahdo ururinta qaybintu maaha hawl adag sida ay u muuqato jaleecada hore. Waxaan ku guuleysanay inaan uruurino oo aan ku gudubno imtixaanno dhammaan qaybaha aan u isticmaalno wax-soo-saarkayaga, sidoo kale wax dhibaato ah nagama qabin geyntooda iyo fulinta hawlgallada aasaasiga ah ee deegaanka imtixaanka.
Marka lagu daro qaybaha jira ee Bigtop, waa suurtagal in lagu daro wax kale, xitaa horumarinta software-kaaga. Waxaas oo dhami si toos ah ayaa otomaatig ah ugu socda waxayna ku habboon yihiin fikradda CI/CD.
gunaanad
Sida cad, qaybinta habkan loo soo ururiyey waa inaan isla markiiba loo dirin wax-soo-saarka. Waxaad u baahan tahay inaad fahamto haddii ay jirto baahi dhab ah oo lagu dhisayo oo lagu taageerayo qaybintaada, markaa waxaad u baahan tahay inaad lacag iyo wakhti ku maalgeliso tan.
Si kastaba ha ahaatee, marka lagu daro habka saxda ah iyo koox xirfad leh, waa suurtogal in la sameeyo iyada oo aan xal ganacsi la helin.
Waxaa muhiim ah in la ogaado in mashruuca Bigtop laftiisa uu u baahan yahay horumar oo uusan u muuqan mid si firfircoon u socda maanta. Sidoo kale ma cadda rajada laga qabo in Hadoop 3 uu ka soo muuqdo. ka Arenadata, kaas oo, marka lagu daro heerka
Waxaa jira tiro ka kooban qaybo dheeraad ah (Ranger, Knox, NiFi).
Dhanka Rostelecom, anaga Bigtop waa mid ka mid ah fursadaha la tixgelinayo maanta. Haddaynu doorano iyo haddii kaleba, wakhti ayaa sheegi doona.
Lifaaqa
Si loogu daro qayb cusub golaha, waxaad u baahan tahay inaad ku darto sharraxaaddeeda bigtop.bom iyo ./bigtop-packages. Waxaad isku dayi kartaa inaad tan sameyso adigoo isbarbar dhigaya qaybaha jira. Isku day inaad ogaato. Ma adka sida ay u muuqato jaleecada hore.
Maxay kula tahay? Waxaan ku farxi doonaa inaan aragno ra'yigaaga faallooyinka waxaanan kuugu mahadcelineynaa dareenkaaga!
Maqaalka waxaa diyaariyay kooxda maamulka xogta ee Rostelecom
Source: www.habr.com