Mhlawumbi akukho mfihlo ukuba unyaka ophelileyo yayingunyaka weenguqu ezinkulu ze-Apache Hadoop. Kunyaka ophelileyo, i-Cloudera kunye neHortonworks zidibene (ngokusisiseko, ukufunyanwa kokugqibela), kunye ne-Map, ngenxa yeengxaki zemali ezinzulu, zathengiswa kuHewlett Packard. Kwaye ukuba kwiminyaka embalwa ngaphambili, kwimeko yofakelo-kwindawo, ukhetho lwaluhlala kufuneka lwenziwe phakathi kweCloudera kunye neHortonworks, namhlanje, yeha, asinalo olu khetho. Enye into eyothusayo yayikukuba i-Cloudera yabhengeza ngoFebruwari walo nyaka ukuba iyakuyeka ukukhulula iindibano zokubini zokusasazwa kwayo kwindawo yokugcina uluntu, kwaye ngoku zifumaneka kuphela ngobhaliso oluhlawulweyo. Ngokuqinisekileyo, kusenokwenzeka ukukhuphela iinguqulelo zamva nje ze-CDH kunye ne-HDP ekhutshwe ngaphambi kokuphela kwe-2019, kwaye inkxaso yabo ilindeleke unyaka omnye ukuya kwemibini. Kodwa yintoni enokwenziwa ngokulandelayo? Kwabo bebehlawula ngaphambili umrhumo, akukho nto itshintshileyo. Kwaye abo bangafuni ukutshintshela kwinguqu ehlawulwayo yokusabalalisa, kodwa kwangaxeshanye bafuna ukukwazi ukufumana iinguqulelo zamva nje zamacandelo eqela, kunye neepatches kunye nolunye uhlaziyo, silungiselele eli nqaku. Kuyo siza kuqwalasela iindlela ezinokubakho zokuphuma kule meko.
Inqaku lingaphezulu kophononongo. Ayiyi kuqulatha uthelekiso lonikezelo kunye nohlalutyo olucacileyo kubo, kwaye akuyi kubakho iindlela zokupheka zokuzifakela kunye nokuzilungiselela. Kuya kwenzeka ntoni? Siza kuthetha ngokufutshane malunga nokusabalalisa okunjalo njenge-Arenadata Hadoop, efanelwe ngokufanelekileyo ingqalelo yethu ngenxa yokufumaneka kwayo, enqabileyo kakhulu namhlanje. Kwaye ke siza kuthetha ngeVanilla Hadoop, ngakumbi malunga nokuba "inokuphekwa" njani kusetyenziswa i-Apache Bigtop. Ngaba ulungile? Emva koko wamkelekile kwikati.
Arenadata Hadoop
Le yikhithi entsha ngokupheleleyo kwaye, okwangoku, eyaziwa kancinci yokuhambisa uphuhliso lwasekhaya. Ngelishwa, okwangoku kuHabrΓ© kukho kuphela .
Ulwazi oluninzi lunokufumaneka kwigosa iprojekthi. Iinguqulelo zamva nje zokusasazwa zisekelwe kwi-Hadoop 3.1.2 yenguqulo ye-3, kunye ne-2.8.5 yenguqulo ye-2.
Ulwazi malunga nemephu yendlela inokufumaneka .
Ujongano loMphathi weQela leArenadata
Imveliso engundoqo yeArenadata yi , esetyenziselwa ukufaka, ukuqwalasela nokubeka iliso kwiisombululo zesoftware yenkampani. I-ADCM ihanjiswa simahla, kwaye ukusebenza kwayo kwandiswa ngokufaka iinyanda, eziyisethi yeencwadi zokudlala. Iinyanda zohlulwe zibe ziindidi ezimbini: ishishini kunye noluntu. Ezi zamva ziyafumaneka ngokukhuphela mahala kwiwebhusayithi yeArenadata. Kwakhona kunokwenzeka ukuba uphuhlise i-bundle yakho kwaye uyidibanise ne-ADCM.
Ukuthunyelwa kunye nokulawulwa kwe-Hadoop 3, inguqu yoluntu ye-bundle inikezelwa ngokubambisana ne-ADCM, kodwa i-Hadoop 2 ikhona kuphela. njengenye indlela. Ngokuphathelele iindawo zokugcina kunye neepakethi, zivulelekile ukufikelela kuluntu, zinokukhutshelwa kwaye zifakwe ngendlela eqhelekileyo yazo zonke iinxalenye zeqela. Ngokubanzi, ukuhanjiswa kubonakala kunomdla kakhulu. Ndiqinisekile ukuba kuya kubakho abo baqhelene nezisombululo ezifana ne-Cloudera Manager kunye ne-Ambari, kwaye ngubani oya kuyithanda i-ADCM ngokwayo. Kwabanye, kuya kuba yinto enkulu kunye nokuhanjiswa ukubuyisela endaweni.
Ukuba sithetha ngezinto ezingalunganga, ziya kufana nazo zonke ezinye izabelo zeHadoop. Oko kukuthi:
- Into ebizwa ngokuba "yi-vendor lock-in". Ukusebenzisa imizekelo ye-Cloudera kunye neHortonworks, sele siqaphele ukuba kukho umngcipheko wokutshintsha umgaqo-nkqubo wenkampani.
- Ukushiyeka okubalulekileyo ngasemva kweApache phezulu.
I-vanilla Hadoop
Njengoko uyazi, i-Hadoop ayiyona imveliso ye-monolithic, kodwa, eneneni, yonke igalaksi yeenkonzo ezijikeleze inkqubo yefayile ye-HDFS. Bambalwa abantu abaya kwanela ngeqela lefayile enye. Abanye bafuna iHive, abanye Presto, kwaye ke kukho iHBase kunye nePhoenix; I-Spark iya isetyenziswa. Kuluhlu kunye nokulayisha idatha, i-Oozie, iSqoop kunye neFlume zifunyanwa ngamanye amaxesha. Kwaye ukuba umba wokhuseleko uvela, i-Kerberos ngokubambisana neRanger ifika ngokukhawuleza engqondweni.
Iinguqulelo zeBinary zamacandelo eHadoop ziyafumaneka kwiwebhusayithi yeprojekthi nganye ye-ecosystem ngendlela yeetarballs. Ungazikhuphela kwaye uqalise ukufakela, kodwa ngemeko enye: ukongeza ekuhlanganiseni ngokuzimeleyo iipakethe ezivela kwibhinari "ekrwada", onokuthi ufune ukuyenza, awuyi kuba naluqiniseko ngokuhambelana kweenguqulelo ezikhutshelweyo zecandelo ngalinye. enye. Inketho ekhethiweyo kukwakha usebenzisa i-Apache Bigtop. IBigtop iya kukuvumela ukuba wakhe ukusuka kwiApache maven repositories, qhuba iimvavanyo kunye nokwakha iipakethi. Kodwa, yintoni ebaluleke kakhulu kuthi, iBigtop iya kudibanisa ezo nguqulelo zamacandelo aya kuhambelana omnye komnye. Siza kuthetha ngayo ngokubanzi ngezantsi.
Apache Bigtop
I-Apache Bigtop sisixhobo sokwakha, ukupakisha kunye nokuvavanya inani le
iiprojekthi zomthombo ovulekileyo, njengeHadoop kunye neGreenplum. I-Bigtop inokuninzi
ukhupho. Ngexesha lokubhalwa, ukukhutshwa okuzinzileyo kwamva nje kwakuyinguqulo 1.4,
kwaye kwinkosi kwakukho 1.5. Iinguqulelo ezahlukeneyo zokukhutshwa zisebenzisa iinguqulelo ezahlukeneyo
amacandelo. Ngokomzekelo, kwi-1.4 i-Hadoop core components ine-version 2.8.5, kunye ne-master
2.10.0. Ukuqulunqwa kwamacandelo axhaswayo kuyatshintsha. Into ephelelwe lixesha kwaye
into engenakuhlaziyeka iyahamba, kwaye endaweni yayo ifika into entsha, ngakumbi kwimfuno, kwaye
ayisiyiyo into evela kwintsapho yakwa-Apache ngokwayo.
Ukongeza, iBigtop inabaninzi .
Xa saqala ukuqhelana neBigtop, okokuqala samangaliswa kukuthozama kwayo, xa kuthelekiswa nezinye iiprojekthi ze-Apache, ukuxhaphaka kunye nokuthandwa, kunye noluntu oluncinci kakhulu. Oku kulandela ukuba kukho ulwazi oluncinci kwimveliso, kwaye ukukhangela izisombululo kwiingxaki eziye zavela kwiiforamu kunye noluhlu lokuposa alunako ukuvelisa nantoni na. Ekuqaleni, kwaba ngumsebenzi onzima kuthi ukugqiba indibano epheleleyo yokusasazwa ngenxa yeempawu zesixhobo ngokwaso, kodwa siza kuthetha ngale nto kamva kancinane.
Njengomntu ohlekisayo, abo ngaxesha lithile babenomdla kwiiprojekthi zendalo yeLinux njengeGentoo kunye neLFS banokukufumanisa kumnandi kakhulu ukusebenza ngale nto kwaye bakhumbule la maxesha "epic" xa thina ngokwethu sasikhangela (okanye sibhala) ebuilds kwaye wakha rhoqo iMozilla ngeepatshi ezintsha.
Inzuzo enkulu yeBigtop kukuvuleka kunye nokuguquguquka kwezixhobo ezisekelwe kuzo. Isekwe kwiGradle kunye neApache Maven. I-Gradle yaziwa kakhulu njengesixhobo uGoogle asisebenzisayo ukwakha i-Android. Iguquguquka, kwaye, njengoko besitsho, "ivavanyiwe edabini." IMaven sisixhobo esiqhelekileyo sokwakha iiprojekthi kwiApache ngokwayo, kwaye kuba uninzi lweemveliso zayo zikhutshwa ngeMaven, ayinakwenziwa ngaphandle kwayo apha. Kufanelekile ukunikela ingqalelo kwi-POM (imodeli yeprojekthi yeprojekthi) - ifayile ye-xml "esisiseko" echaza yonke into efunekayo ukuze uMaven asebenze neprojekthi yakho, apho wonke umsebenzi wakhiwe khona. Ngqo
iindawo zeMaven kwaye kukho imiqobo abathi abasebenzisi bokuqala beBigtop bahlangane nayo.
Zenza
Ngoko ufanele uqale phi? Yiya kwiphepha lokukhuphela kwaye ukhuphele inguqulelo yamva nje ezinzileyo njengendawo yogcino. Ungafumana nezinto zakudala zokubini eziqokelelwe yiBigtop apho. Ngendlela, phakathi kwabaphathi bephakheji eqhelekileyo, i-YUM kunye ne-APT zixhaswa.
Kungenjalo, unokukhuphela ukhupho lwamva nje oluzinzileyo ngokuthe ngqo
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitUkudibanisa kwi "bigtop" ...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΎΠ±ΡΠ΅ΠΊΡΠΎΠ²: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, Π³ΠΎΡΠΎΠ²ΠΎ.
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ: 100% (20503/20503), Π³ΠΎΡΠΎΠ²ΠΎ.
Updating files: 100% (1998/1998), Π³ΠΎΡΠΎΠ²ΠΎ.Isiphumo ./bigtop ulawulo lujongeka ngolu hlobo:
./bigtop-bigpetstore - usetyenziso lwedemo, imizekelo yokwenziwa
./bigtop-ci - CI izixhobo, jenkins
./bigtop-data-generators - ukuveliswa kwedatha, i-synthetics, iimvavanyo zomsi, njl.
./bigtop-deploy - izixhobo zokusasaza
./bigtop-packages β ulungelelwaniso, izikripthi, iipetshi zokudityaniswa, eyona ndawo iphambili yesixhobo
./bigtop-test-framework - isakhelo sovavanyo
./bigtop-tests - iimvavanyo ngokwazo, umthwalo kunye nomsi
./bigtop_toolchain β okusingqongileyo ukulungiselela ukudibanisa, ukulungisa imeko-bume ukuze isixhobo sisebenze
./build - Yakha ulawulo olusebenzayo
./dl β uluhlu lwemithombo yolwazi ekhutshelweyo
./docker -Ukwakha kwimifanekiso ye-docker, uvavanyo
./gradle -I-gradle config
./output β uvimba weefayili apho ii-artifacts zokwakha ziya khona
./provisioner - ukubonelela
Eyona nto inomdla kuthi kweli nqanaba lolona qwalaselo luphambili ./bigtop/bigtop.bom, apho sibona onke amacandelo axhaswayo aneenguqulelo. Kulapho sinokucacisa inguqu eyahlukileyo yemveliso (ukuba ngokukhawuleza sifuna ukuzama ukuyakha) okanye inguqulo yokwakha (ukuba, umzekelo, songeze isiqwenga esibalulekileyo).
Uluhlu olungaphantsi lukwanomdla omkhulu ./bigtop/bigtop-packages, ehambelana ngqo nenkqubo yokudibanisa amacandelo kunye neepakethe kunye nabo.
Ngoko, sikhuphe i-archive, siyikhuphe okanye senze i-clone kwi-github, ngaba singaqala ukwakha?
Hayi, masilungise okusingqongileyo kuqala.
Ukulungisa iNdawo
Kwaye apha sidinga ukurhoxa kancinci. Ukwakha phantse nayiphi na imveliso enzima kakhulu okanye engaphantsi, udinga indawo ethile - kwimeko yethu, le yi-JDK, iilayibrari ezikwabelwana ngazo, iifayile zentloko, njl., izixhobo, umzekelo, i-ant, ivy2 kunye nokunye okuninzi. Enye yeenketho zokufumana imeko-bume oyidingayo kwiBigtop kukufaka izinto eziyimfuneko kwinginginya yokwakha. Ndinokuba ndiphosakele kulandelelwano, kodwa kubonakala ngathi ngenguqulo 1.0 bekukho inketho yokwakha kwimifanekiso yeDocker esele iqwalaselwe kwaye ifikeleleke, enokufumaneka apha.
Ngokumalunga nokulungiselela okusingqongileyo, kukho umncedisi wale nto - iPuppet.
Ungasebenzisa le miyalelo ilandelayo, baleka kulawulo lweengcambu
isixhobo, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Okanye ngokuthe ngqo ngonopopi:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Ngelishwa, iingxaki zinokuvela sele zikweli nqanaba. Ingcebiso ngokubanzi apha kukusebenzisa unikezelo oluxhaswayo, ukuya kumhla kwinginginya yokwakha, okanye uzame indlela yedocker.
Ndibano
Yintoni esinokuzama ukuyiqokelela? Impendulo yalo mbuzo iya kunikwa yimveliso yomyalelo
./gradlew tasks
KwiPackage imisebenzi icandelo kukho inani leemveliso ezizezokugqibela zeBigtop.
Zinokuchongwa ngesimamva -rpm okanye -pkg-ind (kwimeko yesakhiwo
kwidokhi). Kwimeko yethu, eyona nto inomdla yiHadoop.
Makhe sizame ukwakha kwimekobume yeseva yethu yokwakha:
./gradlew hadoop-rpmI-Bigtop ngokwayo iya kukhuphela imithombo eyimfuneko efunekayo yecandelo elithile kwaye iqalise ukuhlanganisa. Ke, ukusebenza kwesixhobo kuxhomekeke kugcino lweMaven kunye neminye imithombo, oko kukuthi, ifuna ukufikelela kwi-Intanethi.
Ngethuba lokusebenza, imveliso esemgangathweni yenziwa. Ngamanye amaxesha yona kunye nemiyalezo yempazamo inokukunceda uqonde ukuba yintoni engalunganga. Kwaye ngamanye amaxesha kufuneka ufumane ulwazi olongezelelweyo. Kule meko kuyafaneleka ukongeza iingxoxo --info okanye --debug, kwaye inokuba luncedo βstacktrace. Kukho indlela efanelekileyo yokuvelisa isethi yedatha yofikelelo olulandelayo kuluhlu lokuposa, isitshixo --scan.
Ngoncedo lwayo, i-bigtop iya kuqokelela lonke ulwazi kwaye ilubeke kwi-gradle, emva koko iya kubonelela ngekhonkco,
ngokulandela oko, umntu onobuchule uya kuba nako ukuqonda ukuba kutheni indibano isilele.
Nceda qaphela ukuba olu khetho lunokuveza ulwazi ongafuniyo, olunje ngamagama abasebenzisi, iindawo zokuhlala, izinto eziguquguqukayo zemekobume, njalo njalo, lumka ke.
Amaxesha amaninzi iimpazamo zisisiphumo sokungakwazi ukufumana naziphi na iikhomponenti eziyimfuneko kwindibano. Ngokuqhelekileyo, unokuyilungisa ingxaki ngokudala isiqwenga sokulungisa into ethile kwimithombo, umzekelo, iidilesi kwi-pom.xml kwiingcambu zeengcambu zemithombo. Oku kwenziwa ngokudala kunye nokubeka kulawulo olufanelekileyo ./bigtop/bigtop-packages/src/common/oozie/ isiqwenga, umzekelo, kwifom patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
Okunokwenzeka ukuba, ngexesha lokufunda eli nqaku, akuyi kufuneka wenze oku ngasentla uzilungise.
Xa uzisa naziphi na iipatches kunye notshintsho kwindlela yokuhlanganisa, unokufuna "ukusetha kwakhona" indibano usebenzisa umyalelo wokucoca:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedLo msebenzi uya kubuyisela umva zonke iinguqu kwindibano yeli candelo, emva koko indibano iya kwenziwa kwakhona. Ngeli xesha siza kuzama ukwakha iprojekthi kumfanekiso wedokhi:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
ΠΌΠ½ΠΎΠ³ΠΎ Π²ΡΠ²ΠΎΠ΄Π°
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedUlwakhiwo lwenziwa phantsi kwe-CentOS, kodwa lunokwenziwa phantsi kwe-Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indUkongeza kwiipakethi zokwakha ezahlukeneyo zokuhanjiswa kweLinux, isixhobo sinokwenza indawo yokugcina kunye neepakethi ezihlanganisiweyo, umzekelo:
./gradlew yumUnokukhumbula kwakhona malunga novavanyo lomsi kunye nokuthunyelwa eDocker.
Yenza iqoqo leendawo ezintathu:
./gradlew -Pnum_instances=3 docker-provisionerYenza uvavanyo lomsi kwiqela leendawo ezintathu:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerCima iqela:
./gradlew docker-provisioner-destroyFumana imiyalelo yokudibanisa ngaphakathi kwizikhongozeli zedocker:
./gradlew docker-provisioner-sshBonisa isimo:
./gradlew docker-provisioner-statusUnokufunda ngakumbi malunga nemisebenzi yokusasazwa kuxwebhu.
Ukuba sithetha ngovavanyo, kukho inani elikhulu lazo, ngakumbi umsi kunye nokudibanisa. Uhlalutyo lwabo lungaphaya kwemida yeli nqaku. Manditsho nje ukuba ukudibanisa ikhithi yokuhambisa ayingomsebenzi onzima njengoko kunokubonakala ekuqaleni. Sikwazile ukuqokelela kunye nokuphumelela iimvavanyo kuwo onke amacandelo esiwasebenzisayo kwimveliso yethu, kwaye asizange sibe neengxaki zokuwahambisa kunye nokwenza imisebenzi esisiseko kwindawo yovavanyo.
Ukongeza kwizinto ezikhoyo kwiBigtop, kuyenzeka ukongeza nantoni na eyenye, nkqu nophuhliso lwesoftware yakho. Konke oku kuzenzekela ngokugqibeleleyo kwaye ingena kwingqikelelo yeCI/CD.
isiphelo
Ngokucacileyo, ukuhanjiswa okuhlanganiswe ngolu hlobo akufanele kuthunyelwe ngokukhawuleza kwimveliso. Kufuneka uqonde ukuba ukuba kukho imfuneko yokwenene yokwakha kunye nokuxhasa ukuhanjiswa kwakho, ngoko kufuneka utyale imali kunye nexesha kule nto.
Nangona kunjalo, ngokudibanisa nendlela efanelekileyo kunye neqela leengcali, kunokwenzeka ukuba wenze ngaphandle kwezisombululo zorhwebo.
Kubalulekile ukuqaphela ukuba iprojekthi yeBigtop ngokwayo ifuna uphuhliso kwaye ayibonakali iphuhliswa ngokusebenzayo namhlanje. Ithemba leHadoop 3 elivela kuyo nalo alicaci.Ngendlela, ukuba unesidingo sangempela sokwakha iHadoop 3, unokujonga ukusuka Arenadata, apho, ukongeza umgangatho
Kukho inani lamacandelo ongezelelweyo (Ranger, Knox, NiFi).
Ngokuphathelele iRostelecom, kuthi iBigtop yenye yeenketho eziqwalaselwayo namhlanje. Enoba siyakhetha okanye akunjalo, ixesha liya kuchaza.
isihlomelo
Ukubandakanya icandelo elitsha kwindibano, kufuneka udibanise inkcazelo yalo kwi-bigtop.bom kunye ./bigtop-packages. Ungazama ukwenza oku ngokufanisa kunye namacandelo akhoyo. Zama ukuyiqonda. Akunzima njengoko kubonakala ekuqaleni.
Ingaba ucinga ntoni? Siya kuvuya ukubona uluvo lwakho kumazwana kwaye siyabonga ngokuqwalasela kwakho!
Inqaku lilungiswe liqela lolawulo lwedatha yeRostelecom
umthombo: www.habr.com