Cishe akuyona imfihlo ukuthi unyaka odlule bekungunyaka wezinguquko ezinkulu ze-Apache Hadoop. Ngonyaka odlule, i-Cloudera ne-Hortonworks yahlanganiswa (empeleni, ukutholwa kokugcina), futhi i-Map, ngenxa yezinkinga ezinkulu zezezimali, yathengiswa ku-Hewlett Packard. Futhi uma eminyakeni embalwa ngaphambili, endabeni yokufakwa kwezakhiwo, ukukhetha ngokuvamile kwakudingeka kwenziwe phakathi kwe-Cloudera ne-Hortonworks, namuhla, maye, asinakho lokhu kukhetha. Okunye okumangazayo ukuthi i-Cloudera yamemezela ngoFebhuwari walo nyaka ukuthi izoyeka ukukhipha imihlangano kanambambili yokusatshalaliswa kwayo endaweni yokugcina umphakathi, futhi manje isitholakala kuphela ngokubhaliselwe okukhokhelwayo. Vele, kusengenzeka ukulanda izinguqulo zakamuva ze-CDH ne-HDP ezikhishwe ngaphambi kokuphela kuka-2019, futhi ukusekelwa kwazo kulindeleke unyaka owodwa kuya kwemibili. Kodwa yini okufanele uyenze ngokulandelayo? Kulabo abebekhokhela ukubhalisa ngaphambilini, akukho okushintshile. Futhi kulabo abangafuni ukushintshela kunguqulo ekhokhelwayo yokusabalalisa, kodwa ngesikhathi esifanayo bafuna ukukwazi ukuthola izinguqulo zakamuva zezingxenye zeqoqo, kanye nama-patches nezinye izibuyekezo, silungiselele lesi sihloko. Kuyo sizocubungula izindlela ezingenzeka zokuphuma kulesi simo.
I-athikili iyisibuyekezo. Ngeke iqukathe ukuqhathaniswa kokusatshalaliswa nokuhlaziya okuningiliziwe kwakho, futhi ngeke kube khona izindlela zokupheka zokuzifaka nokuyilungiselela. Kuzokwenzekani? Sizokhuluma kafushane mayelana nokusabalalisa okunjalo njenge-Arenadata Hadoop, okufanele ukunakekelwa kwethu ngokufanele ngenxa yokutholakala kwayo, okuyivelakancane kakhulu namuhla. Bese sizokhuluma ngeVanilla Hadoop, ikakhulukazi ngokuthi "ingaphekwa" kanjani kusetshenziswa i-Apache Bigtop. Ulungile? Bese ukwamukela ikati.
Arenadata Hadoop
Lena entsha ngokuphelele futhi, okwamanje, isethi yokusabalalisa eyaziwa kancane yokuthuthukiswa kwasekhaya. Ngeshwa, okwamanje ku-HabrΓ© kukhona kuphela .
Olunye ulwazi lungatholakala kwisikhulu iphrojekthi. Izinguqulo zakamuva zokusabalalisa zisekelwe ku-Hadoop 3.1.2 yenguqulo 3, kanye ne-2.8.5 yenguqulo yesi-2.
Ulwazi mayelana nemephu yomgwaqo lungatholakala .
I-Arenadata Cluster Manager Interface
Umkhiqizo oyinhloko we-Arenadata ngu , esetshenziselwa ukufaka, ukulungisa nokuqapha izixazululo zesofthiwe yenkampani ehlukahlukene. I-ADCM isakazwa mahhala, futhi ukusebenza kwayo kunwetshwa ngokungeza izinqwaba, okuyisethi yezincwadi zokudlala ezinengqondo. Izinqwaba zihlukaniswe zibe izinhlobo ezimbili: ibhizinisi kanye nomphakathi. Lezi zokugcina ziyatholakala ukuze zilandwe mahhala kuwebhusayithi ye-Arenadata. Kungenzeka futhi ukuthi uthuthukise inqwaba yakho futhi uyixhume ku-ADCM.
Ngokusatshalaliswa nokuphathwa kwe-Hadoop 3, inguqulo yomphakathi yenqwaba inikezwa ngokuhlanganyela ne-ADCM, kodwa ku-Hadoop 2 kukhona kuphela njengenye indlela. Ngokuqondene namaqoqo anamaphakheji, avulekele ukufinyelela komphakathi, angalandwa futhi afakwe ngendlela evamile yazo zonke izingxenye zeqoqo. Sekukonke, ukusabalalisa kubukeka kuthakazelisa kakhulu. Nginesiqiniseko sokuthi kuzoba khona abajwayele izixazululo ezifana ne-Cloudera Manager ne-Ambari, futhi abazoyithanda i-ADCM uqobo. Kwabanye, kuzoba futhi enkulu ukuthi ukusatshalaliswa ukuze kufakwe endaweni.
Uma sikhuluma ngobubi, zizofana nazo zonke ezinye ukusatshalaliswa kwe-Hadoop. Okungukuthi:
- Okubizwa ngokuthi "ukhiye wokuthengisa". Ukusebenzisa izibonelo ze-Cloudera ne-Hortonworks, sesivele saqaphela ukuthi kuhlale kunengozi yokuguqula inqubomgomo yenkampani.
- Ukubambezeleka okuphawulekayo ngemuva kwe-Apache ekhuphuka nomfula.
I-vanilla Hadoop
Njengoba wazi, i-Hadoop ayiwona umkhiqizo we-monolithic, kodwa, empeleni, i-galaxy yonke yezinsizakalo ezizungeze uhlelo lwayo lwefayela olusabalalisiwe lwe-HDFS. Bambalwa abantu abazoba neqoqo lefayela elilodwa elanele. Abanye badinga i-Hive, abanye iPresto, bese kuba khona i-HBase ne-Phoenix; i-Spark iya ngokuya isetshenziswa. Ngomculo we-orchestration nokulayisha idatha, i-Oozie, i-Sqoop ne-Flume ngezinye izikhathi iyatholakala. Futhi uma inkinga yokuphepha ivela, khona-ke i-Kerberos ngokuhlanganyela ne-Ranger ifika ngokushesha engqondweni.
Izinguqulo kanambambili zezingxenye ze-Hadoop ziyatholakala kuwebhusayithi yephrojekthi ngayinye ye-ecosystem ngendlela yama-tarballs. Ungawalanda futhi uqale ukuwafaka, kodwa ngombandela owodwa: ngaphezu kokuhlanganisa amaphakheji ngokuzimela kusuka kumabhinari "okuhlaza", okungenzeka ukuthi ufuna ukukwenza, ngeke ube nethemba lokuhambisana kwezinguqulo ezilandiwe zezingxenye ngayinye. okunye. Inketho ekhethwayo ukwakha usebenzisa i-Apache Bigtop. I-Bigtop izokuvumela ukuthi wakhe kusuka kumakhosombe e-Apache maven, sebenzisa izivivinyo futhi wakhe amaphakheji. Kodwa, okubaluleke kakhulu kithi, i-Bigtop izohlanganisa lezo zinguqulo zezingxenye ezizohambisana nezinye. Sizokhuluma ngakho ngokuningiliziwe ngezansi.
I-Apache Bigtop
I-Apache Bigtop iyithuluzi lokwakha, ukupakisha nokuhlola inombolo
amaphrojekthi omthombo ovulekile, njengeHadoop neGreenplum. I-Bigtop inokuningi
ukukhishwa. Ngesikhathi sokubhala, ukukhishwa kwakamuva okuzinzile kwakuyinguqulo 1.4,
futhi enkosini kwakukhona 1.5. Izinguqulo ezihlukene zokukhishwa zisebenzisa izinguqulo ezahlukene
izingxenye. Isibonelo, ku-1.4 izingxenye eziyinhloko ze-Hadoop zinenguqulo 2.8.5, futhi ku-master
2.10.0. Ukwakheka kwezingxenye ezisekelwayo nakho kuyashintsha. Into ephelelwe yisikhathi futhi
okungavuseleleki kuyahamba, futhi endaweni yakho kuza into entsha, efunwa kakhulu, futhi
akuyona into evela emndenini wakwa-Apache uqobo.
Ngaphezu kwalokho, i-Bigtop inabaningi .
Lapho siqala ukujwayelana ne-Bigtop, okokuqala samangala ngokuzithoba kwayo, uma kuqhathaniswa namanye amaphrojekthi we-Apache, ukusabalala nokuthandwa, kanye nomphakathi omncane kakhulu. Lokhu kulandela ukuthi kunolwazi oluncane ngomkhiqizo, futhi ukucinga izisombululo zezinkinga ezivele ezingosini kanye nohlu lwama-mail kungase kungatheli lutho nhlobo. Ekuqaleni, kwaba umsebenzi onzima kithi ukuqedela umhlangano ophelele wokusabalalisa ngenxa yezici zethuluzi ngokwalo, kodwa sizokhuluma ngalokhu kamuva.
Njengomuntu ohlekisayo, labo abake baba nesithakazelo kumaphrojekthi anjalo wendawo yonke ye-Linux njengeGentoo ne-LFS bangase bakuthole kumnandi kakhulu ukusebenza nale nto futhi bakhumbule lezo zikhathi "ezinhle" lapho thina ngokwethu sifuna (noma ngisho nokubhala) ebuilds futhi yakha kabusha i-Mozilla njalo ngeziqephu ezintsha.
Inzuzo enkulu ye-Bigtop ukuvuleka nokusebenzisa izinto ezihlukahlukene kwamathuluzi esekelwe kuwo. Isekelwe ku-Gradle ne-Apache Maven. I-Gradle yaziwa kakhulu njengethuluzi i-Google elisebenzisayo ukwakha i-Android. Iyavumelana nezimo, futhi, njengoba besho, βihlolwe impi.β IMaven iyithuluzi elijwayelekile lamaphrojekthi wokwakha e-Apache uqobo, futhi njengoba iningi lemikhiqizo yayo ikhishwa ngeMaven, bekungeke kwenziwe ngaphandle kwayo nalapha. Kuyafaneleka ukunaka i-POM (imodeli yento yephrojekthi) - ifayela le-xml βeliyisisekeloβ elichaza yonke into edingekayo ukuze uMaven asebenze nephrojekthi yakho, lapho wonke umsebenzi owakhelwe khona. Impela ngo
izingxenye ze-Maven futhi kunezinye izithiyo abasebenzisi bokuqala be-Bigtop abavame ukuhlangana nazo.
Hlanganisa
Ngakho kufanele uqale kuphi? Yiya ekhasini lokulanda bese ulanda inguqulo yakamuva ezinzile njengendawo yomlando. Ungathola futhi ama-artifact kanambambili aqoqwe yi-Bigtop lapho. Ngendlela, phakathi kwabaphathi bephakheji abajwayelekile, i-YUM ne-APT ziyasekelwa.
Okunye, ungalanda ukukhishwa kwakamuva okuzinzile ngokuqondile
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitIhlanganisa "phezulu"β¦
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΎΠ±ΡΠ΅ΠΊΡΠΎΠ²: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, Π³ΠΎΡΠΎΠ²ΠΎ.
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ: 100% (20503/20503), Π³ΠΎΡΠΎΠ²ΠΎ.
Updating files: 100% (1998/1998), Π³ΠΎΡΠΎΠ²ΠΎ.Umphumela we-./bigtop directory ubukeka kanjena:
./bigtop-bigpetstore - izinhlelo zokusebenza zedemo, izibonelo zokwenziwa
./bigtop-ci - Amathuluzi e-CI, ama-jenkins
./bigtop-data-generators - ukukhiqizwa kwedatha, okokwenziwa, ukuhlolwa kwentuthu, njll.
./bigtop-deploy - amathuluzi okuthumela
./bigtop-packages - Izilungiselelo, imibhalo, iziqephu zokuhlanganisa, ingxenye enkulu yethuluzi
./bigtop-test-framework - uhlaka lokuhlola
./bigtop-tests - izivivinyo ngokwazo, zilayishe futhi zibheme
./bigtop_toolchain - indawo yokuhlanganisa, ukulungisa indawo ukuze ithuluzi lisebenze
./build - Yakha inkomba yokusebenza
./dl β umkhombandlela wemithombo elandiwe
./docker - Ukwakhiwa kwezithombe ze-docker, ukuhlolwa
./gradle - i-gradle config
./output - uhla lwemibhalo lapho izinto zokwakha ziya khona
./provisioner - ukuhlinzeka
Into ethakazelisa kakhulu kithi kulesi sigaba ukulungiselelwa okuyinhloko ./bigtop/bigtop.bom, lapho sibona zonke izingxenye ezisekelwayo ezinezinguqulo. Kulapho singacacisa khona inguqulo ehlukile yomkhiqizo (uma sivele sifuna ukuzama ukuyakha) noma inguqulo yokwakha (uma, ngokwesibonelo, sengeze ipheshi elibalulekile).
I-subdirectory nayo inentshisekelo enkulu ./bigtop/bigtop-packages, ehlobene ngokuqondile nenqubo yokuhlanganisa izingxenye namaphakheji nazo.
Ngakho-ke, silande ingobo yomlando, siyikhiphe noma senze i-clone kusuka ku-github, singakwazi ukuqala ukwakha?
Cha, masilungise imvelo kuqala.
Ukulungisa Imvelo
Futhi lapha sidinga ukubaleka okuncane. Ukwakha cishe noma yimuphi umkhiqizo oyinkimbinkimbi kakhulu noma ongaphansi, udinga indawo ethile - kithi, lena i-JDK, imitapo yolwazi ehlanganyelwe, amafayela enhlokweni, njll., amathuluzi, isibonelo, intuthwane, ivy2 nokunye okuningi. Enye yezinketho zokuthola imvelo oyidingayo ku-Bigtop ukufaka izingxenye ezidingekayo kumsingathi wokwakha. Ngingaba nephutha ngokulandelana kwezikhathi, kodwa kubonakala sengathi ngenguqulo 1.0 bekukhona nenketho yokwakha ezithombeni ze-Docker ezilungiselelwe ngaphambili nezifinyelelekayo, ezingatholakala lapha.
Ngokuqondene nokulungiselela imvelo, kukhona umsizi walokhu - I-Puppet.
Ungasebenzisa imiyalo elandelayo, gijima kusuka kuhlu lwezimpande
ithuluzi, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Noma ngokuqondile nge-puppet:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Ngeshwa, kungase kuphakame ubunzima kakade kulesi sigaba. Iseluleko esivamile lapha ukusebenzisa ukusabalalisa okusekelwayo, ukwaziswa kwamuva kumsingathi wokwakha, noma zama umzila wedokhu.
Umhlangano
Yini esingazama ukuyiqoqa? Impendulo yalo mbuzo izonikezwa ngokuphuma komyalo
./gradlew tasks
Esigabeni semisebenzi yephakheji kunenani lemikhiqizo engama-artifact okugcina e-Bigtop.
Angabonakala ngesijobelelo -rpm noma -pkg-ind (uma kwakhiwa
esitokisini). Esimweni sethu, okuthakazelisa kakhulu yi-Hadoop.
Ake sizame ukwakha endaweni yeseva yethu yokwakha:
./gradlew hadoop-rpmI-Bigtop ngokwayo izolanda imithombo edingekayo yengxenye ethile futhi iqale ukuhlanganisa. Ngakho-ke, ukusebenza kwethuluzi kuncike kumakhosombe e-Maven neminye imithombo, okungukuthi, kudinga ukufinyelela kwe-inthanethi.
Ngesikhathi sokusebenza, ukukhishwa okujwayelekile kukhiqizwa. Kwesinye isikhathi yona kanye nemilayezo yephutha ingakusiza uqonde ukuthi yini engahambanga kahle. Futhi ngezinye izikhathi udinga ukuthola ulwazi olwengeziwe. Kulokhu kufanelekile ukwengeza izimpikiswano --info noma --debug, futhi ingase ibe usizo βstacktrace. Kukhona indlela elula yokwenza isethi yedatha yokufinyelela okulandelayo ohlwini lwama-imeyili, ukhiye --scan.
Ngosizo lwayo, i-bigtop izoqoqa lonke ulwazi futhi ilufake kancane kancane, emva kwalokho izohlinzeka ngesixhumanisi,
ngokulandela lokho, umuntu onekhono uzokwazi ukuqonda ukuthi kungani umhlangano wehlulekile.
Sicela uqaphele ukuthi le nketho ingase iveze ulwazi ongafuni, njengamagama abasebenzisi, izindawo, okuguquguqukayo kwendawo, njll., ngakho qaphela.
Ngokuvamile amaphutha awumphumela wokungakwazi ukuthola noma yiziphi izingxenye ezidingekayo ukuze kuhlanganiswe. Ngokuvamile, ungakwazi ukulungisa inkinga ngokwakha isiqeshana ukuze ulungise okuthile emithonjeni, isibonelo, amakheli aku-pom.xml kumkhombandlela wezimpande wemithombo. Lokhu kwenziwa ngokudala nokuyibeka ohlwini lwemibhalo olufanele ./bigtop/bigtop-packages/src/common/oozie/ isichibi, isibonelo, efomini patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
Ngokunokwenzeka, ngesikhathi sokufunda lesi sihloko, ngeke kudingeke wenze lokhu okungenhla uzilungise ngokwakho.
Lapho wethula noma yimaphi ama-patches nezinguquko kumshini wokuhlanganisa, kungase kudingeke "ukusetha kabusha" umhlangano usebenzisa umyalo wokuhlanza:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedLokhu kusebenza kuzobuyisela emuva zonke izinguquko ekuhlanganisweni kwale ngxenye, ngemva kwalokho ukuhlanganisa kuzokwenziwa futhi. Kulokhu sizozama ukwakha iphrojekthi esithombeni sedokhu:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
ΠΌΠ½ΠΎΠ³ΠΎ Π²ΡΠ²ΠΎΠ΄Π°
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedUkwakhiwa kwenziwa ngaphansi kwe-CentOS, kodwa futhi kungenziwa ngaphansi kwe-Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indNgokungeziwe kumaphakheji wokwakha wokusabalalisa okuhlukahlukene kwe-Linux, ithuluzi lingakha indawo yokugcina ngamaphakheji ahlanganisiwe, isibonelo:
./gradlew yumUngakhumbula futhi mayelana nokuhlolwa kwentuthu kanye nokuthunyelwa ku-Docker.
Dala iqoqo lamanodi amathathu:
./gradlew -Pnum_instances=3 docker-provisionerYenza ukuhlolwa kwentuthu kuqoqo lamanodi amathathu:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerSusa iqoqo:
./gradlew docker-provisioner-destroyThola imiyalo yokuxhuma ngaphakathi kweziqukathi ze-docker:
./gradlew docker-provisioner-sshBonisa isimo:
./gradlew docker-provisioner-statusUngafunda kabanzi mayelana nemisebenzi Yokuthunyelwa kumadokhumenti.
Uma sikhuluma ngokuhlolwa, kunenani elikhulu kakhulu, ikakhulukazi intuthu nokuhlanganiswa. Ukuhlaziya kwabo kungaphezu kobubanzi balesi sihloko. Mangisho nje ukuthi ukuhlanganisa ikhithi yokusabalalisa akuwona umsebenzi onzima njengoba kungase kubonakale ekuqaleni. Sikwazile ukuhlanganisa nokuphumelela izivivinyo kuzo zonke izakhi esizisebenzisayo ekukhiqizeni kwethu, futhi asibanga nazinkinga zokuzithumela nokwenza imisebenzi eyisisekelo endaweni yokuhlola.
Ngaphezu kwezingxenye ezikhona ku-Bigtop, kuyenzeka ungeze noma yini enye, ngisho nokuthuthukiswa kwesofthiwe yakho. Konke lokhu ku-othomathikhi ngokuphelele futhi kuhambisana nomqondo we-CI/CD.
isiphetho
Ngokusobala, ukusatshalaliswa okuhlanganiswe ngale ndlela akufanele kuthunyelwe ngokushesha ekukhiqizeni. Udinga ukuqonda ukuthi uma kunesidingo sangempela sokwakha nokusekela ukusatshalaliswa kwakho, khona-ke udinga ukutshala imali nesikhathi kulokhu.
Kodwa-ke, ngokuhlanganiswa nendlela efanele kanye neqembu lochwepheshe, kungenzeka ukwenza ngaphandle kwezixazululo zezentengiso.
Kubalulekile ukuqaphela ukuthi iphrojekthi ye-Bigtop ngokwayo idinga ukuthuthukiswa futhi ayibonakali ingathuthukiswa ngokuqhubekayo namuhla. Ithemba lokuthi i-Hadoop 3 ivela kuyo nayo ayicacile. Ngendlela, uma unesidingo sangempela sokwakha i-Hadoop 3, ungabheka kusuka ku-Arenadata, lapho, ngaphezu kokujwayelekile
Kunenombolo yezingxenye ezengeziwe (Ranger, Knox, NiFi).
Ngokuqondene neRostelecom, kithi iBigtop ingenye yezinketho ezicatshangelwa namuhla. Ukuthi siyakukhetha noma cha, isikhathi sizosho.
isithasiselo
Ukuze ufake ingxenye entsha ekuhlanganiseni, udinga ukungeza incazelo yayo kokuthi bigtop.bom kanye ./bigtop-packages. Ungazama ukwenza lokhu ngokufanisa nezingxenye ezikhona. Zama ukukuqonda. Akunzima njengoba kubonakala ekuqaleni.
UCABANGANI? Sizojabula ukubona umbono wakho kumazwana futhi siyabonga ngokunaka kwakho!
I-athikili ilungiselelwe ithimba lokuphatha idatha le-Rostelecom
Source: www.habr.com