Nws yog tej zaum tsis pub leejtwg paub tias xyoo tas los yog ib xyoos ntawm kev hloov pauv loj rau Apache Hadoop. Xyoo tas los, Cloudera thiab Hortonworks tau sib koom ua ke (qhov tseem ceeb, qhov yuav tau tom kawg), thiab Mapr, vim muaj teeb meem nyiaj txiag loj, tau muag rau Hewlett Packard. Thiab yog tias ob peb xyoos dhau los, nyob rau hauv rooj plaub ntawm kev teeb tsa hauv tsev, kev xaiv feem ntau yuav tsum tau ua ntawm Cloudera thiab Hortonworks, hnub no, alas, peb tsis muaj qhov kev xaiv no. Lwm qhov xav tsis thoob yog qhov tseeb tias Cloudera tau tshaj tawm thaum Lub Ob Hlis xyoo no tias nws yuav tsum tsis txhob tso tawm binary sib sau ua ke ntawm nws qhov kev faib tawm rau hauv cov chaw khaws ntaub ntawv pej xeem, thiab tam sim no lawv tsuas yog muaj los ntawm kev them nyiaj rau npe. Tau kawg, nws tseem tuaj yeem rub tawm cov ntawv tshiab ntawm CDH thiab HDP tso tawm ua ntej kawg ntawm 2019, thiab kev txhawb nqa rau lawv xav tau rau ib mus rau ob xyoos. Tab sis yuav ua li cas ntxiv? Rau cov uas yav tas los them rau ib tug subscription, tsis muaj dab tsi hloov. Thiab rau cov neeg uas tsis xav hloov mus rau qhov them version ntawm kev faib tawm, tab sis tib lub sijhawm xav kom tau txais qhov tseeb versions ntawm pawg pawg, nrog rau thaj ua rau thaj thiab lwm yam hloov tshiab, peb tau npaj cov kab lus no. Hauv nws peb yuav xav txog cov kev xaiv ua tau kom tawm ntawm qhov xwm txheej no.
Cov kab lus yog ntau ntawm kev tshuaj xyuas. Nws yuav tsis muaj qhov sib piv ntawm kev faib tawm thiab kev soj ntsuam ntxaws ntawm lawv, thiab yuav tsis muaj cov zaub mov txawv rau kev txhim kho thiab teeb tsa lawv. Yuav ua li cas? Peb yuav tham luv luv txog xws li kev faib khoom xws li Arenadata Hadoop, uas tsim nyog tsim nyog peb mloog vim nws muaj, uas tsis tshua muaj hnub no. Thiab tom qab ntawd peb mam li tham txog Vanilla Hadoop, feem ntau hais txog yuav ua li cas nws tuaj yeem "siav" siv Apache Bigtop. Npaj txhij? Ces txais tos miv.
Arenadata Hadoop
Qhov no yog ib qho tshiab kiag li thiab, raws li tseem, me me-paub cov khoom faib khoom ntawm kev loj hlob hauv tsev. Hmoov tsis zoo, tam sim no ntawm Habre tsuas muaj .
Xav paub ntau ntxiv tuaj yeem pom ntawm official qhov project. Qhov tseeb versions ntawm kev faib yog raws li Hadoop 3.1.2 rau version 3, thiab 2.8.5 rau version 2.
Cov ntaub ntawv hais txog txoj hauv kev tuaj yeem nrhiav tau .
Arenadata Cluster Manager Interface
Arenadata cov khoom tseem ceeb yog , uas yog siv los nruab, teeb tsa thiab saib xyuas ntau lub tuam txhab software daws teeb meem. ADCM tau muab faib dawb, thiab nws cov haujlwm tau nthuav dav los ntawm kev ntxiv cov pob khoom, uas yog cov txheej txheem ntawm ansible-playbooks. Cov pob khoom tau muab faib ua ob hom: kev lag luam thiab zej zog. Cov tom kawg muaj rau rub tawm dawb los ntawm Arenadata lub vev xaib. Nws kuj tseem tuaj yeem tsim koj tus kheej pob thiab txuas rau ADCM.
Rau kev xa tawm thiab kev tswj hwm ntawm Hadoop 3, lub zej zog version ntawm lub pob yog muab nrog ADCM, tab sis rau Hadoop 2 tsuas muaj ua lwm txoj. Raws li rau repositories nrog pob, lawv qhib rau pej xeem nkag, lawv tuaj yeem rub tawm thiab nruab rau hauv txoj kev niaj zaus rau tag nrho cov khoom ntawm pawg. Zuag qhia tag nrho, qhov kev faib tawm zoo li nthuav heev. Kuv paub tseeb tias yuav muaj cov neeg siv cov kev daws teeb meem xws li Cloudera Manager thiab Ambari, thiab leej twg yuav nyiam ADCM nws tus kheej. Rau qee tus, nws tseem yuav yog qhov loj ntxiv uas qhov kev faib tawm rau ntshuam hloov.
Yog tias peb tham txog qhov tsis zoo, lawv yuav zoo ib yam li rau tag nrho lwm qhov kev faib tawm Hadoop. Xws li:
- Lub npe hu ua "tus neeg muag khoom kaw-hauv". Siv cov piv txwv ntawm Cloudera thiab Hortonworks, peb twb pom tau hais tias yeej ib txwm muaj kev pheej hmoo ntawm kev hloov tuam txhab txoj cai.
- Qhov tseem ceeb lag tom qab Apache nce toj.
Vanilla Hadoop
Raws li koj paub, Hadoop tsis yog cov khoom lag luam monolithic, tab sis, qhov tseeb, tag nrho galaxy ntawm cov kev pabcuam nyob ib puag ncig nws cov ntaub ntawv faib HDFS. Tsawg tus neeg yuav muaj txaus ntawm ib pawg ntaub ntawv. Qee qhov xav tau Hive, lwm tus Presto, thiab tom qab ntawd muaj HBase thiab Phoenix; Spark tau siv ntau ntxiv. Rau orchestration thiab cov ntaub ntawv thauj khoom, Oozie, Sqoop thiab Flume qee zaum pom. Thiab yog tias qhov teeb meem ntawm kev ruaj ntseg tshwm sim, ces Kerberos nrog Ranger tam sim ntawd los rau hauv siab.
Binary versions ntawm Hadoop Cheebtsam muaj nyob rau ntawm lub vev xaib ntawm txhua qhov ecosystem tej yaam num nyob rau hauv daim ntawv ntawm tarballs. Koj tuaj yeem rub tawm lawv thiab pib teeb tsa, tab sis nrog ib qho xwm txheej: ntxiv rau nws tus kheej sib sau cov pob khoom los ntawm "raws" binaries, uas koj yuav xav ua, koj yuav tsis muaj kev ntseeg siab rau kev sib raug zoo ntawm cov downloaded versions ntawm cov khoom nrog txhua. lwm yam. Qhov kev xaiv zoo tshaj yog tsim siv Apache Bigtop. Bigtop yuav tso cai rau koj tsim los ntawm Apache maven repositories, khiav kev xeem thiab tsim cov pob. Tab sis, dab tsi tseem ceeb heev rau peb, Bigtop yuav sib sau ua ke cov qauv ntawm cov khoom uas yuav sib haum nrog ib leeg. Peb yuav tham txog nws hauv kev nthuav dav ntxiv hauv qab no.
Apache Bigtop
Apache Bigtop yog ib qho cuab yeej rau kev tsim, ntim thiab sim ntau yam
qhib qhov project, xws li Hadoop thiab Greenplum. Bigtop muaj ntau
tso tawm. Thaum lub sijhawm sau ntawv, qhov tseeb tso tawm ruaj khov yog version 1.4,
thiab hauv tus tswv muaj 1.5. Txawv versions ntawm kev tso tawm siv txawv versions
cov khoom. Piv txwv li, rau 1.4 Hadoop core Cheebtsam muaj version 2.8.5, thiab hauv tus tswv
2.10.0 Nws. Qhov sib xyaw ntawm cov khoom txhawb nqa kuj hloov pauv. Ib yam dab tsi outdated thiab
lub unrenewable ploj mus, thiab nyob rau hauv nws qhov chaw los ib yam dab tsi tshiab, ntau xav tau, thiab
nws tsis yog ib yam dab tsi los ntawm Apache tsev neeg nws tus kheej.
Tsis tas li ntawd, Bigtop muaj ntau yam .
Thaum peb pib paub nrog Bigtop, peb tau ua ntej ntawm txhua qhov xav tsis thoob los ntawm nws qhov kev coj ua, piv nrog lwm cov haujlwm Apache, kev nthuav dav thiab muaj koob meej, nrog rau cov zej zog me me. Nws ua raws los ntawm qhov no tias muaj cov ntaub ntawv tsawg kawg nkaus ntawm cov khoom, thiab kev tshawb nrhiav cov kev daws teeb meem uas tau tshwm sim hauv cov rooj sib tham thiab cov npe xa ntawv yuav tsis muaj dab tsi tshwm sim. Thaum xub thawj, nws tau dhau los ua ib txoj haujlwm nyuaj rau peb ua kom tiav kev sib sau ua ke ntawm kev faib tawm vim cov yam ntxwv ntawm cov cuab yeej nws tus kheej, tab sis peb yuav tham txog qhov no me ntsis tom qab.
Raws li ib tug teaser, cov neeg uas nyob rau hauv ib lub sij hawm tau txaus siab rau tej yaam num ntawm Linux ntug li Gentoo thiab LFS tej zaum yuav pom nws nostalgically qab ntxiag mus ua hauj lwm nrog rau qhov no thiab nco ntsoov cov "epic" lub sij hawm thaum peb tus kheej tab tom nrhiav (los yog txawm sau ntawv) ebuilds thiab tsis tu ncua rebuilt Mozilla nrog thaj ua rau thaj tshiab.
Qhov txiaj ntsig loj ntawm Bigtop yog qhov qhib thiab ntau yam ntawm cov cuab yeej uas nws tau ua raws. Nws yog raws li Gradle thiab Apache Maven. Gradle yog lub npe hu ua cov cuab yeej Google siv los tsim Android. Nws yog hloov tau, thiab, raws li lawv hais, "kev sib ntaus sib tua-kuaj." Maven yog ib qho cuab yeej siv rau kev tsim cov phiaj xwm hauv Apache nws tus kheej, thiab txij li feem ntau ntawm nws cov khoom raug tso tawm los ntawm Maven, nws tsis tuaj yeem ua tsis tau yam tsis muaj qhov no. Nws yog ib qho tsim nyog yuav tsum tau them nyiaj rau POM (qhov project qauv) - "qhov tseem ceeb" xml cov ntaub ntawv piav qhia txhua yam tsim nyog rau Maven ua haujlwm nrog koj qhov project, nyob ib puag ncig uas txhua txoj haujlwm tau tsim. Raws nraim ntawm
qhov chaw ntawm Maven thiab muaj qee qhov teeb meem uas thawj zaug Bigtop cov neeg siv feem ntau ntsib.
Xyaum ua haujlwm
Yog li koj yuav tsum pib qhov twg? Mus rau nplooj ntawv rub tawm thiab rub tawm qhov tseeb ruaj khov version ua ib qho archive. Koj tuaj yeem nrhiav binary artifacts sau los ntawm Bigtop muaj. Los ntawm txoj kev, ntawm cov neeg tswj hwm pob, YUM thiab APT tau txais kev txhawb nqa.
Xwb, koj tuaj yeem rub tawm qhov tseeb tso tawm ncaj qha los ntawm
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitCloning hauv "bigtop"β¦
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΎΠ±ΡΠ΅ΠΊΡΠΎΠ²: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, Π³ΠΎΡΠΎΠ²ΠΎ.
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ: 100% (20503/20503), Π³ΠΎΡΠΎΠ²ΠΎ.
Updating files: 100% (1998/1998), Π³ΠΎΡΠΎΠ²ΠΎ.Qhov tshwm sim ./bigtop directory zoo li no:
./bigtop-bigpetstore - daim ntawv thov demo, piv txwv hluavtaws
./bigtop-ci - CI cov cuab yeej, jenkins
./bigtop-data-generators - cov ntaub ntawv tsim, hluavtaws, rau kev sim pa luam yeeb, thiab lwm yam.
./bigtop-deploy - cov cuab yeej siv
./bigtop-packages - configs, scripts, thaj ua rau thaj rau kev sib dhos, qhov tseem ceeb ntawm cov cuab yeej
./bigtop-test-framework β kev xeem lub moj khaum
./bigtop-tests - kuaj lawv tus kheej, thauj khoom thiab haus luam yeeb
./bigtop_toolchain - ib puag ncig rau kev sib dhos, npaj ib puag ncig rau cov cuab yeej ua haujlwm
./build - tsim cov ntaub ntawv ua haujlwm
./dl - directory rau downloaded qhov chaw
./docker - tsev hauv docker duab, kuaj
./gradle - gradle config
./output - daim ntawv teev npe uas tsim artifacts mus
./provisioner - kev npaj
Qhov nthuav tshaj plaws rau peb nyob rau theem no yog lub ntsiab config ./bigtop/bigtop.bom, nyob rau hauv uas peb pom tag nrho cov kev txhawb zog Cheebtsam nrog versions. Qhov no yog qhov uas peb tuaj yeem hais qhia txog qhov sib txawv ntawm cov khoom lag luam (yog tias peb xav sim tsim nws) lossis tsim qauv (yog tias, piv txwv li, peb ntxiv ib qho tseem ceeb).
Lub subdirectory kuj muaj kev txaus siab heev ./bigtop/bigtop-packages, uas yog ncaj qha ntsig txog cov txheej txheem ntawm assembling Cheebtsam thiab pob khoom nrog lawv.
Yog li, peb rub tawm lub archive, unpacked los yog ua clone los ntawm github, peb puas tuaj yeem pib tsim?
Tsis yog, cia peb npaj ib puag ncig ua ntej.
Npaj ib puag ncig
Thiab ntawm no peb xav tau ib tug me me retreat. Txhawm rau tsim yuav luag txhua yam khoom lag luam ntau dua lossis tsawg dua, koj xav tau ib puag ncig ib puag ncig - hauv peb rooj plaub, qhov no yog JDK, tib lub tsev qiv ntawv sib koom, cov ntaub ntawv header, thiab lwm yam, cov cuab yeej, piv txwv li, ntsaum, ivy2 thiab ntau ntxiv. Ib qho ntawm cov kev xaiv kom tau txais ib puag ncig koj xav tau rau Bigtop yog txhawm rau txhim kho cov khoom tsim nyog ntawm tus tswv tsev tsim. Kuv tuaj yeem ua yuam kev hauv lub sijhawm, tab sis nws zoo li nrog version 1.0 kuj tseem muaj qhov kev xaiv los tsim hauv cov duab ua ntej thiab siv tau Docker, uas tuaj yeem pom ntawm no.
Raws li kev npaj ib puag ncig, muaj tus pab rau qhov no - Puppet.
Koj tuaj yeem siv cov lus txib hauv qab no, khiav ntawm lub hauv paus directory
cuab yeej, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Los yog ncaj qha ntawm tus menyuam roj hmab:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Hmoov tsis zoo, kev nyuaj yuav tshwm sim nyob rau theem no. Cov lus qhia dav dav ntawm no yog siv kev txhawb nqa kev faib tawm, mus txog hnub tim ntawm tus tswv tsev tsim, lossis sim ua txoj hauv kev docker.
Sib dhos
Peb tuaj yeem sim sau dab tsi? Cov lus teb rau lo lus nug no yuav muab los ntawm cov zis ntawm cov lus txib
./gradlew tasks
Nyob rau hauv cov khoom ntim khoom muaj ntau yam khoom uas yog cov khoom qub kawg ntawm Bigtop.
Lawv tuaj yeem raug txheeb xyuas los ntawm lub ntsiab lus -rpm lossis -pkg-ind (nyob rau hauv rooj plaub ntawm lub tsev
hauv docker). Hauv peb cov ntaub ntawv, qhov nthuav tshaj plaws yog Hadoop.
Wb sim tsim nyob rau hauv ib puag ncig ntawm peb tsim server:
./gradlew hadoop-rpmBigtop nws tus kheej yuav rub tawm cov peev txheej tsim nyog xav tau rau ib qho tshwj xeeb thiab pib sib dhos. Yog li, cov cuab yeej ua haujlwm yog nyob ntawm Maven repositories thiab lwm qhov chaw, uas yog, nws yuav tsum tau siv Internet.
Thaum lub sijhawm ua haujlwm, cov qauv tsim tawm yog tsim. Qee lub sij hawm nws thiab cov lus yuam kev tuaj yeem pab koj nkag siab qhov tsis raug. Thiab qee zaum koj yuav tsum tau txais cov ntaub ntawv ntxiv. Hauv qhov no nws tsim nyog ntxiv cov lus sib cav --info los yog --debug, thiab tej zaum kuj yuav pab tau βstacktrace. Muaj ib txoj hauv kev yooj yim los tsim cov ntaub ntawv teeb tsa rau kev nkag mus rau cov npe xa ntawv, tus yuam sij --scan.
Nrog nws cov kev pab, bigtop yuav sau tag nrho cov ntaub ntawv thiab muab tso rau hauv gradle, tom qab uas nws yuav muab ib tug txuas,
los ntawm kev ua raws li qhov no, tus neeg muaj peev xwm yuav nkag siab tias vim li cas lub rooj sib txoos ua tsis tiav.
Thov nco ntsoov tias qhov kev xaiv no yuav nthuav tawm cov ntaub ntawv koj tsis xav tau, xws li cov npe siv, cov nodes, ib puag ncig hloov pauv, thiab lwm yam, yog li ceev faj.
Feem ntau qhov yuam kev yog qhov tshwm sim ntawm qhov tsis muaj peev xwm tau txais cov khoom tsim nyog rau kev sib dhos. Feem ntau, koj tuaj yeem kho qhov teeb meem los ntawm kev tsim ib thaj los kho ib yam dab tsi hauv cov peev txheej, piv txwv li, chaw nyob hauv pom.xml hauv cov npe hauv paus ntawm cov peev txheej. Qhov no yog ua los ntawm kev tsim thiab muab tso rau hauv daim ntawv teev npe tsim nyog ./bigtop/bigtop-packages/src/common/oozie/ thaj, piv txwv li, hauv daim ntawv patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
Feem ntau, thaum lub sijhawm nyeem cov lus no, koj yuav tsis tau ua cov saum toj no kho koj tus kheej.
Thaum qhia txog tej thaj ua rau thaj thiab kev hloov pauv rau lub rooj sib txoos, koj yuav tsum tau "rov pib dua" lub rooj sib txoos siv cov lus hais kom huv:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedQhov kev ua haujlwm no yuav thim rov qab txhua qhov kev hloov pauv rau lub rooj sib txoos ntawm cov khoom siv no, tom qab ntawd lub rooj sib txoos yuav rov ua dua. Lub sijhawm no peb yuav sim tsim qhov project hauv daim duab docker:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
ΠΌΠ½ΠΎΠ³ΠΎ Π²ΡΠ²ΠΎΠ΄Π°
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedKev tsim tau ua tiav hauv CentOS, tab sis kuj tuaj yeem ua tiav hauv Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indNtxiv rau kev tsim cov pob khoom rau ntau yam kev faib tawm Linux, lub cuab yeej tuaj yeem tsim cov chaw khaws cia nrog cov pob khoom sib xyaw, piv txwv li:
./gradlew yumKoj tseem tuaj yeem nco ntsoov txog kev sim pa luam yeeb thiab xa mus rau Docker.
Tsim ib pawg ntawm peb cov nodes:
./gradlew -Pnum_instances=3 docker-provisionerKhiav kev sim pa luam yeeb hauv ib pawg ntawm peb lub nodes:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerRho tawm ib pawg:
./gradlew docker-provisioner-destroyTau txais cov lus txib rau kev sib txuas hauv docker ntim:
./gradlew docker-provisioner-sshQhia xwm txheej:
./gradlew docker-provisioner-statusKoj tuaj yeem nyeem ntxiv txog Kev xa cov dej num hauv cov ntaub ntawv.
Yog tias peb tham txog kev sim, muaj ntau tus ntawm lawv, feem ntau yog haus luam yeeb thiab kev koom ua ke. Lawv qhov kev txheeb xyuas yog dhau ntawm cov kab lus no. Cia kuv hais tias kev sib sau cov khoom siv faib khoom tsis yog ib txoj haujlwm nyuaj li nws yuav zoo li thaum xub thawj siab ib muag. Peb tau tswj hwm los sib sau ua ke thiab dhau qhov kev xeem ntawm tag nrho cov khoom uas peb siv hauv peb cov khoom, thiab peb kuj tsis muaj teeb meem siv lawv thiab ua haujlwm yooj yim hauv qhov chaw sim.
Ntxiv rau cov khoom uas twb muaj lawm hauv Bigtop, nws muaj peev xwm ntxiv lwm yam, txawm tias koj tus kheej tsim software. Tag nrho cov no yog automated zoo kawg nkaus thiab haum rau CI / CD lub tswv yim.
xaus
Obviously, qhov kev faib muab tso ua ke nyob rau hauv no txoj kev yuav tsum tsis txhob tam sim ntawd xa mus rau ntau lawm. Koj yuav tsum nkag siab tias yog tias muaj qhov xav tau tiag tiag los tsim thiab txhawb nqa koj qhov kev faib tawm, ces koj yuav tsum tau nqis peev nyiaj thiab sijhawm hauv qhov no.
Txawm li cas los xij, ua ke nrog txoj hauv kev zoo thiab pab pawg tshaj lij, nws muaj peev xwm ua tau yam tsis muaj kev lag luam daws teeb meem.
Nws yog ib qho tseem ceeb uas yuav tsum nco ntsoov tias qhov project Bigtop nws tus kheej yog qhov xav tau ntawm kev txhim kho thiab tsis zoo li yuav tsum tau tsim kho niaj hnub no. Lub zeem muag ntawm Hadoop 3 tshwm sim hauv nws kuj tseem tsis meej. Los ntawm txoj kev, yog tias koj xav tau tiag tiag los tsim Hadoop 3, koj tuaj yeem saib ntawm los ntawm Arenadata, uas, ntxiv rau tus qauv
Muaj ntau cov khoom ntxiv (Ranger, Knox, NiFi).
Raws li rau Rostelecom, rau peb Bigtop yog ib qho ntawm cov kev xaiv tau txiav txim siab hnub no. Txawm peb xaiv los tsis yog, lub sijhawm yuav qhia.
Lus Qhia Ntxiv
Txhawm rau suav nrog cov khoom siv tshiab hauv kev sib dhos, koj yuav tsum ntxiv nws cov lus piav qhia rau bigtop.bom thiab ./bigtop-packages. Koj tuaj yeem sim ua qhov no los ntawm kev sib piv nrog cov khoom uas twb muaj lawm. Sim ua kom paub tseeb. Nws tsis yooj yim li nws zoo li thaum xub thawj siab ib muag.
Koj xav li cas? Peb yuav zoo siab pom koj lub tswv yim hauv cov lus thiab ua tsaug rau koj cov lus qhia!
Tsab ntawv tau npaj los ntawm Rostelecom pab pawg tswj xyuas cov ntaub ntawv
Tau qhov twg los: www.hab.com