O ṣee ṣe kii ṣe aṣiri pe ọdun to kọja jẹ ọdun ti awọn ayipada nla fun Apache Hadoop. Ni ọdun to kọja, Cloudera ati Hortonworks dapọ (ni pataki, gbigba ti igbehin), ati Mapr, nitori awọn iṣoro owo pataki, ti ta si Hewlett Packard. Ati pe ti awọn ọdun diẹ sẹyin, ninu ọran ti awọn fifi sori ile-ile, yiyan nigbagbogbo ni lati ṣe laarin Cloudera ati Hortonworks, loni, alas, a ko ni yiyan yii. Iyalenu miiran ni otitọ pe Cloudera kede ni Kínní ti ọdun yii pe yoo dawọ idasilẹ awọn apejọ alakomeji ti pinpin rẹ sinu ibi ipamọ gbogbo eniyan, ati pe wọn wa bayi nikan nipasẹ ṣiṣe alabapin sisan. Nitoribẹẹ, o tun ṣee ṣe lati ṣe igbasilẹ awọn ẹya tuntun ti CDH ati HDP ti a tu silẹ ṣaaju opin 2019, ati pe atilẹyin fun wọn ni a nireti fun ọdun kan si meji. Ṣugbọn kini lati ṣe nigbamii? Fun awọn ti wọn sanwo tẹlẹ fun ṣiṣe alabapin, ko si ohun ti o yipada. Ati fun awọn ti ko fẹ lati yipada si ẹya isanwo ti pinpin, ṣugbọn ni akoko kanna fẹ lati ni anfani lati gba awọn ẹya tuntun ti awọn paati iṣupọ, ati awọn abulẹ ati awọn imudojuiwọn miiran, a ti pese nkan yii. Ninu rẹ a yoo ṣe akiyesi awọn aṣayan ti o ṣee ṣe lati jade kuro ninu ipo yii.
Nkan naa jẹ diẹ sii ti atunyẹwo. Kii yoo ni lafiwe ti awọn pinpin ati itupalẹ alaye ti wọn, ati pe kii yoo si awọn ilana fun fifi sori ẹrọ ati tunto wọn. Kini yoo ṣẹlẹ? A yoo sọ ni ṣoki nipa iru pinpin bii Arenadata Hadoop, eyiti o tọ si akiyesi wa nitori wiwa rẹ, eyiti o ṣọwọn pupọ loni. Ati lẹhinna a yoo sọrọ nipa Fanila Hadoop, nipataki nipa bii o ṣe le jẹ “jinna” ni lilo Apache Bigtop. Ṣetan? Lẹhinna kaabo si ologbo.
Arenadata Hadoop
Eyi jẹ tuntun patapata ati, sibẹsibẹ, ohun elo pinpin ti a mọ diẹ ti idagbasoke ile. Laanu, ni akoko yii lori Habré nikan wa .
Alaye siwaju sii le ṣee ri lori osise ise agbese. Awọn ẹya tuntun ti pinpin da lori Hadoop 3.1.2 fun ẹya 3, ati 2.8.5 fun ẹya 2.
Alaye nipa ọna opopona le ṣee ri .
Arenadata Cluster Manager Interface
Arenadata ká mojuto ọja ni , eyi ti o ti lo lati fi sori ẹrọ, tunto ati ki o bojuto awọn orisirisi ile ise solusan. ADCM ti pin laisi idiyele, ati pe iṣẹ ṣiṣe rẹ pọ si nipasẹ fifi awọn edidi kun, eyiti o jẹ eto awọn iwe-iṣere ti o ṣeeṣe. Awọn idii ti pin si awọn oriṣi meji: iṣowo ati agbegbe. Awọn igbehin wa fun igbasilẹ ọfẹ lati oju opo wẹẹbu Arenadata. O tun ṣee ṣe lati ṣe agbekalẹ lapapo tirẹ ki o so pọ si ADCM.
Fun imuṣiṣẹ ati iṣakoso ti Hadoop 3, ẹya agbegbe ti lapapo ni a funni ni apapo pẹlu ADCM, ṣugbọn fun Hadoop 2 nikan wa bi yiyan. Bi fun awọn ibi ipamọ pẹlu awọn idii, wọn ṣii si iraye si gbogbo eniyan, wọn le ṣe igbasilẹ ati fi sii ni ọna deede fun gbogbo awọn paati ti iṣupọ naa. Ìwò, awọn pinpin wulẹ gidigidi awon. Mo ni idaniloju pe awọn ti o faramọ awọn iṣeduro bii Cloudera Manager ati Ambari yoo wa, ati pe yoo fẹ ADCM funrararẹ. Fun diẹ ninu, yoo tun jẹ afikun nla ti pinpin fun agbewọle fidipo.
Ti a ba sọrọ nipa awọn alailanfani, wọn yoo jẹ kanna bi fun gbogbo awọn pinpin Hadoop miiran. Eyun:
- Ohun ti a npe ni "Titiipa-titaja". Lilo awọn apẹẹrẹ ti Cloudera ati Hortonworks, a ti mọ tẹlẹ pe ewu nigbagbogbo wa ti iyipada eto imulo ile-iṣẹ.
- Aisun pataki lẹhin Apache oke.
Fanila Hadoop
Bi o ṣe mọ, Hadoop kii ṣe ọja monolithic, ṣugbọn, ni otitọ, gbogbo galaxy ti awọn iṣẹ ni ayika eto faili pinpin HDFS rẹ. Awọn eniyan diẹ yoo ni akojọpọ faili kan ti o to. Diẹ ninu awọn nilo Hive, awọn miiran Presto, ati lẹhinna HBase ati Phoenix wa; Spark ti wa ni lilo siwaju sii. Fun orchestration ati ikojọpọ data, Oozie, Sqoop ati Flume ni a rii nigba miiran. Ati pe ti ibeere aabo ba waye, lẹhinna Kerberos ni apapo pẹlu Ranger lẹsẹkẹsẹ wa si ọkan.
Awọn ẹya alakomeji ti awọn paati Hadoop wa lori oju opo wẹẹbu ti ọkọọkan awọn iṣẹ akanṣe ilolupo ni irisi tarballs. O le ṣe igbasilẹ wọn ki o bẹrẹ fifi sori ẹrọ, ṣugbọn pẹlu ipo kan: ni afikun si apejọ awọn idii ni ominira lati awọn alakomeji “aise”, eyiti o ṣeese julọ lati ṣe, iwọ kii yoo ni igbẹkẹle eyikeyi ninu ibamu ti awọn ẹya ti o gbasilẹ ti awọn paati pẹlu ọkọọkan. miiran. Aṣayan ayanfẹ ni lati kọ nipa lilo Apache Bigtop. Bigtop yoo gba ọ laaye lati kọ lati awọn ibi ipamọ maven Apache, ṣiṣe awọn idanwo ati kọ awọn idii. Ṣugbọn, kini o ṣe pataki pupọ fun wa, Bigtop yoo ṣajọ awọn ẹya ti awọn paati ti yoo ni ibamu pẹlu ara wọn. A yoo sọrọ nipa rẹ ni alaye diẹ sii ni isalẹ.
Apache Bigtop
Apache Bigtop jẹ ohun elo fun kikọ, apoti ati idanwo nọmba kan ti
awọn iṣẹ akanṣe orisun ṣiṣi, gẹgẹbi Hadoop ati Greenplum. Bigtop ni ọpọlọpọ
awọn idasilẹ. Ni akoko kikọ, idasilẹ iduroṣinṣin tuntun jẹ ẹya 1.4,
ati ninu oluwa ni 1.5. Awọn ẹya oriṣiriṣi ti awọn idasilẹ lo awọn ẹya oriṣiriṣi
irinše. Fun apẹẹrẹ, fun 1.4 Hadoop mojuto irinše ni version 2.8.5, ati ni titunto si
2.10.0. Awọn akopọ ti awọn paati atilẹyin tun n yipada. Nkankan igba atijọ ati
awọn unrenewable lọ kuro, ati ninu awọn oniwe-ibi ba wa ni nkankan titun, siwaju sii ni eletan, ati
kii ṣe dandan nkankan lati inu idile Apache funrararẹ.
Ni afikun, Bigtop ni ọpọlọpọ .
Nigba ti a bẹrẹ lati mọ Bigtop, a ni akọkọ gbogbo iyalenu nipasẹ iwọntunwọnsi rẹ, ni ifiwera pẹlu awọn iṣẹ akanṣe Apache miiran, itankalẹ ati olokiki, ati agbegbe kekere kan. O tẹle lati eyi pe alaye kekere wa lori ọja naa, ati wiwa awọn ojutu si awọn iṣoro ti o dide lori awọn apejọ ati awọn atokọ ifiweranṣẹ le ma mu ohunkohun jade rara. Ni akọkọ, o jade lati jẹ iṣẹ-ṣiṣe ti o nira fun wa lati pari apejọ pipe ti pinpin nitori awọn ẹya ara ẹrọ ti ara rẹ, ṣugbọn a yoo sọrọ nipa eyi diẹ diẹ.
Gẹgẹbi teaser, awọn ti o nifẹ ni akoko kan iru awọn iṣẹ akanṣe ti Agbaye Linux bi Gentoo ati LFS le rii pe o ni idunnu lati ṣiṣẹ pẹlu nkan yii ki o ranti awọn akoko “apọju” wọnyẹn nigbati awa funrara wa n wa (tabi paapaa kikọ) ebuilds ati tun ṣe Mozilla nigbagbogbo pẹlu awọn abulẹ tuntun.
Anfani nla ti Bigtop ni ṣiṣi ati iyipada ti awọn irinṣẹ lori eyiti o da lori. O da lori Gradle ati Apache Maven. Gradle jẹ ohun ti a mọ daradara bi irinṣẹ Google nlo lati kọ Android. O rọ, ati, bi wọn ṣe sọ, “idanwo ogun.” Maven jẹ ọpa boṣewa fun awọn iṣẹ ṣiṣe ile ni Apache funrararẹ, ati pe nitori ọpọlọpọ awọn ọja rẹ ni a tu silẹ nipasẹ Maven, ko le ṣee ṣe laisi rẹ nibi boya. O tọ lati san ifojusi si POM (awoṣe ohun elo akanṣe) - faili xml “ipilẹṣẹ” ti n ṣapejuwe ohun gbogbo ti o ṣe pataki fun Maven lati ṣiṣẹ pẹlu iṣẹ akanṣe rẹ, eyiti gbogbo iṣẹ ti kọ. Gangan ni
awọn apakan ti Maven ati pe diẹ ninu awọn idiwọ wa ti awọn olumulo Bigtop igba akọkọ pade nigbagbogbo.
Ṣaṣeṣe
Nitorina nibo ni o yẹ ki o bẹrẹ? Lọ si oju-iwe igbasilẹ ati ṣe igbasilẹ ẹya iduroṣinṣin tuntun bi ile ifi nkan pamosi. O tun le wa awọn ohun-ọṣọ alakomeji ti Bigtop gba nibẹ. Nipa ọna, laarin awọn alakoso package ti o wọpọ, YUM ati APT ni atilẹyin.
Ni omiiran, o le ṣe igbasilẹ idasilẹ iduroṣinṣin tuntun taara lati
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitCloning ni “bigtop”…
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
Получение объектов: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, готово.
Определение изменений: 100% (20503/20503), готово.
Updating files: 100% (1998/1998), готово.Abajade ./bigtop liana dabi nkan bi eleyi:
./bigtop-bigpetstore - awọn ohun elo demo, awọn apẹẹrẹ sintetiki
./bigtop-ci - CI irinṣẹ, jenkins
./bigtop-data-generators - iran data, sintetiki, fun awọn idanwo ẹfin, ati bẹbẹ lọ.
./bigtop-deploy - imuṣiṣẹ irinṣẹ
./bigtop-packages - awọn atunto, awọn iwe afọwọkọ, awọn abulẹ fun apejọ, apakan akọkọ ti ọpa naa
./bigtop-test-framework - igbeyewo ilana
./bigtop-tests - awọn igbeyewo ara wọn, fifuye ati ẹfin
./bigtop_toolchain - ayika fun apejọ, ngbaradi ayika fun ọpa lati ṣiṣẹ
./build - kọ ṣiṣẹ liana
./dl - itọsọna fun awọn orisun ti a gba lati ayelujara
./docker - ile ni awọn aworan docker, idanwo
./gradle - gradle konfigi
./output – awọn liana ibi ti Kọ onisebaye går
./provisioner - ipese
Ohun ti o nifẹ julọ fun wa ni ipele yii ni atunto akọkọ ./bigtop/bigtop.bom, ninu eyiti a rii gbogbo awọn paati atilẹyin pẹlu awọn ẹya. Eyi ni ibiti a ti le pato ẹya ti ọja ti o yatọ (ti a ba fẹ gbiyanju lati kọ ọ lojiji) tabi ẹya kikọ (ti o ba jẹ pe, fun apẹẹrẹ, a ṣafikun alemo pataki kan).
Awọn subdirectory jẹ tun ti awọn nla anfani ./bigtop/bigtop-packages, eyiti o ni ibatan taara si ilana ti apejọ awọn paati ati awọn idii pẹlu wọn.
Nitorinaa, a ṣe igbasilẹ iwe-ipamọ, ṣiṣi silẹ tabi ṣe oniye kan lati github, ṣe a le bẹrẹ kikọ?
Rara, jẹ ki a mura ayika silẹ ni akọkọ.
Ngbaradi ayika
Ati pe nibi a nilo isinmi kekere kan. Lati kọ eyikeyi diẹ sii tabi kere si ọja eka, o nilo agbegbe kan - ninu ọran wa, eyi ni JDK, awọn ile-ikawe pinpin kanna, awọn faili akọsori, ati bẹbẹ lọ, awọn irinṣẹ, fun apẹẹrẹ, ant, ivy2 ati pupọ diẹ sii. Ọkan ninu awọn aṣayan lati gba agbegbe ti o nilo fun Bigtop ni lati fi sori ẹrọ awọn paati pataki lori ile-iṣẹ kọ. Mo le jẹ aṣiṣe ninu iwe-akọọlẹ, ṣugbọn o dabi pe pẹlu ẹya 1.0 tun wa aṣayan lati kọ ni atunto tẹlẹ ati awọn aworan Docker wiwọle, eyiti o le rii nibi.
Bi fun igbaradi ayika, oluranlọwọ wa fun eyi - Puppet.
O le lo awọn aṣẹ wọnyi, ṣiṣe lati inu iwe ilana gbongbo
irinṣẹ, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Tabi taara nipasẹ puppet:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Laanu, awọn iṣoro le dide tẹlẹ ni ipele yii. Imọran gbogbogbo nibi ni lati lo pinpin atilẹyin, titi di oni lori agbalejo kikọ, tabi gbiyanju ipa ọna docker.
Apejọ
Kini a le gbiyanju lati gba? Idahun si ibeere yii ni yoo fun nipasẹ abajade ti aṣẹ naa
./gradlew tasks
Ni apakan awọn iṣẹ-ṣiṣe Package awọn nọmba kan ti awọn ọja wa ti o jẹ awọn ohun-ini ikẹhin ti Bigtop.
Wọn le ṣe idanimọ nipasẹ suffix -rpm tabi -pkg-ind (ninu ọran ti ile
ni docker). Ninu ọran wa, ohun ti o nifẹ julọ ni Hadoop.
Jẹ ki a gbiyanju lati kọ ni agbegbe ti olupin kikọ wa:
./gradlew hadoop-rpmBigtop funrararẹ yoo ṣe igbasilẹ awọn orisun pataki ti o nilo fun paati kan pato ati bẹrẹ apejọ. Nitorinaa, iṣẹ ọpa naa da lori awọn ibi ipamọ Maven ati awọn orisun miiran, iyẹn ni, o nilo iraye si Intanẹẹti.
Lakoko iṣẹ, iṣelọpọ boṣewa ti ipilẹṣẹ. Nigba miiran o ati awọn ifiranṣẹ aṣiṣe le ṣe iranlọwọ fun ọ ni oye ohun ti ko tọ. Ati nigba miiran o nilo lati gba alaye afikun. Ni idi eyi o tọ lati ṣafikun awọn ariyanjiyan --info tabi --debug, ati pe o tun le wulo –stacktrace. Ọna ti o rọrun wa lati ṣe ipilẹṣẹ data ṣeto fun iraye si atẹle si awọn atokọ ifiweranṣẹ, bọtini --scan.
Pẹlu iranlọwọ rẹ, bigtop yoo gba gbogbo alaye naa ki o si fi sii ni gradle, lẹhin eyi yoo pese ọna asopọ kan,
nipa titẹle eyiti, eniyan ti o ni oye yoo ni anfani lati loye idi ti apejọ naa kuna.
Jọwọ ṣe akiyesi pe aṣayan yii le ṣafihan alaye ti o ko fẹ, gẹgẹbi awọn orukọ olumulo, awọn apa, awọn oniyipada ayika, ati bẹbẹ lọ, nitorina ṣọra.
Nigbagbogbo awọn aṣiṣe jẹ abajade ti ailagbara lati gba eyikeyi awọn paati pataki fun apejọ. Ni deede, o le ṣatunṣe iṣoro naa nipa ṣiṣẹda alemo kan lati ṣatunṣe ohunkan ninu awọn orisun, fun apẹẹrẹ, awọn adirẹsi ni pom.xml ninu itọsọna root ti awọn orisun. Eyi ni a ṣe nipasẹ ṣiṣẹda ati gbigbe si inu itọsọna ti o yẹ ./bigtop/bigtop-packages/src/common/oozie/ patch, fun apẹẹrẹ, ni fọọmu patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
O ṣeese julọ, ni akoko kika nkan yii, iwọ kii yoo ni lati ṣe atunṣe loke funrararẹ.
Nigbati o ba n ṣafihan eyikeyi awọn abulẹ ati awọn iyipada si ẹrọ apejọ, o le nilo lati “tunto” apejọ naa ni lilo pipaṣẹ mimọ:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedIšišẹ yii yoo yi gbogbo awọn ayipada pada si apejọ ti paati yii, lẹhin eyi apejọ naa yoo tun ṣe lẹẹkansi. Ni akoko yii a yoo gbiyanju lati kọ iṣẹ akanṣe ni aworan docker:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
много вывода
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedA ṣe itumọ ti labẹ CentOS, ṣugbọn tun le ṣee ṣe labẹ Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indNi afikun si awọn idii ile fun ọpọlọpọ awọn pinpin Linux, ọpa le ṣẹda ibi ipamọ kan pẹlu awọn idii ti a ṣajọpọ, fun apẹẹrẹ:
./gradlew yumO tun le ranti nipa awọn idanwo ẹfin ati imuṣiṣẹ ni Docker.
Ṣẹda iṣupọ ti awọn apa mẹta:
./gradlew -Pnum_instances=3 docker-provisionerṢiṣe awọn idanwo ẹfin ni iṣupọ ti awọn apa mẹta:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerPa iṣupọ kan rẹ:
./gradlew docker-provisioner-destroyGba awọn aṣẹ fun sisopọ inu awọn apoti docker:
./gradlew docker-provisioner-sshṢe afihan ipo:
./gradlew docker-provisioner-statusO le ka diẹ sii nipa awọn iṣẹ-ṣiṣe imuṣiṣẹ ninu iwe.
Ti a ba sọrọ nipa awọn idanwo, nọmba nla wa ninu wọn, paapaa ẹfin ati isọpọ. Onínọmbà wọn ti kọja aaye ti nkan yii. Jẹ ki n kan sọ pe apejọ ohun elo pinpin kii ṣe iṣẹ ṣiṣe ti o nira bi o ṣe le dabi ni iwo akọkọ. A ṣakoso lati ṣajọ ati ṣe awọn idanwo lori gbogbo awọn paati ti a lo ninu iṣelọpọ wa, ati pe a ko ni awọn iṣoro gbigbe wọn ati ṣiṣe awọn iṣẹ ipilẹ ni agbegbe idanwo naa.
Ni afikun si awọn paati ti o wa tẹlẹ ni Bigtop, o ṣee ṣe lati ṣafikun ohunkohun miiran, paapaa idagbasoke sọfitiwia tirẹ. Gbogbo eyi jẹ adaṣe ni pipe ati pe o baamu si imọran CI/CD.
ipari
O han ni, pinpin ti a ṣajọpọ ni ọna yii ko yẹ ki o firanṣẹ lẹsẹkẹsẹ si iṣelọpọ. O nilo lati ni oye pe ti iwulo gidi ba wa lati kọ ati atilẹyin pinpin rẹ, lẹhinna o nilo lati nawo owo ati akoko ninu eyi.
Sibẹsibẹ, ni apapo pẹlu ọna ti o tọ ati ẹgbẹ alamọdaju, o ṣee ṣe pupọ lati ṣe laisi awọn solusan iṣowo.
O ṣe pataki lati ṣe akiyesi pe iṣẹ akanṣe Bigtop funrararẹ nilo idagbasoke ati pe ko dabi pe o ni idagbasoke ni itara loni. Ifojusọna ti Hadoop 3 han ninu rẹ ko tun ṣe akiyesi. Nipa ọna, ti o ba ni iwulo gidi lati kọ Hadoop 3, o le wo. lati Arenadata, ninu eyiti, ni afikun si boṣewa
Nibẹ ni o wa nọmba kan ti afikun irinše (Ranger, Knox, NiFi).
Bi fun Rostelecom, fun wa Bigtop jẹ ọkan ninu awọn aṣayan ti a gbero loni. Boya a yan tabi rara, akoko yoo sọ.
ÀFIKÚN
Lati ṣafikun paati tuntun ninu apejọ, o nilo lati ṣafikun apejuwe rẹ si bigtop.bom ati ./bigtop-packages. O le gbiyanju lati ṣe eyi nipasẹ afiwe pẹlu awọn paati ti o wa tẹlẹ. Gbiyanju lati ro ero rẹ. Ko nira bi o ṣe dabi ni wiwo akọkọ.
Kini o le ro? A yoo dun lati ri ero rẹ ninu awọn asọye ati pe o ṣeun fun akiyesi rẹ!
Nkan naa ti pese sile nipasẹ ẹgbẹ iṣakoso data Rostelecom
orisun: www.habr.com