ʻAʻole paha he mea huna i ka makahiki i hala he makahiki o nā loli nui no Apache Hadoop. I ka makahiki i hala, ua hui ʻo Cloudera a me Hortonworks (ʻo ia hoʻi, ka loaʻa ʻana o ka mea hope), a ʻo Mapr, ma muli o nā pilikia kālā koʻikoʻi, ua kūʻai ʻia iā Hewlett Packard. A inā he mau makahiki ma mua, i ka hihia o ka hoʻonohonoho ʻana ma ka hale, pono ke koho pinepine ʻia ma waena o Cloudera a me Hortonworks, i kēia lā, auwe, ʻaʻohe o mākou koho. ʻO kekahi mea kupanaha, ʻo ia ka mea i hoʻolaha ai ʻo Cloudera i Pepeluali o kēia makahiki e hoʻōki ia i ka hoʻokuʻu ʻana i nā hui binary o kāna hoʻolaha ʻana i loko o ka waihona lehulehu, a ke loaʻa nei lākou i kēia manawa ma o ke kau inoa uku. ʻOiaʻiʻo, hiki nō ke hoʻoiho i nā mana hou o CDH a me HDP i hoʻokuʻu ʻia ma mua o ka hopena o 2019, a ke manaʻo ʻia nei ke kākoʻo iā lākou no hoʻokahi a ʻelua makahiki. Akā he aha ka hana aʻe? No ka poʻe i uku mua no ke kau inoa, ʻaʻohe mea i loli. A no ka poʻe makemake ʻole e hoʻololi i ka mana i uku ʻia o ka hāʻawi ʻana, akā i ka manawa like makemake e hiki ke loaʻa i nā mana hou o nā ʻāpana cluster, a me nā patch a me nā mea hou aku, ua hoʻomākaukau mākou i kēia ʻatikala. I loko o ia mea e noʻonoʻo mākou i nā koho hiki ke puka i waho o kēia kūlana.
He loiloi hou ka ʻatikala. ʻAʻole i loko o ka hoʻohālikelike ʻana o nā māhele a me kahi kikoʻī kikoʻī o lākou, a ʻaʻohe mea ʻike no ka hoʻokomo ʻana a me ka hoʻonohonoho ʻana iā lākou. He aha ka hopena? E kamaʻilio pōkole mākou e pili ana i ka hāʻawi ʻana e like me Arenadata Hadoop, ka mea i kūpono i ko mākou nānā ʻana ma muli o kona hiki ke komo, ʻaʻole loa i kēia lā. A laila e kamaʻilio mākou e pili ana i ka Vanilla Hadoop, e pili ana i ke ʻano e hiki ai ke "moʻa" me Apache Bigtop. Mākaukau? A laila hoʻokipa i ka pōpoki.
ʻO Arenadata Hadoop
He mea hou loa kēia, a i kēia manawa, ʻike liʻiliʻi ʻia ka puʻupuʻu hoʻomohala home. ʻO ka mea pōʻino, i kēia manawa ma Habré wale nō .
Hiki ke loaʻa ka ʻike hou aku ma ka luna papahana. Hoʻokumu ʻia nā mana hou o ka hāʻawi ʻana ma Hadoop 3.1.2 no ka mana 3, a me 2.8.5 no ka mana 2.
Hiki ke loaʻa nā ʻike e pili ana i ka palapala alanui .
ʻO Arenadata Cluster Manager Interface
ʻO ka huahana koʻikoʻi o Arenadata , i hoʻohana ʻia e hoʻokomo, hoʻonohonoho a nānā i nā ʻōnaehana polokalamu ʻoihana like ʻole. Hāʻawi ʻia ʻo ADCM me ka uku ʻole ʻia, a hoʻonui ʻia kāna hana ma ka hoʻohui ʻana i nā pūʻolo, kahi pūʻulu o nā ansible-playbooks. Hoʻokaʻawale ʻia nā pūʻulu i ʻelua ʻano: ʻoihana a me ke kaiāulu. Loaʻa nā mea hope no ka hoʻoiho manuahi mai ka pūnaewele Arenadata. Hiki nō ke hoʻomohala i kāu pūʻolo ponoʻī a hoʻohui iā ia i ADCM.
No ka hoʻonohonoho ʻana a me ka hoʻokele ʻana o Hadoop 3, hāʻawi ʻia kahi ʻano kaiāulu o ka pūpū i hui pū me ADCM, akā no Hadoop 2 aia wale nō. ma ke ʻano he ʻokoʻa. No nā waihona me nā pūʻolo, ua wehe ʻia lākou i ka lehulehu, hiki ke hoʻoiho a hoʻokomo ʻia ma ke ala maʻamau no nā ʻāpana āpau o ka hui. Ma ke ʻano holoʻokoʻa, he mea hoihoi loa ka hoʻolaha ʻana. ʻIke wau aia kekahi poʻe i maʻa i nā hoʻonā e like me Cloudera Manager a me Ambari, a makemake lākou iā ADCM ponoʻī. No kekahi, e lilo ia i mea hoʻohui nui i ka hāʻawi ʻana no ka hookomo ana mai.
Inā mākou e kamaʻilio e pili ana i nā hemahema, e like lākou me nā hoʻohele ʻē aʻe Hadoop. ʻO ia hoʻi:
- ʻO ka mea i kapa ʻia ʻo "mea kūʻai aku laka-in". Ke hoʻohana nei i nā hiʻohiʻona o Cloudera a me Hortonworks, ua ʻike mua mākou aia mau ka pilikia o ka hoʻololi ʻana i nā kulekele ʻoihana.
- He lag nui ma hope o Apache i uka.
Vanila Hadoop
E like me kāu e ʻike ai, ʻaʻole ʻo Hadoop kahi huahana monolithic, akā, ʻoiaʻiʻo, he galaxy holoʻokoʻa o nā lawelawe a puni kāna ʻōnaehana faila HDFS. He liʻiliʻi ka poʻe e lawa i hoʻokahi pūʻulu faila. Pono kekahi iā Hive, ʻo Presto kekahi, a laila aia ʻo HBase a me Phoenix; Hoʻohana nui ʻia ʻo Spark. No ka orchestration a me ka hoʻouka ʻikepili, loaʻa ʻo Oozie, Sqoop a me Flume i kekahi manawa. A inā e kū mai ka pilikia o ka palekana, hiki koke mai ʻo Kerberos i hui pū me Ranger.
Loaʻa nā mana binary o nā ʻāpana Hadoop ma ka pūnaewele o kēlā me kēia papahana kaiaola ma ke ʻano o nā tarballs. Hiki iā ʻoe ke hoʻoiho iā lākou a hoʻomaka i ka hoʻonohonoho ʻana, akā me ke ʻano hoʻokahi: i ka hoʻohui ʻana i nā pūʻulu kūʻokoʻa mai nā binaries "raw", āu e makemake nui ai e hana, ʻaʻole ʻoe e hilinaʻi i ka hoʻohālikelike ʻana o nā mana i hoʻoiho ʻia me kēlā me kēia. ʻē aʻe. ʻO ka koho i makemake ʻia e kūkulu me ka hoʻohana ʻana iā Apache Bigtop. E ʻae ʻo Bigtop iā ʻoe e kūkulu mai nā waihona waihona maven Apache, holo i nā hoʻokolohua a kūkulu i nā pūʻulu. Akā, ʻo ka mea nui loa iā mākou, e hōʻuluʻulu ʻo Bigtop i kēlā mau ʻano o nā ʻāpana e kūpono kekahi i kekahi. E kamaʻilio mākou e pili ana i ia mea ma lalo nei.
ʻO Apache Bigtop
He mea hana ʻo Apache Bigtop no ke kūkulu ʻana, ka hoʻopaʻa ʻana a me ka hoʻāʻo ʻana i kekahi mau mea
nā papahana open source, e like me Hadoop a me Greenplum. Nui nā Bigtop
hoʻokuʻu. I ka manawa kākau, ʻo ka hoʻokuʻu paʻa hou loa he mana 1.4,
a ma ka haku he 1.5. Hoʻohana ʻia nā mana like ʻole o nā hoʻokuʻu
ʻāpana. No ka laʻana, no ka 1.4 Hadoop mau mea nui he mana 2.8.5, a ma master
2.10.0. Ke hoʻololi nei ke ʻano o nā mea i kākoʻo ʻia. Kekahi mea kahiko a
ka unrenewable hele aku, a ma kona wahi, he mea hou, oi i koi, a
ʻaʻole ia he mea mai ka ʻohana Apache ponoʻī.
Eia kekahi, nui nā Bigtop .
I ka wā i hoʻomaka ai mākou e kamaʻāina me Bigtop, ua kahaha mua mākou i kona haʻahaʻa, i ka hoʻohālikelike ʻia me nā papahana Apache ʻē aʻe, prevalence a kaulana, a me kahi kaiāulu liʻiliʻi loa. Ma muli o kēia ka liʻiliʻi o ka ʻike e pili ana i ka huahana, a ʻo ka ʻimi ʻana i nā hoʻonā i nā pilikia i kū mai ma nā ʻaha kūkā a me nā papa leka uila ʻaʻole hiki ke loaʻa i kekahi mea. I ka wā mua, ua lilo ia i mea paʻakikī no mākou e hoʻopau i ka hui piha o ka hāʻawi ʻana ma muli o nā hiʻohiʻona o ka hāmeʻa ponoʻī, akā e kamaʻilio mākou e pili ana i kēia ma hope.
Ma ke ʻano he teaser, ʻo ka poʻe i makemake i kekahi manawa i nā papahana o ke ao Linux e like me Gentoo a me LFS e ʻoluʻolu paha ka hana me kēia mea a hoʻomanaʻo i kēlā mau manawa "epic" a mākou e ʻimi nei (a i ʻole ke kākau ʻana) kūkulu hou a kūkulu hou ʻo Mozilla me nā ʻāpana hou.
ʻO ka pōmaikaʻi nui o Bigtop ka wehe a me ka versatility o nā mea hana i hoʻokumu ʻia ai. Hoʻokumu ʻia ia ma Gradle a me Apache Maven. Ua kaulana loa ʻo Gradle ma ke ʻano he mea hana a Google e kūkulu ai i ka Android. He maʻalahi, a, e like me kā lākou e ʻōlelo nei, "hoʻāʻo kaua." He mea hana maʻamau ʻo Maven no ke kūkulu ʻana i nā papahana ma Apache ponoʻī, a ʻoiai ua hoʻokuʻu ʻia ka hapa nui o kāna mau huahana ma o Maven, ʻaʻole hiki ke hana ʻia me ka ʻole o ia ma ʻaneʻi. Pono e hoʻolohe i ka POM (project object model) - ka "fundamental" xml file e wehewehe ana i nā mea āpau e pono ai no Maven e hana me kāu papahana, kahi i kūkulu ʻia ai nā hana āpau. pololei ma
mau ʻāpana o Maven a aia kekahi mau mea keakea e ʻike pinepine ai nā mea hoʻohana Bigtop i ka manawa mua.
E aʻo
No laila ma hea ʻoe e hoʻomaka ai? E hele i ka ʻaoʻao hoʻoiho a hoʻoiho i ka mana paʻa hou loa ma ke ʻano he waihona. Hiki iā ʻoe ke loaʻa nā mea kiʻi binary i hōʻiliʻili ʻia e Bigtop ma laila. Ma ke ala, ma waena o nā mana hoʻokele maʻamau, kākoʻo ʻia ʻo YUM a me APT.
ʻO kahi ʻē aʻe, hiki iā ʻoe ke hoʻoiho pololei i ka hoʻokuʻu paʻa hou loa mai
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitHoʻopaʻa ʻia i ka "bigtop"…
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
Получение объектов: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, готово.
Определение изменений: 100% (20503/20503), готово.
Updating files: 100% (1998/1998), готово.ʻO ka hopena ./bigtop directory e like me kēia:
./bigtop-bigpetstore - nā noi demo, nā laʻana synthetic
./bigtop-ci — Mea paahana CI, jenkins
./bigtop-data-generators - hana ʻikepili, synthetics, no ka hoʻāʻo ʻana i ka uahi, etc.
./bigtop-deploy - nā mea hana hoʻolaha
./bigtop-packages - configs, scripts, patch no ka hui, ka hapa nui o ka mea hana
./bigtop-test-framework — ka hoao ana
./bigtop-tests - nā ho'āʻo iā lākou iho, hoʻouka a uahi
./bigtop_toolchain — ka honua no ka hui ʻana, hoʻomākaukau i ke kaiapuni no ka mea hana e hana
./build - kūkulu i ka papa kuhikuhi hana
./dl — papa kuhikuhi no nā kumu i hoʻoiho ʻia
./docker - ke kūkulu ʻana i nā kiʻi docker, hoʻāʻo
./gradle - gradle config
./output - ka papa kuhikuhi kahi e kūkulu ai i nā mea waiwai
./provisioner - hoʻolako
ʻO ka mea hoihoi loa iā mākou i kēia pae ʻo ka config nui ./bigtop/bigtop.bom, ma laila mākou e ʻike ai i nā mea i kākoʻo ʻia me nā mana. ʻO kēia kahi e hiki ai iā mākou ke kuhikuhi i kahi mana ʻokoʻa o ka huahana (inā makemake mākou e hoʻāʻo e kūkulu iā ia) a i ʻole kahi mana kūkulu (inā, no ka laʻana, ua hoʻohui mākou i kahi pākuʻi koʻikoʻi).
He hoihoi nui ka subdirectory ./bigtop/bigtop-packages, pili pono i ke kaʻina hana o ka hui ʻana i nā ʻāpana a me nā pūʻolo me lākou.
No laila, ua hoʻoiho mākou i ka waihona, wehe a hana i kahi clone mai github, hiki iā mākou ke hoʻomaka e kūkulu?
ʻAʻole, e hoʻomākaukau mua kākou i ke kaiapuni.
Hoʻomākaukau i ke Kaiapuni
A ma ʻaneʻi mākou e pono ai i kahi hoʻomaha liʻiliʻi. No ke kūkulu ʻana i kahi huahana ʻoi aku a liʻiliʻi paha, pono ʻoe i kahi kaiapuni - i kā mākou hihia, ʻo ia ka JDK, nā waihona like like, nā faila poʻo, a me nā mea hana, no ka laʻana, ant, ivy2 a me nā mea hou aku. ʻO kekahi o nā koho e loaʻa ai ke kaiapuni āu e pono ai no Bigtop ʻo ka hoʻokomo ʻana i nā mea pono i ka hale kūkulu. Hiki iaʻu ke hewa i ka chronology, akā me he mea lā me ka mana 1.0 aia kekahi koho e kūkulu i nā kiʻi Docker i hoʻonohonoho mua ʻia a hiki ke loaʻa, hiki ke loaʻa ma aneʻi.
No ka hoʻomākaukau ʻana i ke kaiapuni, aia kahi mea kōkua no kēia - Puppet.
Hiki iā ʻoe ke hoʻohana i kēia mau kauoha, holo mai ka papa kuhikuhi kumu
mea hana, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
A i ʻole ma o ka papeti:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"ʻO ka mea pōʻino, hiki mai nā pilikia i kēia manawa. ʻO ka ʻōlelo aʻoaʻo maʻamau ma aneʻi ʻo ka hoʻohana ʻana i ka hāʻawi ʻana i kākoʻo ʻia, i kēia lā ma ka hale kūkulu hale, a i ʻole e hoʻāʻo i ke ala docker.
Hōʻuluʻulu
He aha kā mākou e hoʻāʻo ai e hōʻiliʻili? E hāʻawi ʻia ka pane i kēia nīnau e ka puka o ke kauoha
./gradlew tasks
Ma ka ʻāpana hana Package he nui nā huahana i hana hope loa o Bigtop.
Hiki ke ʻike ʻia lākou e ka suffix -rpm a i ʻole -pkg-ind (ma ke ʻano o ke kūkulu ʻana
ma ka docker). I kā mākou hihia, ʻo ka mea hoihoi loa ʻo Hadoop.
E hoʻāʻo kākou e kūkulu i loko o ke kaiapuni o kā mākou kikowaena kikowaena:
./gradlew hadoop-rpmNa Bigtop ponoʻī e hoʻoiho i nā kumu pono e pono ai no kahi mea kikoʻī a hoʻomaka i ka hui. No laila, pili ka hana o ka mea hana i nā waihona waihona Maven a me nā kumu ʻē aʻe, ʻo ia hoʻi, pono ia e komo i ka pūnaewele.
I ka wā o ka hana, hana ʻia ka huahana maʻamau. I kekahi manawa hiki iā ia a me nā memo hewa ke kōkua iā ʻoe e hoʻomaopopo i ka mea i hewa. A i kekahi manawa pono ʻoe e kiʻi i ka ʻike hou aʻe. I kēia hihia, pono e hoʻohui i nā manaʻo --info ai ole ia, --debug, a he pono no hoi –stacktrace. Aia kahi ala maʻalahi e hoʻopuka i kahi hoʻonohonoho ʻikepili no ke komo ʻana i nā papa inoa leka uila, ke kī --scan.
Me kāna kōkua, e hōʻiliʻili ʻo bigtop i nā ʻike āpau a waiho i loko o gradle, a laila e hāʻawi i kahi loulou,
ma ka hahai ʻana i ia mea, hiki i ke kanaka mākaukau ke hoʻomaopopo i ke kumu i hāʻule ʻole ai ka ʻaha.
E ʻoluʻolu e hoʻomaopopo ʻoe hiki i kēia koho ke hōʻike i ka ʻike āu e makemake ʻole ai, e like me nā inoa inoa, nā nodes, nā ʻano hoʻololi kaiapuni, a pēlā aku, no laila e makaʻala.
ʻO ka hapa pinepine ka hopena o ka hiki ʻole ke loaʻa i nā mea pono no ka hui ʻana. ʻO ka maʻamau, hiki iā ʻoe ke hoʻoponopono i ka pilikia ma ka hana ʻana i kahi pākuʻi e hoʻoponopono i kekahi mea i nā kumu, no ka laʻana, nā ʻōlelo i ka pom.xml i ka papa kuhikuhi kumu o nā kumu. Hana ʻia kēia ma ka hana ʻana a me ka waiho ʻana i loko o ka papa kuhikuhi kūpono ./bigtop/bigtop-packages/src/common/oozie/ patch, no ka laʻana, ma ke ʻano patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
ʻO ka mea nui, i ka wā e heluhelu ai i kēia ʻatikala, ʻaʻole pono ʻoe e hana i ka hoʻoponopono ʻana ma luna nei iā ʻoe iho.
I ka hoʻokomo ʻana i nā ʻāpana a me nā hoʻololi i ka mīkini hui, pono paha ʻoe e "hōʻano hou" i ka hui me ka hoʻohana ʻana i ke kauoha hoʻomaʻemaʻe:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedNa kēia hana e hoʻihoʻi i nā hoʻololi a pau i ka hui o kēia ʻāpana, a laila e hana hou ʻia ka hui. I kēia manawa e hoʻāʻo mākou e kūkulu i ka papahana ma kahi kiʻi docker:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
много вывода
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedUa hana ʻia ka hana ma lalo o CentOS, akā hiki ke hana ʻia ma lalo o Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indMa kahi o ke kūkulu ʻana i nā pūʻolo no nā māhele Linux like ʻole, hiki i ka mea hana ke hana i kahi waihona me nā pūʻulu i hui ʻia, no ka laʻana:
./gradlew yumHiki iā ʻoe ke hoʻomanaʻo e pili ana i nā hoʻāʻo uahi a me ka waiho ʻana ma Docker.
E hana i kahi hui o ʻekolu node:
./gradlew -Pnum_instances=3 docker-provisionerE holo i nā hoʻāʻo uahi ma kahi hui o ʻekolu mau node:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerHoloi i kahi pūʻulu:
./gradlew docker-provisioner-destroyE kiʻi i nā kauoha no ka hoʻohui ʻana i loko o nā pahu docker:
./gradlew docker-provisioner-sshHōʻike kūlana:
./gradlew docker-provisioner-statusHiki iā ʻoe ke heluhelu hou aʻe e pili ana i nā hana Deployment ma ka palapala.
Inā mākou e kamaʻilio e pili ana i nā hoʻāʻo, aia ka nui o lākou, ʻoi aku ka uahi a me ka hoʻohui. ʻOi aku ka nui o kā lākou ʻatikala ma waho o kēia ʻatikala. E ʻōlelo wale wau ʻaʻole paʻakikī ka hoʻohui ʻana i kahi pahu hoʻolaha e like me ka mea i ʻike mua ʻia. Ua hiki iā mākou ke hōʻuluʻulu a hāʻawi i nā hoʻokolohua ma nā mea āpau a mākou e hoʻohana ai i kā mākou hana ʻana, a ʻaʻohe pilikia o ka lawe ʻana iā lākou a me ka hana ʻana i nā hana maʻamau i ke kaiapuni hoʻāʻo.
Ma waho aʻe o nā mea i loaʻa i Bigtop, hiki ke hoʻohui i kekahi mea ʻē aʻe, ʻo kāu hoʻomohala polokalamu ponoʻī. Hoʻopili pono ʻia kēia mau mea āpau a kūpono i ka manaʻo CI/CD.
hopena
ʻIke loa, ʻaʻole pono e hoʻouna koke ʻia ka puʻunaue i hōʻuluʻulu ʻia ma kēia ala i ka hana. Pono ʻoe e hoʻomaopopo inā he pono maoli ke kūkulu a kākoʻo i kāu hāʻawi ʻana, a laila pono ʻoe e hoʻolilo i ke kālā a me ka manawa i kēia.
Eia nō naʻe, i ka hui pū ʻana me ke ala kūpono a me kahi hui ʻoihana, hiki ke hana me ka ʻole o nā hoʻonā pāʻoihana.
He mea nui e hoʻomaopopo i ka pono o ka hoʻomohala ʻana o ka papahana Bigtop ponoʻī a ʻaʻole ʻike ʻia e hoʻomohala ʻia i kēia lā. ʻAʻole maopopo ka manaʻo o Hadoop 3 i loko o ia mea. Ma ke ala, inā he pono maoli ʻoe e kūkulu iā Hadoop 3, hiki iā ʻoe ke nānā mai Arenadata, kahi i hoʻohui ʻia i ka maʻamau
Aia kekahi mau mea hou aku (Ranger, Knox, NiFi).
No Rostelecom, no mākou ʻo Bigtop kekahi o nā koho i noʻonoʻo ʻia i kēia lā. Inā mākou e koho a ʻaʻole paha, na ka manawa e haʻi.
Pākuʻi
No ka hoʻokomo ʻana i kahi mea hou i ka hui, pono ʻoe e hoʻohui i kāna wehewehe ʻana iā bigtop.bom a me ./bigtop-packages. Hiki iā ʻoe ke hoʻāʻo e hana i kēia ma ka hoʻohālikelike me nā ʻāpana i loaʻa. E ho'āʻo e noʻonoʻo. ʻAʻole paʻakikī e like me ka mea i ʻike mua ʻia.
Pehea kou manaʻo? E hauʻoli mākou e ʻike i kou manaʻo ma nā ʻōlelo a mahalo iā ʻoe no kou nānā ʻana!
Ua hoʻomākaukau ʻia ka ʻatikala e ka hui hoʻokele data Rostelecom
Source: www.habr.com