Mwina sizobisika kuti chaka chatha chinali chaka chakusintha kwakukulu kwa Apache Hadoop. Chaka chatha, Cloudera ndi Hortonworks adagwirizanitsa (makamaka, kupeza komaliza), ndipo Mapr, chifukwa cha mavuto aakulu azachuma, adagulitsidwa kwa Hewlett Packard. Ndipo ngati zaka zingapo m'mbuyomo, pankhani ya kukhazikitsa pamalopo, kusankha nthawi zambiri kumayenera kupangidwa pakati pa Cloudera ndi Hortonworks, lero, tsoka, tilibe chisankho ichi. Chodabwitsa china chinali chakuti Cloudera adalengeza mu February chaka chino kuti isiya kutulutsa misonkhano yamabizinesi yomwe imagawidwa m'malo osungira anthu ambiri, ndipo tsopano ikupezeka pokhapokha polembetsa. Zachidziwikire, ndizothekabe kutsitsa mitundu yaposachedwa ya CDH ndi HDP yomwe idatulutsidwa kumapeto kwa chaka cha 2019, ndipo thandizo lawo likuyembekezeka kwa chaka chimodzi kapena ziwiri. Koma chotani chotsatira? Kwa iwo omwe adalipira kale zolembetsa, palibe chomwe chasintha. Ndipo kwa iwo omwe safuna kusinthira ku mtundu wolipidwa wogawira, koma nthawi yomweyo akufuna kuti athe kulandira matembenuzidwe atsopano a zigawo zamagulu, komanso zigamba ndi zosintha zina, takonzekera nkhaniyi. M'menemo tiwona njira zomwe zingatheke kuti tituluke mumkhalidwewu.
Nkhaniyi ndi yobwerezabwereza. Sizikhala ndi kufananitsa kwa magawo ndi kusanthula mwatsatanetsatane, ndipo sipadzakhala maphikidwe oyika ndikuwakonza. Kodi chidzachitike n'chiyani? Tidzakambirana mwachidule za kugawa koteroko monga Arenadata Hadoop, yomwe imayenera kusamala chifukwa cha kupezeka kwake, komwe kuli kosowa kwambiri lero. Kenako tikambirana za Vanilla Hadoop, makamaka za momwe "ingaphike" pogwiritsa ntchito Apache Bigtop. Mwakonzeka? Ndiye kulandiridwa kwa mphaka.
Arenadata Hadoop
Izi ndi zatsopano komanso, pakadali pano, zida zogawa zachitukuko zapakhomo sizidziwika. Tsoka ilo, pakadali pano pa HabrΓ© pali kokha .
Zambiri zitha kupezeka pa boma polojekiti. Mitundu yaposachedwa yogawayi idakhazikitsidwa pa Hadoop 3.1.2 ya mtundu 3, ndi 2.8.5 ya mtundu 2.
Zambiri za mapu amsewu zitha kupezeka .
Arenadata Cluster Manager Interface
Chogulitsa chachikulu cha Arenadata ndi , yomwe imagwiritsidwa ntchito kukhazikitsa, kukonza ndi kuyang'anira njira zosiyanasiyana zamapulogalamu amakampani. ADCM imagawidwa kwaulere, ndipo magwiridwe ake amakulitsidwa ndikuwonjezera mitolo, yomwe ndi gulu la mabuku omvera. Mitolo imagawidwa m'magulu awiri: bizinesi ndi anthu. Zomalizazi zilipo kuti zitsitsidwe kwaulere patsamba la Arenadata. Ndizothekanso kupanga mtolo wanu ndikulumikiza ku ADCM.
Pakutumizidwa ndi kuyang'anira Hadoop 3, mtundu wamagulu amgululi umaperekedwa molumikizana ndi ADCM, koma Hadoop 2 ilipo yokha. ngati njira ina. Ponena za nkhokwe zokhala ndi mapaketi, ndizotsegukira kuti anthu azitha kuzipeza, zitha kutsitsidwa ndikuyika mwanjira yanthawi zonse zamagulu onse agululo. Ponseponse, kugawa kumawoneka kosangalatsa kwambiri. Ndikukhulupirira kuti padzakhala omwe adazolowera mayankho monga Cloudera Manager ndi Ambari, ndi omwe angakonde ADCM yokha. Kwa ena, zidzakhalanso zazikulu kuphatikiza kuti kugawa zolowa m'malo.
Ngati tilankhula za zovuta zake, zidzakhala zofanana ndi zogawa zina zonse za Hadoop. Izi:
- Zomwe zimatchedwa "vendor lock-in". Pogwiritsa ntchito zitsanzo za Cloudera ndi Hortonworks, tazindikira kale kuti nthawi zonse pali chiopsezo chosintha ndondomeko ya kampani.
- Kutsalira kwakukulu kuseri kwa Apache kumtunda.
Vanila Hadoop
Monga mukudziwira, Hadoop si mankhwala monolithic, koma, kwenikweni, mlalang'amba wonse wa mautumiki mozungulira anagawa file dongosolo HDFS. Anthu ochepa adzakhala ndi zokwanira file gulu limodzi. Ena amafunikira Hive, ena Presto, ndiyeno pali HBase ndi Phoenix; Spark ikugwiritsidwa ntchito kwambiri. Poyimba ndi kutsitsa deta, Oozie, Sqoop ndi Flume nthawi zina amapezeka. Ndipo ngati nkhani ya chitetezo ikabuka, ndiye kuti Kerberos molumikizana ndi Ranger nthawi yomweyo amabwera m'maganizo.
Mitundu ya Binary ya zida za Hadoop ikupezeka patsamba lililonse lazinthu zachilengedwe monga ma tarballs. Mutha kuzitsitsa ndikuyamba kuyika, koma ndi chikhalidwe chimodzi: kuphatikiza pakusonkhanitsa mapaketi pawokha kuchokera kumabina "yaiwisi", zomwe mungafune kuchita, simudzakhala ndi chidaliro chilichonse pakugwirizana kwamitundu yotsitsidwa ndi iliyonse. zina. Njira yomwe mungakonde ndikumanga pogwiritsa ntchito Apache Bigtop. Bigtop ikulolani kuti mupange kuchokera ku ma Apache maven repositories, kuyesa mayeso ndi kupanga phukusi. Koma, chomwe chili chofunikira kwambiri kwa ife, Bigtop isonkhanitsa mitundu yazigawo zomwe zimagwirizana. Tidzakambirana mwatsatanetsatane pansipa.
Apache Bigtop
Apache Bigtop ndi chida chomangira, kulongedza ndikuyesa zingapo
mapulojekiti otseguka, monga Hadoop ndi Greenplum. Bigtop ili ndi zambiri
zotulutsa. Panthawi yolemba, kutulutsidwa kokhazikika kwaposachedwa kunali mtundu 1.4,
ndi mbuye anali 1.5. Zotulutsa zosiyanasiyana zimagwiritsa ntchito mitundu yosiyanasiyana
zigawo. Mwachitsanzo, kwa 1.4 Hadoop core components ali ndi version 2.8.5, ndi master
2.10.0. Kupangidwa kwa zigawo zothandizira kukusinthanso. Chinachake chachikale ndi
zosasinthika zimachoka, ndipo m'malo mwake pamabwera chinthu chatsopano, chofunikira kwambiri, ndi
sichofunikira kwenikweni kuchokera ku banja la Apache.
Kuphatikiza apo, Bigtop ili ndi zambiri .
Titayamba kudziwana ndi Bigtop, choyamba tidadabwa ndi kudzichepetsa kwake, poyerekeza ndi ntchito zina za Apache, kufalikira ndi kutchuka, komanso dera laling'ono kwambiri. Izi zimachokera ku izi kuti pali chidziwitso chochepa pa malonda, ndipo kufunafuna njira zothetsera mavuto omwe atuluka pamabwalo ndi mndandanda wamakalata sikungabweretse kalikonse. Poyamba, inali ntchito yovuta kwa ife kuti tikwaniritse msonkhano wathunthu wa kugawa chifukwa cha mawonekedwe a chida chokha, koma tidzakambirana za izi pambuyo pake.
Monga woseketsa, iwo omwe nthawi ina anali ndi chidwi ndi ntchito za Linux monga Gentoo ndi LFS atha kupeza kuti ndizosangalatsa kugwira ntchito ndi chinthu ichi ndikukumbukira nthawi "zowopsa" zomwe ife tinali kuyang'ana (kapena ngakhale kulemba) ebuilds ndikumanganso Mozilla pafupipafupi ndi zigamba zatsopano.
Ubwino waukulu wa Bigtop ndikutseguka komanso kusinthasintha kwa zida zomwe zidakhazikitsidwa. Zimakhazikitsidwa ndi Gradle ndi Apache Maven. Gradle imadziwika bwino ngati chida chomwe Google amagwiritsa ntchito popanga Android. Imasinthasintha, ndipo, monga amanenera, "kuyesedwa pankhondo." Maven ndi chida chokhazikika pama projekiti omanga ku Apache pomwe, ndipo popeza zambiri zomwe zimapangidwa zimatulutsidwa kudzera ku Maven, sizingachitike popanda izi. Ndikoyenera kumvera POM (chitsanzo cha polojekiti) - fayilo "yofunikira" xml yofotokoza zonse zofunika kuti Maven agwire ntchito ndi polojekiti yanu, pomwe ntchito zonse zimamangidwa. Ndendende pa
mbali za Maven ndipo pali zopinga zina zomwe ogwiritsa ntchito a Bigtop oyamba amakumana nazo.
Yesetsani
Ndiye muyambire kuti? Pitani ku tsamba lotsitsa ndikutsitsa mtundu waposachedwa ngati malo osungira. Mutha kupezanso zinthu zakale zamabina zomwe zasonkhanitsidwa ndi Bigtop pamenepo. Mwa njira, pakati pa oyang'anira phukusi wamba, YUM ndi APT amathandizidwa.
Kapenanso, mutha kutsitsa kumasulidwa kokhazikika kwaposachedwa kuchokera
github:
$ git clone --branch branch-1.4 https://github.com/apache/bigtop.gitKujambula mu "bigtop" ...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 40217 (delta 14), reused 10 (delta 1), pack-reused 40171
ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΎΠ±ΡΠ΅ΠΊΡΠΎΠ²: 100% (40217/40217), 43.54 MiB | 1.05 MiB/s, Π³ΠΎΡΠΎΠ²ΠΎ.
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΠΉ: 100% (20503/20503), Π³ΠΎΡΠΎΠ²ΠΎ.
Updating files: 100% (1998/1998), Π³ΠΎΡΠΎΠ²ΠΎ.Chotsatira cha ./bigtop chikuwoneka motere:
./bigtop-bigpetstore - ntchito zowonetsera, zitsanzo zopangira
./bigtop-ci - CI toolkit, jenkins
./bigtop-data-generators - kupanga deta, zopangira, zoyesa utsi, etc.
./bigtop-deploy - zida zotumizira
./bigtop-packages - configs, scripts, zigamba za msonkhano, gawo lalikulu la chida
./bigtop-test-framework - chimango choyesera
./bigtop-tests - mayesero okha, katundu ndi kusuta
./bigtop_toolchain - chilengedwe kuti asonkhane, kukonzekera chilengedwe chida ntchito
./build - kupanga chikwatu ntchito
./dl - chikwatu kwa magwero otsitsidwa
./docker - kumanga muzithunzi za docker, kuyesa
./gradle - Gradle config
./output - chikwatu komwe zinthu zakale zimapita
./provisioner - kupereka
Chosangalatsa kwambiri kwa ife pakadali pano ndikusintha kwakukulu ./bigtop/bigtop.bom, momwe timawonera zigawo zonse zothandizidwa ndi mitundu. Apa ndi pamene tingathe kufotokoza mtundu wina wa mankhwala (ngati mwadzidzidzi tikufuna kuyesa kumanga) kapena kumanga (ngati, mwachitsanzo, tawonjezera chigamba chachikulu).
Subdirectory ilinso ndi chidwi chachikulu ./bigtop/bigtop-packages, zomwe zimagwirizana mwachindunji ndi ndondomeko yosonkhanitsa zigawo ndi mapaketi nawo.
Chifukwa chake, tidatsitsa zosungira, kuzimasula kapena kupanga chojambula kuchokera ku github, tingayambe kumanga?
Ayi, tiyeni tikonze chilengedwe kaye.
Kukonzekera Chilengedwe
Ndipo apa tikufunika kuthawirako pang'ono. Kuti mupange pafupifupi chinthu china chilichonse kapena chocheperako, muyenera malo ena - kwa ife, iyi ndi JDK, malaibulale omwe amagawana nawo, mafayilo amutu, ndi zina zambiri, zida, mwachitsanzo, nyerere, ivy2 ndi zina zambiri. Chimodzi mwazosankha zopezera chilengedwe chomwe mungafune ku Bigtop ndikuyika zofunikira pazomangamanga. Ndikhoza kukhala ndikulakwitsa pakuwerengera nthawi, koma zikuwoneka kuti ndi mtundu 1.0 panalinso njira yopangira zithunzi za Docker zomwe zidakonzedweratu komanso zopezeka, zomwe zingapezeke pano.
Ponena za kukonzekera chilengedwe, pali wothandizira pa izi - Chidole.
Mukhoza kugwiritsa ntchito malamulo otsatirawa, kuthamanga kuchokera muzu directory
chida, ./bigtop:
./gradlew toolchain
./gradlew toolchain-devtools
./gradlew toolchain-puppetmodules
Kapena mwachindunji kudzera pa chidole:
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::installer"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::deployment-tools"
puppet apply --modulepath=<path_to_bigtop> -e "include bigtop_toolchain::development-tools"Tsoka ilo, zovuta zitha kubwera kale pakadali pano. Upangiri wamba apa ndikugwiritsa ntchito kugawa kothandizidwa, zaposachedwa panyumba yomanga, kapena kuyesa njira ya docker.
Msonkhano
Kodi tingayese kusonkhanitsa chiyani? Yankho la funso ili lidzaperekedwa ndi zotsatira za lamulo
./gradlew tasks
Mugawo la Package tasks pali zinthu zingapo zomwe ndi zomaliza za Bigtop.
Atha kudziwika ndi suffix -rpm kapena -pkg-ind (pankhani yomanga
mu docker). Kwa ife, chosangalatsa kwambiri ndi Hadoop.
Tiyeni tiyese kupanga m'malo a seva yathu yomanga:
./gradlew hadoop-rpmBigtop yokha idzatsitsa zofunikira pagawo linalake ndikuyamba kusonkhana. Chifukwa chake, ntchito ya chidacho imadalira malo osungira a Maven ndi magwero ena, ndiye kuti, imafunikira intaneti.
Panthawi yogwira ntchito, kutulutsa kokhazikika kumapangidwa. Nthawi zina ndi zolakwika mauthenga angakuthandizeni kumvetsa chimene chinalakwika. Ndipo nthawi zina muyenera kudziwa zambiri. Pamenepa ndi bwino kuwonjezera mfundo --info kapena --debug, komanso zingakhale zothandiza βstacktrace. Pali njira yabwino yopangira ma data kuti mupeze mndandanda wamakalata, makiyi --scan.
Ndi chithandizo chake, bigtop idzasonkhanitsa zidziwitso zonse ndikuziyika pang'onopang'ono, pambuyo pake zidzapereka ulalo,
potsatira izi, munthu waluso azitha kumvetsetsa chifukwa chake msonkhano unalephereka.
Chonde dziwani kuti izi zitha kuwulula zambiri zomwe simukuzifuna, monga mayina olowera, ma node, zosintha zachilengedwe, ndi zina zambiri, choncho samalani.
Nthawi zambiri zolakwika zimakhala chifukwa cholephera kupeza zigawo zilizonse zofunika pakusonkhanitsa. Nthawi zambiri, mutha kukonza vutolo popanga chigamba kuti mukonze zina m'magwero, mwachitsanzo, ma adilesi mu pom.xml mu bukhu la mizu ya magwero. Izi zimachitika popanga ndikuziyika mu bukhu loyenera ./bigtop/bigtop-packages/src/common/oozie/ chigamba, mwachitsanzo, mu mawonekedwe patch2-fix.diff.
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
<repositories>
<repository>
<id>central</id>
- <url>http://repo1.maven.org/maven2</url>
+ <url>https://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
Mwinamwake, panthawi yomwe mukuwerenga nkhaniyi, simudzasowa kuchita zomwe zili pamwambazi.
Mukayambitsa zigamba zilizonse ndikusintha pamakina osonkhanitsira, mungafunike "kukonzanso" msonkhanowo pogwiritsa ntchito lamulo loyeretsa:
./gradlew hadoop-clean
> Task :hadoop_vardefines
> Task :hadoop-clean
BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executedOpaleshoni iyi idzabwezeretsanso zosintha zonse pagulu la gawoli, kenako msonkhanowo udzachitidwanso. Nthawi ino tiyesa kupanga polojekitiyi mu chithunzi cha docker:
./gradlew -POS=centos-7 -Pprefix=1.2.1 hadoop-pkg-ind
> Task :hadoop-pkg-ind
Building 1.2.1 hadoop-pkg on centos-7 in Docker...
+++ dirname ./bigtop-ci/build.sh
++ cd ./bigtop-ci/..
++ pwd
+ BIGTOP_HOME=/tmp/bigtop
+ '[' 6 -eq 0 ']'
+ [[ 6 -gt 0 ]]
+ key=--prefix
+ case $key in
+ PREFIX=1.2.1
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ key=--os
+ case $key in
+ OS=centos-7
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=--target
+ case $key in
+ TARGET=hadoop-pkg
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z x ']'
+ '[' -z x ']'
+ '[' '' == true ']'
+ IMAGE_NAME=bigtop/slaves:1.2.1-centos-7
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 '!=' x86_64 ']'
++ docker run -d bigtop/slaves:1.2.1-centos-7 /sbin/init
+
CONTAINER_ID=0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8
+ trap 'docker rm -f
0ce5ac5ca955b822a3e6c5eb3f477f0a152cd27d5487680f77e33fbe66b5bed8' EXIT
....
ΠΌΠ½ΠΎΠ³ΠΎ Π²ΡΠ²ΠΎΠ΄Π°
....
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-namenode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-secondarynamenode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-zkfc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-journalnode-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-datanode-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-httpfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-resourcemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-nodemanager-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-proxyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-yarn-timelineserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-mapreduce-historyserver-2.8.5-
1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-client-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-conf-pseudo-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-doc-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-libhdfs-devel-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-hdfs-fuse-2.8.5-1.el7.x86_64.rpm
Wrote: /bigtop/build/hadoop/rpm/RPMS/x86_64/hadoop-debuginfo-2.8.5-1.el7.x86_64.rpm
+ umask 022
+ cd /bigtop/build/hadoop/rpm//BUILD
+ cd hadoop-2.8.5-src
+ /usr/bin/rm -rf /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-2.8.5-1.el7.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.uQ2FCn
+ exit 0
+ umask 022
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.CwDb22
+ cd /bigtop/build/hadoop/rpm//BUILD
+ rm -rf hadoop-2.8.5-src
+ exit 0
[ant:touch] Creating /bigtop/build/hadoop/.rpm
:hadoop-rpm (Thread[Task worker for ':',5,main]) completed. Took 38 mins 1.151 secs.
:hadoop-pkg (Thread[Task worker for ':',5,main]) started.
> Task :hadoop-pkg
Task ':hadoop-pkg' is not up-to-date because:
Task has not declared any outputs despite executing actions.
:hadoop-pkg (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
BUILD SUCCESSFUL in 40m 37s
6 actionable tasks: 6 executed
+ RESULT=0
+ mkdir -p output
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/build .
+ docker cp
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb:/bigtop/output .
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
+ '[' 0 -ne 0 ']'
+ docker rm -f ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
Error: No such container:
ac46014fd9501bdc86b6c67d08789fbdc6ee46a2645550ff6b6712f7d02ffebb
BUILD SUCCESSFUL in 41m 24s
1 actionable task: 1 executedKumangaku kunachitika pansi pa CentOS, komanso kutha kuchitidwa pansi pa Ubuntu:
./gradlew -POS=ubuntu-16.04 -Pprefix=1.2.1 hadoop-pkg-indKuphatikiza pakupanga phukusi la magawo osiyanasiyana a Linux, chidachi chitha kupanga chosungira chokhala ndi mapaketi ophatikizidwa, mwachitsanzo:
./gradlew yumMutha kukumbukiranso za mayeso a utsi ndi kutumizidwa ku Docker.
Pangani gulu la node zitatu:
./gradlew -Pnum_instances=3 docker-provisionerYesetsani kuyesa utsi mugulu la node zitatu:
./gradlew -Pnum_instances=3 -Prun_smoke_tests docker-provisionerChotsani gulu:
./gradlew docker-provisioner-destroyPezani malamulo olumikizira mkati mwa zotengera za docker:
./gradlew docker-provisioner-sshOnetsani udindo:
./gradlew docker-provisioner-statusMutha kuwerenga zambiri za Deployment tasks muzolemba.
Ngati tikambirana za mayeso, pali ambiri ndithu, makamaka utsi ndi kuphatikiza. Kusanthula kwawo sikungatheke m'nkhani ino. Ndiloleni ndingonena kuti kusonkhanitsa zida zogawa sizovuta monga momwe zingawonekere poyang'ana koyamba. Tidakwanitsa kusonkhanitsa ndikuyesa zida zonse zomwe timagwiritsa ntchito popanga, ndipo sitinakhalenso ndi zovuta kuziyika ndikuchita ntchito zoyambira pamayeso.
Kuphatikiza pazigawo zomwe zilipo ku Bigtop, ndizotheka kuwonjezera china chilichonse, ngakhale pulogalamu yanu yopanga mapulogalamu. Zonsezi zimangochitika zokha ndipo zimagwirizana ndi lingaliro la CI/CD.
Pomaliza
Mwachiwonekere, kugawa komwe kumapangidwa motere sikuyenera kutumizidwa nthawi yomweyo kupanga. Muyenera kumvetsetsa kuti ngati pali chosowa chenicheni chomanga ndikuthandizira kugawa kwanu, ndiye kuti muyenera kuyika ndalama ndi nthawi mu izi.
Komabe, kuphatikiza njira yoyenera ndi gulu la akatswiri, ndizotheka kuchita popanda njira zamalonda.
Ndikofunika kuzindikira kuti pulojekiti ya Bigtop yokha ikufunika chitukuko ndipo sikuwoneka kuti ikupangidwa mwakhama lero. Chiyembekezo cha Hadoop 3 kuwonekera momwemo sichidziwikanso. Mwa njira, ngati muli ndi chosowa chenicheni chomanga Hadoop 3, mukhoza kuyang'ana kuchokera ku Arenadata, momwe, kuwonjezera pa muyezo
Pali zowonjezera zowonjezera (Ranger, Knox, NiFi).
Ponena za Rostelecom, kwa ife Bigtop ndi imodzi mwazosankha zomwe zikuganiziridwa lero. Kaya tisankhe kapena ayi, nthawi idzatiuza.
Zakumapeto
Kuti muphatikizepo china chatsopano pagulu, muyenera kuwonjezera kufotokozera kwake ku bigtop.bom ndi ./bigtop-packages. Mutha kuyesa kuchita izi pofanizira ndi zigawo zomwe zilipo. Yesani kulingalira. Sizovuta monga momwe zimawonekera poyang'ana koyamba.
Mukuganiza chiyani? Tidzakhala okondwa kuwona malingaliro anu mu ndemanga ndipo zikomo chifukwa cha chidwi chanu!
Nkhaniyi idakonzedwa ndi gulu loyang'anira data la Rostelecom
Source: www.habr.com