\u0441\u0435\u0440\u0432\u0435\u0440\u0430<\/a>, ashtu edhe n\u00eb HDFS.<\/p>\nN\u00eb \u00e7do transformim bazuar n\u00eb modelin e parazgjedhur, mund t\u00eb krijoni skenarin tuaj n\u00eb gjuh\u00ebn PDL, e cila \u00ebsht\u00eb paksa e ngjashme me shell. <\/p>\n
Me an\u00eb t\u00eb gjuh\u00ebs PDL, ju mund t\u00eb zgjatni funksionalitetin e transformimeve dhe, konkretisht, mund t\u00eb gjeneroni n\u00eb m\u00ebnyr\u00eb dinamike ( gjat\u00eb ekzekutimit) fragma t\u00eb rast\u00ebsishme t\u00eb kodit n\u00eb var\u00ebsi t\u00eb parametrave t\u00eb ekzekutimit.<\/p>\n
Po ashtu, n\u00eb Ab Initio integrimi me OS-n\u00eb \u00ebsht\u00eb m\u00ebse i zhvilluar p\u00ebrmes shell-it. Konkretisht n\u00eb Sberbank p\u00ebrdoret linux ksh. Mund t\u00eb shk\u00ebmbeni me shell variablat dhe t\u2019i p\u00ebrdorni ato si parametra t\u00eb grafik\u00ebve. Mund t\u00eb th\u00ebrrisni ekzekutimin e grafik\u00ebve Ab Initio nga shell dhe t\u00eb administroni Ab Initio.<\/p>\n
P\u00ebrve\u00e7 Ab Initio GDE, d\u00ebrgesa p\u00ebrfshin shum\u00eb produkte t\u00eb tjer\u00eb. Ka nj\u00eb sistem Co>Operation me pretendimin p\u00ebr t\u2019u quajtur sistem operativ. Ka Control>Center, ku mund t\u00eb vendosni n\u00eb program dhe t\u00eb monitoroni flukset e ngarkes\u00ebs. Ka produkte p\u00ebr zhvillim n\u00eb nj\u00eb nivel m\u00eb t\u00eb thjesht\u00eb sesa lejon Ab Initio GDE.<\/p>\n
P\u00ebrshkrimi i korniz\u00ebs MDW dhe pun\u00ebve p\u00ebr p\u00ebrshtatjen e saj p\u00ebr GreenPlum<\/h3>\n \nS\u00eb bashku me produktet e tij, shit\u00ebsi ofron produktin MDW (Metadata Driven Warehouse), i cili \u00ebsht\u00eb nj\u00eb konfigurues grafik\u00ebsh, i destinuar p\u00ebr t\u00eb ndihmuar n\u00eb detyrat tipike t\u00eb mbushjes s\u00eb depozita t\u00eb dh\u00ebnash ose vault-eve t\u00eb dh\u00ebnash.<\/p>\n
Ai p\u00ebrmban parser\u00eb metadatas p\u00ebrdoruesish (specifik p\u00ebr projektin) dhe gjenerator\u00eb kodi \u201cnd\u00ebr shfletues\u201d.<\/p>\n
N\u00eb rastet e Hive dhe RDBMS, gjenerohen grafet e ndryshme p\u00ebr p\u00ebrdit\u00ebsimin fillestar dhe inkremental t\u00eb t\u00eb dh\u00ebnave.<\/p>\n
N\u00eb rastin e Hive, t\u00eb dh\u00ebnat e d\u00ebrguara t\u00eb dyt\u00ebs kombinohen p\u00ebrmes Ab Initio Join me t\u00eb dh\u00ebnat q\u00eb kan\u00eb qen\u00eb n\u00eb tabel\u00eb para p\u00ebrdit\u00ebsimit. Ngarkuesit e t\u00eb dh\u00ebnave n\u00eb MDW (si n\u00eb Hive, ashtu edhe n\u00eb RDBMS) jo vet\u00ebm q\u00eb futin t\u00eb dh\u00ebna t\u00eb reja nga d\u00eblta, por gjithashtu mbyllin periudhat e aktualitetit t\u00eb t\u00eb dh\u00ebnave, sipas \u00e7el\u00ebsave primar\u00eb p\u00ebr t\u00eb cil\u00ebt ka ardhur d\u00eblta. P\u00ebr m\u00eb tep\u00ebr, duhet t\u00eb rilexohet p\u00ebrs\u00ebri pjesa e pandryshuar e t\u00eb dh\u00ebnave. Megjithat\u00eb, kjo \u00ebsht\u00eb e nevojshme, pasi n\u00eb Hive nuk ka operacione t\u00eb fshirjes ose p\u00ebrdit\u00ebsimit.<\/p>\n
K\u00ebshtu, arrit\u00ebm n\u00eb p\u00ebrfundimin se n\u00eb rastin e Hive, MDW duhet t\u00eb b\u00ebj\u00eb p\u00ebrs\u00ebri t\u00eb gjith\u00eb tabel\u00ebn, sepse Hive nuk ka funksion p\u00ebrdit\u00ebsimi. Dhe asgj\u00eb m\u00eb e mir\u00eb se rileximi i plot\u00eb i t\u00eb dh\u00ebnave gjat\u00eb p\u00ebrdit\u00ebsimit nuk \u00ebsht\u00eb krijuar. N\u00eb rastin e RDBMS, p\u00ebrkundrazi, krijuesit e produktit e konsideruan t\u00eb nevojshme t\u00eb besojn\u00eb lidhjen dhe p\u00ebrdit\u00ebsimin e tabelave p\u00ebr p\u00ebrdorimin e SQL.<\/p>\n
P\u00ebr projektin n\u00eb Sberbank, ne krijuam nj\u00eb implementim t\u00eb ri t\u00eb rip\u00ebrdorsh\u00ebm t\u00eb ngarkuesit t\u00eb baz\u00ebs s\u00eb t\u00eb dh\u00ebnave p\u00ebr GreenPlum. Kjo ishte e bazuar n\u00eb versionin q\u00eb MDW gjeneron p\u00ebr Teradata. Pik\u00ebrisht Teradata, dhe jo Oracle, ishte m\u00eb e p\u00ebrshtatshme p\u00ebr k\u00ebt\u00eb, pasi gjithashtu \u00ebsht\u00eb nj\u00eb sistem MPP. M\u00ebnyrat e pun\u00ebs, si dhe sintaksa e Teradata dhe GreenPlum, rezultuan t\u00eb ken\u00eb ngjashm\u00ebri.<\/p>\n
Shembujt e dallimeve kritike p\u00ebr MDW midis RDBMS-ve t\u00eb ndryshme jan\u00eb si m\u00eb posht\u00eb. N\u00eb GreenPlum, ndryshe nga Teradata, kur krijoni tabela, duhet t\u00eb shkruani nj\u00eb klauzol\u00eb<\/p>\n
distributed by<\/code><\/pre>\n\nN\u00eb Teradata shkruhet<\/p>\n
fshij <table> të gjitha<\/code><\/pre>\n\n, nd\u00ebrsa n\u00eb GreenPlum shkruhet<\/p>\n
fshij nga <table><\/code><\/pre>\n\nN\u00eb Oracle, p\u00ebr q\u00ebllime optimizimi, shkruhet<\/p>\n
delete from t where rowid in ()<\/code><\/pre>\n\n, nd\u00ebrsa n\u00eb Teradata dhe GreenPlum shkruhet<\/p>\n
delete from t where exists (select * from delta where delta.pk=t.pk)<\/code><\/pre>\n\nGjithashtu, duhet t\u00eb theksojm\u00eb se p\u00ebr funksionimin e Ab Initio me GreenPlum, ishte e nevojshme t\u00eb instalonim klientin e GreenPlum n\u00eb t\u00eb gjitha nodet e klasterit Ab Initio. Kjo p\u00ebr shkak se ne u lidh\u00ebm me GreenPlum nj\u00ebher\u00ebsh nga t\u00eb gjitha nyjet e klasterit ton\u00eb. Dhe q\u00eb leximi nga GreenPlum t\u00eb ishte paralel dhe \u00e7do rrjedh\u00eb paralele e Ab Initio t\u00eb lexonte porcionin e saj t\u00eb t\u00eb dh\u00ebnave nga GreenPlum, duhej t\u00eb vendosnim n\u00eb seksionin \u201cku\u201d t\u00eb pyetjeve SQL nj\u00eb konstrukcion t\u00eb kuptuesh\u00ebm p\u00ebr Ab Initio<\/p>\n
where ABLOCAL()<\/code><\/pre>\n\ndhe t\u00eb p\u00ebrcaktonim vler\u00ebn e k\u00ebsaj konstrukcioni, duke e caktuar parametrin e transformimit q\u00eb lexon nga DB<\/p>\n
ablocal_expr=\u00abstring_concat(\"mod(t.\", string_filter_out(\"{$TABLE_KEY}\",\"{}\"), \",\", (decimal(3))(number_of_partitions()),\"=\" , (decimal(3))(this_partition()))\u00bb<\/code><\/pre>\n\n, e cila kompilohhet n\u00eb di\u00e7ka si<\/p>\n
mod(sk,10)=3<\/code><\/pre>\n\n, dmth, na duhet t\u00eb tregojm\u00eb GreenPlum nj\u00eb filt\u00ebr t\u00eb qart\u00eb p\u00ebr secil\u00ebn parti. P\u00ebr bazat e tjera t\u00eb t\u00eb dh\u00ebnave (Teradata, Oracle), Ab Initio mund ta kryej\u00eb k\u00ebt\u00eb paralelizim automatikisht.<\/p>\n
Karakteristikat krahasuese t\u00eb performanc\u00ebs s\u00eb Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum<\/h3>\n \nN\u00eb Sberbank u zhvillua nj\u00eb eksperiment p\u00ebr t\u00eb krahasuar performanc\u00ebn e grafik\u00ebve t\u00eb gjeneruar nga MDW n\u00eb lidhje me Hive dhe GreenPlum. N\u00eb kuad\u00ebr t\u00eb eksperimentit, n\u00eb rastin e Hive kishte 5 node n\u00eb t\u00eb nj\u00ebjtin klaster me Ab Initio, nd\u00ebrsa n\u00eb rastin e GreenPlum kishte 4 node n\u00eb nj\u00eb klaster t\u00eb ve\u00e7ant\u00eb. DMth, Hive kishte nj\u00eb avantazh t\u00eb caktuar mbi GreenPlum \u201cn\u00eb harduer\u201d.<\/p>\n
U shqyrtuan dy \u00e7ift grafesh q\u00eb realizonin t\u00eb nj\u00ebjtin detyr\u00eb t\u00eb p\u00ebrdit\u00ebsimit t\u00eb t\u00eb dh\u00ebnave n\u00eb Hive dhe n\u00eb GreenPlum. N\u00eb k\u00ebt\u00eb rast, u nis\u00ebn grafik\u00ebt e gjeneruar nga konfiguratori MDW:<\/p>\n
\nngarkesa fillestare + ngarkesa inkrimetuese e t\u00eb dh\u00ebnave t\u00eb rast\u00ebsishme t\u00eb gjeneruara n\u00eb tabel\u00ebn Hive<\/li>\n ngarkesa fillestare + ngarkesa inkrimetuese e t\u00eb dh\u00ebnave t\u00eb rast\u00ebsishme t\u00eb gjeneruara n\u00eb nj\u00eb tabel\u00eb t\u00eb ngjashme GreenPlum<\/li>\n<\/ul>\n\nN\u00eb t\u00eb dyja rastet (Hive dhe GreenPlum) u lan\u00e7uan ngarkesa n\u00eb 10 rrjedha paralele n\u00eb t\u00eb nj\u00ebjtin klaster Ab Initio. T\u00eb dh\u00ebnat nd\u00ebrmjet\u00ebse p\u00ebr llogaritjet Ab Initio u ruajt\u00ebn n\u00eb HDFS (n\u00eb terma t\u00eb Ab Initio u p\u00ebrdor MFS layout duke p\u00ebrdorur HDFS). Nj\u00eb rresht i t\u00eb dh\u00ebnave t\u00eb krijuara rast\u00ebsisht zinte 200 byte n\u00eb t\u00eb dyja rastet.<\/p>\n
Rezultati doli si m\u00eb posht\u00eb:<\/p>\n
Hive:<\/b><\/p>\n
Ngarkesa fillestare n\u00eb Hive<\/b><\/p>\n
Numri i rreshtave t\u00eb futur
Koh\u00ebzgjatja e ngarkes\u00ebs fillestare
Ngarkesa inkrementale n\u00eb Hive<\/b><\/p>\n
Numri i rreshtave q\u00eb ishin n\u00eb
Numri i rreshtave t\u00eb dyt\u00eb q\u00eb u aplikuan n\u00eb
Koh\u00ebzgjatja e ngarkes\u00ebs inkrementale2 541<\/b><\/p>\n
\nGreenPlum:<\/b><\/p>\n
Ngarkesa fillestare n\u00eb GreenPlum<\/b><\/p>\n
Numri i rreshtave t\u00eb futur
Koh\u00ebzgjatja e ngarkes\u00ebs fillestare
Ngarkesa inkrementale n\u00eb GreenPlum<\/b><\/p>\n
Numri i rreshtave q\u00eb ishin n\u00eb
Numri i rreshtave t\u00eb dyt\u00eb q\u00eb u aplikuan n\u00eb
Koh\u00ebzgjatja e ngarkes\u00ebs inkrementale321<\/b><\/p>\n
\nDuket se shpejt\u00ebsia e ngarkes\u00ebs fillestare si n\u00eb Hive ashtu edhe n\u00eb GreenPlum varet linearisht nga volumi i t\u00eb dh\u00ebnave dhe p\u00ebr shkak t\u00eb pajisjeve m\u00eb t\u00eb mira \u00ebsht\u00eb pak m\u00eb e shpejt\u00eb p\u00ebr Hive se p\u00ebr GreenPlum.<\/p>\n
Ngarkesa inkrementale n\u00eb Hive gjithashtu varet linearisht nga volumi i t\u00eb dh\u00ebnave t\u00eb ngarkuara m\u00eb par\u00eb n\u00eb tabelat e synuara dhe kalon mjaft ngadal\u00eb me rritjen e volumit. Kjo shkaktohet nga nevoja p\u00ebr t\u00eb riparuar plot\u00ebsisht tabel\u00ebn e synuar. Kjo do t\u00eb thot\u00eb se aplikimi i ndryshimeve t\u00eb vogla n\u00eb tabela t\u00eb m\u00ebdha nuk \u00ebsht\u00eb nj\u00eb opsion i mir\u00eb p\u00ebr p\u00ebrdorim n\u00eb Hive.<\/p>\n
Ngarkesa inkrementale n\u00eb GreenPlum varion pak nga volumi i t\u00eb dh\u00ebnave t\u00eb ngarkuara m\u00eb par\u00eb n\u00eb tabelat e synuara dhe kalon mjaft shpejt. Kjo arrihet fal\u00eb SQL Joins dhe arkitektur\u00ebs s\u00eb GreenPlum, e cila lejon operacionin delete.<\/p>\n
Pra, GreenPlum b\u00ebn derdhjen e diferenc\u00ebs duke p\u00ebrdorur metod\u00ebn delete+insert, nd\u00ebrsa n\u00eb Hive nuk ka operacione delete ose update, prandaj e gjith\u00eb masa e t\u00eb dh\u00ebnave n\u00eb p\u00ebrdit\u00ebsimin inkremental duhej t\u00eb shkruhej p\u00ebrs\u00ebri n\u00eb t\u00ebr\u00ebsi. M\u00eb e dukshme \u00ebsht\u00eb krahasimi i qelizave t\u00eb theksuara me ndihm\u00ebn e bold, pasi kjo p\u00ebrkon me variantin m\u00eb t\u00eb zakonsh\u00ebm t\u00eb shfryt\u00ebzimit t\u00eb ngarkesave q\u00eb k\u00ebrkojn\u00eb shum\u00eb burime. Shikojm\u00eb se GreenPlum fitoi nga Hive n\u00eb k\u00ebt\u00eb test me 8 her\u00eb.<\/p>\n
Puna e Ab Initio me GreenPlum n\u00eb modalitetin Near Real Time<\/h3>\n \nN\u00eb k\u00ebt\u00eb eksperiment do t\u00eb shqyrtojm\u00eb mund\u00ebsin\u00eb q\u00eb Ab Initio t\u00eb b\u00ebj\u00eb p\u00ebrdit\u00ebsimin e tabel\u00ebs GreenPlum me grupe t\u00eb dh\u00ebnash t\u00eb krijuara rast\u00ebsisht n\u00eb nj\u00eb mod t\u00eb af\u00ebrt me koh\u00ebn reale. Do t\u00eb shqyrtojm\u00eb tabel\u00ebn GreenPlum dev42_1_db_usl.TESTING_SUBJ_org_finval, me t\u00eb cil\u00ebn do t\u00eb punohet.<\/p>\n
Do t\u00eb p\u00ebrdorim tre grafe Ab Initio p\u00ebr t\u00eb punuar me t\u00eb:<\/p>\n
1) Grafi Create_test_data.mp \u2013 krijon n\u00eb 10 rrjedha paralele skedar\u00ebt me t\u00eb dh\u00ebna n\u00eb HDFS me 6 000 000 rreshta. T\u00eb dh\u00ebnat jan\u00eb rast\u00ebsore, struktura e tyre \u00ebsht\u00eb organizuar p\u00ebr futje n\u00eb tabel\u00ebn ton\u00eb.<\/p>\n
\ngjeneroni 6 000 000 rreshta testues.<\/li>\n b\u00ebni ngarkimin fillestar duke futur 6 000 000 rreshta testues n\u00eb nj\u00eb tabel\u00eb bosh.<\/li>\n p\u00ebrs\u00ebri 5 her\u00eb ngarkimin inkremental.\n\ngjeneroni 6 000 000 rreshta testues.<\/li>\n b\u00ebni nj\u00eb futur\u00eb inkrementale me 6 000 000 rreshta testues n\u00eb tabel\u00eb (n\u00eb k\u00ebt\u00eb rast, t\u00eb dh\u00ebnave t\u00eb vjetra u caktohet nj\u00eb koh\u00eb skadimi valid_to_ts dhe futen t\u00eb dh\u00ebna m\u00eb t\u00eb fresk\u00ebta me t\u00eb nj\u00ebjtin \u00e7el\u00ebs primar).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\nKy skenar imiton modin e funksionimit t\u00eb nj\u00eb sistemi t\u00eb biznesit real \u2013 n\u00eb koh\u00eb reale shfaqet nj\u00eb sasi e konsiderueshme t\u00eb dh\u00ebnash t\u00eb reja dhe menj\u00ebher\u00eb derdhen n\u00eb GreenPlum.<\/p>\n
Tani do t\u00eb shikojm\u00eb logun e pun\u00ebs s\u00eb skenarit:<\/p>\n
Start Create_test_data.input.pset n\u00eb 2020-06-04 11:49:11.
K\u00ebshtu duket kjo pamje:<\/p>\n
Grafiku
Krijo_test_data.input.pset
mdw_load.day_one.current.
Krijo_test_data.input.pset
mdw_load.regular.current.
Krijo_test_data.input.pset
mdw_load.regular.current.
Krijo_test_data.input.pset
mdw_load.regular.current.
Krijo_test_data.input.pset
mdw_load.regular.current.
Krijo_test_data.input.pset
mdw_load.regular.current.
\nV\u00ebrejm\u00eb se 6 000 000 rreshta inkrementi p\u00ebrpunohen brenda 3 minutash, q\u00eb \u00ebsht\u00eb mjaft shpejt.
select valid_from_ts, valid_to_ts, count(1), min(sk), max(sk) from dev42_1_db_usl.TESTING_SUBJ_org_finval group by valid_from_ts, valid_to_ts order by 1,2;<\/code><\/pre>\n\n
P\u00ebrfundim<\/h2>\n \nAktualisht Ab Initio p\u00ebrdoret n\u00eb Sberbank p\u00ebr nd\u00ebrtimin e Nj\u00eb Shtrese Semantike t\u00eb Dh\u00ebnash (ECS). Ky projekt parashikon nd\u00ebrtimin e nj\u00eb versioni t\u00eb vet\u00ebm t\u00eb gjendjes s\u00eb entiteteve t\u00eb ndryshme t\u00eb biznesit t\u00eb bank\u00ebs. Informacioni vjen nga burime t\u00eb ndryshme, riplikat e t\u00eb cilave p\u00ebrgatiten n\u00eb Hadoop. N\u00eb p\u00ebrputhje me nevojat e biznesit, p\u00ebrgatitet modeli i t\u00eb dh\u00ebnave dhe p\u00ebrshkruhen transformimet e t\u00eb dh\u00ebnave. Ab Initio ngarkon informacionin n\u00eb ECS dhe t\u00eb dh\u00ebnat e ngarkuara jo vet\u00ebm q\u00eb paraqesin interes p\u00ebr biznesin p\u00ebr vet\u00eb, por gjithashtu sh\u00ebrbejn\u00eb si burim p\u00ebr nd\u00ebrtimin e vitrinave t\u00eb t\u00eb dh\u00ebnave. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, funksionaliteti i produktit lejon p\u00ebrdorimin e sistemeve t\u00eb ndryshme si pranues (Hive, Greenplum, Teradata, Oracle), duke ofruar mund\u00ebsin\u00eb p\u00ebr t\u00eb p\u00ebrgatitur t\u00eb dh\u00ebnat p\u00ebr biznesin n\u00eb formate t\u00eb ndryshme q\u00eb k\u00ebrkohet.<\/p>\n
Mund\u00ebsit\u00eb e Ab Initio jan\u00eb t\u00eb shumta, p\u00ebr shembull, framework-u i bashkangjitur MDW ofron mund\u00ebsin\u00eb p\u00ebr t\u00eb nd\u00ebrtuar historikun teknik dhe t\u00eb biznesit t\u00eb t\u00eb dh\u00ebnave \"nga kutia\". P\u00ebr zhvilluesit Ab Initio ofron mund\u00ebsin\u00eb q\u00eb \"t\u00eb mos shpikin bi\u00e7iklet\u00ebn\", por t\u00eb p\u00ebrdorin nj\u00eb s\u00ebr\u00eb komponentesh funksionale t\u00eb gatshme, t\u00eb cilat jan\u00eb n\u00eb thelb biblioteka t\u00eb nevojshme p\u00ebr pun\u00ebn me t\u00eb dh\u00ebnat.<\/p>\n
Autori - ekspert i komunitetit profesional t\u00eb Sberbank SberProfi DWH\/BigData. Komuniteti profesional SberProfi DWH\/BigData \u00ebsht\u00eb p\u00ebrgjegj\u00ebs p\u00ebr zhvillimin e kompetencave n\u00eb fusha t\u00eb tilla si ekosistemi Hadoop, Teradata, Oracle DB, GreenPlum, si dhe n\u00eb mjetet BI Qlik, SAP BO, Tableau dhe t\u00eb tjera.<\/p><\/blockquote>\n
Burimi: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438. \u0415\u0451 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 \u0441\u0432\u0435\u043b\u043e\u0441\u044c \u043a \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a\u0443 \u043f\u043e \u0437\u0430\u043f\u0443\u0441\u043a\u0443 \u043a\u043e\u043c\u0430\u043d\u0434 spark-submit. \u041d\u0430 \u0440\u044b\u043d\u043a\u0435 \u0438\u043c\u0435\u043b\u043e\u0441\u044c \u043d\u0435 \u0442\u0430\u043a \u043c\u043d\u043e\u0433\u043e \u0430\u043d\u0430\u043b\u043e\u0433\u043e\u0432, \u0432 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0435 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u044b\u0445 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u0442\u0435\u043c \u043e\u0431\u044a\u0451\u043c\u043e\u043c \u0434\u0430\u043d\u043d\u044b\u0445, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u043c\u044b \u0438\u043c\u0435\u0435\u043c \u0434\u0435\u043b\u043e \u043a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c. […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":87610,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-87609","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t