{"id":90783,"date":"2020-08-06T01:42:19","date_gmt":"2020-08-05T23:42:19","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1"},"modified":"2020-08-06T01:42:19","modified_gmt":"2020-08-05T23:42:19","slug":"kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1","title":{"rendered":"Kuidas Google'i BigQuery andmete anal\u00fc\u00fcsi demokratiseeris. Osa 1","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><b><i>Tere, Habr! Praegu on OTUSis avatud uus kursusegrupp. <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/wDZQ\/\">\u201eAndmeinsener\u201c<\/a><\/noindex>Kursuse algusele eelnevalt oleme traditsiooniliselt valmistanud teile huvitava materjali t\u00f5lke. <\/i><\/b><\/p>\n<p>\nIga p\u00e4ev k\u00fclastab Twitterit \u00fcle saja miljoni inimese, et teada saada, mis maailmas toimub, ja arutada neid teemasid. Iga tweet ja igasugune muu kasutaja tegevus genereerib s\u00fcndmuse, mis on Twitteri sisese andmeanal\u00fc\u00fcsi jaoks saadaval. Sajad t\u00f6\u00f6tajad anal\u00fc\u00fcsivad ja visualiseerivad neid andmeid, ning nende kogemuse parandamine on Twitteri andmeplatvormi meeskonna peamine prioriteet. <\/p>\n<p>Usume, et erineva tehniliste oskustega kasutajad peavad olema v\u00f5imelised leidma andmeid ja p\u00e4\u00e4sema h\u00e4sti toimivatele SQL-p\u00f5histe anal\u00fc\u00fcsi- ja visualiseerimist\u00f6\u00f6riistadele. See v\u00f5imaldaks t\u00e4iesti uuel kasutajagrupil, kellel on v\u00e4hem tehniline taust, sealhulgas andmeanal\u00fc\u00fctikutel ja tootmeeskonna juhtidel, andmetest teavet v\u00e4lja tuua, v\u00f5imaldades neil paremini m\u00f5ista ja kasutada Twitteri v\u00f5imalusi. Nii democratiseerime andmeanal\u00fc\u00fcsi Twitteris. <noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Aja meie t\u00f6\u00f6riistade ja sisemiste andmeanal\u00fc\u00fcsi v\u00f5imaluste t\u00e4iustamisega oleme olnud tunnistajaks Twitteri teenuse paranemisele. Sellegipoolest on veel palju ruumi kasvada. Praegused t\u00f6\u00f6riistad, nagu Scalding, n\u00f5uavad programmeerimiskogemust. SQL-p\u00f5hised anal\u00fc\u00fcsi t\u00f6\u00f6riistad, nagu Presto ja Vertica, kohtavad suurte andmemahtude korral j\u00f5udlusprobleeme. Meil on ka probleem andmete jaotamisega mitmete s\u00fcsteemide vahel ilma pideva ligip\u00e4\u00e4suta neile.<\/p>\n<p>Eelmisel aastal kuulutasime v\u00e4lja <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/infrastructure\/2018\/a-new-collaboration-with-google-cloud.html\">uue koost\u00f6\u00f6 Google'iga<\/a><\/noindex>, mille raames viime osa meie <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/infrastructure\/2019\/the-start-of-a-journey-into-the-cloud.html\">andme-infrastruktuurist<\/a><\/noindex> Google Cloud Platformile (GCP). Oleme j\u00f5udnud j\u00e4reldusele, et Google Cloudi t\u00f6\u00f6riistade <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/products\/big-data\/\">Big Data<\/a><\/noindex> abi aitab meil edendada anal\u00fc\u00fcsi, visualiseerimise ja masin\u00f5ppe demokraatimist Twitteris:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/\">BigQuery<\/a><\/noindex>: ettev\u00f5tte andmesalvestus SQL-mootoriga, mis p\u00f5hineb <noindex><a rel=\"nofollow\" href=\"https:\/\/ai.google\/research\/pubs\/pub36632\">Dremelil<\/a><\/noindex>, mis on tuntud oma kiirusest, lihtsusest ja suudab toime tulla <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery-ml\/docs\/bigqueryml-intro\">masin\u00f5ppega.<\/a><\/noindex>.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/datastudio.google.com\/overview\">Data Studio:<\/a><\/noindex> t\u00f6\u00f6riist suurte andmete visualiseerimiseks, mis pakub koost\u00f6\u00f6funktsioone nagu Google Docsis.<\/li>\n<\/ul>\n<p>\nSelles artiklis r\u00e4\u00e4gime oma kogemustest nende t\u00f6\u00f6riistadega: mida me oleme teinud, mida \u00f5ppinud ja mida plaanime teha j\u00e4rgmiseks. Praegu keskendume pakendatud ja interaktiivsele anal\u00fc\u00fcsile. Reaalajas anal\u00fc\u00fcsi arutame j\u00e4rgmises artiklis.<\/p>\n<h2>Andmehoidlate ajalugu Twitteris<\/h2>\n<p>\nEnne kui s\u00fcveneme BigQuery'sse, tasub l\u00fchidalt kokku v\u00f5tta andmehoidlate ajalugu Twitteris. 2011. aastal tehti Twitteris andmeanal\u00fc\u00fcse Verticas ja Hadoopis. MapReduce'i loomiseks kasutatud Hadoopi t\u00f6\u00f6de jaoks kasutasime Pigi. 2012. aastal asendasime Pigi Scaldinguga, millel oli Scala API, mis pakkus eeliseid nagu keerukate torujuhtmete loomise v\u00f5imalus ja testimise lihtsus. Siiski oli paljudele andmeanal\u00fc\u00fctikutele ja tootemanageritele, kes olid harjunud t\u00f6\u00f6tama SQL-iga, see siiski piisavalt j\u00e4rsk \u00f5ppimisk\u00f5ver. Umbes 2016. aastal hakkasime kasutama Presto't SQL-iga Hadoopi andmete liidese jaoks. Spark pakkus Python'i liidest, mis teeb selle heaks valikuks ad hoc andmeuuringute ja masin\u00f5ppe jaoks.<\/p>\n<p>Alates 2018. aastast oleme andmete anal\u00fc\u00fcsimiseks ja visualiseerimiseks kasutanud j\u00e4rgmisi t\u00f6\u00f6riistu:<\/p>\n<ul>\n<li>Scalding tootmisliinidele<\/li>\n<li>Scalding ja Spark ad hoc andmeanal\u00fc\u00fcsiks ja masin\u00f5ppeks<\/li>\n<li>Vertica ja Presto ad hoc ja interaktiivseks SQL-anal\u00fc\u00fcsiks <\/li>\n<li>Druid v\u00e4ikese interaktiivse, uuriva ja madala latentsusega juurdep\u00e4\u00e4suks ajasari andmetele<\/li>\n<li>Tableau, Zeppelin ja Pivot andmete visualiseerimiseks<\/li>\n<\/ul>\n<p>\nOleme avastanud, et kuigi need t\u00f6\u00f6riistad pakuvad v\u00e4ga v\u00f5imsaid v\u00f5imalusi, oleme kogenud raskusi nende v\u00f5imaluste k\u00e4ttesaadavuse rakendamisega laiemale publikule Twitteris. Laiendades oma platvormi Google Cloudi abil, keskendume oma anal\u00fc\u00fctika t\u00f6\u00f6riistade lihtsustamisele kogu Twitteri ulatuses.<\/p>\n<h2>Google'i BigQuery andmetehoidla <\/h2>\n<p>\nM\u00f5ned meeskonnad Twitteris on juba lisanud BigQuery oma tootmisprotsessidesse. Kasutades nende kogemusi, oleme hakanud hindama BigQuery v\u00f5imalusi k\u00f5ikide Twitteri kasutusjuhtude jaoks. Meie eesm\u00e4rk oli pakkuda BigQueryd kogu ettev\u00f5ttele ning ka standardiseerida ja toetada seda Data Platformi t\u00f6\u00f6riistade kogumina. See oli mitmel p\u00f5hjusel keeruline. Me pidime looma infrastruktuuri usaldusv\u00e4\u00e4rsete suurte andmemahtude vastuv\u00f5tmiseks, toetama ettev\u00f5tte tasandi andmehaldust, tagama n\u00f5uetekohase ligip\u00e4\u00e4su kontrolli ja kliendi konfidentsiaalsuse tagamise. Samuti pidime looma s\u00fcsteemid ressursside jaotamiseks, j\u00e4lgimiseks ja tasude tagastamiseks, et meeskonnad saaksid BigQueryd t\u00f5husalt kasutada.<\/p>\n<p>Novembris 2018. aastal vabastasime BigQuery ja Data Studio alfarelise kogu ettev\u00f5tte jaoks. Pakume Twitteri t\u00f6\u00f6tajatele m\u00f5ned meie k\u00f5ige sagedamini kasutatavad tabelid, milles on puhastatud isikuandmed. BigQuery-d kasutas rohkem kui 250 kasutajat erinevatest meeskondadest, sealhulgas inseneri-, finants- ja turundusosakondadest. \u00c4sja viidi l\u00e4bi umbes 8000 p\u00e4ringut, t\u00f6\u00f6tledes umbes 100 PB kuus, v\u00e4lja arvatud ajastatud p\u00e4ringud. Saades v\u00e4ga positiivset tagasisidet, otsustasime edasi liikuda ja pakkuda BigQuery-d peamise vahendina andmete haldamiseks Twitteris.<\/p>\n<p>Siin on meie Google BigQuery andmehoidla k\u00f5rgetasemeline arhitektuuri skeem. <\/p>\n<p><img decoding=\"async\" alt=\"Kuidas Google&#039;i BigQuery andmete anal\u00fc\u00fcsi demokratiseeris. Osa 1\" src=\"\/wp-content\/uploads\/2020\/08\/6a3e0b43752ebbac641c43519c8c2fba.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nKopeerime andmed kohalikest Hadoopi klusteritest Google Cloud Storage'i (GCS), kasutades sisemist t\u00f6\u00f6riista Cloud Replicator. Seej\u00e4rel kasutame Apache Airflow'd torude loomiseks, mis kasutavad \u201e<noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/docs\/reference\/bq-cli-reference#bq_load\">bq_load<\/a><\/noindex>\u201c andmete laadimiseks GCS-ist BigQuery-sse. Kasutame Presto't Parquet v\u00f5i Thrift-LZO andmekogumite p\u00e4rimise jaoks GCS-is. BQ Blaster on sisemine Scalding t\u00f6\u00f6riist HDFS Vertica ja Thrift-LZO andmekogumite laadimiseks BigQuery-sse.<\/p>\n<p>J\u00e4rgmistes jaotistes arutame meie l\u00e4henemist ja teadmisi kasutatavuse, j\u00f5udluse, andmehaldamise, s\u00fcsteemi t\u00f6\u00f6kindluse ja kulude osas.<\/p>\n<h4>Kasutusmugavus<\/h4>\n<p>\nOleme avastanud, et BigQuery kasutamine oli kasutajatele lihtne, kuna see ei n\u00f5udnud tarkvara installimist ja kasutajad p\u00e4\u00e4sesid sellele ligi intuitiivse veebiliidese kaudu. Siiski pidi kasutajad tutvuma m\u00f5nede GCP funktsioonide ja kontseptidega, sealhulgas ressursside nagu projektid, andmekogud ja tabelid. Oleme v\u00e4lja t\u00f6\u00f6tanud \u00f5ppematerjalid ja juhendid, et aidata kasutajatel alustada. P\u00f5hiteadmiste omandamine aitas kasutajatel andmekogudes navigeerida, vaadata tabelite skeemi ja andmeid, sooritada lihtsaid p\u00e4ringuid ning visualiseerida tulemusi Data Studios.<\/p>\n<p>Meie eesm\u00e4rk andmete sisestamisel BigQuerys oli tagada andmekogude HDFS v\u00f5i GCS sujuv \u00fcleslaadimine \u00fche klikiga. Me k\u00e4sitlesime <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/composer\/\">Cloud Composer<\/a><\/noindex> (hallatav Airflow), kuid ei suutnud seda kasutada meie \u201eDomain Restricted Sharing\u201d turvamudeli t\u00f5ttu (lisainfot leiate altpoolt jaotises \u201eAndmete haldamine\u201d). Katsetasime Google Data Transfer Service'i (DTS) kasutamist BigQuery koormust\u00f6\u00f6de korraldamiseks. Kuigi DTS seadistamine oli kiire, ei olnud see paindlik s\u00f5ltuvustega konveierite loomisel. Meie alfa versiooni jaoks l\u00f5ime oma Apache Airflow keskkonna GCE-s ja valmistame selle ette tootmisse viimiseks ning suudame toetada rohkem andmeallikaid, nagu Vertica.<\/p>\n<p>BigQuery-sse andmete edastamiseks loovad kasutajad lihtsaid SQL-andmepipe, kasutades ajastatud p\u00e4ringuid. Kompleksete mitmeastmeliste konveierite jaoks, kus on s\u00f5ltuvused, plaanime kasutada kas meie oma Airflow infrastruktuuri v\u00f5i Cloud Composerit koos <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/dataflow\/\">Cloud Dataflow<\/a><\/noindex>.<\/p>\n<h4>Tootlikkus<\/h4>\n<p>\nBigQuery on loodud \u00fcldotstarbeliste SQL-p\u00e4ringute jaoks, mis t\u00f6\u00f6tlevad suuri andmekoguseid. Seda ei ole m\u00f5eldud madala latentsusega p\u00e4ringuteks, mis n\u00f5uavad suurt l\u00e4bilaskvust, mis on vajalikud tehinguandmebaasi jaoks, ega madala latentsusega ajaseeria anal\u00fc\u00fcsi jaoks. <noindex><a rel=\"nofollow\" href=\"https:\/\/druid.apache.org\/\">Apache Druid<\/a><\/noindex>. Interaktiivsete anal\u00fc\u00fcsik\u00fcsimuste jaoks ootavad meie kasutajad alla \u00fche minuti vastusaega. Olime pidanud kavandama BigQuery kasutamise viisi, mis vastab nende ootustele. Et tagada meie kasutajatele ettearvatav j\u00f5udlus, kasutasime BigQuery funktsionaalsust, mis on saadaval fikseeritud tasu alusel klientidele, mis v\u00f5imaldab projektide omanikitel reserveerida minimaalsed plokid oma p\u00e4ringute jaoks. <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/docs\/slots\">Plokk<\/a><\/noindex> BigQuery on arvutusv\u00f5imsuse \u00fchik, mis on vajalik SQL p\u00e4ringute t\u00e4itmiseks. <\/p>\n<p>Oleme anal\u00fc\u00fcsinud rohkem kui 800 p\u00e4ringut, mis t\u00f6\u00f6tlevad umbes 1 TB andmeid, ja avastanud, et keskmine t\u00e4itmise aeg oli 30 sekundit. Samuti selgus, et t\u00f6\u00f6tluse tulemuslikkus s\u00f5ltub tugevasti meie sloti kasutamisest erinevates projektides ja \u00fclesannetes. Pidi olema selgelt eristatud meie tootmis- ja ad hoc slotide reservid, et hoida tootlikkust tootmistsenaariumide ja interaktiivse anal\u00fc\u00fcsi jaoks. See avaldas suurt m\u00f5ju meie disainile slotide reserveerimise ja projektide hierarhia osas.<\/p>\n<p><i>Andmehalduse, funktsionaalsuse ja s\u00fcsteemide kulude kohta r\u00e4\u00e4gime juba l\u00e4hiajal t\u00f5lke teises osas, aga praegu kutsume k\u00f5iki huvilisi <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/wDZQ\/\">tasuta otse\u00fclekandele<\/a><\/noindex>, kus on v\u00f5imalus p\u00f5hjalikult tutvuda kursusega ning esitada k\u00fcsimusi meie eksperdile \u2014 Egor Mateshukile (kogenud andmeinsener, MaximaTelecom). <\/i><\/p>\n<p><\/p>\n<h2>Loe edasi:<\/h2>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/501380\/\">Data Build Tool v\u00f5i mis seondub Andmehoidla ja Smuutiga<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">Sissejuhatus Delta Lake'i: sundrakendamine ja skeemi evolutsioon<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/503132\/\">K\u00f5rge kiirus Apache Parquet Pythonis koos Apache Arrow<\/a><\/noindex><\/li>\n<\/ul>\n<p>Allikas: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/513780\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043c\u044b \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u0438\u043b\u0438 \u0434\u043b\u044f \u0432\u0430\u0441 \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u043e\u0433\u043e \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0430. \u041a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c \u0431\u043e\u043b\u0435\u0435 \u0441\u0442\u0430 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 \u0447\u0435\u043b\u043e\u0432\u0435\u043a \u043f\u043e\u0441\u0435\u0449\u0430\u044e\u0442 Twitter, \u0447\u0442\u043e\u0431\u044b \u0443\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u0432 \u043c\u0438\u0440\u0435, \u0438 \u043e\u0431\u0441\u0443\u0434\u0438\u0442\u044c \u044d\u0442\u043e. \u041a\u0430\u0436\u0434\u044b\u0439 \u0442\u0432\u0438\u0442 \u0438 \u043b\u044e\u0431\u043e\u0435 \u0434\u0440\u0443\u0433\u043e\u0435 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u044e\u0442 \u0441\u043e\u0431\u044b\u0442\u0438\u0435, \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0435 \u0434\u043b\u044f \u0432\u043d\u0443\u0442\u0440\u0435\u043d\u043d\u0435\u0433\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":90784,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-90783","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043c\u044b \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u0438\u043b\u0438 \u0434\u043b\u044f \u0432\u0430\u0441 \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u043e\u0433\u043e \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0430. \u041a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c \u0431\u043e\u043b\u0435\u0435 \u0441\u0442\u0430 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 \u0447\u0435\u043b\u043e\u0432\u0435\u043a \u043f\u043e\u0441\u0435\u0449\u0430\u044e\u0442 Twitter, \u0447\u0442\u043e\u0431\u044b \u0443\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u0432 \u043c\u0438\u0440\u0435, \u0438 \u043e\u0431\u0441\u0443\u0434\u0438\u0442\u044c \u044d\u0442\u043e. \u041a\u0430\u0436\u0434\u044b\u0439 \u0442\u0432\u0438\u0442 \u0438 \u043b\u044e\u0431\u043e\u0435 \u0434\u0440\u0443\u0433\u043e\u0435 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u044e\u0442 \u0441\u043e\u0431\u044b\u0442\u0438\u0435, \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0435 \u0434\u043b\u044f \u0432\u043d\u0443\u0442\u0440\u0435\u043d\u043d\u0435\u0433\u043e\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"et_EE\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 1 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043c\u044b \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u0438\u043b\u0438 \u0434\u043b\u044f \u0432\u0430\u0441 \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u043e\u0433\u043e \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0430. \u041a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c \u0431\u043e\u043b\u0435\u0435 \u0441\u0442\u0430 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 \u0447\u0435\u043b\u043e\u0432\u0435\u043a \u043f\u043e\u0441\u0435\u0449\u0430\u044e\u0442 Twitter, \u0447\u0442\u043e\u0431\u044b \u0443\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u0432 \u043c\u0438\u0440\u0435, \u0438 \u043e\u0431\u0441\u0443\u0434\u0438\u0442\u044c \u044d\u0442\u043e. \u041a\u0430\u0436\u0434\u044b\u0439 \u0442\u0432\u0438\u0442 \u0438 \u043b\u044e\u0431\u043e\u0435 \u0434\u0440\u0443\u0433\u043e\u0435 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u044e\u0442 \u0441\u043e\u0431\u044b\u0442\u0438\u0435, \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0435 \u0434\u043b\u044f \u0432\u043d\u0443\u0442\u0440\u0435\u043d\u043d\u0435\u0433\u043e\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-05T23:42:19+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-05T23:42:19+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kuidas Google'i BigQuery on andmeanal\u00fc\u00fcsi demokratiseerinud. Osa 1 | ProHoster","description":"Tere, Habr! Just n\u00fc\u00fcd on OTUS avatud uue \"Data Engineer\" kursuse komplekti registreerimiseks. Kursuse alguse eel\u00f5htul oleme traditsiooniliselt valmistanud teile huvitava materjali t\u00f5lke. Igal p\u00e4eval k\u00fclastab Twitterit \u00fcle saja miljoni inimese, et teada saada, mis maailmas toimub, ja arutada seda. Iga tweet ja iga kasutaja tegevus genereerib s\u00fcndmuse, mis on kergesti k\u00e4ttesaadav.","canonical_url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"et_EE","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 1 | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043c\u044b \u0442\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u0438\u043b\u0438 \u0434\u043b\u044f \u0432\u0430\u0441 \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u043e\u0433\u043e \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0430. \u041a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c \u0431\u043e\u043b\u0435\u0435 \u0441\u0442\u0430 \u043c\u0438\u043b\u043b\u0438\u043e\u043d\u043e\u0432 \u0447\u0435\u043b\u043e\u0432\u0435\u043a \u043f\u043e\u0441\u0435\u0449\u0430\u044e\u0442 Twitter, \u0447\u0442\u043e\u0431\u044b \u0443\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u0432 \u043c\u0438\u0440\u0435, \u0438 \u043e\u0431\u0441\u0443\u0434\u0438\u0442\u044c \u044d\u0442\u043e. \u041a\u0430\u0436\u0434\u044b\u0439 \u0442\u0432\u0438\u0442 \u0438 \u043b\u044e\u0431\u043e\u0435 \u0434\u0440\u0443\u0433\u043e\u0435 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u044e\u0442 \u0441\u043e\u0431\u044b\u0442\u0438\u0435, \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0435 \u0434\u043b\u044f \u0432\u043d\u0443\u0442\u0440\u0435\u043d\u043d\u0435\u0433\u043e","og:url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-1","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-05T23:42:19+00:00","article:modified_time":"2020-08-05T23:42:19+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"90783","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:42:25","updated":"2022-10-03 03:00:18"},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/90783","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/comments?post=90783"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/90783\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media\/90784"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media?parent=90783"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/categories?post=90783"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/tags?post=90783"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}