{"id":91903,"date":"2020-08-20T07:42:18","date_gmt":"2020-08-20T05:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2"},"modified":"2020-08-20T07:42:18","modified_gmt":"2020-08-20T05:42:18","slug":"kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","title":{"rendered":"Jak Google BigQuery demokratizoval anal\u00fdzu dat. \u010c\u00e1st 2","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><b><i>Ahoj, Habr! Pr\u00e1v\u011b te\u010f je v OTUS otev\u0159en\u00e1 registrace na nov\u00fd b\u011bh kurzu <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/wDZQ\/\">\u201eDatov\u00fd in\u017een\u00fdr\u201c<\/a><\/noindex>. V p\u0159edve\u010der spu\u0161t\u011bn\u00ed kurzu pokra\u010dujeme v sd\u00edlen\u00ed u\u017eite\u010dn\u00fdch materi\u00e1l\u016f. <\/i><\/b><\/p>\n<p><i><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/\">\u010c\u00edst prvn\u00ed \u010d\u00e1st<\/a><\/noindex><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Jak Google BigQuery demokratizoval anal\u00fdzu dat. \u010c\u00e1st 2\" src=\"\/wp-content\/uploads\/2020\/08\/7b9314bffd954cd408807280f8956c21.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h4>Spr\u00e1va dat<\/h4>\n<p>\nSiln\u00e1 spr\u00e1va dat (Strong Data Governance) je z\u00e1kladn\u00ed princip Twitter Engineering. Proto\u017ee implementujeme BigQuery na na\u0161i platformu, zam\u011b\u0159ujeme se na objevov\u00e1n\u00ed dat, kontrolu p\u0159\u00edstupu, bezpe\u010dnost a ochranu soukrom\u00ed.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Pro objevov\u00e1n\u00ed a spr\u00e1vu dat jsme roz\u0161\u00ed\u0159ili na\u0161i \u00farove\u0148 p\u0159\u00edstupu k dat\u016fm (Data Access Layer \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), abychom poskytli n\u00e1stroje jak pro m\u00edstn\u00ed data, tak pro data Google Cloud, p\u0159i\u010dem\u017e nab\u00edz\u00edme jednotn\u00e9 rozhran\u00ed a API na\u0161im u\u017eivatel\u016fm. Jak Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/data-catalog\/\">Data Catalog<\/a><\/noindex> se posouv\u00e1 k ve\u0159ejnosti, za\u0159ad\u00edme ho do na\u0161ich projekt\u016f, abychom u\u017eivatel\u016fm nab\u00eddli funkce, jako je vyhled\u00e1v\u00e1n\u00ed podle sloupc\u016f.<\/p>\n<p>BigQuery umo\u017e\u0148uje snadn\u00e9 sd\u00edlen\u00ed dat a p\u0159\u00edstup k nim, ale museli jsme do ur\u010dit\u00e9 m\u00edry tento proces kontrolovat, abychom zabr\u00e1nili exfiltraci dat. Mezi dal\u0161\u00edmi n\u00e1stroji jsme vybrali dv\u011b funkce:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/resource-manager\/docs\/organization-policy\/restricting-domains\">Domain restricted sharing<\/a><\/noindex>: beta funkce, kter\u00e1 zakazuje u\u017eivatel\u016fm sd\u00edlet datov\u00e9 sady BigQuery s u\u017eivateli mimo Twitter.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/vpc-service-controls\/\">VPC service controls<\/a><\/noindex>: prvek zabezpe\u010den\u00ed, kter\u00fd br\u00e1n\u00ed exfiltraci dat a vy\u017eaduje, aby u\u017eivatel\u00e9 m\u011bli p\u0159\u00edstup k BigQuery z ur\u010dit\u00fdch IP adres.<\/li>\n<\/ul>\n<p>\nImplementovali jsme po\u017eadavky na autentizaci, autorizaci a audit (AAA) pro zaji\u0161t\u011bn\u00ed bezpe\u010dnosti n\u00e1sleduj\u00edc\u00edm zp\u016fsobem:<\/p>\n<ul>\n<li> Autentizace: pou\u017eili jsme u\u017eivatelsk\u00e9 \u00fa\u010dty GCP pro ad hoc dotazy a \u00fa\u010dty slu\u017eeb pro pracovn\u00ed dotazy.<\/li>\n<li>Autorizace: vy\u017eadovali jsme, aby ka\u017ed\u00e1 datov\u00e1 sada m\u011bla \u00fa\u010det slu\u017eby vlastn\u00edka a skupinu \u010dten\u00e1\u0159\u016f.<\/li>\n<li>Audit: exportovali jsme protokoly BigQuery Stackdriver, kter\u00e9 obsahovaly podrobnosti o proveden\u00fdch dotazech, do datov\u00e9 sady BigQuery pro snadnou anal\u00fdzu.<\/li>\n<\/ul>\n<p>\nAbychom zajistili \u0159\u00e1dn\u00e9 zpracov\u00e1n\u00ed osobn\u00edch \u00fadaj\u016f u\u017eivatel\u016f Twitteru, mus\u00edme registrovat v\u0161echny datov\u00e9 sady BigQuery, anotovat osobn\u00ed \u00fadaje, udr\u017eovat \u0159\u00e1dn\u00e9 uchov\u00e1v\u00e1n\u00ed a odstra\u0148ovat (\u010distit) data, kter\u00e1 byla u\u017eivateli smaz\u00e1na.<\/p>\n<p>Zva\u017eovali jsme Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/dlp\/\">Cloud Data Loss Prevention API<\/a><\/noindex>, kter\u00e1 pou\u017e\u00edv\u00e1 strojov\u00e9 u\u010den\u00ed k klasifikaci a \u00faprav\u011b citliv\u00fdch \u00fadaj\u016f, ale rozhodli jsme se pro ru\u010dn\u00ed anotaci datov\u00e9ho souboru kv\u016fli p\u0159esnosti. Pl\u00e1nujeme pou\u017e\u00edt API pro prevenci ztr\u00e1ty dat, abychom doplnili u\u017eivatelskou anotaci.<\/p>\n<p>Na Twitteru jsme vytvo\u0159ili \u010dty\u0159i kategorie d\u016fv\u011brnosti pro datov\u00e9 soubory v BigQuery, uveden\u00e9 zde v sestupn\u00e9m po\u0159ad\u00ed citlivosti:<\/p>\n<ul>\n<li>Vysoce citliv\u00e9 datov\u00e9 soubory jsou dostupn\u00e9 na z\u00e1klad\u011b z\u00e1sady nejmen\u0161\u00edch privilegijn\u00edch pr\u00e1v. Ka\u017ed\u00fd datov\u00fd soubor m\u00e1 zvl\u00e1\u0161tn\u00ed skupinu \u010dten\u00e1\u0159\u016f a budeme sledovat pou\u017e\u00edv\u00e1n\u00ed jednotliv\u00fdch \u00fa\u010dt\u016f.<\/li>\n<li>Datov\u00e9 soubory st\u0159edn\u00ed citlivosti (jednosm\u011brn\u00e9 pseudonymy s pou\u017eit\u00edm solen\u00e9 hashov\u00e1n\u00ed) neobsahuj\u00ed osobn\u00ed informace (Personally Identifiable Information \u2013 PII) a jsou k dispozici pro \u0161ir\u0161\u00ed skupinu zam\u011bstnanc\u016f. To p\u0159edstavuje dobrou rovnov\u00e1hu mezi ohledy na d\u016fv\u011brnost a u\u017eite\u010dnost dat. Umo\u017e\u0148uje to zam\u011bstnanc\u016fm prov\u00e1d\u011bt analytick\u00e9 \u00fakoly, jako je v\u00fdpo\u010det po\u010dtu u\u017eivatel\u016f, kte\u0159\u00ed pou\u017eili funkci, ani\u017e by v\u011bd\u011bli, kdo jsou skute\u010dn\u00ed u\u017eivatel\u00e9.<\/li>\n<li>Datov\u00e9 soubory s n\u00edzkou citlivost\u00ed obsahuj\u00ed ve\u0161ker\u00e9 informace, kter\u00e9 identifikuj\u00ed u\u017eivatele. To je dobr\u00fd p\u0159\u00edstup z hlediska d\u016fv\u011brnosti, ale nelze ho pou\u017e\u00edt pro anal\u00fdzu na \u00farovni u\u017eivatele.<\/li>\n<li>Ve\u0159ejn\u00e9 datov\u00e9 soubory (vydan\u00e9 mimo Twitter) jsou dostupn\u00e9 v\u0161em zam\u011bstnanc\u016fm Twitteru.<\/li>\n<\/ul>\n<p>\nPokud jde o registraci, pou\u017eili jsme napl\u00e1novan\u00e9 \u00fakoly k v\u00fdpisu datov\u00fdch soubor\u016f BigQuery a jejich registraci do vrstvy p\u0159\u00edstupu k dat\u016fm (<noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), \u00falo\u017ei\u0161t\u011b metadat Twitteru. U\u017eivatel\u00e9 budou anotovat datov\u00e9 soubory informacemi o d\u016fv\u011brnosti a tak\u00e9 uv\u00e1d\u011bt dobu uchov\u00e1v\u00e1n\u00ed. Pokud jde o \u010di\u0161t\u011bn\u00ed, hodnot\u00edme v\u00fdkon a n\u00e1klady obou mo\u017enost\u00ed: <i>1.<\/i> \u00daklid datov\u00fdch soubor\u016f v GCS pomoc\u00ed n\u00e1stroj\u016f, jako je Scalding, a jejich nahr\u00e1v\u00e1n\u00ed do BigQuery; <i>2.<\/i> Pou\u017eit\u00ed p\u0159\u00edkaz\u016f DML BigQuery. Pravd\u011bpodobn\u011b pou\u017eijeme kombinaci obou metod, abychom splnili po\u017eadavky r\u016fzn\u00fdch skupin a dat.<\/p>\n<h4>Funk\u010dnost syst\u00e9mu<\/h4>\n<p>\nJeliko\u017e je BigQuery spravovanou slu\u017ebou, nebylo pot\u0159eba zapojovat t\u00fdm SRE z Twitteru do spr\u00e1vy syst\u00e9m\u016f nebo pln\u011bn\u00ed pohotovostn\u00edch povinnost\u00ed. Bylo snadn\u00e9 zajistit velkou kapacitu jak pro \u00falo\u017ei\u0161t\u011b, tak pro v\u00fdpo\u010detn\u00ed v\u00fdkon. Mohli jsme zm\u011bnit rezervaci slot\u016f vytv\u00e1\u0159en\u00edm tiket\u016f na podpo\u0159e Google. Ur\u010dili jsme, \u017ee se d\u00e1 vylep\u0161it nap\u0159\u00edklad samoobsluha pro rozd\u011blen\u00ed slot\u016f a vylep\u0161en\u00ed \u0159\u00eddic\u00edho panelu pro monitoring, a tyto po\u017eadavky jsme p\u0159edali Googlu.<\/p>\n<h4>N\u00e1klady<\/h4>\n<p>\nNa\u0161e p\u0159edb\u011b\u017en\u00e1 anal\u00fdza uk\u00e1zala, \u017ee n\u00e1klady na dotazy pro BigQuery a Presto byly na stejn\u00e9 \u00farovni. Zakoupili jsme sloty za <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#flat_rate_pricing\">pevnou<\/a><\/noindex> cenu, abychom m\u011bli stabiln\u00ed m\u011bs\u00ed\u010dn\u00ed n\u00e1klady m\u00edsto placen\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#on_demand_pricing\">na z\u00e1klad\u011b popt\u00e1vky<\/a><\/noindex> za TB zpracovan\u00fdch dat. Toto rozhodnut\u00ed bylo tak\u00e9 zalo\u017eeno na n\u00e1zorech u\u017eivatel\u016f, kte\u0159\u00ed necht\u011bli myslet na n\u00e1klady p\u0159ed ka\u017ed\u00fdm dotazem.<\/p>\n<p>Ulo\u017een\u00ed dat v BigQuery p\u0159ineslo n\u00e1klady nav\u00edc ke n\u00e1klad\u016fm na GCS. N\u00e1stroje jako Scalding vy\u017eaduj\u00ed, aby datov\u00e9 soubory byly v GCS, a pro p\u0159\u00edstup k BigQuery jsme museli nahr\u00e1t stejn\u00e9 datov\u00e9 soubory ve form\u00e1tu BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/blog\/products\/gcp\/inside-capacitor-bigquerys-next-generation-columnar-storage-format\">Capacitor<\/a><\/noindex>. Pracujeme na propojen\u00ed Scalding s datov\u00fdmi soubory BigQuery, co\u017e odstran\u00ed pot\u0159ebu ukl\u00e1dat datov\u00e9 sady jak v GCS, tak v BigQuery.<\/p>\n<p>Pro vz\u00e1cn\u00e9 p\u0159\u00edpady, kter\u00e9 vy\u017eadovaly m\u00e9n\u011b \u010dast\u00e9 dotazy na des\u00edtky petabajt\u016f, jsme usoudili, \u017ee ukl\u00e1d\u00e1n\u00ed datov\u00fdch soubor\u016f v BigQuery nen\u00ed ekonomicky v\u00fdhodn\u00e9, a pou\u017eili jsme Presto pro p\u0159\u00edm\u00fd p\u0159\u00edstup k datov\u00fdm soubor\u016fm v GCS. K tomu se d\u00edv\u00e1me na BigQuery External Data Sources.<\/p>\n<h3>Dal\u0161\u00ed kroky<\/h3>\n<p>\nZaznamenali jsme velk\u00fd z\u00e1jem o BigQuery od doby vyd\u00e1n\u00ed alfa verze. P\u0159id\u00e1v\u00e1me v\u00edce datov\u00fdch sad a v\u00edce t\u00fdm\u016f do BigQuery. Vyv\u00edj\u00edme konektory pro analytick\u00e9 n\u00e1stroje, jako je Scalding, pro \u010dten\u00ed a z\u00e1pis do \u00falo\u017ei\u0161t\u011b BigQuery. Zva\u017eujeme n\u00e1stroje jako Looker a Apache Zeppelin pro vytv\u00e1\u0159en\u00ed podnikov\u00fdch zpr\u00e1v o kvalit\u011b a pozn\u00e1mek pomoc\u00ed datov\u00fdch soubor\u016f BigQuery.<\/p>\n<p>Spolupr\u00e1ce s Googlem byla velmi plodn\u00e1 a t\u011b\u0161\u00edme se na pokra\u010dov\u00e1n\u00ed a rozvoj tohoto partnerstv\u00ed. Pracovali jsme s Googlem na implementaci na\u0161eho vlastn\u00edho <noindex><a rel=\"nofollow\" href=\"https:\/\/developers.google.com\/issue-tracker\/guides\/partner-access\">Partner Issue Tracker<\/a><\/noindex>, abychom mohli p\u0159\u00edmo zas\u00edlat \u017e\u00e1dosti Googlu. N\u011bkter\u00e9 z nich, jako nap\u0159\u00edklad BigQuery Parquet loader, ji\u017e byly Googlem realizov\u00e1ny.<\/p>\n<p>Zde je n\u011bkolik na\u0161ich vysoce prioritn\u00edch po\u017eadavk\u016f na funkce pro Google:<\/p>\n<ul>\n<li>N\u00e1stroje pro pohodln\u00e9 p\u0159ij\u00edm\u00e1n\u00ed dat a podpora form\u00e1tu LZO-Thrift.<\/li>\n<li>Hodinov\u00e9 segmentov\u00e1n\u00ed<\/li>\n<li>Vylep\u0161en\u00ed v oblasti kontroly p\u0159\u00edstupu, jako jsou opr\u00e1vn\u011bn\u00ed na \u00farovni tabulek, \u0159\u00e1dk\u016f a sloupc\u016f.<\/li>\n<li>BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/external-data-sources\">Extern\u00ed zdroje dat<\/a><\/noindex> s integrac\u00ed a podporou Hive Metastore pro form\u00e1t LZO-Thrift.<\/li>\n<li>Vylep\u0161en\u00e1 integrace katalogu dat v u\u017eivatelsk\u00e9m rozhran\u00ed BigQuery<\/li>\n<li>Self-service pro distribuci a monitorov\u00e1n\u00ed slot\u016f.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Z\u00e1v\u011br<\/h3>\n<p>\nDemokratizace anal\u00fdzy dat, vizualizace a strojov\u00e9ho u\u010den\u00ed bezpe\u010dn\u00fdm zp\u016fsobem je nejvy\u0161\u0161\u00ed prioritou t\u00fdmu Data Platform. Identifikovali jsme Google BigQuery a Data Studio jako n\u00e1stroje, kter\u00e9 mohou pomoci dos\u00e1hnout tohoto c\u00edle, a loni jsme zp\u0159\u00edstupnili BigQuery Alpha pro celou spole\u010dnost. <\/p>\n<p>Zjistili jsme, \u017ee dotazy v BigQuery byly jednoduch\u00e9 a efektivn\u00ed. Pro p\u0159\u00edjem a transformaci dat jsme pou\u017eili n\u00e1stroje Google pro jednoduch\u00e9 pipeline, ale pro slo\u017eit\u00e9 pipeline jsme museli vytvo\u0159it vlastn\u00ed infrastrukturu Airflow. V oblasti spr\u00e1vy dat slu\u017eby BigQuery pro autentizaci, autorizaci a audit spl\u0148uj\u00ed na\u0161e pot\u0159eby. Pro spr\u00e1vu metadat a dodr\u017eov\u00e1n\u00ed soukrom\u00ed jsme pot\u0159ebovali v\u011bt\u0161\u00ed flexibilitu a museli jsme vytvo\u0159it vlastn\u00ed syst\u00e9my. BigQuery, jako spravovan\u00e1 slu\u017eba, byl snadn\u00fd na ovl\u00e1d\u00e1n\u00ed. N\u00e1klady na dotazy byly srovnateln\u00e9 s existuj\u00edc\u00edmi n\u00e1stroji. Ulo\u017een\u00ed dat v BigQuery m\u011blo n\u00e1klady nav\u00edc ke n\u00e1klad\u016fm na GCS.<\/p>\n<p>Obecn\u011b BigQuery dob\u0159e funguje pro obecnou SQL anal\u00fdzu. Pozorujeme velk\u00fd z\u00e1jem o BigQuery a pracujeme na p\u0159enesen\u00ed v\u011bt\u0161\u00edho po\u010dtu dataset\u016f, zapojen\u00ed v\u00edce t\u00fdm\u016f a vytv\u00e1\u0159en\u00ed v\u00edce pipeline s BigQuery. Na Twitteru jsou pou\u017e\u00edv\u00e1na r\u016fzn\u00e1 data, pro kter\u00e1 bude pot\u0159eba kombinace takov\u00fdch n\u00e1stroj\u016f, jako jsou Scalding, Spark, Presto a Druid. M\u00e1me v \u00famyslu pokra\u010dovat v rozvoji na\u0161ich analytick\u00fdch n\u00e1stroj\u016f a poskytovat jasn\u00e9 doporu\u010den\u00ed na\u0161im u\u017eivatel\u016fm, jak nejl\u00e9pe vyu\u017e\u00edt na\u0161e nab\u00eddky.<\/p>\n<h3>Slova vd\u011b\u010dnosti<\/h3>\n<p>\nR\u00e1d bych pod\u011bkoval sv\u00fdm spoluautor\u016fm a \u010dlen\u016fm t\u00fdmu, An\u017ee D\u017ea a Willu Pascucci, za jejich skv\u011blou spolupr\u00e1ci a tvrdou pr\u00e1ci na tomto projektu. Tak\u00e9 bych cht\u011bl pod\u011bkovat in\u017een\u00fdr\u016fm a mana\u017eer\u016fm z n\u011bkolika t\u00fdm\u016f na Twitteru a Googlu, kte\u0159\u00ed n\u00e1m pomohli a u\u017eivatel\u016fm BigQuery na Twitteru poskytli cennou zp\u011btnou vazbu.<\/p>\n<p>Pokud m\u00e1te z\u00e1jem pracovat na t\u011bchto \u00faloh\u00e1ch, pod\u00edvejte se na na\u0161e <noindex><a rel=\"nofollow\" href=\"https:\/\/careers.twitter.com\/content\/careers-twitter\/en\/jobs-search.html?q=&amp;team=careers-twitter%3Ateam%2Finfrastructure-engineering\">voln\u00e9 pozice<\/a><\/noindex> v t\u00fdmu Data Platform.<\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eD9S\/\">Kvalita dat v DWH \u2014 konzistence datov\u00e9ho skladu<\/a><\/noindex><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/513782\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441 \u0432\u0430\u043c\u0438 \u043f\u043e\u043b\u0435\u0437\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c. \u0427\u0438\u0442\u0430\u0442\u044c \u043f\u0435\u0440\u0432\u0443\u044e \u0447\u0430\u0441\u0442\u044c \u0423\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0421\u0438\u043b\u044c\u043d\u043e\u0435 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 (Strong Data Governance) \u2014 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043f\u0440\u0438\u043d\u0446\u0438\u043f Twitter Engineering. \u041f\u043e\u0441\u043a\u043e\u043b\u044c\u043a\u0443 \u043c\u044b \u0432\u043d\u0435\u0434\u0440\u044f\u0435\u043c BigQuery \u0432 \u043d\u0430\u0448\u0443 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443, \u043c\u044b \u043a\u043e\u043d\u0446\u0435\u043d\u0442\u0440\u0438\u0440\u0443\u0435\u043c\u0441\u044f \u043d\u0430 \u043e\u0431\u043d\u0430\u0440\u0443\u0436\u0435\u043d\u0438\u0438 \u0434\u0430\u043d\u043d\u044b\u0445, \u043a\u043e\u043d\u0442\u0440\u043e\u043b\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u0430, \u0431\u0435\u0437\u043e\u043f\u0430\u0441\u043d\u043e\u0441\u0442\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91904,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91903","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Jak BigQuery od Googlu demokratizoval anal\u00fdzu dat. \u010c\u00e1st 2 | ProHoster","description":"Ahoj, Habr! Pr\u00e1v\u011b te\u010f je v OTUS otev\u0159en z\u00e1pis na nov\u00fd b\u011bh kurzu \"Data Engineer\".","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-20T05:42:18+00:00","article:modified_time":"2020-08-20T05:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91903","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:18:28","updated":"2022-10-08 11:10:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=91903"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/91903\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/91904"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=91903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=91903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=91903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}