{"id":91903,"date":"2020-08-20T07:42:18","date_gmt":"2020-08-20T05:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2"},"modified":"2020-08-20T07:42:18","modified_gmt":"2020-08-20T05:42:18","slug":"kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","title":{"rendered":"Si BigQuery nga Google demokratizoi analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><b><i>P\u00ebrsh\u00ebndetje, Habr! Aktualisht n\u00eb OTUS \u00ebsht\u00eb hapur regjistrimi p\u00ebr nj\u00eb grup t\u00eb ri t\u00eb kursit <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/wDZQ\/\">Inxhinier i t\u00eb Dh\u00ebnave<\/a><\/noindex>. Para fillimin e kursit, vazhdojm\u00eb t\u00eb ndajm\u00eb me ju materiale t\u00eb dobishme. <\/i><\/b><\/p>\n<p><i><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/\">Lexoni pjes\u00ebn e par\u00eb<\/a><\/noindex><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Si BigQuery nga Google demokratizoi analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2\" src=\"\/wp-content\/uploads\/2020\/08\/7b9314bffd954cd408807280f8956c21.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h4>Menaxhimi i t\u00eb dh\u00ebnave<\/h4>\n<p>\nMenaxhimi i fort\u00eb i t\u00eb dh\u00ebnave (Strong Data Governance) \u00ebsht\u00eb parimi kryesor i Twitter Engineering. Nd\u00ebrsa po implementojm\u00eb BigQuery n\u00eb platform\u00ebn ton\u00eb, ne p\u00ebrqendrohemi n\u00eb zbulimin e t\u00eb dh\u00ebnave, kontrollin e aksesit, sigurin\u00eb dhe privat\u00ebsin\u00eb.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>P\u00ebr zbulimin dhe menaxhimin e t\u00eb dh\u00ebnave, ne kemi zgjeruar nivelin ton\u00eb t\u00eb aksesit n\u00eb t\u00eb dh\u00ebna (Data Access Layer - <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), n\u00eb m\u00ebnyr\u00eb q\u00eb t\u00eb ofrojm\u00eb mjete si p\u00ebr t\u00eb dh\u00ebnat lokale ashtu edhe p\u00ebr t\u00eb dh\u00ebnat n\u00eb Google Cloud, duke ofruar nj\u00eb nd\u00ebrfaqe t\u00eb vetme dhe API p\u00ebr p\u00ebrdoruesit tan\u00eb. Nd\u00ebrsa Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/data-catalog\/\">Data Catalog<\/a><\/noindex> po shkon drejt k\u00ebt\u00eb k\u00ebndv\u00ebshtrim publik, ne do ta p\u00ebrfshijm\u00eb n\u00eb projektet tona p\u00ebr t\u00eb ofruar p\u00ebrdoruesve funksione si k\u00ebrkimi p\u00ebr kolonat.<\/p>\n<p>BigQuery lejon shk\u00ebmbimin dhe aksesin e leht\u00eb t\u00eb t\u00eb dh\u00ebnave, por na duhej t\u00eb kishim nj\u00eb kontroll n\u00eb nj\u00eb far\u00eb mase p\u00ebr t\u00eb parandaluar ekzsfiltrimin e t\u00eb dh\u00ebnave. Nd\u00ebr mjete t\u00eb tjera, ne zgjodh\u00ebm dy funksione:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/resource-manager\/docs\/organization-policy\/restricting-domains\">Nd partage t\u00eb kufizuar n\u00eb domen<\/a><\/noindex>: nj\u00eb funksion beta q\u00eb ndalon p\u00ebrdoruesit t\u00eb ndajn\u00eb grupe t\u00eb dh\u00ebnash BigQuery me p\u00ebrdoruesit jasht\u00eb Twitter.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/vpc-service-controls\/\">Kontrollet e sh\u00ebrbimit VPC<\/a><\/noindex>: nj\u00eb mjet kontrolli q\u00eb parandalon ekzsfiltrimin e t\u00eb dh\u00ebnave dhe k\u00ebrkon q\u00eb p\u00ebrdoruesit t\u00eb ken\u00eb akses n\u00eb BigQuery nga gamat e njohura t\u00eb adresave IP.<\/li>\n<\/ul>\n<p>\nKemi implementuar k\u00ebrkesat p\u00ebr autentikim, autorizim dhe auditim (AAA) p\u00ebr t\u00eb siguruar sigurin\u00eb n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb:<\/p>\n<ul>\n<li> Autentikimi: ne p\u00ebrdor\u00ebm llogarit\u00eb e p\u00ebrdoruesve GCP p\u00ebr k\u00ebrkesa ad hoc dhe llogarit\u00eb sh\u00ebrbimi p\u00ebr k\u00ebrkesat e pun\u00ebs.<\/li>\n<li>Autorizimi: k\u00ebrkuam q\u00eb \u00e7do grup t\u00eb dh\u00ebnash t\u00eb kishte nj\u00eb llogari sh\u00ebrbimi pronari dhe nj\u00eb grup lexuesish.<\/li>\n<li>Auditimi: ne eksportuam logjet e sh\u00ebrbimit BigQuery, t\u00eb cilat p\u00ebrmbanin informacione t\u00eb detajuara mbi ekzekutimin e k\u00ebrkesave, n\u00eb grupin e t\u00eb dh\u00ebnave t\u00eb BigQuery p\u00ebr leht\u00ebsimin e analiz\u00ebs.<\/li>\n<\/ul>\n<p>\nP\u00ebr t\u00eb siguruar p\u00ebrpunimin e duhur t\u00eb t\u00eb dh\u00ebnave personale t\u00eb p\u00ebrdoruesve t\u00eb Twitter, ne duhet t\u00eb regjistrojm\u00eb t\u00eb gjitha grupet e t\u00eb dh\u00ebnave BigQuery, t\u00eb shpallim t\u00eb dh\u00ebnat personale, t\u00eb mbajm\u00eb ruajtjen e duhur dhe t\u00eb fshijm\u00eb (pastruar) t\u00eb dh\u00ebnat q\u00eb jan\u00eb fshir\u00eb nga p\u00ebrdoruesit.<\/p>\n<p>Ne shqyrtuam Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/dlp\/\">Cloud Data Loss Prevention API<\/a><\/noindex>, e cila p\u00ebrdor m\u00ebsimin e makin\u00ebs p\u00ebr klasifikimin dhe redaktimin e t\u00eb dh\u00ebnave t\u00eb ndjeshme, por mor\u00ebm vendim p\u00ebr annotimin manual t\u00eb grupit t\u00eb t\u00eb dh\u00ebnave p\u00ebr shkak t\u00eb sakt\u00ebsis\u00eb. Ne planifikojm\u00eb t\u00eb p\u00ebrdorim API-n\u00eb e Parandalimit t\u00eb Humbjes s\u00eb T\u00eb Dh\u00ebnave p\u00ebr t\u00eb plot\u00ebsuar annotimin e p\u00ebrdoruesve.<\/p>\n<p>N\u00eb Twitter kemi krijuar kat\u00ebr kategori privat\u00ebsie p\u00ebr grupe t\u00eb dh\u00ebnash n\u00eb BigQuery, t\u00eb listuara k\u00ebtu n\u00eb rend t\u00eb zbrit\u00ebs s\u00eb ndjeshm\u00ebris\u00eb:<\/p>\n<ul>\n<li>Grupet e t\u00eb dh\u00ebnave me ndjeshm\u00ebri t\u00eb lart\u00eb jan\u00eb t\u00eb arritshme sipas nevoj\u00ebs mbi parimin e privilegjeve m\u00eb t\u00eb vogla. Secili grup t\u00eb dh\u00ebnash ka nj\u00eb grup lexuesish t\u00eb ve\u00e7ant\u00eb, dhe ne do t\u00eb monitorojm\u00eb p\u00ebrdorimin e llogarive t\u00eb ve\u00e7anta.<\/li>\n<li>Grupet e t\u00eb dh\u00ebnave me ndjeshm\u00ebri mesatare (pseudonime unidirezionale me hashing t\u00eb kriptuar) nuk p\u00ebrmbajn\u00eb informacion personal (Informacion Personal Identifikues \u2014 PII) dhe jan\u00eb t\u00eb arritshme p\u00ebr nj\u00eb grup m\u00eb t\u00eb madh punonj\u00ebsish. Kjo \u00ebsht\u00eb nj\u00eb balanc\u00eb e mir\u00eb midis shqet\u00ebsimeve t\u00eb privat\u00ebsis\u00eb dhe p\u00ebrdorshm\u00ebris\u00eb s\u00eb t\u00eb dh\u00ebnave. Kjo lejon punonj\u00ebsit t\u00eb kryejn\u00eb analiza, si\u00e7 \u00ebsht\u00eb num\u00ebrimi i p\u00ebrdoruesve q\u00eb kan\u00eb p\u00ebrdorur funksionalitetin, pa ditur kush jan\u00eb p\u00ebrdoruesit e v\u00ebrtet\u00eb.<\/li>\n<li>Grupet e t\u00eb dh\u00ebnave me ndjeshm\u00ebri t\u00eb ul\u00ebt p\u00ebrmbajn\u00eb t\u00eb gjith\u00eb informacionin q\u00eb identifikon p\u00ebrdoruesin. Kjo \u00ebsht\u00eb nj\u00eb qasje e mir\u00eb n\u00eb aspektin e privat\u00ebsis\u00eb, por nuk mund t\u00eb p\u00ebrdoret p\u00ebr analiza n\u00eb nivelin e p\u00ebrdoruesit.<\/li>\n<li>Grupet e t\u00eb dh\u00ebnave publike (t\u00eb l\u00ebshuara jasht\u00eb Twitter) jan\u00eb t\u00eb arritshme p\u00ebr t\u00eb gjith\u00eb punonj\u00ebsit e Twitter.<\/li>\n<\/ul>\n<p>\nSa i p\u00ebrket regjistrimit, ne p\u00ebrdor\u00ebm detyra t\u00eb planifikuara p\u00ebr t\u00eb listuar grupet e t\u00eb dh\u00ebnave BigQuery dhe p\u00ebr t'i regjistruar ato n\u00eb Shtres\u00ebn e Qasjes s\u00eb T\u00eb Dh\u00ebnave (<noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), depoja e t\u00eb dh\u00ebnave t\u00eb Twitter. P\u00ebrdoruesit do t\u00eb annotojn\u00eb grupet e t\u00eb dh\u00ebnave me informacion mbi privat\u00ebsin\u00eb, si dhe do t\u00eb specifikojn\u00eb periudh\u00ebn e ruajtjes. Sa i p\u00ebrket pastrimit, ne po vler\u00ebsojm\u00eb performanc\u00ebn dhe koston e dy opsioneve: <i>1.<\/i> Pastrimi i grupeve t\u00eb t\u00eb dh\u00ebnave n\u00eb GCS duke p\u00ebrdorur mjete si Scalding dhe ngarkimi i tyre n\u00eb BigQuery; <i>2.<\/i> P\u00ebrdorimi i operator\u00ebve DML t\u00eb BigQuery. Probabilisht do t\u00eb p\u00ebrdorim nj\u00eb kombinim t\u00eb t\u00eb dyjave p\u00ebr t\u00eb p\u00ebrmbushur k\u00ebrkesat e grupeve t\u00eb ndryshme dhe t\u00eb dh\u00ebnave.<\/p>\n<h4>Funksionaliteti i sistemit<\/h4>\n<p>\nDuke qen\u00eb se BigQuery \u00ebsht\u00eb nj\u00eb sh\u00ebrbim i menaxhuar, nuk ishte e nevojshme t\u00eb angazhoheshin ekipet SRE t\u00eb Twitter n\u00eb menaxhimin e sistemeve ose n\u00eb kryerjen e detyrave t\u00eb sh\u00ebrbimit. Ishte leht\u00eb t\u00eb ofronim kapacitet t\u00eb madh si p\u00ebr ruajtjen ashtu edhe p\u00ebr llogaritjen. Ne mund t\u00eb ndryshonim rezervimin e slot\u00ebve duke krijuar bileta n\u00eb mb\u00ebshtetje t\u00eb Google. E identifikuam se duhej p\u00ebrmir\u00ebsuar, p\u00ebr shembull, vet\u00eb-sh\u00ebrbimi p\u00ebr shp\u00ebrndarjen e slot\u00ebve dhe p\u00ebrmir\u00ebsimi i panelit p\u00ebr monitorimin dhe i d\u00ebrguam k\u00ebto k\u00ebrkesa n\u00eb Google.<\/p>\n<h4>\u00c7mimi<\/h4>\n<p>\nAnaliza jon\u00eb paraprake tregoi se kostot e k\u00ebrkesave p\u00ebr BigQuery dhe Presto ishin n\u00eb t\u00eb nj\u00ebjtin nivel. Ne blejm\u00eb slot\u00eb me <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#flat_rate_pricing\">\u00e7mim fiks<\/a><\/noindex> p\u00ebr t\u00eb pasur nj\u00eb kostos s\u00eb stabilizuar mujor n\u00eb vend q\u00eb t\u00eb paguanim <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#on_demand_pricing\">n\u00eb k\u00ebrkes\u00eb<\/a><\/noindex> p\u00ebr TB t\u00eb dh\u00ebnash t\u00eb p\u00ebrpunuara. Ky vendim gjithashtu bazohej n\u00eb feedback-un e p\u00ebrdoruesve, t\u00eb cil\u00ebt nuk d\u00ebshironin t\u00eb mendonin p\u00ebr kostot para se t\u00eb kryenin \u00e7do k\u00ebrkes\u00eb.<\/p>\n<p>Ruajtja e t\u00eb dh\u00ebnave n\u00eb BigQuery solli shpenzime p\u00ebrve\u00e7 kostove p\u00ebr GCS. Vegla t\u00eb tilla si Scalding k\u00ebrkojn\u00eb q\u00eb setet e dh\u00ebnave t\u00eb jen\u00eb n\u00eb GCS, dhe p\u00ebr t\u00eb pasur qasje n\u00eb BigQuery, na duhej t\u00eb ngarkonim t\u00eb nj\u00ebjtat sete t\u00eb dh\u00ebnash n\u00eb formatin BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/blog\/products\/gcp\/inside-capacitor-bigquerys-next-generation-columnar-storage-format\">Capacitor<\/a><\/noindex>. Ne po punojm\u00eb mbi lidhjen e Scalding me setet e dh\u00ebnave BigQuery, e cila do t\u00eb eliminoj\u00eb nevoj\u00ebn p\u00ebr t\u00eb ruajtur setet e dh\u00ebnash si n\u00eb GCS ashtu edhe n\u00eb BigQuery.<\/p>\n<p>P\u00ebr raste t\u00eb rralla q\u00eb k\u00ebrkonin k\u00ebrkesa t\u00eb pakta p\u00ebr dhjet\u00ebra petabyte, ne vendos\u00ebm se ruajtja e seteve t\u00eb dh\u00ebnave n\u00eb BigQuery nuk ishte ekonomikisht e efektshme dhe p\u00ebrdor\u00ebm Presto p\u00ebr qasje direkte n\u00eb setet e dh\u00ebnave n\u00eb GCS. P\u00ebr k\u00ebt\u00eb, ne jemi duke shqyrtuar Burimet e Dh\u00ebnave t\u00eb Jashtme t\u00eb BigQuery.<\/p>\n<h3>Hapat e ardhsh\u00ebm<\/h3>\n<p>\nNe kemi v\u00ebn\u00eb re nj\u00eb interes t\u00eb madh p\u00ebr BigQuery q\u00eb nga l\u00ebshimi i alaf\u00ebs. Ne po shtojm\u00eb m\u00eb shum\u00eb sete t\u00eb dh\u00ebnash dhe m\u00eb shum\u00eb ekipe n\u00eb BigQuery. Ne po zhvillojm\u00eb lidh\u00ebs p\u00ebr veglat e analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave, si\u00e7 \u00ebsht\u00eb Scalding, p\u00ebr t\u00eb lexuar dhe shkruar n\u00eb depo BigQuery. Ne po shqyrtojm\u00eb vegla si Looker dhe Apache Zeppelin p\u00ebr t\u00eb krijuar raporte korporatash mbi cil\u00ebsin\u00eb dhe sh\u00ebnime duke p\u00ebrdorur sete t\u00eb dh\u00ebnash BigQuery.<\/p>\n<p>Bashk\u00ebpunimi me Google ka qen\u00eb shum\u00eb produktiv dhe ne jemi t\u00eb g\u00ebzuar t\u00eb vazhdojm\u00eb dhe zhvillojm\u00eb k\u00ebt\u00eb partneritet. Ne kemi punuar me Google p\u00ebr t\u00eb zbatuar sistemin ton\u00eb t\u00eb <noindex><a rel=\"nofollow\" href=\"https:\/\/developers.google.com\/issue-tracker\/guides\/partner-access\">Partner Issue Tracker<\/a><\/noindex>, p\u00ebr t\u00eb d\u00ebrguar drejtp\u00ebrdrejt k\u00ebrkesat n\u00eb Google. Disa prej tyre, si ngarkuesi BigQuery Parquet, jan\u00eb realisht zbatuar nga Google.<\/p>\n<p>K\u00ebtu jan\u00eb disa nga k\u00ebrkesat tona m\u00eb prioritare p\u00ebr ve\u00e7ori p\u00ebr Google:<\/p>\n<ul>\n<li>Mjetet p\u00ebr pranim t\u00eb dh\u00ebnash t\u00eb leht\u00eb dhe mb\u00ebshtetje p\u00ebr formatin LZO-Thrift.<\/li>\n<li>Segmentimi me or\u00eb<\/li>\n<li>P\u00ebrmir\u00ebsime n\u00eb fush\u00ebn e kontrollit t\u00eb aksesit, si lejet n\u00eb nivel tabela, rreshti dhe kolona.<\/li>\n<li>BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/external-data-sources\">Burimet e t\u00eb Dh\u00ebnave t\u00eb Jashtme<\/a><\/noindex> me integrim dhe mb\u00ebshtetje p\u00ebr Hive Metastore p\u00ebr formatin LZO-Thrift.<\/li>\n<li>Integrim i p\u00ebrmir\u00ebsuar i katalogut t\u00eb t\u00eb dh\u00ebnave n\u00eb nd\u00ebrfaqen BigQuery<\/li>\n<li>Sh\u00ebrbime vet\u00eb-sh\u00ebrbimi p\u00ebr shp\u00ebrndarjen dhe monitorimin e hap\u00ebsirave.<\/li>\n<\/ul>\n<p><\/p>\n<h3>P\u00ebrfundim<\/h3>\n<p>\nDemokratizimi i analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave, vizualizimeve dhe m\u00ebsimit t\u00eb makineris\u00eb n\u00eb nj\u00eb m\u00ebnyr\u00eb t\u00eb sigurt \u00ebsht\u00eb prioriteti m\u00eb i lart\u00eb p\u00ebr ekipin e Platform\u00ebs s\u00eb t\u00eb Dh\u00ebnave. Ne kemi identifikuar Google BigQuery dhe Data Studio si mjete q\u00eb mund t\u00eb ndihmojn\u00eb n\u00eb arritjen e k\u00ebsaj qellimi dhe lan\u00e7uam vitin e kaluar BigQuery Alpha p\u00ebr t\u00eb gjith\u00eb kompanin\u00eb. <\/p>\n<p>Kemi zbuluar se k\u00ebrkesat n\u00eb BigQuery ishin t\u00eb thjeshta dhe efikase. P\u00ebr pranim dhe p\u00ebrpunim t\u00eb dh\u00ebnash kemi p\u00ebrdorur mjete nga Google p\u00ebr linja t\u00eb thjeshta, por p\u00ebr linja m\u00eb komplekse na duhej t\u00eb ndihmonim infrastruktur\u00ebn ton\u00eb Airflow. N\u00eb fush\u00ebn e menaxhimit t\u00eb t\u00eb dh\u00ebnave, sh\u00ebrbimet e BigQuery p\u00ebr autentikimin, autorizimin dhe auditimin plot\u00ebsojn\u00eb nevojat tona. P\u00ebr menaxhimin e metadata dhe ndershm\u00ebrin\u00eb na d\u00ebshirohej fleksibilitet i madh dhe na duhej t\u00eb krijonim sisteme tona. BigQuery, duke qen\u00eb sh\u00ebrbim i menaxhuar, ishte i leht\u00eb p\u00ebr t'u operuar. Kostot e k\u00ebrkesave ishin t\u00eb ngjashme me mjetet ekzistuese. Ruajtja e t\u00eb dh\u00ebnave n\u00eb BigQuery kishte shpenzime p\u00ebrve\u00e7 kostove p\u00ebr GCS.<\/p>\n<p>P\u00ebrgjith\u00ebsisht, BigQuery funksionon mir\u00eb p\u00ebr analiz\u00ebn e p\u00ebrgjithshme SQL. Ne v\u00ebrejm\u00eb interes t\u00eb madh p\u00ebr BigQuery dhe po punojm\u00eb p\u00ebr t\u00eb transferuar m\u00eb shum\u00eb grupe t\u00eb dh\u00ebnash, p\u00ebr t\u00eb angazhuar m\u00eb shum\u00eb ekipe dhe p\u00ebr t\u00eb krijuar m\u00eb shum\u00eb linja me BigQuery. N\u00eb Twitter p\u00ebrdoren t\u00eb dh\u00ebna t\u00eb ndryshme, p\u00ebr t\u00eb cilat do t\u00eb nevojiten nj\u00eb kombinim i mjeteve si Scalding, Spark, Presto dhe Druid. Ne synojm\u00eb t\u00eb vazhdojm\u00eb t\u00eb rrisim mjetet tona t\u00eb analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave dhe t\u00eb ofrojm\u00eb rekomandime t\u00eb qarta p\u00ebr p\u00ebrdoruesit tan\u00eb se si t\u00eb shfryt\u00ebzojn\u00eb m\u00eb s\u00eb miri ofertat tona.<\/p>\n<h3>Fjal\u00eb fal\u00ebnderimi<\/h3>\n<p>\nD\u00ebshiroj t\u00eb fal\u00ebnderoj bashk\u00ebautorin tim dhe shok\u00ebt e ekipit, Anju Dja dhe Will Pascucci, p\u00ebr bashk\u00ebpunimin e tyre t\u00eb shk\u00eblqyer dhe pun\u00ebn e r\u00ebnd\u00ebsishme n\u00eb k\u00ebt\u00eb projekt. Gjithashtu, dua t\u00eb fal\u00ebnderoj inxhinier\u00ebt dhe menaxher\u00ebt nga disa ekipe n\u00eb Twitter dhe Google, t\u00eb cil\u00ebt na ndihmuan dhe p\u00ebrdoruesve t\u00eb BigQuery n\u00eb Twitter, duke ofruar feedback t\u00eb vlefsh\u00ebm.<\/p>\n<p>N\u00ebse jeni t\u00eb interesuar t\u00eb punoni n\u00eb k\u00ebto detyra, shikoni <noindex><a rel=\"nofollow\" href=\"https:\/\/careers.twitter.com\/content\/careers-twitter\/en\/jobs-search.html?q=&amp;team=careers-twitter%3Ateam%2Finfrastructure-engineering\">vakancat<\/a><\/noindex> n\u00eb ekipin Data Platform.<\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eD9S\/\">Cil\u00ebsia e t\u00eb dh\u00ebnave n\u00eb DWH \u2014 konsistenca e magazin\u00ebs s\u00eb t\u00eb dh\u00ebnave<\/a><\/noindex><\/p>\n<p>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/513782\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441 \u0432\u0430\u043c\u0438 \u043f\u043e\u043b\u0435\u0437\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c. \u0427\u0438\u0442\u0430\u0442\u044c \u043f\u0435\u0440\u0432\u0443\u044e \u0447\u0430\u0441\u0442\u044c \u0423\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0421\u0438\u043b\u044c\u043d\u043e\u0435 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 (Strong Data Governance) \u2014 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043f\u0440\u0438\u043d\u0446\u0438\u043f Twitter Engineering. \u041f\u043e\u0441\u043a\u043e\u043b\u044c\u043a\u0443 \u043c\u044b \u0432\u043d\u0435\u0434\u0440\u044f\u0435\u043c BigQuery \u0432 \u043d\u0430\u0448\u0443 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443, \u043c\u044b \u043a\u043e\u043d\u0446\u0435\u043d\u0442\u0440\u0438\u0440\u0443\u0435\u043c\u0441\u044f \u043d\u0430 \u043e\u0431\u043d\u0430\u0440\u0443\u0436\u0435\u043d\u0438\u0438 \u0434\u0430\u043d\u043d\u044b\u0445, \u043a\u043e\u043d\u0442\u0440\u043e\u043b\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u0430, \u0431\u0435\u0437\u043e\u043f\u0430\u0441\u043d\u043e\u0441\u0442\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91904,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91903","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Si BigQuery nga Google e demokratizoi analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2 | ProHoster","description":"P\u00ebrsh\u00ebndetje, Habr! Tani p\u00ebr tani, n\u00eb OTUS \u00ebsht\u00eb e hapur regjistrimi p\u00ebr grupin e ri t\u00eb kursit \"Inxhinier i T\u00eb Dh\u00ebnave\".","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-20T05:42:18+00:00","article:modified_time":"2020-08-20T05:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91903","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:18:28","updated":"2022-10-08 11:10:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=91903"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91903\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/91904"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=91903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=91903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=91903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}