{"id":91903,"date":"2020-08-20T07:42:18","date_gmt":"2020-08-20T05:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2"},"modified":"2020-08-20T07:42:18","modified_gmt":"2020-08-20T05:42:18","slug":"kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","title":{"rendered":"Si e demokratizoi BigQuery nga Google analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><b><i>P\u00ebrsh\u00ebndetje, Habr! Tani p\u00ebr tani n\u00eb OTUS \u00ebsht\u00eb hapur nj\u00eb grup p\u00ebr nj\u00eb raund t\u00eb ri t\u00eb kursit. <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/wDZQ\/\">\u00abInxhiner i T\u00eb Dh\u00ebnave\u00bb<\/a><\/noindex>. N\u00eb prag t\u00eb nisjes s\u00eb kursit vazhdojm\u00eb t\u00eb ndajm\u00eb me ju materiale t\u00eb dobishme. <\/i><\/b><\/p>\n<p><i><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/\">Lexoni pjes\u00ebn e par\u00eb<\/a><\/noindex><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Si e demokratizoi BigQuery nga Google analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2\" src=\"\/wp-content\/uploads\/2020\/08\/7b9314bffd954cd408807280f8956c21.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h4>Menaxhimi i t\u00eb dh\u00ebnave<\/h4>\n<p>\nMenaxhimi i fort\u00eb i t\u00eb dh\u00ebnave (Strong Data Governance) \u00ebsht\u00eb parimi kryesor i Twitter Engineering. Nd\u00ebrsa po implementojm\u00eb BigQuery n\u00eb platform\u00ebn ton\u00eb, ne p\u00ebrqendrohemi n\u00eb zbulimin e t\u00eb dh\u00ebnave, kontrollet e aksesit, sigurin\u00eb dhe privat\u00ebsin\u00eb.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>P\u00ebr zbulimin dhe menaxhimin e t\u00eb dh\u00ebnave, ne zgjeruam nivelin ton\u00eb t\u00eb aksesit n\u00eb t\u00eb dh\u00ebna (Data Access Layer \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), p\u00ebr t\u00eb ofruar mjete si p\u00ebr t\u00eb dh\u00ebnat lokale ashtu edhe p\u00ebr t\u00eb dh\u00ebnat e Google Cloud, duke siguruar nj\u00eb nd\u00ebrfaqe t\u00eb vetme dhe API p\u00ebr p\u00ebrdoruesit tan\u00eb. Nd\u00ebrkoh\u00eb q\u00eb Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/data-catalog\/\">Data Catalog<\/a><\/noindex> shkon drejt publikimit, ne do ta p\u00ebrfshijm\u00eb at\u00eb n\u00eb projektet tona, p\u00ebr t'u ofruar p\u00ebrdoruesve funksionalitete si k\u00ebrkimi sipas kolonave.<\/p>\n<p>BigQuery lejon ndarjen dhe aksesin e leht\u00eb n\u00eb t\u00eb dh\u00ebna, por na duhej t\u00eb kontrollonim k\u00ebt\u00eb deri n\u00eb nj\u00eb far\u00eb niveli, p\u00ebr t\u00eb parandaluar eksfiltrimin e t\u00eb dh\u00ebnave. Mes mjeteve t\u00eb tjera, zgjodh\u00ebm dy funksionalitete:<\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/resource-manager\/docs\/organization-policy\/restricting-domains\">Ndarja e kufizuar e domain-it<\/a><\/noindex>: nj\u00eb funksion beta q\u00eb ndalon p\u00ebrdoruesit t\u00eb ndajn\u00eb grupe t\u00eb dh\u00ebnash BigQuery me p\u00ebrdoruesit jasht\u00eb Twitter.<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/vpc-service-controls\/\">Kontrollet e sh\u00ebrbimeve VPC<\/a><\/noindex>: nj\u00eb element kontrolli q\u00eb parandalon eksfiltrimin e t\u00eb dh\u00ebnave dhe k\u00ebrkon q\u00eb p\u00ebrdoruesit t\u00eb ken\u00eb akses n\u00eb BigQuery nga ranges IP t\u00eb njohura.<\/li>\n<\/ul>\n<p>\nNe zbatuam k\u00ebrkesat p\u00ebr autentifikim, autorizim dhe auditi (AAA) p\u00ebr t\u00eb siguruar sigurin\u00eb si m\u00eb posht\u00eb:<\/p>\n<ul>\n<li> Autentifikimi: p\u00ebrdor\u00ebm llogarit\u00eb e p\u00ebrdoruesve GCP p\u00ebr k\u00ebrkesa ad hoc dhe llogarit\u00eb e sh\u00ebrbimeve p\u00ebr k\u00ebrkesa pune.<\/li>\n<li>Autorizimi: k\u00ebrkuam q\u00eb \u00e7do grup t\u00eb dh\u00ebnash t\u00eb ket\u00eb nj\u00eb llogari sh\u00ebrbimi pronari dhe nj\u00eb grup lexuesish.<\/li>\n<li>Auditi: eksportuam regjistrat e stek-drivers BigQuery, t\u00eb cilat p\u00ebrmbanin informacion t\u00eb detajuar p\u00ebr ekzekutimin e k\u00ebrkesave, n\u00eb nj\u00eb grup t\u00eb dh\u00ebnash BigQuery p\u00ebr leht\u00ebsi analize.<\/li>\n<\/ul>\n<p>\nP\u00ebr t\u00eb siguruar p\u00ebrpunimin e duhur t\u00eb t\u00eb dh\u00ebnave personale t\u00eb p\u00ebrdoruesve Twitter, ne duhet t\u00eb regjistrojm\u00eb t\u00eb gjitha grupet e dh\u00ebnash BigQuery, t\u00eb anotojm\u00eb t\u00eb dh\u00ebnat personale, t\u00eb mbajm\u00eb ruajtjen e duhur dhe t\u00eb fshijm\u00eb (pastrimin) t\u00eb dh\u00ebnat q\u00eb jan\u00eb fshir\u00eb nga p\u00ebrdoruesit.<\/p>\n<p>Ne shqyrtuam Google <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/dlp\/\">Cloud Data Loss Prevention API<\/a><\/noindex>, i cili p\u00ebrdor m\u00ebsimin automatik p\u00ebr klasifikimin dhe redaktimin e t\u00eb dh\u00ebnave t\u00eb ndjeshme, por vendos\u00ebm t\u00eb favorizojm\u00eb anotimin manual t\u00eb grupit t\u00eb dh\u00ebnash p\u00ebr shkak t\u00eb sakt\u00ebsis\u00eb. Ne planifikojm\u00eb t\u00eb p\u00ebrdorim Data Loss Prevention API p\u00ebr t\u00eb plot\u00ebsuar anotimin e p\u00ebrdoruesve.<\/p>\n<p>N\u00eb Twitter kemi krijuar kat\u00ebr kategori privat\u00ebsie p\u00ebr grupet e dh\u00ebnash n\u00eb BigQuery, t\u00eb renditura k\u00ebtu sipas ndjeshm\u00ebris\u00eb:<\/p>\n<ul>\n<li>Grupet e dh\u00ebnash me ndjeshm\u00ebri t\u00eb lart\u00eb jan\u00eb t\u00eb disponueshme sipas nevoj\u00ebs sipas parimit t\u00eb privilegjeve minimale. \u00c7do grup t\u00eb dh\u00ebnash ka nj\u00eb grup t\u00eb ve\u00e7ant\u00eb lexuesish dhe ne do t\u00eb monitorojm\u00eb p\u00ebrdorimin e llogarive t\u00eb ve\u00e7anta.<\/li>\n<li>Grupet e dh\u00ebnash me ndjeshm\u00ebri t\u00eb mesme (pseudonime nj\u00eban\u00ebsore duke p\u00ebrdorur hashing t\u00eb kriptuar) nuk p\u00ebrmbajn\u00eb informacion personal (Personally Identifiable Information \u2014 PII) dhe jan\u00eb t\u00eb disponueshme p\u00ebr nj\u00eb grup m\u00eb t\u00eb gjer\u00eb punonj\u00ebsish. Kjo ofron nj\u00eb balanc\u00eb t\u00eb mir\u00eb mes shqet\u00ebsimeve t\u00eb privat\u00ebsis\u00eb dhe dobis\u00eb s\u00eb t\u00eb dh\u00ebnave. Kjo lejon punonj\u00ebsit t\u00eb kryejn\u00eb detyra analitike, si num\u00ebrimin e p\u00ebrdoruesve q\u00eb kan\u00eb p\u00ebrdorur funksionin, pa e ditur kush jan\u00eb p\u00ebrdoruesit real\u00eb.<\/li>\n<li>Grupet e dh\u00ebnash me ndjeshm\u00ebri t\u00eb ul\u00ebt p\u00ebrmbajn\u00eb t\u00eb gjitha informacionet q\u00eb identifikojn\u00eb p\u00ebrdoruesin. Kjo \u00ebsht\u00eb nj\u00eb qasje e mir\u00eb nga pik\u00ebpamja e privat\u00ebsis\u00eb, por nuk mund t\u00eb p\u00ebrdoret p\u00ebr analiza n\u00eb nivel p\u00ebrdoruesi.<\/li>\n<li>Grupet publike t\u00eb dh\u00ebnash (t\u00eb l\u00ebshuara jasht\u00eb Twitter) jan\u00eb t\u00eb disponueshme p\u00ebr t\u00eb gjith\u00eb punonj\u00ebsit e Twitter.<\/li>\n<\/ul>\n<p>\nSa i p\u00ebrket regjistrimit, ne p\u00ebrdor\u00ebm detyra t\u00eb planifikuara p\u00ebr t\u00eb listuar grupet e dh\u00ebnash BigQuery dhe p\u00ebr t'i regjistruar ato n\u00eb Data Access Layer (<noindex><a rel=\"nofollow\" href=\"https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2016\/discovery-and-consumption-of-analytics-data-at-twitter.html\">DAL<\/a><\/noindex>), depozita e metadatatit t\u00eb Twitter. P\u00ebrdoruesit do t\u00eb anotojn\u00eb grupet e dh\u00ebnash me informacion n\u00eb lidhje me privat\u00ebsin\u00eb, si dhe do t\u00eb specifikojn\u00eb koh\u00ebn e ruajtjes. Sa i p\u00ebrket pastrimit, ne po vler\u00ebsojm\u00eb performanc\u00ebn dhe kostot e dy opsioneve: <i>1.<\/i> Pastrimi i grupeve t\u00eb dh\u00ebnash n\u00eb GCS duke p\u00ebrdorur mjete si Scalding, dhe ngarkimi i tyre n\u00eb BigQuery; <i>2.<\/i> P\u00ebrdorimi i operacioneve BigQuery DML. Ne, me siguri, do t\u00eb p\u00ebrdorim nj\u00eb kombinim t\u00eb t\u00eb dyja metodave p\u00ebr t\u00eb p\u00ebrmbushur k\u00ebrkesat e grupeve dhe t\u00eb dh\u00ebnave t\u00eb ndryshme.<\/p>\n<h4>Funksionaliteti i sistemit<\/h4>\n<p>\nDuke, pasi q\u00eb BigQuery \u00ebsht\u00eb nj\u00eb sh\u00ebrbim i menaxhuar, nuk ishte e nevojshme t\u00eb p\u00ebrfshiheshin ekipet SRE t\u00eb Twitter n\u00eb menaxhimin e sistemeve ose realizimin e detyrave t\u00eb nat\u00ebs. Ishte e leht\u00eb t\u00eb sigurohej kapacitet i madh p\u00ebr ruajtje dhe llogaritje. Mund t\u00eb ndryshonim rezervimin e slot\u00ebve duke krijuar bileta n\u00eb mb\u00ebshtetje t\u00eb Google. E zbuluam se kishte mund\u00ebsi p\u00ebr p\u00ebrmir\u00ebsim, si shembull vet\u00eb-sh\u00ebrbimi p\u00ebr shp\u00ebrndarjen e slot\u00ebve dhe p\u00ebrmir\u00ebsimi i tabelave p\u00ebr monitorimin, dhe ia kaluam k\u00ebto k\u00ebrkesa n\u00eb Google.<\/p>\n<h4>\u00c7mimi<\/h4>\n<p>\nAnaliza jon\u00eb paraprake tregoi se kostot e k\u00ebrkesave p\u00ebr BigQuery dhe Presto ishin n\u00eb t\u00eb nj\u00ebjtin nivel. Kemi bler\u00eb slot\u00eb me <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#flat_rate_pricing\">\u00e7mim fiks<\/a><\/noindex> , p\u00ebr t\u00eb pasur nj\u00eb kosto t\u00eb q\u00ebndrueshme mujore n\u00eb vend q\u00eb t\u00eb paguanim <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/pricing#on_demand_pricing\">p\u00ebr k\u00ebrkes\u00eb<\/a><\/noindex> p\u00ebr TB t\u00eb dh\u00ebnash t\u00eb p\u00ebrpunuara. Kjo zgjidhje u bazua gjithashtu n\u00eb shqet\u00ebsimet e p\u00ebrdoruesve q\u00eb nuk d\u00ebshironin t\u00eb mendonin p\u00ebr kostot p\u00ebrpara se t\u00eb b\u00ebnin \u00e7do k\u00ebrkes\u00eb.<\/p>\n<p>Ruajtja e t\u00eb dh\u00ebnave n\u00eb BigQuery sjell shpenzime n\u00eb p\u00ebrve\u00e7 dhe kostot p\u00ebr GCS. Vegla si Scalding k\u00ebrkojn\u00eb q\u00eb grumbujt e t\u00eb dh\u00ebnave t\u00eb jen\u00eb n\u00eb GCS, dhe p\u00ebr t'u aksesuar me BigQuery duhej t\u00eb ngarkohej t\u00eb nj\u00ebjtin grumbull t\u00eb t\u00eb dh\u00ebnave n\u00eb formatin BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/blog\/products\/gcp\/inside-capacitor-bigquerys-next-generation-columnar-storage-format\">Capacitor<\/a><\/noindex>. Ne po punojm\u00eb p\u00ebr t\u00eb lidhur Scalding me grumbujt e t\u00eb dh\u00ebnave BigQuery, q\u00eb do t\u00eb eliminoj\u00eb nevoj\u00ebn p\u00ebr t\u00eb mbajtur grumbujt e t\u00eb dh\u00ebnave si n\u00eb GCS ashtu edhe n\u00eb BigQuery.<\/p>\n<p>P\u00ebr rastet e rralla q\u00eb k\u00ebrkonin k\u00ebrkesa t\u00eb pakta p\u00ebr dhjet\u00ebra petabayt, ne vendos\u00ebm se ruajtja e grumbujve t\u00eb t\u00eb dh\u00ebnave n\u00eb BigQuery nuk ishte e favorshme ekonomikisht dhe p\u00ebrdor\u00ebm Presto p\u00ebr qasje t\u00eb drejtp\u00ebrdrejt n\u00eb grumbujt e t\u00eb dh\u00ebnave n\u00eb GCS. P\u00ebr k\u00ebt\u00eb, ne po shqyrtojm\u00eb Burimet e t\u00eb Dh\u00ebnave t\u00eb Jashtme BigQuery.<\/p>\n<h3>Hapat e ardhsh\u00ebm<\/h3>\n<p>\nNe kemi v\u00ebn\u00eb re nj\u00eb interes t\u00eb madh p\u00ebr BigQuery q\u00eb nga l\u00ebshimi i alfa. Po shtojm\u00eb m\u00eb shum\u00eb grumbuj t\u00eb t\u00eb dh\u00ebnave dhe m\u00eb shum\u00eb ekipe n\u00eb BigQuery. Po zhvillojm\u00eb konektor\u00eb p\u00ebr veglat e analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave, si Scalding, p\u00ebr leximin dhe shkruarjen n\u00eb depozit\u00ebn BigQuery. Po shqyrtojm\u00eb vegla si Looker dhe Apache Zeppelin p\u00ebr t\u00eb krijuar raporte korporate mbi cil\u00ebsin\u00eb dhe sh\u00ebnime duke p\u00ebrdorur grumbujt e t\u00eb dh\u00ebnave BigQuery.<\/p>\n<p>Bashk\u00ebpunimi me Google ka qen\u00eb shum\u00eb produktiv, dhe ne jemi t\u00eb g\u00ebzuar t\u00eb vazhdojm\u00eb dhe zhvillojm\u00eb k\u00ebt\u00eb partneritet. Ne punuam me Google p\u00ebr t\u00eb zbatuar <noindex><a rel=\"nofollow\" href=\"https:\/\/developers.google.com\/issue-tracker\/guides\/partner-access\">Partner Issue Tracker<\/a><\/noindex>, p\u00ebr t\u00eb d\u00ebrguar k\u00ebrkesa drejtp\u00ebrdrejt n\u00eb Google. Disa prej tyre, si ngarkuesi BigQuery Parquet, jan\u00eb tashm\u00eb implementuar nga Google.<\/p>\n<p>Ja disa nga k\u00ebrkesat tona t\u00eb prioritetit t\u00eb lart\u00eb p\u00ebr Google:<\/p>\n<ul>\n<li>Vegla p\u00ebr pranimin e t\u00eb dh\u00ebnave n\u00eb m\u00ebnyr\u00eb t\u00eb leht\u00eb dhe mb\u00ebshtetje p\u00ebr formatin LZO-Thrift.<\/li>\n<li>Segmentimi me or\u00eb<\/li>\n<li>P\u00ebrmir\u00ebsime n\u00eb kontrollin e aksesit, si lejet n\u00eb nivel tabelash, rreshtash dhe kolonash.<\/li>\n<li>BigQuery <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/bigquery\/external-data-sources\">Burimet e t\u00eb Dh\u00ebnave t\u00eb Jashtme<\/a><\/noindex> me integrimin dhe mb\u00ebshtetje p\u00ebr Hive Metastore p\u00ebr formatin LZO-Thrift.<\/li>\n<li>Integrimi i p\u00ebrmir\u00ebsuar i katalogut t\u00eb t\u00eb dh\u00ebnave n\u00eb nd\u00ebrfaqen e p\u00ebrdoruesit BigQuery<\/li>\n<li>Vet\u00eb-sh\u00ebrbimi p\u00ebr shp\u00ebrndarjen dhe monitorimin e slot\u00ebve.<\/li>\n<\/ul>\n<p><\/p>\n<h3>P\u00ebrfundimi<\/h3>\n<p>\nDemokratizimi i analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave, vizualizimit dhe m\u00ebsimit t\u00eb makineris\u00eb n\u00eb nj\u00eb m\u00ebnyr\u00eb t\u00eb sigurt \u00ebsht\u00eb prioriteti m\u00eb i lart\u00eb p\u00ebr ekipin e Platform\u00ebs s\u00eb T\u00eb Dh\u00ebnave. Ne e kemi identifikuar Google BigQuery dhe Data Studio si vegla q\u00eb mund t\u00eb ndihmojn\u00eb n\u00eb arritjen e k\u00ebtij q\u00ebllimi, dhe kemi l\u00ebshuar vitin e kaluar BigQuery Alpha p\u00ebr t\u00eb gjith\u00eb kompanin\u00eb. <\/p>\n<p>Kemi zbuluar se k\u00ebrkesat n\u00eb BigQuery ishin t\u00eb thjeshta dhe efikase. P\u00ebr pranimin dhe transformimin e t\u00eb dh\u00ebnave kemi p\u00ebrdorur veglat e Google p\u00ebr konvikte t\u00eb thjeshta, por p\u00ebr konvikte t\u00eb nd\u00ebrlikuara na duhej t\u00eb nd\u00ebrtoshim infrastruktur\u00ebn ton\u00eb Airflow. N\u00eb fush\u00ebn e menaxhimit t\u00eb t\u00eb dh\u00ebnave, sh\u00ebrbimet BigQuery p\u00ebr autentifikim, autorizim dhe auditim p\u00ebrmbushin nevojat tona. P\u00ebr menaxhimin e metadatos dhe respektimin e privat\u00ebsis\u00eb, na nevojitej fleksibilitet m\u00eb i madh dhe na duhej t\u00eb krijonim sistemet tona. BigQuery, si nj\u00eb sh\u00ebrbim i menaxhuar, ishte i leht\u00eb p\u00ebr tu operuar. Kostot p\u00ebr k\u00ebrkesat ishin t\u00eb ngjashme me veglat ekzistuese. Ruajtja e t\u00eb dh\u00ebnave n\u00eb BigQuery solli shpenzime p\u00ebrve\u00e7 kostove p\u00ebr GCS.<\/p>\n<p>N\u00eb p\u00ebrgjith\u00ebsi, BigQuery funksionon mir\u00eb p\u00ebr analiz\u00ebn e p\u00ebrgjithshme SQL. V\u00ebm\u00eb re nj\u00eb interes t\u00eb madh p\u00ebr BigQuery, dhe ne po punojm\u00eb p\u00ebr t\u00eb transferuar m\u00eb shum\u00eb grumbuj t\u00eb t\u00eb dh\u00ebnave, angazhuar m\u00eb shum\u00eb ekipe dhe nd\u00ebrtuar m\u00eb shum\u00eb konvikte me BigQuery. N\u00eb Twitter p\u00ebrdoren t\u00eb dh\u00ebna t\u00eb ndryshme, p\u00ebr t\u00eb cilat do t\u00eb duhej nj\u00eb kombinim i veglave si Scalding, Spark, Presto dhe Druid. Ne synojm\u00eb t\u00eb vazhdojm\u00eb t\u00eb rrisim veglat tona t\u00eb analiz\u00ebs s\u00eb t\u00eb dh\u00ebnave dhe t'u japim p\u00ebrdoruesve tan\u00eb rekomandime t\u00eb qarta se si mund t\u00eb shfryt\u00ebzojn\u00eb m\u00eb s\u00eb miri ofertat tona.<\/p>\n<h3>Fjal\u00eb fal\u00ebnderimi<\/h3>\n<p>\nD\u00ebshiroj t\u00eb fal\u00ebnderoj bashk\u00ebautorit dhe shok\u00ebt e ekipit tim, Anju Dja dhe Will Pascucci, p\u00ebr bashk\u00ebpunimin e shk\u00eblqyer dhe pun\u00ebn e tyre t\u00eb madhe n\u00eb k\u00ebt\u00eb projekt. Gjithashtu, dua t\u00eb fal\u00ebnderoj inxhinier\u00ebt dhe menaxher\u00ebt nga disa ekipe n\u00eb Twitter dhe Google, t\u00eb cil\u00ebt na ndihmuan ne dhe p\u00ebrdoruesit e BigQuery n\u00eb Twitter, duke ofruar komente t\u00eb vlefshme.<\/p>\n<p>N\u00ebse jeni t\u00eb interesuar p\u00ebr t\u00eb punuar n\u00eb k\u00ebto detyra, shqyrtoni <noindex><a rel=\"nofollow\" href=\"https:\/\/careers.twitter.com\/content\/careers-twitter\/en\/jobs-search.html?q=&amp;team=careers-twitter%3Ateam%2Finfrastructure-engineering\">mund\u00ebsit\u00eb tona t\u00eb pun\u00ebs<\/a><\/noindex> n\u00eb ekipin e Data Platform.<\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eD9S\/\">Cil\u00ebsia e t\u00eb dh\u00ebnave n\u00eb DWH \u2014 konsistenca e depozit\u00ebs s\u00eb t\u00eb dh\u00ebnave<\/a><\/noindex><\/p>\n<p>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/513782\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb. \u0412 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441 \u0432\u0430\u043c\u0438 \u043f\u043e\u043b\u0435\u0437\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c. \u0427\u0438\u0442\u0430\u0442\u044c \u043f\u0435\u0440\u0432\u0443\u044e \u0447\u0430\u0441\u0442\u044c \u0423\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0421\u0438\u043b\u044c\u043d\u043e\u0435 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 (Strong Data Governance) \u2014 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u043f\u0440\u0438\u043d\u0446\u0438\u043f Twitter Engineering. \u041f\u043e\u0441\u043a\u043e\u043b\u044c\u043a\u0443 \u043c\u044b \u0432\u043d\u0435\u0434\u0440\u044f\u0435\u043c BigQuery \u0432 \u043d\u0430\u0448\u0443 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0443, \u043c\u044b \u043a\u043e\u043d\u0446\u0435\u043d\u0442\u0440\u0438\u0440\u0443\u0435\u043c\u0441\u044f \u043d\u0430 \u043e\u0431\u043d\u0430\u0440\u0443\u0436\u0435\u043d\u0438\u0438 \u0434\u0430\u043d\u043d\u044b\u0445, \u043a\u043e\u043d\u0442\u0440\u043e\u043b\u0435 \u0434\u043e\u0441\u0442\u0443\u043f\u0430, \u0431\u0435\u0437\u043e\u043f\u0430\u0441\u043d\u043e\u0441\u0442\u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91904,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91903","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.0.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.0.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-20T05:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Si BigQuery i Google demokratizoi analiz\u00ebn e t\u00eb dh\u00ebnave. Pjesa 2 | ProHoster","description":"P\u00ebrsh\u00ebndetje, Habr! Tani n\u00eb OTUS ka nj\u00eb regjistrim t\u00eb hapur p\u00ebr grupin e ri t\u00eb kursit \"Data Engineer\".","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a BigQuery \u043e\u0442 Google \u0434\u0435\u043c\u043e\u043a\u0440\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b \u0430\u043d\u0430\u043b\u0438\u0437 \u0434\u0430\u043d\u043d\u044b\u0445. \u0427\u0430\u0441\u0442\u044c 2 | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u044f\u043c\u043e \u0441\u0435\u0439\u0447\u0430\u0441 \u0432 OTUS \u043e\u0442\u043a\u0440\u044b\u0442 \u043d\u0430\u0431\u043e\u0440 \u043d\u0430 \u043d\u043e\u0432\u044b\u0439 \u043f\u043e\u0442\u043e\u043a \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kak-bigquery-ot-google-demokratiziroval-analiz-dannyh-chast-2","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-20T05:42:18+00:00","article:modified_time":"2020-08-20T05:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91903","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:18:28","updated":"2022-10-08 11:10:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=91903"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91903\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/91904"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=91903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=91903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=91903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}