Lumela, Habr! Ho ingolisa bakeng sa mokhoa o mocha oa thuto ho bulehile hona joale OTUS Ka tebello ea ho qala thupelo, re tsoela pele ho arolelana litaba tse molemo le uena.
Tsamaiso ea data
Taolo e matla ea data ke molao-motheo oa mantlha oa Twitter Engineering. Ha re ntse re kopanya BigQuery sethaleng sa rona, re tsepamisa maikutlo ho sibollo ea data, taolo ea phihlello, ts'ireletso le boinotšing.
Ho nolofalletsa ho sibolloa le taolo ea data, re ekelitse Layer ea ho Finyella Lintlha (DAL) ho kenyelletsa: ) ho fana ka lisebelisoa bakeng sa boitsebiso bo teng le ba Google Cloud, ho fana ka sebopeho se kopaneng le API bakeng sa basebelisi ba rona. Joalo ka Google Ha e ntse e tsoela pele ho fumaneha ka kakaretso, re tla e kenyelletsa mererong ea rona ho fa basebelisi likarolo tse kang ho batla likholumo.
BigQuery e etsa hore ho be bonolo ho arolelana le ho fihlella data, empa re ne re hloka taolo e itseng ho thibela ts'ebetso ea data. Har'a lisebelisoa tse ling, re khethile likarolo tse peli:
- : Karolo ea beta e thibelang basebelisi ho arolelana datha tsa BigQuery le basebelisi ba kantle ho Twitter.
- : Taolo e thibelang phetiso ea data mme e hloka hore basebelisi ba kene ho BigQuery ho tsoa methating e tsebahalang ea IP.
Re kentse tšebetsong litlhoko tsa netefatso, tumello le tlhahlobo (AAA) ho netefatsa ts'ireletso ka tsela e latelang:
- Netefatso: Re sebelisitse liakhaonto tsa basebelisi ba GCP bakeng sa likopo tsa maemo a tšohanyetso le liakhaonto tsa lits'ebeletso bakeng sa likopo tsa tlhahiso.
- Tumello: Re ne re hloka hore datha e ngoe le e ngoe e be le ak'haonte ea litšebeletso tsa beng le sehlopha sa babali.
- Tlhahlobo: Re rometse litlaleho tsa li-stack tsa BigQuery, tse nang le lintlha tse qaqileng tsa ho botsa, ho pokello ea boitsebiso ea BigQuery bakeng sa tlhahlobo e bonolo.
Ho netefatsa ts'ebetso e nepahetseng ea data ea basebelisi ba Twitter, re tlameha ho ngolisa datha tsohle tsa BigQuery, re hlakise lintlha tsa hau, re boloke polokelo e nepahetseng, 'me re hlakole (hlakola) data e hlakotsoeng ke basebelisi.
Re shebile Google , e sebelisang ho ithuta ka mochini ho hlophisa le ho hlophisa lintlha tsa bohlokoa, empa e khethile tlhaloso ea letsoho ea pokello ea lintlha ka lebaka la ho nepahala. Re rera ho sebelisa API ea Thibelo ea ho Lahleheloa ha data ho tlatsana le litlatsetso tsa tloaelo.
Ho Twitter, re thehile mekhahlelo e mene ea lekunutu bakeng sa li-datasets ho BigQuery, tse thathamisitsoeng mona ka tatellano e theohang ea kutloisiso:
- Li-datasets tsa bohlokoa haholo li fumaneha ka mokhoa oo ho hlokahalang ka oona ho latela molao-motheo oa menyetla e fokolang. Lethathamo le leng le le leng le na le sehlopha se ikemetseng sa babali, 'me re tla beha leihlo tšebeliso ea li-account tsa motho ka mong.
- Li-datasets tsa kutloisiso e mahareng (lintlha tsa maiketsetso tsa tsela e le 'ngoe tse sebelisang hashing e letsoai) ha li na lintlha tsa motho ka mong (PII) mme li fumaneha ho sehlopha se seholo sa basebetsi. Sena se fana ka teka-tekano e ntle lipakeng tsa litaba tsa lekunutu le ts'ebeliso ea data. Sena se lumella basebetsi ho etsa mesebetsi ea tlhahlobo, joalo ka ho bala palo ea basebelisi ba sebelisitseng karolo, ntle le ho tseba hore na basebelisi ba nnete ke bo-mang.
- Li-datasets tse fokolang tse nang le lintlha tsohle tse khethollang basebelisi. Ena ke mokhoa o motle ho latela pono ea lekunutu, empa e ke ke ea sebelisoa bakeng sa tlhahlobo ea boemo ba basebelisi.
- Lintlha tsa sechaba (tse lokollotsoeng ka ntle ho Twitter) li fumaneha ho basebeletsi bohle ba Twitter.
Ha e le mabapi le ho rema lifate, re sebelisitse mesebetsi e reriloeng ho bala li-dataset tsa BigQuery le ho li kenya ka har'a Layer ea ho Fumana Boitsebiso (), polokelo ea metadata ea Twitter. Basebelisi ba tla hlakisa li-dataset ka lintlha tsa lekunutu le ho bolela nako ea ho e boloka. Mabapi le ho hloekisa, re lekola ts'ebetso le litšenyehelo tsa likhetho tse peli: 1. Ho hloekisa li-database ho GCS ho sebelisa lisebelisoa tse kang Scalding le ho li kenya ho BigQuery; 2. Ho sebelisa li-operator tsa BigQuery DML. Mohlomong re tla sebelisa motsoako oa mekhoa e 'meli ho fihlela litlhoko tsa lihlopha tse fapaneng le data.
Ts'ebetso ea sistimi
Hobane BigQuery ke ts'ebeletso e laoloang, ho ne ho sa hlokahale ho kenyelletsa sehlopha sa Twitter sa SRE taolong ea sistimi kapa mesebetsing ea mohala. Ho fana ka bokhoni bo bongata bakeng sa polokelo le komporo ho ne ho le bonolo. Re ka fetola lipehelo tsa slot ka ho theha litekete ka tšehetso ea Google. Re hloaile libaka tse lokelang ho ntlafatsoa, joalo ka kabo ea ho ithusa le li-deshboard tse ntlafalitsoeng tsa tlhahlobo, 'me ra fetisetsa likopo tsena ho Google.
ditjeho tsa ho
Tlhahlobo ea rona ea pele e bonts'itse hore litšenyehelo tsa ho botsa BigQuery le Presto li ne li tšoana. Re ile ra reka slots ho theko ho ba le litšenyehelo tse tsitsitseng tsa khoeli le khoeli sebakeng sa ho lefa ka TB ea data e sebetsitsoeng. Qeto ena e ne e ipapisitse le maikutlo a basebelisi, ba neng ba sa batle ho nahana ka litšenyehelo pele ba etsa potso ka 'ngoe.
Ho boloka lintlha ho BigQuery ho kentse litšenyehelo ho GCS. Lisebelisoa tse kang Scalding li hloka li-datasets ho GCS, 'me ho fihlella BigQuery, re ile ra tlameha ho kenya li-dataset tse tšoanang ka sebopeho sa BigQuery. Re ntse re sebeletsa ho hokela Scalding ho li-dataset tsa BigQuery, e leng se tla felisa tlhoko ea ho boloka datha ho GCS le BigQuery.
Bakeng sa linyeoe tse sa tloaelehang tse hlokang ho botsoa khafetsa ka mashome a li-petabyte, re nkile qeto ea hore ho boloka li-database ho BigQuery ho ne ho sa bitse chelete e ngata 'me re sebelisitse Presto bakeng sa phihlello e tobileng ho li-database tsa GCS. Bakeng sa sena, re nahana ka Mehloli ea Boitsebiso ea Kantle ea BigQuery.
Mehato e latelang
Re bone thahasello e kholo ho BigQuery ho tloha ha alpha e lokolloa. Re eketsa li-database le litaelo ho BigQuery. Re nts'etsapele likhokahano tsa lisebelisoa tsa tlhahlobo ea data joalo ka Scalding ho bala le ho ngolla polokelo ea BigQuery. Re ntse re hlahloba lisebelisoa tse kang Looker le Apache Zeppelin bakeng sa ho etsa litlaleho tsa boleng ba khoebo le litlhaloso re sebelisa li-database tsa BigQuery.
Tšebelisano ea rona le Google e bile le litholoana tse ntle haholo, 'me re thabela ho tsoela pele le ho nts'etsapele ts'ebelisano ena. Re sebelitse le Google ho kenya tšebetsong ea rona ho romella lipotso ka kotloloho ho Google. Tse ling tsa tsona, joalo ka BigQuery Parquet loader, li se li kentsoe tšebetsong ke Google.
Mona ke tse ling tsa likopo tsa rona tse tlang pele ho Google:
- Lisebelisoa tsa ho amohela data le tšehetso e bonolo bakeng sa sebopeho sa LZO-Thrift.
- Karohano ea hora
- Lintlafatso taolong ea phihlello, joalo ka tafole, mola, le litumello tsa boemo ba kholomo.
- kgolohadi ka kopanyo ea Hive Metastore le tšehetso bakeng sa sebopeho sa LZO-Thrift.
- Khokahano e ntlafetseng ea Lethathamo la Boitsebiso ho BigQuery UI
- Ho itšebeletsa bakeng sa kabo ea slot le ho beha leihlo.
fihlela qeto e
Ho etsa tlhahlobo ea lintlha tsa demokrasi, ho bona, le ho ithuta ka mochini ka mokhoa o sireletsehileng ke lintho tse tlang pele ho sehlopha sa Data Platform. Re khethile Google BigQuery le Studio Studio e le lisebelisoa tse ka re thusang ho fihlela sepheo sena, 'me selemong se fetileng re lokolotse BigQuery Alpha ho khamphani kaofela.
Re fumane hore lipotso tsa BigQuery li bonolo ebile li sebetsa hantle. Bakeng sa ho kenya le ho fetola lintlha, re sebelisitse lisebelisoa tsa Google bakeng sa lipeipi tse bonolo, empa bakeng sa liphaephe tse rarahaneng, re ile ra tlameha ho iketsetsa lisebelisoa tsa rona tsa Airflow. Bakeng sa taolo ea data, netefatso ea BigQuery, tumello, le lits'ebeletso tsa tlhahlobo ea libuka li ile tsa fihlela litlhoko tsa rona. Bakeng sa taolo ea metadata le boinotšing, re ne re hloka ho fetoha le maemo 'me re ile ra tlameha ho iketsetsa litsamaiso tsa rona. BigQuery, joalo ka ts'ebeletso e laoloang, e ne e le bonolo ho sebetsa. Litšenyehelo tsa ho botsa li ne li tšoana le lisebelisoa tse teng. Ho boloka boitsebiso ho BigQuery ho bile le litšenyehelo ho phaella ho tsa GCS.
Ka kakaretso, BigQuery e sebetsa hantle bakeng sa tlhahlobo e akaretsang ea SQL. Re bone thahasello e kholo ho BigQuery, 'me re ntse re sebeletsa ho fallisa datha tse ling, ho sebelisana le lihlopha tse ngata, le ho haha liphaephe tse ling tsa BigQuery. Twitter e sebelisa mefuta e fapaneng ea li-datasets, tse tla hloka motsoako oa lisebelisoa tse kang Scalding, Spark, Presto, le Druid. Re ikemiselitse ho tsoela pele ho theha lisebelisoa tsa rona tsa tlhahlobo ea data le ho fana ka tataiso e hlakileng ho basebelisi ba rona mabapi le mokhoa oa ho sebelisa linyehelo tsa rona hantle.
Mantsoe a teboho
Ke rata ho leboha bangoli-'moho le 'na, Anju Jha le Will Pascucci, ka tšebelisano-'moho ea bona e babatsehang le mosebetsi o boima mosebetsing ona. Ke rata hape ho leboha baenjiniere le batsamaisi ba lihlopha tse 'maloa tsa Twitter le Google ba re thusitseng, le basebelisi ba BigQuery ho Twitter ba faneng ka maikutlo a bohlokoa.
Haeba u thahasella ho sebetsa mesebetsing ena, ka kopo sheba rona sehlopheng sa Data Platform.
Source: www.habr.com
