Hona joale, hoo e batlang e le k'hamphani e 'ngoe le e' ngoe lefatšeng e bokella lipalo-palo mabapi le liketso tsa mosebedisi ho mohloli oa marang-rang. Tšusumetso e hlakile - lik'hamphani li batla ho tseba hore na sehlahisoa sa bona / sebaka sa marang-rang se sebelisoa joang le ho utloisisa basebelisi ba bona hamolemo. Ehlile, ho na le lisebelisoa tse ngata 'marakeng ho rarolla bothata bona - ho tsoa lits'ebetsong tsa analytics tse fanang ka data ka mokhoa oa li-dashboards le li-graph (mohlala. ) ho Customer Data Platform, e u lumellang ho bokella le ho bokella lintlha ho tsoa mehloling e fapaneng sebakeng sa polokelo efe kapa efe (mohlala ).
Empa re fumane bothata bo so kang bo rarolloa. Ho ile ha tsoaloa joalo - ts'ebeletso ea tlhahlobo ea mohloli o bulehileng. Bala ka hore na ke hobane'ng ha re nkile qeto ea ho nts'etsapele ts'ebeletso ea rona, hore na e re file eng, le hore na phello e bile efe (ka likotoana tsa khoutu).
Ke hobane'ng ha re lokela ho ntlafatsa tšebeletso ea rona?
E ne e le lilemo tsa bo-2019, re ile ra pholoha ka hohle kamoo re ka khonang. Ka XNUMX, re thehile API First Customer Data Platform kSense, e entseng hore ho khonehe ho bokella lintlha tse tsoang mehloling e fapaneng (lipapatso tsa Facebook, Stripe, Salesforce, Google play, Google Analytics, joalo-joalo) bakeng sa tlhahlobo e bonolo haholoanyane ea data, ho khetholla litšepiso, joalo-joalo. Re hlokometse hore basebelisi ba bangata ba sebelisa sethala sa rona bakeng sa tlhahlobo ea data ka ho khetheha Google Analytics (ka mor'a mona GA). Re buile le basebelisi ba bang mme ra fumana hore ba hloka lintlha tsa analytics bakeng sa sehlahisoa sa bona seo ba se fumanang ba sebelisa GA, empa mme ho ba bangata, GA User interface ha se maemo a bonolo. Re bile le lipuisano tse lekaneng le basebelisi ba rona mme ra hlokomela hore ba bangata ba ne ba sebelisa sethala sa Segment (eo, ka tsela, e neng e le tsatsi le leng feela. ).
Ba kentse pixel ea Segment javascript mohloling oa bona oa marang-rang mme data mabapi le boitšoaro ba basebelisi ba bona e kentsoe polokelong e boletsoeng (mohlala Postgres). Empa Segment e boetse e na le mathata a eona - theko. Ka mohlala, haeba mohloli oa marang-rang o na le 90,000 MTU (basebelisi ba lateloang khoeli le khoeli), joale u lokela ho lefa ~ 1,000 $ ka khoeli ho motho ea kenang chelete. Ho ne ho boetse ho e-na le bothata ba boraro - li-extensions tse ling tsa sebatli (joalo ka AdBlock) li thibetse pokello ea analytics hobane ... Likopo tsa http tse tsoang ho sebatli li rometsoe libakeng tsa GA le Segment. Ho ipapisitsoe le litakatso tsa bareki ba rona, re thehile ts'ebeletso ea li-analytics e bokellang lethathamo le felletseng la data (ntle le sampole), e sa lefelloeng 'me e ka sebetsa molemong oa rona.
Tšebeletso e sebetsa joang
Ts'ebeletso e na le likarolo tse tharo: pixel ea javascript (eo hamorao re ileng ra e ngola hape ka mongolo), karolo ea seva e kengoa ts'ebetsong ka puo ea GO, 'me ho ne ho reriloe ho sebelisa Redshift le BigQuery joalo ka polokelong ea ntlo (hamorao ba ile ba eketsa tšehetso bakeng sa Postgres, ClickHouse le Snowflake).
Ho ile ha etsoa qeto ea ho tlohela sebopeho sa liketsahalo tsa GA le Segment se sa fetoha. Sohle se neng se hlokahala e ne e le ho qopitsa liketsahalo tsohle ho tsoa mohloling oa marang-rang moo pixel e kentsoeng ho backend ea rona. Joalokaha ho bonahala, sena ha se thata ho se etsa. Pixel ea Javascript e tlositse mokhoa oa mantlha oa laeborari ea GA ka e ncha, e kopitsitseng ketsahalo eo tsamaisong ea rona.
//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
ga(tracker => {
var originalSendHitTask = tracker.get('sendHitTask');
tracker.set('sendHitTask', (model) => {
var payLoad = model.get('hitPayload');
//отправка оригинального события в GA
originalSendHitTask(model);
let jsonPayload = this.parseQuery(payLoad);
//отправка события в наш сервис
this.send3p('ga', jsonPayload);
});
});
}
Ka Segment pixel ntho e 'ngoe le e' ngoe e bonolo haholoanyane; e na le mekhoa e mahareng, eo e 'ngoe ea eona re e sebelisitseng.
//'analytics' - стандартное название переменной Segment
if (window.analytics) {
if (window.analytics.addSourceMiddleware) {
window.analytics.addSourceMiddleware(chain => {
try {
//дублирование события в наш сервис
this.send3p('ajs', chain.payload);
} catch (e) {
LOG.warn('Failed to send an event', e)
}
//отправка оригинального события в Segment
chain.next(chain.payload);
});
} else {
LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
}
} else {
LOG.warn('Analytics.js listener is not set.');
}
Ntle le ho kopitsa liketsahalo, re ekelitse bokhoni ba ho romella json ntle le mabaka:
//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
product_price: 399.99,
price_currency: 'USD'
product_release_start: '2020-09-25T12:38:27.763000Z'
});
E latelang, a re bue ka karolo ea seva. The backend e lokela ho amohela likopo tsa http, li tlatse ka tlhahisoleseling e eketsehileng, mohlala, data ea geo (kea leboha bakeng sa sena) 'me u e tlalehe ka har'a database. Re ne re batla ho etsa hore tšebeletso e be bonolo kamoo ho ka khonehang e le hore e ka sebelisoa ka mokhoa o fokolang. Re kentse tšebetsong ts'ebetso ea ho khetholla schema ea data ho latela sebopeho sa ketsahalo ea json e tlang. Mefuta ea data e hlalosoa ka boleng. Lintho tse hahelletsoeng lia bola 'me li fokotsoe ho ba sebopeho se bataletseng:
//входящий json
{
"field_1": {
"sub_field_1": "text1",
"sub_field_2": 100
},
"field_2": "text2",
"field_3": {
"sub_field_1": {
"sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}
}
}
//результат
{
"field_1_sub_field_1": "text1",
"field_1_sub_field_2": 100,
"field_2": "text2",
"field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}Leha ho le joalo, li-arrays hajoale li fetoloa likhoele hobane Ha se li-database tsohle tsa likamano tse tšehetsang likarolo tse pheta-phetoang. Hape hoa khoneha ho fetola mabitso a libaka kapa ho a hlakola ka ho sebelisa melao ea boikhethelo ea 'mapa. Li u lumella ho fetola schema ea data ha ho hlokahala kapa ho fetolela mofuta o mong oa data ho o mong. Mohlala, haeba lebala la json le na le khoele e nang le setempe sa linako (lebala_3_tlase_1_tlale_le_1 ho tloha mohlaleng o ka holimo), joale e le hore u thehe tšimo sebakeng sa polokelo ea litaba ka mofuta oa setempe sa nako, u lokela ho ngola molao oa 'mapa ho tlhophiso. Ka mantsoe a mang, mofuta oa data oa tšimo o khethoa pele ka boleng ba json, ebe ho sebelisoa molao oa mofuta oa mofuta (haeba o lokiselitsoe). Re khethile mefuta e 4 ea mantlha ea data: STRING, FLOAT64, INT64 le TIMESTAMP. Melao ea 'mapa le mofuta oa ho lahla e shebahala tjena:
rules:
- "/field_1/subfield_1 -> " #правило удаления поля
- "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
- "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа
Algorithm ea ho khetha mofuta oa data:
- fetola sebopeho sa json ho sebopeho se bataletseng
- ho khetholla mofuta oa data oa masimo ka boleng
- ho sebelisa melao ea 'mapa le ho thaepa melao ea ho etsa
Ebe ho tsoa ho sebopeho sa json se tlang:
{
"product_id": "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
"product_price": 399.99,
"price_currency": "USD",
"product_type": "supplies",
"product_release_start": "2020-09-25T12:38:27.763000Z",
"images": {
"main": "picture1",
"sub": "picture2"
}
}
schema ea data e tla fumanoa:
"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying
Re boetse re nahanne hore mosebelisi o lokela ho khona ho hlophisa karohano kapa ho arola data ho database ho latela mekhoa e meng le ho kenya ts'ebetsong bokhoni ba ho beha lebitso la tafole ka nako le nako kapa tlhophisong. Mohlaleng o ka tlase, ketsahalo e tla bolokoa tafoleng e nang le lebitso le baloang ho latela boleng ba sehlahisoa_mofuta le masimo a _timestamp (mohlala. thepa_2020_10):
tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'
Leha ho le joalo, sebopeho sa liketsahalo tse tlang se ka fetoha ka nako ea ts'ebetso. Re kentse tšebetsong algorithm ho lekola phapang lipakeng tsa sebopeho sa tafole e teng le sebopeho sa ketsahalo e tlang. Haeba phapang e fumaneha, tafole e tla nchafatsoa ka likarolo tse ncha. Ho etsa sena, sebelisa potso ea patch SQL:
#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varyingmehaho
Ke hobane'ng ha u hloka ho ngola liketsahalo ho tsamaiso ea lifaele, 'me u se ke ua li ngolla ka ho toba ho database? Li-database ha li sebetse hantle kamehla ha li sebetsana le lipalo tse ngata tse kentsoeng (). Ho etsa sena, Logger o ngola liketsahalo tse kenang faeleng 'me ka goroutine e arohaneng (khoele) Sebali sa faele se bala faele, ebe data e fetoloa ebe e khethoa. Ka mor'a hore mookameli oa Letlapa a etse bonnete ba hore schema ea tafole e nakong, data e tla ngolloa ho database ka batch e le 'ngoe. Ka mor'a moo, re ile ra eketsa bokhoni ba ho ngola lintlha ka ho toba ho database, empa re sebelisa mokhoa ona bakeng sa liketsahalo tse seng ngata - ka mohlala, liphetoho.
Mohloli o Bulehileng le merero ea bokamoso
Ka nako e 'ngoe, tšebeletso e ile ea qala ho shebahala eka ke sehlahisoa se feletseng' me ra etsa qeto ea ho e lokolla ho Open Source. Hajoale, likhokahano le Postgres, ClickHouse, BigQuery, Redshift, S3, Snowflake li kentsoe tšebetsong. Likopano tsohle li tšehetsa mekhoa ea batch le ea ho phallela ea ho kenya data. Tšehetso e ekelitsoeng bakeng sa likopo ka API.
Morero oa hona joale oa ho kopanya o shebahala tjena:
Le ha ts'ebeletso e ka sebelisoa ka boikemelo (mohlala ho sebelisa Docker), le rona re na le eona , eo ho eona u ka hlophisang ho kopanya le polokelo ea data, eketsa CNAME sebakeng sa hau sa marang-rang 'me u shebe lipalo-palo tsa palo ea liketsahalo. Merero ea rona ea hang-hang ke ho eketsa bokhoni ba ho kopanya eseng feela lipalo-palo ho tsoa mohloling oa marang-rang, empa hape le lintlha tse tsoang mehloling ea data ea kantle le ho li boloka polokelong efe kapa efe eo u e ratang!
→
→
→
Re tla thaba haeba EventNative e thusa ho rarolla mathata a hau!
Ke basebelisi ba ngolisitsoeng feela ba ka kenyang letsoho phuputsong. ka kopo.
Ke sistimi efe ea pokello ea lipalo e sebelisoang k'hamphaning ea hau?
-
48,0%Google Analytics12
-
4,0%Karolo ea 1
-
16,0%E 'ngoe (ngola litlhalosong)4
-
32,0%O kentse tshebetsong tshebeletso ya hao8
Basebelisi ba 25 ba ile ba khetha. Basebelisi ba 6 ba ile ba hana.
Source: www.habr.com