Nhoroondo yeyakavhurika sosi yedu: maitiro atakaita sevhisi yekuongorora muGo uye kuita kuti iwanikwe pachena

Parizvino, inenge kambani yese pasi rose inounganidza zviverengero nezvezviito zvevashandisi pawebhu sosi. Kukurudzira kwakajeka - makambani anoda kuziva kuti chigadzirwa / webhusaiti yavo inoshandiswa sei uye kunzwisisa zviri nani vashandisi vavo. Ehe, kune huwandu hukuru hwezvishandiso pamusika kugadzirisa dambudziko iri - kubva kune analytics masisitimu anopa data muchimiro chedhibhodhi uye magirafu (semuenzaniso. Google Analytics) kune Mutengi Data Platform, iyo inokutendera iwe kuunganidza uye kuunganidza data kubva kwakasiyana masosi mune chero imba yekuchengetera (semuenzaniso Chikamu).

Asi takawana dambudziko risati ragadziriswa. Ndizvo zvakazvarwa EventNative -Open-source analytics sevhisi. Verenga nezve chikonzero nei takasarudza kuvandudza sevhisi yedu, zvayakatipa, uye kuti mhedzisiro yaive yei (nezvidimbu zvekodhi).

Nei tichifanira kukudziridza basa redu pachedu?

Akanga ari makore makumi mapfumbamwe, takararama nepose pataigona. 2019, takagadzira iyo API Yekutanga Mutengi Data Platform kSense, izvo zvakaita kuti zvikwanise kuunganidza data kubva kwakasiyana masosi (Facebook ads, Stripe, Salesforce, Google play, Google Analytics, nezvimwewo) kuti iwedzere nyore kuongorora data, kuona zvinovimbika, nezvimwe. Takaona kuti vashandisi vazhinji vanoshandisa chikuva chedu chekuongorora data kunyanya Google Analytics (pano GA). Takataura nevamwe vashandisi uye takaona kuti vanoda data yeanalytics yechigadzirwa chavo chavanotambira vachishandisa GA, asi. Google samples data uye kune vazhinji, iyo GA Mushandisi interface haisi iyo chiyero chekureruka. Takava nehurukuro dzakakwana nevashandisi vedu uye takaona kuti vazhinji vaishandisawo Segment platform (iyo, nenzira, yaingova rimwe zuva. yakatengeswa ne3.2 bhiriyoni).

Vakaisa Segment javascript pixel pane yavo webhu sosi uye data nezve maitiro evashandisi vavo yakaiswa mudura rakatsanangurwa (semuenzaniso Postgres). Asi Segment zvakare ine kuderera kwayo - mutengo. Semuenzaniso, kana webhu saiti ine 90,000 MTU (mwedzi inoteverwa vashandisi), saka iwe unofanirwa kubhadhara ~ 1,000 $ pamwedzi kune cashier. Paive zvakare nedambudziko rechitatu - mamwe mabrowser ekuwedzera (seAdBlock) akavhara kuunganidzwa kweanalytics nekuti... http zvikumbiro kubva kubrowser zvakatumirwa kuGA uye Segment domains. Zvichienderana nezvido zvevatengi vedu, isu takagadzira sevhisi yekuongorora iyo inounganidza seti yakazara yedata (isina sampling), yemahara uye inogona kushanda pachedu.

Mashandiro anoita sevhisi

Iyo sevhisi ine zvikamu zvitatu: javascript pixel (iyo yatakazonyora zvakare mutypescript), sevha chikamu chinoitwa mumutauro weGO, uye yakarongwa kushandisa Redshift neBigQuery sedura remukati (gare gare ivo vakawedzera rutsigiro Postgres, ClickHouse uye Snowflake).

Zvakasarudzwa kusiya chimiro cheGA uye Segment zviitiko zvisina kuchinjwa. Chaingodiwa kwaive kudzokorora zviitiko zvese kubva pawebhu sosi iyo pixel inoiswa kune yedu backend. Sezvazvinoitika, izvi hazvina kuoma kuita. Iyo Javascript pixel yakapfuura iyo yekutanga raibhurari nzira yeGA neitsva, iyo yakadhirodha chiitiko ichi musystem yedu.

//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
    ga(tracker => {
        var originalSendHitTask = tracker.get('sendHitTask');
        tracker.set('sendHitTask', (model) => {
            var payLoad = model.get('hitPayload');
            //отправка оригинального события в GA
            originalSendHitTask(model);
            let jsonPayload = this.parseQuery(payLoad);
            //отправка события в наш сервис
            this.send3p('ga', jsonPayload);
        });
    });
}

NeSegment pixel zvese zviri nyore; ine middleware nzira, imwe yatakashandisa.

//'analytics' - стандартное название переменной Segment
if (window.analytics) {
    if (window.analytics.addSourceMiddleware) {
        window.analytics.addSourceMiddleware(chain => {
            try {
		//дублирование события в наш сервис
                this.send3p('ajs', chain.payload);
            } catch (e) {
                LOG.warn('Failed to send an event', e)
            }
	    //отправка оригинального события в Segment
            chain.next(chain.payload);
        });
    } else {
        LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
    }
} else {
    LOG.warn('Analytics.js listener is not set.');
}

Pamusoro pekukopa zviitiko, isu takawedzera kugona kutumira zvisina tsarukano json:

//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
    product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
    product_price: 399.99,
    price_currency: 'USD'
    product_release_start: '2020-09-25T12:38:27.763000Z'
});

Tevere, ngatitaure nezve server chikamu. Iyo backend inofanirwa kugamuchira zvikumbiro zve http, zvizadze nerumwe ruzivo, semuenzaniso, geo data (ndatenda maxmind nokuda kweizvi) uye nyora mudhatabhesi. Isu taida kuita kuti sevhisi ive nyore sezvinobvira kuitira kuti igone kushandiswa nekugadziriswa kushoma. Isu takaita mashandiro ekuona iyo data schema zvichienderana nechimiro cheinouya json chiitiko. Mhando dze data dzinotsanangurwa nehunhu. Nested zvinhu zvinoora uye zvakaderedzwa kuita flat chimiro:

//входящий json
{
  "field_1":  {
    "sub_field_1": "text1",
    "sub_field_2": 100
  },
  "field_2": "text2",
  "field_3": {
    "sub_field_1": {
      "sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
    }
  }
}

//результат
{
  "field_1_sub_field_1":  "text1",
  "field_1_sub_field_2":  100,
  "field_2": "text2",
  "field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}

Nekudaro, arrays parizvino anongoshandurwa kuita tambo nekuti Haasi ese ane hukama dhatabhesi anotsigira akadzokororwa minda. Zvinogonekawo kushandura mazita eminda kana kuadzima uchishandisa sarudzo yemitemo yemepu. Ivo vanokutendera iwe kuti uchinje iyo data schema kana zvichidikanwa kana kushandura imwe mhando yedata kune imwe. Semuenzaniso, kana munda wejson uine tambo ine timestamp (field_3_sub_field_1_sub_sub_field_1 kubva pamuenzaniso uri pamusoro), saka kuti ugadzire munda mudhatabhesi nerudzi rwestamp yenguva, unofanirwa kunyora mutemo wemepu mukugadzirisa. Mune mamwe mazwi, iyo data data yemunda inotemerwa kutanga neiyo json kukosha, uyezve mhando yekukanda mutemo (kana yakagadziriswa) inoshandiswa. Taona mhando ina dzemadata makuru: STRING, FLOAT4, INT64 uye TIMESTAMP. Iyo mepu uye yemhando yekukanda mitemo inoita seizvi:

rules:
  - "/field_1/subfield_1 -> " #правило удаления поля
  - "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
  - "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа

Algorithm yekusarudza rudzi rwe data:

• shandura json chimiro kuita flat chimiro
• kusarudza mhando yedata yeminda nemitengo
• kushandisa mepu uye kutaipa mitemo yekukanda

Zvadaro kubva kune inouya json chimiro:

{
    "product_id":  "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
    "product_price": 399.99,
    "price_currency": "USD",
    "product_type": "supplies",
    "product_release_start": "2020-09-25T12:38:27.763000Z",
    "images": {
      "main": "picture1",
      "sub":  "picture2"
    }
}

iyo data schema ichawanikwa:

"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying

Isu takafunga zvakare kuti mushandisi anofanira kukwanisa kugadzirisa kugovera kana kupatsanura data mudhatabhesi zvinoenderana nemamwe maitiro uye kuita kugona kuseta zita retafura nekusingaperi kana. kutaura mukugadzirisa. Mumuenzaniso uri pazasi, chiitiko chichachengetwa patafura ine zita rakaverengerwa zvichienderana nehunhu hwechigadzirwa_type uye _timestamp minda (semuenzaniso. zvekushandisa_2020_10):

tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'

Nekudaro, chimiro chezviitiko zvinouya chinogona kuchinja panguva yekumhanya. Isu takaita algorithm yekutarisa musiyano pakati pechimiro chetafura iripo uye chimiro chechiitiko chiri kuuya. Kana mutsauko ukawanikwa, tafura ichagadziridzwa neminda mitsva. Kuti uite izvi, shandisa chigamba SQL mubvunzo:

#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varying

akitekicha

Sei uchifanira kunyora zviitiko kune faira system, uye kwete kungozvinyora zvakananga kune database? Databases haiwanzo kuita zvakanaka kana ichibata nehuwandu hukuru hwekuisa (Postgres mazano) Kuti uite izvi, Logger inonyora zviitiko zvinouya kune faira uye mune yakaparadzana goroutine (shinda) File reader inoverenga faira, ipapo iyo data inoshandurwa uye yakatemwa. Mushure mekunge Tafura maneja ave nechokwadi chekuti tafura schema iri-kusvika-zuva, iyo data ichanyorwa kune dhatabhesi mune rimwe batch. Zvadaro, isu takawedzera kugona kunyora data zvakananga kune dhatabhesi, asi isu tinoshandisa iyi modhi kune zviitiko zvisina kuwanda - semuenzaniso, kutendeuka.

Open Source uye zvirongwa zvenguva yemberi

Pane imwe nguva, sevhisi yakatanga kutaridzika sechigadzirwa chakazara uye isu takasarudza kuiburitsa kuOpen Source. Parizvino, kubatanidzwa nePostgres, ClickHouse, BigQuery, Redshift, S3, Snowflake yakaitwa. Zvese zvakabatanidzwa zvinotsigira ese batch uye yekufambisa modhi yekurodha data. Yakawedzera rutsigiro rwezvikumbiro kuburikidza neAPI.

Ikozvino yekubatanidza chirongwa chinotaridzika seizvi:

Kunyangwe iyo sevhisi inogona kushandiswa yakazvimiririra (semuenzaniso kushandisa Docker), isu tinewo hosted version, umo iwe unogona kuseta kubatanidzwa nedura re data, wedzera CNAME kune yako domain uye tarisa manhamba ehuwandu hwezviitiko. Zvirongwa zvedu zvekukurumidza ndezvekuwedzera kugona kuunganidza kwete chete zviverengero kubva pawebhu sosi, asiwo data kubva kunze data masosi uye uzvichengete kune chero chengetedzo yesarudzo yako!

→ GitHub
→ Zvinyorwa
→ usimbe

Tichafara kana EventNative ikabatsira kugadzirisa matambudziko ako!

Vashandisi vakanyoresa chete ndivo vanogona kutora chikamu muongororo. Nyorera mu, Munogamuchirwa.

Ndeipi nhamba yekuunganidza system inoshandiswa mukambani yako?

• 48,0%Google Analytics12

• 4,0%Chikamu1

• 16,0%Imwe (nyora mumashoko)4

• 32,0%Wakaita sevhisi yako8

25 vashandisi vakavhota. 6 vashandisi vakaramba.

Source: www.habr.com