Umlando womthombo wethu ovulekile: indlela esenze ngayo isevisi yezibalo ku-Go futhi sayenza yatholakala esidlangalaleni

Njengamanje, cishe yonke inkampani emhlabeni iqoqa izibalo mayelana nezenzo zabasebenzisi kusisetshenziswa sewebhu. Isisusa sicacile - izinkampani zifuna ukwazi ukuthi umkhiqizo/iwebhusayithi yazo isetshenziswa kanjani futhi zibaqonde kangcono abasebenzisi bazo. Yiqiniso, kunenani elikhulu lamathuluzi emakethe okuxazulula le nkinga - kusukela kuzinhlelo zokuhlaziya ezihlinzeka ngedatha ngesimo samadeshibhodi namagrafu (isibonelo. -Google Analytics) ku-Customer Data Platform, ekuvumela ukuthi uqoqe futhi uhlanganise idatha emithonjeni ehlukene kunoma yisiphi isitoreji (isibonelo, Ingxenye).

Kodwa sithole inkinga engakaxazululwa. Wazalwa kanjalo I-EventNative - isevisi yokuhlaziya umthombo ovulekile. Mayelana nokuthi kungani siye ukuze sithuthukise insizakalo yethu, ukuthi yasinika ini nokuthi kwenzekani ekugcineni (ngezingcezu zekhodi), funda ngaphansi kokusikwa.

Kungani kufanele sithuthukise eyethu inkonzo?

Kwakuyiminyaka yamashumi ayisishiyagalolunye, sasinda ngokusemandleni ethu. 2019, sithuthukise i-First Customer Data Platform API kSense, evumele ukuhlanganisa idatha evela emithonjeni ehlukene (izikhangiso ze-Facebook, i-Stripe, i-Salesforce, i-Google play, i-Google Analytics, njll.) ukuze uthole ukuhlaziywa kwedatha okulula kakhulu, ukuhlonza okuncikile, njll. Siqaphele ukuthi abasebenzisi abaningi basebenzisa inkundla yethu yokuhlaziya idatha, ikakhulukazi i-Google Analytics (eyaziwa lapha nge-GA). Sikhulume nabanye abasebenzisi futhi sathola ukuthi badinga idatha yezibalo zomkhiqizo wabo, abayithola kusetshenziswa i-GA, kodwa Amasampula edatha ye-Google futhi kwabaningi i-GA User Interface akuyona indinganiso yokufaneleka. Sibe nezingxoxo ezanele nabasebenzisi bethu futhi sabona ukuthi abaningi baphinde basebenzise inkundla Yesigaba (okuyinto, ngendlela, bekuyizinsuku ezimbalwa ezedlule. ithengiswe ngamaRandi ayizigidi eziyizinkulungwane ezingu-3.2).

Bafake i-Segment javascript pixel kusisetshenziswa sabo sewebhu futhi idatha yabo yokuziphatha komsebenzisi yalayishwa kusizindalwazi esicacisiwe (isb. Postgres). Kodwa Ingxenye nayo inokususa kwayo - inani. Isibonelo, uma insiza yewebhu ine-90,000 MTU (abasebenzisi abalandelwa njalo ngenyanga), kuzomele ukhokhe ~ $1,000 ngenyanga kumuntu ophatha imali. Kuphinde kwaba nenkinga yesithathu - ezinye izandiso zesiphequluli (ezifana ne-AdBlock) zivimbe ukuqoqwa kwezibalo. Izicelo ze-http ezisuka kusiphequluli zithunyelwe kusizinda se-GA kanye neSigaba. Ngokusekelwe esifisweni samakhasimende ethu, sidale isevisi yezibalo eqoqa isethi egcwele yedatha (ngaphandle kwamasampula), mahhala futhi engasebenza kwingqalasizinda yethu.

Isebenza kanjani isevisi

Le sevisi iqukethe izingxenye ezintathu: iphikseli ye-javascript (esiphinde siyibhale kabusha ku-typescript), ingxenye yeseva esetshenziswe ngolimi lwe-GO, futhi kwakuhlelwe ukuthi kusetshenziswe i-Redshift ne-BigQuery njengendawo egciniwe yangaphakathi (kamuva yengeze usekelo lwe-Postgres , ClickHouse kanye Snowflake).

Isakhiwo semicimbi i-GA kanye neSegment inqume ukushiya ingashintshiwe. Okwakudingeka nje kwakuwukuphinda yonke imicimbi kusuka kusisetshenziswa sewebhu lapho i-pixel ifakwe khona kungemuva lethu. Njengoba kuvela, lokhu kulula ukukwenza. Iphikseli ye-Javascript ikhiphe indlela yelabhulali ye-GA yokuqala ngentsha ephindeke umcimbi kusistimu yethu.

//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
    ga(tracker => {
        var originalSendHitTask = tracker.get('sendHitTask');
        tracker.set('sendHitTask', (model) => {
            var payLoad = model.get('hitPayload');
            //отправка оригинального события в GA
            originalSendHitTask(model);
            let jsonPayload = this.parseQuery(payLoad);
            //отправка события в наш сервис
            this.send3p('ga', jsonPayload);
        });
    });
}

Ngephikseli Yesigaba, yonke into ilula, inezindlela ezimaphakathi, futhi sasebenzisa eyodwa yazo.

//'analytics' - стандартное название переменной Segment
if (window.analytics) {
    if (window.analytics.addSourceMiddleware) {
        window.analytics.addSourceMiddleware(chain => {
            try {
		//дублирование события в наш сервис
                this.send3p('ajs', chain.payload);
            } catch (e) {
                LOG.warn('Failed to send an event', e)
            }
	    //отправка оригинального события в Segment
            chain.next(chain.payload);
        });
    } else {
        LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
    }
} else {
    LOG.warn('Analytics.js listener is not set.');
}

Ngaphezu kokukopisha imicimbi, sengeze ikhono lokuthumela i-json ngokungafanele:

//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
    product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
    product_price: 399.99,
    price_currency: 'USD'
    product_release_start: '2020-09-25T12:38:27.763000Z'
});

Okulandelayo, ake sikhulume ngohlangothi lweseva. I-backend kufanele yamukele izicelo ze-http, igcwalise ngolwazi olwengeziwe, isibonelo, i-geodata (siyabonga ungqondongqondo ngayo) bese ubhalela kusizindalwazi. Besifuna ukwenza isevisi ibe lula ngangokunokwenzeka ukuze isetshenziswe ngokucushwa okuncane. Sisebenzise umsebenzi wokunquma i-schema sedatha ngokusekelwe esakhiweni somcimbi ongenayo we-json. Izinhlobo zedatha zichazwa ngamavelu. Izinto ezifakwe isidleke ziyabola futhi zehliswe zibe yisakhiwo esiyisicaba:

//входящий json
{
  "field_1":  {
    "sub_field_1": "text1",
    "sub_field_2": 100
  },
  "field_2": "text2",
  "field_3": {
    "sub_field_1": {
      "sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
    }
  }
}

//результат
{
  "field_1_sub_field_1":  "text1",
  "field_1_sub_field_2":  100,
  "field_2": "text2",
  "field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}

Nokho, amalungu afanayo okwamanje aguqulelwa nje kuyunithi yezinhlamvu. akuzona zonke izizindalwazi ezihlobene ezisekela izinkambu eziphindaphindiwe. Kuyenzeka futhi ukuthi uguqule amagama ezinkambu noma uwasuse usebenzisa imithetho oyikhethayo yokwenza imephu. Zikuvumela ukuthi uguqule i-schema yedatha, uma kunesidingo, noma uphonsa uhlobo lwedatha olulodwa kolunye. Isibonelo, uma inkambu ye-json iqukethe iyunithi yezinhlamvu enesitembu sesikhathi (inkambu_3_inkambu_engaphansi_1_inkambu_engaphansi_1 kusukela kusibonelo esingenhla), ukuze udale inkambu kusizindalwazi ngohlobo lwesitembu sesikhathi, udinga ukubhala umthetho wokwenza imephu ekucushweni. Ngamanye amazwi, uhlobo lwedatha yenkambu lunqunywa kuqala inani le-json, bese kusetshenziswa umthetho wokusakaza wohlobo (uma ulungisiwe). Sihlonze izinhlobo zedatha eziyinhloko ezingu-4: STRING, FLOAT64, INT64 kanye ne-TIMESTAMP. Imithetho yemephu nokulingisa ibukeka kanje:

rules:
  - "/field_1/subfield_1 -> " #правило удаления поля
  - "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
  - "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа

I-algorithm yokunquma uhlobo lwedatha:

• guqula i-json struct ibe yi-flat struct
• ukucacisa uhlobo lwedatha yezinkambu ngamavelu
• ukusebenzisa imithetho yemephu nokuthayipha imithetho yokulingisa

Bese kusuka kusakhiwo se-json engenayo:

{
    "product_id":  "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
    "product_price": 399.99,
    "price_currency": "USD",
    "product_type": "supplies",
    "product_release_start": "2020-09-25T12:38:27.763000Z",
    "images": {
      "main": "picture1",
      "sub":  "picture2"
    }
}

I-schema yedatha izotholakala:

"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying

Siphinde sacabanga ukuthi umsebenzisi kufanele akwazi ukusetha ukwahlukanisa noma ukuhlukanisa idatha kusizindalwazi ngokwezinye izinqubo futhi asebenzise ikhono lokusetha igama lethebula njengenjwayelo noma isisho ekucushweni. Esibonelweni esingezansi, umcimbi uzolondolozwa kuthebula elinegama elibalwe ngokusekelwe kumanani ohlobo lomkhiqizo kanye nezinkambu _zesitembu sesikhathi (isibonelo izimpahla_2020_10):

tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'

Nokho, ukwakheka kwemicimbi engenayo kungashintsha ngesikhathi sokusebenza. Senze i-algorithm yokuhlola umehluko phakathi kwesakhiwo sethebula elikhona kanye nesakhiwo somcimbi ongenayo. Uma umehluko utholakala, ithebula lizobuyekezwa ngezinkambu ezintsha. Ukuze wenze lokhu, sebenzisa umbuzo we-patch SQL:

#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varying

bokwakha

Kungani udinga ukubhala izehlakalo ohlelweni lwefayela, futhi ungavele uzibhale ngqo kusizindalwazi? Imininingo egciniwe ayihlali ibonisa ukusebenza okuphezulu ngenani elikhulu lokufakwayo (izincomo ze-postgres). Ukwenza lokhu, i-Logger ibhala imicimbi engenayo efayeleni futhi kakade ku-goroutine ehlukile (uchungechunge) Isifundi sefayela sifunda ifayela, bese ukuguqulwa kanye nencazelo ye-schema yedatha kwenzeka. Ngemva kokuba imenenja yeThebula iqinisekise ukuthi i-schema yethebula ihambisana nesikhathi, idatha izobhalwa kusizindalwazi ngeqeqebana elilodwa. Kamuva, sengeze ikhono lokubhala idatha ngokuqondile kusizindalwazi, kodwa sisebenzisa le modi ezenzakalweni ezingeziningi - isibonelo, ukuguqulwa.

Umthombo ovulekile nezinhlelo zesikhathi esizayo

Ngesinye isikhathi, isevisi yaba njengomkhiqizo ogcwele futhi sanquma ukuyibeka ku-Open Source. Okwamanje, ukuhlanganiswa ne-Postgres, ClickHouse, BigQuery, Redshift, S3, Snowflake sekwenziwe. Konke ukuhlanganiswa kusekela kokubili izindlela zokulayisha iqoqo kanye nokusakaza idatha. Ukwesekwa okwengeziwe kwezicelo nge-API.

Uhlelo lwamanje lokuhlanganisa lubukeka kanje:

Nakuba isevisi ingasetshenziswa ngokuzimela (ngokwesibonelo, usebenzisa i-Docker), sinayo futhi inguqulo ephethwe, lapho ungasetha khona ukuhlanganiswa nenqolobane yedatha, wengeze i-CNAME esizindeni sakho, futhi ubuke izibalo zenani lemicimbi. Izinhlelo zethu ezisheshayo ziwukwengeza ikhono lokuhlanganisa hhayi nje izibalo ezivela esisetshenziswa sewebhu, kodwa nedatha evela emithonjeni yedatha yangaphandle futhi uyilondoloze kunoma isiphi isitoreji osithandayo!

→ GitHub
→ Imibhalo
→ ukuxega

Sizojabula uma i-EventNative izokusiza ukuxazulula izinkinga zakho!

Abasebenzisi ababhalisiwe kuphela abangabamba iqhaza kuhlolovo. Ngena ngemvume, wamukelekile.

Iluphi uhlelo lokuqoqwa kwezibalo olusetshenziswa enkampanini yakho

• 48,0%IGoogle Analytics12

• 4,0%Ingxenye1

• 16,0%Okunye (bhala emazwaneni) 4

• 32,0%Usebenzise isevisi yakho8

Bangu-25 abasebenzisi abavotile. Abasebenzisi abangu-6 bagobile.

Source: www.habr.com