O le tala faasolopito o la matou punaoa tatala: le auala na matou faia ai se au'aunaga au'ili'ili i Go ma fa'aavanoaina fa'alaua'itele

I le taimi nei, toetoe lava o kamupani uma i le lalolagi e aoina faʻamaumauga e uiga i gaioiga a tagata faʻaoga i luga o se upega tafaʻilagi. E manino le faʻaosofia - o kamupani e fia iloa pe faʻafefea ona faʻaogaina a latou oloa / upega tafaʻilagi ma sili atu ona malamalama ia latou tagata faʻaoga. O le mea moni, o loʻo i ai se numera tele o meafaigaluega i luga o le maketi e foia ai lenei faʻafitauli - mai faʻamaumauga faʻapitoa e maua ai faʻamatalaga i foliga o dashboards ma kalafi (mo se faʻataʻitaʻiga Analytics Google) i le Customer Data Platform, lea e mafai ai ona e aoina ma faʻapipiʻi faʻamatalaga mai punaoa eseese i soʻo se fale teu oloa (mo se faʻataʻitaʻiga Vaega).

Ae na matou maua se faʻafitauli e leʻi foia. Sa faapea ona fanau mai EventNative - fa'amatalaga fa'amatalaga fa'amatalaga matala. Faitau e uiga i le mafuaʻaga na matou filifili ai e atiina ae a matou lava auaunaga, o le a le mea na tuʻuina mai ia i matou, ma pe o le a le iʻuga (faʻatasi ai ma fasi code).

Aisea e tatau ai ona tatou atiina ae a tatou lava auaunaga?

O le nineties, sa matou ola i le mea sili sa matou mafaia. 2019, na matou atiaeina le API First Customer Data Platform kSense, lea na mafai ai ona tuʻufaʻatasia faʻamatalaga mai punaoa eseese (Facebook ads, Stripe, Salesforce, Google play, Google Analytics, ma isi) mo le sili atu ona faigofie suʻesuʻega faʻamatalaga, faʻamaonia faʻalagolago, ma isi. Ua matou matauina le tele o tagata fa'aoga latou te fa'aogaina la matou fa'avae mo su'esu'ega fa'amaumauga fa'apitoa Google Analytics (fa'asolo mai GA). Na matou talanoa i nisi o tagata faʻaoga ma iloa ai latou te manaʻomia faʻamatalaga auiliiliga mo a latou oloa latou te mauaina e faʻaaoga ai GA, ae Google fa'ata'ita'iga fa'amaumauga ma mo le toʻatele, o le GA User interface e le o se tulaga faʻaoga faigofie. Na lava a matou talanoaga ma a matou tagata faʻaoga ma iloa ai o le toʻatele o loʻo faʻaaogaina le Segment platform (lea, i le ala, na o le isi aso. faatau atu mo le $3.2 piliona).

Na latou faʻapipiʻiina se Pika javascript Segment i luga o la latou punaoa i luga o le upega tafaʻilagi ma faʻamatalaga e uiga i amioga a latou tagata faʻaoga na utaina i totonu o faʻamaumauga faʻamaonia (mo se faʻataʻitaʻiga Postgres). Ae o lo'o iai fo'i le itu i lalo ole Vaega - ole tau. Mo se faʻataʻitaʻiga, afai o se upega tafaʻilagi e 90,000 MTU (tagata faʻaoga masina), ona e manaʻomia lea ona e totogia le ~1,000 $ i le masina i le cashier. Sa i ai foi se faʻafitauli lona tolu - o nisi faʻaopoopoga o suʻesuʻega (e pei o le AdBlock) na poloka le aoina o auʻiliʻili ona... O talosaga http mai le su'esu'ega na lafo i le GA ma le Vaega. Faʻavae i luga o manaʻoga o matou tagata faʻatau, ua matou fatuina se auʻaunaga auʻiliʻili e aoina ai se seti atoa o faʻamaumauga (e aunoa ma se faʻataʻitaʻiga), e leai se totogi ma e mafai ona galue i luga oa tatou lava atinaʻe.

Auala e galue ai le auaunaga

O le tautua e aofia ai vaega e tolu: o le javascript pixel (lea na matou toe tusia mulimuli ane i le typescript), o le vaega o le server o loʻo faʻatinoina i le gagana GO, ma sa fuafua e faʻaaoga Redshift ma BigQuery o se faʻamaumauga i totonu o le fale (mulimuli ane latou faʻaopoopoina le lagolago mo Postgres, ClickHouse ma Snowflake).

Na tonu e tu'u le fa'atulagaina o le GA ma le Vaega o mea na tutupu e le suia. Pau lava le mea na manaʻomia o le faʻaluaina o mea uma na tutupu mai le upega tafaʻilagi o loʻo faʻapipiʻi ai le pika i le pito i tua. E pei ona aliali mai, e le faigata ona fai. O le Javascript pixel na suitulaga i le uluai GA faletusi metotia ma se mea fou, lea na faʻaluaina le mea na tupu i totonu o la matou polokalama.

//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
    ga(tracker => {
        var originalSendHitTask = tracker.get('sendHitTask');
        tracker.set('sendHitTask', (model) => {
            var payLoad = model.get('hitPayload');
            //отправка оригинального события в GA
            originalSendHitTask(model);
            let jsonPayload = this.parseQuery(payLoad);
            //отправка события в наш сервис
            this.send3p('ga', jsonPayload);
        });
    });
}

Faatasi ai ma le Vaega pixel e faigofie mea uma; o loʻo i ai auala middleware, o se tasi na matou faʻaaogaina.

//'analytics' - стандартное название переменной Segment
if (window.analytics) {
    if (window.analytics.addSourceMiddleware) {
        window.analytics.addSourceMiddleware(chain => {
            try {
		//дублирование события в наш сервис
                this.send3p('ajs', chain.payload);
            } catch (e) {
                LOG.warn('Failed to send an event', e)
            }
	    //отправка оригинального события в Segment
            chain.next(chain.payload);
        });
    } else {
        LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
    }
} else {
    LOG.warn('Analytics.js listener is not set.');
}

I le faʻaopoopoga i le kopiina o mea na tutupu, ua matou faʻaopoopoina le gafatia e lafo ai le json faʻamaonia:

//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
    product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
    product_price: 399.99,
    price_currency: 'USD'
    product_release_start: '2020-09-25T12:38:27.763000Z'
});

Sosoo ai, tatou talanoa e uiga i le vaega server. E tatau i le pito i tua ona talia talosaga http, faʻatumu i faʻamatalaga faaopoopo, mo se faʻataʻitaʻiga, geo data (faʻafetai maxmind mo lenei) ma faamaumau i totonu o le database. Matou te mananaʻo e faʻafaigofie le auaunaga ina ia mafai ona faʻaaogaina i se faʻatulagaga itiiti. Ua matou faʻatinoina le faʻatinoga o le fuafuaina o le faʻasologa o faʻamaumauga e faʻavae i luga o le fausaga o le mea na tupu i le json. O ituaiga faʻamatalaga e faʻamatalaina i tau. O mea fa'aputu ua pala ma fa'aitiitia i se fausaga mafolafola:

//входящий json
{
  "field_1":  {
    "sub_field_1": "text1",
    "sub_field_2": 100
  },
  "field_2": "text2",
  "field_3": {
    "sub_field_1": {
      "sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
    }
  }
}

//результат
{
  "field_1_sub_field_1":  "text1",
  "field_1_sub_field_2":  100,
  "field_2": "text2",
  "field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}

Ae ui i lea, o arrays o loʻo faʻaliliuina nei i manoa ona E le'o fa'amaumauga uma feso'ota'iga e lagolagoina matā'upu fai soo. E mafai fo'i ona sui igoa ole fanua pe tape ile fa'aogaina o tulafono fa'afanua. Latou te fa'atagaina oe e sui le fa'asologa o fa'amaumauga pe a mana'omia pe fa'aliliu le tasi ituaiga fa'amatalaga i le isi. Mo se faʻataʻitaʻiga, afai o se fanua json o loʻo i ai se manoa ma faʻailoga taimi (field_3_sub_field_1_sub_sub_field_1 mai le faʻataʻitaʻiga o loʻo i luga), ona mafai ai lea ona fatuina se fanua i totonu o faʻamaumauga faʻatasi ma le ituaiga timestamp, e tatau ona e tusia se tulafono faʻafanua i le faʻatulagaga. I se isi faaupuga, o le ituaiga faʻamatalaga o le fanua e faʻamoemoe muamua i le tau o le json, ona faʻaaogaina ai lea o le tulafono faʻatonu (pe a faʻatulagaina). E 4 ituaiga fa'amaumauga autu ua matou iloa: STRING, FLOAT64, INT64 ma TIMESTAMP. O le fa'afanua ma le tu'aiga tu'aiga tulafono e fa'apea:

rules:
  - "/field_1/subfield_1 -> " #правило удаления поля
  - "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
  - "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа

Algorithm mo le fuafuaina o le ituaiga faʻamatalaga:

• liliu le fausaga o le json i le fausaga mafolafola
• fuafuaina le ituaiga fa'amaumauga o fanua e ala i tau
• fa'aogaina tulafono fa'afanua ma tu'aiga tu'aiga

Ona mai le fausaga json ulufale mai:

{
    "product_id":  "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
    "product_price": 399.99,
    "price_currency": "USD",
    "product_type": "supplies",
    "product_release_start": "2020-09-25T12:38:27.763000Z",
    "images": {
      "main": "picture1",
      "sub":  "picture2"
    }
}

o le a maua le fuafuaga fa'amaumauga:

"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying

Matou te manatu foi e tatau i le tagata faʻaoga ona mafai ona faʻapipiʻi le vaeluaga poʻo le vaevaeina o faʻamaumauga i totonu o faʻamaumauga e tusa ai ma isi taʻiala ma faʻatino le mafai ona seti le igoa laulau i se taimi tumau pe fa'aaliga i le faatulagaga. I le faʻataʻitaʻiga o loʻo i lalo, o le mea na tupu o le a faʻasaoina i luga o se laulau ma se igoa ua fuafuaina e faʻatatau i tau o oloa_type ma _timestamp fanua (mo se faʻataʻitaʻiga sapalai_2020_10):

tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'

Ae ui i lea, o le fausaga o mea o loʻo oʻo mai e mafai ona suia i le taimi o le taʻavale. Ua matou faʻatinoina se algorithm e siaki ai le eseesega i le va o le fausaga o se laulau o loʻo i ai ma le fausaga o se mea e oʻo mai. Afai e maua se eseesega, o le a faʻafouina le laulau i fanua fou. Ina ia faia lenei mea, faʻaaoga le patch SQL query:

#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varying

atiga

Aisea e te manaʻomia ai le tusia o mea na tutupu i le faila faila, ae le naʻo le tusia saʻo i le database? E le lelei i taimi uma fa'amaumauga fa'amaumauga pe a feagai ma se numera tele o fa'aofi (Fautuaga Postgres). Ina ia faia lenei mea, e tusia e Logger mea e oʻo mai i se faila ma i se isi goroutine (filo) E faitau e le faila faila le faila, ona liua lea o faʻamatalaga ma fuafua. A maeʻa ona faʻamautinoa e le Pule o le laulau o loʻo faʻafouina le faʻasologa o le laulau, o le a tusia faʻamatalaga i le database i le tasi vaega. Mulimuli ane, matou te faʻaopoopoina le tomai e tusi saʻo ai faʻamatalaga i le database, ae matou te faʻaogaina lenei faiga mo mea e le tele - mo se faʻataʻitaʻiga, liua.

Open Source ma fuafuaga mo le lumanaʻi

I se taimi, o le auaunaga na amata ona foliga mai o se oloa atoatoa ma na matou filifili e faʻasalalau i Open Source. I le taimi nei, faʻatasi ma Postgres, ClickHouse, BigQuery, Redshift, S3, Snowflake ua faʻatinoina. O tu'ufa'atasiga uma e lagolagoina uma le fa'aputuga ma le fa'afefeina o auala o le utaina o fa'amaumauga. Fa'aopoopo le lagolago mo talosaga e ala ile API.

O le polokalame tu'ufa'atasiga o lo'o iai nei e pei o lenei:

E ui o le auaunaga e mafai ona faʻaaogaina tutoʻatasi (mo se faʻataʻitaʻiga faʻaaoga Docker), matou te maua foi talimalo lomiga, lea e mafai ai ona e setiina le tuʻufaʻatasia ma se faleteuoloa faʻamaumauga, faʻaopoopo se CNAME i lau vaega ma vaʻai fuainumera i le numera o mea na tutupu. O a matou fuafuaga vave o le faʻaopoopoina lea o le gafatia e faʻapipiʻi e le gata o fuainumera mai se upega tafaʻilagi, ae faʻapea foʻi faʻamaumauga mai faʻamatalaga mai fafo ma faʻasaoina i soʻo se teuina o lau filifiliga!

→ GitHub
→ Tusi
→ faatuai

Matou te fiafia pe a fesoasoani EventNative e foia ou faafitauli!

Na'o tagata fa'aigoaina e mafai ona auai i le su'esu'ega. Saini ese j, faʻamolemole.

O le a le faiga o le aoina o fuainumera o lo'o fa'aogaina i lau kamupani?

• 48,0%Google Analytics12

• 4,0%Vaega1

• 16,0%O le isi (tusi i fa'amatalaga)4

• 32,0%Fa'atino lau tautua8

25 tagata fa'aoga na palota. 6 tagata fa'aoga na le fa'aaogaina.

puna: www.habr.com