Mbiri ya malo athu otseguka: m'mene tinapangira ntchito yowunikira mu Go ndikupangitsa kuti ipezeke poyera

Pakadali pano, pafupifupi kampani iliyonse padziko lapansi imasonkhanitsa ziwerengero za ogwiritsa ntchito pa intaneti. Zolimbikitsa zikuwonekeratu - makampani amafuna kudziwa momwe malonda / tsamba lawo limagwiritsidwira ntchito komanso kumvetsetsa bwino ogwiritsa ntchito. Inde, pali zida zambiri pamsika kuti zithetse vutoli - kuchokera ku machitidwe a analytics omwe amapereka deta mu mawonekedwe a dashboards ndi ma graph (mwachitsanzo. Analytics Google) kupita ku Customer Data Platform, yomwe imakupatsani mwayi wosonkhanitsa ndikusonkhanitsa deta kuchokera kuzinthu zosiyanasiyana m'nyumba yosungiramo zinthu iliyonse (mwachitsanzo Chigawo).

Koma tapeza vuto lomwe silinathebe. Chomwecho chinabadwa EventNative - ntchito yowunikira magwero otseguka. Werengani za chifukwa chake tidaganiza zopanga ntchito yathu, zomwe zidatipatsa, komanso zotsatira zake (ndi zidutswa za code).

N’chifukwa chiyani tiyenera kukulitsa utumiki wathu?

Zinali zaka za makumi asanu ndi anayi, tinapulumuka momwe tingathere. 2019, tinapanga API First Customer Data Platform kSense, zomwe zinapangitsa kuti zitheke kusonkhanitsa deta kuchokera kumagwero osiyanasiyana (zotsatsa za Facebook, Stripe, Salesforce, Google play, Google Analytics, ndi zina zotero) kuti muthe kusanthula deta mosavuta, kuzindikira kudalira, ndi zina zotero. Tawona kuti ogwiritsa ntchito ambiri amagwiritsa ntchito nsanja yathu pakusanthula deta makamaka Google Analytics (yotsatira pano GA). Tidalankhula ndi ogwiritsa ntchito ena ndipo tidapeza kuti amafunikira data ya analytics yazinthu zawo zomwe amalandila pogwiritsa ntchito GA, koma Zitsanzo za data za Google ndipo kwa ambiri, mawonekedwe a GA Ogwiritsa ntchito siwoyenera. Tinali ndi zokambirana zokwanira ndi ogwiritsa ntchito athu ndipo tinazindikira kuti ambiri akugwiritsanso ntchito gawo la Gawo (lomwe, mwa njira, linali tsiku lina chabe. idagulitsidwa $3.2 biliyoni).

Adayika ma pixel a Segment javascript pa tsamba lawo lawebusayiti ndipo zambiri zokhudzana ndi machitidwe a ogwiritsa ntchito zidakwezedwa munkhokwe yomwe yatchulidwa (mwachitsanzo Postgres). Koma Gawo lilinso ndi zovuta zake - mtengo. Mwachitsanzo, ngati tsamba lawebusayiti lili ndi 90,000 MTU (otsatira omwe amatsatiridwa pamwezi), ndiye kuti muyenera kulipira ~ 1,000 $ pamwezi kwa wosunga ndalama. Panalinso vuto lachitatu - zowonjezera msakatuli (monga AdBlock) zaletsa kusonkhanitsa kwa analytics chifukwa ... Zopempha za http kuchokera kwa osatsegula zidatumizidwa kumadera a GA ndi Segment. Kutengera zofuna za makasitomala athu, tapanga ntchito yowunikira yomwe imasonkhanitsa deta yonse (popanda sampuli), ndi yaulere ndipo imatha kugwira ntchito zathu zokha.

Momwe ntchitoyo imagwirira ntchito

Ntchitoyi ili ndi magawo atatu: pixel ya javascript (yomwe tidalembanso m'malembedwe), gawo la seva likugwiritsidwa ntchito muchilankhulo cha GO, ndipo idakonzedwa kuti igwiritse ntchito Redshift ndi BigQuery ngati nkhokwe yamkati (kenako adawonjezera thandizo la Postgres, ClickHouse ndi Snowflake).

Zinasankhidwa kusiya mapangidwe a GA ndi Segment zochitika zosasinthika. Zomwe zimafunikira ndikubwereza zochitika zonse kuchokera pa intaneti pomwe pixel imayikidwa kumbuyo kwathu. Monga momwe zikukhalira, izi sizovuta kuchita. Pixel ya Javascript idatsitsa njira yoyambirira ya library ya GA ndi yatsopano, yomwe idabwereza zomwe zidachitika m'dongosolo lathu.

//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
    ga(tracker => {
        var originalSendHitTask = tracker.get('sendHitTask');
        tracker.set('sendHitTask', (model) => {
            var payLoad = model.get('hitPayload');
            //отправка оригинального события в GA
            originalSendHitTask(model);
            let jsonPayload = this.parseQuery(payLoad);
            //отправка события в наш сервис
            this.send3p('ga', jsonPayload);
        });
    });
}

Ndi pixel ya Gawo chilichonse ndi chosavuta; ili ndi njira zapakati, imodzi yomwe tidagwiritsa ntchito.

//'analytics' - стандартное название переменной Segment
if (window.analytics) {
    if (window.analytics.addSourceMiddleware) {
        window.analytics.addSourceMiddleware(chain => {
            try {
		//дублирование события в наш сервис
                this.send3p('ajs', chain.payload);
            } catch (e) {
                LOG.warn('Failed to send an event', e)
            }
	    //отправка оригинального события в Segment
            chain.next(chain.payload);
        });
    } else {
        LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
    }
} else {
    LOG.warn('Analytics.js listener is not set.');
}

Kuphatikiza pa kukopera zochitika, tawonjezeranso kuthekera kotumiza json mosasamala:

//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
    product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
    product_price: 399.99,
    price_currency: 'USD'
    product_release_start: '2020-09-25T12:38:27.763000Z'
});

Kenako, tiyeni tikambirane za gawo la seva. The backend ayenera kuvomereza zopempha http, kuwadzaza ndi zina zambiri, mwachitsanzo, geo data (zikomo maxmind za izi) ndikuzilemba mu database. Tinkafuna kuti ntchitoyi ikhale yosavuta momwe tingathere kuti igwiritsidwe ntchito ndi kasinthidwe kochepa. Takhazikitsa magwiridwe antchito owunikira schema ya data kutengera kapangidwe ka chochitika cha json chomwe chikubwera. Mitundu ya data imatanthauzidwa ndi zikhalidwe. Zinthu zokhala ndi zisa zimawola ndikusinthidwa kukhala lathyathyathya:

//входящий json
{
  "field_1":  {
    "sub_field_1": "text1",
    "sub_field_2": 100
  },
  "field_2": "text2",
  "field_3": {
    "sub_field_1": {
      "sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
    }
  }
}

//результат
{
  "field_1_sub_field_1":  "text1",
  "field_1_sub_field_2":  100,
  "field_2": "text2",
  "field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}

Komabe, ma arrays pakadali pano amangosinthidwa kukhala zingwe chifukwa Sikuti nkhokwe zonse zaubale zimathandizira magawo obwerezabwereza. Ndikothekanso kusintha mayina am'munda kapena kuwachotsa pogwiritsa ntchito malamulo opangira mapu. Amakulolani kuti musinthe schema ya data ngati kuli kofunikira kapena kusintha mtundu wina wa data kukhala wina. Mwachitsanzo, ngati gawo la json lili ndi chingwe chokhala ndi nthawi (field_3_sub_field_1_sub_sub_field_1 kuchokera ku chitsanzo pamwambapa), ndiye kuti mupange gawo mu nkhokwe ndi mtundu wa stamp yanthawi, muyenera kulemba lamulo la mapu pakukonza. Mwa kuyankhula kwina, mtundu wa deta wamunda umatsimikiziridwa poyamba ndi mtengo wa json, ndiyeno lamulo loponyera mtundu (ngati litakonzedwa) likugwiritsidwa ntchito. Tazindikira mitundu 4 ya data: STRING, FLOAT64, INT64 ndi TIMESTAMP. Malamulo opangira mapu ndi mitundu amawoneka motere:

rules:
  - "/field_1/subfield_1 -> " #правило удаления поля
  - "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
  - "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа

Algorithm yodziwira mtundu wa data:

• sinthani json structure kukhala yosalala
• kudziwa mtundu wa data wa minda ndi ma values
• kugwiritsa ntchito mapu ndi kulemba malamulo opangira

Kenako kuchokera pamapangidwe a json omwe akubwera:

{
    "product_id":  "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
    "product_price": 399.99,
    "price_currency": "USD",
    "product_type": "supplies",
    "product_release_start": "2020-09-25T12:38:27.763000Z",
    "images": {
      "main": "picture1",
      "sub":  "picture2"
    }
}

schema ya data idzapezedwa:

"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying

Tidaganizanso kuti wogwiritsa ntchito azitha kugawa magawo kapena kugawa zidziwitso mu nkhokwe molingana ndi njira zina ndikukhazikitsa luso loyika dzina la tebulo nthawi zonse kapena mawu mu kasinthidwe. Muchitsanzo chomwe chili pansipa, chochitikacho chidzasungidwa patebulo lomwe lili ndi dzina lowerengedwa kutengera mikhalidwe ya product_type ndi _timestamp minda (mwachitsanzo. katundu_2020_10):

tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'

Komabe, mapangidwe a zochitika zomwe zikubwera zimatha kusintha panthawi yothamanga. Takhazikitsa algorithm kuti tiwone kusiyana pakati pa kapangidwe ka tebulo lomwe lilipo ndi kapangidwe ka chochitika chomwe chikubwera. Ngati kusiyana kukupezeka, tebulo lidzasinthidwa ndi magawo atsopano. Kuti muchite izi, gwiritsani ntchito chigamba cha SQL funso:

#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varying

zomangamanga

Chifukwa chiyani muyenera kulemba zochitika ku fayilo yamafayilo, osati kungolemba mwachindunji ku database? Ma database samachita bwino nthawi zonse pochita ndi zoyika zambiri (Malangizo a Postgres). Kuti achite izi, Logger amalemba zochitika zomwe zikubwera ku fayilo ndipo mu goroutine (ulusi) wowerenga fayilo amawerenga fayilo, ndiye kuti deta imasinthidwa ndikutsimikiziridwa. Woyang'anira Table akaonetsetsa kuti schema ya tebulo ili yatsopano, deta idzalembedwa ku database mu batch imodzi. Pambuyo pake, tinawonjezera luso lolemba deta mwachindunji ku database, koma timagwiritsa ntchito njirayi pazinthu zomwe sizili zambiri - mwachitsanzo, kutembenuka.

Open Source ndi mapulani amtsogolo

Panthawi ina, ntchitoyi inayamba kuwoneka ngati chinthu chokwanira ndipo tinaganiza zomasula ku Open Source. Pakadali pano, kuphatikiza ndi Postgres, ClickHouse, BigQuery, Redshift, S3, Snowflake yakhazikitsidwa. Zophatikizira zonse zimathandizira mitundu yonse ya batch ndi kutsitsa ndikutsitsa deta. Thandizo lowonjezera pazofunsira kudzera pa API.

Chiwembu chophatikiza chamakono chikuwoneka motere:

Ngakhale ntchitoyo ingagwiritsidwe ntchito paokha (mwachitsanzo kugwiritsa ntchito Docker), tilinso nayo mtundu wochititsa, momwe mungakhazikitsire kuphatikiza ndi malo osungiramo deta, onjezerani CNAME ku domain yanu ndikuwona ziwerengero za chiwerengero cha zochitika. Zolinga zathu zaposachedwa ndikuwonjezera kuthekera kophatikiza osati ziwerengero zokha kuchokera pa intaneti, komanso deta kuchokera kuzinthu zakunja za data ndikuzisunga kusungidwe kulikonse komwe mungafune!

→ GitHub
→ Zolemba
→ lochedwa

Tidzakhala okondwa ngati EventNative ikuthandizani kuthetsa mavuto anu!

Ogwiritsa ntchito olembetsedwa okha ndi omwe angatenge nawo gawo pa kafukufukuyu. Lowani muakauntichonde.

Ndi ndondomeko yanji yotolera ziwerengero yomwe imagwiritsidwa ntchito pakampani yanu?

• 48,0%Google Analytics12

• 4,0%Gawo 1

• 16,0%Wina (lembani mu ndemanga)4

• 32,0%Yambani ntchito yanu8

Ogwiritsa 25 adavota. Ogwiritsa ntchito 6 adakana.

Source: www.habr.com