Imbali yomthombo wethu ovulekileyo: indlela esenze ngayo inkonzo yohlalutyo kwi-Go kwaye sayenza yafumaneka esidlangalaleni

Okwangoku, phantse yonke inkampani emhlabeni iqokelela izibalo malunga nezenzo zabasebenzisi kwisixhobo sewebhu. Inkuthazo icacile - iinkampani zifuna ukwazi ukuba imveliso / iwebhusayithi yazo isetyenziswa njani kwaye ibaqonde ngcono abasebenzisi bazo. Ngokuqinisekileyo, kukho inani elikhulu lezixhobo kwiimarike ukusombulula le ngxaki - ukusuka kwiinkqubo zokuhlalutya ezibonelela ngedatha ngendlela yeedeshibhodi kunye neegrafu (umzekelo, google Analytics) kwiPlatform yeDatha yoMthengi, ekuvumela ukuba uqokelele kwaye udibanise idatha kwimithombo eyahlukeneyo kuyo nayiphi na indawo yokugcina (umzekelo, Icandelo).

Kodwa sifumene ingxaki engekasonjululwa. Ngoko wazalwa IsiganekoNative -inkonzo yohlahlelo oluvulelekileyo. Malunga nesizathu sokuba siye ukuphuhlisa inkonzo yethu, yintoni eyasinika yona kunye nento eyenzekayo ekugqibeleni (kunye neengcezu zekhowudi), funda phantsi kokusikwa.

Kutheni le nto kufuneka siphuhlise inkonzo yethu?

Kwakuyiminyaka yamashumi asithoba, sasinda kangangoko sinako. Ngo-2019, siphuhlise iPlatifomu yoMthengi wokuQala we-API kSense, eyavumela ukudibanisa idatha kwimithombo eyahlukeneyo (iintengiso ze-Facebook, i-Stripe, i-Salesforce, i-Google play, i-Google Analytics, njl.) ukwenzela uhlalutyo lwedatha olulula ngakumbi, ukuchonga ukuxhomekeka, njl. Siye saqaphela ukuba abasebenzisi abaninzi basebenzisa iqonga lethu lokuhlalutya idatha, ngokukodwa i-Google Analytics (emva koku kuthiwa yi-GA). Sithethe nabanye abasebenzisi kwaye safumanisa ukuba bafuna idatha yabo yohlalutyo lwemveliso, abayifumana besebenzisa i-GA, kodwa Iisampulu zedatha kaGoogle kwaye kuninzi lwe-GA User Interface ayingomgangatho woncedo. Saba neengxoxo ezaneleyo nabasebenzisi bethu kwaye saqonda ukuba abaninzi basebenzise iqonga leCandelo (ekuthi, ngendlela, yayiyientsuku ezimbalwa ezidlulileyo. ithengiswe ngeebhiliyoni zeerandi ezi-3.2).

Bafakele iSegment javascript pixel kwisixhobo sabo sewebhu kwaye idatha yabo yokuziphatha komsebenzisi yalayishwa kwisiseko sedatha esixeliweyo (umz. Postgres). Kodwa iSegment nayo ineminus yayo - ixabiso. Umzekelo, ukuba umthombo wewebhu une-90,000 MTU (abasebenzisi abalandelwayo ngenyanga), ngoko kufuneka uhlawule ~ $ 1,000 ngenyanga kwi-cashier. Kwakhona kwakukho ingxaki yesithathu - ezinye izandiso zesiphequluli (ezifana ne-AdBlock) zithintele ingqokelela ye-analytics. Izicelo ze-http ezivela kwisikhangeli zithunyelwe kwi-GA kunye neSegment domains. Ngokusekelwe kumnqweno wabathengi bethu, senze inkonzo yohlalutyo eqokelela isethi epheleleyo yedatha (ngaphandle kwesampuli), ngaphandle kwentlawulo kwaye inokusebenza kwiziseko zethu.

Isebenza njani inkonzo

Inkonzo inamacandelo amathathu: i-pixel ye-javascript (esithi kamva siyibhale ngokutsha kwi-typescript), inxalenye yomncedisi ephunyezwe kulwimi lwe-GO, kwaye kwacwangciswa ukusebenzisa i-Redshift kunye ne-BigQuery njengendawo yogcino lwangaphakathi (kamva bongeze inkxaso ye-Postgres). , ClickHouse kunye Snowflake).

Ubume bemicimbi ye-GA kunye neCandelo bagqibe ekubeni bahambe bengatshintshwanga. Ekuphela kwento eyayifuneka yayikukuphinda zonke iziganeko kwisixhobo sewebhu apho ipixel ifakwe kwi-backend yethu. Njengoko kuvela, oku kulula ukwenza. I-pixel yeJavascript ikhuphele ngaphezulu indlela yethala leencwadi ye-GA yantlandlolo ngentsha ephinda isiganeko kwinkqubo yethu.

//'ga' - стандартное название переменной Google Analytics
if (window.ga) {
    ga(tracker => {
        var originalSendHitTask = tracker.get('sendHitTask');
        tracker.set('sendHitTask', (model) => {
            var payLoad = model.get('hitPayload');
            //отправка оригинального события в GA
            originalSendHitTask(model);
            let jsonPayload = this.parseQuery(payLoad);
            //отправка события в наш сервис
            this.send3p('ga', jsonPayload);
        });
    });
}

NgePixel yeCandelo, yonke into ilula, ineendlela eziphakathi, kwaye sisebenzise enye yazo.

//'analytics' - стандартное название переменной Segment
if (window.analytics) {
    if (window.analytics.addSourceMiddleware) {
        window.analytics.addSourceMiddleware(chain => {
            try {
		//дублирование события в наш сервис
                this.send3p('ajs', chain.payload);
            } catch (e) {
                LOG.warn('Failed to send an event', e)
            }
	    //отправка оригинального события в Segment
            chain.next(chain.payload);
        });
    } else {
        LOG.warn("Invalid interceptor state. Analytics js initialized, but not completely");
    }
} else {
    LOG.warn('Analytics.js listener is not set.');
}

Ukongeza kwiminyhadala yokukopa, songeze ukukwazi ukuthumela i-json ngokungenasizathu:

//Отправка событий с произвольным json объектом
eventN.track('product_page_view', {
    product_id: '1e48fb70-ef12-4ea9-ab10-fd0b910c49ce',
    product_price: 399.99,
    price_currency: 'USD'
    product_release_start: '2020-09-25T12:38:27.763000Z'
});

Okulandelayo, masithethe ngecala lomncedisi. I-backend kufuneka yamkele izicelo ze-http, zigcwalise ngolwazi olongezelelweyo, umzekelo, i-geodata (enkosi ingqondo enkulu ngayo) kwaye ubhale kuvimba weenkcukacha. Besifuna ukwenza inkonzo ibe lula kangangoko ukuze ikwazi ukusetyenziswa ngoqwalaselo oluncinci. Siphumeze umsebenzi wokumisela i-schema yedatha esekelwe kwisakhiwo sesiganeko esingenayo i-json. Iintlobo zedatha zichazwa ngamaxabiso. Izinto ezifakwe kwindlwane ziyabola kwaye zincitshiswe kwisakhiwo esisicaba:

//входящий json
{
  "field_1":  {
    "sub_field_1": "text1",
    "sub_field_2": 100
  },
  "field_2": "text2",
  "field_3": {
    "sub_field_1": {
      "sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
    }
  }
}

//результат
{
  "field_1_sub_field_1":  "text1",
  "field_1_sub_field_2":  100,
  "field_2": "text2",
  "field_3_sub_field_1_sub_sub_field_1": "2020-09-25T12:38:27.763000Z"
}

Nangona kunjalo, uluhlu lwangoku luguqulelwa nje lube yimitya. ayingabo bonke oovimba bonxibelelwano abaxhasa imimandla ephindaphindiweyo. Kuyenzeka ukuba utshintshe amagama entsimi okanye uwasuse usebenzisa imithetho ekhethwayo yokwenza imephu. Bakuvumela ukuba utshintshe i-schema yedatha, ukuba kukho imfuneko, okanye ulahle uhlobo lwedatha enye kwenye. Umzekelo, ukuba umhlaba we json uqulathe umtya onesitampu sexesha (ibala_3_ibala_elingaphantsi_1_ibala_elingaphantsi_1 ukusuka kumzekelo ongentla), ngoko ukuze udale umhlaba kwisiseko sedatha kunye nodidi lwesitampu sexesha, kufuneka ubhale umthetho wokwenza imephu kuqwalaselo. Ngamanye amazwi, udidi lwedatha yentsimi lumiselwa kuqala ngexabiso le-json, kwaye ngoko umgaqo wokuphosa wodidi (ukuba uqwalaselwe) uyasetyenziswa. Sichonge iintlobo ezi-4 zedatha eziphambili: STRING, FLOAT64, INT64 kunye neTIMESTAMP. Imithetho yemephu kunye nokuphosa ijongeka ngolu hlobo:

rules:
  - "/field_1/subfield_1 -> " #правило удаления поля
  - "/field_2/subfield_1 -> /field_10/subfield_1" #правило переноса поля
  - "/field_3/subfield_1/subsubfield_1 -> (timestamp) /field_20" #правило переноса поля и приведения типа

I-algorithm yokumisela uhlobo lwedatha:

• guqula i-json struct kwisakhiwo esisicaba
• ukugqiba uhlobo lwedatha yemihlaba ngamaxabiso
• ukusebenzisa imephu kunye nokuchwetheza imithetho yokuphosa

Emva koko ukusuka kwisakhiwo se-json engenayo:

{
    "product_id":  "1e48fb70-ef12-4ea9-ab10-fd0b910c49ce",
    "product_price": 399.99,
    "price_currency": "USD",
    "product_type": "supplies",
    "product_release_start": "2020-09-25T12:38:27.763000Z",
    "images": {
      "main": "picture1",
      "sub":  "picture2"
    }
}

Isicwangciso sedatha siya kufunyanwa:

"product_id" character varying,
"product_price" numeric (38,18),
"price_currency" character varying,
"product_type" character varying,
"product_release_start" timestamp,
"images_main" character varying,
"images_sub" character varying

Siphinde sacinga ukuba umsebenzisi kufuneka akwazi ukuseta ulwahlulo okanye ukwahlula-hlula idatha kwisiseko sedatha ngokwezinye iindlela kwaye aphumeze ukukwazi ukuseta igama letafile njengesiqhelo okanye intetho kuqwalaselo. Kulo mzekelo ungezantsi, isiganeko siya kugcinwa kwitafile enegama elibalwe ngokusekelwe kumaxabiso ohlobo lwemveliso kunye _nesitampu sexesha (umzekelo iimpahla_2020_10):

tableName: '{{.product_type}}_{{._timestamp.Format "2006_01"}}'

Nangona kunjalo, ulwakhiwo lweziganeko ezingenayo lunokutshintsha ngexesha lokusebenza. Sisebenzise i-algorithm yokukhangela umahluko phakathi kwesakhiwo setafile ekhoyo kunye nesakhiwo sesiganeko esingenayo. Ukuba umahluko ufunyenwe, itheyibhile iya kuhlaziywa ngemimandla emitsha. Ukwenza oku, sebenzisa umbuzo we-patch SQL:

#Пример для Postgres
ALTER TABLE "schema"."table" ADD COLUMN new_column character varying

izakhiwo

Kutheni kufuneka ubhale iziganeko kwisixokelelwano sefayile, kwaye ungazibhali nje ngokuthe ngqo kwisiseko sedatha? Ugcino lwedatha alusoloko lubonisa ukusebenza okuphezulu ngenani elikhulu lofakelo (postgres iingcebiso). Ukwenza oku, i-Logger ibhala iziganeko ezingenayo kwifayile kwaye sele ikwi-goroutine eyahlukileyo (umsonto) Umfundi wefayile ufunda ifayile, ngoko ukuguqulwa kunye nenkcazo ye-schema yedatha kwenzeka. Emva kokuba umphathi weThebhile eqinisekisa ukuba i-schema yetafile ihlaziyiwe, idatha iya kubhalwa kwisiseko sedatha kwibhetshi enye. Emva koko, songeze amandla okubhala idatha ngokuthe ngqo kwisiseko sedatha, kodwa sisebenzisa le mowudi yeziganeko ezingekho ninzi - umzekelo, ukuguqulwa.

Umthombo ovulekileyo kunye nezicwangciso zexesha elizayo

Ngexesha elithile, inkonzo yaba njengemveliso epheleleyo kwaye sagqiba ekubeni siyibeke kwi-Open Source. Okwangoku, ukudibanisa kunye ne-Postgres, i-ClickHouse, i-BigQuery, i-Redshift, i-S3, i-Snowflake iphunyeziwe. Zonke udibaniso zixhasa zombini ibhetshi kunye neendlela zokulayisha idatha. Inkxaso eyongeziweyo yezicelo nge-API.

Iskimu sohlanganiso sangoku sijongeka ngolu hlobo:

Nangona inkonzo inokusetyenziswa ngokuzimeleyo (umzekelo, usebenzisa i-Docker), nathi sinakho inguqulelo ebanjwe, apho unokuseta udibaniso kunye nendawo yokugcina idatha, yongeza i-CNAME kwindawo yakho, kwaye ujonge izibalo kwinani leziganeko. Izicwangciso zethu zangoku kukongeza ukukwazi ukudibanisa kungekuphela nje izibalo ezivela kwisixhobo sewebhu, kodwa kunye nedatha evela kwimithombo yedatha yangaphandle kwaye uyigcine kuyo nayiphi na indawo yokugcina oyikhethileyo!

→ GitHub
→ Amaxwebhu
→ Musa

Siya kuvuya ukuba i-EventNative iya kukunceda ukusombulula iingxaki zakho!

Ngabasebenzisi ababhalisiweyo kuphela abanokuthatha inxaxheba kuphando. Ngena, ndiyacela.

Yeyiphi inkqubo yokuqokelela izibalo esetyenziswa kwinkampani yakho

• 48,0%Uhlalutyo lukaGoogle12

• 4,0%Icandelo1

• 16,0%Okunye (bhala kwizimvo) 4

• 32,0%Iphumeze inkonzo yakho8

Bali-25 abasebenzisi abavotileyo. Abasebenzisi abasi-6 abakhange.

umthombo: www.habr.com