Ma analytics a seva

Ili ndi gawo lachiwiri lazolemba za analytical systems (link ku part 1).

Masiku ano palibenso kukayikira kulikonse kuti kukonza deta mosamala ndi kutanthauzira zotsatira kungathandize pafupifupi mtundu uliwonse wa bizinesi. Pachifukwa ichi, machitidwe owunikira akuchulukirachulukira ndi magawo, ndipo kuchuluka kwa zoyambitsa ndi zochitika za ogwiritsa ntchito muzofunsira zikukulirakulira.
Chifukwa cha izi, makampani akupereka owunika awo zambiri zochulukirapo kuti azisanthula ndikusintha kukhala zisankho zomveka. Kufunika kwa kachitidwe ka analytics kwa kampani sikuyenera kunyalanyazidwa, ndipo dongosolo lokha liyenera kukhala lodalirika komanso lokhazikika.

Makasitomala owunika

Kusanthula kwamakasitomala ndi ntchito yomwe kampani imalumikizana ndi tsamba lake kapena kugwiritsa ntchito kwake kudzera mu SDK yovomerezeka, imaphatikizana ndi codebase yake ndikusankha zoyambitsa zochitika. Pali zovuta zodziwikiratu panjira iyi: zonse zomwe zasonkhanitsidwa sizingasinthidwe ndendende momwe mukufunira chifukwa cha kuchepa kwa ntchito iliyonse yomwe mungasankhe. Mwachitsanzo, pa dongosolo limodzi sizingakhale zophweka kuyendetsa ntchito za MapReduce, pa ina simungathe kuyendetsa chitsanzo chanu. Choyipa china chidzakhala bilu yanthawi zonse (yosangalatsa) ya mautumiki.
Pali njira zambiri zowunikira makasitomala pamsika, koma posachedwa akatswiri akukumana ndi mfundo yakuti palibe ntchito yapadziko lonse yoyenera ntchito iliyonse (pamene mitengo ya mautumiki onsewa ikukwera nthawi zonse). Zikatero, makampani nthawi zambiri amasankha kupanga ma analytics awo omwe ali ndi makonda ndi kuthekera koyenera.

Ofufuza za seva

Ma analytics a seva ndi ntchito yomwe imatha kutumizidwa mkati mwa kampani pa ma seva ake komanso (nthawi zambiri) ndi zoyesayesa zake. Muchitsanzo ichi, zochitika zonse za ogwiritsa ntchito zimasungidwa pa ma seva amkati, zomwe zimalola opanga kuyesa ma database osiyanasiyana osungira ndikusankha zomangamanga zosavuta kwambiri. Ndipo ngakhale mutafunabe kugwiritsa ntchito kusanthula kwamakasitomala a chipani chachitatu pazinthu zina, zitha kutero.
Ma analytics a seva atha kutumizidwa m'njira ziwiri. Choyamba: sankhani zida zina zotseguka, zikhazikitseni pamakina anu ndikupanga malingaliro abizinesi.

Плюсы
Минусы

Mutha kusintha chilichonse chomwe mukufuna
Izi nthawi zambiri zimakhala zovuta ndipo zimafuna opanga osiyana

Chachiwiri: tengani mautumiki a SaaS (Amazon, Google, Azure) m'malo modzitumizira nokha. Tidzakambirana za SaaS mwatsatanetsatane mu gawo lachitatu.

Плюсы
Минусы

Zitha kukhala zotsika mtengo pama voliyumu apakatikati, koma ndi kukula kwakukulu zitha kukhala zodula kwambiri
Sizingatheke kulamulira magawo onse

Utsogoleri umasamutsidwa kwathunthu kumapewa a wothandizira
Sizidziwika nthawi zonse zomwe zili mkati mwautumiki (sizingafunike)

Momwe mungasonkhanitsire ma analytics a seva

Ngati tikufuna kusiya kugwiritsa ntchito ma analytics a kasitomala ndikudzipangira tokha, choyamba tiyenera kuganiza za kamangidwe ka dongosolo latsopanoli. Pansipa ndikuwuzani sitepe ndi sitepe zomwe muyenera kuziganizira, chifukwa chake sitepe iliyonse ikufunika komanso zida zomwe mungagwiritse ntchito.

1. Kulandira deta

Monga momwe zimakhalira ndi ma analytics a makasitomala, choyamba, akatswiri amakampani amasankha mitundu ya zochitika zomwe akufuna kuphunzira m'tsogolomu ndikuzisonkhanitsa pamndandanda. Kawirikawiri, zochitika izi zimachitika mwadongosolo, lotchedwa "chithunzi cha zochitika."
Kenako, taganizirani kuti pulogalamu yam'manja (tsamba lawebusayiti) ili ndi ogwiritsa ntchito nthawi zonse (zida) ndi ma seva ambiri. Kuti musamutse zochitika kuchokera ku zida kupita ku ma seva, gawo lapakati likufunika. Kutengera ndi kamangidwe kake, pakhoza kukhala mizere ingapo ya zochitika zosiyanasiyana.
Apache Kafka Ndi pub/sub queue, yomwe imagwiritsidwa ntchito ngati mzere wosonkhanitsa zochitika.

Malingana ndi positi pa Quora mu 2014, mlengi wa Apache Kafka adaganiza zopatsa dzina la pulogalamuyo pambuyo pa Franz Kafka chifukwa "ndi dongosolo lokonzekera kulemba" komanso chifukwa chokonda ntchito za Kafka. - Wikipedia

Mu chitsanzo chathu, pali ambiri opanga deta ndi ogula deta (zipangizo ndi ma seva), ndipo Kafka imathandiza kuwagwirizanitsa wina ndi mzake. Ogula adzafotokozedwa mwatsatanetsatane m'masitepe otsatirawa, kumene iwo adzakhala mitu yayikulu. Tsopano tingoganizira opanga ma data okha (zochitika).
Kafka amaphatikiza malingaliro amzere ndi magawo; ndibwino kuti muwerenge zambiri za izi kwina (mwachitsanzo, mu zolemba). Popanda kulowa mwatsatanetsatane, tiyerekeze kuti pulogalamu yam'manja imayambitsidwa ma OS awiri osiyanasiyana. Kenako mtundu uliwonse umapanga mtsinje wake wosiyana. Opanga amatumiza zochitika ku Kafka, zimalembedwa pamzere woyenera.

(chithunzi kuchokera pano)

Nthawi yomweyo, Kafka imakulolani kuti muwerenge m'machunks ndikukonza zochitika zingapo mumagulu ang'onoang'ono. Kafka ndi chida chothandiza kwambiri chomwe chimayenderana bwino ndi zosowa zomwe zikukula (mwachitsanzo, potengera zochitika).
Nthawi zambiri shard imodzi imakhala yokwanira, koma zinthu zimakhala zovuta kwambiri pokulitsa (monga momwe zimakhalira nthawi zonse). Mwinamwake palibe amene angafune kugwiritsa ntchito shard imodzi yokha pakupanga, popeza zomangamanga ziyenera kukhala zolekerera zolakwika. Kuphatikiza pa Kafka, pali yankho lina lodziwika bwino - RabbitMQ. Sitinagwiritse ntchito popanga ngati mzere wowunikira zochitika (ngati muli ndi chidziwitso chotere, tiuzeni za izi m'mawu!). Komabe, tidagwiritsa ntchito AWS Kinesis.

Tisanapite ku sitepe yotsatira, tiyenera kutchulanso gawo limodzi lowonjezera la dongosolo - kusungirako chipika chaiwisi. Izi sizosanjikiza zofunika, koma zingakhale zothandiza ngati china chake sichikuyenda bwino ndipo mizere ya zochitika ku Kafka yakhazikitsidwanso. Kusunga zipika zaiwisi sikufuna yankho lovuta komanso lokwera mtengo; mutha kungolemba penapake molondola (ngakhale pa hard drive).

2. Kukonza mitsinje ya zochitika

Pambuyo pokonzekera zochitika zonse ndikuziyika mumizere yoyenera, timapita ku sitepe yokonza. Pano ndikukuuzani za njira ziwiri zomwe zimagwiritsidwa ntchito kwambiri.
Njira yoyamba ndikupangitsa Spark Streaming mu Apache system. Zogulitsa zonse za Apache zimakhala pa HDFS, fayilo yotetezeka yokhala ndi mafayilo amafayilo. Spark Streaming ndi chida chosavuta kugwiritsa ntchito chomwe chimayendetsa bwino deta ndi masikelo. Komabe, zingakhale zovuta kusunga.
Njira ina ndikudzipangira nokha chowongolera zochitika. Kuti muchite izi, muyenera, mwachitsanzo, kulemba pulogalamu ya Python, kumanga ku Docker ndikulembetsa pamzere wa Kafka. Zoyambitsa zikafika kwa othandizira ma docker, kukonza kumayamba. Ndi njira iyi, muyenera kusunga mapulogalamu nthawi zonse.
Tiyerekeze kuti tasankha imodzi mwazosankha zomwe tafotokozazi ndikupitilira pakukonza komweko. Mapurosesa akuyenera kuyamba ndikuwona kutsimikizika kwa deta, kusefa zinyalala ndi zochitika "zosweka". Kuti titsimikizire timagwiritsa ntchito nthawi zambiri Cerberus. Pambuyo pake, mutha kupanga mapu a data: deta yochokera kumadera osiyanasiyana imasinthidwa ndikukhazikika kuti iwonjezeredwe patebulo wamba.

3. Database

Gawo lachitatu ndikusunga zochitika zokhazikika. Pogwira ntchito ndi makina owunikira okonzeka, tidzayenera kuwapeza pafupipafupi, chifukwa chake ndikofunikira kusankha nkhokwe yabwino.
Ngati deta ikugwirizana bwino ndi ndondomeko yokhazikika, mukhoza kusankha clickhouse kapena database ina ya columnar. Mwanjira iyi ma aggregations adzagwira ntchito mwachangu kwambiri. Choyipa ndichakuti chiwembucho chimakhazikika mokhazikika ndipo chifukwa chake sikungatheke kuwonjezera zinthu zosasinthika popanda kusinthidwa (mwachitsanzo, pakachitika chochitika chosavomerezeka). Koma mutha kuwerengera mwachangu kwambiri.
Kwa deta yosasinthika, mutha kutenga NoSQL, mwachitsanzo, Apache cassandra. Imagwira pa HDFS, imabwereza bwino, mutha kukweza nthawi zambiri, ndipo imalekerera zolakwika.
Mukhozanso kukweza chinthu chosavuta, mwachitsanzo, MongoDB. Ndi yochedwa kwambiri komanso ya mavoliyumu ang'onoang'ono. Koma kuphatikiza ndikuti ndikosavuta kwambiri ndipo ndikofunikira poyambira.

4. Kuphatikizika

Popeza tidasunga mosamala zochitika zonse, tikufuna kusonkhanitsa zidziwitso zonse zofunika kuchokera pagulu lomwe lafika ndikukonzanso nkhokwe. Padziko lonse lapansi, tikufuna kupeza ma dashboard ndi ma metric oyenerera. Mwachitsanzo, sonkhanitsani mbiri ya ogwiritsa ntchito kuchokera kuzochitika ndikuyesa machitidwe. Zochitika zimasonkhanitsidwa, kusonkhanitsidwa, ndikusungidwanso (m'matebulo a ogwiritsa ntchito). Panthawi imodzimodziyo, mukhoza kupanga dongosolo kuti muthe kugwirizanitsa fyuluta kwa aggregator-coordinator: sonkhanitsani ogwiritsa ntchito kuchokera kumtundu wina wa chochitika.
Pambuyo pake, ngati wina pagulu amangofuna ma analytics apamwamba, machitidwe a analytics akunja akhoza kulumikizidwa. Mutha kutenga Mixpanel kachiwiri. koma popeza ndi okwera mtengo, sizinthu zonse za ogwiritsa ntchito zomwe zimatumizidwa kumeneko, koma zomwe zimafunikira. Kuti tichite izi, tifunika kupanga wogwirizanitsa yemwe angasamutsire zochitika zosaphika kapena china chake chomwe ife tokha tinachiphatikiza kale ku machitidwe akunja, APIs kapena nsanja zotsatsa.

5. Kutsogolo

Muyenera kulumikiza kutsogolo kwa dongosolo lopangidwa. Chitsanzo chabwino ndi utumiki redash, ndi database ya GUI yomwe imathandizira kupanga dashboards. Momwe kuyankhulana kumagwirira ntchito:

1. Wogwiritsa amapanga funso la SQL.
2. Poyankha amalandira chizindikiro.
3. Imapanga 'zowonera zatsopano' ndikupeza chithunzi chokongola chomwe mungadzisungire nokha.

Zowoneka muutumiki ndikuzisintha zokha, mutha kusintha mwamakonda ndikutsata kuwunika kwanu. Redash ndi yaulere ngati mumadzichitira nokha, koma monga SaaS idzawononga $ 50 pamwezi.

Pomaliza

Mukamaliza masitepe onse pamwambapa, mupanga ma analytics anu a seva. Chonde dziwani kuti izi sizophweka monga kungolumikiza ma analytics a makasitomala, chifukwa chirichonse chiyenera kukonzedwa nokha. Chifukwa chake, musanapange dongosolo lanu, ndiyenera kufananiza kufunikira kwa dongosolo lalikulu la analytics ndi zinthu zomwe mukulolera kugawirako.
Ngati mwachita masamu ndikupeza kuti ndalamazo ndizokwera kwambiri, mu gawo lotsatira ndikambirana za momwe mungapangire ma analytics otsika mtengo a seva.

Zikomo powerenga! Ndidzakhala wokondwa kufunsa mafunso mu ndemanga.

Source: www.habr.com