Kwa miyezi isanu ndi umodzi yapitayi ndakhala ndikupanga dongosolo lolimbana ndi chinyengo (zachinyengo, chinyengo, ndi zina zotero) popanda maziko oyambirira a izi. Malingaliro amasiku ano omwe tapeza ndikukhazikitsa m'dongosolo lathu amatithandiza kuzindikira ndi kusanthula zochitika zambiri zachinyengo. M'nkhaniyi, ndikufuna kunena za mfundo zomwe timatsatira komanso zomwe tinachita kuti tikwaniritse dongosolo lathu lamakono, popanda kulowa mu gawo laukadaulo.
Mfundo za dongosolo lathu
Mukamva mawu ngati "zokha" ndi "zachinyengo," nthawi zambiri mumayamba kuganiza za kuphunzira pamakina, Apache Spark, Hadoop, Python, Airflow, ndi matekinoloje ena ochokera ku Apache Foundation ecosystem ndi gawo la Science Science. Ndikuganiza kuti pali mbali imodzi yogwiritsira ntchito zidazi zomwe sizimatchulidwa kawirikawiri: zimafuna zofunikira zina mubizinesi yanu musanayambe kuzigwiritsa ntchito. Mwachidule, mufunika nsanja ya data yamabizinesi yomwe imaphatikizapo nyanja ya data ndi nyumba yosungiramo zinthu. Koma bwanji ngati mulibe nsanja yotere ndipo mukufunikirabe kupanga izi? Mfundo zotsatirazi zomwe ndikugawana pansipa zatithandiza kufikira pomwe titha kuyang'ana kwambiri pakuwongolera malingaliro athu m'malo mopeza zomwe zimagwira ntchito. Komabe, iyi si gawo la polojekitiyi. Palinso zinthu zambiri mu ndondomekoyi kuchokera kuzinthu zamakono ndi zamakono.
Mfundo 1: Phindu la Bizinesi Choyamba
Timayika "mtengo wabizinesi" patsogolo pa zoyesayesa zathu zonse. Nthawi zambiri, njira iliyonse yowunikira yokha ndi ya gulu la machitidwe ovuta omwe ali ndi mlingo wapamwamba wa automation ndi zovuta zamakono. Kupanga yankho lathunthu kudzatenga nthawi yochuluka ngati mupanga izo kuchokera pachiyambi. Tinaganiza zoyika phindu labizinesi patsogolo ndi kukwanira kwaukadaulo kachiwiri. Mβmoyo weniweni, izi zikutanthauza kuti sitivomereza ukadaulo wapamwamba ngati chiphunzitso. Timasankha ukadaulo womwe umatithandizira kwambiri pakadali pano. M'kupita kwa nthawi, zikhoza kuwoneka kuti tidzayenera kukhazikitsanso ma modules ena. Uku ndiye kunyengerera komwe tidavomereza.
Mfundo 2: Nzeru zowonjezera
Ndikubetcha kuti anthu ambiri omwe sakhudzidwa kwambiri pakupanga mayankho ophunzirira makina angaganize kuti m'malo mwa anthu ndiye cholinga. M'malo mwake, mayankho ophunzirira makina siangwiro ndipo m'malo ena okha ndi omwe angasinthidwe. Tinakana lingaliro ili kuyambira pachiyambi pazifukwa zingapo: deta yosalinganika pazochitika zachinyengo komanso kulephera kupereka mndandanda wazinthu zamitundu yophunzirira makina. Mosiyana ndi izi, tinasankha njira yanzeru yowonjezera. Ili ndi lingaliro lina laluntha lochita kupanga lomwe limayang'ana kwambiri gawo lothandizira la AI, kutsindika mfundo yakuti matekinoloje ozindikira amapangidwa kuti apititse patsogolo luntha la anthu m'malo molowa m'malo mwake. [1]
Poganizira izi, kupanga njira yophunzirira makina onse kuyambira pachiyambi kungafune khama lalikulu, zomwe zingachedwetse kupanga phindu la bizinesi yathu. Tidaganiza zopanga makina omwe akukula mopitilira muyeso motsogozedwa ndi akatswiri athu amadomeni. Gawo lovuta pakupanga dongosolo lotere ndiloti liyenera kupereka ofufuza athu milandu osati pokhapokha ngati ndi ntchito yachinyengo kapena ayi. Nthawi zambiri, vuto lililonse lamakasitomala ndi nkhani yokayikitsa yomwe akatswiri amayenera kufufuza ndikuyankha mwanjira ina. Gawo lochepa chabe la milandu yomwe yafotokozedwayi ingatchulidwedi ngati chinyengo.
Mfundo 3: Platform Rich Analytics
Gawo lovuta kwambiri la dongosolo lathu ndikutsimikizira komaliza mpaka kumapeto kwa kayendetsedwe ka kayendetsedwe kake. Openda ndi okonza ayenera kupeza mosavuta ma seti akale omwe ali ndi ma metrics onse omwe amagwiritsidwa ntchito pounika. Kuphatikiza apo, nsanja ya data iyenera kupereka njira yosavuta yolumikizira ma metric omwe alipo ndi atsopano. Njira zomwe timapanga, ndipo izi sizinthu zamapulogalamu chabe, ziyenera kutilola kuwerengeranso nthawi zam'mbuyomu, kuwonjezera ma metrics atsopano ndikusintha zolosera. Titha kukwaniritsa izi mwa kusonkhanitsa deta yonse yomwe makina athu opanga amapanga. Pankhaniyi, deta pang'onopang'ono kukhala vuto. Tifunika kusunga kuchuluka kwa data yomwe sitigwiritsa ntchito ndikuyiteteza. Zikatero, deta idzakhala yosafunikira pakapita nthawi, komabe pamafunika khama lathu kuti tiziwongolera. Kwa ife, kusunga deta sikunali kwanzeru, choncho tinaganiza zotengera njira ina. Tinaganiza zokonza masitolo a nthawi yeniyeni mozungulira mabungwe omwe tikufuna kuwayika m'magulu, ndikusunga zomwe zimatilola kuyang'ana nthawi zaposachedwa komanso zofunikira. Chovuta pakuchita izi ndikuti dongosolo lathu ndi losiyana kwambiri, lomwe lili ndi malo ambiri osungiramo data ndi ma module apulogalamu omwe amafunikira kukonzekera mosamala kuti agwire ntchito mosasintha.
Malingaliro opangira dongosolo lathu
Tili ndi zigawo zinayi zazikuluzikulu m'dongosolo lathu: makina omeza, computational, kusanthula kwa BI ndi njira yotsata. Amakwaniritsa zolinga zawozawo, ndipo timawalekanitsa potsatira njira zopangira.
Mapangidwe otengera makontrakitala
Choyamba, tidavomereza kuti zigawo ziyenera kudalira ma data ena (makontrakitala) omwe amadutsa pakati pawo. Izi zimapangitsa kuti zikhale zosavuta kuphatikizira pakati pawo osati kukakamiza kupangidwa kwapadera (ndi dongosolo) la zigawo. Mwachitsanzo, nthawi zina izi zimatilola kuphatikizira mwachindunji njira yolowera ndi njira yowunikira chenjezo. Zikatero, izi zidzachitika motsatira mgwirizano wa chenjezo womwe wagwirizana. Izi zikutanthauza kuti zigawo zonsezi zidzaphatikizidwa pogwiritsa ntchito mgwirizano womwe chigawo china chilichonse chingagwiritse ntchito. Sitikuwonjezera mgwirizano kuti tiwonjezere zidziwitso ku dongosolo lolondolera kuchokera pamakina olowetsa. Njirayi imafuna kugwiritsa ntchito chiwerengero chodziwikiratu cha makontrakitala ndi kufewetsa dongosolo ndi mauthenga. M'malo mwake, timatenga njira yotchedwa "Contract First Design" ndikuyigwiritsa ntchito pamakontrakitala otsatsa. [2]
Kukhamukira kulikonse
Kupulumutsa ndi kuyang'anira boma mu dongosolo mosalephera kumabweretsa zovuta pakukhazikitsa kwake. Kawirikawiri, boma liyenera kupezeka kuchokera ku gawo lililonse, liyenera kukhala lokhazikika ndikupereka mtengo wamakono pamagulu onse, ndipo liyenera kukhala lodalirika ndi zikhalidwe zolondola. Kuphatikiza apo, kukhala ndi mafoni osungirako mosalekeza kuti mutengenso dziko laposachedwa kumawonjezera kuchuluka kwa magwiridwe antchito a I/O ndi zovuta zama algorithms omwe amagwiritsidwa ntchito pamapaipi athu anthawi yeniyeni. Chifukwa cha izi, tinaganiza zochotsa kusungirako kwa boma, ngati n'kotheka, kwathunthu ku dongosolo lathu. Njirayi imafuna kuti deta yonse yofunikira iphatikizidwe mu block block (uthenga). Mwachitsanzo, ngati tifunika kuwerengera chiwerengero cha ziwonetsero zina (chiwerengero cha ntchito kapena zochitika zomwe zili ndi makhalidwe ena), timawerengera pamtima ndikupanga mitsinje yamtengo wapatali. Ma module odalira adzagwiritsa ntchito magawo ndi batching kuti agawanitse mtsinjewo kukhala mabungwe ndikugwira ntchito pazotsatira zaposachedwa. Njirayi inathetsa kufunika kokhala ndi diski yosungiramo deta yotereyi. Dongosolo lathu limagwiritsa ntchito Kafka ngati meseji broker ndipo litha kugwiritsidwa ntchito ngati nkhokwe ndi KSQL. [3] Koma kuzigwiritsa ntchito zikanamanga yankho lathu kwambiri ku Kafka, ndipo tinaganiza kuti tisagwiritse ntchito. Njira yomwe tinasankha imatilola kuti tilowe m'malo mwa Kafka ndi uthenga wina wogulitsa popanda kusintha kwakukulu mkati mwa dongosolo.
Lingaliro ili silikutanthauza kuti sitigwiritsa ntchito disk yosungirako ndi nkhokwe. Kuti tiyese ndi kusanthula machitidwe a dongosolo, tiyenera kusunga deta yochuluka pa disk yomwe imayimira ma metrics ndi mayiko osiyanasiyana. Mfundo yofunika apa ndikuti ma algorithms a nthawi yeniyeni sadalira deta yotere. Nthawi zambiri, timagwiritsa ntchito deta yosungidwa kuti tifufuze popanda intaneti, kuchotsa zolakwika ndi kufufuza zochitika zinazake ndi zotsatira zomwe dongosolo limapanga.
Mavuto a dongosolo lathu
Pali mavuto ena omwe tawathetsa mpaka kufika pamlingo wina, koma amafuna mayankho oganiza bwino. Tsopano ndingofuna kuzitchula pano chifukwa mfundo iliyonse ili ndi nkhani yakeyake.
- Tikufunikabe kufotokozera ndondomeko ndi ndondomeko zomwe zimathandizira kusonkhanitsa deta yothandiza komanso yofunikira kuti tipeze deta yathu yodzipangira yokha, kufufuza, ndi kufufuza.
- Kuphatikizika kwa kusanthula kwa anthu kumabweretsa njira yodzipangira zokha makinawo kuti asinthe ndi zomwe zachitika posachedwa. Uku sikungosintha mtundu wathu, komanso kukonzanso njira zathu ndikuwongolera kumvetsetsa kwathu deta.
- Kupeza malire pakati pa njira yotsimikizika ya IF-ELSE ndi ML. Wina adati, "ML ndi chida chaosimidwa." Izi zikutanthauza kuti mudzafuna kugwiritsa ntchito ML pomwe simukumvetsetsanso momwe mungakwaniritsire ndikuwongolera ma aligorivimu anu. Kumbali inayi, njira yodziwikiratu simalola kuzindikira zolakwika zomwe sizinali kuyembekezera.
- Timafunikira njira yosavuta yoyesera malingaliro athu kapena kulumikizana pakati pa ma metric mu data.
- Dongosololi liyenera kukhala ndi magawo angapo a zotsatira zowona. Milandu yachinyengo ndi gawo laling'ono chabe la milandu yonse yomwe ingaganizidwe kuti ndi yabwino padongosolo. Mwachitsanzo, akatswiri akufuna kulandira milandu yonse yokayikitsa kuti itsimikizidwe, ndipo gawo laling'ono chabe laiwo ndi lachinyengo. Dongosololi liyenera kuwonetsa bwino milandu yonse kwa akatswiri, mosasamala kanthu kuti ndi zachinyengo zenizeni kapena zongokayikira.
- Dongosolo la data liyenera kupezanso ma seti a mbiri yakale ndi mawerengedwe opangidwa ndikuwerengedwa pa ntchentche.
- Ingotumizani mosavuta komanso mwachangu zida zilizonse zamakina m'malo osachepera atatu: kupanga, kuyesa (beta) ndi opanga.
- Ndipo pomalizira pake. Tiyenera kupanga pulatifomu yoyezera magwiridwe antchito pomwe titha kusanthula zitsanzo zathu. [4]
powatsimikizira
Source: www.habr.com