Ku Skyeng timagwiritsa ntchito Amazon Redshift, kuphatikiza makulitsidwe ofanana, chifukwa chake tapeza nkhaniyi ndi Stefan Gromoll, woyambitsa dotgo.com, pa intermix.io yosangalatsa. Pambuyo pomasulira, pang'ono zomwe takumana nazo kuchokera kwa katswiri wa data Daniyar Belkhodzhaev.
amalola makulitsidwe powonjezera node atsopano ku tsango. Kufunika kothana ndi kuchuluka kwa zopempha kungayambitse kuperekedwa kwa node mopitilira muyeso. Concurrency Scaling, m'malo mowonjezera ma node atsopano, imawonjezera mphamvu zamakompyuta ngati pakufunika.
Amazon Redshift parallel scaling imapatsa magulu a Redshift mphamvu zowonjezera kuti athe kuthana ndi kuchuluka kwa pempho. Zimagwira ntchito posuntha zopempha kumagulu atsopano "ofanana" kumbuyo. Zopempha zimayendetsedwa motengera kasinthidwe ndi malamulo a WLM.
Kukweza mitengo yofananira kumatengera mtundu wangongole wokhala ndi gawo laulere. Pamwamba pa ngongole zaulere, malipiro amatengera nthawi yomwe Parallel Scaling Cluster ikufunsira.
Wolembayo adayesa makulitsidwe ofanana pagulu limodzi lamkati. Mu positi iyi, alankhula za zotsatira za mayeso ndikupereka malangizo amomwe mungayambire.
Zofunikira zamagulu
Kuti mugwiritse ntchito makulitsidwe ofanana, gulu lanu la Amazon Redshift liyenera kukwaniritsa izi:
- nsanja: EC2-VPC;
- mtundu wa node: dc2.8xlarge, ds2.8xlarge, dc2.large kapena ds2.xlarge;
- chiwerengero cha nodes: kuyambira 2 mpaka 32 (magulu amodzi a node sakuthandizidwa).
Mitundu yovomerezeka yofunsira
Parallel makulitsidwe sikoyenera kwa mitundu yonse ya mafunso. Mu mtundu woyamba, imangowerengera zopempha zomwe zimakwaniritsa zinthu zitatu:
- ZOSANKHA mafunso amawerengedwa-okha (ngakhale mitundu yambiri ikukonzekera);
- funso silikunena za tebulo lomwe lili ndi kalembedwe ka INTERLEAVED;
- Funso siligwiritsa ntchito Amazon Redshift Spectrum kutengera matebulo akunja.
Kuti mutumize ku Parallel Scaling Cluster, pempho liyenera kukhala pamzere. Komanso, mafunso oyenera pamzere , sikuyenda pamagulu a sikelo zofananira.
Mizere ndi SQA zimafunikira masinthidwe oyenera . Tikupangira kukhathamiritsa WLM yanu poyamba - izi zichepetsa kufunikira kofananako. Ndipo izi ndizofunikira chifukwa makulitsidwe ofanana ndi aulere kwa maola angapo. AWS imati makulitsidwe ofanana adzakhala aulere kwa 97% yamakasitomala, zomwe zimatifikitsa ku nkhani yamitengo.
Mtengo wa makulitsidwe ofanana
AWS imapereka mtundu wangongole pakukweza kofananira. Gulu lililonse logwira ntchito Amadziunjikira ma kirediti pa ola limodzi, mpaka ola limodzi la makulitsidwe aulere ofananira patsiku.
Mumalipira kokha pamene kugwiritsa ntchito kwanu kwa Parallel Scaling Clusters kupitirira kuchuluka kwa ngongole zomwe mwalandira.
Mtengo wake umawerengedwa pamphindi pa sekondi iliyonse pakufunidwa kwa gulu lofananira lomwe limagwiritsidwa ntchito pamwamba pa mtengo waulere. Mumalipidwa panthawi yonse ya zopempha zanu, ndikulipiritsa osachepera mphindi imodzi nthawi iliyonse Parallel Scaling Cluster ikatsegulidwa. Mtengo wa sekondi iliyonse pofunidwa umawerengedwa potengera mfundo zamitengo , ndiye kuti, zimatengera mtundu wa node ndi kuchuluka kwa node mumagulu anu.
Kukhazikitsa Parallel Scaling
Kukula kofanana kumayambika pamzere uliwonse wa WLM. Pitani ku AWS Redshift console ndikusankha Workload Management kuchokera kumanzere kumanzere. Sankhani gulu la WLM la cluster lanu kuchokera pazotsatira zotsikira pansi.
Mudzawona ndime yatsopano yotchedwa "Concurrency Scaling Mode" pafupi ndi mzere uliwonse. Chokhazikika ndi "Olemala". Dinani "Sinthani" ndipo mutha kusintha zosintha pamzere uliwonse.
Kukhazikika
Kukweza kofanana kumagwira ntchito potumiza zopempha zoyenera kumagulu atsopano odzipatulira. Magulu atsopano ali ndi kukula kofanana (mtundu ndi chiwerengero cha node) monga gulu lalikulu.
Nambala yosasinthika yamagulu omwe amagwiritsidwa ntchito pokulitsa magawo ofanana ndi amodzi (1), omwe amatha kukonza mpaka magulu khumi (10).
Chiwerengero chonse cha magulu okulira kofanana chikhoza kukhazikitsidwa ndi max_concurrency_scaling_clusters parameter. Kuchulukitsa mtengo wa parameter iyi kumapereka magulu owonjezera owonjezera.
Kuwunikira
Pali ma graph angapo owonjezera omwe amapezeka mu AWS Redshift console. Tchati cha Max Configured Concurrency Scaling Clusters chikuwonetsa mtengo wa max_concurrency_scaling_clusters pakapita nthawi.
Chiwerengero cha magulu omwe akugwira ntchito chikuwonetsedwa mu mawonekedwe a ogwiritsa ntchito mu gawo la "Concurrency Scaling Activity":
Mu tabu ya Queries, pali mzati wosonyeza ngati funsolo lidachitidwa pagulu lalikulu kapena mgulu lofananira:
Mosasamala kanthu kuti funso linalake lidachitidwa mu gulu lalikulu kapena kudzera mu gulu lokulitsa lofananira, limasungidwa mu stl_query.concurrency_scaling_status.
Mtengo wa 1 ukuwonetsa kuti funsoli lidachitidwa pagulu lofananira, pomwe zikhalidwe zina zikuwonetsa kuti zidachitika mgulu loyamba.
Chitsanzo:
Zambiri zakukweza ndalama zimasungidwanso m'matebulo ndi mawonedwe ena, monga SVCS_CONCURRENCY_SCALING_USAGE. Kuphatikiza apo, pali matebulo angapo am'ndandanda omwe amasunga zambiri zokhudzana ndi kukula kofananira.
Zotsatira
Olembawo adayamba kukweza pamzere umodzi mgulu lamkati pafupifupi 18:30:00 GMT pa 29.03.2019/3/20. Anasintha max_concurrency_scaling_clusters parameter kukhala 30 pafupifupi 00:29.03.2019:XNUMX pa XNUMX/XNUMX/XNUMX.
Kuti tiyerekezere pamzere wopempha, tachepetsa mipata ya pamzerewu kuchoka pa 15 kufika pa 5.
Pansipa pali tchati cha dashboard ya intermix.io yomwe ikuwonetsa kuchuluka kwa zopempha zomwe zikuchitika ndikuima pamzere mutachepetsa kuchuluka kwa malo olowera.
Tikuwona kuti nthawi yodikirira zopempha pamzere yawonjezeka, ndipo nthawi yayitali imakhala yopitilira mphindi 5.
Nayi chidziwitso chofunikira kuchokera ku konsoli ya AWS pazomwe zidachitika panthawiyi:
Redshift idakhazikitsa masango atatu (3) ofanana makulitsidwe monga adakonzedwa. Zikuwoneka kuti maguluwa sanagwiritsidwe ntchito mochepera, ngakhale zopempha zambiri mgulu lathu zidayikidwa pamzere.
Grafu yogwiritsira ntchito imagwirizana ndi makulitsidwe a graph:
Pambuyo pa maola angapo, olembawo adayang'ana pamzerewu ndipo zikuwoneka ngati zopempha za 6 zikuyenda mofanana. Tinayesanso mwachisawawa zopempha ziwiri kudzera mu mawonekedwe ogwiritsira ntchito. Sitinayang'ane momwe tingagwiritsire ntchito mfundozi pamene magulu angapo ofanana akugwira ntchito nthawi imodzi.
anapezazo
Kuchulukitsa kofanana kumatha kuchepetsa nthawi yomwe zopempha zimathera pamzere panthawi yochulukirachulukira.
Kutengera zotsatira za mayeso oyambira, zidapezeka kuti zinthu zofunsira zotsitsa zasintha pang'ono. Komabe, kufanana kokha sikunathetse mavuto onse ogwirizana.
Izi ndichifukwa cha zoletsa pamitundu yamafunso omwe angagwiritse ntchito makulitsidwe ofanana. Mwachitsanzo, olembawo ali ndi matebulo ambiri okhala ndi makiyi amtundu wa interleaved, ndipo ntchito yathu yambiri ndikulemba.
Ngakhale makulitsidwe ofananira si njira yachilengedwe yokhazikitsa WLM, kugwiritsa ntchito izi ndikosavuta komanso kosavuta.
Chifukwa chake, wolemba akupangira kuti mugwiritse ntchito pamzere wanu wa WLM. Yambani ndi gulu limodzi lofananira ndikuwunika kuchuluka kwazomwe zikuchitika kudzera pa kontrakitala kuti muwone ngati masango atsopanowa akugwiritsidwa ntchito mokwanira.
Monga AWS imawonjezera chithandizo chamitundu yamafunso ndi matebulo owonjezera, kukweza kofananira kuyenera kukhala kothandiza kwambiri.
Ndemanga kuchokera kwa Daniyar Belkhodzhaev, Skyeng Data Engineer
Ife ku Skyeng tinawonanso nthawi yomweyo kuthekera kokulirakulira kofanana.
Magwiridwe ake ndi okongola kwambiri, makamaka poganizira kuti AWS ikuyerekeza kuti ogwiritsa ntchito ambiri safunikanso kulipira zowonjezera.Zidachitika kuti pakati pa mwezi wa April tinali ndi zopempha zachilendo ku gulu la Redshift. Panthawi imeneyi, nthawi zambiri timagwiritsa ntchito Concurrency Scaling; nthawi zina gulu lowonjezera limagwira ntchito maola 24 patsiku osayima.
Izi zidapangitsa kuti, ngati sikuthetsa vutolo ndi mizere, ndiye kuti zinthuzo zikhale zovomerezeka.
Zomwe tikuwona makamaka zimagwirizana ndi zomwe anyamata ochokera ku intermix.io.
Tidawonanso kuti ngakhale panali zopempha zomwe zikudikirira pamzere, si zopempha zonse zomwe zidatumizidwa mwachangu kugulu lofananira. Zikuoneka kuti izi zimachitika chifukwa gulu lofananira limatenga nthawi kuti liyambe. Chotsatira chake, panthawi yonyamula katundu wanthawi yochepa timakhalabe ndi mizere yaying'ono, ndipo ma alarm omwe ali nawo amakhala ndi nthawi yoyambitsa.
Titachotsa zolemetsa zachilendo mu Epulo, ife, monga momwe AWS amayembekezera, tinalowa mwa apo ndi apo - mkati mwa chizolowezi chaulere.
Mutha kutsata makulitsidwe anu ofanana mu AWS Cost Explorer. Muyenera kusankha Service - Redshift, Usage Type - CS, mwachitsanzo USW2-CS:dc2.large.Mutha kuwerenga zambiri zamitengo mu Russian
Source: www.habr.com