PaSkyeng tinoshandisa Amazon Redshift, kusanganisira parallel scaling, saka takawana chinyorwa ichi naStefan Gromoll, muvambi we dotgo.com, ye intermix.io inonakidza. Mushure mekushandura, zvishoma zvechiitiko chedu kubva kune data engineer Daniyar Belkhodzhaev.
inobvumira kuyera nekuwedzera node nyowani kune cluster. Iko kudikanwa kwekutarisana nenhamba yepamusoro yezvikumbiro inogona kutungamira kune yakawandisa-kupa manode. Concurrency Scaling, kusiyana nekuwedzera node itsva, inowedzera simba rekombuta sezvinodiwa.
Amazon Redshift parallel scaling inopa Redshift masumbu ekuwedzera kugona kubata peak chikumbiro mavhoriyamu. Inoshanda nekufambisa zvikumbiro kune matsva "parallel" masumbu kumashure. Zvikumbiro zvinofambiswa zvichienderana neWLM kumisikidzwa nemitemo.
Parallel scaling pricing yakavakirwa pachikwereti modhi ine yemahara tier. Pamusoro pezvikwereti zvemahara, kubhadhara kunoenderana nenguva iyo Parallel Scaling Cluster inoitisa zvikumbiro.
Munyori akaedza kuenzana kuyera pane rimwe remasumbu emukati. Mune ino post, iye achataura nezve bvunzo mhinduro uye kupa matipi ekuti ungatanga sei.
Cluster zvinodiwa
Kuti ushandise parallel scaling, yako Amazon Redshift cluster inofanirwa kuita zvinotevera zvinodiwa:
- chikuva: EC2-VPC;
- node mhando: dc2.8xlarge, ds2.8xlarge, dc2.guru kana ds2.xlarge;
- nhamba yemanodhi: kubva pa2 kusvika ku32 (single node clusters haatsigirwe).
Mhando dzekukumbira dzinogamuchirwa
Parallel scaling haina kukodzera kumarudzi ese emibvunzo. Mushanduro yekutanga, inongogadzirisa kuverenga zvikumbiro zvinogutsa mamiriro matatu:
- SARUDZA mibvunzo inoverengwa-chete (kunyangwe mamwe marudzi akarongwa);
-mubvunzo hautauri tafura ine INTERLEAVED yekuronga maitiro;
-Mubvunzo haushandise Amazon Redshift Spectrum kuratidza ekunze matafura.
Kuti uendeswe kuParallel Scaling Cluster, chikumbiro chinofanira kuiswa mumutsara. Pamusoro pezvo, mibvunzo inokodzera mutsara , haizomhanye pazvikero zvakaenzanirana.
Mitsetse uye SQA inoda kurongeka kwakaringana . Isu tinokurudzira optimize WLM yako kutanga - izvi zvinoderedza kudiwa kwekuenzana kuyera. Uye izvi zvakakosha nekuti parallel scaling ndeyemahara chete kweimwe nhamba yemaawa. AWS inoti parallel scaling ichave yemahara kune 97% yevatengi, izvo zvinotisvitsa kunyaya yemitengo.
Mutengo weparallel scaling
AWS inopa kiredhiti modhi yekufananidza kuyera. Chikwata chega chega chinoshanda Inounganidza makiredhiti paawa, inosvika awa imwe yemahara parallel scaling kiredhiti pazuva.
Unobhadhara chete kana kushandiswa kwako kweParallel Scaling Clusters kwakapfuura huwandu hwezvikwereti zvawakagamuchira.
Mutengo unoverengerwa pasekondi yega yega pa-inoda chiyero cheparallel cluster inoshandiswa pamusoro peyero yemahara. Unobhadhariswa chete kwenguva yezvikumbiro zvako, uine muripo wepasi peminiti imwe nguva imwe neimwe Parallel Scaling Cluster inovhurwa. Mari yesekondi yega yega pa-inodiwa inoverengerwa zvichienderana nemisimboti yemitengo , ndiko kuti, zvinoenderana nerudzi rwenodhi uye nhamba yemanodhi musumbu rako.
Kutangisa Parallel Scaling
Parallel scaling inotangwa pamutsara wega wega weWLM. Enda kuAWS Redshift koni uye sarudza Workload Management kubva kuruboshwe rwekutenderera menyu. Sarudza cluster yako WLM parameter boka kubva pane inotevera yekudonha-pasi menyu.
Iwe uchaona koramu itsva inonzi "Concurrency Scaling Mode" padivi pemutsetse wega wega. Iyo default ndeye "Disabled". Dzvanya "Edit" uye unogona kushandura marongero emutsetse wega wega.
Kugadziriswa
Parallel scaling inoshanda nekutumira zvikumbiro zvakakodzera kumasumbu matsva akazvitsaurira. Masumbu matsva ane saizi yakafanana (rudzi uye nhamba yemanodhi) sesumbu guru.
Nhamba yekusarudzika yemasumbu anoshandiswa kuenzanirana kuyera imwe (1), ine kugona kugadzirisa kusvika kune gumi (10) masumbu.
Huwandu hwemasumbu ekuenzana kuyera hunogona kusetwa neiyo max_concurrency_scaling_clusters parameter. Kuwedzera kukosha kweiyi parameter kunopa mamwe masumbu akawandisa.
Kuongorora
Kune akati wandei ekuwedzera magirafu anowanikwa muAWS Redshift koni. Chati yeMax Configured Concurrency Scaling Clusters inoratidza kukosha kwemax_concurrency_scaling_clusters nekufamba kwenguva.
Huwandu hwemasumbu anoshanda ekuyera anoratidzwa mune yevashandisi interface muchikamu che "Concurrency Scaling Activity":
MuQueries tab, pane koramu inoratidza kana mubvunzo wakaitwa musumbu guru kana muparallel scaling cluster:
Zvisinei nekuti mumwe muvhunzo wakaitwa muboka guru here kana kuti kuburikidza nechikwata chekuenzanisa, chinochengetwa mu stl_query.concurrency_scaling_status.
Kukosha kwe1 kunoratidza kuti muvhunzo wakaitwa muparallel scale cluster, nepo humwe hunhu hunoratidza kuti hwakaitwa muboka rekutanga.
Muenzaniso:
Concurrency scaling information inochengetwawo mune mamwe matafura uye maonero, seSVCS_CONCURRENCY_SCALING_USAGE. Pamusoro pezvo, kune akati wandei ematafura ekatalogi anochengeta ruzivo nezve parallel scaling.
Mhinduro
Vanyori vakatanga kuyera mutsara mumwe muboka remukati nguva dzinokwana 18:30:00 GMT musi wa29.03.2019/3/20. Yakashandura max_concurrency_scaling_clusters parameter kuita 30 nenguva dzinosvika 00:29.03.2019:XNUMX musi waXNUMX/XNUMX/XNUMX.
Kutevedzera mutsara wekukumbira, takadzikisa huwandu hwenzvimbo dzemutsetse uyu kubva pa15 kusvika pa5.
Pazasi pane intermix.io dashboard chati inoratidza huwandu hwezvikumbiro zvirikumhanya uye kuita mutsetse mushure mekudzikisa huwandu hwenzvimbo.
Tinoona kuti nguva yekumirira yezvikumbiro mumutsara yakawedzera, uye nguva yakawanda inopfuura maminitsi mashanu.
Heino ruzivo rwakakodzera kubva kuAWS console nezve zvakaitika panguva iyi:
Redshift yakatangisa matatu (3) anoenderana kuyera masumbu sekugadziriswa. Zvinoita sekuti masumbu aya haana kushandiswa zvishoma, kunyangwe zvikumbiro zvakawanda muchikwata chedu zvakaiswa mumutsara.
Girafu rekushandisa rinoenderana neyekuyera chiitiko girafu:
Mushure memaawa mashoma, vanyori vakatarisa mutsara uye zvaiita sekunge zvikumbiro zvitanhatu zvaimhanya pamwe chete. Isu takaedzawo zvisina tsarukano zvikumbiro zviviri kuburikidza nemushandisi interface. Hatina kutarisa mashandisiro ezvikoshero izvi kana akati wandei akafanana masumbu ari kushanda kamwechete.
zvakawanikwa
Parallel scaling inogona kuderedza nguva yekukumbira inopedzwa mumutsara panguva yekuremerwa kwepamusoro.
Zvichienderana nemhedzisiro yebvunzo yekutanga, zvakazoitika kuti mamiriro ekurodha zvikumbiro akawedzera zvishoma. Nekudaro, parallel scaling chete haina kugadzirisa matambudziko ese emari.
Izvi zvinokonzerwa nezvirambidzo pamhando dzemibvunzo inogona kushandisa parallel scaling. Semuenzaniso, vanyori vane matafura akawanda ane interleaved sort kiyi, uye mazhinji ebasa redu nderekunyora.
Kunyangwe kuenzana kuyera isiri mhinduro yepasirese yekumisikidza WLM, kushandisa iyi ficha iri nyore uye yakatwasuka.
Naizvozvo, munyori anokurudzira kuishandisa kune yako WLM mitsetse. Tanga nesumbu rimwe rakafanana uye tarisa peak mutoro kuburikidza nekoni kuti uone kana masumbu matsva ari kushandiswa zvizere.
Sezvo AWS inowedzera tsigiro yemamwe marudzi emibvunzo uye matafura, parallel scaling inofanira zvishoma nezvishoma kuwedzera uye inoshanda.
Mhinduro kubva kuna Daniyar Belkhodzhaev, Skyeng Data Engineer
Isu paSkyeng zvakare takabva taona mukana uri kubuda wekuyera kuenzanirana.
Kushanda kwacho kunoyevedza, kunyanya tichifunga kuti AWS inofungidzira kuti vashandisi vazhinji havazotofanira kubhadhara yakawedzera pazviri.Zvakaitika kuti pakati paApril takava nekusajairika kwezvikumbiro kuRedshift cluster. Munguva iyi, taiwanzoenda kuConcurrency Scaling; dzimwe nguva boka rekuwedzera raishanda maawa makumi maviri nemana pazuva risingamire.
Izvi zvakaita kuti zvigoneke, kana zvisiri kugadzirisa zvachose dambudziko nemitsara, zvino kuita kuti mamiriro acho agamuchirwe.
Zvatinoona zvinonyanya kuenderana nezvinofungwa nevakomana kubva intermix.io.
Takaonawo kuti kunyange zvazvo paiva nezvikumbiro zvakamirira mumutsetse, hazvisi zvikumbiro zvose zvakakurumidza kuendeswa kuchikwata chakafanana. Sezviri pachena izvi zvinoitika nekuti iyo parallel cluster ichiri kutora nguva kutanga. Nekuda kweizvozvo, mukati menguva pfupi-yepamusoro mitoro isu tichine mitsetse midiki, uye maalarm anowirirana ane nguva yekukonzeresa.
Sezvo tabvisa mitoro isina kujairika muna Kubvumbi, isu, sezvaitarisirwa neAWS, takapinda apo neapo maitiro ekushandisa - mukati memahara emahara.
Iwe unogona kuteedzera yako yakafanana kuyera mitengo muAWS Mutengo Explorer. Iwe unofanirwa kusarudza Service - Redshift, Usage Type - CS, semuenzaniso USW2-CS:dc2.large.Iwe unogona kuverenga zvakawanda nezvemitengo muRussia
Source: www.habr.com