Ntawm Skyeng peb siv Amazon Redshift, suav nrog kev sib npaug sib npaug, yog li peb pom cov kab lus no los ntawm Stefan Gromoll, tus tsim ntawm dotgo.com, rau intermix.io nthuav. Tom qab kev txhais lus, me ntsis ntawm peb kev paub los ntawm cov ntaub ntawv engineer Daniyar Belkhodzhaev.
tso cai scaling los ntawm kev ntxiv cov nodes tshiab rau pawg. Qhov xav tau los tiv thaiv tus naj npawb ntawm qhov kev thov tuaj yeem ua rau muaj ntau tshaj ntawm cov nodes. Concurrency Scaling, as opposed to add new nodes, tsub kom suav zog raws li xav tau.
Amazon Redshift parallel scaling muab Redshift pawg ntxiv peev xwm los tswj cov ntawv thov siab kawg. Nws ua haujlwm los ntawm kev txav cov lus thov mus rau cov pawg tshiab "parallel" hauv keeb kwm yav dhau. Kev thov raug xa mus raws li WLM teeb tsa thiab cov cai.
Parallel scaling pricing is based on a credit model with a free tier. Tshaj li cov qhab nia pub dawb, kev them nyiaj yog nyob ntawm lub sijhawm Parallel Scaling Cluster ua cov txheej txheem thov.
Tus neeg sau tau sim ntsuas qhov sib npaug ntawm ib qho ntawm cov pawg sab hauv. Hauv tsab ntawv no, nws yuav tham txog cov txiaj ntsig kev xeem thiab muab cov lus qhia txog yuav pib li cas.
Cov kev cai ntawm pawg
Txhawm rau siv qhov sib npaug sib npaug, koj Amazon Redshift pawg yuav tsum ua raws li cov cai hauv qab no:
- platform: EC2-VPC;
- hom node: dc2.8xlarge, ds2.8xlarge, dc2.large lossis ds2.xlarge;
- tus naj npawb ntawm nodes: los ntawm 2 mus rau 32 (ib leeg ntawm pawg tsis tau txais kev txhawb nqa).
Txais hom kev thov
Parallel scaling tsis haum rau txhua hom lus nug. Hauv thawj version, nws tsuas yog ua cov ntawv thov uas txaus siab rau peb yam xwm txheej:
- SELECT cov lus nug tsuas yog nyeem nkaus xwb (txawm tias muaj ntau hom kev npaj);
- cov lus nug tsis siv lub rooj nrog INTERLEAVED sorting style;
- Cov lus nug tsis siv Amazon Redshift Spectrum los siv cov rooj sab nraud.
Yuav kom xa mus rau Parallel Scaling Cluster, qhov kev thov yuav tsum tau queued. Tsis tas li ntawd, cov lus nug tsim nyog rau lub queue , yuav tsis khiav ntawm parallel scale pawg.
Queues thiab SQA xav tau kev teeb tsa kom raug . Peb pom zoo kom optimizing koj WLM ua ntej - qhov no yuav txo qhov kev xav tau ntawm qhov sib npaug. Thiab qhov no yog qhov tseem ceeb vim tias kev ntsuas qhov sib npaug tsuas yog pub dawb rau qee lub sijhawm. AWS thov tias qhov sib npaug sib npaug yuav pub dawb rau 97% ntawm cov neeg siv khoom, uas coj peb mus rau qhov teeb meem ntawm tus nqi.
Tus nqi ntawm parallel scaling
AWS muab cov qauv qiv nyiaj rau kev ntsuas qhov sib npaug. Txhua pawg active Ua kom cov qhab nia ib teev, mus txog ib teev ntawm cov qhab nia sib npaug pub dawb hauv ib hnub.
Koj tsuas yog them thaum koj qhov Parallel Scaling Clusters siv ntau dua li cov qhab nia koj tau txais.
Tus nqi yog xam ntawm tus nqi ib-thib ob ntawm qhov xav tau rau ib pawg sib npaug uas yog siv los ntawm tus nqi pub dawb. Koj tsuas yog them rau lub sijhawm ntawm koj qhov kev thov, nrog tus nqi tsawg kawg nkaus ntawm ib feeb txhua lub sijhawm ua haujlwm Parallel Scaling Cluster. Tus nqi ib-thib ob ntawm qhov xav tau yog xam raws li cov qauv nqi dav dav , uas yog, nws nyob ntawm hom node thiab tus naj npawb ntawm cov nodes hauv koj pawg.
Launching Parallel Scaling
Parallel scaling yog ua rau txhua WLM kab. Mus rau AWS Redshift console thiab xaiv Workload Management los ntawm cov ntawv qhia zaub mov sab laug. Xaiv koj pawg pawg WLM parameter los ntawm cov ntawv qhia zaub mov nram qab no.
Koj yuav pom kab tshiab hu ua "Concurrency Scaling Mode" ntawm ib sab ntawm txhua kab. Lub neej ntawd yog "Disabled". Nyem "Kho kom raug" thiab koj tuaj yeem hloov qhov chaw rau txhua kab.
Configuration
Parallel scaling ua hauj lwm los ntawm kev xa cov lus thov tsim nyog rau cov pab pawg tshiab. Cov pawg tshiab muaj qhov loj tib yam (hom thiab tus naj npawb ntawm cov nodes) raws li pawg loj.
Lub neej ntawd tus naj npawb ntawm pawg siv rau qhov sib npaug scaling yog ib (1), muaj peev xwm teeb tsa mus txog rau tag nrho kaum (10) pawg.
Tag nrho cov naj npawb ntawm pawg rau kev ntsuas qhov sib npaug tuaj yeem raug teeb tsa los ntawm max_concurrency_scaling_clusters parameter. Kev nce tus nqi ntawm qhov parameter no muab cov kab rov ua dua ntxiv.
Saib xyuas
Muaj ntau cov duab ntxiv muaj nyob hauv AWS Redshift console. Daim ntawv qhia Max Configured Concurrency Scaling Clusters qhia tus nqi ntawm max_concurrency_scaling_clusters dhau sijhawm.
Tus naj npawb ntawm cov scaling active pawg yog tso tawm nyob rau hauv tus neeg siv interface nyob rau hauv lub "Concurrency Scaling Activity" seem:
Hauv Cov Lus Nug tab, muaj ib kab ntawv qhia seb cov lus nug tau ua tiav hauv pawg loj lossis hauv pawg sib npaug sib npaug:
Txawm hais tias cov lus nug tshwj xeeb tau ua tiav hauv pawg tseem ceeb lossis los ntawm kev sib txuas ua ke scaling pawg, nws yog khaws cia hauv stl_query.concurrency_scaling_status.
Tus nqi ntawm 1 qhia tau hais tias cov lus nug raug tua nyob rau hauv parallel scale pawg, thaum lwm qhov tseem ceeb qhia tau hais tias nws tau raug tua nyob rau hauv thawj pawg.
Piv Txwv:
Cov ntaub ntawv sib npaug sib npaug kuj tseem khaws cia rau hauv qee lub rooj thiab kev pom, xws li SVCS_CONCURRENCY_SCALING_USAGE. Tsis tas li ntawd, muaj ntau lub rooj catalog uas khaws cov ntaub ntawv hais txog kev sib npaug.
Π Π΅Π·ΡΠ»ΡΡΠ°ΡΡ
Cov kws sau ntawv tau pib qhov sib npaug sib npaug rau ib kab hauv pawg sab hauv ntawm kwv yees li 18:30:00 GMT ntawm 29.03.2019/3/20. Hloov cov max_concurrency_scaling_clusters parameter rau 30 ntawm kwv yees li 00:29.03.2019:XNUMX ntawm XNUMX/XNUMX/XNUMX.
Txhawm rau simulate ib daim ntawv thov, peb txo tus naj npawb ntawm qhov qhib rau qhov kab no los ntawm 15 mus rau 5.
Hauv qab no yog ib daim ntawv qhia intermix.io dashboard qhia tus naj npawb ntawm kev thov khiav thiab queuing tom qab txo tus naj npawb ntawm cov slots.
Peb pom tias lub sijhawm tos rau kev thov hauv kab tau nce, nrog rau lub sijhawm siab tshaj plaws yog ntau tshaj 5 feeb.
Nov yog cov ntaub ntawv cuam tshuam los ntawm AWS console txog dab tsi tshwm sim thaum lub sijhawm no:
Redshift launched peb (3) parallel scaling clusters raws li configured. Nws zoo nkaus li tias cov pawg no tau siv tsis tau, txawm tias muaj ntau qhov kev thov hauv peb pawg tau teem caij.
Kev siv daim duab cuam tshuam nrog qhov ntsuas qhov ntsuas qhov ntsuas:
Tom qab ob peb teev, cov kws sau ntawv tau tshuaj xyuas cov kab thiab nws zoo li 6 qhov kev thov tau khiav ntawm qhov sib npaug. Peb kuj random sim ob qhov kev thov los ntawm tus neeg siv interface. Peb tsis tau kuaj xyuas seb yuav siv cov txiaj ntsig no li cas thaum ob peb pawg sib txuas ua haujlwm ib zaug.
tshawb pom
Parallel scaling tuaj yeem txo cov sij hawm thov siv nyob rau hauv kab thaum lub sij hawm siab tshaj.
Raws li cov txiaj ntsig ntawm qhov kev sim yooj yim, nws tau pom tias qhov xwm txheej nrog kev thov kev thauj khoom tau txhim kho ib nrab. Txawm li cas los xij, qhov sib npaug ntawm qhov ntsuas ib leeg tsis daws txhua qhov teeb meem sib txuam.
Qhov no yog vim muaj kev txwv ntawm hom lus nug uas tuaj yeem siv qhov sib npaug sib npaug. Piv txwv li, cov neeg sau ntawv muaj ntau lub rooj nrog cov yuam sij sib cuam tshuam, thiab feem ntau ntawm peb cov haujlwm yog sau.
Txawm hais tias qhov sib npaug sib npaug tsis yog qhov kev daws teeb meem thoob ntiaj teb rau kev teeb tsa WLM, siv cov yam ntxwv no yooj yim thiab ncaj nraim.
Yog li ntawd, tus sau pom zoo siv nws rau koj WLM queues. Pib nrog ib pawg sib npaug thiab saib xyuas qhov siab tshaj plaws los ntawm lub console los txiav txim seb cov pawg tshiab puas tau siv tag nrho.
Raws li AWS ntxiv kev txhawb nqa rau cov lus nug ntxiv thiab cov rooj, kev sib txuas ua ke yuav tsum maj mam ua kom zoo dua.
Hais lus los ntawm Daniyar Belkhodzhaev, Skyeng Data Engineer
Peb ntawm Skyeng kuj tau pom tam sim ntawd qhov tshwm sim muaj peev xwm ntawm kev sib npaug sib npaug.
Cov kev ua haujlwm yog qhov txaus nyiam heev, tshwj xeeb tshaj yog xav tias AWS kwv yees tias cov neeg siv feem ntau yuav tsis tau them nyiaj ntxiv rau nws.Nws thiaj li tshwm sim tias nyob rau nruab nrab Lub Plaub Hlis peb muaj qhov txawv txav ntawm kev thov rau Redshift pawg. Nyob rau lub sijhawm no, peb feem ntau tau mus rau Concurrency Scaling; qee zaum ib pawg ntxiv tau ua haujlwm 24 teev hauv ib hnub yam tsis tau nres.
Qhov no ua rau nws ua tau, yog tias tsis daws qhov teeb meem nrog cov kab, ces tsawg kawg yog ua kom qhov teeb meem tau txais.
Peb cov kev soj ntsuam feem ntau coincide nrog cov impressions ntawm cov txiv neej los ntawm intermix.io.
Peb kuj tau pom tias txawm hais tias muaj kev thov tos hauv cov kab, tsis yog txhua qhov kev thov raug xa mus tam sim ntawd rau pawg sib npaug. Pom tau tias qhov no tshwm sim vim hais tias cov pawg sib npaug tseem siv sijhawm los pib. Yog li ntawd, thaum lub sij hawm luv luv lub ncov loads peb tseem muaj me me queues, thiab cov coj lub tswb muaj sij hawm los ua.
Tau tshem ntawm qhov tsis xws luag hauv lub Plaub Hlis, peb, raws li AWS xav tau, nkag mus rau qee lub sijhawm siv hom - nyob rau hauv cov qauv dawb.
Koj tuaj yeem taug qab koj cov nqi sib npaug sib npaug hauv AWS Cost Explorer. Koj yuav tsum xaiv Service - Redshift, Usage Type - CS, piv txwv li USW2-CS:dc2.large.Koj tuaj yeem nyeem ntxiv txog cov nqi hauv Lavxias
Tau qhov twg los: www.hab.com