I-Amazon Redshift Parallel Scaling Guide kanye nemiphumela yokuhlola

Kwa-Skyeng sisebenzisa i-Amazon Redshift, okuhlanganisa ukukala okufanayo, ngakho-ke sithole lesi sihloko sikaStefan Gromoll, umsunguli we-dotgo.com, se-intermix.io sithakazelisayo. Ngemva kokuhumusha, okuncane kokuhlangenwe nakho kwethu okuvela kunjiniyela wedatha u-Daniyar Belkhodzhaev.

I-Amazon Redshift Architecture ivumela ukukala ngokungeza amanodi amasha kuqoqo. Isidingo sokubhekana nenani eliphakeme lezicelo singaholela ekuhlinzekeni ngokweqile kwama-node. I-Concurrency Scaling, ngokuphambene nokwengeza ama-node amasha, kwandisa amandla ekhompiyutha njengoba kudingeka.

Ukukala okuhambisanayo kwe-Amazon Redshift kunikeza amaqoqo e-Redshift umthamo owengeziwe wokusingatha amavolumu ezicelo eziphezulu. Isebenza ngokuhambisa izicelo kumaqoqo amasha "ahambisanayo" ngemuva. Izicelo zihanjiswa ngokususelwe ekucushweni kwe-WLM nemithetho.

Intengo yokukala ehambisanayo isuselwe kumodeli yekhredithi enesigaba samahhala. Ngaphezu kwamakhredithi amahhala, inkokhelo isekelwe esikhathini i-Parallel Scaling Cluster icubungula izicelo.

Umbhali uhlole ukukala okuhambisanayo kwelinye lamaqoqo angaphakathi. Kulokhu okuthunyelwe, uzokhuluma ngemiphumela yokuhlolwa futhi anikeze namathiphu okuthi ungaqala kanjani.

Izidingo zeqoqo

Ukuze usebenzise ukukala okufanayo, iqoqo lakho le-Amazon Redshift kufanele lihlangabezane nalezi zidingo ezilandelayo:

- inkundla: I-EC2-VPC;
- uhlobo lwe-node: dc2.8xlarge, ds2.8xlarge, dc2.large noma ds2.xlarge;
- inombolo yama-node: ukusuka ku-2 ukuya ku-32 (amaqoqo e-node eyodwa awasekelwe).

Izinhlobo zesicelo ezamukelekile

Ukukala okuhambisanayo akuzifanele zonke izinhlobo zemibuzo. Enguqulweni yokuqala, icubungula kuphela izicelo ezifundwayo ezenelisa izimo ezintathu:

— KHETHA imibuzo ifundwa kuphela (yize kuhlelwa ezinye izinhlobo);
— umbuzo awubhekiseli etafuleni elinesitayela sokuhlunga INTERLEAVED;
- Umbuzo awusebenzisi i-Amazon Redshift Spectrum ukubhekisela kumathebula angaphandle.

Ukuze sihanjiswe ku-Parallel Scaling Cluster, isicelo kufanele sifakwe kulayini. Ukwengeza, imibuzo iyafaneleka kulayini I-SQA (Ukusheshisa Kombuzo Omfushane), ngeke isebenze ngamaqoqo esikali afanayo.

Ulayini kanye ne-SQA kudinga ukucushwa okufanele I-Redshift Workload Management (WLM). Sincoma ukuthi uthuthukise i-WLM yakho kuqala - lokhu kuzonciphisa isidingo sokukala okuhambisanayo. Futhi lokhu kubalulekile ngoba ukukala okuhambisanayo kumahhala ngenani elithile lamahora. I-AWS ithi ukukala okuhambisanayo kuzoba mahhala ku-97% wamakhasimende, okusiletha odabeni lwamanani entengo.

Izindleko zokukala okufanayo

I-AWS inikeza imodeli yekhredithi yokukala okufanayo. Iqoqo ngalinye elisebenzayo I-Amazon Redshift Iqongelela amakhredithi njalo ngehora, kufika ehoreni elilodwa lamakhredithi okukala ahambisanayo amahhala ngosuku.

Ukhokha kuphela uma ukusetshenziswa kwakho kwe-Parallel Scaling Clusters kudlula inani lamakhredithi owatholile.

Izindleko zibalwa ngenani lesekhondi ngalinye elifunwayo leqoqo elihambisanayo elisetshenziswa ngaphezu kwenani lamahhala. Ukhokhiswa kuphela isikhathi sobude bezicelo zakho, nenkokhelo encane yomzuzu owodwa isikhathi ngasinye lapho i-Parallel Scaling Cluster icushwa. Izinga lentengo lesekhondi ngalinye libalwa ngokusekelwe ezimisweni zentengo ezijwayelekile I-Amazon Redshift, okusho ukuthi, kuya ohlotsheni lwenodi kanye nenani lamanodi kuqoqo lakho.

Kwethulwa I-Parallel Scaling

Ukukala okuhambisanayo kuyaqaliswa kulayini ngamunye we-WLM. Iya kukhonsoli ye-AWS Redshift bese ukhetha Ukuphathwa Komthwalo Womsebenzi kumenyu engakwesokunxele yokuzulazula. Khetha iqembu lepharamitha yeqoqo lakho le-WLM kumenyu yokudonsela phansi elandelayo.

Uzobona ikholomu entsha ebizwa nge-"Concurrency Scaling Mode" eduze komugqa ngamunye. Okuzenzakalelayo "Kukhutshaziwe". Chofoza "Hlela" futhi ungakwazi ukushintsha izilungiselelo zomugqa ngamunye.

Ukucushwa

Ukukala okuhambisanayo kusebenza ngokudlulisela izicelo ezifanele kumaqoqo amasha azinikezele. Amaqoqo amasha anosayizi ofanayo (uhlobo nenombolo yamanodi) njengeqoqo elikhulu.

Inombolo ezenzakalelayo yamaqoqo asetshenziselwa ukukala okuhambisanayo yinye (1), enekhono lokumisa ingqikithi yamaqoqo ayishumi (10).
Isamba senani lamaqoqo esikali esihambisanayo singasethwa yipharamitha ye-max_concurrency_scaling_clusters. Ukwenyusa inani lale pharamitha kunikeza amaqoqo engeziwe angasebenzi.

Ukuqapha

Kunamagrafu amaningana angeziwe atholakala kukhonsoli ye-AWS Redshift. Ishadi Le-Max Configured Concurrency Scaling Clusters libonisa inani lama-max_concurrency_scaling_clusters ngokuhamba kwesikhathi.

Inombolo yamaqoqo esikali asebenzayo iboniswa kusixhumi esibonakalayo somsebenzisi esigabeni esithi “Umsebenzi Wokukala Ngokuvumelana Kwemali”:

Kuthebhu Yemibuzo, kukhona ikholomu ebonisa ukuthi ingabe umbuzo usetshenziswe kuqoqo eliyinhloko noma kuqoqo lesikali elihambisanayo:

Kungakhathaliseki ukuthi umbuzo othile wenziwa kuqoqo elikhulu noma ngeqoqo lesikali elihambisanayo, ugcinwa kokuthi stl_query.concurrency_scaling_status.

Inani elingu-1 libonisa ukuthi umbuzo usetshenziswe kuqoqo lesikali esilinganayo, kuyilapho amanye amanani ebonisa ukuthi wasetshenziswa kuqoqo eliyinhloko.

Isibonelo:

Ulwazi lokukala lokuvumelana ngemali luphinde lugcinwe kwamanye amathebula nokubukwa, okufana ne-SVCS_CONCURRENCY_SCALING_USAGE. Ngaphezu kwalokho, kunenani lamathebula ekhathalogi agcina ulwazi mayelana nokukala okuhambisanayo.

Imiphumela

Ababhali baqale ukukala okuhambisanayo komugqa owodwa kuqoqo langaphakathi cishe ngo-18:30:00 GMT ngomhla ka-29.03.2019/3/20. Kushintshwe ipharamitha ye-max_concurrency_scaling_clusters yaba ngu-30 cishe ngo-00:29.03.2019:XNUMX ngo-XNUMX/XNUMX/XNUMX.

Ukuze silingise ulayini wesicelo, sehlise inani lezikhala zalo mugqa ukusuka ku-15 ukuya ku-5.

Ngezansi kuneshadi ledeshibhodi le-intermix.io elibonisa inani lezicelo ezisebenzayo futhi ezikulayini ngemva kokunciphisa inani lezikhala.

Siyabona ukuthi isikhathi sokulinda sezicelo kulayini sinyukile, kanti isikhathi esiphezulu singaphezu kwamaminithi angu-5.

Nalu ulwazi olufanele oluvela kukhonsoli ye-AWS mayelana nokwenzeka ngalesi sikhathi:

I-Redshift yethule amaqoqo okukala okulinganayo amathathu (3) njengoba amisiwe. Kubonakala sengathi lawa maqoqo asetshenziswe kancane, nakuba izicelo eziningi kuqoqo lethu bezikulayini.

Igrafu yokusetshenziswa ihlobana negrafu yomsebenzi wokukala:

Ngemuva kwamahora ambalwa, ababhali bahlole ulayini futhi kubukeka sengathi izicelo eziyisi-6 bezisebenza ngokuhambisanayo. Siphinde sahlola izicelo ezimbili ngokungahleliwe ngokusebenzisa isixhumi esibonakalayo somsebenzisi. Asihlolanga ukuthi asetshenziswa kanjani lawa manani uma amaqoqo amaningana afanayo esebenza ngesikhathi esisodwa.

okutholakele

Ukukala okuhambisanayo kunganciphisa isikhathi esichithwa izicelo kulayini phakathi nomthwalo omkhulu.

Ngokusekelwe emiphumeleni yokuhlolwa okuyisisekelo, kuvele ukuthi isimo esinezicelo zokulayisha siye sathuthuka kancane. Kodwa-ke, ukukala okuhambisanayo kukodwa akuzange kuzixazulule zonke izinkinga ze-concurrency.

Lokhu kungenxa yemikhawulo ezinhlotsheni zemibuzo ezingasebenzisa ukukala okuhambisanayo. Isibonelo, ababhali banamathebula amaningi anokhiye bokuhlunga abahlukanisiwe, futhi umsebenzi wethu omningi uyabhala.

Nakuba ukukala okuhambisanayo akusona isixazululo sasemhlabeni wonke sokusetha i-WLM, ukusebenzisa lesi sici kulula futhi kuqondile.

Ngakho-ke, umbhali uncoma ukuthi uyisebenzise kolayini bakho be-WLM. Qala ngeqoqo elilodwa elihambisanayo bese uqapha umthamo omkhulu ngokusebenzisa ikhonsoli ukuze unqume ukuthi amaqoqo amasha asetshenziswa ngokugcwele yini.

Njengoba i-AWS yengeza ukusekelwa kwezinhlobo zemibuzo eyengeziwe namathebula, ukukala okuhambisanayo kufanele kancane kancane kusebenze kahle kakhulu.

Ukuphawula okuvela ku-Daniyar Belkhodzhaev, Skyeng Data Engineer

Thina kwa-Skyeng siphinde sabona ngokushesha ithuba elivelayo lokukala okufanayo.
Ukusebenza kukhanga kakhulu, ikakhulukazi uma kucatshangelwa ukuthi i-AWS ilinganisela ukuthi abasebenzisi abaningi ngeke baze bakhokhe imali eyengeziwe ngakho.

Kwenzeka ukuthi maphakathi no-April saba nokushayisana okungavamile kwezicelo kuqoqo le-Redshift. Ngalesi sikhathi, sasivame ukuphendukela ku-Concurrency Scaling; kwesinye isikhathi iqoqo elengeziwe lalisebenza amahora angama-24 ngosuku ngaphandle kokuma.

Lokhu kwenza kube nokwenzeka, uma kungenjalo ukuxazulula ngokuphelele inkinga ngemigqa, okungenani ukwenza isimo samukeleke.

Esikuqaphelayo kakhulu kuhambisana nokuvela kwabafana abavela ku-intermix.io.

Siphinde saqaphela ukuthi nakuba bekunezicelo ezilindile kulayini, akuzona zonke izicelo ezisheshe zathunyelwa ku-parallel cluster. Ngokusobala lokhu kwenzeka ngoba iqoqo elihambisanayo lisathatha isikhathi ukuqala. Ngenxa yalokho, phakathi nemithwalo ephakeme yesikhathi esifushane sisenolayini abancane, futhi ama-alamu ahambisanayo anesikhathi sokuqalisa.

Ngemva kokulahla imithwalo engavamile ngo-April, thina, njengoba i-AWS yayilindele, sangena kumodi yokusebenzisa ngezikhathi ezithile - ngaphakathi kwenkambiso yamahhala.
Ungakwazi ukulandelela izindleko zakho zokukala okufanayo ku-AWS Cost Explorer. Udinga ukukhetha Isevisi - I-Redshift, Uhlobo Lokusetshenziswa - CS, isibonelo USW2-CS:dc2.large.

Ungafunda kabanzi mayelana namanani ngesiRashiya lapha.

Source: www.habr.com