Zopereka za Yandex pazosungidwa zotsatirazi zidzawunikidwanso.
- Dinani Nyumba
- Odyssey
- Kuchira pakanthawi kochepa (WAL-G)
- PostgreSQL (kuphatikiza ma loger, Amcheck, heapcheck)
- Greenplum
Video:
Moni Dziko Lapansi! Dzina langa ndine Andrey Borodin. Ndipo zomwe ndimachita ku Yandex.Cloud ndikukhazikitsa nkhokwe zotseguka zaubale mokomera makasitomala a Yandex.Cloud ndi Yandex.Cloud.
Munkhani iyi, tikambirana za zovuta zomwe ma database otseguka amakumana nawo. N’chifukwa chiyani kuli kofunikira? Chifukwa mavuto ang'onoang'ono omwe, monga udzudzu, amakhala njovu. Zimakhala zazikulu mukakhala ndi masango ambiri.
Koma sindicho chinthu chachikulu. Zinthu zosaneneka zimachitika. Zinthu zomwe zimachitika m'modzi mwamilandu miliyoni. Ndipo m'malo amtambo, muyenera kukonzekera izi, chifukwa zinthu zodabwitsa zimakhala zotheka kwambiri ngati china chake chilipo.
Koma! Ubwino wa malo otsegula ndi chiyani? Chowonadi ndi chakuti muli ndi mwayi wongoganiza kuti muthane ndi vuto lililonse. Muli ndi code source, muli ndi chidziwitso cha pulogalamu. Timagwirizanitsa ndikugwira ntchito.
Kodi pali njira zotani zogwirira ntchito pa pulogalamu yotseguka?
- Njira yowongoka kwambiri ndiyo kugwiritsa ntchito mapulogalamu. Ngati mugwiritsa ntchito ma protocol, ngati mugwiritsa ntchito miyezo, ngati mugwiritsa ntchito mawonekedwe, ngati mulemba mafunso mu pulogalamu yotseguka, ndiye kuti mumathandizira kale.
- Mukukulitsa chilengedwe chake. Mumakulitsa mwayi wozindikira cholakwikacho msanga. Mumawonjezera kudalirika kwa dongosolo lino. Mumawonjezera kupezeka kwa opanga pamsika. Mukuwongolera pulogalamuyo. Ndinu othandizira kale ngati mwangopanga masitayilo ndikusintha china chake pamenepo.
- Njira ina yomveka ndikuthandizira pulogalamu yotseguka. Mwachitsanzo, pulogalamu yodziwika bwino ya Google Summer of Code, pomwe Google imalipira ophunzira ambiri ochokera padziko lonse lapansi ndalama zomveka kuti apange mapulogalamu otseguka omwe amakwaniritsa zofunikira zina zamalayisensi.
- Iyi ndi njira yosangalatsa kwambiri chifukwa imalola kuti pulogalamuyo isinthe popanda kusuntha kuyang'ana kutali ndi anthu ammudzi. Google, monga chimphona chaukadaulo, sichikunena kuti tikufuna izi, tikufuna kukonza cholakwika ichi ndipo apa ndipamene tiyenera kukumba. Google imati: "Chitani zomwe mumachita. Ingogwirani ntchito momwe mwakhala mukuchitira ndipo zonse zikhala bwino. "
- Njira yotsatila yotenga nawo mbali poyera ndi kutenga nawo mbali. Mukakhala ndi vuto mu pulogalamu yotseguka ndipo pali opanga, opanga anu amayamba kuthetsa mavutowo. Amayamba kupangitsa kuti zomangamanga zanu zikhale zogwira mtima, mapulogalamu anu mwachangu komanso odalirika.
Chimodzi mwazinthu zodziwika bwino za Yandex pamapulogalamu otseguka ndi ClickHouse. Iyi ndi database yomwe idabadwa ngati yankho ku zovuta zomwe Yandex.Metrica ikukumana nazo.
Ndipo monga nkhokwe, idapangidwa poyera kuti apange chilengedwe ndikuchikulitsa pamodzi ndi opanga ena (osati mkati mwa Yandex okha). Ndipo tsopano iyi ndi pulojekiti yayikulu yomwe makampani ambiri osiyanasiyana akukhudzidwa.
Mu Yandex.Cloud, tidapanga ClickHouse pamwamba pa Yandex Object Storage, i.e. pamwamba pa kusungirako mitambo.
N’chifukwa chiyani zimenezi zili zofunika mumtambo? Chifukwa nkhokwe iliyonse imagwira ntchito pamakona atatu awa, mu piramidi iyi, muulamuliro wamitundu yamakumbukiro. Muli ndi zolembera zofulumira koma zazing'ono komanso ma SSD otsika mtengo koma ocheperako, ma hard drive ndi zida zina zama block. Ndipo ngati mukuchita bwino pamwamba pa piramidi, ndiye kuti muli ndi database yachangu. ngati mukuchita bwino pansi pa piramidi iyi, ndiye kuti muli ndi database ya scaled. Ndipo pankhaniyi, kuwonjezera wosanjikiza wina kuchokera pansipa ndi njira yomveka yowonjezerera scalability ya database.
Kodi zikanatheka bwanji? Iyi ndi mfundo yofunika kwambiri mu lipotili.
- Titha kugwiritsa ntchito ClickHouse pa MDS. MDS ndi mawonekedwe osungiramo mtambo a Yandex. Ndizovuta kwambiri kuposa protocol wamba ya S3, koma ndiyoyenera kwambiri pachida cha block. Ndi bwino kujambula deta. Pamafunika zambiri mapulogalamu. Opanga mapulogalamu adzapanga, ndizabwino, ndizosangalatsa.
- S3 ndi njira yodziwika bwino yomwe imapangitsa mawonekedwe kukhala osavuta pamtengo wocheperako kumitundu ina yantchito.
Mwachibadwa, pofuna kupereka magwiridwe antchito ku chilengedwe chonse cha ClickHouse ndikuchita ntchito yomwe ikufunika mkati mwa Yandex.Cloud, tinaganiza zowonetsetsa kuti gulu lonse la ClickHouse lidzapindula nalo. Tidakhazikitsa ClickHouse pa S3, osati ClickHouse pa MDS. Ndipo iyi ndi ntchito yambiri.
Zolemba:
"Filesystem abstraction layer"
"Kuphatikiza kwa AWS SDK S3"
"Base kukhazikitsa IDisk interafce kwa S3"
"Kuphatikizika kwa injini zosungira logi ndi mawonekedwe a IDisk"
"Log injini yothandizira S3 ndi SeekableReadBuffer"
"Story Stripe Log S3 thandizo"
"Story MergeTree chithandizo choyambirira cha S3"
"MergeTree thandizo lonse la S3"
"Thandizani ReplicatedMergeTree pa S3"
"Onjezani zidziwitso zosasinthika ndi mitu yosungiramo s3"
"S3 yokhala ndi masinthidwe amphamvu a proxy"
"S3 yokhala ndi proxy resolution"
Uwu ndi mndandanda wa zopempha zokoka kuti mugwiritse ntchito mawonekedwe a fayilo ku ClickHouse. Ichi ndi chiwerengero chachikulu cha zopempha kukoka.
Zolemba:
"DiskS3 hardlinks kukhazikitsa mulingo woyenera"
"S3 HTTP kasitomala - Pewani kukopera mayankho pamakumbukiro"
"Pewani kukopera mayankho athunthu kukumbukira mu S3 HTTP
kasitomala"
"Kutha kuyika chizindikiro ndi index mafayilo a S3 disk"
"Sungani magawo kuchokera ku DiskLocal kupita ku DiskS3 mofanana"
Koma ntchitoyi sinathere pamenepo. Pambuyo popangidwa, ntchito ina idafunikira kuti izi zitheke.
Zolemba:
"Onjezani SelectedRows ndi zochitika za SelectedBytes"
"Onjezani zochitika za mbiri kuchokera ku pempho la S3 kupita ku system.events"
"Onjezani QueryTimeMicroseconds, SelectQueryTimeMicroseconds ndi InsertQueryTimeMicroseconds"
Ndiyeno kunali kofunikira kuti zidziwike, kukhazikitsa kuyang'anira ndikuzipangitsa kuti zitheke.
Ndipo zonsezi zidachitidwa kuti gulu lonse, chilengedwe chonse cha ClickHouse, chilandire zotsatira za ntchitoyi.
Tiyeni tipitirire ku nkhokwe zamalonda, ku OLTP databases, zomwe zili pafupi ndi ine ndekha.
Ili ndiye gawo lotseguka lachitukuko la DBMS. Anyamatawa akuchita zamatsenga mumsewu kuti apititse patsogolo ma database otseguka.
Imodzi mwa mapulojekiti, pogwiritsa ntchito chitsanzo chomwe tingalankhule za momwe tingachitire ndi zomwe timachita, ndi Connection Pooler ku Postgres.
Postgres ndi njira yosungiramo zinthu. Izi zikutanthauza kuti database iyenera kukhala ndi ma network ochepa momwe angathere omwe amasamalira zochitika.
Kumbali ina, m'malo amtambo, zomwe zimachitika nthawi zambiri zimakhala pamene maulumikizidwe chikwi amabwera pagulu limodzi nthawi imodzi. Ndipo ntchito yolumikizirana ndi kunyamula maulumikizidwe zikwizikwi pamalumikizidwe ang'onoang'ono a seva.
Titha kunena kuti cholumikizira cholumikizira ndi wogwiritsa ntchito foni yemwe amakonzanso ma byte kuti afikire bwino database.
Tsoka ilo, palibe mawu abwino a Chirasha oti agwirizane. Nthawi zina amatchedwa multiplexer kugwirizana. Ngati mukudziwa chomwe mungatchule cholumikizira cholumikizira, onetsetsani kuti mwandiuza, ndikhala wokondwa kulankhula chilankhulo chaukadaulo cha Chirasha.
Tidafufuza zolumikizira zomwe zinali zoyenera pagulu loyang'aniridwa la postgres. Ndipo PgBouncer inali chisankho chabwino kwambiri kwa ife. Koma tidakumana ndi zovuta zingapo ndi PgBouncer. Zaka zambiri zapitazo, Volodya Borodin anapereka malipoti kuti timagwiritsa ntchito PgBouncer, timakonda chirichonse, koma pali ma nuances, pali chinachake choti tigwiritse ntchito.
Ndipo tinagwira ntchito. Tinakonza zovuta zomwe tidakumana nazo, tidayika Bouncer, ndikuyesera kukankhira zopempha kumtunda. Koma ulusi umodzi wofunikira unali wovuta kugwira nawo ntchito.
Tidayenera kutolera ma cascades kuchokera ku ma Bouncer okhala ndi zigamba. Tikakhala ndi ma Bouncer ambiri okhala ndi ulusi umodzi, zolumikizira pamwamba zimasamutsidwa kupita kugawo lamkati la Bouncers. Iyi ndi dongosolo losasamalidwa bwino lomwe ndi lovuta kupanga ndikukweza mmbuyo ndi mtsogolo.
Tinafika pamapeto kuti tidapanga cholumikizira chathu, chomwe chimatchedwa Odyssey. Tinalemba izo kuchokera pachiyambi.
Mu 2019, pamsonkhano wa PgCon, ndidapereka chowerengera ichi kwa otukula. Tsopano tili ndi nyenyezi zosachepera 2 pa GitHub, mwachitsanzo, polojekitiyi ndi yamoyo, polojekitiyi ndi yotchuka.
Ndipo ngati mupanga gulu la Postgres mu Yandex.Cloud, ndiye kuti lidzakhala gulu lomwe lili ndi Odyssey yomangidwa, yomwe imakonzedwanso pakukweza gululo kumbuyo kapena kutsogolo.
Kodi taphunzirapo chiyani pa ntchitoyi? Kuyambitsa pulojekiti yopikisana nthawi zonse ndi sitepe yaukali, ndi njira yowonjezereka pamene tikunena kuti pali mavuto omwe sakuthetsedwa mwamsanga, omwe sakuthetsedwa mu nthawi zomwe zingatigwirizane ndi ife. Koma uwu ndi muyeso wogwira mtima.
PgBouncer idayamba kukula mwachangu.
Ndipo tsopano ntchito zina zawonekera. Mwachitsanzo, pgagroal, yomwe imapangidwa ndi opanga Red Hat. Amatsata zolinga zofanana ndikugwiritsa ntchito malingaliro ofanana, koma, ndithudi, ndi enieni awo, omwe ali pafupi ndi opanga pgagroal.
Nkhani ina yogwira ntchito ndi gulu la postgres ikubwereranso pakapita nthawi. Uku ndikuchira pambuyo pakulephera, uku ndikuchira kuchokera ku zosunga zobwezeretsera.
Pali zosunga zobwezeretsera zambiri ndipo zonse ndizosiyana. Pafupifupi wogulitsa aliyense wa Postgres ali ndi yankho lake losunga zobwezeretsera.
Ngati mutenga machitidwe onse osunga zobwezeretsera, pangani mawonekedwe amtundu ndikuwerengera moseketsa chotsimikizira mu matrix awa, chikhala ziro. Kodi izi zikutanthauza chiyani? Bwanji ngati mutenga fayilo yosunga zobwezeretsera, ndiye kuti singasonkhanitsidwe kuchokera ku zidutswa za ena onse. Ndilopadera pakukhazikitsidwa kwake, ndilopadera mu cholinga chake, ndi lapadera mu malingaliro omwe amaikidwa mmenemo. Ndipo onse ndi achindunji.
Pamene tikugwira ntchitoyi, CitusData inayambitsa ntchito ya WAL-G. Izi ndi zosunga zobwezeretsera zomwe zidapangidwa ndi diso ku chilengedwe chamtambo. Tsopano CitusData ili kale gawo la Microsoft. Ndipo panthawiyo, tidakonda kwambiri malingaliro omwe adayikidwa muzotulutsa zoyambirira za WAL-G. Ndipo tinayamba kuthandizira pa ntchitoyi.
Tsopano pali ambiri omwe akupanga pulojekitiyi, koma omwe amathandizira 10 ku WAL-G akuphatikizapo 6 Yandexoids. Tinabweretsa malingaliro athu ambiri kumeneko. Ndipo, zowona, tidazikhazikitsa tokha, tidaziyesa tokha, tidazipanga tokha, timazigwiritsa ntchito tokha, timadziwa komwe tingapite, ndikulumikizana ndi gulu lalikulu la WAL-G.
Ndipo m'malingaliro athu, tsopano dongosolo losunga zobwezeretserali, kuphatikizaponso kuyesayesa kwathu, lakhala loyenera ku chilengedwe chamtambo. Uwu ndiye mtengo wabwino kwambiri wothandizira Postgres mumtambo.
Zikutanthauza chiyani? Tinkalimbikitsa lingaliro lalikulu: zosunga zobwezeretsera ziyenera kukhala zotetezeka, zotsika mtengo kuti zigwiritsidwe ntchito komanso kuti zibwezeretse mwachangu momwe zingathere.
Chifukwa chiyani kuyenera kukhala kotsika mtengo? Pamene palibe chosweka, simuyenera kudziwa kuti muli ndi zosunga zobwezeretsera. Chilichonse chimagwira ntchito bwino, mumawononga CPU yaying'ono momwe mungathere, mumagwiritsa ntchito zochepa za disk yanu momwe mungathere, ndipo mumatumiza ma byte ochepa pa intaneti momwe mungathere kuti musasokoneze malipiro a ntchito zanu zamtengo wapatali.
Ndipo chilichonse chikasweka, mwachitsanzo, woyang'anira adagwetsa deta, china chake chalakwika, ndipo muyenera kubwereranso zakale, mumachira ndi ndalama zonse, chifukwa mukufuna kuti deta yanu ibwererenso mwachangu.
Ndipo tinalimbikitsa lingaliro losavuta ili. Ndipo, zikuwoneka kwa ife, tinakwanitsa kuzikwaniritsa.
Koma si zokhazo. Tinkafuna chinthu chimodzi chaching'ono. Tinkafuna ma database osiyanasiyana. Osati makasitomala athu onse amagwiritsa ntchito Postgres. Anthu ena amagwiritsa ntchito MySQL, MongoDB. M'deralo, opanga ena athandizira FoundationDB. Ndipo mndandandawu ukukula mosalekeza.
Anthu ammudzi amakonda lingaliro la database ikuyendetsedwa pamalo oyendetsedwa mumtambo. Ndipo Madivelopa amasunga nkhokwe zawo, zomwe zitha kuthandizidwa chimodzimodzi pamodzi ndi Postgres ndi makina athu osunga zobwezeretsera.
Kodi taphunzirapo chiyani m’nkhaniyi? Zogulitsa zathu, monga gawo lachitukuko, si mizere ya code, si mawu, si mafayilo. Zogulitsa zathu sizimakoka zopempha. Awa ndi malingaliro omwe timapereka kwa anthu ammudzi. Uwu ndi ukatswiri waukadaulo komanso kusuntha kwaukadaulo kupita kumalo amtambo.
Pali database ngati Postgres. Ndimakonda kwambiri Postgres pachimake. Ndimathera nthawi yambiri ndikupanga maziko a Postgres ndi anthu ammudzi.
Koma apa ziyenera kunenedwa kuti Yandex.Cloud ili ndi kukhazikitsidwa kwamkati kwa database yoyendetsedwa. Ndipo idayamba kalekale mu Yandex.Mail. Ukatswiri womwe tsopano wapangitsa kuti Postgres atsogolere adasonkhanitsidwa pomwe makalata amafuna kusamukira ku Postgres.
Imelo ili ndi zofunikira zofanana kwambiri ndi mtambo. Zimafunikira kuti muzitha kukulitsa kukula kosayembekezereka nthawi iliyonse mu data yanu. Ndipo makalatawo anali ndi katundu kale ndi mazana mamiliyoni mamiliyoni a makalata ochuluka a ogwiritsa ntchito omwe nthawi zonse amapempha zambiri.
Ndipo izi zinali zovuta kwambiri kwa gulu lomwe limapanga Postgres. Kalelo, mavuto alionse amene tinkakumana nawo ankauzidwa kwa anthu ammudzi. Ndipo mavutowa adakonzedwa, ndikuwongoleredwa ndi anthu amdera lina m'malo ena ngakhale pamlingo wolipiridwa wothandizidwa ndi nkhokwe zina komanso bwino. Ndiko kuti, mutha kutumiza kalata ku PgSQL hacker ndikulandila yankho mkati mwa mphindi 40. Thandizo lolipidwa m'ma database ena lingaganize kuti pali zinthu zofunika kwambiri kuposa cholakwika chanu.
Tsopano kuyika kwamkati kwa Postgres ndi ma petabytes ena a data. Izi ndi mamiliyoni a zopempha pa sekondi imodzi. Awa ndi masauzande masauzande. Ndi yayikulu kwambiri.
Koma pali nuance. Sizikhala pamayendedwe apamwamba a netiweki, koma pa Hardware yosavuta. Ndipo pali malo oyesera makamaka zinthu zatsopano zosangalatsa.
Ndipo panthawi ina m'malo oyesera tidalandira uthenga wosonyeza kuti zosintha zamkati mwa ma index a database zidaphwanyidwa.
Kusasinthika ndi mtundu wina wa ubale womwe timayembekezera kukhala nawo nthawi zonse.
Mkhalidwe wovuta kwambiri kwa ife. Zimasonyeza kuti deta ina yatayika. Ndipo kutaya deta ndi chinthu choopsa kwambiri.
Lingaliro lambiri lomwe timatsatira pazosungidwa zoyendetsedwa ndilakuti ngakhale mutayesetsa, zidzakhala zovuta kutaya deta. Ngakhale mutawachotsa mwadala, mudzafunikirabe kunyalanyaza kusowa kwawo kwa nthawi yayitali. Chitetezo cha data ndi chipembedzo chomwe timatsatira mwachangu.
Ndipo apa pabuka mkhalidwe wosonyeza kuti pangakhale mkhalidwe umene sitingakonzekere. Ndipo tinayamba kukonzekera izi.
Chinthu choyamba chimene tinachita chinali kukwirira zipika kuchokera m’magulu masauzande amenewa. Tidapeza kuti ndi magulu ati omwe anali pa disks omwe ali ndi vuto la firmware omwe amataya zosintha zamasamba. Adalemba ma code onse a Postgres. Ndipo tidalemba mauthenga omwe akuwonetsa kuphwanya zosintha zamkati ndi code yomwe idapangidwa kuti izindikire chivundi cha data.
Chigawochi chinavomerezedwa ndi anthu onse popanda kukambirana zambiri, chifukwa muzochitika zilizonse zinali zoonekeratu kuti chinachake choipa chachitika ndipo chiyenera kuuzidwa kwa chipikacho.
Pambuyo pa izi, tidafika pomwe tili ndi zowunikira zomwe zimasanthula zipika. Ndipo ngati pali mauthenga okayikitsa, amadzutsa woyang'anira ntchitoyo, ndipo woyang'anira ntchitoyo amakonza.
Koma! Kusanthula zipika ndi ntchito yotsika mtengo pagulu limodzi komanso yokwera mtengo kwambiri pamagulu chikwi.
Tinalemba zowonjezera zotchedwa . Zimapanga maonekedwe a database momwe mungathere mosavuta ndikusankha ziwerengero pa zolakwika zakale. Ndipo ngati tifunika kudzutsa woyang'anira ntchito, ndiye kuti tidzapeza izi popanda kuyang'ana mafayilo a gigabyte, koma pochotsa ma byte angapo pa tebulo la hashi.
Kuwonjezedwa uku kwalandiridwa, mwachitsanzo, munkhokwe ya . Ngati mukufuna kugwiritsa ntchito, mutha kuyiyika nokha. Inde ndi gwero lotseguka.
Koma si zokhazo. Tidayamba kugwiritsa ntchito Amcheck, chowonjezera chomangidwa ndi anthu, kuti tipeze zolakwika zosasinthika m'ma index.
Ndipo tidapeza kuti ngati mugwiritsa ntchito pamlingo, pali nsikidzi. Tinayamba kuwakonza. Zokonza zathu zalandiridwa.
Tidazindikira kuti kukulitsa uku sikungathe kusanthula ma index a GiST & GIT. Tidawathandizira. Koma chithandizochi chikukambidwabe ndi anthu ammudzi, chifukwa ichi ndi ntchito yatsopano ndipo pali zambiri zambiri kumeneko.
Ndipo tidazindikiranso kuti poyang'ana zolemba za kuphwanya kwa mtsogoleri wobwereza, pa mbuye, zonse zimagwira ntchito bwino, koma pazofananira, pazotsatira, kufunafuna ziphuphu sizothandiza. Sikuti zosintha zonse zimafufuzidwa. Ndipo wosasintha wina anativutitsa kwambiri. Ndipo tidakhala chaka chimodzi ndi theka tikulumikizana ndi anthu ammudzi kuti tithandizire cheke chofananira.
Tinalemba kachidindo kamene kayenera kutsatira zonse zomwe zingatheke ... ndondomeko. Tinakambirana chigambachi kwa nthawi ndithu ndi Peter Gaghan wochokera ku Crunchy Data. Anayenera kusintha pang'ono mtengo wa B womwe ulipo ku Postgres kuti avomereze chigambachi. Iye analandiridwa. Ndipo tsopano kuyang'ana ma index pa replicas kwakhalanso kothandiza kuti muwone zophwanya zomwe tidakumana nazo. Ndiko kuti, izi ndi zophwanya zomwe zingayambitsidwe ndi zolakwika mu disk firmware, nsikidzi mu Postgres, nsikidzi mu Linux kernel, ndi zovuta za hardware. Mndandanda wambiri wamagwero amavuto omwe timakonzekera.
Koma pambali pa ma index, pali gawo ngati mulu, mwachitsanzo, malo omwe deta imasungidwa. Ndipo palibe zosintha zambiri zomwe zitha kufufuzidwa.
Tili ndi chowonjezera chotchedwa Heapcheck. Tinayamba kukulitsa. Ndipo mofananira, pamodzi ndi ife, kampani ya EnterpriseDB idayambanso kulemba gawo, lomwe adatcha Heapcheck mwanjira yomweyo. Kokha tidachitcha kuti PgHeapcheck, ndipo amangochitcha kuti Heapcheck. Iwo ali ndi ntchito zofanana, siginecha yosiyana pang'ono, koma ndi malingaliro omwewo. Iwo anakhazikitsa izo bwinoko pang'ono m'malo ena. Ndipo adaziyika poyera kale.
Ndipo tsopano tikupanga kukulitsa kwawo, chifukwa sikulinso kufalikira kwawo, koma kufalikira kwa anthu ammudzi. Ndipo m'tsogolomu, iyi ndi gawo la kernel yomwe idzaperekedwa kwa aliyense kuti adziwe za mavuto amtsogolo.
M'malo ena, tidafika pozindikira kuti tili ndi zolakwika m'machitidwe athu owunikira. Mwachitsanzo, dongosolo 1C. Mukamagwiritsa ntchito nkhokwe, Postgres nthawi zina amalembamo zomwe amatha kuwerenga, koma pg_dump sangathe kuwerenga.
Izi zimawoneka ngati katangale ku makina athu ozindikira zovuta. Ogwira ntchitoyo adadzutsidwa. Woyang'anira ntchitoyo adayang'ana zomwe zikuchitika. Patapita nthawi, kunabwera kasitomala n’kunena kuti ndili ndi vuto. Wantchitoyo anafotokoza vuto. Koma vuto lili mu Postgres pachimake.
Ndapeza zokambirana za gawoli. Ndipo adalemba kuti tidakumana ndi izi ndipo zidali zosasangalatsa, munthu amadzuka usiku kuti adziwe kuti ndi chani.
Anthu ammudzi adayankha, "O, tifunikadi kukonza."
Ndili ndi fanizo losavuta. Ngati mukuyenda mu nsapato yomwe ili ndi mchenga wa mchenga, ndiye kuti, kwenikweni, mukhoza kusuntha - palibe vuto. Ngati mumagulitsa nsapato kwa anthu zikwizikwi, ndiye tiyeni tipange nsapato zopanda mchenga konse. Ndipo ngati mmodzi wa ogwiritsira ntchito nsapato zanu adzathamanga marathon, ndiye kuti mukufuna kupanga nsapato zabwino kwambiri, ndikuzikweza kwa ogwiritsa ntchito anu onse. Ndipo ogwiritsa ntchito mosayembekezereka nthawi zonse amakhala mumtambo. Nthawi zonse pamakhala ogwiritsa ntchito omwe amagwiritsa ntchito gululo mwanjira ina yoyambirira. Muyenera kukonzekera izi nthawi zonse.
Taphunzirapo chiyani apa? Tinaphunzira chinthu chosavuta: chofunika kwambiri ndikufotokozera anthu ammudzi kuti pali vuto. Ngati anthu ammudzi azindikira vutoli, ndiye kuti mpikisano wachilengedwe umayamba kuti athetse vutoli. Chifukwa aliyense amafuna kuthetsa vuto lalikulu. Mavenda onse, obera onse amamvetsetsa kuti iwo eni amatha kupondaponda, chifukwa chake akufuna kuwachotsa.
Ngati mukugwira ntchito pa vuto, koma samakuvutitsani aliyense koma inu, koma inu ntchito mwadongosolo ndipo potsirizira pake amaona vuto, ndiye kukoka pempho lanu ndithu kulandiridwa. Chigawo chanu chidzavomerezedwa, zokometsera zanu kapena zopempha zanu zokongoletsedwa zidzawunikiridwa ndi anthu ammudzi. Kumapeto kwa tsiku, timapanga database kukhala yabwino kwa wina ndi mnzake.
Dongosolo losangalatsa la database ndi Greenplum. Ndi database yofananira kwambiri yotengera Postgres codebase, yomwe ndimayidziwa bwino.
Ndipo Greenplum ili ndi magwiridwe antchito osangalatsa - onjezerani matebulo okongoletsedwa. Awa ndi matebulo omwe mutha kuwonjezerapo mwachangu. Iwo akhoza kukhala columnar kapena mzere.
Koma panalibe kusanja, mwachitsanzo, panalibe magwiridwe antchito pomwe mutha kukonza zomwe zili patebulo molingana ndi dongosolo lomwe lili mu imodzi mwazolozera.
Anyamata a m’takisiyo anabwera kwa ine n’kunena kuti: “Andrey, ukumudziwa Postgres. Ndipo apa zili pafupifupi zofanana. Sinthani kukhala mphindi 20. Inu mutenge izo ndi kuchita izo.” Ndinaganiza kuti inde, ndikudziwa Postgres, kusintha kwa mphindi 20 - ndiyenera kuchita izi.
Koma ayi, sizinali mphindi 20, ndinazilemba kwa miyezi ingapo. Pamsonkhano wa PgConf.Russia, ndinapita kwa Heikki Linakangas wochokera ku Pivotal ndi kumufunsa kuti: “Kodi pali vuto lililonse pankhaniyi? Chifukwa chiyani palibe kuphatikiza kwamatebulo kowonjezera?" Iye anati: “Mumatengera deta. Mumakonza, mumakonzanso. Ndi ntchito chabe." Ine: "Inde, muyenera kungotenga ndikuchita." Iye akuti: “Inde, timafunikira manja aulere kuti tichite izi.” Ndinaganiza kuti ndiyeneradi kuchita zimenezi.
Ndipo miyezi ingapo pambuyo pake ndinapereka pempho lachikoka lomwe linakhazikitsa izi. Pempho lokopali lidawunikiridwa ndi Pivotal pamodzi ndi anthu ammudzi. Inde, panali nsikidzi.
Koma chochititsa chidwi kwambiri ndichakuti pempholi likaphatikizidwa, nsikidzi zidapezeka ku Greenplum komwe. Tapeza kuti milu ya tebulo nthawi zina imasokoneza kusinthana ikaphatikizidwa. Ndipo ichi ndi chinthu chomwe chiyenera kukonzedwa. Ndipo iye ali pamalo amene ndangowakhudza kumene. Ndipo zomwe ndinachita mwachibadwa zinali - chabwino, ndiroleni inenso ndichite izi.
Ndakonza cholakwika ichi. Anatumiza pempho kukoka kwa okonza. Iye anaphedwa.
Pambuyo pake zidapezeka kuti ntchitoyi ikufunika kupezedwa mu Greenplum version ya PostgreSQL 12. Ndiko kuti, ulendo wa mphindi 20 ukupitirizabe ndi zatsopano zosangalatsa. Zinali zosangalatsa kukhudza chitukuko chamakono, kumene anthu ammudzi akudula zinthu zatsopano komanso zofunika kwambiri. Wazizira.
Koma sizinathere pamenepo. Pambuyo pa zonse, zidapezeka kuti tikufunika kulemba zolemba pazonsezi.
Ndinayamba kulemba zolemba. Mwamwayi, olemba zolemba ochokera ku Pivotal adabwera. Chingerezi ndi chilankhulo chawo. Iwo anandithandiza ndi zolembedwa. M'malo mwake, iwonso adalembanso zomwe ndidapereka m'Chingerezi chenicheni.
Ndipo apa, zikuwoneka, ulendowo unatha. Ndipo kodi mukudziwa zimene zinachitika pamenepo? Anyamata a m’takisiyo anabwera kwa ine n’kunena kuti: “Pakadalipo maulendo aŵiri, iliyonse kwa mphindi 10.” Ndipo ndiwauze chiyani? Ndidati tsopano ndipereka lipoti pamlingo, ndiye tiwona zochitika zanu, chifukwa iyi ndi ntchito yosangalatsa.
Tiphunzilapo ciani pa nkhani imeneyi? Chifukwa kugwira ntchito ndi gwero lotseguka nthawi zonse kumagwira ntchito ndi munthu wina, nthawi zonse kumagwira ntchito ndi anthu ammudzi. Chifukwa pagawo lililonse ndimagwira ntchito ndi wopanga ena, woyesa, ena owononga, ena olemba zolemba, ena omanga. Sindinagwire ntchito ndi Greenplum, ndimagwira ntchito ndi anthu ozungulira Greenplum.
Koma! Palinso mfundo ina yofunika - ndi ntchito basi. Ndiko kuti, mubwere, kumwa khofi, kulemba code. Mitundu yonse ya zosintha zosavuta zimagwira ntchito. Chitani mwachizolowezi - zikhala bwino! Ndipo ndi ntchito yosangalatsa. Pali pempho la ntchitoyi kuchokera kwa makasitomala a Yandex.Cloud, ogwiritsa ntchito magulu athu mkati mwa Yandex ndi kunja. Ndipo ndikuganiza kuti kuchuluka kwa mapulojekiti omwe tikuchita nawo kudzawonjezeka komanso kuzama kwakutengapo gawo kwathu kudzawonjezekanso.
Ndizomwezo. Tiyeni tipitirire ku mafunso.
Gawo la mafunso
Moni! Tili ndi gawo lina la mafunso ndi mayankho. Ndipo mu studio Andrei Borodin. Uyu ndi munthu amene wangokuuzani za zopereka za Yandex.Cloud ndi Yandex kuti mutsegule gwero. Lipoti lathu tsopano siliri kwathunthu za Mtambo, koma nthawi yomweyo ife timachokera ku matekinoloje oterowo. Popanda zomwe mudachita mkati mwa Yandex, sipakanakhala ntchito ku Yandex.Cloud, kotero zikomo kuchokera kwa ine ndekha. Ndipo funso loyamba kuchokera pawailesi: "Kodi projekiti iliyonse yomwe mudatchulapo idalembedwa bwanji?"
Makina osungira mu WAL-G amalembedwa mu Go. Iyi ndi imodzi mwamapulojekiti atsopano omwe tagwirapo. Ali ndi zaka zitatu zokha. Ndipo database nthawi zambiri imakhala yodalirika. Ndipo izi zikutanthauza kuti nkhokwezo ndi zakale kwambiri ndipo nthawi zambiri zimalembedwa mu C. Ntchito ya Postgres idayamba pafupifupi zaka 3 zapitazo. Ndiye C30 inali chisankho choyenera. Ndipo Postgres yalembedwa pamenepo. Zambiri zamakono monga ClickHouse nthawi zambiri zimalembedwa mu C ++. Kukula kwadongosolo lonse kumakhazikitsidwa mozungulira C ndi C ++.
Funso lochokera kwa manejala wathu wazachuma, yemwe amayang'anira ndalama pa Cloud: "N'chifukwa chiyani Cloud imagwiritsa ntchito ndalama pothandizira gwero lotseguka?"
Pali yankho losavuta kwa woyang'anira zachuma pano. Timachita izi kuti ntchito zathu zikhale bwino. Kodi tingachite bwino bwanji? Titha kuchita zinthu moyenera, mwachangu, ndikupangitsa kuti zinthu ziwonjezeke. Koma kwa ife, nkhaniyi ikunena za kudalirika. Mwachitsanzo, mu zosunga zobwezeretsera timawunika 100% ya zigamba zomwe zimagwira ntchito. Tikudziwa chomwe code ndi. Ndipo ndife omasuka kutulutsa mitundu yatsopano yopanga. Ndiko kuti, choyamba, ndi za chidaliro, za kukonzekera chitukuko ndi za kudalirika
Funso lina: "Kodi zofunikira za ogwiritsa ntchito akunja omwe amakhala ku Yandex.Cloud ndizosiyana ndi ogwiritsa ntchito mkati omwe amakhala mumtambo wamkati?"
Mbiri ya katundu ndi, ndithudi, yosiyana. Koma kuchokera ku dipatimenti yanga, milandu yonse yapadera komanso yosangalatsa imapangidwa pa katundu wosagwirizana. Madivelopa okhala ndi malingaliro, opanga omwe amachita zosayembekezereka, amatha kupezeka mkati ndi kunja. Pankhani imeneyi, tonse ndife ofanana. Ndipo, mwinamwake, chinthu chokhacho chofunikira mkati mwa ntchito ya Yandex ya databases chidzakhala chakuti mkati mwa Yandex tili ndi chiphunzitso. Panthawi ina, malo ena opezeka amalowa mumthunzi, ndipo ntchito zonse za Yandex ziyenera kupitiriza kugwira ntchito ngakhale izi. Izi ndi zosiyana pang'ono. Koma zimapanga chitukuko chochuluka cha kafukufuku pa mawonekedwe a database ndi network stack. Kupanda kutero, makhazikitsidwe akunja ndi amkati amatulutsa zopempha zomwezo za mawonekedwe ndi zopempha zofananira kuti muwongolere kudalirika ndi magwiridwe antchito.
Funso lotsatira: "Kodi inuyo panokha mukumva bwanji ponena kuti zambiri zomwe mumachita zimagwiritsidwa ntchito ndi Mitambo ina?" Sitidzatchula ena enieni, koma mapulojekiti ambiri omwe anachitidwa ku Yandex.Cloud amagwiritsidwa ntchito mumitambo ya anthu ena.
Izi ndizabwino. Choyamba, ndi chizindikiro chakuti tachita bwino. Ndipo zimakankha ego. Ndipo ndife otsimikiza kuti tinasankha bwino. Kumbali inayi, ichi ndi chiyembekezo chakuti m'tsogolomu izi zidzatibweretsera malingaliro atsopano, zopempha zatsopano kuchokera kwa ogwiritsa ntchito chipani chachitatu. Nkhani zambiri pa GitHub zimapangidwa ndi oyang'anira machitidwe, ma DBA pawokha, omanga pawokha, mainjiniya pawokha, koma nthawi zina anthu omwe ali ndi chidziwitso mwadongosolo amabwera ndikuti mu 30% yamilandu ina tili ndi vutoli ndipo tiyeni tiganizire momwe tingalithetsere. Izi ndi zomwe tikuyembekezera kwambiri. Tikuyembekezera kugawana zomwe takumana nazo ndi nsanja zina zamtambo.
Munayankhula zambiri za marathon. Ndikudziwa kuti mudathamanga marathon ku Moscow. Zotsatira zake? Adapeza anyamata aku PostgreSQL?
Ayi, Oleg Bartunov amathamanga kwambiri. Anamaliza ola limodzi patsogolo panga. Pazonse, ndine wokondwa ndi momwe ndafikira. Kwa ine, kungomaliza kunali kupindula. Ponseponse, ndizodabwitsa kuti pali othamanga ambiri mdera la postgres. Zikuwoneka kwa ine kuti pali mtundu wina wa ubale pakati pa masewera a aerobic ndi chikhumbo cha mapulogalamu a machitidwe.
Kodi mukunena kuti palibe othamanga ku ClickHouse?
Ndikudziwa motsimikiza kuti alipo. ClickHouse ndinso nkhokwe. Mwa njira, Oleg tsopano akundilembera kuti: "Kodi tithamangire lipoti?" Ili ndi lingaliro labwino.
Funso lina kuchokera pawailesi yaku Nikita: "N'chifukwa chiyani munakonza nokha cholakwika ku Greenplum osachipereka kwa achinyamata?" Zowona, sizikudziwika bwino kuti cholakwikacho ndi chiyani komanso ndi ntchito iti, koma mwina zikutanthauza zomwe mudazikamba.
Inde, kwenikweni, zikanaperekedwa kwa winawake. Inali chabe code yomwe ndinangoyisintha. Ndipo zinali zachibadwa kupitiriza kuchita zimenezo nthawi yomweyo. M'malo mwake, lingaliro logawana ukadaulo ndi gulu ndi lingaliro labwino. Tigawana nawo ntchito za Greenplum pakati pa mamembala onse agawo lathu.
Popeza tikukamba za achinyamata, nali funso. Munthuyo adaganiza zopanga gawo loyamba ku Postgres. Kodi ayenera kuchita chiyani kuti akwaniritse cholinga choyamba?
Ili ndi funso lochititsa chidwi: "Kuyambira pati?" Nthawi zambiri zimakhala zovuta kuyamba ndi china chake mu kernel. Mu Postgres, mwachitsanzo, pali mndandanda wazomwe mungachite. Koma kwenikweni, ili ndi pepala la zomwe adayesera kuchita, koma sizinapambane. Izi ndi zinthu zovuta. Ndipo nthawi zambiri mumatha kupeza zofunikira pazachilengedwe, zowonjezera zina zomwe zitha kuwongolera, zomwe zimakopa chidwi chochepa kuchokera kwa opanga ma kernel. Ndipo, motero, pali mfundo zambiri za kukula kumeneko. Pa Google Summer of code program, chaka chilichonse gulu la postgres limapereka mitu yambiri yomwe ingayankhidwe. Chaka chino tinali ndi, ndikuganiza, ophunzira atatu. Mmodzi adalembanso mu WAL-G pamitu yomwe ili yofunika ku Yandex. Ku Greenplum, chilichonse ndi chosavuta kuposa gulu la Postgres, chifukwa obera a Greenplum amachitira zopempha zabwino kwambiri ndikuyamba kuwunika nthawi yomweyo. Kutumiza chigamba ku Postgres ndi nkhani ya miyezi, koma Greenplum idzabwera tsiku limodzi ndikuwona zomwe mwachita. Chinanso ndikuti Greenplum iyenera kuthetsa mavuto omwe alipo. Greenplum sigwiritsidwa ntchito kwambiri, kotero kupeza vuto lanu ndikovuta. Ndipo choyamba, tiyenera kuthetsa mavuto, ndithudi.
Source: www.habr.com