🥇Kugadziriswa kwemafaira masisitimu kudiki LinuxMaitiro Ekugadzira Makopi Anoshanda eDhaibhaiti reMySQL reTerabyte nhatu mumasekondi makumi maviri

Zita rangu ndinonzi Yuri, uye ndiri mukuru wechikwata chekutonga masisitimu kuCitymobil. Nhasi, ndichagovana ruzivo rwangu nehunyanzvi hwekugadzira masisitimu emafaira. Linux Ndichatsanangura mashandisirwo ainogona kuitwa mumaitiro eCI/CD ekambani. Tichaongorora mamiriro ezvinhu apo, kuti tiedze otomatiki kodhi patinenge tichiiendesa kuproduction, tinoda makopi eMySQL ekuverenga-kunyora ari pedyo neproduction version sezvinobvira.

Nhanganyaya: Sei uchipa mazano akaipa?

Mubvunzo une musoro, nekuti kune nzira dzakasimbiswa dzekutamisa dhatabhesi schemas munzvimbo dzekuyedza. Sei uchitowedzera iyo huru isina-shared DBMS kune mavhoriyamu akadaro? Uye haisi yese data inodiwa pakuyedzwa. Ndichaedza kutsanangura.

Rinenge gore rapfuura, pakatarisana nekukura kwekukura kwetaxi aggregator (muna 2018, nzendo dzakapedzwa dzakawedzera kanokwana kagumi neshanu), huwandu hwe data, mutoro pamaseva, uye kuwanda kwekuburitswa kwakawedzera. Takazviwana tiri mune inotevera situation:

Iyo huru MySQL dhatabhesi yakakura kusvika kunosvika zana matafura anosvika 1000 TB, uye yakaramba ichikura.
Pakanga pasina nzira yekukurumidza kuve sharded uye kuparadza hwaro. Izvi zvakanga zvisingabvumirwi nemaitiro ekare "Ndinonyora mudhatabhesi zvandinoda uye kuti ndinoda sei", boka reJOINs uye tafura yemukati inotsamira.
Pakanga pasina nzira yekufambisa iyo database schema munzvimbo dzekuyedza.
Pakanga pasina otomatiki kuyedzwa kwekodhi panguva yekutumirwa.

Ndaida kugadzirisa dambudziko rekupedzisira nekukurumidza sezvinobvira. Miedzo yePostman yakanga yatonyorwa kuti iedze iyo huru PHP monolith, asi yakashaya dhatabhesi rezvino. Panguva imwecheteyo, isu hatigone kugadzira mufananidzo wehusiku, kuita tenzi uye kuisiya kuti iparadzwe zvidimbu masikati: nhamba huru kwazvo yekuburitswa uye shanduko, kusanganisira mune data uye dhatabhesi schema, ingadai yakaita. chigadziko hachishande pakati pezuva. Uye kumisa kuburitswa kwezuva rekushanda chete kungave kusashanda.

Kunyange zvakadaro, basa racho rakapedzwa: takagamuchira nzvimbo yokutanga yokushanda mukati memavhiki maviri. Yakave neshanduko dzakawanda mugore rapfuura uye ichiri kushandiswa.

Tevere, ini ndichatsanangura zvakadzama matanho nematanho ekuvandudza kwemhinduro yedu. Iwe uchaona kuti nzira iyi inokodzera kodzero yayo yekuvepo.

Chii chinonzi "thin redundancy"?
Iyi ihardware kana software tekinoroji (rimwe zita ishoma mavhoriyamu) inobvumidza iwe kugovera yakawanda yeinodiwa sosi pane iripo. Muchiitiko ichi, vhoriyamu yakagoverwa inofanirwa kusangana nemaitiro ekungo-zvakakwana (sezvazvinoda) uye-mu-nguva (yenguva inodiwa). Chaizvoizvo, kuchengetedza kwakatetepa kunoshandiswa mumasisitimu akasiyana ekuchengetedza kuti ape dhisiki nzvimbo mumavhoriyamu anodiwa, kupfuura ayo aripo chaizvo. Iyo tekinoroji inotsigirwa neakasiyana faira masisitimu, semuenzaniso, LVM2, ZFS, BTRFS. Inoshandiswa zvakanyanya mu virtualization hypervisors. Kuchengetedza kwakaonda kwakatibvumira kukurumidza kugadzira kubva pamifananidzo yechikamu chikuru nedata semakopi akawanda echikamu ichi sezvataida (dhairekitori redata reMySQL DBMS).

Chekutanga kumira, Thin LVM tekinoroji

Ichi chitsauko chinogona zvakare kudaidzwa kuti "Maitiro ekuita zvinokurumidza snapshots zvemavhoriyamu makuru edata uchishandisa Mutete LVM, kuderedza kugadzikana kwefaira system uye MySQL DBMS kusvika kumazinga asina kunaka."

Sezvo isu takatoshandisa LVM kuvaka iyo huru OS partitions, takafunga kutanga nayo. Kutanga, taida muchina wakasiyana wemuviri - mufananidzo wedhatabhesi yedu huru yeMySQL, yataigona kugadzira mufananidzo weiyo replica pakukumbira uye kuisimudza padivi peimwe MySQL muenzaniso. Panguva yekuyedzwa, isu takabvumira kushandura mashandiro kuti ashandiswe pachiitiko ichi, uye pakupera kwemiedzo, takaidzima zvakachengeteka. Sevha yekumisikidza yaive seizvi:

2 x Intel Silver 4114 (10x2,2 GHz HT)
8 x 32 GB DDR4
8 x 1920 GB Intel SSD muAdaptec RAID controller muRAID-10

Iwe unogona kunyora chinyorwa chakasiyana pamusoro pemusoro wekusarudza pakati peRAID controller uye software RAID MD. Rega ndingotaura kuti sarudzo yedu yakakanganiswa nezvinhu zviviri:

Panguva iyo dambudziko rakagadzirwa, takaisa ese maDBMS pane RAID controllers, saka tinogona kutaura kuti izvi zvakaitika kare.
Musiyano mukuita pakati pesynthetic faira system bvunzo uye bvunzo neakasiyana MySQL mashandiro aive mashoma.

Isu takapatsanura yakaguma RAID-10: isu takagadzira imwechete Vhoriyamu Boka (VG) yevhoriyamu yese (ine pamusoro peinosvika 6,7 GB) uye takagadzira chikamu chine musoro (Logical Volume, LV) ye50 GB system. Mumamiriro ezvinhu akajairwa, tinotsanangura imwe nzvimbo sechikamu cheMySQL. Asi isu taida backup yakaonda, saka takatanga tagadzira inodaidzwa dziva, mukati matakagadzira chikamu che /var/lib/mysql ine 3,5 TB (zvichienderana neyakafungidzirwa dhatabhesi mavhoriyamu):

lvcreate -l 100%FREE -T vga/thin
lvcreate -V 3.5T -T vga/thin -n mysql

Isu takafometa iyo partition mu ext4, takaiisa, takarekodha replica uye tikawana iyo yekutanga kumira. Ipapo takaita chinosunga muchimiro che API, iyo inofanirwa kugadzira snapshots, simudza MySQL dhatabhesi muenzaniso pane yakapihwa chiteshi uye kudzima iyo yakagadzirwa muenzaniso. Sezvo izvi zvichishandisa nharembozha chete, takasarudza bash yenguva dzose semutauro wekunyora, uye takaisa yakavhurika sosi mhinduro yekubatanidza iyo HTTP → bash API. goexpose, rakanyorwa muna Go.

Rimwe zuva tichaburitsa zvinyorwa zvedu zvebash mune yakavhurika sosi, asi ikozvino ini ndinongotsanangura iyo huru algorithm:

Kugadzira iyo huru snapshot snapmain:

Kumisa iyo huru replica.
Isu tinoisa block pane maoperation neiyo snapmain snapshot.
Gadzira imwe snapshot itsva.
Tanga MySQL uye bvisa kukiya.

Kugadzira dhatabhesi pane inopokana port kubva snapmain:

Isu tinoisa kukiya pane yakatarwa dhatabhesi muenzaniso (port).
Isu tinotarisa kana kusikwa kweiyo main snapshot kwakavharwa. Kana iripo, saka tinomirira uye toongororazve masekonzi mashanu.
Isu tinotarisa kana paine yekare LV chikamu chemuenzaniso.
3.1 Kana iripo, saka shandisa kuuraya -9 kumisa chiitiko cheMySQL uye kudzima chikamu cheLV.
Isu tinogadzira chiitiko chitsva kubva snapmain.
Isu tinogadzirira uye tinoisa madhairekitori echiitiko ichi.
Isu tinobvisa hunhu hwemuranda (mafaira) uye totanga iyo MySQL muenzaniso.
Ngatimuite tenzi.
Tinobvisa kuvhara.

Kubvisa dhatabhesi pane yakasarudzika port:

Isu tinoisa kukiya pane yakatarwa dhatabhesi muenzaniso (port).
Uraya iyo MySQL muenzaniso uchishandisa kuuraya -9.
Ngatiburutse madhairekitori.
Isu tinodzima chikamu cheLV uye tinobvisa kukiya.

Mienzaniso yemirairo yekubatanidza zvikamu zvechiitiko chitsva che database:

lvcreate -n stage_3307 -s vga/snapmain
lvchange -ay -K vga/stage_3307
mount -o noatime,nodiratime,data=writeback /dev/mapper/vga-stage_3307 /mnt/stage_3307

Zvino ini ndichakuudza nezve dambudziko guru ratakasangana naro pakushandisa mutete redundancy. Isu takanamira pakuita kweSSD madhiraivha. Izvi zvakaitika nekuda kwemaitiro eThin LVM: inoshanda padanho remudziyo ine yakaderera-level chunks yeiyo default saizi ye4 MB. Zvairatidzika sei:

Gadzira mufananidzo kubva pachikamu chikuru /var/lib/mysql.
Tinotanga kudzokorora kuti tibate natenzi.
Chero shanduko mumatafura e replica inomanikidza chekare, isina kuchinjika chunks yedata kuti ichengetwe muchikamu chesnapshot.
Chero shanduko kune yakasimudzwa bvunzo muenzaniso inokonzeresa yekare, isina kugadziridzwa chunks yedata kuti ichengetwe muchikamu chakamisikidzwa chechiitiko ichocho.
Isu tinowana kuremerwa kwe100% I/O mashandiro pane mudziyo, kudzikira kwechero mashandiro uye zvishoma nezvishoma lag replica.
Pakupera kwezuva rekushanda tinowana kumira kuri maawa akawanda kumashure.

Mabatiro atakaita izvi kuti tiwane mhedzisiro (mapoinzi makuru):

RAID controller:

Mhando dzese dze caching dzakadzimwa nekusarudzika.
Seta kunyora kumashure (kana data ichipinda mubuffer, kunyora kunopedzwa kusati kwaitwa kuchengetedza chaiko ku diski).

Faira system:

Pagomo rekuti /var/lib/mysql takanyora noatime,nodiratime,data=writeback
Yakaremara ext4 matanda uchishandisa tune2fs.

MySQL:

Prescribed innodb_flush_method = O_DSYNC (yakawedzera kukurumidza kurekodha, nokudaro kuderedza kuvimbika).
Kutema miti kwakadzimwa, hatidi matanda.
Prescribed innodb_buffer_pool_size = 4G (iyo diki yeInnoDB pool size, iyo inokurumidza MySQL inovhara kana yamira, uye nekukurumidza isu tichagadzira mufananidzo).

Iyi haisi rondedzero yakazara, kunyanya yeMySQL. Nekudaro, iyo yasara shanduko idiki uye kazhinji haiwanzo kana kushanda nemazvo. Somuenzaniso, mukuedza kuburutsa madhisiki, takatoenda nawo innodb_parallel_doublewrite_path mu/dev/shm, iyo mune dzimwe nguva yakatiponesa kusvika kumasekonzi mashanu kana tichitanga chiitiko chisina kubviswa.

Sei tichimisa MySQL tisati tatora mufananidzo? Mushure mezvose, isu tinogona kuibvisa kubva pane inoshanda replica. Ndizvozvo, asi iyo nyowani dhatabhesi muenzaniso pane ino snapshot ichaonekwa yakakuvadzwa nekusarudzika uye inoda yakazara scanner pakutanga. Kumisa replica inokurumidza kukurumidza, kunyangwe ichizopedzisira yave iyo yakareba kushanda mukuita kwese.

Nekuda kweizvozvo, takagamuchira nguva dzakagamuchirwa uye takagadzirira-kushandisa. Kunyangwe, sezvinogona kuonekwa kubva kune yakanyatsojeka girafu yekudzokorora lag yeiyo huru replica, mamiriro acho achiri kure nekunaka:

Pakati pezvimwe zvikanganiso, zvakakosha kucherechedza kusagoneka kusingagoneke kwekutarisa iyo Thin LVM dziva: mukuwedzera kune iyo system standard iostat mabasa, hazvigoneke kunzwisisa, semuenzaniso, iyo dziva chinhu chiri kuburitsa mutoro mukuru pane iyo faira system.

Zvakasiyana, zvakakosha kuti ticherechedze imwe hombe dhizaini yakabatana neiyo optimization yakatsanangurwa pamusoro: isu takagamuchira YOLO stand. Inenge kamwe chete mwedzi mumwe kana miviri ext4 yaisakwanisa kumirisana nekushungurudzwa kwakadaro uye yakaparara zvisingagadzirisike, ichida kugadziridzwa uye kurodha zvakare replica. Takunda mukumhanya, takakanganisa kugadzikana zvisina tariro.

Ndeapi metrics aunofanirwa kutarisa paunenge uchishandisa Thin LVM:

Data yemvura yakatetepa%
Din pool metadata %

Kana kumira kwedu kunopukunyuka kupera kwenzvimbo yedata (yakakwana kuchenesa madhisiki), ipapo kupera kwenzvimbo yemetadata kuchaita kuti pave nekuputsika kwedziva uye kudiwa kwekuivakazve kubva pakutanga.

Iyo faira system mukati medziva inove yakakamukana zvakanyanya nekufamba kwenguva. Ini ndinokurudzira kumhanya iyo cron command kamwe pazuva fstrim -v /var/lib/mysql.

Subtotals:

Iyo tekinoroji iri nyore kushandisa, senge LVM pachayo, uye haidi yakakosha mainjiniya qualifications.
Iyo yakanyatsokodzera kune diki uye isina kurodha dhatabhesi. Iyo diki dhatabhesi, mashoma chunks anofamba achitenderedza faira system mukati medziva, uye anodzikisa mutoro pamadhisiki.
Kubasa redu, takatanga kutsvaga mamwe magadzirirwo, ayo achakurukurwa muchikamu chinotevera.

Chechipiri kumira, ZFS tekinoroji

Ndakamboshanda neZFS file system kare kare, asi kare ikako, ZFS yaishanda zvakanaka pamhuri yayo yeSolaris OS. Paiva nechiteshi cheFreeBSD chine nhanho yakanaka yekuitwa. Paivawo nechiteshi chisina kupera chekuisa. Linux, iyo vanhu vashoma vaiishandisa. Nekuda kwechimiro chayo chekuchengetera data cheB-tree (zvineiwo, MySQL's InnoDB ine chimiro chakafanana chekuchengetera), ZFS haina kushanda zvakanaka pakuisa mafaira akawanda. Izvi, pamwe chete nekudiwa kwekudzidza tambo usati washandisa, zvakaita kuti ndishandise system iyi kwenguva refu. Ext4 ne xfs zvakabuda zvikava standard. Asi tichifunga kuti ZFS yakakodzera zvatinoda, uye Linux-version, tichitarisa wongororo, yakura kuva chigadzirwa chine musoro chose (kunyange zvazvo chisingatsigirwi zvizere, ndosaka kuisa system paZFS kubva pakutanga kuchingogoneka chete nerubatsiro rwemhando dzakasiyana dzevoodoo), takasarudza kuiedza.

Nezvikonzero zviri pachena, chimiro chakasarudzwa neyakafanana dhizaini (kunze kweiyo RAID controller). Isu takaisa masere 1920 GB SSD madhiraivha. Pakanga pasina chishuwo chekunyora yedu yetiweki mufananidzo kurodha sevha kuti iburitse ZFS, saka isu takadzima makumi mashanu GB emadhisiki ese ndokugadzira MD RAID-50 pavari kune iyo system. Iyo yakasara 10 GB pane imwe diski yakasanganiswa kuita ZFS analogue yeRAID-1950:

zpool create zpool mirror /dev/sda2 /dev/sdb2 mirror /dev/sdc2 /dev/sdd2 mirror /dev/sde2 /dev/sdf2 mirror /dev/sdg2 /dev/sdh2

Isu takagadzira zvikamu zveMySQL:

zfs create zpool/mysql
zfs set compression=gzip zpool/mysql
zfs set recordsize=128k zpool/mysql
zfs set atime=off zpool/mysql
zfs create zpool/mysql/data
zfs set recordsize=16k zpool/mysql/data
zfs set primarycache=metadata zpool/mysql/data
zfs set mountpoint=/var/lib/mysql zpool/mysql/data

Ndokumbira utarise kuti tagonesa kudzvanywa kwedata ye gzip. Tine zvakawanda zvekushandisa processor pane sevha uye hazvina kushandiswa zvizere.Nekuda kweizvozvo, 3 TB yedatabase yedu yakashandurwa kuita 1,6 TB, uye sezvo iyo isina kusimba link, senge mune yapfuura kesi, ndiyo yakanyanya disk performance, iyo data shoma zviri nani, isu Tinowana bhonasi yakanaka kubva kuZFS kubva pakutanga! Munguva yekumhanyisa nguva pakuremerwa kwakazara, zvinotora kusvika ku4 cores kuti gzip irambe ichimhanya, asi isu hatina basa nazvo.

Ipapo kuiswa kwacho kwakakurumidza. Iyo MySQL replica marongero akatamiswa kubva paLVM kumira sekopi kabhoni. Ini ndaifanira kupedza imwe nguva ndichinyora zvakare zvinyorwa ndichishandisa ZFS mirairo, asi kazhinji maalgorithms akaramba akafanana. Muenzaniso wekugadzira snapshot:

zfs set snapdir=visible zpool/mysql/data
zfs create zpool/stage_3307
zfs clone zpool/mysql/data@snapmain zpool/stage_3307/data
zfs set mountpoint=/mnt/stage_3307 zpool/stage_3307/data

Kubva pakuwedzera tuning: takafambisa ZFS zvikamu zvine metadata uye l2arc uye zil matanda mundangariro. Zvebasa redu, sezvazvakazoitika gare gare, izvi zvaive zvisingaite, asi ikozvino takasiya iyi optimization; zviri nyore kuchinja kana zvichidikanwa. Imwe yeakaipa mhedzisiro ndeyekuti mushure mekutangazve sevha, iwe unofanirwa kudzokorora inoenderana nzvimbo dzekurangarira. Hapana data yakarasika. Zpool chimiro chekucheka:

logs
      /dev/shm/zil_slog.img  ONLINE       0     0     0
cache
      /dev/shm/l2arc.img     ONLINE       0     0     0

Mukugadzirisa uku, takatanga kuyedza kumira uye tikawana mhedzisiro yakanaka: iine maviri panguva imwe chete achimhanya dhatabhesi zviitiko (uye inoshanda main replica) mumifananidzo, takawana 50-60% disk kushandiswa.

Isu takabvisa dambudziko redu guru, sezvinoonekwa mune replication lag graph (enzanisa nerapfuura girafu muThin LVM chikamu):

Pamusoro pe uye nekutenda kune izvi, isu takawedzera zvakanyanya mashandiro ese: kugadzira zvachose snapshot nekumira uye kutanga replica kunotora kusvika kumasekonzi makumi mana, kutumira chiitiko chitsva cheMySQL kubva pamufananidzo kunotora kusvika kumasekonzi makumi maviri. Izvo zvinonyanya kugutsa isu tese uye yedu chirongwa chekodhi bvunzo.

Subtotals:

Mhedzisiro yacho yakagutsa zvizere kuda kwedu kuwana kopi yedhatabhesi rekugadzira yekuyedza kodhi.
Iyo tekinoroji inoda kupinda: iwe unofanirwa kunzwisisa kuti ZFS chii uye maitiro ekushanda nayo.
Hatina kutarisa chimiro chazvino cheZFS nenhamba huru (inopfuura miriyoni imwe) yemafaira madiki. Asi isu tinofungidzira kuti dambudziko rinoramba riripo, saka ini handingakurudzire iyi faira system kune chero faira rekuchengetedza.

Chii chinotevera?

Hapana chimwe chekuita mukati mechimiro chechimiro; isu tinogutsikana nemhedzisiro. Zvichida mune ramangwana isu tichawedzera kusarudzika kwematafura asingadikanwi pakuyedzwa kune yekumisikidza yekumisikidza; izvi zvichawedzera kuderedza saizi yedatabase. Hatina kuyedza iyo BTRFS system uye kuita kwayo yakaonda redundancy tekinoroji. Zvisinei, basa rakadaro harichakoshi, sezvo chinangwa chikuru chakaitwa. Kazhinji, hongu, ndinoda kubva kure nenzira inotsanangurwa pamusoro - shandisa kushanda dhatabhesi kutamira kunzvimbo yekuyedza, gadzira yakaparadzana bvunzo dhatabhesi redunhu, uye tanga sharding huru dhatabhesi. Isu tave kutoisa zvakawanda zveizvi mukuita, izvo zvatichataura nezvazvo mune ramangwana zvinyorwa.

Migumisiro

Dambudziko rekutanga rakagadziriswa, kunyangwe nenzira isina kujairika. Mhedziso dzepakati dzakatsanangura zvakanakira uye zvakaipira zveimwe neimwe yetekinoroji inoshandiswa, saka ngatisarudzei kuti ndeipi tekinoroji inogona kushandiswa uye riinhi:

Thin LVM - yediki dhatabhesi uye kana iwe usingade kana usina nguva yekudzidza ZFS.
ZFS - kana uine ruzivo rwekushanda nayo kana mukana wekupedza nguva uchiidzidza mune chero mamiriro ezvinhu.

Pamwero wepamusoro wemharidzo, chinyorwa ichi hachisi kungofananidza tekinoroji yemafaira maviri. Pfungwa huru yandinoda kuburitsa nekusimbisa ndeyekuti haufanirwe kutya kufunga kunze kwebhokisi mune bhizinesi-akakosha mamiriro uye kutora chete akagadzirira-akagadzirwa mabikirwo. Pane imwe nguva, taigona kudzungudza misoro yedu sese mudhipatimendi rehunyanzvi uye tichiti basa rekugadzira matatu-terabyte makopi edhatabheti isingasviki miniti hazvigoneke, uye hatidi tekinoroji ine njodzi, ngatiite. zvakanaka. Zvaigoneka, asi tingadai takarasikirwa nemwedzi mitanhatu kusvika pagore uye nzendo zhinji dzevatengi (kufamba ndicho chiratidzo chedu chikuru chebhizinesi) pasina kuyedzwa uye panguva yekushandisa. Nekuita kunze kwebhokisi, hatina kurasikirwa nenguva yakawanda pakuita, takawana ruzivo mune hutsva uye hwakakanganwa matekinoroji, uye kupa kuyedzwa panguva yataida chaizvo. Pasina mubvunzo, izvi zvakave nemhedzisiro yakanaka pane zvese zviratidzo zvedu. Sarudzo inogara iri yako, uye isu, kune chikamu chedu, ticharamba tichitaura pamusoro pezvinofadza zvazvino uye zveramangwana kubudirira mune yedu blog.

Source: www.habr.com