Muaj ntau RAM pub dawb, NVMe Intel P4500 thiab txhua yam yog qeeb heev - zaj dab neeg ntawm kev ua tsis tiav ntawm kev sib pauv muab faib

Hauv tsab xov xwm no, kuv yuav tham txog qhov xwm txheej uas nyuam qhuav tshwm sim nrog ib qho ntawm cov servers hauv peb VPS huab, uas ua rau kuv poob rau ob peb teev. Kuv tau teeb tsa thiab daws teeb meem Linux servers txog li 15 xyoo, tab sis qhov teeb meem no tsis haum rau kuv qhov kev coj ua txhua - Kuv tau ua ntau qhov kev xav tsis tseeb thiab tau txais kev xav me ntsis ua ntej kuv tuaj yeem txiav txim siab qhov laj thawj ntawm qhov teeb meem thiab daws nws. .

Npaj

Peb ua haujlwm huab cua nruab nrab, uas peb tsim cov qauv servers nrog cov teeb tsa hauv qab no - 32 cores, 256 GB RAM thiab 4500TB PCI-E Intel P4 NVMe tsav. Peb yeej nyiam qhov kev teeb tsa no vim tias nws tshem tawm qhov xav tau kev txhawj xeeb txog IO nyiaj siv ua haujlwm los ntawm kev muab cov kev txwv kom raug ntawm VM hom piv txwv. Vim NVMe Intel P4500 muaj kev ua tau zoo, peb tuaj yeem muab ob qho tag nrho IOPS kev muab rau cov tshuab thiab cov ntaub ntawv khaws cia rau cov neeg rau zaub mov thaub qab nrog xoom IOWAIT.

Peb yog ib tug ntawm cov neeg ntseeg qub uas tsis siv hyperconverged SDN thiab lwm yam stylish, fashionable, cov tub ntxhais hluas cov khoom khaws cia VM ntim, ntseeg hais tias qhov yooj yim system, qhov yooj yim nws yog los daws nws nyob rau hauv tej yam kev mob ntawm "lub ntsiab guru tau ploj mus. mus rau roob.” Raws li qhov tshwm sim, peb khaws VM ntim hauv QCOW2 hom hauv XFS lossis EXT4, uas yog xa mus rau sab saum toj ntawm LVM2.

Peb kuj raug yuam kom siv QCOW2 los ntawm cov khoom peb siv rau orchestration - Apache CloudStack.

Txhawm rau ua qhov thaub qab, peb nqa tag nrho cov duab ntawm qhov ntim raws li LVM2 snapshot (yog, peb paub tias LVM2 snapshots qeeb, tab sis Intel P4500 pab peb tawm ntawm no ib yam nkaus). Peb ua lvmcreate -s .. thiab nrog kev pab dd peb xa cov ntaub ntawv thaub qab mus rau cov chaw taws teeb tswj nrog ZFS cia. Ntawm no peb tseem muaj kev vam meej me ntsis - tom qab tag nrho, ZFS tuaj yeem khaws cov ntaub ntawv hauv daim ntawv compressed, thiab peb tuaj yeem kho nws sai sai. DD los yog tau txais ib tus neeg VM ntim siv mount -o loop ....

Koj tuaj yeem, tau kawg, tshem tawm tsis yog tag nrho cov duab ntawm LVM2 ntim, tab sis mount cov ntaub ntawv kaw lus hauv RO thiab luam cov duab QCOW2 lawv tus kheej, txawm li cas los xij, peb tau ntsib nrog qhov tseeb tias XFS ua phem los ntawm qhov no, thiab tsis yog tam sim ntawd, tab sis nyob rau hauv txoj kev tsis tuaj yeem pom. Peb yeej tsis nyiam nws thaum hypervisor hosts "lo" dheev nyob rau hnub so, hmo ntuj lossis hnub so vim qhov ua yuam kev tsis meej thaum lawv yuav tshwm sim. Yog li ntawd, rau XFS peb tsis siv snapshot mounting nyob rau hauv RO txhawm rau rho tawm ntim, peb tsuas luam tag nrho LVM2 ntim.

Qhov ceev ntawm thaub qab mus rau tus neeg rau zaub mov backup yog txiav txim siab nyob rau hauv peb rooj plaub los ntawm kev ua tau zoo ntawm lub backup server, uas yog hais txog 600-800 MB / s rau incompressible cov ntaub ntawv; ib tug ntxiv limiter yog 10Gbit / s channel uas lub backup server txuas nrog. mus rau pawg.

Hauv qhov no, cov ntawv luam theej ntawm 8 hypervisor servers ib txhij muab tso rau hauv ib lub server backup. Yog li, lub disk thiab network subsystems ntawm lub backup server, ua qeeb qeeb, tsis txhob cia lub disk subsystems ntawm lub hypervisor hosts overload, vim lawv tsuas yog tsis muaj peev xwm ua tau, hais, 8 GB / sec, uas lub hypervisor hosts tau yooj yim. tsim tawm.

Cov txheej txheem luam tawm saum toj no tseem ceeb heev rau zaj dab neeg ntxiv, suav nrog cov ntsiab lus - siv lub nrawm Intel P4500 tsav, siv NFS thiab, tej zaum, siv ZFS.

Backup dab neeg

Ntawm txhua qhov hypervisor node peb muaj qhov sib faib me me SWAP ntawm 8 GB hauv qhov loj me, thiab peb "yuav tawm" lub hypervisor node nws tus kheej siv. DD los ntawm daim duab siv. Rau lub kaw lus ntim ntawm cov servers, peb siv 2xSATA SSD RAID1 lossis 2xSAS HDD RAID1 ntawm LSI lossis HP kho vajtse tswj. Feem ntau, peb tsis quav ntsej txhua yam nyob hauv, txij li peb lub kaw lus ntim ua haujlwm hauv "yuav luag nyeem nkaus xwb", tsuas yog SWAP. Thiab txij li thaum peb muaj ntau RAM ntawm lub server thiab nws yog 30-40% dawb, peb tsis xav txog SWAP.

txheej txheem thaub qab. Txoj haujlwm no zoo li no:

#!/bin/bash

mkdir -p /mnt/backups/volumes

DIR=/mnt/images-snap
VOL=images/volume
DATE=$(date "+%d")
HOSTNAME=$(hostname)

lvcreate -s -n $VOL-snap -l100%FREE $VOL
ionice -c3 dd iflag=direct if=/dev/$VOL-snap bs=1M of=/mnt/backups/volumes/$HOSTNAME-$DATE.raw
lvremove -f $VOL-snap

Xyuam xim rau ionice -c3, qhov tseeb, qhov no yog qhov tsis muaj txiaj ntsig rau NVMe li, txij li IO teem sijhawm rau lawv tau teeb tsa:

cat /sys/block/nvme0n1/queue/scheduler
[none] 

Txawm li cas los xij, peb muaj ntau cov kab ke qub txeeg qub teg nrog cov qauv SSD RAIDs, rau lawv qhov no tseem ceeb, yog li lawv tsiv AS yog. Zuag qhia tag nrho, qhov no tsuas yog ib qho nthuav cov cai uas piav qhia qhov tsis muaj txiaj ntsig ionice nyob rau hauv cov ntaub ntawv ntawm xws li ib tug configuration.

Ua tib zoo mloog tus chij iflag=direct rau DD. Peb siv ncaj qha IO hla dhau qhov tsis muaj cache kom tsis txhob hloov tsis tsim nyog ntawm IO buffers thaum nyeem ntawv. Txawm li cas los xij, oflag=direct peb tsis yog vim peb tau ntsib ZFS kev ua haujlwm teeb meem thaum siv nws.

Peb tau siv cov tswv yim no tau zoo rau ntau xyoo tsis muaj teeb meem.

Thiab ces nws pib... Peb pom tias ib qho ntawm cov nodes tsis tau rov qab lawm, thiab yav dhau los tau khiav nrog IOWAIT loj heev ntawm 50%. Thaum sim nkag siab tias vim li cas kev luam ntawv tsis tshwm sim, peb ntsib cov xwm txheej hauv qab no:

Volume group "images" not found

Peb pib xav txog "qhov kawg tau los rau Intel P4500," txawm li cas los xij, ua ntej muab lub server hloov pauv, nws tseem tsim nyog ua qhov thaub qab. Peb kho LVM2 los ntawm kev kho cov metadata los ntawm LVM2 thaub qab:

vgcfgrestore images

Peb launched ib tug thaub qab thiab pom no oil painting:

Ib zaug ntxiv peb tau tu siab heev - nws tau pom tseeb tias peb tsis tuaj yeem ua neej zoo li no, vim tias txhua tus VPSs yuav raug kev txom nyem, uas txhais tau tias peb yuav raug kev txom nyem ib yam nkaus. Dab tsi tshwm sim yog tsis meej kiag li - iostat pom IOPS IOPS thiab qhov siab tshaj IOWAIT. Tsis muaj cov tswv yim uas tsis yog "cia hloov NVMe," tab sis qhov kev pom tshwm sim nyob rau lub sijhawm.

Kev txheeb xyuas qhov xwm txheej ib kauj ruam zuj zus

Keeb kwm magazine. Ob peb hnub dhau los, ntawm tus neeg rau zaub mov no yuav tsum tsim VPS loj nrog 128 GB RAM. Zoo li muaj lub cim xeeb txaus, tab sis kom nyob rau sab nyab xeeb, peb faib lwm 32 GB rau kev sib pauv faib. VPS tau tsim, ua tiav nws txoj haujlwm tiav thiab qhov xwm txheej tsis nco qab, tab sis SWAP muab faib tseem nyob.

Configuration Nta. Rau tag nrho cov huab servers qhov parameter vm.swappiness tau teem rau default 60. Thiab SWAP tau tsim los ntawm SAS HDD RAID1.

Dab tsi tshwm sim (raws li cov neeg kho). Thaum thaub qab DD tsim ntau cov ntaub ntawv sau, uas tau muab tso rau hauv RAM buffers ua ntej sau rau NFS. System core, coj los ntawm txoj cai swappiness, tau tsiv ntau nplooj ntawv ntawm VPS nco mus rau thaj chaw sib pauv, uas nyob ntawm qhov qeeb HDD RAID1 ntim. Qhov no ua rau IOWAIT loj hlob heev, tab sis tsis yog vim IO NVMe, tab sis vim yog IO HDD RAID1.

Qhov teeb meem raug daws li cas. Lub 32GB kev sib pauv faib tau raug kaw. Qhov no siv sijhawm 16 teev; koj tuaj yeem nyeem nyias txog yuav ua li cas thiab yog vim li cas SWAP kaw qeeb heev. Cov chaw tau hloov lawm swappiness rau tus nqi sib npaug 5 tag nrho saum huab.

Yuav ua li cas qhov no tsis tshwm sim?. Ua ntej, yog SWAP nyob rau ntawm SSD RAID lossis NVMe ntaus ntawv, thiab qhov thib ob, yog tias tsis muaj NVMe ntaus ntawv, tab sis lub tshuab qeeb qeeb uas yuav tsis tsim cov ntaub ntawv ntim li no - ironically, qhov teeb meem tshwm sim vim tias NVMe nrawm dhau.

Tom qab ntawd, txhua yam pib ua haujlwm zoo li ua ntej - nrog xoom IOWAIT.

Tau qhov twg los: www.hab.com