Lab: Setting up lvm, raid on linux

A small digression: this lr is synthetic.
Some tasks that are described here can be done much easier, but since the task of l / r is to get to know
with the functionality of raid, lvm, then some operations are artificially complicated.

Requirements for tools to perform LR:

• Virtualization tools such as Virtualbox
• Linux installation image, for example Debian9
• Internet access to download multiple packages
• Connect via ssh to the installed VM (optional)

ATTENTION

This laboratory work is related to such a delicate matter as data security - this is such an area,
which allows you to lose all your data due to the smallest mistake - one extra letter or number.
Because you're doing lab work, you're not in any danger, unless you have to start doing it all over again.
In real life, everything is much more serious, so you should enter the drive names very carefully, understanding
what exactly you are executing with the current command and what disks you are working with.

The second important point is the naming of disks and partitions: depending on the situation, disk numbers may differ
from the values ​​presented in the commands in the lab.
So, for example, if you remove the sda ​​drive from the array and then add a new drive, then the new drive will be displayed
on a system named sda. If you reboot before adding a new disk, then the new
the disk will be named sdb and the old one will be named sda

This lab should be run as superuser (root) because most of the commands require
elevated privileges and it doesn't make sense to constantly elevate privileges via sudo

Materials for study

• RAID
• LVM
• Disk naming in Linux OS
• What is a section
• What is a partition table and where is it stored
• What is grub

Utilities used

1) view disk information

• lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
• fdisk -l
  2) viewing information and working with LVM
• pvs
• pvextend
• pvcreate
• pvresize
• vgs
• vgreduce
• lvs
• liveextend
  3) viewing information and working with RAID
• cat /proc/mdstat
• Mdadm
  4) mount points
• mount
• umount
• cat / etc / fstab
• cat / etc / mtab
  5) disk repartitioning
• fdisk /dev/XXX
  6) copy partitions
• dd if=/dev/xxx of=/dev/yyy
  7) work with the partition table
• partx
• sfdisk
• mkfs.ext4
  8) work with the bootloader
• grub-install /dev/XXX
• update-grub
  9) misc
• lsof
• apt
• Rsync

Laboratory work consists of 3 parts:

• setting up a working system using lvm, raid
• emulation of a failure of one of the disks
• replacement of disks on the fly, with the addition of new disks and the transfer of partitions.

Task 1 (Installing OS and configuring LVM, RAID)

1) Create a new virtual machine, giving it the following characteristics:

• 1 gb ram
• 1 cpu
• 2 hdd (name them ssd1, ssd2 and assign equal size, check hot swap and ssd)
• SATA controller configured with 4 ports

2) Start installing Linux and when you reach the selection of hard drives, do the following:

• Partitioning method: manual, after which you should see this picture:
  

• Set up a separate partition under /boot: Select the first disk and create a new partition table on it

  • Partition size: 512M
  • Mount point: /boot
  • Repeat the setup for the second disk, but since you can’t mount /boot 2 times at the same time, select mount point: none as a result, you get the following (picture with a jamb, too lazy to redo it):
    

• RAID setup:

  • Select free space on the first disk and set the partition type to physical volume for RAID
  • Select "Done setting up the partition"
  • Repeat exactly the same setup for the second disk, resulting in the following:
    
  • Select "Configure software RAID"
  • Create MD device
  • Software RAID device type: Select a mirrored array
  • Active devices for the RAID XXXX array: Select both drives
  • Spare devices: Leave 0 as default
  • Active devices for the RAID XX array: select the partitions you created under the raid
  • Finish
  • As a result, you should get this picture:
    

• LVM Configuration: Select Configure the Logical Volume Manager

  • Keep current partition layout and configure LVM: Yes
  • Create volume group
  • Volume group name: system
  • Devices for the new volume group: Select your created RAID
  • Create logical volume
  • logical volume name: root
  • logical volume size: 25 of your disk size
  • Create logical volume
  • logical volume name: var
  • logical volume size: 25 of your disk size
  • Create logical volume
  • logical volume name: log
  • logical volume size: 15 of your disk size
  • By selecting Display configuration details you should get the following picture:
    
  • Once you have completed the LVM setup, you should see the following:
    

• Partition layout: select each volume created in LVM in turn and partition them, for example, for root like this:

  • Use as: ext4
  • mount point: /
  • the result of partitioning the root partition should look like this:
    
  • repeat the partitioning operation for var and log selecting the appropriate mount points (/var and /var/log manually enter), resulting in the following result:
    
  • Select Finish Partitioning
  • You will be asked a few questions about the fact that you have an unmounted partition and no swap configured. Both questions should be answered in the negative.

• The final result should look like this:
  
  3) Finish installing the OS by putting grub on the first device (sda) and boot the system.
  4) Copy the contents of the /boot partition from the sda ​​disk (ssd1) to the sdb disk (ssd2)

  dd if=/dev/sda1 of=/dev/sdb1

  5) Install grub on the second device:

• view disks in the system:

  fdisk -l
  lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

• List all the disks that the previous command gave you and describe what kind of disk it is

• Find the disk where grub was not installed and install this:
  grub-install /dev/sdb

• view information about the current raid with cat /proc/mdstat and write down what you see.

• look at the output of the commands: pvs, vgs, lvs, mount and write down exactly what you saw

Describe in your own words what you did and what result you got as a result of the task done.

After completing this task, it is recommended that you save a backup copy of the folder with the virtual machine, or make
vagrantbox: https://t.me/bykvaadm/191

Result: Virtual machine with disks ssd1, ssd2

Task 2 (Emulation of the failure of one of the disks)

1) If you check the hot swap box, then you can delete disks on the fly

• Delete disk ssd1 in machine properties
• Find the directory where your virtual machine files are stored and delete ssd1.vmdk
  2) Make sure your virtual machine is still running
  3) Reboot the virtual machine and make sure it still works
  4) check the status of the RAID array: cat /proc/mdstat
  5) add a new disk of the same size in the VM interface and name it ssd3
  6) follow the operations:
• see that a new disk has arrived in the system with the fdisk -l command
• copy the partition table from the old disk to the new one: sfdisk -d /dev/XXXX | sfdisk /dev/YYY
• see the result with fdisk -l
• Add a new drive to the raid array: mdadm --manage /dev/md0 --add /dev/YYY
• Look at the output: cat /proc/mdstat. You should see that synchronization has started.
  7) Now you need to manually synchronize the non-RAID partitions.
  To do this, we will use the dd utility, copying from a “live” disk to a new one that you recently installed

  dd if=/dev/XXX of=/dev/YYY

  8) Once sync is complete, install grub to the new drive
  9) Reboot the VM to make sure everything works
  Describe in your own words what you did and what result you got as a result of the task done.
  Result: Removed ssd1, kept ssd2, added ssd3.

  Task 3 (Adding new disks and moving the partition)

  This is the most difficult and voluminous task of all presented.
  Check very carefully what you are doing and with what disks and partitions.
  It is recommended that you make a copy before running it.
  This job is independent of Job #2, it can be done after Job #1 with disc names adjusted.
  The second part of the task of this laboratory should lead to exactly the same state that was after the first part.

  To make it easier for you to work, I can recommend that you do not physically remove disks from the host machine, but only
  disconnect them in the properties of the machine. From the point of view of the OS in the VM, it will look exactly the same, but you can
  in which case, connect the disk back and continue the work by rolling back a couple of points, if
  you're having problems. For example, you might have mishandled or forgotten to copy the /boot partition to the new drive.
  I can only advise you to double-check which disks and partitions you are working with several times, and even better
  write out on a piece of paper the correspondence of disks, partitions and the “physical” disk number. Beautiful and understandable tree
  team draws lsblk, use it as often as possible to analyze what you have done and what needs to be done.

  To history...

  Imagine that your server has been running for a long time on 2 ssd drives, when suddenly ...

  1) Simulate the failure of the ssd2 disk by removing the disk from the VM properties and rebooting
  2) View the current status of disks and RAID:

  cat /proc/mdstat
  fdisk -l
  lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

  3) You are lucky - the authorities allowed you to purchase several new disks:

  2 large SATA for the long overdue task of making a partition with logs on a separate disk

  2 SSD to replace the deceased, as well as to replace the still functioning one.

  Please note that the server basket only supports the installation of 4 disks at a time,
  so you can't add all the disks at once.

  The volume of the HDD is 2 times larger than the SSD.
  SSD volume select 1,25 times the former SSD.

  4) Add one new ssd disk, naming it ssd4, and after adding, check what happened:

  fdisk -l
  lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

  5) First of all, you should take care of the safety of the data of the old disk.
  This time we will be transferring data using LVM:

  • First of all, you need to copy the file table from the old disk to the new one:

    sfdisk -d /dev/XXX | sfdisk /dev/YYY

    Substitute the correct drives for x,y and see what this command does.

    Run the lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT command and compare its output with the previous call.
    What has changed?
    use the dd command to copy the /boot data to the new drive

    dd if=/dev/XXX of=/dev/YYY

    if /boot is still mounted on the old drive, it should be remounted to the live drive:

    mount | grep boot # смотрим куда смонтирован диск
    lsblk # смотрим какие диски есть в системе и смотрим есть ли диск, полученный из предыдущего пункта
    umount /boot # отмонтируем /boot
    mount -a # выполним монтирование всех точек согласно /etc/fstab. 
    # Поскольку там указана точка монтирования /dev/sda, то будет выполнено корректное перемонтирование на живой диск

    Install bootloader on new ssd drive

    grub-install /dev/YYY

    Why are we doing this operation?

    create a new raid array with only one new ssd disk included:

    mdadm --create --verbose /dev/md63 --level=1 --raid-devices=1 /dev/YYY

    The command above will not work without specifying a special key.
    Read the help and add this key to the command.

    Use the cat /proc/mdstat command to check the result of your operation. What has changed?
    Run the lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT command and compare its output with the previous call.
    What has changed?
    6) The next step is to configure LVM
    run the pvs command to view information about the current physical volumes
    create a new physical volume including the previously created RAID array:

    pvcreate /dev/md63

    Run the lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT command and compare its output with the previous call.
    What has changed?
    Run the pvs command again. What has changed?
    Increase the size of the Volume Group system with the following command:

    vgextend system /dev/md63

    Run the commands and write down what you saw and what changed.

    vgdisplay system -v
    pvs
    vgs
    lvs -a -o+devices

    On what physical disk are LV var,log,root currently located?

    Move the data from the old disk to the new one, substituting the correct device names.

    pvmove -i 10 -n /dev/system/root /dev/md0 /dev/md63 

    Repeat the operation for all logical volumes

    Run the commands and write down what you saw and what changed.

    vgdisplay system -v
    pvs
    vgs
    lvs -a -o+devices
    lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

    Let's change our VG by removing the disk of the old raid from it. Substitute the correct raid name.

    vgreduce system /dev/md0

    Run the commands and write down what you saw and what changed.

    lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
    pvs
    vgs

    For the sake of beauty, remount /boot to the second ssd drive (ssd4) and run lsblk. As a result, the ssd3 disk does not
    nothing should be mounted. Carefully check that the /boot partition is not empty! ls /boot should show
    multiple files and folders. Examine what is stored in this section and write down which file directory is responsible for what.
    7) remove the ssd3 disk and add ssd5, hdd1, hdd2 according to the above technical specifications, eventually getting:
    ssd4 - the first new ssd
    ssd5 - second new ssd
    hdd1 - first new hdd
    hdd2 - second new hdd

    8) Check what happened after adding disks:

    fdisk -l
    lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

    9) Restore the operation of the main raid array:

    • copy the partition table, substituting the correct disks:

      sfdisk -d /dev/XXX | sfdisk /dev/YYY

    • Note that when we copied the partition table from the old drive, the new size seemed to
      does not use the entire capacity of the hard drive.
      Therefore, we will soon need to resize this partition and expand the raid.
      See for yourself by typing the command:

      lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      10) copy boot partition /boot from ssd4 to ssd5

      dd if=/dev/XXX of=/dev/YYY

      11) Install grub on a new disk (ssd5)
      12) resize the second partition of the ssd5 disk

      run the utility for working with disk partitioning:

      fdisk /dev/XXX

      enter the d key to delete the existing partition (choose 2)
      enter the key n to create a new partition
      enter the p key to specify the partition type "primary"
      enter key 2 so that the new partition has a second number
      First sector: press enter to accept the automatically calculated size of the beginning of the partition
      Last sector: press enter to accept the automatically calculated size of the end of the section
      enter the key l to see a list of all possible partition types and find Linux raid auto in it
      enter the key t to change the type of partition created (2) and enter the number found in the previous step.
      enter w to write the change to disk.
      12) re-read the partition table and check the result

      partx -u /dev/XXX
      lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      add a new disk to the current raid array (don't forget to substitute the correct disks)

      mdadm --manage /dev/md63 --add /dev/sda2

      Let's expand the number of disks in our array to 2 pieces:

      mdadm --grow /dev/md63 --raid-devices=2

      Look at the result: we have 2 arrays marked up, but both sections included in this array have different sizes

      lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      13) increase the size of the partition on the ssd4 disk

      run the utility for working with disk partitioning:

      fdisk /dev/XXX

      enter the d key to delete the existing partition (choose 2)
      enter the key n to create a new partition
      enter the p key to specify the partition type "primary"
      enter key 2 so that the new partition has a second number
      First sector: press enter to accept the automatically calculated size of the beginning of the partition
      Last sector: press enter to accept the automatically calculated size of the end of the section
      At the end of the markup, select No to leave the signature of the section belonging to the array.
      enter w to write the change to disk.
      12) re-read the partition table and check the result

      partx -u /dev/XXX
      lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      note now sda2, sdc2 partitions are > than the size of the raid device.

      13) at this stage, the size of the raid can now be expanded

      mdadm --grow /dev/md63 --size=max
      lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT # check result

      View lsblk and note what has changed
      14) However, although we changed the size of the raid, the sizes of vg root,var,log did not change

      • see what the PV size is:

        pvs

      • expand the size of our PV:

        pvresize /dev/md63

      • see what the PV size is:

        pvs

        15) Add the newly appeared place VG var, root

        lvs # посмотрим сколько сейчас размечено
        lvextend -l +50%FREE /dev/system/root
        lvextend -l +100%FREE /dev/system/var
        lvs # проверьте что получилось

        At this point, you have completed the migration of the main array to the new disks. work with ssd1,ssd2 finished

        16) Our next task is to move /var/log to new disks, for this we will create a new array and lvm on hdd disks.

        • let's see what names the new hdd disks have

          fdisk -l

        • create a raid array

          mdadm --create /dev/md127 --level=1 --raid-devices=2 /dev/sdc /dev/sdd

        • create a new PV on the raid from large disks

          pvcreate data /dev/md127

        • create a group in this PV called data

          vgcreate data /dev/md127

        • create a logical volume the size of all free space and call it val_log

          lvcreate -l 100%FREE -n var_log data # lvs # посмотрим результат

        • format the created partition in ext4

          mkfs.ext4 /dev/mapper/data-var_log

        • let's see the result

          lsblk

          17) transfer log data from the old partition to the new one

          mount a temporary new log storage

          mount /dev/mapper/data-var_log /mnt

          sync partitions

          apt install rsync
          rsync -avzr /var/log/ /mnt/

          find out what processes are currently working with /var/log

          apt install lsof
          lsof | grep '/var/log'

          stop these processes

          systemctl stop rsyslog.service syslog.socket

          perform the final synchronization of partitions (those data that could have changed since the last synchronization)

          rsync -avzr /var/log/ /mnt/

          swap sections

          umount /mnt
          umount /var/log
          mount /dev/mapper/data-var_log /var/log

          check what happened

          lsblk

          18) Edit /etc/fstab
          fstab - a file in which the rules are written according to which partitions will be mounted upon boot
          our task is to find the line where /var/log is mounted and fix the device system-log on data-var_log

          19) The most important thing at this stage is not to forget to change the section table (ext4, for example). Since no matter how we change any raid, lvm - until the FS on the partition is notified that now the partition size has changed, we will not be able to use the new space. Use command resize2fs to change the FS.

          20) Final chord

          • let's do a reboot. If you did everything right, you will again get into your OS (this is necessary in order to make sure that everything works. This step does not make any sense except for self-testing)
          • check that everything we wanted to do was actually done:

            pvs
            lvs
            vgs
            lsblk
            cat /proc/mdstat

          21) [OPTIONAL] Follow the steps

          • reboot by pressing F12 to boot to different disks to make sure you can boot
            from any of the ssd drives, so that we are not afraid of the failure of one of them
          • you now have an unnecessary LV log in the VG system. Allocate this space between root or var, but instead of using
            designs 100%FREE specify the size by hand using the -L switch:

            -L 500M

          • fix the problem that /boot is on two partitions without synchronization, you don’t need to do this correctly,
            it's added here as an example. Don't forget to copy the contents of /boot somewhere first.
            • create a new raid and include sda1,sda2 in it
            • include these partitions in the existing raid and restore /boot in the main raid, but without mounting it.

Source: habr.com