🥇Laboratory Work: Setting up LVM, RAID on Linux

A Brief Digression: This lab work is synthetic.
Some tasks described here can be done much more easily, but since the purpose of the lab work is to familiarize with
the functionality of RAID and LVM, some operations are artificially complicated.

Requirements for the tools to complete the lab work:

Virtualization tools, such as VirtualBox
A Linux installation image, for example Debian 9
Internet access to download several packages
SSH connection to the installed VM (optional)

WARNING

This laboratory work is related to the delicate matter of data preservation—this is an area
that can lead to the loss of all your data due to the smallest mistake—one extra letter or digit.
Since you are conducting laboratory work, nothing is at stake, except you might have to start over.
In real life, things are much more serious, so you should be very careful when entering disk names, understanding
what exactly you are executing with the current command and which disks you are working with.

The second important point is disk and partition naming: depending on the situation, the disk numbers may differ
from the values presented in the commands in the lab work.
For example, if you remove disk sda from the array and then add a new disk, the new disk will appear
in the system with the name sda. However, if you reboot before adding the new disk, the new
disk will have the name sdb, and the old one will be named sda.

The laboratory work must be performed as a superuser (root) since most commands require
elevated privileges, and there is no point in constantly elevating privileges using sudo.

Materials for Study

RAID
LVM
Disk Naming in the Linux OS
What is a Partition?
What is a Partition Table and Where is it Stored?
What is GRUB?

Used Utilities

1) Viewing Disk Information

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
fdisk -l
2) Viewing Information and Working with LVM
pvs
pvextend
pvcreate
pvresize
vgs
vgreduce
lvs
lvextend
3) Viewing Information and Working with RAID
cat /proc/mdstat
mdadm
4) Mount Points
mount
umount
cat /etc/fstab
cat /etc/mtab
5) Resizing the Disk
fdisk /dev/XXX
6) Copying Partitions
dd if=/dev/xxx of=/dev/yyy
7) Working with the Partition Table
partx
sfdisk
mkfs.ext4
8) Working with the Bootloader
grub-install /dev/XXX
update-grub
9) Miscellaneous
lsof
apt
rsync

The lab work consists of 3 parts:

setting up a working system using LVM, RAID
simulating the failure of one of the disks
hot-swapping disks, adding new disks, and migrating partitions.

Task 1 (Installing the OS and configuring LVM, RAID)

1) Create a new virtual machine with the following specifications:

1 GB RAM
1 CPU
2 HDDs (name them ssd1, ssd2, assign equal size, check hot swap and SSD options)
The SATA controller is configured for 4 ports

2) Start the installation of Linux, and upon reaching the hard disk selection, do the following:

Partitioning method: manual, after which you should see the following view:
Setting up a separate partition for /boot: Select the first disk and create a new partition table on it
- Partition size: 512M
- Mount point: /boot
- Repeat the setup for the second disk, but since you cannot mount /boot twice simultaneously, choose mount point: none, resulting in the following (image with mistake, too lazy to redo):
RAID setup:
- Select free space on the first disk and set the partition type as physical volume for RAID
- Select 'Done setting up the partition'
- Repeat the exact same setup for the second disk, resulting in the following:
- Select 'Configure software RAID'
- Create MD device
- Software RAID device type: Select mirror array
- Active devices for the RAID XXXX array: Select both disks
- Spare devices: Leave 0 by default
- Active devices for the RAID XX array: select the partitions you created for RAID
- Finish
- Ultimately, you should see the following view:
LVM setup: Select 'Configure the Logical Volume Manager'
- Keep current partition layout and configure LVM: Yes
- Create volume group
- Volume group name: system
- Devices for the new volume group: Select your created RAID
- Create logical volume
- logical volume name: root
- logical volume size: 25 of your disk size
- Create logical volume
- logical volume name: var
- logical volume size: 25 of your disk size
- Create logical volume
- logical volume name: log
- logical volume size: 15 of your disk size
- Selecting 'Display configuration details' you should see the following view:
- After completing the LVM setup, you should see the following:
Partition layout: sequentially select each volume created in LVM and partition them, for example, for root like this:
- Use as: ext4
- mount point: /
- the result of partitioning the root partition should look like this:
- repeat the partitioning operation for var and log by choosing the appropriate mount points (/var and /var/log entered manually), resulting in the following:
- Select 'Finish Partitioning'
- You will be asked a few questions about the unmounted partition and whether swap is configured. You should respond negatively to both questions.
The final result should look like this:

3) Complete the OS installation by placing grub on the first device (sda) and boot the system.
4) Copy the contents of the /boot partition from disk sda (ssd1) to disk sdb (ssd2)
```
dd if=/dev/sda1 of=/dev/sdb1
```
5) Install grub on the second device:

check the disks in the system:

fdisk -l
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

List all the disks reported by the previous command and describe what each disk is.
Find the disk where grub was not installed and perform the installation:
grub-install /dev/sdb
view current raid information with the command cat /proc/mdstat and note what you see.
review the outputs of the commands: pvs, vgs, lvs, mount and note exactly what you saw

Describe in your own words what you did and what result you achieved from the completed task

After completing this task, it is recommended to save a backup of the folder with the virtual machine or create
vagrant box: https://t.me/bykvaadm/191

Result: Virtual machine with disks ssd1, ssd2

Task 2 (Simulating the failure of one of the disks)

1) If you checked the hot swap option, you can remove disks on the fly

Remove disk ssd1 in the machine properties
Find the directory where your virtual machine files are stored and delete ssd1.vmdk
2) Ensure that your virtual machine is still running
3) Restart the virtual machine and confirm that it is still running
4) check the status of the RAID array: cat /proc/mdstat
5) add a new disk of the same size in the VM interface and name it ssd3
6) perform operations:
check that the new disk has appeared in the system with the command fdisk -l
copy the partition table from the old disk to the new one: sfdisk -d /dev/XXXX | sfdisk /dev/YYY
check the result with the command fdisk -l
Add the new disk to the raid array: mdadm —manage /dev/md0 —add /dev/YYY
Check the result: cat /proc/mdstat. You should see that synchronization has started
7) Now manually synchronize the partitions that are not included in the RAID.
To do this, use the dd utility, copying from the ‘live’ disk to the new one you just installed
```
dd if=/dev/XXX of=/dev/YYY
```
8) After synchronization is complete, install grub on the new disk
9) Restart the VM to ensure everything is working.
Describe in your own words what you did and what result you achieved from the completed task
Result: Disk ssd1 removed, disk ssd2 retained, disk ssd3 added.
Task 3 (Adding new disks and moving a partition)
This is the most complex and extensive task of all presented.
Carefully check what you are doing and with which disks and partitions.
It is recommended to make a backup before performing this.
This task is independent of task No. 2; it can be performed after task No. 1 with adjustments to the disk names.
The second part of this laboratory task should restore the exact state that was after completing the first part.
To make it easier for you to work, I recommend not physically removing disks from the host machine, but just
disconnecting them in the machine properties. From the OS's perspective in the VM, it will look absolutely the same, but you will be able to
reconnect the disk later if needed and continue the work from a few steps back, in case you encounter problems.
For example, you might have executed incorrectly or forgotten to copy the /boot partition to the new disk.
I can only advise you to double-check several times which disks and partitions you are working with, and even better,
write down on a piece of paper the correspondence of disks, partitions, and the 'physical' disk number. A neat and clear tree
is drawn by the command lsblk, use it as often as possible to analyze what you have done and what needs to be done.
To the story…
Imagine that your server has been running for a long time on 2 SSDs, and suddenly…
1) Simulate the failure of disk ssd2 by removing the disk from the VM properties and rebooting
2) Check the current state of disks and RAID:
```
cat /proc/mdstat
fdisk -l
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
3) You are lucky — management allowed the purchase of several new disks:
2 large capacity SATA for a long-awaited task of moving the logs partition to a separate disk
2 SSDs to replace the failed one, as well as to replace the still functioning one.
It should be noted that the basket server supports the installation of only 4 disks at the same time,
so you cannot add all the disks at once.
Choose the HDD capacity to be 2 times larger than the SSD.
Choose the SSD capacity to be 1.25 times larger than the previous SSDs.
4) Add one new SSD disk, naming it ssd4, and after addition check what happened:
```
fdisk -l
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
5) First and foremost, we need to ensure the safety of the data on the old disk.
This time we will transfer the data using LVM:
- first, we need to copy the file table from the old disk to the new one:
```
sfdisk -d /dev/XXX | sfdisk /dev/YYY
```
  Replace x,y with the correct disks and analyze what this command does.
  Run the command lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT and compare its output with the previous call.
  What has changed?
  use the dd command to copy the /boot data to the new disk
```
dd if=/dev/XXX of=/dev/YYY
```
  if /boot is still mounted on the old disk, it should be remounted to the live disk:
```
mount | grep boot # check where the disk is mounted
lsblk # see what disks are in the system and check if the disk obtained in the previous step is present
umount /boot # unmount /boot
mount -a # mount all points according to /etc/fstab.
# Since the mount point /dev/sda is specified there, it will correctly remount on the live disk
```
  Install the bootloader on the new SSD disk
```
grub-install /dev/YYY
```
  Why are we performing this operation?
  create a new RAID array including only the new SSD disk:
```
mdadm --create --verbose /dev/md63 --level=1 --raid-devices=1 /dev/YYY
```
  The command above will not work without specifying a special key.
  Read the documentation and add this key to the command.
  Use the command cat /proc/mdstat to check the result of your operation. What has changed?
  Run the command lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT and compare its output with the previous call.
  What has changed?
  6) The next step is to configure LVM
  run the pvs command to view information about the current physical volumes
  create a new physical volume by including the previously created RAID array:
```
pvcreate /dev/md63
```
  Run the command lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT and compare its output with the previous call.
  What has changed?
  Run the pvs command again. What has changed?
  We will increase the size of the Volume Group 'system' with the following command:
```
vgextend system /dev/md63
```
  Execute the commands and note what you saw and what has changed.
```
vgdisplay system -v
pvs
vgs
lvs -a -o+devices
```
  On which physical disk are the LV var, log, root currently located?
  Move the data from the old disk to the new one, substituting the correct device names.
```
pvmove -i 10 -n /dev/system/root /dev/md0 /dev/md63 
```
  Repeat the operation for all logical volumes
  Execute the commands and note what you saw and what has changed.
```
vgdisplay system -v
pvs
vgs
lvs -a -o+devices
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
  We will modify our VG by removing the disk of the old RAID. Substitute the correct RAID name.
```
vgreduce system /dev/md0
```
  Execute the commands and note what you saw and what has changed.
```
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
pvs
vgs
```
  To enhance the setup, remount /boot on the second SSD disk (ssd4) and run lsblk. As a result, ssd3 will not be on the disk.
  nothing should be mounted. Please carefully check that the /boot partition is not empty! ls /boot should show
  several files and folders. Investigate what is stored in this partition and note which file/directory is responsible for what.
  7) remove the ssd3 disk and add ssd5, hdd1, hdd2 according to the specifications above, resulting in:
  ssd4 — the first new ssd
  ssd5 — the second new ssd
  hdd1 — the first new hdd
  hdd2 — the second new hdd
  8) Check what happened after adding the disks:
```
fdisk -l
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
  9) Restore the operation of the main RAID array:
  - execute the copying of the partition table, substituting the correct disks:
```
sfdisk -d /dev/XXX | sfdisk /dev/YYY
```
  - Note that when we copied the partition table from the old disk, it showed that the new size
    does not use the entire disk space.
    Therefore, we will need to change the size of this partition and expand the RAID soon.
    Check this yourself by entering the command:
```
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
    10) copy the boot partition /boot from disk ssd4 to ssd5
```
dd if=/dev/XXX of=/dev/YYY
```
    11) install grub on the new disk (ssd5)
    12) let's change the size of the second partition of disk ssd5
    launch the utility for working with disk partitions:
```
fdisk /dev/XXX
```
    enter key d to delete the existing partition (choose 2)
    enter key n to create a new partition
    enter key p to specify the partition type as 'primary'
    enter key 2 so that the new partition has the second number
    First sector: press enter to agree with the automatically calculated size of the beginning of the partition
    Last sector: press enter to agree with the automatically calculated size of the end of the partition
    enter key l to see a list of all possible partition types and find 'Linux raid auto' in it
    enter key t to change the type of the created partition (2) and enter the number found in the previous step.
    enter key w to write the changes to the disk.
    12) reread the partition table and check the result
```
partx -u /dev/XXX
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
    add the new disk to the current RAID array (don't forget to substitute the correct disks)
```
mdadm --manage /dev/md63 --add /dev/sda2
```
    We will increase the number of disks in our array to 2:
```
mdadm --grow /dev/md63 --raid-devices=2
```
    Check the result: we have two arrays allocated, but both partitions included in this array have different sizes
```
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
    13) increase the size of the partition on disk ssd4
    launch the utility for working with disk partitions:
```
fdisk /dev/XXX
```
    enter key d to delete the existing partition (choose 2)
    enter key n to create a new partition
    enter key p to specify the partition type as 'primary'
    enter key 2 so that the new partition has the second number
    First sector: press enter to agree with the automatically calculated size of the beginning of the partition
    Last sector: press enter to agree with the automatically calculated size of the end of the partition
    At the end of the markup, select No to keep the section signature belonging to the array.
    enter key w to write the changes to the disk.
    12) reread the partition table and check the result
```
partx -u /dev/XXX
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
```
    Note that now the sda2 and sdc2 partitions are larger than the RAID device size.
    13) At this stage, the RAID size can now be expanded.
```
mdadm --grow /dev/md63 --size=max
lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT # check result
```
    Review lsblk and note what has changed.
    14) However, although we have changed the RAID size, the sizes of vg root, var, log have not changed.
    - Check the size of the PV:
```
pvs
```
    - Let's expand the size of our PV:
```
pvresize /dev/md63
```
    - Check the size of the PV:
```
pvs
```
      15) Let's add the newly available space to VG var, root.
```
lvs # check how much is currently allocated
lvextend -l +50%FREE /dev/system/root
lvextend -l +100%FREE /dev/system/var
lvs # check the results
```
      At this stage, you have completed the migration of the main array to the new disks. Work with ssd1 and ssd2 is completed.
      16) Our next task is to move /var/log to the new disks. For this, we will create a new array and LVM on HDD disks.
      - Let's check what names the new HDD disks have.
        fdisk -l
      - Create a RAID array.
        mdadm --create /dev/md127 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
      - Let's create a new PV on the RAID from the large disks.
        pvcreate data /dev/md127
      - Let's create a group in this PV named data.
        vgcreate data /dev/md127
      - Let's create a logical volume with the size of all free space and name it var_log.
        lvcreate -l 100%FREE -n var_log data # lvs # check the result
      - Format the created partition to ext4.
        mkfs.ext4 /dev/mapper/data-var_log
      - See the result.
        lsblk
        17) Transfer the log data from the old partition to the new one.
        Temporarily mount the new log storage.
        mount /dev/mapper/data-var_log /mnt
        Perform synchronization of partitions.
        apt install rsync rsync -avzr /var/log/ /mnt/
        Determine which processes are currently working with /var/log.
        apt install lsof lsof | grep '/var/log'
        Stop these processes.
        systemctl stop rsyslog.service syslog.socket
        Perform final synchronization of partitions (the data that may have changed since the last synchronization).
        rsync -avzr /var/log/ /mnt/
        Swap the partitions.
        umount /mnt umount /var/log mount /dev/mapper/data-var_log /var/log
        Check the result.
        lsblk
        18) Edit /etc/fstab.
        fstab is a file that records the rules by which partitions will be mounted on boot.
        Our task is to find the line where /var/log is mounted and correct the device. system-log to data-var_log
        19) The most important thing at this stage is to remember to change the partition table (ext4, for example). No matter how much we change different RAID and LVM settings, as long as the file system on the partition is not notified that its size has changed, we will not be able to use the new space. Use the command resize2fs to change the file system.
        20) The final touch
        
        is to reboot. If you've done everything correctly, you'll return to your OS (this is to ensure that everything works. This step serves no purpose other than self-checking)
        
        perform checks to ensure that everything we intended to do has indeed been accomplished:
        pvs lvs vgs lsblk cat /proc/mdstat
        21) [OPTIONAL] Take actions
        
        reboot by pressing F12 to select different disks during startup, to verify that you can boot
        from any of the SSDs, so that we do not fear the failure of any one of them.
        
        now you have an unnecessary LV log in the VG system. Allocate this space between root or var, but instead of using
        the 100%FREE option, specify the size manually with the -L option:
        -L 500M
        
        resolve the issue of /boot being on two partitions without synchronization; this should not be done correctly,
        this is added here for example. Do not forget to back up the contents of /boot somewhere first.
        
        create a new RAID and include sda1, sda2 in it.
        
        include these partitions in the existing RAID and restore /boot in the main RAID, but without mounting it.