ProHoster > Blog > Administration > LINSTOR Storage and Its Integration with OpenNebula

LINSTOR Storage and Its Integration with OpenNebula

LINSTOR Storage and Its Integration with OpenNebula

Recently, the team at LINBIT introduced their new SDS solution — Linstor. This completely open storage solution is based on proven technologies: DRBD, LVM, ZFS. Linstor combines simplicity with a well-designed architecture, allowing for stability and impressive results.

Today, I would like to talk about it in more detail and demonstrate how easily it can be integrated with OpenNebula using linstor_un — a new driver I developed specifically for this purpose.

Linstor, in combination with OpenNebula, allows for the creation of a fast and reliable cloud that can be deployed effortlessly on your own infrastructure.

Linstor Architecture

Linstor is neither a file system nor a block storage on its own; Linstor is an orchestrator that provides an abstraction layer enabling the automation of volume creation in LVM or ZFS and replicating them using DRBD9.

Breaking Stereotypes

But wait, DRBD? — Why automate it and how can this even work?

Let’s recall the past when DRBD8 was very popular. Its standard use involved creating one large block device and slicing it into many smaller pieces using LVM. A kind of mdadm RAID-1 but with network replication.

This approach has its drawbacks, and with the arrival of DRBD9, the principles of storage architecture changed; now, a separate DRBD device is created for each virtual machine.

The approach with independent block devices allows for better space utilization in the cluster and adds a number of additional features. For example, for each such device, you can define the number of replicas, their locations, and individual settings. They can be easily created/deleted, snapshots can be taken, sizes modified, encryption enabled, and much more. It's worth noting that DRBD9 also supports quorum, which helps avoid split-brain situations.

Resources and Backends

When creating a new block device, Linstor places the required number of replicas on different nodes within the cluster. Each such replica will be referred to as a DRBD resource.

Resources come in two types:

  • Data Resource — represent a DRBD device located on a node in an LVM or ZFS pool.
    Currently, there is support for several backends, and their number is constantly growing. Support is available for LVM, ThinLVM, and ZFS. The latter two allow for the creation and use of snapshots.
  • Diskless resource — represents a DRBD device located on a node without a backend, but allowing it to be treated like a regular block device; all read/write operations will be redirected to data resources. The closest equivalent to diskless resources is an iSCSI LUN.

Each DRBD resource can have up to 8 replicas, and only one of them can be active by default — Primary, all others will be Secondary and their use will be impossible as long as there is at least one Primary, meaning they will simply replicate data between themselves.

By mounting the DRBD device in the system, it automatically becomes Primary, thus even a Diskless resource, in DRBD terminology, can be Primary.

So why is Linstor needed?

By delegating all resource-intensive tasks to the kernel, Linstor essentially represents a regular Java application that allows for easy automation of DRBD resource creation.
Each resource created by it will represent an independent DRBD cluster that operates independently, regardless of the state of the control plane and other DRBD resources.

Linstor consists of just two components:

  • Linstor-controller — The main controller that provides an API for creating and managing resources. It also communicates with satellites, checking for available space on them, and sends tasks for creating and removing new resources. It runs in a single instance and uses a database that can be either internal (H2) or external (PostgreSQL, MySQL, MariaDB).
  • Linstor-satellite — Installed on all storage nodes, provides the controller with information about available space, and executes tasks received from the controller to create and delete new volumes and DRBD devices on top of them.

Linstor operates with the following key concepts:

  • Node — a physical server on which DRBD resources will be created and used.
  • Storage Pool — an LVM or ZFS pool created on a node where DRBD resources will be placed. A diskless pool is also possible — a pool that contains only diskless resources.
  • Resource Definition — Definition of a resource, essentially this is a prototype that describes the name and all its properties.
  • Volume Definition — Definition of a volume. Each resource can consist of multiple volumes, each volume must have a size.
  • Resource — A created instance of a block device, each resource must be placed on a specific node and in some storage pool.

Installing Linstor

I recommend using Ubuntu as the system, because there is a ready PPA:

add-apt-repository ppa:linbit/linbit-drbd9-stack
apt-get update

Or Debian, where Linstor can be installed from the official repository for Proxmox:

wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
PVERS=5 && echo "deb http://packages.linbit.com/proxmox/ proxmox-$PVERS drbd-9.0" > 
    /etc/apt/sources.list.d/linbit.list
apt-get update

Controller

It's quite simple here:

apt-get install linstor-controller linstor-client
systemctl enable linstor-controller
systemctl start linstor-controller

Storage nodes

Currently, an in-tree kernel module DRBD8, unfortunately, it is not suitable for us and we need to install DRBD9:

apt-get install drbd-dkms

As practice shows, most difficulties arise precisely because the DRBD8 module is loaded into the system, not DRBD9. Fortunately, this can be easily checked by running:

modprobe drbd
cat /proc/drbd

If you see version: 9 — then everything is fine, if version: 8 — then something went wrong and you need to take additional steps to find out why.

Now let's install linstor-satellite and drbd-utils:

apt-get install linstor-satellite drbd-utils
systemctl enable linstor-satellite
systemctl start linstor-satellite

Creating a cluster

Storage pools and nodes

As a backend, we will take ThinLVM, as it is the simplest and supports snapshots.
Install lvm2, if you haven’t done so yet, and let’s create a ThinLVM pool on all our storage nodes:

sudo vgcreate drbdpool /dev/sdb
sudo lvcreate -L 800G -T drbdpool/thinpool

All subsequent actions can be performed directly on the controller:

Let's add our nodes:

linstor node create node1 127.0.0.11
linstor node create node2 127.0.0.12
linstor node create node3 127.0.0.13

Let's create the storage pools:

linstor storage-pool create lvmthin node1 data drbdpool/thinpool
linstor storage-pool create lvmthin node2 data drbdpool/thinpool
linstor storage-pool create lvmthin node3 data drbdpool/thinpool

Now let's check the created pools:

linstor storage-pool list

If everything is done correctly, we should see something like:

+-------------------------------------------------------------------------------------------------------+
| StoragePool | Node  | Driver   | PoolName          | FreeCapacity | TotalCapacity | SupportsSnapshots |
|-------------------------------------------------------------------------------------------------------|
| data        | node1 | LVM_THIN | drbdpool/thinpool |       64 GiB |        64 GiB | true              |
| data        | node2 | LVM_THIN | drbdpool/thinpool |       64 GiB |        64 GiB | true              |
| data        | node3 | LVM_THIN | drbdpool/thinpool |       64 GiB |        64 GiB | true              |
+-------------------------------------------------------------------------------------------------------+

DRBD resources

Now let's try to create our new DRBD resource:

linstor resource-definition create myres
linstor volume-definition create myres 1G
linstor resource create myres --auto-place 2

Let's check the created resources:

linstor resource list 

+------------------------------------------------------------------------------------------------------+
| Node  | Resource | StoragePool | VolumeNr | MinorNr | DeviceName    | Allocated  | InUse  |    State |
|------------------------------------------------------------------------------------------------------|
| node1 | myres    | data        | 0        | 1084    | /dev/drbd1084 | 52 KiB     | Unused | UpToDate |
| node2 | myres    | data        | 0        | 1084    | /dev/drbd1084 | 52 KiB     | Unused | UpToDate |
+------------------------------------------------------------------------------------------------------+

Great! — we see that the resource was created on the first two nodes, we can also try to create a diskless resource on the third:

linstor resource create --diskless node3 myres

On the nodes, you will always find this device as /dev/drbd1084 or /dev/drbd/by-res/myres/0

This is how Linstor works, you can get more information from the official documentation..

Now I will explain how to integrate it with OpenNebula

Setting up OpenNebula

I won’t go into depth about the OpenNebula setup process, as all steps are detailed in the official documentation., which I recommend you refer to, I will only talk about integrating OpenNebula with Linstor.

linstor_un

To solve this task, I wrote my own driver — linstor_un, currently it is available as a plugin and should be installed separately.

The entire installation is done on the frontend OpenNebula nodes and does not require additional actions on the compute nodes.

First of all, we need to ensure that we have jq and linstor-client:

apt-get install jq linstor-client

The command linstor node list should output the list of nodes. All OpenNebula compute nodes must be added to the Linstor cluster.

Let's download and install the plugin:

curl -L https://github.com/OpenNebula/addon-linstor_un/archive/master.tar.gz | tar -xzvf - -C /tmp

mv /tmp/addon-linstor_un-master/vmm/kvm/* /var/lib/one/remotes/vmm/kvm/

mkdir -p /var/lib/one/remotes/etc/datastore/linstor_un
mv /tmp/addon-linstor_un-master/datastore/linstor_un/linstor_un.conf /var/lib/one/remotes/etc/datastore/linstor_un/linstor_un.conf

mv /tmp/addon-linstor_un-master/datastore/linstor_un /var/lib/one/remotes/datastore/linstor_un
mv /tmp/addon-linstor_un-master/tm/linstor_un /var/lib/one/remotes/tm/linstor_un

rm -rf /tmp/addon-linstor_un-master

Now we need to add it to the OpenNebula config, for which we perform the simple steps described here.

After which we will restart OpenNebula:

systemctl restart opennebula

And add our datastores, the system:

cat > system-ds.conf <<EOT
NAME="linstor-system"
TYPE="SYSTEM_DS"
STORAGE_POOL="data"
AUTO_PLACE="2"
CLONE_MODE="snapshot"
CHECKPOINT_AUTO_PLACE="1"
BRIDGE_LIST="node1 node2 node3"
TM_MAD="linstor_un"
EOT

onedatastore create system-ds.conf

And the images datastore:

cat > images-ds.conf <<EOT
NAME="linstor-images"
TYPE="IMAGE_DS"
STORAGE_POOL="data"
AUTO_PLACE="2"
BRIDGE_LIST="node1 node2 node3"
DISK_TYPE="BLOCK"
DS_MAD="linstor_un"
TM_MAD="linstor_un"
EOT

onedatastore create images-ds.conf

  • Parameter AUTO_PLACE indicates the number of data replicas that will be created for each new image in OpenNebula.
  • Parameter CLONE_MODE specifies how images will be cloned when creating new virtual machines, snapshot — will create a snapshot of the image and deploy the virtual machine from the snapshot, copy — will make a full copy of the image for every virtual machine.
  • In BRIDGE_LIST it is recommended to specify all nodes that will be used for performing image cloning operations.

A complete list of supported parameters is provided in README the project.

This concludes the configuration, and you can now load an appliance from the official OpenNebula Marketplace and create virtual machines from it.

Link to the project:
https://github.com/OpenNebula/addon-linstor_un

Source: habr.com

Buy reliable website hosting with DDoS protection, VPS VDS servers 🔥 Buy reliable website hosting with DDoS protection, VPS VDS servers | ProHoster