Are there any of us (Ceph users) who do not love 'professional extremes'?
Hardly β otherwise, we wouldn't be rolling around with this extremely interesting and fun product.
Many who have worked with Ceph have encountered one not very frequent (or rather, very infrequent) but sometimes in-demand case β connecting Ceph via iSCSI or FC. Why? Well, for example, to present an image from Ceph to a still unvirtualized Windows or Solaris server. Or to a virtualized server via a hypervisor that doesn't support Ceph β and there are many, as we know. For instance? Well, for instance, HyperV or ESXi, which are widely used. And if the task arises to present an image from Ceph to a guest machine, it becomes quite an intriguing challenge.
So, hereβs what we have:
- an already functioning Ceph cluster
- an existing image that needs to be presented via iSCSI
- The name of the pool mypool, the name of the image myimage
Ready to begin?
First of all, when we talk about FC or iSCSI, we encounter entities like the initiator and the target. The target is essentially the server, while the initiator is the client. Our task is to present the Ceph image to the initiator with minimal effort. This means we need to set up the target. But where, on which computer?
Fortunately, in the Ceph cluster, we have at least one component with a fixed IP address that has one of the most important Ceph components configured on it, and that component is the monitor. Therefore, we install the iSCSI target on the monitor (and the initiator as well, at least for testing). I did this on CentOS, but the solution will also work for any other distribution β just install the packages in a way that suits your distribution.
# yum -y install iscsi-initiator-utils targetcli
What is the purpose of the installed packages?
- targetcli β a utility for managing the SCSI target built into the Linux kernel
- iscsi-initiator-utils β a package with utilities used to manage the iSCSI initiator, which is also built into the Linux kernel
To submit an image via iSCSI to the initiator, there are two possible scenarios β use a userspace target backend or connect the image as a block device visible to the operating system and export it via iSCSI. We will take the second approach β the userspace backend is still in 'experimental' status and is slightly unprepared for production use. Additionally, there are pitfalls that can be discussed at length and (oh horror!) debated.
If we use a somewhat stable distribution with a long support cycle, then the kernel will be some ancient version. For example, in CentOS7 it's 3.10.*, in CentOS8 it's 4.19. We are interested in having at least kernel 5.3 (or likely 5.4) and newer. Why? Because by default, images in Ceph come with a set of options that are not compatible with older kernels. This means we need to include a repository with a new kernel for our distribution (for example, for CentOS it's elrepo), install the new kernel, and reboot the system to work with it:
- Connect to the selected monitor for the experiment
- Add the elrepo repositories as per the instructions β
- Install the kernel: yum -y --enablerepo=elrepo-kernel install kernel-ml
- Reboot the server with the monitor (we have three monitors, right?)
Connect the image as a block device
# rbd map mypool/myimage
/dev/rbd0
Now we just need to configure the target. In this example, I will configure the target in so-called demo mode β without authentication, visible and accessible to everyone. In a production environment, you would likely want to configure authentication β but that is a bit out of scope for today's just-for-fun exercise.
Create a backend named disk1 mapped to the file /dev/rbd/mypool/myimage. The specified file is an automatically created symbolic link by the udev daemon to /dev/rbd0. We use this symbolic link specifically because the rbd device name can change due to the order of how Ceph images are connected to the host.
Create the backend:
# targetcli /backstores/block create disk1 /dev/rbd/mypool/myimage
Create the iSCSI target:
# targetcli /iscsi create iqn.2020-01.demo.ceph:mypool
Connect the backend as a LUN to the target:
# targetcli /iscsi/iqn.2020-01.demo.ceph:mypool/tpg1/luns create /backstores/block/disk1
Fine-tune the target for demo mode:
# targetcli /iscsi/iqn.2020-01.demo.ceph:mypool/tpg1/ set
> attribute demo_mode_write_protect=0
# targetcli /iscsi/iqn.2020-01.demo.ceph:mypool/tpg1/ set
> attribute generate_node_acls=1
# targetcli /iscsi/iqn.2020-01.demo.ceph:mypool/tpg1/ set
> attribute cache_dynamic_acls=1
Save the configuration:
# targetcli saveconfig
Check for the target's availability:
# iscsiadm -m discovery -t st -p 127.0.0.1:3260
127.0.0.1:3260,1 iqn.2020-01.demo.ceph:mypool
Connect the target:
# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2020-01.demo.ceph:mypool, portal: 127.0.0.1,3260] (multiple)
Login to [iface: default, target: iqn.2020-01.demo.ceph:mypool, portal: 127.0.0.1,3260] successful.
If you've done everything correctly, a new disk will appear on the server that looks like a SCSI device but is actually an image from Ceph, which can be accessed through the iSCSI target. To avoid boot issues, it's best to remove the connected disk and the detected target from the local initiator:
# iscsiadm -m node --logout
# iscsiadm -m discoverydb -o delete -t st -p 127.0.0.1:3260
All that's left is to persist the configuration so that the image connects automatically and the target starts after the connection. Starting the target involves two steps β connecting the RBD and actually starting the target.
First, let's configure automatic connection of RBD images to the host. This is done by adding lines to the file /etc/ceph/rbdmap:
# cat /etc/ceph/rbdmap
# RbdDevice Parameters
mypool/myimage id=admin
# systemctl enable rbdmap
The restoration of the target configuration is a bit more complex β we need to write a unit for systemd that will restore the configuration:
# cat /usr/lib/systemd/system/scsi-target.service
[Unit]
Description=Start iSCSI target
After=network-online.target rbdmap.service
Before=remote-fs-pre.target
Wants=network-online.target remote-fs-pre.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/targetcli restoreconfig
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload
# systemctl enable scsi-target
The final test β let's reboot our monitor (which is now the iSCSI target) once again. It should be noted that if we hadn't cleared the initiator's database with the command iscsiadm -n discoverydb -o delete β¦ we might have ended up with a non-bootable or slow-booting server.
What remains?
Configure the initiator on the server where we want to provide the target.
How to ensure the high availability of our target?
Similarly, targets can be configured on other monitors to establish multipath (VMware understands this and will work fine, Hyper-V will not understand β it requires SCSI locking). Since the Ceph client does not use caching from the kernel, this is quite workable. Another option is to create a clustered resource from three components β a dedicated an IP address target and the rbdmap and scsi-target services, and manage this resource through clustering tools (who said pacemaker?)
Instead of a conclusion
As you can see, this article is somewhat of a joke β but I tried to quickly address several quite popular topics simultaneously using examples β iSCSI target, which does not necessarily have to export Ceph images β but can, for instance, export LVM volumes, the basics of working with the iSCSI initiator (how to scan the target, how to connect to the target, disconnect, remove the target entry from the database), writing your own unit for systemd, and some other topics.
I hope that even if you donβt go through this entire experiment in full, at least some part of this article will be useful to you.
Source: habr.com
