ProHoster > Blog > Administration > Launching Jupyter into LXD Orbit

Launching Jupyter into LXD Orbit

Have you ever experimented with code or system utilities on Linux in a way that doesn't shake the base system and tear it all apart when a code error occurs that should run with root privileges?

But what about the fact that, say, you need to test or run a whole cluster of various microservices on one machine? A hundred or even a thousand?

With virtual machines managed by a hypervisor, such tasks can and will be solved, but at what cost? For example, a container in LXD based on the Alpine Linux distribution minimally consumes only 7.60MB RAM, and where the root partition after startup occupies 9.5MB! How do you like that, Elon Musk? I recommend you check out basic features of LXD - container systems in Linux

After it became generally clear what LXD containers are, let's go further and think about what if there was such a combine platform where one could safely run code for the host, generate graphs, dynamically (interactively) link UI- widgets with your code, complete the code with blackjack text… formatting? Some kind of interactive blog? Wow... I want it! Want! πŸ™‚

Look under the cat where we will launch in a container jupyter lab - the next generation of the user interface instead of the outdated Jupyter Notebook, and also install Python modules such as NumPy, pandas, Matplotlib, IPyWidgets which will allow you to do all of the above and save it all in a special file - an IPython laptop.

Launching Jupyter into LXD Orbit

Take-off plan ^

Launching Jupyter into LXD Orbit

We throw in a short action plan to make it easier for us to implement the scheme above:

  • Install and run the container based on the distribution Alpine Linux. We will use this distribution kit as it is aimed at minimalism and we will install only the most necessary software into it, nothing more.
  • Let's add an additional virtual disk in the container to which we will give a name - hostfs and mount to the root FS. This disk will make it possible to use files on the host from a given directory inside the container. Thus, the data will be independent of the container. If the container is deleted, the data will remain on the host. Also, this scheme is useful for sharing the same data between many containers without using the regular networking mechanisms of the container distribution.
  • Install Bash, sudo, necessary libraries, add and configure a system user
  • Install Python, modules and compile binary dependencies for them
  • Install and run jupyter lab, customize the appearance, install extensions for it.

In this article, we will start by launching the container, we will not consider installing and configuring LXD, you can find all this in another article - Core Features of LXD - Linux Container Systems.

Installing and configuring the base system ^

We create a container with a command in which we specify the image - alpine3, identifier for the container - jupyterlab and optionally configuration profiles:

lxc init alpine3 jupyterlab --profile=default --profile=hddroot

Here I am using a config profile hddroot which tells to create a container with a root partition in storage pool located on the physical HDD:

lxc profile show hddroot

config: {}
description: ""
devices:
  root:
    path: /
    pool: hddpool
    type: disk
name: hddroot
used_by: []
lxc storage show hddpool

config:
  size: 10GB
  source: /dev/loop1
  volatile.initial_source: /dev/loop1
description: ""
name: hddpool
driver: btrfs
used_by:
- /1.0/images/ebd565585223487526ddb3607f5156e875c15a89e21b61ef004132196da6a0a3
- /1.0/profiles/hddroot
status: Created
locations:
- none

This gives me the opportunity to experiment with containers on the HDD, saving the resources of the SSD disk that I also have in my system πŸ™‚ for which I have created a separate configuration profile ssdroot.

After the container is created, it is in the state STOPPED, so we need to start it by running the init system in it:

lxc start jupyterlab

Display a list of containers in LXD using the key -c which indicates which columns to display:

lxc list -c ns4b
+------------+---------+-------------------+--------------+
|    NAME    |  STATE  |       IPV4        | STORAGE POOL |
+------------+---------+-------------------+--------------+
| jupyterlab | RUNNING | 10.0.5.198 (eth0) | hddpool      |
+------------+---------+-------------------+--------------+

When creating the container, the IP address was chosen randomly, since we used a configuration profile default which was previously configured in the article Core Features of LXD - Linux Container Systems.

We will change this IP address to a more memorable one by creating a network interface at the container level, and not at the configuration profile level, as it is now in the current configuration. This is not required, you can skip it.

Creating a network interface eth0 which we link with the switch (network bridge) lxdbr0 in which we enabled NAT according to the last article and the container will now have access to the Internet, and we also assign a static IP address to the interface - 10.0.5.5:

lxc config device add jupyterlab eth0 nic name=eth0 nictype=bridged parent=lxdbr0 ipv4.address=10.0.5.5

After adding the device, the container must be restarted:

lxc restart jupyterlab

Checking the status of the container:

lxc list -c ns4b
+------------+---------+------------------+--------------+
|    NAME    |  STATE  |       IPV4       | STORAGE POOL |
+------------+---------+------------------+--------------+
| jupyterlab | RUNNING | 10.0.5.5 (eth0)  | hddpool      |
+------------+---------+------------------+--------------+

Basic software installation and system setup ^

To administer our container, you need to install the following software:

Package
Description

bash
The GNU Bourne Again shell

bash completion
Programmable completion for the bash shell

sudo
Give certain users the ability to run some commands as root

shadow
Password and account management tool suite with support for shadow files and PAM

tzdata
Sources for time zone and daylight saving time data

dwarf
Pico editor clone with enhancements

Additionally, you can install support in the man-pages system by installing the following packages βˆ’ man man-pages mdocml-apropos less

lxc exec jupyterlab -- apk add bash bash-completion sudo shadow tzdata nano

Let's analyze the commands and keys that we used:

  • lxc - Calling LXD client
  • exec - LXD client method that runs a command in a container
  • jupyterlab β€” Container ID
  • -- β€” A special key that specifies not to interpret further keys as keys for lxc and pass the rest of the string as is to the container
  • apk - Alpine Linux distribution package manager
  • add - A package manager method that installs the packages specified after the command

Next, set up a time zone in the system Europe/Moscow:

lxc exec jupyterlab -- cp /usr/share/zoneinfo/Europe/Moscow /etc/localtime

After setting the time zone, the package tzdata is no longer needed in the system, it will take up space, therefore, delete it:

lxc exec jupyterlab -- apk del tzdata

Checking the timezone:

lxc exec jupyterlab -- date

Wed Apr 15 10:49:56 MSK 2020

In order not to spend a lot of time setting up Bash for new users in the container, the following steps will copy ready-made skel files from the host system into it. This will allow you to decorate Bash in a container in an interactive way. My host system is Manjaro Linux and copied files /etc/skel/.bash_profile, /etc/skel/.bashrc, /etc/skel/.dir_colors in principle, they are suitable for Alpine Linux and do not cause critical problems, but you may have a different distribution and you need to figure it out yourself in case of an error running Bash in a container.

Copy the skel files to the container. Key --create-dirs will create the necessary directories if they don't exist:

lxc file push /etc/skel/.bash_profile jupyterlab/etc/skel/.bash_profile --create-dirs
lxc file push /etc/skel/.bashrc jupyterlab/etc/skel/.bashrc
lxc file push /etc/skel/.dir_colors jupyterlab/etc/skel/.dir_colors

For an already existing root user, copy the skel files just copied into the container to the home directory:

lxc exec jupyterlab -- cp /etc/skel/.bash_profile /root/.bash_profile
lxc exec jupyterlab -- cp /etc/skel/.bashrc /root/.bashrc
lxc exec jupyterlab -- cp /etc/skel/.dir_colors /root/.dir_colors

Alpine Linux installs a system shell for users /bin/sh, we will replace it with root user in Bash:

lxc exec jupyterlab -- usermod --shell=/bin/bash root

That root the user was not passwordless, he needs to set a password. The following command will generate and set a new random password for it, which you will see on the console screen after it is executed:

lxc exec jupyterlab -- /bin/bash -c "PASSWD=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 12); echo "root:$PASSWD" | chpasswd && echo "New Password: $PASSWD""

New Password: sFiXEvBswuWA

Also, let's create a new system user - jupyter for which we will later configure jupyter lab:

lxc exec jupyterlab -- useradd --create-home --shell=/bin/bash jupyter

Generate and set a password for it:

lxc exec jupyterlab -- /bin/bash -c "PASSWD=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 12); echo "jupyter:$PASSWD" | chpasswd && echo "New Password: $PASSWD""

New Password: ZIcbzWrF8tki

Next, run two commands, the first will create a system group sudo, and the second one will add the user to it jupyter:

lxc exec jupyterlab -- groupadd --system sudo
lxc exec jupyterlab -- groupmems --group sudo --add jupyter

See what groups the user belongs to jupyter:

lxc exec jupyterlab -- id -Gn jupyter

jupyter sudo

All right, let's move on.

Allow all users who are members of the group sudo use command sudo. To do this, run the following script, where sed uncomment the parameter line in the configuration file /etc/sudoers:

lxc exec jupyterlab -- /bin/bash -c "sed --in-place -e '/^#[ t]*%sudo[ t]*ALL=(ALL)[ t]*ALL$/ s/^[# ]*//' /etc/sudoers"

Installing and configuring JupyterLab ^

jupyter lab is a Python application, so we must first install this interpreter. Also, jupyter lab we will install using python package manager pip, and not the system one, because in the system repository it may be outdated, and therefore, we must manually resolve the dependencies for it by installing the following packages βˆ’ python3 python3-dev gcc libc-dev zeromq-dev:

lxc exec jupyterlab -- apk add python3 python3-dev gcc libc-dev zeromq-dev

Update python modules and package manager pip to the current version:

lxc exec jupyterlab -- python3 -m pip install --upgrade pip setuptools wheel

Set jupyter lab via package manager pip:

lxc exec jupyterlab -- python3 -m pip install jupyterlab

Since the extensions in jupyter lab are experimental and are not officially shipped with the jupyterlab package, so we have to install and configure it manually.

Install NodeJS and the package manager for it - NPM, since jupyter lab uses them for its extensions:

lxc exec jupyterlab -- apk add nodejs npm

To extensions for jupyter lab which we will install worked, they need to be installed in the user directory, since the application will be launched from the user jupyter. The problem is that there is no parameter in the launch command that can be passed a directory, the application only accepts an environment variable and therefore we must define it. To do this, we will write a variable export command JUPYTERLAB_DIR in the user's environment jupyter, to file .bashrc, which is executed every time the user logs in:

lxc exec jupyterlab -- su -l jupyter -c "echo -e "nexport JUPYTERLAB_DIR=$HOME/.local/share/jupyter/lab" >> .bashrc"

The next command will install a special extension - the extension manager in jupyter lab:

lxc exec jupyterlab -- su -l jupyter -c "export JUPYTERLAB_DIR=$HOME/.local/share/jupyter/lab; jupyter labextension install --no-build @jupyter-widgets/jupyterlab-manager"

Now everything is ready for the first launch jupyter lab, but we can still install some useful extensions:

  • toc - Table of Contents, generates a list of headings in an article/notebook
  • jupyterlab-horizon-theme - UI Theme
  • jupyterlab_neon_theme - UI Theme
  • jupyterlab-ubu-theme - Another one theme from the author of this article πŸ™‚ But in this case, the installation from the GitHub repository will be shown

So, run the following commands in sequence to install these extensions:

lxc exec jupyterlab -- su -l jupyter -c "export JUPYTERLAB_DIR=$HOME/.local/share/jupyter/lab; jupyter labextension install --no-build @jupyterlab/toc @mohirio/jupyterlab-horizon-theme @yeebc/jupyterlab_neon_theme"
lxc exec jupyterlab -- su -l jupyter -c "wget -c https://github.com/microcoder/jupyterlab-ubu-theme/archive/master.zip"
lxc exec jupyterlab -- su -l jupyter -c "unzip -q master.zip && rm master.zip"
lxc exec jupyterlab -- su -l jupyter -c "export JUPYTERLAB_DIR=$HOME/.local/share/jupyter/lab; jupyter labextension install --no-build jupyterlab-ubu-theme-master"
lxc exec jupyterlab -- su -l jupyter -c "rm -r jupyterlab-ubu-theme-master"

After installing the extensions, we must compile them, since earlier, during installation, we specified the key --no-build to save time. Now we can speed up a lot by compiling them together in one go:

lxc exec jupyterlab -- su -l jupyter -c "export JUPYTERLAB_DIR=$HOME/.local/share/jupyter/lab; jupyter lab build"

Now run the following two commands for the first run jupyter lab. It would be possible to launch it with one command, but in this case, the launch command, which is hard to keep in mind, will be remembered by bash in the container, and not on the host, where there are already enough commands to write them to history πŸ™‚

Log in to the container as a user jupyter:

lxc exec jupyterlab -- su -l jupyter

Next run jupyter lab with keys and parameters as given:

[jupyter@jupyterlab ~]$ jupyter lab --ip=0.0.0.0 --no-browser

Navigate in a web browser to http://10.0.5.5:8888 and on the opened page enter token access that you will see in the console. Copy it and paste it on the page, then click Login. After logging in, go to the extensions menu on the left, as shown in the figure below, where you will be prompted, when activating the extension manager, to accept the security risks of installing extensions from third parties for which the command JupyterLab development is not responsible for:

Launching Jupyter into LXD Orbit

However, we will isolate the entire jupyter lab and put it in a container so that third-party extensions that require and use NodeJS cannot at least steal data on disk other than what we open inside the container. Get to your private documents on the host in /home processes from the container are unlikely to succeed, and if they do, then you need to have file privileges on the host system, since we run the container in non-privileged mode. Based on this information, you can assess the risk of including extensions in jupyter lab.

Created IPython notebooks (pages in jupyter lab) will now be created in the user's home directory - /home/jupyter, but in our plans to share data (share) between the host and the container, therefore, return to the console and stop jupyter lab by executing hotkey - CTRL+C and answering y on request. Then terminate the user's interactive session jupyter by performing a hotkey CTRL+D.

Sharing data with the host ^

To share data with the host, you need to create a device in the container that allows you to do this, and for this, run the following command where we specify the following keys:

  • lxc config device add - Command adds device configuration
  • jupyter β€” Identifier of the container to which the configuration is added
  • hostfs β€” Device ID. You can set any name.
  • disk - Device type is indicated
  • path - Specifies the path in the container to which LXD will mount this device
  • source - The source is specified, the path to the directory on the host that you want to share with the container. Specify the path according to your preference
lxc config device add jupyterlab hostfs disk path=/mnt/hostfs source=/home/dv/projects/ipython-notebooks

For catalog /home/dv/projects/ipython-notebooks the permission must be set to the container user which currently has a UID equal to SubUID + UID, see chapter Safety. Container Privileges Article Core Features of LXD - Linux Container Systems.

Set the permission on the host where the container user will be the owner jupyter, and the variable $USER will list your host user as a group:

sudo chown 1001000:$USER /home/dv/projects/ipython-notebooks

Hello World! ^

If you still have a console session open in the container with jupyter labthen restart it with a new key --notebook-dir by setting the value /mnt/hostfs as the path to the root of the notebooks in the container for the device we created in the previous step:

jupyter lab --ip=0.0.0.0 --no-browser --notebook-dir=/mnt/hostfs

Then go to the page http://10.0.5.5:8888 and create your first laptop by clicking the button on the page as shown in the picture below:

Launching Jupyter into LXD Orbit

Then, in the field on the page, enter the Python code that will output the classic Hello World!. When finished entering, press CTRL+ENTER or the "play" button on the top toolbar to have JupyterLab do it:

Launching Jupyter into LXD Orbit

This is almost all ready to use, but it will not be interesting if we do not install additional Python modules (full-fledged applications) that allow us to significantly extend the standard features of Python in jupyter labso let's move on πŸ™‚

PS The interesting thing is that the old implementation Jupyter under the codename Jupyter Notebook has not gone away and it exists in parallel with jupyter lab. To switch to the old version, follow the link by adding a suffix to the address/tree, and the transition to the new version is carried out with the suffix /lab, but it doesn't have to be specified:

Extending Python ^

In this section, we will install such powerful Python language modules as NumPy, pandas, Matplotlib, IPyWidgets the results of which are integrated into laptops jupyter lab.

Before installing the listed Python modules through the package manager pip we must first resolve system dependencies in Alpine Linux:

  • g++ - Needed to compile modules, as some of them are implemented in the language C++ and are connected to Python at runtime as binary modules
  • freetype-dev - dependency for Python module Matplotlib

Install dependencies:

lxc exec jupyterlab -- apk add g++ freetype-dev

There is one problem, in the current state of the Alpine Linux distribution, it will not work to compile a new version of NumPy, a compilation error will fly out that I could not resolve:

ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly

Therefore, we will install this module as a system package that distributes an already compiled version, but a little older than what is currently available on the site:

lxc exec jupyterlab -- apk add py3-numpy py3-numpy-dev

Next, install Python modules through the package manager pip. Be patient as some modules will compile and take a few minutes. Compilation took ~15 minutes on my machine:

lxc exec jupyterlab -- python3 -m pip install pandas ipywidgets matplotlib

Clear installation caches:

lxc exec jupyterlab -- rm -rf /home/*/.cache/pip/*
lxc exec jupyterlab -- rm -rf /root/.cache/pip/*

Testing modules in JupyterLab ^

If you are running jupyter lab, restart it so that the newly installed modules are activated. To do this, in the console session, press CTRL+C where you have it running and enter y to stop request and then restart jupyter lab by pressing the arrow on the keyboard "up" so as not to enter the command again and then Enter to start it:

jupyter lab --ip=0.0.0.0 --no-browser --notebook-dir=/mnt/hostfs

Go to the page http://10.0.5.5:8888/lab or refresh the page in the browser, and then enter the following code in a new notebook cell:

%matplotlib inline

from ipywidgets import interactive
import matplotlib.pyplot as plt
import numpy as np

def f(m, b):
    plt.figure(2)
    x = np.linspace(-10, 10, num=1000)
    plt.plot(x, m * x + b)
    plt.ylim(-5, 5)
    plt.show()

interactive_plot = interactive(f, m=(-2.0, 2.0), b=(-3, 3, 0.5))
output = interactive_plot.children[-1]
output.layout.height = '350px'
interactive_plot

You should get a result like in the picture below, where IPyWidgets generates a UI element on the page that interacts interactively with the source code, and Matplotlib displays the result of the code as a picture as a graph of the function:

Launching Jupyter into LXD Orbit

Many examples IPyWidgets you can find in tutorials here

What else? ^

You are great if you stayed and reached the very end of the article. I deliberately did not post the finished script at the end of the article that would install jupyter lab in "one click" to reward the hard workers πŸ™‚ But you can do it yourself, because you already know how, by collecting commands into a single Bash script πŸ™‚

You can also:

  • Set a network name for the container instead of an IP address by writing it in a simple /etc/hosts and in the browser type the address http://jupyter.local:8888
  • Play around with the resource limit for the container, for this read the chapter in basic LXD capabilities or get more information on the LXD developer site.
  • Change theme:

Launching Jupyter into LXD Orbit

And much more you can do! That's all. I wish you success!

UPDATE: 15.04.2020/18/30 XNUMX:XNUMX - Fixed errors in the "Hello, World!"
UPDATE: 16.04.2020/10/00 XNUMX:XNUMX am - Corrected and added text in the extension manager activation description jupyter lab
UPDATE: 16.04.2020/10/40 XNUMX:XNUMX am - Fixed found errors in the text and slightly improved the chapter "Installing basic software and system configuration"

Source: habr.com

Add a comment