Making Python and Bash friends: smart-env and python-shell libraries

Good day to all.

Today, Python is one of the most used languages ​​in the field of creating not only software products directly, but also providing their infrastructure. As a result, many devops, willingly or against it, had to learn a new language for later use as an addition to the good old Bash scripts. However, Bash and Python profess different approaches to writing code and have certain features, which means that porting Bash scripts to the “snake language” sometimes turns out to be a capacious and far from trivial task.

To make life easier for devops, many useful Python libraries and utilities have been created and continue to be created. This article describes two new libraries at once, created by the author of this post - smart env и python shell - and designed to save the devops from having to pay a lot of attention to the intricacies of working with Python, leaving room for more interesting tasks. The scope of the libraries is environment variables and the launch of external utilities.

Who is interested, please under the cat.

New bikes?

It would seem, why create new packages for fairly mundane operations? What prevents you from using os.environ and subprocess.<method or class of your choice> directly?

Evidence in favor of each of the libraries will be given separately.

smart-env library

Before writing your own brainchild, it is useful to get into the Internet and look for ready-made solutions. Of course, there is a risk of not finding what you need, but this is more of an "insured event". As a rule, the approach works and saves a lot of time and effort.

According to the results Search the following was revealed:

• there are packages that really wrap calls to os.environ, but at the same time require a bunch of distracting actions (creating an instance of a class, special parameters in calls, etc.);
• there are good packages, which, however, are tightly tied to a certain ecosystem (mainly to web frameworks like Django) and therefore, without a file, they are not at all universal;
• there are rare attempts to do something new. For example, add typing and explicitly parse the values ​​of variables by calling methods of the form

  get_<typename>(var_name)

  Or one more solution, which, however, does not support the now disgraced Python 2 (which, despite official RIP, there are still mountains of written code and entire ecosystems);

• there are school-student crafts, it is not at all clear why they ended up in upstream PyPI and only create problems with the naming of new packages (in particular, the name “smart-env” is a necessary measure).

And this list can be continued for a long time. However, the above points were enough to light up the idea of ​​​​making something convenient and universal.

Requirements that were set before writing smart-env:

• The most simple scheme of use
• Easily configurable data typing support
• Compatibility with Python 2.7
• Good code coverage with tests

In the end, all this was possible to implement. Here is an example of usage:

from smart_env import ENV

print(ENV.HOME)  # Equals print(os.environ['HOME'])

# assuming you set env variable MYVAR to "True"

ENV.enable_automatic_type_cast()

my_var = ENV.MY_VAR  # Equals boolean True

ENV.NEW_VAR = 100  # Sets a new environment variable

As you can see from the example, to work with a new class, it is enough to import it (you do not need to create an instance - minus an extra step). Access to any environment variable is achieved by referring to it as an ENV class variable, which, in fact, makes this class an intuitive wrapper for the native system environment, while simultaneously turning it into a possible variant of the configuration object of almost any system (a similar approach, for example, is achieved in Django , only there the configuration object is directly the settings module/package).

Enabling/disabling the automatic typing support mode is achieved using two methods — enable_automatic_type_cast() and disable_automatic_type_cast(). This can be handy if the environment variable contains a serialized JSON-like object, or even just a boolean constant (explicitly setting the DEBUG variable in Django by comparing the environment variable with "valid" strings is one of the common cases). But now there is no need to explicitly convert strings - most of the necessary actions are already embedded in the bowels of the library and are just waiting for a signal to act. 🙂 In general, typing works transparently and supports almost all available built-in data types (frozenset, complex and bytes were not tested).

The requirement for Python 2 support was implemented with little or no sacrifice (removing typing and some of the "sugar candy" of recent versions of Python 3), in particular thanks to the ubiquitous six (to solve problems using metaclasses).

But there are also a few limitations:

• Support for Python 3 implies version 3.5 and higher (their presence in your project is the result of either laziness or lack of need for improvements, since it is difficult to come up with an objective reason why you are still sitting on 3.4);
• In Python 2.7, the library does not support deserialization of set literals. Description here. But, if someone wants to implement - welcome:);

The library also professes an exception mechanism in case of parsing errors. If the string could not be recognized by any of the available parsers, the value remains a string (rather, for reasons of convenience and backward compatibility with the familiar logic of how variables work in Bash).

python shell library

Now I’ll talk about the second library (I’ll omit the description of the shortcomings of the existing analogues - it is similar to that described for smart-env. Analogs - here и here).

In general, the idea of ​​implementation and requirements for it are similar to those described for smart-env, as can be seen from the example:

from python_shell import Shell

Shell.ls('-l', '$HOME')  # Equals "ls -l $HOME"

command = Shell.whoami()  # Equals "whoami"
print(command.output)  # prints your current user name

print(command.command)  # prints "whoami"
print(command.return_code)  # prints "0"
print(command.arguments)  # prints ""

Shell.mkdir('-p', '/tmp/new_folder')  # makes a new folder

The idea is:

1. A single class that represents Bash in the Python world;
2. Every Bash command is called as a function of the Shell class;
3. The call parameters of each function are then passed to the corresponding Bash command call;
4. Each command is executed "here and now" at the time of its call, i.e. the synchronous approach works;
5. it is possible to access the output of the command in stdout, as well as its return code;
6. If the command is not present in the system, an exception is thrown.

As in the case of smart-env, Python 2 support is provided (although a little more sacrificial blood was required) and there is no support for Python 3.0-3.4.

Library Development Plans

You can use the libraries now: both are posted on the official PyPI. Sources are available on Github (see below).

Both libraries will be developed based on feedback collected from interested parties. And, if in smart-env it may be difficult to come up with a variety of new features, then in python-shell there is definitely more to add:

• support for non-blocking calls;
• the possibility of interactive communication with the team (working with stdin);
• adding new properties (for example, property to get the output from stderr);
• implementation of a catalog of available commands (for use with the dir() function);
• etc.

references

1. smart-env library: Github и P&IP
2. python shell library: Github и P&IP
3. Telegram channel library updates

UPD 23.02.2020/XNUMX/XNUMX:
* Repositories moved, relevant links updated
* The version of python-shell==1.0.1 is being prepared for release on 29.02.2020/XNUMX/XNUMX. Changes include support for autocomplete commands and the dir(Shell) command, running commands with an invalid Python identifier, and bug fixes.

Source: habr.com