Good day to everyone.
Today, Python is one of the most widely used languages not only for creating software products but also for managing their infrastructure. As a result, many DevOps have had to learn a new language, whether they wanted to or not, to complement good old Bash scripts. However, Bash and Python have different coding approaches and certain peculiarities, making the porting of Bash scripts to the "snake language" sometimes a complex and far from trivial task.
To simplify the lives of DevOps, many useful libraries and utilities have been created and continue to emerge in Python. This article describes two new libraries created by the author of this post — and — designed to free DevOps from having to pay much attention to the nuances of working with Python, leaving room for more interesting tasks. The scope of the libraries is environment variables and launching external utilities.
If you're interested, please read on.
New "bicycles"?
It seems, why create new packages for quite mundane operations? What prevents using os.environ and subprocess directly.?
I will provide evidence in favor of each of the libraries separately.
The smart-env library
Before creating your own solution, it's useful to browse the Internet for ready-made solutions. Of course, there is a risk of not finding what you need, but this is more of an "insurance case." Usually, this approach works and saves a lot of time and effort.
According to the results of the following was revealed:
- there are packages that do wrap calls to os.environ, but require a lot of distracting actions (creating an instance of a class, special parameters in calls, etc.);
- there are decent packages that, however, are tightly bound to a specific ecosystem (mainly web frameworks like Django) and are therefore not universally applicable without some adjustments;
- there are rare attempts to create something new. For example, and explicitly parsing variable values by calling methods like
get_(var_name)Or here's , which, however, does not support now-deprecated Python 2 (on which, despite) , there are still mountains of written code and whole ecosystems);
- there are school and student projects that somehow ended up in the upstream PyPI and only create problems with naming new packages (in particular, the name “smart-env” is a forced measure).
And this list can go on for a long time. However, the above points were enough to spark the idea of creating something convenient and versatile.
Requirements set for writing smart-env:
- A maximally simple usage scheme
- Easily configurable data typing support
- Compatibility with Python 2.7
- Good test coverage of the code
In the end, all of this was successfully implemented. Here’s an example of usage:
from smart_env import ENV
print(ENV.HOME) # Equals print(os.environ['HOME'])
# assuming you set env variable MYVAR to "True"
ENV.enable_automatic_type_cast()
my_var = ENV.MY_VAR # Equals boolean True
ENV.NEW_VAR = 100 # Sets a new environment variable
As seen from the example, to work with the new class it is sufficient to import it (no need to create an instance — eliminating unnecessary action). Access to any environment variable is achieved by referencing it as a class variable ENV, which effectively makes this class an intuitive wrapper around the native system environment, while simultaneously transforming it into a potential configuration object for almost any system (a similar approach is achieved in Django, where the configuration object is the settings module/package itself).
Enabling/disabling automatic type casting mode is achieved using two methods — enable_automatic_type_cast() and disable_automatic_type_cast(). This can be useful if an environment variable holds a serialized JSON-like object or even just a boolean constant (explicitly defining the DEBUG variable in Django by comparing the environment variable with 'valid' strings is one of the common occurrences). But now there is no need to explicitly convert strings — most of the required actions are already built into the depths of the library, just waiting for a signal to act. 🙂 In general, typing works transparently and supports almost all built-in data types (frozenset, complex, and bytes were not tested).
The requirement for Python 2 support has been implemented with almost no sacrifices (giving up typing and some 'sweet features' from the latest versions of Python 3), primarily thanks to the ubiquitous six (to handle metaclass usage issues).
However, there are some limitations:
- Python 3 support implies version 3.5 and above (having them in your project is either due to laziness or a lack of need for improvements, as it's hard to come up with an objective reason why you are still stuck on 3.4);
- In Python 2.7, the library does not support the deserialization of set literals. Description . But if someone wants to implement it — welcome:);
The library also follows an exception mechanism in case of parsing errors. If a string cannot be recognized by any of the available parsers, the value remains a string (mainly for convenience and backward compatibility with the usual behavior of variables in Bash).
The python-shell library
Now I'll talk about the second library (I’ll skip the description of the shortcomings of existing analogs — it's similar to what was described for smart-env. The analogs are — and ).
Overall, the implementation idea and its requirements are similar to those described for smart-env, which is evident from the example:
from python_shell import Shell
Shell.ls('-l', '$HOME') # Equals "ls -l $HOME"
command = Shell.whoami() # Equals "whoami"
print(command.output) # prints your current user name
print(command.command) # prints "whoami"
print(command.return_code) # prints "0"
print(command.arguments) # prints ""
Shell.mkdir('-p', '\/tmp\/new_folder') # makes a new folder
The idea is as follows:
- A single class embodying Bash in the Python world;
- Each Bash command is called as a method of the Shell class;
- The parameters for each method call are then passed to the corresponding Bash command;
- Each command is executed 'here and now' at the moment of its call; that is, a synchronous approach is employed;
- there is the ability to access the command's output in stdout, as well as its return code;
- If the command is not present in the system — an exception is raised.
Similar to smart-env, support for Python 2 is provided (though it required a bit more sacrifice) and support for Python 3.0-3.4 is absent.
Plans for library development
Both libraries can be used now: they are both available on the official PyPI. The source code is available on GitHub (see below).
Both libraries will evolve based on the feedback collected from interested parties. While it may be challenging to come up with a variety of new features in smart-env, there is definitely more to add in python-shell:
- support for non-blocking calls;
- the ability to interactively communicate with the team (working with stdin);
- adding new properties (for example, a property for retrieving output from stderr);
- implementation of a directory of available commands (to be used with the dir() function);
- etc.
Links
- The smart-env library: and
- The python-shell library: and
- library updates
UPD 23.02.2020:
* Repositories have been moved, corresponding links updated
* Version python-shell==1.0.1 is set to release on 29.02.2020. Changes include support for command auto-complete and the command dir(Shell), execution of commands with Python-invalid identifiers, and bug fixes.
Source: habr.com
