PrefaceBy the will of fate, I transitioned from the world of academia (medicine) to the realm of information technology, where I have to apply my knowledge of experimental design methodology and data analysis strategies, albeit with a new stack of technologies for me. In the process of mastering these technologies, I encounter several challenges that I am, fortunately, managing to overcome. Perhaps this post will be beneficial for those who are also just starting to work on Apache projects.
So, to the point.Inspired by Yuri Emelyanov's insights on the capabilities of Apache Airflow in automating analytical procedures, I was eager to start using the proposed library set in my work. For those who are unfamiliar with Apache Airflow, a brief overview may be of interest. on the website of the N.E. Bauman National Library.
Since standard instructions for running Airflow don't seem to apply in a Windows environment, and using would be excessive in my case, I began searching for alternative solutions. Fortunately for me, I was not the first to tread this path, so I was able to find a wonderful on installing Apache Airflow on Windows 10 without using Docker. But, as often happens, difficulties arise when following the recommended steps, and I assume I'm not the only one facing this. Therefore, I would like to share my experience of installing Apache Airflow, hoping to save someone some time.
Let's go through the steps of the instruction (spoiler — everything went well until step 5):
1. Install the Windows Subsystem for Linux for the subsequent installation of Linux distributions.
This is the lesser of the problems, as they say:
Control Panel → Programs → Programs and Features → Turn Windows features on or off → Windows Subsystem for Linux.
2. Install the chosen Linux distribution.
I used the app .
3. Install and update pip.
sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update
sudo apt-get install python-pip.4. Install Apache Airflow.
export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow.5. Initialize the database.
And it was precisely here that my small troubles began. The instructions state to enter the command airflow initdb and proceed to the next step. However, I consistently received the response airflow: command not found.. It is reasonable to assume that difficulties arose during the installation of Apache Airflow and the necessary files are simply not where they should be. Having made sure everything is in its proper place, I decided to try specifying the full path to the airflow file (it should look like this: full/path/to/file/airflow initdb). But there was no miracle, and the response was the same airflow: command not found.. I tried using a relative path to the file (./.local/bin/airflow initdb), which led to a new error ModuleNotFoundError: No module named 'json', which can be overcome by upgrading the library werkzeug (in my case to version 0.15.4):
pip install werkzeug==0.15.4You can read more about werkzeug .
After this simple manipulation, the command ./.local/bin/airflow initdb was executed successfully.
6. Starting the Airflow server
At this point, the issues with accessing airflow are not yet over. Running the command ./.local/bin/airflow webserver -p 8080 resulted in an error No such file or directory. An experienced Ubuntu user would probably have tried to overcome such difficulties with file access by using the command export PATH=$PATH:~/.local/bin/ (that is, adding the /.local/bin/ directory to the existing executable file search path defined by the PATH variable), but this post is aimed at those who primarily work with Windows and may not consider such a solution obvious.
After the aforementioned manipulation, the command ./.local/bin/airflow webserver -p 8080 was successfully executed.
7. URL: :8080/
If everything went well in the previous steps, you are ready to conquer analytical heights.
I hope the experience described above for installing Apache Airflow on Windows 10 will be helpful for novice users and speed up their entry into the universe of modern analytical tools.
Next time I would like to continue the theme and share my experience using Apache Airflow in the analysis of mobile application user behavior.
Source: habr.com
