ืืืขื ืืืจ ืฆืืืจืืืื ืืื ืืืขืจ ืืืืืื ืืงืจืืื ืืืืื, ืืืจ ืคึผืืจืืึทืืืงืื ืืจืขืคื ืฉืืืขืจืืงืืืื ืืื ืืขืจืืื ืขื ืคืื ืืจืืขืื ืืื ืืืืขืจ ืืืฉืืจืื. ืืื ืืื ืืขื ืืึธืืขื ื ืืืขื ืืืจ ืืจืขืคื ืืื, ืขืก ืืื ื ืื ืฉืืขื ืืืง ืืขื ืื ืืึทืงืืืืขื ืืืืฉืึทื ืืื ืึทืจืืืงืืขื ืืืึธืก ืืืึธืื ืืขืืคึฟื ืืื ืื ืงืึธืคึผืข ืืื ืืขื ืคึผืจืึธืืืขื.
ืืึธืก ืืื ืืขืืืขื ืืขืจ ืคืึทื, ืืืฉื, ืืื 2015, ืืื ืืขืฉืึทืก ืื "ืืื ืืึทืืึท ืืืืืข" ืคึผืจืึธืืจืึทื ืืืจ ืืขืืืืื ื ืึท Hadoop ืงื ืืื ืืื ืกืคึผืึทืจืง ืคึฟืึทืจ 35 ืกืืืืึทืืืืื ืืึทืก ื ืืฆืขืจืก. ืขืก ืืื ืืขืืืขื ื ืื ืงืืึธืจ ืืื ืฆื ืฆืืืจืืืื ืขืก ืคึฟืึทืจ ืึทืืึท ืึท ืคืึทื ื ืืฆื YARN. ืฆืื ืกืืฃ, ืืืขื ืืืจ ืืึธืื ืขืก ืืืืกืืขืคืื ืขื ืืื ืืขืืื ืืขื ืืขื ืืืขื ืืืืฃ ืืื ืืืขืจ ืืืืื, ืืืจ ืืืื ืืื ืืืื ืืขืืื ืืืึท .
ืคึผืจืขืืืกืืึธืจื
ืืึธืก ืืึธื ืืืจ ืืืขืื ืจืขืื ืืืขืื ืึท ืึทื ืืขืจืฉ ืคึผืจืึธืืจืึทื - . ืืื ืืืขืจ ืคึผืึทืจืืืกืึทืคึผืึทื ืฅ ืืืืขื ืฆืืืื ืืืืคึผืก ืคืื ืึทืจืงืึทืืขืงืืฉืขืจ ืืืืฃ ืืื: ืืึทืืืืึท ืืื ืงืึทืคึผืึท. ืืื ืืื ืื ืืึทืืืืึท ืึทืจืงืึทืืขืงืืฉืขืจ, ืืื ืึท ืืืื ืคืื ืคึผืขืงื ืคึผืจืึทืกืขืกืื ื, ืึทืืจืคืืึธืื ืืื ืืขื ืืฆื ืฆื ืึทืจืืืขืจืคืืจื ืืึธืืก ืคืื HDFS ืฆื ClickHouse.
ืืืขืก ืืื ืืืื ืืื. ืืื ืืื ืืืืขื ืืืืขืจ ืืืืื ืคึผืืืคึผืืืื ื. ืึธืืขืจ, ืขืก ืืื ืึท "ืึธืืขืจ": ืึทืืข ืืื ืืืขืจ ืืืืื ืืขื ืขื ืืขืงื ืึทืืึทืืืฉืืงืึทืื ืึทืืืึทื ืกืืจืืข ืคึฟืื ืื ืคืื ื ืคืื ืืืื ืื ื ืคืื ืื ืืขืจื ืขื ืคึผืจืึธืฆืขืก ืืื. ืฆื ืงืึธื ืืจืึธืืืจื ืื ืืึทืืึธืจืึทืืึธืจืืข, ืืืจ ื ืืฆื ืึธืืึทืืึทืืืง ืืฉืขืงืงืขืจืก: ืืขืจ ืืึทืืืืืืงืืขืจ ืืึทืจืฃ ืืืื ืฆื ืืืื ืคืขืจืืขื ืืขืืข ืืฉืืื, ืืื ืื "ืืฉืขืง" ืงื ืขืคึผื, ืืื ื ืึธื ืขืืืขืืข ืืึธื ืขืจ ืืขื ืขืืืขืืข ืขืงืกืืขื ืืขื ืืึทืืขืจืงืื ืืขื ืืืืฃ ืืืึธืก ืขืจ ืืื ืืขืืื. ืืื ืขืก ืืื ืืื ืืขื ืืึธืืขื ื ืึทื ืืืจ ืึธื ืืืืื ืฆื ืฆืืืึทื ื ืืื ืืืขืจ ืคึผืจืึธืืืขื.
ืื ืืืขืจืึทืคืึทืงืืืฉืึทื ืคืื ืืขื ืืึทื ืืื ืกืืจืึทืงืืฉืขืจื ืืื ืืึธืก: ืืืจ ืฉืืงื ืึท ืงืึธื ืืจืึธื ืืึทืื ืคึผืึทืงืึทื ืฆื ืื ืืึทืืืืืืงืืขืจ ืก ืงืึทืคืงืึท, ืืขืืึธืื ืืึธืืืื ืืจืึทื ืกืคืขืจืก ืืขื ืืึทืื ืคึผืึทืงืึทื ืฆื HDFS, ืืื ืึทืืจืคืืึธืื ื ืขืื ืืขื ืืึทืื ืคึผืึทืงืึทื ืืื ืืืืื ืขืก ืืื ืงืืืงืืึธืืกืข. ืืขืจ ืงืื ืฅ ืืื ืึทื ืึทืืจืคืืึธืื ืืึธื ื ืื ืืึธืื ืฆื ืืึธื ืืึธืก ืืื ืคืึทืงืืืฉ ืฆืืื, ืขืก ืืื ืขืก ืืืื ืึท ืคึผืืึทื: ืืขืืขืจ 15 ืืื ืื ืขืก ื ืขืื ืึท ืคึผืืึทืฅ ืคืื ืืขืงืขืก ืืื ืืคึผืืึธืึทืืก ืืื.
ืขืก ืืืฒึทืื ืืื ืืืืก, ืึทื ืืืจ ืืึทืจืคึฟื ืืื ืึทืืืื ืืืืคึฟื ืคึฟืึทืจืืึทื ื ืฆื ืฆืื ืื ืืืืขืจ DAG ืืฉืขืช ืืขืจ ืืฉืขืงืขืจ ืืืืคึฟื ืืึธ ืืื ืืืฆื. ื ืึธื ืืึธืึธืืืื ื, ืืืจ ืืขืคึฟืื ืขื ืึทื ืคึฟืึทืจ ืฉืคึผืขืืขืจ ืืืขืจืกืืขืก ืคืื ืึทืืจืคืืึธืื ืขืก ืืื ืึท ืึทืืื ืืขืจืืคืขื ืข . ืื ืืืึธืจื experimental, ืคืื ืงืืจืก, ืขืก ืกืึธืื ืืก ืกืงืขืจื, ืึธืืขืจ ืืืึธืก ืฆื ืืึธื ... ืคึผืืืฆืืื ื ืขืก ื ืขืื ืึทืืืขืง.
ืืืืึทืืขืจ, ืืืจ ืืืขืื ืืึทืฉืจืืึทืื ืื ืืื ืฆืข ืืจื: ืคืื ืื ืกืืึธืืื ื ืึทืืจืคืืึธืื ืฆื ืืืฉืขื ืขืจืืืืื ื ืึท POST ืืขืื ืืืึธืก ืืจืืืขืจื ืื DAG ื ืืฆื ืื ืขืงืกืคึผืขืจืืืขื ืืึทื ืึทืคึผื. ืืืจ ืืืขืื ืึทืจืืขืื ืืื Ubuntu 16.04.
1. ืึทืืจืคืืึธืื ืื ืกืืึทืืืจืื ื
ืืึธืืืจ ืงืึธื ืืจืึธืืืจื ืึทื ืืืจ ืืึธืื Python 3 ืืื Virtualenv.
$ python3 --version
Python 3.6.6
$ virtualenv --version
15.2.0ืืืื ืืืื ืขืจ ืคืื ืืขื ืืื ืคืขืื ืืืง, ืื ืกืืึทืืืจื ืขืก.
ืืืฆื ืืึธืื ืืื ืื ืฉืึทืคึฟื ืึท ืืืขืืืืืึทืืขืจ ืืื ืืืึธืก ืืืจ ืืืขืื ืคืึธืจืืขืฆื ืฆื ืึทืจืืขืื ืืื Airflow.
$ mkdir <your name of directory>
$ cd /path/to/your/new/directory
$ virtualenv -p which python3 venv
$ source venv/bin/activate
(venv) $ืื ืกืืึทืืืจื Airflow:
(venv) $ pip install airflowืื ืืืขืจืกืืข ืืืจ ืืขืืจืืขื ืืืืฃ: 1.10.
ืืืฆื ืืืจ ืืึทืจืคึฟื ืฆื ืฉืึทืคึฟื ืึท ืืืขืืืืืึทืืขืจ airflow_home, ืืื DAG ืืขืงืขืก ืืื Airflow ืคึผืืืืื ืก ืืืขื ืืืื ืืืื. ื ืึธื ืงืจืืืืืื ื ืื ืืืขืืืืืึทืืขืจ, ืฉืืขืื ืื ืกืืืืืืข ืืืึทืืขืืืืืง AIRFLOW_HOME.
(venv) $ cd /path/to/my/airflow/workspace
(venv) $ mkdir airflow_home
(venv) $ export AIRFLOW_HOME=<path to airflow_home>ืืขืจ ืืืืึทืืขืจ ืฉืจืื ืืื ืฆื ืืืืคื ืึท ืืึทืคึฟืขื ืืืึธืก ืืืขื ืฉืึทืคึฟื ืืื ืื ืืฉืึทืืืื ืึท ืืึทืืึทืคืืึธืื ืืึทืืึทืืืืก ืืื SQLite:
(venv) $ airflow initdbืื ืืึทืืึทืืืืก ืืืขื ืืืื ืืืฉืืคื ืืื airflow.db default.
ืืึธืืืจ ืงืึธื ืืจืึธืืืจื ืืืื Airflow ืืื ืืื ืกืืึทืืืจื:
$ airflow version
[2018-11-26 19:38:19,607] {__init__.py:57} INFO - Using executor SequentialExecutor
[2018-11-26 19:38:19,745] {driver.py:123} INFO - Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
[2018-11-26 19:38:19,771] {driver.py:123} INFO - Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
v1.10.0ืืืื ืืขืจ ืืึทืคึฟืขื ืืขืืจืืขื, Airflow ืืืฉืืคื ืืืื ืืืืืขื ืข ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข airflow.cfg ะฒ AIRFLOW_HOME:
$ tree
.
โโโ airflow.cfg
โโโ unittests.cfgAirflow ืืื ืึท ืืืขื ืฆืืืื ื. ืขืก ืงืขื ืขื ืืืื ืืึธื ืืฉื ืืืจื ืืืืคื ืื ืืึทืคึฟืขื:
(venv) $ airflow webserver --port 8081ืืืจ ืงืขื ื ืืืฆื ืฉืืึธืื ืื ืืืขื ืฆืืืื ื ืืื ืึท ืืืขืืขืจืขืจ ืืืืฃ ืคึผืึธืจื 8081 ืืืืฃ ืืขืจ ืืึทืืขืืึธืก ืืื ืึทืืจืคืืึธืื ืืื ืคืืืกื ืืืง, ืืืฉื: <hostname:8081>.
2. ืืจืืขืื ืืื ืขืงืกืคึผืขืจืืืขื ืืึทื ืึทืคึผื
ืืื ืืขื ืคืื ื, Airflow ืืื ืงืึทื ืคืืืืขืจื ืืื ืืจืืื ืฆื ืืืื. ืึธืืขืจ, ืืืจ ืืืื ืืึทืจืคึฟื ืฆื ืืืืคื ืื ืขืงืกืคึผืขืจืืืขื ืืึทื ืึทืคึผื. ืืื ืืืขืจ ืืฉืขืงืงืขืจืก ืืขื ืขื ืืขืฉืจืืื ืืื ืคึผืืืืึธื, ืึทืืื ืึทืืข ืจืืงืืืขืก ืืืขื ืืืื ืืื ืขืก ืืื ืื ืืืืืืึธืืขืง requests.
ืืื ืคืึทืงื, ืื ืึทืคึผื ืฉืืื ืึทืจืืขื ืคึฟืึทืจ ืคึผืฉืื ืจืืงืืืขืก. ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืขื ืืงืฉื ืึทืืึทืื ืืืจ ืฆื ืคึผืจืืืืจื ืืืึทื ืึธืคึผืขืจืึทืฆืืข:
>>> import requests
>>> host = <your hostname>
>>> airflow_port = 8081 #ะฒ ะฝะฐัะตะผ ัะปััะฐะต ัะฐะบะพะน, ะฐ ะฟะพ ะดะตัะพะปัั 8080
>>> requests.get('http://{}:{}/{}'.format(host, airflow_port, 'api/experimental/test').text
'OK'ืืืื ืืืจ ืืึทืงืืืขื ืึทืืึท ืึท ืึธื ืืึธื ืืื ืขื ืืคืขืจ, ืขืก ืืืื ืึทื ืึทืืฅ ืึทืจืืขื.
ืึธืืขืจ, ืืืขื ืืืจ ืืืืื ืฆื ืฆืื ืื ืึท DAG, ืืืจ ืืขื ืขื ืคืืืกื ืืื ืื ืคืึทืงื ืึทื ืืขื ืืืคึผ ืคืื ืืขืื ืงืขื ืขื ื ืื ืืืื ืืขืืืื ืึธื ืึธืืขื ืืึทืงืืืฉืึทื.
ืฆื ืืึธื ืืึธืก, ืืืจ ืืึทืจืคึฟื ืฆื ืืึธื ืึท ื ืืืขืจ ืคืื ืืขืจ ืกืืขืคึผืก.
ืขืจืฉืืขืจ, ืืืจ ืืึทืจืคึฟื ืฆื ืืืืื ืืขื ืฆื ืื ืงืึทื ืคืืืืขืจืืืฉืึทื:
[api]
auth_backend = airflow.contrib.auth.backends.password_authืืขืจื ืึธื ืืืจ ืืึทืจืคึฟื ืฆื ืฉืึทืคึฟื ืืืื ืืึทื ืืฆืขืจ ืืื ืึทืืืื ืืกืืจืึทืืึธืจ ืจืขืื:
>>> import airflow
>>> from airflow import models, settings
>>> from airflow.contrib.auth.backends.password_auth import PasswordUser
>>> user = PasswordUser(models.Admin())
>>> user.username = 'new_user_name'
>>> user.password = 'set_the_password'
>>> session = settings.Session()
>>> session.add(user)
>>> session.commit()
>>> session.close()
>>> exit()ืืขืจื ืึธื ืืืจ ืืึทืจืคึฟื ืฆื ืฉืึทืคึฟื ืึท ืืึทื ืืฆืขืจ ืืื ื ืึธืจืืึทื ืจืขืื ืืืึธืก ืืืขื ืืืื ืขืจืืืืื ืฆื ืฆืื ืื ืื DAG.
>>> import airflow
>>> from airflow import models, settings
>>> from airflow.contrib.auth.backends.password_auth import PasswordUser
>>> user = PasswordUser(models.User())
>>> user.username = 'newprolab'
>>> user.password = 'Newprolab2019!'
>>> session = settings.Session()
>>> session.add(user)
>>> session.commit()
>>> session.close()
>>> exit()ืืืฆื ืึทืืฅ ืืื ืืจืืื.
3. ืงืึทืืขืจ ืึท ืคึผืึธืกืื ืืขืื
ืื POST ืืขืื ืืื ืืืขื ืงืืงื ืืื ืืึธืก:
>>> dag_id = newprolab
>>> url = 'http://{}:{}/{}/{}/{}'.format(host, airflow_port, 'api/experimental/dags', dag_id, 'dag_runs')
>>> data = {"conf":"{"key":"value"}"}
>>> headers = {'Content-type': 'application/json'}
>>> auth = ('newprolab', 'Newprolab2019!')
>>> uri = requests.post(url, data=json.dumps(data), headers=headers, auth=auth)
>>> uri.text
'{n "message": "Created <DagRun newprolab @ 2019-03-27 10:24:25+00:00: manual__2019-03-27T10:24:25+00:00, externally triggered: True>"n}n'ืื ืืงืฉื ืืื ืืฆืืื ืคึผืจืึทืกืขืกื.
ืืขืจืืืขืจ, ืืืจ ืืขืื ืื DAG ืขืืืขืืข ืืึธื ืฆื ืคึผืจืึธืฆืขืก ืืื ืืึทืื ืึท ืืงืฉื ืฆื ืื ClickHouse ืืืฉ, ืืจืืื ื ืฆื ืืึทืคึผื ืื ืงืึธื ืืจืึธื ืืึทืื ืคึผืึทืงืึทื.
ืืฉืขืง ืืขืขื ืืืงื.
ืืงืืจ: www.habr.com