ืืขืื, ืืืืจ! ืืื ืืขื ืึทืจืืืงื ืืื ืืืืื ืฆื ืจืขืื ืืืขืื ืืืื ืืจืืืก ืืขืฆืืึทื ืคึฟืึทืจ ืืขืืืขืืึธืคึผืื ื ืคึผืขืงื ืืึทืื ืคึผืจืึทืกืขืกืื ื ืคึผืจืึทืกืขืกืึทื, ืืืฉื, ืืื ืื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ ืคืื ืึท ืคึฟืืจืืข DWH ืึธืืขืจ ืืืื DataLake. ืืืจ ืืืขืื ืจืขืื ืืืขืื ืึทืคึผืึทืืฉื ืึทืืจืคืืึธืื (ืืขืจื ืึธื ืจืืคืขืจื ืฆื ืืื ืึทืืจืคืืึธืื). ืขืก ืืื ืืืืืืฉืขืจืืืง ืืืคึผืจืืืืื ืคืื ืืคืืขืจืงืืึทืืงืืื ืืืืฃ Habrรฉ, ืืื ืืื ืืขืจ ืืืืคึผื ืืืื ืืื ืืืขื ืคึผืจืืืืจื ืฆื ืืืืขืจืฆืืืื ืืืจ ืึทื ืืืึท ืืื ืืกืืขืจ ืึทืืจืคืืึธืื ืืื ืืืขืจื ืฆื ืงืืงื ืืื ืืืขื ืืฉืืืื ื ืึท ืกืงืขืืืฉืืืขืจ ืคึฟืึทืจ ืืืื ETL / ELT ืคึผืจืึทืกืขืกืึทื.
ืืื ืึทืืขืจ, ืืื ืืขืฉืจืืื ืึท ืกืขืจืืข ืคืื โโืึทืจืืืงืืขื ืืืืฃ ืื ืืขืืข ืคืื โโDWH ืืืขื ืืื ืืขืืจืืขื ืืื ืืื ืงืึธืคืฃ ืืึทื ืง. ืืืฆื ืืื ืืื ืืขืืืืจื ืึท ืืืื ืคืื ืื Mail.Ru ืืจืืคืข ืืึทื ืฉืึทืคึฟื ืืื ืืื ืึทื ืืืืืงืืขื ืึท ืคึผืืึทืืคืึธืจืืข ืคึฟืึทืจ ืืึทืื ืึทื ืึทืืืกืืก ืืื ืื ืืืืืื ื ืืขืื ื. ืึทืงืืืึทืืื, ืืื ื ืืึทืขืก ืืื ืืฉืืงืึทืืืข ืกืึทืืืฉืึทื ื ืืขืจืฉืืึทื ืขื, ืืืื ืืึทื ืฉืึทืคึฟื ืืื ืืื ืืืขืื ืจืขืื ืืึธ ืืืขืื ืืื ืืืขืจ ืคึผืืึทืืคืึธืจืืข ืคึฟืึทืจ ืืึทืื ืึทื ืึทืืืืืงืก.
ืคึผืจืึธืืึธื
ืึทืืื, ืืึธืื ืก ืึธื ืืืืื. ืืืึธืก ืืื ืึทืืจืคืืึธืื? ืืึธืก ืืื ืึท ืืืืืืึธืืขืง (ืึธืืขืจ ) ืฆื ืึทื ืืืืืงืืขื, ืคึผืืึทื ืืื ืืึธื ืืืึธืจ ืึทืจืืขื ืคึผืจืึทืกืขืกืึทื. ืื ืืืืคึผื ืฉืืจืื ืคืื ืึทืืจืคืืึธืื: ืคึผืืืืึธื ืงืึธื ืืื ืืขื ืืฆื ืฆื ืืึทืฉืจืืึทืื (ืึทื ืืืืืงืืขื) ืคึผืจืึทืกืขืกืึทื. ืืึธืก ืืื ืึท ืคึผืืึทืฅ ืคืื ืึทืืืืึทื ืืืืืฉืื ืคึฟืึทืจ ืึธืจืืึทื ืืืืื ื ืืืื ืคึผืจืืืขืงื ืืื ืึทื ืืืืืงืืื ื: ืืื ืขืกืึทื ืก, ืืืื (ืืืฉื) ETL ืคึผืจืืืขืงื ืืื ืืืืื ืึท ืคึผืืืืึธื ืคึผืจืืืขืงื, ืืื ืืืจ ืงืขื ืขื ืึธืจืืึทื ืืืืจื ืขืก ืืื ืืืจ ืืืืื, ืืขื ืืืขื ืืื ืืฉืืื ืื ืกืคึผืขืกืืคืืงืก ืคืื ืื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ, ืืึทื ืฉืึทืคึฟื ืืจืืืก ืืื ืื ืืขืจืข ืืืืขืจืคืขื ืืฉื. ืื ืกืืจืืืขื ืืึทืืื ืึทืืฅ ืืื ืคึผืฉืื. ื ืืฆื ืคึฟืึทืจ ืืืึทืฉืคึผืื PyCharm + Git. ืขืก ืืื ืืืื ืืขืจืืขื ืืื ืืืืขืจ ืืึทืงืืืขื!
ืืืฆื ืืึธืื ืก ืงืืง ืืื ืื ืืืืคึผื ืขื ืืืืื ืคืื Airflow. ืืืจื ืคึฟืึทืจืฉืืืื ืืืืขืจ ืขืกืึทื ืก ืืื ืฆืื, ืืืจ ืงืขื ืขื ืึธืคึผืืืืึทื ืึธืจืืึทื ืืืืจื ืืืื ืคึผืจืึธืฆืขืก ืึทืจืงืึทืืขืงืืฉืขืจ. ืืึธืืขืจ ืื ืืืืคึผื ืขื ืืืื ืืื ืื ืืืจืขืงืืขื ืึทืกืืงืืืง ืืจืึทืคืืง (ืืขืจื ืึธื ืจืืคืขืจื ืฆื ืืื DAG).
DAG
ื DAG ืืื ืึท ืืื ืื ืืคืึทื ืคืึทืจืืึทื ื ืคืื ืืืื ืืึทืกืงืก ืืืึธืก ืืืจ ืืืืื ืฆื ืคืึทืจืขื ืืืงื ืืื ืึท ืฉืืจืขื ื ืืืคืืื ื ืกืืงืืืึทื ืก ืืืื ืึท ืกืคึผืขืฆืืคืืฉ ืคึผืืึทื. Airflow ืืื ืึท ืืึทืงืืืขื ืืืขื ืฆืืืื ื ืคึฟืึทืจ ืืจืืขืื ืืื DAGs ืืื ืื ืืขืจืข ืขื ืืืืื:
ืื DAG ืงืขื ืงืืงื ืืื ืืึธืก:
ืืขืจ ืืขืืืขืืึธืคึผืขืจ, ืืืขื ืืืืืื ืื ื ืึท DAG, ืืืื ืึท ืืึทื ื ืคืื ืึธืคึผืขืจืืืืขืจื ืืืืฃ ืืืึธืก ืืึทืกืงืก ืืื ืื DAG ืืืขื ืืืื ืืขืืืื. ืืึธ ืืืจ ืงืืืขื ืฆื ืื ืื ืืขืจ ืืืืืืืง ืขื ืืืื: ืึทืืจืคืืึธืื ืึธืคึผืขืจืึทืืึธืจ.
Operators
ืึทื ืึธืคึผืขืจืึทืืึธืจ ืืื ืึทื ืขื ืืืื ืืืืฃ ืื ืืงืขืจ ืคืื ืืืึธืก ืึทืจืืขื ืื ืกืืึทื ืกืื ืืขื ืขื ืืืฉืืคื, ืืืึธืก ืืืฉืจืืืื ืืืึธืก ืืืขื ืคึผืึทืกืืจื ืืขืฉืึทืก ืื ืืืจืืคืืจืื ื ืคืื ืึท ืึทืจืืขื ืืืึทืฉืคึผืื. ืฉืืื ืึทื ืืืึทืืื ืึท ืกืืื ืคืื ืึธืคึผืขืจืืืืขืจื ืืจืืื ืฆื ื ืืฆื. ืืืืฉืคืืื:
- BashOperator - ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืขืงืกืึทืงืืืืื ื ืึท ืืึทืฉ ืืึทืคึฟืขื.
- PythonOperator - ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืจืืคื ืคึผืืืืึธื ืงืึธื.
- EmailOperator - ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืฉืืงื E- ืืจืืื.
- ืืืืคึผืึธืคึผืขืจืึทืืึธืจ - ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืืจืืขืื ืืื ืืืืคึผ ืจืืงืืืขืก.
- SqlOperator - ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืขืงืกืึทืงืืืืื ื ืกืงื ืงืึธื.
- ืกืขื ืกืึธืจ ืืื ืึทื ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืืืืจืื ืคึฟืึทืจ ืึท ืืขืฉืขืขื ืืฉ (ืืขืจ ืึธื ืงืืืขื ืคืื ืื ืคืืจืืื ืื ืฆืืื, ืื ืืืืกืืขื ืคืื ืื ืคืืจืืื ืื ืืขืงืข, ืึท ืฉืืจื ืืื ืื ืืึทืืึทืืืืก, ืึท ืขื ืืคืขืจ ืคืื ืื API, ืืื"ื ื).
ืขืก ืืขื ืขื ืืขืจ ืกืคึผืขืฆืืคืืฉ ืึธืคึผืขืจืืืืขืจื: DockerOperator, HiveOperator, S3FileTransferOperator, PrestoToMysqlOperator, SlackOperator.
ืืืจ ืงืขื ื ืืืื ืึทื ืืืืืงืืขื ืึธืคึผืขืจืืืืขืจื ืืืืืจื ืืืืฃ ืืืื ืืืืืขื ืข ืงืขืจืึทืงืืขืจืืกืืืงืก ืืื ื ืืฆื ืืื ืืื ืืืื ืคึผืจืืืขืงื. ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืืจ ืืืฉืืคื MongoDBToHiveViaHdfsTransfer, ืึทื ืึธืคึผืขืจืึทืืึธืจ ืคึฟืึทืจ ืขืงืกืคึผืึธืจืืื ื ืืึธืงืืืขื ืื ืคึฟืื MongoDB ืฆื Hive, ืืื ืขืืืขืืข ืึธืคึผืขืจืืืืขืจื ืคึฟืึทืจ ืืจืืขืื ืืื : CHLoadFromHiveOperator ืืื CHTableLoaderOperator. ืืกืขื ืฉืึทืื, ืืื ืืึทืื ืืื ืึท ืคึผืจืืืขืงื ืึธืคื ืืขื ืืฆื ืงืึธื ืืขืืืื ืืืืฃ ืืงืขืจืืืง ืกืืืืืืึทื ืฅ, ืืืจ ืงืขื ืขื ืืจืึทืืื ืืืขืื ืืืืขื ืขืก ืืื ืึท ื ืืึทืข ืืขืจืงืืขืจืื ื. ืืึธืก ืืืขื ืคืึทืจืคึผืึธืฉืขืืขืจื ืืืืึทืืขืจ ืึทื ืืืืืงืืื ื, ืืื ืืืจ ืืืขื ืืงืกืคึผืึทื ื ืืืื ืืืืืืึธืืขืง ืคืื ืึธืคึผืขืจืืืืขืจื ืืื ืื ืคึผืจืืืขืงื.
ืืืืึทืืขืจ, ืึทืืข ืื ืื ืกืืึทื ืกืื ืคืื ืืึทืกืงืก ืืึทืจืคึฟื ืฆื ืืืื ืขืงืกืึทืงืืืืึทื, ืืื ืืืฆื ืืืจ ืืืขืื ืจืขืื ืืืขืื ืื ืกืงืขืืืฉืืืขืจ.
ืกืืฉืขืืืืขืจ
ืึทืืจืคืืึธืื ืก ืึทืจืืขื ืกืงืขืืืฉืืืขืจ ืืื ืืขืืืื ืืืืฃ . ืกืขืืืขืจืื ืืื ืึท ืคึผืืืืึธื ืืืืืืึธืืขืง ืืืึธืก ืึทืืึทืื ืืืจ ืฆื ืึธืจืืึทื ืืืืจื ืึท ืจืื ืคึผืืืก ืืืกืื ืืงืจืึทื ืึทืก ืืื ืคืื ืื ืืขืจืืขืืืืื ืืืจืืคืืจืื ื ืคืื ืืึทืกืงืก. ืืืืฃ ืื ืึทืืจืคืืึธืื ืืืึทื, ืึทืืข ืืึทืกืงืก ืืขื ืขื ืฆืขืืืืื ืืื ืคึผืึธืึธืืก. ืคึผืึธืึธืืก ืืขื ืขื ืืืฉืืคื ืืึทื ืืืึทืื. ืืืคึผืืงืึทืืื, ืืืืขืจ ืฆืื ืืื ืฆื ืืึทืืจืขื ืขืฆื ืื ืืืขืจืงืืึธืื ืคืื ืืจืืขืื ืืื ืื ืืงืืจ ืึธืืขืจ ืฆื ืืืคึผ ืืึทืกืงืก ืืื ืื DWH. ืคึผืึธืึธืืก ืงืขื ืขื ืืืื ืืขืจืืื ืืืจื ืื ืืืขื ืฆืืืื ื:
ืืขืืขืจ ืืขืงื ืืื ืึท ืฉืืขืืจ ืืืืฃ ืื ื ืืืขืจ ืคืื ืกืืึธืฅ. ืืืขื ืงืจืืืืืื ื ืึท DAG, ืขืก ืืื ืืขืืขืื ืึท ืืขืงื:
ALERT_MAILS = Variable.get("gv_mail_admin_dwh")
DAG_NAME = 'dma_load'
OWNER = 'Vasya Pupkin'
DEPENDS_ON_PAST = True
EMAIL_ON_FAILURE = True
EMAIL_ON_RETRY = True
RETRIES = int(Variable.get('gv_dag_retries'))
POOL = 'dma_pool'
PRIORITY_WEIGHT = 10
start_dt = datetime.today() - timedelta(1)
start_dt = datetime(start_dt.year, start_dt.month, start_dt.day)
default_args = {
'owner': OWNER,
'depends_on_past': DEPENDS_ON_PAST,
'start_date': start_dt,
'email': ALERT_MAILS,
'email_on_failure': EMAIL_ON_FAILURE,
'email_on_retry': EMAIL_ON_RETRY,
'retries': RETRIES,
'pool': POOL,
'priority_weight': PRIORITY_WEIGHT
}
dag = DAG(DAG_NAME, default_args=default_args)
dag.doc_md = __doc__ื ืืขืงื ืืืคืืื ื ืืืืฃ ืื DAG ืืืจืื ืงืขื ืขื ืืืื ืึธืืืืขืจืจืืื ืืืืฃ ืื ืึทืจืืขื ืืืจืื.
ื ืืึทืืื ืืขืจ ืคึผืจืึธืฆืขืก, ืกืืฉืขืืืืขืจ, ืืื ืคืึทืจืึทื ืืืืึธืจืืืขื ืคึฟืึทืจ ืกืงืขืืืฉืืืื ื ืึทืืข ืืึทืกืงืก ืืื ืึทืืจืคืืึธืื. ืึทืงืืืึทืืื, ืกืืฉืขืืืืขืจ ืืืื ืืื ืึทืืข ืื ืืึทืงืึทื ืืงืก ืคืื ืืึทืฉืืขืืืงื ืืึทืกืงืก ืคึฟืึทืจ ืืืจืืคืืจืื ื. ืื ืึทืจืืขื ืืืื ืืืจื ืขืืืขืืข ืกืืึทืืขืก ืืืืืขืจ ืืืื ืขืงืกืึทืงืืืืึทื:
- ืื ืคืจืืขืจืืืงืข ืืึทืกืงืก ืืึธืื ืฉืืื ืืืจืืืขืงืึธืื ืืื ืื DAG; ืึท ื ืืึทืข ืงืขื ืขื ืืืื ืงืื.
- ืื ืจืื ืืื ืืืืกืืขืฉืืขืื ืืืคึผืขื ืืื ื ืืืืฃ ืื ืืืืืขืจืงืืึทื ืคืื ืืึทืกืงืก (ืคึผืจืืืึธืจืึทืืื ืงืขื ืขื ืืืื ืืืื ืงืึทื ืืจืึธืืื), ืืื ืืืื ืขืก ืืื ืึท ืคืจืื ืฉืคึผืขืืื ืืื ืื ืืขืงื, ืื ืึทืจืืขื ืงืขื ืขื ืืืื ืืขื ืืืขื ืืื ืึธืคึผืขืจืึทืฆืืข.
- ืืืื ืขืก ืืื ืึท ืคึฟืจืืึท ืึทืจืืขืืขืจ ืกืขืืืขืจืื, ืื ืึทืจืืขื ืืื ืืขืฉืืงื ืฆื ืืื; ืื ืึทืจืืขื ืืืึธืก ืืืจ ืคึผืจืึธืืืจืึทืื ืืื ืืขื ืคึผืจืึธืืืขื ืืืืื ืืื ืึธื ืืื ืืืื ืึธืืขืจ ืื ืืขืจื ืึธืคึผืขืจืึทืืึธืจ.
ืคึผืฉืื ืืขื ืื.
ืกืืฉืขืืืืขืจ ืืืืคื ืืืืฃ ืื ืืึทื ื ืคืื ืึทืืข DAGs ืืื ืึทืืข ืืึทืกืงืก ืืื DAGs.
ืคึฟืึทืจ ืกืืฉืขืืืืขืจ ืฆื ืึธื ืืืืื ืืจืืขืื ืืื DAG, ืื DAG ืืึทืจืฃ ืฉืืขืื ืึท ืคึผืืึทื:
dag = DAG(DAG_NAME, default_args=default_args, schedule_interval='@hourly')ืขืก ืืื ืึท ืกืืื ืคืื ืคืึทืจืืืง ืคึผืจืขืกืขืฅ: @once, @hourly, @daily, @weekly, @monthly, @yearly.
ืืืจ ืงืขื ื ืืืื ื ืืฆื Cron ืืืืกืืจืืงื:
dag = DAG(DAG_NAME, default_args=default_args, schedule_interval='*/10 * * * *')ืืืจืืคืืจืื ื ืืึธื
ืฆื ืคึฟืึทืจืฉืืืื ืืื ืึทืืจืคืืึธืื ืึทืจืืขื, ืขืก ืืื ืืืืืืืง ืฆื ืคึฟืึทืจืฉืืืื ืืืึธืก ืื ืืืจืืคืืจืื ื ืืึธื ืืื ืคึฟืึทืจ ืึท DAG. ืืื ืึทืืจืคืืึธืื, DAG ืืื ืึทื ืขืงืกืขืงืืืืึธื ืืึธื ืืืกืืขืกืืื ื, ื"ื ืืืคึผืขื ืืื ื ืืืืฃ ืื ืึทืจืืขื ืคึผืืึทื ืคืื ืื DAG, ืึทืจืืขื ืื ืกืืึทื ืกืื ืืขื ืขื ืืืฉืืคื ืคึฟืึทืจ ืืขืืขืจ ืขืงืกืขืงืืืืึธื ืืึธื. ืืื ืคึฟืึทืจ ืืขืืขืจ ืขืงืกืขืงืืืืึธื ืืึธื, ืืึทืกืงืก ืงืขื ืขื ืืืื ืฉืืึทืขื-ืขืงืกืึทืงืืืืึทื - ืึธืืขืจ, ืืืฉื, ืึท DAG ืงืขื ืขื ืึทืจืืขืื ืกืืืืึทืืืืื ืืึทืกืื ืืื ืขืืืขืืข ืขืงืกืขืงืืืืึธื ืืึทืืขืก. ืืึธืก ืืื ืงืืืจ ืืขืืืืื ืืึธ:
ืฆืื ืืึทืืืืขืจื (ืึธืืขืจ ืืคึฟืฉืจ ืืืืง: ืขืก ืืขืคึผืขื ืืก ืืืืฃ ืื ืกืืืืึทืฆืืข), ืืืื ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืื ืึทืจืืขื ืืื ืื DAG ืืื ืงืขืจืขืงืืึทื, ืื ืืืจืืคืืจืื ื ืืื ืื ืคืจืืขืจืืืงืข ืขืงืกืขืงืืืืึธื ืืึธื ืืืขื ืคืึธืจืืขืฆื ืืื ืื ืึทืืืฉืึทืกืืืึทื ืฅ. ืืึธืก ืืื ืืื ืืืื ืืืจ ืืึทืจืคึฟื ืฆื ืจืืงืึทืืงืืึทืืืื ืืึทืื ืืื ืคืึทืจืืึทื ืืขื ืืืื ืคึผืืจืืึทืื ืืื ืึท ื ืืึทืข ืึทืืืขืจืืืึทื, ืึธืืขืจ ืืึธืก ืืื ืฉืืขืื ืืืืึทื ืื ืจืขืคึผืจืึธืืืกืืืืืืื ืคืื ืืขืจ ืจืขืืืืืึทื ืืื ืคืึทืจืคืึทืื (ืคืื ืงืืจืก, ืงืืื ืขืจ ืืึทืืขืจื ืืืจ ืฆื ืฆืืจืืงืงืืืขื ืื ืคืืจืืื ืื ืืืขืจืกืืข ืคืื โโโโืื ืืงืืจ ืงืึธื ืคึฟืื Git ืืื ืจืขืืขื ืขื ืืืึธืก ืืืจ ืืึทืจืคึฟื ืืืื ืืึธื, ืืื ืืืจ ืืึทืจืคึฟื ืขืก).
ืืืฉืขื ืขืจืืืืื ื ืืึทืกืงืก
ืื ืืืคึผืืึทืืขื ืืืืฉืึทื ืคืื ืื DAG ืืื ืงืึธื ืืื Python, ืึทืืื ืืืจ ืืึธืื ืึท ืืืืขืจ ืืึทืงืืืขื ืืืขื ืฆื ืจืขืืืฆืืจื ืื ืกืืืข ืคืื โโืงืึธื ืืืขื ืืจืืขืื, ืืืฉื, ืืื ืฉืขืจื ืงืืืืื. ืืื ืก ืืึธืื ืืืจ ืืึธืื ืืจืื MySQL ืฉืึทืจืื ืืื ืึท ืืงืืจ, ืืืจ ืืึทืจืคึฟื ืฆื ืงืจืืื ืืื ืืขืืขืจ ืืื ืงืืืึทืื ืขืืืขืืข ืืึทืื. ืืขืจืฆื, ืื ืืืคึผืขื ืืึทื ืืื ืืื ืืื ืคึผืึทืจืึทืืขื. ืื ืคึผืืืืึธื ืงืึธื ืืื ืื DAG ืงืขื ืงืืงื ืืื ืืึธืก:
connection_list = lv.get('connection_list')
export_profiles_sql = '''
SELECT
id,
user_id,
nickname,
gender,
{{params.shard_id}} as shard_id
FROM profiles
'''
for conn_id in connection_list:
export_profiles = SqlToHiveViaHdfsTransfer(
task_id='export_profiles_from_' + conn_id,
sql=export_profiles_sql,
hive_table='stg.profiles',
overwrite=False,
tmpdir='/data/tmp',
conn_id=conn_id,
params={'shard_id': conn_id[-1:], },
compress=None,
dag=dag
)
export_profiles.set_upstream(exec_truncate_stg)
export_profiles.set_downstream(load_profiles)ืื DAG ืงืืงื ืืื ืืึธืก:
ืืื ืืขื ืคืึทื, ืืืจ ืงืขื ืขื ืืืืื ืึธืืขืจ ืึทืจืึธืคึผื ืขืืขื ืึท ืฉืึทืจื ืืืจื ืคืฉืื ืึทืืืฉืึทืกืืื ื ืื ืกืขืืืื ืืก ืืื ืึทืคึผืืืืืื ื ืื DAG. ืืึทืงืืืขื!
ืืืจ ืงืขื ื ืืืื ื ืืฆื ืืขืจ ืงืึธืืคึผืืืฆืืจื ืงืึธื ืืืจ, ืืืฉื, ืึทืจืืขื ืืื ืงืืืืื ืืื ืื ืคืึธืจืขื ืคืื ืึท ืืึทืืึทืืืืก ืึธืืขืจ ืืึทืฉืจืืึทืื ืึท ืืืฉ ืกืืจืืงืืืจ, ืึท ืึทืืืขืจืืืึทื ืคึฟืึทืจ ืืจืืขืื ืืื ืึท ืืืฉ, ืืื, ืืขื ืืืขื ืืื ืืฉืืื ืื ืคึฟืขืึดืงืืืื ืคืื ืื DWH ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ, ืืืฉืขื ืขืจืืื ืึท ืคึผืจืึธืฆืขืก ืคึฟืึทืจ ืืึธืืืื ื N ืืืฉื ืืื ืืืื ืกืืึธืจืืืืฉ. ืึธืืขืจ, ืคึฟืึทืจ ืืืึทืฉืคึผืื, ืืจืืขืื ืืื ืึท ืึทืคึผื ืืืึธืก ืืื ื ืืฉื ืฉืืืฆื ืืจืืขืื ืืื ืึท ืคึผืึทืจืึทืืขืืขืจ ืืื ืื ืคืึธืจืขื ืคืื ืึท ืจืฉืืื, ืืืจ ืงืขื ืขื ืืืฉืขื ืขืจืืื N ืืึทืกืงืก ืืื ืึท DAG ืคึฟืื ืืขืจ ืจืฉืืื, ืืึทืืจืขื ืขืฆื ืื ืคึผืึทืจืึทืืขืืืืึทื ืคืื ืจืืงืืืขืก ืืื ืื API ืฆื ืึท ืืขืงื, ืืื ืกืงืจืึทืคึผ ืื ื ืืืืืง ืืึทืื ืคืื ืื ืึทืคึผื. ืคืืขืงืกืึทืืึทื!
ืจืืคึผืึทืืึทืืึธืจื
ืึทืืจืคืืึธืื ืืื ืืืื ืืืืืขื ืข ืืึทืงืขื ื ืจืืคึผืึทืืึทืืึธืจื, ืึท ืืึทืืึทืืืืก (ืงืขื ืืืื MySQL ืึธืืขืจ Postgres, ืืืจ ืืึธืื Postgres), ืืืึธืก ืกืืึธืจื ืื ืืึทืกืงืก ืกืืึทืฅ, ืืึทืืก, ืงืฉืจ ืกืขืืืื ืืก, ืืืืืืืข ืืืขืจืืึทืืึทืื, ืืื"ื ื. ืืึธ ืืื ืืืึธืื ืืื ืืื ืงืขื ืืึธืื ืึทื ืื ืจืืคึผืึทืืึทืืึธืจื ืืื ืึทืืจืคืืึธืื ืืื ืืืืขืจ ืคึผืฉืื (ืืืขืื 20 ืืืฉื) ืืื ืืึทืงืืืขื ืืืื ืืืจ ืืืืื ืฆื ืืืืขื ืืืื ืืืืืขื ืข ืคึผืจืึทืกืขืกืึทื ืืืืฃ ืฉืคึผืืฅ. ืืื ืืขืืขื ืง ืื 100500 ืืืฉื ืืื ืื Informatica ืจืืคึผืึทืืึทืืึธืจื, ืืืึธืก ืืื ืฆื ืืืื ืืขืืขืจื ื ืคึฟืึทืจ ืึท ืืึทื ื ืฆืืึทื ืืืืืขืจ ืืืจ ืคึฟืึทืจืฉืืืื ืืื ืฆื ืืืืขื ืึท ืึธื ืคึฟืจืขื.
ืืึธื ืืืึธืจืื ื
ืืขืืขืื ืื ืคึผืึทืฉืืขืก ืคืื ืื ืจืืคึผืึทืืึทืืึธืจื, ืืืจ ืงืขื ืขื ืืืืขื ืึท ืึทืจืืขื ืืึธื ืืืึธืจืื ื ืคึผืจืึธืฆืขืก ืืืึธืก ืืื ืืึทืงืืืขื ืคึฟืึทืจ ืืืจ. ืืืจ ื ืืฆื ืึท ื ืึธืืขืคึผืึทื ืืื Zeppelin, ืืื ืืืจ ืงืืงื ืืื ืื ืกืืึทืืืก ืคืื ืืึทืกืงืก:
ืืึธืก ืงืขื ืืืื ืืืื ืื ืืืขื ืฆืืืื ื ืคืื Airflow ืืื:
ืื ืึทืืจืคืืึธืื ืงืึธื ืืื ืึธืคึฟื ืืงืืจ, ืึทืืื ืืืจ ืืึธืื ืฆืืืขืืขืื ืึทืืขืจืืื ื ืฆื ืืขืืขืืจืึทื. ืืขืืขืจ ืคืืืกื ืืืง ืืืึทืฉืคึผืื ืคืื ืึท ืึทืจืืขื, ืืืื ืึท ืืขืืช ืึทืงืขืจื, ืกืคึผืึทืืก ืื ืืจืืคึผืข ืืื ืืขืืขืืจืึทื, ืืื ืื ืืื ืฆืข ืึทื ืืืืืงืืื ื ืืื ืฉืืืฆื ืืึทื ืฉืึทืคึฟื ืืืฉืืืื.
ืืืจ ืืึทืงืืืขื ืึท ืคึผืื ืืืขื ืขื ืืคืขืจ ืืืจื ืืขืืขืืจืึทื (ืืืื ืคืืจืืื ืื), ืืื ืืืจื Zeppelin ืืืจ ืืึทืงืืืขื ืึท ืงืืืืขืืืืง ืืืื ืคืื ืืึทืกืงืก ืืื ืึทืืจืคืืึธืื.
ืืื ืืึทื ืฅ
ืึทืืจืคืืึธืื ืืื ืืคึฟืจื ืึธืคึฟื ืืงืืจ, ืืื ืืืจ ืืึธื ื ืืฉื ืืขืจืืืึทืจืื ืืืจืึทืงืึทืื ืคืื ืขืก. ืืืื ืฆืืืขืืจืืื ืฆื ืฉืืขืื ืื ืฆืืื ืืื ืื ืฆื ืืืืขื ืึท ืืืืืื ื ืืืึธืก ืึทืจืืขื. ืืขืจ ืฆืื ืืื ืึทืืฉืืืืึทืืึทื, ืืืืืื ืืืจ, ืขืก ืืื ืืืขืจื ืขืก. ืืืืงืืึทื ืคืื ืึทื ืืืืืงืืื ื, ืืืืืืงืืึทื, ืื ืคืื ืึทืืื ื ื ืืึท ืคึผืจืึทืกืขืกืึทื - ืืืจ ืืืขื ืืื ืขืก. ืคืื ืงืืจืก, ืืืจ ืืึทืจืคึฟื ืฆื ืืึทืฆืึธืื ืึท ืคึผืืึทืฅ ืคืื ืืคืืขืจืงืืึทืืงืืึทื ืฆื ืืขืจ ืึธืจืืึทื ืืืึทืฆืืข ืคืื โโืื ืคึผืจืืืขืงื, ืื ืคืขืกืืงืืึทื ืคืื ืื ืึทืืจืคืืึธืื ืืื: ืืืจืึทืงืึทืื ืืึธื ื ืื ืคึผืึทืกืืจื.
ืืืฆื ืืืจ ืืึธืื ืึทืืจืคืืึธืื ืืจืืขืื ืืขืืืขื ืืืขืื 6,5 ืืืืื ื ืืึทืกืงืก. ืืื ืืขื ืขื ืืึทื ืฅ ืึทื ืืขืจืฉ ืืื ืืึทืจืึทืงืืขืจ. ืขืก ืืขื ืขื ืืึทืกืงืก ืคืื ืืึธืืืื ื ืืึทืื ืืื ืื ืืืืคึผื DWH ืคึฟืื ืคืืืข ืคืึทืจืฉืืืขื ืข ืืื ืืืืขืจ ืกืคึผืขืฆืืคืืฉ ืงืืืืื, ืขืก ืืขื ืขื ืืึทืกืงืก ืคืื ืงืึทืืงืืึทืืืืืื ื ืกืืึธืจืคืจืึทื ืฅ ืืื ืื ืืืืคึผื DWH, ืขืก ืืขื ืขื ืืึทืกืงืก ืคืื ืืจืืืกืืขืื ืืึทืื ืืื ืึท ืฉื ืขื DWH, ืขืก ืืขื ืขื ืคืืืข, ืคืืืข ืคืึทืจืฉืืืขื ืข ืืึทืกืงืก - ืืื Airflow. ืืฉืื ืืื ืึทืืข ืึทืจืืืฃ ืืึธื ื ืึธื ืืึธื. ืืขืจืขืื ืืื ื ืืืขืจื, ืืึธืก ืืื 2,3 ืืืืื ื ELT ืืึทืกืงืก ืคืื ืืืขืจืืื ื ืงืึทืืคึผืืขืงืกืืื ืืื DWH (Hadoop), ืืขืขืจืขื. 2,5 ืืื ืืขืจื ืืึทืืึทืืืืกืื ืืงืืจืื, ืืึธืก ืืื ืึท ืงืึธืืขืงืืืื ืคึฟืื 4 ETL ืืขืืืขืืึธืคึผืขืจืก, ืืืึธืก ืืขื ืขื ืฆืขืืืืื ืืื ETL ืืึทืื ืคึผืจืึทืกืขืกืื ื ืืื DWH ืืื ELT ืืึทืื ืคึผืจืึทืกืขืกืื ื ืืื DWH ืืื ืืึธื ืืขืจ ืืืื ืึทืืืื ืืกืืจืึทืืึธืจ, ืืืืก ืืืื ืืื ืื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ ืคืื ืื ืืื ืกื.
ืคึผืืึทื ืก ืคึฟืึทืจ ืื ืฆืืงืื ืคึฟื
ืื ื ืืืขืจ ืคืื ืคึผืจืึทืกืขืกืึทื ืืื ืื ืขืืืึทืืึทืืื ืืจืึธืืื ื, ืืื ืื ืืืืคึผื ืืึทื ืืืึธืก ืืืจ ืืืขืื ืืึธื ืืื ืืขืจืืื ืขื ืคืื ืื ืึทืืจืคืืึธืื ืื ืคืจืึทืกืืจืึทืงืืฉืขืจ ืืื ืกืงืืืืื ื. ืืืจ ืืืืื ืฆื ืืืืขื ืึท ืึทืืจืคืืึธืื ืงื ืืื, ืึทืืึทืงืืื ืึท ืคึผืึธืจ ืคืื ืืขืืก ืคึฟืึทืจ ืกืขืืืขืจืื ืืืขืจืก, ืืื ืืึทืื ืึท ืืื-ืืืคึผืืืงืืืืื ื ืงืึธืคึผ ืืื ืึทืจืืขื ืกืงืขืืืฉืืืื ื ืคึผืจืึทืกืขืกืึทื ืืื ืึท ืจืืคึผืึทืืึทืืึธืจื.
ืขืคึผืืืึธื
ืืึธืก, ืคืื ืงืืจืก, ืืื ื ืืฉื ืึทืืฅ ืืืึธืก ืืื ืืืึธืื ืืื ืฆื ืืึธืื ืืืขืื ืึทืืจืคืืึธืื, ืึธืืขืจ ืืื ืืขืคืจืืืื ืฆื ืืืืืคึผืื ืงื ืื ืืืืคึผื ืคืื ืงืื. ืึทืคึผืขืืื ืงืืื ืืื ืขืกื, ืคึผืจืืืืจื ืขืก ืืื ืืืจ ืืืขื ืืื ืขืก :)
ืืงืืจ: www.habr.com