🥇Ekumashure mabasa eFaust, Chikamu I: Nhanganyaya

Ko ndakazorarama sei zvakadai?

Munguva pfupi yapfuura ndaifanira kushanda kumashure kwepurojekiti yakaremerwa, umo zvaive zvakafanira kuronga nguva dzose kuitwa kwenhamba yakawanda yemabasa ekumashure nekuverenga kwakaoma uye zvikumbiro zvebasa rechitatu. Iyo purojekiti ndeye asynchronous uye ndisati ndauya, yaive neyakapusa nzira yekutangisa-cron-mabasa: loop yekutarisa nguva iripo uye kuvhura mapoka emakoroti kuburikidza nekuungana - nzira iyi yakave inogamuchirwa kusvika paine gumi nemaviri nemazana emakoroti akadaro. , zvisinei, apo nhamba yavo yakapfuura zviuru zviviri, ndaifanira kufunga nezve kuronga mutsara webasa wenguva dzose nebhuroka, vashandi vakati wandei, zvichingodaro.

Kutanga ndakasarudza kuedza Celery, yandakanga ndamboshandisa. Nekuda kweasynchronous chimiro cheprojekiti, ndakanyura mumubvunzo ndikaona chinyorwa,uye ronga, yakagadzirwa nemunyori wechinyorwa.

Ini ndichataura izvi, chirongwa ichi chinonakidza uye chinoshanda zvakabudirira mune mamwe maapplication echikwata chedu, uye munyori pachake anoti akakwanisa kuiburitsa mukugadzira nekushandisa asynchronous dziva. Asi, zvinosuruvarisa, hazvina kunyatsondikodzera, sezvazvakazoitika dambudziko nekutangwa kweboka kwemabasa (ona. boka) Panguva yekunyora nyaya yakatovharwa, zvisinei, basa rave nemwedzi. Chero zvazvingava, rombo rakanaka kune munyori uye zvose zvakanakisisa, sezvo pane zvatova kushanda zvinhu pa lib ... kazhinji, pfungwa iri mandiri uye chigadziro chakashanduka kuva chakanyorova kwandiri. Uye zvakare, mamwe mabasa aive ne2-3 http zvikumbiro kune akasiyana masevhisi, saka kunyangwe kana optimize mabasa, isu tinogadzira zviuru zvina zveTCP zvinongedzo, anenge maawa maviri ega ega - hazvina kunaka kwazvo ... ndinoda kugadzira musangano werudzi rumwe basa pakutanga vashandi. Zvimwe zvishoma nezve nhamba huru yezvikumbiro kuburikidza neaiohttp pano.

Panyaya iyi, ndakatanga kutarisa dzimwe nzira ndokuiwana! Vagadziri vecelery, kunyanya, sezvandinozvinzwisisa Bvunza Solem, yakasikwa Faust, pakutanga kweprojekti utsotsi. Faust inofemerwa neKafka Streams uye inoshanda neKafka sebroker, rocksdb inoshandiswawo kuchengetedza mhedzisiro kubva kubasa revashandi, uye chinonyanya kukosha ndechekuti raibhurari iri asynchronous.

Uyezve, unogona kutarisa kuenzanisa nekukurumidza celery uye faust kubva kune vagadziri vekupedzisira: misiyano yavo, mutsauko pakati pemabhuroka, kuita kwekutanga basa. Zvese zviri nyore, zvisinei, chinhu chakanaka mufaust chinokwezva kutarisa - yakanyorwa data yekufambisa kune iyo musoro.

Tinoita sei?

Saka, mune pfupi pfupi yezvinyorwa, ini ndinokuratidza nzira yekuunganidza data kubva kumashure mabasa uchishandisa Faust. Mhedzisiro yeprojekiti yedu yemuenzaniso ichave, sekureva kunoita zita, alphavantage.co. Ini ndicharatidza manyorero maajenti (kunyura, misoro, zvikamu), maitiro ekuita nguva dzose (cron) kuuraya, iyo yakanyanya nyore faust cli mirairo (kuputira pamusoro pekudzvanya), kusanganisa kwakapusa, uye kumagumo isu tichabatanidza datadog ( kushanda kunze kwebhokisi) uye edza kune chimwe chinhu kuti uone. Kuchengeta iyo data yakaunganidzwa isu tinoshandisa mongodb uye mota yekubatanidza.

PS Tichifunga nekuvimba kwakanyorwa pfungwa yekutarisa, ndinofunga kuti muverengi pakupera kwechinyorwa chekupedzisira acharamba achitaridzika seizvi:

Project zvinodiwa

Nekuda kwekuti ini ndakatovimbisa, ngatiite runyoro rudiki rwezvinofanira kuitwa nesevhisi:

Isa zvibatiso uye mucherechedzo wazvo (kusanganisira purofiti uye kurasikirwa, chiyero chemari, kubuda kwemari - kwegore rapfuura) - nguva dzose
Isa nhoroondo yedata (yega rega rekutengesa gore, tsvaga yakanyanyisa kukosha yemutengo wekuvhara wekutengesa) - nguva dzose
Isa data razvino rekutengesa - nguva nenguva
Isa rondedzero yakagadziridzwa yezviratidzo kune yega yega chengetedzo - nguva dzose

Sezvinotarisirwa, tinosarudza zita repurojekiti kubva pakutanga: Horton

Tiri kugadzirira zvivakwa

Musoro wacho wakasimba, zvisinei, zvese zvaunofanirwa kuita kunyora diki config ye docker-compose ne kafka (uye zookeeper - mumudziyo mumwe), kafdrop (kana tichida kutarisa mameseji mumisoro), mongodb. Tinowana [docker-compose.yml](https://github.com/Egnod/horton/blob/562fa5ec14df952cd74760acf76e141707d2ef58/docker-compose.yml) yefomu rinotevera:

version: '3'

services:
  db:
    container_name: horton-mongodb-local
    image: mongo:4.2-bionic
    command: mongod --port 20017
    restart: always
    ports:
      - 20017:20017
    environment:
      - MONGO_INITDB_DATABASE=horton
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password

  kafka-service:
    container_name: horton-kafka-local
    image: obsidiandynamics/kafka
    restart: always
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

  kafdrop:
    container_name: horton-kafdrop-local
    image: 'obsidiandynamics/kafdrop:latest'
    restart: always
    ports:
      - '9000:9000'
    environment:
      KAFKA_BROKERCONNECT: kafka-service:29092
    depends_on:
      - kafka-service

Hapana chakaoma apa zvachose. Vateereri vaviri vakaziviswa kafka: imwe (yemukati) yekushandisa mukati metiweki inoumbwa, uye yechipiri (yekunze) yezvikumbiro kubva kunze, saka vakaitumira kunze. 2181 - zookeeper port. Zvimwe, ndinofunga, zvakajeka.

Kugadzirira skeleton yeprojekti

Mune iyo yekutanga vhezheni, chimiro chepurojekiti yedu chinofanira kutaridzika seizvi:

horton
├── docker-compose.yml
└── horton
    ├── agents.py *
    ├── alphavantage.py *
    ├── app.py *
    ├── config.py
    ├── database
    │   ├── connect.py
    │   ├── cruds
    │   │   ├── base.py
    │   │   ├── __init__.py
    │   │   └── security.py *
    │   └── __init__.py
    ├── __init__.py
    ├── records.py *
    └── tasks.py *

*Zvese zvandakaona Hatisati tazvibata, tinongogadzira mafaira asina chinhu.**

Isu takagadzira chimiro. Zvino ngatiwedzerei zvinodiwa zvinoenderana, nyora iyo config uye ubatanidze kune mongodb. Ini handisi kuzopa iwo mameseji akazara emafaira ari muchinyorwa, kuti urege kunonoka, asi ini ndichapa zvinongedzo kune dzinodiwa shanduro.

Ngatitangei nekutsamira uye meta nezve purojekiti - pyproject.toml

Tevere, tinotanga kuisa zvinotsamira uye kugadzira virtualenv (kana iwe unogona kugadzira iyo venv dhairekitori pachako uye kumisa nharaunda):

pip3 install poetry (если ещё не установлено)
poetry install

Zvino ngatigadzirirei config.yml - Credentials uye kupi kugogodza. Iwe unogona pakarepo kuisa data yealphavantage ipapo. Zvakanaka, ngatienderere mberi config.py - bvisa data rekushandisa kubva kune yedu config. Hongu, ndinoreurura, ndakashandisa lib yangu - sitri.

Kana uchibatanidza kuMongo, zvese zviri nyore. zvakaziviswa kirasi yevatengi kubatana uye base class zvemacruds, kuti zvive nyore kuita mibvunzo pane zvakaunganidzwa.

Chii chichatevera kuitika?

Chinyorwa chacho hachina kureba, sezvo pano ndiri kungotaura nezvekukurudzira uye kugadzirira, saka usandipe mhosva - ndinovimbisa kuti chikamu chinotevera chichava nekuita uye magirafu.

Saka, mune ino chikamu chinotevera isu:

Ngatinyorei mutengi mudiki kune alphavantage paaiohttp nezvikumbiro zvemagumo atinoda.
Ngatigadzirei mumiririri anozounganidza data pamusoro pezvivimbiso nemitengo yenhoroondo yavo.

Project code

Kodhi yechikamu ichi

Source: www.habr.com

Kumashure Mabasa paFaust, Chikamu I: Nhanganyaya