🥇Ayyukan bango don Faust, Sashe na I: Gabatarwa

Ta yaya na zo rayuwa haka?

Ba da dadewa ba dole ne in yi aiki a kan baya na aikin da aka ɗora sosai, wanda ya zama dole don tsara aiwatar da aiwatar da ayyuka na yau da kullum na babban adadin ayyuka na baya tare da ƙididdiga masu rikitarwa da buƙatun sabis na ɓangare na uku. Aikin ba daidai ba ne kuma kafin in zo, yana da hanya mai sauƙi don ayyukan ƙaddamar da cron: madauki don bincika lokacin yanzu da ƙaddamar da ƙungiyoyin coroutines ta hanyar tattarawa - wannan hanyar ta zama abin karɓa har sai an sami ɗaruruwan da ɗaruruwan irin waɗannan coroutines. , duk da haka, lokacin da adadin su ya wuce dubu biyu, dole ne in yi tunani game da shirya jerin ayyuka na yau da kullum tare da dillali, ma'aikata da yawa, da sauransu.

Da farko na yanke shawarar gwada Celery, wanda na yi amfani da shi a baya. Saboda yanayin asynchronous na aikin, na nutse cikin tambayar na gani labarinkazalika aikin, wanda marubucin labarin ya kirkira.

Zan faɗi wannan, aikin yana da ban sha'awa sosai kuma yana aiki sosai cikin nasara a cikin sauran aikace-aikacen ƙungiyarmu, kuma marubucin da kansa ya ce ya sami damar fitar da shi zuwa samarwa ta hanyar amfani da tafkin asynchronous. Amma, abin takaici, bai dace da ni sosai ba, kamar yadda ya kasance matsala tare da ƙaddamar da ayyuka (duba. kungiyar). A lokacin rubutawa batun An riga an rufe, duk da haka, ana gudanar da aiki tsawon wata guda. A kowane hali, sa'a ga marubucin da duk mafi kyau, tun da an riga an riga an yi aiki a kan lib ... gaba ɗaya, ma'anar yana cikin ni kuma kayan aiki ya zama mai laushi a gare ni. Bugu da ƙari, wasu ayyuka suna da buƙatun 2-3 http zuwa ayyuka daban-daban, don haka ko da lokacin inganta ayyuka, muna ƙirƙirar haɗin TCP dubu 4, kusan kowane sa'o'i 2 - ba mai kyau ba ... Ina so in ƙirƙiri zaman don nau'in nau'i ɗaya. aiki lokacin fara ma'aikata. Ƙari kaɗan game da ɗimbin buƙatun ta hanyar aiohttp a nan.

Game da wannan, na fara dubawa madadin kuma ya same shi! Masu kirkiro seleri, musamman, kamar yadda na fahimta Tambayi Sulaiman, an halicce su Faust, asali don aikin robinhood. Faust yana da wahayi daga Kafka Streams kuma yana aiki tare da Kafka a matsayin dillali, rocksdb kuma ana amfani dashi don adana sakamako daga aikin wakilai, kuma abu mafi mahimmanci shine cewa ɗakin karatu bai dace ba.

Hakanan, kuna iya duba saurin kwatanta seleri da faust daga masu kirkiro na karshen: bambance-bambancen su, bambance-bambance tsakanin dillalai, aiwatar da wani aiki na farko. Komai abu ne mai sauqi qwarai, duk da haka, kyakkyawan yanayin a cikin faust yana jan hankali - bayanan da aka buga don watsawa ga batun.

Me muke yi?

Don haka, a cikin ɗan gajeren jerin labarai, zan nuna muku yadda ake tattara bayanai daga ayyukan baya ta amfani da Faust. Tushen aikin misalin mu zai kasance, kamar yadda sunan ya nuna, alphavantage.co. Zan nuna yadda ake rubuta wakilai ( nutsewa, batutuwa, ɓangarorin), yadda ake yin kisa na yau da kullun (cron), umarni mafi dacewa faust cli (a kunsa a kan dannawa), gungu mai sauƙi, kuma a ƙarshe za mu haɗe da datadog ( aiki daga cikin akwatin) kuma gwada wani abu don gani. Don adana bayanan da aka tattara za mu yi amfani da mongodb da mota don haɗi.

PS Yin la'akari da amincewa da abin da aka rubuta batun game da saka idanu, ina tsammanin cewa mai karatu a ƙarshen labarin da ya gabata zai kasance kamar haka:

Bukatun aikin

Saboda gaskiyar cewa na riga na yi alkawari, bari mu yi ɗan ƙaramin jerin abubuwan da ya kamata sabis ɗin ya iya yi:

Loda bayanan tsaro da bayyaninsu (ciki har da riba da asara, takardar ma'auni, tsabar kuɗi - na shekarar da ta gabata) - akai-akai.
Loda bayanan tarihi (na kowace shekara ta kasuwanci, sami matsananciyar ƙimar farashin rufewar ciniki) - akai-akai
Loda sabbin bayanan ciniki - akai-akai
Loda lissafin da aka keɓance na alamomi don kowane tsaro - akai-akai

Kamar yadda aka zata, mun zaɓi suna don aikin daga karce: Horton

Muna shirya abubuwan more rayuwa

Taken tabbas yana da ƙarfi, duk da haka, duk abin da kuke buƙatar yi shine rubuta ƙaramin tsari don docker-compose tare da kafka (da zookeeper - a cikin akwati ɗaya), kafdrop (idan muna son duba saƙonni a cikin batutuwa), mongodb. Mun samu [docker-compose.yml](https://github.com/Egnod/horton/blob/562fa5ec14df952cd74760acf76e141707d2ef58/docker-compose.yml) na fom mai zuwa:

version: '3'

services:
  db:
    container_name: horton-mongodb-local
    image: mongo:4.2-bionic
    command: mongod --port 20017
    restart: always
    ports:
      - 20017:20017
    environment:
      - MONGO_INITDB_DATABASE=horton
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password

  kafka-service:
    container_name: horton-kafka-local
    image: obsidiandynamics/kafka
    restart: always
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

  kafdrop:
    container_name: horton-kafdrop-local
    image: 'obsidiandynamics/kafdrop:latest'
    restart: always
    ports:
      - '9000:9000'
    environment:
      KAFKA_BROKERCONNECT: kafka-service:29092
    depends_on:
      - kafka-service

Babu wani abu mai rikitarwa a nan kwata-kwata. An bayyana masu sauraro biyu don kafka: ɗaya (na ciki) don amfani a cikin hanyar sadarwa mai haɗawa, da na biyu (na waje) don buƙatun daga waje, don haka sun tura shi waje. 2181 - tashar jiragen ruwa na zookeeper. Sauran, ina tsammanin, a bayyane yake.

Ana shirya kwarangwal na aikin

A cikin sigar asali, tsarin aikinmu yakamata yayi kama da haka:

horton
├── docker-compose.yml
└── horton
    ├── agents.py *
    ├── alphavantage.py *
    ├── app.py *
    ├── config.py
    ├── database
    │   ├── connect.py
    │   ├── cruds
    │   │   ├── base.py
    │   │   ├── __init__.py
    │   │   └── security.py *
    │   └── __init__.py
    ├── __init__.py
    ├── records.py *
    └── tasks.py *

*Duk abin da na lura Ba mu taɓa shi ba tukuna, kawai muna ƙirƙirar fayiloli marasa komai.**

Mun halicci tsari. Yanzu bari mu ƙara abin dogaro, rubuta config kuma haɗa zuwa mongodb. Ba zan ba da cikakken rubutun fayiloli a cikin labarin ba, don kada in jinkirta shi, amma zan samar da hanyoyin haɗi zuwa sigogin da suka dace.

Bari mu fara da dogaro da meta game da aikin - pyproject.toml

Na gaba, za mu fara shigar da abin dogaro da ƙirƙirar virtualenv (ko kuma kuna iya ƙirƙirar babban fayil ɗin venv da kanku kuma kunna yanayin):

pip3 install poetry (если ещё не установлено)
poetry install

Yanzu bari mu ƙirƙira daidaita.yml - Takaddun shaida da inda za a buga. Kuna iya nan da nan sanya bayanai don haruffa a wurin. To, mu ci gaba zuwa config.py - Cire bayanai don aikace-aikacen daga tsarin mu. Ee, na furta, na yi amfani da lib na - sitri.

Lokacin haɗi zuwa Mongo, komai yana da sauƙi. sanar aji na abokin ciniki don haɗawa da tushe aji don cruds, don sauƙaƙa yin tambayoyi akan tarin.

Me zai faru a gaba?

Labarin ba shi da tsayi sosai, tun da yake a nan ina magana ne kawai game da dalili da shirye-shirye, don haka kada ku zargi ni - Na yi alkawari cewa sashi na gaba zai sami aiki da zane-zane.

To, a wannan bangare na gaba mu:

Bari mu rubuta ƙaramin abokin ciniki don haruffa akan aiohttp tare da buƙatun ƙarshen abubuwan da muke buƙata.
Bari mu ƙirƙiri wani wakili wanda zai tattara bayanai kan aminci da farashin tarihi a gare su.

Lambar aikin

Code don wannan bangare

source: www.habr.com

Ayyukan Fage akan Faust, Sashe na I: Gabatarwa