🥇Imisebenzi yangasemva yeFaust, iCandelo I: Intshayelelo

Ndenze njani ukuba ndiphile ngolu hlobo?

Kungekudala kwafuneka ndisebenze kwi-backend yeprojekthi elayishwe kakhulu, apho kwakuyimfuneko ukuququzelela ukuphunyezwa rhoqo kwenani elikhulu lemisebenzi yangasemva kunye nezibalo ezinzima kunye nezicelo zeenkonzo zomntu wesithathu. Iprojekthi i-asynchronous kwaye ngaphambi kokuba ndifike, yayinendlela elula yokwenza imisebenzi yokuqalisa i-cron: i-loop ejonga ixesha langoku kunye nokusungula amaqela e-coroutines ngokuhlanganisa- le ndlela iye yamkeleka de kwabakho ishumi elinambini kunye namakhulu ee-coroutines. , nangona kunjalo, xa inani labo lidlula amawaka amabini, kwafuneka ndicinge malunga nokulungelelanisa umgca wemisebenzi eqhelekileyo kunye nomthengisi, abasebenzi abaninzi, njalo njalo.

Okokuqala ndagqiba ekubeni ndizame iCelery, endandikhe ndayisebenzisa ngaphambili. Ngenxa yobume beprojekthi, ndiye ndangena kumbuzo ndabona inqaku, kwaye iprojekthi, eyenziwe ngumbhali wenqaku.

Ndiya kuthetha oku, iprojekthi inomdla kakhulu kwaye isebenza ngempumelelo kwezinye izicelo zeqela lethu, kwaye umbhali ngokwakhe uthi ukwazile ukuyikhupha kwimveliso ngokusebenzisa i-asynchronous pool. Kodwa, ngelishwa, ayizange indifanele ngokwenene, njengoko kwavelayo ingxaki ngokusungulwa kweqela lemisebenzi (bona. iqela). Ngexesha lokubhala khupha sele ivaliwe, nangona kunjalo, umsebenzi uqhubekile inyanga. Kwimeko nayiphi na into, inhlanhla kumbhali kunye nazo zonke izinto ezilungileyo, ekubeni sele sele zisebenza izinto kwi-lib ... ngokubanzi, inqaku likum kwaye isixhobo sajika saba ngumanzi kum. Ukongeza, eminye imisebenzi ine-2-3 yezicelo ze-http kwiinkonzo ezahlukeneyo, ngoko ke naxa silungiselela imisebenzi, senza amawaka ama-4 onxibelelwano lwe-TCP, malunga neeyure ezi-2 - hayi kakhulu ... Ndingathanda ukwenza iseshoni yohlobo olunye umsebenzi xa uqala abasebenzi. Okungakumbi malunga nenani elikhulu lezicelo nge aiohttp apha.

Kule nkalo, ndaqala ukukhangela ezinye iindlela kwaye wayifumana! Abadali be-celery, ngokukodwa, njengoko ndiyiqonda Buza uSolem, yadalwa Faust, ekuqaleni iprojekthi ukuphanga. I-Faust iphefumlelwe yi-Kafka Streams kwaye isebenza noKafka njengomthengisi, i-rocksdb iphinda isetyenziswe ukugcina iziphumo ezivela kumsebenzi wama-agent, kwaye into ebaluleke kakhulu kukuba ilayibrari i-asynchronous.

Kwakhona, unokujonga uthelekiso olufutshane i-celery kunye ne-faust evela kubadali bokugqibela: ukungafani kwabo, ukungafani phakathi kwabathengi, ukuphunyezwa komsebenzi osisiseko. Yonke into ilula, nangona kunjalo, inqaku elihle kwi-faust litsala ingqalelo - idatha echwetheziweyo yokudluliselwa kwisihloko.

Senza ntoni?

Ke, kuthotho olufutshane lwamanqaku, ndiza kukubonisa indlela yokuqokelela idatha kwimisebenzi yangasemva usebenzisa iFaust. Umthombo weprojekthi yethu yomzekelo iya kuba, njengoko igama libonisa, alphavantage.co. Ndiza kubonisa indlela yokubhala ii-agent (i-sink, izihloko, izahlulo), indlela yokwenza rhoqo (cron) ukubulawa, eyona miyalelo ilula kakhulu ye-cli faust (i-wrapper phezu kokuchofoza), i-clustering elula, kwaye ekugqibeleni siya kuqhoboshela idathadog ( ukusebenza ngaphandle kwebhokisi) kwaye uzame into yokubona. Ukugcina idatha eqokelelweyo siya kusebenzisa i-mongodb kunye ne-motor ukudibanisa.

P.S. Ukujonga ngokuzithemba ekwabhalwa ngalo umhlathi malunga nokubeka iliso, ndicinga ukuba umfundi osekupheleni kwenqaku lokugqibela usajongeka ngolu hlobo:

Iimfuno zeprojekthi

Ngenxa yokuba sele ndithembisile, masenze uluhlu oluncinci lwento ekufuneka ikwazi ukuyenza inkonzo:

Layisha izibambiso kunye nesishwankathelo sazo (kubandakanywa iingeniso kunye nelahleko, i-balance sheet, ukuhamba kwemali - kulo nyaka uphelileyo) - rhoqo
Layisha idatha yembali (kunyaka ngamnye wokurhweba, fumana amaxabiso agqithisileyo exabiso lokuvala lokuthengisa) - rhoqo
Layisha idatha yamva nje yokurhweba - rhoqo
Layisha uluhlu olulungiselelweyo lwezalathi zokhuseleko ngalunye - rhoqo

Njengoko kulindelekile, sikhetha igama leprojekthi ukusuka ekuqaleni: Horton

Silungisa iziseko zophuhliso

Isihloko sinamandla ngokuqinisekileyo, nangona kunjalo, konke okufuneka ukwenze kukubhala uqwalaselo oluncinci lwe-docker-qamba ngekafka (kunye nomgcini-mgcini-kwisikhongozeli esinye), kafdrop (ukuba sifuna ukujonga imiyalezo kwizihloko), mongodb. Sifumana [docker-compose.yml](https://github.com/Egnod/horton/blob/562fa5ec14df952cd74760acf76e141707d2ef58/docker-compose.yml) ngolu hlobo lulandelayo:

version: '3'

services:
  db:
    container_name: horton-mongodb-local
    image: mongo:4.2-bionic
    command: mongod --port 20017
    restart: always
    ports:
      - 20017:20017
    environment:
      - MONGO_INITDB_DATABASE=horton
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password

  kafka-service:
    container_name: horton-kafka-local
    image: obsidiandynamics/kafka
    restart: always
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

  kafdrop:
    container_name: horton-kafdrop-local
    image: 'obsidiandynamics/kafdrop:latest'
    restart: always
    ports:
      - '9000:9000'
    environment:
      KAFKA_BROKERCONNECT: kafka-service:29092
    depends_on:
      - kafka-service

Akukho nto inzima apha konke konke. Abaphulaphuli ababini babhengezwa nge-kafka: enye (yangaphakathi) yokusetyenziswa ngaphakathi kwenethiwekhi edibeneyo, kwaye eyesibini (yangaphandle) yezicelo ezivela ngaphandle, ngoko bayithumela ngaphandle. 2181 - izibuko lomgcini wezilwanyana. Eminye, ndiyacinga, icacile.

Ukulungiselela i-skeleton yeprojekthi

Kuhlobo olusisiseko, ubume beprojekthi yethu kufuneka bubukeke ngolu hlobo:

horton
├── docker-compose.yml
└── horton
    ├── agents.py *
    ├── alphavantage.py *
    ├── app.py *
    ├── config.py
    ├── database
    │   ├── connect.py
    │   ├── cruds
    │   │   ├── base.py
    │   │   ├── __init__.py
    │   │   └── security.py *
    │   └── __init__.py
    ├── __init__.py
    ├── records.py *
    └── tasks.py *

*Yonke into ndiyiqaphele Asikayichukumisi, sidala iifayile ezingenanto.**

Sakha isakhiwo. Ngoku masenze ukuxhomekeka okuyimfuneko, bhala uqwalaselo kwaye uqhagamshele kwimongodb. Andiyi kubonelela ngokubhaliweyo okupheleleyo kweefayile kwinqaku, ukwenzela ukuba ungayilibazisi, kodwa ndiya kunika amakhonkco kwiinguqulelo eziyimfuneko.

Masiqale ngokuxhomekeka kunye nemeta malunga neprojekthi - pyproject.toml

Okulandelayo, siqala ukufaka ukuxhomekeka kunye nokwenza i-virtualenv (okanye unokwenza ifolda ye-venv ngokwakho kwaye uvule imeko-bume):

pip3 install poetry (если ещё не установлено)
poetry install

Ngoku masidale config.yml -Iziqinisekiso kunye nendawo yokunkqonkqoza. Ungabeka ngoko nangoko idatha yealphavantage apho. Kulungile, masiqhubele phambili config.py -khupha idatha yosetyenziso kulungiselelo lwethu. Ewe, ndiyavuma, ndisebenzise i-lib yam- sitri.

Xa uqhagamshela kwiMongo, yonke into ilula kakhulu. ibhengezwe iklasi yomxhasi ukudibanisa kunye iklasi yesiseko kwicruds, ukwenza kube lula ukwenza imibuzo ngengqokelela.

Kuza kwenzeka ntoni emva koko?

Inqaku alikho lide kakhulu, kuba apha ndithetha kuphela ngenkuthazo kunye nokulungiselela, ngoko ungandibeki tyala - ndiyathembisa ukuba inxalenye elandelayo iya kuba nentshukumo kunye nemizobo.

Ke, kweli candelo lilandelayo siza:

Masibhale umxhasi omncinci wealphavantage kwi-aiohttp kunye nezicelo zeesiphelo esizifunayo.
Masidale i-arhente eya kuqokelela idatha kwi-securities kunye namaxabiso embali kubo.

Ikhowudi yeprojekthi

Ikhowudi yale nxalenye

umthombo: www.habr.com

Imvelaphi yeMisebenzi kwiFaust, iCandelo I: Intshayelelo