🥇Imisebenzi yangemuva ka-Faust, Ingxenye I: Isingeniso

Kwenzeka kanjani ukuba ngiphile kanje?

Esikhathini esingengakanani esidlule kwakudingeka ngisebenze ngemuva kwephrojekthi elayishwe kakhulu, lapho kwakudingeka khona ukuhlela ukukhishwa okuvamile kwenani elikhulu lemisebenzi yangemuva ngezibalo eziyinkimbinkimbi kanye nezicelo zezinsizakalo zezinkampani zangaphandle. Le phrojekthi i-asynchronous futhi ngaphambi kokuthi ngifike, ibinomshini olula wokwethulwa kwe-cron: iluphu ebheka isikhathi samanje kanye nokwethulwa kwamaqembu ama-coroutines ngokuhlangana - le ndlela yamukeleka kuze kube yilapho sekukhona inqwaba namakhulu ama-coroutines anjalo. , nokho, lapho inani labo lidlula izinkulungwane ezimbili, kwadingeka ngicabange ngokuhlela umugqa wemisebenzi evamile nomthengisi, izisebenzi eziningana, njalonjalo.

Okokuqala nginqume ukuzama i-Celery, engangiyisebenzise ngaphambili. Ngenxa yokungavumelani kwephrojekthi, ngingene embuzweni ngabona indatshanafuthi iphrojekthi, edalwe umbhali we-athikili.

Ngizokusho lokhu, iphrojekthi iyathakazelisa kakhulu futhi isebenza ngempumelelo kwezinye izinhlelo zokusebenza zethimba lethu, futhi umbhali ngokwakhe uthi ukwazile ukuyikhipha ekukhiqizeni ngokusebenzisa ichibi elingavumelanisiwe. Kodwa, ngeshwa, akuzange kungifanele ngempela, njengoba kwenzeka inkinga ngokuqaliswa kweqembu kwemisebenzi (bona. iqembu). Ngesikhathi sokuloba Ukukhishwa isivele ivaliwe, nokho, umsebenzi usunenyanga uqhubeka. Kunoma yikuphi, inhlanhla kumlobi kanye nakho konke okuhle kakhulu, njengoba sekuvele kukhona izinto ezisebenzayo ku-lib ... ngokujwayelekile, iphuzu likimi futhi ithuluzi laphenduka laba manzi kimi. Ngaphezu kwalokho, eminye imisebenzi yayinezicelo ezingu-2-3 ze-http ezinsizeni ezahlukene, ngakho-ke ngisho nalapho silungiselela imisebenzi, sakha uxhumo lwe-TCP oluyizinkulungwane ezingu-4, cishe njalo ngamahora angu-2 - akukuhle kakhulu... Ngingathanda ukudala iseshini yohlobo olulodwa umsebenzi lapho uqala abasebenzi. Okuthe xaxa mayelana nenani elikhulu lezicelo nge-aiohttp lapha.

Mayelana nalokhu, ngaqala ukubheka ezinye izindlela futhi wayithola! Abadali besilimo esidliwayo esinamagatsha anamanzi, ikakhulukazi, njengoba ngikuqonda Buza uSolem, kwadalwa Faust, ekuqaleni kwephrojekthi ubugebengu. I-Faust iphefumulelwe yi-Kafka Streams futhi isebenza no-Kafka njengomthengisi, i-rocksdb iphinde isetshenziselwe ukugcina imiphumela evela emsebenzini wama-ejenti, futhi into ebaluleke kakhulu ukuthi umtapo wezincwadi awuvumelanisi.

Futhi, ungabheka ukuqhathanisa okusheshayo isilimo esidliwayo esinamagatsha anamanzi kanye ne-faust evela kubadali bokugcina: umehluko wabo, umehluko phakathi kwabathengi, ukuqaliswa komsebenzi wokuqala. Yonke into ilula, nokho, isici esihle ku-faust sidonsa ukunaka - idatha ethayiphiwe ukuze idluliselwe esihlokweni.

Senzenjani?

Ngakho-ke, ochungechungeni olufushane lwezihloko, ngizokukhombisa ukuthi ungayiqoqa kanjani idatha kusuka emisebenzini yangemuva usebenzisa i-Faust. Umthombo wephrojekthi yethu yesibonelo uzoba, njengoba igama liphakamisa, alphavantage.co. Ngizobonisa indlela yokubhala ama-ejenti (ukucwilisa, izihloko, ukwahlukanisa), indlela yokwenza okuvamile (cron) ukubulawa, imiyalo elula kakhulu ye-faust cli (i-wrapper phezu kokuchofoza), ukuhlanganisa okulula, futhi ekugcineni sizonamathisela idathadog ( ukusebenza ngaphandle kwebhokisi) bese uzama okuthile ongakubona. Ukugcina idatha eqoqiwe sizosebenzisa i-mongodb ne-motor ukuze sixhumeke.

I-PS Ngokubheka ukuzethemba okwabhalwa ngakho iphuzu mayelana nokuqapha, ngicabanga ukuthi umfundi ekupheleni kwesihloko sokugcina usazobukeka kanje:

Izidingo zephrojekthi

Ngenxa yokuthi sengivele ngithembisile, masenze uhlu oluncane lwalokho okumele kwenziwe yisevisi:

Layisha izibambiso kanye nohlaka lwazo (okuhlanganisa inzuzo nokulahlekelwa, ishidi lebhalansi, ukuhamba kwemali - ngonyaka odlule) - njalo
Layisha idatha yomlando (ngonyaka ngamunye wokuhweba, thola amanani adlulele wentengo yokuvala yokuhweba) - njalo
Layisha idatha yokuhweba yakamuva - njalo
Layisha uhlu olwenziwe ngokwezifiso lwezinkomba zokuvikela ngakunye - njalo

Njengoba kulindelekile, sikhetha igama lephrojekthi kusukela ekuqaleni: Horton

Silungisa ingqalasizinda

Isihloko siqinile impela, nokho-ke, okudingeka ukwenze nje ukubhala ukulungiselelwa okuncane kwe-docker-compose nge-kafka (kanye ne-zookeeper - esitsheni esisodwa), i-kafdrop (uma sifuna ukubheka imilayezo ezihlokweni), mongodb. Siyathola [i-docker-compose.yml](https://github.com/Egnod/horton/blob/562fa5ec14df952cd74760acf76e141707d2ef58/docker-compose.yml) kuleli fomu elilandelayo:

version: '3'

services:
  db:
    container_name: horton-mongodb-local
    image: mongo:4.2-bionic
    command: mongod --port 20017
    restart: always
    ports:
      - 20017:20017
    environment:
      - MONGO_INITDB_DATABASE=horton
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password

  kafka-service:
    container_name: horton-kafka-local
    image: obsidiandynamics/kafka
    restart: always
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

  kafdrop:
    container_name: horton-kafdrop-local
    image: 'obsidiandynamics/kafdrop:latest'
    restart: always
    ports:
      - '9000:9000'
    environment:
      KAFKA_BROKERCONNECT: kafka-service:29092
    depends_on:
      - kafka-service

Akukho lutho oluyinkimbinkimbi lapha. Abalaleli ababili bamenyezelwe nge-kafka: eyodwa (yangaphakathi) ukuze isetshenziswe ngaphakathi kwenethiwekhi eyinhlanganisela, neyesibili (yangaphandle) yezicelo ezivela ngaphandle, ngakho bayidlulisela ngaphandle. 2181 - ichweba lezookeeper. Okunye, ngicabanga, kucacile.

Ukulungisa uhlaka lwamathambo ephrojekthi

Kunguqulo eyisisekelo, ukwakheka kwephrojekthi yethu kufanele kubukeke kanje:

horton
├── docker-compose.yml
└── horton
    ├── agents.py *
    ├── alphavantage.py *
    ├── app.py *
    ├── config.py
    ├── database
    │   ├── connect.py
    │   ├── cruds
    │   │   ├── base.py
    │   │   ├── __init__.py
    │   │   └── security.py *
    │   └── __init__.py
    ├── __init__.py
    ├── records.py *
    └── tasks.py *

*Konke engikuqaphelile Asikayithinti, sivele sidale amafayela angenalutho.**

Sakhe isakhiwo. Manje ake sengeze ukuncika okudingekayo, bhala i-config bese uxhuma ku-mongodb. Ngeke nginikeze umbhalo ogcwele wamafayela ku-athikili, ukuze ngingayibambezeli, kodwa ngizohlinzeka ngezixhumanisi ezinguqulweni ezidingekayo.

Ake siqale ngokuncika kanye nemeta mayelana nephrojekthi - pyproject.toml

Okulandelayo, siqala ukufaka ukuncika nokudala i-virtualenv (noma ungakha ifolda ye-venv ngokwakho futhi wenze kusebenze imvelo):

pip3 install poetry (если ещё не установлено)
poetry install

Manje masidale config.yml - Imininingwane nokuthi ungangqongqoza kuphi. Ungabeka ngokushesha idatha ye-alphavantage lapho. Awu, asiqhubekele phambili config.py - khipha idatha yohlelo lokusebenza ku-config yethu. Yebo, ngiyavuma, ngasebenzisa i-lib yami - sitri.

Uma uxhuma ku-Mongo, yonke into ilula. kumenyezelwe isigaba seklayenti ukuxhuma futhi ikilasi lesisekelo kuma-cruds, ukwenza kube lula ukwenza imibuzo ngamaqoqo.

Kuzokwenzekani ngokulandelayo?

Lesi sihloko asiside kakhulu, njengoba lapha ngikhuluma kuphela ngokugqugquzela kanye nokulungiselela, ngakho ungangisoli - ngiyathembisa ukuthi ingxenye elandelayo izoba nesenzo kanye nemifanekiso.

Ngakho, kule ngxenye elandelayo si:

Masibhale iklayenti elincane le-alphavantage ku-aiohttp ngezicelo zamaphoyinti esiwadingayo.
Masidale i-ejenti ezoqoqa idatha yezibambiso nezintengo zomlando yazo.

Ikhodi yephrojekthi

Ikhodi yale ngxenye

Source: www.habr.com

Imisebenzi yangemuva ku-Faust, Ingxenye I: Isingeniso