Pehea wau i ola ai e like me kēia?
ʻAʻole i liʻuliʻu, pono wau e hana ma ke kua o kahi papahana i hoʻouka nui ʻia, kahi e pono ai e hoʻonohonoho i ka hoʻokō maʻamau o ka nui o nā hana hope me nā helu paʻakikī a me nā noi no nā lawelawe ʻaoʻao ʻekolu. Asynchronous ka papahana a ma mua o koʻu hiki ʻana mai, he hana maʻalahi ia no nā hana cron-launching: kahi loop e nānā ana i ka manawa o kēia manawa a me ka hoʻomaka ʻana o nā hui o nā coroutines ma o ka hui ʻana - ua ʻae ʻia kēia ala a hiki i ka nui a me nā haneli o ia mau coroutines. , akā naʻe, i ka wā i ʻoi aku ai ko lākou helu ma mua o ʻelua tausani, pono wau e noʻonoʻo e pili ana i ka hoʻonohonoho ʻana i kahi laina hana maʻamau me kahi mea hoʻolimalima, nā limahana, a pēlā aku.
Ua hoʻoholo mua wau e hoʻāʻo i ka Celery, ka mea aʻu i hoʻohana ai ma mua. Ma muli o ke ʻano asynchronous o ka papahana, luʻu wau i ka nīnau a ʻike , a ʻo kekahi , hana ʻia e ka mea kākau o ka ʻatikala.
E ʻōlelo wau i kēia, hoihoi loa ka papahana a hana maikaʻi loa i nā noi ʻē aʻe o kā mākou hui, a ʻo ka mea kākau ponoʻī i ʻōlelo ua hiki iā ia ke ʻōwili iā ia i loko o ka hana me ka hoʻohana ʻana i kahi wai asynchronous. Akā, ʻo ka mea pōʻino, ʻaʻole ia i kūpono iaʻu, e like me ka mea i ʻike ʻia me ka hoʻomaka pū ʻana o nā hana (e nānā. ). I ka manawa kākau ua pani ʻia, akā naʻe, ua holo ka hana no hoʻokahi mahina. ʻO kēlā me kēia hihia, pōmaikaʻi i ka mea kākau a me nā mea maikaʻi a pau, no ka mea, aia nā mea e hana nei ma ka lib ... ma ke ʻano nui, aia ka helu i loko oʻu a ua lilo ka mea hana i mea pulu iaʻu. Eia kekahi, ua loaʻa i kekahi mau hana he 2-3 mau noi http i nā lawelawe like ʻole, no laila, i ka wā e hoʻomaikaʻi ai i nā hana, hana mākou i 4 tausani mau pilina TCP, ma kahi o kēlā me kēia 2 hola - ʻaʻole maikaʻi loa ... makemake wau e hana i kahi kau no hoʻokahi ʻano hana i ka wā e hoʻomaka ai i nā limahana. ʻO kahi mea hou aʻe e pili ana i ka nui o nā noi ma o aiohttp .
Ma kēia mea, hoʻomaka wau e nānā nā koho ʻē aʻe a loaa! ʻO nā mea hana i ka celery, e like me kaʻu i hoʻomaopopo ai , ua hanaia , mua no ka papahana . Hoʻoulu ʻia ʻo Faust e Kafka Streams a hana pū me Kafka ma ke ʻano he mea kūʻai aku, ua hoʻohana pū ʻia ʻo rocksdb e mālama i nā hopena mai ka hana a nā ʻelele, a ʻo ka mea nui loa ʻo ia ka waihona asynchronous.
Eia kekahi, hiki iā ʻoe ke nānā celery a me faust mai nā mea hana o ka hope: ko lākou ʻokoʻa, ʻokoʻa ma waena o nā brokers, hoʻokō i kahi hana haʻahaʻa. He mea maʻalahi nā mea a pau, akā naʻe, kahi hiʻohiʻona maikaʻi i ka faust e huki i ka manaʻo - typed data no ka hoʻouna ʻana i ke kumuhana.
He aha kā mākou e hana ai?
No laila, ma kahi pōkole o nā ʻatikala, e hōʻike wau iā ʻoe pehea e hōʻiliʻili ai i ka ʻikepili mai nā hana hope me ka hoʻohana ʻana iā Faust. ʻO ke kumu no kā mākou papahana laʻana, e like me ka manaʻo o ka inoa, . E hōʻike wau i ke kākau ʻana i nā mea hana (ke kuʻi, nā kumuhana, nā ʻāpana), pehea e hana ai i ka hoʻokō maʻamau (cron), nā kauoha faust cli maʻalahi (kahi ʻōwili ma luna o ke kaomi), ʻohi maʻalahi, a ma ka hopena e hoʻopili mākou i kahi datadog ( hana ma waho o ka pahu) a ho'āʻo i kekahi mea e ʻike. No ka mālama ʻana i ka ʻikepili i hōʻiliʻili ʻia e hoʻohana mākou i ka mongodb a me ka motor no ka pilina.
PS Ma muli o ka hilinaʻi i kākau ʻia ai ka manaʻo e pili ana i ka nānā ʻana, manaʻo wau e ʻike mau ka mea heluhelu ma ka hope o ka ʻatikala hope loa e like me kēia:
Pono papahana
Ma muli o kaʻu i hoʻohiki ai, e hana mākou i kahi papa inoa liʻiliʻi o nā mea e hiki ai i ka lawelawe ke hana:
- Hoʻouka i nā palekana a me kahi ʻike o ia mau mea (me nā loaʻa a me nā poho, ka pale kaulike, ke kahe kālā - no ka makahiki i hala) - maʻamau
- Hoʻouka i ka ʻikepili mōʻaukala (no kēlā me kēia makahiki kālepa, e ʻike i nā waiwai koʻikoʻi o ke kumukūʻai pani o ke kālepa) - mau
- Hoʻouka i ka ʻikepili kālepa hou loa - mau
- Hoʻouka i kahi papa inoa o nā hōʻailona no kēlā me kēia palekana - i kēlā me kēia manawa
E like me ka mea i manaʻo ʻia, koho mākou i kahi inoa no ka papahana mai ka wā kahiko: puolo
Ke hoʻomākaukau nei mākou i ka ʻōnaehana
He ikaika maoli ka inoa, akā naʻe, ʻo nā mea a pau āu e hana ai, ʻo ia ke kākau i kahi config liʻiliʻi no ka docker-compose me kafka (a me ka zookeeper - i loko o kahi pahu), kafdrop (inā makemake mākou e nānā i nā memo ma nā kumuhana), mongodb. Loaʻa iā mākou [docker-compose.yml]() o ke ano penei:
version: '3'
services:
db:
container_name: horton-mongodb-local
image: mongo:4.2-bionic
command: mongod --port 20017
restart: always
ports:
- 20017:20017
environment:
- MONGO_INITDB_DATABASE=horton
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=admin_password
kafka-service:
container_name: horton-kafka-local
image: obsidiandynamics/kafka
restart: always
ports:
- "2181:2181"
- "9092:9092"
environment:
KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:29092,EXTERNAL://localhost:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
KAFKA_RESTART_ATTEMPTS: "10"
KAFKA_RESTART_DELAY: "5"
ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"
kafdrop:
container_name: horton-kafdrop-local
image: 'obsidiandynamics/kafdrop:latest'
restart: always
ports:
- '9000:9000'
environment:
KAFKA_BROKERCONNECT: kafka-service:29092
depends_on:
- kafka-serviceʻAʻohe mea paʻakikī ma ʻaneʻi. ʻElua mau mea hoʻolohe i haʻi ʻia no kafka: hoʻokahi (i loko) no ka hoʻohana ʻana i loko o ka pūnaewele hui, a ʻo ka lua (waho) no nā noi mai waho, no laila ua hoʻouna lākou i waho. 2181 — awa kiai holoholona. ʻO ke koena, manaʻo wau, maopopo.
Hoʻomākaukau i ka iwi o ka papahana
Ma ka mana kumu, e like ke ano o ke ano o ko makou papahana penei:
horton
├── docker-compose.yml
└── horton
├── agents.py *
├── alphavantage.py *
├── app.py *
├── config.py
├── database
│ ├── connect.py
│ ├── cruds
│ │ ├── base.py
│ │ ├── __init__.py
│ │ └── security.py *
│ └── __init__.py
├── __init__.py
├── records.py *
└── tasks.py **ʻO nā mea a pau aʻu i ʻike ai ʻAʻole mākou e hoʻopā iā ia, hana mākou i nā faila hakahaka.**
Ua hana mākou i kahi hale. I kēia manawa e hoʻohui i nā mea hilinaʻi e pono ai, e kākau i ka config a hoʻohui iā mongodb. ʻAʻole wau e hāʻawi i ka kikokikona piha o nā faila i ka ʻatikala, i ʻole e hoʻopaneʻe, akā e hāʻawi wau i nā loulou i nā mana kūpono.
E hoʻomaka kākou me nā hilinaʻi a me nā meta e pili ana i ka papahana -
A laila, hoʻomaka mākou e hoʻokomo i nā hilinaʻi a hana i kahi virtualenv (a i ʻole hiki iā ʻoe ke hana i ka waihona venv iā ʻoe iho a hoʻāla i ke kaiapuni):
pip3 install poetry (если ещё не установлено)
poetry installI kēia manawa e hana kākou - Nā palapala hōʻoia a me kahi e kīkēkē ai. Hiki iā ʻoe ke waiho koke i ka ʻikepili no ka alphavantage ma laila. ʻAe, e neʻe kāua i ka - wehe i ka ʻikepili no ka noi mai kā mākou config. ʻAe, haʻi wau, ua hoʻohana au i kaʻu lib - .
Ke hoʻopili nei iā Mongo, maʻalahi nā mea a pau. hoolahaia e hoʻohui a no nā cruds, i mea e maʻalahi ai ka hana ʻana i nā nīnau ma nā hōʻiliʻili.
He aha ka hope?
ʻAʻole lōʻihi ka ʻatikala, ʻoiai eia wau e kamaʻilio wale nei e pili ana i ka hoʻoikaika a me ka hoʻomākaukau ʻana, no laila, mai hōʻino mai iaʻu - ke hoʻohiki nei au e loaʻa i ka ʻaoʻao aʻe nā hana a me nā kiʻi.
No laila, ma kēia ʻāpana aʻe mākou:
- E kākau i kahi mea kūʻai liʻiliʻi no ka alphavantage ma aiohttp me nā noi no nā helu hope a mākou e pono ai.
- E hana kākou i luna nāna e hōʻiliʻili i ka ʻikepili no nā waiwai palekana a me nā kumukūʻai mōʻaukala no lākou.
Source: www.habr.com