Pulogalamu ya ProHoster > Blog > Ulamuliro > Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

kulowa

Привет!

Munkhaniyi ndigawana zomwe ndakumana nazo pomanga kamangidwe ka microservice pulojekiti yogwiritsa ntchito neural network.

Tiyeni tikambirane zofunikira za zomangamanga, yang'anani pazithunzi zosiyanasiyana zamapangidwe, santhulani chilichonse mwazomangamanga zomalizidwa, ndikuwunikanso ukadaulo waukadaulo.

Kusangalala kuwerenga!

Mawu ochepa onena za vutoli ndi yankho lake

Lingaliro lalikulu ndikuwunika kukopa kwa munthu pamlingo wa mfundo khumi potengera chithunzi.

Munkhaniyi tisiya kufotokoza ma neural network omwe amagwiritsidwa ntchito komanso njira yokonzekera ndi kuphunzitsa. Komabe, m'modzi mwa zofalitsa zotsatirazi, tidzabwereranso pakuwunika njira yowunika mozama.

Tsopano tidutsa paipi yowunikira pamlingo wapamwamba, ndipo tiyang'ana kwambiri kuyanjana kwa ma microservices malinga ndi kapangidwe ka polojekiti yonse. 

Pogwira ntchito yowunikira kukopa, ntchitoyi idagawika m'zigawo zotsatirazi:

  1. Kusankha nkhope muzithunzi
  2. Muyezo wa munthu aliyense
  3. Perekani zotsatira

Yoyamba imathetsedwa ndi mphamvu zophunzitsidwa kale MTCNN. Kwachiwiri, neural neural network idaphunzitsidwa pa PyTorch, pogwiritsa ntchito ResNet34 - kuchokera pamlingo wa "quality / liwiro la kutanthauzira pa CPU"

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Chithunzi chogwira ntchito chapaipi yowunikira

Kusanthula zofunikira za kamangidwe ka polojekiti

M'moyo wozungulira ML magawo a ntchito yomanga ndi makina opangira ma modeli nthawi zambiri amakhala pakati pa nthawi yowononga kwambiri komanso yowononga ndalama.

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Kuzungulira kwa moyo wa polojekiti ya ML

Ntchitoyi ndi chimodzimodzi - chigamulo chinapangidwa kuti titseke payipi yowunika kukhala ntchito yapaintaneti, yomwe idafunikira kuti tidzilowetse muzomangamanga. Zofunikira zotsatirazi zidadziwika:

  1. Kusungidwa kwa chipika chogwirizana - mautumiki onse ayenera kulemba zipika pamalo amodzi, ziyenera kukhala zosavuta kuzisanthula
  2. Kuthekera kwa makulitsidwe opingasa a ntchito yowunika - monga momwe mungapangire Bottleneck
  3. Ndalama zomwezo za purosesa ziyenera kuperekedwa kuti ziwunikire chithunzi chilichonse kuti tipewe kugawika kwa nthawi yofotokozera.
  4. Kutumiza mwachangu (re) kwa mautumiki ena onse ndi stack yonse
  5. Kutha, ngati kuli kofunikira, kugwiritsa ntchito zinthu wamba muzochita zosiyanasiyana

zomangamanga

Pambuyo pofufuza zofunikira, zidawonekeratu kuti zomangamanga za microservice zimagwirizana bwino.

Pofuna kuchotsa mutu wosafunikira, Telegraph API idasankhidwa kukhala kutsogolo.

Choyamba, tiyeni tiwone chithunzi cha mapangidwe omalizidwa, kenaka pita ku kufotokozera kwa zigawo zonse, ndikukhazikitsanso ndondomeko yoyendetsera bwino zithunzi.

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Chithunzi chojambula cha zomangamanga zomalizidwa

Tiyeni tilankhule mwatsatanetsatane za gawo lililonse lachithunzichi, kutanthauza Udindo Umodzi powunika zithunzi.

Microservice "attrai-telegram-bot"

Microservice iyi imaphatikiza zochitika zonse ndi Telegraph API. Pali zochitika zazikulu ziwiri: kugwira ntchito ndi chithunzi chokhazikika ndikugwira ntchito ndi zotsatira za payipi yowunikira. Tiyeni tiwone zochitika zonse ziwirizi.

Mukalandira uthenga womwe uli ndi chithunzi:

  1. Kusefera kumachitika, kokhala ndi macheke awa:
    • Kupezeka kwa kukula koyenera kwa chithunzi
    • Chiwerengero cha zithunzi zomwe zili kale pamzere
  2. Mukadutsa kusefa koyamba, chithunzicho chimasungidwa mu voliyumu ya docker
  3. Ntchito imapangidwa pamzere wa "to_estimate", womwe umaphatikizapo, mwa zina, njira yopita ku chithunzi chomwe chili mu voliyumu yathu.
  4. Ngati masitepe omwe ali pamwambawa akwaniritsidwa bwino, wogwiritsa ntchito adzalandira uthenga wokhala ndi nthawi yokonza zithunzi, yomwe imawerengedwa potengera kuchuluka kwa ntchito zomwe zili pamzere. Ngati cholakwika chichitika, wogwiritsa ntchitoyo adzadziwitsidwa bwino potumiza uthenga wonena zomwe mwina zalakwika.

Komanso, microservice iyi, ngati wogwira ntchito ya udzu winawake, imamvetsera pamzere wa "after_estimate", womwe umapangidwira ntchito zomwe zadutsa paipi yowunika.

Mukalandira ntchito yatsopano kuchokera ku "after_estimate":

  1. Ngati chithunzicho chakonzedwa bwino, timatumiza zotsatira kwa wogwiritsa ntchito; ngati sichoncho, timadziwitsa za cholakwika.
  2. Kuchotsa chithunzi chomwe chiri chotsatira cha njira yowunikira

Evaluation microservice "attrai-estimator"

Microservice iyi ndi ntchito ya udzu winawake ndipo imayika chilichonse chokhudzana ndi payipi yowunikira zithunzi. Pali njira imodzi yokha yogwirira ntchito pano - tiyeni tiyipende.

Mukalandira ntchito yatsopano kuchokera ku "to_estimate":

  1. Tiyeni tiyendetse chithunzichi kudzera munjira yowunika:
    1. Kutsegula chithunzi mu kukumbukira
    2. Timabweretsa chithunzicho kukula kofunikira
    3. Kupeza nkhope zonse (MTCNN)
    4. Timawunika nkhope zonse (timakulunga nkhope zomwe zapezeka pamapeto omaliza kukhala gulu ndikuwonetsa ResNet34)
    5. Perekani chithunzi chomaliza
      1. Tiyeni tijambule mabokosi omangira
      2. Kujambula mavoti
  2. Kuchotsa chithunzi chamakonda (choyambirira).
  3. Kusunga zotuluka kuchokera paipi yowunika
  4. Timayika ntchitoyi pamzere wa "after_estimate", womwe umamvetsedwa ndi "attrai-telegram-bot" microservice yomwe takambirana pamwambapa.

Graylog (+ mongoDB + Elasticsearch)

greylog ndi yankho la kasamalidwe ka chipika chapakati. Mu polojekitiyi, idagwiritsidwa ntchito pazolinga zake.

Chosankha chinagwera pa iye, osati pa nthawi zonse mbawala stack, chifukwa chosavuta kugwira nawo ntchito kuchokera ku Python. Zomwe muyenera kuchita kuti mulowe ku Graylog ndikuwonjezera GELFTCPHndler kuchokera phukusi imvi kwa ena onse odula mizu ya python microservice yathu.

Monga munthu yemwe poyamba ankangogwira ntchito ndi ELK stack, ndinali ndi chidziwitso chabwino ndikugwira ntchito ndi Graylog. Chokhacho chomwe chikukhumudwitsa ndikupambana kwa mawonekedwe a Kibana pa intaneti ya Graylog.

KaluluMQ

KaluluMQ ndi uthenga broker kutengera AMQP protocol.

Mu polojekitiyi idagwiritsidwa ntchito ngati yokhazikika kwambiri komanso yoyesedwa nthawi broker wa Selari ndipo amagwira ntchito mokhazikika.

Redis

Redis ndi NoSQL DBMS yomwe imagwira ntchito ndi ma key-value data structures

Nthawi zina pamakhala kufunika kogwiritsa ntchito zinthu wamba zomwe zimakhazikitsa ma data mumitundu yosiyanasiyana ya Python.

Mwachitsanzo, Redis imasunga hashmap ya mawonekedwe a "telegram_user_id => chiwerengero cha ntchito zomwe zimagwira pamzere," zomwe zimakulolani kuchepetsa chiwerengero cha zopempha kuchokera kwa wogwiritsa ntchito m'modzi kupita ku mtengo wina ndipo, potero, kupewa kuukira kwa DoS.

Tiyeni tikonze ndondomeko yokonza bwino zithunzi

  1. Wogwiritsa amatumiza chithunzi ku Telegraph bot
  2. "attrai-telegram-bot" ilandila uthenga kuchokera ku Telegraph API ndikuyigawa
  3. Ntchito yokhala ndi chithunzicho imawonjezedwa pamzere wosasinthika "to_estimate"
  4. Wogwiritsa amalandira uthenga wokhala ndi nthawi yowunika yomwe idakonzedwa
  5. "attrai-estimator" imatenga ntchito kuchokera pamzere wa "to_estimate", imayendetsa zongoyerekeza ndikuyika ntchitoyo pamzere wa "after_estimate"
  6. "attrai-telegram-bot" kumvetsera pamzere wa "after_estimate", imatumiza zotsatira kwa wogwiritsa ntchito.

DevOps

Pomaliza, mutayang'ananso zomanga, mutha kupita ku gawo losangalatsanso - DevOps

Gulu la Docker

 

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Gulu la Docker  - makina ophatikizira, omwe amagwira ntchito mkati mwa Docker Engine ndipo amapezeka kunja kwa bokosi.

Pogwiritsa ntchito "gulu lankhondo", ma node onse mumagulu athu akhoza kugawidwa mu mitundu iwiri - wogwira ntchito ndi woyang'anira. Pamakina amtundu woyamba, magulu a zotengera (miluko) amayikidwa, makina amtundu wachiwiri ali ndi udindo wokweza, kusanja ndi kuwongolera. zinthu zina zabwino. Oyang'anira nawonso ndi antchito mwachisawawa.

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Gulu lokhala ndi mtsogoleri m'modzi ndi antchito atatu

Kukula kocheperako komwe kungatheke ndi 1 node; makina amodzi nthawi imodzi amakhala ngati woyang'anira mtsogoleri komanso wogwira ntchito. Malingana ndi kukula kwa polojekitiyo komanso zofunikira zochepa za kulekerera zolakwika, adaganiza kuti agwiritse ntchito njirayi.

Ndikuyang'ana m'tsogolo, ndikunena kuti kuyambira pakupanga koyamba, komwe kunali pakati pa mwezi wa June, sipanakhalepo mavuto okhudzana ndi gulu lamagulu (koma izi sizikutanthauza kuti bungwe loterolo ndilovomerezeka mwanjira iliyonse. mapulojekiti, omwe ali ndi zofunikira zololera zolakwika).

Docker Stack

Mumayendedwe agulu, ali ndi udindo wotumiza milu (maseti a ntchito za docker) docker stack

Imathandizira ma docker-compose configs, kukulolani kuti mugwiritsenso ntchito magawo a deploy.  

Mwachitsanzo, pogwiritsa ntchito magawowa, zothandizira pazochitika zilizonse zowunika za microservice zinali zochepa (timagawa ma N cores muzochitika za N, mu microservice yokhayo timachepetsa kuchuluka kwa ma cores omwe PyTorch amagwiritsa ntchito)

attrai_estimator:
  image: 'erqups/attrai_estimator:1.2'
  deploy:
    replicas: 4
    resources:
      limits:
        cpus: '4'
    restart_policy:
      condition: on-failure
      …

Ndikofunika kuzindikira kuti Redis, RabbitMQ ndi Graylog ndi ntchito zabwino kwambiri ndipo sizingayesedwe mosavuta monga "attrai-estimator"

Kuwonetseratu funso - bwanji Kubernetes?

Zikuwoneka kuti kugwiritsa ntchito Kubernetes m'mapulojekiti ang'onoang'ono ndi apakatikati ndizovuta; magwiridwe antchito onse ofunikira atha kupezeka kuchokera ku Docker Swarm, yomwe ndi yosavuta kugwiritsa ntchito kwa oimba nyimbo komanso ili ndi chotchinga chochepa cholowera.

Zachilengedwe

Zonsezi zidayikidwa pa VDS ndi izi:

  • CPU: 4 core Intel® Xeon® Gold 5120 CPU @ 2.20GHz
  • RAM: 8 GB
  • SSD: 160GB

Pambuyo poyesa katundu wamba, zidawoneka kuti ndi kuchuluka kwa ogwiritsa ntchito, makinawa angakhale okwanira.

Koma, atangotumizidwa, ndinayika ulalo ku imodzi mwazithunzi zodziwika kwambiri ku CIS (yup, yemweyo), pambuyo pake anthu adachita chidwi ndi maola angapo ntchitoyo idakonza bwino makumi masauzande a zithunzi. Panthawi imodzimodziyo, panthawi yowonjezereka, CPU ndi RAM zothandizira sizinagwiritsidwe ntchito ngakhale theka.

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network
Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Zithunzi zina

Chiwerengero cha ogwiritsa ntchito mwapadera ndi zopempha zowunikira kuyambira pomwe zidatumizidwa, kutengera tsiku

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

Kugawa kwa nthawi yowunikira mapaipi

Chidule cha kamangidwe ka ntchito pakuwunika mawonekedwe kutengera neural network

anapezazo

Mwachidule, nditha kunena kuti kapangidwe kake ndi njira yoyimba zotengera zidadzilungamitsa - ngakhale panthawi yapamwamba panalibe madontho kapena kugwa mu nthawi yokonza. 

Ndikuganiza kuti mapulojekiti ang'onoang'ono ndi apakatikati omwe amagwiritsa ntchito nthawi yeniyeni ya ma neural network pa CPU munjira yawo akhoza kutengera zomwe zafotokozedwa m'nkhaniyi.

Ndiwonjeza kuti poyamba nkhaniyi inali yayitali, koma kuti ndisatumize kuwerenga kwanthawi yayitali, ndinaganiza zosiya mfundo zina m'nkhaniyi - tidzabwereranso m'mabuku amtsogolo.

Mutha kuyika bot pa Telegraph - @AttraiBot, igwira ntchito mpaka kumapeto kwa autumn 2020. Ndiroleni ndikukumbutseni kuti palibe deta ya ogwiritsa ntchito yomwe imasungidwa - kapena zithunzi zoyambirira, kapena zotsatira za payipi yowunikira - chilichonse chimathetsedwa pambuyo pokonza.

Source: www.habr.com

Kuwonjezera ndemanga