I-OpenVINO hackathon: ukuqonda ilizwi kunye neemvakalelo kwiRaspberry Pi

NgoNovemba 30-Disemba 1 eNizhny Novgorod yabanjwa OpenVINO hackathon. Abathathi-nxaxheba bacelwa ukuba benze iprototype yesisombululo semveliso usebenzisa i-Intel OpenVINO toolkit. Abaququzeleli bacebise uluhlu lwezihloko eziqikelelwayo ezinokukhokelwa xa ukhetha umsebenzi, kodwa isigqibo sokugqibela sahlala kumaqela. Ukongezelela, ukusetyenziswa kweemodeli ezingabandakanywanga kwimveliso kwakhuthazwa.

Kweli nqaku siza kukuxelela malunga nendlela esenza ngayo iprototype yethu yemveliso, apho ekugqibeleni sathatha indawo yokuqala.

Amaqela angaphezu kwe-10 athathe inxaxheba kwi-hackathon. Kumnandi ukuba abanye babo besuka kweminye imimandla. Indawo ye-hackathon yayiyi-complex ye-"Kremlinsky kwi-Pochain", apho iifoto zamandulo zaseNizhny Novgorod zazixhonywe ngaphakathi, kwi-entourage! (Ndiyakukhumbuza ukuba okwangoku iofisi ephakathi ye-Intel itholakala eNizhny Novgorod). Abathathi-nxaxheba banikwe iiyure ze-26 ukuba babhale ikhowudi, kwaye ekugqibeleni kufuneka babonise isisombululo sabo. Inzuzo eyahlukileyo yayibubukho beseshoni yedemo ukuqinisekisa ukuba yonke into ecwangcisiweyo iphunyeziwe kwaye ayizange ihlale ingcamango kwinkcazo. Urhwebo, amashwamshwam, ukutya, yonke into yayilapho!

Ukongeza, i-Intel ibonelela ngeekhamera, iRaspberry PI, iNeural Compute Stick 2.

Ukukhetha umsebenzi

Enye yezona ndawo zinzima kakhulu zokulungiselela i-hackathon ye-free-form ukukhetha umngeni. Ngoko nangoko sagqiba kwelokuba size nento eyayingekabikho kwimveliso, ekubeni isaziso sasisithi oku kwamkelwe kakhulu.

Emva kokuhlalutya imizekelo, ezibandakanyiweyo kwimveliso ekukhutshweni kwangoku, sifikelela kwisigqibo sokuba uninzi lwazo luxazulula iingxaki ezahlukeneyo zombono wekhompyutha. Ngaphezu koko, kunzima kakhulu ukuza nengxaki kwinkalo yombono wekhompyutheni engenakusombululwa ngokusebenzisa i-OpenVINO, kwaye nokuba umntu unokwakheka, kunzima ukufumana imodeli eqeqeshwe kwangaphambili kwindawo yoluntu. Sithatha isigqibo sokugrumba kwelinye icala - malunga nokulungiswa kwentetho kunye nohlalutyo. Makhe siqwalasele umsebenzi onomdla wokwazi iimvakalelo kwintetho. Kufuneka kuthiwe i-OpenVINO sele inemodeli emisela iimvakalelo zomntu ngokusekelwe ebusweni babo, kodwa:

• Kwithiyori, kunokwenzeka ukwenza i-algorithm edibeneyo eya kusebenza kuzo zombini isandi kunye nomfanekiso, okufuneka unike ukwanda kokuchaneka.
• Iikhamera zihlala zineengile yokujonga emxinwa; ngaphezu kwekhamera enye iyafuneka ukugubungela indawo enkulu; isandi asinamda onjalo.

Masiphuhlise umbono: masithathe umbono wecandelo lokuthengisa njengesiseko. Unokulinganisa ukwaneliseka kwabathengi kwiivenkile zokuphuma. Ukuba omnye wabathengi akanelisekanga ngenkonzo kwaye uqala ukuphakamisa ithoni yakhe, unokufowunela umlawuli ngokukhawuleza uncedo.
Kule meko, kufuneka songeze ukuqatshelwa kwelizwi lomntu, oku kuya kusivumela ukuba sihlukanise abasebenzi bevenkile kubathengi kwaye sinikeze uhlalutyo kumntu ngamnye. Ewe, ukongeza, kuya kwenzeka ukuhlalutya ukuziphatha kwabasebenzi bevenkile ngokwabo, ukuvavanya umoya kwiqela, kuvakala kakuhle!

Senza iimfuno zesisombululo sethu:

• Ubungakanani obuncinci besixhobo ekujoliswe kuso
• Ukusebenza ngexesha lokwenyani
• Ixabiso eliphantsi
• Easy scalability

Ngenxa yoko, sikhetha iRaspberry Pi 3 c njengesixhobo esijoliswe kuko I-Intel NCS 2.

Apha kubalulekile ukuqaphela into enye ebalulekileyo ye-NCS - isebenza kakuhle kunye ne-architectures ye-CNN eqhelekileyo, kodwa ukuba ufuna ukuqhuba imodeli enemigangatho yesiko kuyo, ngoko ulindele ukulungelelaniswa kwezinga eliphantsi.

Inye nje into encinci ekufuneka uyenzile: kufuneka ufumane imakrofoni. Umbhobho oqhelekileyo we-USB uya kwenza, kodwa awuyi kukhangeleka kakuhle kunye ne-RPI. Kodwa nalapha isisombululo ngokoqobo "silele kufutshane." Ukurekhoda ilizwi, sithatha isigqibo sokusebenzisa ibhodi yeBhonethi yeLizwi kwikhithi Google AIY Voice Kit, apho kukho i-microphone ye-stereo enentambo.

Khuphela iRaspbian ukusuka Uvimba weeprojekthi ze-AIY kwaye uyilayishe kwi-flash drive, vavanya ukuba imakrofoni iyasebenza usebenzisa lo myalelo ulandelayo (iya kurekhoda umsindo wemizuzwana emi-5 ubude kwaye uyigcine kwifayile):

arecord -d 5 -r 16000 test.wav

Ndimele ndiqaphele ngokukhawuleza ukuba imakrofoni inomdla kakhulu kwaye ithatha ingxolo kakuhle. Ukulungisa oku, masiye kwi-alsamixer, khetha i-Bamba izixhobo kwaye unciphise inqanaba lesignali yegalelo ukuya kwi-50-60%.

Siwuguqula umzimba ngefayile kwaye yonke into iyalingana, unokuwuvala ngesiciko

Ukongeza iqhosha lesalathisi

Ngelixa sithatha i-AIY Voice Kit ngaphandle, siyakhumbula ukuba kukho iqhosha le-RGB, ukukhanya okungasemva kwayo kunokulawulwa yisoftware. Sikhangela "iGoogle AIY Led" kwaye sifumana amaxwebhu: https://aiyprojects.readthedocs.io/en/latest/aiy.leds.html
Kutheni ungasebenzisi eli qhosha ukubonisa imvakalelo eyaziwayo, sineeklasi ezisi-7 kuphela, kwaye iqhosha linemibala esi-8, ngokwaneleyo!

Sidibanisa iqhosha nge-GPIO kwi-Voice Bonnet, layisha iilayibrari eziyimfuneko (sele zifakwe kwikhithi yokusabalalisa kwiiprojekthi ze-AIY)

from aiy.leds import Leds, Color
from aiy.leds import RgbLeds

Masenze i-dict apho imvakalelo nganye iya kuba nombala ohambelanayo ngendlela ye-RGB Tuple kunye nento yeklasi aiy.leds.Leds, apho siya kuhlaziya umbala:

led_dict = {'neutral': (255, 255, 255), 'happy': (0, 255, 0), 'sad': (0, 255, 255), 'angry': (255, 0, 0), 'fearful': (0, 0, 0), 'disgusted':  (255, 0, 255), 'surprised':  (255, 255, 0)} 
leds = Leds()

Kwaye ekugqibeleni, emva kokubikezela okutsha kwemvakalelo, siya kuhlaziya umbala weqhosha ngokuhambelana nayo (ngesitshixo).

leds.update(Leds.rgb_on(led_dict.get(classes[prediction])))

Iqhosha, tshisa!

Ukusebenza ngezwi

Siza kusebenzisa i-pyaudio ukubamba umlambo ukusuka kwi-microphone kunye ne-webrtcvad ukucoca ingxolo kunye nokubona ilizwi. Ukongeza, siya kwenza umgca apho siya kongeza ngokulinganayo kwaye sisuse izicatshulwa zelizwi.

Ekubeni i-webrtcvad inomda kubungakanani beqhekeza elinikeziweyo - kufuneka lilingane ne-10/20/30ms, kwaye uqeqesho lomzekelo wokuqaphela iimvakalelo (njengoko siza kufunda kamva) lwenziwa kwi-dataset ye-48kHz, siya bamba amaqhekeza obungakanani 48000x20ms/1000x1( mono)=960 bytes. I-Webrtcvad izakubuyisela iNyaniso/Ububuxoki kwisiqwenga ngasinye kwezi, ezihambelana nobukho okanye ukungabikho kwevoti kwisiqwenga.

Masisebenzise ingqiqo ilandelayo:

• Siza kongeza kuluhlu loo maqhekeza apho kukho ivoti; ukuba akukho voti, ngoko siya kwandisa ikhawuntala yeenqununu ezingenanto.
• Ukuba ikhawuntala yeziqwenga ezingenanto yi>=30 (600 ms), ngoko sijonga ubungakanani boluhlu lweziqwenga eziqokelelweyo; ukuba yi>250, ngoko songeza kumgca; ukuba akunjalo, sicinga ukuba ubude. yerekhodi akwanelanga ukuyondla kumzekelo ukuchonga isithethi.
• Ukuba i-counter of chunks engenanto isekho <30, kwaye ubungakanani boluhlu lwee-chunks eziqokelelweyo zidlula i-300, ngoko siya kongeza iqhekeza kumgca wokuqikelela okuchanekileyo. (kuba iimvakalelo zidla ngokutshintsha ngokuhamba kwexesha)

 def to_queue(frames):
    d = np.frombuffer(b''.join(frames), dtype=np.int16)
    return d

framesQueue = queue.Queue()
def framesThreadBody():
    CHUNK = 960
    FORMAT = pyaudio.paInt16
    CHANNELS = 1
    RATE = 48000

    p = pyaudio.PyAudio()
    vad = webrtcvad.Vad()
    vad.set_mode(2)
    stream = p.open(format=FORMAT,
                channels=CHANNELS,
                rate=RATE,
                input=True,
                frames_per_buffer=CHUNK)
    false_counter = 0
    audio_frame = []
    while process:
        data = stream.read(CHUNK)
        if not vad.is_speech(data, RATE):
            false_counter += 1
            if false_counter >= 30:
                if len(audio_frame) > 250:              
                    framesQueue.put(to_queue(audio_frame,timestamp_start))
                    audio_frame = []
                    false_counter = 0

        if vad.is_speech(data, RATE):
            false_counter = 0
            audio_frame.append(data)
            if len(audio_frame) > 300:                
                    framesQueue.put(to_queue(audio_frame,timestamp_start))
                    audio_frame = []

Lixesha lokukhangela iimodeli eziqeqeshwe kwangaphambili kwi-domain yoluntu, yiya kwi-github, i-Google, kodwa khumbula ukuba sinomda kwi-architecture esetyenzisiweyo. Le yinxalenye enzima kakhulu, kuba kufuneka uvavanye iimodeli kwidatha yegalelo lakho, kwaye ukongeza, uziguqulele kwifomathi yangaphakathi ye-OpenVINO - IR (Umelelo oluPhakathi). Sazama malunga ne-5-7 izisombululo ezahlukeneyo ezivela kwi-github, kwaye ukuba imodeli yokuqaphela iimvakalelo zasebenza ngokukhawuleza, ngoko ngokuqaphela ilizwi kwafuneka silinde ixesha elide - basebenzisa izakhiwo eziyinkimbinkimbi.

Sigxila koku kulandelayo:

• Iimvakalelo ezivela kwilizwi - https://github.com/alexmuhr/Voice_Emotion
  Isebenza ngokwalo mgaqo ulandelayo: i-audio inqunywe kwiindinyana zobukhulu obuthile, kwisicatshulwa ngasinye esisikhethayo I-MFCC kwaye ke uzingenise njengegalelo kwi-CNN
• Ukuqondwa kwelizwi - https://github.com/linhdvu14/vggvox-speaker-identification
  Apha, endaweni ye-MFCC, sisebenza kunye ne-spectrogram, emva kwe-FFT sondla isignali kwi-CNN, apho kwisiphumo sifumana ukubonakaliswa kwe-vector yezwi.

Okulandelayo siza kuthetha ngokuguqula iimodeli, ngokuqala ngethiyori. I-OpenVINO ibandakanya iimodyuli ezininzi:

• Vula iMveliso yoMzi wezilwanyana, imifuziselo enokusetyenziswa kwaye ibandakanywe kwimveliso yakho
• IModeli ye-Optimzer, enkosi apho unokuguqula khona imodeli ukusuka kwiifomathi ezahlukeneyo zesakhelo (Tensorflow, ONNX njl) kwifomathi yoMmeli oPhakathi, esiya kusebenza ngayo ngakumbi.
• I-Inference Engine ikuvumela ukuba usebenzise iimodeli kwifomathi ye-IR kwiiprosesa ze-Intel, iitshiphusi ezininzi kunye neNeural Compute Stick accelerators.
• Olona guqulelo lusebenzayo lwe-OpenCV (kunye nenkxaso ye-Inference Engine)
  Imodeli nganye kwifomathi ye-IR ichazwa ngeefayile ezimbini: .xml kunye .bin.
  Iimodeli ziguqulelwa kwifomathi ye-IR ngeModel Optimizer ngolu hlobo lulandelayo:

  python /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --input_model speaker.hdf5.pb --data_type=FP16 --input_shape [1,512,1000,1]

  --data_type ikuvumela ukuba ukhethe ifomathi yedatha apho imodeli iya kusebenza ngayo. FP32, FP16, INT8 ziyaxhaswa. Ukukhetha olona didi lwedatha lufanelekileyo kunokunika unyuso olululo lokusebenza.
  --input_shape ibonisa ubungakanani bedatha yegalelo. Ukukwazi ukuyitshintsha ngokuguquguqukayo kubonakala kukho kwi-C ++ API, kodwa asizange siyimbe kude kwaye siyilungisele enye yeemodeli.
  Okulandelayo, makhe sizame ukulayisha imodeli esele iguqulwe kwifomati ye-IR ngemodyuli ye-DNN kwi-OpenCV kwaye uyithumele kuyo.

  import cv2 as cv
  emotionsNet = cv.dnn.readNet('emotions_model.bin',
                            'emotions_model.xml')
  emotionsNet.setPreferableTarget(cv.dnn.DNN_TARGET_MYRIAD)

  Umgca wokugqibela kule meko ikuvumela ukuba uqondise kwakhona izibalo kwi-Neural Compute Stick, izibalo ezisisiseko zenziwa kwiprosesa, kodwa kwimeko ye-Raspberry Pi le nto ayiyi kusebenza, uya kufuna intonga.

  Okulandelayo, ingqiqo ihamba ngolu hlobo lulandelayo: sahlula i-audio yethu kwiifestile zobungakanani obuthile (kuthina yi-0.4 s), siguqulela nganye yezi festile zibe yi-MFCC, esizondla ngayo kwigridi:

  emotionsNet.setInput(MFCC_from_window)
  result = emotionsNet.forward()

  Okulandelayo, masithathe olona didi luqhelekileyo kuzo zonke iifestile. Isisombululo esilula, kodwa kwi-hackathon akudingeki ukuba uze nento engabonakaliyo, kuphela ukuba unayo ixesha. Sisenomsebenzi omninzi ekufuneka siwenze, ngoko ke masiqhubele phambili - siza kujongana nokwamkelwa kwelizwi. Kuyimfuneko ukwenza uhlobo oluthile lwesiseko sedatha apho iispectrograms zamazwi arekhodiweyo kwangaphambili ziya kugcinwa. Ekubeni ixesha lincinane, siza kuyicombulula le ngxaki kangangoko sinako.

  Okukuthi, senza iskripthi sokurekhoda isicatshulwa sezwi (sisebenza ngendlela efanayo njengoko kuchazwe ngasentla, kuphela xa kuphazanyiswa kwikhibhodi kuya kugcina ilizwi kwifayile).

  Masizame:

  python3 voice_db/record_voice.py test.wav

  Sirekhoda amazwi abantu abaliqela (kwimeko yethu, amalungu amathathu eqela)
  Okulandelayo, kwilizwi ngalinye elirekhodiweyo senza uguqulo olukhawulezayo lwesine, sifumane ispectrogram kwaye siyigcine njenge numpy array (.npy):

  for file in glob.glob("voice_db/*.wav"):
          spec = get_fft_spectrum(file)
          np.save(file[:-4] + '.npy', spec)

  Iinkcukacha ezininzi kwifayile create_base.py
  Ngenxa yoko, xa siqhuba iskripthi esiphambili, siya kufumana ukufakwa kwezi spectrograms kwasekuqaleni:

  for file in glob.glob("voice_db/*.npy"):
      spec = np.load(file)
      spec = spec.astype('float32')
      spec_reshaped = spec.reshape(1, 1, spec.shape[0], spec.shape[1])
      srNet.setInput(spec_reshaped)
      pred = srNet.forward()
      emb = np.squeeze(pred)

  Emva kokufumana i-embedding ukusuka kwicandelo elivakalayo, siya kukwazi ukugqiba ukuba ngubani na ngokuthatha umgama we-cosine ukusuka kwipaseji ukuya kuwo onke amazwi kwi-database (encinci, kunokwenzeka) - kwidemo sibeka umda. ku0.3):

          dist_list = cdist(emb, enroll_embs, metric="cosine")
          distances = pd.DataFrame(dist_list, columns = df.speaker)

  Ekugqibeleni, ndingathanda ukuqaphela ukuba isantya se-inference sasikhawuleza kwaye senza ukuba sikwazi ukongeza i-1-2 imodeli engaphezulu (isampula imizuzwana eyi-7 ithatha i-2.5 ukubethelela). Sasingasenalo ixesha lokongeza iimodeli ezintsha kwaye sigxile ekubhaleni iprototype yesicelo sewebhu.

  Usetyenziso lwewebhu

  Inqaku elibalulekileyo: sithatha i-router kunye nathi ukusuka ekhaya kwaye simise inethiwekhi yethu yendawo, inceda ukudibanisa isixhobo kunye neelaptops kwinethiwekhi.

  I-backend yitshaneli yomyalezo wokugqibela phakathi kwe-front kunye ne-Raspberry Pi, esekelwe kwi-teknoloji ye-websocket (http phezu kwe-tcp protocol).

  Inqanaba lokuqala kukufumana ulwazi olucwangcisiweyo oluvela kwi-raspberry, oko kukuthi, i-predictors epakishwe kwi-json, egcinwe kwi-database yesiqingatha sohambo lwabo ukwenzela ukuba izibalo zenziwe malunga nemvelaphi yeemvakalelo zomsebenzisi kwixesha. Le ipakethi ke ithunyelwa kwi-frontend, esebenzisa urhumo kwaye ifumana iipakethi ukusuka kwisiphelo se-websocket. Yonke indlela yokubuyela umva yakhiwe ngolwimi lwesigolang; yakhethwa kuba ifanelekele imisebenzi engahambelaniyo, ethi iigoroutines ziyibambe kakuhle.
  Xa ufikelela ekupheleni, umsebenzisi ubhalisiwe kwaye wangena kwisakhiwo, ngoko umyalezo wakhe ufunyenwe. Bobabini umsebenzisi kunye nomyalezo bangeniswa kwi-hub eqhelekileyo, apho imiyalezo sele ithunyelwe ngokuqhubekayo (kwicala elibhalisiweyo), kwaye ukuba umsebenzisi uvala uxhumano (i-raspberry okanye ngaphambili), ngoko ukubhaliswa kwakhe kucinyiwe kwaye uyasuswa. ihabhu.

  
  Silindele uxhulumaniso oluvela ngasemva

  Isiphelo sangaphambili sisicelo sewebhu esibhalwe kwiJavaScript usebenzisa ilayibrari yeReact ukukhawulezisa kunye nokwenza lula inkqubo yophuhliso. Injongo yesi sicelo kukubona ngeso lengqondo idatha efunyenwe kusetyenziswa i-algorithms esebenza kwicala elingasemva kwaye ngqo kwiRaspberry Pi. Iphepha linomzila wecandelo eliphunyeziweyo usebenzisa i-react-router, kodwa iphepha eliphambili elinomdla kwiphepha eliphambili, apho umlambo oqhubekayo wedatha ufunyanwa ngexesha langempela kwi-server usebenzisa i-WebSocket iteknoloji. I-Raspberry Pi ibona ilizwi, inquma ukuba lelomntu othile ukusuka kwisiseko sedatha esibhalisiweyo, kwaye ithumela uluhlu olunokwenzeka kumxhasi. Umxhasi ubonisa idatha efanelekileyo yamva nje, ubonisa i-avatar yomntu onokuthi uthethe kwi-microphone, kunye nemvakalelo abiza ngayo amagama.

  
  Iphepha lasekhaya elinoqikelelo oluhlaziyiweyo

  isiphelo

  Akunakwenzeka ukugqiba yonke into njengoko bekucwangcisiwe, besingenaxesha, ke ithemba eliphambili lalikwidemo, ukuba yonke into iya kusebenza. Kumboniso bathetha ngendlela esebenza ngayo yonke into, yeyiphi imizekelo abayithathileyo, zeziphi iingxaki abadibana nazo. Okulandelayo yaba yinxalenye yedemo - iingcali zajikeleza igumbi ngokungacwangciswanga kwaye zaya kwiqela ngalinye ukuba lijonge iprototype esebenzayo. Baphinde basibuza imibuzo, wonke umntu waphendula indima yakhe, bayishiya iwebhu kwilaptop, kwaye yonke into yasebenza ngokwenyani njengoko bekulindelekile.

  Makhe ndiqaphele ukuba ixabiso lilonke lesisombululo sethu laliyi-150 yeedola:

  • Raspberry Pi 3 ~ $35
  • Google AIY Voice Bonnet (ungathabatha umrhumo wesithethi) ~ 15$
  • Intel NCS 2 ~ 100$

  Uphucula njani:

  • Sebenzisa ubhaliso olusuka kumxhasi - cela ukufunda okubhaliweyo okwenziwa ngokungenamkhethe
  • Yongeza ezinye iimodeli ezimbalwa: unokumisela isini kunye nobudala ngelizwi
  • Yahlula ngaxeshanye izandi zamazwi (diarization)

  Uvimba: https://github.com/vladimirwest/OpenEMO

  
  Sidiniwe kodwa sonwabile

  Ukuqukumbela, ndingathanda ukuthi enkosi kubaququzeleli nabathathi-nxaxheba. Phakathi kweeprojekthi zamanye amaqela, thina ngokwethu sisithandile isisombululo sokubeka iliso kwiindawo zokupaka simahla. Kuthi, yayingamava apholileyo okuntywiliselwa kwimveliso kunye nophuhliso. Ndiyathemba ukuba iziganeko ezininzi ezinomdla ziya kubanjwa kwimimandla, kubandakanywa nezihloko ze-AI.

umthombo: www.habr.com