ProHoster > Blog > Utongi > DeepPavlov yevagadziri: #1 NLP maturusi uye chatbot kusikwa

DeepPavlov yevagadziri: #1 NLP maturusi uye chatbot kusikwa

Mhoroi mose! Tiri kuvhura nhevedzano yezvinyorwa zvakatsaurirwa kugadzirisa matambudziko anoshanda ane chekuita nemutauro wechisikigo kugadzirisa (Natural Mutauro Processing kana kungoti NLP) uye kugadzira dialog agents (chatbots) vachishandisa raibhurari yakavhurika-sosi. DeepPavlov, iyo iri kuvandudzwa nechikwata chedu kuMIPT Laboratory yeNeural Systems uye Kudzidza Kwakadzika. Chinangwa chikuru cheiyi nhevedzano kusuma DeepPavlov kune akasiyana siyana evagadziri uye kuratidza magadzirisiro aungaita akaiswa matambudziko eNLP pasina kuve neruzivo rwakadzama muMuchina Kudzidza uye PhD muMasvomhu.

Mabasa eNLP anosanganisira kuona manzwiro echinyorwa, kupatsanura mazita masangano, kuona izvo anopindira anoda kubva kubhoti yako: odha pizza kana kuwana ruzivo rwekumashure, nezvimwe zvakawanda. Unogona kuverenga zvakawanda nezveNLP mabasa uye nzira pano.

Muchikamu chino, tinokuratidza maitiro ekumhanyisa sevha yeREST ine pre-yakadzidziswa NLP modhi, yakagadzirira kushandisa pasina kumwe kugadziridzwa kana kudzidziswa.

DeepPavlov yevagadziri: #1 NLP maturusi uye chatbot kusikwa

Kuiswa kweDeepPavlov

Pano uye pazasi, mirairo yeLinux ichapihwa. ZveWindows, ona yedu zvinyorwa

  • Gadzira uye shandisa nharaunda chaiyo ine yazvino inotsigirwa vhezheni yePython: 
    virtualelnv env -p python3.7
source env/bin/activate
  • Isa DeepPavlov munzvimbo chaiyo: 
    pip install deeppavlov

Kutangisa sevha yeREST neiyo DeepPavlov modhi

Tisati taparura sevha neiyo DeepPavlov modhi kekutanga, zvichave zvinobatsira kutaura nezve mamwe maficha ekuvakwa kweraibhurari.

Chero modhi muDP ine:

  • Python code;
  • Zvinodhawunirodha zvikamu - serialized kudzidziswa mhedzisiro pane chaiyo data (kumisikidza, huremu hweneural network, nezvimwewo);
  • Iyo faira yekumisikidza (inozonzi iyo config), iyo ine ruzivo nezve makirasi anoshandiswa nemuenzaniso, maURL ezvikamu zvakatorwa, Python kutsamira, nezvimwe.

Isu tichakuudza zvimwe nezve izvo zviri pasi pehodhi yeDeepPavlov mune zvinotevera zvinyorwa, nekuti ikozvino zvakwana kuti isu tizive izvozvo:

  • Chero muenzaniso muDeepPavlov unozivikanwa nezita rekugadzirisa kwayo;
  • Kuti umhanye iyo modhi, unofanirwa kurodha zvikamu zvaro kubva kune DeepPavlov maseva;
  • Zvakare, kuti umhanye modhi, unofanirwa kuisa iyo Python raibhurari yaanoshandisa.

Yekutanga modhi yatichaparura ichave yemitauro yakawanda Yakatumidzwa Entity Recognition (NER). Iyo modhi inorongedza mazwi echinyorwa zvichienderana nerudzi rwemazita emasangano kwavari (mazita akakodzera, mazita enzvimbo, mazita emari, nezvimwe). Gadzirisa zita reshanduro yazvino yeNER: 

ner_ontonotes_bert_mult

Isu tinotangisa iyo REST server nemuenzaniso:

  1. Isu tinoisa iyo yemhando yekutsamira yakatsanangurwa mune yayo config mune inoshanda virtual nharaunda: 
    python -m deeppavlov install ner_ontonotes_bert_mult
  2. Dhawunirodha serialized modhi zvikamu kubva kuDeepPavlov maseva: 
    python -m deeppavlov download ner_ontonotes_bert_mult

    Izvo zvikamu zve serialized zvichatorwa kuDeepPavlov imba dhairekitori, iyo inowanikwa nekusarudzika 

    ~/.deeppavlov

    Paunenge uchirodha, iyo hashi yezvinhu zvakatodhawunirwa zvinotariswa zvichipesana nemaheshi ezvikamu zviri paseva. Kana paine mutambo, kurodha kunodarikwa uye mafaera aripo anoshandiswa. Saizi yezvikamu zvakadhawunirwa zvinogona kusiyana paavhareji kubva pa0.5 kusvika 8 Gb, mune dzimwe nguva inosvika 20 Gb mushure mekusunungura.

  3. Isu tinotangisa iyo REST server nemuenzaniso: 
    python -m deeppavlov riseapi ner_ontonotes_bert_mult -p 5005

Nekuda kwekuita uyu murairo, sevha yeREST ine modhi ichavhurwa pachiteshi 5005 yemuchina wemuchina (iyo default port ndeye 5000).

Mushure mekutanga modhi, Swagger ine API zvinyorwa uye kugona kuyedza kunogona kuwanikwa paURL http://127.0.0.1:5005. Ngatiedzei modhi nekuitumira kune yekupedzisira http://127.0.0.1:5005/model POST chikumbiro neinotevera JSON zvemukati:

{
  "x": [
    "В МФТИ можно добраться на электричке с Савёловского Вокзала.",
    "В юго-западной Руси стог жита оценен в 15 гривен"
  ]
}

Mukupindura isu tinofanirwa kugamuchira inotevera JSON:

[
  [
    ["В", "МФТИ", "можно", "добраться", "на", "электричке", "с", "Савёловского", "Вокзала", "."],
    ["O", "B-FAC", "O", "O", "O", "O", "O", "B-FAC", "I-FAC", "O"]
  ],
  [
    ["В", "юго", "-", "западной", "Руси", "стог", "жита", "оценен", "в", "15", "гривен"],
    ["O", "B-LOC", "I-LOC", "I-LOC", "I-LOC", "O", "O", "O", "O", "B-MONEY", "I-MONEY"]
  ]
]

Tichishandisa iyi mienzaniso, isu tichaongorora iyo DeepPavlov REST API.

API DeepPavlov

Imwe neimwe yeDeepPavlov modhi ine kanenge imwe yekuisa nharo. MuREST API, nharo dzinopihwa mazita, mazita avo ndiwo makiyi eduramazwi rinouya. Muzviitiko zvakawanda, nharo ndiyo chinyorwa chinoda kugadziriswa. Ruzivo rwakawanda nezve nharo uye kukosha kwakadzoserwa nemodheru inogona kuwanikwa muchikamu cheMODELS chezvinyorwa DeepPavlov

Mumuenzaniso, runyoro rwetambo mbiri dzakapfuudzwa kunharo x, imwe neimwe yakapihwa markup yakaparadzana. MuDeepPavlov, ese mamodheru anotora seyekupinza rondedzero (batch) yehunhu hunogadziriswa zvakazvimirira.

Izwi rekuti "batch" rinoreva ndima yekudzidza muchina uye rinoreva batch yezvakazvimirira yekuisa kukosha inogadziriswa nealgorithm kana neural network panguva imwe chete. Iyi nzira inobvumira iwe kuderedza (kazhinji zvakanyanya) nguva iyo modhi inobata chimwe chinhu chebatch zvichienzaniswa neiyo kukosha kwakapfuudzwa kune yekuisa zvakasiyana. Asi mhedzisiro yekugadziriswa inopihwa chete mushure mekunge zvinhu zvese zvagadziriswa. Naizvozvo, kana uchigadzira batch inouya, zvichave zvakafanira kufunga nezvekumhanya kweiyo modhi uye inodiwa yekugadzirisa nguva kune yega yega zvinhu zvayo.

Kana paine kupokana kwakati wandei kune iyo DeepPavlov modhi, imwe neimwe yadzo inogamuchira batch yayo yehukoshero, uye pakubuda kwayo modhi inogara ichiburitsa batch rimwe remhinduro. Zvinhu zvebatch inobuda ndiwo mhedzisiro yekugadzirisa zvinhu zvemabhechi anouya ane index yakafanana.

Mumuenzaniso uri pamusoro apa, mhedzisiro yemuenzaniso yaive yekudimbura tambo imwe neimwe kuita tokens (mazwi nezviratidzo zvepumisikidzo) uye kurongedza chiratidzo chehukama kune yakapihwa zita (zita resangano, mari) yainomiririra. Iye zvino muenzaniso ner_ontonotes_bert_mult inokwanisa kuziva 18 mhando dzemasangano ane mazita, tsananguro yakadzama inogona kuwanikwa pano.

Mamwe emhando dzekunze-kwe-bhokisi naDeepPavlov

Pamusoro peNER, iwo anotevera kunze-kwe-the-bhokisi modhi anowanikwa muDeepPavlov panguva yekunyora:

Rugwaro Mubvunzo Mhinduro

Pindura mubvunzo kune chinyorwa nechidimbu chechinyorwa ichi. Muenzaniso config: squad_ru_bert_infer

Muenzaniso chikumbiro:

{
  "context_raw": [
    "DeepPavlov разрабатывается лабораторией МФТИ.",
    "В юго-западной Руси стог жита оценен в 15 гривен."
  ],
  "question_raw": [
    "Кем разрабатывается DeepPavlov?",
    "Сколько стоил стог жита на Руси?"
  ]
}

Mhinduro:

[
  ["лабораторией МФТИ", 27, 31042.484375],
  ["15 гривен", 39, 1049.598876953125]
]

Insult Detection

Kuonekwa kwekuvapo kwekutuka kune munhu anonyorerwa chinyorwa (panguva yekunyora - chete Chirungu). Model config: anotuka_kaggle_conv_bert

Muenzaniso chikumbiro:


{
  "x": [
    "Money talks, bullshit walks.",
    "You are not the brightest one."
  ]
}

Mhinduro:

[
  ["Not Insult"],
  ["Insult"]
]

Sentiment Kuongorora

Kurongeka kwemanzwiro echinyorwa (chakanaka, chisina kwaakarerekera, chisina kunaka). Model config: rusentiment_elmo_twitter_cnn

Muenzaniso chikumbiro:

{
  "x": [
    "Мне нравится библиотека DeepPavlov.",
    "Я слышал о библиотеке DeepPavlov.",
    "Меня бесят тролли и анонимусы."
  ]
}

Mhinduro:

[
  ["positive"],
  ["neutral"],
  ["negative"]
]

Paraphrase Detection

Kuona kana zvinyorwa zviviri zvakasiyana zvine zvazvinoreva zvakafanana. Model config: stand_paraphraser_ru

Chikumbiro:

{
  "text_a": [
    "Город погружается в сон, просыпается Мафия.",
    "Президент США пригрозил расторжением договора с Германией."
  ],
  "text_b": [
    "Наступает ночь, все жители города пошли спать, а преступники проснулись.",
    "Германия не собирается поддаваться угрозам со стороны США."
  ]
}

Mhinduro:

[
  [1],
  [0]
]

Rondedzero yazvino yeese kunze-kwe-iyo-bhokisi DeepPavlov modhi inogona kuwanikwa nguva dzose pano.

mhedziso

Muchikamu chino, takazivana neDeepPavlov API uye mamwe ezvinyorwa zvekugadzirisa zvinyorwa zveraibhurari zvakapiwa kunze kwebhokisi. Izvo zvinofanirwa kutakurwa mupfungwa kuti kune chero basa reNLP, mhedzisiro yakanakisa ichawanikwa paunenge uchidzidzisa modhi pane data seti inoenderana nenzvimbo yezvidzidzo (domain) yebasa racho. Mukuwedzera, kunyange mamwe mamodheru haagoni, mumusimboti, kudzidziswa kune ese mamiriro.

Muzvinyorwa zvinotevera tichatarisa mamwe marongero eraibhurari, kutanga DeepPavlov kubva kuDocker, uye tobva taenderera mberi kumamodeli ekudzidzisa. Uye usakanganwa kuti DeepPavlov ane forum - bvunza mibvunzo yako maererano neraibhurari nemhando. Ndinokutendai nekuteerera kwenyu!

Source: www.habr.com

Voeg