Pulogalamu ya ProHoster > Blog > Ulamuliro > DeepPavlov kwa opanga: # 1 zida za NLP ndi kupanga chatbot

DeepPavlov kwa opanga: # 1 zida za NLP ndi kupanga chatbot

Moni nonse! Tikutsegula mndandanda wa zolemba zomwe zikuthandizira kuthetsa mavuto okhudzana ndi chilankhulo chachilengedwe (Natural Language Processing kapena kungoti NLP) ndikupanga ma dialog agents (chatbots) pogwiritsa ntchito laibulale yotsegula. DeepPavlov, yomwe ikupangidwa ndi gulu lathu ku MIPT Laboratory of Neural Systems and Deep Learning. Cholinga chachikulu cha mndandandawu ndikudziwitsa DeepPavlov kwa opanga osiyanasiyana ndikuwonetsa momwe mungathetsere mavuto a NLP ogwiritsidwa ntchito popanda kudziwa mozama mu Machine Learning ndi PhD mu Masamu.

Ntchito za NLP zikuphatikiza kudziwa momwe mawuwo amamvera, kugawa mabungwe omwe adatchulidwa, kudziwa zomwe wolowa nawo akufuna ku bot yanu: kuyitanitsa pitsa kapena kudziwa zakumbuyo, ndi zina zambiri. Mutha kuwerenga zambiri za ntchito ndi njira za NLP apa.

M'nkhaniyi, tikuwonetsani momwe mungayendetsere seva ya REST yokhala ndi zitsanzo za NLP zophunzitsidwa kale, zokonzeka kugwiritsa ntchito popanda kukonzanso kapena maphunziro ena.

DeepPavlov kwa opanga: # 1 zida za NLP ndi kupanga chatbot

Kuyika kwa DeepPavlov

Apa ndi pansipa, malangizo a Linux adzaperekedwa. Kwa Windows, onani athu zolemba

  • Pangani ndi yambitsani malo omwe ali ndi mtundu wa Python womwe ulipo: 
    virtualelnv env -p python3.7
source env/bin/activate
  • Ikani DeepPavlov pamalo enieni: 
    pip install deeppavlov

Kukhazikitsa seva ya REST ndi mtundu wa DeepPavlov

Tisanayambe seva ndi chitsanzo cha DeepPavlov kwa nthawi yoyamba, zingakhale zothandiza kulankhula za zina za zomangamanga za laibulale.

Mtundu uliwonse mu DP uli ndi:

  • Python kodi;
  • Zida zotsitsidwa - zotsatira zamaphunziro otsatizana pazidziwitso zinazake (zowonjezera, zolemera za neural network, etc.);
  • Fayilo yosinthira (yomwe imadziwika kuti config), yomwe ili ndi zambiri zamakalasi omwe amagwiritsidwa ntchito ndi mtunduwo, ma URL a zida zotsitsidwa, kudalira kwa Python, ndi zina zambiri.

Tikuwuzani zambiri zomwe zili pansi pa DeepPavlov m'nkhani zotsatirazi, pakuti tsopano ndi zokwanira kuti tidziwe kuti:

  • Mtundu uliwonse mu DeepPavlov umadziwika ndi dzina la kasinthidwe kake;
  • Kuti muyendetse chitsanzocho, muyenera kukopera zigawo zake kuchokera ku maseva a DeepPavlov;
  • Komanso, kuti muyendetse chitsanzocho, muyenera kukhazikitsa malaibulale a Python omwe amagwiritsa ntchito.

Mtundu woyamba womwe tikhazikitse ukhala zinenero zambiri Zotchedwa Entity Entity Recognition (NER). Chitsanzocho chimayika mawu olembedwa molingana ndi mtundu wa mabungwe omwe ali nawo (mayina oyenerera, mayina a malo, mayina a ndalama, ndi zina). Konzani dzina la mtundu waposachedwa kwambiri wa NER: 

ner_ontonotes_bert_mult

Timatsegula seva ya REST ndi chitsanzo:

  1. Timayika zodalira zachitsanzo zomwe zafotokozedwa mu kasinthidwe kake m'malo omwe akugwira ntchito: 
    python -m deeppavlov install ner_ontonotes_bert_mult
  2. Tsitsani zigawo zamitundu yotsatiridwa kuchokera ku maseva a DeepPavlov: 
    python -m deeppavlov download ner_ontonotes_bert_mult

    Zida zosakanizidwa zidzatsitsidwa ku bukhu lanyumba la DeepPavlov, lomwe limapezeka mwachisawawa 

    ~/.deeppavlov

    Mukatsitsa, hashi ya zida zomwe zidatsitsidwa kale zimayang'aniridwa motsutsana ndi ma hashes a zigawo zomwe zili pa seva. Ngati pali machesi, kutsitsa kumadumphidwa ndipo mafayilo omwe alipo amagwiritsidwa ntchito. Kukula kwa zida zotsitsidwa kumatha kusiyanasiyana kuchokera pa 0.5 mpaka 8 Gb, nthawi zina kufika 20 Gb mutatsegula.

  3. Timatsegula seva ya REST ndi chitsanzo: 
    python -m deeppavlov riseapi ner_ontonotes_bert_mult -p 5005

Chifukwa chotsatira lamuloli, seva ya REST yokhala ndi chitsanzo idzakhazikitsidwa pa doko 5005 la makina osungira (doko lokhazikika ndi 5000).

Pambuyo poyambitsa chitsanzo, Swagger yokhala ndi zolemba za API ndi kuthekera koyesa kungapezeke pa URL http://127.0.0.1:5005. Tiyeni tiyese chitsanzocho potumiza kumapeto http://127.0.0.1:5005/model POST pempho ndi izi za JSON:

{
  "x": [
    "В МФТИ можно добраться на электричке с Савёловского Вокзала.",
    "В юго-западной Руси стог жита оценен в 15 гривен"
  ]
}

Poyankha tiyenera kulandira JSON zotsatirazi:

[
  [
    ["В", "МФТИ", "можно", "добраться", "на", "электричке", "с", "Савёловского", "Вокзала", "."],
    ["O", "B-FAC", "O", "O", "O", "O", "O", "B-FAC", "I-FAC", "O"]
  ],
  [
    ["В", "юго", "-", "западной", "Руси", "стог", "жита", "оценен", "в", "15", "гривен"],
    ["O", "B-LOC", "I-LOC", "I-LOC", "I-LOC", "O", "O", "O", "O", "B-MONEY", "I-MONEY"]
  ]
]

Pogwiritsa ntchito zitsanzozi, tidzasanthula DeepPavlov REST API.

API DeepPavlov

Mtundu uliwonse wa DeepPavlov uli ndi mfundo imodzi yolowera. Mu REST API, mikangano imatchulidwa, mayina awo ndi makiyi a dikishonale yomwe ikubwera. Nthaŵi zambiri, mkangano ndi malemba omwe amafunika kukonzedwa. Zambiri zokhudzana ndi mikangano ndi mfundo zomwe zabwezedwa ndi zitsanzo zitha kupezeka mu gawo la MODELS la zolembedwa DeepPavlov

Muchitsanzo, mndandanda wa zingwe ziwiri zidaperekedwa ku mkangano x, womwe uliwonse unapatsidwa chizindikiro chosiyana. Mu DeepPavlov, mitundu yonse imatengera mndandanda (mgulu) wazinthu zomwe zimasinthidwa paokha.

Mawu akuti "batch" amatanthauza gawo la kuphunzira pamakina ndipo amatanthauza gulu lazinthu zodziyimira pawokha zomwe zimakonzedwa ndi algorithm kapena neural network nthawi imodzi. Njirayi imakulolani kuti muchepetse (nthawi zambiri kwambiri) nthawi yomwe chitsanzocho chimagwiritsa ntchito chinthu chimodzi cha batch poyerekeza ndi mtengo womwewo womwe umaperekedwa kuzinthuzo mosiyana. Koma zotsatira zake zimaperekedwa pokhapokha zinthu zonse zitakonzedwa. Choncho, popanga batch yomwe ikubwera, zidzakhala zofunikira kuganizira liwiro lachitsanzo ndi nthawi yofunikira yopangira chinthu chilichonse.

Ngati pali mikangano ingapo ku chitsanzo cha DeepPavlov, aliyense wa iwo amalandira mtengo wake wamtengo wapatali, ndipo pazotsatira chitsanzocho nthawi zonse chimapanga gulu limodzi la mayankho. Zinthu za batch yotuluka ndi zotsatira za kukonza zinthu zamagulu omwe akubwera ndi index yomweyi.

Muchitsanzo chapamwambachi, chotsatira cha chitsanzo chinali kuthyola chingwe chilichonse kukhala zizindikiro (mawu ndi zizindikiro zopumira) ndikuyika chizindikiro chokhudzana ndi dzina la bungwe (dzina la bungwe, ndalama) lomwe likuyimira. Panopa chitsanzo ner_ontonotes_bert_mult wokhoza kuzindikira mitundu 18 ya mabungwe otchulidwa, kufotokozera mwatsatanetsatane kungapezeke apa.

Zitsanzo zina zakunja za DeepPavlov

Kuphatikiza pa NER, mitundu yotsatirayi yakunja ikupezeka ku DeepPavlov panthawi yolemba:

Kuyankha Mafunso a Malemba

Yankhani funso ku lembalo ndi chidutswa cha lemba ili. Model config: squad_ru_bert_infer

Funsani chitsanzo:

{
  "context_raw": [
    "DeepPavlov разрабатывается лабораторией МФТИ.",
    "В юго-западной Руси стог жита оценен в 15 гривен."
  ],
  "question_raw": [
    "Кем разрабатывается DeepPavlov?",
    "Сколько стоил стог жита на Руси?"
  ]
}

Zotsatira:

[
  ["лабораторией МФТИ", 27, 31042.484375],
  ["15 гривен", 39, 1049.598876953125]
]

Kuzindikira mwachipongwe

Kuzindikira kukhalapo kwa chipongwe kwa munthu amene lembalo lidalembedwera (panthawi yolemba - cha Chingerezi chokha). Kukonzekera kwachitsanzo: kutukwana_kaggle_conv_bert

Funsani chitsanzo:


{
  "x": [
    "Money talks, bullshit walks.",
    "You are not the brightest one."
  ]
}

Zotsatira:

[
  ["Not Insult"],
  ["Insult"]
]

Kusanthula Maganizo

Kugawika kwa kumverera kwa mawu (zabwino, zandale, zosokoneza). Model config: rusentiment_elmo_twitter_cnn

Funsani chitsanzo:

{
  "x": [
    "Мне нравится библиотека DeepPavlov.",
    "Я слышал о библиотеке DeepPavlov.",
    "Меня бесят тролли и анонимусы."
  ]
}

Zotsatira:

[
  ["positive"],
  ["neutral"],
  ["negative"]
]

Kuzindikira kwa Mawu

Kuwona ngati malemba awiri osiyana ali ndi tanthauzo lofanana. Model config: stand_paraphraser_en

Pempho:

{
  "text_a": [
    "Город погружается в сон, просыпается Мафия.",
    "Президент США пригрозил расторжением договора с Германией."
  ],
  "text_b": [
    "Наступает ночь, все жители города пошли спать, а преступники проснулись.",
    "Германия не собирается поддаваться угрозам со стороны США."
  ]
}

Zotsatira:

[
  [1],
  [0]
]

Mndandanda wamakono wamitundu yonse ya kunja kwa bokosi ya DeepPavlov ukhoza kupezeka nthawi zonse apa.

Pomaliza

M'nkhaniyi, tidadziwa DeepPavlov API ndi zina mwazolemba zalaibulale zomwe zaperekedwa m'bokosilo. Tiyenera kukumbukira kuti pa ntchito iliyonse ya NLP, zotsatira zabwino zidzakwaniritsidwa pophunzitsa chitsanzo pa seti ya deta yogwirizana ndi gawo la phunziro (malo) a ntchitoyo. Kuphatikiza apo, ngakhale zitsanzo zambiri sizingaphunzitsidwe pazochitika zonse.

M'nkhani zotsatirazi tiwona zina zowonjezera laibulale, ndikuyambitsa DeepPavlov kuchokera ku Docker, ndiyeno tidzapita ku zitsanzo zophunzitsira. Ndipo musaiwale kuti DeepPavlov ali msonkhano - funsani mafunso okhudza laibulale ndi zitsanzo. Zikomo chifukwa chakumvetsera!

Source: www.habr.com

Kuwonjezera ndemanga