ProHoster > Блог > Kev tswj hwm > DeepPavlov rau cov neeg tsim tawm: # 1 NLP cov cuab yeej thiab chatbot tsim

DeepPavlov rau cov neeg tsim tawm: # 1 NLP cov cuab yeej thiab chatbot tsim

Nyob zoo sawv daws! Peb tab tom qhib ntau cov ntawv mob siab rau los daws cov teeb meem uas muaj feem xyuam nrog kev ua cov lus ntuj (Natural Language Processing lossis NLP) thiab tsim cov neeg sawv cev sib tham (chatbots) siv lub tsev qiv ntawv qhib. DeepPavlov, uas yog tsim los ntawm peb pab neeg ntawm MIPT Laboratory of Neural Systems thiab Deep Learning. Lub hom phiaj tseem ceeb ntawm koob yog los qhia DeepPavlov rau ntau tus neeg tsim khoom thiab qhia seb koj tuaj yeem daws cov teeb meem NLP li cas yam tsis muaj kev paub tob hauv Machine Learning thiab PhD hauv Kev Ua lej.

NLP cov dej num suav nrog kev txiav txim siab qhov kev xav ntawm cov ntawv nyeem, txheeb xyuas cov npe npe, txiav txim siab seb tus interlocutor xav tau dab tsi los ntawm koj tus bot: xaj pizza lossis tau txais cov ntaub ntawv keeb kwm yav dhau, thiab ntau ntxiv. Koj tuaj yeem nyeem ntxiv txog NLP cov dej num thiab txoj hauv kev S, SѓS,.

Hauv tsab xov xwm no, peb yuav qhia koj yuav ua li cas khiav REST server nrog cov qauv NLP ua ntej, npaj siv yam tsis muaj kev teeb tsa lossis kev cob qhia ntxiv.

DeepPavlov rau cov neeg tsim tawm: # 1 NLP cov cuab yeej thiab chatbot tsim

Installation ntawm DeepPavlov

Ntawm no thiab hauv qab no, cov lus qhia rau Linux yuav muab. Rau Windows, saib peb cov ntaub ntawv

  • Tsim thiab qhib ib puag ncig virtual nrog cov kev txhawb nqa tam sim no ntawm Python: 
    virtualelnv env -p python3.7
source env/bin/activate
  • Nruab DeepPavlov hauv ib puag ncig virtual: 
    pip install deeppavlov

Tua tawm REST server nrog DeepPavlov qauv

Ua ntej peb tso cov neeg rau zaub mov nrog DeepPavlov qauv thawj zaug, nws yuav muaj txiaj ntsig los tham txog qee qhov nta ntawm lub tsev qiv ntawv architecture.

Txhua tus qauv hauv DP muaj xws li:

  • Python code;
  • Downloadable Cheebtsam - serialized kev kawm tau los ntawm tej ntaub ntawv (embeddings, luj ntawm neural networks, thiab lwm yam);
  • Cov ntaub ntawv teeb tsa (tom qab no hu ua config), uas muaj cov ntaub ntawv hais txog cov chav kawm siv los ntawm cov qauv, URLs ntawm cov khoom rub tawm, Python dependencies, thiab lwm yam.

Peb yuav qhia koj ntxiv txog dab tsi nyob hauv qab hood ntawm DeepPavlov hauv cov ntawv hauv qab no, rau tam sim no nws txaus rau peb paub tias:

  • Txhua tus qauv hauv DeepPavlov raug txheeb xyuas los ntawm lub npe ntawm nws cov teeb tsa;
  • Txhawm rau khiav tus qauv, koj yuav tsum rub tawm nws cov khoom los ntawm DeepPavlov servers;
  • Tsis tas li ntawd, txhawm rau khiav tus qauv, koj yuav tsum nruab Python cov tsev qiv ntawv nws siv.

Thawj tus qauv peb yuav tso tawm yuav yog ntau hom lus Named Entity Recognition (NER). Tus qauv faib cov lus sau raws li hom npe uas lawv koom nrog (cov npe tsim nyog, npe thaj chaw, npe ntawm cov nyiaj, thiab lwm yam). Config lub npe rau tam sim no tam sim no version ntawm NER: 

ner_ontonotes_bert_mult

Peb tso REST server nrog tus qauv:

  1. Peb nruab cov qauv kev vam khom tau teev tseg hauv nws qhov teeb tsa rau hauv qhov chaw ua haujlwm virtual: 
    python -m deeppavlov install ner_ontonotes_bert_mult
  2. Rub tawm serialized qauv Cheebtsam los ntawm DeepPavlov servers: 
    python -m deeppavlov download ner_ontonotes_bert_mult

    Cov serialized Cheebtsam yuav raug rub mus rau hauv DeepPavlov home directory, uas yog nyob ntawm lub neej ntawd 

    ~/.deeppavlov

    Thaum rub tawm, lub hash ntawm cov khoom uas twb tau rub tawm tau raug tshuaj xyuas tawm tsam cov hashs ntawm cov khoom nyob rau ntawm lub server. Yog tias muaj qhov sib tw, qhov rub tawm tau hla thiab cov ntaub ntawv uas twb muaj lawm raug siv. Qhov ntau thiab tsawg ntawm cov khoom rub tawm tuaj yeem sib txawv ntawm qhov nruab nrab ntawm 0.5 txog 8 Gb, qee zaum nce mus txog 20 Gb tom qab unzipping.

  3. Peb tso REST server nrog tus qauv: 
    python -m deeppavlov riseapi ner_ontonotes_bert_mult -p 5005

Raws li qhov tshwm sim ntawm kev ua raws li cov lus txib no, REST server nrog tus qauv yuav raug tso tawm ntawm chaw nres nkoj 5005 ntawm lub tshuab host (qhov chaw nres nkoj yog 5000).

Tom qab pib tus qauv, Swagger nrog API cov ntaub ntawv thiab lub peev xwm los ntsuas tuaj yeem pom ntawm URL http://127.0.0.1:5005. Cia peb sim tus qauv los ntawm kev xa mus rau qhov kawg http://127.0.0.1:5005/model POST thov nrog cov ntsiab lus JSON hauv qab no:

{
  "x": [
    "В МФТИ можно добраться на электричке с Савёловского Вокзала.",
    "В юго-западной Руси стог жита оценен в 15 гривен"
  ]
}

Hauv kev teb peb yuav tsum tau txais JSON hauv qab no:

[
  [
    ["В", "МФТИ", "можно", "добраться", "на", "электричке", "с", "Савёловского", "Вокзала", "."],
    ["O", "B-FAC", "O", "O", "O", "O", "O", "B-FAC", "I-FAC", "O"]
  ],
  [
    ["В", "юго", "-", "западной", "Руси", "стог", "жита", "оценен", "в", "15", "гривен"],
    ["O", "B-LOC", "I-LOC", "I-LOC", "I-LOC", "O", "O", "O", "O", "B-MONEY", "I-MONEY"]
  ]
]

Siv cov piv txwv no, peb yuav txheeb xyuas qhov DeepPavlov REST API.

API DeepPavlov

Txhua tus qauv DeepPavlov muaj tsawg kawg yog ib qho kev sib cav. Hauv REST API, cov lus sib cav muaj npe, lawv cov npe yog cov yuam sij ntawm phau ntawv txhais lus tuaj. Feem ntau, cov lus sib cav yog cov ntawv nyeem uas yuav tsum tau ua. Xav paub ntau ntxiv txog kev sib cav thiab cov txiaj ntsig rov qab los ntawm cov qauv tuaj yeem pom hauv MODELS ntu ntawm cov ntaub ntawv DeepPavlov

Hauv qhov piv txwv, ib daim ntawv teev npe ntawm ob txoj hlua tau dhau mus rau kev sib cav x, txhua tus tau muab cais cais. Nyob rau hauv DeepPavlov, tag nrho cov qauv coj raws li input ib daim ntawv teev (batch) ntawm qhov tseem ceeb uas tau ua tiav ntawm nws tus kheej.

Lub sij hawm "batch" yog hais txog kev kawm tshuab thiab hais txog ib pawg ntawm cov tswv yim ywj pheej uas ua tiav los ntawm algorithm lossis neural network ib txhij. Txoj hauv kev no tso cai rau koj los txo qis (feem ntau tseem ceeb) lub sij hawm tus qauv ua ib feem ntawm ib pawg piv rau tib tus nqi dhau mus rau cov tswv yim sib cais. Tab sis cov txiaj ntsig ua tiav tsuas yog muab tom qab tag nrho cov ntsiab lus tau ua tiav. Yog li ntawd, thaum tsim ib qho batch tuaj, nws yuav tsum tau coj mus rau hauv tus account qhov ceev ntawm tus qauv thiab lub sij hawm ua yuav tsum tau rau txhua yam ntawm nws tus kheej hais.

Yog hais tias muaj ntau qhov kev sib cav rau tus qauv DeepPavlov, txhua tus ntawm lawv tau txais nws tus kheej batch ntawm qhov tseem ceeb, thiab ntawm cov zis tus qauv yeej ib txwm ua ib pawg ntawm cov lus teb. Cov ntsiab lus ntawm cov khoom tawm mus yog cov txiaj ntsig ntawm kev ua cov ntsiab lus ntawm cov khoom tuaj nrog tib qhov ntsuas.

Hauv qhov piv txwv saum toj no, qhov tshwm sim ntawm tus qauv yog txhawm rau txhawm rau txhua txoj hlua rau hauv tokens (cov lus thiab cov cim cim) thiab faib cov token txheeb ze rau lub npe (lub koom haum npe, txiaj) uas nws sawv cev. Tam sim no tus qauv ner_ontonotes_bert_mult muaj peev xwm paub txog 18 hom ntawm cov npe npe, cov lus piav qhia ntxaws tuaj yeem pom S, SѓS,.

Lwm cov qauv tawm ntawm lub thawv los ntawm DeepPavlov

Ntxiv rau NER, cov qauv tawm hauv qab no muaj nyob hauv DeepPavlov thaum lub sijhawm sau ntawv:

Cov lus nug teb

Teb cov lus nug rau cov ntawv nrog ib feem ntawm cov ntawv no. Qauv config: squad_ru_bert_infer

Thov piv txwv:

{
  "context_raw": [
    "DeepPavlov разрабатывается лабораторией МФТИ.",
    "В юго-западной Руси стог жита оценен в 15 гривен."
  ],
  "question_raw": [
    "Кем разрабатывается DeepPavlov?",
    "Сколько стоил стог жита на Руси?"
  ]
}

Tshwm sim:

[
  ["лабораторией МФТИ", 27, 31042.484375],
  ["15 гривен", 39, 1049.598876953125]
]

Insult Detection

Kev tshawb pom ntawm qhov muaj kev thuam rau tus neeg uas cov ntawv nyeem tau hais txog (thaum lub sijhawm sau ntawv - tsuas yog lus Askiv). Qauv config: insults_kaggle_conv_bert

Thov piv txwv:


{
  "x": [
    "Money talks, bullshit walks.",
    "You are not the brightest one."
  ]
}

Tshwm sim:

[
  ["Not Insult"],
  ["Insult"]
]

Kev Ntsuam Xyuas Kev Tshawb Xyuas

Kev faib tawm ntawm cov ntawv xa tawm (zoo, nruab nrab, tsis zoo). Qauv config: rusentiment_elmo_twitter_cnn

Thov piv txwv:

{
  "x": [
    "Мне нравится библиотека DeepPavlov.",
    "Я слышал о библиотеке DeepPavlov.",
    "Меня бесят тролли и анонимусы."
  ]
}

Tshwm sim:

[
  ["positive"],
  ["neutral"],
  ["negative"]
]

Paraphrase Detection

Kev txiav txim siab seb ob phau ntawv sib txawv muaj lub ntsiab lus zoo ib yam. Qauv config: stand_paraphraser_ru

Thov:

{
  "text_a": [
    "Город погружается в сон, просыпается Мафия.",
    "Президент США пригрозил расторжением договора с Германией."
  ],
  "text_b": [
    "Наступает ночь, все жители города пошли спать, а преступники проснулись.",
    "Германия не собирается поддаваться угрозам со стороны США."
  ]
}

Tshwm sim:

[
  [1],
  [0]
]

Daim ntawv teev npe tam sim no ntawm txhua tus qauv tawm ntawm DeepPavlov tuaj yeem nrhiav pom no.

xaus

Hauv tsab xov xwm no, peb tau paub nrog DeepPavlov API thiab qee qhov kev sau ntawv muaj peev xwm ntawm lub tsev qiv ntawv muab tawm ntawm lub thawv. Nws yuav tsum tau nco ntsoov tias rau txhua txoj haujlwm NLP, qhov txiaj ntsig zoo tshaj plaws yuav ua tiav thaum kev cob qhia tus qauv ntawm cov ntaub ntawv teeb tsa sib raug rau thaj chaw kawm (domain) ntawm txoj haujlwm. Tsis tas li ntawd, txawm tias ntau tus qauv tsis tuaj yeem, hauv paus ntsiab lus, raug cob qhia rau txhua qhov xwm txheej.

Hauv cov ntawv hauv qab no peb yuav saib cov chaw ntxiv ntawm lub tsev qiv ntawv, pib DeepPavlov los ntawm Docker, thiab tom qab ntawd peb yuav mus rau cov qauv kev cob qhia. Thiab tsis txhob hnov ​​qab tias DeepPavlov muaj kev sab laj - nug koj cov lus nug txog lub tsev qiv ntawv thiab cov qauv. Ua tsaug rau koj mloog!

Tau qhov twg los: www.hab.com

Ntxiv ib saib