DeepPavlov ืคึฟืึทืจ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก: #1 NLP ืžื›ืฉื™ืจื™ื ืื•ืŸ ืฉืึทืคื•ื ื’ ืคื•ืŸ ื˜ืฉืึทื˜ื‘ืึธื˜

ื ื’ื•ื˜ืŸ ื™ืขื“ืขืจ! ืžื™ืจ ืขืคึฟืขื ืขืŸ ืึท ืกืขืจื™ืข ืคื•ืŸ โ€‹โ€‹โ€‹โ€‹ืึทืจื˜ื™ืงืœืขืŸ ื•ื•ืึธืก ื–ืขื ืขืŸ ื“ืขื“ืึทืงื™ื™ื˜ืึทื“ ืฆื• ืกืึทืœื•ื•ื™ื ื’ ืคึผืจืึทืงื˜ื™ืฉ ืคึผืจืึธื‘ืœืขืžืก ืฉื™ื™ึทื›ื•ืช ืฆื• ื ืึทื˜ื™ืจืœืขืš ืฉืคึผืจืึทืš ืคึผืจืึทืกืขืกื™ื ื’ (ื ืึทื˜ื•ืจืึทืœ ืฉืคึผืจืึทืš ืคึผืจืึทืกืขืกื™ื ื’ ืึธื“ืขืจ ืคืฉื•ื˜ NLP) ืื•ืŸ ืงืจื™ื™ื™ื˜ื™ื ื’ ื“ื™ืึทืœืึธื’ ืื’ืขื ื˜ืŸ (ื˜ืฉืึทื˜ื‘ืึธืฅ) ื ื™ืฆืŸ ืึทืŸ ืึธืคึฟืŸ-ืžืงื•ืจ ื‘ื™ื‘ืœื™ืึธื˜ืขืง. ื“ื™ืคึผืคึผืึทื•ื•ืœืึธื•ื•, ื•ื•ืึธืก ืื™ื– ื“ืขื•ื•ืขืœืึธืคึผืขื“ ื“ื•ืจืš ืื•ื ื“ื–ืขืจ ืžืึทื ืฉืึทืคึฟื˜ ืื™ืŸ ื“ื™ MIPT ืœืึทื‘ืึธืจืึทื˜ืึธืจื™ืข ืคื•ืŸ โ€‹โ€‹ื ืขื•ืจืึทืœ ืกื™ืกื˜ืขืžืขืŸ ืื•ืŸ ื˜ื™ืฃ ืœืขืจื ืขืŸ. ื“ืขืจ ื”ื•ื™ืคึผื˜ ืฆื™ืœ ืคื•ืŸ ื“ื™ ืกืขืจื™ืข ืื™ื– ืฆื• ื‘ืึทืงืขื ืขืŸ DeepPavlov ืฆื• ืึท ื‘ืจื™ื™ื˜ ืงื™ื™ื˜ ืคื•ืŸ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก ืื•ืŸ ื•ื•ื™ื™ึทื–ืŸ ื•ื•ื™ ืื™ืจ ืงืขื ืขืŸ ืกืึธืœื•ื•ืข ื’ืขื•ื•ืขื ื“ื˜ NLP ืคึผืจืึธื‘ืœืขืžืก ืึธืŸ ืึท ื˜ื™ืฃ ื•ื•ื™ืกืŸ ืื™ืŸ ืžืึทืฉื™ืŸ ืœืขืจื ืขืŸ ืื•ืŸ PhD ืื™ืŸ ืžืึทื˜ื”ืขืžืึทื˜ื™ืงืก.

NLP ื˜ืึทืกืงืก ืึทืจื™ื™ึทื ื ืขืžืขืŸ ื“ื™ื˜ืขืจืžืึทื ื™ื ื’ ื“ื™ ืกืขื ื˜ื™ืžืขื ื˜ ืคื•ืŸ ื“ื™ ื˜ืขืงืกื˜, ืคึผืึทืจืกื™ื ื’ ื’ืขื”ื™ื™ืกืŸ ืขื ื˜ื™ื˜ื™ื–, ื“ื™ื˜ืขืจืžืึทื ื™ื ื’ ื•ื•ืึธืก ื“ื™ ื™ื ื˜ืขืจืœืึทืงื™ืึทื˜ืขืจ ื•ื•ื™ืœ ืคื•ืŸ ื“ื™ื™ืŸ ื‘ืึธื˜: ืกื“ืจ ืคึผื™ืฆืข ืึธื“ืขืจ ื‘ืึทืงื•ืžืขืŸ ื”ื™ื ื˜ืขืจื’ืจื•ื ื˜ ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข, ืื•ืŸ ืคื™ืœ ืžืขืจ. ืื™ืจ ืงืขื ื˜ ืœื™ื™ืขื ืขืŸ ืžืขืจ ื•ื•ืขื’ืŸ NLP ื˜ืึทืกืงืก ืื•ืŸ ืžืขื˜ื”ืึธื“ืก ื“ืึธ.

ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ, ืžื™ืจ ื•ื•ืขื˜ ื•ื•ื™ื™ึทื–ืŸ ืื™ืจ ื•ื•ื™ ืฆื• ืœื•ื™ืคืŸ ืึท REST ืกืขืจื•ื•ืขืจ ืžื™ื˜ ืคืึทืจ-ื˜ืจื™ื™ื ื“ NLP ืžืึธื“ืขืœืก, ื’ืจื™ื™ื˜ ืฆื• ื ื•ืฆืŸ ืึธืŸ ื ืึธืš ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืึธื“ืขืจ ื˜ืจื™ื™ื ื™ื ื’.

DeepPavlov ืคึฟืึทืจ ื“ืขื•ื•ืขืœืึธืคึผืขืจืก: #1 NLP ืžื›ืฉื™ืจื™ื ืื•ืŸ ืฉืึทืคื•ื ื’ ืคื•ืŸ ื˜ืฉืึทื˜ื‘ืึธื˜

ื™ื ืกื˜ืึทืœื™ืจื•ื ื’ ืคื•ืŸ DeepPavlov

ื“ืึธ ืื•ืŸ ืื•ื ื˜ืŸ, ื™ื ืกื˜ืจืึทืงืฉืึทื ื– ืคึฟืึทืจ ืœื™ื ื•ืงืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ื’ืขื’ืขื‘ืŸ. ืคึฟืึทืจ Windows, ื–ืขืŸ ืื•ื ื“ื–ืขืจ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ

  • ืฉืึทืคึฟืŸ ืื•ืŸ ืึทืงื˜ืึทื•ื•ื™ื™ื˜ ืึท ื•ื•ื™ืจื˜ื•ืึทืœ ืกื•ื•ื™ื•ื•ืข ืžื™ื˜ ื“ื™ ืงืจืึทื ื˜ ื’ืขืฉื˜ื™ืฆื˜ ื•ื•ืขืจืกื™ืข ืคื•ืŸ โ€‹โ€‹โ€‹โ€‹ืคึผื™ื˜ื”ืึธืŸ:
    virtualelnv env -p python3.7
    source env/bin/activate
  • ื™ื ืกื˜ืึทืœื™ืจืŸ DeepPavlov ืื™ืŸ ืึท ื•ื•ื™ืจื˜ื•ืึทืœ ืกื•ื•ื™ื•ื•ืข:
    pip install deeppavlov
    

ืœืึธื ื˜ืฉื™ื ื’ ืึท REST ืกืขืจื•ื•ืขืจ ืžื™ื˜ ื“ื™ DeepPavlov ืžืึธื“ืขืœ

ืื™ื™ื“ืขืจ ืžื™ืจ ืงืึทื˜ืขืจ ืึท ืกืขืจื•ื•ืขืจ ืžื™ื˜ ื“ื™ DeepPavlov ืžืึธื“ืขืœ ืคึฟืึทืจ ื“ื™ ืขืจืฉื˜ืขืจ ืžืึธืœ, ืขืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ื ื•ืฆื™ืง ืฆื• ืจืขื“ืŸ ื•ื•ืขื’ืŸ ืขื˜ืœืขื›ืข ืคึฟืขื™ึดืงื™ื™ื˜ืŸ ืคื•ืŸ ื“ื™ ื‘ื™ื‘ืœื™ืึธื˜ืขืง ืึทืจืงืึทื˜ืขืงื˜ืฉืขืจ.

ืงื™ื™ืŸ ืžืึธื“ืขืœ ืื™ืŸ DP ื‘ืืฉื˜ื™ื™ื˜ ืคื•ืŸ:

  • ืคึผื™ื˜ื”ืึธืŸ ืงืึธื“;
  • ื“ืึธื•ื•ื ืœืึธืึทื“ืึทื‘ืœืข ืงืึทืžืคึผืึธื•ื ืึทื ืฅ - ืกื™ืจื™ืึทืœื™ื–ืขื“ ื˜ืจื™ื™ื ื™ื ื’ ืจืขื–ื•ืœื˜ืึทื˜ืŸ ืื•ื™ืฃ ืกืคึผืขืฆื™ืคื™ืฉ ื“ืึทื˜ืŸ (ืขืžื‘ืขื“ื™ื ื’ื–, ื•ื•ื™ื™ืฅ ืคื•ืŸ ื ืขื•ืจืึทืœ ื ืขื˜ื•ื•ืึธืจืงืก, ืืื–"ื• ื•);
  • ื ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ื˜ืขืงืข (ื“ืขืจื ืึธืš ืจื™ืคืขืจื“ ืฆื• ื•ื•ื™ ื“ื™ ืงืึธื ืคื™ื’), ื•ื•ืึธืก ื›ึผื•ืœืœ ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข ื•ื•ืขื’ืŸ ื“ื™ ืงืœืืกืŸ ื’ืขื ื™ืฆื˜ ื“ื•ืจืš ื“ื™ ืžืึธื“ืขืœ, URL ืก ืคื•ืŸ ื“ืึทื•ื ืœืึธื•ื“ื™ื“ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ, ืคึผื™ื˜ื”ืึธืŸ ื“ื™ืคึผืขื ื“ืึทื ืกื™ื–, ืืื–"ื• ื•.

ืžื™ืจ ื•ื•ืขืœืŸ ื–ืึธื’ืŸ ืื™ืจ ืžืขืจ ื•ื•ืขื’ืŸ ื•ื•ืึธืก ืื™ื– ืื•ื ื˜ืขืจ ื“ื™ ืงืึทืคึผื˜ืขืจ ืคื•ืŸ DeepPavlov ืื™ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืึทืจื˜ื™ืงืœืขืŸ, ืื™ืฆื˜ ืขืก ืื™ื– ื’ืขื ื•ื’ ืคึฟืึทืจ ืื•ื ื“ื– ืฆื• ื•ื•ื™ืกืŸ ืึทื–:

  • ืงื™ื™ืŸ ืžืึธื“ืขืœ ืื™ืŸ DeepPavlov ืื™ื– ื™ื™ื“ืขื ืึทืคื™ื™ื“ ื“ื•ืจืš ื“ื™ ื ืึธืžืขืŸ ืคื•ืŸ ื–ื™ื™ึทืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ;
  • ืฆื• ืœื•ื™ืคืŸ ื“ืขื ืžืึธื“ืขืœ, ืื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ืึธืคึผืœืึธื“ื™ืจืŸ ื“ื™ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืคื•ืŸ ื“ื™ DeepPavlov ืกืขืจื•ื•ืขืจืก;
  • ืื•ื™ืš, ืฆื• ืœื•ื™ืคืŸ ื“ืขื ืžืึธื“ืขืœ, ืื™ืจ ื“ืึทืจืคึฟืŸ ืฆื• ื™ื ืกื˜ืึทืœื™ืจืŸ ื“ื™ ืคึผื™ื˜ื”ืึธืŸ ืœื™ื™ื‘ืจืขืจื™ื– ืขืก ื ื™ืฆื˜.

ื“ืขืจ ืขืจืฉื˜ืขืจ ืžืึธื“ืขืœ ื•ื•ืึธืก ืžื™ืจ ื•ื•ืขืœืŸ ืงืึทื˜ืขืจ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืžืึทืœื˜ื™ื™ืœื™ื ื’ื•ื•ืึทืœ ื’ืขื”ื™ื™ืกืŸ ืขื ื˜ื™ื˜ื™ ืจืขืงืึธื’ื ื™ื˜ื™ืึธืŸ (NER). ื“ืขืจ ืžืึธื“ืขืœ ืงืœืึทืกืึทืคื™ื™ื– ื˜ืขืงืกื˜ ื•ื•ืขืจื˜ืขืจ ืœื•ื™ื˜ ื“ื™ ื˜ื™ืคึผ ืคื•ืŸ ื’ืขื”ื™ื™ืกืŸ ืขื ื˜ื™ื˜ื™ื– ืฆื• ื•ื•ืึธืก ื–ื™ื™ ื’ืขื”ืขืจืŸ (ืจืขื›ื˜ ื ืขืžืขืŸ, ื“ื–ืฉื™ืึทื’ืจืึทืคื™ืงืึทืœ ื ืขืžืขืŸ, ื ืขืžืขืŸ ืคื•ืŸ ืงืขืจืึทื ืกื™ื–, ืื•ืŸ ืื ื“ืขืจืข). ืงืึธื ืคื™ื’ ื ืึธืžืขืŸ ืคึฟืึทืจ ื“ื™ ืœืขืฆื˜ืข ืœืขืฆื˜ืข ื•ื•ืขืจืกื™ืข ืคื•ืŸ โ€‹โ€‹NER:

ner_ontonotes_bert_mult

ืžื™ืจ ืงืึทื˜ืขืจ ื“ื™ REST ืกืขืจื•ื•ืขืจ ืžื™ื˜ ื“ืขื ืžืึธื“ืขืœ:

  1. ืžื™ืจ ื™ื ืกื˜ืึทืœื™ืจืŸ ื“ื™ ืžืึธื“ืขืœ ื“ื™ืคึผืขื ื“ืึทื ืกื™ื– ืกืคึผืขืกื™ืคื™ืขื“ ืื™ืŸ ื–ื™ื™ึทืŸ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ ืื™ืŸ ื“ื™ ืึทืงื˜ื™ื•ื• ื•ื•ื™ืจื˜ื•ืึทืœ ืกื•ื•ื™ื•ื•ืข:
    python -m deeppavlov install ner_ontonotes_bert_mult
    
  2. ืืจืืคืงืืคื™ืข ืกืขืจื™ืึทืœื™ื–ืขื“ ืžืึธื“ืขืœ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืคื•ืŸ DeepPavlov ืกืขืจื•ื•ืขืจืก:
    python -m deeppavlov download ner_ontonotes_bert_mult
    

    ื“ื™ ืกื™ืจื™ืึทืœื™ื–ืขื“ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ื•ื•ืขื˜ ื–ื™ื™ืŸ ื“ืึทื•ื ืœืึธื•ื“ื™ื“ ืฆื• ื“ื™ DeepPavlov ื”ื™ื™ื ื•ื•ืขื’ื•ื•ื™ื™ึทื–ืขืจ, ื•ื•ืึธืก ืื™ื– ืœื™ื’ืŸ ื“ื•ืจืš ืคืขืœื™ืงื™ื™ึทื˜

    ~/.deeppavlov

    ื•ื•ืขืŸ ื“ืึทื•ื ืœืึธื•ื“ื™ื ื’, ื“ื™ ื”ืึทืฉ ืคื•ืŸ ืฉื•ื™ืŸ ื“ืึทื•ื ืœืึธื•ื“ื™ื“ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืื™ื– ืึธืคึผื’ืขืฉื˜ืขืœื˜ ืงืขื’ืŸ ื“ื™ ื”ืึทืฉืขืก ืคื•ืŸ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืœื™ื’ืŸ ืื•ื™ืฃ ื“ื™ ืกืขืจื•ื•ืขืจ. ืื•ื™ื‘ ืขืก ืื™ื– ืึท ื’ืœื™ื™ึทื›ืŸ, ื“ื™ ืืจืืคืงืืคื™ืข ืื™ื– ืกืงื™ืคึผื˜ ืื•ืŸ ื™ื’ื–ื™ืกื˜ื™ื ื’ ื˜ืขืงืขืก ื–ืขื ืขืŸ ื’ืขื ื™ืฆื˜. ื“ื™ ืกื™ื–ืขืก ืคื•ืŸ ื“ืึทื•ื ืœืึธื•ื“ื™ื“ ืงืึทืžืคึผืึธื•ื ืึทื ืฅ ืงืขื ืขืŸ ื‘ื™ื™ึทื˜ืŸ ืื™ืŸ ื“ื•ืจื›ืฉื ื™ื˜ืœืขืš ืคื•ืŸ 0.5 ืฆื• 8 ื’ื™ื’ืื‘ื™ื™ื˜, ืื™ืŸ ืขื˜ืœืขื›ืข ืงืึทืกืขืก ืจื™ื˜ืฉื™ื ื’ 20 ื’ื™ื’ืื‘ื™ื™ื˜ ื ืึธืš ืึทื ื–ื™ืคึผื™ื ื’.

  3. ืžื™ืจ ืงืึทื˜ืขืจ ื“ื™ REST ืกืขืจื•ื•ืขืจ ืžื™ื˜ ื“ืขื ืžืึธื“ืขืœ:
    python -m deeppavlov riseapi ner_ontonotes_bert_mult -p 5005
    

ื•ื•ื™ ืึท ืจืขื–ื•ืœื˜ืึทื˜ ืคื•ืŸ ืขืงืกืึทืงื™ื•ื˜ื™ื ื’ ื“ืขื ื‘ืึทืคึฟืขืœ, ืึท REST ืกืขืจื•ื•ืขืจ ืžื™ื˜ ืึท ืžืึธื“ืขืœ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืœืึธื ื˜ืฉื˜ ืื•ื™ืฃ ืคึผืึธืจื˜ 5005 ืคื•ืŸ ื“ืขืจ ื‘ืึทืœืขื‘ืึธืก ืžืึทืฉื™ืŸ (ื“ื™ ืคืขืœื™ืงื™ื™ึทื˜ ืคึผืึธืจื˜ ืื™ื– 5000).

ื ืึธืš ื™ื ื™ื˜ื™ืึทืœื™ื–ื™ื ื’ ื“ื™ ืžืึธื“ืขืœ, Swagger ืžื™ื˜ ืึทืคึผื™ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ ืื•ืŸ ื“ื™ ืคื™ื™ื™ืงื™ื™ื˜ ืฆื• ืคึผืจื•ื‘ื™ืจืŸ ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขืคึฟื•ื ืขืŸ ืื™ืŸ ื“ื™ URL http://127.0.0.1:5005. ืœืึธืžื™ืจ ืคึผืจื•ื‘ื™ืจืŸ ื“ื™ ืžืึธื“ืขืœ ื“ื•ืจืš ืฉื™ืงืŸ ืขืก ืฆื• ื“ื™ ืขื ื“ืคึผื•ื™ื ื˜ http://127.0.0.1:5005/model POST ื‘ืขื˜ืŸ ืžื™ื˜ ื“ื™ ืคืืœื’ืขื ื“ืข JSON ืื™ื ื”ืึทืœื˜:

{
  "x": [
    "ะ’ ะœะคะขะ˜ ะผะพะถะฝะพ ะดะพะฑั€ะฐั‚ัŒัั ะฝะฐ ัะปะตะบั‚ั€ะธั‡ะบะต ั ะกะฐะฒั‘ะปะพะฒัะบะพะณะพ ะ’ะพะบะทะฐะปะฐ.",
    "ะ’ ัŽะณะพ-ะทะฐะฟะฐะดะฝะพะน ะ ัƒัะธ ัั‚ะพะณ ะถะธั‚ะฐ ะพั†ะตะฝะตะฝ ะฒ 15 ะณั€ะธะฒะตะฝ"
  ]
}

ืื™ืŸ ืขื ื˜ืคืขืจ ืžื™ืจ ื–ืึธืœ ื‘ืึทืงื•ืžืขืŸ ื“ื™ ืคืืœื’ืขื ื“ืข JSON:

[
  [
    ["ะ’", "ะœะคะขะ˜", "ะผะพะถะฝะพ", "ะดะพะฑั€ะฐั‚ัŒัั", "ะฝะฐ", "ัะปะตะบั‚ั€ะธั‡ะบะต", "ั", "ะกะฐะฒั‘ะปะพะฒัะบะพะณะพ", "ะ’ะพะบะทะฐะปะฐ", "."],
    ["O", "B-FAC", "O", "O", "O", "O", "O", "B-FAC", "I-FAC", "O"]
  ],
  [
    ["ะ’", "ัŽะณะพ", "-", "ะทะฐะฟะฐะดะฝะพะน", "ะ ัƒัะธ", "ัั‚ะพะณ", "ะถะธั‚ะฐ", "ะพั†ะตะฝะตะฝ", "ะฒ", "15", "ะณั€ะธะฒะตะฝ"],
    ["O", "B-LOC", "I-LOC", "I-LOC", "I-LOC", "O", "O", "O", "O", "B-MONEY", "I-MONEY"]
  ]
]

ืžื™ื˜ ื“ื™ ื‘ื™ื™ืฉืคื™ืœืŸ, ืžื™ืจ ื•ื•ืขืœืŸ ืึทื ืึทืœื™ื™ื– ื“ื™ DeepPavlov REST API.

API DeepPavlov

ื™ืขื“ืขืจ DeepPavlov ืžืึธื“ืขืœ ื”ืื˜ ืœืคึผื—ื•ืช ืื™ื™ืŸ ืึทืจื™ื™ึทื ืฉืจื™ื™ึทื‘ ืึทืจื’ื•ืžืขื ื˜. ืื™ืŸ ื“ื™ REST API, ืึทืจื’ื•ืžืขื ื˜ืŸ ื–ืขื ืขืŸ ื’ืขื”ื™ื™ืกืŸ, ื–ื™ื™ืขืจ ื ืขืžืขืŸ ื–ืขื ืขืŸ ื“ื™ ืฉืœื™ืกืœืขืŸ ืคื•ืŸ ื“ื™ ื™ื ืงืึทืžื™ื ื’ ื•ื•ืขืจื˜ืขืจื‘ื•ืš. ืื™ืŸ ืจื•ื‘ึฟ ืงืึทืกืขืก, ื“ื™ ืึทืจื’ื•ืžืขื ื˜ ืื™ื– ื“ืขืจ ื˜ืขืงืกื˜ ื•ื•ืึธืก ื“ืึทืจืฃ ื–ื™ื™ืŸ ืคึผืจืึทืกืขืกื˜. ืžืขืจ ืื™ื ืคึฟืึธืจืžืึทืฆื™ืข ื•ื•ืขื’ืŸ ืึทืจื’ื•ืžืขื ื˜ืŸ ืื•ืŸ ื•ื•ืึทืœื•ืขืก ืื•ืžื’ืขืงืขืจื˜ ื“ื•ืจืš ืžืึธื“ืขืœืก ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขืคึฟื•ื ืขืŸ ืื™ืŸ ื“ื™ ืžืึธื“ืขืœืก ืึธืคึผื˜ื™ื™ืœื•ื ื’ ืคื•ืŸ ื“ื™ ื“ืึทืงื™ื•ืžืขื ื˜ื™ื™ืฉืึทืŸ ื“ื™ืคึผืคึผืึทื•ื•ืœืึธื•ื•

ืื™ืŸ ื“ืขื ื‘ื™ื™ึทืฉืคึผื™ืœ, ืึท ืจืฉื™ืžื” ืคื•ืŸ ืฆื•ื•ื™ื™ ืกื˜ืจื™ื ื’ืก ืื™ื– ื“ื•ืจื›ื’ืขื’ืื ื’ืขืŸ ืฆื• ืึทืจื’ื•ืžืขื ื˜ X, ื™ืขื“ืขืจ ืคื•ืŸ ื•ื•ืึธืก ืื™ื– ื’ืขื’ืขื‘ืŸ ื‘ืึทื–ื•ื ื“ืขืจ ืžืึทืจืงืึทืคึผ. ืื™ืŸ DeepPavlov, ืึทืœืข ืžืึธื“ืขืœืก ื ืขืžืขืŸ ื•ื•ื™ ืึทืจื™ื™ึทื ืฉืจื™ื™ึทื‘ ืึท ืจืฉื™ืžื” (ืคึผืขืงืœ) ืคื•ืŸ ื•ื•ืึทืœื•ืขืก ื•ื•ืึธืก ื–ืขื ืขืŸ ืคึผืจืึทืกืขืกื˜ ื™ื ื“ื™ืคึผืขื ื“ืึทื ื˜ืœื™.

ื“ืขืจ ื˜ืขืจืžื™ืŸ "ืคึผืขืงืœ" ืจืขืคืขืจืก ืฆื• ื“ื™ ืคืขืœื“ ืคื•ืŸ ืžืึทืฉื™ืŸ ืœืขืจื ืขืŸ ืื•ืŸ ืจืขืคืขืจืก ืฆื• ืึท ืคึผืขืงืœ ืคื•ืŸ ืคืจื™ื™ึท ืึทืจื™ื™ึทื ืฉืจื™ื™ึทื‘ ื•ื•ืึทืœื•ืขืก ืคึผืจืึทืกืขืกื˜ ืกื™ื™ืžืึทืœื˜ื™ื™ื ื™ืึทืกืœื™ ื“ื•ืจืš ืึทืŸ ืึทืœื’ืขืจื™ื“ืึทื ืึธื“ืขืจ ื ืขื•ืจืึทืœ ื ืขืฅ. ื“ืขืจ ืฆื•ื’ืึทื ื’ ืึทืœืึทื•ื– ืื™ืจ ืฆื• ืจืขื“ื•ืฆื™ืจืŸ (ืึธืคื˜ ื‘ืื˜ื™ื™ื˜ื™ืง) ื“ื™ ืฆื™ื™ื˜ ื•ื•ืขืŸ ื“ื™ ืžืึธื“ืขืœ ืคึผืจืึทืกืขืกืึทื– ืื™ื™ืŸ ืขืœืขืžืขื ื˜ ืคื•ืŸ ืึท ืคึผืขืงืœ ืงืึทืžืคึผืขืจื“ ืฆื• ื“ื™ ื–ืขืœื‘ืข ื•ื•ืขืจื˜ ื“ื•ืจื›ื’ืขื’ืื ื’ืขืŸ ืฆื• ื“ื™ ืึทืจื™ื™ึทื ืฉืจื™ื™ึทื‘ ืกืขืคึผืขืจืึทื˜ืœื™. ืื‘ืขืจ ื“ื™ ืคึผืจืึทืกืขืกื™ื ื’ ืจืขื–ื•ืœื˜ืึทื˜ ืื™ื– ืืจื•ื™ืก ื‘ืœื•ื™ื– ื ืึธืš ืึทืœืข ืขืœืขืžืขื ื˜ืŸ ื–ืขื ืขืŸ ืคึผืจืึทืกืขืกื˜. ื“ืขืจื™ื‘ืขืจ, ื•ื•ืขืŸ ื“ื–ืฉืขื ืขืจื™ื™ื˜ื™ื ื’ ืึท ื™ื ืงืึทืžื™ื ื’ ืคึผืขืงืœ, ืขืก ื•ื•ืขื˜ ื–ื™ื™ืŸ ื ื™ื™ื˜ื™ืง ืฆื• ื ืขืžืขืŸ ืื™ืŸ ื—ืฉื‘ื•ืŸ ื“ื™ ื’ื™ื›ืงื™ื™ึทื˜ ืคื•ืŸ ื“ื™ ืžืึธื“ืขืœ ืื•ืŸ ื“ื™ ืคืืจืœืื ื’ื˜ ืคึผืจืึทืกืขืกื™ื ื’ ืฆื™ื™ื˜ ืคึฟืึทืจ ื™ืขื“ืขืจ ืคื•ืŸ ื–ื™ื™ึทืŸ ื™ื—ื™ื“ ืขืœืขืžืขื ื˜ืŸ.

ืื•ื™ื‘ ืขืก ื–ืขื ืขืŸ ืขื˜ืœืขื›ืข ืึทืจื’ื•ืžืขื ื˜ืŸ ืฆื• ื“ื™ DeepPavlov ืžืึธื“ืขืœ, ื™ืขื“ืขืจ ืคื•ืŸ ื–ื™ื™ ื ืขืžื˜ ื–ื™ื™ึทืŸ ืื™ื™ื’ืŸ ืคึผืขืงืœ ืคื•ืŸ ื•ื•ืึทืœื•ืขืก, ืื•ืŸ ืื™ืŸ ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ื“ื™ ืžืึธื“ืขืœ ืฉื˜ืขื ื“ื™ืง ื˜ืจืื’ื˜ ืื™ื™ืŸ ืคึผืขืงืœ ืคื•ืŸ ืขื ื˜ืคึฟืขืจืก. ื“ื™ ืขืœืขืžืขื ื˜ืŸ ืคื•ืŸ ื“ื™ ืึทื•ื˜ื’ืึธื•ื™ื ื’ ืคึผืขืงืœ ื–ืขื ืขืŸ ื“ื™ ืจืขื–ื•ืœื˜ืึทื˜ืŸ ืคื•ืŸ ืคึผืจืึทืกืขืกื™ื ื’ ื“ื™ ืขืœืขืžืขื ื˜ืŸ ืคื•ืŸ ื“ื™ ื™ื ืงืึทืžื™ื ื’ ื‘ืึทื˜ืฉืึทื– ืžื™ื˜ ื“ืขืจ ื–ืขืœื‘ื™ืงืขืจ ืื™ื ื“ืขืงืก.

ืื™ืŸ ื“ื™ ืื•ื™ื‘ืŸ ื‘ื™ื™ึทืฉืคึผื™ืœ, ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ืคื•ืŸ ื“ื™ ืžืึธื“ืขืœ ืื™ื– ื’ืขื•ื•ืขืŸ ืฆื• ื‘ืจืขื›ืŸ ืึทืจืึธืคึผ ื™ืขื“ืขืจ ืฉื˜ืจื™ืงืœ ืื™ืŸ ื˜ืึธืงืขื ืก (ื•ื•ืขืจื˜ืขืจ ืื•ืŸ ืคึผื•ื ืงื˜ื•ืึทืฆื™ืข ืžืึทืจืงืก) ืื•ืŸ ืงืœืึทืกื™ืคื™ืฆื™ืจืŸ ื“ื™ ืกื™ืžืขืŸ ืงืึธืจืขื•ื• ืฆื• ื“ื™ ื’ืขื”ื™ื™ืกืŸ ืขื ื˜ื™ื˜ื™ (ืึธืจื’ืึทื ื™ื–ืึทืฆื™ืข ื ืึธืžืขืŸ, ืงืจืึทื ื˜ืงื™ื™ึทื˜) ื•ื•ืึธืก ืขืก ืจืขืคึผืจืึทื–ืขื ืฅ. ื“ืขืจื•ื•ื™ื™ึทืœ ื“ื™ ืžืึธื“ืขืœ ner_ontonotes_bert_mult ื•ื•ืึธืก ืื™ื– ื‘ื™ื›ื•ืœืช ืฆื• ื“ืขืจืงืขื ืขืŸ 18 ื˜ื™ื™ืคึผืก ืคื•ืŸ ื’ืขื”ื™ื™ืกืŸ ืขื ื˜ื™ื˜ื™ื–, ืึท ื“ื™ื˜ื™ื™ืœื“ ื‘ืึทืฉืจื™ื™ึทื‘ื•ื ื’ ืงืขื ืขืŸ ื–ื™ื™ืŸ ื’ืขืคึฟื•ื ืขืŸ ื“ืึธ.

ืื ื“ืขืจืข ืื•ื™ืก-ืคื•ืŸ-ื“ืขื-ืงืขืกื˜ืœ ืžืึธื“ืขืœืก ืคื•ืŸ DeepPavlov

ืื™ืŸ ืึทื“ื™ืฉืึทืŸ ืฆื• NER, ื“ื™ ืคืืœื’ืขื ื“ืข ืื•ื™ืก-ืคื•ืŸ-ื“ืขื-ืงืขืกื˜ืœ ืžืึธื“ืขืœืก ื–ืขื ืขืŸ ื‘ื ื™ืžืฆื ืื™ืŸ DeepPavlov ืื™ืŸ ื“ืขืจ ืฆื™ื™ื˜ ืคื•ืŸ ืฉืจื™ื™ื‘ืŸ:

ื˜ืขืงืกื˜ ืคืจืื’ืข ืึทื ืกื•ื•ืขืจื™ื ื’

ืขื ื˜ืคึฟืขืจืŸ ื“ื™ ืงืฉื™ื ืฆื• ื“ื™ ื˜ืขืงืกื˜ ืžื™ื˜ ืึท ืคืจืึทื’ืžืขื ื˜ ืคื•ืŸ ื“ืขื ื˜ืขืงืกื˜. ืžืึธื“ืขืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ: squad_ru_bert_infer

ื‘ื™ื™ึทืฉืคึผื™ืœ ื‘ืงืฉื”:

{
  "context_raw": [
    "DeepPavlov ั€ะฐะทั€ะฐะฑะฐั‚ั‹ะฒะฐะตั‚ัั ะปะฐะฑะพั€ะฐั‚ะพั€ะธะตะน ะœะคะขะ˜.",
    "ะ’ ัŽะณะพ-ะทะฐะฟะฐะดะฝะพะน ะ ัƒัะธ ัั‚ะพะณ ะถะธั‚ะฐ ะพั†ะตะฝะตะฝ ะฒ 15 ะณั€ะธะฒะตะฝ."
  ],
  "question_raw": [
    "ะšะตะผ ั€ะฐะทั€ะฐะฑะฐั‚ั‹ะฒะฐะตั‚ัั DeepPavlov?",
    "ะกะบะพะปัŒะบะพ ัั‚ะพะธะป ัั‚ะพะณ ะถะธั‚ะฐ ะฝะฐ ะ ัƒัะธ?"
  ]
}

ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜:

[
  ["ะปะฐะฑะพั€ะฐั‚ะพั€ะธะตะน ะœะคะขะ˜", 27, 31042.484375],
  ["15 ะณั€ะธะฒะตะฝ", 39, 1049.598876953125]
]

ื‘ืึทืœื™ื™ื“ื™ืงื•ื ื’ ื“ืขื˜ืขืงืฉืึทืŸ

ื“ื™ื˜ืขืงืฉืึทืŸ ืคื•ืŸ ื“ืขื ื‘ื™ื™ึทื–ื™ื™ึทืŸ ืคื•ืŸ ืึท ื‘ืึทืœื™ื™ื“ื™ืงื•ื ื’ ืฆื• ื“ืขืจ ืžืขื ื˜ืฉ ืฆื• ื•ื•ืขืžืขืŸ ื“ืขืจ ื˜ืขืงืกื˜ ืื™ื– ื’ืขืจืขื“ื˜ (ืื™ืŸ ื“ืขืจ ืฆื™ื™ื˜ ืคื•ืŸ ืฉืจื™ื™ื‘ืŸ - ื‘ืœื•ื™ื– ืคึฟืึทืจ ืขื ื’ืœื™ืฉ). ืžืึธื“ืขืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ: insults_kaggle_conv_bert

ื‘ื™ื™ึทืฉืคึผื™ืœ ื‘ืงืฉื”:


{
  "x": [
    "Money talks, bullshit walks.",
    "You are not the brightest one."
  ]
}

ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜:

[
  ["Not Insult"],
  ["Insult"]
]

ืกืขื ื˜ื™ืžืขื ื˜ ืึทื ืึทืœื™ืกื™ืก

ืงืœืึทืกืึทืคืึทืงื™ื™ืฉืึทืŸ ืคื•ืŸ ื˜ืขืงืกื˜ ืกืขื ื˜ื™ืžืขื ื˜ (positive, ื ื™ื™ื˜ืจืึทืœ, ื ืขื’ืึทื˜ื™ื•ื•). ืžืึธื“ืขืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ: rusentiment_elmo_twitter_cnn

ื‘ื™ื™ึทืฉืคึผื™ืœ ื‘ืงืฉื”:

{
  "x": [
    "ะœะฝะต ะฝั€ะฐะฒะธั‚ัั ะฑะธะฑะปะธะพั‚ะตะบะฐ DeepPavlov.",
    "ะฏ ัะปั‹ัˆะฐะป ะพ ะฑะธะฑะปะธะพั‚ะตะบะต DeepPavlov.",
    "ะœะตะฝั ะฑะตััั‚ ั‚ั€ะพะปะปะธ ะธ ะฐะฝะพะฝะธะผัƒัั‹."
  ]
}

ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜:

[
  ["positive"],
  ["neutral"],
  ["negative"]
]

ืคึผืึทืจืึทืคืจืึทื– ื“ืขื˜ืขืงืฉืึทืŸ

ื‘ืึทืฉื˜ื™ืžืขืŸ ืฆื™ ืฆื•ื•ื™ื™ ืคืึทืจืฉื™ื“ืขื ืข ื˜ืขืงืกื˜ืŸ ื”ืึธื‘ืŸ ื“ื™ ื–ืขืœื‘ืข ื˜ื™ื™ึทื˜ืฉ. ืžืึธื“ืขืœ ืงืึทื ืคื™ื’ื™ืขืจื™ื™ืฉืึทืŸ: stand_paraphraser_en

ื‘ืขื˜ืŸ:

{
  "text_a": [
    "ะ“ะพั€ะพะด ะฟะพะณั€ัƒะถะฐะตั‚ัั ะฒ ัะพะฝ, ะฟั€ะพัั‹ะฟะฐะตั‚ัั ะœะฐั„ะธั.",
    "ะŸั€ะตะทะธะดะตะฝั‚ ะกะจะ ะฟั€ะธะณั€ะพะทะธะป ั€ะฐัั‚ะพั€ะถะตะฝะธะตะผ ะดะพะณะพะฒะพั€ะฐ ั ะ“ะตั€ะผะฐะฝะธะตะน."
  ],
  "text_b": [
    "ะะฐัั‚ัƒะฟะฐะตั‚ ะฝะพั‡ัŒ, ะฒัะต ะถะธั‚ะตะปะธ ะณะพั€ะพะดะฐ ะฟะพัˆะปะธ ัะฟะฐั‚ัŒ, ะฐ ะฟั€ะตัั‚ัƒะฟะฝะธะบะธ ะฟั€ะพัะฝัƒะปะธััŒ.",
    "ะ“ะตั€ะผะฐะฝะธั ะฝะต ัะพะฑะธั€ะฐะตั‚ัั ะฟะพะดะดะฐะฒะฐั‚ัŒัั ัƒะณั€ะพะทะฐะผ ัะพ ัั‚ะพั€ะพะฝั‹ ะกะจะ."
  ]
}

ื“ืขืจ ืจืขื–ื•ืœื˜ืึทื˜:

[
  [1],
  [0]
]

ืื™ืจ ืงืขื ืขืŸ ืฉื˜ืขื ื“ื™ืง ื’ืขืคึฟื™ื ืขืŸ ื“ื™ ืงืจืึทื ื˜ ืจืฉื™ืžื” ืคื•ืŸ ืึทืœืข ื“ืขืขืคึผืคึผืึทื•ื•ืœืึธื•ื• ืžืึธื“ืขืœืก ื•ื•ืึธืก ื–ืขื ืขืŸ ื ื™ืฉื˜-ืคื•ืŸ-ื“ืขื-ืงืขืกื˜ืœ ื“ืึธ.

ืกืึธืฃ

ืื™ืŸ ื“ืขื ืึทืจื˜ื™ืงืœ, ืžื™ืจ ื–ืขื ืขืŸ ื‘ืึทืงืึทื ื˜ ืžื™ื˜ ื“ื™ DeepPavlov API ืื•ืŸ ืขื˜ืœืขื›ืข ืคื•ืŸ โ€‹โ€‹ื“ื™ ื˜ืขืงืกื˜ ืคึผืจืึทืกืขืกื™ื ื’ ืงื™ื™ืคึผืึทื‘ื™ืœืึทื˜ื™ื– ืคื•ืŸ ื“ืขืจ ื‘ื™ื‘ืœื™ืึธื˜ืขืง ืฆื•ื’ืขืฉื˜ืขืœื˜ ืื•ื™ืก ืคื•ืŸ ื“ื™ ืงืขืกื˜ืœ. ืขืก ื–ืึธืœ ื–ื™ื™ืŸ ื’ืขื˜ืจืื’ืŸ ืื™ืŸ ืžื™ื™ื ื•ื ื’ ืึทื– ืคึฟืึทืจ ืงื™ื™ืŸ NLP ืึทืจื‘ืขื˜, ื“ืขืจ ื‘ืขืกื˜ืขืจ ืจืขื–ื•ืœื˜ืึทื˜ ื•ื•ืขื˜ ื–ื™ื™ืŸ ืึทื˜ืฉื™ื•ื•ื“ ื•ื•ืขืŸ ื˜ืจื™ื™ื ื™ื ื’ ื“ื™ ืžืึธื“ืขืœ ืื•ื™ืฃ ืึท ื“ืึทื˜ืŸ ืฉื˜ืขืœืŸ ืงืึธืจืึทืกืคึผืึทื ื“ื™ื ื’ ืฆื• ื“ื™ ื•ื ื˜ืขืจื˜ืขื ื™ืง ื’ืขื’ื ื˜ (ืคืขืœื“) ืคื•ืŸ ื“ื™ ืึทืจื‘ืขื˜. ืื™ืŸ ื“ืขืจืฆื•, ืืคื™ืœื• ืžืขืจ ืžืึธื“ืขืœืก ืงืขื ืขืŸ ื ื™ื˜ ื–ื™ื™ืŸ ื˜ืจื™ื™ื ื“ ืื™ืŸ ืคึผืจื™ื ืฆื™ืคึผ ืคึฟืึทืจ ืึทืœืข ืกื™ื˜ื•ืึทื˜ื™ืึธื ืก.

ืื™ืŸ ื“ื™ ืคืืœื’ืขื ื“ืข ืึทืจื˜ื™ืงืœืขืŸ, ืžื™ืจ ื•ื•ืขืœืŸ ืงื•ืงืŸ ืื™ืŸ ื ืึธืš ืกืขื˜ื˜ื™ื ื’ืก ืคื•ืŸ ื“ืขืจ ื‘ื™ื‘ืœื™ืึธื˜ืขืง, ืงืึทื˜ืขืจ DeepPavlov ืคึฟื•ืŸ Docker, ืื•ืŸ ื“ืขืจื ืึธืš ืžื™ืจ ื•ื•ืขืœืŸ ืคืึธืจื–ืขืฆืŸ ืฆื• ื˜ืจื™ื™ื ื™ื ื’ ืžืึธื“ืขืœืก. ืื•ืŸ ื˜ืึธืŸ ื ื™ื˜ ืคืึทืจื’ืขืกืŸ ืึทื– DeepPavlov ื”ืื˜ ื“ื™ ืคืึธืจื•ื - ืคืจืขื’ืŸ ื“ื™ื™ืŸ ืคึฟืจืื’ืŸ ื•ื•ืขื’ืŸ ื“ื™ ื‘ื™ื‘ืœื™ืึธื˜ืขืง ืื•ืŸ ืžืึธื“ืขืœืก. ื“ืื ืง ืื™ืจ ืคึฟืึทืจ ื“ื™ื™ืŸ ื•ืคืžืขืจืงื–ืึทืžืงื™ื™ึทื˜!

ืžืงื•ืจ: www.habr.com

ืœื™ื™ื’ืŸ ืึท ื‘ืึทืžืขืจืงื•ื ื’