Pa Habr!
Sitimakonda kuyika pano zomasulira zazaka ziwiri, zopanda ma code komanso zamaphunziro - koma lero tisiya. Tikukhulupirira kuti vuto lomwe lili pamutu wa nkhaniyi likudetsa nkhawa owerenga athu ambiri, ndipo mwawerenga kale ntchito yofunikira pazachisinthiko zomwe positiyi ikutsutsana nayo poyambirira kapena muwerenga tsopano. Takulandirani kumphaka!
Mu Marichi 2017, OpenAI idapanga mafunde pagulu lophunzirira mwakuya ndi pepala "" Ntchitoyi idafotokoza zotsatira zochititsa chidwi zokomera mfundo yakuti reinforcement learning (RL) siinasinthe, ndipo pophunzitsa ma neural network ovuta, ndikofunikira kuyesa njira zina. Kenako mkangano unayambika ponena za kufunikira kwa maphunziro olimbikitsira komanso momwe akuyenera kukhala ngati ukadaulo "woyenera kukhala nawo" pakuphunzitsa kuthetsa mavuto. Apa ndikufuna kunena kuti matekinoloje awiriwa sayenera kuonedwa ngati mpikisano, imodzi yomwe ili bwino kuposa ina; m'malo mwake, pamapeto pake amakwaniritsana. Zowonadi, ngati mukuganiza pang'ono pazomwe zimafunika kuti mupange ndi machitidwe oterowo, omwe pakukhalapo kwawo angakhale okhoza kuphunzira, kuweruza ndi kukonzekera, ndiye kuti pafupifupi tidzafika pamapeto kuti izi kapena zophatikizana zidzafunika. Mwa njira, inali njira yophatikizira imeneyi yomwe chilengedwe chinabwera, chomwe chinapatsa zinyama ndi zinyama zina zapamwamba zanzeru zovuta panthawi ya chisinthiko.
Njira Zachisinthiko
Lingaliro lalikulu la pepala la OpenAI linali lakuti, m'malo mogwiritsa ntchito maphunziro olimbikitsana pamodzi ndi kubwereza kwachikhalidwe, adaphunzitsidwa bwino ndi neural network kuti athetse mavuto ovuta pogwiritsa ntchito zomwe adazitcha "njira yosinthika" (ES). Njira ya ES iyi imakhala ndi kusunga zolemera zolemera pa intaneti, kuphatikizapo othandizira angapo omwe amagwira ntchito mofanana ndi kugwiritsa ntchito magawo osankhidwa kuchokera kugawa uku. Wothandizira aliyense amagwira ntchito m'malo ake, ndipo akamaliza kuchuluka kwa magawo kapena magawo a gawo, algorithm imabweretsa mphotho yowonjezereka, yowonetsedwa ngati kulimba mtima. Poganizira za mtengo uwu, kugawidwa kwa magawo kungasinthidwe kwa othandizira opambana, kulepheretsa osapambana. Pobwereza ntchito yotereyi nthawi zambiri ndi kutenga nawo mbali kwa mazana a othandizira, ndizotheka kusuntha kugawa zolemera kumalo omwe angalole kuti ogwira ntchito apange ndondomeko yapamwamba yothetsera ntchito yomwe apatsidwa. Zowonadi, zotsatira zomwe zafotokozedwa m'nkhaniyi ndi zochititsa chidwi: zikuwonetsedwa kuti ngati muthamangitsa othandizira chikwi limodzi, ndiye kuti kuyenda kwa anthropomorphic pamiyendo iwiri kumatha kuphunziridwa pasanathe theka la ola (pamene ngakhale njira zapamwamba kwambiri za RL zimafuna kugwiritsa ntchito ndalama zambiri. kuposa ola limodzi pa izi). Kuti mumve zambiri, ndikupangira kuwerenga zabwino kwambiri kuchokera kwa olemba kuyesera, komanso .
Njira zosiyanasiyana zophunzitsira kuyenda mowongoka kwa anthropomorphic, kuphunzira pogwiritsa ntchito njira ya ES kuchokera ku OpenAI.
Bokosi lakuda
Phindu lalikulu la njirayi ndikuti likhoza kufanana mosavuta. Ngakhale njira za RL, monga A3C, zimafuna kuti chidziwitso chisinthidwe pakati pa ulusi wogwira ntchito ndi seva ya parameter, ES imangofunika kuyerekezera kwa thupi ndi chidziwitso chogawa magawo. Ndi chifukwa cha kuphweka kumeneku kuti njirayi ili patsogolo kwambiri kuposa njira zamakono za RL ponena za luso lokulitsa. Komabe, zonsezi sizibwera pachabe: muyenera kukhathamiritsa maukonde molingana ndi mfundo ya black box. Pankhaniyi, "bokosi lakuda" limatanthawuza kuti panthawi yophunzitsira mawonekedwe amkati mwa netiweki amanyalanyazidwa kwathunthu, ndipo zotsatira zake zonse (mphoto ya gawo) zimagwiritsidwa ntchito, ndipo zimatengera ngati zolemera za netiweki inayake zidzakwaniritsidwa. kutengera kwa mibadwo yotsatira. M'mikhalidwe yomwe sitilandira ndemanga zambiri kuchokera ku chilengedwe-komanso m'mabvuto ambiri amtundu wa RL kutuluka kwa mphotho kumakhala kochepa kwambiri-vuto limachokera ku "bokosi lakuda" mpaka "bokosi lakuda kwathunthu." Pankhaniyi, mutha kuwonjezera zokolola zambiri, kotero, zowona, kunyengerera koteroko ndikoyenera. "Ndani amafunikira ma gradients ngati ali ndi phokoso lopanda chiyembekezo?" - ili ndilo lingaliro lonse.
Komabe, m'malo omwe mayankho amakhala achangu, zinthu zimayamba kusokonekera kwa ES. Gulu la OpenAI likufotokoza momwe maukonde osavuta a MNIST adaphunzitsidwa pogwiritsa ntchito ES, ndipo nthawi ino maphunzirowo anali ocheperako nthawi 1000. Chowonadi ndi chakuti chizindikiro cha gradient mumagulu azithunzi ndi chidziwitso kwambiri chokhudza momwe mungaphunzitsire gulu labwino la maukonde. Chifukwa chake, vuto limakhala locheperako ndi njira ya RL komanso zambiri zokhala ndi mphotho zochepa m'malo omwe amapanga phokoso laphokoso.
Yankho lachilengedwe
Ngati tiyesa kuphunzira kuchokera ku chitsanzo cha chilengedwe, kuganizira njira zopangira AI, ndiye kuti nthawi zina AI ikhoza kuganiziridwa ngati . Kupatula apo, chilengedwe chimagwira ntchito mwazovuta zomwe asayansi apakompyuta alibe. Pali lingaliro lakuti njira yongoganizira chabe yothetsera vuto linalake ingathe kupereka mayankho ogwira mtima kuposa njira zowonetsera. Komabe, ndikuganiza kuti zingakhale zopindulitsa kuyesa momwe machitidwe osinthika amagwirira ntchito pansi pa zovuta zina (Dziko Lapansi) apanga othandizira (nyama, makamaka zoyamwitsa) zomwe zimatha kusinthasintha komanso zovuta. Ngakhale zina mwazolepheretsa izi sizikugwira ntchito m'maiko ofananirako a sayansi ya data, zina zili bwino.
Titapenda luntha la zinyama zoyamwitsa, tikuwona kuti zimapangidwira chifukwa cha zovuta zomwe zimagwirizanitsa njira ziwiri zogwirizana kwambiri: kuphunzira kuchokera ku zochitika za ena ΠΈ kuphunzira mwa kuchita. Zakale nthawi zambiri zimafanana ndi chisinthiko choyendetsedwa ndi kusankhidwa kwachilengedwe, koma apa ndimagwiritsa ntchito mawu ochulukirapo kuti ndiganizire za epigenetics, microbiomes, ndi njira zina zomwe zimathandiza kugawana zochitika pakati pa zamoyo zomwe sizikugwirizana ndi majini. Njira yachiwiri, kuphunzira kuchokera kuzochitika, ndizo zonse zomwe nyama imatha kuphunzira m'moyo wake wonse, ndipo chidziwitsochi chimatsimikiziridwa mwachindunji ndi kugwirizana kwa nyamayi ndi dziko lakunja. Gululi limaphatikizapo chilichonse kuyambira kuphunzira kuzindikira zinthu mpaka kudziwa bwino kulumikizana komwe kumachitika pakuphunzira.
Mwachidule, njira ziwirizi zomwe zimachitika mwachilengedwe zitha kufananizidwa ndi njira ziwiri zokongoletsera ma neural network. Njira zachisinthiko, pomwe chidziwitso chokhudza ma gradients chimagwiritsidwa ntchito kusinthira zamoyo, zimayandikira kuphunzira kuchokera ku zomwe ena adakumana nazo. Mofananamo, njira za gradient, kumene kupeza chimodzi kapena china kumabweretsa kusintha kwa khalidwe la wothandizira, mofanana ndi kuphunzira kuchokera ku zochitika zanu. Ngati tilingalira za mitundu ya khalidwe lanzeru kapena maluso omwe njira iliyonse ya njira ziwirizi imayambira pa zinyama, kuyerekezerako kumawonekera kwambiri. M'zochitika zonsezi, "njira zachisinthiko" zimalimbikitsa kuphunzira za makhalidwe omwe amalola munthu kukhala ndi thanzi labwino (lokwanira kuti akhalebe ndi moyo). Kuphunzira kuyenda kapena kuthawa ku ukapolo nthawi zambiri kumakhala kofanana ndi makhalidwe "achibadwa" omwe ali "olimba" mu zinyama zambiri pamtundu wa chibadwa. Kuphatikiza apo, chitsanzochi chikutsimikizira kuti njira zachisinthiko zimagwira ntchito pomwe chizindikiro cha mphotho chimakhala chosowa kwambiri (mwachitsanzo, kulera bwino mwana). Zikatero, nβkosatheka kugwirizanitsa mphothoyo ndi zochita zinazake zomwe mwina zakhala zikuchitika zaka zambiri izi zisanachitike. Kumbali ina, ngati tilingalira nkhani yomwe ES imalephera, ndiko kugawanika kwa zithunzi, zotsatira zake zimafanana modabwitsa ndi zotsatira za maphunziro a zinyama zomwe zimapezedwa muzoyesa zambiri zamaganizo zamakhalidwe zomwe zachitika pa 100-kuphatikiza zaka.
Kuphunzira kwa Zinyama
Njira zomwe zimagwiritsidwa ntchito polimbikitsa kuphunzira nthawi zambiri zimatengedwa mwachindunji kuchokera ku zolemba zamaganizidwe , ndipo opareshoni conditioning anaphunziridwa pogwiritsa ntchito nyama psychology. Mwa njira, Richard Sutton, m'modzi mwa omwe adayambitsa maphunziro olimbikitsa, ali ndi digiri ya bachelor mu psychology. Pankhani yoyendetsera ntchito, nyama zimaphunzira kugwirizanitsa mphotho kapena chilango ndi machitidwe enaake. Ophunzitsa ndi ochita kafukufuku amatha kusokoneza mgwirizano wa mphothoyi mwanjira ina, kupangitsa nyama kuwonetsa luntha kapena machitidwe ena. Komabe, kuwongolera magwiridwe antchito, monga momwe kumagwiritsidwira ntchito pakufufuza zanyama, sikuli kanthu koma mawonekedwe odziwika bwino amtundu womwewo pamaziko omwe nyama zimaphunzira m'miyoyo yawo yonse. Timalandila nthawi zonse zidziwitso za kulimbikitsidwa kochokera ku chilengedwe ndikusintha machitidwe athu moyenerera. M'malo mwake, akatswiri ambiri odziwa zamatsenga komanso asayansi ozindikira amakhulupirira kuti anthu ndi nyama zina zimagwira ntchito kwambiri ndipo amaphunzira kulosera zomwe zidzachitike m'tsogolomu kutengera mphotho zomwe zingachitike.
Udindo waukulu wa kulosera pakuphunzira kuchokera ku zomwe zachitika zimasintha zomwe zafotokozedwa pamwambapa m'njira zazikulu. Chizindikiro chomwe poyamba chinkaganiziridwa kuti ndi chochepa kwambiri (mphoto ya episodic) imakhala yochuluka kwambiri. Mwachidziwitso, zinthu zili ngati izi: nthawi iliyonse, ubongo wa nyama yoyamwitsa umawerengera zotsatira zochokera kumagulu ovuta a zokopa ndi zochita, pamene nyama imangomizidwa mumtsinje uwu. Pankhaniyi, khalidwe lomaliza la nyama limapereka chizindikiro champhamvu chomwe chiyenera kugwiritsidwa ntchito kutsogolera kusintha kwa maulosi ndi chitukuko cha khalidwe. Ubongo umagwiritsa ntchito zizindikilo zonsezi kuti ukwaniritse zolosera (ndipo, molingana ndi zomwe zachitika) m'tsogolomu. Chidule cha njira iyi chikuperekedwa m'buku labwino kwambiri "β wasayansi wozindikira komanso wafilosofi Andy Clark. Ngati tiwonjezera kulingalira kotereku pakuphunzitsa zida zopangira, ndiye kuti cholakwika chachikulu pakuphunzirira kolimbikitsa chimawululidwa: chizindikiro chomwe chimagwiritsidwa ntchito mu paradigm ndi chofooka mopanda chiyembekezo poyerekeza ndi momwe chingakhalire (kapena chiyenera kukhala). Pazifukwa zomwe sizingatheke kuonjezera machulukitsidwe azizindikiro (mwina chifukwa mwachibadwa ndi ofooka kapena okhudzana ndi reactivity yotsika), ndi bwino kusankha njira yophunzitsira yomwe ili yofanana bwino, mwachitsanzo, ES.
Maphunziro olemera a neural network
Kumanga pa mfundo za zochitika zapamwamba za ubongo zomwe zimapezeka mu ubongo wa mammalian, womwe umakhala wotanganidwa nthawi zonse kuneneratu, kupita patsogolo kwaposachedwa kwapangidwa polimbikitsa maphunziro, omwe tsopano akuganizira kufunika kwa maulosi oterowo. Nditha kukupangirani ntchito ziwiri zofanana:
M'mapepala onsewa, olemba amawonjezera ndondomeko yosasinthika ya ma neural network awo ndi zotsatira zolosera za momwe chilengedwe chidzakhalire m'tsogolomu. M'nkhani yoyamba, kulosera kumagwiritsidwa ntchito pamitundu yosiyanasiyana ya miyeso, ndipo chachiwiri, kulosera kumagwiritsidwa ntchito pakusintha kwa chilengedwe ndi khalidwe la wothandizira monga choncho. Pazochitika zonsezi, chizindikiro chochepa chokhudzana ndi kulimbikitsana kwabwino chimakhala cholemera kwambiri komanso chodziwitsa zambiri, zomwe zimalola kuphunzira mofulumira komanso kupeza makhalidwe ovuta kwambiri. Kusintha koteroko kumangopezeka ndi njira zomwe zimagwiritsa ntchito chizindikiro cha gradient, osati ndi njira zomwe zimagwira ntchito pa mfundo ya "black box", monga ES.
Kuphatikiza apo, kuphunzira kuchokera kuzomwe zachitika komanso njira zowongolera ndizothandiza kwambiri. Ngakhale pazochitika zomwe zinali zotheka kuphunzira vuto linalake pogwiritsa ntchito njira ya ES mofulumira kuposa kugwiritsa ntchito kulimbikitsa maphunziro, phindu linapindula chifukwa chakuti njira ya ES inaphatikizapo zambiri zambiri kuposa RL. Poganizira nkhaniyi pa mfundo za maphunziro a nyama, tikuwona kuti zotsatira za kuphunzira kuchokera ku chitsanzo cha munthu wina zimadziwonetsera pambuyo pa mibadwo yambiri, pamene nthawi zina chochitika chimodzi chokha chokha chimakhala chokwanira kuti nyamayo iphunzirepo phunziro kwamuyaya. Ngakhale ngati Ngakhale sizikugwirizana ndi njira zachikhalidwe, ndizomveka kwambiri kuposa ES. Pali, mwachitsanzo, njira monga , pomwe ma Q-makhalidwe amasungidwa panthawi yophunzitsidwa, pambuyo pake pulogalamuyo imayang'ana musanachitepo kanthu. Zotsatira zake ndi njira ya gradient yomwe imakupatsani mwayi wophunzirira kuthana ndi mavuto mwachangu kuposa kale. M'nkhani yonena za neural episodic control, olembawo amatchula za hippocampus yaumunthu, yomwe imatha kusunga zambiri za chochitika ngakhale zitachitika kamodzi, motero, zimasewera. m'njira yokumbukira. Njira zoterezi zimafuna mwayi wopita ku bungwe lamkati la wothandizira, lomwenso, mwa kutanthauzira, zosatheka mu ES paradigm.
Ndiye bwanji osaphatikiza?
Ndizotheka kuti zambiri za nkhaniyi zitha kusiya malingaliro kuti ndikulimbikitsa njira za RL. Komabe, ndikuganiza kuti m'kupita kwanthawi njira yabwino yothetsera vutoli ndikuphatikiza njira zonse ziwiri, kotero kuti iliyonse igwiritsidwe ntchito pazochitika zomwe zili zoyenera. Mwachiwonekere, pankhani ya mfundo zambiri zogwira ntchito kapena mukakhala ndi zizindikiro zochepa zolimbikitsira, ES imapambana, makamaka ngati muli ndi mphamvu zamakompyuta zomwe mungathe kuchita nawo maphunziro ofanana. Kumbali ina, njira zochepetsera kugwiritsa ntchito maphunziro olimbikitsa kapena kuphunzira koyang'aniridwa zidzakhala zothandiza tikakhala ndi mayankho ochulukirapo ndipo tifunika kuphunzira momwe tingathetsere vuto mwachangu komanso ndi data yochepa.
Kutembenukira ku chilengedwe, timapeza kuti njira yoyamba, makamaka, imayala maziko achiwiri. Ichi ndichifukwa chake, pakupita kwa chisinthiko, nyama zoyamwitsa zapanga ubongo zomwe zimawalola kuphunzira bwino kwambiri kuchokera kuzizindikiro zovuta zochokera ku chilengedwe. Kotero, funso limakhala lotseguka. Mwina njira zachisinthiko zitithandiza kupanga njira zophunzirira zogwira mtima zomwe zingakhalenso zothandiza panjira zophunzirira mopepuka. Ndi iko komwe, njira yopezeka mwachilengedwe imakhaladi yopambana kwambiri.
Source: www.habr.com