Hai Habr!
Ba sau da yawa ba mu yanke shawarar buga fassarorin rubutun da suka kasance shekaru biyu ba, ba tare da lamba ba kuma a sarari na yanayin ilimi - amma a yau za mu keɓanta. Muna fatan cewa matsalar da aka gabatar a cikin taken labarin ta damu da yawancin masu karatunmu, kuma kun riga kun karanta ainihin aikin dabarun juyin halitta wanda wannan sakon ya yi jayayya a cikin asali ko za ku karanta shi yanzu. Barka da zuwa cat!
A cikin Maris 2017, OpenAI ya yi taguwar ruwa a cikin al'ummar ilmantarwa mai zurfi tare da takarda "" Wannan aikin ya bayyana sakamako mai ban sha'awa a cikin gaskiyar cewa ƙarfafa ilmantarwa (RL) bai zama kullun ba, kuma lokacin horar da hanyoyin sadarwar jijiyoyi, yana da kyau a gwada wasu hanyoyin. Sai wata muhawara ta barke game da mahimmancin ƙarfafa koyo da kuma yadda ya cancanci matsayinsa a matsayin fasaha na "dole ne" don koyar da warware matsala. Anan ina so in ce bai kamata a dauki wadannan fasahohin guda biyu a matsayin masu takara ba, wanda daya daga cikinsu ya fi sauran; akasin haka, a ƙarshe suna haɗa juna. Lalle ne, idan kun yi tunani kadan game da abin da ake bukata don ƙirƙirar da kuma irin wadannan tsare-tsare, wadanda a duk tsawon rayuwarsu za su iya koyo, yanke hukunci da tsarawa, to lalle ne za mu kai ga yanke shawarar cewa za a bukaci wannan ko wancan hadadden bayani. Af, daidai wannan hadadden bayani ne yanayi ya zo da shi, wanda ya baiwa dabbobi masu shayarwa da sauran manyan dabbobi da hadaddun hankali a lokacin juyin halitta.
Dabarun Juyin Halitta
Babban jigon takarda na OpenAI shine, maimakon yin amfani da ilmantarwa na ƙarfafawa tare da al'ada na al'ada, sun sami nasarar horar da hanyar sadarwa ta hanyar sadarwa don magance matsaloli masu rikitarwa ta amfani da abin da suka kira "dabarun juyin halitta" (ES). Wannan tsarin na ES ya ƙunshi ci gaba da rarraba ma'auni na cibiyar sadarwa, wanda ya haɗa da wakilai da yawa da ke aiki a layi daya da kuma amfani da sigogi da aka zaɓa daga wannan rarraba. Kowane wakili yana aiki a cikin nasa muhallin, kuma bayan kammala ƙayyadadden adadin jigogi ko matakai na jigo, algorithm ɗin yana dawo da lada mai tarin yawa, wanda aka bayyana azaman ƙimar dacewa. Yin la'akari da wannan darajar, za a iya karkatar da rarraba sigogi zuwa ga wakilai masu nasara, da hana masu nasara. Ta hanyar maimaita irin wannan aiki miliyoyin sau tare da halartar daruruwan wakilai, yana yiwuwa a motsa rarraba ma'auni zuwa sararin samaniya wanda zai ba da damar wakilai su tsara manufofi masu kyau don magance aikin da aka ba su. Lalle ne, sakamakon da aka gabatar a cikin labarin yana da ban sha'awa: an nuna cewa idan kun gudanar da wakilai dubu a cikin layi daya, to, ana iya koyan locomotion na anthropomorphic a kan kafafu biyu a cikin ƙasa da rabin sa'a (yayin da har ma hanyoyin RL mafi ci gaba suna buƙatar ciyarwa da yawa). fiye da awa daya akan wannan). Don ƙarin cikakkun bayanai, Ina ba da shawarar karanta mafi kyau daga marubutan gwajin, haka nan .
Daban-daban dabaru don koyar da anthropomorphic tafiya madaidaiciya, karatu ta amfani da hanyar ES daga OpenAI.
Akwatin baki
Babban fa'idar wannan hanyar ita ce ana iya daidaita ta cikin sauƙi. Yayin da hanyoyin RL, kamar A3C, suna buƙatar musayar bayanai tsakanin zaren ma'aikaci da uwar garken siga, ES kawai yana buƙatar ƙididdigan dacewa da bayanan rarraba ma'auni. Saboda wannan sauƙi ne wannan hanya ta yi nisa a gaban hanyoyin RL na zamani dangane da iyawar ƙira. Duk da haka, duk wannan ba ya zo a banza: dole ne ka inganta cibiyar sadarwa bisa ga ka'idar akwatin baki. A wannan yanayin, "akwatin baƙar fata" yana nufin cewa a lokacin horar da tsarin na ciki na cibiyar sadarwa an yi watsi da shi gaba daya, kuma kawai ana amfani da sakamakon gaba ɗaya (lada ga abin da ya faru), kuma ya dogara da shi ko ma'auni na wani cibiyar sadarwa zai kasance. za a gadar da al'ummomi masu zuwa. A cikin yanayin da ba mu sami ra'ayi mai yawa daga yanayin ba - kuma a yawancin matsalolin RL na gargajiya ladaran lada ba su da yawa - matsalar ta tashi daga kasancewa "akwatin baki" zuwa "akwatin baki cikakke." A wannan yanayin, zaka iya ƙara yawan yawan aiki, don haka, ba shakka, irin wannan sulhu ya dace. "Wa ke buƙatar gradients idan suna da hayaniya ko ta yaya?" - wannan shine ra'ayi na gaba ɗaya.
Koyaya, a cikin yanayin da martani ya fi aiki, abubuwa sun fara yin kuskure ga ES. Ƙungiyar OpenAI ta bayyana yadda aka horar da hanyar sadarwa mai sauƙi ta MNIST ta amfani da ES, kuma wannan lokacin horon ya kasance sau 1000 a hankali. Gaskiyar ita ce siginar gradient a cikin rarrabuwar hoto yana da cikakken bayani game da yadda ake koyar da cibiyar sadarwa mafi kyawu. Don haka, matsalar ta yi ƙasa da dabarar RL kuma tare da ƙarancin lada a cikin mahallin da ke samar da ƙararrawa.
Maganin yanayi
Idan muka yi ƙoƙari mu koyi daga misalin yanayi, tunani game da hanyoyin bunkasa AI, to, a wasu lokuta ana iya tunanin AI kamar . Bayan haka, yanayi yana aiki a cikin iyakokin da masana kimiyyar kwamfuta ba su da shi. Akwai ra'ayi cewa tsarin ka'ida kawai don magance wata matsala na iya samar da ingantattun mafita fiye da hanyoyin da za'a bi. Duk da haka, har yanzu ina tsammanin zai dace a gwada yadda tsarin da ke aiki a ƙarƙashin wasu ƙuntatawa (Duniya) ya haifar da wakilai (dabbobi, musamman ma masu shayarwa) masu iya sassauƙa da ɗabi'a mai rikitarwa. Yayin da wasu daga cikin waɗannan ƙuntatawa ba sa aiki a cikin duniyar kimiyyar bayanai da aka kwaikwayi, wasu suna da kyau.
Bayan da muka yi nazari kan halayyar dabbobi masu shayarwa, mun ga cewa an kafa ta ne sakamakon hadadden tasirin juna na matakai biyu masu alaka da juna: koyo daga abubuwan wasu и koyo ta hanyar aikatawa. Na farko ana daidaita shi da juyin halitta wanda zabin yanayi ke tafiyar da shi, amma a nan na yi amfani da kalma mai faɗi don yin la'akari da epigenetics, microbiomes, da sauran hanyoyin da ke ba da damar raba abubuwan da ke tsakanin kwayoyin halitta marasa alaƙa. Hanya ta biyu, koyo daga gogewa, ita ce dukkan bayanan da dabba ke sarrafa ta koyo a tsawon rayuwarta, kuma wannan bayanin yana samuwa ne kai tsaye ta hanyar hulɗar wannan dabba da duniyar waje. Wannan rukunin ya haɗa da komai daga koyo zuwa gane abubuwa zuwa ƙwarewar sadarwa da ke cikin tsarin ilmantarwa.
Kusan magana, waɗannan matakai guda biyu da ke faruwa a yanayi ana iya kwatanta su da zaɓuɓɓuka biyu don inganta hanyoyin sadarwar jijiya. Dabarun juyin halitta, inda ake amfani da bayanai game da gradients don sabunta bayanai game da kwayoyin halitta, suna kusa da koyo daga kwarewar wasu. Hakazalika, hanyoyin gradient, inda samun ɗaya ko wani ƙwarewa ke haifar da ɗayan ko wani canji a cikin halayen wakilin, suna kama da koyo daga kwarewar mutum. Idan muka yi la'akari da nau'o'in halaye na basira ko iyawar da kowane ɗayan waɗannan hanyoyin biyu ke tasowa a cikin dabbobi, kwatancen yana ƙara bayyana. A cikin duka biyun, "hanyoyin juyin halitta" suna haɓaka nazarin halayen halayen halayen da ke ba mutum damar haɓaka wani yanayin dacewa (wanda ya isa ya rayu). Koyon tafiya ko tserewa daga zaman talala a yawancin lokuta daidai yake da ƙarin halaye na “haɓaka” waɗanda “masu wahala” a cikin dabbobi da yawa a matakin kwayoyin halitta. Bugu da kari, wannan misalin yana tabbatar da cewa hanyoyin juyin halitta suna amfani da su a lokuta inda siginar lada ba ta da yawa (misali, gaskiyar nasarar renon jariri). A irin wannan yanayin, ba shi yiwuwa a daidaita ladan tare da kowane takamaiman ayyuka da ƙila an yi shekaru da yawa kafin faruwar wannan gaskiyar. A gefe guda, idan muka yi la'akari da shari'ar da ES ta kasa, wato rarraba hoto, sakamakon yana da kyau kwatankwacin sakamakon koyo na dabba da aka samu a cikin gwaje-gwajen dabi'a marasa adadi da aka gudanar sama da shekaru 100-plus.
Koyo daga Dabbobi
Hanyoyin da aka yi amfani da su wajen ƙarfafa ilmantarwa suna a yawancin lokuta ana ɗaukar su kai tsaye daga wallafe-wallafen tunani akan , kuma an yi nazarin yanayin aiki ta hanyar amfani da ilimin halin dabba. Af, Richard Sutton, ɗaya daga cikin masu haɓaka koyon ƙarfafawa, yana da digiri na farko a cikin ilimin halin ɗan adam. A cikin mahallin yanayin aiki, dabbobi suna koyon alaƙar lada ko hukunci tare da takamaiman yanayin ɗabi'a. Masu horarwa da masu bincike na iya sarrafa wannan ƙungiyar lada ta hanya ɗaya ko wata, tada hankalin dabbobi don nuna hankali ko wasu halaye. Duk da haka, operant conditioning, kamar yadda aka yi amfani da shi a cikin binciken dabba, ba kome ba ne face nau'i mai mahimmanci na nau'i ɗaya na yanayin yanayin da dabbobi ke koya a duk rayuwarsu. Kullum muna karɓar sigina na ingantaccen ƙarfafawa daga yanayi kuma muna daidaita halayenmu daidai. A gaskiya ma, yawancin masana kimiyyar neuroscientists da masana kimiyyar fahimi sun yi imanin cewa mutane da sauran dabbobi a zahiri suna aiki a matakin da ya fi girma kuma suna ci gaba da koyan hasashen sakamakon halayensu a yanayi na gaba dangane da yuwuwar lada.
Matsayin tsakiya na tsinkaya a cikin koyo daga gwaninta yana canza yanayin da aka kwatanta a sama ta hanyoyi masu mahimmanci. Siginar da a baya an yi la'akari da shi ba ta da yawa (ladan episodic) ta zama mai yawa sosai. A bisa ka'ida, yanayin wani abu ne kamar haka: a kowane lokaci, kwakwalwar dabbar dabbar tana ƙididdige sakamako bisa ga hadadden rafi na motsa jiki da ayyuka, yayin da dabbar ke nutsewa cikin wannan rafi. A wannan yanayin, hali na ƙarshe na dabba yana ba da sigina mai ƙarfi wanda dole ne a yi amfani da shi don jagorantar daidaitawar hasashen da haɓaka halayen. Kwakwalwa tana amfani da duk waɗannan sigina don inganta hasashe (kuma, bisa ga haka, ingancin ayyukan da aka ɗauka) a nan gaba. An ba da bayyani na wannan hanyar a cikin mafi kyawun littafin "Masanin kimiyya kuma masanin falsafa Andy Clark. Idan muka fitar da irin wannan dalili zuwa horar da ma'aikatan wucin gadi, to, an bayyana aibi mai mahimmanci a cikin ƙarfafa ilmantarwa: siginar da aka yi amfani da shi a cikin wannan yanayin ba shi da ƙarfi idan aka kwatanta da abin da zai iya zama (ko ya kamata). A cikin lokuta inda ba zai yiwu ba don ƙara yawan siginar sigina (watakila saboda yana da rauni a cikin asali ko kuma yana hade da ƙananan matakan sake kunnawa), yana da kyau a fi son tsarin horo wanda ya dace da kyau, misali, ES.
Ingantattun horarwa na hanyoyin sadarwa na jijiyoyi
Gina kan ka'idodin ayyukan jijiyoyi mafi girma da ke cikin kwakwalwar mammalian, wanda ke aiki akai-akai don yin tsinkaya, an sami ci gaba na baya-bayan nan don ƙarfafa ilmantarwa, wanda a yanzu yayi la'akari da muhimmancin irin wannan tsinkaya. Nan da nan zan iya ba ku shawarar ayyuka iri ɗaya guda biyu:
A cikin waɗannan takaddun guda biyu, marubutan sun ƙara ƙa'idar da aka saba amfani da su na hanyoyin sadarwar su tare da sakamako na tsinkaya game da yanayin muhalli a nan gaba. A cikin kasidar farko, ana amfani da tsinkaya akan nau'ikan ma'auni iri-iri, kuma a cikin na biyu, ana amfani da tsinkaya ga canje-canje a cikin muhalli da halayen wakili kamar haka. A cikin duka biyun, siginar da ba ta dace ba da ke da alaƙa da ingantaccen ƙarfafawa ya zama mafi arziƙi da ƙarin bayani, yana ba da damar koyo da sauri da kuma samun ƙarin halaye masu rikitarwa. Irin waɗannan haɓakawa suna samuwa ne kawai tare da hanyoyin da ke amfani da siginar gradient, kuma ba tare da hanyoyin da ke aiki akan ƙa'idar "akwatin baƙar fata", kamar ES.
Bugu da kari, koyo daga gogewa da hanyoyin gradient sun fi tasiri sosai. Ko da a cikin lokuta inda zai yiwu a yi nazarin wata matsala ta amfani da hanyar ES da sauri fiye da yin amfani da ilmantarwa na ƙarfafawa, an samu riba saboda gaskiyar cewa dabarun ES ya ƙunshi bayanai da yawa fiye da RL. Idan muka yi la’akari da ka’idojin ilmantarwa a cikin dabbobi, mun lura cewa sakamakon koyo daga misalin wani yana bayyana kansa bayan al’ummomi da yawa, yayin da wani lokaci wani lamari guda daya da kansa ya isa dabbar ta koyi darasi har abada. Yayin so Duk da yake bai dace da hanyoyin gradient na gargajiya ba, yana da hankali fiye da ES. Akwai, alal misali, hanyoyin kamar , inda Q-darajar da aka adana a lokacin horo, bayan haka shirin ya duba su kafin daukar mataki. Sakamakon shine hanyar gradient wanda ke ba ku damar koyon yadda ake magance matsalolin da sauri fiye da da. A cikin labarin game da kula da episodic na jijiyoyi, marubutan sun ambaci hippocampus na ɗan adam, wanda ke da ikon riƙe bayanai game da wani taron ko da bayan gwaninta guda ɗaya kuma, sabili da haka, wasa. a cikin tsarin tunawa. Irin waɗannan hanyoyin suna buƙatar samun dama ga ƙungiyar cikin gida na wakili, wanda kuma, ta ma'anar, ba zai yiwu ba a cikin yanayin ES.
Don haka, me zai hana a haɗa su?
Wataƙila yawancin wannan labarin na iya barin tunanin cewa ina ba da shawarar hanyoyin RL. Duk da haka, a zahiri ina tsammanin cewa a cikin dogon lokaci mafi kyawun mafita ita ce haɗa hanyoyin biyu, ta yadda kowane ana amfani da shi a cikin yanayin da ya fi dacewa. A bayyane yake, a cikin yanayin manufofin amsawa da yawa ko kuma cikin yanayi tare da sigina kaɗan na ingantacciyar ƙarfafawa, ES ta yi nasara, musamman idan kuna da ikon sarrafa kwamfuta a hannun ku wanda zaku iya gudanar da horo iri ɗaya. A daya hannun, hanyoyin gradient ta amfani da ƙarfafa koyo ko kulawa za su kasance da amfani lokacin da muka sami damar samun amsa mai yawa kuma muna buƙatar koyon yadda ake magance matsala cikin sauri da ƙarancin bayanai.
Idan muka juya ga dabi'a, zamu ga cewa hanya ta farko, a zahiri, ta kafa harsashi na biyu. Wannan shine dalilin da ya sa, a tsawon lokacin juyin halitta, dabbobi masu shayarwa sun haɓaka kwakwalwa wanda ke ba su damar koyo sosai daga sigina masu rikitarwa da ke fitowa daga yanayi. Don haka, tambayar ta kasance a buɗe. Wataƙila dabarun juyin halitta zasu taimaka mana ƙirƙira ingantaccen tsarin gine-ginen koyo waɗanda kuma zasu kasance masu amfani ga hanyoyin koyo a hankali. Bayan haka, maganin da aka samo ta yanayi yana da nasara sosai.
source: www.habr.com