Gulu la ofufuza ochokera ku yunivesite ya Stanford, University of California ku Berkeley, ETH Zurich, Graduate School of Economics, Carnegie Mellon University, komanso Yandex ndi Meta, asindikiza kachidindo ka injini yogwiritsira ntchito zilankhulo zazikulu pazachuma. -kachitidwe kolemetsa. Mwachitsanzo, injiniyo imapereka kuthekera kopanga magwiridwe antchito monga ChatGPT ndi Copilot pogwiritsa ntchito chitsanzo cha OPT-175B chophunzitsidwa kale, chokhala ndi magawo mabiliyoni a 175, pakompyuta yokhazikika yokhala ndi khadi yamasewera ya NVIDIA RTX3090 yokhala ndi 24GB ya kukumbukira kanema. Khodiyo imalembedwa ku Python, imagwiritsa ntchito dongosolo la PyTorch ndipo imagawidwa pansi pa chilolezo cha Apache 2.0.
Mulinso script yachitsanzo yopanga ma bots omwe amakulolani kutsitsa imodzi mwazilankhulo zomwe zimapezeka pagulu ndikuyamba kulankhulana (mwachitsanzo, poyendetsa lamulo "python apps/chatbot.py -model facebook/opt-30b β -percent 0 100 100 0 100 0 "). Monga maziko, akukonzekera kugwiritsa ntchito chilankhulo chachikulu chofalitsidwa ndi Facebook, chophunzitsidwa pamagulu a BookCorpus (mabuku 10 zikwi), CC-Stories, Pile (OpenSubtitles, Wikipedia, DM Mathematics, HackerNews, etc.), Pushshift. io (kutengera Reddit data) ndi CCNewsV2 (nkhani zakale). Chitsanzo chimakwirira ma tokeni pafupifupi 180 biliyoni (800 GB ya data). Masiku 33 ogwirira ntchito limodzi ndi 992 NVIDIA A100 80GB GPUs adagwiritsidwa ntchito pophunzitsa chitsanzocho.
Mukamagwiritsa ntchito mtundu wa OPT-175B pamakina okhala ndi NVIDIA T4 GPU (16GB), injini ya FlexGen idawonetsa magwiridwe antchito mpaka 100 mwachangu kuposa mayankho omwe adaperekedwa kale, kupangitsa kuti zilankhulo zazikulu zizipezeka mosavuta ndikuzilola kuti ziziyenda pamakina osadzipereka. ma accelerator. Nthawi yomweyo, FlexGen imatha kufananiza kuwerengera ndi ma GPU angapo. Kuti muchepetse kukula kwachitsanzocho, pulogalamu yophatikizira ya parameter ndi njira yosungiramo chikumbutso imagwiritsidwanso ntchito.
Pakadali pano, FlexGen imangothandizira mitundu ya zilankhulo za OPT, koma mtsogolomo opanga akulonjezanso kuwonjezera thandizo kwa BLOOM (magawo mabiliyoni 176, amathandizira zilankhulo 46 ndi zilankhulo 13), CodeGen (ikhoza kupanga code m'zilankhulo 22) ndi Zithunzi za GLM. Chitsanzo cha zokambirana ndi bot zochokera FlexGen ndi OPT-30B chitsanzo:
Munthu: Dzina la phiri lalitali kwambiri padziko lonse lapansi ndi liti?
Wothandizira: Everest.
Munthu: Ndikukonzekera ulendo wachikumbutso chathu. Kodi tingachite chiyani?
Wothandizira: Chabwino, pali zinthu zingapo zomwe mungachite pa tsiku lanu lokumbukira. Choyamba, mutha kusewera makadi. Chachiwiri, mukhoza kupita kokayenda. Chachitatu, mukhoza kupita ku nyumba yosungiramo zinthu zakale.
Source: opennet.ru