{"id":54494,"date":"2019-12-27T00:00:00","date_gmt":"2019-12-26T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/dzhedajskaya-tehnika-umensheniya-svertochnyh-setej-pruning"},"modified":"2020-02-18T14:02:30","modified_gmt":"2020-02-18T11:02:30","slug":"dzhedajskaya-tehnika-umensheniya-svertochnyh-setej-pruning","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/news\/dzhedajskaya-tehnika-umensheniya-svertochnyh-setej-pruning","title":{"rendered":"T\u00eb gjith\u00eb teknik\u00eb Jedi t\u00eb reduktimit t\u00eb rrjeteve konvencionale \u2014 pruning","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"T\u00eb<\/p>\n

<\/p>\n

Para ty s\u00ebrish nj\u00eb detyr\u00eb p\u00ebr detektimin e objekteve. Prioriteti \u00ebsht\u00eb shpejt\u00ebsia e operimit me sakt\u00ebsi t\u00eb pranueshme. Merr arkitektur\u00ebn YOLOv3 dhe vazhdoje trajnimin. Sakt\u00ebsia (mAp75) \u00ebsht\u00eb m\u00eb shum\u00eb se 0.95. Por shpejt\u00ebsia e ekzekutimit ende mbetet e ul\u00ebt. Damn. <\/p>\n

<\/p>\n

Sot do ta kalojm\u00eb kualifikimin. N\u00ebn titullin do t\u00eb shqyrtojm\u00eb Model Pruning<\/strong> \u2014 prerja e pjes\u00ebve t\u00eb tep\u00ebrta t\u00eb rrjetit p\u00ebr t\u00eb p\u00ebrshpejtuar Inferenc\u00ebn pa humbur sakt\u00ebsin\u00eb. P\u00ebrshtatsh\u00ebm \u2014 nga ku, sa dhe si mund t\u00eb priten. Do t\u00eb shqyrtojm\u00eb si ta b\u00ebjm\u00eb k\u00ebt\u00eb manualisht dhe ku mund t\u00eb automatizohet. N\u00eb fund \u2014 nj\u00eb depozit\u00eb n\u00eb keras.<\/p>\n

<\/a><\/noindex><\/p>\n

Hyrje<\/h3>\n

<\/p>\n

N\u00eb pun\u00ebn time t\u00eb kaluar, n\u00eb Macroscopin e Permit, kam fituar nj\u00eb zakon \u2014 gjithmon\u00eb t\u00eb monitoroj koh\u00ebn e ekzekutimit t\u00eb algoritmeve. Dhe koha e ekzekutimit t\u00eb rrjeteve gjithmon\u00eb duhet t\u00eb kontrollohet p\u00ebrmes filtrit t\u00eb p\u00ebrshtatshm\u00ebris\u00eb. Zakonisht, state-of-the-art n\u00eb prodhim nuk kalojn\u00eb k\u00ebt\u00eb filter, q\u00eb m\u00eb ka \u00e7uar te Pruning. <\/p>\n

<\/p>\n

Pruning \u00ebsht\u00eb nj\u00eb tem\u00eb e vjet\u00ebr, e cila u diskutua n\u00eb lecturat e Stanfordit<\/a><\/noindex> n\u00eb vitin 2017. Ideja kryesore \u2014 reduktimi i madh\u00ebsis\u00eb s\u00eb rrjetit t\u00eb trajnuar pa humbur sakt\u00ebsin\u00eb p\u00ebrmes heqjes s\u00eb nodeve t\u00eb ndryshme. D\u00ebgjohet bukur, por rrall\u00eb d\u00ebgjoj p\u00ebr aplikimin e tij. Ndoshta mungojn\u00eb implementimet, nuk ka artikuj n\u00eb gjuh\u00ebn ruse ose thjesht t\u00eb gjith\u00eb mendojn\u00eb se pruning \u00ebsht\u00eb nj\u00eb know-how dhe heshtin.
\nPor le t\u00eb fillojm\u00eb.<\/p>\n

<\/p>\n

Nj\u00eb shikim n\u00eb biologji<\/h3>\n

<\/p>\n

M\u00eb p\u00eblqen kur n\u00eb Deep Learning hyjn\u00eb ide nga biologia. Ato, si dhe evolucioni, jan\u00eb t\u00eb besueshme (dhe a e dinte se ReLU \u00ebsht\u00eb shum\u00eb e ngjashme me funksionin e aktivizimit t\u00eb neuron\u00ebve n\u00eb tru<\/a><\/noindex>?) <\/p>\n

<\/p>\n

Procesi i Model Pruning \u00ebsht\u00eb gjithashtu i ngjash\u00ebm me biolgjin\u00eb. Reagimi i rrjetit k\u00ebtu mund t\u00eb krahasohet me plastik\u00ebn e trurit. Disa shembuj interesant\u00eb jan\u00eb n\u00eb librin Norman Doidge<\/a><\/noindex>:<\/p>\n

<\/p>\n

    \n
  1. Tru i nj\u00eb gruaje q\u00eb ka pasur vet\u00ebm nj\u00eb gjysm\u00eb q\u00eb nga lindja, e ka rikrijuar veten p\u00ebr t\u00eb kryer funksionet e gjysm\u00ebs s\u00eb munguar.<\/li>\n
  2. Nj\u00eb djal\u00eb p\u00ebrjashtoi pjes\u00ebn e trurit q\u00eb p\u00ebrgjigjej p\u00ebr shikimin. Me kalimin e koh\u00ebs, pjes\u00eb t\u00eb tjera t\u00eb trurit mor\u00ebn p\u00ebrsip\u00ebr k\u00ebto funksione. (nuk po p\u00ebrpiqemi ta p\u00ebrs\u00ebrisim)<\/li>\n<\/ol>\n

    <\/p>\n

    Ashtu si nga modeli juaj mund t\u00eb priten disa konvolucione t\u00eb dob\u00ebta. N\u00eb rastin m\u00eb t\u00eb keq, konvolucionet e mbetura do t\u00eb ndihmojn\u00eb p\u00ebr t\u00eb z\u00ebvend\u00ebsuar ato t\u00eb prera. <\/p>\n

    <\/p>\n

    Doni Transfer Learning apo po m\u00ebsoni nga fillimi?<\/h3>\n

    <\/p>\n

    Opsioni num\u00ebr nj\u00eb.<\/strong> Po p\u00ebrdorni Transfer Learning n\u00eb Yolov3. Retina, Mask-RCNN ose U-Net. Por shpesh na nevojitet t\u00eb njohim 80 klasa objektesh, si n\u00eb COCO. N\u00eb praktik\u00ebn time, gjith\u00e7ka \u00ebsht\u00eb e kufizuar n\u00eb 1-2 klasa. Mund t\u00eb supozohet se arkitektura p\u00ebr 80 klasa \u00ebsht\u00eb e tep\u00ebrt k\u00ebtu. E mendoj se arkitektura duhet t\u00eb zvog\u00eblohet. P\u00ebr m\u00eb tep\u00ebr, d\u00ebshirojm\u00eb ta b\u00ebjm\u00eb k\u00ebt\u00eb pa humbur pesha t\u00eb paratanuara q\u00eb kemi.<\/p>\n

    <\/p>\n

    Opsioni num\u00ebr dy.<\/strong> Ndoshta keni shum\u00eb t\u00eb dh\u00ebna dhe burime llogarit\u00ebse ose thjesht keni nevoj\u00eb p\u00ebr nj\u00eb arkitektur\u00eb super t\u00eb personalizuar. Nuk ka r\u00ebnd\u00ebsi. Por po e m\u00ebsoni rrjetin nga fillimi. Renditja e zakonshme \u2014 shikojm\u00eb struktur\u00ebn e t\u00eb dh\u00ebnave, p\u00ebrzgjedhim nj\u00eb arkitektur\u00eb TEP\u00cbR KOHA dhe shtojm\u00eb dropaout p\u00ebr t\u00eb parandaluar mbip\u00ebrgatitjen. Kam par\u00eb dropaout 0.6, Karl. <\/p>\n

    <\/p>\n

    N\u00eb t\u00eb dyja rastet, rrjeti mund t\u00eb zvog\u00eblohet. T\u00eb stimuluar. Tani shkojm\u00eb t\u00eb m\u00ebsojm\u00eb se \u00e7far\u00eb \u00ebsht\u00eb pruning.<\/p>\n

    <\/p>\n

    Algoritmi i p\u00ebrgjithsh\u00ebm<\/h3>\n

    <\/p>\n

    Ne vendos\u00ebm se mund t\u00eb hiqnim konvolucionet. Kjo duket shum\u00eb e thjesht\u00eb:<\/p>\n

    <\/p>\n

    \"T\u00eb<\/p>\n

    <\/p>\n

    Heqja e \u00e7do konvolucioni \u00ebsht\u00eb nj\u00eb stres p\u00ebr rrjetin, q\u00eb zakonisht \u00e7on n\u00eb rritjen e ndonj\u00eb gabimi. Nga nj\u00ebra an\u00eb, kjo rritje gabimi \u00ebsht\u00eb nj\u00eb tregues se sa sakt\u00ebsisht po heqim konvolucionet (p\u00ebr shembull, nj\u00eb rritje e madhe tregon se po b\u00ebjm\u00eb di\u00e7ka t\u00eb gabuar). Por nj\u00eb rritje e vog\u00ebl \u00ebsht\u00eb plot\u00ebsisht e pranueshme dhe shpesh eliminohet nga provimi m\u00eb i leht\u00eb i m\u00ebvonsh\u00ebm me nj\u00eb LR t\u00eb vog\u00ebl. Shtojm\u00eb hapin e m\u00ebsimit:<\/p>\n

    <\/p>\n

    \"T\u00eb<\/p>\n

    <\/p>\n

    Tani duhet t\u00eb kuptojm\u00eb se kur duam t\u00eb ndalim ciklin ton\u00eb LearningPruning. K\u00ebtu mund t\u00eb ket\u00eb variante ekzotike, kur na nevojitet t\u00eb zvog\u00eblojm\u00eb rrjetin n\u00eb nj\u00eb madh\u00ebsi dhe shpejt\u00ebsi t\u00eb caktuar (p\u00ebr shembull, p\u00ebr pajisje mobile). Megjithat\u00eb, varianti m\u00eb i zakonsh\u00ebm \u2014 \u00ebsht\u00eb vazhdimi i ciklit derisa gabimi t\u00eb b\u00ebhet m\u00eb i lart\u00eb se e lejuara. Shtojm\u00eb kushtin:<\/p>\n

    <\/p>\n

    \"T\u00eb<\/p>\n

    <\/p>\n

    Pra, algoritmi b\u00ebhet i qart\u00eb. Mbetej t\u00eb shqyrtonim se si t\u00eb p\u00ebrcaktojm\u00eb konvolucionet q\u00eb do t\u00eb hiqen.<\/p>\n

    <\/p>\n

    K\u00ebrkimi i konvolucion\u00ebve p\u00ebr t'u hequr<\/h3>\n

    <\/p>\n

    Na duhet t\u00eb heqim disa mb\u00ebshtjellje. T\u00eb godasim n\u00eb mes dhe t\u00eb \"q\u00ebllojm\u00eb\" \u00e7do gj\u00eb \u00ebsht\u00eb nj\u00eb ide e keqe, ndon\u00ebse do t\u00eb funksionoj\u00eb. Por pasi kemi mendimin, mund t\u00eb mendojm\u00eb dhe t\u00eb p\u00ebrpiqemi t\u00eb identifikojm\u00eb mb\u00ebshtjelljet \"e dob\u00ebt\" p\u00ebr t'i hequr. Ka disa opsione:<\/p>\n

    <\/p>\n

      \n
    1. Masa e vog\u00ebl L1 ose low_magnitude_pruning<\/a><\/noindex>. Ideja \u00ebsht\u00eb se konvolucionet me vlera t\u00eb vogla peshe, kontribuojn\u00eb pak n\u00eb vendimmarrjen p\u00ebrfundimtare. <\/li>\n
    2. Masa m\u00eb e vog\u00ebl L1 duke marr\u00eb parasysh mesataren dhe devijimin standard. Plot\u00ebsojm\u00eb me vler\u00ebsimin e natyr\u00ebs s\u00eb shp\u00ebrndarjes.<\/li>\n
    3. Maskimi i konvolucioneve dhe p\u00ebrjashtimi i atyre q\u00eb ndikojn\u00eb m\u00eb pak n\u00eb sakt\u00ebsin\u00eb p\u00ebrfundimtare<\/a><\/noindex>. P\u00ebrcaktim m\u00eb t\u00eb sakt\u00eb t\u00eb konvolucionve t\u00eb dob\u00ebta, por shum\u00eb k\u00ebrkon koh\u00eb dhe burime. <\/li>\n
    4. T\u00eb tjera <\/li>\n<\/ol>\n

      <\/p>\n

      \u00c7do variant ka t\u00eb drejt\u00eb p\u00ebr jet\u00ebn dhe karakteristikat e tij t\u00eb zbatimit. K\u00ebtu do t\u00eb shqyrtojm\u00eb variantin me mas\u00ebn m\u00eb t\u00eb vog\u00ebl L1.<\/p>\n

      <\/p>\n

      Procesi manual p\u00ebr YOLOv3<\/h3>\n

      <\/p>\n

      Arkitektura origjinale p\u00ebrmban blloqe t\u00eb mbetura. Por sa madh\u00ebshtore ishin ato p\u00ebr rrjetet e thella, ato do na pengonin disi. Problemi \u00ebsht\u00eb se nuk mund t\u00eb fshihen verifikimet me indekse t\u00eb ndryshme n\u00eb k\u00ebto nivele:<\/p>\n

      <\/p>\n

      \"T\u00eb<\/p>\n

      <\/p>\n

      Prandaj, do t\u00eb ndajm\u00eb nivelet nga t\u00eb cilat mund t\u00eb fshijm\u00eb lirsh\u00ebm verifikimet:<\/p>\n

      <\/p>\n

      \"T\u00eb<\/p>\n

      <\/p>\n

      Tani do t\u00eb nd\u00ebrtojm\u00eb nj\u00eb cik\u00ebl pune:<\/p>\n

      <\/p>\n

        \n
      1. Shkarkojm\u00eb aktivizimet<\/li>\n
      2. Vler\u00ebsojm\u00eb sa mund t\u00eb presim <\/li>\n
      3. Presim<\/li>\n
      4. M\u00ebsojm\u00eb p\u00ebr 10 epoka me LR=1e-4 <\/li>\n
      5. Testojm\u00eb <\/li>\n<\/ol>\n

        <\/p>\n

        Shkarkimi i konvolucioneve \u00ebsht\u00eb i dobish\u00ebm p\u00ebr t\u00eb vler\u00ebsuar se sa mund t\u00eb fshihet n\u00eb nj\u00eb hap t\u00eb caktuar. Shembujt e shkarkimit:<\/p>\n

        <\/p>\n

        \"T\u00eb<\/p>\n

        <\/p>\n

        V\u00ebrejm\u00eb se pothuajse kudo 5% e konvolucioneve kan\u00eb nj\u00eb norm\u00eb t\u00eb ul\u00ebt L1 dhe ne mund t'i fshijm\u00eb ato. N\u00eb \u00e7do hap, nj\u00eb shkarkim i till\u00eb \u00ebsht\u00eb p\u00ebrs\u00ebritur dhe \u00ebsht\u00eb b\u00ebr\u00eb vler\u00ebsimi se nga cilat nivele dhe sa mund t\u00eb presim. <\/p>\n

        <\/p>\n

        I gjith\u00eb procesi \u00ebsht\u00eb realizuar n\u00eb 4 hapa (numrat k\u00ebtu dhe kudo jan\u00eb p\u00ebr RTX 2060 Super):<\/p>\n

        <\/p>\n\n\n\n\n\n\n\n\n\n
        Hapi<\/th>\nmAp75<\/th>\nNumri i parametrave, mln<\/th>\nMadh\u00ebsia e rrjetit, mb<\/th>\nNga origjinali, %<\/th>\nKoha e ekzekutimit, ms<\/th>\nKushti i prerjes<\/th>\n<\/tr>\n<\/thead>\n
        0<\/td>\n0.9656<\/td>\n60<\/td>\n241<\/td>\n100<\/td>\n180<\/td>\n\u2014<\/td>\n<\/tr>\n
        1<\/td>\n0.9622<\/td>\n55<\/td>\n218<\/td>\n91<\/td>\n175<\/td>\n5% e t\u00eb gjithave<\/td>\n<\/tr>\n
        2<\/td>\n0.9625<\/td>\n50<\/td>\n197<\/td>\n83<\/td>\n168<\/td>\n5% e t\u00eb gjithave<\/td>\n<\/tr>\n
        3<\/td>\n0.9633<\/td>\n39<\/td>\n155<\/td>\n64<\/td>\n155<\/td>\n15% p\u00ebr nivelet me 400+ konvolucione<\/td>\n<\/tr>\n
        4<\/td>\n0.9555<\/td>\n31<\/td>\n124<\/td>\n51<\/td>\n146<\/td>\n10% p\u00ebr nivelet me 100+ konvolucione<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        <\/p>\n

        N\u00eb hapin 2 u shtua nj\u00eb efekt pozitiv - u ndikua nga madh\u00ebsia e grupit 4, q\u00eb e p\u00ebrshpejtoi procesin e rinovimit.
        \nN\u00eb hapin 4, procesi u ndal, pasi madje edhe st\u00ebrvitja e gjat\u00eb nuk e \u00e7oi mAp75 deri n\u00eb vlerat e m\u00ebparshme.
        \nRezultati ishte p\u00ebrshpejtimi i inferenc\u00ebs me 15%<\/strong>, zvog\u00eblimi i madh\u00ebsis\u00eb me 35% <\/strong>dhe pa humbur sakt\u00ebsin\u00eb. <\/p>\n

        <\/p>\n

        Automatizimi p\u00ebr arkitekturat m\u00eb t\u00eb thjeshta<\/h3>\n

        <\/p>\n

        P\u00ebr arkitekturat m\u00eb t\u00eb thjeshta t\u00eb rrjeteve (pa blloqe kushtore add, concatenate dhe residual), \u00ebsht\u00eb plot\u00ebsisht e mundur t\u00eb orientoheni nga p\u00ebrpunimi i t\u00eb gjitha konvolucioneve dhe t\u00eb automatizoni procesin e prerjes s\u00eb konvolucioneve.<\/p>\n

        <\/p>\n

        Nj\u00eb variant t\u00eb till\u00eb e kam realizuar k\u00ebtu<\/a><\/noindex>.
        \nE gjith\u00eb kjo \u00ebsht\u00eb e thjesht\u00eb: ju nevojitet vet\u00ebm funksioni i humbjes, optimizatori dhe gjenerator\u00ebt e grupit:<\/p>\n

        <\/p>\n

        import pruning\nfrom keras.optimizers import Adam\nfrom keras.utils import Sequence\n\ntrain_batch_generator = BatchGenerator...\nscore_batch_generator = BatchGenerator...\n\nopt = Adam(lr=1e-4)\npruner = pruning.Pruner(\"config.json\", \"categorical_crossentropy\", opt)\n\npruner.prune(train_batch, valid_batch)<\/code><\/pre>\n
        <\/p>\n
        N\u00ebse nevojitet, mund t\u00eb ndryshoni parametrat e konfigurimeve:<\/p>\n
        <\/p>\n
        {\n    \"input_model_path\": \"model.h5\",\n    \"output_model_path\": \"model_pruned.h5\",\n    \"finetuning_epochs\": 10, # numri i epokave p\u00ebr st\u00ebrvitje midis hapave t\u00eb prerjes\n    \"stop_loss\": 0.1, # humbja p\u00ebr t\u00eb ndalur procesin\n    \"pruning_percent_step\": 0.05, # pjesa e konvolucioneve p\u00ebr t'u fshir\u00eb n\u00eb \u00e7do hap prerjeje\n    \"pruning_standart_deviation_part\": 0.2 # shkelje p\u00ebr kufizimin e pjes\u00ebs s\u00eb prer\u00eb\n}<\/code><\/pre>\n
        <\/p>\n
        P\u00ebr m\u00eb tep\u00ebr, \u00ebsht\u00eb realizuar nj\u00eb kufizim mbi baz\u00ebn e devijimit standard. Q\u00ebllimi \u00ebsht\u00eb t\u00eb kufizojm\u00eb pjes\u00ebn e hequr, duke p\u00ebrjashtuar mb\u00ebshtjelljet me masa L1 \"t\u00eb mjaftueshme\":<\/p>\n
        <\/p>\n
        \"T\u00eb<\/p>\n
        <\/p>\n
        Me k\u00ebt\u00eb, lejojm\u00eb q\u00eb t\u00eb fshijm\u00eb vet\u00ebm konvolucionet e dob\u00ebta nga shp\u00ebrndarjet si ato t\u00eb djathta dhe t\u00eb mos ndikojm\u00eb n\u00eb fshirjen nga shp\u00ebrndarjet si ato t\u00eb majta:<\/p>\n
        <\/p>\n
        \"T\u00eb<\/p>\n
        <\/p>\n
        Te afrohesh shp\u00ebrndarja me norm\u00ebn, koeficienti pruning_standart_deviation_part mund t\u00eb p\u00ebrcaktohet nga:<\/p>\n
        <\/p>\n
        \"T\u00eb
        \nUn\u00eb rekomandoj nj\u00eb toleranc\u00eb prej 2 sigma. Ose mund t\u00eb mos orientoheni nga kjo karakteristik\u00eb, duke l\u00ebn\u00eb vler\u00ebn < 1.0.<\/p>\n
        <\/p>\n
        N\u00eb fund, rezulton nj\u00eb grafik i madh\u00ebsis\u00eb s\u00eb rrjetit, humbjeve dhe koh\u00ebs s\u00eb ekzekutimit t\u00eb rrjetit gjat\u00eb gjith\u00eb testit, t\u00eb normalizuar n\u00eb 1.0. P\u00ebr shembull, k\u00ebtu madh\u00ebsia e rrjetit \u00ebsht\u00eb zvog\u00ebluar pothuajse dy her\u00eb pa humbur n\u00eb cil\u00ebsi (nj\u00eb rrjet konvolucioni i vog\u00ebl me 100k pesh\u00eb):<\/p>\n
        <\/p>\n
        \"T\u00eb<\/p>\n
        <\/p>\n
        Shpejt\u00ebsia e ekzekutimit \u00ebsht\u00eb n\u00ebn ndikimin e fluktuacioneve normale dhe praktikisht nuk ka ndryshuar. K\u00ebshtu \u00ebsht\u00eb e arsyeshme:<\/p>\n
        <\/p>\n
          \n
        1. Numri i konvolucioneve ndryshon nga t\u00eb p\u00ebrshtatshmet (32, 64, 128) n\u00eb ato jo shum\u00eb p\u00ebrshtatsh\u00ebm p\u00ebr kartat grafike - 27, 51 etj. K\u00ebtu mund t\u00eb gaboj, por me siguri kjo ndikon.<\/li>\n
        2. Arkitektura nuk \u00ebsht\u00eb e gjer\u00eb, por e vazhdueshme. Duke zvog\u00ebluar gjer\u00ebsin\u00eb, ne nuk prekim thell\u00ebsin\u00eb. K\u00ebshtu, zvog\u00eblojm\u00eb ngarkes\u00ebn, por nuk ndryshojm\u00eb shpejt\u00ebsin\u00eb.<\/li>\n<\/ol>\n
          <\/p>\n
          Prandaj, p\u00ebrmir\u00ebsimi u shpreh n\u00eb zvog\u00eblimin e ngarkes\u00ebs CUDA gjat\u00eb ekzekutimit me 20-30%, por jo n\u00eb zvog\u00eblimin e koh\u00ebs s\u00eb ekzekutimit.<\/p>\n
          <\/p>\n
          P\u00ebrfundimet<\/h3>\n
          <\/p>\n
          Le t\u00eb reflektojm\u00eb. Ne shqyrtuam 2 variante t\u00eb prerjes - p\u00ebr YOLOv3 (kur duhet t\u00eb punojm\u00eb manualisht) dhe p\u00ebr rrjetet me arkitektura m\u00eb t\u00eb thjeshta. Ndjehet se n\u00eb t\u00eb dy rastet, \u00ebsht\u00eb e mundur t\u00eb arrihet zvog\u00eblimi i madh\u00ebsis\u00eb s\u00eb rrjetit dhe p\u00ebrshpejtimi pa humbur sakt\u00ebsin\u00eb. Rezultatet:<\/p>\n
          <\/p>\n