{"id":75722,"date":"2020-03-28T07:42:04","date_gmt":"2020-03-28T05:42:04","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-uskorili-kodirovanie-video-v-vosem-raz"},"modified":"2020-03-28T07:42:04","modified_gmt":"2020-03-28T05:42:04","slug":"kak-my-uskorili-kodirovanie-video-v-vosem-raz","status":"publish","type":"post","link":"https:\/\/prohoster.info\/az\/blog\/administrirovanie\/kak-my-uskorili-kodirovanie-video-v-vosem-raz","title":{"rendered":"Videonun kodla\u015fd\u0131r\u0131lmas\u0131n\u0131 nec\u0259 s\u0259kkiz d\u0259f\u0259 s\u00fcr\u0259tl\u0259ndirdik","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Videonun
\n
\nH\u0259r g\u00fcn milyonlarla izl\u0259yici internetd\u0259 videolar\u0131 izl\u0259yir. Lakin, bir videonun istifad\u0259\u00e7il\u0259r\u0259 t\u0259qdim edilm\u0259si \u00fc\u00e7\u00fcn onu yaln\u0131z server\u0259 y\u00fckl\u0259m\u0259k kifay\u0259t etmir, eyni zamanda emal etm\u0259k d\u0259 laz\u0131md\u0131r. Bu proses n\u0259 q\u0259d\u0259r s\u00fcr\u0259tli ba\u015f verirs\u0259, xidm\u0259t v\u0259 onun istifad\u0259\u00e7il\u0259ri \u00fc\u00e7\u00fcn bir o q\u0259d\u0259r yax\u015f\u0131d\u0131r. <\/p>\n

M\u0259nim ad\u0131m Askar Kamalovdur, bir il \u0259vv\u0259l Yandex-in video texnologiyalar\u0131 komandas\u0131nda i\u015f\u0259 ba\u015flad\u0131m. Bu g\u00fcn sizl\u0259r\u0259 video kodla\u015fd\u0131rma prosesinin paralell\u0259\u015fdirilm\u0259si vasit\u0259sil\u0259 istifad\u0259\u00e7il\u0259r\u0259 video \u00e7atd\u0131rma m\u00fcdd\u0259tini nec\u0259 \u0259h\u0259miyy\u0259tli d\u0259r\u0259c\u0259d\u0259 s\u00fcr\u0259tl\u0259ndirdiyimiz haqq\u0131nda q\u0131sa m\u0259lumat ver\u0259c\u0259y\u0259m. <\/p>\n

Bu post, x\u00fcsus\u0259n d\u0259 daha \u0259vv\u0259l video xidm\u0259tl\u0259rin arxas\u0131nda n\u0259l\u0259rin ba\u015f verdiyin\u0259 d\u00fc\u015f\u00fcnm\u0259y\u0259nl\u0259r \u00fc\u00e7\u00fcn maraql\u0131 olacaq. \u015e\u0259rhl\u0259rd\u0259 suallar veril\u0259 bil\u0259r v\u0259 g\u0259l\u0259c\u0259k postlar \u00fc\u00e7\u00fcn m\u00f6vzular t\u0259klif edil\u0259 bil\u0259r.
\n<\/a><\/noindex>
\nBir ne\u00e7\u0259 s\u00f6z bu m\u0259s\u0259l\u0259nin mahiyy\u0259ti haqq\u0131nda. Yandex yaln\u0131z ba\u015fqa saytlar\u0131n videolar\u0131n\u0131 tapma\u011fa k\u00f6m\u0259k etmir, eyni zamanda \u00f6z xidm\u0259tl\u0259ri \u00fc\u00e7\u00fcn videolar\u0131 saxlay\u0131r. Bu, f\u0259rdi proqramlar, efird\u0259 idman yar\u0131\u015flar\u0131, KinoPoisk-d\u0259 filml\u0259r v\u0259 ya Zen v\u0259 X\u0259b\u0259rl\u0259rd\u0259ki klipl\u0259r olsun \u2014 bunlar\u0131n ham\u0131s\u0131 serverl\u0259rimiz\u0259 y\u00fckl\u0259nir. \u0130stifad\u0259\u00e7il\u0259r videolar\u0131 izl\u0259y\u0259 bilsinl\u0259r, bu videolar\u0131 haz\u0131rlamaq laz\u0131md\u0131r: laz\u0131m olan formata \u00e7evirm\u0259k, \u00f6n g\u00f6r\u00fcnt\u00fc yaratmaq v\u0259 ya h\u0259tta texnologiya vasit\u0259sil\u0259 i\u015fl\u0259tm\u0259k. DeepHD<\/a><\/noindex>. Haz\u0131rlanmam\u0131\u015f fayl sad\u0259c\u0259 yer tutur. Burada yaln\u0131z hardware-in optimal istifad\u0259si deyil, eyni zamanda istifad\u0259\u00e7il\u0259r\u0259 m\u0259zmunun \u00e7atd\u0131r\u0131lma s\u00fcr\u0259ti d\u0259 \u00f6n\u0259mlidir. M\u0259s\u0259l\u0259n: h\u0259qiq\u0259t\u0259n h\u0259lledici bir an\u0131n video \u00e7\u0259kili\u015fi, hadis\u0259d\u0259n bir d\u0259qiq\u0259 sonra axtar\u0131la bil\u0259r.<\/p>\n

Dava kodla\u015fd\u0131rma<\/h3>\n

\nBel\u0259likl\u0259, istifad\u0259\u00e7inin m\u0259mnuniyy\u0259ti, videonun n\u0259 q\u0259d\u0259r tez \u0259l\u00e7atan olmas\u0131ndan as\u0131l\u0131d\u0131r. V\u0259 bu, \u0259sas\u0259n, transkodu s\u00fcr\u0259tind\u0259n as\u0131l\u0131d\u0131r. \u018fg\u0259r video yay\u0131m\u0131n\u0131n s\u00fcr\u0259tind\u0259 ciddi t\u0259l\u0259bl\u0259r yoxdursa, problem d\u0259 yoxdur. Siz birl\u0259\u015fdirilmi\u015f, b\u00f6l\u00fcnm\u0259z bir fayl al\u0131rs\u0131n\u0131z, onu \u00e7evirdikd\u0259n sonra yerl\u0259\u015fdirirsiniz. Yolun \u0259vv\u0259lind\u0259 biz d\u0259 bel\u0259 i\u015fl\u0259yirdik:<\/p>\n

\"Videonun
\n
\nM\u00fc\u015ft\u0259ri videonu saxlanmaya y\u00fckl\u0259yir, Analyzer komponenti meta m\u0259lumatlar\u0131 toplay\u0131r v\u0259 videonu konversiya \u00fc\u00e7\u00fcn Worker komponentin\u0259 \u00f6t\u00fcr\u00fcr. B\u00fct\u00fcn m\u0259rh\u0259l\u0259l\u0259r s\u0131rayla icra olunur. Bu zaman kodla\u015fd\u0131rma \u00fc\u00e7\u00fcn bir \u00e7ox server ola bil\u0259r, lakin yaln\u0131z biri konkret videonun emal\u0131 il\u0259 m\u0259\u015f\u011fuldur. Sad\u0259, \u015f\u0259ffaf sxema. Bunun \u00fcst\u00fcnl\u00fckl\u0259ri burada ba\u015fa \u00e7at\u0131r. Bel\u0259 bir sxema yaln\u0131z \u015faquli \u015f\u0259kild\u0259 geni\u015fl\u0259nir (daha g\u00fccl\u00fc serverl\u0259rin al\u0131nmas\u0131 yolu il\u0259). <\/p>\n

Aral\u0131ql\u0131 n\u0259tic\u0259 il\u0259 ard\u0131c\u0131l kodla\u015fd\u0131rma<\/h3>\n

\nM\u00fcxatir\u0259t g\u00f6zl\u0259m\u0259sini bir az yumaq \u00fc\u00e7\u00fcn, m\u00fc\u0259ssis\u0259l\u0259r s\u00fcr\u0259tli kodla\u015fd\u0131rma variant\u0131 t\u0259rtib etdil\u0259r. Aldad\u0131c\u0131 ad, \u00e7\u00fcnki \u0259slind\u0259 tam kodla\u015fd\u0131rma ard\u0131c\u0131ld\u0131r v\u0259 eyni d\u0259r\u0259c\u0259d\u0259 uzun s\u00fcr\u0259r. Amma n\u0259tic\u0259d\u0259 aral\u0131q bir versiya meydana g\u0259lir. M\u0259qs\u0259d budur: m\u00fcmk\u00fcn q\u0259d\u0259r s\u00fcr\u0259tl\u0259 a\u015fa\u011f\u0131 t\u0259svirli video versiyas\u0131n\u0131 haz\u0131rlay\u0131b yerl\u0259\u015fdirm\u0259k, sonra is\u0259 daha y\u00fcks\u0259k t\u0259svirli versiyalar\u0131. <\/p>\n

Bir t\u0259r\u0259fd\u0259n, video daha tez bir zamanda \u0259ld\u0259 edilir. V\u0259 bu, m\u00fch\u00fcm hadis\u0259l\u0259r \u00fc\u00e7\u00fcn faydal\u0131d\u0131r. Lakin dig\u0259r t\u0259r\u0259fd\u0259n, g\u00f6r\u00fcnt\u00fc bulan\u0131q olur, bu da izl\u0259yicil\u0259ri rahats\u0131z edir.<\/p>\n

Bel\u0259likl\u0259, videonu yaln\u0131z s\u00fcr\u0259tl\u0259 emal etm\u0259k deyil, h\u0259m d\u0259 keyfiyy\u0259tini qorumaq da laz\u0131m. Bu, m\u00fcasir videoxidm\u0259t istifad\u0259\u00e7il\u0259rinin g\u00f6zl\u0259diyi bir \u015feydir. G\u00f6r\u00fcn\u0259 bil\u0259r ki, \u0259n g\u00fccl\u00fc serverl\u0259ri sat\u0131n almaq kifay\u0259tdir (v\u0259 onlar\u0131 m\u00fcnt\u0259z\u0259m olaraq yenil\u0259m\u0259k). Amma bu, bir \u00e7\u0131x\u0131lmaz yoldur, \u00e7\u00fcnki h\u0259mi\u015f\u0259 \u0259n g\u00fccl\u00fc avadanl\u0131qda bel\u0259 s\u00fcr\u0259tl\u0259nm\u0259y\u0259 s\u0259b\u0259b olacaq bir video olacaq. <\/p>\n

Paralel kodla\u015fd\u0131rma<\/h3>\n

\n\u00c7\u0259tin bir tap\u015f\u0131r\u0131\u011f\u0131 daha \u00e7ox az \u00e7\u0259tin tap\u015f\u0131r\u0131qlara b\u00f6lm\u0259k v\u0259 onlar\u0131 m\u00fcxt\u0259lif serverl\u0259rd\u0259 paralel olaraq h\u0259ll etm\u0259k daha s\u0259m\u0259r\u0259lidir. Bu, videolar \u00fc\u00e7\u00fcn MapReduce kimidir. Bu v\u0259ziyy\u0259td\u0259 bir serverin m\u0259hsuldarl\u0131\u011f\u0131na ba\u011fl\u0131 qalm\u0131r\u0131q v\u0259 yeni ma\u015f\u0131nlar \u0259lav\u0259 ed\u0259r\u0259k horizontal olaraq \u00f6l\u00e7\u0259 bil\u0259rik. <\/p>\n

Qeyd etm\u0259k laz\u0131md\u0131r ki, videonu ki\u00e7ik par\u00e7alara ay\u0131rmaq, paralel emal etm\u0259k v\u0259 sonra birl\u0259\u015fdirm\u0259k ideyas\u0131 h\u0259r hans\u0131 bir sirr deyil. Bu yana\u015fman\u0131 tapmaq \u00fc\u00e7\u00fcn \u00e7oxlu istinadlar tapa bil\u0259rsiniz (m\u0259s\u0259l\u0259n, Habr-da bir postu t\u00f6vsiy\u0259 edir\u0259m DistVIDc<\/a><\/noindex>). Lakin bu, b\u00fct\u00f6vl\u00fckd\u0259 asanla\u015fm\u0131r, \u00e7\u00fcnki art\u0131q haz\u0131r bir h\u0259lli g\u00f6t\u00fcr\u00fcb \u00f6z sisteminiz\u0259 daxil etm\u0259k olmur. Bizim infrastrukturumuza, videomuza v\u0259 h\u0259tta y\u00fck\u00fcm\u00fcz\u0259 uy\u011funla\u015fd\u0131rma laz\u0131md\u0131r. \u00dcmumilikd\u0259, \u00f6z\u00fcn\u00fc yazmaq daha asand\u0131r. <\/p>\n

Bel\u0259likl\u0259, yeni arxitektura daxilind\u0259 ard\u0131c\u0131l kodla\u015fd\u0131rma il\u0259 monolit Worker bloku, Segmenter, Tcoder, Combiner mikroservisl\u0259rin\u0259 b\u00f6l\u00fcnm\u00fc\u015fd\u00fcr.<\/p>\n

\"Videonun
\n<\/p>\n

    \n
  1. Segmenter, videonu t\u0259xmin\u0259n 10 saniy\u0259lik fraqmentl\u0259r\u0259 ay\u0131r\u0131r. Fraqmentl\u0259r bir v\u0259 ya bir ne\u00e7\u0259 GOP-dan ibar\u0259tdir (group of pictures<\/a><\/noindex>). H\u0259r GOP m\u00fcst\u0259qil \u015f\u0259kild\u0259 kodlan\u0131r v\u0259 ba\u015fqalar\u0131na ba\u011fl\u0131 olmadan de\u015fifr\u0259 olunur, y\u0259ni par\u00e7alar bir-birind\u0259n m\u00fcst\u0259qil \u015f\u0259kild\u0259 yenid\u0259n qurula bil\u0259r. Bu c\u00fcr seqmentl\u0259\u015fdirm\u0259 gecikm\u0259ni azald\u0131r, emal\u0131 daha tez ba\u015flama\u011fa imkan verir.<\/li>\n
  2. Tcoder h\u0259r par\u00e7an\u0131 emal edir. O, qeydiyyat s\u0131ras\u0131n\u0131 al\u0131r, par\u00e7as\u0131n\u0131 anbarlardan y\u00fckl\u0259yir, m\u00fcxt\u0259lif d\u0259qiqlikl\u0259rd\u0259 kodlay\u0131r (xat\u0131rladaq ki, pleyer ba\u011flant\u0131 s\u00fcr\u0259tin\u0259 \u0259saslanaraq versiyan\u0131 se\u00e7\u0259 bil\u0259r), daha sonra n\u0259tic\u0259ni geri anbara y\u0131\u011f\u0131r v\u0259 par\u00e7as\u0131n\u0131 m\u0259lumat bazas\u0131nda emal olunmu\u015f kimi i\u015far\u0259l\u0259yir. B\u00fct\u00fcn par\u00e7alar\u0131 emal etdikd\u0259n sonra, Tcoder n\u00f6vb\u0259ti komponent \u00fc\u00e7\u00fcn n\u0259tic\u0259l\u0259rin yarad\u0131lmas\u0131 tap\u015f\u0131r\u0131\u011f\u0131n\u0131 g\u00f6nd\u0259rir.<\/li>\n
  3. Combiner n\u0259tic\u0259l\u0259ri bir araya g\u0259tirir: Tcoder t\u0259r\u0259find\u0259n haz\u0131rlanm\u0131\u015f b\u00fct\u00fcn par\u00e7alar\u0131 y\u00fckl\u0259yir v\u0259 m\u00fcxt\u0259lif d\u0259qiqlikl\u0259r \u00fc\u00e7\u00fcn ax\u0131nlar formala\u015fd\u0131r\u0131r.<\/li>\n<\/ol>\n

    \nS\u0259s haqq\u0131nda bir ne\u00e7\u0259 s\u00f6z. \u018fn populyar audio kodekl\u0259rd\u0259n biri olan AAC-n\u0131n m\u0259d\u0259ni bir x\u00fcsusiyy\u0259ti var. \u018fg\u0259r par\u00e7alar\u0131 ayr\u0131-ayr\u0131l\u0131qda kodlayarsan, onlar\u0131 seamless birl\u0259\u015fdirm\u0259k m\u00fcmk\u00fcn olmayacaq. Ke\u00e7idl\u0259r diqq\u0259t \u00e7\u0259k\u0259c\u0259k. Video kodekl\u0259rind\u0259 bel\u0259 bir problem yoxdur. Teorik c\u0259h\u0259td\u0259n m\u00fcr\u0259kk\u0259b texniki h\u0259ll axtara bil\u0259rik, amma bu oyun sad\u0259c\u0259 buna d\u0259y\u0259r (audio video il\u0259 m\u00fcqayis\u0259d\u0259 \u0259h\u0259miyy\u0259tind\u0259 daha az yer tutur). Bu s\u0259b\u0259bd\u0259n, videonun kodlanmas\u0131 paralel olaraq yerin\u0259 yetirilir, audio is\u0259 tamamil\u0259 emal edilir.<\/p>\n

    N\u0259tic\u0259l\u0259r<\/h3>\n

    \nVideonun paralel emal\u0131 say\u0259sind\u0259, videonu sistem\u0259 y\u00fckl\u0259dikd\u0259n sonra istifad\u0259\u00e7il\u0259r \u00fc\u00e7\u00fcn \u0259l\u00e7atan olmaq aras\u0131nda gecikm\u0259ni \u0259h\u0259miyy\u0259tli d\u0259r\u0259c\u0259d\u0259 azald\u0131q. M\u0259s\u0259l\u0259n, daha \u0259vv\u0259l bir bu\u00e7uk saatl\u0131q FullHD film \u00fc\u00e7\u00fcn m\u00fcxt\u0259lif keyfiyy\u0259tl\u0259ri olan bir ne\u00e7\u0259 tam versiyan\u0131n yarad\u0131lmas\u0131 iki saat \u00e7\u0259k\u0259 bil\u0259rdi. \u0130ndi bu i\u015f 15 d\u0259qiq\u0259 \u00e7\u0259kir. Daha da ir\u0259lil\u0259y\u0259r\u0259k, paralel emal zaman\u0131 daha y\u00fcks\u0259k d\u0259qiqlikli versiyan\u0131, k\u00f6hn\u0259 yana\u015fma il\u0259 a\u015fa\u011f\u0131 d\u0259qiqliqli versiyadan daha tez yarad\u0131r\u0131q. <\/p>\n

    V\u0259 h\u0259l\u0259 bir \u015fey. K\u00f6hn\u0259 yana\u015fmada ya serverl\u0259r \u00e7atmad\u0131, ya da onlar taps\u0131z i\u015fl\u0259rl\u0259 bo\u015f qald\u0131. Paralel kodlama, avadanl\u0131qlar\u0131n istifad\u0259 nisb\u0259tini art\u0131rma\u011fa imkan verir. \u0130ndi minl\u0259rl\u0259 serverd\u0259n ibar\u0259t klasterimiz h\u0259mi\u015f\u0259 n\u0259s\u0259 il\u0259 m\u0259\u015f\u011fuldur. <\/p>\n

    H\u0259qiq\u0259t\u0259n d\u0259 h\u0259l\u0259 d\u0259 ir\u0259lil\u0259y\u0259 bil\u0259c\u0259yimiz yerl\u0259r var. M\u0259s\u0259l\u0259n, video biz\u0259 tamamil\u0259 g\u0259lm\u0259mi\u015fd\u0259n \u0259vv\u0259l par\u00e7alar\u0131 emal etm\u0259y\u0259 ba\u015flasaq, zaman\u0131 \u0259h\u0259miyy\u0259tli d\u0259r\u0259c\u0259d\u0259 qazana bil\u0259rik. Dedikl\u0259ri kimi, daha ir\u0259lid\u0259 daha \u00e7ox \u015fey var. <\/p>\n

    \u015e\u0259rhl\u0259rd\u0259 video i\u015fl\u0259ri sah\u0259sind\u0259 hans\u0131 m\u0259s\u0259l\u0259l\u0259r haqq\u0131nda oxumaq ist\u0259diyinizi yaz\u0131n. <\/p>\n

    S\u0259naye m\u00fct\u0259x\u0259ssisl\u0259rinin t\u0259cr\u00fcb\u0259sin\u0259 faydal\u0131 linkl\u0259r<\/h3>\n

    <\/p>\n