{"id":39133,"date":"2019-10-31T22:28:01","date_gmt":"2019-10-31T19:28:01","guid":{"rendered":"https:\/\/prohoster.info\/blog\/monitoring-nagruzochnoe-testirovanie-prognozirovanie-i-otsutstvie-sboev\/"},"modified":"2019-10-31T22:28:01","modified_gmt":"2019-10-31T19:28:01","slug":"monitoring-nagruzochnoe-testirovanie-prognozirovanie-i-otsutstvie-sboev","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/monitoring-nagruzochnoe-testirovanie-prognozirovanie-i-otsutstvie-sboev","title":{"rendered":"Monitoring + load testing = forecasting and preventing failures","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

The IT department of VTB has faced emergency situations in system operations several times when the load on them increased significantly. Therefore, a need arose to develop and test a model that would predict peak loads on critical systems. To achieve this, the bank's IT specialists set up monitoring, analyzed data, and learned to automate forecasts. We will discuss which tools helped predict load and whether they were successful in optimizing operations in a brief article.<\/p>\n

\"Monitoring
\n<\/a><\/noindex>
\nProblems with high-load services occur across almost all industries, but they are critical for the financial sector. At the decisive moment, all combat units must be ready, which means it was essential to know in advance what might happen and even to determine the day when load would spike and which systems would face it. Failures must be fought against and prevented, so the necessity of implementing a predictive analytics system was never in question. It was essential to modernize systems based on monitoring data.<\/p>\n

Knee-jerk analytics<\/h2>\n

\nThe payroll project is one of the most sensitive in case of failure. It is also the most understandable for forecasting, so they decided to start with it. Due to the high interconnectivity during peak load moments, other subsystems, including remote banking services (RBS), could also experience problems. For instance, clients, delighted by SMS notifications about incoming funds, began to use them actively. During this time, the load could spike by more than an order of magnitude.\u00a0<\/p>\n

Prvo modela za predikciju smo izradili ru\u010dno. Uzeli smo izvoz podataka za poslednju godinu i izra\u010dunali u koje dane se o\u010dekuju maksimalni vrhovi: na primer, 1., 15. i 25. u mesecu, kao i na kraju meseca. Ovaj model je zahtevao ozbiljne ljudske resurse i nije pru\u017eao ta\u010dnu prognozu. Ipak, otkrio je uska mesta gde je bilo potrebno dodati dodatni \u201ehardver\u201c i omogu\u0107io optimizaciju procesa ispla\u0107ivanja novca dogovorom sa klju\u010dnim klijentima: kako ne bismo ispla\u0107ivali plate \u201eu jednom potezu\u201c, transakcije iz razli\u010ditih regiona su raspore\u0111ene po vremenu. Sada ih obra\u0111ujemo u delovima koje IT infrastruktura banke mo\u017ee \u201esvariti\u201c bez prekida.<\/p>\n

Nakon postizanja prvog pozitivnog rezultata, pre\u0161li smo na automatizaciju prognoziranja. \u010cekalo je jo\u0161 desetak kriti\u010dnih ta\u010daka.<\/p>\n

Kompleksan pristup<\/h2>\n

\nU VTB-u je implementiran sistem nadzora kompanije MicroFocus. Odande smo uzeli prikupljanje podataka za prognoziranje, sistem skladi\u0161tenja i sistem generisanja izve\u0161taja. U su\u0161tini, monitoring je ve\u0107 postojao, samo je trebalo dodati metrike, modul za predikciju i kreirati nove izve\u0161taje. Ovo re\u0161enje podr\u017eava spolja\u0161nji izvo\u0111a\u010d \u201eTehnoserv\u201c, tako da su glavni radovi na implementaciji projekta pasti na njegove stru\u010dnjake, ali smo model izgradili samostalno. Sistem prognoziranja napravljen je na osnovu Prophet - ovaj otvoreni proizvod razvila je Facebook. Lako se koristi i lako se integri\u0161e sa postoje\u0107im sredstvima za sveobuhvatan monitoring i Vertica. Drugim re\u010dima, sistem analizira grafikon optere\u0107enja i na osnovu Furijevih nizova pravi njegov ekstrapolaciju. Tako\u0111e postoji mogu\u0107nost dodavanja odre\u0111enih koeficijenata po danima, uzetih iz na\u0161eg modela. Metrike se skupljaju bez ljudske intervencije, prognoza se automatski prera\u010dunava jednom nedeljno, a novi izve\u0161taji se \u0161alju zainteresovanim stranama.\u00a0<\/p>\n

Ta metoda razkriva glavne cikli\u010dnosti, kot so letni, mese\u010dni, \u010detrtletni in tedenski. Izpla\u010dila pla\u010d in akontacij, obdobja dopustov, prazniki in razprodaje - vse to vpliva na \u0161tevilo poizvedb v sistemih. Izkazalo se je, da se nekateri cikli prekrivajo, medtem ko glavnino obremenitve (75 %) na sisteme predstavlja Centralni federativni okrug. Pravni in fizi\u010dni subjekti se obna\u0161ajo razli\u010dno. Medtem ko je obremenitev s strani \u00bbfizi\u010dnih\u00ab oseb razmeroma enakomerno razporejena po dnevih v tednu (gre za zelo veliko \u0161tevilo manj\u0161ih transakcij), pa pri podjetjih 99,9 % obremenitve nastane v delovnem \u010dasu, pri \u010demer so lahko transakcije kratke, lahko pa se obdelujejo tudi ve\u010d minut ali celo ure.<\/p>\n

\"Monitoring
\n
\nNa podlagi pridobljenih podatkov so dolo\u010deni dolgotrajni trendi. Nov sistem je pokazal, da ljudje mno\u017ei\u010dno prehajajo na DBO. To je znano, vendar nismo pri\u010dakovali tak\u0161nih obse\u017enosti in sprva temu nismo verjeli: \u0161tevilo poizvedb v bankovnih uradih se izjemno hitro zmanj\u0161uje, hkrati pa nara\u0161\u010da \u0161tevilo daljinskih transakcij. Tako se tudi obremenitev sistemov pove\u010duje in se bo \u0161e naprej pove\u010devala. Trenutno napovedujemo obremenitev do februarja 2020. Normalne dni lahko napovemo s 3 % napako, vr\u0161ne pa z 10 % napako. To je dober rezultat.<\/p>\n

Skrbi<\/h2>\n

\nBrez te\u017eav, kakor obi\u010dajno, ni \u0161lo. Mehanizem ekstrapolacije z uporabo Fourierovih vrst slabo prehaja skozi ni\u010d - vemo, da ob koncu tedna pravne osebe generirajo malo transakcij, vendar napovedni modul daje vrednosti dale\u010d od ni\u010d. Lahko bi jih prisilno popravili, vendar to ni na\u0161 na\u010din. Poleg tega smo morali re\u0161iti te\u017eavo brezglavega pridobivanja podatkov iz sistemov-izvorov. Redno zbiranje informacij zahteva resne ra\u010dunalni\u0161ke vire, zato smo zgradili hitre predpomnilnike s pomo\u010djo replikacije, poslovne podatke pa pridobivamo \u017ee iz replik. Odsotnost dodatne obremenitve na glavnih sistemih v takih primerih je blokirajo\u010de zahtevana.<\/p>\n

Novi izzivi<\/h2>\n

\nZadati cilj predvi\u0111anja vrhunca je re\u0161en: od maja ove godine nije bilo povezanih prekida usled preoptere\u0107enja u banci, a novi sistem predvi\u0111anja je odigrao zna\u010dajnu ulogu u tome. Da, ispostavilo se da to nije bilo dovoljno, a sada banka \u017eeli da razume koliko su ti vrhunci opasni. Potrebne su nam prognoze koriste\u0107i metrike iz optere\u0107enja testiranja, a to ve\u0107 funkcioni\u0161e za oko 30% kriti\u010dnih sistema, dok su ostali u procesu obezbe\u0111ivanja prognoza. U slede\u0107oj fazi planiramo da predvi\u0111amo optere\u0107enje na sistemima ne u poslovnim transakcijama, ve\u0107 u terminima IT infrastrukture, tj. spusti\u0107emo se na ni\u017ei nivo. Pored toga, potrebno je potpuno automatizovati prikupljanje metrika i njihovo kori\u0161\u0107enje za izradu prognoza, kako ne bismo morali da se bavimo ekportovanjem. U tome nema ni\u0161ta izvanredno - jednostavno spajamo monitoring i testiranje optere\u0107enja u skladu sa najboljim svetskim praksama.
\n
Izvor: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

IT-\u0434\u0435\u043f\u0430\u0440\u0442\u0430\u043c\u0435\u043d\u0442\u0443 \u0412\u0422\u0411 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0440\u0430\u0437 \u043f\u0440\u0438\u0445\u043e\u0434\u0438\u043b\u043e\u0441\u044c \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u0442\u044c\u0441\u044f \u0441 \u043d\u0435\u0448\u0442\u0430\u0442\u043d\u044b\u043c\u0438 \u0441\u0438\u0442\u0443\u0430\u0446\u0438\u044f\u043c\u0438 \u0432 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441\u0438\u0441\u0442\u0435\u043c, \u043a\u043e\u0433\u0434\u0430 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u043d\u0430 \u043d\u0438\u0445 \u043c\u043d\u043e\u0433\u043e\u043a\u0440\u0430\u0442\u043d\u043e \u0432\u043e\u0437\u0440\u0430\u0441\u0442\u0430\u043b\u0430. \u041f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u043e\u044f\u0432\u0438\u043b\u0430\u0441\u044c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e\u0441\u0442\u044c \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438 \u043e\u043f\u0440\u043e\u0431\u043e\u0432\u0430\u0442\u044c \u043c\u043e\u0434\u0435\u043b\u044c, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u0440\u0435\u0434\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u043b\u0430 \u0431\u044b \u043f\u0438\u043a\u043e\u0432\u0443\u044e \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0443 \u043d\u0430 \u043a\u0440\u0438\u0442\u0438\u0447\u043d\u044b\u0435 \u0441\u0438\u0441\u0442\u0435\u043c\u044b. \u0414\u043b\u044f \u044d\u0442\u043e\u0433\u043e IT-\u0441\u043f\u0435\u0446\u0438\u0430\u043b\u0438\u0441\u0442\u044b \u0431\u0430\u043d\u043a\u0430 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433, \u043f\u0440\u043e\u0430\u043d\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043b\u0438 \u0434\u0430\u043d\u043d\u044b\u0435 \u0438 \u043d\u0430\u0443\u0447\u0438\u043b\u0438\u0441\u044c \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043f\u0440\u043e\u0433\u043d\u043e\u0437\u044b. \u041a\u0430\u043a\u0438\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u043f\u043e\u043c\u043e\u0433\u043b\u0438 \u0441\u043f\u0440\u043e\u0433\u043d\u043e\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0443 \u0438 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c \u043b\u0438 \u0441 \u0438\u0445 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":29368,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-39133","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Monitoring + testiranje optere\u0107enja = prognoziranje i izostanak prekida | ProHoster","description":"IT odeljenju VTB-a nekoliko puta je bilo potrebno da se suo\u010di sa vanrednim situacijama u radu sistema, kada je optere\u0107enje na njima vi\u0161estruko poraslo.","canonical_url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/monitoring-nagruzochnoe-testirovanie-prognozirovanie-i-otsutstvie-sboev","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sl_SI","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 + \u043d\u0430\u0433\u0440\u0443\u0437\u043e\u0447\u043d\u043e\u0435 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 = \u043f\u0440\u043e\u0433\u043d\u043e\u0437\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0438 \u043e\u0442\u0441\u0443\u0442\u0441\u0442\u0432\u0438\u0435 \u0441\u0431\u043e\u0435\u0432 | ProHoster","og:description":"IT-\u0434\u0435\u043f\u0430\u0440\u0442\u0430\u043c\u0435\u043d\u0442\u0443 \u0412\u0422\u0411 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0440\u0430\u0437 \u043f\u0440\u0438\u0445\u043e\u0434\u0438\u043b\u043e\u0441\u044c \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u0442\u044c\u0441\u044f \u0441 \u043d\u0435\u0448\u0442\u0430\u0442\u043d\u044b\u043c\u0438 \u0441\u0438\u0442\u0443\u0430\u0446\u0438\u044f\u043c\u0438 \u0432 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441\u0438\u0441\u0442\u0435\u043c, \u043a\u043e\u0433\u0434\u0430 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0430 \u043d\u0430 \u043d\u0438\u0445 \u043c\u043d\u043e\u0433\u043e\u043a\u0440\u0430\u0442\u043d\u043e \u0432\u043e\u0437\u0440\u0430\u0441\u0442\u0430\u043b\u0430.","og:url":"https:\/\/prohoster.info\/sl\/blog\/administrirovanie\/monitoring-nagruzochnoe-testirovanie-prognozirovanie-i-otsutstvie-sboev","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:28:01+00:00","article:modified_time":"2019-10-31T19:28:01+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"39133","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 00:53:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:55:36","updated":"2026-01-24 00:53:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/39133","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/comments?post=39133"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/posts\/39133\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media\/29368"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/media?parent=39133"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/categories?post=39133"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sl\/wp-json\/wp\/v2\/tags?post=39133"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}