{"id":70862,"date":"2020-02-22T06:40:55","date_gmt":"2020-02-22T03:40:55","guid":{"rendered":"https:\/\/prohoster.info\/blog\/sber-ds-platforma-kotoraya-pozvolyaet-sozdavat-i-vnedryat-modeli-dazhe-bez-koda"},"modified":"2020-03-03T16:14:35","modified_gmt":"2020-03-03T13:14:35","slug":"sber-ds-platforma-kotoraya-pozvolyaet-sozdavat-i-vnedryat-modeli-dazhe-bez-koda","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/sber-ds-platforma-kotoraya-pozvolyaet-sozdavat-i-vnedryat-modeli-dazhe-bez-koda","title":{"rendered":"Sber.DS \u2013 o platform\u0103 care permite crearea \u0219i implementarea de modele chiar \u0219i f\u0103r\u0103 cod","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Idei \u0219i \u00eent\u00e2lniri despre procesele care pot fi automatizate apar zilnic \u00een afaceri de diferite dimensiuni. Dar, pe l\u00e2ng\u0103 timpul necesar pentru a crea un model, trebuie s\u0103 se aloce timp pentru evaluarea \u0219i verificarea faptului c\u0103 rezultatul ob\u021binut nu este \u00eent\u00e2mpl\u0103tor. Dup\u0103 implementare, orice model trebuie monitorizat \u0219i verificat periodic.<\/p>\n

<\/p>\n

Acestea sunt etape care trebuie parcurse \u00een orice companie, indiferent de dimensiunea acesteia. C\u00e2nd ne referim la amploarea \u0219i structurile legate de Sberbank, num\u0103rul ajust\u0103rilor fine cre\u0219te considerabil. P\u00e2n\u0103 la sf\u00e2r\u0219itul anului 2019, Sberbank folosea deja peste 2000 de modele. Nu este suficient s\u0103 dezvol\u021bi un model; este necesar\u0103 integrarea cu sistemele industriale, dezvoltarea vitrinelor de date pentru construirea modelelor \u0219i asigurarea controlului func\u021bion\u0103rii acestuia pe cluster.<\/p>\n

\"Sber.DS<\/a><\/noindex><\/p>\n

Echipa noastr\u0103 dezvolt\u0103 platforma Sber.DS. Aceasta permite rezolvarea sarcinilor de \u00eenv\u0103\u021bare automat\u0103, accelereaz\u0103 procesul de verificare a ipotezelor, simplific\u0103, \u00een general, procesul de dezvoltare \u0219i validare a modelelor, precum \u0219i controleaz\u0103 rezultatul muncii modelului \u00een PROM.<\/p>\n

<\/p>\n

Pentru a nu v\u0103 \u00een\u0219ela a\u0219tept\u0103rile, vreau s\u0103 v\u0103 spun din timp c\u0103 aceast\u0103 postare este introducerea \u0219i, mai jos, discut\u0103m despre ceea ce se afl\u0103, \u00een principiu, sub capot\u0103 \u00een platforma Sber.DS. Povestea ciclului de via\u021b\u0103 al modelului, de la crea\u021bie la implementare, o vom relata separat. <\/p>\n

\n<\/a><\/noindex><\/p>\n

Sber.DS este format din mai multe componente, cele cheie fiind biblioteca, sistemul de dezvoltare \u0219i sistemul de execu\u021bie a modelului. <\/p>\n

\n\"Sber.DS
\n<\/p>\n

Biblioteca controleaz\u0103 ciclul de via\u021b\u0103 al modelului de la momentul \u00een care apare ideea dezvolt\u0103rii sale p\u00e2n\u0103 la implementarea \u00een PROM, monitorizarea \u0219i scoaterea din uz. Multe dintre func\u021bionalit\u0103\u021bile bibliotecii sunt dictate de regulile autorit\u0103\u021bilor de reglementare, de exemplu, raportarea \u0219i stocarea seturilor de date de antrenament \u0219i validare. De fapt, aceasta este un registru al tuturor modelelor noastre.<\/p>\n

<\/p>\n

Sistemul de dezvoltare este destinat dezvolt\u0103rii vizuale a modelelor \u0219i metodologiilor de validare. Modelele create trec printr-o validare ini\u021bial\u0103 \u0219i sunt livrate sistemului de execu\u021bie pentru \u00eendeplinirea func\u021biilor lor de afaceri. De asemenea, \u00een sistemul de execu\u021bie, modelul poate fi plasat pe monitor cu scopul de a rula periodic metodologiile de validare pentru a controla func\u021bionarea acestuia. <\/p>\n

<\/p>\n

Sistemul include mai multe tipuri de noduri. Unele sunt destinate conect\u0103rii la diverse surse de date, altele - transform\u0103rii datelor ini\u021biale \u0219i \u00eembog\u0103\u021birii acestora (etichete). Exist\u0103 multe noduri pentru construirea diverselor modele \u0219i pentru validarea acestora. Dezvoltatorul poate \u00eenc\u0103rca date din orice surs\u0103, le poate transforma, filtra, vizualiza datele intermediare \u0219i le poate \u00eemp\u0103r\u021bi \u00een p\u0103r\u021bi.<\/p>\n

<\/p>\n

De asemenea, platforma con\u021bine module deja preg\u0103tite, care pot fi trasate \u00een zona de proiectare. Toate ac\u021biunile se desf\u0103\u0219oar\u0103 printr-o interfa\u021b\u0103 vizualizat\u0103. Practic, se poate rezolva o sarcin\u0103 f\u0103r\u0103 a scrie o singur\u0103 linie de cod.<\/p>\n

<\/p>\n

Dac\u0103 func\u021bionalit\u0103\u021bile \u00eencorporate nu sunt suficiente, sistemul ofer\u0103 posibilitatea de a crea rapid propriile module. Am implementat un mod de dezvoltare integrat\u0103 pe baza Jupyter Kernel Gateway<\/a><\/noindex> pentru cei care creeaz\u0103 module noi \"de la zero\".<\/p>\n

\n\"Sber.DS
\n<\/p>\n

Arhitectura Sber.DS este construit\u0103 pe microservicii. Exist\u0103 multe p\u0103reri despre ceea ce \u00eenseamn\u0103 microservicii. Unii consider\u0103 c\u0103 este suficient s\u0103 \u00eemparti codul monolitic \u00een p\u0103r\u021bi, dar acestea continu\u0103 s\u0103 acceseze aceea\u0219i baz\u0103 de date. La noi, un microserviciu trebuie s\u0103 comunice cu alt microserviciu doar prin REST API. Niciun acces direct la baza de date.<\/p>\n

<\/p>\n

Ne str\u0103duim ca serviciile s\u0103 nu devin\u0103 foarte mari \u0219i lente: un exemplu nu ar trebui s\u0103 consume mai mult de 4-8 gigaocte\u021bi de memorie RAM \u0219i trebuie s\u0103 asigure posibilitatea scal\u0103rii orizontale a cererilor prin lansarea de noi instan\u021be. Fiecare serviciu comunic\u0103 cu celelalte doar prin REST API (Open API<\/a><\/noindex>). Echipa responsabil\u0103 pentru serviciu trebuie s\u0103 men\u021bin\u0103 compatibilitatea invers\u0103 a API-ului p\u00e2n\u0103 la ultimul client care \u00eel utilizeaz\u0103.<\/p>\n

<\/p>\n

Nucleul aplica\u021biei este scris \u00een Java folosind Spring Framework. Solu\u021bia a fost proiectat\u0103 ini\u021bial pentru o desf\u0103\u0219urare rapid\u0103 \u00een infrastructura cloud, de aceea aplica\u021bia a fost construit\u0103 folosind un sistem de containerizare Red Hat OpenShift<\/a><\/noindex> (Kubernetes<\/a><\/noindex>). Platforma se dezvolt\u0103 constant, at\u00e2t \u00een ceea ce prive\u0219te extinderea func\u021bionalit\u0103\u021bii de afaceri (se adaug\u0103 noi conectori, AutoML), c\u00e2t \u0219i \u00een ceea ce prive\u0219te eficien\u021ba tehnologic\u0103. <\/p>\n

<\/p>\n

Una dintre \u00abcaracteristicile\u00bb platformei noastre const\u0103 \u00een faptul c\u0103 putem rula cod dezvoltat \u00een interfa\u021ba vizual\u0103 pe orice sistem de execu\u021bie a modelului Sberbank. Acum exist\u0103 deja dou\u0103: una pe Hadoop \u0219i cealalt\u0103 pe OpenShift (Docker). Nu ne oprim aici \u0219i cre\u0103m module de integrare pentru a rula cod pe orice infrastructur\u0103, inclusiv on-premise \u0219i \u00een cloud. \u00cen ceea ce prive\u0219te capacit\u0103\u021bile de integrare eficient\u0103 \u00een ecosistemul Sberbank, avem de asemenea planuri de a sus\u021bine lucrul cu mediile de execu\u021bie existente. \u00cen viitor, solu\u021bia poate fi integrat\u0103 flexibil \u201edin cutie\u201d \u00een orice peisaj al oric\u0103rei organiza\u021bii.<\/p>\n

<\/p>\n

Cei care au \u00eencercat vreodat\u0103 s\u0103 men\u021bin\u0103 o solu\u021bie care ruleaz\u0103 Python pe Hadoop \u00een PRO, \u0219tiu c\u0103 e pu\u021bin s\u0103 preg\u0103te\u0219ti \u0219i s\u0103 livrezi mediu Python pentru fiecare nod de data. O mul\u021bime de biblioteci C\/C++ pentru \u00eenv\u0103\u021barea automat\u0103 care utilizeaz\u0103 module Python nu \u00ee\u021bi vor permite s\u0103 te odihne\u0219ti lini\u0219tit. Nu trebuie s\u0103 ui\u021bi s\u0103 actualizezi pachetele atunci c\u00e2nd adaugi biblioteci sau servere noi, men\u021bin\u00e2nd compatibilitatea \u00eenapoi cu codul modelului deja implementat.<\/p>\n

<\/p>\n

Exist\u0103 c\u00e2teva abord\u0103ri pentru a face acest lucru. De exemplu, preg\u0103tirea anticipat\u0103 a c\u00e2torva biblioteci frecvent utilizate \u0219i integrarea acestora \u00een PRO. \u00cen distribu\u021bia Hadoop de la Cloudera, acest lucru se face de obicei folosind parcel<\/a><\/noindex>. De asemenea, acum \u00een Hadoop apare posibilitatea de a rula docker<\/a><\/noindex>-containere. \u00cen unele cazuri simple, codul poate fi livrat \u00eempreun\u0103 cu pachetul python.eggs<\/a><\/noindex>.<\/p>\n

<\/p>\n

Banca abordeaz\u0103 foarte serios securitatea rul\u0103rii codului ter\u021b, astfel \u00eenc\u00e2t utiliz\u0103m la maximum noile capacit\u0103\u021bi ale nucleului Linux, unde procesul lansat \u00eentr-un mediu izolat Linux namespace<\/a><\/noindex>, poate fi restric\u021bionat, de exemplu, \u00een ceea ce prive\u0219te accesul la re\u021bea \u0219i la discurile locale, ceea ce reduce semnificativ capacit\u0103\u021bile codului mali\u021bios. Zonele de date ale fiec\u0103rui departament sunt protejate \u0219i accesibile doar proprietarilor acestor date. Platforma garanteaz\u0103 c\u0103 datele dintr-o zon\u0103 pot ajunge \u00een alta doar printr-un proces de publicare a datelor cu control la toate etapele, de la accesul la surse p\u00e2n\u0103 la livrarea datelor \u00een vitrina \u021bint\u0103.<\/p>\n

\n\"Sber.DS
\n<\/p>\n

Anul acesta, ne propunem s\u0103 finaliz\u0103m MVP-ul lans\u0103rii modelelor scrise \u00een Python\/R\/Java pe Hadoop. Ne-am fixat o sarcin\u0103 ambi\u021bioas\u0103 de a \u00eenv\u0103\u021ba s\u0103 rul\u0103m orice mediu utilizator pe Hadoop, pentru a nu restric\u021biona utilizatorii platformei noastre.<\/p>\n

<\/p>\n

De asemenea, s-a dovedit c\u0103 mul\u021bi speciali\u0219ti DS au cuno\u0219tin\u021be excelente de matematic\u0103 \u0219i statistic\u0103, creeaz\u0103 modele remarcabile, dar nu sunt foarte familiariza\u021bi cu transform\u0103rile de date mari, av\u00e2nd nevoie de ajutorul inginerilor no\u0219tri de date pentru preg\u0103tirea seturilor de antrenament. Am decis s\u0103 ajut\u0103m colegii \u0219i s\u0103 cre\u0103m module convenabile pentru transformarea standardizat\u0103 \u0219i preg\u0103tirea caracteristicilor pentru modelele bazate pe motorul Spark. Aceasta va permite s\u0103 dedic\u0103m mai mult timp dezvolt\u0103rii modelelor \u0219i s\u0103 nu a\u0219tept\u0103m ca inginerii de date s\u0103 preg\u0103teasc\u0103 un nou set de date.<\/p>\n

<\/p>\n

La noi lucreaz\u0103 oameni cu cuno\u0219tin\u021be din diferite domenii: Linux \u0219i DevOps, Hadoop \u0219i Spark, Java \u0219i Spring, Scala \u0219i Akka, OpenShift \u0219i Kubernetes. Data viitoare, v\u0103 vom vorbi despre biblioteca de modele, despre cum trece o model\u0103 prin ciclul de via\u021b\u0103 \u00een cadrul companiei, precum \u0219i despre cum se desf\u0103\u0219oar\u0103 validarea \u0219i implementarea.<\/p>\n

Sursa: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0418\u0434\u0435\u0438 \u0438 \u0432\u0441\u0442\u0440\u0435\u0447\u0438 \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u0435\u0449\u0451 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u044b \u043c\u043e\u0436\u043d\u043e \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c, \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u044e\u0442 \u0432 \u0431\u0438\u0437\u043d\u0435\u0441\u0435 \u0440\u0430\u0437\u043d\u043e\u0433\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0430 \u0435\u0436\u0435\u0434\u043d\u0435\u0432\u043d\u043e. \u041d\u043e \u043f\u043e\u043c\u0438\u043c\u043e \u0442\u043e\u0433\u043e, \u0447\u0442\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u043c\u043e\u0436\u0435\u0442 \u0443\u0445\u043e\u0434\u0438\u0442\u044c \u043d\u0430 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u043c\u043e\u0434\u0435\u043b\u0438, \u043d\u0443\u0436\u043d\u043e \u043f\u043e\u0442\u0440\u0430\u0442\u0438\u0442\u044c \u0435\u0433\u043e \u043d\u0430 \u0435\u0451 \u043e\u0446\u0435\u043d\u043a\u0443 \u0438 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0443 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e \u043f\u043e\u043b\u0443\u0447\u0430\u0435\u043c\u044b\u0439 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u043d\u0435 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u044b\u043c. \u041f\u043e\u0441\u043b\u0435 \u0432\u043d\u0435\u0434\u0440\u0435\u043d\u0438\u044f \u043b\u044e\u0431\u0443\u044e \u043c\u043e\u0434\u0435\u043b\u044c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043f\u043e\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u043d\u0430 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0438 \u043f\u0435\u0440\u0438\u043e\u0434\u0438\u0447\u0435\u0441\u043a\u0438 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0442\u044c. \u0418 \u044d\u0442\u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":70863,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-70862","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Sber.DS \u2014 platform\u0103 care permite crearea \u0219i implementarea de modele chiar \u0219i f\u0103r\u0103 cod | ProHoster","description":"Ideile \u0219i \u00eent\u00e2lnirile despre ce alte procese ar putea fi automatizate apar zilnic \u00een afaceri de diferite dimensiuni.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/sber-ds-platforma-kotoraya-pozvolyaet-sozdavat-i-vnedryat-modeli-dazhe-bez-koda","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Sber.DS \u2014 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c \u0438 \u0432\u043d\u0435\u0434\u0440\u044f\u0442\u044c \u043c\u043e\u0434\u0435\u043b\u0438 \u0434\u0430\u0436\u0435 \u0431\u0435\u0437 \u043a\u043e\u0434\u0430 | ProHoster","og:description":"\u0418\u0434\u0435\u0438 \u0438 \u0432\u0441\u0442\u0440\u0435\u0447\u0438 \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u0435\u0449\u0451 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u044b \u043c\u043e\u0436\u043d\u043e \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u0442\u044c, \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u044e\u0442 \u0432 \u0431\u0438\u0437\u043d\u0435\u0441\u0435 \u0440\u0430\u0437\u043d\u043e\u0433\u043e \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0430 \u0435\u0436\u0435\u0434\u043d\u0435\u0432\u043d\u043e.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/sber-ds-platforma-kotoraya-pozvolyaet-sozdavat-i-vnedryat-modeli-dazhe-bez-koda","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-22T03:40:55+00:00","article:modified_time":"2020-03-03T13:14:35+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"70862","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:12:23","updated":"2022-09-28 08:13:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/70862","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=70862"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/70862\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/70863"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=70862"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=70862"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=70862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}