{"id":97982,"date":"2020-10-23T14:42:15","date_gmt":"2020-10-23T12:42:15","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes"},"modified":"2020-11-18T00:58:47","modified_gmt":"2020-11-17T22:58:47","slug":"devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes","title":{"rendered":"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/92dc510aa9d785a31d310816b18bf854.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Salut tuturor! Numele meu este Oleg Sidorenkov, iar eu lucrez ca \u0219ef al echipei de infrastructur\u0103 la compania DomClick. Folosim \u201eCubul\u201d \u00een produc\u021bie de mai bine de trei ani \u0219i, \u00een aceast\u0103 perioad\u0103, am trecut prin multe momente interesante. Ast\u0103zi v\u0103 voi \u00eemp\u0103rt\u0103\u0219i cum, cu o abordare corect\u0103, pute\u021bi extrage \u0219i mai mult\u0103 performan\u021b\u0103 din Kubernetes \u201evanilla\u201d pentru clusterul vostru. Ready steady go! <\/p>\n<p>To\u021bi \u0219ti\u021bi foarte bine c\u0103 Kubernetes este un sistem scalabil cu cod surs\u0103 deschis pentru orchestrarea containerelor; sau, altfel spus, 5 binare care fac magie gestion\u00e2nd ciclul de via\u021b\u0103 al microserviciilor voastre \u00een mediul server. \u00cen plus, este un instrument destul de flexibil, pe care-l pute\u021bi construi ca pe un set de Lego, pentru a maximiza personalizarea \u00een func\u021bie de diverse sarcini.<\/p>\n<p>\u0218i p\u0103rea c\u0103 totul este bine: ad\u0103uga\u021bi servere \u00een cluster ca pe ni\u0219te bu\u0219teni \u00een foc \u0219i nu mai ave\u021bi griji. Dar dac\u0103 v\u0103 pas\u0103 de ecologie, ve\u021bi reflecta: \u201eCum pot men\u021bine focul \u00een sobe \u0219i, \u00een acela\u0219i timp, s\u0103 protejez p\u0103durea?\u201d. Cu alte cuvinte, cum s\u0103 g\u0103si\u021bi modalit\u0103\u021bi de a \u00eembun\u0103t\u0103\u021bi infrastructura \u0219i de a reduce costurile.<\/p>\n<h2>1. Monitoriza\u021bi resursele echipelor \u0219i aplica\u021biilor<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/91e2e60b47ee985b024532c8b685230c.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Una dintre cele mai banale, dar eficiente metode este introducerea requests\/limits. \u00cemp\u0103r\u021bi\u021bi aplica\u021biile pe namespaces \u0219i namespaces pe echipele de dezvoltare. Seta\u021bi aplica\u021biei valori pentru consumul de CPU, memorie \u0219i stocare efemer\u0103 \u00eenainte de deployment.<\/p>\n<pre><code>resources:\n   requests:\n     memory: 2Gi\n     cpu: 250m\n   limits:\n     memory: 4Gi\n     cpu: 500m<\/code><\/pre>\n<p>Prin experien\u021b\u0103, am ajuns la concluzia c\u0103 nu este bine s\u0103 exagera\u021bi cererile fa\u021b\u0103 de limite cu mai mult de dou\u0103 ori. Volumul clusterului este calculat pe baza cererilor, iar dac\u0103 ve\u021bi defini aplica\u021biilor o diferen\u021b\u0103 \u00een resurse, de exemplu, de 5-10 ori, imagina\u021bi-v\u0103 ce se va \u00eent\u00e2mpla cu nodul vostru c\u00e2nd acesta se va umple de poduri \u0219i va primi brusc o \u00eenc\u0103rc\u0103tur\u0103. Nimic bun. Cel pu\u021bin, throttling, iar la maxim, ve\u021bi spune adio lucr\u0103torului \u0219i ve\u021bi avea o \u00eenc\u0103rcare ciclic\u0103 pe celelalte noduri dup\u0103 ce podurile vor \u00eencepe s\u0103 migreze.<\/p>\n<p>\u00cen plus, cu ajutorul <code>limitranges<\/code> pute\u021bi s\u0103 stabili\u021bi la \u00eenceput pentru container valori pentru resurse \u2014 minime, maxime \u0219i implicite:<\/p>\n<pre><code>\u279c  ~ kubectl describe limitranges --namespace ops\nName:       limit-range\nNamespace:  ops\nType        Resource           Min   Max   Default Request  Default Limit  Max Limit\/Request Ratio\n----        --------           ---   ---   ---------------  -------------  -----------------------\nContainer   cpu                50m   10    100m             100m           2\nContainer   ephemeral-storage  12Mi  8Gi   128Mi            4Gi            -\nContainer   memory             64Mi  40Gi  128Mi            128Mi          2<\/code><\/pre>\n<p>Nu uita\u021bi s\u0103 limita\u021bi resursele namespace-ului pentru a \u00eempiedica o echip\u0103 s\u0103 consume toate resursele cluster-ului:<\/p>\n<pre><code>\u279c  ~ kubectl describe resourcequotas --namespace ops\nName:                   resource-quota\nNamespace:              ops\nResource                Used          Hard\n--------                ----          ----\nlimits.cpu              77250m        80\nlimits.memory           124814367488  150Gi\npods                    31            45\nrequests.cpu            53850m        80\nrequests.memory         75613234944   150Gi\nservices                26            50\nservices.loadbalancers  0             0\nservices.nodeports      0             0<\/code><\/pre>\n<p>Dup\u0103 cum se vede din descriere <code>resourcequotas<\/code>, dac\u0103 echipa ops dore\u0219te s\u0103 desf\u0103\u0219oare pod-uri care vor consuma \u00eenc\u0103 10 cpu, atunci planificatorul nu va permite acest lucru \u0219i va genera o eroare:<\/p>\n<pre><code>Error creating: pods \"nginx-proxy-9967d8d78-nh4fs\" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=5,requests.cpu=5, used: limits.cpu=77250m,requests.cpu=53850m, limited: limits.cpu=10,requests.cpu=10<\/code><\/pre>\n<p>Pentru a rezolva o problem\u0103 similar\u0103, se poate scrie un instrument, de exemplu, ca <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/pauljamm\/team-operator\">aceasta<\/a><\/noindex>, capabil s\u0103 stocheze \u0219i s\u0103 comite starea resurselor echipelor.<\/p>\n<h2>2. Alege\u021bi un stocare optim\u0103 a fi\u0219ierelor<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/c7f8bdbf5490acc9061695a6d608015e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Aici a\u0219 dori s\u0103 discut despre volumele persistente \u0219i subsistemul de stocare al nodurilor worker Kubernetes. Sper c\u0103 nimeni nu folose\u0219te \u201eCube\u201d pe HDD \u00een produc\u021bie, dar uneori chiar \u0219i un SSD obi\u0219nuit devine insuficient. Ne-am confruntat cu problema c\u0103 jurnalele supraincarc\u0103 discul din cauza opera\u021biunilor de intrare-ie\u0219ire, iar solu\u021biile nu sunt foarte multe: <\/p>\n<ul>\n<li>\n<p>Utiliza\u021bi SSD-uri de \u00eenalt\u0103 performan\u021b\u0103 sau trece\u021bi la NVMe (dac\u0103 gestiona\u021bi propriul hardware).<\/p>\n<\/li>\n<li>\n<p>Reduce\u021bi nivelul de jurnalizare.<\/p>\n<\/li>\n<li>\n<p>Face\u021bi o \u201e\u00eembinare inteligent\u0103\u201d a pod-urilor care supra\u00eencarc\u0103 discul (<code>podAntiAffinity<\/code>).<\/p>\n<\/li>\n<\/ul>\n<p>Screenshot-ul de mai sus arat\u0103 ce se \u00eent\u00e2mpl\u0103 cu nginx-ingress-controller pe disk c\u00e2nd jurnalizarea access_logs este activat\u0103 (~12 mii jurnale\/sec.). Aceast\u0103 stare poate duce, desigur, la degradarea tuturor aplica\u021biilor de pe acest nod.<\/p>\n<p>\u00cen ceea ce prive\u0219te PV, din p\u0103cate, nu am testat toate <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/persistent-volumes\/#types-of-persistent-volumes\">tipurile<\/a><\/noindex> Volume persistente. Folosi\u021bi cea mai bun\u0103 op\u021biune care se potrive\u0219te nevoilor dumneavoastr\u0103. Istoric, o mic\u0103 parte din servicii necesit\u0103 volume RWX, iar de mult timp folosim stocare NFS pentru acest scop. Este ieftin \u0219i\u2026 suficient. Sigur, am avut parte de multe nepl\u0103ceri cu el \u2014 dar am \u00eenv\u0103\u021bat s\u0103-l optimiz\u0103m, \u0219i acum nu mai avem dureri de cap. Dac\u0103 este posibil, trece\u021bi la stocarea obiect S3.<\/p>\n<h2>3. Colecta\u021bi imagini optimizate<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/c25ce405d1eb4c01f031e18b0bfa4836.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Cel mai bine este s\u0103 utiliza\u021bi imagini optimizate pentru containere, astfel \u00eenc\u00e2t Kubernetes s\u0103 le poat\u0103 accesa mai rapid \u0219i s\u0103 le execute mai eficient.&nbsp;<\/p>\n<p>Optimizarea \u00eenseamn\u0103 c\u0103 imaginile:<\/p>\n<ul>\n<li>\n<p>con\u021bin o singur\u0103 aplica\u021bie sau \u00eendeplinesc o singur\u0103 func\u021bie;<\/p>\n<\/li>\n<li>\n<p>sunt de dimensiuni mici, deoarece imaginile mari sunt mai greu de transferat prin re\u021bea;<\/p>\n<\/li>\n<li>\n<p>au puncte finale pentru verificarea st\u0103rii de func\u021bionare \u0219i a disponibilit\u0103\u021bii, prin care Kubernetes poate lua m\u0103suri \u00een caz de nefunc\u021bionare;<\/p>\n<\/li>\n<li>\n<p>utilizeaz\u0103 sisteme de operare prietenoase cu containere (cum ar fi Alpine sau CoreOS), care sunt mai rezistente la erori de configurare;<\/p>\n<\/li>\n<li>\n<p>folosesc compil\u0103ri multi-etap\u0103, astfel \u00eenc\u00e2t s\u0103 pute\u021bi desf\u0103\u0219ura doar aplica\u021biile compilate, nu \u0219i sursele \u00eenso\u021bitoare.<\/p>\n<\/li>\n<\/ul>\n<p>Exist\u0103 multe instrumente \u0219i servicii care permit verificarea \u0219i optimizarea imaginilor \u00een timp real. Este important s\u0103 le men\u021bine\u021bi \u00eentotdeauna actualizate \u0219i verificate pentru securitate. \u00cen final, ob\u021bine\u021bi: <\/p>\n<ol>\n<li>\n<p>Reducerea \u00eenc\u0103rc\u0103rii de re\u021bea pe \u00eentregul cluster.<\/p>\n<\/li>\n<li>\n<p>Reducerea timpului de lansare a containerului.<\/p>\n<\/li>\n<li>\n<p>Un volum mai mic al \u00eentregului dvs. registry Docker.<\/p>\n<\/li>\n<\/ol>\n<h2>4. Utiliza\u021bi memoria cache DNS<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/9f4384462a77dc528d7911c56f684f2d.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>\u00cen ceea ce prive\u0219te sarcinile mari, f\u0103r\u0103 ajustarea sistemului DNS al clusterului, via\u021ba poate fi destul de nepl\u0103cut\u0103. \u00cen trecut, dezvoltatorii Kubernetes sus\u021bineau solu\u021bia lor kube-dns. A fost implementat\u0103 \u0219i la noi, dar aceast\u0103 aplica\u021bie nu a fost ajustat\u0103 special \u0219i nu oferea performan\u021ba necesar\u0103, de\u0219i, aparent, sarcina era simpl\u0103. Apoi a ap\u0103rut coredns, la care am trecut \u0219i nu am mai avut probleme, devenind ulterior serviciul DNS implicit \u00een K8s. La un moment dat, am ajuns la 40.000 de solicit\u0103ri pe secund\u0103 la sistemul DNS, iar aceast\u0103 solu\u021bie a devenit insuficient\u0103. Totu\u0219i, din fericire, a ap\u0103rut Nodelocaldns, denumit \u0219i cache local de noduri, <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/tasks\/administer-cluster\/nodelocaldns\/\">NodeLocal DNSCache<\/a><\/noindex>.<\/p>\n<p>De ce folosim asta? \u00cen nucleul Linux exist\u0103 un bug care, atunci c\u00e2nd se face multiple apeluri prin conntrack NAT pe UDP, duce la o stare de competi\u021bie pentru scrierea \u00een tabelele conntrack, iar o parte din trafic prin NAT se pierde (fiecare solicitare prin Service este NAT). Nodelocaldns rezolv\u0103 aceast\u0103 problem\u0103 prin eliminarea NAT-ului \u0219i actualizarea conexiunii la TCP c\u0103tre DNS-urile upstream, precum \u0219i prin cache-ul local al solicit\u0103rilor DNS c\u0103tre upstream-uri (inclusiv un cache negativ scurt de 5 secunde).<\/p>\n<h2>5. Scalati podurile automat at\u00e2t orizontal, c\u00e2t \u0219i vertical<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/3804a14685a55160fa7d9f3226eab1fa.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Pute\u021bi spune cu \u00eencredere c\u0103 toate microserviciile dvs. sunt preg\u0103tite pentru o cre\u0219tere a sarcinii de lucru de dou\u0103 sau trei ori? Cum s\u0103 aloca\u021bi corect resursele aplica\u021biilor dvs.? Men\u021binerea unui num\u0103r excesiv de poduri \u00een func\u021bie de sarcina de lucru poate fi redundant\u0103, iar men\u021binerea la limit\u0103 risc\u0103 s\u0103 provoace opriri din cauza unei cre\u0219teri bru\u0219te a traficului pe serviciu. Echilibrul este realizat cu ajutorul serviciilor de scalare, cum ar fi <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/tasks\/run-application\/horizontal-pod-autoscale\/\">Horizontal Pod Autoscaler<\/a><\/noindex> \u0219i <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/autoscaler\/tree\/master\/vertical-pod-autoscaler\">Vertical Pod Autoscaler<\/a><\/noindex>. <\/p>\n<p><strong>VPA<\/strong> care permite cre\u0219terea automat\u0103 a requests\/limits containerelor din pod \u00een func\u021bie de utilizarea real\u0103. Cum poate fi util? Dac\u0103 ave\u021bi poduri care nu pot fi scalate orizontal dintr-un anumit motiv (ceea ce nu este tocmai sigur), pute\u021bi \u00eencerca s\u0103 l\u0103sa\u021bi VPA s\u0103 gestioneze modificarea resurselor sale. Caracteristica sa const\u0103 \u00een sistemul de recomand\u0103ri bazat pe date istorice \u0219i actuale din metric-server, a\u0219adar, dac\u0103 nu dori\u021bi s\u0103 schimba\u021bi automat requests\/limits, pute\u021bi pur \u0219i simplu s\u0103 monitoriza\u021bi resursele recomandate pentru containerele dvs. \u0219i s\u0103 optimiza\u021bi set\u0103rile pentru economisirea CPU \u0219i memorie \u00een cluster. <\/p>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/3965950aa3315faa4bb3e3ff5bd61955.png\" style=\"display:block;margin: 0 auto;\" \/>Imaginea este preluat\u0103 de la https:\/\/levelup.gitconnected.com\/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231<\/p>\n<p>Planificatorul din Kubernetes se bazeaz\u0103 \u00eentotdeauna pe requests. Indiferent de valoarea pe care o pune\u021bi acolo, planificatorul va c\u0103uta un nod potrivit, baz\u00e2ndu-se pe aceasta. Valorile limits sunt necesare kubelet-ului pentru a \u00een\u021belege c\u00e2nd s\u0103 throttleze sau s\u0103 omoare podul. \u0218i deoarece singurul parametru important este valoarea requests, VPA va lucra cu acesta. De fiecare dat\u0103 c\u00e2nd stabili\u021bi scalarea vertical\u0103 a aplica\u021biei, defini\u021bi cum ar trebui s\u0103 fie requests. Dar ce se va \u00eent\u00e2mpla cu limits? Acest parametru va fi, de asemenea, scalat propor\u021bional.<\/p>\n<p>De exemplu, iat\u0103 set\u0103rile obi\u0219nuite ale podului:<\/p>\n<pre><code>resurse:\n   cereri:\n     memorie: 250Mi\n     cpu: 200m\n   limite:\n     memorie: 500Mi\n     cpu: 350m<\/code><\/pre>\n<p>Mecanismul de recomandare stabile\u0219te c\u0103 aplica\u021bia dumneavoastr\u0103 necesit\u0103 300m CPU \u0219i 500Mi pentru a func\u021biona corespunz\u0103tor. Ve\u021bi primi urm\u0103toarele set\u0103ri:<\/p>\n<pre><code>resurse:\n   cereri:\n     memorie: 500Mi\n     cpu: 300m\n   limite:\n     memorie: 1000Mi\n     cpu: 525m<\/code><\/pre>\n<p>A\u0219a cum s-a men\u021bionat mai sus, aceasta este o scalare propor\u021bional\u0103 bazat\u0103 pe raportul cereri\/limite din manifest:<\/p>\n<ul>\n<li>\n<p>CPU: 200m \u2192 300m: raport 1:1.75;<\/p>\n<\/li>\n<li>\n<p>Memorie: 250Mi \u2192 500Mi: raport 1:2.<\/p>\n<\/li>\n<\/ul>\n<p>\u00cen ceea ce prive\u0219te <strong>HPA<\/strong>, aici mecanismul de func\u021bionare este mai transparent. Se stabilesc valori prag pentru statistici, cum ar fi CPU \u0219i memorie, iar dac\u0103 media tuturor replicilor dep\u0103\u0219e\u0219te pragul, aplica\u021bia se scaleaz\u0103 cu +1 pod p\u00e2n\u0103 c\u00e2nd valoarea scade sub prag sau p\u00e2n\u0103 c\u00e2nd se atinge num\u0103rul maxim de replici.<\/p>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/f7a3fa28177d66be925867e38f5bbed6.png\" style=\"display:block;margin: 0 auto;\" \/>Imaginea este preluat\u0103 de la https:\/\/levelup.gitconnected.com\/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231<\/p>\n<p>Pe l\u00e2ng\u0103 statisticile obi\u0219nuite, cum ar fi CPU \u0219i memorie, pute\u021bi configura praguri pe metrici personalizate din Prometheus \u0219i s\u0103 lucra\u021bi cu acestea, dac\u0103 considera\u021bi c\u0103 este cea mai precis\u0103 defini\u021bie despre c\u00e2nd s\u0103 scala\u021bi aplica\u021bia dumneavoastr\u0103. Odat\u0103 ce aplica\u021bia se stabilizeaz\u0103 sub limita stabilit\u0103, HPA va \u00eencepe s\u0103 scaleze podurile \u00een jos p\u00e2n\u0103 la num\u0103rul minim de replici sau p\u00e2n\u0103 c\u00e2nd sarcina va \u00eendeplini pragul stabilit.<\/p>\n<h2>6. Nu uita\u021bi de Node Affinity \u0219i Pod Affinity<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/e718191d9e8ff25fd3b2d65cba9b3b73.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Nu toate nodurile func\u021bioneaz\u0103 pe hardware identical, nu toate podurile trebuie s\u0103 ruleze aplica\u021bii ce necesit\u0103 capacitate de procesare intens\u0103. Kubernetes permite specializarea nodurilor \u0219i podurilor prin: <strong>Node Affinity<\/strong> \u0219i <strong>Pod Affinity<\/strong>.<\/p>\n<p>Dac\u0103 ave\u021bi noduri potrivite pentru opera\u021biuni cu intensitate de procesare, pentru eficien\u021b\u0103 maxim\u0103 este mai bine s\u0103 lega\u021bi aplica\u021biile la noduri corespunz\u0103toare. Pentru aceasta utiliza\u021bi: <code>nodeSelector<\/code> cu eticheta nodului.<\/p>\n<p>S\u0103 presupunem c\u0103 ave\u021bi dou\u0103 noduri: unul cu <code>CPUType=HIGHFREQ<\/code> \u0219i multe nuclee rapide, altul cu <code>MemoryType=HIGHMEMORY<\/code> multe memorie \u0219i o performan\u021b\u0103 mai mare. Cel mai simplu este s\u0103 aloca\u021bi desf\u0103\u0219urarea podului nodului <code>HIGHFREQ<\/code>, ad\u0103ug\u00e2nd \u00een sec\u021biunea <code>spec<\/code> un selector ca acesta:<\/p>\n<pre><code>\u2026\nnodeSelector:\n\tCPUType: HIGHFREQ<\/code><\/pre>\n<p>O modalitate mai costisitoare \u0219i specific\u0103 de a face acest lucru este prin utilizarea sec\u021biunii <code>nodeAffinity<\/code> \u00een c\u00e2mpul <code>affinity<\/code> Exist\u0103 dou\u0103 op\u021biuni: <code>spec<\/code>: configurare strict\u0103 (planificatorul va desf\u0103\u0219ura podurile doar pe noduri specifice (\u0219i nic\u0103ieri altundeva));<\/p>\n<ul>\n<li>\n<p><code>requiredDuringSchedulingIgnoredDuringExecution<\/code>preferredDuringSchedulingIgnoredDuringExecution<\/p>\n<\/li>\n<li>\n<p><code>preferredDuringSchedulingIgnoredDuringExecution<\/code>: configurare flexibil\u0103 (schedulerul va \u00eencerca s\u0103 desf\u0103\u0219oare pe noduri specifice, iar dac\u0103 nu reu\u0219e\u0219te, va \u00eencerca s\u0103 desf\u0103\u0219oare pe urm\u0103torul nod disponibil).<\/p>\n<\/li>\n<\/ul>\n<p>Pute\u021bi specifica o sintax\u0103 de control al etichetelor nodurilor, de exemplu, <code>In<\/code>, <code>NotIn<\/code>, <code>Exists<\/code>, <code>DoesNotExist<\/code>, <code>Gt<\/code> sau <code>Lt<\/code>. Totu\u0219i, re\u021bine\u021bi c\u0103 metodele complexe \u00een liste lungi de etichete vor \u00eencetini luarea deciziilor \u00een situa\u021bii critice. Cu alte cuvinte, nu complica\u021bi.<\/p>\n<p>A\u0219a cum s-a men\u021bionat mai sus, Kubernetes permite specificarea asocierii podurilor curente. Adic\u0103 pute\u021bi face astfel \u00eenc\u00e2t anumite poduri s\u0103 lucreze \u00eempreun\u0103 cu alte poduri \u00een aceea\u0219i zon\u0103 de disponibilitate (relevant pentru cloud-uri) sau noduri.<\/p>\n<p>\u00cen <code>podAffinity<\/code> c\u00e2mpuri <code>affinity<\/code> Exist\u0103 dou\u0103 op\u021biuni: <code>spec<\/code> sunt disponibile acelea\u0219i c\u00e2mpuri ca \u0219i \u00een cazul <code>nodeAffinity<\/code>: <code>requiredDuringSchedulingIgnoredDuringExecution<\/code><strong> <\/strong>\u0219i <code>preferredDuringSchedulingIgnoredDuringExecution<\/code>. Singura diferen\u021b\u0103 este c\u0103 <code>matchExpressions<\/code> va asocia podurile cu nodul pe care deja ruleaz\u0103 un pod cu aceast\u0103 etichet\u0103.<\/p>\n<p>De asemenea, Kubernetes ofer\u0103 c\u00e2mpul <code>podAntiAffinity<\/code>, care, spre deosebire, nu asociaz\u0103 podul cu nodul care are anumite poduri.<\/p>\n<p>Referitor la expresii, <code>nodeAffinity<\/code> se poate da acela\u0219i sfat: \u00eencerca\u021bi s\u0103 men\u021bine\u021bi simplitatea \u0219i logica regulilor, nu trebuie s\u0103 \u00eencerca\u021bi s\u0103 suprasolicita\u021bi specifica\u021bia podurilor cu un set complex de reguli. Este foarte u\u0219or s\u0103 crea\u021bi o regul\u0103 care s\u0103 nu corespund\u0103 condi\u021biilor cluster-ului, cre\u00e2nd o sarcin\u0103 suplimentar\u0103 pentru scheduler \u0219i sc\u0103z\u00e2nd performan\u021ba general\u0103.<\/p>\n<h2>7. Taints &amp; Tolerations<\/h2>\n<p>Exist\u0103 \u0219i o alt\u0103 modalitate de a gestiona schedulerul. Dac\u0103 ave\u021bi un cluster mare cu sute de noduri \u0219i mii de microservicii, atunci este foarte greu s\u0103 nu permite\u021bi anumitor poduri s\u0103 fie plasate pe anumite noduri.<\/p>\n<p>Acesta este sprijinit de mecanismul taints \u2014 reguli interzic\u0103toare. De exemplu, \u00een anumite scenarii, pute\u021bi interzice anumitor noduri s\u0103 ruleze poduri. Pentru a aplica taint unui nod specific, trebuie s\u0103 folosi\u021bi op\u021biunea <code>taint<\/code> \u00een kubectl. Specifica\u021bi cheia \u0219i valoarea, apoi taint cum ar fi <code>NoSchedule<\/code> sau <code>NoExecute<\/code>:<\/p>\n<pre><code>$ kubectl taint nodes node10 node-role.kubernetes.io\/ingress=true:NoSchedule<\/code><\/pre>\n<p>De asemenea, este important de men\u021bionat c\u0103 mecanismul taint suport\u0103 trei efecte principale: <code>NoSchedule<\/code>, <code>NoExecute<\/code> \u0219i <code>PreferNoSchedule<\/code><strong>. <\/strong><\/p>\n<ul>\n<li>\n<p><code>NoSchedule<\/code><strong> <\/strong>\u00eenseamn\u0103 c\u0103 at\u00e2ta timp c\u00e2t \u00een specifica\u021bia podului nu va exista o \u00eenregistrare corespunz\u0103toare <code>tolerations<\/code>, acesta nu va putea fi desf\u0103\u0219urat pe nod (\u00een acest exemplu <code>node10<\/code>).<\/p>\n<\/li>\n<li>\n<p><code>PreferNoSchedule <\/code>\u2014 o versiune simplificat\u0103 <code>NoSchedule<\/code>. \u00cen acest caz, schedulerul va \u00eencerca s\u0103 nu distribuie podurile care nu au o \u00eenregistrare corespunz\u0103toare <code>tolerations<\/code> pe nod, dar aceasta nu este o restric\u021bie strict\u0103. Dac\u0103 \u00een cluster nu vor exista resurse, atunci podurile vor \u00eencepe s\u0103 se desf\u0103\u0219oare pe acest nod.<\/p>\n<\/li>\n<li>\n<p><code>NoExecute<\/code><strong> <\/strong>\u2014 acest efect declan\u0219eaz\u0103 evacuarea imediat\u0103 a podurilor care nu au o \u00eenregistrare corespunz\u0103toare <code>tolerations<\/code>.<\/p>\n<\/li>\n<\/ul>\n<p>Este interesant c\u0103 acest comportament poate fi anulat prin intermediul mecanismului de toleran\u021be. Este convenabil atunci c\u00e2nd exist\u0103 un nod \"interzis\" \u0219i trebuie s\u0103 plasa\u021bi pe el doar servicii infrastructurale. Cum se face asta? Permite\u021bi doar acele poduri pentru care exist\u0103 toleran\u021ba potrivit\u0103.<\/p>\n<p>Iat\u0103 cum va ar\u0103ta specifica\u021bia podului:<\/p>\n<pre><code>spec:\n   toleran\u021be:\n     - cheie: \"node-role.kubernetes.io\\\/ingress\"\n        operator: \"Equal\"\n        valoare: \"true\"\n        efect: \"NoSchedule\"<\/code><\/pre>\n<p>Asta nu \u00eenseamn\u0103 c\u0103 la urm\u0103torul redeploy podul va ajunge exact pe acest nod, nu este un mecanism de Node Affinity \u0219i <code>nodeSelector<\/code>. Dar combin\u00e2nd mai multe func\u021bii, pute\u021bi ob\u021bine o configurare foarte flexibil\u0103 a scheduler-ului.<\/p>\n<h2>8. Configura\u021bi prioritatea desf\u0103\u0219ur\u0103rii podurilor<\/h2>\n<p>Ceea ce a\u021bi configurat ca \u0219i legare a podurilor de noduri nu \u00eenseamn\u0103 c\u0103 toate podurile trebuie s\u0103 fie procesate cu aceea\u0219i prioritate. De exemplu, s-ar putea s\u0103 dori\u021bi s\u0103 desf\u0103\u0219ura\u021bi unele poduri mai devreme dec\u00e2t altele.<\/p>\n<p>Kubernetes ofer\u0103 diferite moduri de a configura prioritatea podurilor (Pod Priority and Preemption). Configurarea const\u0103 din mai multe p\u0103r\u021bi: obiectul <code>PriorityClass<\/code><strong> <\/strong>\u0219i descrierea c\u00e2mpului <code>priorityClassName<\/code><strong> <\/strong>\u00een specifica\u021bia podului. S\u0103 vedem un exemplu:<\/p>\n<pre><code>apiVersion: scheduling.k8s.io\\\/v1\nkind: PriorityClass\nmetadata:\n  name: high-priority\nvalue: 99999\nglobalDefault: false\ndescription: \"Aceast\u0103 clas\u0103 de prioritate ar trebui s\u0103 fie utilizat\u0103 doar pentru podurile foarte importante\"<\/code><\/pre>\n<p>Cre\u0103m <code>PriorityClass<\/code>, \u00eei d\u0103m un nume, o descriere \u0219i o valoare.<strong> <\/strong>Cu c\u00e2t <code>valoare<\/code>, cu at\u00e2t prioritatea este mai mare. Valoarea poate fi orice num\u0103r \u00eentreg de 32 de bi\u021bi, mai mic sau egal cu 1 000 000 000. Valori mai mari sunt rezervate pentru podurile critice de sistem, care, \u00een general, nu pot fi \u00eenl\u0103turate.<strong> <\/strong>\u00cenlocuirea va avea loc doar dac\u0103 podul de \u00eenalt\u0103 prioritate nu are loc unde s\u0103 se desf\u0103\u0219oare, atunci unele poduri de pe un anumit nod vor fi evacuate. Dac\u0103 acest mecanism este prea rigid pentru dvs., pute\u021bi ad\u0103uga op\u021biunea <code>preemptionPolicy: Never<\/code>, \u0219i atunci nu va exista \u00eenlocuire, podul va fi pus primul \u00een a\u0219teptare \u0219i va a\u0219tepta p\u00e2n\u0103 c\u00e2nd scheduler-ul g\u0103se\u0219te resurse libere pentru el.<\/p>\n<p>Apoi cre\u0103m un pod \u00een care specific\u0103m numele <code>priorityClassName<\/code>:<\/p>\n<pre><code>apiVersion: v1\nkind: Pod\nmetadata:\n  name: static-web\n  labels:\n    role: myrole\n spec:\n  containers:\n    - name: web\n      image: nginx\n      ports:\n        - name: web\n          containerPort: 80\n          protocol: TCP\n  priorityClassName: high-priority\n          <\/code><\/pre>\n<p>Pute\u021bi crea c\u00e2te clase de prioritate dori\u021bi, de\u0219i se recomand\u0103 s\u0103 nu exagera\u021bi (s\u0103 v\u0103 limita\u021bi la prioritate mic\u0103, medie \u0219i mare). <\/p>\n<p>Astfel, \u00een caz de necesitate, ve\u021bi putea \u00eembun\u0103t\u0103\u021bi eficien\u021ba desf\u0103\u0219ur\u0103rii serviciilor critice, cum ar fi nginx-ingress-controller, coredns etc.<\/p>\n<h2>9. Optimizarea clusterului ETCD<\/h2>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/f5917adfa943c789b6c784f43305851b.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>ETCD poate fi numit creierul \u00eentregului cluster. Este foarte important s\u0103 men\u021bine\u021bi func\u021bionarea acestei baze de date la un nivel ridicat, deoarece de aceasta depinde viteza opera\u021biunilor \u00een \u201eKube\u201d. O solu\u021bie standard, dar totu\u0219i bun\u0103, ar fi s\u0103 men\u021bine\u021bi clusterul ETCD pe nodurile master, pentru a avea o \u00eent\u00e2rziere minim\u0103 fa\u021b\u0103 de kube-apiserver. Dac\u0103 nu reu\u0219i\u021bi s\u0103 face\u021bi acest lucru, atunci plasa\u021bi ETCD c\u00e2t mai aproape posibil, av\u00e2nd o l\u0103\u021bime de band\u0103 bun\u0103 \u00eentre participan\u021bi. De asemenea, ave\u021bi grij\u0103 la c\u00e2te noduri din ETCD pot ie\u0219i din func\u021biune f\u0103r\u0103 a afecta clusterul.<\/p>\n<p><img decoding=\"async\" alt=\"Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes\" src=\"\/wp-content\/uploads\/2020\/10\/cdd2c32eefed253c6ff774899c367ccc.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Fi\u021bi con\u0219tien\u021bi c\u0103 o cre\u0219tere excesiv\u0103 a num\u0103rului de participan\u021bi \u00een cluster poate spori disponibilitatea, dar poate afecta performan\u021ba; totul trebuie s\u0103 fie \u00eentr-o m\u0103sur\u0103 rezonabil\u0103.<\/p>\n<p>C\u00e2nd vine vorba de configurarea serviciului, recomand\u0103rile sunt pu\u021bine:<\/p>\n<ol>\n<li>\n<p>S\u0103 ave\u021bi hardware bun, \u00een func\u021bie de dimensiunea clusterului (puteti citi <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/etcd-io\/etcd\/blob\/master\/Documentation\/op-guide\/hardware.md\">aici<\/a><\/noindex>).<\/p>\n<\/li>\n<li>\n<p>S\u0103 ajusta\u021bi c\u00e2teva parametrii, dac\u0103 a\u021bi dispersat clusterul \u00eentre c\u00e2teva centre de date sau re\u021beaua \u0219i discurile dumneavoastr\u0103 las\u0103 de dorit (pute\u021bi citi <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/etcd-io\/etcd\/blob\/master\/Documentation\/tuning.md\">aici<\/a><\/noindex>).<\/p>\n<\/li>\n<\/ol>\n<h2>Concluzie<\/h2>\n<p>\u00cen acest articol sunt descrise punctele pe care echipa noastr\u0103 \u00eencearc\u0103 s\u0103 le respecte. Acesta nu este un ghid pas cu pas, ci variante care pot fi utile pentru optimizarea costurilor indirecte ale clusterului. Este clar c\u0103 fiecare cluster este unic, iar solu\u021biile de configurare pot varia foarte mult, a\u0219a c\u0103 ar fi interesant s\u0103 primim feedback de la voi: cum v\u0103 monitoriza\u021bi clusterul Kubernetes, cu ce instrumente \u00eembun\u0103t\u0103\u021bi\u021bi func\u021bionarea lui. \u00cemp\u0103rt\u0103\u0219i\u021bi-v\u0103 experien\u021ba \u00een comentarii, va fi interesant de aflat. <\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/domclick\/blog\/520968\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041e\u043b\u0435\u0433 \u0421\u0438\u0434\u043e\u0440\u0435\u043d\u043a\u043e\u0432, \u044f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 \u0414\u043e\u043c\u041a\u043b\u0438\u043a \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u0435\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b. \u041c\u044b \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0438\u0440\u0443\u0435\u043c \u00ab\u041a\u0443\u0431\u0438\u043a\u00bb \u0432 \u043f\u0440\u043e\u0434\u0435 \u0443\u0436\u0435 \u0431\u043e\u043b\u044c\u0448\u0435 \u0442\u0440\u0451\u0445 \u043b\u0435\u0442, \u0438 \u0437\u0430 \u044d\u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043f\u0435\u0440\u0435\u0436\u0438\u043b\u0438 \u0441 \u043d\u0438\u043c \u043c\u043d\u043e\u0433\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u0445 \u043c\u043e\u043c\u0435\u043d\u0442\u043e\u0432. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u043f\u043e\u0432\u0435\u0434\u0430\u044e \u0432\u0430\u043c, \u043a\u0430\u043a \u043f\u0440\u0438 \u043f\u0440\u0430\u0432\u0438\u043b\u044c\u043d\u043e\u043c \u043f\u043e\u0434\u0445\u043e\u0434\u0435 \u043c\u043e\u0436\u043d\u043e \u0432\u044b\u0436\u0430\u0442\u044c \u0438\u0437 \u00ab\u0432\u0430\u043d\u0438\u043b\u044c\u043d\u043e\u0433\u043e\u00bb Kubernetes \u0435\u0449\u0451 \u0431\u043e\u043b\u044c\u0448\u0435 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0433\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430. Ready steady [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":97983,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-97982","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0414\u0435\u0432\u044f\u0442\u044c \u0441\u043e\u0432\u0435\u0442\u043e\u0432 \u043f\u043e \u043f\u043e\u0432\u044b\u0448\u0435\u043d\u0438\u044e \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 Kubernetes | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-10-23T12:42:15+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-11-17T22:58:47+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Nou\u0103 sfaturi pentru \u00eembun\u0103t\u0103\u021birea performan\u021bei Kubernetes | ProHoster","description":"Salut tuturor!","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0414\u0435\u0432\u044f\u0442\u044c \u0441\u043e\u0432\u0435\u0442\u043e\u0432 \u043f\u043e \u043f\u043e\u0432\u044b\u0448\u0435\u043d\u0438\u044e \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 Kubernetes | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442!","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/devyat-sovetov-po-povysheniyu-proizvoditelnosti-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-10-23T12:42:15+00:00","article:modified_time":"2020-11-17T22:58:47+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"97982","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:08:27","updated":"2022-09-30 17:30:03","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/97982","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=97982"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/97982\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/97983"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=97982"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=97982"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=97982"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}