{"id":40721,"date":"2020-02-03T14:41:59","date_gmt":"2020-02-03T11:41:59","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka"},"modified":"2020-02-03T14:41:59","modified_gmt":"2020-02-03T11:41:59","slug":"nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","status":"publish","type":"post","link":"https:\/\/prohoster.info\/no\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","title":{"rendered":"V\u00e5r erfaring med utvikling av CSI-driver i Kubernetes for Yandex.Cloud","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"V\u00e5r erfaring med utvikling av CSI-driver i Kubernetes for Yandex.Cloud\" src=\"\/wp-content\/uploads\/2020\/02\/86adf77b2ee08c426bf79c5b9b2d3d1e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nVi er glade for \u00e5 kunngj\u00f8re at selskapet \u00abFlant\u00bb utvider sitt bidrag til Open Source-verkt\u00f8y for Kubernetes med lanseringen av <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">alphaversjon av CSI-driveren<\/a><\/noindex> (Container Storage Interface) for Yandex.Cloud.<\/p>\n<p>Men f\u00f8r vi g\u00e5r i detaljer om implementeringen, la oss svare p\u00e5 sp\u00f8rsm\u00e5let: hvorfor er dette n\u00f8dvendig, n\u00e5r Yandex allerede har tjenesten <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/managed-kubernetes\/\">Managed Service for Kubernetes.<\/a><\/noindex>.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Innledning<\/h2>\n<p><\/p>\n<h3>Hvorfor det?<\/h3>\n<p>\nInnen v\u00e5r virksomhet, helt fra starten av driften av Kubernetes i produksjon (dvs. i flere \u00e5r n\u00e5), har vi utviklet et eget verkt\u00f8y (deckhouse), som vi for \u00f8vrig ogs\u00e5 planlegger \u00e5 gj\u00f8re tilgjengelig som et Open Source-prosjekt snart. Med det konfigurerer og setter vi opp alle v\u00e5re klynger p\u00e5 en ensartet m\u00e5te, og i \u00f8yeblikket har vi mer enn 100 av dem, p\u00e5 de mest varierte maskinvarekonfigurasjoner og p\u00e5 alle tilgjengelige skytjenester.<\/p>\n<p>Klyngene som bruker deckhouse inneholder alle n\u00f8dvendige komponenter for drift: lastbalanserere, overv\u00e5king med praktiske diagrammer, m\u00e5linger og varsler, autentisering av brukere gjennom eksterne leverand\u00f8rer for tilgang til alle dashboard og s\u00e5 videre. Det er ikke meningsfylt \u00e5 sette opp en s\u00e5 \u00aboppgradert\u00bb klynge i en managed-l\u00f8sning, ettersom det ofte enten er umulig eller vil f\u00f8re til at man m\u00e5 deaktivere halve komponentene.<\/p>\n<p><i><b>NB<\/b>: Dette er v\u00e5r erfaring, og den er ganske spesifikk. Vi p\u00e5st\u00e5r absolutt ikke at alle b\u00f8r h\u00e5ndtere oppsett av Kubernetes-klynger selv, i stedet for \u00e5 bruke ferdige l\u00f8sninger. For \u00e5 si det slik, vi har ingen reell erfaring med driften av Kubernetes fra Yandex, og vi vil ikke gi noen vurdering av denne tjenesten i denne artikkelen.<\/i><\/p>\n<h3>Hva er det og for hvem?<\/h3>\n<p>\nS\u00e5, vi har allerede snakket om den moderne tiln\u00e6rmingen til lagringsl\u00f8sninger i Kubernetes: <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/424211\/\">hvordan CSI er konstruert,<\/a><\/noindex> og <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/465417\/\">hvordan samfunnet kom<\/a><\/noindex> til denne tiln\u00e6rmingen.<\/p>\n<p>I dag har mange store leverand\u00f8rer av skytjenester utviklet drivere for bruk av sine \u00abskytjenester\u00bb disker som Persistent Volume i Kubernetes. Hvis leverand\u00f8ren ikke har en slik driver, men alle n\u00f8dvendige funksjoner tilbys via API, er det ingenting som hindrer deg i \u00e5 lage driveren selv. Slik har vi gjort det med Yandex.Cloud.<\/p>\n<p>Vi baserte utviklingen p\u00e5 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/digitalocean\/csi-digitalocean\">CSI-driveren for DigitalOcean-skyen<\/a><\/noindex> og noen ideer fra <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-sigs\/gcp-compute-persistent-disk-csi-driver\">driveren for GCP,<\/a><\/noindex>fordi samhandlingen med APIene til disse skyene (Google og Yandex) har noen likheter. Spesielt er APIene til b\u00e5de <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/compute\/docs\/api\/how-tos\/api-requests-responses#handling_api_responses\">GCP<\/a><\/noindex>og <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/api-design-guide\/concepts\/about-async\">Yandex<\/a><\/noindex> return an object <code>Operation<\/code> for tracking the status of long operations (for example, creating a new disk). To interact with Yandex.Cloud API, the <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/yandex-cloud\/go-sdk\">Yandex.Cloud Go SDK<\/a><\/noindex>.<\/p>\n<p>Result of the completed work <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">published on GitHub<\/a><\/noindex> and may be useful for those who for some reason use their own installation of Kubernetes on Yandex.Cloud virtual machines (but not a managed cluster) and would like to use (order) disks through CSI.<\/p>\n<h2>Implementering<\/h2>\n<p><\/p>\n<h3>Hovedmuligheter<\/h3>\n<p>\nAt this time, the driver supports the following features:<\/p>\n<ul>\n<li> Ordering disks in all cluster zones according to the topology of the existing nodes in the cluster;<\/li>\n<li> Deleting previously ordered disks;<\/li>\n<li> Offline resize for disks (Yandex.Cloud <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/compute\/operations\/disk-control\/update#change-disk-size\">st\u00f8tter ikke<\/a><\/noindex> increasing disks that are mounted to a virtual machine). Below is more information on how to modify the driver to perform resizing as painlessly as possible.<\/li>\n<\/ul>\n<p>\nIn the future, support for creating and deleting disk snapshots is planned.<\/p>\n<h3>The main challenge and its solution<\/h3>\n<p>\nThe absence in the Yandex.Cloud API of the ability to increase disks in real-time is a limitation that complicates the resize operation for PV (Persistent Volume): in such cases, the pod of the application using the disk must be stopped, which can cause application downtime.<\/p>\n<p>I henhold til <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/container-storage-interface\/spec\">CSI-spesifikasjonen<\/a><\/noindex>, if the CSI controller reports that it can only perform disk resizing 'offline' (<code>VolumeExpansion.OFFLINE<\/code>), then the process of increasing the disk must proceed as follows:<\/p>\n<blockquote><p>If the plugin has only <code>VolumeExpansion.OFFLINE<\/code> expansion capability and volume is currently published or available on a node then <code>ControllerExpandVolume<\/code> MUST be called ONLY after either:<\/p>\n<ul>\n<li> The plugin has controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability and <code>ControllerUnpublishVolume<\/code> has been invoked successfully.<\/li>\n<\/ul>\n<p>\nOR ELSE<\/p>\n<ul>\n<li> The plugin does NOT have controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability, the plugin has node <code>STAGE_UNSTAGE_VOLUME<\/code> capability, and <code>NodeUnstageVolume<\/code> has been completed successfully.<\/li>\n<\/ul>\n<p>\nOR ELSE<\/p>\n<ul>\n<li>The plugin does NOT have controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability, nor node <code>STAGE_UNSTAGE_VOLUME<\/code> capability, and <code>NodeUnpublishVolume<\/code> has completed successfully.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nEssentially, this means the need to detach the disk from the virtual machine before increasing it.<\/p>\n<p>However, unfortunately, <b>implementering av<\/b> the CSI specifications via sidecars do not meet these requirements:<\/p>\n<ul>\n<li> In the sidecar container <code>csi-attacher<\/code>, which is supposed to ensure the necessary gap between mountings, this functionality is simply not implemented during offline resizing. A discussion on this was initiated. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-csi\/external-attacher\/issues\/207\">her<\/a><\/noindex>.<\/li>\n<li> Hva er egentlig en sidecar-beholder i denne konteksten? Selve CSI-pluginen h\u00e5ndterer ikke interaksjonen med Kubernetes API, men reagerer bare p\u00e5 gRPC-anropene som sendes til den av sidecar-beholdere. Sistnevnte <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes-csi.github.io\/docs\/sidecar-containers.html\">utvikles<\/a><\/noindex> er fellesskapet til Kubernetes.<\/li>\n<\/ul>\n<p>\nI v\u00e5rt tilfelle (CSI-plugin) ser operasjonen for \u00e5 utvide disken slik ut:<\/p>\n<ol>\n<li> Vi mottar gRPC-anropet <code>ControllerExpandVolume<\/code>;<\/li>\n<li> Vi fors\u00f8ker \u00e5 utvide disken i API-en, men f\u00e5r en feil om at operasjonen ikke kan utf\u00f8res, ettersom disken er montert;<\/li>\n<li> Vi lagrer diskens ID i et kart som inneholder disker som trenger \u00e5 f\u00e5 utvidet. Videre vil vi for enkelhets skyld kalle dette kartet for <code>volumeResizeRequired<\/code>;<\/li>\n<li> Vi fjerner manuelt pod-en som bruker disken. Kubernetes vil deretter starte den p\u00e5 nytt. For \u00e5 forhindre at disken rekker \u00e5 bli montert (<code>ControllerPublishVolume<\/code>) f\u00f8r utvidelsesoperasjonen er fullf\u00f8rt ved montasjen, sjekker vi at disken fortsatt befinner seg i <code>volumeResizeRequired<\/code> og returnerer en feil;<\/li>\n<li> CSI-driveren pr\u00f8ver \u00e5 utf\u00f8re resize-operasjonen p\u00e5 nytt. Hvis operasjonen lykkes, fjerner vi disken fra <code>volumeResizeRequired<\/code>;<\/li>\n<li> Siden diskens ID mangler i <code>volumeResizeRequired<\/code>, <code>ControllerPublishVolume<\/code> operasjonen gjennomf\u00f8res vellykket, disken monteres, pod-en startes.<\/li>\n<\/ol>\n<p>\nAlt ser ganske enkelt ut, men som alltid finnes det skjulte fallgruver. Utvidelsen av disker h\u00e5ndteres av <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes-csi.github.io\/docs\/external-resizer.html\">external-resizer<\/a><\/noindex>, som i tilfelle av en feil under operasjonen <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-csi\/external-resizer\/blob\/master\/vendor\/k8s.io\/client-go\/util\/workqueue\/default_rate_limiters.go#L41\">bruker en k\u00f8<\/a><\/noindex> med eksponentiell \u00f8kning av tidsavbruddet opp til 1000 sekunder:<\/p>\n<pre><code class=\"go\">func DefaultControllerRateLimiter() RateLimiter {\n  return NewMaxOfRateLimiter(\n  NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second),\n  \\\/\\\/ 10 qps, 100 bucket size.  Dette er kun for retry-hastighet og det er bare den generelle faktoren (ikke per element)\n  &amp;BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)},\n  )\n}<\/code><\/pre>\n<p>\nDette kan periodisk f\u00f8re til at diskens utvidelsesoperasjon strekker seg over 15+ minutter og dermed gj\u00f8r den tilknyttede pod-en utilgjengelig.<\/p>\n<p>Det eneste alternativet som enkelt og smertefritt gjorde at vi kunne redusere potensialet for nedetid, var \u00e5 bruke v\u00e5r egen versjon av external-resizer med et maksimalt tidsavbrudd <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/external-resizer\/blob\/faster-workqueue\/pkg\/controller\/controller.go#L81\">p\u00e5 5 sekunder<\/a><\/noindex>:<\/p>\n<pre><code class=\"go\">workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 5*time.Second)<\/code><\/pre>\n<p>\nVi ans\u00e5 det ikke som n\u00f8dvendig \u00e5 umiddelbart initiere en diskusjon og patchere external-resizer, siden offline resize av disker er et levn som snart vil forsvinne hos alle skytjenesteleverand\u00f8rer.<\/p>\n<h2>Hvordan begynne \u00e5 bruke det?<\/h2>\n<p>\nDriveren st\u00f8ttes i Kubernetes versjon 1.15 og h\u00f8yere. For at driveren skal fungere, m\u00e5 f\u00f8lgende krav v\u00e6re oppfylt:<\/p>\n<ul>\n<li> Flagget <code>--allow-privileged<\/code> innstilt til <code>true<\/code> for API-serveren og kubelet;<\/li>\n<li> Aktiverte <code>--feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true<\/code> for API-serveren og kubelet;<\/li>\n<li> Monteringsspredning (<noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/volumes\/#mount-propagation\">mount propagation<\/a><\/noindex>) m\u00e5 v\u00e6re aktivert i klyngen. N\u00e5r du bruker Docker, m\u00e5 daemonen konfigureres slik at delte monteringsobjekter (shared mounts) er tillatt.<\/li>\n<\/ul>\n<p>\nAlle n\u00f8dvendige trinn for selve installasjonen <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver#installing-driver\">er beskrevet i README<\/a><\/noindex>. Installasjonen inneb\u00e6rer opprettelse av objekter i Kubernetes fra manifester.<\/p>\n<p>For at driveren skal fungere, trenger du f\u00f8lgende:<\/p>\n<ul>\n<li> Angi i manifestet katalog-ID (<code>folder-id<\/code>) fra Yandex.Cloud (<noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/resource-manager\/operations\/folder\/get-id\">se dokumentasjonen<\/a><\/noindex>);<\/li>\n<li> For \u00e5 samhandle med Yandex.Cloud API i CSI-driveren, brukes en tjenestekonto. I Secret-manifestet m\u00e5 du overf\u00f8re <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.com\/docs\/iam\/concepts\/authorization\/key\">autoriserte n\u00f8kkel<\/a><\/noindex> fra tjenestekontoen. I dokumentasjonen <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/iam\/quickstart-sa\">beskrevet<\/a><\/noindex>, hvordan du oppretter en tjenestekonto og f\u00e5r n\u00f8kler.<\/li>\n<\/ul>\n<p>\nGenerelt \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">pr\u00f8v<\/a><\/noindex>, og vi vil v\u00e6re glade for tilbakemelding og <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\/issues\">nye issues<\/a><\/noindex>, hvis du st\u00f8ter p\u00e5 noen problemer!<\/p>\n<h2>Videre st\u00f8tte<\/h2>\n<p>\nSom en konklusjon vil vi gjerne p\u00e5peke at denne CSI-driveren ikke ble utviklet av stor lyst til \u00e5 leke med applikasjonsprogrammering i Go, men ut fra et sterkt behov i selskapet. \u00c5 opprettholde v\u00e5r egen implementasjon synes vi ikke er b\u00e6rekraftig, s\u00e5 hvis Yandex viser interesse og \u00f8nsker \u00e5 fortsette \u00e5 st\u00f8tte driveren, vil vi gjerne overf\u00f8re depotet til deres disposisjon.<\/p>\n<p>Dessuten har Yandex sannsynligvis sin egen implementasjon av CSI-driveren i den administrerte Kubernetes-klyngen, som kan lanseres som Open Source. En slik utvikling ser vi ogs\u00e5 som gunstig \u2014 fellesskapet kan dra nytte av en velpr\u00f8vd driver fra tjenesteleverand\u00f8ren, og ikke fra et tredjeparts selskap.<\/p>\n<h2>P.S.<\/h2>\n<p>\nAlso read in our blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/465417\/\">Volumplugins for lagring i Kubernetes: fra Flexvolume til CSI<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/424211\/\">Forst\u00e5 Container Storage Interface (i Kubernetes og andre steder)<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/455543\/\">Er det enkelt og praktisk \u00e5 sette opp et Kubernetes-kluster? Vi lanserer addon-operator<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/455543\/\">Utvider og supplere Kubernetes (oversikt og video av foredraget)<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Kilde: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/486190\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f \u00ab\u0424\u043b\u0430\u043d\u0442\u00bb \u043f\u043e\u043f\u043e\u043b\u043d\u044f\u0435\u0442 \u0441\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434 \u0432 Open Source-\u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u0434\u043b\u044f Kubernetes, \u0432\u044b\u043f\u0443\u0441\u0442\u0438\u0432 \u0430\u043b\u044c\u0444\u0430-\u0432\u0435\u0440\u0441\u0438\u044e \u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 CSI (Container Storage Interface) \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430. \u041d\u043e \u043f\u0435\u0440\u0435\u0434 \u0442\u0435\u043c, \u043a\u0430\u043a \u043f\u0435\u0440\u0435\u0439\u0442\u0438 \u043a \u0434\u0435\u0442\u0430\u043b\u044f\u043c \u0440\u0435\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438, \u043e\u0442\u0432\u0435\u0442\u0438\u043c \u043d\u0430 \u0432\u043e\u043f\u0440\u043e\u0441, \u0437\u0430\u0447\u0435\u043c \u044d\u0442\u043e \u0432\u043e\u043e\u0431\u0449\u0435 \u043d\u0443\u0436\u043d\u043e, \u043a\u043e\u0433\u0434\u0430 \u0443 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 \u0443\u0436\u0435 \u0435\u0441\u0442\u044c \u0443\u0441\u043b\u0443\u0433\u0430 Managed Service for Kubernetes. \u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u0417\u0430\u0447\u0435\u043c \u044d\u0442\u043e? \u0412\u043d\u0443\u0442\u0440\u0438 \u043d\u0430\u0448\u0435\u0439 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438, \u0435\u0449\u0451 \u0441 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":40722,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-40721","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/no\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"nn_NO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0430\u0448 \u043e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 CSI-\u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 \u0432 Kubernetes \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/no\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-03T11:41:59+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-03T11:41:59+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47V\u00e5r erfaring med \u00e5 utvikle CSI-driveren i Kubernetes for Yandex.Cloud | ProHoster","description":"Gleder oss over \u00e5 kunngj\u00f8re at selskapet.","canonical_url":"https:\/\/prohoster.info\/no\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"nn_NO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0430\u0448 \u043e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 CSI-\u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 \u0432 Kubernetes \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430 | ProHoster","og:description":"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.","og:url":"https:\/\/prohoster.info\/no\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-03T11:41:59+00:00","article:modified_time":"2020-02-03T11:41:59+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"40721","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:32:40","updated":"2022-09-30 15:42:14","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/40721","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/comments?post=40721"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/40721\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/media\/40722"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/media?parent=40721"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/categories?post=40721"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/tags?post=40721"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}