{"id":40721,"date":"2020-02-03T14:41:59","date_gmt":"2020-02-03T11:41:59","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka"},"modified":"2020-02-03T14:41:59","modified_gmt":"2020-02-03T11:41:59","slug":"nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","title":{"rendered":"Our experience in developing a CSI driver in Kubernetes for Yandex.Cloud","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Our experience in developing a CSI driver in Kubernetes for Yandex.Cloud\" src=\"\/wp-content\/uploads\/2020\/02\/86adf77b2ee08c426bf79c5b9b2d3d1e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWe are pleased to announce that Flant has expanded its contribution to Open Source tools for Kubernetes by releasing <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">the alpha version of the CSI driver<\/a><\/noindex> (Container Storage Interface) for Yandex.Cloud.<\/p>\n<p>But before diving into the implementation details, let's address the question of why this is needed at all, given that Yandex already offers <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/managed-kubernetes\/\">Managed Service for Kubernetes<\/a><\/noindex>.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Introduction<\/h2>\n<p><\/p>\n<h3>Why is this needed?<\/h3>\n<p>\nWithin our company, since the very beginning of running Kubernetes in production (i.e., for several years now), we have been developing our own tool (deckhouse), which we also plan to make available as an Open Source project soon. With it, we configure and manage all our clusters uniformly, currently numbering over 100, deployed on various hardware configurations and across all available cloud services.<\/p>\n<p>Clusters using deckhouse incorporate all the components necessary for operation: load balancers, monitoring with convenient graphs, metrics, and alerts, user authentication via external providers for access to all dashboards, and so on. Such an 'enhanced' cluster makes little sense to implement in a managed solution, as often this either isn't possible or would require disabling half of its components.<\/p>\n<p><i><b>NB<\/b>: This is our experience, and it is quite specific. We do not claim that everyone should independently deploy Kubernetes clusters instead of using ready-made solutions. By the way, we do not have any real-world experience with Yandex's Kubernetes service, and we will not make any assessments of this service in the present article.<\/i><\/p>\n<h3>What is this and for whom?<\/h3>\n<p>\nSo, we have already discussed the modern approach to storage in Kubernetes: <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/424211\/\">how CSI works<\/a><\/noindex> and <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/465417\/\">how the community arrived<\/a><\/noindex> at such an approach.<\/p>\n<p>Currently, many large cloud service providers have developed drivers to use their 'cloud' disks as Persistent Volumes in Kubernetes. However, if a provider does not have such a driver but provides all necessary functions through an API, there is nothing preventing us from implementing a driver ourselves. That\u2019s how we came to create one for Yandex.Cloud.<\/p>\n<p>We based our development on <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/digitalocean\/csi-digitalocean\">the CSI driver for DigitalOcean<\/a><\/noindex> and a few ideas from <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-sigs\/gcp-compute-persistent-disk-csi-driver\">the driver for GCP<\/a><\/noindex>, as the interaction with the APIs of these clouds (Google and Yandex) has several similarities. In particular, both APIs and <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/compute\/docs\/api\/how-tos\/api-requests-responses#handling_api_responses\">GCP<\/a><\/noindex>, and <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/api-design-guide\/concepts\/about-async\">Yandex<\/a><\/noindex> return an object <code>Operation<\/code> for tracking the status of long operations (for example, creating a new disk). To interact with the Yandex.Cloud API, the <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/yandex-cloud\/go-sdk\">Yandex.Cloud Go SDK<\/a><\/noindex>.<\/p>\n<p>The result of the work done <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">is published on GitHub<\/a><\/noindex> and may be useful for those who for some reason use their own Kubernetes installation on Yandex.Cloud virtual machines (but not a ready-made managed cluster) and would like to use (order) disks via CSI.<\/p>\n<h2>Implementation<\/h2>\n<p><\/p>\n<h3>Key Features<\/h3>\n<p>\nCurrently, the driver supports the following features:<\/p>\n<ul>\n<li> Ordering disks in all cluster zones according to the topology of the available nodes in the cluster;<\/li>\n<li> Deleting previously ordered disks;<\/li>\n<li> Offline resize for disks (Yandex.Cloud <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/compute\/operations\/disk-control\/update#change-disk-size\">does not support<\/a><\/noindex> increases the disks that are mounted to the virtual machine). For information on how the driver had to be modified to perform the resize as painlessly as possible, see below.<\/li>\n<\/ul>\n<p>\nIn the future, support for creating and deleting snapshots of disks is planned.<\/p>\n<h3>The main challenge and overcoming it<\/h3>\n<p>\nThe absence of the ability to increase disks in real-time in the Yandex.Cloud API is a limitation that complicates the resize operation for PV (Persistent Volume): in this case, it is necessary for the application pod using the disk to be stopped, which can cause application downtime.<\/p>\n<p>According to <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/container-storage-interface\/spec\">CSI specification<\/a><\/noindex>, if the CSI controller reports that it can only perform the disk resize \u2018offline\u2019 (<code>VolumeExpansion.OFFLINE<\/code>), then the process of increasing the disk must proceed as follows:<\/p>\n<blockquote><p>If the plugin has only <code>VolumeExpansion.OFFLINE<\/code> expansion capability and the volume is currently published or available on a node then <code>ControllerExpandVolume<\/code> MUST be called ONLY after either:<\/p>\n<ul>\n<li> The plugin has controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability and <code>ControllerUnpublishVolume<\/code> has been invoked successfully.<\/li>\n<\/ul>\n<p>\nOR ELSE<\/p>\n<ul>\n<li> The plugin does NOT have controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability, the plugin has node <code>STAGE_UNSTAGE_VOLUME<\/code> capability, and <code>NodeUnstageVolume<\/code> has been completed successfully.<\/li>\n<\/ul>\n<p>\nOR ELSE<\/p>\n<ul>\n<li>The plugin does NOT have controller <code>PUBLISH_UNPUBLISH_VOLUME<\/code> capability, nor node <code>STAGE_UNSTAGE_VOLUME<\/code> capability, and <code>NodeUnpublishVolume<\/code> has completed successfully.<\/li>\n<\/ul>\n<\/blockquote>\n<p>\nEssentially, this means that the disk must be detached from the virtual machine before it can be increased.<\/p>\n<p>However, unfortunately, <b>implementation<\/b> the CSI specifications through sidecars do not meet these requirements:<\/p>\n<ul>\n<li> In the sidecar container <code>csi-attacher<\/code>, which should be responsible for ensuring the necessary interval between mounts, this functionality has simply not been implemented during offline resizing. This discussion was initiated <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-csi\/external-attacher\/issues\/207\">here<\/a><\/noindex>.<\/li>\n<li> What is a sidecar container in this context? The CSI plugin itself does not interact with the Kubernetes API; it only responds to gRPC calls sent to it by sidecar containers. The latter <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes-csi.github.io\/docs\/sidecar-containers.html\">are developed<\/a><\/noindex> is managed by the Kubernetes community.<\/li>\n<\/ul>\n<p>\nIn our case (CSI plugin), the disk expansion operation looks like this:<\/p>\n<ol>\n<li> We receive a gRPC call <code>ControllerExpandVolume<\/code>;<\/li>\n<li> We attempt to expand the disk in the API but receive an error about the inability to perform the operation since the disk is mounted;<\/li>\n<li> We save the disk ID in a map containing disks for which the expansion operation needs to be performed. For brevity, we will refer to this map as <code>volumeResizeRequired<\/code>;<\/li>\n<li> We manually delete the pod that uses the disk. Kubernetes will restart it. To prevent the disk from being mounted (<code>ControllerPublishVolume<\/code>) before the expansion operation is complete during the mount attempt, we check that this disk is still in <code>volumeResizeRequired<\/code> and return an error;<\/li>\n<li> The CSI driver attempts to retry the resize operation. If the operation is successful, we remove the disk from <code>volumeResizeRequired<\/code>;<\/li>\n<li> Since the disk ID is not present in <code>volumeResizeRequired<\/code>, <code>ControllerPublishVolume<\/code> is successful, the disk is mounted, and the pod starts running.<\/li>\n<\/ol>\n<p>\nEverything seems quite simple, but as always, there are pitfalls. Disk resizing is handled by <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes-csi.github.io\/docs\/external-resizer.html\">external-resizer<\/a><\/noindex>, which, in case of an error during the operation, <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes-csi\/external-resizer\/blob\/master\/vendor\/k8s.io\/client-go\/util\/workqueue\/default_rate_limiters.go#L41\">uses a queue<\/a><\/noindex> with an exponential increase in timeout up to 1000 seconds:<\/p>\n<pre><code class=\"go\">func DefaultControllerRateLimiter() RateLimiter {\n  return NewMaxOfRateLimiter(\n  NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second),\n  \\\/\\\/ 10 qps, 100 bucket size. This is only for retry speed and its only the overall factor (not per item)\n  &amp;BucketRateLimiter{Limiter: rate.NewLimiter(rate.Limit(10), 100)},\n  )\n}<\/code><\/pre>\n<p>\nThis can periodically lead to the disk expansion operation stretching over 15+ minutes, thus causing the corresponding pod to be unavailable.<\/p>\n<p>The only option that allowed us to reduce potential downtime relatively easily was to use our version of external-resizer with a maximum timeout limit <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/external-resizer\/blob\/faster-workqueue\/pkg\/controller\/controller.go#L81\">of 5 seconds<\/a><\/noindex>:<\/p>\n<pre><code class=\"go\">workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 5*time.Second)<\/code><\/pre>\n<p>\nWe didn\u2019t find it necessary to urgently initiate a discussion and patch external-resizer because offline disk resizing is an artifact that will soon disappear with all cloud providers.<\/p>\n<h2>How to get started?<\/h2>\n<p>\nThe driver is supported in Kubernetes version 1.15 and above. The following requirements must be met for the driver to function:<\/p>\n<ul>\n<li> Flag <code>--allow-privileged<\/code> set to <code>true<\/code> for the API server and kubelet;<\/li>\n<li> Enabled <code>--feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true<\/code> for the API server and kubelet;<\/li>\n<li> Mount propagation (<noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/storage\/volumes\/#mount-propagation\">mount propagation<\/a><\/noindex>) must be enabled in the cluster. When using Docker, the daemon must be configured to allow shared mounts.<\/li>\n<\/ul>\n<p>\nAll necessary installation steps <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver#installing-driver\">are described in README<\/a><\/noindex>. The installation involves creating objects in Kubernetes from manifests.<\/p>\n<p>To use the driver, you'll need the following:<\/p>\n<ul>\n<li> Specify the folder ID in the manifest (<code>folder-id<\/code>) of Yandex.Cloud (<noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/resource-manager\/operations\/folder\/get-id\">see documentation<\/a><\/noindex>);<\/li>\n<li> . To interact with Yandex.Cloud API in the CSI driver, a service account is used. In the Secret manifest, you need to provide <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.com\/docs\/iam\/concepts\/authorization\/key\">authorized keys<\/a><\/noindex> from the service account. The documentation explains <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.yandex.ru\/docs\/iam\/quickstart-sa\">is described<\/a><\/noindex>how to create a service account and obtain the keys.<\/li>\n<\/ul>\n<p>\nIn general, <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\">please try<\/a><\/noindex>, and we would appreciate your feedback and <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/yandex-csi-driver\/issues\">new issues<\/a><\/noindex>, if you encounter any problems!<\/p>\n<h2>Further support<\/h2>\n<p>\nIn conclusion, we would like to note that we developed this CSI driver not out of a strong desire to play with writing applications in Go, but due to a pressing need within the company. We don't see maintaining our own implementation as feasible, so if Yandex shows interest and decides to continue supporting the driver, we would be happy to transfer the repository to their disposal.<\/p>\n<p>Additionally, Yandex probably has its own implementation of a CSI driver in the managed Kubernetes cluster that could be released as Open Source. Such a development option also seems favorable to us \u2014 the community will be able to use a proven driver from the service provider rather than a third-party company.<\/p>\n<h2>P.S.<\/h2>\n<p>\nAlso read in our blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/465417\/\">Volume plugins for storage in Kubernetes: from Flexvolume to CSI<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/424211\/\">Understanding the Container Storage Interface (in Kubernetes and beyond)<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/455543\/\">Is setting up a Kubernetes cluster easy and convenient? Announcing the addon-operator<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/455543\/\">Expanding and complementing Kubernetes (overview and presentation video)<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/486190\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f \u00ab\u0424\u043b\u0430\u043d\u0442\u00bb \u043f\u043e\u043f\u043e\u043b\u043d\u044f\u0435\u0442 \u0441\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434 \u0432 Open Source-\u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u0434\u043b\u044f Kubernetes, \u0432\u044b\u043f\u0443\u0441\u0442\u0438\u0432 \u0430\u043b\u044c\u0444\u0430-\u0432\u0435\u0440\u0441\u0438\u044e \u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 CSI (Container Storage Interface) \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430. \u041d\u043e \u043f\u0435\u0440\u0435\u0434 \u0442\u0435\u043c, \u043a\u0430\u043a \u043f\u0435\u0440\u0435\u0439\u0442\u0438 \u043a \u0434\u0435\u0442\u0430\u043b\u044f\u043c \u0440\u0435\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438, \u043e\u0442\u0432\u0435\u0442\u0438\u043c \u043d\u0430 \u0432\u043e\u043f\u0440\u043e\u0441, \u0437\u0430\u0447\u0435\u043c \u044d\u0442\u043e \u0432\u043e\u043e\u0431\u0449\u0435 \u043d\u0443\u0436\u043d\u043e, \u043a\u043e\u0433\u0434\u0430 \u0443 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 \u0443\u0436\u0435 \u0435\u0441\u0442\u044c \u0443\u0441\u043b\u0443\u0433\u0430 Managed Service for Kubernetes. \u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u0417\u0430\u0447\u0435\u043c \u044d\u0442\u043e? \u0412\u043d\u0443\u0442\u0440\u0438 \u043d\u0430\u0448\u0435\u0439 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438, \u0435\u0449\u0451 \u0441 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":40722,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-40721","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0430\u0448 \u043e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 CSI-\u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 \u0432 Kubernetes \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-03T11:41:59+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-03T11:41:59+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Our experience in developing a CSI driver in Kubernetes for Yandex.Cloud | ProHoster","description":"We are pleased to announce that the company.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0430\u0448 \u043e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 CSI-\u0434\u0440\u0430\u0439\u0432\u0435\u0440\u0430 \u0432 Kubernetes \u0434\u043b\u044f \u042f\u043d\u0434\u0435\u043a\u0441.\u041e\u0431\u043b\u0430\u043a\u0430 | ProHoster","og:description":"\u0420\u0430\u0434\u044b \u043e\u0431\u044a\u044f\u0432\u0438\u0442\u044c, \u0447\u0442\u043e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f.","og:url":"https:\/\/prohoster.info\/en\/blog\/nash-opyt-razrabotki-csi-drajvera-v-kubernetes-dlya-yandeks-oblaka","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-03T11:41:59+00:00","article:modified_time":"2020-02-03T11:41:59+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"40721","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:32:40","updated":"2022-09-30 15:42:14","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/40721","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=40721"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/40721\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/40722"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=40721"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=40721"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=40721"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}