{"id":94739,"date":"2020-09-20T19:42:27","date_gmt":"2020-09-20T17:42:27","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/prakticheskij-primer-podklyucheniya-hranilishha-na-baze-ceph-v-klaster-kubernetes"},"modified":"2021-01-02T13:05:08","modified_gmt":"2021-01-02T11:05:08","slug":"prakticheskij-primer-podklyucheniya-hranilishha-na-baze-ceph-v-klaster-kubernetes","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/prakticheskij-primer-podklyucheniya-hranilishha-na-baze-ceph-v-klaster-kubernetes","title":{"rendered":"A practical example of connecting a Ceph-based storage to a Kubernetes cluster","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

The Container Storage Interface (CSI) is a unified interface for interaction between Kubernetes and storage systems. We've briefly covered it already, talked<\/a><\/noindex>and today we will take a closer look at the connection between CSI and Ceph: we'll demonstrate how to connect Ceph storage<\/a><\/noindex> to a Kubernetes cluster.
\nThis article provides real, albeit slightly simplified examples for easier understanding. We will not discuss the installation and configuration of Ceph and Kubernetes clusters.<\/p>\n

Are you interested in how this works?<\/p>\n

\"A<\/p>\n

<\/a><\/noindex><\/p>\n

So, you have a Kubernetes cluster set up, for example, using kubespray.<\/a><\/noindex>Nearby, there is a Ceph cluster \u2014 it can also be deployed using, for example, this set of playbooks.<\/a><\/noindex>I hope it\u2019s unnecessary to mention that for production, there should be a network between them with a bandwidth of at least 10 Gbps.<\/p>\n

If you have all this, let's get started!<\/p>\n

First, let's log in to one of the Ceph cluster nodes and check that everything is fine:<\/p>\n

ceph health\nceph -s<\/code><\/pre>\n
Next, we will create a pool for RBD disks:<\/p>\n
ceph osd pool create kube 32\nceph osd pool application enable kube rbd<\/code><\/pre>\n
Now, let's move to the Kubernetes cluster. The first thing we'll do is install the Ceph CSI driver for RBD. We will install it as we should, using Helm.
\nWe add the repository containing the chart and get a set of chart variables for ceph-csi-rbd:<\/p>\n
helm repo add ceph-csi https:\/\/ceph.github.io\/csi-charts\nhelm inspect values ceph-csi\/ceph-csi-rbd > cephrbd.yml<\/code><\/pre>\n
Now we need to fill in the cephrbd.yml file. To do this, we will find out the cluster ID and the IP addresses of the monitors in Ceph:<\/p>\n
ceph fsid  # this will show us the clusterID\nceph mon dump  # this will show the IP addresses of the monitors<\/code><\/pre>\n
We will insert the obtained values into the cephrbd.yml file. Simultaneously, we will enable the creation of PSP (Pod Security Policies). The options in the sections nodeplugin,<\/strong> and provisioner,<\/strong> are already in the file, and can be modified as shown below:<\/p>\n
csiConfig:\n  - clusterID: \"bcd0d202-fba8-4352-b25d-75c89258d5ab\"\n    monitors:\n      - \"v2:172.18.8.5:3300\/0,v1:172.18.8.5:6789\/0\"\n      - \"v2:172.18.8.6:3300\/0,v1:172.18.8.6:6789\/0\"\n      - \"v2:172.18.8.7:3300\/0,v1:172.18.8.7:6789\/0\"\n\nnodeplugin:\n  podSecurityPolicy:\n    enabled: true\n\nprovisioner:\n  podSecurityPolicy:\n    enabled: true<\/code><\/pre>\n
Next, all that remains is to install the chart in the Kubernetes cluster.<\/p>\n
helm upgrade -i ceph-csi-rbd ceph-csi\/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace<\/code><\/pre>\n
Great, the RBD driver is working!
\nLet's create a new StorageClass in Kubernetes. This will require a little more work with Ceph.<\/p>\n
We will create a new user in Ceph and grant it write access to the pool kube.<\/strong>:<\/p>\n
ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'<\/code><\/pre>\n
Now let's view the access key right there:<\/p>\n
ceph auth get-key client.rbdkube<\/code><\/pre>\n
The command will output something like:<\/p>\n
AQCO9NJbhYipKRAAMqZsnqqS\/T8OYQX20xIa9A==<\/code><\/pre>\n
We will insert this value into a Secret in the Kubernetes cluster \u2014 where it's needed userKey.<\/strong>:<\/p>\n
---\napiVersion: v1\nkind: Secret\nmetadata:\n  name: csi-rbd-secret\n  namespace: ceph-csi-rbd\nstringData:\n  # The key values correspond to the username and its key, as specified in\n  # the Ceph cluster. The user ID must have access to the pool,\n  # specified in the storage class\n  userID: rbdkube\n  userKey:<\/code><\/pre>\n
And we create our secret:<\/p>\n
kubectl apply -f secret.yaml<\/code><\/pre>\n
Next, we need a manifest similar to this StorageClass:<\/p>\n
---\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n   name: csi-rbd-sc\nprovisioner: rbd.csi.ceph.com\nparameters:\n   clusterID: \n   pool: kube\n\n   imageFeatures: layering\n\n   # These secrets must contain data for authentication\n   # to your pool.\n   csi.storage.k8s.io\/provisioner-secret-name: csi-rbd-secret\n   csi.storage.k8s.io\/provisioner-secret-namespace: ceph-csi-rbd\n   csi.storage.k8s.io\/controller-expand-secret-name: csi-rbd-secret\n   csi.storage.k8s.io\/controller-expand-secret-namespace: ceph-csi-rbd\n   csi.storage.k8s.io\/node-stage-secret-name: csi-rbd-secret\n   csi.storage.k8s.io\/node-stage-secret-namespace: ceph-csi-rbd\n\n   csi.storage.k8s.io\/fstype: ext4\n\nreclaimPolicy: Delete\nallowVolumeExpansion: true\nmountOptions:\n  - discard<\/code><\/pre>\n
You need to fill in clusterID<\/strong>, which we already learned with the command ceph fsid<\/em>, and apply this manifest in the Kubernetes cluster:<\/p>\n
kubectl apply -f storageclass.yaml<\/code><\/pre>\n
To check the operation of clusters in tandem, let's create a PVC (Persistent Volume Claim) like this:<\/p>\n
apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: rbd-pvc\nspec:\n  accessModes:\n  - ReadWriteOnce\n  resources:\n    requests:\n      storage: 1Gi\n  storageClassName: csi-rbd-sc<\/code><\/pre>\n
Let's immediately see how Kubernetes created the requested volume in Ceph:<\/p>\n
kubectl get pvc\nkubectl get pv<\/code><\/pre>\n
Everything seems great! But how does it look on the Ceph side?
\nWe get a list of volumes in the pool and view information about our volume:<\/p>\n
rbd ls -p kube\nrbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # of course, there will be a different volume ID provided by the previous command<\/code><\/pre>\n
Now, let's see how RBD volume resizing works.
\nWe change the volume size in the pvc.yaml manifest to 2Gi and apply it:<\/p>\n
kubectl apply -f pvc.yaml<\/code><\/pre>\n
Let's wait for the changes to take effect and check the volume size again.<\/p>\n
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653\n\nkubectl get pv\nkubectl get pvc<\/code><\/pre>\n
We see that the size of the PVC has not changed. To find out why, we can request the YAML description of the PVC from Kubernetes:<\/p>\n
kubectl get pvc rbd-pvc -o yaml<\/code><\/pre>\n
And here is the problem:<\/p>\n
message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending<\/em><\/p>\n
This means the disk has increased, but the filesystem on it has not.
\nTo enlarge the filesystem, the volume must be mounted. Our created PVC\/PV is not being used in any way right now.<\/p>\n
We can create a test Pod, for example like this:<\/p>\n
---\napiVersion: v1\nkind: Pod\nmetadata:\n  name: csi-rbd-demo-pod\nspec:\n  containers:\n    - name: web-server\n      image: nginx:1.17.6\n      volumeMounts:\n        - name: mypvc\n          mountPath: \/data\n  volumes:\n    - name: mypvc\n      persistentVolumeClaim:\n        claimName: rbd-pvc\n        readOnly: false<\/code><\/pre>\n
And now let's take a look at the PVC:<\/p>\n
kubectl get pvc<\/code><\/pre>\n
The size has changed, everything is fine.<\/p>\n
In the first part, we worked with the RBD block device (which stands for Rados Block Device), but this approach isn't suitable when multiple microservices need simultaneous access to the disk. For file operations rather than disk images, CephFS is a much better fit.
\nUsing the example of Ceph and Kubernetes clusters, we will configure CSI and the other necessary entities to work with CephFS.<\/p>\n
We will extract values from the required new Helm chart:<\/p>\n
helm inspect values ceph-csi\/ceph-csi-cephfs > cephfs.yml<\/code><\/pre>\n
We need to fill in the cephfs.yml file again. As before, Ceph commands will help:<\/p>\n
ceph fsid\nceph mon dump<\/code><\/pre>\n
We fill in the values file like this:<\/p>\n
csiConfig:\n  - clusterID: \"bcd0d202-fba8-4352-b25d-75c89258d5ab\"\n    monitors:\n      - \"172.18.8.5:6789\"\n      - \"172.18.8.6:6789\"\n      - \"172.18.8.7:6789\"\n\nnodeplugin:\n  httpMetrics:\n    enabled: true\n    containerPort: 8091\n  podSecurityPolicy:\n    enabled: true\n\nprovisioner:\n  replicaCount: 1\n  podSecurityPolicy:\n    enabled: true<\/code><\/pre>\n
Note that the addresses of the monitors are specified in the straightforward address:port format. For mounting cephfs on the node, these addresses are passed to the kernel module, which does not yet support v2 monitor protocols.
\nWe change the port for httpMetrics (where Prometheus will request metrics for monitoring) to avoid conflicts with the nginx-proxy installed by Kubespray. You might not need to do this.<\/p>\n
We install the Helm chart in the Kubernetes cluster:<\/p>\n
helm upgrade -i ceph-csi-cephfs ceph-csi\/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace<\/code><\/pre>\n
Now, let's move to the Ceph data storage to create a separate user there. The documentation states that the CephFS provisioner requires cluster administrator access. However, we will create a separate user fs<\/em> with limited permissions:<\/p>\n
ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'<\/code><\/pre>\n
And let's immediately check its access key, which we will need later:<\/p>\n
ceph auth get-key client.fs<\/code><\/pre>\n
We will create separate Secret and StorageClass.
\nNothing new; we have seen this before with RBD:<\/p>\n
---\napiVersion: v1\nkind: Secret\nmetadata:\n  name: csi-cephfs-secret\n  namespace: ceph-csi-cephfs\nstringData:\n  # Necessary for dynamically created volumes\n  adminID: fs\n  adminKey:<\/code><\/pre>\n
We apply the manifest:<\/p>\n
kubectl apply -f secret.yaml<\/code><\/pre>\n
And now \u2013 a separate StorageClass:<\/p>\n
---\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n  name: csi-cephfs-sc\nprovisioner: cephfs.csi.ceph.com\nparameters:\n  clusterID: \n\n  # Name of the CephFS file system where the volume will be created\n  fsName: cephfs\n\n  # (optional) Pool in Ceph where the volume data will be stored\n  # pool: cephfs_data\n\n  # (optional) Comma-separated mount options for Ceph-fuse\n  # for example:\n  # fuseMountOptions: debug\n\n  # (optional) Comma-separated mount options for CephFS for kernel\n  # See man mount.ceph for a list of these options. For example:\n  # kernelMountOptions: readdir_max_bytes=1048576,norbytes\n\n  # Secrets must contain access tokens for admin and\/or Ceph user.\n  csi.storage.k8s.io\/provisioner-secret-name: csi-cephfs-secret\n  csi.storage.k8s.io\/provisioner-secret-namespace: ceph-csi-cephfs\n  csi.storage.k8s.io\/controller-expand-secret-name: csi-cephfs-secret\n  csi.storage.k8s.io\/controller-expand-secret-namespace: ceph-csi-cephfs\n  csi.storage.k8s.io\/node-stage-secret-name: csi-cephfs-secret\n  csi.storage.k8s.io\/node-stage-secret-namespace: ceph-csi-cephfs\n\n  # (optional) The driver can use either ceph-fuse (fuse), \n  # or ceph kernelclient (kernel).\n  # If not specified, the default volume mounting will be used,\n  # determined by searching for ceph-fuse and mount.ceph\n  # mounter: kernel\nreclaimPolicy: Delete\nallowVolumeExpansion: true\nmountOptions:\n  - debug<\/code><\/pre>\n
Let's fill this in clusterID<\/strong> and apply it in Kubernetes:<\/p>\n
kubectl apply -f storageclass.yaml<\/code><\/pre>\n
Check<\/h2>\n
To verify, as in the previous example, let's create a PVC:<\/p>\n
---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: csi-cephfs-pvc\nspec:\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 5Gi\n  storageClassName: csi-cephfs-sc<\/code><\/pre>\n
And check for the presence of PVC\/PV:<\/p>\n
kubectl get pvc\nkubectl get pv<\/code><\/pre>\n
If you want to look at the files and directories in CephFS, you can mount this file system somewhere. For example, as shown below.<\/p>\n
Let's go to one of the nodes in the Ceph cluster and perform the following actions:<\/p>\n
# \u0422\u043e\u0447\u043a\u0430 \u043c\u043e\u043d\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f\nmkdir -p \/mnt\/cephfs\n\n# \u0421\u043e\u0437\u0434\u0430\u0451\u043c \u0444\u0430\u0439\u043b \u0441 \u043a\u043b\u044e\u0447\u043e\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0430\u0442\u043e\u0440\u0430\nceph auth get-key client.admin >\/etc\/ceph\/secret.key\n\n# \u0414\u043e\u0431\u0430\u0432\u043b\u044f\u0435\u043c \u0437\u0430\u043f\u0438\u0441\u044c \u0432 \/etc\/fstab\n# !! \u0418\u0437\u043c\u0435\u043d\u044f\u0435\u043c ip \u0430\u0434\u0440\u0435\u0441 \u043d\u0430 \u0430\u0434\u0440\u0435\u0441 \u043d\u0430\u0448\u0435\u0433\u043e \u0443\u0437\u043b\u0430\necho \"172.18.8.6:6789:\/ \/mnt\/cephfs ceph name=admin,secretfile=\/etc\/ceph\/secret.key,noatime,_netdev    0       2\" >> \/etc\/fstab\n\nmount \/mnt\/cephfs<\/code><\/pre>\n
Of course, this kind of FS mounting on a Ceph node is only suitable for training purposes, which is what we're doing in our Slurm courses<\/a><\/noindex>. I don't think anyone would do this in production due to the high risk of accidentally deleting important files.<\/p>\n
Lastly, let's check how resizing a volume works in CephFS. We return to Kubernetes and edit our PVC manifest \u2014 let's increase the size, for example, to 7Gi.<\/p>\n
We will apply the edited file:<\/p>\n
kubectl apply -f pvc.yaml<\/code><\/pre>\n
Let's look in the mounted directory to see how the quota has changed:<\/p>\n
getfattr -n ceph.quota.max_bytes<\/code><\/pre>\n
You might need to install the package system for this command to work attr<\/strong>.<\/p>\n
Fear makes the eyes wide, but hands get to work<\/h2>\n
At first glance, all these spells and lengthy YAML manifests may seem complicated, but in practice, Slayer students grasp them quite quickly.
\nIn this article, we did not delve into the intricacies \u2014 for that, there is official documentation. If you're interested in details on configuring Ceph storage with a Kubernetes cluster, these links will help:<\/p>\n
General Principles of Kubernetes with Volumes<\/a><\/noindex>
\nRBD Documentation<\/a><\/noindex>
\nIntegration of RBD and Kubernetes from the Perspective of Ceph<\/a><\/noindex>
\nIntegration of RBD and Kubernetes from the Perspective of CSI<\/a><\/noindex>
\nGeneral Documentation on CephFS<\/a><\/noindex>
\nIntegration of CephFS and Kubernetes from the Perspective of CSI<\/a><\/noindex><\/p>\n
In the Slayer course Kubernetes Base<\/a><\/noindex> , you can go a bit further and deploy a real application in Kubernetes that will use CephFS as storage for files. Through GET\/POST requests, you will be able to transfer files and retrieve them from Ceph.<\/p>\n
If you're more interested in data storage, then sign up for the new course on Ceph<\/a><\/noindex>. While beta testing is ongoing, the course can be obtained at a discount, and you can influence its content.<\/p>\n
Author of the article: Alexander Shvalov, practicing engineer Southbridge<\/a><\/noindex>, Certified Kubernetes Administrator, author and developer of Slayer courses.<\/em><\/p>\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
Container Storage Interface (CSI) \u2014 \u044d\u0442\u043e \u0443\u043d\u0438\u0444\u0438\u0446\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u0432\u0437\u0430\u0438\u043c\u043e\u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f Kubernetes \u0438 \u0441\u0438\u0441\u0442\u0435\u043c \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445. \u0412\u043a\u0440\u0430\u0442\u0446\u0435 \u043e \u043d\u0451\u043c \u043c\u044b \u0443\u0436\u0435 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u043b\u0438, \u0430 \u0441\u0435\u0433\u043e\u0434\u043d\u044f \u043f\u043e\u0434\u0440\u043e\u0431\u043d\u0435\u0435 \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u0441\u0432\u044f\u0437\u043a\u0443 CSI \u0438 Ceph: \u043f\u043e\u043a\u0430\u0436\u0435\u043c, \u043a\u0430\u043a \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u0442\u044c \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 Ceph \u043a \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0443 Kubernetes. \u0412 \u0441\u0442\u0430\u0442\u044c\u0435 \u043f\u0440\u0438\u0432\u0435\u0434\u0435\u043d\u044b \u0440\u0435\u0430\u043b\u044c\u043d\u044b\u0435, \u0445\u043e\u0442\u044f \u0438 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u0443\u043f\u0440\u043e\u0449\u0451\u043d\u043d\u044b\u0435 \u0434\u043b\u044f \u0443\u0434\u043e\u0431\u0441\u0442\u0432\u0430 \u0432\u043e\u0441\u043f\u0440\u0438\u044f\u0442\u0438\u044f \u043f\u0440\u0438\u043c\u0435\u0440\u044b. \u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0443 \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0443 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u0432 Ceph \u0438 Kubernetes […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":94740,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-94739","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47 Practical example of connecting Ceph-based storage to a Kubernetes cluster | ProHoster","description":"Container Storage Interface (CSI) is a standardized interface for communication between Kubernetes and storage systems.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/prakticheskij-primer-podklyucheniya-hranilishha-na-baze-ceph-v-klaster-kubernetes","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043f\u0440\u0438\u043c\u0435\u0440 \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0430 \u043d\u0430 \u0431\u0430\u0437\u0435 Ceph \u0432 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 Kubernetes | ProHoster","og:description":"Container Storage Interface (CSI) \u2014 \u044d\u0442\u043e \u0443\u043d\u0438\u0444\u0438\u0446\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u0432\u0437\u0430\u0438\u043c\u043e\u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f Kubernetes \u0438 \u0441\u0438\u0441\u0442\u0435\u043c \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/prakticheskij-primer-podklyucheniya-hranilishha-na-baze-ceph-v-klaster-kubernetes","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-20T17:42:27+00:00","article:modified_time":"2021-01-02T11:05:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"94739","title":null,"description":"","keywords":"","keyphrases":null,"primary_term":null,"canonical_url":"","og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:19:21","updated":"2022-09-30 10:03:10","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/94739","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=94739"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/94739\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/94740"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=94739"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=94739"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=94739"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}