{"id":39249,"date":"2019-10-31T22:28:41","date_gmt":"2019-10-31T19:28:41","guid":{"rendered":"https:\/\/prohoster.info\/blog\/cse-kubernetes-dlya-teh-kto-v-vcloud-e\/"},"modified":"2019-10-31T22:28:41","modified_gmt":"2019-10-31T19:28:41","slug":"cse-kubernetes-dlya-teh-kto-v-vcloud-e","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/cse-kubernetes-dlya-teh-kto-v-vcloud-e","title":{"rendered":"CSE: Kubernetes for those in vCloud,","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"CSE:
\nHello everyone!<\/b><\/p>\n

It so happened that our small team, not to say recently, and certainly not suddenly, has grown to the point of transferring some (and eventually all) products to Kubernetes. <\/p>\n

There were many reasons for this, but our story isn't about the controversy.<\/p>\n

We had a limited selection for the infrastructure foundation. We chose between vCloud Director and vCloud Director. We picked the newer one and decided to start. <\/p>\n

Once again flipping through 'The Hard Way,' I quickly concluded that a tool for automating at least basic processes, such as deployment and sizing, was needed yesterday. A deep dive into Google revealed a product called VMware Container Service Extension (CSE) \u2014 an open-source product that automates the creation and sizing of k8s clusters for those in vCloud.
\n<\/a><\/noindex><\/p>\n

Disclaimer: CSE has its limitations, but it fit our needs perfectly. The solution must also be supported by the cloud provider, but since the server part is also open-source, demand its availability from your nearest manager \ud83d\ude42.<\/p><\/blockquote>\n

To begin using it, an administrator account in the vCloud organization is required, along with a pre-created routed network for the cluster (internet access is necessary from this network during the deployment, so don't forget to configure Firewall\/NAT). Addressing doesn't matter. In this example, we'll take 10.0.240.0\/24.<\/p>\n

\"CSE:<\/p>\n

Since the cluster will need to be managed post-creation, it's advisable to have a VPN with routing to the created network. We use a standard SSL-VPN configured on our organization's Edge Gateway.<\/p><\/blockquote>\n

Next, you need to install the CSE client from which the k8s clusters will be managed. In my case, it's my work laptop and a couple of well-hidden containers that handle automation. <\/p>\n

The client requires Python version 3.7.3 or higher and the installed module vcd-cli<\/a><\/noindex>, so let's install both.<\/p>\n

pip3 install vcd-cli\n\npip3 install container-service-extension\n<\/code><\/pre>\n
\nAfter installation, we check the version of CSE and get the following:<\/p>\n
# vcd cse version\nError: No such command \"cse\".\n<\/code><\/pre>\n
\nUnexpectedly, but fixable. It turned out that CSE needs to be hooked as a module to vcd-cli.
\nTo do this, you first need to log in to vcd-cli in our organization:<\/p>\n
# vcd login MyCloud.provider.com org-dev admin\nPassword: \nadmin logged in, org: 'org-dev', vdc: 'org-dev_vDC01'\n<\/code><\/pre>\n
\nAfter that, vcd-cli will create a configuration file ~\/.vcd-cli\/profiles.yaml<\/i>
\nYou need to add the following at the end:<\/p>\n
extensions:\n  - container_service_extension.client.cse\n<\/code><\/pre>\n
\nAfter that, we check again:<\/p>\n
# vcd cse version\nCSE, Container Service Extension for VMware vCloud Director, version 2.5.0\n<\/code><\/pre>\n
\nThe client installation stage is complete. Let's try to deploy the first cluster.
\nCSE has several sets of usage parameters, all of which can be viewed here.<\/a><\/noindex><\/p>\n
First, let's create keys for passwordless access to the future cluster. This is important because by default, password login to the nodes will be disabled, and if keys are not set, it can lead to a lot of work through the virtual machines' consoles, which is not exactly convenient.<\/p>\n
# ssh-keygen\nGenerating public\/private rsa key pair.\nEnter file in which to save the key (\/root\/.ssh\/id_rsa): \nCreated directory '\/root\/.ssh'.\nEnter passphrase (empty for no passphrase): \nEnter same passphrase again: \nYour identification has been saved in \/root\/.ssh\/id_rsa.\nYour public key has been saved in \/root\/.ssh\/id_rsa.pub.\n<\/code><\/pre>\n
\nLet's try to start creating the cluster:<\/p>\n
vcd cse cluster create MyCluster --network k8s_cluster_net --ssh-key ~\/.ssh\/id_rsa.pub --nodes 3 --enable-nfs<\/code><\/pre>\n
\nIf we receive an error Error: Session has expired or user not logged in. Please re-login.<\/i> \u2014 re-login to vcd-cli in vCloud as described above and try again.<\/p>\n
This time everything is fine, and the cluster creation task has started.<\/p>\n
cluster operation: Creating cluster vApp 'MyCluster' (38959587-54f4-4a49-8f2e-61c3a3e879e0) from template 'photon-v2_k8-1.12_weave-2.3.0' (revision 1)\n<\/code><\/pre>\n
\nThe task will take about 20 minutes to complete, in the meantime, let's discuss the main launch parameters.<\/p>\n
\u2014network \u2014 the network we created earlier.
\n\u2014ssh-key \u2014 the keys we generated, which will be written to the cluster nodes.
\n\u2014nodes n \u2014 The number of Worker nodes in the cluster. There will always be one Master; this is a limitation of CSE.
\n\u2014enable-nfs \u2014 create an additional node for the NFS share under persistent volumes. It's a bit of a niche option; we'll get back to what it does in more detail later.\n<\/p><\/blockquote>\n
 In the meantime, in vCloud, you can visually observe the cluster creation.
\n\"CSE:
\n
\nOnce the cluster creation task is complete, it is ready for operation.<\/p>\n
Let's check the deployment correctness with the command vcd cse cluster info MyCluster<\/i><\/p>\n
\"CSE:
\n
\nNext, we need to get the cluster configuration for use. kubectl<\/i><\/p>\n
# vcd cse cluster config MyCluster > .\/.kube\/config\n<\/code><\/pre>\n
\nAnd we can check the cluster's status using it:<\/p>\n
\"CSE:
\n
\nAt this point, the cluster can be considered conditionally operational, if it weren't for the persistent volumes issue. Since we are in vCloud, using the vSphere Provider won't work. The option \u2014enable-nfs<\/i> is intended to smooth over this inconvenience, but it was not completely successful. Manual reconfiguration is required.<\/p>\n
First, our node needs to create a separate Independent disk in vCloud. This guarantees that our data will not disappear along with the cluster if it is deleted. We will also connect the disk to NFS. <\/p>\n
# vcd disk create nfs-shares-1 100g --description 'Kubernetes NFS shares'\n# vcd vapp attach mycluster nfsd-9604 nfs-shares-1\n<\/code><\/pre>\n
\nAfter that, let's SSH (you did create keys, right?) into our NFS node and finally connect the disk:<\/p>\n
root@nfsd-9604:~# parted \/dev\/sdb\n(parted) mklabel gpt\nWarning: The existing disk label on \/dev\/sdb will be destroyed and all data on\nthis disk will be lost. Do you want to continue?\nYes\/No? yes\n(parted) unit GB\n(parted) mkpart primary 0 100\n(parted) print\nModel: VMware Virtual disk (scsi)\nDisk \/dev\/sdb: 100GB\nSector size (logical\/physical): 512B\/512B\nPartition Table: gpt\nDisk Flags:\n\nNumber  Start   End    Size   File system  Name     Flags\n 1      0.00GB  100GB  100GB               primary\n\n(parted) quit\nroot@nfsd-9604:~# mkfs -t ext4 \/dev\/sdb1\nCreating filesystem with 24413696 4k blocks and 6111232 inodes\nFilesystem UUID: 8622c0f5-4044-4ebf-95a5-0372256b34f0\nSuperblock backups stored on blocks:\n\t32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,\n\t4096000, 7962624, 11239424, 20480000, 23887872\n\nAllocating group tables: done\nWriting inode tables: done\nCreating journal (32768 blocks): done\nWriting superblocks and filesystem accounting information: done\n<\/code><\/pre>\n
\nCreating a directory for data and mounting the new partition there:<\/p>\n
mkdir \/export\necho '\/dev\/sdb1  \/export   ext4  defaults   0 0' >> \/etc\/fstab\nmount -a\n<\/code><\/pre>\n
\nLet's create five test volumes and share them for the cluster:<\/p>\n
>cd \/export\n>mkdir vol1 vol2 vol3 vol4 vol5\n>vi \/etc\/exports\n#Add this to the end of the file\n\/export\/vol1 *(rw,sync,no_root_squash,no_subtree_check)\n\/export\/vol2 *(rw,sync,no_root_squash,no_subtree_check)\n\/export\/vol3 *(rw,sync,no_root_squash,no_subtree_check)\n\/export\/vol4 *(rw,sync,no_root_squash,no_subtree_check)\n\/export\/vol5 *(rw,sync,no_root_squash,no_subtree_check)\n#:wq! ;)\n#Next - export the volumes\n>exportfs -r\n<\/code><\/pre>\n
\nAfter all this magic, we can create a PV and PVC in our cluster like this:
\nPV:<\/p>\n
cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-vol1\nspec:\n  capacity:\n    storage: 10Gi\n  accessModes:\n    - ReadWriteMany\n  nfs:\n    # Same IP as the NFS host we SSH'ed to earlier.\n    server: 10.150.200.22\n    path: "\/export\/vol1"\nEOF\n<\/code><\/pre>\n
\nPVC:<\/p>\n
cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: nfs-pvc\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: ""\n  resources:\n    requests:\n      storage: 10Gi\nEOF\n<\/code><\/pre>\n
\nThis concludes the story of creating one cluster and begins the story of its life cycle. As a bonus, here are two useful CSE commands that can help save resources at times:<\/p>\n
#\u0423\u0432\u0435\u043b\u0438\u0447\u0438\u0432\u0430\u0435\u043c \u0440\u0430\u0437\u043c\u0435\u0440 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0434\u043e 8 \u0432\u043e\u0440\u043a\u0435\u0440 \u043d\u043e\u0434\n>cse cluster resize MyCluster --network k8s_cluster_net --nodes 8\n\n#\u0412\u044b\u0432\u043e\u0434\u0438\u043c \u043d\u0435\u043d\u0443\u0436\u043d\u044b\u0435 \u043d\u043e\u0434\u044b \u0438\u0437 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0441 \u0438\u0445 \u043f\u043e\u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u043c \u0443\u0434\u0430\u043b\u0435\u043d\u0438\u0435\u043c\n>vcd cse node delete MyCluster node-1a2v node-6685 --yes\n<\/code><\/pre>\n
\nThank you all for your time, if you have any questions - feel free to ask in the comments.
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u0422\u0430\u043a \u0441\u043b\u043e\u0436\u0438\u043b\u043e\u0441\u044c, \u0447\u0442\u043e \u043d\u0430\u0448\u0430 \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u043a\u043e\u043c\u0430\u043d\u0434\u0430, \u043d\u0435 \u0441\u043a\u0430\u0437\u0430\u0442\u044c, \u0447\u0442\u043e\u0431\u044b \u043d\u0435\u0434\u0430\u0432\u043d\u043e, \u0438 \u0443\u0436 \u0442\u043e\u0447\u043d\u043e \u043d\u0435 \u0432\u043d\u0435\u0437\u0430\u043f\u043d\u043e, \u0434\u043e\u0440\u043e\u0441\u043b\u0430 \u0434\u043e \u043f\u0435\u0440\u0435\u043d\u043e\u0441\u0430 \u043d\u0435\u043a\u043e\u0442\u043e\u0440\u044b\u0445 (\u0430 \u0432 \u043f\u0435\u0440\u0441\u043f\u0435\u043a\u0442\u0438\u0432\u0435 \u0438 \u0432\u0441\u0435\u0445) \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432 \u0432 Kubernetes. \u041f\u0440\u0438\u0447\u0438\u043d \u0442\u043e\u043c\u0443 \u0431\u044b\u043b\u043e \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e, \u043d\u043e \u043d\u0430\u0448\u0430 \u0438\u0441\u0442\u043e\u0440\u0438\u044f \u043d\u0435 \u043f\u0440\u043e \u0445\u043e\u043b\u0438\u0432\u0430\u0440. \u0418\u0437 \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043d\u043e\u0439 \u043e\u0441\u043d\u043e\u0432\u044b \u0432\u044b\u0431\u043e\u0440 \u0443 \u043d\u0430\u0441 \u0431\u044b\u043b \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439. vCloud Director \u0438 vCloud Director. \u0412\u044b\u0431\u0440\u0430\u043b\u0438 \u0442\u043e\u0442, \u0447\u0442\u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":29455,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-39249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47CSE: Kubernetes for those in vCloud | ProHoster","description":"Hello everyone! It has turned out that our.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/cse-kubernetes-dlya-teh-kto-v-vcloud-e","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47CSE: Kubernetes \u0434\u043b\u044f \u0442\u0435\u0445 \u043a\u0442\u043e \u0432 vCloud-\u0435 | ProHoster","og:description":"\u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u0422\u0430\u043a \u0441\u043b\u043e\u0436\u0438\u043b\u043e\u0441\u044c, \u0447\u0442\u043e \u043d\u0430\u0448\u0430.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/cse-kubernetes-dlya-teh-kto-v-vcloud-e","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:28:41+00:00","article:modified_time":"2019-10-31T19:28:41+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"39249","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 01:28:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:53:37","updated":"2026-01-24 01:28:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/39249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=39249"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/39249\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/29455"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=39249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=39249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=39249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}