{"id":81173,"date":"2020-05-11T13:42:51","date_gmt":"2020-05-11T11:42:51","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/production-ready-images-for-k8s"},"modified":"2020-05-11T13:42:51","modified_gmt":"2020-05-11T11:42:51","slug":"production-ready-images-for-k8s","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/production-ready-images-for-k8s","title":{"rendered":"Production-ready images for k8s","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>This story is about how we use containers in a production environment, specifically under Kubernetes. The article focuses on collecting metrics and logs from containers, as well as building images.<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/c7e8df02d845138e489c09c55cd616e6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nWe are from the fintech company Exness, which develops services for online trading and fintech products for B2B and B2C. Our R&amp;D has various teams, with over 100 employees in the development department.<\/p>\n<p>We represent the team responsible for the platform that collects and runs code from our developers. Specifically, we handle the collection, storage, and provision of metrics, logs, and events from applications. Currently, we operate approximately three thousand Docker containers in production, maintain a big data storage of 50 TB, and provide architectural solutions built around our infrastructure: Kubernetes, Rancher, and various public cloud providers.\u00a0<\/p>\n<h1>Our motivation<\/h1>\n<p>\nWhat is burning? No one can answer. Where is the fire? It's hard to understand. When did it start? It can be found out, but not immediately.\u00a0<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/3ca2271caf1d02ba14937680cd89d65d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWhy do some containers fail while others are fine? Which container is to blame? Externally, the containers look the same, but inside each has its own Neo.<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/f971583f21e363244f9ea754da2a1f4c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOur developers are skilled professionals. They create good services that bring profit to the company. However, there are hiccups when application containers run amok. One container consumes too much CPU, another uses too much network, a third one overwhelms I\/O operations, and the fourth one does something entirely unclear with sockets. Everything collapses, and the ship sinks.\u00a0<\/p>\n<h1>Agents<\/h1>\n<p>\nTo understand what is happening inside, we decided to place agents directly in the containers.<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/cc9eb47d1250bb866ec9452181479df4.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThese agents are containment programs that maintain the containers in a state where they do not break each other. The agents are standardized, allowing for a unified approach to container maintenance.\u00a0<\/p>\n<p>In our case, agents must provide logs in a standardized format, tagged and throttled. They should also offer standardized metrics that are expandable from the perspective of business applications.<\/p>\n<p>Agents also refer to utilities for operation and maintenance, capable of working in various orchestration systems and supporting different images (Debian, Alpine, CentOS, etc.).<\/p>\n<p>Finally, agents must support a simple CI\/CD process that includes Docker files. Otherwise, the ship will fall apart because containers will start to be delivered on 'crooked' rails.<\/p>\n<h1>The build process and the structure of the target image<\/h1>\n<p>\nTo ensure everything is standardized and manageable, we need to adhere to some standard building process. Therefore, we decided to build containers with containers \u2014 such is the recursion.<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/708715af05bf79ce1eb9c2a8ce2aeca6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nHere, containers are represented by solid outlines. We also decided to include distributions in them so that 'life wouldn't be a raspberry.' We'll explain why this was done below.<br \/>\n\u00a0<br \/>\nAs a result, we created a build tool \u2014 a container of a specific version that references specific versions of distributions and specific versions of scripts.<\/p>\n<p>How do we use it? We have a Docker Hub where the container is stored. We mirror it into our system to eliminate external dependencies. The result is a container marked in yellow. We create a template to install all the necessary distributions and scripts in the container. After that, we assemble a production-ready image: developers put their code and some specific dependencies into it.\u00a0<\/p>\n<p>What are the advantages of this approach?\u00a0<\/p>\n<ul>\n<li>First, complete version control of the build tools \u2013 build container, versions of scripts, and distributions.\u00a0<\/li>\n<li>Second, we achieved standardization: we create templates, intermediate, and production-ready images in the same way.\u00a0<\/li>\n<li>Third, containers provide us with portability. Today we use GitLab, and tomorrow we can switch to TeamCity or Jenkins and still run our containers just the same.\u00a0<\/li>\n<li>Fourth, minimizing dependencies. We deliberately included distributions in the container as this allows us to avoid downloading them every time from the Internet.\u00a0<\/li>\n<li>Fifth, the speed of building has increased \u2013 having local copies of images allows us not to waste time downloading since there is a local image.\u00a0<\/li>\n<\/ul>\n<p>\nIn other words, we have achieved a controlled and flexible building process. We use the same tools to build any containers with complete versioning.\u00a0<\/p>\n<h1>How does our building procedure work?<\/h1>\n<p>\n<img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/22b9a8546005e289f79962188dbb0659.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe build is initiated with a single command, and the process runs in the image (highlighted in red). The developer has a Dockerfile (highlighted in yellow), which we render by substituting variables with values. Simultaneously, we add headers and footers \u2014 these are our agents.\u00a0<\/p>\n<p>The header adds distributions from the respective images. The footer installs our services inside, configures the launch of workloads, logging, and other agents, substitutes the entrypoint, etc.\u00a0<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/4004e57dc45771479e535921a839716d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWe deliberated for a long time on whether to use a supervisor. In the end, we decided that we needed it. We chose S6. The supervisor manages the container: it allows connection in case the main process crashes and provides manual control of the container without recreating it. Logs and metrics are processes executed inside the container. They also need to be monitored somehow, and we do this with the help of the supervisor. Finally, S6 takes on housekeeping, signal handling, and other tasks.<\/p>\n<p>Since we use different orchestration systems, after the build and launch, the container must understand which environment it is in and act accordingly. For example:<br \/>\nThis allows us to build a single image and run it in different orchestration systems, taking into account the specifics of that orchestration system.<\/p>\n<p>\u00a0<img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/6e76396afedbbf0e71d44a7fbb8521da.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFor the same container, we get different process trees in Docker and Kubernetes:<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/83f9d23876192930ffebafcdd549915e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe payload is executed under the S6 supervisor. Note the collector and events \u2014 these are our agents responsible for logs and metrics. They are absent in Kubernetes, but present in Docker. Why?\u00a0<\/p>\n<p>If we look at the specification of a 'pod' (hereinafter \u2013 Kubernetes pod), we see that the events container runs in the pod, where there is a separate collector container performing the function of collecting metrics and logs. We can leverage Kubernetes capabilities: launching containers in the same pod, within the same process and\/or network spaces. Essentially, we can integrate our agents and perform certain functions. And if this same container runs in Docker, it will have all the same capabilities, meaning it will be able to deliver logs and metrics since the agents will be running inside.\u00a0<\/p>\n<h1>Metrics and Logs<\/h1>\n<p>\nDelivering metrics and logs is a complex task. Its solution involves several aspects.<br \/>\nThe infrastructure is created to execute workloads, not for mass log delivery. This means this process should be carried out with minimal resource requirements for the containers. We strive to assist our developers: 'Take a Docker Hub container, run it, and we will be able to deliver the logs.'\u00a0<\/p>\n<p>The second aspect is the limitation of log volume. If multiple containers experience a spike in log volume (an application outputs a stack trace in a loop), this increases the load on the CPU, communication channels, log processing system, which impacts the host's overall performance and other containers on the host, sometimes leading to the host 'crashing.'\u00a0<\/p>\n<p>The third aspect is the need to support as many metric collection methods out of the box as possible. From reading files and polling the Prometheus endpoint to using application-specific protocols.<\/p>\n<p>And the final aspect is the need to minimize resource consumption.<\/p>\n<p>We chose an open-source solution written in Go called Telegraf. This is a universal connector that supports over 140 types of input channels (input plugins) and 30 types of output channels (output plugins). We have modified it, and now we will explain how it is used in our example with Kubernetes.\u00a0<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/9635d571c852e5f9358458874c58d7b0.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLet\u2019s say a developer deploys a workload, and Kubernetes receives a request to create a pod. At this moment, a container named Collector is automatically created for each pod (we use a mutation webhook). The Collector is our agent. At startup, this container configures itself to work with Prometheus and the log collection system.<\/p>\n<ul>\n<li>To do this, it uses pod annotations, and depending on its content, it creates, for example, a Prometheus endpoint.\u00a0\n<\/li>\n<li>Based on the pod specification and the specific settings of the containers, it determines how to deliver the logs.\n<\/li>\n<\/ul>\n<p>\nWe collect logs through the Docker API: developers simply need to write them to stdout or stderr, and then the Collector will take care of the rest. Logs are collected in chunks with some delay to prevent possible host overload.\u00a0<\/p>\n<p>Metrics are collected per workload instance (processes) in containers. Everything is tagged: namespace, pod, and so on, and then converted to Prometheus format \u2013 making it available for collection (except for logs). Additionally, we send logs, metrics, and events to Kafka and beyond.<\/p>\n<ul>\n<li>Logs are accessible in Graylog (for visual analysis);\n<\/li>\n<li>Logs, metrics, and events are sent to Clickhouse for long-term storage.\n<\/li>\n<\/ul>\n<p>\nIt works exactly the same in AWS, where we replace Graylog with Kafka with Cloudwatch. We send logs there, and it becomes very convenient: it\u2019s immediately clear which cluster and container they belong to. The same is true for Google Stackdriver. Our architecture works both on-premise with Kafka and in the cloud.\u00a0<\/p>\n<p>If we don't have Kubernetes with pods, the setup becomes a bit more complex, but it operates on the same principles.<\/p>\n<p><img decoding=\"async\" alt=\"Production-ready images for k8s\" src=\"\/wp-content\/uploads\/2020\/05\/dd8efb33c5c74433a2767cb00caeec2d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nInside the container, the same processes run, orchestrated using S6. All the same processes are launched inside one container.<\/p>\n<h1>As a result,<\/h1>\n<p>\nWe created a comprehensive solution for building and deploying images, with options for collecting and delivering logs and metrics:<\/p>\n<ul>\n<li>We developed a standardized approach to building images, which served as the basis for our CI templates;\n<\/li>\n<li>The agents for data collection are our Telegraf extensions. We have thoroughly tested them in production;\n<\/li>\n<li>We apply a mutation webhook to inject containers with agents into the pods;\u00a0\n<\/li>\n<li>We integrated into the Kubernetes\/Rancher ecosystem;\n<\/li>\n<li>We can run the same containers in different orchestration systems and achieve our desired results;\n<\/li>\n<li>We created a fully dynamic container management configuration.\u00a0\n<\/li>\n<\/ul>\n<p>\nCo-author: <b>Ilya Prudnikov<\/b><br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/exness\/blog\/488392\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u042d\u0442\u0430 \u0438\u0441\u0442\u043e\u0440\u0438\u044f \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u0432 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u043e\u0439 \u0441\u0440\u0435\u0434\u0435, \u0432 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0434 Kubernetes. \u0421\u0442\u0430\u0442\u044c\u044f \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u0430 \u0441\u0431\u043e\u0440\u0443 \u043c\u0435\u0442\u0440\u0438\u043a \u0438 \u043b\u043e\u0433\u043e\u0432 \u0441 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0431\u0438\u043b\u0434\u0443 \u043e\u0431\u0440\u0430\u0437\u043e\u0432. \u041c\u044b \u0438\u0437 \u0444\u0438\u043d\u0442\u0435\u0445-\u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Exness, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0437\u0430\u043d\u0438\u043c\u0430\u0435\u0442\u0441\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u043e\u0439 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0434\u043b\u044f \u043e\u043d\u043b\u0430\u0439\u043d-\u0442\u0440\u0435\u0439\u0434\u0438\u043d\u0433\u0430 \u0438 \u0444\u0438\u043d\u0442\u0435\u0445-\u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432 \u0434\u043b\u044f B2B \u0438 \u04122\u0421. \u0412 \u043d\u0430\u0448\u0435\u043c R&amp;D \u043c\u043d\u043e\u0433\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043c\u0430\u043d\u0434, \u0432 \u043e\u0442\u0434\u0435\u043b\u0435 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 100 + \u0441\u043e\u0442\u0440\u0443\u0434\u043d\u0438\u043a\u043e\u0432. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":81174,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-81173","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u042d\u0442\u0430 \u0438\u0441\u0442\u043e\u0440\u0438\u044f \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u0432 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u043e\u0439 \u0441\u0440\u0435\u0434\u0435, \u0432 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0434 Kubernetes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/production-ready-images-for-k8s\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Production-ready images for k8s | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u042d\u0442\u0430 \u0438\u0441\u0442\u043e\u0440\u0438\u044f \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u0432 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u043e\u0439 \u0441\u0440\u0435\u0434\u0435, \u0432 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0434 Kubernetes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/production-ready-images-for-k8s\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-11T11:42:51+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-11T11:42:51+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Production-ready images for k8s | ProHoster","description":"This story is about how we use containers in a product environment, especially under Kubernetes.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/production-ready-images-for-k8s","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Production-ready images for k8s | ProHoster","og:description":"\u042d\u0442\u0430 \u0438\u0441\u0442\u043e\u0440\u0438\u044f \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u0432 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u043e\u0439 \u0441\u0440\u0435\u0434\u0435, \u0432 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0438 \u043f\u043e\u0434 Kubernetes.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/production-ready-images-for-k8s","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-11T11:42:51+00:00","article:modified_time":"2020-05-11T11:42:51+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"81173","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:03:22","updated":"2022-09-28 06:12:17","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/81173","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=81173"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/81173\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/81174"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=81173"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=81173"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=81173"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}