{"id":93294,"date":"2020-09-05T07:42:58","date_gmt":"2020-09-05T05:42:58","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/logirovanie-v-kubernetes-kak-sobirat-hranit-parsit-i-obrabatyvat-logi"},"modified":"2020-09-05T07:42:58","modified_gmt":"2020-09-05T05:42:58","slug":"logirovanie-v-kubernetes-kak-sobirat-hranit-parsit-i-obrabatyvat-logi","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/logirovanie-v-kubernetes-kak-sobirat-hranit-parsit-i-obrabatyvat-logi","title":{"rendered":"Logging in Kubernetes: How to Collect, Store, Parse, and Process Logs","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Let's dive into the basics of logging in Docker and Kubernetes, and then explore two tools that can confidently be used in production: Grafana Loki and the EFK stack (Elasticsearch + Fluent Bit + Kibana).<\/p>\n

<\/p>\n

The material of this article is a summary of an open lecture from the \"Slurm\" school<\/a><\/noindex>. If there is a desire, and especially a production necessity, you can undergo full training \u2014 sign up for a course on Monitoring and Logging Infrastructure in Kubernetes<\/a><\/noindex>.<\/em><\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/a><\/noindex><\/p>\n

Logging in Docker<\/h2>\n

<\/p>\n

At the Kubernetes level, applications run in pods, but at the lower level, they still typically operate in Docker. Therefore, it is necessary to configure logging in such a way as to collect logs from containers. Since Docker runs the containers, we need to understand how logging works at the Docker level.<\/p>\n

<\/p>\n

I hope every reader knows: application logs should be written to stdout\/stderr, not inside the container. The Docker Daemon aggregates logs, and it works specifically with those logs sent to stdout\/stderr. Moreover, writing logs inside the container can lead to problems: the container bloats due to the growing log (since there is likely no Logrotate in the container), and the Docker Daemon is unaware of this log.<\/p>\n

<\/p>\n

Docker has several logging drivers or plugins for collecting container logs. In the free version, Docker Community Edition (CE) has fewer logging drivers than in the commercial Docker Enterprise Edition (EE).<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

I have never used Docker EE in practice: at Southbridge, we strive to stick to open-source solutions, and most clients do not need the additional features of Docker EE. <\/p>\n

<\/p>\n

Logging drivers in Docker CE:<\/p>\n

<\/p>\n

local<\/strong> \u2014 logging to internal Docker Daemon files;
\njson-file<\/strong> \u2014 creating a json-log in the directory of each container;
\njournald<\/strong> \u2014 sending logs to journald.<\/p>\n

<\/p>\n

Logging settings in Docker are located in the daemon.json file. <\/p>\n

<\/p>\n

In the 'log-driver' field, specify the plugin, and in the 'log-opts' field, specify its settings. In the example above, the 'json-file' plugin is indicated, with a log size limit of 'max-size': '10m'; a file count limit (rotation settings) of 'max-file': '3'; as well as values that will be attached to the logs.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

Some logging driver settings can be set through the command utility. This is convenient if a particular container needs to be started with a different logging driver. <\/p>\n

<\/p>\n

Here is what the logging scheme in Docker looks like:<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

How the scheme works: the logging driver, for example, json-file, creates files. Log collectors (Rsyslog, Fluentd, Logagent, and others) collect these files and forward them for storage in Elastic, Sematext, or other storage solutions.<\/p>\n

<\/p>\n

Features of logging in Kubernetes<\/h2>\n

<\/p>\n

Simplified, the logging scheme in Kubernetes looks like this: there is a pod, a container is running in it, and the container sends logs to stdout\/stderr. Then Docker creates a file and writes logs, which can then be rotated.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

Let's consider the features of logging in Kubernetes.<\/p>\n

<\/p>\n

Preserving logs between deployments<\/strong>This is a mandatory condition for correct logging setup. If logs are not saved between deployments, the logs from the previous application version will be overwritten with the release of a new version, and restarting the container will also risk log loss. Kubernetes has a flag \u2014previous that allows you to view the application logs prior to the last Pod restart, but not deeper.<\/p>\n

<\/p>\n

Aggregating logs from all instances<\/strong>. If microservices are hosted in the cloud, the cloud provider is responsible for system monitoring. If the microservices are on your own hardware, you need to collect not only the container logs but also the system logs.<\/p>\n

<\/p>\n

Previously, there were no convenient tools for collecting logs both from the system and from microservices. Usually, one tool would collect system logs (e.g., Rsyslog), and a second would collect Docker logs (e.g., journal-bit with the Docker log driver configured for journald). They tried using journal-bit to collect logs from both containers (setting the Docker log driver to write logs to journald) and from the system (in CentOS 7, systemd and journald are already available). The solution works, but it isn't perfect. If there are many logs, journal-bit starts to lag, and messages are lost.<\/p>\n

<\/p>\n

The experiments continued \u2014 and another way was found. In CentOS 7, the main system logs (messages, audit, secure) are duplicated in var-log as files. In Docker, you can also configure log saving to JSON files. Accordingly, these files from CentOS 7 and Docker can be collected together.<\/p>\n

<\/p>\n

Over time, the ELK Stack solution became popular. This is a combination of several tools: Elasticsearch, Logstash, and Kibana.<\/p>\n

<\/p>\n

Elasticsearch stores logs from containers, Logstash collects logs from instances, and Kibana allows for processing the received logs and building graphs from them. For some time, the ELK Stack was actively used, but, in my opinion, its time is fading. I will explain later why.<\/p>\n

<\/p>\n

Adding metadata<\/strong>Pods, applications, and containers can run anywhere. Moreover, a single application can have multiple instances. Logs are recorded in a uniform format, but we need to understand which replica it is, which Pod is writing it, and in which namespace it resides. This is why logs need to have metadata added.<\/p>\n

<\/p>\n

Parsing logs<\/strong>Interestingly, the costs of supporting a logging and monitoring system can exceed the expenses of the primary application. When you have tens or hundreds of thousands of logs flying in per second, this seems reasonable, but you still need to know the limits. One way to find this limit is through log parsing. <\/p>\n

<\/p>\n

Typically, it is not necessary to collect and store all logs; you should only store a portion \u2014 for example, logs with a status of 'warning' or 'error.' If we are talking about logs from nginx or ingress controllers, then only those logs with a status different from 200 should be stored. However, this is not a universal recommendation: if you are building some form of analytics on Nginx logs, then it is evident that they should be collected. <\/p>\n

<\/p>\n

It is not advisable to blindly filter logs because the filtered data may not suffice for proper analytics. On the other hand, perhaps analytics should be conducted at the metrics collection level rather than at the logging level. This way, you won't have to store hundreds of thousands of lines with a 200 status code. One approach is to obtain information about traffic and errors from the metrics of ingress controllers. <\/p>\n

<\/p>\n

In general, you need to think carefully about what you want to store and how long; otherwise, you may end up in a situation where the logging system consumes more resources than the main project.<\/p>\n

<\/p>\n

There is currently no standard solution for logging<\/strong>. Unlike monitoring, where there is one most common solution, Prometheus, logging does not have a standard. <\/p>\n

<\/p>\n

In this lecture, we will examine two tools: one popular and the other gaining popularity. Aside from these, there are others, but we will not address them in this article.<\/p>\n

<\/p>\n

Considering all the aforementioned features, logging in Kubernetes can now be visualized in the following diagram:<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

The container log remains, rotation is in place, but an agent collector appears, which picks up logs and sends them for storage (in the diagram \u2014 to the Logging Backend). The agent operates on each node and is typically running in Kubernetes.<\/p>\n

<\/p>\n

Now let's look at the tools for logging.<\/p>\n

<\/p>\n

\u2014 our new log aggregation system, which helped confirm that all ingesters behaved appropriately during and after the failure.<\/h2>\n

<\/p>\n

\u2014 our new log aggregation system, which helped confirm that all ingesters behaved appropriately during and after the failure.<\/a><\/noindex> has recently emerged but has already become quite well known. Its advantages include easy installation, low resource consumption, and no need to install Elasticsearch since it stores data in a TSDB (time series database). In the previous article, I mentioned that Prometheus stores data in such a database, and this is one of the many similarities between the two products. The developers even claim that Loki is 'Prometheus for the world of logging.'<\/p>\n

<\/p>\n

A brief digression about TSDB for those who haven't read about it. the previous article<\/a><\/noindex>: TSDB is excellent for storing a large amount of data in time series but is not intended for long-term storage. If for some reason you need to keep logs for more than two weeks, it's better to set up their transfer to another database.<\/p>\n

<\/p>\n

Another advantage of Loki is that for data visualization, Grafana is used. It's very convenient: in Grafana, we can view monitoring data and there connect to Loki to see logs. Graphs can be built from logs.<\/p>\n

<\/p>\n

The architecture of Loki looks approximately like this: <\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

A DaemonSet deploys an agent\u2014Promtail or Fluent Bit\u2014on all servers in the cluster. The agent collects logs. Loki retrieves them and stores them in TSDB. Metadata is immediately added to the logs, which is convenient: you can filter by Pods, namespaces, container names, and even labels.<\/p>\n

<\/p>\n

Installation instructions for Loki<\/a><\/noindex><\/p>\n

<\/p>\n

Loki works in the familiar Grafana interface. Loki even has its own query language, called LogQL, which resembles PromQL in Prometheus in name and syntax. The Loki interface provides query suggestions, so there's no need to memorize them.<\/p>\n

<\/p>\n

Documentation for the LogQL language<\/a><\/noindex><\/p>\n

<\/p>\n

\"Logging
\nLoki in the Grafana interface<\/em><\/p>\n

<\/p>\n

Using filters, Loki allows you to find codes (like '400', '404', and any others); view logs from the entire node; and filter all logs containing the word 'error'. Clicking on a log reveals a card with all event information.<\/p>\n

<\/p>\n

Loki has enough tools that allow you to extract the necessary logs, although honestly, there could technically be more. Loki is currently actively developing and gaining popularity. <\/p>\n

<\/p>\n

Elastic + Fluent Bit + Kibana (EFK Stack)<\/h2>\n

<\/p>\n

The EFK stack is a more classic, yet equally popular logging tool. <\/p>\n

<\/p>\n

At the beginning of the article, ELK (Elasticsearch + Logstash + Kibana) was mentioned, but this stack has become outdated due to the resource-intensive and not very performant Logstash. Instead, a lighter and more efficient solution called Fluent Bit<\/a><\/noindex> has emerged as an even lighter and more efficient data collector. <\/p>\n

<\/p>\n

According to the developers, Fluent Bit is over 100 times more performant than Fluentd: \"where Fluentd consumes 20 MB of RAM, Fluent Bit will consume 150 KB\" \u2014 a direct quote from the documentation. Given this, Fluent Bit has become more widely adopted.<\/p>\n

<\/p>\n

Fluent Bit has fewer capabilities than Fluentd, but it addresses the primary needs, which is why we mainly use Fluent Bit.<\/p>\n

<\/p>\n

The EFK stack operates as follows: the agent collects logs from all pods (typically, this is a DaemonSet running on all cluster servers) and sends them to storage (Elasticsearch, PostgreSQL, or Kafka). Kibana connects to the storage to retrieve all the necessary information.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

Kibana<\/a><\/noindex> It presents the information in a user-friendly web interface. There are graphs, filters, and much more.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

You can create entire dashboards from the logs.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

Capabilities of Fluent Bit<\/h2>\n

<\/p>\n

As Fluent Bit is generally less known than Logstash, let's examine it in more detail. Fluent Bit can logically be divided into 6 modules, with some modules supporting plugins that extend Fluent Bit's functionality.<\/p>\n

<\/p>\n

\"Logging<\/p>\n

<\/p>\n

Input Module<\/strong> collects logs from files, systemd services, and even tcp-sockets (you just need to specify the endpoint, and Fluent Bit will start accessing it). These capabilities are sufficient to collect logs from both the system and containers.<\/p>\n

<\/p>\n

In production, we most often use plugins tail<\/a><\/noindex> (you can point it at a folder with logs) and systemd<\/a><\/noindex> (you can specify which services to collect logs from).<\/p>\n

<\/p>\n

Parser Module<\/strong> standardizes the logs. By default, Nginx logs are strings. Using a plugin, you can convert this string into JSON: specifying fields and their values. Working with JSON is much easier than with string logs, as it allows for more flexible sorting options.<\/p>\n

<\/p>\n

Filter Module<\/strong>. At this level, unnecessary logs are filtered out. For example, only logs with a \"warning\" value or certain labels are sent for storage. The selected logs are buffered.<\/p>\n

<\/p>\n

Buffer Module<\/strong>Fluent Bit has two types of buffers: a memory buffer and a disk buffer. A buffer is a temporary storage for logs needed in case of errors or failures. Everyone wants to save on RAM, so a disk buffer is usually chosen. However, it is important to keep in mind that before logging goes to disk, it is still unloaded into memory.<\/p>\n

<\/p>\n

Routing\/Output Module<\/strong> contains the rules and addresses for sending logs. As mentioned, logs can be sent to Elasticsearch, PostgreSQL, or, for example, Kafka. <\/p>\n

<\/p>\n

Interestingly, logs can be sent from Fluent Bit to Fluentd. Since the former is more lightweight and less functional, it can gather logs and send them to Fluentd, where they can be further processed and sent to storage with the help of additional plugins.<\/p>\n

<\/p>\n

If you plan to use Elasticsearch...<\/p>\n

Finally, two tips for those who plan to use Elasticsearch as a log storage in production. <\/p>\n

    \n
  1. Set up alerts using ElastAlert<\/a><\/noindex>. This program extracts important messages from the general log stream and creates alerts via email or another channel. However, not so long ago there was a sad news that the project may soon cease to exist.<\/a><\/noindex><\/em>.<\/li>\n
  2. Rotate logs using the application Curator<\/a><\/noindex> or by calling the Elasticsearch API. Elastic, in general, is currently making significant strides in managing index lifecycles without using third-party tools. Overall, there is no sense in storing logs for too long: it is unlikely that any log will be needed after two weeks \u2014 if it's truly critical, it will definitely have been processed within that time. As a last resort, old logs can be archived and sent somewhere for long-term storage. I have heard of special logs that must be kept by law for up to 5 years. Personally, I have not encountered such a situation, but I would not equate this information to regular logs, and I might even store them separately.<\/li>\n<\/ol>\n

    \n<\/p><\/blockquote>\n

    To be continued...<\/p>\n

    <\/p>\n

    Author: Marsel Ibryaev, certified Kubernetes administrator, practicing engineer at Southbridge<\/a><\/noindex>, speaker and course developer Sleurm<\/a><\/noindex>.<\/em><\/p>\n

    Source: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

    \u0420\u0430\u0437\u0431\u0435\u0440\u0451\u043c \u043e\u0441\u043d\u043e\u0432\u044b \u043b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0432 Docker \u0438 Kubernetes, \u0430 \u0437\u0430\u0442\u0435\u043c \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u0434\u0432\u0430 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0436\u043d\u043e \u0441\u043c\u0435\u043b\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u043d\u0430 \u043f\u0440\u043e\u0434\u0430\u043a\u0448\u0435\u043d\u0435: Grafana Loki \u0438 \u0441\u0442\u0435\u043a EFK (Elasticsearch + Fluent Bit + Kibana). \u041c\u0430\u0442\u0435\u0440\u0438\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 \u2014 \u0432\u044b\u0436\u0438\u043c\u043a\u0430 \u0438\u0437 \u043e\u0442\u043a\u0440\u044b\u0442\u043e\u0439 \u043b\u0435\u043a\u0446\u0438\u0438 \u0448\u043a\u043e\u043b\u044b \u00ab\u0421\u043b\u0451\u0440\u043c\u00bb. \u0415\u0441\u043b\u0438 \u0435\u0441\u0442\u044c \u0436\u0435\u043b\u0430\u043d\u0438\u0435 \u0438 \u0442\u0435\u043c \u0431\u043e\u043b\u0435\u0435 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u0430\u044f \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e\u0441\u0442\u044c \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0439\u0442\u0438 \u043f\u043e\u043b\u043d\u043e\u0435 \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u0435 \u2014 \u0437\u0430\u043f\u0438\u0441\u044b\u0432\u0430\u0439\u0442\u0435\u0441\u044c \u043d\u0430 \u043a\u0443\u0440\u0441 \u043f\u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":93295,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-93294","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Logging in Kubernetes: how to collect, store, parse, and process logs | ProHoster","description":"We will explore the basics of logging in Docker and Kubernetes, and then look at two tools that can be safely used in production: Grafana Loki and the EFK stack (Elasticsearch + Fluent Bit + Kibana)..","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/logirovanie-v-kubernetes-kak-sobirat-hranit-parsit-i-obrabatyvat-logi","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0432 Kubernetes: \u043a\u0430\u043a \u0441\u043e\u0431\u0438\u0440\u0430\u0442\u044c, \u0445\u0440\u0430\u043d\u0438\u0442\u044c, \u043f\u0430\u0440\u0441\u0438\u0442\u044c \u0438 \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0442\u044c \u043b\u043e\u0433\u0438 | ProHoster","og:description":"\u0420\u0430\u0437\u0431\u0435\u0440\u0451\u043c \u043e\u0441\u043d\u043e\u0432\u044b \u043b\u043e\u0433\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0432 Docker \u0438 Kubernetes, \u0430 \u0437\u0430\u0442\u0435\u043c \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u0434\u0432\u0430 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0436\u043d\u043e \u0441\u043c\u0435\u043b\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u043d\u0430 \u043f\u0440\u043e\u0434\u0430\u043a\u0448\u0435\u043d\u0435: Grafana Loki \u0438 \u0441\u0442\u0435\u043a EFK (Elasticsearch + Fluent Bit + Kibana)..","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/logirovanie-v-kubernetes-kak-sobirat-hranit-parsit-i-obrabatyvat-logi","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-05T05:42:58+00:00","article:modified_time":"2020-09-05T05:42:58+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"93294","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:50:29","updated":"2022-09-27 18:43:48","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/93294","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=93294"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/93294\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/93295"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=93294"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=93294"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=93294"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}