{"id":78676,"date":"2020-04-21T07:42:22","date_gmt":"2020-04-21T05:42:22","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya"},"modified":"2020-04-21T07:42:22","modified_gmt":"2020-04-21T05:42:22","slug":"devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya","title":{"rendered":"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<h1>Part 1: Web \/ Android<\/h1>\n<p>\n <u>Note<\/u>: <i>this article is a translation of the original article\u00a0<\/i><noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/@altunin.alexey.a\/devops-tools-are-not-only-for-devops-building-test-automation-infrastructure-from-scratch-18e351dde938?source=friends_link&amp;sk=54dcbf6ab96c5592d093b50e441f942e\"><i>\u201cDevOps tools are not only for DevOps. Building test automation infrastructure from scratch.\u201d<\/i><\/a><\/noindex><i> However, all illustrations, links, quotes, and terms are preserved in their original language to avoid distorting meaning during the translation. I wish you an enjoyable learning experience!<\/i><\/p>\n<p><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/3d2ffb8331713a18755213cafac02144.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n Currently, the DevOps profession is one of the most sought-after in the IT industry. If you check popular job search websites and filter by salary, you will see that DevOps-related vacancies are at the top of the list. However, it is important to understand that this primarily applies to the 'Senior' position, which implies that the candidate has a high level of skills, knowledge of technologies, and tools. This also comes with a high degree of responsibility related to the seamless operation of production. However, we seem to have forgotten what DevOps really is. Originally, it was not a specific person or department. If you search for definitions of this term, you will find many beautiful and correct nouns, such as methodology, practices, cultural philosophy, group of concepts, and so on.<\/p>\n<p>My specialization is QA automation engineer, but I believe it shouldn't be limited to just writing auto-tests or developing test framework architecture. In 2020, knowledge of automation infrastructure is also essential. It allows for organizing the automation process independently, from launching tests to providing results to all stakeholders in accordance with set objectives. As a result, DevOps skills are a necessary factor for performing this job. And all of this is good, but unfortunately, there is a problem (<i>spoiler: this article attempts to simplify this issue<\/i>). It\u2019s about the fact that DevOps is complex. And that is obvious since companies will not pay a lot for something that is easy to do\u2026 In the world of DevOps, there is a multitude of tools, terms, and practices that one must master. This is especially challenging at the beginning of one\u2019s career and depends on the accumulated technical experience.<\/p>\n<p><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/20712bd4acf881d5346b87f5a2b359d2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nSource: <noindex><a rel=\"nofollow\" href=\"http:\/\/maximelanciauxbi.blogspot.com\/2017\/04\/devops-tools.html\"> http:\/\/maximelanciauxbi.blogspot.com\/2017\/04\/devops-tools.html<\/a><\/noindex><\/p>\n<p>Here we will conclude the introductory part and focus on the purpose of this article.\u00a0<\/p>\n<h3>What this article is about<\/h3>\n<p>\nIn this article, I am going to share my experience in building an infrastructure for test automation. There are many sources of information online about various tools and how to use them, but I would like to consider them exclusively in the context of automation. I believe that many automation engineers are familiar with the situation where the tests created are run by no one but themselves, and no one takes care of maintaining them. As a result, the tests become obsolete and time is spent on updating them. Again, at the beginning of one\u2019s career, this can be quite a difficult task: to properly decide which tools should help solve this problem, how to choose, configure, and maintain them. Some testers seek help from DevOps (the people) and, let\u2019s be honest, this approach works. In many cases, it may be the only option since we lack visibility of all dependencies. But, as we know, DevOps are very busy people, as they must think about the infrastructure of the entire company, deployment, monitoring, microservices, and other similar tasks depending on the organization\/team. As is often the case, automation is not a priority. In this situation, we must try to do everything we can from start to finish. This will reduce dependencies, speed up workflows, enhance our skills, and allow us to see a broader picture of what is happening.<\/p>\n<p>This article presents the most in-demand and popular tools, showing how to use them for a step-by-step construction of an automation infrastructure. Each group is represented by tools that have been tested from personal experience. However, this does not mean you should use the same ones. The tools themselves are not significant; they emerge and become obsolete. Our engineering task is to understand the basic principles: why we need this group of tools and what work tasks we can solve using them. Therefore, at the end of each section, I leave links to similar tools that may be used in your organization.<\/p>\n<h3>What this article lacks<\/h3>\n<p>\nLet me reiterate that this article is not about specific tools, so there will be no code snippets from the documentation or descriptions of specific commands. However, at the end of each section, I provide links for further study.<\/p>\n<p>This is done because:\u00a0<\/p>\n<ul>\n<li>this material is very easy to find in various sources (documentation, books, video courses);<\/li>\n<li>if we dive deeper, we would have to write 10, 20, or 30 parts of this article (whereas the plan is for 2-3);<\/li>\n<li>I simply do not want to waste your time, as you may want to use other tools to achieve the same goals.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Practice<\/h3>\n<p>\nI really hope that this material will be useful for every reader, rather than just read and forgotten. In any learning, practice is a very important component. For this, I have prepared<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/AlexeyAltunin\/devops-tools-for-qa-automation-infrastructure\"> a GitHub repository with a step-by-step guide on how to do everything from scratch<\/a><\/noindex>. You will also have homework to ensure that you do not mindlessly copy lines of commands. <\/p>\n<h3>Plan<\/h3>\n<p><\/p>\n<p><strong>Step<\/strong><br \/>\n<strong>Technology<\/strong><br \/>\n<strong>Tools<\/strong><\/p>\n<p><strong>1<\/strong><br \/>\nLocal running (prepare web \/ android demo tests and run it locally)\u00a0<br \/>\nNode.js, Selenium, Appium<\/p>\n<p><strong>2<\/strong><br \/>\nVersion control systems\u00a0<br \/>\nGit<\/p>\n<p><strong>3<\/strong><br \/>\nContainerisation<br \/>\nDocker, Selenium Grid, Selenoid (Web, Android)<\/p>\n<p><strong>4<\/strong><br \/>\nCI \/ CD<br \/>\nGitlab CI<\/p>\n<p><strong>5<\/strong><br \/>\nCloud platforms<br \/>\nGoogle Cloud Platform<\/p>\n<p><strong>6<\/strong><br \/>\nOrchestration<br \/>\nKubernetes<\/p>\n<p><strong>7<\/strong><br \/>\nInfrastructure as code (IaC)<br \/>\nTerraform, Ansible<\/p>\n<p><\/p>\n<h3>Structure of each section<\/h3>\n<p>\nTo maintain the narrative visually, each section is described according to the following plan:<\/p>\n<ul>\n<li>brief description of the technology,<\/li>\n<li>value for automation infrastructure,<\/li>\n<li>illustration of the current state of the infrastructure,<\/li>\n<li>links for study,<\/li>\n<li>similar tools.<\/li>\n<\/ul>\n<p><\/p>\n<h2>1. Local test running<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\nThis is just a preliminary step for launching demo tests locally and for verifying that they pass successfully. In the practical part, Node.js is used, but the programming language and platform are not important; you can use whatever is used in your company.\u00a0<\/p>\n<p>However, for automation tools, I recommend using Selenium WebDriver for web platforms and Appium for Android platforms, respectively, as in the next steps we will be using Docker images that are specifically tailored to work with these tools. Moreover, referring to job requirements, these tools are the most in demand in the market.<\/p>\n<p>As you may have noticed, we are only considering web and Android tests. Unfortunately, iOS is a completely different story (thanks to Apple). I plan to showcase solutions and practices related to iOS in the following parts.<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\nFrom an infrastructure perspective, local execution offers no value. You are just checking that tests work on a local machine in local browsers and simulators. Nonetheless, this is a necessary starting point.<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/504f28744e29938f66be41c32d9a7dc6.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/selenium.dev\/\">Selenium<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"http:\/\/appium.io\/docs\/en\/about-appium\/getting-started\/\">Appium<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/developer.android.com\/studio\/run\/emulator\">Android Emulator<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/SeleniumHQ\/selenium\/tree\/master\/javascript\/node\/selenium-webdriver\/example\">Selenium JS GitHub examples<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/appium\/appium\/tree\/master\/sample-code\/javascript-wd\">Appium JS GitHub examples<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li>any programming language you prefer, combined with Selenium\/Appium for testing;<\/li>\n<li>any tests;<\/li>\n<li>any test runner.<\/li>\n<\/ul>\n<p><\/p>\n<h2>2. Version Control Systems (Git)<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n It will be no surprise to anyone if I say that a version control system is an extremely important part of development, both in teams and individually. Based on various sources, it can be confidently stated that Git is the most popular representative. A version control system offers numerous advantages, such as code sharing, version storage, reverting to previous branches, project history monitoring, and backups. We will not discuss each point in detail, as I am sure you are well acquainted with this and use it in your everyday work. However, if you are not, I recommend pausing reading this article and addressing this gap as soon as possible.<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\n And here you may reasonably ask, \"Why is he telling us about Git? Everyone knows and uses it for both development code and auto-test code.\" You would be absolutely right, but in this article we are discussing infrastructure, and this section serves as a preview for Section 7: \"Infrastructure as Code (IaC).\" For us, this means that the entire infrastructure, including test environments, is described in code, and thus we can apply versioning systems to it and gain similar advantages as for development and automation code.<\/p>\n<p>We will explore IaC in more detail in step 7, but even now you can start using Git locally by creating a local repository. The complete picture will expand when we add a remote repository to the infrastructure.<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/f17c1d2b2f18aa814c44f7c443e49416.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.atlassian.com\/git\/tutorials\/what-is-version-control\">What is version control<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/git-scm.com\/\">Git<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/git-scm.com\/book\/en\/v2\/Getting-Started-About-Version-Control\">Getting Started \u2014 About Version Control<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/subversion.apache.org\/\">SVN<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.mercurial-scm.org\/guide\">Mercurial<\/a><\/noindex>\u00a0<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/azure.microsoft.com\/en-us\/services\/devops\/server\/\">Microsoft TFS (now part of Azure DevOps Services)<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h2>3. Containerization (Docker)<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n To demonstrate how containerization has changed the game, let's travel back a few decades. Back then, people would buy and use server machines to run applications. But in most cases, the necessary resources for running those applications were not known in advance. As a result, companies spent money on purchasing expensive, powerful servers, but a part of that capacity was not fully utilized.<\/p>\n<p>The next stage of evolution was virtual machines (VMs), which addressed the issue of spending on unused resources. This technology allowed applications to run independently of one another within a single server by providing fully isolated space. However, unfortunately, any technology has its drawbacks. Running VMs requires a full operating system, which consumes CPU, RAM, storage, and, depending on the OS, license costs need to be considered. These factors affect boot speed and complicate portability.<\/p>\n<p>And here we arrive at containerization. Once again, this technology resolved the previous problem, as containers do not require a full OS, which frees up a significant amount of resources and provides a fast and flexible solution for portability.<\/p>\n<p>Certainly, containerization technology is not new and was first introduced in the late 1970s. During that time, a lot of research, developments, and attempts were made. But it was Docker that adapted this technology and made it accessible to the masses. Nowadays, when we talk about containers, we often mean Docker. When we discuss Docker containers, we are referring to Linux containers. We can use Windows and macOS systems to run containers, but it's important to understand that an additional layer is introduced in such cases. For example, Docker on Mac seamlessly runs containers inside a lightweight Linux VM. We will revisit this topic when discussing running Android emulators within containers, as there is a very important nuance that needs to be explored in more detail.<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\n We've established that containerization and Docker are great. Let's look at this in the context of automation, as every tool or technology must solve some problem. Let's highlight the obvious automation testing issues in the context of UI tests:<\/p>\n<ul>\n<li>a vast number of dependencies when installing Selenium, especially Appium;<\/li>\n<li>compatibility issues between versions of browsers, emulators, and drivers;<\/li>\n<li>the lack of isolated space for browsers\/emulators, which is particularly critical for parallel execution;<\/li>\n<li>it is difficult to manage and maintain when needing to run 10, 50, 100, or even 1000 browsers simultaneously.<\/li>\n<\/ul>\n<p>\nBut since Selenium is the most popular automation tool and Docker is the most popular containerization tool, it should be no surprise that someone has attempted to combine them to create a powerful tool for addressing the aforementioned problems. Let's take a closer look at such solutions.\u00a0<\/p>\n<p><u>Selenium grid in docker<\/u><\/p>\n<p>This tool is the most popular Selenium framework in the world for launching multiple browsers on multiple machines and managing them from a central hub. To start, at least two components need to be registered: Hub and Node(s). The Hub is the central unit that receives all requests from tests and distributes them to the corresponding Nodes. For each Node, we can configure a specific setup, such as specifying the desired browser and its version. However, we still need to take care of the compatible browser drivers and install them on the necessary Nodes. For this reason, the Selenium grid is not used in its pure form, except when we need to work with browsers that cannot be installed on Linux OS. In all other cases, a much more flexible and proper solution would be to use Docker images to run the Selenium grid Hub and Nodes. This approach greatly simplifies node management, as we can select the desired image with already installed compatible versions of browsers and drivers.<\/p>\n<p>Despite the negative reviews regarding its stability, especially when launching a large number of Nodes simultaneously, the Selenium grid remains the most popular tool for running Selenium tests in parallel. It's important to note that various enhancements and modifications to this tool are continuously emerging in the open-source community, addressing various bottlenecks.<\/p>\n<p><u>Selenoid for Web<\/u><\/p>\n<p>This tool is a breakthrough in the world of Selenium, as it works out of the box and has made the lives of many automation engineers significantly easier. First and foremost, it is not just another modification of Selenium Grid. Instead, the developers have created a completely new version of Selenium Hub in Golang, which, combined with lightweight Docker images for various browsers, has propelled the development of test automation. Moreover, with Selenium Grid, we must define all required browsers and their versions in advance, which is not an issue when working with a single browser. However, when multiple supported browsers are involved, Selenoid is the number one solution, thanks to the 'browser on demand' feature. All we need to do is pre-load the necessary browser images and update the configuration file that Selenoid interacts with. Once Selenoid receives a request from the tests, it will automatically start the required container with the needed browser. After the test is completed, Selenoid will terminate the container, thereby freeing up resources for subsequent requests. This approach completely eliminates the known issue of 'node degradation' that we often encounter with Selenium Grid.<\/p>\n<p>But unfortunately, Selenoid is still not a silver bullet. We have obtained the 'browser on demand' feature, but the 'resources on demand' feature is still not available. To use Selenoid, we must deploy it on physical hardware or a VM, which means we need to know in advance how many resources to allocate. I believe this is not a problem for small projects that run 10, 20, or even 30 browsers in parallel. But what if we need 100, 500, 1000, or more? It makes no sense to maintain and pay for such a large number of resources constantly. In sections 5 and 6 of this article, we will discuss solutions that allow scalability, significantly reducing the company's expenses.<\/p>\n<p><u>Selenoid for Android<\/u><\/p>\n<p>After the success of Selenoid as a tool for web automation, people wanted something similar for Android. And it happened \u2013 Selenoid was released with support for Android. From a high-level user perspective, the principle is similar to web automation. The only difference is that instead of containers with browsers, Selenoid runs containers with Android emulators. In my opinion, this is currently the most powerful free tool for running Android tests in parallel.<\/p>\n<p>I would really prefer not to talk about the drawbacks of this tool, as I genuinely like it. However, there are still the same limitations regarding scaling, which also apply to web automation. Additionally, there's another restriction that may come as a surprise if we are setting up the tool for the first time. To run Android images, we need a physical machine or a VM with nested virtualization support. In the practical guide, I demonstrate how to enable this on a Linux VM. However, if you are a macOS user and want to deploy Selenoid locally, then running Android tests will not be possible. But you can always run a Linux VM locally with 'nested virtualization' set up and deploy Selenoid inside.<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n In the context of this article, we will add 2 tools to illustrate the infrastructure. These are Selenium Grid for web tests and Selenoid for Android tests. In the GitHub guide, I will also show how to use Selenoid to run web tests.\u00a0<\/p>\n<p><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/1202ae79eba9cffc965f3a5b2648c755.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.docker.com\/resources\/what-container\">Docker: what container is<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.docker.com\/get-started\">Docker: get started<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/containerjournal.com\/topics\/container-ecosystems\/5-container-alternatives-to-docker\/\">5 Container Alternatives to Docker<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/SeleniumHQ\/selenium\/wiki\/Grid2\">Selenium Grid<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/SeleniumHQ\/docker-selenium\">Selenium docker<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/aerokube.com\/selenoid\/latest\/\">Selenoid<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/aerokube\/selenoid\">Selenoid Github<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/developer.android.com\/studio\/run\/emulator-acceleration#vm-linux\">KVM setup for Android emulators<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Docker-Deep-Dive-Nigel-Poulton-ebook\/dp\/B01LXWQUFF\/ref=pd_sim_351_1\/147-1129855-1615632?_encoding=UTF8&amp;pd_rd_i=B01LXWQUFF&amp;pd_rd_r=f8dcd666-e707-4f85-99c6-24ecec56ea74&amp;pd_rd_w=rMnHL&amp;pd_rd_wg=u17vC&amp;pf_rd_p=bab57536-7c8f-4781-a8ed-3e270b9cd303&amp;pf_rd_r=KN6EXNGBWJ6B9QNJE4M2&amp;psc=1&amp;refRID=KN6EXNGBWJ6B9QNJE4M2\">Docker Deep Dive book<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li>There are other containerization tools, but Docker is the most popular. If you want to try something else, keep in mind that the tools we discussed for parallel running Selenium tests will not work out of the box.\u00a0\u00a0<\/li>\n<li>As mentioned, there are many modifications of Selenium Grid, for example,<noindex><a rel=\"nofollow\" href=\"https:\/\/opensource.zalando.com\/zalenium\/\"> Zalenium<\/a><\/noindex>. <\/li>\n<\/ul>\n<p><\/p>\n<h2>4. CI \/ CD<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n Continuous integration is quite popular in development and is on par with version control systems. Despite this, I feel there's some confusion around the terminology. In this paragraph, I would like to describe three variations of this technology from my perspective. You can find many articles online with various interpretations, and it's perfectly fine if your views differ. What's most important is that you are aligned with your colleagues.<\/p>\n<p>So, there are three terms: CI \u2014 Continuous Integration, CD \u2014 Continuous Delivery, and again CD \u2014 Continuous Deployment.<i>From now on, I will use these terms in English.<\/i>. Each variation adds a few additional steps to your development pipeline. But the word <u>continuous<\/u> (continuous) is the most important. In this context, we mean something that happens from start to finish, without interruptions or manual intervention. Let's take a look at CI &amp; CD and CD in this context.<\/p>\n<ul>\n<li><strong>Continuous Integration \u2013<\/strong> is the initial step of evolution. After new code is sent to the server, we expect to receive quick feedback that everything is fine with our changes. Typically, CI involves running static code analysis tools and unit\/internal API tests. This allows us to get insights about our code just seconds\/minutes later.<\/li>\n<li><strong>Continuous Delivery<\/strong> is a more advanced step where we run integration\/UI tests. However, at this stage, we do not get results as quickly as with CI. First, these types of tests take longer to complete. Secondly, before the run, we need to deploy our changes to a test\/staging environment. Moreover, when it comes to mobile development, there is an additional step for building our application.<\/li>\n<li><strong>Continuous Deployment<\/strong> implies that we automatically release our changes to production if all acceptance tests have passed in earlier stages. In addition to this, after the release stage, we can configure various steps, such as launching smoke tests on production and collecting relevant metrics. Continuous Deployment is only possible with good coverage of automated tests. If any manual interventions are required, including testing, then this is no longer <u>Continuous<\/u> (continuous). Then we can say that our pipeline only adheres to the Continuous Delivery practice.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\n In this section, I must clarify that when we talk about end-to-end UI tests, it implies that we need to deploy our changes and related services to testing environments. Continuous Integration is not applicable for this task, and we must ensure the implementation of at least Continuous Delivery practices. Continuous Deployment also makes sense in the context of UI tests if we plan to run them in production.<\/p>\n<p>And before we look at the illustration of the architectural changes, I want to say a few words about GitLab CI. Unlike other CI\/CD tools, GitLab provides a remote repository and many additional features. Thus, GitLab is more than just CI. It includes code management out of the box, Agile management, CI\/CD pipelines, logging tools, and metric collection. The architecture of GitLab consists of GitLab CI\/CD and GitLab Runner. Here is a brief description from the official site:<\/p>\n<blockquote><p>GitLab CI\/CD is a web application with an API that stores its state in a database, manages projects\/builds, and provides a user interface. GitLab Runner is an application that processes builds. It can be deployed separately and works with GitLab CI\/CD through an API. For running tests, you need both a GitLab instance and a Runner.<\/p><\/blockquote>\n<p><\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/a05ece39b6c6a9aaec4177d4b5242722.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/martinfowler.com\/articles\/continuousIntegration.html\">Martin Fowler: Continuous Integration<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/docs.gitlab.com\/ee\/ci\/\">GitLab CI\/CD<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Continuous-Integration-Improving-Software-Reducing\/dp\/0321336380\">Continuous Integration book<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Continuous-Delivery-Deployment-Automation-Addison-Wesley-ebook-dp-B003YMNVC0\/dp\/B003YMNVC0\/ref=mt_kindle?_encoding=UTF8&amp;me=&amp;qid=\">Continuous Delivery book<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/jenkins.io\/\">Jenkins<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.jetbrains.com\/teamcity\/\">TeamCity<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/travis-ci.com\/\">Travis<\/a><\/noindex>\u00a0<\/li>\n<li>And many others<\/li>\n<\/ul>\n<p><\/p>\n<h2>5. Cloud platforms<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n In this section, we will discuss a popular trend known as 'public clouds'. Despite the tremendous benefits provided by the virtualization and containerization technologies mentioned above, we still require computing resources. Companies either purchase expensive servers or rent data centers, but in such cases, it is necessary to calculate (sometimes unrealistically) how many resources we will need, whether we will use them 24\/7, and for what purposes. For instance, production requires a server that operates round the clock, but do we need similar resources for testing during off-hours? This also depends on the type of testing being performed. Load\/stress tests that we plan to run during non-working hours can be an example, as we want to obtain results the following day. However, round-the-clock server availability is definitely not necessary for end-to-end automated tests and particularly for manual testing environments. In such situations, it would be ideal to access only the resources we need on demand, utilize them, and stop paying when they are no longer required. Moreover, it would be fantastic to have them instantly by making a few mouse clicks or running a couple of scripts. This is the purpose of public clouds. Let\u2019s take a look at the definition:<\/p>\n<blockquote><p>The public cloud is defined as computing services offered by third-party providers over the public Internet, making them available to anyone who wants to use or purchase them. They may be free or sold on-demand, allowing customers to pay only per usage for the CPU cycles, storage, or bandwidth they consume.<\/p><\/blockquote>\n<p>\n There is a common belief that public clouds are expensive. However, their key idea is to reduce company expenses. As previously mentioned, public clouds allow us to get resources on demand and pay only for the time we use them. Additionally, we sometimes forget that employees receive salaries, and specialists are also a costly resource. It is important to consider that public clouds significantly ease the maintenance of infrastructure, allowing engineers to focus on more critical tasks.\u00a0<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\n What specific resources do we need for end-to-end UI testing? Primarily, these are virtual machines or clusters (we will discuss Kubernetes in the next section) for running browsers and emulators. The more browsers and emulators we want to run simultaneously, the more CPU and memory we require, leading to higher costs. Thus, public clouds in the context of test automation allow us to launch a large number (100, 200, 1000 \u2026) of browsers\/emulators on demand, get testing results as quickly as possible, and stop paying for such resource-intensive capabilities.\u00a0<\/p>\n<p>The most popular cloud providers are Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). The practical guide provides examples using GCP, but overall it doesn't matter what you use for automation tasks. They all offer roughly the same functionality. Usually, when choosing a provider, the guide focuses on the entire infrastructure of the company and business requirements, which is beyond the scope of this article. Automation engineers might be more interested in comparing the use of cloud providers with cloud platforms specifically for testing purposes, such as Sauce Labs, BrowserStack, BitBar, and so on. So let's do that! In my opinion, Sauce Labs is the most well-known cloud testing farm, which is why I chose it for comparison.\u00a0<\/p>\n<p><u>GCP vs. Sauce Labs for automation purposes:<\/u><\/p>\n<p>Let's say we need to run 8 web tests and 8 Android tests simultaneously. For this, we will use GCP and launch 2 virtual machines with Selenoid. On the first, we will spin up 8 containers with browsers. On the second \u2013 8 containers with emulators. Let\u2019s take a look at the prices:\u00a0\u00a0<\/p>\n<p><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/e4688fd2e6e5593d2e1c22fc57f59484.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n To run one container with Chrome, we will need <strong>n1-standard-1 <\/strong>machine. In the case of Android, it will be <strong>n1-standard-4 <\/strong>for one emulator. In fact, a more flexible and cheaper method is to set specific custom values for CPU\/Memory, but currently, for comparison with Sauce Labs, this is not critical.<\/p>\n<p>Here are the rates for using Sauce Labs:<\/p>\n<p><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/1367d2b59bbef9603b4b4be166e8d721.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nI assume you've already noticed the difference, but I will still provide a table with calculations for our task:<\/p>\n<p><strong>Required resources<\/strong><br \/>\n<strong>Monthly<\/strong><br \/>\n<strong>Working hours<\/strong><strong>(8 a.m \u2014 8 p.m)<\/strong><br \/>\n<strong>Working hours<\/strong><strong>+ Preemptible<\/strong><\/p>\n<p><strong>GCP for Web<\/strong><br \/>\nn1-standard-1 x 8 = n1-standard-8<br \/>\n$194.18<br \/>\n23 days * 12h * 0.38 = $104.88\u00a0<br \/>\n23 days * 12h * 0.08 = $22.08<\/p>\n<p><strong>Sauce Labs for Web<\/strong><br \/>\nVirtual Cloud 8 parallel tests<br \/>\n$1.559<br \/>\n\u2014<br \/>\n\u2014<\/p>\n<p><strong>GCP for Android<\/strong><br \/>\nn1-standard-4 x 8: n1-standard-16<br \/>\n$776.72<br \/>\n23 days * 12h * 1.52 = $419.52\u00a0<br \/>\n23 days * 12h * 0.32 = $88.32<\/p>\n<p><strong>Sauce Labs for Android<\/strong><br \/>\nReal Device Cloud 8 parallel tests<br \/>\n$1.999<br \/>\n\u2014<br \/>\n\u2014<\/p>\n<p>\nAs you can see, the cost difference is enormous, especially if tests are run only during the working twelve-hour interval. But you can cut costs even more by using preemptible machines. What are they?<\/p>\n<blockquote><p>A preemptible VM is an instance that you can create and run at a much lower price than normal instances. However, Compute Engine might terminate (preempt) these instances if it requires access to those resources for other tasks. Preemptible instances are excess Compute Engine capacity, so their availability varies with usage.<\/p>\n<p>If your apps are fault-tolerant and can withstand possible instance preemptions, then preemptible instances can significantly reduce your Compute Engine costs. For example, batch processing jobs can run on preemptible instances. If some of those instances terminate during processing, the job slows but does not completely stop. Preemptible instances complete your batch processing tasks without placing additional workload on your existing instances and without requiring you to pay full price for additional normal instances.<\/p><\/blockquote>\n<p>\n And this is still not the end! In reality, I\u2019m sure that no one runs tests for 12 hours straight. If that\u2019s the case, you can automatically start and stop virtual machines when they are not needed. Actual usage time can drop to 6 hours a day. Then the payment in the context of our task will shrink to just $11 per month for 8 browsers. Isn\u2019t that wonderful? But with preemptible machines, we need to be cautious and prepared for interruptions and unstable performance, though these situations can be anticipated and handled programmatically. It\u2019s worth it!<\/p>\n<p>But I\u2019m by no means saying \u2018never use cloud testing farms.\u2019 They have a number of advantages. First of all, it\u2019s not just a virtual machine but a complete solution for test automation with a built-in set of features: remote access, logs, screenshots, video recording, various browsers, and physical mobile devices. In many situations, this can be an irreplaceable luxurious alternative. Testing platforms are especially useful for iOS automation, where public clouds can offer only Linux\/Windows systems. But the discussion about iOS will be in the following articles. I always recommend assessing the situation and basing decisions on the tasks: in some cases, it\u2019s cheaper and more efficient to use public clouds, while in others, testing platforms are definitely worth the money.<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/f68770c9e27677b9d762af574726c90e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/gcp\/\">GCP<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/compute\/docs\/instances\/preemptible\">GCP: Preemptible<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/compute\/vm-instance-pricing\">GCP: VM pricing<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/saucelabs.com\/pricing\">SauceLabs<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.inovex.de\/blog\/comparing-cloud-providers\/\">Comparing Cloud Providers<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar tools:<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/\">AWS<\/a><\/noindex>\u00a0<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/azure.microsoft.com\/en-us\/\">Azure<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h2>6. Orchestration<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n I have good news \u2013 we are almost at the end of the article! At this moment, our automation infrastructure consists of web and Android tests, which we run through GitLab CI in parallel, using Docker-supported tools: Selenium grid and Selenoid. Moreover, we utilize virtual machines created via GCP to host containers with browsers and emulators. To reduce costs, we only run these virtual machines on demand and stop them when testing is not in progress. Is there anything else that could enhance our infrastructure? The answer is yes! Meet Kubernetes (K8s)!<\/p>\n<p>First, let\u2019s see how the terms orchestration, cluster, and Kubernetes are interrelated. At a high level, orchestration is a system that deploys and manages applications. For test automation, these containerized applications are Selenium grid and Selenoid. Docker and K8s complement each other. The former is used for deploying applications, while the latter is for orchestration. In turn, K8s acts as a cluster. The task of the cluster is to use VMs as Nodes, enabling the installation of various functionalities, programs, and services within a single server (cluster). If any Node fails, other Nodes will take over, ensuring uninterrupted operation of our application. In addition, K8s has important functionality related to scaling, which allows us to automatically obtain the optimal number of resources based on load and set limits.<\/p>\n<p>To be honest, manually setting up Kubernetes from scratch is quite a non-trivial task. I'll leave a link to a well-known practical guide called 'Kubernetes The Hard Way', and if you're interested, you can practice on your own. However, fortunately, there are alternative methods and tools. The easiest one is to use Google Kubernetes Engine (GKE) in GCP, which will provide a ready-made cluster after just a few clicks. For beginners, I recommend using this approach, as it allows you to focus on learning how to use K8s for your tasks instead of figuring out how the internal components should integrate with each other.\u00a0<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\nLet's consider some significant features provided by K8s:<\/p>\n<ul>\n<li>application deployment: utilizing a multi-node cluster instead of VMs;<\/li>\n<li>dynamic scaling: reduces expenses on resources that are only used on demand;<\/li>\n<li>self-healing: automatic recovery of pods (which in turn restores the containers);<\/li>\n<li>rolling updates and rollbacks without downtime: updating tools, browsers, and emulators does not interrupt the current users' activities.<\/li>\n<\/ul>\n<p>\n But K8s is still not a silver bullet. To understand all the advantages and limitations regarding the tools we are discussing (Selenium grid, Selenoid), let's briefly discuss the structure of K8s. A cluster contains two types of nodes: Master Nodes and Worker Nodes. Master Nodes are responsible for management, deployment, and scheduling decisions. Worker Nodes are where applications run. Nodes also contain the container runtime environment. In our case, this is Docker, which handles container-related operations. However, there are alternative solutions, such as<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/containerd\/containerd\"> containerd<\/a><\/noindex>. It is important to understand that scaling or self-healing does not directly relate to the containers. This is implemented by adding\/removing the number of pods, which in turn contain the containers (usually one container per pod, but depending on the task, there can be more). The high-level hierarchy consists of worker nodes, within which there are pods, and within those are running containers.<\/p>\n<p>Scaling is a key function and can be applied to both nodes within a cluster node pool and pods within a node. There are 2 types of scaling that relate to both nodes and pods. The first type, horizontal, involves increasing the number of nodes\/pods. This type is more preferred. The second type, vertical, involves increasing the sizes of nodes\/pods rather than their quantity.<\/p>\n<p>Now let's look at our tools in the context of the aforementioned terms.<\/p>\n<p><u>Selenium grid<\/u><\/p>\n<p>As mentioned earlier, Selenium grid is a very popular tool, and it's no surprise that it has been containerized. Consequently, it's not surprising that Selenium grid can be deployed in K8s. An example of how to do this can be found in the official K8s repository. As usual, I will attach the links at the end of the section. In addition, the practical guide shows how to do this using Terraform. There is also a guide on how to scale the number of pods that contain containers with browsers. However, the automatic scaling function in the context of K8s is still not a fully clear task. When I started studying, I didn't find any practical guides or recommendations. After several studies and experiments with the support of the DevOps team, we chose the approach of raising containers with the required browsers inside a single pod located within one worker node. This method allows us to implement a strategy for horizontal scaling of nodes by increasing their number. I hope that in the future the situation will change, and we will see more and more descriptions of the best approaches and ready solutions, especially after the release of Selenium grid 4 with a changed internal architecture.<\/p>\n<p><u>Selenoid<\/u>:<\/p>\n<p>Currently, deploying Selenoid in K8s is the biggest disappointment. They are not compatible. Theoretically, we can raise a Selenoid container within a pod, but when Selenoid starts running containers with browsers, they will still be inside the same pod. This makes scaling impossible, and as a result, the operation of Selenoid within the cluster will not differ from its operation within a virtual machine. The end of the story.<\/p>\n<p><u>Moon<\/u>:<\/p>\n<p>Knowing this bottleneck when working with Selenoid, developers released a more powerful tool called Moon. This tool was originally designed to work with Kubernetes and, as a result, it can and should utilize the autoscaling feature. Moreover, I would say that at the moment this is <u>only<\/u> the tool in the world of Selenium that has native K8s cluster support out of the box (<i>no longer, see the next tool <\/i>). The key feature of Moon that provides this support is:\u00a0<\/p>\n<blockquote><p>Completely stateless. Selenoid stores in memory information about currently running browser sessions. If for some reason its process crashes \u2014 then all running sessions are lost. Moon, on the contrary, has no internal state and can be replicated across data centers. Browser sessions remain alive even if one or more replicas go down.<\/p><\/blockquote>\n<p>So, Moon is a great solution, but with one problem: it is not free. The price depends on the number of sessions. You can only run 0-4 sessions for free, which is not particularly useful. However, starting from the fifth session, you will have to pay $5 for each. The situation may vary from company to company, but in our case, using Moon makes no sense. As I described above, we can launch VMs with Selenium Grid on demand or increase the number of Nodes in the cluster. Approximately for one pipeline, we launch 500 browsers and stop all resources after the tests finish. If we used Moon, we would have to pay an additional $500 x 5 = $2500 per month, regardless of how often we run tests. And again, I'm not saying 'don't use Moon.' For your tasks, it may be an indispensable solution, especially if you have many projects\/teams in your organization and need a huge shared cluster for everyone. As always, I will leave a link at the end and recommend that you do all necessary calculations in the context of your task.<\/p>\n<p><u>Callisto<\/u>: (<i><strong>Attention! This is not in the original article and is included only in the Russian translation.<\/strong><\/i>)<\/p>\n<p>As I mentioned, Selenium is a very popular tool, and the IT field is evolving rapidly. While I was working on the translation, a new promising tool called Callisto emerged (hello Cypress and other Selenium killers). It natively operates with K8s and allows running Selenoid containers in pods, distributed across Nodes. Everything works right out of the box, including auto-scaling. Fantastic, but it needs testing. I have already managed to deploy this tool and conduct a few experiments. However, it's too early to draw conclusions; after obtaining long-term results, I may publish a review in future articles. For now, I'm leaving just links for independent research.\u00a0\u00a0<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/p>\n<h3><img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/c9c2ffd21df2a261947df809dcec7108.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Links for Further Study<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/setup\/\">Kubernetes: setup<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kelseyhightower\/kubernetes-the-hard-way\">Kubernetes: the hard way<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/examples\/tree\/master\/staging\/selenium\">Kubernetes: Selenium<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/kubernetes-engine\">GCP: Kubernetes engine<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/aerokube.com\/moon\/latest\/\">Moon<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Kubernetes-Book-Version-November-2018-ebook\/dp\/B072TS9ZQZ\/ref=pd_sim_351_1\/147-1129855-1615632?_encoding=UTF8&amp;pd_rd_i=B072TS9ZQZ&amp;pd_rd_r=489f0d27-dd2c-4e33-b23c-a420affcba5d&amp;pd_rd_w=HLjs5&amp;pd_rd_wg=eGbU7&amp;pf_rd_p=bab57536-7c8f-4781-a8ed-3e270b9cd303&amp;pf_rd_r=G71DN3V8127KHPXQ6JFN&amp;psc=1&amp;refRID=G71DN3V8127KHPXQ6JFN\">The Kubernetes Book<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=MmdGyqNqs9w\">Distributed Automation using Kubernetes<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/SeleniumHQ\/selenium\/wiki\/Selenium-Grid-4\">Selenium Grid 4<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/wrike\/callisto\">Callisto GitHub<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/software-testing.ru\/events\/review\/3317-wrike-techclub\">Wrike TechClub (Callisto)<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/docs.docker.com\/get-started\/swarm-deploy\/\">Docker swarm<\/a><\/noindex>\u00a0<\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/mesosphere.github.io\/marathon\/\">Marathon<\/a><\/noindex>\u00a0<\/li>\n<\/ul>\n<p><\/p>\n<h2>7. Infrastructure as Code (IaC)<\/h2>\n<p><\/p>\n<h3>Brief description of the technology<\/h3>\n<p>\n And here we are at the final section. Usually, this technology and the related tasks do not fall under the responsibility of automation engineers. And there are reasons for that. Firstly, in many organizations, infrastructure issues are managed by the DevOps department, and development teams don't concern themselves too much with what enables the pipeline to function or how to maintain everything related to it. Secondly, let's be honest, the practice of 'Infrastructure as Code (IaC)' is still not widely implemented in many companies. However, it has definitely become a popular trend, and it's important to try to be involved in related processes, approaches, and tools. Or at least stay informed.<\/p>\n<p>Let\u2019s start with the motivation for using this approach. We have already discussed that to run tests in GitlabCI, we need at least the resources to launch a Gitlab Runner. To run containers with browsers\/emulators, we need to allocate a VM or a cluster. In addition to the testing resources, we need a significant amount of power to maintain development, staging, and production environments, which also includes databases, automated schedules, network configurations, load balancers, user permissions, and more. The key issue lies in the efforts required to support all of this. There are several ways in which we can make changes and roll out updates. For instance, in the context of GCP, we can use the UI console in the browser and perform all actions by clicking buttons. An alternative can be using API calls to interact with cloud entities or employing the gcloud command-line utility for the necessary operations. However, when there is a truly large number of various entities and infrastructure elements, it becomes difficult or even impossible to perform all operations manually. Moreover, all these manual actions are uncontrolled. We cannot submit them for review before execution, utilize a version control system, or quickly revert changes that led to an incident. To address such issues, engineers have created and are still creating automated bash\/shell scripts, which are not much better than the previous methods, as they are not particularly easy to read, understand, maintain, and modify quickly in a procedural style.<\/p>\n<p>In this article and practical guide, I use two tools related to IaC practice: Terraform and Ansible. Some believe it makes no sense to use them simultaneously since their functionalities are similar and interchangeable. However, the reality is that they are designed to address completely different tasks. The fact that these tools should complement each other was confirmed during a joint presentation by developers from HashiCorp and RedHat. The conceptual difference is that Terraform is a provisioning tool for managing servers themselves, while Ansible is a configuration management tool aimed at installing, configuring, and managing software on those servers.<\/p>\n<p>Another key distinguishing feature of these tools is their coding style. Unlike Bash and Ansible, Terraform uses a declarative style based on describing the desired end state that needs to be achieved. For instance, if we plan to create 10 VMs and apply changes via Terraform, we will have 10 VMs. If the script is applied again, nothing will happen, as we already have 10 VMs, and Terraform is aware of this because it maintains the current infrastructure state in a state file. On the other hand, Ansible follows a procedural approach, and if we ask it to create 10 VMs, we will get 10 VMs on the first run, just like with Terraform. However, after rerunning, we will end up with 20 VMs. This is the crucial distinction. In the procedural style, we do not store the current state and merely describe the sequence of steps that should be executed. Of course, we can handle different scenarios and add checks for resource existence and current state, but it is pointless to spend our time and effort controlling this logic. Moreover, it increases the risk of making mistakes.\u00a0<\/p>\n<p>Summarizing the above, it can be concluded that Terraform, with its declarative notation, is more suitable for provisioning servers, while configuration management tasks are better delegated to Ansible. Having understood this, let's look at usage examples in the context of automation.<\/p>\n<h3>Value for Automation Infrastructure<\/h3>\n<p>\n It's important to understand that the test automation infrastructure should be viewed as part of the company's overall infrastructure. This means that all IaC practices must be applied globally to the resources of the entire organization. Who is responsible for this depends on your processes. The DevOps team is more experienced in these matters and sees the whole picture. However, QA engineers are more involved in the automation build process and the structure of the pipeline, which allows them to better identify all necessary changes and opportunities for improvement. The best option is to work together, sharing knowledge and ideas to achieve the expected results.\u00a0<\/p>\n<p>Let me provide a few examples of using Terraform and Ansible in the context of test automation and the tools we discussed earlier:<\/p>\n<p>1. Describe the necessary characteristics and parameters of VMs and clusters using Terraform.<\/p>\n<p>2. Install the necessary testing tools with Ansible: Docker, Selenoid, Selenium Grid, and load the required versions of browsers\/emulators.<\/p>\n<p>3. Describe the characteristics of the VM in which the GitLab Runner will be launched using Terraform.<\/p>\n<p>4. Install GitLab Runner and the necessary accompanying tools with Ansible, and set configurations and settings.<\/p>\n<h3>Illustration of the Current State of the Infrastructure<\/h3>\n<p>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/5d793a800a4ef62e181e6cb3a3f0544e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h3>Resources for study:<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.terraform.io\/\">Terraform<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.ansible.com\/\">Ansible<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.hashicorp.com\/resources\/ansible-terraform-better-together\">Ansible and Terraform better together<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/blog.gruntwork.io\/a-comprehensive-guide-to-terraform-b3d32832baca\">A Comprehensive Guide to Terraform<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/blog.gruntwork.io\/why-we-use-terraform-and-not-chef-puppet-ansible-saltstack-or-cloudformation-7989dad2865c\">Why we use Terraform and not Chef, Puppet, Ansible, SaltStack, or CloudFormation<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/solutions\/infrastructure-as-code\">GCP: IaC<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<h3>Similar Tools<\/h3>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.chef.io\/\">Chef<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/puppet.com\/\">Puppet<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/saltstack.com\/\">SaltStack<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/www.vagrantup.com\/\">Vagrant<\/a><\/noindex><\/li>\n<\/ul>\n<h2>Let's summarize!<\/h2>\n<p><\/p>\n<p><strong>Step<\/strong><br \/>\n<strong>Technology<\/strong><br \/>\n<strong>Tools<\/strong><br \/>\n<strong>Value for Automation Infrastructure<\/strong><\/p>\n<p><strong>1<\/strong><br \/>\nLocal running<br \/>\nNode.js, Selenium, Appium<\/p>\n<ul>\n<li>The most popular tools for web and mobile<\/li>\n<li>Support for many languages and platforms (including Node.js)<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>2<\/strong><br \/>\nVersion control systems\u00a0<br \/>\nGit<\/p>\n<ul>\n<li>Similar benefits with development code<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>3<\/strong><br \/>\nContainerisation<br \/>\nDocker, Selenium Grid, Selenoid (Web, Android)<\/p>\n<ul>\n<li>Parallel test execution<\/li>\n<li>Isolated environments<\/li>\n<li>Simple, flexible version updating<\/li>\n<li>Dynamic stopping of unused resources<\/li>\n<li>Easy to set up<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>4<\/strong><br \/>\nCI \/ CD<br \/>\nGitlab CI<\/p>\n<ul>\n<li>Tests are part of the pipeline<\/li>\n<li>Fast feedback<\/li>\n<li>Visibility for the entire company\/team<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>5<\/strong><br \/>\nCloud platforms<br \/>\nGoogle Cloud Platform<\/p>\n<ul>\n<li>On-demand resources (we pay only when we need them)<\/li>\n<li>Easy to manage and update<\/li>\n<li>Visibility and control of all resources<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>6<\/strong><br \/>\nOrchestration<br \/>\nKubernetes<br \/>\nIn the context of containers with browsers\/emulators inside pods:<\/p>\n<ul>\n<li>Scaling \/ auto-scaling<\/li>\n<li>Self-healing<\/li>\n<li>Updates and rollbacks without interruptions<\/li>\n<\/ul>\n<p><\/p>\n<p><strong>7<\/strong><br \/>\nInfrastructure as code (IaC)<br \/>\nTerraform, Ansible<\/p>\n<ul>\n<li>Similar benefits with development infrastructure<\/li>\n<li>All the advantages of versioning code<\/li>\n<li>Easy to make changes and maintain<\/li>\n<li>Fully automated<\/li>\n<\/ul>\n<p><\/p>\n<h2>Mind map diagrams: the evolution of infrastructure<\/h2>\n<p>\nstep1: Local<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/507908ba097a603adc3b301ab2f94a8d.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step2: VCS<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/71c5170293d98375d8ab86d05eb47a33.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step3: Containerization\u00a0<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/da33442a592851e07c3f5800c3ee9ead.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step4: CI\/CD\u00a0<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/900f18b45124412a1ad00d3901efb40e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step5: Cloud Platforms<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/840e5063141ac8787612e2fc46b5d690.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step6: Orchestration<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/4651b552efdc893695cbf359c6992e9e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>step7: IaC<br \/>\n<img decoding=\"async\" alt=\"DevOps tools are not just for DevOps. The process of building test automation infrastructure from scratch\" src=\"\/wp-content\/uploads\/2020\/04\/e784130f570260fc6c3aa263916e3db5.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>What's next?<\/h2>\n<p>\nSo, this is the end of the article. However, I'd like to establish some agreements with you in conclusion.<\/p>\n<p><u>From your side<\/u><br \/>\n As mentioned at the beginning, I want the article to provide practical benefits and help you apply the insights gained in real work. I'm adding again<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/AlexeyAltunin\/devops-tools-for-qa-automation-infrastructure\"> the link to the practical guide<\/a><\/noindex>.<\/p>\n<p>But even after this, do not stop; practice, study the relevant links and books, learn how it works in your company, find areas to improve, and participate in this. Good luck!<\/p>\n<p><u>From my side<\/u><\/p>\n<p>The title indicates that this was only the first part. Despite being quite lengthy, important topics are still not discussed here. In the second part, I plan to explore automation infrastructure in the context of iOS. Due to Apple\u2019s restrictions on launching iOS simulators only on macOS systems, our set of solutions is limited. For example, we are unable to use Docker to run the simulator or public clouds to run virtual machines. But this does not mean there aren't other alternatives. I will try to keep you updated on the latest solutions and modern tools!<\/p>\n<p>I also haven't mentioned quite significant topics related to monitoring. In part 3, I plan to review the most popular tools for infrastructure monitoring, as well as what data and metrics to consider.<\/p>\n<p>And finally. In the future, I plan to release a video course on building test infrastructure and popular tools. Currently, there is a lot of content online about DevOps, but all materials are presented in the context of development, not test automation. In this regard, I really need feedback; would such a course be interesting and valuable for the testing and automation community? Thank you in advance!<\/h3>\n<\/h3>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/497918\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0427\u0430\u0441\u0442\u044c 1: Web \/ Android \u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435: \u0434\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u043e\u043c \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b\u044c\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438\u00a0\u00abDevOps tools are not only for DevOps. Building test automation infrastructure from scratch\u00bb. \u041e\u0434\u043d\u0430\u043a\u043e \u0432\u0441\u0435 \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0430\u0446\u0438\u0438, \u0441\u0441\u044b\u043b\u043a\u0438, \u0446\u0438\u0442\u0430\u0442\u044b \u0438 \u0442\u0435\u0440\u043c\u0438\u043d\u044b \u0441\u043e\u0445\u0440\u0430\u043d\u0435\u043d\u044b \u043d\u0430 \u044f\u0437\u044b\u043a\u0435 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b\u0430, \u0447\u0442\u043e\u0431\u044b \u0438\u0437\u0431\u0435\u0436\u0430\u0442\u044c \u0438\u0441\u043a\u0430\u0436\u0435\u043d\u0438\u044f \u0441\u043c\u044b\u0441\u043b\u0430 \u043f\u0440\u0438 \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u0435 \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a. \u0416\u0435\u043b\u0430\u044e \u0432\u0430\u043c \u043f\u0440\u0438\u044f\u0442\u043d\u043e\u0433\u043e \u0438\u0437\u0443\u0447\u0435\u043d\u0438\u044f! \u0412 \u043d\u0430\u0441\u0442\u043e\u044f\u0449\u0435\u0435 \u0432\u0440\u0435\u043c\u044f \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u044c [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":78677,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-78676","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.1.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0427\u0430\u0441\u0442\u044c 1: Web \/ Android \u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435: \u0434\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u043e\u043c \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b\u044c\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.1.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47DevOps \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u0434\u043b\u044f DevOps. \u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u044f \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0441 \u043d\u0443\u043b\u044f | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0427\u0430\u0441\u0442\u044c 1: Web \/ Android \u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435: \u0434\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u043e\u043c \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b\u044c\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-04-21T05:42:22+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-04-21T05:42:22+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47DevOps tools not just for DevOps. The process of building test automation infrastructure from scratch | ProHoster","description":"Part 1: Web \/ Android Note: this article is a translation of the original article into English","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47DevOps \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u0434\u043b\u044f DevOps. \u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u044f \u0438\u043d\u0444\u0440\u0430\u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u044b \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0441 \u043d\u0443\u043b\u044f | ProHoster","og:description":"\u0427\u0430\u0441\u0442\u044c 1: Web \/ Android \u041f\u0440\u0438\u043c\u0435\u0447\u0430\u043d\u0438\u0435: \u0434\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0430\u0442\u044c\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434\u043e\u043c \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u0438\u0439 \u044f\u0437\u044b\u043a \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b\u044c\u043d\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0438","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/devops-instrumenty-ne-tolko-dlya-devops-proczess-postroeniya-infrastruktury-avtomatizaczii-testirovaniya-s-nulya","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-04-21T05:42:22+00:00","article:modified_time":"2020-04-21T05:42:22+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"78676","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:53:59","updated":"2022-10-05 06:14:03","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/78676","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=78676"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/78676\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/78677"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=78676"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=78676"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=78676"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}