{"id":87467,"date":"2020-07-08T13:42:11","date_gmt":"2020-07-08T11:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/rukovodstvo-dlya-nachinayushhih-sozdaem-devops-pajplajn"},"modified":"2020-07-08T13:42:11","modified_gmt":"2020-07-08T11:42:11","slug":"rukovodstvo-dlya-nachinayushhih-sozdaem-devops-pajplajn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rukovodstvo-dlya-nachinayushhih-sozdaem-devops-pajplajn","title":{"rendered":"Beginner's Guide: Creating a DevOps Pipeline","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

If you're new to DevOps, check out this guide to creating your first pipeline in five stages.<\/i><\/p>\n

\"Beginner's<\/p>\n

DevOps has become the standard solution for fixing slow, disjointed, or non-functional software development processes. The problem is that if you are a newcomer to DevOps and unsure where to start, you may lack an understanding of these methodologies. This article focuses on defining the DevOps pipeline and providing a five-step guide to create one. Although this tutorial is not exhaustive, it should provide you with a foundation to begin your journey and expand your knowledge in the future. But let's begin with some background.<\/p>\n

My DevOps Journey<\/h3>\n

\nPreviously, I worked in Citi Group's cloud team, developing a web application for Infrastructure-as-a-Service (IaaS) to manage Citi's cloud infrastructure. However, I was always interested in how to make the development process more efficient and bring about positive cultural changes within the development team. The answer I found was in a book recommended by Greg Lavender, Citi's Chief Technology Officer for cloud architecture and infrastructure. The book was titled \"The Phoenix Project\"The Phoenix Project<\/a><\/noindex>, which explains the principles of DevOps while reading like a novel.<\/p>\n

The table on the back of the book shows how frequently various companies deploy their systems in a release environment:<\/p>\n

Amazon: 23,000 times a day
\nGoogle: 5,500 times a day
\nNetflix: 500 times a day
\nFacebook: Once a day
\nTwitter: 3 times a week
\nTypical company: Once every 9 months<\/p>\n

How are the frequencies of Amazon, Google, and Netflix even possible? It's all because these companies figured out how to create an almost perfect DevOps pipeline.
\n<\/a><\/noindex>
\nWe were far from it until we implemented DevOps at Citi. At that time, my team had different environments, but deployment on the development server was entirely manual. All developers had access only to a single development server based on IBM WebSphere Application Server Community Edition. The issue was that the server would shut down every time multiple users attempted to deploy simultaneously, so developers had to communicate their intentions to each other, which was quite painful. Additionally, there were issues with low-level code test coverage, cumbersome manual deployment processes, and a lack of tracking capability for code deployments related to a specific task or user story.<\/p>\n

I realized something needed to be done and found a like-minded colleague. We decided to collaborate on creating the initial DevOps pipeline \u2013 he set up a virtual machine and Tomcat application server while I worked on Jenkins, integrated Atlassian Jira and BitBucket, and focused on code test coverage. This side project was highly successful: we nearly fully automated many processes, achieved almost 100% uptime for our development server, provided tracking and improved code test coverage, and added the ability to link branches in Git with tasks in Jira or deployments. Most of the tools we used to build our DevOps pipeline were open source.<\/p>\n

Now I understand how simple our DevOps pipeline was: we didn't use extensions like Jenkins files or Ansible. Yet, this simple pipeline worked well, possibly due to the Pareto principle (also known as the 80\/20 rule).<\/p>\n

A brief introduction to DevOps and the CI\/CD pipeline<\/h3>\n

\nIf you ask several people, \"What is DevOps?\", you will likely receive a variety of answers. Like Agile, DevOps has evolved to encompass many different disciplines, but most people agree on a few things: DevOps is a software development practice or software development lifecycle (SDLC) where the central principle is to change the culture in which developers and non-developers coexist in an environment where:<\/p>\n

Operations that were previously performed manually are automated;
\nEveryone does what they do best;
\nThe number of deployments over a given period increases; throughputs increase;
\nDevelopment flexibility improves.<\/p>\n

While having the right software tools is not the only requirement for creating a DevOps environment, some tools are essential. A key tool is Continuous Integration and Continuous Deployment (CI\/CD). In this pipeline, environments have various stages (e.g., DEV, INT, TST, QA, UAT, STG, PROD), many operations are automated, and developers can write high-quality code while achieving development flexibility and high deployment frequency.<\/p>\n

This article outlines a five-step approach to building a DevOps pipeline, similar to the one depicted in the following diagram, using open-source tools.<\/p>\n

Step 1: CI\/CD Methods<\/h3>\n

\nThe first thing you need is a tool for CI\/CD. Jenkins, an open-source tool based on Java and distributed under the MIT license, is the means that popularized the DevOps movement and has become the de facto standard.<\/p>\n

So what is Jenkins? Think of it as a kind of magical universal remote that can communicate with various services and tools and organize them. By itself, a CI\/CD tool like Jenkins is useless, but it becomes more powerful as it connects to various tools and services.<\/p>\n

Jenkins is just one of many open-source tools for CI\/CD that you can use to build a DevOps pipeline.<\/p>\n

Jenkins: Creative Commons and MIT
\nTravis CI: MIT
\nCruiseControl: BSD
\nBuildbot: GPL
\nApache Gump: Apache 2.0
\nCabie: GNU<\/p>\n

Here's what DevOps processes look like with a CI\/CD tool:<\/p>\n

\"Beginner's<\/p>\n

You have a CI\/CD tool running on your localhost, but there\u2019s not much you can do at this point. Let\u2019s move on to the next stage in your journey into the world of DevOps.<\/p>\n

Step 2: Managing Source Control Systems<\/h3>\n

\nThe best (and perhaps easiest) way to check if your CI\/CD tool can perform magic is to integrate it with a Source Control Management (SCM) tool. Why do you need source control? Suppose you're developing an application. Every time you create this application, you're coding, regardless of whether you\u2019re using Java, Python, C++, Go, Ruby, JavaScript, or any of the myriad programming languages out there. The code you write is called source code. In the beginning, especially when you're working alone, it's probably okay to put everything into a local directory. But as the project grows and you invite others to collaborate, you need a way to avoid conflicts while effectively sharing modifications. You also need a way to restore previous versions because making backups and copying\/pasting them is outdated. You (and your team members) need something better.<\/p>\n

This is where source control becomes virtually essential. This tool saves your code in repositories, keeps track of versions, and coordinates the work of project participants.<\/p>\n

While there are many source control tools, Git is the standard, and that's true. I highly recommend using Git, though there are other open-source options if you prefer.<\/p>\n

Git: GPLv2 and LGPL v2.1
\nSubversion: Apache 2.0
\nConcurrent Versions System (CVS): GNU
\nVesta: LGPL
\nMercurial: GNU GPL v2+<\/p>\n

This is what a DevOps pipeline looks like with the addition of source control tools.<\/p>\n

\"Beginner's<\/p>\n

A CI\/CD tool can automate the processes of checking, retrieving source code, and collaboration among team members. Not bad? But how do we turn this into a working application that billions of people can use and appreciate?<\/p>\n

Step 3: Creating a Build Automation Tool<\/h3>\n

\nGreat! You can check the code and make changes in the version control system, as well as invite your friends to collaborate on the development. However, you haven't created an application yet. To make a web application, it needs to be compiled and packaged into a deployable format or run as an executable file. (Note that an interpreted programming language like JavaScript or PHP does not require compilation).<\/p>\n

Utilize a build automation tool. Regardless of which build automation tool you decide to use, they all have the same goal: to compile the source code into some desired format and automate the tasks of cleaning, compiling, testing, and deploying in a specific environment. Build tools will vary depending on your programming language, but here are some common open-source options.<\/p>\n

Title
\nLicense
\nProgramming language<\/p>\n

Maven
\nApache 2.0
\n Java.<\/p>\n

Ant
\nApache 2.0
\nJava.<\/p>\n

Gradle
\nApache 2.0
\nJava.<\/p>\n

Bazel
\nApache 2.0
\nJava.<\/p>\n

Make
\nGNU
\nN\/A<\/p>\n

Grunt
\nMIT
\nJavaScript<\/p>\n

Gulp
\nMIT
\nJavaScript<\/p>\n

Buildr
\nApache
\nRuby<\/p>\n

Rake
\nMIT
\nRuby<\/p>\n

A-A-P
\nGNU
\nPython<\/p>\n

SCons
\nMIT
\nPython<\/p>\n

BitBake
\nGPLv2
\nPython<\/p>\n

Cake
\nMIT
\nC#<\/p>\n

ASDF
\nExpat (MIT)
\nLISP<\/p>\n

Cabal
\n(the software distribution system), the first source codes already had version number 4.3. Initially, the DNS server was used by staff in the university's laboratories. Until version 4.8.3, the development of BIND was overseen by members of the university's Computer Systems Research Group (CSRG), but in the late 1980s, the DNS server broke free from the university \u2014 it was handed over to Paul Vixie from the corporation
\nHaskell<\/p>\n

Awesome! You can place the build automation tool configuration files in the version control system and allow your CI\/CD tool to put everything together.<\/p>\n

\"Beginner's<\/p>\n

All good, right? But where to deploy your application?<\/p>\n

Step 4: Server for Web Applications<\/h3>\n

\nSo far, you have a packaged file that can be either executable or installable. For any application to be truly useful, it must provide some service or interface, but you need a container to host your application.<\/p>\n

A server for web applications serves as that container. The server provides an environment in which the logic of the deployable package can be defined. It also offers an interface and provides web services, opening sockets to the outside world. You need an HTTP server, as well as some environment (like a virtual machine) for it to run. For now, let's assume you'll learn more about this later (though I'll talk about containers below).<\/p>\n

There are several open-source servers for web applications.<\/p>\n

Title
\nLicense
\nProgramming language<\/p>\n

Tomcat
\nApache 2.0
\nJava.<\/p>\n

Jetty
\nApache 2.0
\nJava.<\/p>\n

WildFly
\nGNU Lesser Public
\nJava.<\/p>\n

GlassFish
\nCDDL & GNU Lesser Public
\nJava.<\/p>\n

Django
\n3-Clause BSD
\nPython<\/p>\n

Tornado
\nApache 2.0
\nPython<\/p>\n

Gunicorn
\nMIT
\nPython<\/p>\n

Python
\nMIT
\nPython<\/p>\n

Rails
\nMIT
\nRuby<\/p>\n

Node.js
\nMIT
\nJavascript<\/p>\n

Your DevOps pipeline is almost ready to use. Good job!<\/p>\n

\"Beginner's<\/p>\n

While you can stop here and handle the integration yourself, code quality is crucial for application developers, and it requires attention.<\/p>\n

Step 5: Code Testing Coverage<\/h3>\n

\nImplementing tests can be another cumbersome requirement, but developers must catch any errors in the application early on and enhance code quality to ensure user satisfaction. Fortunately, numerous open-source tools exist for testing your code and generating recommendations for improving its quality. Even better, most CI\/CD tools can integrate with these tools to automate the process.<\/p>\n

Code testing consists of two parts: testing frameworks that help write and run tests, and tools that provide suggestions to improve code quality.<\/p>\n

Code Testing Systems<\/b><\/p>\n

Title
\nLicense
\nProgramming language<\/p>\n

JUnit
\nEclipse Public License
\nJava.<\/p>\n

EasyMock
\nApache
\nJava.<\/p>\n

Mockito
\nMIT
\nJava.<\/p>\n

PowerMock
\nApache 2.0
\nJava.<\/p>\n

Pytest
\n MIT
\nPython<\/p>\n

Hypothesis
\nMozilla
\nPython<\/p>\n

Tox
\nMIT
\nPython<\/p>\n

Code Improvement Recommendations Systems<\/b><\/p>\n

Title
\nLicense
\nProgramming language<\/p>\n

Cobertura
\nGNU
\nJava.<\/p>\n

CodeCover
\nEclipse Public (EPL)
\nJava.<\/p>\n

Coverage.py
\n Apache 2.0
\nPython<\/p>\n

Emma
\nCommon Public License
\nJava.<\/p>\n

JaCoCo
\n Eclipse Public License
\nJava.<\/p>\n

Hypothesis
\n Mozilla
\nPython<\/p>\n

Tox
\nMIT
\nPython<\/p>\n

Jasmine
\nMIT
\nJavaScript<\/p>\n

Karma
\nMIT
\nJavaScript<\/p>\n

Mocha
\n MIT
\nJavaScript<\/p>\n

Jest
\nMIT
\nJavaScript<\/p>\n

Note that most of the tools and frameworks mentioned above are written for Java, Python, and JavaScript, as C++ and C# are proprietary programming languages (although GCC is open source).<\/p>\n

Now that you have implemented code coverage tools, your DevOps pipeline should resemble the diagram shown at the beginning of this guide.<\/p>\n

Additional steps<\/h3>\n

\nContainers<\/b><\/p>\n

As I mentioned, you can host your server on a virtual machine or a server, but containers are a popular solution.<\/p>\n

What are containers? In short, a virtual machine requires a large amount of operating system memory exceeding the application's size, while a container only needs a few libraries and configurations to run the application. Clearly, virtual machines still have significant use cases, but a container is a lightweight solution for hosting applications, including application servers.<\/p>\n

Although there are other container options, the most popular are Docker and Kubernetes.<\/p>\n

Docker: Apache 2.0
\nKubernetes: Apache 2.0<\/p>\n

Intermediate Automation Tools<\/b><\/p>\n

Our DevOps pipeline is primarily focused on collaborative application creation and deployment, but there are many other tasks that can be accomplished using DevOps tools. One of these is the use of Infrastructure as Code (IaC) tools, also known as intermediate automation tools. These tools assist in automating the installation, management, and other tasks for middleware. For example, an automation tool can pull applications like web application servers, databases, and monitoring tools, with the correct configurations, and deploy them to the application server.<\/p>\n

Here are some open-source intermediate automation tools:<\/p>\n

Ansible: GNU Public
\nSaltStack: Apache 2.0
\nChef: Apache 2.0
\nPuppet: Apache or GPL<\/p>\n

\"Beginner's<\/p>\n

Learn more about how to gain an in-demand profession from scratch or Level Up your skills and salary by taking paid online courses at SkillFactory:<\/p>\n