If you're new to DevOps, take a look at this guide to building your first five-step pipeline.
DevOps has become the standard solution for fixing slow, disconnected, or broken software development processes. The problem is that if you're new to DevOps and don't know where to start, then you may lack understanding of these techniques. This article will focus on defining a DevOps pipeline, and will also offer instructions on how to create it in five steps. Although this tutorial is not exhaustive, it should give you a foundation to get started and expand your knowledge in the future. But let's start with history.
My DevOps Journey
I previously worked for the Citi Group cloud team developing an Infrastructure-as-a-Service (IaaS) web application to manage Citi's cloud infrastructure, but I've always been interested in how to make the development process more efficient and bring positive cultural change to the development team. I found the answer in a book recommended by Greg Lavender, Citi's CTO for Cloud Architecture and Infrastructure. The book was called "The Phoenix Project" () and explains DevOps principles while reading like a novel.
The table on the back of the book shows how often different companies deploy their systems in a release environment:
Amazon: 23 per day
Google: 5 per day
Netflix: 500 per day
Facebook: Once a day
Twitter: 3 times a week
Typical company: Once every 9 months
How are the frequencies of Amazon, Google and Netflix even possible? This is because these companies figured out how to make a near-perfect DevOps pipeline.
We were far from it until we implemented DevOps at Citi. At that time, my team had different environments, but deployment to the development server was completely manual. All developers had access to only one development server based on IBM WebSphere Application Server Community Edition. The problem was that the server would shut down whenever multiple users tried to deploy at the same time, so the developers had to let each other know their intentions, which was pretty painful. In addition, there were problems with low-level code test coverage, cumbersome manual deployment processes, and the inability to track the deployment of code associated with a particular task or user story.
I realized that something needed to be done, and I found a like-minded colleague. We decided to collaborate on building the initial DevOps pipeline - he set up a Tomcat virtual machine and application server while I worked on Jenkins, integrated Atlassian Jira and BitBucket, and worked on code test coverage. This side project was very successful: we almost completely automated many processes, reached almost 100% uptime on our development server, provided tracking and improved code test coverage, and added the ability to link branches in Git to issues in Jira or deployment. Most of the tools we used to build our DevOps pipeline were open source.
Now I understand how simple our DevOps pipeline was: we didn't use extensions like Jenkins files or Ansible. However, this simple pipeline worked well, perhaps thanks to the Pareto principle (also known as the 80/20 rule).
A Brief Introduction to DevOps and the CI/CD Pipeline
If you ask a few people "What is DevOps?" you will probably get a few different answers. DevOps, like Agile, has evolved to encompass many different disciplines, but most people will agree on a few things: DevOps is a software development practice or software development life cycle (SDLC) whose central tenet is to change the culture in which developers and non-developers exist in an environment where:
Automated operations that were previously performed manually;
Everyone does what he does best;
The number of implementations for a certain period of time increases; Increased throughput;
Increased development flexibility.
While having the right software tools isn't the only thing you need to create a DevOps environment, some tools are essential. The key tool is continuous integration and continuous deployment (CI/CD). In this pipeline, environments have different stages (e.g. DEV, INT, TST, QA, UAT, STG, PROD), many operations are automated, and developers can write high-quality code, achieve development agility, and high deployment frequency.
This article describes a five-step approach to building a DevOps pipeline similar to the one shown in the following diagram using open source tools.
Step 1: CI/CD Methods
The first thing you need is a CI/CD tool. Jenkins, an open source tool based on Java and licensed under the MIT license, is the tool that popularized DevOps and became the de facto standard.
So what is Jenkins? Think of it as some kind of magical universal remote control that can talk to and organize various services and tools. On its own, a CI/CD tool like Jenkins is useless, but it gets more powerful as it connects to different tools and services.
Jenkins is just one of many open source CI/CD tools that you can use to build your DevOps pipeline.
Jenkins: Creative Commons and MIT
Travis CI: MIT
CruiseControl:BSD
Buildbot: GPL
Apache Gump: Apache 2.0
Cabie: GNU
Here's what DevOps processes look like with a CI/CD tool:
You have a CI/CD tool running on your localhost, but there isn't much you can do at the moment. Let's move on to the next step in our DevOps journey.
Step 2: Managing source code control systems
The best (and possibly easiest) way to test that your CI/CD tool can do magic is to integrate with a source code control (SCM) tool. Why do you need source control? Let's say you are developing an application. Whenever you create an application, you are programming, whether you use Java, Python, C++, Go, Ruby, JavaScript, or any of a gazillion programming languages. The code you write is called source code. In the beginning, especially when you're working alone, it's probably fine to put everything in a local directory. But as the project gets bigger and you invite other people to contribute, you need a way to avoid conflicts while sharing modifications effectively. You also need a way to restore previous versions, because creating backups and copying/pasting into them is already obsolete. You (and your teammates) need something better.
This is where source code control becomes almost a necessity. This tool keeps your code in repositories, keeps track of versions, and coordinates the work of project participants.
Although there are many source code control tools out there, Git is the standard, and rightfully so. I highly recommend using Git, although there are other open source options if you like.
Git: GPLv2 and LGPL v2.1
Subversion: Apache 2.0
Concurrent Versions System (CVS): GNU
Vesta: LGPL
Mercurial: GNU GPL v2+
This is what a DevOps pipeline looks like with the addition of source code controls.
A CI/CD tool can automate the processes of verification, source code acquisition, and collaboration between members. Not bad? But how do you make it into a working application so that billions of people can use and appreciate it?
Step 3: Create a Build Automation Tool
Great! You can check the code and make changes to the source code control system, as well as invite your friends to contribute to the development. But you haven't created an application yet. To make a web application, it must be compiled and packaged in a deployable package format or run as an executable file. (Note that an interpreted programming language such as JavaScript or PHP does not need to be compiled.)
Use a build automation tool. Regardless of which build automation tool you choose to use, they all share the same goal: to build source code into some desired format and automate the task of cleaning, compiling, testing, and deploying to a specific environment. Build tools will vary depending on your programming language, but here are some common open source options.
Name
License
Programming language
Maven
Apache 2.0
Java
Ant
Apache 2.0
Java
gradle
Apache 2.0
Java
Basel
Apache 2.0
Java
Make
GNU
N/A
Grunt
MIT
JavaScript
Gulp
MIT
JavaScript
Builder
Apache
Ruby
rake
MIT
Ruby
AAP
GNU
Python
SCons
MIT
Python
bitbake
GPLv2
Python
Cake
MIT
C#
ASDF
Expat (MIT)
LISP
Cabal
BSD
Haskell
Great! You can put the build automation tool configuration files into source control and let your CI/CD tool put everything together.
It's all good, isn't it? But where to deploy your application?
Step 4: Web Application Server
So far, you have a packaged file that can be either executable or installable. For any application to be truly useful, it must provide some kind of service or interface, but you need a container to host your application.
The web application server is just such a container. The server provides an environment in which the logic of the package being deployed can be defined. The server also provides an interface and offers web services by opening sockets to the outside world. You need an HTTP server as well as some environment (like a virtual machine) to set it up. For now, let's assume you learn more about this further (although I'll cover containers below).
There are several open source web application servers.
Name
License
Programming language
Tomcat
Apache 2.0
Java
Jetty
Apache 2.0
Java
wildfly
GNU Lesser Public
Java
GlassFish
CDDL & GNU Less Public
Java
Django
3-ClauseBSD
Python
Tornado
Apache 2.0
Python
gunicorn
MIT
Python
Python
MIT
Python
Rails
MIT
Ruby
Node.js
MIT
javascript
Your DevOps pipeline is almost ready to use. Good job!
While you can stop there and do the integration yourself, code quality is an important thing for an application developer to worry about.
Step 5: Code testing coverage
Implementing tests can be another cumbersome requirement, but developers must catch any bugs in the application early and improve code quality to ensure end users are satisfied. Luckily, there are many open source tools available to test your code and make recommendations to improve its quality. Better yet, most CI/CD tools can plug into these tools and automate the process.
Code testing comes in two parts: code testing frameworks that help you write and run tests, and suggestion tools that help improve code quality.
Code testing systems
Name
License
Programming language
Junit
Eclipse Public License
Java
Easy Mock
Apache
Java
mockito
MIT
Java
powermock
Apache 2.0
Java
Pytest
MIT
Python
hypothesis
Mozilla
Python
Tox
MIT
Python
Code improvement recommendation systems
Name
License
Programming language
Coverage
GNU
Java
codecover
Eclipse Public (EPL)
Java
Coverage.py
Apache 2.0
Python
Emma
Common Public License
Java
JaCoCo
Eclipse Public License
Java
hypothesis
Mozilla
Python
Tox
MIT
Python
Jasmine
MIT
JavaScript
Karma
MIT
JavaScript
Mocha
MIT
JavaScript
Jest
MIT
JavaScript
Note that most of the tools and frameworks mentioned above are written for Java, Python, and JavaScript, since C++ and C# are proprietary programming languages ββ(even though GCC is open source).
Now that you've implemented code coverage tools, your DevOps pipeline should look similar to the diagram shown at the beginning of this tutorial.
Additional steps
Containers
As I said before, you can host your server in a virtual machine or server, but containers are a popular solution.
What are containers? The short explanation is that a virtual machine needs a huge amount of operating system memory, more than the size of an application, while a container only needs a few libraries and configurations to run an application. Obviously, there are still important uses for a virtual machine, but a container is a lightweight solution for hosting an application, including an application server.
While other container options exist, Docker and Kubernetes are the most popular.
Docker: Apache 2.0
Kubernetes: Apache 2.0
Intermediate Automation Tools
Our DevOps pipeline is primarily focused on co-creating and deploying applications, but there are many other things you can do with DevOps tools. One of these is the use of Infrastructure as Code (IaC) tools, which are also known as middleware automation tools. These tools help automate installation, management, and other tasks for middleware. So, for example, an automation tool can extract applications like a web application server, a database, and a monitoring tool with the right configurations and deploy them to an application server.
Here are some open source middleware automation tools:
Ansible: GNU Public
SaltStack: Apache 2.0
Chef: Apache 2.0
Puppet: Apache or GPL
Find out the details of how to get a sought-after profession from scratch or Level Up in terms of skills and salary by completing SkillFactory paid online courses:
- (12 months)
more courses
- (12 weeks)
- (12 months)
- (9 months)
- (9 months)
Useful
Source: habr.com