Have you learned Git commands but want to understand how Continuous Integration (CI) works in reality? Or maybe you want to optimize your daily activities? This course will give you practical skills in continuous integration using a GitHub repository. This course is not meant to be a wizard that you can just click on, on the contrary, you will do the same things that people actually do at work, in the same way that they do it. I will explain the theory as you go through the relevant steps.

What do we do?

As we go along, we will gradually build up a list of typical CI steps, which is a great way to remember this list. In other words, we will create a list of activities that developers perform when implementing continuous integration, implementing continuous integration. We will also use a simple test suite to bring our CI process closer to the real thing.

This GIF schematically shows the commits in your repository as you progress through the course. As you can see, there is nothing complicated and only the most necessary.

You will go through the following standard CI scenarios:

• Work on a feature;
• Application of autotests for quality assurance;
• Implementation of the priority task;
• Conflict resolution when merging branches (merge conflict);
• The occurrence of an error in a production environment.

What will you learn?

You will be able to answer questions like:

• What is continuous integration (CI)?
• What types of autotests are used in CI, and in response to what actions do they run?
• What is a pull request and when are they needed?
• What is Test Driven Development (TDD) and how does it relate to CI?
• Merge or apply changes on top (rebase)?
• Rollback or repair in the next version?

At first I translated things like "pull request" everywhere, but as a result I decided to return phrases in English in some places in order to reduce the degree of madness in the text. I will occasionally use a "programming surzhik" like the fancy verb "commit" where people actually use it at work.

What is continuous integration?

Continuous Integration, or CI, is a technical practice in which each team member integrates their code into a common repository at least once a day, and the resulting code must at least build without errors.

There are controversies about this term.

The point of contention is the frequency of integration. Some argue that merging code once a day is not enough to actually integrate continuously. An example is a team where everyone takes a fresh code in the morning and integrates once in the evening. While this is a reasonable objection, it is generally believed that the definition of "once a day" is quite practical, specific, and suitable for teams of different sizes.

Another objection is that C++ has not been the only language used in development for a long time, and simply requiring assembly without errors as a validation method is rather weak. A set of tests (for example, unit tests run locally) must also succeed. At the moment, the community is gravitating toward making such a requirement mandatory, and in the future, "build + unit tests" will probably become commonplace, if it hasn't already.

Continuous Integration differs from continuous supply (Continuous Delivery, CD) in that it does not require a release candidate after each integration cycle.

List of steps that we will use throughout the course

1. Pull in the latest code. Create a branch from master. Start working.
2. Create commits on your new branch. Build and test locally. Pass? Go to the next step. fail? Fix errors or tests and try again.
3. Push to your remote repository or remote branch.
4. Create a pull request. Discuss the changes, add more commits as discussion continues. Make tests pass on the feature branch.
5. Merge/rebase commits from master. Make tests pass on the merge result.
6. Deploy from the feature branch to production.
7. If everything is good in production for some period of time, merge changes to master.

️ Preparation

Make sure you have the right software

To take this course you will need Node.js и Git client.

You can use any Git client, but I will only list commands for the command line.

Make sure you have a Git client installed that supports the command line

If you don't already have a command-line Git client installed, you can find installation instructions here. here.

Prepare the repository

You will need to create a personal copy (fork) repository-template with the code for the course on GitHub. Let's agree to call this personal copy course repository.

Done? If you haven't changed the default settings, your course repository is most likely named continuous-integration-team-scenarios-students, it's in your GitHub account and the URL looks like this

https://github.com/<ваше имя ползователя на GitHub>/continuous-integration-team-scenarios-students

I will simply call this address <URL репозитория>.

angle brackets like <тут> will mean that you must replace such an expression with the appropriate value.

Make sure that GitHub actions enabled for this course repository. If they are not enabled, please enable them by clicking the big button in the middle of the page, which you can access by clicking Actions in the GitHub interface.

You will not be able to complete the course following my instructions unless GitHub Actions are enabled.

You can always use GitHub's ability to display Markdown to see the current state of the list we're composing, here

https://github.com/<your GitHub user name>/continuous-integration-team-scenarios-students/blob/master/ci.md

About answers

While the best way to complete this course is to do it yourself, you may find it difficult to complete.

If you feel that you do not understand what to do and cannot continue, you can peep into the thread solution, which is in your starter repository.
Please do not merge solution в master during the course. You can use this branch to figure out what to do, or to compare your code with the author's, using all the possibilities that Git gives us. If you are completely lost, you can completely replace your branch master on a branch solution and then reset your working directory to the course step you want.

Only use this if you really need it.

Commit your code

git add .
git commit -m "Backing up my work"

These commands

• rename master в master-backup;
• rename solution в master;
• switch (checkout) to a new branch master and overwrite the contents of the working directory;
• create a "solution" branch from "master" (which used to be "solution") in case you need a "solution" branch in the future.

git branch -m master master-backup
git branch -m solution master
git checkout master -f
git branch solution

After these steps, you can use git log master to figure out which commit you need.
You can reset your working directory to this commit like so:

git reset --hard <the SHA you need>

If you are happy with the result, at some point you will need to publish your version of the repository to a remote repository (remote). Don't forget to explicitly specify the remote branch when you do this.

git push --force origin master

Please note that we use git push --force. You probably don't want to do this very often, but we have a very specific scenario here with one user of the repository who, in addition, understands what he is doing.

Starting working

Let's start compiling our list of CI steps. You usually start this step by pulling the latest code from a remote repository, but we don't have a local repository yet, so we clone it from a remote one instead.

️ Task: update local repository, create a branch from master, start working

1. Clone the course repository from <URL репозитория>.
2. Run npm install in the course repository directory; we need it to install Jest, which we use to run tests.
3. Create a branch and name it feature. Switch to this thread.

4. Add test code to ci.test.js between comments asking you to do so.

   it('1. pull latest code', () => {
     expect(/.*pull.*/ig.test(fileContents)).toBe(true);
   });
   
   it('2. add commits', () => {
     expect(/.*commit.*/ig.test(fileContents)).toBe(true);
   });
   
   it('3. push to the remote branch with the same name', () => {
     expect(/.*push.*/ig.test(fileContents)).toBe(true);
   });
   
   it('4. create a pull request and continue working', () => {
     expect(/.*pulls+request.*/ig.test(fileContents)).toBe(true);
   });

5. Add text with first 4 steps to file ci.md.

   1. Pull in the latest code. Create a branch from `master`. Start working.    
   2. Create commits on your new branch. Build and test locally.  
   Pass? Go to the next step. Fail? Fix errors or tests and try again.  
   3. Push to your remote repository or remote branch.  
   4. Create a pull request. Discuss the changes, add more commits  
   as discussion continues. Make tests pass on the feature branch.  

   commands

# Клонируйте репозиторий курса
git clone <repository URL>
cd <repository name>

# Выполните npm install в каталоге репозитория курса; он установит Jest, который мы используем для запуска тестов.
npm install

# Создайте ветку и назовите ее feature. Переключитесь на эту в ветку.
git checkout -b feature

# Отредактируйте ci.test.js как описано выше.
# Отредактируйте ci.md как описано выше

Create commits on a new branch, build and test locally

We're going to set up the tests to run before the commit, and then commit the code.

Typical Scenarios When Tests Run Automatically

• Locally:
  • Continuously or in response to appropriate code changes;
  • On save (for interpreted or JIT-compiled languages);
  • On build (when compilation is required);
  • On commit;
  • When publishing to a shared repository.

• On the build server or build environment:
  • When code is published to a personal branch/repository.
  • The code in this thread is being tested.
  • A potential merge result is tested (usually with master).
  • As a continuous integration step/continuous delivery pipeline

Generally, the faster a test suite runs, the more often you can afford to run it. A typical staging might look like this.

• Rapid unit tests - at build, in CI pipeline
• Slow unit tests, fast component and integration tests - at commit, in the CI pipeline
• Slow component and integration tests - in the CI pipeline
• Security testing, load testing, and other long or expensive tests - in CI / CD pipelines, but only in certain modes / stages / build pipelines, for example, when preparing a release candidate or when starting manually.

️ Quest

I suggest running the tests manually first using the command npm test. After that, let's add a git hook to run our tests on commit. There is one catch: Git hooks are not considered part of the repository and therefore cannot be cloned from GitHub along with the rest of the course content. To install hook you need to run install_hook.sh or copy file repo/hooks/pre-commit to local directory .git/hooks/.
When you commit, you will see that the tests are run and they check if certain keywords are present in the list.

1. Run the tests manually by running the command npm test in your course repository folder. Verify that the tests have been run.
2. Install a commit hook (pre-commit hook) by running install_hook.sh.
3. Commit the changes to the local repository.
4. Make sure the tests are run before the commit.

Your repository should look like this after completing these steps.

commands

# Установите pre-commit hook выполнив install_hook.sh.  

# Закоммитьте изменения в локальный репозиторий. Используйте "Add first CI steps" в качестве сообщения при коммите.
git add ci.md ci.test.js
git commit -m "Add first CI steps"

# Убедитесь, что тесты запускаются перед коммитом.  

Publish code to a remote repository or remote branch

After finishing working locally, developers usually make their code public so that it can eventually be integrated with the public. With GitHub, this is usually achieved by publishing the work either to a personal copy of the repository (personal fork, fork) or to a personal branch.

• With forks, the developer clones the remote shared repository, creating a personal remote copy of it, also known as a fork. After that, he clones this personal repository so that he can work with it locally. When the work is done and the commits are created, he puts them in his fork, where they are available to others and can be integrated into a common repository. This approach is commonly used in open source projects on GitHub. It is also used in my advanced course [Team Work and CI with Git] (http://devops.redpill.solutions/).
• Another approach is to use only one remote repository and only count the branch master the shared repository is "protected". In this scenario, individual developers publish their code to the branches of the remote repository so that others can see this code, if everything is in order, merge with master shared repository.

In this particular course, we will be using a workflow that uses branches.

Let's publish our code.

️ Quest

• Post changes to a remote branch with the same name as your work branch

commands

git push --set-upstream origin feature

Create a pull request

Create a pull request with a title Steps review... Install feature as "head branch" and master as "base branch".

Make sure you have installed master before repository fork as a "base branch", I will not respond to change requests to the course content repository.

In GitHub slang, "base branch" is the branch on which you base your work, and "head branch" is the branch containing the proposed changes.

Discuss changes, add new commits as discussion continues

Pull request(PR)

Pull request(PR) is a way to discuss and document code, as well as conduct code review. Pull requests are named after the general way in which individual changes are integrated into the overall code. Usually a person clones a remote official project repository and works on the code locally. After that, he places the code in his personal remote repository and asks those responsible for the official repository to pick it up (pull) its code to their local repositories where they review and possibly integrate(go) his. This concept is also known by other names, for example, merge request.

In fact, you don't have to use the pull request feature of GitHub or similar platforms. Development teams may use other means of communication, including face-to-face communication, voice calls, or email, but there are still a number of reasons to use such forum-style pull requests. Here are some of them:

• organized discussions related to specific changes in the code;
• as a place to view feedback on work in progress from both autotests and peers;
• formalization of code checks;
• so that you can later find out the reasons and considerations behind this or that piece of code.

Usually you create a pull request when you need to discuss something or get feedback. For example, if you're working on a feature that can be implemented in multiple ways, you can create a pull request before the first line of code is written to share your ideas and discuss your plans with collaborators. If the work is more simple, a pull request is opened when something has already been done, committed and can be discussed. In some scenarios, you may want to open a PR just for quality control reasons: to run automated tests or initiate code reviews. Whatever you decide, don't forget to @mention the people you want approval in your pull request.

Typically, when creating a PR, you do the following.

• Specify what you propose to change and where.
• Write a description explaining the purpose of the change. You may want:
  • add something important that is not obvious from the code, or something useful for understanding the context, such as relevant #bugs and commit numbers;
  • @mention anyone you want to work with, or you can @mention them in the comments later;
  • ask colleagues to help with something or check something specific.

After you open a PR, the tests configured to run in such cases are executed. In our case, this will be the same test suite that we ran locally, but there may be additional tests and checks in a real project.

Please wait while the tests are completed. You can see the status of the tests at the bottom of the PR thread in the GitHub interface. Continue when tests are completed.

️ Add a note about the arbitrariness of the list of CI steps

The list used in this course is arbitrary and subjective, we must add a note about this.

️ Challenge: Create a pull request for this note

1. Switch to branch master.
2. Create a branch named bugfix.
3. Add note text to the end of the file ci.md.

   > **GitHub flow** is sometimes used as a nickname to refer to a flavor of trunk-based development  
   when code is deployed straight from feature branches. This list is just an interpretation  
   that I use in my [DevOps courses](http://redpill.solutions).  
   The official tutorial is [here](https://guides.github.com/introduction/flow/).

4. Commit the changes.
5. Publish a branch bugfix to a remote repository.
6. Create a pull request named Adding a remark with head branch bugfix and base branchmaster.

Make sure you have installed master before repository fork as a "base branch", I will not respond to change requests to the course content repository.

This is what your repository should look like.

commands

# Переключитесь на ветку master. Создайте ветку bugfix.
git checkout master

# Создайте ветку bugfix-remark.
git checkout -b bugfix

# Добавьте текст примечания внизу ci.md.

# Закоммитьте изменения
git add ci.md
git commit -m "Add a remark about the list being opinionated"

# Опубликуйте ветку bugfix в удалённый репозиторий.
git push --set-upstream origin bugfix

# Создайте pull request при помощи интерфейса GitHub как описано выше

Approve the pull request "Adding a remark"

️ Quest

1. Create a pull request.
2. Click "Merge pull request".
3. Click "Confirm merge".
4. Click "Delete branch", we don't need it anymore.

This is a diagram of the commits after the merge.

️ Keep working and adding tests

Collaborating on a pull request often results in extra work being done. This is usually the result of a code review or discussion, but in our course we are going to model this by adding new items to our list of CI steps.

With continuous integration, some test coverage is usually applied. Test coverage requirements vary and are usually found in a document with a title like "contribution guidelines". We'll go easy and add a test for each line in our checklist.

When running jobs, first try to commit the tests. If you installed correctly pre-commit hook before, the newly added test will run, fail, and nothing will be committed. Note that this is how we know that our tests actually test something. Curiously, if we started with the code before the tests, passing the tests could either mean that the code works as expected, or that the tests don't actually test anything. Also, if we hadn't written the tests in the first place, we might have forgotten them altogether, since there would be nothing to remind us of it.

Test Driven Development (TDD)

TDD recommends writing tests before code. A typical workflow using TDD looks like this.

1. Add a test.
2. Run all tests and verify that the new test fails.
3. Write code.
4. Run tests, make sure all tests pass.
5. Refactor your code.
6. Repeat.

Because test results that fail are typically shown in red, and tests that pass are shown in green, the cycle is also known as red-green-refactor.

️ Quest

First try to commit the tests and let them fail, then add and commit the CI step list text itself. You will see that the tests pass ("green").
Then publish the new code to a remote repository and watch the tests run in the GitHub interface at the bottom of the pull request discussion and the PR status is updated.

1. Switch to branch feature.

2. Add these tests to ci.test.js after the last call it (...);.

   it('5. Merge/rebase commits from master. Make tests pass on the merge result.', () => {
     expect(/.*merge.*commits.*testss+pass.*/ig.test(fileContents)).toBe(true);
   });
   
   it('6. Deploy from the feature branch to production.', () => {
     expect(/.*Deploy.*tos+production.*/ig.test(fileContents)).toBe(true);
   });
   
   it('7. If everything is good in production for some period of time, merge changes to master.', () => {
     expect(/.*merge.*tos+master.*/ig.test(fileContents)).toBe(true);
   });

3. Try committing the tests. If pre-commit hook is set, the commit attempt will fail.
4. Then add this text to ci.md.

   5. Merge/rebase commits from master. Make tests pass on the merge result.  
   6. Deploy from the feature branch with a sneaky bug to production.
   7. If everything is good in production for some period of time, merge changes to master. 

5. Make and commit changes locally.
6. Post changes to branch feature.

Now you should have something like this

commands

# Переключительна ветку feature
git checkout feature

# Добавить тесты в ci.test.js как описано выше

# Добавьте в индекс ci.test.js чтобы позже закоммитить
git add ci.test.js

# Попытайтесь закоммитить тесты. Если pre-commit hook установлены, коммит не произойдёт.
git commit

# Теперь добавьте текст в ci.md как описано выше

# Внесите изменения и закоммитьте их
git add ci.md
git commit -m "Add the remaining CI steps"

# Опубликуйте изменения в ветку feature
git push

Merge conflict

Go to change request Steps review.

Even though we didn't do anything wrong and the tests for our code passed, we still can't merge the branch feature и master. This is because the other thread bugfix was merged with master while we were working on this PR.
This creates a situation where the remote branch master has a newer version than the one we based the branch on feature. Because of this, we can't just rewind HEAD master to the end of the branch feature. In this situation, we need to either merge or apply commits feature over(rebase) master. GitHub can actually do an automatic merge if there are no conflicts. Alas, in our situation, both branches have competing changes in the file ci.md. This situation is known as a merge conflict and we need to resolve it manually.

Merge or rebase

Go

• Creates an additional merge commit and saves the work history.
  • Preserves the original branch commits with their original timestamps and authors.
  • Stores the commit SHAs and links to commits in change request discussions.
• Requires one-time conflict resolution.
• Makes the story non-linear.
  • The history can be difficult to read due to the large number of branches (reminds me of an IDE cable).
  • Makes automatic debugging more difficult, for example, doing git bisect less useful - it will only find the merge commit.

rebase

• Replays commits from the current branch on top of the base one one after the other.
  • New commits are generated with new SHAs, causing GitHub commits to match the original pull requests but not the corresponding comments.
  • Commits can be recombined and changed in the process, or even merged into one.
• You may need to resolve multiple conflicts.
• Allows you to maintain a linear history.
  • The story may be easier to read, as long as it is not too long for no good reason.
  • Automatic debugging and troubleshooting is somewhat easier: makes it possible git bisect, can make automatic rollbacks more distinct and predictable.
• Requires publishing a branch with reverted commits with a flag --force when used with change requests.

Teams usually agree to always use the same strategy when they need to merge changes. This can be a "pure" merge or a "pure" commit on top, or something in between, such as doing a commit over interactive mode(git rebase -i) locally for branches not published to a shared repository, but merge for "public" branches.

Here we will use merge.

️ Quest

1. Make sure the code is in the local branch master updated from a remote repository.
2. Switch to branch feature.
3. Initiate a merge with a branch master. A merge conflict will be reported due to competing changes in ci.md.
4. Resolve the conflict so that both our list of CI steps and a note about it remain in the text.
5. Post a merge commit to a remote branch feature.
6. Check the status of the pull request in the GitHub UI, wait until the merge is resolved.

commands

# Убедитесь, что код в локальное ветке `master` обновлён из удалённого репозитория.
git checkout master
git pull

# Переключитесь на ветку feature
git checkout feature

# Инициируйте слияние с веткой master 
git merge master

# A merge conflict related to concurrent changes to ci.md will be reported
# => Auto-merging ci.md
#    CONFLICT (content): Merge conflict in ci.md
#    Automatic merge failed; fix conflicts and then commit the result.

# Разрешите конфликт так, чтобы и наш список шагов CI, и замечание о нем остались в тексте.
# отредактируйте ci.md чтоб он не содержал маркеров конфликта слияния
git add ci.md
git merge --continue
# при коммите можете оставить сообщение по умолчанию

# Опубликуйте коммит слияния в удаленную ветку feature.
git push

# Проверьте статус запроса на изменения в пользовательском интерфейсе GitHub, дождитесь пока слияние не будет разрешено.

Great job!

You are done with the list and now you need to approve the pull request in master.

️ Task: Approve the pull request "Steps review"

1. Open a pull request.
2. Click "Merge pull request".
3. Click "Confirm merge".
4. Click "Delete branch" since we don't need it anymore.

This is your repository at the moment

Product error

They say that "testing can be used to show the presence of errors, but never to show their absence." Despite the fact that we had tests and they did not show us any errors, an insidious error crept into production.

In a scenario like this, we need to take care of:

• that is deployed on the productive;
• branch code master with an error, from which developers can start new work.

Rollback or fix in the next version?

"Rolling back" is the act of deploying a known-good earlier version to a production environment and reverting the commits that contain the error. "Fixing forward" is the addition of a fix to master and deploying the new version as soon as possible. Because APIs and database schemas change as code is deployed to production, with continuous delivery and good test coverage, rolling back is generally much more difficult and risky than fixing in the next release.

Since rolling back does not carry any risk in our case, we will go this way, because it allows us

• fix the bug on the production as soon as possible;
• make code in master immediately ready to start a new job.

️ Quest

1. Switch to branch master locally.
2. Update the local repository from the remote repository.
3. Revert the PR merge commit Steps review в master.
4. Publish changes to a remote repository.

This is the history of the repository with the merge commit reverted

commands

# Переключитесь на ветку master.
git checkout master

# Обновите локальный репозиторий из удалённого репозитория.
git pull

# Отмените коммит слияния PR Steps review в master.
# Мы отменяем коммит слияния, поэтому нам нужно выбрать ветку истории, которую мы захотим оставить
git show HEAD

# предположим, что коммит, который был последним в ветке master до слияния, был отображён предыдущей командой первым
git revert HEAD -m 1
# можете не менять сообщения коммитов

# Опубликуйте изменения в удалённый репозиторий
git push

️ Self check

Make sure that ci.md no longer contains the text "sneaky bug" after a merge commit is reverted.

Fix CI step list and revert to master

We completely reverted the branch merge commit feature. The good news is that now we have no error in master. The bad news is that our precious list of continuous integration steps is also gone. So, ideally, we want to apply the fix to the commits from feature and return them to master along with the fix.

We can approach the problem in different ways:

• revert the commit, which reverts the merge feature с master;
• migrate commits from the former feature.

Different development teams take different approaches in this case, but we will move useful commits to a separate branch and create a separate pull request for this new branch.

️ Quest

1. Create a branch called feature-fix and switch to it.

2. Migrate all commits from the former branch feature to a new thread. Resolve merge conflicts that occurred during the migration.

   

3. Add a regression test to ci.test.js:

   it('does not contain the sneaky bug', () => {
   expect( /.*sneakys+bug.*/gi.test(fileContents)).toBe(false);
   });

4. Run the tests locally to make sure they don't complete successfully.
5. Delete the text "with a sneaky bug" in ci.md.
6. Add test changes and changes to the list of steps to the index and commit them.
7. Publish the branch to the remote repository.

You should end up with something like

commands

# Создайте ветку под названием feature-fix и переключитесь на нее.
git checkout -b feature-fix

# Перенесите все коммиты из бывшей ветки feature в новую ветку. Разрешите конфликты слияния, которые возникли при переносе.
# используйте историю чтобы узнать хэши коммитов:
# - предшествующего коммиту с первой частью списка: C0
# - добавляющего последние элементы списка: C2
git log --oneline --graph
git cherry-pick C0..C2
# разрешите конфликты слияния
# - отредактируйте ci.md и/или ci.test.js
# - добавьте файлы в индекс
# - выполните "git cherry-pick --continue", можете не менять сообщение коммита

# Добавьте регрессионный тест в ci.test.js
# Запустите тесты локально, чтобы убедиться, что они не завершаются успешно.

# Удалите текст " with a sneaky bug" в ci.md.

# Добавьте в индекс изменения тестов и в списке шагов и закоммитьте их.
git add ci.md ci.test.js
git commit -m "Fix the bug in steps list"

# Опубликуйте ветку в удалённый репозиторий.
git push --set-upstream origin feature-fix

Create a pull request.

Create a pull request with a title Fixing the feature... Install feature-fix as "head branch", and master as "base branch".
Please wait while the tests are completed. You can see the status of the tests at the bottom of the PR discussion.

Make sure you have installed master before repository fork as a "base branch", I will not respond to change requests to the course content repository.

Approve the pull request "Fixing the feature"

Thanks for correction! Please approve the changes in master from pull request.

️ Quest

1. Click "Merge pull request".
2. Click "Confirm merge".
3. Click "Delete branch" since we don't need it anymore.

This is what you should have right now.

Congratulations!

You've completed all the steps that people typically take in a continuous integration process.

If you notice any problems with the course or know how to improve it, please create an issue in course content repositories. This course also has interactive version using GitHub Learning Lab as a platform.

Source: habr.com