🥇Analysis of commits and pull requests in Travis CI, Buddy and AppVeyor using PVS-Studio

In the PVS-Studio analyzer for C and C++ languages on Linux and macOS, starting from version 7.04, a test opportunity has appeared to check the list of specified files. Using the new mode, you can configure the analyzer to check commits and pull requests. This article will show you how to set up a GitHub project file list check in popular CI (Continuous Integration) systems such as Travis CI, Buddy and AppVeyor.

File list check mode

PVS Studio is a tool for detecting errors and potential vulnerabilities in the source code of programs written in C, C++, C# and Java. Works on 64-bit systems on Windows, Linux and macOS.

The version of PVS-Studio 7.04 for Linux and macOS has a mode for checking the list of source files. This works for projects whose build system allows you to generate a file compile_commands.json. It is needed for the analyzer to extract information about the compilation of the specified files. If your build system does not support generating a compile_commands.json file, you can try generating such a file using the utility Bear.

Also, the mode of checking the list of files can be used together with the strace trace of compiler runs (pvs-studio-analyzer trace). To do this, you will first need to carry out a full build of the project and track it so that the analyzer collects complete information about the compilation parameters of all the files being checked.

However, this option has a significant drawback - you will either need to perform a full build trace of the entire project at each launch, which in itself contradicts the idea of a quick commit check. Or, if you cache the tracing result itself, subsequent launches of the analyzer may turn out to be incomplete if the source file dependency structure changes after tracing (for example, a new #include is added to one of the source files).

Therefore, we do not recommend using the file list check mode with a trace log to check commits or pull requests. In case you can do an incremental build when checking a commit, consider using the mode incremental analysis.

The list of source files for analysis is saved to a text file and passed to the analyzer using the parameter -S:

pvs-studio-analyzer analyze ... -f build/compile_commands.json -S check-list.txt

This file specifies relative or absolute paths to files, and each new file must be on a new line. It is permissible to specify not only the names of files for analysis, but also various text. The parser will see that this is not a file and will ignore the line. This can be useful for commenting if files are specified manually. However, often the list of files will be generated during CI parsing, for example files from a commit or a pull request.

Now, using this mode, you can quickly test new code before it gets into the main development branch. In order for the verification system to react to analyzer warnings, the utility plog-converter flag added --indicate-warnings:

plog-converter ... --indicate-warnings ... -o /path/to/report.tasks ...

With this flag, the converter will return a non-zero code if there are warnings in the analyzer report. Using the return code, you can block a precommit hook, commit or pull request, and display the generated analyzer report on the screen, share it or send it by mail.

Note. The first time you start analyzing the list of files, the entire project will be analyzed, because the analyzer needs to generate a file of dependencies of the project's source files on the header files. This is a feature of parsing C and C++ files. In the future, the dependency file can be cached and it will be updated automatically by the analyzer. The advantage of checking commits when using filelist check mode over using incremental parsing mode is that only that file needs to be cached, not the object files.

General principles of pull request analysis

The analysis of the entire project takes a lot of time, so it makes sense to check only some part of it. The problem is that you need to separate the new files from the rest of the project files.

Consider an example of a commit tree with two branches:

Let's pretend that the commit A1 contains a fairly large amount of code that has already been checked. A little earlier we made a branch from the commit A1 and changed some files.

Of course, you noticed that after A1 there were two more commits, but these were also mergers of other branches, because we don’t commit in master. And now the time has come when hotfix ready. Therefore, a pull request appeared for the merge B3 и A3.

Of course, it would be possible to check the entire result of their merge, but this would be too long and unjustified, since only a few files were changed. Therefore, it is more efficient to analyze only the changed ones.

To do this, we get the difference between the branches, being in the HEAD of the branch from which we want to merge into master:

git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list

$MERGE_BASE we will consider in detail later. The fact is that not every CI service provides the necessary information about the base for the merge, so every time you have to come up with new ways to get this data. This will be detailed below in each of the web services described.

So, we got the difference between the branches, or rather, the list of file names that have been changed. Now we need to give the file .pvs-pr.list (we redirected the output above to it) to the analyzer:

pvs-studio-analyzer analyze -j8 
                            -o PVS-Studio.log 
                            -S .pvs-pr.list

After analysis, we need to convert the log file (PVS-Studio.log) into a readable format:

plog-converter -t errorfile PVS-Studio.log --cerr -w

This command will list the errors in stderr (standard error output stream).

Only here we need not only to display errors, but also to inform our service for assembly and testing about the presence of problems. For this, a flag was added to the converter -W (--indicate-warnings). If there is at least one analyzer warning, the return code of the utility plog-converter will change to 2, which in turn will notify the CI service that there are potential errors in the pull request files.

Travis C.I.

The configuration is made in the form of a file .travis.yml. For convenience, I advise you to put everything in a separate bash script with functions that will be called from the file .travis.yml (bash scriptname.sh function_name).

We will add the necessary code to the script on bash, so we get more functionality. In section install let's write the following:

install:
  - bash .travis.sh travis_install

If you had any instructions, you can move them to the script by removing the hyphens.

Let's open the file .travis.sh and add the analyzer setup to the function travis_install():

travis_install() {
  wget -q -O - https://files.viva64.com/etc/pubkey.txt 
    | sudo apt-key add -
  sudo wget -O /etc/apt/sources.list.d/viva64.list 
    https://files.viva64.com/etc/viva64.list
  
  sudo apt-get update -qq
  sudo apt-get install -qq pvs-studio 
}

Now let's add to the section script run analysis:

script:
  - bash .travis.sh travis_script

And in bash script:

travis_script() {
  pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY
  
  if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
    git diff --name-only origin/HEAD > .pvs-pr.list
    pvs-studio-analyzer analyze -j8 
                                -o PVS-Studio.log 
                                -S .pvs-pr.list 
                                --disableLicenseExpirationCheck
  else
    pvs-studio-analyzer analyze -j8 
                                -o PVS-Studio.log 
                                --disableLicenseExpirationCheck
  fi
  
  plog-converter -t errorfile PVS-Studio.log --cerr -w
}

This code must be run after the project is built, for example, if you had a CMake build:

travis_script() {
  CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}"
  cmake $CMAKE_ARGS CMakeLists.txt
  make -j8
}

It will turn out like this:

travis_script() {
  CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}"
  cmake $CMAKE_ARGS CMakeLists.txt
  make -j8
  
  pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY
  
  if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
    git diff --name-only origin/HEAD > .pvs-pr.list
    pvs-studio-analyzer analyze -j8 
                                -o PVS-Studio.log 
                                -S .pvs-pr.list 
                                --disableLicenseExpirationCheck
  else
    pvs-studio-analyzer analyze -j8 
                                -o PVS-Studio.log 
                                --disableLicenseExpirationCheck
  fi
  
  plog-converter -t errorfile PVS-Studio.log --cerr -w
}

You probably already noticed the specified environment variables. $TRAVIS_PULL_REQUEST и $TRAVIS_BRANCH. Travis CI declares them on its own:

$TRAVIS_PULL_REQUEST stores the pull request number, or falseif it is a normal branch;
$TRAVIS_REPO_SLUG stores the name of the project repository.

The algorithm of this function:

Travis CI responds to return codes, so the presence of warnings will tell the service to flag the commit as buggy.

Let's take a closer look at this line of code:

git diff --name-only origin/HEAD > .pvs-pr.list

The fact is that Travis CI automatically merges branches during the analysis of a pull request:

Therefore, we analyze A4And not B3->A3. Because of this feature, we need to calculate the difference from A3, which is just the top of the branch from origin.

One important detail remains - caching the dependencies of header files on compiled translation units (*.c, *.cc, *.cpp, etc.). The analyzer calculates these dependencies at the first start in the mode of checking the list of files and then saves them in the .PVS-Studio directory. Travis CI allows you to cache folders, so we will save directory data .PVS-Studio/:

cache:
  directories:
    - .PVS-Studio/

This code needs to be added to the file .travis.yml. This directory stores various data collected after analysis, which will significantly speed up subsequent runs of file list analysis or incremental analysis. If this is not done, then the analyzer will actually analyze all files every time.

Buddy

Like Travis C.I., Buddy provides the ability to automatically build and test projects that are stored on GitHub. Unlike Travis CI, it is configured in the web interface (bash support is available), so there is no need to store configuration files in the project.

First of all, we need to add a new action to the build line:

Specify the compiler that was used to build the project. Notice the docker container that is installed in this activity. For example, there is a special container for GCC:

Now let's install PVS-Studio and the necessary utilities:

Add the following lines to the editor:

apt-get update && apt-get -y install wget gnupg jq

wget -q -O - https://files.viva64.com/etc/pubkey.txt | apt-key add -
wget -O /etc/apt/sources.list.d/viva64.list 
  https://files.viva64.com/etc/viva64.list

apt-get update && apt-get -y install pvs-studio

Now let's go to the Run tab (the first icon) and add the following code to the corresponding editor field:

pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY

if [ "$BUDDY_EXECUTION_PULL_REQUEST_NO" != '' ]; then
  PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO"
  MERGE_BASE=`wget -qO - 
    https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} 
    | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck 
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck
fi

plog-converter -t errorfile PVS-Studio.log --cerr -w

If you've read the section on Travs-CI, then this code is already familiar to you, however, now there is a new step:

The fact is that now we are analyzing not the result of the merge, but the HEAD of the branch from which the pull request is made:

So we're in a conditional commit B3 and we need to get the difference from A3:

PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO"
  MERGE_BASE=`wget -qO - 
    https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} 
    | jq -r ".base.ref"`
git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list

For determining A3 Let's use the GitHub API:

https://api.github.com/repos/${USERNAME}/${REPO}/pulls/${PULL_REQUEST_ID}

We used the following variables provided by Buddy:

$BUDDY_EXECUTION_PULL_REQEUST_NO - pull request number;
$BUDDY_REPO_SLUG - a combination of username and repository (for example max / test).

Now let's save the changes using the button below, and enable pull request analysis:

Unlike Travis CI, we do not need to specify .pvs-studio for caching, since Buddy automatically caches all files for subsequent launches. Therefore, the last thing left is to save the login and password for PVS-Studio in Buddy. After saving the changes, we will get back to the Pipeline. We need to go to setting variables and add login and key for PVS-Studio:

After that, the appearance of a new pull request or commit will trigger a check. If a commit contains errors, then Buddy will point it out on the pull request page.

AppVeyor

Setting up AppVeyor is similar to Buddy, since everything happens in the web interface and there is no need to add a *.yml file to the project repository.

Let's go to the Settings tab in the project overview:

Let's scroll down this page and enable cache saving for building pull requests:

Now let's go to the Environment tab, where we specify the image to build and the necessary environment variables:

If you have read the previous sections, you are very familiar with these two variables − PVS_KEY и PVS_USERNAME. If not, then let me remind you that they are necessary to check the license of the PVS-Studio analyzer. In the future, we will meet them again in Bash scripts.

On the same page below, specify the folder for caching:

If we do not do this, then we will analyze the entire project instead of a couple of files, but we will get the output based on the specified files. Therefore, it is important to enter the correct directory name.

Now it's time for the script to test. Open the Tests tab and select Script:

Paste the following code into this form:

sudo apt-get update && sudo apt-get -y install jq

wget -q -O - https://files.viva64.com/etc/pubkey.txt 
  | sudo apt-key add -
sudo wget -O /etc/apt/sources.list.d/viva64.list 
  https://files.viva64.com/etc/viva64.list

sudo apt-get update && sudo apt-get -y install pvs-studio

pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY

PWD=$(pwd -L)
if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then
  PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
  MERGE_BASE=`wget -qO - 
    https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} 
    | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck 
                              --dump-files --dump-log pvs-dump.log 
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck
fi

plog-converter -t errorfile PVS-Studio.log --cerr -w

Let's take a look at the following part of the code:

PWD=$(pwd -L)
if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then
  PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
  MERGE_BASE=`wget -qO - 
   https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} 
   | jq -r ".base.ref"`

  git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck 
                              --dump-files --dump-log pvs-dump.log 
                              -S .pvs-pr.list
else
  pvs-studio-analyzer analyze -j8 
                              -o PVS-Studio.log 
                              --disableLicenseExpirationCheck
fi

The rather specific assignment of the value of the pwd command to a variable that should store this default value seems strange at first glance, however, I will explain everything in a moment.

While setting up the analyzer in AppVeyor, I encountered an extremely strange behavior of the analyzer. On the one hand, everything worked correctly, but the analysis did not start. I spent a lot of time noticing that we are in the /home/appveyor/projects/testcalc/ directory, and the analyzer is sure that we are in /opt/appveyor/build-agent/. Then I realized that the $PWD variable is a bit of a lie. For this reason, I manually updated its value before starting the analysis.

And then everything, as before:

Now consider the following snippet:

PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER"
MERGE_BASE=`wget -qO - 
  https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} 
  | jq -r ".base.ref"`

In it, we get the difference between the branches on which the pull request is declared. For this we need the following environment variables:

$APPVEYOR_PULL_REQUEST_NUMBER - pull request number;
$APPVEYOR_REPO_NAME - username and project repository.

Conclusion

Of course, we have not considered all of the possible continuous integration services, however, they all have very similar work specifics. With the exception of caching, each service makes its own "bike", so everything is always different.

Somewhere, like in Travis-CI, a couple of lines of code and caching works flawlessly; somewhere, as in AppVeyor, you just need to specify the folder in the settings; but somewhere you need to create unique keys and try to convince the system to give you the opportunity to overwrite the cached fragment. Therefore, if you want to set up pull request analysis on a continuous integration service that was not discussed above, then first make sure that you will not have problems with caching.

Thank you for your attention. If something does not work out, then feel free to write to us at support. We will advise and help.

If you want to share this article with an English-speaking audience, please use the translation link: Maxim Zvyagintsev. Analysis of commits and pull requests in Travis CI, Buddy and AppVeyor using PVS-Studio.

Source: habr.com

Analysis of commits and pull requests in Travis CI, Buddy and AppVeyor using PVS-Studio