Get test coverage for software written in Rust using Kcov

2018-01-31 12:04:00 UTC

To get test coverage for your Rust code, you can use kcov.

% cargo test
% kcov --include-path ./src ./target/coverage target/debug/<name>-*

It generates pretty HTML and JSON report files that include also coverage score. And, this one liner can extract percentage value from index.json, if you want.

% grep -oE 'covered":"([0-9]*\.[0-9]*|[0-9]*)"' target/coverage/index.json | \
    grep -oE '[0-9]*\.[0-9]*|[0-9]*'
4.5

Ohh, 4.5% .. !!! Anyway, if you run the test on GitLab CI, there is pattern matching on general settings, So we can just like this:

: in .gitlab-ci.yml
% cat target/coverage/index.json

And then, use pattern something like this:

"covered":"(\d+(?:\.\d+)?)",

This seems that it usually works well. But I faced still some problems on CI.

Trouble Shooting

SIGSEGV

According to issue #212 (SimonKagstrom/kcov), you can pass --include-path to avoid this SIGSEGV error.

With this option, it worked fine also on my code. But, for now, in some project, it seems that it might not work, properly. I found pull request #55 (sagiegurari/cargo-make).

Cannot open linux-vdso.so.1

In local machine (Gentoo Linux, 64bit), kcov which is installed via Portage, works fine with my Rust code, but I faced this error on GitLab CI (Debian). This problem also discussed on issue #26 (SimonKagstrom/kcov).

I've installed kcov via apt-get (with cache configuration).

test:
  stage: test
  image: rust:latest
  ...
  variables:
    APT_CACHE_DIR: apt-cache
  before_script:
    # this lines install kcov
    - mkdir -pv $APT_CACHE_DIR && apt-get -qq update
    - apt-get -qq -o dir::cache::archives="$APT_CACHE_DIR" install -y kcov
  ...
  cache:
    untracked: true
    paths:
      - apt-cache
  ...

test:
  ...
  after_script:
    - kcov --include-path src target/coverage target/debug/<name>-*
    - cat target/coverage/index.json
  ...

: output on GitLab CI
$ kcov --include-path src target/coverage target/debug/20min-*

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out

Error: Cannot open linux-vdso.so.1

Hmm, it seems that kcov on Debian is a little bit old (11.1?). And it does not have even `--version` option. See links below (In 01.02.2018, 11.1 is still stable version.)

So, finally, I wrote a build script using newest version (version 34) with cache support on CI.

Build Script

This is a small bash script to build/install kcov into kcov directory on project root.

#!/bin/bash
set -eu

# NOTE:
# if set KCOV_DISCARD_CACHE=true, then it will force installing kcov)

renew="${KCOV_DISCARD_CACHE:-false}"

kcov_dir="kcov"
kcov_bin="${kcov_dir}/bin/kcov"
kcov_url="https://github.com/SimonKagstrom/kcov/archive"
kcov_ver="v34"

if [[ -f "${kcov_bin}" && "${renew}" != "true" ]]; then
  echo "kcov already installed in ${kcov_bin}"
else
  rm -fr $kcov_dir
  mkdir $kcov_dir
  cd $kcov_dir
  curl -sLO ${kcov_url}/${kcov_ver}.tar.gz
  mkdir $kcov_ver
  tar zxvf ${kcov_ver}.tar.gz -C $kcov_ver --strip-components=1
  cd $kcov_ver
  mkdir build
  cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/
  make
  make install DESTDIR=../
fi

And .gitlab-ci.yml will be something like this ( In addition to dependencies listed in INSTALL.md of kcov, you need also cmake):

test:
  stage: test
  image: rust:latest
  variables:
    KCOV_DISCARD_CACHE: "false"
    APT_CACHE_DIR: apt-cache
  before_script:
    - mkdir -pv $APT_CACHE_DIR && apt-get -qq update
    - apt-get -qq -o dir::cache::archives="$APT_CACHE_DIR" install -y
      binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev
      cmake
    - ./bin/build-kcov
    - rustc --version
    - cargo --version
    - ./kcov/bin/kcov --version
  script:
    - cargo test
  after_script:
    - ./kcov/bin/kcov --include-path src target/coverage target/debug/20min-*
    - cat target/coverage/index.json
  cache:
    untracked: true
    paths:
      - apt-cache
      - kcov
  except:
    - tags

Set KCOVDISCARDCACHE as true, if you need to force re:install kcov. The kcov directory will be normally cached on GitLab CI!

I made also make cov target to check it on local, and a ci-runner script run CI docker container on local.

Check it out on my small project repo grauwoelfchen/20min ;)

References