Run Firefox Headless Browser tests on GitLab CI

2018-03-18 06:18:04 UTC

| You have some tests on Firefox Headless mode, and that would work fine on | your local machine. For example, using Karma:

% npm run karma

: run just it like normal
% ./node_modelus/.bin/karma start karma.conf.js

| I wanted here to use karma-firefox-runner as "FirefoxHeadless" for the | tests. (e.g. my project Vergil)

It succeeded well on my local machine. But I faced some issues to run it on CI (GitLab CI).

This is a note to resolve the issues.

Browser is missing

: on CI $ npm run karma > @0.0.1 karma /builds/grauwoelfchen/ > karma start karma.conf.js # tests 0 tests 0 # pass 0 pass 0 # ok ✓ oknpm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! @0.0.1 karma: `karma start karma.conf.js` npm ERR! Exit status 1 npm ERR! ...

This error is likely happend on a situation, that the script can't find the certain browser installed there. In my case, I just forgot to install the Firefox on this environment, and installed it on strange location ;)

| So, update .gitlab-ci.yml. It will look like this. | (This sample is on Node.js image, just installes firefox-esr on Debian)

test:
  stage: test
  image: node:8.9.4
  services:
  variables:
    NODE_ENV: test
    APT_CACHE_DIR: apt-cache
  before_script:
    - mkdir -pv $APT_CACHE_DIR && apt-get -qq update
    - apt-get -qq -o dir::cache::archives="$APT_CACHE_DIR" install -y
      firefox-esr
    - firefox --version
    - node --version
    - npm --version
    - npm install
    - npm run build
  script:
    - npm run karma
  cache:
    untracked: true
    paths:
      - apt-cache
      - node_modules
  except:
    - tags

GDKBACKEND does not match

: on CI $ firefox --version Mozilla Firefox 52.7.2

The browser (Firefox) is now installed, However it won't run tests, and fail to start it on CI environment, because it does not have GDK for the UI and DISPLAY.

18 03 2018 06:45:30.601:ERROR [launcher]: Cannot start FirefoxHeadless Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.604:ERROR [launcher]: FirefoxHeadless stdout: 18 03 2018 06:45:30.605:ERROR [launcher]: FirefoxHeadless stderr: Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.638:ERROR [launcher]: Cannot start FirefoxHeadless Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.638:ERROR [launcher]: FirefoxHeadless stdout: 18 03 2018 06:45:30.638:ERROR [launcher]: FirefoxHeadless stderr: Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.668:ERROR [launcher]: Cannot start FirefoxHeadless Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.669:ERROR [launcher]: FirefoxHeadless stdout: 18 03 2018 06:45:30.669:ERROR [launcher]: FirefoxHeadless stderr: Error: GDK_BACKEND does not match available displays 18 03 2018 06:45:30.670:ERROR [launcher]: FirefoxHeadless failed 2 times (cannot start). Giving up. # tests 0 # pass 0 ✓ ok

Hmm, these tests work on HEADLESS browser. Why does it need those libraries for UI components?

Appearantly, it seems that for right now, Firefox Headless requires some those dependencies what are not even used.

See: https://developer.mozilla.org/en-US/Firefox/Headless_mode

(from "Troubleshooting and further help" section)

On Linux, certain libraries are currently required on your system, even though headless mode doesn't use them, as Firefox links against them. See bug 1372998, for more details and progress towards a fix.

This little bit confused me. I didn't notice that the issue on my local, because it already have those libraries.

On Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1372998

They reported this issue, and talked.

GLib may be unavoidable, but GTK+ and especially Xvfb (and any another X11 implementation) should be optional for Linux systems that run Firefox headlessly.

This comment helped me. It seems that xvfb is needed, at least, for now.

| So let's run tests using `Xvfb` (xvfb-run). | Install `xvfb-run` for Gentoo Linux, `xvfb` for Debian.

: on Gentoo Linux
❯❯❯ equery l -po xvfb-run
 * Searching for xvfb-run ...
[--O] [  ] x11-misc/xvfb-run-1.18.4_p2:0
[I-O] [  ] x11-misc/xvfb-run-1.19.3_p2:0

And check it on local machine (especially to run it fine on Docker container like CI environment).

Run tests xvfb-run on local

: check it on local
% xvfb-run npm run karma

Run tests on local GitLab CI Container

I always do it with local runner using following scripts.

At first, Install runner binary provided from GitLab.

NOTE This runner is still old version. I should migrate it to new one. But it's next time ;)

#!/bin/sh
set -eu

bin_dir=$(dirname $(readlink -f "${0}"))
name="gitlab-ci-multi-runner"
platform="linux-amd64"
version="latest"

indent() {
  sed -u 's/^/       /'
}

echo "Platform: ${platform}"
echo "Version: ${version}"
echo ""
echo "-----> Installing into: ${bin_dir}"

location_base="https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com"
location="${location_base}/${version}/binaries/${name}-${platform}"

curl -sL $location -o $bin_dir/$name

chmod +x $_

echo "Done" | indent

And start docker. As next, create `ci-runner` script like below.

#!/bin/sh
set -x

dir_name=$(dirname $(dirname $(readlink -f $0)))
job_name="${1}"
env_file="${ENV_FILE:-.env.ci}"
env_opts="";

if [ -z "${job_name}" ]; then
  echo "Please specify job name in .gitlab-ci.yml"
  exit 2
fi

if [ ! -f "${env_file}" ]; then
  echo "Please create \`${env_file}\` (cp ${env_file}.sample ${env_file})"
  exit 2
fi

while read line; do
env_opts+=" --env ${line}"
done < "${env_file}"

# ci-runner <job>
echo "${env_opts} ${job_name}" | \
  xargs $dir_name/bin/gitlab-ci-multi-runner exec docker \
  --cache-dir /cache \
  --docker-privileged \
  --docker-volumes $dir_name/tmp/_cache:/cache \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

| Set secret environment variables in `.env` like you do on CI, as you need, | then just run it.

% cp .env.ci.sample .env.ci

: this script takes job name as argument
% ./bin/ci-runner <JOB-NAME>

It will run tests almost same as GitLab CI. You can check it before pushing it on the remote.

Conclusion
  • Use `xvfb` (xvfb-run) for test on Firefox Headless on CI (Firefox 52, 18. March 2018)

  • Check it to run on your local machine same with the CI

Finally, my `.gitlab-ci.yml` looks like this:

test:
  stage: test
  image: node:8.9.4
  services:
  variables:
    NODE_ENV: test
    APT_CACHE_DIR: apt-cache
  before_script:
    - mkdir -pv $APT_CACHE_DIR && apt-get -qq update
    - apt-get -qq -o dir::cache::archives="$APT_CACHE_DIR" install -y
      xvfb firefox-esr
    - firefox --version
    - node --version
    - npm --version
    - npm install
    - npm run build
  script:
    - xvfb-run npm run karma
  cache:
    untracked: true
    paths:
      - apt-cache
      - node_modules
  except:
    - tags

Thank you Gitlab, for the great runner, and Mozilla, for Firefox Headless!

Happy testing ;)

References