Code coverage show-off

If you are like me, than you have a deep desire to write quality code. Sadly, writing unit-tests takes time that you cannot put into new features. Writing unit-tests does not give you the feeling that you are "making progress", especially if you are working on a software that needs a lot of them.

Well, I have to solution to your problem: "Visible test coverage".

Let's say, you have started an open-source project. It was little in the beginning and you were playing around a lot. You put it on GitHub and on npm, just in case anyone wants to use it. And after a period of wild-coding, you notice that you have just created a non-trivial code-base. You also notice that you haven't written a single unit-test and that people are starting to use your program.

Time to write some tests

Time to write some tests, but there is always one problem: Of course, you are working on the project in your free time (mostly evenings and week-ends). Also, you have a live, hobbies and possibly kids. So you are in the same situation as most developers in their daytime jobs nowadays: You don't have enough time.

Not enough time to work on documentation, not enough time to work on tests, because if you are working on this in your free-time, you want to have fun doing it. You want to see progress. You want others to see progress as well. Otherwise your precious time feels wasted.

Visible test-coverage

I encourage you: Write those tests, write all the tests needed to cover the use-cases of your project. If you want to use it in production, if others should use it in production, it would be reckless not to write tests. But how can you preserve the feeling of progress without implementing new features.

Easy: Visible test-coverage. Place badges on your project. There are a lot of services out there, that are free for use with public GitHub-repos that allow you to run tests, collect code coverage reports, check coding-style and other quality-measures. You can put badges in your README that show the current state off your project. These are the badges of my thoughtful-release project:

Travis Build Status Appveyor Build Status Coverage Status

Now, everybody (including yourself) will see that your code coverage is at only 50% and everybody will see that it is increasing over time. You can write tests and see the progress.

And you can use the coverage reports to find untested pieces of code, like in this screenshot of http://coveralls.io.

Coveralls, Travis, Appveyor

I am currently using the following services in my projects:

  • Travis CI is continuous integration service for Linux and OSX.
  • Appveyor is a continous integration service for Windows
  • Coveralls is a service that collects and displays code-coverage reports.

You can sign into all of these services via GitHub; and they are free - as long as you just want to connect public GitHub repositories.

I have set up all three services for my latest NodeJS project thoughtful-release. Here is how you can do it as well:

Travis

Once you sign via with your GitHub you can add your repositories to Travis

Click here to add repos

Select any number of repos you want to execute tests for.

Of course, Travis must have additional information about each repository. What platform
is it running on? What commands should be executed? Those things are specified in a file called .travis.yml in the root-folder of your repository. For my NodeJS projects, I use the following file (derived from the one in the coreflow-templates by Charlike Mike Reagent:

sudo: false  
language: "node_js"  
node_js:  
  - "0.10"
  - "0.12"
  - "iojs"
  - "4"
  - "5"
before_script:  
  - npm install standard
  - standard
script:  
  - npm install istanbul
  - istanbul cover ./node_modules/.bin/_mocha --report lcovonly
after_script:  
  - npm install coveralls
  - cat ./coverage/lcov.info | coveralls

This is actually not the file for thoughtful-release, because this project requires NodeJS 4 at minimum.

I should also mention that this StackOverflow question helped me setting up mocha with istanbul)

The tells Travis to that this is project requires NodeJS and which Node versions it should be tested with. Before the script is executed, the StandardJS style-checker is installed and run. It is not part of the dependencies, because it is rather large and shouldn't be downloaded into every single project on your desktop.

The script itself consists of running the mocha tests with the istanbul-code-coverage tool. Afterwards, we send the coverage reports to Coveralls.

These actions are performed for each commit that is pushed to GitHub and for each Pull-Request, so you always have confidence that your tests run on all kinds of Node versions.

Note: Travis also supports OSX, I have not included that yet. I will update this blog-post, when I do

Coveralls

In the previous section, I wrote that the after_script-part of .travis.yml sends the coverage data to coveralls. Of course, you have to create activate your repository at Coveralls before this works.

Coveralls - add repository

As in Travis, you can then switch your GitHub repositories on and off:

Afterwards, push a commit to any branch. You should see Travis starting to work and when its finished, you should see the coverage report on Coveralls.

Appveyor

Travis CI does not support Windows yet but Appveyor does. So we do the same as before: Sign in and select a repository:

Appveyor reads its build metadata from a different file: appveyor.yml. A sample file for NodeJS can be found in the documentation. Here is the file I am using in thoughtful-release. Note that I cancelled support for NodeJS older then 4.0:

# Test against this version of Node.js
environment:  
 matrix:
  - nodejs_version: "4"
  - nodejs_version: "5"

# Install scripts. (runs after repo cloning)
install:  
  # Get the latest stable version of Node.js or io.js
  - ps: Install-Product node $env:nodejs_version
  # install modules
  - npm install

# Post-install test scripts.
test_script:  
  # Output useful info for debugging.
  - node --version
  - npm --version
  # run tests
  - npm install -g istanbul
  - istanbul cover node_modules/mocha/bin/_mocha --report lcovonly
  - npm install -g coveralls
  - cat ./coverage/lcov.info | coveralls

# Don't actually build.
build: off  

If you are confused by the last line: build: off refers to the usual build-process of Appveyor, which seems to adjusted for .NET-builds. We don't want .NET but NodeJS, so we put everything in the test_script instead.

git push origin master and watch the build running.

After fixing thoughtful-release and its test-cases, which didn't pass at all on Appveyor's Windows-machine, there was one issue with the configuration of this CI service.

While Travis just worked with Coveralls, the [Appveyor-build](https://ci.appveyor.com/project/nknapp/thoughtful-release/build/1.0.28/job/lmq1p26f7nwr6b1q
) ended with an error: Couldn't find a repository matching this job. This was actually an error reported back from Coveralls to Appveyor, because of missing configuration:

The documentation of the node-coveralls-module states that two environment variables must be set in order to send the coverage-reports to Coveralls:

  • COVERALLS_SERVICE_NAME - the name of the service ("appveyor")
  • COVERALLS_REPO_TOKEN - the access-token for the repository.

Without the token, the client is not authorized to access to repository, which results in the above error:

When you open the project-page in Coveralls, you see the access-token in the lower-right corner.

Retrieving the Coveralls repo token

You can then copy its value and add the proper environment variables to the Appveyor project configuration: Settings > Environment > Add variable

Appveyor environment variables

After that, hopefully, commits to any branch and pull-requests should invoke the build and cause coverage reports to be sent to coveralls.

Merging the coverage reports of both CI-builds

What I actually wanted to achieve with setting up builds for multiple platforms, was a merged code coverage: The snippet

function describeOS () {  
  var os = require('os')
  switch (os.type()) {
    case 'Linux':
      return 'this is Linux'
    case 'Darwin':
      return 'this is OSX'
    case 'Windows_NT':
      return 'this is Windows'
    default:
      throw new Error(`Unexpected OS-type ${os.type()}`)
  }
}

will never get 100% code coverage if you only run tests on a single platform. You have to merge the results of Travis and Appveyor to achieve that. Unfortunately, Coveralls is currently not able to do that. Instead, you will get two coverage reports and the last one will be shown by default.

If I find a way to merge reports, I will update this post.

Adding badges:

The documentation of all services contain information about how to include badges: TravisCI status-images, Appveyor status-badges. Coveralls seems to be begging me to add a badge to my repo (although I already have one):

Coveralls status badge

In thoughtful-release, I am using the following code to include the badges in the README

[![Travis Build Status](https://travis-ci.org/nknapp/thoughtful-release.svg?branch=master)](https://travis-ci.org/nknapp/thoughtful-release)
[![Appveyor Build Status](https://ci.appveyor.com/api/projects/status/github/nknapp/thoughtful-release?svg=true&branch=master)](https://ci.appveyor.com/project/nknapp/thoughtful-release)
[![Coverage Status](https://img.shields.io/coveralls/nknapp/thoughtful-release.svg)](https://coveralls.io/r/nknapp/thoughtful-release)

I am not recreating those badges for every project. I am creating the README with Thought, my documentation generator for GitHub projects. It now generates Travis, Coveralls and Appveyor badges depending on the existance and contents of the configuration files and based on the GitHub-URL provided in the package.json.
Sadly, Thought itself does not run on Windows (yet) and has no code-coverage tools enabled. That may be one of the next things to do.

What's next?

Apart from slowly adding coverage badges to all my projects, I think the whole service setup should be automated. All services also provide Rest-APIs for their configuration interfaces. I might integrate their configuration into thoughtful-release.
After all, we should rather write many small modules than few large ones. Many small modules means: "Often log in to Travis and add a repo, often log in to Coveralls and add a repo, often log in to Appveyor and add a repo".
This must be easier or noone will do it...

What about you?

What services are you using to ensure code quality of your open source modules? I'm curious.

Disclaimer

I don't have much time and I really didn't want to postpone this post for another week. There maybe issues with grammar and spelling, because I didn't really proofread everything (and I don't have a editor). Forgive me that.