This post is the text version of a talk I gave at the Melbourne Ruby meet-up earlier this year. I thought some people would prefer the information presented in plain text over the video. For anyone interested, I’ve embedded the video below.
If you don’t want to watch the video, or you found the text on screen hard to see, read on…
If you’re writing a gem, you want to give people the confidence that it works as described - otherwise they might not want to pull it into their project. Part of doing this involves writing good tests.
Most gems have dependencies and most also support multiple versions of Ruby. This presents something of an issue because we need to test our gem against all its dependencies and run these tests using all the different versions of Ruby we support. Only then can we have confidence that it works as expected.
This article is going to focus on using Travis CI to implement and execute our test matrix. If you use a different CI platform, I’m sure the ideas will be broadly applicable, but the specifics will probably be different.
I’ll assume you have a
Rakefile in your project with a
test task set as the default. Unless asked otherwise, Travis simply executes
rake on each Ruby job run, so this is a simple way of getting it to run your test suite.
What Would Travis Do?
Before we get into the nitty-gritty I thought it might be helpful to share a gem that will improve your experience getting all this working.
Setting up this stuff for the first time in Travis can be a bit tedious. You make your changes, commit them, push them, wait for Travis to enqueue the job, wait for the jobs to build and run, figure out why they failed, rinse, repeat.
Enter, What Would Travis Do? a gem that (as its name suggests) simulates the Travis job runner on your local machine.
Setup is a doozy, just install the gem and run
and it takes care of the rest. This does, of course, mean you need to have all the different versions of Ruby you’re testing against installed on your local machine (see here for a quick guide), but it’s worth the hassle just for the improvement in the development experience.
When you pull Travis CI into your project, you create a
travis.yml file that tells Travis how to build and run your jobs. Part of doing this in a Ruby project is providing it with a Ruby version number under the
rvm header. If you specify multiple Ruby version numbers under
rvm, Travis will automatically execute your jobs in each Ruby environment. So, basically, we get multi-Ruby testing out of the box for free.
# travis.yml language: ruby rvm: - 2.3 - 2.4 - 2.5 - 2.6
This doesn’t completely solve the problem, though, becuase we’re still testing our gem against only the dependencies defined in the project
However, Travis also supports running jobs with multiple
Gemfiles, so if we define a separate
Gemfile for each version of our dependencies, we’ll get our test matrix: one job for each
Gemfile and each version of Ruby.
Rather than manually writing out each
Gemfile, we can use the Appraisal gem to simplify the process. Appraisal creates multiple gemfiles by using your project Gemfile or gemspec as a base and then selectively overriding the versions of certain gems you define in its config.
In brief, you need to create a file called
Appraisals in the root of your project directory. Within this file, call the
appraise method and pass it a string identifier and a block containing the gems you want to override. In the example below you can see we’re testing against different versions of ActiveRecord.
# Appraisals appraise 'AR_4.2' do gem 'activerecord', '~> 4.2.11' gem 'sqlite3', '~> 1.3.6' end appraise 'AR_5.0' do gem 'activerecord', '~> 5.0.7' gem 'sqlite3', '~> 1.3.6' end appraise 'AR_5.1' do gem 'activerecord', '~> 5.1.7' gem 'sqlite3' end # and so on, and so forth...
Once you’re done, run
$ bundle exec appraisal install
Appraisal will generate the gemfiles and place them under a
gemfiles/ folder in your project. Then, you need to add them to your
travis.yml file like so:
# travis.yml gemfile: - gemfiles/dependancy-v1.gemfile - gemfiles/dependancy-v2.gemfile - gemfiles/dependancy-v2.gemfile
Now when you kick off a Travis build, it will create a job for each gemfile and each version of Ruby you specified.
Improving the Test Matrix
Now that we’ve created a basic test matrix, it’s time to trim it down to something more useful. You’ll probably find that some of the tests are failing or pointless because you’re now running up-to-date libraries or dependencies against old versions of Ruby, or testing the latest version of Ruby against old libraries like ActiveRecord v4.2.
We can improve the matrix by removing redundant and pointless tests by telling Travis to ignore certain combinations using the
exclude header like so:
# travis.yml matrix: exclude: - rvm: 2.3 gemfile: gemfiles/AR_6.0.gemfile - rvm: 2.4 gemfile: gemfiles/AR_6.0.gemfile - rvm: 2.7 gemfile: gemfiles/AR_4.2.gemfile
In the above example, I’ve excluded testing the ActiveRecord 6.0 Gemfile against Ruby 2.3 and 2.4 because ActiveRecord 6 requires Ruby >= 2.5. I’ve also excluded testing AR 4.2 against Ruby 2.6 becuase it seems a bit pointless (let’s be honest, if anyone is using AR 4.2 in their project when they pull your gem in, they’re extremely unlikely to be using such an up-to-date version of Ruby).
All Together Now
After all this, you should have a
travis.yml file that contains something like the following, plus any additional instructions required to get your jobs running.
# travis.yml language: ruby cache: bundler rvm: - 2.3 - 2.4 - 2.5 - 2.6 gemfile: - gemfiles/AR_4.2.gemfile - gemfiles/AR_5.0.gemfile - gemfiles/AR_5.1.gemfile - gemfiles/AR_5.2.gemfile - gemfiles/AR_6.0.gemfile matrix: exclude: - rvm: 2.3 gemfile: gemfiles/AR_6.0.gemfile - rvm: 2.4 gemfile: gemfiles/AR_6.0.gemfile - rvm: 2.6 gemfile: gemfiles/AR_4.2.gemfile
Keep in mind that I don’t think running these jobs are worth it during normal development, once I have a test matrix like this set up, I only use it in “pre-merge” test jobs to make sure pull requests are able to be merged into master. This ensures changes don’t cause any inintended regressions, but allows development to proceed with a minimum of friction.