Day 10 - Investigate your slowest test(s)

Today’s task is to investigate your slowest tests.

If you happen to use Ruby and RSpec, finding your 10 slowest tests is simply a matter of adding --profile to your test command. If you’re using something else, hopefully a similar flag is only a quick google away.

Once you’ve identified your 10 slowest tests or so, give them each a once-over.

Some things to consider:

  1. Are any of these tests duplicates? It’s easier than you might think to accidentally write the same test more than once. You can’t see that you’ve done this by reviewing a diff in a PR, so this type of duplication can often creep into your codebase over time.

  2. Can any of these tests be replaced with a faster variant? If you’re testing a conditional in the view, can you use a view spec instead of an integration spec? Can you stub expensive calls? Can you perform less setup? Can you hit less of the stack and still feel confident things work? (This video I recorded on integration vs. unit tests)[https://www.youtube.com/watch?v=kBOqaluDf2k] might be helpful.

  3. Are all these tests still pulling their weight? Remember: slow tests act as a constant drag against your forward progress. Make sure the cost is outweighed by the benefits of keeping them around. I encourage you to be pragmatic. If a very slow test is verifying something that’s not mission-critical, it’s worth considering deleting it.

As always, please share how things go. Did you end up making any changes?

Using maven and surefire? This might help: https://gist.github.com/stianlagstad/e3b3aaca66209a122159731cbf325100

Since I don’t have any tests in place yet I used the 20mins to do some research what possibilities are there nowadays for JS testing.

I found a tool for slow tests in phpunit https://github.com/johnkary/phpunit-speedtrap
There are ~ 20/230 test, that took more than 150ms on cheapest MBP2017, but all of them need to interact with the db, so I dont think I can make them faster.

Slowest suite for us was a collection of 6 or so tests which on our CI server (Jenkins) takes about 2 seconds to run.

Running the same tests locally, they take about 0.5 seconds. Not sure if that’s because my local machine is more powerful than the Jenkins worker, because they’re being run slightly differently (on jenkins its in a container, locally it’s not, etc), or because of the DB (this is a Django project, so each test does hit a DB which is a Django thing, and when I run the tests locally it’s hitting a local postgresql instance, on Jenkins it’s a remote postgresql instance).

Digging into the tests a bit, it appeared as though (at least locally) that half the execution time was a call to a library we use for fixing the system time (freezegun) which is super handy, but turns out to be fairly expensive.

That was about where I got to in the 20 minutes, so no code change, but some insight on how expensive that library is.

1 Like

On my project, we have unit tests that run in one CI job, and the full test suite (which adds the feature specs) that runs in another. The first, unit-test-only job is also the one that handles continuous deployment when it’s on the ‘staging’ or ‘production’ branch.

So while the overall suite has feature specs that are the slowest ones, I value shaving time off the unit tests more, because that speeds up deployment. (The slowest feature specs are also among the most useful, catching UI bugs that would otherwise sneak through without causing errors, and can’t easily be sped up.)

So, I made some tweaks to the slowest Ruby unit tests, shaving off maybe 10 seconds from the ~2.30 of unit tests. I wish I could shave more time off of all the travis-ci setup before the tests start, though. The actual builds take about 8-10 mins, but less than half is actually running the tests. (That also includes building the javascript assets and running the javascript tests, which take about 1:40.)

We’re using Google Test, which automatically provides timing information, for our C++ unit testing. The slowest test case (by far) took 100ms per run, all of which was one-time setup, so I moved the setup out into the test suite.

1 Like

I found out that my slowest spec was too slow because of a problem in it’s setup. I was creating data in a common before callback, creating a lot of data in each expectation.

I put my setup inside a before(:context) callback and improved the speed in all the expectations inside this context in almost 3 times.

Thanks, ben

We use Jasmine as the test suite for our node app. Unfortunately I could not find any configuration with Jasmine to get a similar output to --profile with rspec :frowning:

After poking around I found some people had tried this but mostly all on outdated versions of jasmine that relied on a global jasmine variable or were otherwise outdated for current versions.

As a result, I ended up spending more than 20 minutes on this task and writing my own jasmine reporter middleware that could do this and output the tests that are above a certain threshold (100ms by default).

Here’s the package I made, if you find another one that works let me know, maybe it’s better.

yarn add jasmine-slow-spec-reporter

If anybody uses this please let me know! I haven’t tried it with a frontend jasmine setup like Karma but I think it would work just fine, as long as you can inject your own reporter.

Running this on our node app, for 716 specs we had 11 that took > 200ms. This is what the output looks like:

2 Likes

You might consider trying to switch to Jest. It has slow-spec flagging by default, and when I made a similar switch, it was amazing how easy it was to substitute. The switch to Jest also means your JS tests get run in parallel, so it might be a huge speedup for very little work.

1 Like

Oh wow. I just made our slowest test 9x faster because of this: https://blog.codeship.com/faster-rails-tests/

Even though we use Rspec which supposedly optimizes assertions in this form:

expect(page).not_to have_content("abc")

I had created a custom matcher called “be_a_form_with_errors” which rspec doesn’t optimize. I ended up creating a new negated matcher for those cases and started using:

# This:
expect(page).to be_a_form_without_errors

# Instead of this:
expect(page).not_to be_a_form_with_errors

Overall this saved us a few minutes from a full test run, because we were using this in a few spots. Great success! :raised_hands:

1 Like

Glad to have learned about --profile for RSpec.

I looked into the slowest specs that came up but didn’t see anything obvious, so I called it a day. I think falling behind on the challenges this week made me want to rush to “check the box” on this one. Hopefully, I can get back on track next week!

Thanks. I have used jest with frontend projects, but haven’t run it on a node project. Good idea, this is a bit of a legacy project, I’ll see if I can get it running on the backend.

Our test suite degraded badly over time so I’ve started recovering it… still working on getting it back to a clean running state. I’ll add a ticket to check the testing times when it’s done.

For some reason, the package that I was using was slower than the actual one. Now I’m using node-fetch and all the tests reduce their time (not a lot but a little bit)