Tests Pass. You Fail.

Shit breaks. It’d be so much cooler if things just worked the way we expected, but they don’t. For better or worse, we live in a world where everything eventually fails. And software is definitely not an exception.

So we write tests. If 1 + 1 should be 2, we make assertions that that’s always the case. Seems like an all-upside no-brainer.

But could your tests being doing more harm than good?

You Stop Looking Forward

Just because your tests look back doesn’t mean you should. In fact the exact opposite is true. As a developer you have to keep your head up, scanning the horizon for opportunities. Ideally, you’re looking for opportunities to make your product better. But even if you’re not responsible for the decisions that create happier customers, babysitting your test suite is still probably not the best use of your time.

If you’re interested in actually moving the needle, perhaps the better benchmark is not how often your tests pass, but how often they fail.

You Lose Sight of Your Customer

Passing tests are quite a high. Engineers are always looking for cold hard facts. Passing tests take the edge off: “I wrote code to solve a problem, and now this test I wrote proves that I really solved the problem!”

It can be a good thing. If the problem matters and the tests actually test the right things then it’s a great thing. But it’s easy to chase that high by writing tests that don’t test the right thing, test something that’s already tested or, worse, pass despite a feature that still doesn’t work correctly.

Don’t lose sight of your customers in spite of your test coverage.

You Fear Failure

The tolerance for failure is probably a lot higher than you imagine; your customer is probably much more forgiving than you (or your boss) think, especially if you handle it the right way.

OK, that’s not always true. If you’re engineering the rules for credit card approval, the launch sequence for a missle, or the internals of a pacemaker, most of this post is irrelevant. But if you’re like me, trying to break through to meaty part of the adoption curve, then isn’t the ultimate test simply feedback from your users?

If you think about your product from that perspective code coverage suddenly becomes less important. Are users signing up? Are they coming back? Are they raving fans? Is your product a vitamin or a pain-killer? Make those tests pass.

90% of coding is debugging. The other 10% is writing bugs

– @bramcohen

Show-stopping bugs are easy to prevent. At Sqoot we use a few tools that I consider just as important as rake test. Squawk, a gem I wrote that let’s our app tweet, gives us live feedback (good, bad and ugly). Pingdom hits our endpoints every minute to ensure uptime. Hoptoad is invaluable to trap and report on errors.

So I say release some bugs! The benefit of being 10-20% more nimble over the lifetime of a project totally outweighs the cost of breaking things from time to time.

You Slow Down

No matter how lean and mean you think you are tests are a process and process can slow you down:

1. Chasing 100% coverage: Once you start testing, you don’t want to stop. Untested code becomes a second-class citizen. But that’s fine! Not all code is created equal.
2. Calcification: Deleting code is awesome, but when it’s tested, you think twice. This is especially true when you’re new to a project.
3. You become the customer: Tests are one part of your app where the customers are other developers. It’s easy to get lost in a world of tools and process. Unless you’re GitHub, your product is probably not other developers.

“New process is reluctantly introduced only right before the point where things tip into chaos”

– Yishan Wong on Facebook

Now, Before You Roast Me on HN…

By this point, I’ve probably ruffled some feathers. So let me be clear that testing isn’t all downside. There are several critical reasons you have to test:

1. Implementation first thinking: When you write tests you end up thinking about how other parts of your application will use that code. This means less refactoring down the line.
2. Free regression suite: Rarely do engineers go back to test untested code, so it’s good to know that the feature you built works now and will forever. This is especially true for teams.
3. Framing the problem: Originally for PiggyBack.it, I built some code that balanced debts between a bunch of people. To think about that problem conceptually would have taken me forever to solve. By laying out the expectations first, the solution came naturally.

See, even I write tests!

Loaded suite /Users/avand/.rvm/gems/ruby-1.9.2-p0/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
..................................................................................................................................................................................................
Finished in 5.778192 seconds.

194 tests, 290 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 50633
Loaded suite /Users/avand/.rvm/gems/ruby-1.9.2-p0/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
..............................................
Finished in 13.691397 seconds.

46 tests, 56 assertions, 0 failures, 0 errors, 0 skips

If you’re saying to yourself “13 seconds to run 46 tests is slow,” you may want to re-read this post or double-down and check out riot.

Happy Customers are All That Matters

I don’t enjoy writing code for code’s sake. It’s ironic to think you might even evaluate a software engineer by the code he doesn’t write. Coding is fun, it’s addictive and it’s a bit hypnotic. So you have to be really careful that the code you’re writing actually matters!

Are you making your customer happier?

If the line of code you’re working on doesn’t answer that question with a huge “yes” then you’re probably wasting your time.