Dave Schweisguth in a Bottle

Recently I made an error in a Ruby program (a Rails application) that I wouldn’t have been able to make in a Java program. I renamed a model method manually — because another class had a method with the same name, and RubyMine’s Rename Method refactoring would have renamed both methods — and although I renamed the method itself and uses of the method in the model object’s spec, I missed uses of the method in a controller and mocks of it in the controller’s spec. All of the tests passed, but the application broke. “Gee,” I thought, “if I’d had to compile everything first, that couldn’t have happened.”

After a little more thought I realized that while it was true that I couldn’t have gotten that specific error past a compiler, and in fact I wouldn’t even have made it in a language that would only consider two methods of two different classes to be ‘the same’ if they implemented a method which appeared in a supertype, that kind of error is only a small subset of a much larger category of errors that can occur in any language. Instead of calling a nonexistent method, the controller might have called the wrong method, or it might have called the right method with the wrong parameters or expected a different range of return value or a different exception than the method actually had. In Java, the controller might have called an overloaded version of the same method. To catch such errors, in a perfect world every method call would be integration-tested regardless of language. What about in our imperfect world?

In an application that’s fully unit-tested (a reasonable and useful thing to achieve even in an imperfect world), every method will be unit-tested with all of the combinations of inputs and outputs that are needed for the application to function, and every method X that calls another class’s method Y will be unit-tested with a mock implementation of Y which accepts a subset of the inputs and returns a superset of the outputs of the real Y, guaranteeing that the real X and the real Y will work together. X and Y must be unit-tested separately to avoid the combinatorial explosion of tests that would be required to completely cover the code if all tests ran against only real implementations, and unit tests of Y must use a mock X to achieve all of the outputs of X that are necessary to test Y and, often, for acceptable test performance.

But in return for these benefits we pay the price of duplication. Tests of X fully specify its behavior, but tests of Y also contain at least a partial specification of X’s behavior. Where there is duplication, copies can diverge, leading to errors. These are integration errors, and only integration tests can catch them.

For an integration test suite to be complete it must cover every method call from one separately unit-tested module to another, not 100% line coverage but still a lot of testing. If the application is developed driven by integration tests, that level of coverage is automatically achieved. But integration tests are generally harder to write and set up and much slower to run than unit tests, so many projects test-drive with unit tests instead of integration tests at least some of the time and compromise on something less than full integration coverage.

Back to the effect of language: A program in any language can be fully unit-tested. What about integration tests? If you commit to full integration coverage, you’ll catch all integration errors, again regardless of language. If you don’t have complete integration coverage, language and framework matter, because different languages and frameworks allow different types of integration errors. How much they matter is a complex qualitative question that can only be answered by experience. Not having any statistics at hand, I’ll just say that I suspect that the answer to the question posed in the title is yes, static typing does reduce the chance of integration errors, but Ruby and Rails’ test-first culture more than makes up the difference, and its practitioners are happy with the tradeoff.