Dave’s Lazy Testing Technique

I often find there’s an underlying assumption that when software breaks it’s because it wasn’t tested enough. Granted, there’s a lot of crappy, buggy software, but finding any one particular bug doesn’t signify that the app needed to be tested more. Testing is a huge expense and it has diminishing returns. To find 90% of the defects might take x amount of effort. To find 99% would take 10x. So you do have to draw the line and just push it out the door at some point. It’s actually as much a business decision about the tolerance you have for risk and the consequences of something going bad, as it is a technical decision.

What if there was a way to eliminate testing almost altogether, and yet know with virtual certainty there are no bugs? Best of all possible worlds right? In some circumstances, there is. I’ve been at this for 10 years, and I don’t know why it took me so long to find this technique. It doesn’t apply in every situation, but it does more often that you might think. I’ll explain by way of two real-world examples.

The New Foreign Key

We help develop a custom SoftSlate Commerce Java ecommerce website that records the customer’s selected shipping method along with each order. We wanted to move from storing the shipping method in a simple text field, to storing it with a foreign key pointing to a separate database table. Nice, useful refactoring. It occurred to me, why not deploy the project, but still use the text field for a little while afterwards? Populate the foreign key, but have the code still use the original field. We did this and after a period of time I was able to run a simple query to see if the new table was being populated and modified correctly. Sure enough, it revealed a bug, where the foreign key was not getting updated when it should have. So I fixed that and waited. A couple weeks later I ran the same query. Everything matches up. Now I have a tremendous amount of confidence that I can use the foreign key and it will be 100% accurate. I used live usage and data to test my code in a manner that was extremely safe. Deploying now become a matter of updating the code to use my new foreign key instead of the clunky text field.

The New Initialization Function

In a very similar way, we needed to optimize the way a particular object was being initialized. The object represented a rental schedule, but it doesn’t really matter what it was used for. Any bean essentially whose state has to be initialized would qualify. Again, our plan was to keep the old way of initializing it alone, and add the new way. Any client code using the object would still use the old way, but in parallel, we’d initialize the schedule the new way as well, right alongside it. Simply add a toString() method to the object and write both object’s string representations to the log. Let it run for a couple weeks, or however long you like until you’re satisfied, and then go back and check the logs. If the toString() output matches, you can be highly confident the new technique is working exactly as the old one was.

Possibly the commonality here is we were refactoring or optimizing existing code. In general terms, when you are replacing an existing function with a new, improved version that does the same thing, you should be able to use this technique. You do need to have a way to check the old technique against the new technique, either via logs or with a database query. Essentially you are using the live, running application to create your test cases and you are performing your assertions against these real-life test cases. Not only is it an extremely robust test, it doesn’t take any effort on your part. You let the application do your testing for you. Again, I don’t know why it didn’t occur to me years ago.

Anyone ever heard of this technique and if so, what’s it called? If not, I hereby dub it “Dave’s Lazy Testing Method”.