Dave’s Lazy Testing Technique

I often find there’s an underlying assumption that when software breaks it’s because it wasn’t tested enough. Granted, there’s a lot of crappy, buggy software, but finding any one particular bug doesn’t signify that the app needed to be tested more. Testing is a huge expense and it has diminishing returns. To find 90% of the defects might take x amount of effort. To find 99% would take 10x. So you do have to draw the line and just push it out the door at some point. It’s actually as much a business decision about the tolerance you have for risk and the consequences of something going bad, as it is a technical decision.

What if there was a way to eliminate testing almost altogether, and yet know with virtual certainty there are no bugs? Best of all possible worlds right? In some circumstances, there is. I’ve been at this for 10 years, and I don’t know why it took me so long to find this technique. It doesn’t apply in every situation, but it does more often that you might think. I’ll explain by way of two real-world examples.

The New Foreign Key

We help develop a custom SoftSlate Commerce Java ecommerce website that records the customer’s selected shipping method along with each order. We wanted to move from storing the shipping method in a simple text field, to storing it with a foreign key pointing to a separate database table. Nice, useful refactoring. It occurred to me, why not deploy the project, but still use the text field for a little while afterwards? Populate the foreign key, but have the code still use the original field. We did this and after a period of time I was able to run a simple query to see if the new table was being populated and modified correctly. Sure enough, it revealed a bug, where the foreign key was not getting updated when it should have. So I fixed that and waited. A couple weeks later I ran the same query. Everything matches up. Now I have a tremendous amount of confidence that I can use the foreign key and it will be 100% accurate. I used live usage and data to test my code in a manner that was extremely safe. Deploying now become a matter of updating the code to use my new foreign key instead of the clunky text field.

The New Initialization Function

In a very similar way, we needed to optimize the way a particular object was being initialized. The object represented a rental schedule, but it doesn’t really matter what it was used for. Any bean essentially whose state has to be initialized would qualify. Again, our plan was to keep the old way of initializing it alone, and add the new way. Any client code using the object would still use the old way, but in parallel, we’d initialize the schedule the new way as well, right alongside it. Simply add a toString() method to the object and write both object’s string representations to the log. Let it run for a couple weeks, or however long you like until you’re satisfied, and then go back and check the logs. If the toString() output matches, you can be highly confident the new technique is working exactly as the old one was.

Possibly the commonality here is we were refactoring or optimizing existing code. In general terms, when you are replacing an existing function with a new, improved version that does the same thing, you should be able to use this technique. You do need to have a way to check the old technique against the new technique, either via logs or with a database query. Essentially you are using the live, running application to create your test cases and you are performing your assertions against these real-life test cases. Not only is it an extremely robust test, it doesn’t take any effort on your part. You let the application do your testing for you. Again, I don’t know why it didn’t occur to me years ago.

Anyone ever heard of this technique and if so, what’s it called? If not, I hereby dub it “Dave’s Lazy Testing Method”.

About David Tobey

I'm a web developer and consultant based in lovely Schenectady, NY, where I run SoftSlate, LLC. In my twenties I worked in book publishing, where I met my wife. In my thirties I switched careers and became a computer programmer. I am still in the brainstorming phase for what to do in my forties, and I am quickly running out of time to make a decision. If you have any suggestions, please let me know. UPDATE: I have run out of time and received no (realistic) suggestions. I guess it's programming for another decade.
This entry was posted in Deep Thoughts, How To's, Programming, Web Ops. Bookmark the permalink.

6 Responses to Dave’s Lazy Testing Technique

  1. Ivank2139 says:

    Very interesting technique. I did some refactoring and created new classes alongside old ones and then wrote unit tests for both with performance timing to show the refactored code was better. I could not deploy it for use though so I never could utilize your method. It does look interesting though and my current project does include a huge amount of refactoring if we get the go ahead.

  2. tea42 says:

    I've been developing a similar concept which I analogize to Double Entry Bookkeeping where a "test" is simply an alternate implementation of the same API. Comparing the results of the two would identify potential bugs. I like your idea of putting both into production.

  3. Greg says:

    If you take a snapshot of your DB on Monday, and record all HTTP requests & responses for a couple of days, then by the time you're finished coding on Wednesday, you can do exactly the test you described by putting your new code on top of your saved DB image, replaying the traffic, and checking the result — no need to wait until after the code is done to get confirmation that it works.

  4. David Tobey says:

    @Greg – that's a good idea. what tools would you use to record all the app's http requests and then rerun them? I guess you wouldn't be able to do this if any of the application code depended on dates or time intervals. Ie, if code behaved differently if certain things in the db were timestamped in the past.

  5. David Tobey says:

    The comment got munged during the migration of this blog, but a person named John who seemed to know what he was talking about said this:

    We call it parallel ops … but your name is good too :)

    I knew there was an official name for it!

  6. Pingback: Testing Has a Point of Diminishing Returns | SoftSlate Blog

Leave a Reply