Comments for SoftSlate Blog http://www.softslate.com/blog From David Tobey, lead developer of the Java shopping cart SoftSlate Commerce Tue, 06 Sep 2011 17:32:36 +0000 hourly 1 http://wordpress.org/?v=3.0.1 Comment on Testing Has a Point of Diminishing Returns by BlueCollarCritic http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-76 BlueCollarCritic Tue, 06 Sep 2011 17:32:36 +0000 http://www.softslate.com/blog/?p=107#comment-76 Having done quite a bit of testing myself when I worked in support/custom development for a West coast software company (Not Microsoft) I can tell you that your metrics about diminishing returns match up with what I recall during my testing. While we all would like to have perfect code there is no such thing and so you have to go with good enough. That said, there is one thing that the testing department (at least where I worked) could and should observe and did not (for whatever reason) and that is : Never assume the way you (as a tester or developer or even someone in support) use the product is how a customer does or even how most of your customers do. The biggest mistake I observed was the mindset (shared by t9o many at that company ) of “that’s how everyone does it” and that resulted in more bugs getting into production code then should have. Having done quite a bit of testing myself when I worked in support/custom development for a West coast software company (Not Microsoft) I can tell you that your metrics about diminishing returns match up with what I recall during my testing. While we all would like to have perfect code there is no such thing and so you have to go with good enough. That said, there is one thing that the testing department (at least where I worked) could and should observe and did not (for whatever reason) and that is :

Never assume the way you (as a tester or developer or even someone in support) use the product is how a customer does or even how most of your customers do.

The biggest mistake I observed was the mindset (shared by t9o many at that company ) of “that’s how everyone does it” and that resulted in more bugs getting into production code then should have.

]]> Comment on Automated Functional Testing with JMeter by Testing Has a Point of Diminishing Returns | SoftSlate Blog http://www.softslate.com/blog/2011/08/functional-and-regression-testing-with-jmeter.html#comment-54 Testing Has a Point of Diminishing Returns | SoftSlate Blog Wed, 31 Aug 2011 15:25:26 +0000 http://www.softslate.com/blog/?p=44#comment-54 [...] Company Home » Company » Blog ← Automated Functional Testing with JMeter [...] [...] Company Home » Company » Blog ← Automated Functional Testing with JMeter [...]

]]> Comment on Testing Has a Point of Diminishing Returns by tom http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-53 tom Wed, 31 Aug 2011 13:55:36 +0000 http://www.softslate.com/blog/?p=107#comment-53 In my experience, it seems that most programmers test to make something work, not to break it. I have never experienced the diminishing return, but the other side where bare bones testing is done. It seems directly proportional to the programmers ego, that being, my code is infallible since I wrote it, so there is no way it could break. Or, testing is tedious, so pass it on. In my experience, it seems that most programmers test to make something work, not to break it. I have never experienced the diminishing return, but the other side where bare bones testing is done. It seems directly proportional to the programmers ego, that being, my code is infallible since I wrote it, so there is no way it could break. Or, testing is tedious, so pass it on.

]]> Comment on Testing Has a Point of Diminishing Returns by Chris Hemedinger http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-52 Chris Hemedinger Wed, 31 Aug 2011 13:21:54 +0000 http://www.softslate.com/blog/?p=107#comment-52 The concept of "good enough" quality for software has been around for a while, and is a formal description of your observations here. See http://www.satisfice.com/articles/gooden2.pdf for an example. The concept of “good enough” quality for software has been around for a while, and is a formal description of your observations here. See http://www.satisfice.com/articles/gooden2.pdf for an example.

]]> Comment on Testing Has a Point of Diminishing Returns by Frank http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-51 Frank Wed, 31 Aug 2011 13:05:49 +0000 http://www.softslate.com/blog/?p=107#comment-51 I think it's important to separate the time/resource cost spent uncovering bugs from the cost/tradeoff spent fixing and retesting them. Defect tracking systems are glutted with years-old bugs that will never be fixed: some are related to Browsers, OS, and hardware platforms no longer supported. Others involve features slated to be obsoleted and/or circumvented by new features. Bug statuses and priorities are often misleading or meaningless from a cost perspective: To the tester who specializes in obscure administrative functions a bug may scream showstopper, yet the field and support engineers know for a fact that no customer has come within miles of that particular function. How many "cosmetic" bugs have been pushed to the top of the queue because they're in highly visible locations that customers (and demo-ing salepeople) are sure to notice. This is why managers who reduce everything to bug counts aren't doing anyone any favors. I think it’s important to separate the time/resource cost spent uncovering bugs from the cost/tradeoff spent fixing and retesting them.

Defect tracking systems are glutted with years-old bugs that will never be fixed: some are related to Browsers, OS, and hardware platforms no longer supported. Others involve features slated to be obsoleted and/or circumvented by new features.

Bug statuses and priorities are often misleading or meaningless from a cost perspective: To the tester who specializes in obscure administrative functions a bug may scream showstopper, yet the field and support engineers know for a fact that no customer has come within miles of that particular function. How many “cosmetic” bugs have been pushed to the top of the queue because they’re in highly visible locations that customers (and demo-ing salepeople) are sure to notice.

This is why managers who reduce everything to bug counts aren’t doing anyone any favors.

]]> Comment on Testing Has a Point of Diminishing Returns by John Sposato http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-50 John Sposato Wed, 31 Aug 2011 13:01:57 +0000 http://www.softslate.com/blog/?p=107#comment-50 If the return on testing is finding bugs, then yes there are diminishing returns. If the return on testing is confidence in knowing there are very few bugs, then I don't think that curve holds. Obviously, we can't test every line of code all the time. However, we use our clients to test frequent beta releases to get feedback we'd never get from ourselves. The non-technical domain expert will always find things you can't find in-house. Testing is always going on, just because the guys/gals writing the software aren't doing it doesn't mean it's not happening. I'm testing Firefox right now as I write this comment. :) If the return on testing is finding bugs, then yes there are diminishing returns. If the return on testing is confidence in knowing there are very few bugs, then I don’t think that curve holds.

Obviously, we can’t test every line of code all the time. However, we use our clients to test frequent beta releases to get feedback we’d never get from ourselves. The non-technical domain expert will always find things you can’t find in-house.

Testing is always going on, just because the guys/gals writing the software aren’t doing it doesn’t mean it’s not happening.

I’m testing Firefox right now as I write this comment. :)

]]> Comment on Testing Has a Point of Diminishing Returns by Jim Rush http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-49 Jim Rush Wed, 31 Aug 2011 12:37:33 +0000 http://www.softslate.com/blog/?p=107#comment-49 As a general statement, I agree that as you do more testing, you get less value out of the results. A more important observation, in my opinion, is that all tests are not created equally. Instead of focusing on the number of tests, focus on ROI based thinking. Also, you haven't even discussed manual versus automated tests. Regression testing versus initial testing (many edge cases are worth testing once, but if are in a static area of code, may not justify testing again). X-ability testing. All of these things are far, far more important than finding that point of diminishing value in tests because your metric, number of tests, isn't the best way to think about the challenge of assuring quality. It's just another metric to help you balance your efforts and judge your effectiveness. As a general statement, I agree that as you do more testing, you get less value out of the results. A more important observation, in my opinion, is that all tests are not created equally.

Instead of focusing on the number of tests, focus on ROI based thinking. Also, you haven’t even discussed manual versus automated tests. Regression testing versus initial testing (many edge cases are worth testing once, but if are in a static area of code, may not justify testing again). X-ability testing.

All of these things are far, far more important than finding that point of diminishing value in tests because your metric, number of tests, isn’t the best way to think about the challenge of assuring quality. It’s just another metric to help you balance your efforts and judge your effectiveness.

]]> Comment on Testing Has a Point of Diminishing Returns by Wooton Hague http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-48 Wooton Hague Wed, 31 Aug 2011 05:37:41 +0000 http://www.softslate.com/blog/?p=107#comment-48 Byte shift testing is the best testing methodology. It can be done during the entire development frame of the project and when deployed can be used to administer alterations and changes to the deployment. Byte shift testing is the best testing methodology. It can be done during the entire development frame of the project and when deployed can be used to administer alterations and changes to the deployment.

]]> Comment on Dave’s Lazy Testing Technique by Testing Has a Point of Diminishing Returns | SoftSlate Blog http://www.softslate.com/blog/2011/07/great-testing-technique.html#comment-43 Testing Has a Point of Diminishing Returns | SoftSlate Blog Tue, 30 Aug 2011 21:31:45 +0000 http://www.softslate.com/blog/?p=11#comment-43 [...] it. For example, a couple of the most cost-effective testing techniques we do regularly include parallel ops and automated functional testing. Those are good ways to catch more defects without going beyond the [...] [...] it. For example, a couple of the most cost-effective testing techniques we do regularly include parallel ops and automated functional testing. Those are good ways to catch more defects without going beyond the [...]

]]> Comment on Testing Has a Point of Diminishing Returns by Luis Gutierrez http://www.softslate.com/blog/2011/08/testing-has-a-point-of-diminishing-returns.html#comment-41 Luis Gutierrez Tue, 30 Aug 2011 16:07:26 +0000 http://www.softslate.com/blog/?p=107#comment-41 In my experience in the hardware world, I would say that you are roughly correct, testing does have diminishing results over time. However, they are techniques that can greatly increase how many bugs you find, and make the curve look more 1:1. In particular, pseudo-randmon testing is very interesting. It allows you to hit hard-to-think corner cases easily. The problem with it, is that to do it properly you not only need a custom-built language to randomize all the inputs, but you also need a reference model (which are easy to do for simple tests, but very hard for the whole system), and functional coverage. Without coverage, your random testing is blind, and there is now real way of knowing what is it exactly that you are testing. Code coverage helps here, but to truly understand what your random tests are doing you need a more elaborate way of measuring what is happening in your application. have I executed this function with all the relevant values for the inputs? have I executed this function AFTER a calling this? The hardware world solved this problems with the creation of specific languages generating random stimulus and for capturing your functional "coverage points" and "coverage groups", and by developing a series of ever-refined methodologies to achieve this. Take a look at specman (e language), System Verilog and SystemC verification library. In my experience in the hardware world, I would say that you are roughly correct,
testing does have diminishing results over time.
However, they are techniques that can greatly increase how many bugs you find, and make the curve look more 1:1.

In particular, pseudo-randmon testing is very interesting. It allows you to hit hard-to-think corner cases easily. The problem with it, is that to do it properly you not only need a custom-built language to randomize all the inputs, but you also need a reference model (which are easy to do for simple tests, but very hard for the whole system), and functional coverage.

Without coverage, your random testing is blind, and there is now real way of knowing what is it exactly that you are testing. Code coverage helps here, but to truly understand what your random tests are doing you need a more elaborate way of measuring what is happening in your application. have I executed this function with all the relevant values for the inputs? have I executed this function AFTER a calling this?

The hardware world solved this problems with the creation of specific languages generating random stimulus and for capturing your functional “coverage points” and “coverage groups”, and by developing a series of ever-refined methodologies to achieve this.
Take a look at specman (e language), System Verilog and SystemC verification library.

]]>