Authors

Latest text of pad PkSyMpSzag
Saved April 10, 2013
GR Testers - 
 
Testing Death Match - Manual vs Automated Testing
 
Matt Heusser describes a common scenario where the "test automation consultant" gets a call that says "we need someone to come in and fix this automation bed except the guy that developed it everything was cool and he's been gone for six months and nothing works.  Its a GUI and we can't get the automation stuff to work.  Can you fix it?"
 
Often times, the answer is "No." Ppl don;t want to hear that.  Whaddayado?
 
So, going around the table - who has a stake in the game?  Well, NO ONE! 
 
Cool
 
David hoppe chimes in - 
 
This is a GPS device.  No buttons.  No Nuthin. Now what?
 
So, there is a GPS receiver and a transmitter that sends to a display unit.  There is a jiggler to tell when it is moving.  How do you test it?  (Pete comment:  Dude, really?)
 
Rob's comment was to build a EM silent room and send defined signals to the device and see what happens.  Sigh
 
Matt raises the question of "Can we use this /stuff/ for something other than this explicit purpose?"
 
Be careful to not get into the trap of Re-testing vs Regression testing (Pete, insert your definition of each)
 
When to automate?  Regression testing vs. "retesting".  The two can both be automated, but they ask different questions and should be automated differently.
 
Think about pushing automation farther upstream in the process; include automation in the unit tests to make coverage easier before turning over to the test team.
 
When the work you did before is becoming irrelevant, you are becoming a machine & you should be automated.  Or, when you can program your automation to be more "exploratory", thereby becoming more like manual testing... See Doug Hoffmn's Machine assisted exploratory testing work.
 
So, you can "explore" but how do you know if it is "good" or "bad"?  That requires an 'Oracle' - a way to determine if the answer is "right" - Pete's suggestion is that sometimes self-identifying data may be of help.
 
Damian - One shop (NPI) they had terrabytes of data, but really, it was almost impossible to make use of it.  How can u run a simple validation that stuff is not broken?  They ran reports nightly that exercised the paths in question.  Something similar to Continuous Integration Testing - run the old version and the new version 
 
Matt story - problems found sometimes are the result of building the automation suite instead of the execution of it.  Does that count?
 
When the automation finds a "bug" - is it the SUT or the automation suite that has the bug?  Hmmmm
 
Matt - I don't believe in 'best practices' but I have 'preferences' - if the dev writes "Assert True" = "True" u may have a problem////
David - There is nothing wrong with Assert True = True ... as long as you actually deal with it correctly later...
 
So... after a long digression to Agile ...
 
What about manual vs automated testing?
 
if you are working on an app that is going to the apple store - probably testig once & forget it.
If you have an app that needs to be exercised carefully... there are continuous changes over time... does it make sense to build some repeatable tests?  Probably.  Can some of these be automated?  Maybe
 
Damian - I intereviewed at a place that said "we are 100% automated" - Matt - They are lying
Pete-  maybe to themselves as well as everyone else...
 
The context of the question is the key to the discussion.  What is it that you want to learn from the application? 
 
In some cases, large scale, big data tests can be of value - in other cases, the interesting permutations get really complex to automate.
 
It is the customer interaction that makes it extremely challenging 
 
How many times will developers say something like "No one would ever do that!"  And then it happens?  Too bloody often.
 
Lesson fomr this conversation - Which one is better?  It depends.  In some cases, it makes sense to automate work, in others, not so much
 
TFS 2010 - has an interesting possibility to record an "exploratory" session - then play it back at will.
Snagit - 
Others....
 
ok - desert... :)