Does anyone have any experience of tools which can perform automated testing
in an mv environment?

If so, what tools would they recommend?

Thanks,
Steve H.

Steve Haddon wrote:
I've had a lot of success using AccuTerm. It's VBScript is
industry-standard and it's rock solid. I can install, configure,
boot, test, and uninstall an environment entirely from scripts.
Underlying Accuterm are COM components which wrap telnet, so the same
operations can be controlled from outside of AccuTerm, and I've done
this too. I used it like this from VB6, haven't tried with .NET yet,
but that shouldn't be a problem. Along the same lines, code for
simple telnet clients is available in almost all languages, so if you
want to get a little closer to the metal you can telnet in directly
and manage the data flow yourself, without AccuTerm.

By wrapping processes such as the above in your own higher-level
container, you can create a test environment which follows a script
and takes different actions depending on what comes back from the
server. The more dynamic it is the tougher it is, but I've done it,
so I know it's possible.

Outside of this, in the past I researched various automated testing
utilities and found most of them to be very expensive. If you find
something you like, please let us know.

HTH
T

Thanks Tony.

We use Dynamic Connect (terminal emulator) which is bundled with Universe,
so I would prefer not to have to introduce another emulator. I was also
hoping for something at the server (rather than client) end.

A quick Google had already taken me to "DejaGnu" - see here:

http://www.gnu.org/software/dejagnu/

but I haven't tried it. I was hoping someone else had already gone through
the evaluation process and would be able to give me pros & cons on the
various options. Let's see if anyone else has anything to offer!

Thanks again,
Steve H.


Tony Gravagno wrote:

I understand your position. Following my "tools are irrelevant"
mantra, I'll convey some experience in this area which may help you to
develop your tests, regardless of which tools you use. Note, no
terminal emulators are required for the following.

When I was QA Manager at Pick Systems we had a standard test suite of
a few thousand programs which tested various aspects of the
environment. All of these programs were controlled by a single
program. The original control program and many of the unit tests were
written by Dick himself. In short, here's the framework:
1) Select all programs that should pass for the current platform and
release.
2) Run the program.
3) No matter what the program does, it should return a result with the
word "pass" or "fail".
4) Record the result.

Tests were run a few times. Distinctions were made between consistent
failures, random failures, failure the first time but success
thereafter, tests that only passed or failed after other specific
tests, tests that only passed or failed after a clean system boot,
logoff, or other event, and a lot of other analysis was performed.

Tests were run in a specific order which was recorded along with the
results. They were run in reverse order. Then they were run in
random order many times, often repeatedly over several days to see if
any random execution of commands eventually induced errors. If errors
were reported, the tests were re-run using the exact same recorded
order to see if the ordering had anything to do with the tests.
Sometimes tests fail after being run some number of times. For
example, execution of specific statements would sometimes cause aborts
after 1024 or 32767 times.

Those tests and many more are still run today at Raining Data. Many
things can be learned from these procedures which follow industry
standards as determined by various professional QA organizations.

1) Unit testing is extremely important. Only test one thing. Test it
in as isolated and simple an environment as possible.
2) Test in many ways, repeatedly and check the results under various
conditions. Results must be consistent even if the environment
changes.
3) Do not let the testing environment itself influence the tests - the
observer should not affect the observed.
4) "Test to destroy" was my motto, and while we couldn't always manage
it, I tried to test systems under stressful conditions, like running
tests on multiple ports simultaneously, running on Very large files or
with Very large items or with Very large numbers or strings, etc.
That's not always fair to the system or developers but it sometimes
reveals interesting results.

This testing process must be continually managed:
1) New features require new test which must be incorporated into the
standard test set.
2) When a bug is fixed the test that was not expected to pass on a
prior release must be flagged so that it is expected to pass in the
new release.
3) Known bugs should have their tests flagged so that they are not run
in unpatched releases, or if they are run the testing people need to
be aware of known issues so that time isn't wasted trying to figure
out why some test is failing.
4) When bugs are reported a test should be written to prove there is a
bug. This should fail in the platform in which it was reported.
After a fix the test should pass when run by the developer/engineer.
This test should then be incorporated into the standard test set to
ensure the same problem doesn't come up in later releases.
5) After you run the tests that should not pass, run as many tests as
possible that you do not expect to pass. You can get valuable
information by investigating why things work when they shouldn't,
including finding undocumented fixes, inconsistent fixes, fixes that
were accidentally made as a byproduct of some other change, etc.
6) Don't assume the test set is perfect. A failure may be the result
of a defined change, so the test may need to be changed for different
releases. Sometimes a test will incorrectly pass or fail because the
test itself was not properly coded. Tests need to be continually
reviewed by different people to ensure they do what they were intended
to do, and that they do it as described above, effectively and
minimally.
7) Some failures may indicate a need for new documentation rather than
a code change.
8) Some tests cannot be automated, they must be done manually unless
special software is written or purchased. The tests that cannot be
performed should be understood to try to prevent known "possible"
issues from getting into the field.
9) Sometimes instinct is important and you need to spend time running
tests in different ways to see if you can trick the system into
failing in a controlled manner. This indicates that the people who
test the system are as important as the tests being run - QA people
aren't just low level developers or grunts who press buttons.

Over the years I've seen many bugs in D3, and others have been
reported in these forums. I can often tell from the nature of the
issue which tests were not being run or which procedures were not
being followed which should have caught the bugs. When I use software
from other companies and find bugs, I recognize the sort of testing
they are not doing. Sometimes I send a note to the vendor to
encourage them to enhance their testing processes and I tell them
exactly what sort of procedure would have caught the specific error.
Unfortunately many vendors take such commentary indignantly. Well,
they're the ones with code bugs. It's up to them if they want to
continue to have them or if they want to fix the fundamental process
which causes bugs to get into the field. Bugs have tangible costs in
terms of support and re-work, and intangible costs in terms of
customer satisfaction, lost sales, diminished market confidence, etc.

Most developers assume they know how to write tests or how to QA
software. This is a tragic error. I'm frequently told that software
is "in the QA cycle". This is usually just a professional way of
saying someone just ran their program and they don't see any runtime
errors. While most people feel they build quality into their
software, learning a programming language has nothing to do with
software quality. We aren't born knowing how to write quality
software or how to write testing software. Debugging your own code
has nothing to do with knowing how to establish QA processes. A title
like Senior Developer, IT Manager, CTO, or CIO does not imply
knowledge of QA principles. Writing application code is completely
different from writing QA tests.

Developers should generally not write the unit tests for their own
code because they will write the test to pass, not to test or
challenge the components.

QA tests should challenge code to fail, but only in scenarios which
emulate real world usage. Sometimes even invalid usage to determine
how well the system performs when human beings make simple mistakes.
Every test should be separate, like buffer overruns, tests with zero,
null, invalid but likely/probably parameters, etc.

Given the above it may become evident why RD and some other providers
insist on "someone" creating a reproducible case with a short bit of
code before anyone attempts a fix. Some bugs simply cannot be
reproduced on demand, but most can.

After I actually started reading about proper QA techniques I felt
pretty ignorant about this whole other side of our business. (Oh
yeah, and the material is boring as hell but it's required reading for
someone truly interested in the ongoing practice of the assurance of
quality.) After learning something about standard QA practices, and
understanding the processes in place at Pick Systems, I came to find
that the processes we had in place were absolutely industry standard,
and the only time the process failed is when people cut corners, or
when resources were cut, including time, people, or financing.
Conversely, I also had to learn that there was no such thing as "done"
and letting go of a release that has known issues is a painful
process, no matter how small they may be.

I would be remiss if I didn't offer services to help companies to
create an effective QA plan, write tests, perform "test to destroy"
testing, etc.. Errr, offer is on the table for anyone interested.

HTH
TG@ removethisNebula-RnD.com



Steve Haddon wrote:

Thanks Tony,

That was a huge amount of very useful info and just in case anyone missed
your plug at the bottom here it is again:

"I would be remiss if I didn't offer services to help companies to create an
effective QA plan, write tests, perform "test to destroy" testing, etc..
Errr, offer is on the table for anyone interested."

We're actually actively recruiting a tester right now (Note. We're in the
UK), but I know there will be challenges for them, if they want to bring QA
Tools into a "dumb terminal environment". Hence my original post.

Regards,
Steve H.