UP | HOME
../../

Software Testing

Table of Contents

1 A Famous Quote

"Testing shows the presence, not the absence, of bugs" – Edsger W. Dijkstra, 1969, Turning Award Winner 1972

2 Overview/Goals

2.1 Overview

  1. You may have understood testing on a very intuitive basis
  2. Make this understanding more rigorous
  3. Software testing v testing in engineering

2.2 Assumptions

  1. Assumption: Did not practice Correct by Design.
  2. Assumption: Life cycle: We are "done" with development. We shall test and debug. Fix errors, and loop over this step.
  3. Assumption: We know what the results of a test ought to be. But, how do we know? Not as simple as knowing sqrt(x)!

2.3 Goals of Software Test Design

  1. "Hope" to fail.
  2. Not randomly generated tests
  3. Known examples of hangs and crashes
  4. 100% coverage of source code
  5. Edge cases
  6. Pass "old" tests (Regression Tests)
  7. Minimize test costs (How "much" testing?)

2.4 Goal of Debugging

  1. Identify the location where errant behavior begins
  2. Definition of a "Errant Behavior"
  3. Can we recognize errant behavior? How soon?
  4. Repeatability of Tests (i.e., test results)
    1. In the presence of non-determinacy (due to concurrency and distributed-ness)

2.5 Test Suites

  1. "Smoke" Test description on Wikipedia
  2. Conformance/ Acceptance Tests [part of Requirements and Specs]
  3. Regression description on Wikipedia
  4. Unit tests
  5. Integration tests
  6. Fuzzing https://en.wikipedia.org/wiki/Fuzzing

3 Definition of a "Test"

  1. test io pair: input and its expected output
    1. "test vector" refers to the inputs given at one time
  2. input includes: actual arguments, state of files, databases, …
    1. everything the program "reads"
  3. output: as above
    1. everything the program "writes"
  4. test fails if observed output != expected
  5. hangs v crashes; both => fail

4 Test Generation

  1. Generate the io pairs, with high likelihood of test failures
  2. Assume for now that input is text, and the output is text
  3. when it is a test of an "internal" component, the io may not be a "text/text" pair
    1. Unit testing solves this problem

4.1 Research on: Formal Specs => Tests

  1. Offutt, Jeff, and Aynur Abdurazik. "Generating tests from UML specifications." In UML99 – The Unified Modeling Language, pp. 416-429. Springer Berlin Heidelberg, 1999. Reference
  2. Pradel, Michael, and Thomas R. Gross. "Leveraging test generation and specification mining for automated bug detection without false positives." In 34th International Conference on Software Engineering (ICSE), pp. 288-298. IEEE, 2012. Reference

5 Test Coverage

  1. Can we construct tests that exercise every line of code?
  2. Statement coverage: execute each statement (at least) once
  3. Branch coverage: Execute each branch after a condition statement
  4. Path coverage: Execute all paths through the code
  5. Tools to track coverage
    1. C/C++: gprof, gcov/lcov, googletest, …
    2. Java: eCobertura, CodeCover, EclEmma, IdeaJ
  6. Does 100% coverage and no failures, guarantee that software is fault-free?

6 Testing Techniques

  1. Black box testing (aka functional testing): Uses no info re internals
  2. White box testing (aka structural testing): Uses all info re internals
  3. Worst/Special Case Testing
    1. Errors seem to cluster at boundaries
    2. What if values fall outside their ranges
    3. "Torture" (Stress) Tests
    4. Donald E. Knuth, "A torture test for TEX", http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tripman.pdf, 163pp, Version 3, January 1990.

7 Equivalence Class Testing

  1. Goal: Reduction of human/computer testing effort
  2. Divide the test vectors into subsets so that test coverage is the same
  3. The program is likely constructed so that it either succeeds or fails for each of the values in that class
  4. To test for robustness: for each equivalence class of valid inputs, devise equivalence classes of invalid inputs
  5. "cover" a class: Construct tests choosing one value from the class
  6. The goal is to cover all equivalence classes

8 Unit testing, Integration Testing

  1. OOP and programming language (PL) neutral term "unit" cf. class, method
  2. Unit test: test one method of one isolated class with one thread
  3. Tools for unit testing: JUnit and TestNG (both for any JVM based PL)
  4. Integration test: Test (all) the units together. It occurs after unit testing and before validation testing.
  5. see separate lectures
    1. ./testing-units.html
    2. ./testing-integrated.html

9 Testing of Interactive Programs

  1. Input generated in "real" time
  2. Output observed in "real" time
  3. Correctness judgement?
  4. Book: Don Libes, "Exploring Expect: A Tcl-based Toolkit for Automating Interactive Programs", O'Reilly Media, Inc., 2010, 606 pages. Look it up in WSU Library Safari Books.
  5. The "expect" is a standard package in Debian/Ubuntu systems

10 Test Driven Development (TDD)

  1. Testing impacts other phases of life cycle.
  2. We assumed above that implementation is "done". But, testing should be done during development.
  3. TDD is done as follows. Repeat until code is "complete":
    1. Add a test.
    2. Run all tests and see the new one fail.
    3. Make a (little) change.
    4. Run all tests and see them succeed.
    5. Refactor to remove duplication.
  4. ../Maintenance/KentBeck_TDD_byexample.pdf book extract 137pp. Ken Beck made popular the TDD, as a phrase.
    1. Full book: Kent Beck, Addison-Wesley, Boston, MA, 2003. 216 pp. ISBN 0321146530. Reference.
    2. Book review by Charles Ashbacher in Journal of Object Technology, vol. 2, no. 2, March-April 2003, pp. 203-204.

11 Project Specific Lecture

  1. Summer 2018 Project DITAA Testing
  2. https://www.jetbrains.com/help/idea/performing-tests.html
  3. https://www.jetbrains.com/help/idea/viewing-code-coverage-results.html
  4. IntelliJ videos, several.

12 References

  1. Glenford J. Myers, Corey Sandler, and Tom Badgett, The Art of Software Testing, 3rd Edition, John Wiley & Sons, Inc. 2011. Reference.
  2. http://en.wikipedia.org/wiki/Mutation_testing Required Reading.
  3. Pfleeger and Atlee, Software Engineering, 4e, slides are at http://wps.prenhall.com/esm_pfleeger_softengtp_4/. Following slides are Required Reading.
    1. ./Pfleeger-Atlee-08.ppt Chapter 8 - Testing the Programs
    2. ./Pfleeger-Atlee-09.ppt Chapter 9 - Testing the System
    3. ./Pfleeger-Atlee-10.ppt Chapter 10 - Delivering the System
  4. Barr, Earl T., et al. "The [Test] Oracle Problem in Software Testing: A Survey." IEEE transactions on software engineering 41.5 (2015): 507-525pp. "Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviur from potentially incorrect behavior is called the "test oracle problem". http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6963470. Required Reading.
  5. http://courses.cs.washington.edu/courses/cse331/16wi/L08/L08-Testing.pdf 40+ slides. Recommended Reading.
  6. Test-Driven Development – Extensive Tutorial. Open Source ebook https://leanpub.com/tdd-ebook Recommended Reading.
  7. ./TDD/tdd-calpoly-slides.pdf slides Recommended Reading.
  8. ./TDD/tdd-bneilsen-2006.pdf 30+ slides, cs.aau.dk; by Brian Nielsen + Arne Skou Recommended Reading.

13 End

Copyright © 2018 pmateti@wright.edu • www.wright.edu/~pmateti 2018-07-03