Software Testing Fundamentals

Software testing is a critical process for evaluating software quality by revealing failures, exposing defects, and ensuring requirements are met, ultimately providing evidence for product readiness and user satisfaction.

Core Principles

Quality has two faces: conformance to requirements (producer view) and fitness for use (customer view).
Testing is not the entire quality management process but provides evidence for product quality.
The cost of fixing defects increases exponentially the later they are found in the development lifecycle.
Precise language is crucial: distinguish between faults (static defects), errors (incorrect internal states), and failures (observable wrong behavior).
Exhaustive testing is impossible; prioritize testing where risk is highest.
Test early and continuously to reduce costs and improve quality.
Testing shows the presence of defects, not their absence.
Verification checks if the software is built right (against specifications), while validation checks if the right software is built (for user needs).

Action Steps

Exercise the software against its expected behavior (Evaluate).
Observe and record any wrong behavior before users experience it (Reveal failures).
Provide evidence that the product is ready to proceed or ship (Assure quality).

Key Terms

Fault: A static defect in the software, a mistake in the source code.
Error: An incorrect internal state produced when a fault is executed.
Failure: Observable wrong behavior relative to the requirements.
Verification: Checking if the software is being built right, ensuring outputs of each phase meet previous requirements.
Validation: Checking if the right software is being built, ensuring the finished software complies with intended use and user satisfaction.
Static Testing: Examining software artifacts (code, requirements, design) for defects without executing the program.
Dynamic Testing: Executing the software with real inputs and observing its behavior to find defects.
Cyclomatic Complexity: A metric measuring the number of linearly independent paths in a program, indicating its complexity.
Test Automation: Using software tools to automatically run and validate test cases, enhancing testing efficiency.
Non-Functional Requirement (NFR): A requirement that specifies a quality attribute of the software, such as performance, reliability, or usability.

Pro Tips

Static testing, performed early in the lifecycle (requirements, design, code), is cheaper for fixing defects than dynamic testing.
Automated testing provides fast, repeatable results and is ideal for regression testing, high-volume tests, and stable features.
Non-functional requirements (NFRs) must be unambiguous, doable, verifiable, in-scope, and genuinely non-functional, often including specific metrics like response times or resource usage.

Pitfalls to Avoid

Delayed testing leads to insufficient resources, missed design flaws, complexities in debugging, and project delays.
Relying solely on functional testing can result in a system that passes all tests but is unusable, unsafe, or unreliable.
Assuming "bad people" will be stopped before reaching the software in security testing can lead to vulnerabilities.

Real World Examples

Mars Climate Orbiter failure: A metric-vs-imperial unit mix-up caused the probe to go too low, resulting in a $327M loss.
Knight Capital trading engine failure: Untested trade-engine deployment led to a loss of the sum in 45 minutes, costing $440M.
THERAC-25 radiation overdoses: A race condition in the software delivered massive radiation overdoses, causing 3 deaths between 1985–87.

Timeline

1985–87: THERAC-25 incidents: Race condition caused massive radiation overdoses, resulting in 3 deaths.
1990–1999: Mars Climate Orbiter failure due to unit mix-up ($327M loss).
1996: Ariane 5 Flight 501 self-destructed due to conversion overflow (~$370M loss).
2002: Kent Beck published "Test-Driven Development: By Example", a foundational text.
2003: NE Blackout: Alarm-system software error left operators blind, affecting 55M people.
2012: Knight Capital trading engine failure ($440M loss).
2016: Tricentis Fail Watch reported 606 software failures worldwide, costing $1.7T and affecting half the world's population.
2018–19: Boeing 737 MAX crashes (346 lives lost) due to inadequately tested MCAS.