Skip to content
CS Capstone Handbook

Testing Strategy

Your team has code merging every week, some of it written by AI tools, and a project partner who will judge the result by whether it does what they need. This page covers what to test automatically and in what proportion, how to keep the suite trustworthy, and how to check with real users that you’re building the right thing. It doesn’t cover the CI workflow file, branch protection, or accessibility testing.

Testing answers two questions. Verification asks whether the code does what you specified: unit tests, integration tests, CI, exploratory testing. Validation asks whether what you specified solves the user’s problem: usability sessions, beta feedback, experiments, and demos to the people who’ll use the product.

Thinking About Your Testing Strategy

Section titled “Thinking About Your Testing Strategy”

Decide what confidence your project needs before choosing a framework, because a data pipeline and a public web app need different tests. Four questions settle most of it:

  • Who uses it, and when do they first touch it? Software handed to non-technical users after the team leaves needs usability testing. A pipeline whose only user is a data scientist needs reproducibility checks.
  • Where would a failure be most expensive? A payment flow or an authentication path earns automated tests; an about page doesn’t. A model pipeline earns validation against a known baseline.
  • When will real users see it? Put user sessions on the board in the sprint you intend to run them, because work that isn’t on the board has no owner and no time set aside for it.
  • What does your Definition of Done already promise? If it requires green tests in CI (continuous integration), you need the pipeline before you need more kinds of test. If it requires sign-off from the project partner, you need a demo and a feedback loop.

Write the answers into your working agreement or CONTRIBUTING.md in a few lines: which kinds of testing the team does, when automated tests are required, and when users are involved.

Testing as Part of Your Checks

Section titled “Testing as Part of Your Checks”

Your checks are everything that catches a bad change before it reaches a user:

  • tests that fail when behavior breaks, on the paths that matter;
  • CI on every pull request, required to pass before merge;
  • review gates on the hard-to-reverse decisions (schema, authentication, deployment, data migrations);
  • a staging or preview environment, so the first run of a change isn’t in production;
  • a rollback path you’ve actually tried.

Delegate to AI tools only as far as those checks would catch a mistake, and strengthen the checks before you delegate more, because a tool’s output can look right and still be wrong in ways nobody reads closely enough to see. The Generative AI guide makes that argument in full, and Audit Your Safety Net measures the gap on your own repository.

Verification: Does the Code Work?

Section titled “Verification: Does the Code Work?”

Verification covers automated tests, which run without anyone watching, and manual checks, which need a person using the system.

Automated Testing

Section titled “Automated Testing”

Automated tests are code that exercises your code. They run in seconds or minutes, give the same answer every time they run (when they’re written well), and catch a regression before a user does.

Unit Tests

Section titled “Unit Tests”

A unit test checks one function, method, or component in isolation. It’s fast, cheap to write, and when it fails it points at the line that broke.

def test_calculate_total_with_discount():
    result = calculate_total(price=100, discount=0.2)
    assert result == 80.0

Write unit tests for business logic, data transformations, and utility functions, where the inputs and outputs are well defined.

Integration Tests

Section titled “Integration Tests”

An integration test checks that several components work together: a function that queries a database, an API endpoint that authenticates a user and returns data, a pipeline that reads from one source and writes to another.

def test_create_user_stores_in_database(db_session):
    response = client.post("/users", json={"email": "test@example.com"})
    assert response.status_code == 201
    assert db_session.query(User).filter_by(email="test@example.com").one()

Integration tests are slower than unit tests, but they catch what a unit test can’t see: a configuration error, a schema mismatch, a wrong assumption about how a library or service behaves.

End-to-End Tests

Section titled “End-to-End Tests”

An end-to-end (E2E) test checks the system the way a user meets it: open the product, work through a workflow, and assert that the expected result appears. For a web application, that might mean logging in through a browser, creating a resource, and finding it on a dashboard. For a mobile app or a game, it means launching the build on a device, simulator, or emulator and driving its interface. For a data pipeline, it means feeding raw data in and checking the final output.

E2E tests are the slowest and the most likely to break for reasons unrelated to your change, but they catch failures no other test reaches: broken routing, a missing environment variable, a UI element that never renders, a workflow that breaks across a service boundary.

Contract Tests

Section titled “Contract Tests”

When a frontend consumes a backend API, a contract test checks that both sides agree on the shape of requests and responses, so a breaking change fails at the interface instead of at runtime. If different teammates own the two sides, even a light version (validating API responses against an OpenAPI schema) catches the mismatch before it reaches a demo.

The Testing Pyramid

Section titled “The Testing Pyramid”

The testing pyramid is a model for balancing these types: many fast, focused unit tests at the base, fewer integration tests in the middle, and a few slow, broad E2E tests at the top. Fowler credits Mike Cohn’s Succeeding with Agile (2009) with making it known.

    /   E2E   \        Few, slow, broad
   /───────────\
  / Integration \      Some, moderate speed
 /───────────────\
/   Unit Tests    \    Many, fast, focused

The pyramid has a competitor, and Fowler himself hedges it. He notes that it rests on the assumption that broad tests are slow, expensive, and brittle compared with focused ones, that this is usually true but has exceptions, and that if your high-level tests are fast, reliable, and cheap to change, lower-level tests aren’t needed. Kent C. Dodds’s testing trophy draws integration tests as its largest band, where each band’s size is, in general, the focus that kind of test deserves, on the principle that a test gives more confidence the more closely it resembles the way the software is used, and he points out that a unit test can’t ensure you’re calling a dependency correctly. Mike Wacker, on Google’s Testing Blog, defends the pyramid with the 70/20/10 split of unit, integration, and end-to-end tests that Google often suggests as a first guess, arguing that you should say no to more end-to-end tests because they give slow, flaky feedback and a failure is hard to trace to the code that caused it. Which shape fits depends on how expensive your integration tests actually are, so measure that before you commit to a shape.

What to Test

Section titled “What to Test”

Spend tests where a failure would hurt, not where tests are easiest to write:

  • Test behavior, not implementation. A test that asserts “the output is sorted” survives a refactor; one that asserts “the function calls sort() internally” breaks every time the code changes, even when the behavior hasn’t.
  • Cover the critical paths: authentication, payment processing, the core data pipeline, the primary user interaction. Edge cases in a settings page can wait.
  • Test the boundaries first (empty inputs, maximum values, invalid data, concurrent access), because a boundary is where a single wrong comparison flips the result, and it’s the case a happy-path test never reaches. If a function accepts a list, test it with an empty list, one item, and a very large list.
  • Don’t test framework code. If the framework guarantees a 404 for an unregistered route, test your code instead of theirs.
  • Don’t chase a coverage number. Use coverage to find code no test executes; Some Truths below explains why a target backfires.

Accessibility

Section titled “Accessibility”

Run automated accessibility checks in the same CI run as the rest of your suite, so a missing label or an unreachable button fails the build the way a broken function does.

Accessibility TestingWhich accessibility checks to run in CI, and what automated checks miss.

Manual and Exploratory Testing

Section titled “Manual and Exploratory Testing”

Automated tests check what the team told them to check. A person using the system notices what nobody thought to write a test for: a button that works but is invisible on a dark background, a form that submits successfully and confuses every user, a workflow that’s technically correct and takes twelve clicks instead of two.

Exploratory testing is unscripted manual testing: someone uses the system without a checklist, looking for anything that feels wrong. Run it after a major feature is complete and before a demo or release, when there’s a whole workflow to explore.

Smoke testing is a quick manual check that the most critical workflows still function after a deployment or a significant change. It answers one question, whether the system is broken outright, before anyone spends time on more detailed verification.

Validation: Does the Product Solve the Right Problem?

Section titled “Validation: Does the Product Solve the Right Problem?”

The methods below differ mostly in how many users they need: a usability round needs about five, a beta a small group, and an A/B test enough traffic for statistics. Pick the one your number of real users can support, and use sprint demos with the project partner whatever that number is.

Usability Testing

Section titled “Usability Testing”

Watch real users, or reasonable proxies, attempt key tasks in your system, and note where they hesitate, make mistakes, or say they’re confused. Keep each round small and run several. Nielsen and Landauer (1993) modeled problem discovery across 11 studies and, for their medium example, estimated the best benefit-to-cost ratio at four evaluations; Nielsen later argued for about five users per round, because each added user reveals less that’s new and the budget is better spent testing the redesign.

Test before the product is finished. Paper prototypes, clickable mockups, and half-built features are all testable, and a problem found in a mockup costs a redraw instead of a rewrite.

Plan each session: pick the tasks to test, recruit the participants, and write a script. “Try it and tell me what you think” gets opinions; a task such as “book a room for Thursday” shows you where the user gets stuck.

Testing with Assistive Technology Users

Section titled “Testing with Assistive Technology Users”

Only someone who relies on assistive technology daily can tell you whether the software is good rather than merely compliant. Accessibility Testing covers how to recruit for that session and what the closest substitute is when you can’t.

Beta Testing and Feedback

Section titled “Beta Testing and Feedback”

A beta gives a working version of the software to a small group of users and collects structured feedback on how it behaves in their real conditions. Decide what you want to learn before you distribute it, and ask specific questions tied to your acceptance criteria, because “let us know what you think” gets general impressions you can’t act on.

A/B Testing and Experiments

Section titled “A/B Testing and Experiments”

When success is measurable (conversion rates, task completion times, error rates), an A/B test shows two variations to different users and compares the outcome. It needs enough users for the difference to be statistically meaningful, which a product with a few dozen users can’t reach. The reasoning behind it still applies at any size: state a hypothesis, measure the outcome, and decide on the evidence.

Project Partner Demos as Validation

Section titled “Project Partner Demos as Validation”

Treat every sprint demo as a test of whether you built the right thing. When the project partner watches a feature and says “that’s not what I meant”, you’ve found a requirements bug, and a demo finds it within a sprint of the code being written instead of at the end of the project. The Presentations guide covers how to run a demo that gets that feedback.

Testing Results, Not Just Code

Section titled “Testing Results, Not Just Code”

When the work produces a result, the question a test answers is whether the result is reproducible and trustworthy. The Shipping guide covers what reproducible means and why you reproduce the baseline before attempting anything new. Two checks are testing work:

  • Validate the input data. Check that it meets the expected formats, ranges, and distributions before processing, because a pipeline that silently drops malformed records or produces NaN without warning produces results nobody can trust.
  • Sanity-check the outputs. After a model trains or an analysis runs, verify that the outputs fall in the expected range. Treat a model that suddenly reports 99.9% accuracy as a data-leakage bug until you’ve ruled it out, because test data leaking into training makes a model look far better than it is.

Research output that others will use (a tool, a dataset, a method) needs validation as well: put it in front of the intended audience and watch whether it works for them.

When to Test

Section titled “When to Test”

Testing runs through the whole workflow:

  • As you build. Tests exist before the code merges, whether the team writes them first or alongside, because a change merged without them can break later without anyone noticing.
  • On every pull request. Tests run in CI and a red build blocks the merge, which makes the suite a gate instead of a suggestion. The workflow file is in the DevOps guide; the branch protection that makes red mean blocked is in Git and GitHub.
  • Before every demo and release. Run the full suite plus a manual smoke test. If a workflow matters enough to demo, give it an end-to-end test.
  • After every bug that reaches a user. Write a test that reproduces the failure before you fix it, so the test proves the fix and keeps the bug from coming back (Fowler advises a unit test even when an end-to-end test found the bug). Over the project, that builds a suite shaped by real failures rather than imagined ones.

Organizing and Maintaining Your Test Suite

Section titled “Organizing and Maintaining Your Test Suite”

Organize tests so any teammate can find, run, and change them:

  • Keep tests close to the code they test: a tests/ directory mirroring the source tree, or colocated files (login.test.ts next to login.ts). Either works; pick one and use it everywhere.
  • Name tests descriptively. test_login_with_invalid_password_returns_401 tells a reviewer what’s verified without reading the body.
  • Build test data with fixtures, factories, or builders rather than hardcoding it in each file, so a schema change means one edit instead of forty.

Write the conventions (framework, file layout, how to run the suite) into CONTRIBUTING.md, so every teammate and every AI coding tool follows the same pattern. The Documentation guide covers what else belongs there.

Dealing with Flaky Tests

Section titled “Dealing with Flaky Tests”

A flaky test passes and fails on the same code. Luo et al. (2014) classified the causes of 161 flaky-test fixes from open-source Apache projects: 45% were async waits, 20% concurrency, and 12% test order dependency, 77% together. Each of those has a fix, below, along with the one for tests that depend on the network, the clock, or random numbers:

  • Async wait: the test uses a result before an asynchronous call has produced it, sometimes hidden behind a sleep(). Wait explicitly for the condition instead.
  • Concurrency: threads interleave differently on different runs. Add the missing synchronization or make the code deterministic.
  • Test order dependency: a test relies on state a previous test left behind. Have each test set up and clean up its own state.
  • Network, time, and randomness: a real network, clock, or random generator in the loop. Use a test double or a local instance, freeze the time, and seed the generator.

Luo et al. (2014) also found 78% of flaky tests were flaky from the day they were written, so run a new test many times in a row before you merge it.

Testing Tools

Section titled “Testing Tools”

Pick the framework your stack’s documentation uses, because that’s the one its examples and tutorials assume. Defaults for several stacks follow; for any other, that first rule still applies:

  • Python: pytest.
  • JavaScript and TypeScript: Vitest or Jest, with Testing Library for UI components.
  • Java: JUnit; Go: the built-in testing package; C#: NUnit; C++: GoogleTest.
  • Browser end-to-end, for a web app: Playwright.
  • Unity games: the Unity Test Framework, built on NUnit, which runs Edit Mode tests in the editor and Play Mode tests with the game running. Unreal games: the Automation Test Framework, whose feature and smoke tests run in the engine itself.
  • Mobile: XCTest for iOS unit and UI tests, Espresso for Android UI tests, and Flutter’s integration_test package, which drives the app on a device or emulator.
  • Embedded: unit-test the logic on your own machine with Unity for C (no relation to the game engine) or GoogleTest for C++, then run tests on the board itself, which catches the timing, memory, and driver faults a laptop can’t reproduce. On the Zephyr RTOS, Twister runs the same test suite in an emulator or, with a board connected, flashes and runs it on the device. Hardware-in-the-loop testing goes one step further: the real controller’s signals are wired to a test system that simulates the environment it controls, so you can test a motor controller without the motor, including the faults you couldn’t safely cause on real equipment.

Choose tools the team will run every day. A plain pytest suite that runs on every pull request catches more than an elaborate setup that only one person knows how to start.

Best Practices

Section titled “Best Practices”

These keep a suite trustworthy once it’s large enough that nobody remembers why each test exists:

  • Hold test code to the same standard as production code, because a test nobody can read is a test nobody fixes when it fails.
  • Fix a red suite the day it goes red; a suite that stays red for days teaches the team that red means nothing.
  • Delete tests for removed features and dead code paths, which slow the suite and test nothing anyone depends on.
  • Keep the suite fast enough to run before every push. When it slows down, find the few slow integration or E2E tests responsible and speed them up or move them to a separate job.

Some Truths About Testing

Section titled “Some Truths About Testing”

Coverage is the metric most likely to mislead you, because it records which lines executed, not whether anything was checked; a suite that calls every function and asserts nothing reports excellent coverage. Inozemtseva and Holmes (2014) generated 31,000 test suites for five large Java systems and found only a low to moderate correlation between coverage and fault detection once suite size was controlled, and they suggest coverage should not be a quality target.

Mutation testing measures what coverage can’t: whether the tests would notice a bug. Tools such as PIT and Stryker seed small faults into your code and count how many your tests catch, and Just et al. (2014) found that a suite’s mutant detection correlates with its detection of real faults, independently of coverage. It reruns the suite once per seeded fault, so on a large codebase run it on the critical paths rather than everywhere.

Flaky tests do damage beyond the test itself. On Google’s Testing Blog, Micco (2016) reports that about 1.5% of all test runs at Google are flaky, that almost 16% of their tests have some flakiness, and that about 84% of the pass-to-fail transitions their CI sees involve a flaky test; with that many false alarms, people commonly ignore legitimate failures in those tests. Deleting the test is tempting, but Luo et al. (2014) found that 24% of flaky-test fixes changed the code under test, and 94% of those fixed a real bug, so they argue against simply removing flaky tests. Quarantine a flaky test the day you find it, so it stops training the team to ignore red, then read the failure before you decide between fixing and deleting.

Testing pressure improves design only up to a point. What it rewards is separating decisions from side effects: a function that only computes is trivial to test, and one that computes and writes to a database is not, and the separated version is also easier to change. Past that point, David Heinemeier Hansson argues, the pressure does test-induced design damage: indirection and extra layers added only so a unit test can run fast and in isolation, at the cost of clarity. When a change makes code easier to test and harder to read, prefer the readable version and test it at a higher level, because the reader pays for every extra layer and the test doesn’t need it.

Verification is easier to invest in than validation, because a test suite gives a green or red answer on every push and a user session gives a page of notes once. That asymmetry is how a project ends up with a green suite on a feature nobody wants: nothing in CI checks the specification itself, so a wrong one stays green for as long as nobody puts it in front of a user.

Testing in Industry and Academia

Section titled “Testing in Industry and Academia”

Google’s testing culture turned on a failure. Bender (2020) recounts that in 2005 the Google Web Server’s releases were getting buggier, and at one point more than 80% of its production pushes contained user-affecting bugs that had to be rolled back; the tech lead then required tests with every new code change, run continuously, and within a year emergency pushes had dropped by half, despite a record number of changes.

In academia, testing research often measures a suite with mutation analysis, because the real faults in a program are usually unknowable in advance (Just et al., 2014), and that’s how the coverage study under Some Truths measured fault detection.

When research depends on code, reproducibility plays the role a test suite plays in industry, and the failure it guards against has been documented at scale: Kapoor and Narayanan (2023) surveyed the literature and found data leakage in 17 fields, affecting 294 papers and in some cases leading to wildly overoptimistic conclusions.

References

Section titled “References”

Additional Readings

Section titled “Additional Readings”

Sources and further reading

Activities that exercise this