Testing Strategy

Read this when you decide what to test and how far to go; it gives you the layers of automated testing and the user validation that complements them.

Testing is how a team builds confidence that the software works, keeps working, and solves the right problem. It is not a phase that happens at the end. It is a continuous activity woven into every sprint, every pull request, and every conversation with a project partner. Without a deliberate approach to testing:

Bugs accumulate silently until they surface in a demo or a deployment.
Refactoring becomes terrifying because nobody knows what might break.
The team cannot tell whether a feature actually meets the acceptance criteria or just appears to.
The project partner receives software that technically runs but does not behave the way they expected.

Testing answers two distinct questions. The first is does the code work? This is the domain of automated tests and manual verification: unit tests, integration tests, CI pipelines, exploratory testing, and regression checks. The second is does the product solve the right problem? This is the domain of user testing and validation: usability studies, beta feedback, A/B experiments, and structured conversations with the people who will actually use what you build. A complete testing strategy addresses both. A team that writes excellent unit tests but never puts the product in front of a real user can ship something that works perfectly and is completely useless.

Thinking About Your Testing Strategy

Different projects need different confidence, so decide what yours needs before choosing a framework. Three questions settle most of it:

Who uses it, and when do they first touch it? Software handed to non-technical users after the year ends makes usability testing mandatory. A pipeline whose only user is a data scientist makes reproducibility the thing to test.
Where would a failure be most expensive? A payment flow or an authentication path earns rigorous automated coverage; an about page does not. A model pipeline earns validation against a known baseline.
Which sprints include real users? User testing is planned or it does not happen. Put the sessions on the board in the sprint you intend to run them.
What does your Definition of Done already promise? If it requires green tests in CI, you need the pipeline before you need more test types. If it requires partner sign-off, you need a demo and feedback loop on the board.

Write the answers into your working agreement or CONTRIBUTING.md, in a few lines: which kinds of testing the team does, when automated tests are required, and when users are involved.

Team Working AgreementWhere testing expectations and the Definition of Done live.

Testing as the Safety Net

Most of the code in a Capstone project is now written with AI tools, which is why the useful question is not how much you delegated but what your automation would catch when the delegation goes wrong. That automation is your safety net, and testing is most of it. The net has five parts, and each is checkable:

Tests that fail when behavior breaks, on the paths that matter. A suite that passes after you delete a feature is not a net.
CI on every pull request, green, and required before merge.
Review gates on the one-way doors: schema, authentication, deployment, data migrations, anything you cannot walk back.
A staging or preview environment, so the first run of a change is not production.
A rollback you have tried, not one you believe in.

You may delegate exactly as far as that net catches, and no further. If you want to hand more to the tools, add to the net first. Audit Your Safety Net measures the gap between the two on your own repository; the reasoning behind the rule is in the Generative AI guide. The rest of this guide is how to build the first part of the net well, and how to test the thing the net cannot: whether the product is worth building.

Verification: Does the Code Work?

Verification is about confirming that the system behaves as specified. This includes both automated tests (which run without human intervention) and manual checks (which require a person to interact with the system).

Automated Testing

Automated tests are code that exercises your code. They run fast, repeat reliably, and catch regressions before anyone sees them.

Unit Tests

Unit tests verify that individual functions, methods, or components behave correctly in isolation. They are fast, cheap to write, and provide precise feedback when something breaks.

def test_calculate_total_with_discount():
    result = calculate_total(price=100, discount=0.2)
    assert result == 80.0

Unit tests are especially valuable for business logic, data transformations, and utility functions where the inputs and outputs are well-defined.

Integration Tests

Integration tests verify that multiple components work together correctly: a function that queries a database, an API endpoint that authenticates a user and returns data, or a data pipeline that reads from one source and writes to another.

def test_create_user_stores_in_database(db_session):
    response = client.post("/users", json={"email": "test@example.com"})
    assert response.status_code == 201
    assert db_session.query(User).filter_by(email="test@example.com").one()

Integration tests are slower than unit tests but catch a class of bugs that unit tests cannot: configuration errors, schema mismatches, incorrect assumptions about how a library or service behaves.

End-to-End Tests

End-to-end (E2E) tests verify the system from the user’s perspective: open a browser, click through a workflow, and assert that the expected result appears. For a web application, this might mean logging in, creating a resource, and verifying it appears on a dashboard. For a data pipeline, it might mean feeding raw data in and checking the final output.

E2E tests are the slowest and most brittle, but they catch problems that no other test type can: broken routing, missing environment variables, UI elements that fail to render, or workflows that break across service boundaries.

Contract Tests

In systems with a clear API boundary (for example, a frontend consuming a backend API), contract tests verify that both sides agree on the shape of requests and responses. They catch breaking changes at the interface before they cause failures at runtime.

If your project has a frontend and backend developed by different team members, even lightweight contract testing (such as validating API responses against an OpenAPI schema) can prevent a common class of integration failures.

The Testing Pyramid

A useful mental model for balancing these test types is the testing pyramid: many fast, focused unit tests at the base; fewer integration tests in the middle; a small number of slow, broad E2E tests at the top.

    /   E2E   \        Few, slow, broad
   /───────────\
  / Integration \      Some, moderate speed
 /───────────────\
/   Unit Tests    \    Many, fast, focused

The pyramid is a guideline, not a rule. Some projects (especially UI-heavy ones) benefit from more integration and E2E tests. Research projects may have few unit tests but need strong reproducibility checks. The principle holds: invest more in tests that are fast and reliable, and use expensive tests selectively for what cheaper tests cannot cover.

What to Test

The most common testing mistake is testing the wrong things: writing tests for trivial code while leaving critical paths uncovered.

Test behavior, not implementation. A test that asserts “the function calls sort() internally” is fragile; one that asserts “the output is sorted” is resilient. Tests tied to implementation details break every time the code is refactored, even when the behavior has not changed.
Focus on critical paths. Every application has a handful of workflows that must work: authentication, payment processing, the core data pipeline, the primary user interaction. These deserve thorough test coverage. Edge cases in a settings page are less urgent.
Test the boundaries. Boundary conditions (empty inputs, maximum values, invalid data, concurrent access) are where most bugs live. If a function accepts a list, test it with an empty list, a single item, and a very large list.
Do not test framework code. If a framework guarantees that a route returns a 404 for an unregistered path, you do not need to test that. Test your code, not the tools you are using.
Do not chase coverage numbers. Code coverage measures which lines were executed during testing, not whether the tests are meaningful. A test suite with 90% coverage that only tests happy paths is worse than one with 60% coverage that tests critical paths and edge cases thoroughly. Use coverage as a signal to find blind spots, not as a target to optimize.

Accessibility

Accessibility is a correctness property and it belongs in the same CI run as the rest of your suite: lint rules, axe-core inside the tests you already have, and a page-level budget that fails the build.

Accessibility TestingThe three layers: automated gates, the keyboard and screen-reader pass, and a session with a daily assistive-technology user.

Manual and Exploratory Testing

Automated tests verify that the system does what the team told it to do. Manual testing verifies that the system does what a human expects it to do. Both are necessary.

Exploratory testing is unscripted manual testing where a person uses the system without a predefined checklist, looking for anything that feels wrong: confusing workflows, unexpected states, visual glitches, performance issues. It is especially valuable after a major feature is complete and before a demo or release.

Smoke testing is a quick manual check that the most critical workflows still function after a deployment or significant change. It answers one question: “Is the system fundamentally broken?” before investing time in more detailed verification.

Automated tests alone will never catch everything. A button that works perfectly but is invisible on a dark background, a form that submits successfully but confuses every user, a workflow that is technically correct but takes twelve clicks instead of two: these are problems that only a human will notice.

Validation: Does the Product Solve the Right Problem?

Validation is about confirming that the system is worth building in its current form. Code can pass every automated test and still fail the people it was built for. Validation puts the product in front of real users and asks: does this actually work for you?

Usability Testing

Watch real users (or reasonable proxies) attempt key tasks in your system. Note where they hesitate, make mistakes, or express confusion. Five users are enough to surface the most critical usability problems.

Usability testing does not require a finished product. Paper prototypes, clickable mockups, and partially implemented features are all testable. The earlier you test with users, the cheaper it is to fix what you learn.

Plan usability sessions deliberately. Decide which tasks to test, recruit participants, and prepare a script or task list. Unstructured “try it and tell me what you think” sessions produce less actionable feedback than specific task-based observations.

Testing with Assistive Technology Users

Only someone who relies on assistive technology daily can tell you whether the software is good rather than merely compliant. Accessibility Testing covers how to recruit for that session and what the honest substitute is when you cannot.

Beta Testing and Feedback

Giving a working version of the software to a small group of users and collecting structured feedback is one of the most effective ways to validate that the system works in real conditions. Define what you want to learn before distributing the beta. Open-ended “let us know what you think” produces less useful feedback than specific questions tied to acceptance criteria.

A/B Testing and Experiments

For projects where success is measurable (conversion rates, task completion times, error rates), A/B testing compares two variations to determine which performs better. This requires enough users and enough traffic to be statistically meaningful, which limits its applicability in many Capstone projects, but the thinking behind it (form a hypothesis, measure the outcome, decide based on evidence) applies universally.

Project Partner Demos as Validation

Sprint demos are not just presentations. They are validation opportunities. When the project partner watches a feature in action and says “that is not what I meant,” that is testing. Treat every demo as a chance to confirm (or correct) the team’s understanding of what the product should do.

User ActivitiesUX research methods and activities for learning from users.

PresentationsHow sprint demos serve as feedback loops with stakeholders.

Testing Research Projects

The primary question is not “does this code work?” but “are these results reproducible and trustworthy?” What reproducible means concretely, and why you reproduce the baseline before attempting novelty, is in the Shipping guide. Two checks are testing work rather than plumbing:

Data validation. Check that input data meets expected formats, ranges, and distributions before processing. A pipeline that silently drops malformed records or produces NaN without warning is a source of unreliable results.
Sanity checks on outputs. After a model trains or an analysis runs, verify the outputs fall in the expected range. A model that suddenly reports 99.9% accuracy is reporting a data-leakage bug, not a breakthrough.

Research output that others will use (a tool, a dataset, a methodology) needs validation too: put it in front of the intended audience and watch whether it works for them.

When to Test

Testing is woven into the workflow, not a phase at the end.

As you build. Tests exist before the code merges, whether the team writes them first or alongside. Code merged without them works today and nobody knows about tomorrow.
On every pull request. Tests run in CI and a red build blocks the merge. This is the single most effective quality gate a team has. The workflow file is in the DevOps guide; the branch protection that makes red mean blocked is in Git and GitHub.
In the sprint you planned it. User testing appears on the board as work, or it happens “if we have time”, which means never.
Before every demo and release. Full suite plus a manual smoke test. If a workflow matters enough to demo, it matters enough to have an end-to-end test.
After every incident. Write the test that reproduces the failure before you fix it. Over a year that builds a suite shaped by real problems rather than imagined ones.

Organizing and Maintaining Your Test Suite

Consistent test organization makes it easier for the team to find, run, and maintain tests.

Keep tests close to the code they test. Many frameworks support a tests/ directory mirroring the source structure, or colocated test files (e.g., login.test.ts next to login.ts). Either is fine; pick one and be consistent.
Name tests descriptively. test_login_with_invalid_password_returns_401 tells a reviewer exactly what is being verified without reading the test body.
Use test data thoughtfully. Hardcoded test data scattered across test files becomes a maintenance burden. Use fixtures, factories, or builder patterns to create test data in a central, reusable way.

Document your testing conventions (framework, file organization, how to run tests) in CONTRIBUTING.md so that every team member and every AI coding tool follows the same patterns.

DocumentationWhere testing conventions belong in your repository documentation.

Dealing with Flaky Tests

A flaky test passes sometimes and fails other times with no code change. They are corrosive: they teach the team to ignore failures, which defeats the point of testing at all. Four causes cover almost all of them:

Timing. A sleep() or an assumed duration. Poll or wait explicitly instead.
Shared state. A test that depends on what the previous one left behind. Each test sets up and tears down its own.
External services. A real API or database in the loop. Use a test double, or a local or containerized instance.
Non-determinism. Random data, concurrency, or the clock. Seed the generator, control the concurrency, freeze time.

Fix a flaky test or delete it. A test you cannot trust is worse than no test.

Testing Tools

The right tools depend on your stack. Here are common choices across popular frameworks:

Stack	Test Framework	E2E / Integration
Python	pytest	Playwright, httpx
JavaScript / TypeScript	Vitest, Jest	Playwright, Cypress
React	React Testing Library	Playwright
Go	built-in `testing` package	httptest
Java	JUnit	Selenium
Mobile (React Native)	Jest + React Native Testing Library	Detox

For usability testing, screen and session recording tools like Lookback, Hotjar, or even a simple Zoom recording can capture user interactions for later analysis.

Choose tools your team will actually use. A simple pytest suite maintained diligently beats a sophisticated testing infrastructure that nobody runs.

Best Practices

Treat test code with the same care as production code. Tests that are hard to read, poorly organized, or full of duplication become a burden rather than a safety net.
Fix broken tests immediately. A test suite that stays red for days loses all value as a quality signal.
Delete tests that no longer serve a purpose. Tests for removed features or deprecated code paths add noise and slow down the suite.
Keep tests fast. A test suite that takes ten minutes to run will not be run frequently. If the suite is slow, identify the bottleneck (usually a few slow integration or E2E tests) and optimize or isolate those tests.

Some Truths About Testing

The pyramid this guide opens with is a guideline, and it is worth knowing that it is also contested. Mike Cohn’s original shape assumed integration tests were slow and brittle, which was true of the tooling in 2009. Kent C. Dodds argues for a testing trophy instead: more integration tests than unit tests, on the grounds that integration tests buy more confidence per test and that modern tooling has made them fast enough to write in bulk. Google’s testing team reached a related conclusion from the opposite end, arguing you should say no to more end-to-end tests because they are slow and flaky at scale. Which shape fits depends on how expensive your integration tests actually are, so measure before you commit to one.

Coverage percentage is the metric most likely to mislead you, and the reason is mechanical: coverage records which lines executed, not whether anything was checked. A test suite that calls every function and asserts nothing reports excellent coverage. This is why a coverage target set as a team goal tends to produce tests written to raise the number rather than tests that would catch a regression. Use coverage to find code nobody tested at all, which it is good at, and not as a quality score, which it cannot be.

Flaky tests deserve more attention than they get, because they fail in a way that destroys the value of everything around them. A test that passes and fails without the code changing teaches the team to ignore red builds, and once that habit forms a real failure gets ignored too. Google has written about the scale of this: a meaningful fraction of their test failures are flakes, which is why they invest in detecting and quarantining them. On a student project the correct response is usually to fix it the day it appears or delete it, because a quarantine you never revisit is just a deleted test with extra steps. Some flakiness is genuinely unavoidable, such as a test that depends on a network service, and automatic reruns are a reasonable mitigation there.

Writing testable code does tend to improve design, but the claim is often stated too strongly. What testing pressure actually rewards is separating decisions from side effects, because a function that only computes is trivial to test and a function that computes and writes to a database is not. That is a real design improvement most of the time. It is not a universal law: tests can also push you toward over-abstraction, adding interfaces that exist only so a mock can be injected. If a change makes the code harder to read and easier to test, that is a trade to make deliberately, not an automatic win.

Finally, automated tests answer whether the code does what you specified. They cannot tell you whether you specified the right thing. A suite that is entirely green on a feature nobody wants is a well-tested waste of a term, which is why this guide keeps sending you back to real users and why research projects need reproducibility checks rather than just passing tests. Results you cannot reproduce are not results, and software nobody can use is not shipped.

Testing in Industry and Academia

In industry, both automated testing and user research are baseline expectations. Companies like Google require tests for virtually all production code; their internal testing culture is documented extensively in Software Engineering at Google. At the same time, product teams at companies like Stripe, Airbnb, and Shopify run continuous user research to validate that what they build actually serves their customers. Neither discipline substitutes for the other.

The testing pyramid (or variations like the testing trophy) is a widely used heuristic for automated test strategy. Kent C. Dodds and Martin Fowler have written extensively about this, and their work is worth reading regardless of your stack.

In academia, reproducibility serves the same purpose as automated testing: it allows others to verify that results are valid. Journals increasingly require that computational results be backed by public code and data, with clear instructions for reproducing the findings. A well-tested research codebase is publishable; an untested one is a liability.

The habit of testing holistically, through automated suites, manual exploration, reproducibility checks, and user validation, is one of the most transferable skills a Capstone project can develop.

Additional Readings

The testing literature is opinionated and the disagreements are real, so read at least two of the first three entries and notice where they conflict. The tooling links are here because the practical question, how expensive your integration tests are, is answered by which tools you pick.

Sources and further reading

The Practical Test Pyramid, Ham Vocke on Martin Fowler’s site: the most thorough treatment of the shape and what each layer is actually for.
Static vs Unit vs Integration vs E2E Testing, Kent C. Dodds: the testing trophy, and the argument against the pyramid’s proportions.
Just Say No to More End-to-End Tests, Google Testing Blog: the same argument from the other direction.
Flaky Tests at Google: how big the flakiness problem gets and what to do about it.
Eradicating Non-Determinism in Tests, Martin Fowler: the specific causes of flaky tests and how to remove each one.
Hypothesis: property-based testing, for when example-based tests keep missing the edge case.
Playwright: browser automation for the end-to-end layer, including accessibility assertions.
Test and evaluate, W3C Web Accessibility Initiative: the accessibility half, which automated tools only partly cover.

Activities that exercise this

Test Plan: turns this guide into a plan specific to your project.
Wire an Accessibility Audit Into CI: the automated half of the accessibility check.
Run an Acceptance Pass: checking work against what was asked, which tests cannot do for you.
Keep an Experiment Log: the research equivalent of a test record.