Testing Overview¶

Validibot uses a layered testing strategy. Each layer runs at a different level of the stack and catches different kinds of bugs. Understanding what each layer does helps you pick the right test type when adding new functionality and know which suite to run when debugging a failure.

Test Layers at a Glance¶

How the layers differ¶

Unit and use case tests run with CELERY_TASK_ALWAYS_EAGER=True and Django's test database. They're fast and isolated but don't exercise the real HTTP/Celery/Redis pipeline. They run automatically as part of the normal pytest suite.

The SHACL use-case module also exercises its isolated validator container. It skips when the optional Docker SDK, daemon, or backend image is unavailable. CI always installs and builds those prerequisites. To run that module locally:

uv sync --group dev --extra docker-runner
docker build \
  -f ../validibot-validator-backends/validator_backends/shacl/Dockerfile \
  -t validibot-validator-backend-shacl:latest \
  ../validibot-validator-backends
pytest tests/tests_use_cases/test_shacl_validation.py

Integration tests run against real infrastructure - a local Postgres instance, GCS buckets, and retained Cloud Run Jobs. They verify those cloud adapters but do not exercise the full request lifecycle or primary validator Service handoff. That path requires E2E testing against a deployed environment. The tests require Docker and (for cloud tests) GCP credentials.

E2E tests run against a fully running Validibot environment (Docker Compose stack). They make real HTTP requests to the API and exercise the complete pipeline: API submission, Celery task dispatch, Redis broker, worker execution, and (for advanced validators) Docker container simulation. These catch problems that unit and integration tests can't: task dispatch failures, broker bottlenecks, concurrency bugs, and end-to-end data flow issues.

Which tests to run when¶

Running the test suite¶

# Unit + use case tests (the everyday default)
pytest

# With coverage
pytest --cov=validibot

# A specific app's tests
pytest validibot/validations/tests/

# A specific test file
pytest validibot/validations/tests/test_models.py -v

E2E and integration tests are excluded from the default pytest run via norecursedirs in pyproject.toml. Each has its own just recipe documented in its dedicated guide.

Type-checking baseline¶

Run just typecheck to execute the same production-code mypy guard as CI. The project still has historical Django typing debt, recorded in scripts/mypy_baseline.json by file and error code. Existing counts may fall, but a new category or an increase fails. After intentionally fixing existing diagnostics, ratchet the allowance downward with:

uv run python scripts/check_mypy_baseline.py --update

Do not update the baseline merely to make a new diagnostic pass; fix the new error unless the team has consciously accepted additional debt.

Detailed guides¶

• Integration Tests — retained Cloud Run Jobs, GCS, Selenium browser tests
• E2E and Stress Tests — Concurrent stress tests and EnergyPlus simulation tests against a live stack

Writing tests¶

A few conventions to follow when adding new tests:

• Every test file needs a module-level docstring explaining what the suite covers and why.
• Every test method needs a docstring explaining why the test matters, not just what it does.
• Always include security tests for features that handle user input. Test for XSS, injection, XXE, and other OWASP-style attacks.
• Use range assertions for simulation outputs. EnergyPlus and FMU results vary across versions and platforms, so assert within a reasonable range rather than exact values.
• Tests follow the code: commercial feature tests live in this repo (gated by feature flags), not in the commercial repos.

See AGENTS.md for the full coding standards.