Integration Tests¶

Integration tests verify that Validibot's cloud integrations work correctly. They run against real infrastructure - a local Postgres instance, GCS buckets, retained Cloud Run Jobs, and Selenium-driven browser tests. Unlike unit tests, they exercise individual infrastructure layers but don't test the full HTTP request lifecycle or the primary validator Service handoff (that's what a deployed-environment E2E test is for).

For an overview of all test layers, see the Testing Overview.

What integration tests cover¶

Retained Cloud Run Job execution and status lookup
GCS storage operations (uploading, downloading, signed URLs)
Selenium browser tests (UI flows with a real browser)
Database operations against a real Postgres instance

Prerequisites¶

Docker Compose available
GCP credentials configured (gcloud auth application-default login) for cloud tests
Required environment variables (see below)

Running¶

# Run all integration tests
just local test-integration

# Run a specific test file
just local test-integration tests/tests_integration/test_validator_jobs.py

# Run with extra verbosity
just local test-integration -vvv

The just local test-integration recipe:

Ensures the django Docker image exists (Chromium + chromedriver baked in for Selenium)
Resets and starts Postgres + Mailpit containers
Runs tests inside the Django container
Stops containers when done

Tips:

Set BUILD_DJANGO_IMAGE=1 to force a rebuild after Dockerfile changes
Set SELENIUM_HEADLESS=0 to watch Selenium tests in a browser
If running outside Docker, set CHROME_BIN and CHROMEDRIVER_PATH

Required environment variables¶

For the retained Cloud Run Job integration tests:

export GCP_PROJECT_ID=your-project
export GCS_VALIDATION_BUCKET=your-bucket
export GCP_REGION=us-west1

The Cloud Run Job names are no longer set via env vars. The tests resolve them at runtime via ValidatorConfig.cloud_run_job_name, which is derived from each validator's image_name field (defined in validators/{slug}/config.py). By project convention, the job name equals the image name and follows the validibot-validator-backend-{slug} pattern.

If a test reports it can't resolve the job name, check that the validator's config.py is imported during app startup (see validators/__init__.py) — that's the path that registers ValidatorConfig and makes get_config('ENERGYPLUS') return a non-None value.

These vars are typically exported in your shell (often via source set-env.sh) or loaded from .envs/ in Docker/GCP.

Cloud E2E tests (staging)¶

The integration test directory also includes tests that verify the complete validation flow against a deployed staging environment. The attempt resolves the active ValidatorExecutionDeployment: normally a private Cloud Run Service reached by the provider queue, or a retained Job for an over-budget or explicit rollback route. Either runtime must be able to call back to Django.

What the cloud E2E flow tests:

Submit file via API
Django creates ValidationRun
Django pins the exact deployment and dispatches it: deterministic provider Cloud Task for a Service, or Jobs API call for a retained Job
The selected runtime executes and calls back to the Django worker (via WORKER_URL)
Worker processes the callback and updates the database
Test polls API until completion
Test verifies status and findings

If your environment uses a custom public domain (via load balancer or domain mapping), make sure WORKER_URL is set to the worker *.run.app URL. Otherwise callbacks may accidentally route to the public domain (which points at the web service) and fail.

Running cloud E2E tests¶

# Set environment variables
export E2E_TEST_API_URL=https://your-staging-app.run.app/api/v1
export E2E_TEST_API_TOKEN=your-api-token
export E2E_TEST_WORKFLOW_ID=your-workflow-uuid

# Run
just local test-e2e

Variable	Required	Description
`E2E_TEST_API_URL`	Yes	API base URL (e.g., `https://staging.validibot.com/api/v1`)
`E2E_TEST_API_TOKEN`	Yes	Valid API token with workflow execution permission
`E2E_TEST_WORKFLOW_ID`	Yes	UUID of workflow with EnergyPlus validator step
`E2E_TEST_WORKFLOW_EXPECTS_SUCCESS`	No	Set to `false` if test file should fail validation (default: `true`)

Running in CI¶

# Integration tests (skip callbacks)
- name: Run integration tests
  env:
    GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
    GCS_VALIDATION_BUCKET: ${{ secrets.GCS_VALIDATION_BUCKET }}
    # ... other env vars
  run: just local test-integration

# E2E tests (post-deployment)
- name: Run E2E tests
  env:
    E2E_TEST_API_URL: https://staging.validibot.com/api/v1
    E2E_TEST_API_TOKEN: ${{ secrets.E2E_API_TOKEN }}
    E2E_TEST_WORKFLOW_ID: ${{ vars.E2E_WORKFLOW_ID }}
  run: just local test-e2e

psycopg3 + live_server fix¶

Django's live_server fixture uses a threaded WSGI server, but psycopg3 connections aren't thread-safe. After Selenium tests hit the live server, database connections can become corrupted.

The fix in tests/tests_integration/conftest.py:

Autouse fixture — Resets BAD psycopg3 connections before/after each test
Monkey-patched flush — Resets connections before Django's teardown flush

Additionally, config/settings/test.py sets CONN_MAX_AGE = 0 to disable persistent connections. This is a known Django + psycopg3 issue (Django tickets

32416, #35455).¶

Troubleshooting¶

Tests skip with "missing settings"¶

Ensure all required environment variables are set. Run env | grep -E "(GCP|GCS|E2E)" to check.

E2E test times out¶

Check Cloud Logging for the validation run
Verify the callback URL is correct and reachable
Inspect the attempt's pinned deployment. For Services, correlate its deterministic provider task and revision logs; for Jobs, check the Cloud Run execution status.