Google Cloud Deployment¶

If you are deciding whether GCP is the right target or you want the short version first, start with Deploy to GCP. This page is the deeper Cloud Run runbook.

This guide covers deploying Validibot to Google Cloud Run with support for multiple environments (dev, staging, prod).

Multi-Environment Architecture¶

Validibot supports three deployment stages:

Each stage has isolated: - Cloud Run services (web + worker) - Cloud SQL database instance - Secrets in Secret Manager - Application and validator-provider Cloud Tasks queues - Application, validator-runtime, and provider-invoker service accounts

Shared across stages: - GCS buckets (with stage prefixes in paths) - Artifact Registry (same images, different services) - Cloud KMS keys

Quick Start with justfile¶

All deployment commands accept a stage parameter:

# Deploy to dev
just gcp deploy dev
just gcp deploy-worker dev

# Deploy to production
just gcp deploy prod
just gcp deploy-worker prod

# Deploy both services at once
just gcp deploy-all dev

# Replace the staged revision while a stage remains fully offline
just gcp deploy-maintenance prod

# Run migrations
just gcp migrate dev

# View logs
just gcp logs dev
just gcp logs prod

Run just to see all available commands.

Setting Up a New Environment¶

The just gcp init-stage command works for all stages (dev, staging, prod). The command is idempotent: it checks for existing resources and only creates what is missing, making it safe to re-run. There is no single command that performs the complete first-time installation; infrastructure, generated secret configuration, application deployment, and managed validator setup are separate checkpoints.

Current production resources:

• Service account (web/worker): $GCP_APP_NAME-cloudrun-prod@PROJECT.iam.gserviceaccount.com
• Service account (validator runtime): $GCP_APP_NAME-validator-prod@PROJECT.iam.gserviceaccount.com
• Service account (provider invoker): $GCP_APP_NAME-val-invoker-prod@PROJECT.iam.gserviceaccount.com
• Cloud SQL: $GCP_APP_NAME-db
• Cloud Tasks queue: $GCP_APP_NAME-tasks
• Validator provider queue: $GCP_APP_NAME-validator-provider
• GCS bucket: $GCP_APP_NAME-storage (with public/ and private/ prefixes)
• Secret: django-env

To create a new environment from scratch (or verify existing resources):

Step 1: Initialize Infrastructure¶

# Creates service account, database, Cloud Tasks queue, GCS buckets, and secret placeholder
just gcp init-stage dev      # For dev environment
just gcp init-stage staging  # For staging environment
just gcp init-stage prod     # For production environment

This command creates (example for dev):

• Service account (web/worker): $GCP_APP_NAME-cloudrun-dev@PROJECT.iam.gserviceaccount.com
• Service account (validators): $GCP_APP_NAME-validator-dev@PROJECT.iam.gserviceaccount.com (worker callback/capability renewal only; no ambient storage role)
• Service account (provider invoker): $GCP_APP_NAME-val-invoker-dev@PROJECT.iam.gserviceaccount.com (sole validator Service invoker; no project roles)
• Cloud SQL instance: $GCP_APP_NAME-db-dev (db-f1-micro for dev, db-g1-small for staging, and db-custom-2-8192 for production)
• Database validibot and user validibot_user with generated password
• Cloud Tasks queue: $GCP_APP_NAME-validation-queue-dev
• Validator provider queue: $GCP_APP_NAME-validator-provider-dev
• GCS bucket: $GCP_APP_NAME-storage-dev (with public/ and private/ prefixes)
• Secret placeholder: django-env-dev

For production, resource names have no suffix (e.g., $GCP_APP_NAME-db, $GCP_APP_NAME-storage).

??? info "Cloud SQL connectivity and public IP" The deploys use the Cloud SQL Auth Proxy via --add-cloudsql-instances, which authenticates with IAM instead of IP allowlisting and encrypts traffic. This is a reasonable default for dev/staging and avoids VPC connector costs. If you need network isolation in production, plan a migration to Private IP + Serverless VPC Access and point Cloud Run at the connector; that adds cost/complexity but removes the public IP.

Step 2: Update Environment File with Password¶

Edit the appropriate environment file for your stage:

Replace PASSWORD_FROM_GCP_INIT with the actual password from Step 1. Remember to URL-encode special characters in DATABASE_URL (/ → %2F, = → %3D):

POSTGRES_PASSWORD=<actual-password-here>
DATABASE_URL=postgres://validibot_user:<url-encoded-password>@/validibot?host=/cloudsql/$GCP_PROJECT_ID:$GCP_REGION:<db-instance>

Where <db-instance> is $GCP_APP_NAME-db-dev, $GCP_APP_NAME-db-staging, or $GCP_APP_NAME-db (for prod).

While you're in the env file, also generate and set DJANGO_API_KEY_DIGEST_KEY and DJANGO_MFA_ENCRYPTION_KEY. Also set DJANGO_ALLOWED_HOSTS to the exact Cloud Run/custom hostnames and DJANGO_CSRF_TRUSTED_ORIGINS to the full public HTTPS origin, for example https://app.validibot.com. The API-key digest key must be separate from DJANGO_SECRET_KEY; production refuses to start without it so bearer tokens are never stored using Django's session/signing key as a fallback.

python -c "import secrets; print(secrets.token_urlsafe(32))"

Paste the output as DJANGO_API_KEY_DIGEST_KEY=<value>. Then generate the Fernet key for MFA secret material; the app validates the format at import time so malformed values fail the deploy rather than showing up as mysterious errors during MFA enrolment.

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Paste the output as DJANGO_MFA_ENCRYPTION_KEY=<value>. Generate a fresh key per stage — never reuse across dev / staging / prod. See configure-mfa.md for details.

No Redis setup is needed for the default GCP deployment. The production settings use Django's DatabaseCache backend (persisted in the same Cloud SQL instance) for allauth rate limiting and TOTP replay protection — a zero-marginal-cost option that's fine for pre-release volume. The migration step in Step 5 runs createcachetable automatically. See configure-mfa.md for the upgrade path to Memorystore Redis when traffic warrants it.

Step 3: Upload Secrets to Secret Manager¶

just gcp secrets <stage>  # e.g., just gcp secrets dev|staging|prod

Step 4: Deploy Services¶

# Deploy both web and worker
just gcp deploy-all <stage>
# e.g., just gcp deploy-all dev|staging|prod

The standard deploy recipes require a complete LIVE stage: database ready, normal public runtime ingress, running queues, and every managed scheduler job enabled. To stage the latest application while keeping the site offline, first use just gcp mode-maintenance <stage>, then just gcp deploy-maintenance <stage>. Maintenance entry isolates every runtime, pauses queues and schedulers, and starts or keeps Cloud SQL RUNNABLE. The deploy uses that same database for migrations and deploys web, worker, schedulers, and optional MCP with internal ingress and zero minimum capacity. An exit trap restores runtime isolation after success or failure without stopping the database, so consecutive maintenance operations do not repeatedly restart SQL. The cleanup first inspects Cloud Run and skips services already internal with zero minimum capacity. That matters when a newly selected revision is unhealthy: a redundant service update could otherwise fail before the remaining runtime safeguards are restored.

Use just gcp mode-parked <stage> as the sole explicit database stop after all maintenance work is complete. Database start and stop requests are submitted asynchronously and poll both instance state and the absence of an active control-plane operation, covering slow provider maintenance and eventual state reporting. The default transition deadline is 30 minutes; set GCP_SQL_TRANSITION_TIMEOUT_SECONDS for an exceptional longer operation. mode-status reports LIVE, MAINTENANCE, PARKED, or PARTIALLY_TRANSITIONED from runtime, queue, scheduler, and database signals. Optional MCP is internal, zero-capacity, and explicitly disabled while isolated. Taking the stage online exposes web first and then restores the configured MCP kill-switch value, which runs the normal license check.

Step 5: Database and Application Initialization¶

just gcp deploy-all <stage> already runs migrations and the complete guarded initializer before deploying services. A fresh database receives site/default data, validators and Step I/O, help content, and bundled validator resources. The production image includes the version-controlled curated EPW catalogue in data/weather/, so system EnergyPlus weather resources are reproducible from a clean checkout rather than depending on developer-local files. Before writing schema changes, the migration job runs the read-only check_migration_history preflight. It refuses a database that records one of the deliberately removed pre-2026-07-16 migration tails instead of attempting duplicate operations over that older schema. No separate first-install command is required for migrations or application data. Managed Cloud Run validator backends are a separate release boundary and still require Step 6.

For recovery, just gcp migrate <stage> reruns migration preparation and just gcp setup-data <stage> explicitly refreshes every initialized data concern using the currently deployed image.

Step 6: Deploy Validators¶

# Daemonless crane is preferred; Docker Buildx is the fallback.
crane version  # or: docker info

# A fresh stage has no active managed backend routes. Inspect it, then install
# and activate every latest offered backend release.
just gcp validator-status <stage>
just gcp validator-setup <stage>

Setup reads every offered backend version from validibot-validator-backends/backends.toml, verifies each backend-specific tag and release JSON, copies the exact GHCR digest into GAR without rebuilding, creates release-specific Service and Job resources, imports the database pairs, and runs normal plus Job-only acceptance before activating all backend routes together. A stage-specific reusable management Job performs each backend's import and both acceptance phases in one execution. For later releases, use just gcp validator-update <stage> [backend] instead of setup.

The release-specific Service/Job pair is the provider identity. The database route selects the active pair, and each execution attempt snapshots the exact deployment before contacting GCP. See the private project documentation for the full source-to-GCP flow and the rationale for keeping application logic, backend images, deployment records, and hosted operator policy in their respective repositories: validibot-project/docs/operations/validator-backend-gcp-architecture.md.

Setup and update move a complete LIVE stage into maintenance and restore its initial mode after the operation. Their lower-level acceptance step requires maintenance. Attempt-scoped token delivery and ambient-IAM denial are fixed GCP contracts; there are no storage rollout flags to configure. Image strictness remains configurable, and the production environment selects VALIDATOR_BACKEND_IMAGE_POLICY=digest. Acceptance verifies the signed release, removes any historical validator storage binding, proves effective IAM denial, activates the candidate internally, runs every compatible semantic canary with three concurrent attempts in both Service and Job mode, retains private JSON reports, exercises capability-aware Job routing, parks the accepted candidate until final activation, and restores full maintenance mode. The release smoke records individual timings and min/median/max summaries; it does not claim p50/p95 performance evidence. If any check fails, it restores the prior accepted route but never re-grants ambient storage access.

Use just gcp validator-latency-report prod 168 20 separately when at least 20 real executions exist for an immutable backend, execution kind, and revision. That operational report—not routine release acceptance—is the percentile measurement surface.

Maintenance pauses rather than deletes Cloud Tasks. Before a candidate is deployed or routed, setup and update require both the application queue and the validator-provider queue to be empty, and require every managed execution attempt in the database to be terminal. If work remains, the command changes no provider route and restores the lifecycle mode that was active on entry, so the preserved work continues against the existing accepted release. This gate is required because private acceptance temporarily resumes the shared queues. If attempt-scoped storage authority cannot be prepared before Cloud Tasks is called, the claimed attempt is failed immediately; it is not left DISPATCHING until the acceptance timeout.

Development may use the backend repo's local build/push recipes; production intentionally refuses that unsigned path.

Production mirroring prefers daemonless crane and falls back to Docker Buildx; it never rebuilds the backend and skips a copy when GAR already has the verified digest. Check the operator machine before starting setup or an update:

crane version  # or: docker info

If neither registry client is available, the release command fails immediately before changing a provider resource. If a multi-step setup or update is interrupted, verify the stage is LIVE or MAINTENANCE; use VALIDATOR_PROVIDER_RETENTION_MODE=drain-and-retain for recovery and do not manually delete validator providers. To restore an isolated stage without deleting providers:

VALIDATOR_PROVIDER_RETENTION_MODE=drain-and-retain GCP_YES=1 \
    just gcp mode-live prod

Provider retention policy¶

The hosted installation currently has no users or validation data, so its operator wrapper sets VALIDATOR_PROVIDER_RETENTION_MODE=latest-only. After a release is accepted and active, validator-setup, validator-update, and mode-live reconcile the stage before queues and schedulers resume. The reconciliation requires maintenance mode, a normal active route for every managed backend, no unfinished attempts, and canonical release-specific provider names. It then removes superseded Services and Jobs, including legacy names such as validibot-validator-service-<backend>-v<version>, while retaining historical database rows and evidence.

# Explicit recovery/reconciliation command; run only while in maintenance.
just gcp mode-maintenance prod
just gcp validator-latest-only prod

The manual command requires an explicit confirmation phrase and fails closed if the latest offered release is not active, an attempt is unfinished, or the database route does not match the canonical provider pair. Cloud Run min=0 remains a scale-to-zero setting; it does not disable the active route.

Before serving users or durable validation data, set VALIDATOR_PROVIDER_RETENTION_MODE=drain-and-retain. That mode keeps the previous accepted pair for rollback and pinned unfinished attempts, then uses the normal seven-day drain and validator-cleanup lifecycle. Do not manually delete legacy providers: use the reconciliation command so provider deletion is recorded and active or in-flight routes are protected.

Step 7: Verify Deployment¶

just gcp deploy-all <stage> already synchronizes the managed Cloud Scheduler jobs. just gcp scheduler-setup <stage> remains available as an explicit recovery command; it is not another first-install step.

# Check status and get service URL
just gcp status <stage>

# View logs
just gcp logs <stage>

# List all resources
just gcp list-resources <stage>

# Verify deployment inventory, queues, IAM, callbacks, and timeout chain.
just gcp doctor <stage> --strict

Strict doctor intentionally fails on any warning. VB205 describes the fixed GCS + Cloud Run storage contract; just gcp validator-acceptance separately proves the live token boundary, ambient-IAM denial, and representative execution before traffic is enabled.

Update both DJANGO_ALLOWED_HOSTS and DJANGO_CSRF_TRUSTED_ORIGINS in your stage env file before verification, then run just gcp django secrets <stage> and redeploy. The first setting takes hostnames; the second takes full origins including https://.

Regular Deployments¶

For routine code updates after initial setup:

# Deploy code changes to dev
just gcp deploy dev

# Deploy to both web and worker
just gcp deploy-all dev

# Run migrations if needed
just gcp migrate dev

# Deploy to production
just gcp deploy-all prod
just gcp migrate prod

Custom Domain Setup¶

There are two ways to map a custom domain to your Cloud Run services. Which one you use depends on your GCP region and requirements.

Option A: Cloud Run Domain Mappings (simpler)¶

Cloud Run has a built-in domain mapping feature that handles SSL certificates and DNS routing automatically. This is the simpler option but is only available in certain regions:

Supported regions: asia-east1, asia-northeast1, asia-southeast1, europe-north1, europe-west1, europe-west4, us-central1, us-east1, us-east4, us-west1

Not supported: australia-southeast1, australia-southeast2, and many others. If your region isn't listed above, you must use Option B.

Note: Domain mappings are still in preview and Google notes they may have latency issues. For high-traffic production deployments, Option B is generally recommended regardless of region.

To set up a domain mapping:

gcloud beta run domain-mappings create \
  --service $GCP_APP_NAME-web \
  --domain your-domain.com \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID

Then add the DNS records shown in the command output to your DNS provider.

For full details, see the Cloud Run domain mapping docs.

Option B: Global Application Load Balancer (recommended for production)¶

A Global external HTTP(S) Load Balancer works with all Cloud Run regions and gives you a static IP, CDN integration, and full control over SSL and routing. This is the recommended approach for production, and is required for regions that don't support domain mappings (e.g. australia-southeast1).

The justfile includes an idempotent command that creates the load balancer resources and prints the static IP you need to set in your DNS provider:

just gcp lb-setup prod validibot.com

If you want multiple hostnames on the same cert/load balancer, pass a comma-separated list:

just gcp lb-setup prod "validibot.com,www.validibot.com"

Cost: Global external load balancers have a non-zero base cost even at low traffic. Check GCP pricing before you commit.

DNS records¶

In your DNS provider, create an A record pointing at the static IP printed by lb-setup.

• For the apex/root domain (validibot.com): create an A record for host @ -> the load balancer IP.
• For www.validibot.com (optional): either add another A record pointing at the same IP, or use CNAME www -> validibot.com (and include www.validibot.com in the lb-setup domains list so the cert covers it).

SSL certificate provisioning¶

The load balancer uses a Google-managed certificate. After the DNS change propagates, provisioning typically takes 15-60 minutes.

Useful status commands:

# See the reserved IP (prod)
gcloud compute addresses describe $GCP_APP_NAME-ip --global \
  --project $GCP_PROJECT_ID

# See certificate status (prod)
gcloud compute ssl-certificates describe $GCP_APP_NAME-cert --global \
  --project $GCP_PROJECT_ID

App configuration (both options)¶

• Make sure DJANGO_ALLOWED_HOSTS (in .envs/.production/.google-cloud/.django) includes each exact Cloud Run or custom hostname. Set DJANGO_CSRF_TRUSTED_ORIGINS to each corresponding full HTTPS origin (for example https://app.validibot.com). Do not use a wildcard .run.app host. Then run just gcp django secrets prod and redeploy.
• Set these base URLs in your env file (they serve different purposes):
• SITE_URL: public web base URL (prod: https://validibot.com; dev/staging: the web *.run.app URL is fine).
• WORKER_URL: internal worker base URL (the worker *.run.app URL). Validator Services, retained Jobs, and Cloud Scheduler target the worker service; callbacks should never go to the public domain. You can fetch the current worker URL with:

gcloud run services describe $GCP_APP_NAME-worker \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID \
  --format='value(status.url)'

gcloud run services update $GCP_APP_NAME-web \
  --ingress internal-and-cloud-load-balancing \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID

Timeouts (avoiding "30s" surprises)¶

• Cloud Run request timeouts are configured on the Cloud Run service. This repo deploys with --timeout 3600s (see gcp_cloud_run_request_timeout in justfile).
• Gunicorn is configured to match via GUNICORN_TIMEOUT_SECONDS (defaults to 3600) in compose/production/django/start.sh and compose/production/django/start-worker.sh.
• If using Option B (load balancer): serverless NEGs do not support customizing the backend-service timeout, and the backend service will show timeoutSec=30. If you see requests ending around 30 seconds, check the Cloud Run service --timeout, plus any client-side timeouts (browser, reverse proxy, task runner).

Architecture Overview¶

┌─────────────────────────────────────────────────────────────┐
│                     Google Cloud Platform                    │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐   │
│  │  Cloud Run   │    │  Cloud Run   │    │  Cloud SQL   │   │
│  │  (web)       │───▶│  (worker)    │───▶│  PostgreSQL  │   │
│  │  Port 8000   │    │  Port 8001   │    │              │   │
│  └──────────────┘    └──────────────┘    └──────────────┘   │
│         │                   ▲                               │
│         │                   │ OIDC                          │
│         ▼            ┌──────┴───────┐                       │
│  ┌──────────────┐    │  Cloud       │    ┌──────────────┐   │
│  │  Cloud       │    │  Scheduler   │    │  Cloud       │   │
│  │  Storage     │    │  (cron)      │    │  Secret Mgr  │   │
│  │  (media)     │    └──────────────┘    │  (secrets)   │   │
│  └──────────────┘                        └──────────────┘   │
│                                                              │
│  ┌──────────────┐    ┌──────────────────────────────────┐   │
│  │  Cloud       │    │  Artifact Registry (Docker)       │   │
│  │  Tasks       │    └──────────────────────────────────┘   │
│  │  (async)     │                                           │
│  └──────────────┘                                           │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Prerequisites¶

Before deploying, ensure you have completed the Setup Cheatsheet:

• [x] gcloud CLI installed and authenticated
• [x] Project configured ($GCP_PROJECT_ID)
• [x] Required APIs enabled
• [x] Artifact Registry created ($GCP_APP_NAME)
• [x] Docker authentication configured

For production, also ensure: - [x] Cloud SQL instance created ($GCP_APP_NAME-db) - [x] Database and user created - [x] Secret Manager configured (django-env)

Pre-Deployment Checks¶

Before every deployment, run tests and linting:

# Run the test suite
uv run --group dev pytest

# Run linting
uv run --group dev ruff check

Optionally, run Django's deployment security checks against production settings:

# Check production settings (may require some env vars to be set)
uv run python manage.py check --deploy --settings=config.settings.production

Secrets Management¶

Each stage has its own secrets file and Secret Manager entry:

To update secrets:

# Edit the file
vim .envs/.production/.google-cloud/.django

# Upload to Secret Manager
just gcp secrets prod

# Redeploy to pick up changes
just gcp deploy prod

Operations¶

View Logs¶

# Recent logs
just gcp logs dev

# Follow logs in real-time
just gcp logs-follow dev

# View job logs (migrations, setup)
just gcp job-logs $GCP_APP_NAME-migrate-dev

Check Status¶

# Single stage
just gcp status dev

# All stages
just gcp status-all

Pause/Resume Service¶

# Isolate runtime/work and start or keep the database available
just gcp mode-maintenance dev

# Report LIVE, MAINTENANCE, PARKED, or PARTIALLY_TRANSITIONED
just gcp mode-status dev

# Stage a release or run a management command without opening the stage
just gcp deploy-maintenance dev
just gcp maintenance-management-cmd dev "check_validibot --strict"

# Explicitly stop the database after all maintenance work
just gcp mode-parked dev

# Or restore capacity, queues, schedulers, and ingress
just gcp mode-live dev

The lower-level pause/resume recipes only change web ingress. Use the maintenance recipes when the intent is to take a stage offline or control cost.

List Resources¶

# See all resources for a stage
just gcp list-resources dev

Scheduled Jobs (Cloud Scheduler)¶

# Set up scheduled jobs for a stage
just gcp scheduler-setup dev
just gcp scheduler-setup prod

# List all scheduler jobs
just gcp scheduler-list

# Run a job manually (for testing)
just gcp scheduler-run $GCP_APP_NAME-clear-sessions-dev

# Delete all scheduler jobs for a stage
just gcp scheduler-delete-all dev

Validator backends¶

# Routine read-only view and one-backend update.
just gcp validator-status prod
just gcp validator-update prod energyplus

# Diagnostic/recovery: stage one backend pair without activating it.
just gcp validator-deploy energyplus prod energyplus-v0.15.1

# Diagnostic/recovery: deploy just one execution shape.
just gcp validator-job-deploy energyplus prod energyplus-v0.15.1
just gcp validator-service-deploy energyplus prod energyplus-v0.15.1

# List validator jobs
gcloud run jobs list --filter="name~$GCP_APP_NAME-validator" --region=$GCP_REGION --project=$GCP_PROJECT_ID

Build and Push Docker Image¶

The gcp-deploy commands handle this automatically, but you can also run manually:

# Build for Cloud Run (linux/amd64)
just gcp build

# Push to Artifact Registry
just gcp push

Troubleshooting¶

View detailed logs¶

# Real-time logs for web service
gcloud run services logs tail $GCP_APP_NAME-web-dev --region=$GCP_REGION

# Historical logs with filtering
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=$GCP_APP_NAME-web-dev" --limit=100

Connect to Cloud SQL directly¶

# Using Cloud SQL Auth Proxy
gcloud sql connect $GCP_APP_NAME-db-dev --user=validibot_user --database=validibot

Check secret values¶

gcloud secrets versions access latest --secret=django-env-dev

Common issues¶

"Secret not found" error:

# Ensure secret exists
gcloud secrets describe django-env-dev

# If not, create it
just gcp secrets dev

"Service account not found" error:

# Re-run infrastructure setup
just gcp init-stage dev

Database connection errors:

# Verify Cloud SQL instance is running
gcloud sql instances describe $GCP_APP_NAME-db-dev --format="value(state)"

# Check connection name in secrets matches instance

Local vs Production¶

There is no docker-compose.production.yml — production runs on Cloud Run, not Docker Compose.

Cost Estimates¶

Monthly costs per stage (approximate, Australia region):

Dev uses smaller database tiers to minimize costs. All environments scale to zero when not in use.