Google Cloud Deployment¶

If you are deciding whether GCP is the right target or you want the short version first, start with Deploy to GCP. This page is the deeper Cloud Run runbook.

This guide covers deploying Validibot to Google Cloud Run with support for multiple environments (dev, staging, prod).

Multi-Environment Architecture¶

Validibot supports three deployment stages:

Each stage has isolated: - Cloud Run services (web + worker) - Cloud SQL database instance - Secrets in Secret Manager - Application and validator-provider Cloud Tasks queues - Application, validator-runtime, and provider-invoker service accounts

Shared across stages: - GCS buckets (with stage prefixes in paths) - Artifact Registry (same images, different services) - Cloud KMS keys

Quick Start with justfile¶

All deployment commands accept a stage parameter:

# Deploy to dev
just gcp deploy dev
just gcp deploy-worker dev

# Deploy to production
just gcp deploy prod
just gcp deploy-worker prod

# Deploy both services at once
just gcp deploy-all dev

# Replace the staged revision while a stage remains fully offline
just gcp deploy-maintenance prod

# Run migrations
just gcp migrate dev

# View logs
just gcp logs dev
just gcp logs prod

Run just to see all available commands.

Setting Up a New Environment¶

The gcp-init-stage command works for all stages (dev, staging, prod). The command is idempotent - it checks for existing resources and only creates what's missing, making it safe to re-run.

Current production resources:

• Service account (web/worker): $GCP_APP_NAME-cloudrun-prod@PROJECT.iam.gserviceaccount.com
• Service account (validator runtime): $GCP_APP_NAME-validator-prod@PROJECT.iam.gserviceaccount.com
• Service account (provider invoker): $GCP_APP_NAME-val-invoker-prod@PROJECT.iam.gserviceaccount.com
• Cloud SQL: $GCP_APP_NAME-db
• Cloud Tasks queue: $GCP_APP_NAME-tasks
• Validator provider queue: $GCP_APP_NAME-validator-provider
• GCS bucket: $GCP_APP_NAME-storage (with public/ and private/ prefixes)
• Secret: django-env

To create a new environment from scratch (or verify existing resources):

Step 1: Initialize Infrastructure¶

# Creates service account, database, Cloud Tasks queue, GCS buckets, and secret placeholder
just gcp init-stage dev      # For dev environment
just gcp init-stage staging  # For staging environment
just gcp init-stage prod     # For production environment

This command creates (example for dev):

• Service account (web/worker): $GCP_APP_NAME-cloudrun-dev@PROJECT.iam.gserviceaccount.com
• Service account (validators): $GCP_APP_NAME-validator-dev@PROJECT.iam.gserviceaccount.com (worker callback/capability renewal only; no ambient storage role)
• Service account (provider invoker): $GCP_APP_NAME-val-invoker-dev@PROJECT.iam.gserviceaccount.com (sole validator Service invoker; no project roles)
• Cloud SQL instance: $GCP_APP_NAME-db-dev (db-f1-micro for dev, db-g1-small for staging, and db-custom-2-8192 for production)
• Database validibot and user validibot_user with generated password
• Cloud Tasks queue: $GCP_APP_NAME-validation-queue-dev
• Validator provider queue: $GCP_APP_NAME-validator-provider-dev
• GCS bucket: $GCP_APP_NAME-storage-dev (with public/ and private/ prefixes)
• Secret placeholder: django-env-dev

For production, resource names have no suffix (e.g., $GCP_APP_NAME-db, $GCP_APP_NAME-storage).

??? info "Cloud SQL connectivity and public IP" The deploys use the Cloud SQL Auth Proxy via --add-cloudsql-instances, which authenticates with IAM instead of IP allowlisting and encrypts traffic. This is a reasonable default for dev/staging and avoids VPC connector costs. If you need network isolation in production, plan a migration to Private IP + Serverless VPC Access and point Cloud Run at the connector; that adds cost/complexity but removes the public IP.

Step 2: Update Environment File with Password¶

Edit the appropriate environment file for your stage:

Replace PASSWORD_FROM_GCP_INIT with the actual password from Step 1. Remember to URL-encode special characters in DATABASE_URL (/ → %2F, = → %3D):

POSTGRES_PASSWORD=<actual-password-here>
DATABASE_URL=postgres://validibot_user:<url-encoded-password>@/validibot?host=/cloudsql/$GCP_PROJECT_ID:$GCP_REGION:<db-instance>

Where <db-instance> is $GCP_APP_NAME-db-dev, $GCP_APP_NAME-db-staging, or $GCP_APP_NAME-db (for prod).

While you're in the env file, also generate and set DJANGO_API_KEY_DIGEST_KEY and DJANGO_MFA_ENCRYPTION_KEY. Also set DJANGO_ALLOWED_HOSTS to the exact Cloud Run/custom hostnames and DJANGO_CSRF_TRUSTED_ORIGINS to the full public HTTPS origin, for example https://app.validibot.com. The API-key digest key must be separate from DJANGO_SECRET_KEY; production refuses to start without it so bearer tokens are never stored using Django's session/signing key as a fallback.

python -c "import secrets; print(secrets.token_urlsafe(32))"

Paste the output as DJANGO_API_KEY_DIGEST_KEY=<value>. Then generate the Fernet key for MFA secret material; the app validates the format at import time so malformed values fail the deploy rather than showing up as mysterious errors during MFA enrolment.

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Paste the output as DJANGO_MFA_ENCRYPTION_KEY=<value>. Generate a fresh key per stage — never reuse across dev / staging / prod. See configure-mfa.md for details.

No Redis setup is needed for the default GCP deployment. The production settings use Django's DatabaseCache backend (persisted in the same Cloud SQL instance) for allauth rate limiting and TOTP replay protection — a zero-marginal-cost option that's fine for pre-release volume. The migration step in Step 5 runs createcachetable automatically. See configure-mfa.md for the upgrade path to Memorystore Redis when traffic warrants it.

Step 3: Upload Secrets to Secret Manager¶

just gcp secrets <stage>  # e.g., just gcp secrets dev|staging|prod

Step 4: Deploy Services¶

# Deploy both web and worker
just gcp deploy-all <stage>
# e.g., just gcp deploy-all dev|staging|prod

The standard deploy recipes require a complete LIVE stage: database ready, normal public runtime ingress, running queues, and every managed scheduler job enabled. To stage the latest application while keeping the site offline, first use just gcp mode-maintenance <stage>, then just gcp deploy-maintenance <stage>. Maintenance entry isolates every runtime, pauses queues and schedulers, and starts or keeps Cloud SQL RUNNABLE. The deploy uses that same database for migrations and deploys web, worker, schedulers, and optional MCP with internal ingress and zero minimum capacity. An exit trap restores runtime isolation after success or failure without stopping the database, so consecutive maintenance operations do not repeatedly restart SQL. The cleanup first inspects Cloud Run and skips services already internal with zero minimum capacity. That matters when a newly selected revision is unhealthy: a redundant service update could otherwise fail before the remaining runtime safeguards are restored.

Use just gcp mode-parked <stage> as the sole explicit database stop after all maintenance work is complete. Database start and stop requests are submitted asynchronously and poll both instance state and the absence of an active control-plane operation, covering slow provider maintenance and eventual state reporting. The default transition deadline is 30 minutes; set GCP_SQL_TRANSITION_TIMEOUT_SECONDS for an exceptional longer operation. mode-status reports LIVE, MAINTENANCE, PARKED, or PARTIALLY_TRANSITIONED from runtime, queue, scheduler, and database signals. Optional MCP is internal, zero-capacity, and explicitly disabled while isolated. Taking the stage online exposes web first and then restores the configured MCP kill-switch value, which runs the normal license check.

Step 5: Database and Application Initialization¶

just gcp deploy-all <stage> already runs migrations and the complete guarded initializer before deploying services. A fresh database receives site/default data, validators and Step I/O, help content, and bundled validator resources. The production image includes the version-controlled curated EPW catalogue in data/weather/, so system EnergyPlus weather resources are reproducible from a clean checkout rather than depending on developer-local files. Before writing schema changes, the migration job runs the read-only check_migration_history preflight. It refuses a database that records one of the deliberately removed pre-2026-07-16 migration tails instead of attempting duplicate operations over that older schema. No separate first-install command is required.

For recovery, just gcp migrate <stage> reruns migration preparation and just gcp setup-data <stage> explicitly refreshes every initialized data concern using the currently deployed image.

Step 6: Deploy Validators¶

# Production: inspect and update one independent backend release.
just gcp validator-status prod
just gcp validator-update prod energyplus

The command reads the offered EnergyPlus version from validibot-validator-backends/backends.toml, verifies its backend-specific tag and release JSON, copies the exact GHCR digest into GAR without rebuilding, creates release-specific Service and Job resources, imports the database pairs, and runs normal plus Job-only acceptance before changing EnergyPlus routing.

The stage must already be in maintenance mode. Attempt-scoped token delivery and ambient-IAM denial are fixed GCP contracts; there are no storage rollout flags to configure. Image strictness remains configurable, and the production environment selects VALIDATOR_BACKEND_IMAGE_POLICY=digest. The acceptance command verifies the signed release, removes any historical validator storage binding, proves effective IAM denial, activates the candidate internally, runs the four real canaries and 20-attempt bursts, retains one private JSON report, exercises rollback to capability-aware Jobs, reactivates the accepted Services, and restores full maintenance mode. If any check fails, it restores Job routing but never re-grants ambient storage access.

Development may use the backend repo's local build/push recipes; production intentionally refuses that unsigned path.

If you have legacy jobs from the older validibot-validator-<slug> naming convention, you can safely delete them once you've confirmed no gcloud run jobs execute calls reference them.

Step 7: Set Up Scheduled Jobs¶

# Create Cloud Scheduler jobs for cleanup tasks
just gcp scheduler-setup <stage>

Step 8: Verify Deployment¶

# Check status and get service URL
just gcp status <stage>

# View logs
just gcp logs <stage>

# List all resources
just gcp list-resources <stage>

# Verify deployment inventory, queues, IAM, callbacks, and timeout chain.
just gcp doctor <stage> --strict

Strict doctor intentionally fails on any warning. VB205 describes the fixed GCS + Cloud Run storage contract; just gcp validator-acceptance separately proves the live token boundary, ambient-IAM denial, and representative execution before traffic is enabled.

Update both DJANGO_ALLOWED_HOSTS and DJANGO_CSRF_TRUSTED_ORIGINS in your stage env file before verification, then run just gcp django secrets <stage> and redeploy. The first setting takes hostnames; the second takes full origins including https://.

Regular Deployments¶

For routine code updates after initial setup:

# Deploy code changes to dev
just gcp deploy dev

# Deploy to both web and worker
just gcp deploy-all dev

# Run migrations if needed
just gcp migrate dev

# Deploy to production
just gcp deploy-all prod
just gcp migrate prod

Custom Domain Setup¶

There are two ways to map a custom domain to your Cloud Run services. Which one you use depends on your GCP region and requirements.

Option A: Cloud Run Domain Mappings (simpler)¶

Cloud Run has a built-in domain mapping feature that handles SSL certificates and DNS routing automatically. This is the simpler option but is only available in certain regions:

Supported regions: asia-east1, asia-northeast1, asia-southeast1, europe-north1, europe-west1, europe-west4, us-central1, us-east1, us-east4, us-west1

Not supported: australia-southeast1, australia-southeast2, and many others. If your region isn't listed above, you must use Option B.

Note: Domain mappings are still in preview and Google notes they may have latency issues. For high-traffic production deployments, Option B is generally recommended regardless of region.

To set up a domain mapping:

gcloud beta run domain-mappings create \
  --service $GCP_APP_NAME-web \
  --domain your-domain.com \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID

Then add the DNS records shown in the command output to your DNS provider.

For full details, see the Cloud Run domain mapping docs.

Option B: Global Application Load Balancer (recommended for production)¶

A Global external HTTP(S) Load Balancer works with all Cloud Run regions and gives you a static IP, CDN integration, and full control over SSL and routing. This is the recommended approach for production, and is required for regions that don't support domain mappings (e.g. australia-southeast1).

The justfile includes an idempotent command that creates the load balancer resources and prints the static IP you need to set in your DNS provider:

just gcp lb-setup prod validibot.com

If you want multiple hostnames on the same cert/load balancer, pass a comma-separated list:

just gcp lb-setup prod "validibot.com,www.validibot.com"

Cost: Global external load balancers have a non-zero base cost even at low traffic. Check GCP pricing before you commit.

DNS records¶

In your DNS provider, create an A record pointing at the static IP printed by lb-setup.

• For the apex/root domain (validibot.com): create an A record for host @ -> the load balancer IP.
• For www.validibot.com (optional): either add another A record pointing at the same IP, or use CNAME www -> validibot.com (and include www.validibot.com in the lb-setup domains list so the cert covers it).

SSL certificate provisioning¶

The load balancer uses a Google-managed certificate. After the DNS change propagates, provisioning typically takes 15-60 minutes.

Useful status commands:

# See the reserved IP (prod)
gcloud compute addresses describe $GCP_APP_NAME-ip --global \
  --project $GCP_PROJECT_ID

# See certificate status (prod)
gcloud compute ssl-certificates describe $GCP_APP_NAME-cert --global \
  --project $GCP_PROJECT_ID

App configuration (both options)¶

• Make sure DJANGO_ALLOWED_HOSTS (in .envs/.production/.google-cloud/.django) includes each exact Cloud Run or custom hostname. Set DJANGO_CSRF_TRUSTED_ORIGINS to each corresponding full HTTPS origin (for example https://app.validibot.com). Do not use a wildcard .run.app host. Then run just gcp django secrets prod and redeploy.
• Set these base URLs in your env file (they serve different purposes):
• SITE_URL: public web base URL (prod: https://validibot.com; dev/staging: the web *.run.app URL is fine).
• WORKER_URL: internal worker base URL (the worker *.run.app URL). Validator Services, retained Jobs, and Cloud Scheduler target the worker service; callbacks should never go to the public domain. You can fetch the current worker URL with:

gcloud run services describe $GCP_APP_NAME-worker \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID \
  --format='value(status.url)'

gcloud run services update $GCP_APP_NAME-web \
  --ingress internal-and-cloud-load-balancing \
  --region $GCP_REGION \
  --project $GCP_PROJECT_ID

Timeouts (avoiding "30s" surprises)¶

• Cloud Run request timeouts are configured on the Cloud Run service. This repo deploys with --timeout 3600s (see gcp_cloud_run_request_timeout in justfile).
• Gunicorn is configured to match via GUNICORN_TIMEOUT_SECONDS (defaults to 3600) in compose/production/django/start.sh and compose/production/django/start-worker.sh.
• If using Option B (load balancer): serverless NEGs do not support customizing the backend-service timeout, and the backend service will show timeoutSec=30. If you see requests ending around 30 seconds, check the Cloud Run service --timeout, plus any client-side timeouts (browser, reverse proxy, task runner).

Architecture Overview¶

┌─────────────────────────────────────────────────────────────┐
│                     Google Cloud Platform                    │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐   │
│  │  Cloud Run   │    │  Cloud Run   │    │  Cloud SQL   │   │
│  │  (web)       │───▶│  (worker)    │───▶│  PostgreSQL  │   │
│  │  Port 8000   │    │  Port 8001   │    │              │   │
│  └──────────────┘    └──────────────┘    └──────────────┘   │
│         │                   ▲                               │
│         │                   │ OIDC                          │
│         ▼            ┌──────┴───────┐                       │
│  ┌──────────────┐    │  Cloud       │    ┌──────────────┐   │
│  │  Cloud       │    │  Scheduler   │    │  Cloud       │   │
│  │  Storage     │    │  (cron)      │    │  Secret Mgr  │   │
│  │  (media)     │    └──────────────┘    │  (secrets)   │   │
│  └──────────────┘                        └──────────────┘   │
│                                                              │
│  ┌──────────────┐    ┌──────────────────────────────────┐   │
│  │  Cloud       │    │  Artifact Registry (Docker)       │   │
│  │  Tasks       │    └──────────────────────────────────┘   │
│  │  (async)     │                                           │
│  └──────────────┘                                           │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Prerequisites¶

Before deploying, ensure you have completed the Setup Cheatsheet:

• [x] gcloud CLI installed and authenticated
• [x] Project configured ($GCP_PROJECT_ID)
• [x] Required APIs enabled
• [x] Artifact Registry created ($GCP_APP_NAME)
• [x] Docker authentication configured

For production, also ensure: - [x] Cloud SQL instance created ($GCP_APP_NAME-db) - [x] Database and user created - [x] Secret Manager configured (django-env)

Pre-Deployment Checks¶

Before every deployment, run tests and linting:

# Run the test suite
uv run --group dev pytest

# Run linting
uv run --group dev ruff check

Optionally, run Django's deployment security checks against production settings:

# Check production settings (may require some env vars to be set)
uv run python manage.py check --deploy --settings=config.settings.production

Secrets Management¶

Each stage has its own secrets file and Secret Manager entry:

To update secrets:

# Edit the file
vim .envs/.production/.google-cloud/.django

# Upload to Secret Manager
just gcp secrets prod

# Redeploy to pick up changes
just gcp deploy prod

Operations¶

View Logs¶

# Recent logs
just gcp logs dev

# Follow logs in real-time
just gcp logs-follow dev

# View job logs (migrations, setup)
just gcp job-logs $GCP_APP_NAME-migrate-dev

Check Status¶

# Single stage
just gcp status dev

# All stages
just gcp status-all

Pause/Resume Service¶

# Isolate runtime/work and start or keep the database available
just gcp mode-maintenance dev

# Report LIVE, MAINTENANCE, PARKED, or PARTIALLY_TRANSITIONED
just gcp mode-status dev

# Stage a release or run a management command without opening the stage
just gcp deploy-maintenance dev
just gcp maintenance-management-cmd dev "check_validibot --strict"

# Explicitly stop the database after all maintenance work
just gcp mode-parked dev

# Or restore capacity, queues, schedulers, and ingress
just gcp mode-live dev

The lower-level pause/resume recipes only change web ingress. Use the maintenance recipes when the intent is to take a stage offline or control cost.

List Resources¶

# See all resources for a stage
just gcp list-resources dev

Scheduled Jobs (Cloud Scheduler)¶

# Set up scheduled jobs for a stage
just gcp scheduler-setup dev
just gcp scheduler-setup prod

# List all scheduler jobs
just gcp scheduler-list

# Run a job manually (for testing)
just gcp scheduler-run $GCP_APP_NAME-clear-sessions-dev

# Delete all scheduler jobs for a stage
just gcp scheduler-delete-all dev

Validator backends¶

# Routine read-only view and one-backend update.
just gcp validator-status prod
just gcp validator-update prod energyplus

# Diagnostic/recovery: stage one backend pair without activating it.
just gcp validator-deploy energyplus prod energyplus-v0.15.1

# Diagnostic/recovery: deploy just one execution shape.
just gcp validator-job-deploy energyplus prod energyplus-v0.15.1
just gcp validator-service-deploy energyplus prod energyplus-v0.15.1

# List validator jobs
gcloud run jobs list --filter="name~$GCP_APP_NAME-validator" --region=$GCP_REGION --project=$GCP_PROJECT_ID

Build and Push Docker Image¶

The gcp-deploy commands handle this automatically, but you can also run manually:

# Build for Cloud Run (linux/amd64)
just gcp build

# Push to Artifact Registry
just gcp push

Troubleshooting¶

View detailed logs¶

# Real-time logs for web service
gcloud run services logs tail $GCP_APP_NAME-web-dev --region=$GCP_REGION

# Historical logs with filtering
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=$GCP_APP_NAME-web-dev" --limit=100

Connect to Cloud SQL directly¶

# Using Cloud SQL Auth Proxy
gcloud sql connect $GCP_APP_NAME-db-dev --user=validibot_user --database=validibot

Check secret values¶

gcloud secrets versions access latest --secret=django-env-dev

Common issues¶

"Secret not found" error:

# Ensure secret exists
gcloud secrets describe django-env-dev

# If not, create it
just gcp secrets dev

"Service account not found" error:

# Re-run infrastructure setup
just gcp init-stage dev

Database connection errors:

# Verify Cloud SQL instance is running
gcloud sql instances describe $GCP_APP_NAME-db-dev --format="value(state)"

# Check connection name in secrets matches instance

Local vs Production¶

There is no docker-compose.production.yml — production runs on Cloud Run, not Docker Compose.

Cost Estimates¶

Monthly costs per stage (approximate, Australia region):

Dev uses smaller database tiers to minimize costs. All environments scale to zero when not in use.