writing /

Terraform drift detection with Cloud Build and Cloud Scheduler

A daily drift check for Terraform on GCP that stays silent when nothing changed and posts to Google Chat when something did. With the full Cloud Build config.

Terraform only knows about changes made through Terraform. A hotfix in the console during an incident, a teammate testing something by hand, or a provider changing a default all leave the real infrastructure different from the code. You usually find out at the worst moment: the next apply wants to undo a change nobody remembers.

This post sets up a daily drift check on GCP with three managed services: Cloud Scheduler starts it, Cloud Build runs terraform plan for every module, and Google Chat gets a message only when something differs. No servers, no extra tools.

How it works

  1. Cloud Scheduler calls the Cloud Build API every weekday morning to run a trigger.
  2. Cloud Build checks out the infrastructure repository and runs terraform plan -detailed-exitcode for each module.
  3. The exit code tells you what happened: 0 no changes, 2 drift, anything else a failed plan.
  4. If any module drifted or failed, a second step posts one message to Google Chat with the list and a link to the build. Clean runs send nothing.

The last point matters most. A check that messages every day gets muted within a week.

The Cloud Build config

Save this as cloudbuild-drift.yaml in the infrastructure repository. _MODULES lists the Terraform root modules to check.

steps:
  - id: plan
    name: hashicorp/terraform:1.9
    entrypoint: sh
    args:
      - -c
      - |
        : > /workspace/drift.txt
        for dir in $_MODULES; do
          terraform -chdir="$$dir" init -input=false -no-color > /dev/null || { echo "FAILED  $$dir (init)" >> /workspace/drift.txt; continue; }
          terraform -chdir="$$dir" plan -detailed-exitcode -input=false -no-color -lock-timeout=5m > /dev/null
          code=$$?
          case $$code in
            0) echo "ok      $$dir" ;;
            2) echo "DRIFT   $$dir" | tee -a /workspace/drift.txt ;;
            *) echo "FAILED  $$dir (exit $$code)" | tee -a /workspace/drift.txt ;;
          esac
        done

  - id: notify
    name: gcr.io/cloud-builders/gcloud
    entrypoint: bash
    secretEnv: ['CHAT_WEBHOOK']
    env:
      - 'PROJECT=$PROJECT_ID'
      - 'BUILD_URL=https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=$PROJECT_ID'
    args:
      - -c
      - |
        if [ ! -s /workspace/drift.txt ]; then echo "No drift."; exit 0; fi
        python3 - > /workspace/msg.json <<'PY'
        import json, os
        report = open("/workspace/drift.txt").read().strip()
        text = f"Terraform drift in {os.environ['PROJECT']}:\n{report}\n{os.environ['BUILD_URL']}"
        print(json.dumps({"text": text}))
        PY
        curl -sS -X POST -H 'Content-Type: application/json' --data @/workspace/msg.json "$$CHAT_WEBHOOK"

availableSecrets:
  secretManager:
    - versionName: projects/$PROJECT_ID/secrets/drift-chat-webhook/versions/latest
      env: CHAT_WEBHOOK

substitutions:
  _MODULES: "envs/dev/network envs/dev/gke envs/prod/network envs/prod/gke"

options:
  logging: CLOUD_LOGGING_ONLY

A few details that are easy to get wrong:

  • $$ versus $. Cloud Build replaces $PROJECT_ID, $BUILD_ID and $_MODULES before the step runs. Shell variables need $$ so Cloud Build leaves them alone.
  • The plan loop never stops early. One failing module should not hide drift in the others, so each result is written to drift.txt and the loop continues.
  • The webhook stays secret. The Google Chat URL is a credential: anyone with it can post to your space. It lives in Secret Manager and is only exposed to the notify step.
  • Building the message in Python avoids broken JSON when module paths or error text contain quotes.
  • -lock-timeout=5m waits for a running apply instead of failing the check.

If you want the build itself to turn red on drift, add exit 1 after the curl line.

Permissions

Use a dedicated service account for the trigger, not the default Cloud Build account:

  • Read access to every project the modules manage, for example roles/viewer, plus any extra read roles your resources need for plan (some providers need roles/iam.securityReviewer to read IAM policies).
  • Write access to the state bucket, roles/storage.objectUser, because plan creates and removes a lock file even though it changes nothing.
  • Secret access, roles/secretmanager.secretAccessor on the webhook secret only.

It should not have any role that can change resources. A drift check that can apply is a second, unreviewed deployment path.

The trigger and the schedule

Create the webhook secret, then a manual Cloud Build trigger that points at cloudbuild-drift.yaml on main (Console, or gcloud builds triggers create manual). Then schedule it:

gcloud scheduler jobs create http terraform-drift-daily \
  --location=europe-west6 \
  --schedule="0 6 * * 1-5" \
  --time-zone="Africa/Cairo" \
  --uri="https://cloudbuild.googleapis.com/v1/projects/PROJECT_ID/locations/global/triggers/TRIGGER_ID:run" \
  --http-method=POST \
  --message-body='{"branchName":"main"}' \
  --oauth-service-account-email="drift-scheduler@PROJECT_ID.iam.gserviceaccount.com"

The scheduler’s service account only needs roles/cloudbuild.builds.editor to start the trigger. If your trigger is regional, replace global in the URL with its region.

What a message looks like

Terraform drift in platform-prod:
DRIFT   envs/prod/gke
FAILED  envs/prod/network (exit 1)
https://console.cloud.google.com/cloud-build/builds/...

From there, open the build log, read the plan for the drifted module, and decide: bring the change into Terraform, or let the next reviewed apply put it back.

Where to go next

  • More projects: keep one _MODULES list per repository, or one trigger per environment if the lists get long.
  • Fewer surprises: combine the check with an apply path that only runs from the main branch, so drift can only come from outside Terraform.
  • Ownership: send each team’s modules to its own Chat space by running one trigger per team.
← All writing