writing /
Terraform drift detection with Cloud Build and Cloud Scheduler
A daily drift check for Terraform on GCP that stays silent when nothing changed and posts to Google Chat when something did. With the full Cloud Build config.
Terraform only knows about changes made through Terraform. A hotfix in the console during an incident, a teammate testing something by hand, or a provider changing a default all leave the real infrastructure different from the code. You usually find out at the worst moment: the next apply wants to undo a change nobody remembers.
This post sets up a daily drift check on GCP with three managed services: Cloud Scheduler starts it, Cloud Build runs terraform plan for every module, and Google Chat gets a message only when something differs. No servers, no extra tools.
How it works
- Cloud Scheduler calls the Cloud Build API every weekday morning to run a trigger.
- Cloud Build checks out the infrastructure repository and runs
terraform plan -detailed-exitcodefor each module. - The exit code tells you what happened:
0no changes,2drift, anything else a failed plan. - If any module drifted or failed, a second step posts one message to Google Chat with the list and a link to the build. Clean runs send nothing.
The last point matters most. A check that messages every day gets muted within a week.
The Cloud Build config
Save this as cloudbuild-drift.yaml in the infrastructure repository. _MODULES lists the Terraform root modules to check.
steps:
- id: plan
name: hashicorp/terraform:1.9
entrypoint: sh
args:
- -c
- |
: > /workspace/drift.txt
for dir in $_MODULES; do
terraform -chdir="$$dir" init -input=false -no-color > /dev/null || { echo "FAILED $$dir (init)" >> /workspace/drift.txt; continue; }
terraform -chdir="$$dir" plan -detailed-exitcode -input=false -no-color -lock-timeout=5m > /dev/null
code=$$?
case $$code in
0) echo "ok $$dir" ;;
2) echo "DRIFT $$dir" | tee -a /workspace/drift.txt ;;
*) echo "FAILED $$dir (exit $$code)" | tee -a /workspace/drift.txt ;;
esac
done
- id: notify
name: gcr.io/cloud-builders/gcloud
entrypoint: bash
secretEnv: ['CHAT_WEBHOOK']
env:
- 'PROJECT=$PROJECT_ID'
- 'BUILD_URL=https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=$PROJECT_ID'
args:
- -c
- |
if [ ! -s /workspace/drift.txt ]; then echo "No drift."; exit 0; fi
python3 - > /workspace/msg.json <<'PY'
import json, os
report = open("/workspace/drift.txt").read().strip()
text = f"Terraform drift in {os.environ['PROJECT']}:\n{report}\n{os.environ['BUILD_URL']}"
print(json.dumps({"text": text}))
PY
curl -sS -X POST -H 'Content-Type: application/json' --data @/workspace/msg.json "$$CHAT_WEBHOOK"
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/drift-chat-webhook/versions/latest
env: CHAT_WEBHOOK
substitutions:
_MODULES: "envs/dev/network envs/dev/gke envs/prod/network envs/prod/gke"
options:
logging: CLOUD_LOGGING_ONLY
A few details that are easy to get wrong:
$$versus$. Cloud Build replaces$PROJECT_ID,$BUILD_IDand$_MODULESbefore the step runs. Shell variables need$$so Cloud Build leaves them alone.- The plan loop never stops early. One failing module should not hide drift in the others, so each result is written to
drift.txtand the loop continues. - The webhook stays secret. The Google Chat URL is a credential: anyone with it can post to your space. It lives in Secret Manager and is only exposed to the
notifystep. - Building the message in Python avoids broken JSON when module paths or error text contain quotes.
-lock-timeout=5mwaits for a runningapplyinstead of failing the check.
If you want the build itself to turn red on drift, add exit 1 after the curl line.
Permissions
Use a dedicated service account for the trigger, not the default Cloud Build account:
- Read access to every project the modules manage, for example
roles/viewer, plus any extra read roles your resources need forplan(some providers needroles/iam.securityReviewerto read IAM policies). - Write access to the state bucket,
roles/storage.objectUser, becauseplancreates and removes a lock file even though it changes nothing. - Secret access,
roles/secretmanager.secretAccessoron the webhook secret only.
It should not have any role that can change resources. A drift check that can apply is a second, unreviewed deployment path.
The trigger and the schedule
Create the webhook secret, then a manual Cloud Build trigger that points at cloudbuild-drift.yaml on main (Console, or gcloud builds triggers create manual). Then schedule it:
gcloud scheduler jobs create http terraform-drift-daily \
--location=europe-west6 \
--schedule="0 6 * * 1-5" \
--time-zone="Africa/Cairo" \
--uri="https://cloudbuild.googleapis.com/v1/projects/PROJECT_ID/locations/global/triggers/TRIGGER_ID:run" \
--http-method=POST \
--message-body='{"branchName":"main"}' \
--oauth-service-account-email="drift-scheduler@PROJECT_ID.iam.gserviceaccount.com"
The scheduler’s service account only needs roles/cloudbuild.builds.editor to start the trigger. If your trigger is regional, replace global in the URL with its region.
What a message looks like
Terraform drift in platform-prod:
DRIFT envs/prod/gke
FAILED envs/prod/network (exit 1)
https://console.cloud.google.com/cloud-build/builds/...
From there, open the build log, read the plan for the drifted module, and decide: bring the change into Terraform, or let the next reviewed apply put it back.
Where to go next
- More projects: keep one
_MODULESlist per repository, or one trigger per environment if the lists get long. - Fewer surprises: combine the check with an apply path that only runs from the main branch, so drift can only come from outside Terraform.
- Ownership: send each team’s modules to its own Chat space by running one trigger per team.