case study / Modeso · 2023–now
Terraform governance across 16+ GCP projects
Terraform moved off laptops into one reviewed pipeline, a daily drift check that only speaks when something changed, and a large state split into independent domains.
- Client
- Modeso
- Period
- 2023–now
- Role
- DevOps Engineer
- Result
- one reviewed path, daily drift checks
- Stack
- GCP, Terraform, Cloud Build, Cloud Scheduler, Secret Manager, PAM/IAP
In short
- Moved Terraform off laptops and into Cloud Build, with apply allowed only from the protected main branch.
- Removed the need for permanent local infrastructure credentials; privileged work uses time-bound access.
- Added a daily drift check across the infrastructure repositories that posts to Google Chat only when something changed outside Terraform.
- Split one oversized Terraform state into four domains per environment, with zero-change plans before delivery resumed.
Context
At Modeso I am one of the DevOps engineers running a portfolio of enterprise products for Swiss clients, spread over more than 16 active GCP projects. With one project, Terraform is an infrastructure question. With sixteen, it becomes a governance question: how changes are executed, how drift is noticed, and how state stays manageable.
Constraints
- Terraform could run from different laptops, with different versions and credentials.
- A manual console change could stay unnoticed until the next release picked it up.
- One project had grown a single large state per environment, so plans were slow and a mistake had a wide blast radius.
- Delivery could not stop while any of this changed.
Decisions
One execution path for Terraform
Cloud Build became the only way Terraform reaches the cloud. The pipeline takes the module, environment, Terraform image and operation as substitutions, initializes the right remote backend, and runs terraform plan or terraform apply. Apply only runs from the protected main branch, under a dedicated Cloud Build service account.
That gave the team one Terraform version, central logs for every plan and apply, and review before any change. It also removed the need for permanent local infrastructure credentials. Privileged work moved to time-bound access through Google Privileged Access Manager, with IAP as the private path to protected machines.
A pipeline alone is not a security boundary. It only works together with the IAM model, branch protection, state bucket permissions and the privileged-access process, so all four were changed together.
Drift detection that stays quiet when nothing is wrong
Central applies do not stop every manual change: incidents, emergency fixes and provider behavior still create drift. The drift check works like this:
- Cloud Scheduler calls a Cloud Build trigger with a dedicated service account.
- Cloud Build checks out the infrastructure repository and loops over each domain and environment.
- Each module runs
terraform plan -detailed-exitcode:0means no change,2means drift, anything else is a failed check. - Only drift or failures send a Google Chat message, listing the affected modules with a link to the build.
The webhook URL lives in Secret Manager and is injected only into the notification step. Because clean runs stay silent, the alerts are read instead of muted. The setup is written up step by step in Terraform drift detection with Cloud Build and Cloud Scheduler.
Splitting a large state by operational domain
For one enterprise project, development and production each had a single large state. I split every environment into four independently managed domains:
- GKE cluster and supporting compute
- Cloud SQL
- Load balancing and certificates
- Monitoring and alerting
Each domain got its own backend prefix. Resources were mapped to their new addresses and moved or imported without being recreated, and each domain had to show a zero-change plan before normal delivery resumed. The split followed operational boundaries on purpose: splitting only to make files smaller creates cross-state dependencies that make releases harder.
Results
- One reviewed, logged path for Terraform across 16+ GCP projects, with a single Terraform version and apply only from the main branch.
- No permanent local infrastructure credentials for routine delivery.
- Drift found daily instead of during the next release.
- Smaller, faster plans and a smaller blast radius on the largest project, with less state-lock contention.
Related: the cost work at Modeso is its own case study, Non-production environments that run only when needed.