case study / Modeso · 2023–now
Non-production environments that run only when needed
A month of billing data turned into weekday schedules and on-demand start and stop for test environments, cutting non-production spend by 66.7% on one project.
- Client
- Modeso
- Period
- 2023–now
- Role
- DevOps Engineer
- Result
- 66.7% lower spend on one project
- Stack
- GCP, Cloud SQL, Cloud Run, GKE, Argo CD
In short
- Analysed a month of billing exports for one project with testing, UAT and performance environments: CHF 354.05 per month, mostly Cloud SQL, Cloud Run and networking.
- Put UAT on a weekday schedule and made testing and performance start on demand.
- Measured result: 66.7% lower non-production spend on that project.
- On two larger projects, GKE node pools and Cloud SQL scale down overnight and come back at the size they had, and developers start only the environment they need.
- Planned shutdowns do not page the on-call engineer.
Context
At Modeso, non-production environments ran around the clock whether anyone used them or not. Nobody had decided that; it was simply the default. Changing it meant changing how the environments are operated, not just resizing a few machines once.
Constraints
- Developers, QA and business users must be able to get an environment without waiting for DevOps.
- Developers should not need Cloud SQL or Cloud Run admin rights to start one.
- A planned shutdown must not look like an outage to monitoring.
- After a shutdown, an environment has to come back at its real size, not a guessed one.
- On some projects, several environments share one cluster and its databases.
Decisions
Start from the bill
For the three-environment project, a month of billing exports gave the baseline: CHF 354.05 per month, concentrated in Cloud SQL, Cloud Run and networking. The plan targeted CHF 118.03 per month, a saving of CHF 236.02. Networking was left in the estimate unchanged, so the target stayed conservative until load balancer, NAT and static egress needs could be assessed separately.
Schedules where usage is predictable, on demand where it is not
- UAT runs on a weekday schedule.
- Testing and performance start and stop on demand, with a safety shutdown for environments someone forgot.
- While an environment is off, Cloud Run is scaled to zero and Cloud SQL is stopped.
- The start and stop workflows run under dedicated identities, so a developer can start an environment without holding Cloud SQL or Cloud Run admin rights.
- A monitored rightsizing pilot for the testing database ran alongside.
Scale down, and remember the size
On two larger projects, Kubernetes and database capacity did not need to run at full size overnight. The evening run records the current size of each selected GKE node pool before scaling it to zero, stops the selected Cloud SQL instances, and pauses the alerts the shutdown would trigger. The morning run reads the last successful scale-down, restores each node pool to its recorded size, starts only the databases that were stopped, waits for the platform to settle, and turns the alerts back on.
Restoring the recorded size matters: a hard-coded morning capacity slowly drifts away from what the environment really needs.
One environment at a time
One project has development, acceptance, patch, staging and testing environments on shared cluster capacity. Scale-up there is requested per environment:
- The workflow checks the requested environment and exits if it is already running.
- It counts the environments currently active.
- Application and shared data-node capacity is calculated from that demand, with a safe upper limit.
- Shared database compute starts only with the first environment.
- Stateful workloads come back in a controlled order.
- Only the Argo CD applications for the requested namespace are synced.
Argo CD’s app-of-apps pattern makes the last step possible. Environment value files declare the applications, namespaces and Helm values, so capacity starts first, stateful components become ready, and Argo CD then reconciles only what was asked for.
Results
- 66.7% lower non-production spend on the three-environment project, measured after rollout against the CHF 354.05 baseline.
- Developers and business users start the environment they need themselves, without admin rights and without paying for every environment all the time.
- Environments return at their real size after a shutdown.
- Scheduled shutdowns no longer page the on-call engineer. I cover about two weeks a month of that 24/7 rotation, with alerts routed through PagerDuty and Opsgenie.