The internet connected billions of people and mobile devices, putting computers in every hand. Now, we’re in the middle of the next big technology shift, deploying millions of autonomous AI agents to work alongside employees and end users. Today, we announced new FinOps controls for Gemini Enterprise to help organizations manage project-level AI spend and eliminate token shock. But the sheer scale of the agentic era is placing new constraints at every layer of the stack, including infrastructure. AI workloads are notoriously difficult to architect, resource-intensive, and bursty, which can also lead to scaling bottlenecks and large pools of underutilized — or misutilized — compute resources.
Organizations need insights to help them extract more value from their infrastructure investments. In this blog, we outline best practices for dynamic capacity management — scheduling and utilization strategies to help you run enterprise and AI applications on a single, flexible foundation with predictable cost and performance. These capabilities are designed to augment our on-demand, Spot and committed use discount (CUD) consumption models, which provide flexible pricing and discounting for your workloads. Let’s jump in.
Here’s a quick summary
Three ways you can implement dynamic capacity management:
-
Schedule capacity for planned events. Schedule mission-critical resources (GPUs, TPUs and select VM families) ahead of planned events using calendar mode, or optimize costs for batch jobs with flexible start times using flex-start mode in Dynamic Workload Scheduler. Once you obtain the capacity, those resources are guaranteed for the specified duration.
-
Maintain service continuity by creating a fallback plan for every application. Define automated, prioritized hardware fallback lists using managed instance groups (MIGs) so your apps automatically pivot to the next approved compute option when your preferred option isn’t available.
-
Automate your entire capacity management lifecycle on a single, adaptive control plane. Google Kubernetes Engine (GKE) provides an agent-native environment to orchestrate the entire process — from fallback lists using Custom ComputeClasses, to granular hardware slicing with dynamic resource allocation, so agents can rapidly spin up in secure sandboxes and containers while it dynamically reallocating resources on the fly.
Why architectural flexibility matters
Ninety percent of enterprises want to deploy agents within the next three years, but only 17% of IT leaders feel confident their current IT setup can handle the load. Because these workloads have unique performance needs, organizations are racing to adopt specialized infrastructure, including accelerators (GPUs, TPUs) and CPUs with customized compute, memory, and storage ratios. However, agents also require access to enterprise applications and databases — often at a volume and scale that vastly exceeds typical human usage. Handling the intense demands of both agents and the applications they interact with requires a dynamic infrastructure. Infrastructure teams can leverage custom-designed processors like Google’s Axion to meet these needs, but hardware isn’t a complete solution. They also need ways to use that infrastructure wisely, solving execution inefficiencies to enable more flexibility across the stack.
How to overcome infrastructure constraints
Achieving this kind of flexibility requires a two-pronged approach: securing resources for the demand you can predict, and building automation to respond to the demand you can’t. Combining the two, you can preschedule capacity for planned events and your infrastructure can adapt to unexpected changes without manual intervention.
1. Schedule capacity for planned events
You can secure mission-critical capacity ahead of scheduled milestones, offline training, or anticipated demand surges using Dynamic Workload Scheduler. By scheduling the resources you need up front, you optimize your spend and ensure you get access to the compute resources you need. Dynamic Workload Scheduler supports hardware accelerators (TPUs and GPUs) and select CPUs with two distinct modes:
-
Flex-start mode: Use this for latency-tolerant workloads like batch processing, model training, or offline fine-tuning. Instead of requiring resources immediately, you submit a defined duration request and the system intelligently queues your job, provisioning the resources as soon as capacity becomes available. This maximizes cost-efficiency and drastically improves your ability to obtain high-demand accelerators.
-
Calendar mode: Use this for mission-critical, time-bound events like a major product launch, a scheduled migration, or a seasonal traffic surge. By specifying the exact start and end dates of your event, you create a future reservation. This guarantees the requested capacity will be available when the event begins.
2. Maintain service continuity by creating a fallback plan for every application
Not every spike in traffic is predictable. You also need to plan for unexpected traffic from, say, a breaking news cycle or a sudden market shift that drives a surge in user activity. To help your services get the resources they need without interruption, you need a fallback plan — an automated, prioritized sequence of acceptable hardware configurations. This strategy:
-
Decouples your workloads from a single VM shape, size, or configuration. This allows them to run without manual intervention if your preferred option is unavailable
-
Allows you to execute a progressive tech refresh by adopting the newest VM generations as your primary choice while keeping older generations as an automatic fallback option.
If you run non-containerized workloads on Google Compute Engine, you can dynamically manage capacity with instance flexibility in managed instance groups (MIGs) and bulk VM creation. Instance flexibility lets you specify multiple machine types for your VM instances rather than being limited to a single machine type.
How it works: If your preferred machine type is temporarily unavailable, the MIG automatically provisions a compatible alternative from your list based on real-time capacity. When combined with location flexibility — by specifying multiple zones your MIGs can search within a region — you can drastically improve your provisioning success rate. If your MIGs use Spot VMs, Compute Engine automatically integrates with Spot capacity signals to prioritize machine types that offer longer estimated uptimes and lower risk of pre-emption.
You can also extend instance flexibility to your block storage layer by setting baseline disk defaults and configuring disk overrides so your storage adapts when a VM falls back to a different machine type.
How it works: Most of the time you can simply rely on our default options, omitting ‘disk type’ from the instance template entirely. However, for data disks that will outlive their associated VMs, it’s possible to enable a fast, durable Hyperdisk across multiple VM generations.
While Compute Engine provides instance flexibility for organizations working with virtual machines, GKE goes a step further and automates the entire capacity lifecycle from a single control plane. With GKE custom ComputeClasses, platform teams can design multi-dimensional fallback lists, automatically combine different VM machine families, sizes, and ratios, scale across multiple zones, and shift between on-demand and Spot VMs. By using Dynamic Workload Scheduler as a capacity target, and custom ComputeClasses to define the policy and priority, you can fully automate the capacity management lifecycle.
How it works: Once you’ve set up ComputeClasses, GKE automatically detects when a preferred node configuration is unavailable and falls back to your pre-approved alternative options in order of priority. When active migration is enabled, GKE gracefully migrates workloads back to higher-priority node configurations as capacity becomes available. For short-lived disks such as boot disks, GKE dynamically picks the right defaults based on the instance family. However, for long-term disks that will outlive the VM, you can use Hyperdisk.
Another GKE feature, dynamic resource allocation, helps eliminate wasteful, all-or-nothing hardware assignments by letting developers define advanced rules that dictate how resources are consumed.
How it works: Instead of claiming an entire GPU or TPU, your application specifies its exact parameters — such as total memory or number of cores — and the system allocates the perfect slice of hardware, helping to maximize utilization and reduce costs.
Take the next step toward dynamic infrastructure
Scaling AI shouldn’t mean linearly scaling your infrastructure budget or accumulating more tech debt. As these examples show, the right tools can help you overcome constraints and dramatically alter the value you get from your compute investments. Here are three steps to get started:
-
Audit your workloads for immediate cost-savings: Identify any applications currently tightly coupled to a single VM family, machine type, or availability zone, and map out viable alternative hardware shapes. Look beyond your existing configurations to evaluate new compute options that might better serve or act as alternatives based on your workload-level objectives. Then use Compute Engine MIGs, bulk VM creation or GKE Custom ComputeClasses to adopt them automatically, integrating them into your fallback lists.
-
Commit to a minimum spend for deeply discounted prices: Receive automatic discounts for sustained use, or up to 63% off when you sign up for Compute flexible committed use discounts, where your discount is tied to the resources you use regardless of the specific machine type or location.
-
Engage your account team: Reach out to your Google Cloud account team to craft a tailored capacity management strategy and configure your automated fallback lists.