There’s a lot more to AI infrastructure than just buying accelerators. Indeed, the infrastructure mandate for enterprise AI has shifted from acquiring accelerators to keeping them fully utilized. Next to these expensive accelerators, storage accounts for only a small sliver of total AI spend. However, having the wrong storage in place can put the rest of your committed compute capital at risk.
To optimize your accelerators’ TCO, you need to keep them fully occupied. Very often, the challenge here is data I/O. If accelerators sit waiting to read or write data, you’re wasting money. Across the industry, insufficient storage throughput has become a primary impediment to AI success, starving models of the data they need. And the problem is only going to get worse. Gartner predicts enterprises will deploy over 150,000 agents in the next few years, so the I/O bottleneck and the challenge of orchestrating storage for agentic fleets will only compound.
Organizations are eager to move their AI initiatives forward, but preventing idle compute requires a purpose-built approach to data I/O. With the demands of AI, standard enterprise storage — even a system optimized for fast analytics — will inevitably suffer from bottlenecks that leave compute resources twiddling their proverbial thumbs.
AI is not a monolithic workload
The most common architectural misstep we see is treating AI as a single, general-purpose workload. In reality, the physical movement of data transforms across the AI lifecycle, creating distinct I/O profiles based on how models consume and generate information:
-
In model training, clusters read datasets at high throughput, but pause regularly to write checkpoints to record their state in event of a failure. When thousands of GPUs attempt to save this data in a single, simultaneous burst, standard storage architectures stall. That’s especially true of traditional cloud storage, which is engineered for durability. Writing checkpoints triggers thousands of administrative database transactions to safely record the files, bogging down the system, choking the network, and freezing the entire GPU cluster until the save is complete. Furthermore, saving the data is only one of the problems: If a server fails, slow storage turns a routine checkpoint recovery into a multi-hour delay, leaving expensive accelerators sitting completely idle.
-
Inference, conversely, is real-time and latency-sensitive. As AI services scale to meet user demand, inference clusters constantly load and unload petabyte-scale model repositories quickly. In this case, a slow storage system can leave GPUs in a “cold start” as they wait for data. Also, many models now process context windows that exceed 100,000 tokens, filling up accelerator memory. Without a high-speed external cache to offload this context, when a user asks the next question, the accelerator must recalculate previous inputs, wasting compute on redundant operations.
Industry-leading performance across object and file storage
To solve these I/O challenges and keep expensive accelerators busy, organizations need compute, networking, and storage to be deeply integrated.
As industry analysts have noted, enterprise customers gain a unique performance advantage because Google Cloud runs on the exact same storage foundation as Google’s own consumer products. This foundation is built on Colossus—the planet-scale distributed file system powering Google Search, YouTube, and Gemini. Because this shared infrastructure supports billions of users daily, its reliability is continuously proven against the extreme traffic and hardware demands of Google’s global consumer services. As a result, today, nine out of the world’s ten leading AI frontier model labs run on Google Cloud infrastructure because we deliver the throughput and performance they require to innovate. In fact, we spent the last two years co-designing our storage architecture directly with these frontier labs as well as with Google’s own foundation model teams.
For large-scale training on object storage, our architecture ensures workloads don’t have to wait on data, and successfully absorbs peak bandwidth demands. For real-time inference, enterprise platform companies like Salesforce AI Research leverage our architecture to serve complex agentic inference workloads with up to 40% fewer GPUs. We deliver these outcomes through two distinct, purpose-built storage solutions:
-
Cloud Storage Rapid: For organizations that have standardized on cloud-native object storage, Cloud Storage Rapid provides Colossus scale and performance directly to accelerators, so they can use the data stored on object storage in their AI workflows. This eliminates the need to migrate data into separate, specialized storage silos to achieve the performance that AI workloads require. A single Rapid bucket pushes the boundaries of object storage, delivering up to 15 TB/s of throughput and 20 million queries per second with sub-millisecond latency. This makes it a strong fit for streaming large training datasets and recovering checkpoints without bottlenecking the network.
For bursty workloads such as model loading across multiple locations, Rapid Cache acts as a read accelerator that co-locates active data directly with TPU and GPU clusters. With no code changes, Rapid Cache delivers up to 2.5 TB/s of aggregate read throughput, taking idle cold-start times from hours to minutes.
-
Google Cloud Managed Lustre: For demanding AI and HPC workloads that need a parallel file system, there’s Google Cloud Managed Lustre. Built in partnership with DDN, Managed Lustre provides up to 10 TB/s of throughput per instance, absorbing the intense read-write bursts of synchronous checkpointing and reinforcement learning, so that your expensive accelerators remain fully saturated.
To help organizations scale their workloads without breaking budgets, Managed Lustre’s new Dynamic Tier delivers more Lustre for less by consolidating hot and cold data into a single namespace, eliminating traditional caching performance cliffs. Finally, Managed Lustre Cloud Storage Mirror reduces the downtime and operational friction inherent in manual data transfers. Instead of you having to write custom scripts that leave expensive compute clusters waiting on files, it automatically synchronizes datasets in the background so they are ready before training begins.
Proven in production at scale
This purpose-built storage foundation is already powering customers’ most demanding AI workloads, helping them extract maximum return on investment from their compute resources.
-
Thinking Machines Lab utilizes Cloud Storage Rapid Cache to stabilize read throughput peaks above 1.8 TB/s and achieve an 89% cache hit ratio, completely eradicating rate-limiting surge errors across their multi-region architecture.
-
Shopify deployed 50+ unique Rapid Caches across their worldwide footprint, achieving over 2 TB/s peak throughput and up to a 90% latency reduction from their baseline to meet the data demands of their growing compute scale.
-
Salesforce AI Research uses Managed Lustre as a high-throughput, low-latency external cache to keep their GPUs fully saturated, driving substantial performance gains for LLM inference over previous hardware generations.
-
Atlassian adopted Managed Lustre on Google Kubernetes Engine (GKE) to turn preemption interruptions into quick five-minute restores, delivering nearly 5x higher throughput and boosting their GPU utilization by 70% compared to NVMe-based systems.
-
Harmonic relies on Managed Lustre to train mathematical reasoning models, eliminating 50% of experiment interruptions, cutting startup times in half, and increasing data retrieval speeds by 15x.
The path forward
When advising AI Infrastructure teams designing their stack today, our guidance is straightforward. First, evaluate your token economics rather than just legacy storage metrics. If your chips are waiting on data, optimizing storage is a cost-effective and efficient remedy. Second, avoid creating separate copies of data for every new workload; keep your primary data in a unified foundation and move it to the right tier as needed. Finally, while AI performance grabs the headlines, you can’t create your AI strategy on infrastructure built for a different era. At the same time, AI infrastructure needs enterprise-grade reliability, built-in security, and automated cost discipline for optimized TCO.
The future of enterprise AI belongs to organizations that can turn their data into action. Google Cloud enables you to match the speed of your storage to the speed of your compute, so that you maximize your compute investments.