Are you paying for bored GPUs? How AI starvation eats your budget

Smoke and time don’t make mediocre meat better

I love cooking outdoors. Fire, smoke, time. What I’ve learned doing it: you can buy the best kamado or smoker on the market, but if the butcher can’t deliver, you’re standing there with nothing. And if he does deliver, you still want to know what exactly he’s handing you. Start with mediocre meat, and you end up with mediocre meat that’s spent fourteen hours cooking. Smoke, temperature, and time make a good cut of meat better. They don’t make a mediocre cut good.

I’ve written before about that second part, the quality of what you feed in. That’s where most AI projects run aground during testing. This time, it’s about the supply chain.

Because that expensive GPU is sitting there waiting, and that costs you money while nothing happens. We call that AI starvation. You scale up to a pricier tier while the actual bottleneck was never the compute. The question that interests me is a different one: how did we get to a point where nobody even noticed?

The database that dropped out of sight

Ask yourself one question: who’s managing your database right now? Not which product, not which cloud. Who.

At most organizations we talk to, that question doesn’t produce a name. It produces a service. “That runs on RDS.” “That’s Cloud SQL.” “That’s a container in our Kubernetes cluster.” And that’s usually where it stops, because the word “managed” is right there in the name, so surely it’s handled. At least, that’s what we assume.

AWS, Google, and Azure do exactly what they promise. They keep your database up and running, and they patch the engine. That’s management, and nothing more than that. It isn’t optimization.

You see the same pattern with backups. There’s an automatic snapshot feature running, but that’s a retention window of at most 35 days, in the same account and the same region. Under the shared-responsibility model, you’re the one who defines the backup strategy, and you’re the one who tests whether a restore actually works. Harry wrote about exactly this: everyone makes backups, almost nobody restores them.

You see that same pattern everywhere. Nobody at your cloud provider checks whether your indexes still match how your platform is actually being queried today. Nobody checks whether your data model has kept pace with the four applications that got added since it was built. And nobody asks whether your storage layer can deliver the throughput a training job or a RAG pipeline actually needs.

That work never disappeared. It just fell off the task list the moment the database vanished behind a console screen.

Bored GPUs are a symptom

What we find when we get called into one of these environments is almost never dramatic. It’s the boring side of the trade.

A dataloader firing off a separate query per record instead of fetching in batches. A vector index still sitting on the default parameters from the day someone switched it on. Retrieval going over a network connection to storage in a different region, because that’s just how it happened to be set up. Statistics that were never refreshed again, so the query planner has been making the wrong calls for months.

Each of those costs milliseconds. Multiply that by a few million retrieval actions, and your GPU spends a considerable chunk of the day doing nothing. Your dashboard mostly shows that the GPU is “active.” What it rarely tells you is that, meanwhile, it’s just waiting on your data layer.

How to spot it in your own environment

Five signals you can check without a deep investigation:

• Your GPU monitoring reports high usage, while throughput lags behind what you’d expect.
• Fetching the data takes longer than the inference or the training step itself.
• Your logs show a lot of small, individual queries in places where batches should be.
• Your latency spikes the moment retrieval crosses regions or zones.
• Query plans, statistics, and indexes haven’t been reviewed since go-live.

If you recognize two or more of these, you’re currently paying for compute that’s sitting idle.

One nuance here. AI starvation doesn’t always originate in the database itself. It can just as easily sit in your object storage, your network, your vector store configuration, or how your dataloader was built. What we see is that it tends to surface in the database, because that’s where most of the layers come together. So what you need is someone who looks at the whole path, not just one piece of it.

The real problem is ownership

The temptation now is to say: fine, let’s tune that database. We can do that, and we do. But then you’ll be back in the same spot a year from now.

Because underneath it all, the database no longer has an owner. The data team looks at models and pipelines. The platform team looks at Kubernetes and deployments. The cloud provider looks at uptime. And the database sits somewhere in between, belonging to everyone and therefore to nobody, while every AI initiative in your organization leans on it.

A foundation needs someone who understands it and is responsible for it. That’s exactly what our Database Platform Engineers do. They look at your data model, your indexing, your storage layer, and your retrieval patterns, whether that’s running on your own server, in a container, or on a fully managed hyperscaler service.

Because managed doesn’t mean optimized. And your AI is only as smart as your database.

Where do you start?

Burning through GPU-hour budget right now while the results are underwhelming? Run through the five signals above first, before you go shopping for a different model.

Can’t quite figure it out yourself? That’s what our AI Data Readiness QuickScan is for. Within a few days, you’ll know where the bottlenecks are and what to tackle first. No thick report, just a clear picture.

And if you just want to talk through that question from earlier (who’s actually managing your database right now) feel free to call or email. Even if you’re not a client yet.