The Fastest Machines in History Are Stuck Waiting

I keep thinking about something I heard on the Dwarkesh podcast: GPUs spend a surprising amount of time waiting for other work to finish. The exact number may not have been 15 percent, and it depends on whether we are talking about memory bandwidth, pipeline utilization, or an entire company’s GPU fleet. But the larger idea is more interesting than the number. We have built some of the fastest machines in history, placed thousands of them inside billion-dollar data centers, and much of their performance is still determined by how long they have to wait for everything around them.

A GPU can be ready to perform trillions of calculations, but the model weights may still be moving from memory. Another machine may not have completed its part of the computation. Data may be travelling between racks, or one slower stage may be holding up the entire pipeline. The chip itself is incredibly fast, yet it is constrained by the surrounding system. We tend to imagine the AI race as a competition to acquire more intelligence and more computing power, when a large part of the problem is simply keeping the existing hardware occupied.

This feels like the same coordination problem that appears everywhere else. A customer waits while support contacts another department. A patient waits for information to move between clinics. A designer waits for engineering, engineering waits for product, and product waits for legal. Every individual component may be capable, but the complete system still moves slowly because the handoffs are badly designed. A GPU waiting for another GPU is basically experiencing organizational bureaucracy.

This is why the OpenAI article about GPT-5.6 is more interesting than another benchmark chart. Much of the progress came from improving the machinery around the model: routing requests more effectively, loading fewer unnecessary tools, preserving prompt caches, rewriting GPU kernels, improving speculative decoding, and preventing expensive hardware from sitting idle. The model was even used to help optimize parts of the infrastructure that runs the model. AI is beginning to reorganize its own factory floor.

That creates a recursive loop. Better models can inspect code, discover bottlenecks, test new scheduling strategies, and optimize the systems that serve them. Better infrastructure then makes those models cheaper and faster to run, which allows more experiments and creates better models. The next frontier may therefore come from wasting less intelligence rather than only producing more of it.

I also think this is an interaction design problem. Interaction designers study where people hesitate, which information is missing, what blocks the next action, and where momentum disappears. The same questions can be asked about computational systems. Where is the GPU idle? Which operation is blocking the pipeline? What information is being moved unnecessarily? Which dependency could be removed, predicted, cached, or handled in parallel?

The interface is no longer only between a person and a screen. Interfaces now exist between models, tools, memory, networks, databases, agents, and hardware. Every one of those boundaries creates the possibility of latency. As AI systems become more agentic, they may repeatedly load unnecessary context, call too many tools, wait for sequential operations, or send work through a chain of models that could have been avoided. The model may be intelligent while the workflow around it behaves like a badly managed office.

We often assume that the company with the most GPUs will win, but perhaps the winner will be the company that wastes the least computation. A smaller cluster with better scheduling may outperform a larger one with poor utilization. A smaller model with the right context and tools may outperform a larger model trapped inside a messy system. Intelligence is becoming cheaper, but coordination remains expensive.

The scarce resource may not be computation itself. It may be uninterrupted useful work. The fastest machines in history are already here. The next breakthrough may come from figuring out how to stop making them wait.

Connected reading

• The Product Happens Between the Screens
• AI Makes Organizational Mess Executable
• The Real Bottleneck Is Human Attention
• The Coding Assistant Is Becoming the Development System

AI authorship note

This essay began with Kris Haamer remembering a Dwarkesh Podcast discussion about GPUs spending significant time waiting for memory, communication or other stages of a distributed workload. Kris connected that technical bottleneck to organizational handoffs and interaction design, then used OpenAI’s GPT-5.6 efficiency work as the current example. AI helped verify the podcast context, research the infrastructure concepts, shape the argument, edit the article and prepare the Raw Drafts publishing files. Kris supplied the original observation, framing and final editorial direction.