The Database That Shouldn’t Exist

Somewhere inside Spotify’s data lake is a record of the music I listened to during a particular summer. The information already exists. It has been stored, compressed and arranged inside enormous Parquet files alongside the listening histories of hundreds of millions of other people. Yet retrieving my few hundred records quickly enough for an interactive product is still treated as a separate engineering problem.

Spotify could ask an analytical system such as BigQuery or Trino to find the records. These systems are very good at scanning large amounts of data, but they may spend seconds preparing to answer a question whose result is only a few kilobytes. Alternatively, Spotify could copy my listening history into an online database designed to return it immediately. This would be faster, but it would mean storing another version of data the company already has.

This duplication is normal. Across the technology industry, data is first collected in cheap object storage, transformed into analytical tables and then copied again into databases built for online products. There may be another copy for search, another for machine-learning features and several caches above them. Pipelines keep the copies synchronized. Monitoring systems watch the pipelines. Teams are created to operate the systems that operate the copies.

Spotify is now exploring a different question. What if the historical data does not need to move? What if it only needs an address?

The company calls its proposal Random Access Parquet, or RAP. The central idea is almost embarrassingly literal. Build an external index that records exactly where the data associated with a particular key can be found. For a Spotify user, that index might identify the relevant Parquet files and the exact rows or compressed pages containing their listening history. The application would no longer have to open files, inspect metadata, search columns and progressively narrow down the possibilities. It could ask the index where the records are and retrieve the relevant byte ranges directly from object storage.

Parquet already contains several mechanisms for avoiding unnecessary work. Statistics can show that a row group cannot contain a value. Bloom filters can indicate that a key is definitely absent. Page indexes can narrow the search to a smaller region of a file. These techniques are useful, but they still describe where the data might be. Spotify’s proposed index describes where it is.

That distinction changes the shape of the request. Instead of turning a small lookup into a miniature analytical query, RAP attempts to reduce it to one index lookup followed by a few parallel reads. Spotify also proposes changing how selected Parquet files are physically written. Records belonging to the same key can be placed together. Compression can restart at key boundaries. Fields needed by an online product can be stored together as a single object. Frequently requested values can even be included directly inside the index.

The point is not to transform Parquet into a complete transactional database. It is to make certain kinds of historical data reachable without copying them into one.

Other large technology companies have faced the same problem and generally arrived at a different answer. Netflix stores online user history and application state in systems such as Cassandra and EVCache. LinkedIn built Venice to move large derived datasets into a distributed key-value serving layer. Airbnb’s Chronon computes machine-learning features from warehouse and streaming data, then materializes the current values into an online store. Uber uses systems such as Hudi to manage mutable data in the lake and Pinot when it needs fast analytical queries.

These systems exist for good reasons. They offer predictable latency, high throughput, updates, replication and mature operational controls. A recommendation displayed on every page request should not depend on a cold object-storage lookup. A fraud score that changes continuously should not be reconstructed from a historical Parquet file. A payment system cannot replace transactional guarantees with a clever byte-range request.

Spotify is not showing that these databases are unnecessary. It is questioning how much data deserves to live inside them.

Netflix provides the clearest contrast. It has spent years developing infrastructure for moving warehouse data into online key-value systems and serving user histories through Cassandra and caches. The logic is straightforward: if a product needs the data quickly, put a prepared copy close to the product. Spotify is asking whether the long tail of historical data needs the same treatment as the records requested on every screen.

LinkedIn’s Venice and Airbnb’s Chronon make the same trade in more specialized forms. They accept duplication because recommendation results and machine-learning features must be available with predictable latency. Uber separates mutable lake storage from low-latency analytical serving. These companies are not making a mistake that Spotify has suddenly noticed. They are solving a harder online problem.

RAP is interesting because it creates another category between the warehouse and the serving database. The data can remain too cold to justify a full online copy while still being too important to leave behind a multi-second analytical query.

This is where the economics become strange. Companies have accumulated enormous amounts of historical information because object storage makes retaining it relatively cheap. AI agents and increasingly personalized products now make this history potentially useful. A future assistant might need to retrieve a forgotten playlist, an old purchase, a support conversation, a previous recommendation or an event from several years ago. The company cannot know in advance which obscure record will suddenly become relevant.

The conventional response would be to make more of the archive available through online databases. But this means paying to keep a vast long tail of rarely requested information permanently ready. Most of it may never be read. The infrastructure is provisioned not because the data is hot, but because it might one day be needed quickly.

RAP suggests another arrangement. Keep the long tail cold, but give it an address.

This may be particularly important for AI products because their information requirements are difficult to predict. Traditional applications expose a relatively small number of designed screens and predefined actions. Engineers know which records those screens require and can create the necessary serving tables. An agent can receive an open-ended question and decide that some obscure fragment of history is relevant. The potential retrieval surface is much larger than the product interface that came before it.

It would be extremely expensive to place every fact an agent might someday request into a low-latency database. It would also be wasteful. Much of the value of an agent may come precisely from its ability to reach into information that is normally too old, too specific or too infrequently accessed to justify an online copy.

The deeper issue is that the technology industry has spent years treating the boundary between the data lake and the product as fixed. The lake is for analysis. The database is for interaction. Data moves from one to the other through pipelines. Each system has its own owners, reliability requirements, access controls and operational culture.

Once this division becomes standard architecture, it begins to appear natural. The analytical system is slow at point queries, so another database is necessary. The new database contains a second copy, so synchronization is necessary. Synchronization occasionally fails, so reconciliation tooling is necessary. The pipeline must be observable, so dashboards and alerts are necessary. The copied data has its own retention and deletion requirements, so governance must now operate across several systems.

One limitation creates a workaround. The workaround creates infrastructure. The infrastructure creates an organization.

This is not necessarily bad engineering. It is how reliable systems are built under real constraints. But temporary constraints have a habit of becoming permanent institutions. Even when storage becomes faster, network access improves and file formats become more capable, the compensating architecture remains. The company continues to maintain the answer it developed for an earlier version of the problem.

Spotify’s article is interesting because it reopens the original question. Is the data lake truly unsuitable for online retrieval, or have companies been accessing it through the wrong path? Perhaps object storage was not always too slow. Perhaps the larger problem was repeatedly asking analytical engines to discover information whose physical location could have been recorded in advance.

The proposal is not entirely new. External indexes over Parquet already exist in systems such as Apache Hudi, Iceberg-related tooling and research projects like Rottnest. The database industry has spent decades building indexes that convert questions into physical locations. Spotify’s contribution is not the discovery that indexes are useful. It is the attempt to combine exact indexing, object-storage range reads and deliberately prepared Parquet layouts into a practical serving path for historical data.

There are unresolved problems. Physical indexes become stale when files are rewritten, compacted or deleted. A large company may still need a substantial distributed system simply to store and serve the indexes. A single user’s history may be spread across hundreds of files. Object-storage latency may remain too unpredictable for a core product path. The article provides an architectural argument, not production benchmarks showing that Spotify has replaced its online databases.

The absence of those numbers matters. There is no published p95 or p99 latency, no index-build cost, no demonstrated request volume and no detailed account of what happens when a file moves while an index still points at its old location. The proposal is convincing as a direction, but it should not be retold as Spotify having already turned its data lake into Bigtable.

RAP may therefore become one tier among several rather than a universal replacement. The hottest and most frequently updated records remain in caches and online databases. Interactive analytical datasets remain in systems such as Pinot or ClickHouse. Large exploratory questions continue to go through BigQuery, Trino or Spark. Historical records that are rarely requested but still need to be reachable can remain in Parquet behind an exact index.

This is less dramatic than turning the data lake into a database, but probably more useful. Infrastructure rarely disappears completely. It is pushed away from the places where its cost is no longer justified.

The broader pattern extends beyond Spotify. Modern software is full of systems built to compensate for the limitations of other systems. Data is copied because one storage layer cannot answer quickly enough. Interfaces are added because one tool cannot be trusted by ordinary users. Human review processes appear because automated systems cannot explain themselves. Every workaround may be rational, but each one can outlive the condition that produced it.

Eventually the workaround is no longer perceived as a workaround. It becomes the way serious companies do things.

Spotify is still exploring RAP, and the difficult work would be turning the clean diagram into something that survives production traffic, stale indexes, deleted data and unpredictable access patterns. But the proposal has already made one useful move. It has taken a piece of infrastructure that appears inevitable and made it questionable again.

The data was already there. The second database may have existed because nobody had written down exactly where to look.

Connected reading

• The Data Warehouse Is Becoming an Operator
• Eleven Terabytes Are Not a Story
• The Real Bottleneck Is Human Attention
• The Coding Assistant Is Becoming the Development System

AI authorship note

This essay began with Kris Haamer reading Spotify’s Random Access Parquet proposal and asking how it compared with the ways other large companies serve historical data. AI supported the comparative research, drafting, editing, the three SVG illustrations, metadata and the publishing workflow. Kris selected the argument that temporary technical constraints can harden into permanent infrastructure and approved this version for publication.