What Spotify actually built (and what it cost them)

Spotify shipped "Talk to Spotify" two weeks ago. A conversational AI agent that lives inside the app, understands your listening history, and talks back.

It looks simple. But trust me when I say, its nontrivial.

Spotify has spent over a decade building the infrastructure that makes this possible. They didn't bolt a chatbot onto their app. They built an entire agentic layer from the ground up.

As someone who runs a business offering this outcome as a service (AMP) this update from Spotify intrigued me for two reasons.

The cost, time and effort that went into making this (surfacing the problem that we solve)
The validation from consumers that they want to use agents directly in the applications they love & enjoy every day. Notion were another large SaaS example who adopted this trend earlier this year.

So I figured, lets do some research and investigate further.

The model layer alone is a multi-vendor orchestration problem. Spotify confirmed to TechCrunch they are running "a mix of our own AI and models from multiple providers, based on whatever is best for the task." [1] That means a router that decides in real time: is this a simple query that runs on a fine-tuned Llama? A complex reasoning problem that needs Claude? A voice interaction that needs Whisper for speech-to-text and Sonantic for text-to-speech? An audio understanding task that needs LLark, their multimodal music model? [2]

Each of those models has different latency profiles, cost structures, and failure modes. The router has to handle failover, load balancing, and context passing between completely different architectures. One provider goes down, the whole agent degrades gracefully. That failover logic alone is a full engineering team.

Then there is the retrieval layer. "When did I first listen to this song?" sounds like a casual question. Answering it requires real-time RAG across a user-specific graph of listening history, plus catalog metadata for 100 million tracks, plus podcast and audiobook indices, plus cultural knowledge about artists and albums. All of this has to be indexed, embedded, and queryable in under a few hundred milliseconds.

The embedding infrastructure alone is staggering. Every track, artist, album, and podcast needs a vector representation. Those vectors need to live in a database that can handle billions of similarity searches per day across hundreds of millions of users. That is not a side project. That is a core infrastructure investment.

Then, there is the conversation layer, which is a beast of its own I might add. Maintaining context across turns is trivial in a demo. At Spotify's scale it is a distributed systems problem. Context windows have hard limits. Users interrupt. They change topics. They reference things from three turns ago. The agent has to track state, manage memory, and resolve ambiguities like "make it more upbeat" when "it" refers to a playlist that was shaped across four previous exchanges.

The action layer is where the real complexity lives. This agent does not just chat. It executes. Queue this track. Save that song. Follow this artist. Build that playlist. Each action needs validation, sandboxing, and rollback capability. One bad intent parse and you accidentally unfollowed someone's favorite artist or deleted a playlist they'd curated for years.

Every action has to be traced, logged, and reversible. The observability stack alone, tracking every model call, every retrieval, every action execution across hundreds of millions of users, is a platform in its own right.

The voice stack adds another dimension. Sub-500 millisecond response time is the threshold where a voice interaction feels natural. Above that, it feels broken. Spotify has to run speech recognition, intent parsing, model inference, retrieval, response generation, and speech synthesis in under half a second. That requires aggressive caching, edge inference, and pre-computed embeddings for common queries.

Then there is the training pipeline. They cannot just download Llama and call it a day. Llama is a general-purpose text predictor. It does not know what drill music is. It does not understand the difference between a lo-fi playlist and an ambient sleep mix. It does not know that "The Beatles" and "Beatles" are the same band, or that "Ye" is Kanye West.

Spotify had to fine-tune their models on a proprietary corpus of music metadata, user behaviour, cultural annotations, and genre taxonomy. [2][3] They had to build evaluation frameworks that measure not just fluency, but musical accuracy. Where credability is at stake, mistakes are unacceptable.

Safety and guardrails are the final iceberg. The agent has access to a user's entire musical identity. It can surface explicit content to minors. It can hallucinate album release dates or artist biographies. It can recommend inappropriate content. Spotify needs content moderation, factual verification, and age-gating running in parallel with every response.

So what did this cost?

Spotify's R&D spend was $1.608 billion in 2024 and $1.576 billion in 2025. [4] They do not break out the AI line item, but we can triangulate from industry benchmarks. Building a multi-model conversational agent at enterprise scale runs from $100,000 to $500,000+ for the initial build, with annual team costs of $250,000 to $600,000 for a core AI engineering squad. And thats being generous when you consider the cost of a senior engineers pay package. [5][6] Timelines stretch 20 to 40+ weeks for a production multi-agent system. [5] Monthly operational costs, once live, run $1,500 to $20,000+ depending on model usage and traffic volume. [5]

That is before you factor in the data infrastructure, the embedding pipelines, the voice stack, the safety layer, and the mobile integration. Add it all up and you are looking at millions in direct cost, plus the opportunity cost of pulling engineers off core product work for the better part of a year.

Spotify generated 652,000 automated pull requests in 2024 alone. [7] Their AI coding agent, Honk, is used weekly by nearly every engineer. [8] This is a company with the engineering density, data infrastructure, and capital to make something like this work. And props to them for it

Most companies do not have that. And that is the point.

Spotify just proved the market wants conversational agents inside consumer apps. Users want to talk to their products. They want personalized responses. They want their history and preferences understood and acted on.

But the engineering effort required to build this in-house is monumental. It is not a feature. It is a platform.

Could they have done this with Llama alone? No. Llama is a text-in, text-out reasoning engine. It does not process voice. It does not understand audio. It cannot query a database of 100 million songs. It cannot execute actions in a mobile app. It is one component in a system that requires a dozen specialized subsystems. Spotify literally built their own orchestration layer from the ground up

This is exactly why we built BlueNexus. The router, the connectors, the conversation state, the safety layer, the multi-model orchestration. All of it. Out of the box.

You do not need Spotify's engineering budget to build Spotify-grade agents.

You just need the right platform. And with that, I invite you to try BlueNexusAI today.

Sources

Originally published on LinkedIn.

What Spotify actually built (and what it cost them)

Sources

Subscribe to our blog