Outcome-Driven Data Architecture: A Spotify Listener Personality Case Study

Spotify creates for you a "Listening Personality" that illustrates your personal way of listening to music. Typically it measures indicators like Familiarity/Exploration or Timelessness/Newness and attaches a profile to you. Because of my curiosity about which Spotify Listening Personality I have, I started a data project that evolved into a broader exploration of data system design. This project is a data pipeline in which I demonstrate how daily listening events can be transformed into meaningful behavioral signals. It is implemented with Apache Airflow running in Docker for orchestration and Google Cloud Platform services such as BigQuery storage and analytics. This architecture enables reproducible local development, containerized and stateless execution, and a clear separation of concerns between ingestion, orchestration, and analytics.

Through this project, I explain how everyone can find out more about their personal musical tastes, but, even more important, I highlight a common pitfall in data engineering: starting with technology choices rather than product goals. The talk makes the case for clarifying the desired outcome first and letting it drive architectural and tooling decisions, resulting in simpler, more maintainable, and more cloud-native data pipelines. I, myself, focused on tooling first when I started the project, but after that I corrected that decision, and the project turned out to be a successful one in the end. I am gonna explain to you my initially wrong architectural decisions and what I changed afterwards in order for you not to repeat my mistakes.