蜜芽传媒

Orbit is our pilot for a music discovery service that lets you find your next favourite song by exploring recently played tracks from 蜜芽传媒 Introducing 鈥� the 蜜芽传媒鈥檚 platform for unsigned and undiscovered music. Every track is a crossroads where you have to decide with your ears: do you want to find out what you鈥檙e listening to, or do you want to try a related track? You can quickly move through the spectrum of musical genres, discovering what interests you and moving past what doesn鈥檛. There鈥檚 no AI and no recommendation algorithms 鈥� you decide.

This article looks at how we organise music tracks behind the scenes to allow this kind of navigation. The other article in this series details the associated UX challenges.

A provocation

During a 6-month research project on young people and their music, podcast, and audio habits we found that they felt that music discovery was not intentional anymore: it just happened to them through algorithmic recommendations or, as they browse social media. As a response, we showed them a prototype where music played with no information 鈥� no image, no metadata 鈥� and they had to make choices based on what they heard using a joystick to go from track to track. It was a blunt instrument designed to get a reaction.

The reaction was very positive! Participants liked that it required them to trust their ears and pushed them out of their comfort zone. Their concern, however, was that it felt too random and they couldn鈥檛 quite get to something they liked reliably. For that prototype, we used 50 manually cut samples from the 蜜芽传媒 Introducing database manually graded on them two aspects: how danceable the track is and how much singing there is in it. The directions weren鈥檛 labelled on purpose but it only made sense if you knew how it worked behind the scenes: it was hard to tell what each direction 鈥榤eant鈥� just by listening because the organisation of the tracks was not granular enough.

For Orbit, we wanted to add more intentionality to the design: you should be able to go in a particular direction and feel it progressively get closer to your goal. More importantly, you should be able to feel it just from what you鈥檙e hearing. You can read more about the design work we鈥檝e done in our other article.

In short, we needed a better way to organise the music to give sense of direction to the navigation that was also scalable to thousands of tracks.

Breaking down genres

Music is very often organised by genre 鈥� record shops have been organised this way for a long time. However, young people showed us that genres are less relevant to them today and artists seem to sit across genres more comfortably. Additionally, Spotify suggests that genres are now irrelevant to fans while organising their catalogue into . How many genres exist between well-known and well-understood genres like 鈥渞ock鈥� and 鈥渇unk鈥�? How would you decide?

Genres remain a useful shorthand to describe multiple aspects of a song. If someone says a track is 鈥榬ock-y鈥� you can easily imagine that it has electric guitars (possibly distorted) and maybe sounds quite dark and aggressive. If someone says 鈥榝olk-y鈥� you can see acoustic guitars and a sparser arrangement, something softer and more relaxed.

What if instead of breaking things down by genre, we could break down the genres themselves into component parts to start establishing a finer web of relationships between tracks? This way we could tell how close or far they are from each other sonically, exposing the complex overlap of genres.

Musical similarity

In his , Paul Arzelier shows how we can use mathematical distance between a series of musical descriptors to make more coherent playlists. The shorter the distance, the less difference there is between the tracks. Paul uses , his own open-source library, to extract several  and  features from an audio file and combines them to calculate the distance.

Though this was very promising, we had another hurdle to face: we didn鈥檛 want to make linear playlists but a multi-dimensional exploration tool with a feeling of direction. We wanted users to move based on what they heard: they have to feel the music become more or less 鈥榓ggressive鈥� or 鈥榟appy鈥� in a way they understood so they could follow their ears.

To get these more 鈥榟uman鈥� descriptors we used an open-source library called  from the University of Barcelona. It provides machine learning models to extrapolate musical information from audio files (previously used in another project to analyse the Introducing database) such as danceability but also how aggressive, happy, sad or relaxed a track sounds.

With so many descriptors to choose from, picking only two as we鈥檇 done in the first prototype seemed as reductive as mashing them all into a single indicator. To strike a balance and still distribute the tracks on a 2D plane, we used a dimensionality reduction technique popular in data visualisation: a Principal Component Analysis (PCA).

Visualising connections

A PCA allows you to condense the most important information of a dataset by finding similarity between each data point based on their associated descriptors. In doing so, it compresses many of the descriptors into new 鈥榩rincipal components鈥� which can be used as axes for a visual representation. As a result, each axis cannot be labelled easily as they now represent (鈥榚xplain鈥�) many features at once.

We could pour all the descriptors we assembled from Essentia and Bliss into the PCA and obtain two axes to visualise our dataset. However, we needed a way to make sure that tracks were not just mathematically linked, but that their position made intuitive sense to a human going from track to track. Because the PCA looks for the strongest differences and decides how similar two tracks are based on this, the quality of these relationships entirely depends on the quality of the descriptors: do you have too many, the wrong ones or are they even accurate enough to group and separate tracks effectively?

So to make Orbit, we first built a tool that could change which features were going into the PCA and showed us the resulting plot. It also allowed us to play the tracks by clicking on points in the plot, meaning we could experience the similarity between the tracks with our own ears.

This method had a lot of potential but clearly we needed to fine-tune which descriptors went into the PCA to get a more successful organisation.

Rebuilding genres

To know whether our new organisation was successful, we re-introduced genres into the plot by colour-coding each point using data extracted with Essentia. Now we could see if tracks from a similar genre were still roughly grouped together 鈥� because genres still represent the 鈥榗ompressed鈥� version of many descriptors 鈥� and we could see them appear as areas overlapping with each other, like a Venn diagram with interstitial pockets of micro-genres.

We reduced our 30+ descriptors to just eight 鈥榟uman鈥� ones: aggressive, danceable, electronic, happy, instrumental, party, relaxed, sad. Each track being rated from -1 (not at all) to 1 (very) on each of those. We tried various combinations until we could see some separation between opposed genres (e.g. classical vs dance) but see similarity between others (e.g.funk and hip-hop) while confirming by listening.

The final result makes sense to a computer but should also make sense to a human as they listen. With this list of adjectives, we can now describe tracks across genres and the PCA can position them into space.

Building the star field

The PCA gives us x/y coordinates for each track and we can use the distance between these points to determine their musical similarity 鈥� much like in Paul鈥檚 paper, but in two dimensions and with different indicators. However, for every single track, we need to establish which tracks they鈥檙e connected to.

We used a  to find the closest neighbours for each point, regardless of distance. If a point is an outlier, the distance between two neighbouring tracks may be huge (and therefore their similarity low), but this is actually something we value to create doorways to new different genres. The resulting plot looked a lot like constellations in the night sky, so we dubbed the graph the 鈥榮tar field鈥�.

At this point, we have all the data necessary to build a system that starts with a track and allows you to intuitively navigate based on the features we鈥檝e chosen, at scale. The result is not perfect, but it鈥檚 all we needed to run our pilot with reasonable confidence. The last challenge is to automatically cut representative samples of the tracks that let users make quick decisions.

Audio thumbnails

Our first prototype used manually edited short snippets 鈥� less than 30 seconds that we thought were representative 鈥� to let users quickly decide about what they鈥檙e hearing. In our testing, users only needed a few seconds to choose between listening further or skipping. However, we need to do the artists justice with their work and choosing the 鈥榳rong鈥� sample would be unfair to them. Ideally, the artists themselves would select the most representative part of their track, however this wasn鈥檛 practical for the pilot as we鈥檙e re-using an existing infrastructure.

The process of finding and cutting that snippet is a well-known problem called audio thumbnailing. The  is to find which part of the song is the chorus and use that, as it鈥檚 usually the most engaging part of the song. Traditionally we鈥檇 do this by finding the most 鈥榚nergetic鈥� part of the track 鈥� and the 蜜芽传媒 even has an  to do this. However, this method did not prove very accurate and didn鈥檛 scale well across genres in our testing.

Fortunately, recent advances in machine learning have made song structure analysis easier and we can now more reliably detect the chorus. We used an open-source library called  to do this with . For songs that don鈥檛 have a chorus or where a chorus cannot be found (some electronic or folk music, for example), we trim any silence from the start and cut the first 20 seconds of the track.

Putting it all together

Consequently, we created an automated process using machine learning to cut and organise the tracks in an intuitive way. It facilitates organic discovery by reinforcing a sense of direction during exploration.

It鈥檚 notable that we鈥檙e doing this directly by looking at the track audio and no other external metadata. It doesn鈥檛 matter who listened to it or how much 鈥� it鈥檚 organised in relation to the other tracks in the dataset and how they sound.

By breaking down genres into a set of observable adjectives and rebuilding them on a 2D plane, we鈥檝e created a more flexible and organic layout. Because ease of exploration was our priority, we found a different way to organise and discover music.

Orbit puts the users back in the driving seat, getting them to make conscious decisions about what they want to listen to rather than just taking what an algorithm feeds them. In doing so, it restores the feeling of ownership over musical discoveries and makes finding new tracks as fun and exciting as flipping through records in a bargain bin.

In the first 3 months of the trial, we had over 1 000 000 samples requested and it鈥檚 a very promising start. We鈥檙e now sharing the findings with our colleagues to see how best to take Orbit and its findings forward.