A Beets Plugin for ML-Powered Emotion Prediction

[jaeheonshim]

VibeNet: A beets plugin for music emotion prediction!

I wanted a way to sort my music by a variety of perceptual metrics so that I could create smart playlists for different occasions, like working out, driving, studying etc. There are 7 commonly used features (acousticness, danceability, energy, instrumentalness, liveness, speechiness, valence) for this purpose. For example, a workout mix might have songs with high energy and high valence (happiness) content.

Spotify has these features in their API, but the way they are calculated is proprietary and unknown, so I decided to train my own neural network to locally predict these features from music files! I've made the model open source and created a beets plugin for it so that everyone can classify their songs.

Once your songs are tagged, you can query your music collection by emotion. For instance, if you wanted a party playlist with happy and upbeat songs, simply run this query:

$ beet ls valence:0.6.. energy:0.7..
ABBA - Arrival - Dancing Queen
ABBA - The Visitors - The Visitors
ABBA - The Visitors - When All Is Said and Done
ABC - The Lexicon of Love - Show Me
...

There are 7 different continuous features for you to play around with, so the degree of flexibility in the playlists you can create is quite large. For a detailed explanation of each feature, click here.

The model used is a lightweight CNN (EfficientNet) that runs well even if you don't have a CPU. You can configure the plugin to automatically add these new tags to your files during import, or run it as a CLI command manually. For more details and instructions, check out the GitHub repository: https://github.com/jaeheonshim/vibenet

Also, I would appreciate all the feedback I can get! At least from my perspective, the model spits out pretty reasonable predictions, but I plan to continue iterating with model architecture and training to provide even more accurate predictions.

[simonmarklar]

omg, This looks awesome!! I've been slowly working towards something like this but needed to get my library in order first. I wonder how this will go with a library of techno and minimal