Unable to decode dealer user collection (liked song collection) events #1505

BenJeau · 2025-05-27T04:01:25Z

BenJeau
May 27, 2025

Description

I'm creating a Spotify client to listen passively on events from Spotify. It works great for most of the events, except the hm://collection/collection/:username one, which is used whenever a user adds/remove a song from their liked songs.

Version

Using the latest commit, 8b72954

How to reproduce

Create a new project with the following dependencies + code
Execute cargo run
Login to Spotify in the browser window it opens up
Add/Remove a song from your saved library
View error in the logs

Cargo.toml

[dependencies]
futures-util = "0.3.31"
librespot = { git = "https://github.com/librespot-org/librespot", rev = "8b729540f4ad1e7f8c94ff3bba33095878b24d02" }
protobuf = "3.7.2"
tokio = { version = "1.45.1", features = ["full"] }
tracing = "0.1.41"
tracing-subscriber = "0.3.19"

main.rs - the protobuf message used for hm://collection/collection/:username is certainly not the right one, but it would give another error if it reached that part (I'm also unsure which protobuf schema this would relate to):

use futures_util::stream::StreamExt;
use librespot::{
    core::{Error, Session, SessionConfig, dealer::protocol::Message},
    discovery::Credentials,
    oauth::OAuthClientBuilder,
    protocol::connect::{ClusterUpdate, MemberType, PutStateReason, PutStateRequest},
};
use protobuf::EnumOrUnknown;

const HTML: &str = r#"<!DOCTYPE html>
<html>
  <head>
    <script>
      window.close();
    </script>
  </head>
</html>
"#;

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::DEBUG)
        .init();

    let client = OAuthClientBuilder::new(
        "65b708073fc0480ea92a077233ca87bd",
        "http://127.0.0.1:8898/login",
        vec!["streaming"],
    )
    .open_in_browser()
    .with_custom_message(HTML)
    .build()
    .unwrap();

    let access_token = client.get_access_token_async().await.unwrap();
    let refresh_token = client
        .refresh_token_async(&access_token.refresh_token)
        .await
        .unwrap();
    let credentials = Credentials::with_access_token(refresh_token.access_token);
    let session = Session::new(SessionConfig::default(), None);

    fn extract_connection_id(msg: Message) -> Result<String, Error> {
        let connection_id = msg.headers.get("Spotify-Connection-Id").unwrap();
        Ok(connection_id.to_owned())
    }

    let mut connection_id_update = session
        .dealer()
        .listen_for("hm://pusher/v1/connections/", extract_connection_id)
        .unwrap();
    let mut user_collection_update = session
        .dealer()
        .listen_for::<ClusterUpdate>(
            &format!("hm://collection/collection/{}", session.username()),
            |message| {
                println!("user collection update:\n{message:?}");
                Message::from_raw(message)
            },
        )
        .unwrap();

    session.connect(credentials, true).await.unwrap();
    session.dealer().start().await.unwrap();

    println!("---connected!!!---");
    loop {
        tokio::select! {
            user_collection_update = user_collection_update.next() => {
                println!("user collection update:\n{user_collection_update:?}");
            }
            Some(Ok(connection_id)) = connection_id_update.next() => {
                session.set_connection_id(&connection_id);
                let request = PutStateRequest {
                    member_type: EnumOrUnknown::new(MemberType::CONNECT_STATE),
                    put_state_reason: EnumOrUnknown::new(PutStateReason::NEW_DEVICE),
                    ..Default::default()
                };
                session.spclient().put_connect_state_request(&request).await.unwrap();
            }
        }
    }
}

Log

My logs are extremely verbose, but I've captured where the parsing issues are logged. I removed/added/removed/added the same song link=https://open.spotify.com/track/1kuPgsWuwfNVHTPDBGMMj4, uri="spotify:track:1kuPgsWuwfNVHTPDBGMMj4", uid="ea0019b856d6d45d2cc4"

// removed
Message { headers: {"MC-ETag": "64941807882", "Collection-Update-Id": "66dce80f5c4d7c20", "Collection-Source-Revision": "55320748569", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:23.652329Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. }    


// liked
Message { headers: {"Collection-Update-Id": "9ee4a66d1fa26263", "MC-ETag": "46685679324", "Collection-Source-Revision": "64941807882", "Content-Type": "application/octet-stream"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:31.498298Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. }    

 
// removed
Message { headers: {"MC-ETag": "47316492062", "Collection-Source-Revision": "46685679324", "Collection-Update-Id": "ee516cc0498e02dd", "Content-Type": "application/octet-stream"}, payload: Raw([10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:40.601061Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 125, offset 108. }   


// liked
Message { headers: {"Collection-Source-Revision": "47316492062", "Content-Type": "application/octet-stream", "MC-ETag": "56467871245", "Collection-Update-Id": "ab9637744956f5fa"}, payload: Raw([10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]), uri: "hm://collection/collection/ha9ha10" }
Error { kind: FailedPrecondition, error: Error(WireError(Utf8Error)) }
2025-05-27T03:04:42.219687Z  WARN librespot_core::dealer: failure during data parsing for hm://collection/collection/ha9ha10/json: Invalid state { base64 decoding failed: Invalid symbol 123, offset 0. }

Host (what you are running `librespot` on):

OS: macOS 15.5 (24F74)
Platform: MacBook Pro

Additional context

You can see the raw data is similar throughout the requests:

1st removed: [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1]
1st added: [10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 191, 215, 212, 193, 6, 48, 0]
2nd removed: [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1] (the same)
2nd added: [10, 28, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 202, 215, 212, 193, 6, 48, 0]

I can see the following:

8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40 is common, I'm guessing that the song ID
the last byte seems to be a boolean, 1=removed, 0=added

Ultimately, I'm unfamiliar how this works other from my exploration of this codebase.

BenJeau · 2025-05-28T02:46:16Z

BenJeau
May 28, 2025
Author

Continued my exploration, but no definitive way to parse it yet.

I think the protobuf file for this type of message would look something like

syntax = "proto3";

package spotify;

message Track {
    int32 status = 1;                   // Always 0 (maybe play state?)
    bytes track_id_binary = 2;          // 16-byte binary Spotify track ID
    optional int32 context_field = 3;   // Possible context/playlist ID (4 bytes)
}

message SpotifyLikeAction {
    repeated Track tracks = 1;      // Can contain multiple tracks
    uint64 timestamp = 5;           // Timestamp (0 for unlike, current time for like in seconds)
    bool action_type = 6;           // false = save, true = unsave
}

For [10, 24, 8, 0, 18, 16, 43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30, 40, 0, 48, 1], the parsed message is now SpotifyLikeAction { tracks: [Track { status: 0, track_id_binary: [43, 184, 38, 105, 23, 212, 70, 149, 146, 77, 130, 115, 44, 209, 83, 30], context_field: None }], timestamp: 0, action_type: false }

Steps to convert those binary track_id bytes to the Spotify ID:

Convert bytes to u128 using big endian
Convert the number to base62 using the 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ alphabet
You get the ID 1kuPgsWuwfNVHTPDBGMMj4 which is the correct Spotify ID!

Other things I found out/confirmed:

I saw in a few messages that a single message can contain multiple tracks (why the repeated Track tracks).
The last byte corresponds to save/unsaved. 1=unsaved, 0=saved
Anything between the last 48 byte and the 40 byte before it represents the timestamp in seconds
- For unsaves, it is always 0
- For saves, it is a timestamp in second - I'm unsure what the type of this field is though
- I was unable to get this field and the action_type field to work with protobuf, but was able to with a custom decoder.

0 replies

photovoltex · 2025-05-28T13:31:02Z

photovoltex
May 28, 2025
Maintainer

Interesting discoveries! But what exactly is your issue you encounter? Is it something rudimentary in the implementation or rather a missing protobuf definition?

From your log there is a WireError and a baes64 error/warning, but I can't tell which of those you refer to.

You also use ClusterUpdate as parsing for the collection event, which is usually used for the connect device communication if I'm remember right. So maybe that's the issue, which wouldn't be directly a library issue, right?

If so I would convert this issue into a discussion, so we can still help you but without declaring non existing proto files as bug^^;

0 replies

BenJeau · 2025-05-29T00:22:49Z

BenJeau
May 29, 2025
Author

Sorry for opening an issue if this isn't considered an issue. Yes it is about the parsing of the data from the hm://collection/collection/:usernameevents, I saw it as an issue as the other events didn't have any issue with parsing. I do not know exactly what is the issue, I have a hunch is more complex than just a missing protobuf definition, but I think it is missing a protobuf definition and I could be wrong.

I'm aware that ClusterUpdate isn't the right message format for this, but decided to just put anything there as the errors without it weren't descriptive. In the code itself, the WireError and base64 error comes from the same place (if my memory serves well), just logged/returned differently.

Feel free to convert this as a discussion, I do not mind :)

On another topic, if you know how to "acquire" or extract the protobuf definitions from a Spotify binary, let me know! Either here or at my email

0 replies

photovoltex · 2025-05-29T10:01:23Z

photovoltex
May 29, 2025
Maintainer

Interesting how the issue is converted. Anyhow, no problem, you can see it both ways. It could be considered an bug as there is something missing, but as we do not include all proto files to keep the size somewhat down I would rather consider it more of a "need help" section which seems more appropriate as discussion.

So anyways, you can look at #1424, there is the last proto dump from which the definition were updated last time. There is also a simple tool listed with which you can extract the protobuf definitions of the current clients. Maybe you can find a matching definition there. If not, we will need to add our own protobuf file with the matching definition :)

2 replies

BenJeau May 29, 2025
Author

Aha wasn't aware of that tool, cool! I will look more into those protobuf files - thanks again!

photovoltex May 29, 2025
Maintainer

No problem :) If you want you can still share your discoveries here :D

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unable to decode dealer user collection (liked song collection) events #1505

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Unable to decode dealer user collection (liked song collection) events #1505

Uh oh!

BenJeau May 27, 2025

Description

Version

How to reproduce

Log

Host (what you are running librespot on):

Additional context

Replies: 4 comments · 2 replies

Uh oh!

BenJeau May 28, 2025 Author

Uh oh!

photovoltex May 28, 2025 Maintainer

Uh oh!

BenJeau May 29, 2025 Author

Uh oh!

photovoltex May 29, 2025 Maintainer

Uh oh!

BenJeau May 29, 2025 Author

Uh oh!

photovoltex May 29, 2025 Maintainer

BenJeau
May 27, 2025

Host (what you are running `librespot` on):

Replies: 4 comments 2 replies

BenJeau
May 28, 2025
Author

photovoltex
May 28, 2025
Maintainer

BenJeau
May 29, 2025
Author

photovoltex
May 29, 2025
Maintainer

BenJeau May 29, 2025
Author

photovoltex May 29, 2025
Maintainer