This is what I have seemingly noticed, so I will have to double check the implementation & confirm this is occurring.
My simple idea is to just fetch the whole playlist and set that to be queue tracks.
As we store simply a track id (u32), let's assume a max of 100K tracks per playlist. That means 4 bytes per ID, or 400KB in total.