Plugin for better import incremental sync with upstream

[smichel17]

Before using Beets, my music library was already tagged from other sources, in a way that was basically-good-enough, but also managed manually, some AudioTrack01's, etc. I was considering writing some simple scripts for managing it, but then found Beets. A big draw for me is the ability to sync metadata with "upstream" MusicBrainz (and other sources). I imagine this is a pretty typical story.

Also pretty typical: A big pain point has been the time required to import. Well, kind of. It's actually more about having to spend that time up-front. Incremental import helps, but leaves my library split between two locations, which doesn't play well with other tools. Importing without auto-tagging and then gradually re-importing works, but opens up a separate can of worms:

The problem is that Beets' autotagger uses a bag of heuristics to determine whether an album matches an upstream data source, and there's no way for it to tell the difference between

• "I haven't gotten around to autotagging this yet"
  • vs "I don't want to autotag this!"
• Or "MusicBrainz doesn't know about this album yet (I should upload it)"
  • vs "This is a mix from a friend and I should probably convert it to a playlist eventually."
• Never mind weirder edge cases like "I auto-tagged this and then I changed some things, and I don't want my changes overridden"'
  • vs "The MusicBrainz tags were wrong when I autotagged this but have now been fixed, and I'd like to update to the latest."

I think the simplest solution would be a plugin which allows marking certain tracks/albums to be skipped during re-import.

Does a plugin (or setting I missed) for this exist? If not, I'd like to write one.

[sampsyo]

Hi! Thanks for this note; it's a really clear write-up of a common problem.

I don't think there is an existing plugin that does stuff like this. I seem to recall some discussion about this long ago, under the banner of thinking about the "metadata lifecycle." That was a bit more of an ambitious (probably too ambitious) plan, where the database would record a kind of "to-do" list for each album/track: have we fetched the album art, lyrics, etc. for this track yet? But I think you're right that the low-hanging fruit here is just being able to:

• Have albums in your library that you haven't finished "cleaning" yet. (The default way that beets works is that your "unclean" music is not imported yet, and therefore not in your library database. Importing and "cleaning" happen at the same step.)
• Easily run a "re-import" on unclean music in the library. (You may not be aware of this yet, but doing beet import on music that's already in the library is a way to re-run the autotagger without creating duplicate database entries.)

One "version 1.0" way to realize this plugin might be to just transparently add a field called $pending or something to all albums that get imported with the "as-is" option, and remove this tag whenever the importer successfully applies metadata to it. Then re-running the autotagger on all this stuff would be as simple as doing beet import -L pending:1 or similar.

[smichel17]

it's a really clear write-up of a common problem.

Thanks. I actually re-wrote it twice, trying to figure out how to be relatively concise but also accurate. And because I'm not 100% sure what this plugin will look like when it's finished. If you remember where the "metadata lifecycle" discussion happened (if publicly accessible), I'd be interested to read it (but please don't spend too much time searching 😄).

You may not be aware of this yet, but doing beet import on music that's already in the library is a way to re-run the autotagger without creating duplicate database entries.

Yes, soo... my own music library situation is a bit more complicated 😅; I solved the rewriting problem mostly by leaving out details. I actually first started beets back in… searches …2016…

Here's a long story, since I didn't have time to write a short one

My library was originally in wav format, with only file & folder structure for metadata. Before I found Beets, I wrote a script to convert to FLAC, adding the inferred tags along the way. I've since deleted the wavs (+500GB), although I might be able to find them on a backup somewhere.

I went in with the naive idea that I was going to do a perfect import on the first try. My plan was:

• Create a full backup of my FLACs
• Do an incremental import, moving the files into my library, retagging as I went
• Once I was done and satisfied I hadn't mangled anything, delete the backup.

For many of my albums — when the album was an official release; I was not missing any files; and MB metadata was already present — it went perfectly. In the same situation but when MB metadata was missing entirely, it took some work but I was able to upload my metadata to a new release, and then it went great.

What didn't go so well (basically what I wrote in the OP, but with specific examples this time):

The good: MB data I don't like

Peter Gabriel's first 4 solo albums are all named "Peter Gabriel". MB disambiguates them by the album cover (car, scratch, melt, security ); I prefer numbers (1-4).

• For now, I overwrote my local tags with MB data. Later, I plan to change the disambiguation in my local tags and exclude these from MB sync.

• Really I'd like to be able to tell beets, "Keep everything in sync with upstream, except don't overwrite the disambiguation field". Or something like that— more on this later (I have a separate plugin idea).

Overall, I'm happy enough with the current metadata (besides the disambiguation) that I could change that one thing, exclude these albums from MB sync, and not worry about it further.

The Bad: Mixes

A family friend (let's call him "Mik") makes yearly mixes (e.g. "Summer Mix 2018"). The early mixes have no metadata (AudioTrack01). Later mixes had the artist and title concatenated into the title, with Mik's Mix as the artist and Summer Mix 2018 as the album.

• I imported the early mixes "as is" and have left them untouched. I don't remember exactly how I handled the later mixes, but I managed to move Mik's Mix to albumartist and get the artist & title in their proper fields. Not linked to MB.
  • Rant on albumartist and how most players (don't) handle it later.
• I'm not sure how I'd like to handle these. I know I'd like to have a reference to the original album somehow. I could convert them to playlists, or I could keep the album as-is but include some kind of reference back to the original. I'm not super familiar with MB tagging, but I think this is possible…?

The later mixes are currently "good enough"; the early mixes are still in a sad state. I'd like it if the autotagger could help me classify these (especially with chroma/acoustid). However, I'm afraid to do that because…

The Ugly: Mangled Metadata

Some "mixes" were not really mixes, they were "here's an album (or most of one), and I have some space at the end of the CD, so here's a couple other songs too." For these, I was interested in at least having the full album under the original artist (maybe in addition to in the mix).

• You can't auto-tag a mix tape as an album, so instead it has to be by tracks… but if you do this and have write and move config options enabled, you scatter your mix to the winds.
• It gets worse— sometimes I tried to re-import the singles as an album (to re-group them), and sometimes this produced configurations that I thought were correct at the time but are actually just wrong.

So, I have some random, mistagged songs floating around in some combination of albums and singles, where they match well enough that it's hard to obviously identify them as mis-tagged (especially since I'm not super familiar with them). When I realized this, I think I also re-imported the mixes as-is (from my backup, with copy), so I can't rely on "this mix is in my backup but not my library" to identify the songs, either.

The silver lining is that they're all either singles (which are otherwise rare in my libary) or imperfect matches with MB, so if I can get the rest of my libary into perfect condition, some combination of the duplicates and missing plugins should be able to identify them.

In summary, re-importing has gotten me into a bit of a mess, with some mis-tagged songs/albums. However, all my problems together make up at 20% of my library, and big chunk of that is just the same as the status quo before Beets, so I've been just living with it. Recently, I've been trying to reduce the number of in-progress projects, so I'm finally circling back to finish this one and, in the words of the beets homepage, "get [my] music collection right, once and for all."

Have albums in your library that you haven't finished "cleaning" yet

Agreed. I think ideally this would be the primary workflow. The import command would be split into two commands, add and tag, which would each be simpler and less error-prone. But that's getting ahead of myself— for now I just want to focus on building what will help me fix my library now.

[smichel17]

One "version 1.0" way to realize this plugin might be to just transparently add a field called $pending or something to all albums that get imported with the "as-is" option, and remove this tag whenever the importer successfully applies metadata to it. Then re-running the autotagger on all this stuff would be as simple as doing beet import -L pending:1 or similar.

I agree that adding a field is a good way to go.

Another idea I had was to do it based on the various mb_ids. E.g. beet import mb_albumid::^$.
However, this doesn't distinguish between "pending" albums and those which MB is missing.

I figured a $noautotag flag would let me divide my library into four groups:

	No $noautotag	$noautotag=1
mb_albumid=*	In sync with MB	Local overrides for MB data (Peter Gabriel)
No mb_albumid	Pending "real import"	Mixes and other stuff which doesn't belong in MB

I think there might be value in a slightly more expressive status, to differentiate the upper corner between "MB tags are right/complete, but I want to tag this differently" and "MB tags are not quite right (e.g. missing my release)". Although I also have an idea for a different* plugin to solve the first case, so then the distinction wouldn't be necessary.

*and I need to re-familiarize myself with all the existing plugins, to make sure that the functionality does not already exist :)

[smichel17]

Looking over the list of plugins, it looks like this could be accomplished with IHate.

ihate:
    skip:
        - noautotag:1

So then it's "just" a matter of adding an option to the autotagger "import as-is and add noautotag=1".

[smichel17]

Also works with the mbsync plugin— beet mbsync ^noautotag:1

[Maxr1998]

That's a cool solution, only issue is that you can forget to specify this attribute. Allowing a set of default options in the config would be a good workaround, or, as suggested by others, an integrated noautotag attribute that's ignored by certain commands by default.

[JOJ0]

Hi @smichel17, you gotten around to writing this "pending" plugin or manage to use IHate with it already? I am facing a similar problem I guess.

[smichel17]

I have not. I hope to get back to my bag of things I'd like to do with beets within the next two weeks.

[smichel17]

On the other hand, what I wrote in #4158 (reply in thread) should work, although I have not tested it yet. You just need to add the noautotag attribute yourself, like:

beet modify artist:"Peter Gabriel" noautotag=1

(obviously replacing artist:"Peter Gabriel" with whatever query you need to apply to the right selection of songs)