Here be dragons: I have never seen an import that didn't crash.

[IngwiePhoenix]

First, my setup: I use an OpenWrt router to handle all gateway and server needs in my home network, the only node between it and the internet is an A/VDSL modem (i can never remember which one...). And on the other side, I have an old Synology NAS that I lately outfitted with a set of 8TB HGST HDDs in RAID1. So far, so good. Speeds aren't the greatest, but it works for my single-user needs.

Here is the problem: When I run beets on my router to manage my library mounted via SMB, I continiously get "File not found" errors, which really shouldn't be the case.

Trace:

Importing as-is.
Traceback (most recent call last):
  File "/usr/bin/beet", line 33, in <module>
    sys.exit(load_entry_point('beets==1.6.0', 'console_scripts', 'beet')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/usr/lib/python3.11/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/lib/python3.11/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/usr/lib/python3.11/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/usr/lib/python3.11/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 314, in run
    for msg in _allmsgs(out):
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 467, in pull
    out = coro.send(msg)
          ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/usr/lib/python3.11/site-packages/beets/importer.py", line 1369, in lookup_candidates
    plugins.send('import_task_start', session=session, task=task)
  File "/usr/lib/python3.11/site-packages/beets/plugins.py", line 488, in send
    result = handler(**arguments)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/plugins.py", line 145, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 180, in fingerprint_task
    return fingerprint_task(self._log, task, session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 249, in fingerprint_task
    acoustid_match(log, item.path)
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 89, in acoustid_match
    duration, fp = acoustid.fingerprint_file(util.syspath(path))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/acoustid.py", line 340, in fingerprint_file
    return _fingerprint_file_audioread(path, maxlength)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/acoustid.py", line 280, in _fingerprint_file_audioread
    with audioread.audio_open(path) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/audioread/__init__.py", line 127, in audio_open
    return BackendClass(path)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/audioread/rawread.py", line 59, in __init__
    self._fh = open(filename, 'rb')
               ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: b'/mnt/diskstation/bunker/Music/3LAU/Dance Floor Filth 4/01-01 3LAU - Backsmash (3LAU mashup).mp3'

This is my config:

plugins: duplicates convert fetchart embedart chroma

directory: /mnt/diskstation/bunker/Music

threaded: yes
per_disc_numbering: yes

paths:
  default: $albumartist/$album%aunique{}/$disc-$track $artist - $title
  singleton: Non-Album/$artist/$title
  comp: Compilations/$album%aunique{}/$track $title

import:
  write: yes
  move: yes
  copy: no
  autotag: yes
  link: no
  hardlink: no
  reflink: no
  quiet_fallback: asis
  none_rec_action: asis
  default_action: asis
  languages: en de
  duplicate_action: merge
  bell: yes

duplicates:
  merge: yes
  copy_album_art: yes
  delete_originals: true

spotify:
  mode: list

embedart:
  ifempty: yes

web:
  reverse_proxy: true

And, the command: beet import -w -q . (CWD being /mnt/diskstation/bunker/Music).

I am at my wit's end here and am just frustrated in a way. All I want is for beets to work it's way through my library, sort files to where they should go, add potentially missing tags as far as it can tell without intervention (I plan to run a non--q import down the line to answer any questions it may has) and remove any duplicates that exist - because I am very sure they do. This folder is a culmination of merging several heres and theres into a singular folder and my attempt at finally, at last, putting an end to an unsorted, messy music library.

But no matter how many times I run it, it crashes... all the time. It'll go for a while, but eventually just drop. I wish it had Make's "keep going" flag to accompany -q.

Do you have any idea what I could do to finally make this work out?

Kind regards,
Ingwie

PS.: The library is ~200 GB and growing since 2004.

[arsaboo]

Can you try beet -vv import -w -q .

The verbose output should offer some clues.

[IngwiePhoenix]

Did:

root@FriendlyWrt /m/d/b/Music# beet -vv import -w -q . >~/beet.log 2>~/beet.log
root@FriendlyWrt /m/d/b/Music# cat ~/beet.log
(... snip ...)
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Various Artists - Playlist: May 2006 (9abd2733-c7b9-44db-93ec-750287a5f0ad)
Computing track assignment...
...done.
Success. Distance: 0.84
chroma: acoustid album candidates: 0
Evaluating 7 candidates.
Traceback (most recent call last):
  File "/usr/bin/beet", line 33, in <module>
    sys.exit(load_entry_point('beets==1.6.0', 'console_scripts', 'beet')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/usr/lib/python3.11/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/lib/python3.11/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/usr/lib/python3.11/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/usr/lib/python3.11/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 314, in run
    for msg in _allmsgs(out):
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 467, in pull
    out = coro.send(msg)
          ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/usr/lib/python3.11/site-packages/beets/importer.py", line 1369, in lookup_candidates
    plugins.send('import_task_start', session=session, task=task)
  File "/usr/lib/python3.11/site-packages/beets/plugins.py", line 488, in send
    result = handler(**arguments)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beets/plugins.py", line 145, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 180, in fingerprint_task
    return fingerprint_task(self._log, task, session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 249, in fingerprint_task
    acoustid_match(log, item.path)
  File "/usr/lib/python3.11/site-packages/beetsplug/chroma.py", line 89, in acoustid_match
    duration, fp = acoustid.fingerprint_file(util.syspath(path))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/acoustid.py", line 340, in fingerprint_file
    return _fingerprint_file_audioread(path, maxlength)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/acoustid.py", line 280, in _fingerprint_file_audioread
    with audioread.audio_open(path) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/audioread/__init__.py", line 127, in audio_open
    return BackendClass(path)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/audioread/rawread.py", line 59, in __init__
    self._fh = open(filename, 'rb')
               ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: b'/mnt/diskstation/bunker/Music/3LAU/Dance Floor Filth 4/01-01 3LAU - Backsmash (3LAU mashup).mp3'

Full log: beet.log

[JOJ0]

Beets latest from git? Or which version?

I also use an old synology nas to save my library, slow but works.

You might run into network problems maybe and nothing to do with beets though.

So you are saying that this message just is not true, the file is existing, right?

FileNotFoundError: [Errno 2] No such file or directory: b'/mnt/diskstation/bunker/Music/3LAU/Dance Floor Filth 4/01-01 3LAU - Backsmash (3LAU mashup).mp3'

Just wild guessing but if that is the fact, then it sounds like the problem may lye somewhere around of how the network file system is mounted on your router. How did you mount the smb device exactly? I can briefly check your mount options.....but no expert here as well..

[JOJ0]

My guess is that something in the corotines becomes "desynced" and so it believes the file still exists, although an earlier command moved/deleted it.

Search for similar issues.

Me personally I haven't seen such issues with desynced corotines around the beets forums. Sounds unlikely and still points to something the filesystem is responsible for and not the beets code but you never know actually....

[IngwiePhoenix]

Sounds like a mount issue. Try importing a local file to rule that out.

Tried from local, and even just a single file on the share itself. Both worked fine. But when I let Beet run in a screen and stuff, it just stops. I also tried to verify if the mount is problematic by running a fs speed test (bonnie++). Well, it too crashed but didn't exactly say why. So, I have re-mounted the drive via NFS instead of SMB. Maybe this works better? I'll at least give it a shot. But it does seem like the mount is the problem now ... this is very confusing.

[IngwiePhoenix]

I spent an absurd amount of time fiddling with mount options between SMB and NFS, enabling UNIX permissions on my Synology and a few other settings, even setting up GID and UID mapping in SMB via the apropriate options - but, no luck, it still fails.

But, it did work when disabling multi-threading (config threading: no) and it always breaks when trying to run the chroma plugin, so I will disable that first and try again.

But I am starting to run out of ideas, honestly. Here's the mount options currently in use:

root@FriendlyWrt /m/d/b/Music# mount | grep bunker
192.168.2.170:/volume2/bunker on /mnt/diskstation/bunker.nfs type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.1,local_lock=none,addr=192.168.2.170)
//192.168.2.170/bunker on /mnt/diskstation/bunker type cifs (rw,relatime,vers=3.1.1,cache=none,username=admin,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.2.170,file_mode=0755,dir_mode=0755,nocase,hard,nounix,serverino,mapposix,cifsacl,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,_netdev)

The associated fstab entries:

//192.168.2.170/bunker  /mnt/diskstation/bunker  cifs  _netdev,iocharset=utf8,credentials=/sdcard/.smbcredentials,hard,nocase,cache=none,cifsacl,perm
192.168.2.170:/volume2/bunker /mnt/diskstation/bunker.nfs nfs4 nolock

If you spot any option i could add or something, please do tell. :)

[JOJ0]

I'm sorry no more ideas :-( The OpenWRT you are using: Maybe search if others have mounting/fs problems. I don't know how modern that OS is or what kernel....just fishing in the dark....I'm so sorry I can't help more!

[IngwiePhoenix]

It's alright, mounting did seem to at least be part of the issue :) So thats something!
OpenWrt per-se is actually quite modern; MUSL libc, focused on stability, but up-to-date in most instances. Ther kernel is a "Rockchip kernel", 5.10.110. I did notice missing idmapd or nfsidmap though, so I might have to go and recompile those packages myself (which is quite easy to do, in fact).

Thanks for the pointers!