MPEG Audio Scanner (aka MP3 Scanner) #120
Conversation
Deep scans MPEG Audio files Supports MP1, MP2, MP3
|
Mmmmm tests are failing, is this because I built against the SNDHDR branch? EDIT: I might refactor this before we merge, it literally kept me awake last night. It would make things a lot clearer for expansion/testing/debugging later. |
| case ( | ||
| mpeg_audio_scanner.mp3_id3_match_bytes | ||
| | mpeg_audio_scanner.raw_mp3_match_bytes | ||
| | mpeg_audio_scanner.fffe_match_bytes | ||
| | mpeg_audio_scanner.ffff_match_bytes | ||
| | mpeg_audio_scanner.fffc_match_bytes | ||
| | mpeg_audio_scanner.fffd_match_bytes | ||
| | mpeg_audio_scanner.fffa_match_bytes | ||
| | mpeg_audio_scanner.fff6_match_bytes | ||
| | mpeg_audio_scanner.fff7_match_bytes | ||
| | mpeg_audio_scanner.fff4_match_bytes | ||
| | mpeg_audio_scanner.fff5_match_bytes | ||
| | mpeg_audio_scanner.fff2_match_bytes | ||
| | mpeg_audio_scanner.fff3_match_bytes | ||
| | mpeg_audio_scanner.ffe6_match_bytes | ||
| | mpeg_audio_scanner.ffe7_match_bytes | ||
| | mpeg_audio_scanner.ffe4_match_bytes | ||
| | mpeg_audio_scanner.ffe5_match_bytes | ||
| | mpeg_audio_scanner.ffe2_match_bytes | ||
| | mpeg_audio_scanner.ffe3_match_bytes | ||
| ): |
There was a problem hiding this comment.
| case ( | |
| mpeg_audio_scanner.mp3_id3_match_bytes | |
| | mpeg_audio_scanner.raw_mp3_match_bytes | |
| | mpeg_audio_scanner.fffe_match_bytes | |
| | mpeg_audio_scanner.ffff_match_bytes | |
| | mpeg_audio_scanner.fffc_match_bytes | |
| | mpeg_audio_scanner.fffd_match_bytes | |
| | mpeg_audio_scanner.fffa_match_bytes | |
| | mpeg_audio_scanner.fff6_match_bytes | |
| | mpeg_audio_scanner.fff7_match_bytes | |
| | mpeg_audio_scanner.fff4_match_bytes | |
| | mpeg_audio_scanner.fff5_match_bytes | |
| | mpeg_audio_scanner.fff2_match_bytes | |
| | mpeg_audio_scanner.fff3_match_bytes | |
| | mpeg_audio_scanner.ffe6_match_bytes | |
| | mpeg_audio_scanner.ffe7_match_bytes | |
| | mpeg_audio_scanner.ffe4_match_bytes | |
| | mpeg_audio_scanner.ffe5_match_bytes | |
| | mpeg_audio_scanner.ffe2_match_bytes | |
| | mpeg_audio_scanner.ffe3_match_bytes | |
| ): | |
| case mpeg_bytes if mpeg_bytes in mpeg_audio_scanner.mpeg_audio_signatures | |
| ): |
Would just put those all in a single array
| # The first match wins | ||
| for scanner in (pdf_scanner, python_scanner, json_scanner): | ||
| result = scanner.main(filename, head, foot) | ||
| result = scanner.main(filename, head) |
There was a problem hiding this comment.
| result = scanner.main(filename, head) | |
| result = scanner.main(filename, head, foot) |
Need to pass in all three of these. Some other scanners may use foot or expect exactly 3 inputs
| cached_data = {"path": None, "matched": False, "name_end": [], "name_format": []} | ||
|
|
||
|
|
||
| class mp3_decoding: |
There was a problem hiding this comment.
| class mp3_decoding: | |
| class MP3Decoding: |
Standard class naming should be camel case
| b"WXXX", | ||
| ] | ||
| self.lyric3_tags = [b"IND", b"LYR", b"INF", b"AUT", b"EAL", b"EAR", b"ETT", b"IMG", b"GRE"] | ||
| """Temporary variables are stored here.""" |
There was a problem hiding this comment.
| """Temporary variables are stored here.""" | |
| # Temporary variables are stored here. |
| Not everything here may be used in outputs at this time. | ||
| """ | ||
|
|
||
| def __init__(self, file_path: os.PathLike | str): |
There was a problem hiding this comment.
I assume this class is basically a copy paste from your repo?
There isn't really a reason to use a class here otherwise, but if it's just an easy drop it can keep it that way for ease!
There was a problem hiding this comment.
I'm was using it to make things clearer, but then it got messier as I went on.
| return Match(extension=cached_data["ext"], name=cached_data["name"], mime_type="audio/mpeg", confidence=1.0) | ||
|
|
||
|
|
||
| def main(file_path: os.PathLike | str, head: bytes) -> Optional[Match]: |
There was a problem hiding this comment.
| def main(file_path: os.PathLike | str, head: bytes) -> Optional[Match]: | |
| def main(file_path: os.PathLike | str, head: bytes, _) -> Optional[Match]: |
|
Thanks for the suggestions, I'll integrate this into a new PR I'm going to close this PR for now as it's a bit messy and I want to refactor into something a bit easier to deal with. There will still be classes purely to help keep everything apart (EOF Tags, v2 Tags, VBR and Stream) this should make it easier for anyone else reading the code to follow what does what. Hope you like the scanner so far 😎 |
|
OK rewrite making good progress, the EOF tag scanner now correctly identifies and calculates the size of all tag styles (my math skills took a beating). Much more logical layout for the code. Less temporary data being stored as well. |
Absolutely this is a lot of great work, thank you for the effort! |
2 participants
MPEG Audio Scanner Version 1
This is my first pass at an MP3 Scanner, it ended up being a whole lot more than that, it can scan basically any valid MPEG-1 audio stream. In short, if it's an
.mp1,.mp2or.mp3this should understand what it is.The decoder grew in scope far beyond what PureMagic is aimed at, I'm going to eventually release a full-featured decoder under its own repo as a standalone tool, this is not to compete with PureMagic but it will provide features outside of the PureMagic goals (i.e: Tag recovery/conversion for obscure formats, Data stream checking etc...). As I develop either this or that, code enhancements will pass back and forth, so this scanner will see updates.
Deepscan
I've altered the
Best Matchline to printDeepscan Matchif a scanner returns a positiveconfidence=1result, this makes it clearer to users that we're 100% certain the file is what it is. If the file fails the test we politelyreturn Noneand let a regular magic_data match offer a best match.magic_data
A few changes have been made to accommodate matches for all valid byte combinations of
.mp1,.mp2and.mp3.Changed extension on
ffbbfrom the less common.mpgato.mp3MPEG Audio Scanner:
Overview
To test fully if the file is a true MP3 I've ending up building something close to a full featured MP3 decoder:
return Noneand matches will fall back to the .json as there is a chance it's not an MPEG Audio file (or is highly corrupted).Features
Deep scans any MPEG Audio files, this is a pretty complex scanner we have to account for:
Issues and Limitations
MPEG Audio quirks
MP3's are not quite as standard as one would think, there are a few common assumptions that are wrong, I learned a lot about the format working on this one, some examples are:
ID3: If the file has V2 tags thenID3will be present at byte 0, otherwise it's the MPEG header frame starting with hexffeXorfffXXingmeans VBR andInfomeans CBR: an MP3 can be encoded withInfoand still be a VBR. 🤦 MediaInfo (an awesome tool) checks these flags and bases VBR/CBR-ness from this, if you hex editInfotoXingor vice-versa it changes its report.Tags Tested
ID3v1.x
TAG128 bytes from EOF. The original MP3 tags, limited but everything knows what to do with them.ID3v2.x
ID3at start of file. These are the current standard, big lumps of data for all sorts of info, v2.2, v2.3 and v2.4 all differ slightly but are handled.APE Tag
APETAGEXat the absolute end of file or just before ID3v1TAG. The APE tag is a competitor to the standard ID3 tag. There are two version and both are detected.ID3v1.2 Enhanced Tag
EXT: 256 bytes from EOF. Niche standard used in the late 1990's invented by BirdCageSoft. Their software supports it but I can't find anything else that does. It was designed to overcome the limits of ID3v1 by offering extra tacked on space for tags.The software only runs on XP or earlier (failed on x86 Win 7)
ID3v1 Enhanced Tag
TAG+at 227(ish) bytes from EOF. Another niche standard aimed at addressing similar shortfalls in ID3v1 tags asEXT.One tool created by the spec creators called SpeedTag exists on the WaybackMachine linked below for those wanting to play.
Interestingly the SpeedTag tool does not follow their spec on site and seems to place the TAG+ data slightly earlier in the file, we can still test for it reliably and without affecting overall speed.
There is also a later tool MP3Manager (see LYRICS) created by one or more of the SpeedTag/TAG+ authors, this may have supported TAG+ in an earlier form but it's latest version seems to ignore them.
Browsing between pages is broken due to missing interstitials
Amazingly works on Win 11 x64, just use WinZip or similar to unpack, no need to run the .exe directly
Additional notes regarding
TAG+andEXTDue to the nature of these tags it entirely possible for entries to be corrupted easily by other TAG editors. In addition to getting pushed out of the byte window by other EOF tags, there is the possibility of a regular ID3v1 tag editor altering the base
TAGwithout affecting these two in any way.This is what really put the nail in the coffin for these extended formats, they were a great idea at the times but splitting the tags between two data fields caused weird or short names on devices that could not read them, or they could be easily corrupted by other tag editors.
LYRICS
Large block before ID3v1
TAGprefixed byLYRICSBEGIN. Created to address both the shortfalls of ID3v1 tags and add lyrics to your song. Seems to have been created in part or whole by some of the TAG+ developers. Lyrics3 (v1 and v2) became one of the first widely used standards to successfully add lyric information to MP3s. Lyrics3v2 upgrades allowed for timestamped lyrics for karaoke and other enhancements.The spec for v2 has a size field for calculating the tag size, however the official tool
MP3Manageris bugged by design or accident and breaks their spec by miscalculating the size, some other tools seem to follow the spec going by some files in my library. I've created a rudimentary check that is not ideal but functional. If I develop a more rigid check later for my own decoder project I'll backport it to PureMagic.Installs on Win 11 x64, will ask for a player, just point it at it's own exe for one of the choices if you don't use any it wants.
3DI Tag
3DI10 bytes before the ID3v1TAG. This is a super niche tag, According to the Library of Congress link it was meant to be placed 10 bytes before the ID3v1TAGmarker, or 10 bytes before the end of the file if not.It's purpose as summarised by Google Gemini (about the only source of information I could find on what it was for):
Once ID3v1.1 came along this became less relevant and obviously ID3v2 killed any need for it stone dead. I have no test files so this is a theoretical implementation but no reason for it to not work.
Sample files
mp3
For testing, all VBR files found in the
test\resources\audioare based off3-second synth melodyfrom https://samplelib.com/sample-mp3.html, the files are free or any use restrictions:VBR-Xing-128k-NoTags.mp3VBR file withXingheader and No TagsVBR-Info-128k-NoTags.mp3VBR file withInfoheader and No Tags, MediaInfo will incorrectly call this a CBRVBR-Xing-128k-v1tag-tagplus.mp3VBR file withXingheader, V1 TAGS and ID3v1 Enhanced Tag (TAG+). Almost nothing now can read the TAG+ part and will simply ignore it.VBR-Xing-128k-v1tag-ext.mp3VBR file withXingheader, V1 TAGS and ID3.1v2 (EXT Tag). Almost nothing now will read the EXT part and will simply ignore it.VBR-Xing-128k-v1tag-lyrics3v2.mp3VBR file withXingheader, V1 TAGS and Lyrics3v2.These abominations are purely to stretch the decoder to it's limit, these combinations could exist if someone modified an old file they found buried somewhere.
VBR-Xing-128k-v1tag-tagplus-ape.mp3VBR file withXingheader, V1 TAGS, ID3v1 Enhanced Tag and APE Tags. SpeedTag will no longer be able to see the TAG+ as it's pushed out the byte window it looks at.VBR-Xing-128k-v1tag-ape-tagplus.mp3VBR file withXingheader, V1 TAGS, APE Tags and ID3v1 Enhanced Tag. SpeedTag pushes the APE tags out from their byte window causing them to no longer be seen.I've not added V2 tagged based samples as they do not affect the logic needed to test the end of the file, equally CBR's not encoded via LAME and VBRI's would behave the same.
mp2 and mp1
For testing:
I've not added them to the repo due to lack of notices regarding usage.
We can add the samples from these links if we wish to use them for testing.
Example outputs:
These all come from real files.