Usecase: I have a myriad of finetunes/ablit models that re-use the same mmproj, I'd rather not copy/symlink stuff around. Also some quant makers (even official ones) put mmproj- at the start of the filename.
I'd rather be able to manually pick which mmproj file to load with which model. This would mean getting rid of the auto-detector you have in place, and putting the user in control. the input would work the same as the other gguf loaders, except it will additionally filter with the 'mmproj' word in the filename.
I would be willing to contribute a PR if you would agree to having this feature.
EDIT: I ended up coding this anyway, I've thrown in a PR if you're interested.