Canon EF 28-105 Lens Recognition

[Profoktor]

Describe the bug

When probing exif information with exiv2 for a specific camera/lens combination (see below), for a relevant fraction of the images the lens gets not identified correctly: Although the ID seems to match, exiv2 is unable to identify the LensType-String.

• There is no obvious difference between the files.
• It occurs for images which where taken seconds apart
• It is reproducible per image, i.e. if the LensType cannot be identified by exiv2 the first time, the behaviour remains the same for consecutive calls to exiv2

This also affects downstreams products (as in this case darktable - the import stores the ID instead of the name, and automatic lens correction fails)

To Reproduce

Run exiv2 as shown below on the two provided, consecutively taken images.

• -pv: identical output concerning the LensType: 156
• -pa / -pt: For one image, the lens gets identified correctly, for the other just the id is shown

Expected behavior

exiv2 should resolve the lens for both images correctly, giving the desired output ("EF28-105mm f/3.5-4.5 USM").

Workaround

Place the lens identification in exiv2 config file:

[canon]
156=Canon EF 28-105mm f/3.5-4.5 USM

Desktop:

• OS and version: Windows 10
• Compiler and version (Download-Packages via https://www.exiv2.org/download.html:
  -- MSVC 2019, exiv2-0.27.5-2019msvc64
  -- MinGW, exiv2-0.27.5-MinGW64

Additional context

Camera: Canon EOS 400D
Lens: EF28-105mm f/3.5-4.5 USM (Identified with id 156)

Side Notes:

• The issue came up when trying to migrate from Lightroom to darktable. Lightroom was able to correctly interpret exif data, so I would assume that the files themselves are corrupted.
• The issue has not been seen for other lenses so far (Canon/Tamron with same camera, another Canon lens with another camera)

Console outputs:

// print all
.\exiv2.exe -pa --grep lens/i  .\exif_correct.CR2
Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pa --grep lens/i  .\exif_incorrect.CR2
Exif.CanonCs.LensType                        Short       1  156
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

// print plain exif
.\exiv2.exe -pv --grep lens/i  .\exif_correct.CR2
0x0016 CanonCs      LensType                    Short       1  156
0x0017 CanonCs      Lens                        Short       3  105 28 1
0x0009 CanonCf      LensAFStopButton            Short       1  2048
0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pv --grep lens/i  .\exif_incorrect.CR2
0x0016 CanonCs      LensType                    Short       1  156
0x0017 CanonCs      Lens                        Short       3  105 28 1
0x0009 CanonCf      LensAFStopButton            Short       1  2048
0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM

// print interpreted exif
.\exiv2.exe -pt --grep lens/i  .\exif_correct.CR2
Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pt --grep lens/i  .\exif_incorrect.CR2
Exif.CanonCs.LensType                        Short       1  156
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

[Profoktor]

Sample files:
CR2.zip

[Profoktor]

I set up the build environment:

• Building the release version reproduces the bug
• Building main does not reproduce the bug

Conclusion:
Issue seems to have been fixed after the 0.27.5 Release - Bug can be closed

Out of curiosity - if anyone could give me a pointer to the corresponding PR, I would be delighted :)

[clanmills]

Thank you for reporting this. And thank you for providing test data.

I'm unable reproduce your report.

Using exiv2 0.25 (built from the current 0.27-maintenance branch)

509 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ exiv2 -vVg exiv2
exiv2 0.27.5
exiv2=0.27.5
package_name=exiv2
executable=/usr/local/bin/exiv2
library=/usr/local/lib/libexiv2.0.27.5.dylib
config_path=/Users/rmills/.exiv2
510 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ cd ~/Downloads/CR2/
511 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pv -g lens/i *
exif_correct.CR2      0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2      0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2      0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2      0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2    0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2    0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2    0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
512 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_correct.CR2      Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_correct.CR2      Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_correct.CR2      Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  156
exif_incorrect.CR2    Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_incorrect.CR2    Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_incorrect.CR2    Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
513 rmills@rmillsm1:~/Downloads/CR2 $

Using the 'main' branch:

513 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -vVg exiv2
exiv2 1.0.0.9
exiv2=1.0.0
package_name=exiv2
executable=/usr/local/bin/exiv2
library=/usr/local/lib/libexiv2.1.0.0.9.dylib
config_path=/Users/rmills/.exiv2
514 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pv -g lens/i *
exif_correct.CR2      0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2      0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2      0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2      0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2    0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2    0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2    0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
515 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_correct.CR2      Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_correct.CR2      Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_correct.CR2      Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_incorrect.CR2    Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_incorrect.CR2    Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
516 rmills@rmillsm1:~/Downloads/CR2 $ 

Running the same test with the download of Exiv2 v0.27.5 from exiv2.org

510 rmills@rmillsmm-local:~/Downloads $ tar xf exiv2-0.27.5-Darwin.tar
511 rmills@rmillsmm-local:~/Downloads $ cd exiv2-0.27.5-Darwin
512 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -vVg exiv2
exiv2 0.27.5
exiv2=0.27.5
processpath=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/bin
package_name=exiv2
executable=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/bin/exiv2
library=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/lib/libexiv2.0.27.5.dylib
config_path=/Users/rmills/.exiv2
513 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pv -g lens/i ~/Downloads/CR2/*
exif_correct.CR2  0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2  0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2  0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2  0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2  0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2  0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2  0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
514 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $ 

Please work with me to get "on the same page" so that we agree on the issue being investigated.

I'm going to modify the title of this issue. In general the Lens is not stored in the metadata and Exiv2 has to "guess" the lens by inspecting other metadata. In your case, the lens is stored int the metadata and the tag Exif.Canon.LensModel is a 64 byte ASCII string and correctly identified on all three builds.

It is possible that darktable is using an earlier version of the exiv2 library that does not recognise this lens. The remedy for that is to use the ~/.exiv2 configuration file to identify Lens 156. This is documented in the exiv2 man page exiv2.1.

It's also possible that I'm a little blind this today and failed to understand your report (and my tests). Please forgive and help me to understand.

[Profoktor]

Hi @clanmills,

thanks for your quick response. I'll try to shed some light on the issue.

• In general, the lens can be recognised, i.e. it is part of the database (e.g. the "working image")
• However, for some images/files, the resolution/translation with exiv2 -pt breaks in version 0.27.5. The result is, that exiv2 only reports the bare id and not the resolved LensType
  -- It breaks for the download binaries 0.27.5 (You did not run the -pt option to reproduce the behavior)
  -- It breaks for a "fresh" build of the 0.27-maintenance branch
  Actually, you just demonstrated that as well:

515 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
...
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  156
...

But luckily, this seems to be fixed on the main branch (see my previous comment). I think I should have emphasized that a bit more in the first place as soon as I found out.

So, nothing to do here 👍
(Unless, by chance somebody has an idea what the problem actually was - I'm curious ;))

[clanmills]

I'm still having difficult understanding the scope/versions involved in this issue. Have I understood this correctly -pt -g lenstype/i reports tag Exif.CanonCs.LensType as:

file	branch main	branch 0.27-maintance
exif_correct.CR2	Short 1 Canon EF 28-105mm f/3.5-4.5 USM	Short 1 Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2	Short 1 Canon EF 28-105mm f/3.5-4.5 USM	Short 1 156

921 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g lenstype/i ~/Downloads/CR2/*
exif_correct.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  Exif.CanonCs.LensType                        Short       1  156
922 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ 
922 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ cd ../../main/build
923 rmills@rmillsm1:~/gnu/github/exiv2/main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g lenstype/i ~/Downloads/CR2/*
exif_correct.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
924 rmills@rmillsm1:~/gnu/github/exiv2/main/build $ 

I'll have to investigate why -pt is incorrectly translating the file exif_incorrect.CR2 on branch 0.27-maintenance. The behaviour is unlikely to be "sporadic". I expect to discover a significant difference in those files.

[hassec]

Hi @Profoktor,

as @clanmills mentioned, lens recognition is a bit of a guessing game based on the metadata we find in the file.
The good news is that the lens recognition for pictures taken with canon cameras did get quite a bit better on the main branch with #1692.
But these changes didn't and likely won't make it into the 0.27.x release series.

Now that should hopefully answer your question as to what changed 😉

And as for why you are seeing different results for your different images in this particular case:
it is the different value of CanonCs.MaxAperture for the two images. This value is one of the values that exiv2 uses to guess the correct lens description.
The function that does this guessing work for these images and LensType=156 is:

exiv2/src/canonmn_int.cpp

Line 2862 in 1605fe5

std::ostream& printCsLensByFocalLengthAndMaxAperture(std::ostream& os,

And lastly since you also mention darktable, I just also want to reiterate, that @clanmills suggestion to use the configuration file is really an easy and straightforward fix for this until a future release of exiv2 can fix this 😉

Hope the above helps, but feel free to ping me again if you have any other questions!

[clanmills]

Ahhh, Merry Christmas @hassec. And Merry Christmas @Profoktor

There's a substantial difference in the code in src/canonmn_int.cpp between 0.27-maintenance and main. The function printCsLensTypeByMetadata() is now being used on 'main' for lens Canon lens "recognition/guess-work".

bdd8a386b src/canonmn_int.cpp (Christoph Hasse    2021-06-08 21:56:04 +0200 2737)

Thank You, Christoph for mentioning MaxAperture. It would have taken a lot more digging for me to discover this. However this is a significant difference between the exif_correct and exif_incorrect image - probably better called exif_different. Not sporadic. Exiv2 is deterministic. There are plenty of other differences:

.../main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g canon/i ~/Downloads/CR2/exif_incorrect.CR2 > /tmp/incorrect.txt
.../main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g canon/i ~/Downloads/CR2/exif_incorrect.CR2 > /tmp/incorrect.txt
.../main/build $ diff /tmp/correct.txt /tmp/incorrect.txt 
21,22c21,22
< Exif.CanonCs.MaxAperture                     Short       1  F4
< Exif.CanonCs.MinAperture                     Short       1  F27
---
> Exif.CanonCs.MaxAperture                     Short       1  F3.5
> Exif.CanonCs.MinAperture                     Short       1  F22
35c35
< Exif.Canon.FocalLength                       Short       4  70.0 mm
---
> Exif.Canon.FocalLength                       Short       4  28.0 mm
38c38
< Exif.CanonSi.MeasuredEV                      Short       1  12.50
---
> Exif.CanonSi.MeasuredEV                      Short       1  12.75
40c40
< Exif.CanonSi.TargetShutterSpeed              Short       1  1/128 s
---
> Exif.CanonSi.TargetShutterSpeed              Short       1  1/161 s
44c44
< Exif.CanonSi.CameraTemperature               Short       1  4 °C
---
> Exif.CanonSi.CameraTemperature               Short       1  6 °C
49c49
< Exif.CanonSi.SubjectDistance                 Short       1  18.58 m
---
> Exif.CanonSi.SubjectDistance                 Short       1  6.82 m
51,52c51,52
< Exif.CanonSi.ShutterSpeedValue               Short       1  1/140 s
< Exif.CanonSi.MeasuredEV2                     Short       1  12.25
---
> Exif.CanonSi.ShutterSpeedValue               Short       1  1/161 s
> Exif.CanonSi.MeasuredEV2                     Short       1  12.62
55c55
< Exif.CanonSi.AutoRotate                      Short       1  0
---
> Exif.CanonSi.AutoRotate                      Short       1  3
82c82
< Exif.CanonFi.FileNumber                      Long        1  100-0105
---
> Exif.CanonFi.FileNumber                      Long        1  100-0106
94c94
< Exif.CanonFi.MacroMagnification              SShort      1  136
---
> Exif.CanonFi.MacroMagnification              SShort      1  149
111c111
< Exif.Canon.MeasuredColor                     Short       5  10 496 1024 1024 552
---
> Exif.Canon.MeasuredColor                     Short       5  10 549 1024 1024 604
113c113
< Exif.Canon.VRDOffset                         Long        1  10210790
---
> Exif.Canon.VRDOffset                         Long        1  9622655
115,116c115,116
< Exif.Canon.ColorData                         Short     796  1 850 1024 ...
< Exif.Canon.CRWParam                          Short     11229  22458 4098 ...
---
> Exif.Canon.ColorData                         Short     796  1 850 1024 1024 ...
> Exif.Canon.CRWParam                          Short     11229  22458 4098 0 0 0 ...

As @hassec has said, it's unlikely that we'll back-port printCsLensTypeByMetadata() to 0.27-maintenance. We released 0.27.5 in October and no case was made to back port this change at that time.

I think your original assertion "Case Closed: fixed on main" is correct.

[Profoktor]

Hi @clanmills and @hassec,

Merry Christmas to you as well! And a happy new year, while we're at it 😄

@clanmills : yes, you're correct about the term "sporadic" - I just wanted to point out that it affects only a portion of the files. Maybe "scattered" or "isolated" would be more fitting.

@hassec : Thanks for the pointer! Actually, the culprit being the max/min apperture values makes sense, as they change with the focal length and hence might lead to a misinterpretation.

I'll also give the workaround a try (I manipulated the lensfun xml s.th. lens correction is possible, but helping exiv2 to identify the lens appears to be a more robust approach)

[clanmills]

@Profoktor. I would be helpful if you could ask LensFun/darktable to use Exif.CanonCs.LensModel which is an ascii string added to the metadata by Canon.

I'm not a photographer, so I don't know what LensFun does. However, I know darktable/LensFun (and other applications) want Exiv2 to identify the lens with 100% accuracy. The business of deducing the lens from the metadata is at best a guess. In this case, the data in Exif.CanonCs.LensModel is correct, so why do they use the "guessed" interpreted (-pt) value of Exif.CanonCs.LensType?

[Profoktor]

Hi @clanmills,

the answer to that is quite simple: Canon only provides this information for their own lenses. As soon as you use a third party lens, these fields are left empty.
Only the lens type is provided, which is a short and which exiv2 decodes into something meaningful (I suppose the id is provided by the lens itself, and collides between makes. Adding the fact that there is no indication of the lens maker as well, we end up at the wild guessing we have)

I attached the output for exiv2 -pv for reference:
exif_Tamron.txt
exif_canon.txt