Lens detection issue for Nikon d780 + Tamron 85mm 1.8 VC

[jcubic]

Describe the bug
I'm not sure if this is a bug. In #1270 I was told to use .exiv2 file to identify lenses but I have issue with Tamron 85mm 1.8 VC on Nikon d780.

The lens is identified as VR 85mm f/1.8E

To Reproduce

This file is generated by darktable, because I've had NEF files created for DarkTable White Balance (I've forget that I can generate small jpeg in camera) but it should not matter, I've got the same output for NEF file.

Steps to reproduce the behaviour:

This is the output of a exiv2 comment (I'm using version on Fedora 32):

$ exiv2 --grep lens/i file.jpg
Exif.Nikon3.LensType                         Byte        1  D G VR
Exif.Nikon3.Lens                             Rational    4  85mm F1.8
Exif.Nikon3.LensFStops                       Undefined   4  6.33333
Exif.Nikon3.LensData                         Undefined 108  48 56 48 48 5 245 141 25 51 230 101 60 50 216 67 164 0 247 93 93 208 207 150 146 51 31 168 32 120 180 214 223 202 155 81 236 108 209 27 74 94 87 53 248 160 45 159 246 50 83 89 68 20 201 99 226 70 143 189 208 200 165 103 14 154 11 97 156 188 193 171 122 46 199 69 168 240 29 47 38 2 195 105 244 100 185 243 18 22 255 205 128 24 149 247 63 106 123 113 76 12 177 59 170 254 55 85 88
Exif.Photo.LensSpecification                 Rational    4  850/10 850/10 180/100 180/100
Exif.Photo.LensMake                          Ascii       6  
Exif.Photo.LensModel                         Ascii      15  VR 85mm f/1.8E
Exif.Photo.LensSerialNumber                  Ascii      11
$ exiv2 --version
exiv2 0.27.99.0

There are no Exif.NikonLd3.LensIDNumber

This is the output of same lens on Nikon d750

$ exiv2 --grep lens/i file.jpg
Exif.NikonLd3.LensIDNumber                   Byte        1  229
Exif.NikonLd3.LensFStops                     Byte        1  F6.3
Exif.Nikon3.LensType                         Byte        1  D G VR
Exif.Nikon3.Lens                             Rational    4  85mm F1.8
Exif.Nikon3.LensFStops                       Undefined   4  6.33333
Exif.Photo.LensModel                         Ascii       4  229

Desktop (please complete the following information):

• OS: Fedora 32
• Compiler & Version [Gcc 7.3, Clang-6, MSVC 2017] default fedora package
• Compilation mode and/or compiler flags

Additional context
I'm not sure if this is issue with camera or something with exiv2. Maybe if Nikon d780 have different EXIF tags then exiv2 should use different approach to detect lenses using this camera.
Note: I will not be able to test this camera if the fix will be available, because I've had this camera only for a weekend.

[boardhead]

The sample image you posted has been edited by some software which has removed some metadata.

ExifTool gives this warning: "Can't decrypt Nikon information (no ShutterCount key)"

[clanmills]

Thank You to @jcubic for reporting this. And Thank You to @boardhead for your comments.

I've put 229 into ~/.exiv2 and life is good with Exiv2 v0.27.4.9 (the current development branch for 0.27.4). I've reproduced your report on 0.27.99.0 (current development branch of 'master').

566 rmills@rmillsmm-local:~/gnu/exiv2/team/book $ cat ~/.exiv2 
[nikon]
229=Tamron 85mm 1.8 VC
567 rmills@rmillsmm-local:~/gnu/exiv2/team/book $ exiv2 --grep lens/i ~/Downloads/107146721-2ee22100-694a-11eb-8577-a6c75375a72b.jpg 
Exif.Nikon3.LensType                         Byte        1  D G VR
Exif.Nikon3.Lens                             Rational    4  85mm F1.8
Exif.Nikon3.LensFStops                       Undefined   4  6.33333
Exif.NikonLd4.LensIDNumber                   Byte        1  Tamron 85mm 1.8 VC
Exif.NikonLd4.LensFStops                     Byte        1  F6.3
Exif.NikonLd4.LensID                         Short       1  0
Exif.Photo.LensSpecification                 Rational    4  850/10 850/10 180/100 180/100
Exif.Photo.LensMake                          Ascii       6  
Exif.Photo.LensModel                         Ascii      15  VR 85mm f/1.8E
Exif.Photo.LensSerialNumber                  Ascii      11  
568 rmills@rmillsmm-local:~/gnu/exiv2/team/book $

Can you checkout, build, install and try again with '0.27-maintenance'.

---

@boardhead I'm surprised by your comment. I see the same shutter data on with exiftool and exiv2:

581 rmills@rmillsmm-local:~/gnu/github/exiv2/master/build $ exiftool ~/Downloads/107146721-2ee22100-694a-11eb-8577-a6c75375a72b.jpg  | grep -i shut
Warning                         : Can't decrypt Nikon information (no ShutterCount key)
Shutter Mode                    : Electronic
Mechanical Shutter Count        : 4356
Shutter Count                   : 4554
Shutter Speed                   : 1/500
582 rmills@rmillsmm-local:~/gnu/github/exiv2/master/build $ bin/exiv2 -g shut/i ~/Downloads/107146721-2ee22100-694a-11eb-8577-a6c75375a72b.jpg 
Exif.NikonSi02xx.ShutterCount1               Long        1  0
Exif.NikonSi02xx.ShutterCount2               Undefined   2  4 0
Exif.NikonSi02xx.ShutterCount                Long        1  262148
Exif.Nikon3.ShutterCount                     Long        1  4554
583 rmills@rmillsmm-local:~/gnu/github/exiv2/master/build $ 

Curiously, 0.27-maintenance, reveals both 'Shutter Count' and 'Mechanical Shutter Count'.

583 rmills@rmillsmm-local:~/gnu/github/exiv2/master/build $ exiv2 -g shut/i ~/Downloads/107146721-2ee22100-694a-11eb-8577-a6c75375a72b.jpg 
Exif.NikonSi02xx.ShutterCount1               Long        1  0
Exif.NikonSi02xx.ShutterCount2               Undefined   2  4 0
Exif.NikonSi02xx.ShutterCount                Long        1  262148
Exif.Nikon3.ShutterMode                      Short       1  Electronic
Exif.Nikon3.MechanicalShutterCount           Long        1  4356
Exif.Nikon3.ShutterCount                     Long        1  4554
584 rmills@rmillsmm-local:~/gnu/github/exiv2/master/build $ 

[jcubic]

Will test latest version, in mean time while still have the camera two jpegs basic+small directly from d780, just in case one with electronic shutter in silent mode.

[clanmills]

@jcubic Thanks for sending the files. Both seem to be fine with exiv2 v0.27.4.9

594 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ exiv2 --version
exiv2 0.27.4.9

This program is free software; you can redistribute it and/or
...
595 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ cat ~/.exiv2 
[nikon]
229=Tamron 85mm 1.8 VC
596 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ 

591 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ exiv2 -g lens/i 107151333-504f0700-6962-11eb-9cc8-737ff49a5f9e.JPG 
Exif.Nikon3.LensType                         Byte        1  D G VR
Exif.Nikon3.Lens                             Rational    4  85mm F1.8
Exif.Nikon3.LensFStops                       Undefined   4  6.33333
Exif.NikonLd4.LensIDNumber                   Byte        1  Tamron 85mm 1.8 VC
Exif.NikonLd4.LensFStops                     Byte        1  F6.3
Exif.NikonLd4.LensID                         Short       1  0
Exif.Photo.LensSpecification                 Rational    4  850/10 850/10 180/100 180/100
Exif.Photo.LensMake                          Ascii       6  
Exif.Photo.LensModel                         Ascii      65  VR 85mm f/1.8E
Exif.Photo.LensSerialNumber                  Ascii      11  
592 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ curl -LO https://user-images.githubusercontent.com/280241/107151334-5218ca80-6962-11eb-8c1d-3fe87fb63b42.JPG
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  367k  100  367k    0     0  1202k      0 --:--:-- --:--:-- --:--:-- 1202k
593 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ exiv2 -g lens/i 107151334-5218ca80-6962-11eb-8c1d-3fe87fb63b42.JPG 
Exif.Nikon3.LensType                         Byte        1  D G VR
Exif.Nikon3.Lens                             Rational    4  85mm F1.8
Exif.Nikon3.LensFStops                       Undefined   4  6.33333
Exif.NikonLd4.LensIDNumber                   Byte        1  Tamron 85mm 1.8 VC
Exif.NikonLd4.LensFStops                     Byte        1  F6.3
Exif.NikonLd4.LensID                         Short       1  0
Exif.Photo.LensSpecification                 Rational    4  850/10 850/10 180/100 180/100
Exif.Photo.LensMake                          Ascii       6  
Exif.Photo.LensModel                         Ascii      65  VR 85mm f/1.8E
Exif.Photo.LensSerialNumber                  Ascii      11  
594 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ 

[hassec]

Hi, just wanted to add my 2 cents (while hopefully not only adding useless noise :D )

It seems that the LensData from the D780 is encoded as LensData version 8.0.
Using 0.27-maintenance as @clanmills did in the above comment will print:

Exif.NikonLd4.Version                        Undefined   4  8.00

Technically, one can even see that in @jcubic's first comment that contains:

$ exiv2 --grep lens/i file.jpg
...
Exif.Nikon3.LensData                         Undefined 108  48 56 48 48 5 245 141 25 51 230 101 60 50 216 67 164 0 247 93 93 208 207 150 146 51 31 168 32 120 180 214 223 202 155 81 236 108 209 27 74 94 87 53 248 160 45 159 246 50 83 89 68 20 201 99 226 70 143 189 208 200 165 103 14 154 11 97 156 188 193 171 122 46 199 69 168 240 29 47 38 2 195 105 244 100 185 243 18 22 255 205 128 24 149 247 63 106 123 113 76 12 177 59 170 254 55 85 88
...
$ exiv2 --version
exiv2 0.27.99.0

The beginning of the LensData is not encrypted and contains the version of the LensData, here you can see the first 4 values are
48 56 48 48 which via the ASCII table can be translated to 0 8 0 0 ---> Version 8.0 ;)

Exiv2 only knows about this version since the merging of #1437.
So if you want to use exiv2 for pictures taken from the D780, you unfortunately have to build the 0.27-maintenance branch for now.

[clanmills]

Well said, @hassec What you are saying is totally correct. I didn't say that the change in 0.27.4.9 is #1437 on which you and I cooperated. This team-work stuff really works.

[boardhead]

@clanmills Yes. Interesting. The problem is that something has changed reordered the maker note entries of the first sample image so they are no longer in sequence. The result is that ExifTool doesn't find the ShutterCount when it pre-scans the makernote IFD because it only scans as far as necessary, assuming ordered tag ID's. The second and third samples do not have this problem.

[clanmills]

That's interesting. Yes in tvisitor.cpp, I have a little mechanism in ReportVisitor::visitTag() to defer decoding the Nikon Encrypted data until I know three items:

if ( image.serial_ && image.shutterCount_ && image.lensSize_ ) ...

Exiv2 code has a class TiffFinder : TiffVisitor for searching an ifd. I think that was designed to sniff Exif.Image.Make in IFD0 to ensure that Make is known before we parse makernote. Andreas Huggel is a very good engineer and did an excellent job to think out solutions to everything. So, he uses TiffFinder when decoding nikon files to determine the serial number and shutterCount up front.

My enduring contribution to Exiv2 is to have documented everything and hope that @hassec, @kmilos and others will continue to work on Exiv2 in future.

Thank You everybody for a quick and useful discussion about this issue.

@jcubic I believe we've answered your question and, if you agree, please close the issue.

[jcubic]

Yes, thanks and this is really good job everyone.