Leica M10 raw file (DNG) analysis

As usual when new Leica cameras come out, I took a quick look inside some DNGs from one of Leica's new M10 cameras. Usually, there's not much to see with Leica DNGs - they are typically text-book vanilla DNGs. But with the Leica M10, things are bit more interesting.

The first thing to note is that I obtained sample DNGs from three sources; these three sources will become important later. The sources were:

From the DPReview site.
The "performance proofs" from the official Leica site.
Some shots from a demo camera, shot by "Tobers".

What's in the DNGs

Firstly, let's talk about what's in the files:

There is no version number for the camera firmware in the majority of files. In those where a version number does exist, it is 1.0.1.0. Having a firmware version number isn't a requirement, but is certainly considered to be best practice. All previous Leica cameras did have a firmware version number.
The camera name shows as "LEICA M10"
The image data is 14-bit.
All of the files use lossless JPEG compression
The DNG version is 1.4, with a "backward version" of 1.3. This is the same as most current generation Leica cameras.
In the DNGs I looked at, there are no "opcode" lens corrections, as seen, for example, in the Leica Q.
In all files, there is a embedded named DNG camera profile, or at least parts of one. The profile is named "LEICA M10 ". Note the large number of trailing spaces.
Unusually, the DNGs all contain 3 different JPEG preview images in addition to the main raw image, one of 1440x960, one of 160x120, and finally a full sized preview of 5952x3968. Having the full sized preview is particularly odd, as it takes up a lot of space. In the approximately 25-30 MB files I've seen, the full size preview typically takes up about 1.8 MB. I discuss the possible reasons for this later.
The color matrix doesn't appear very similar to that from the Leica SL. So the likelihood is that speculation that the sensor is the same as that in the SL is probably incorrect.
The profile has a "Looktable". Looktables allow color transforms. For more info see my blog posts on the subject. However, it's a very strange looktable - see below.
There is EXIF data for the lens, which is new. However, it's broken - see below.
All the files had XMP data in them, but what that was there varied - see below.

What's not in the DNG

Some users will be disappointed that there's not an estimated as-shot aperture (or f-stop) in the DNG. Leica have flipped-flopped on this - back in the days of the M8, the estimated aperture wasn't written to the DNG, but could be calculated from the "blue dot" data in the makernotes. I set up CornerFix to read that information, and add the estimated aperture to the DNG, and later Adobe added the ability to read the "blue dot" information to Lightroom as well. In later models, Leica relented, and added the estimated aperture to the DNG. But now it seems to have gone again. And no, the "blue dot" information is no more, so CornerFix won't help you. The information is probably hiding in the makernotes somewhere, but it will take some detective work to root out.

The strange looktable

ProfileLookTableDims: Hues = 3, Sats = 2, Vals = 1
ProfileLookTableData:
h [ 0] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
h [ 0] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
h [ 1] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
h [ 1] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
h [ 2] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
h [ 2] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
ProfileLookTableEncoding: Linear

This is the looktable. Perfectly formed and valid, but does nothing at all. So what's it doing there?

The broken lens EXIF data

One of things that I do to all new DNGs is to run them through Adobe's DNG validation tool. The M10's threw the following warning:

*** Warning: Possible MaxApertureValue conflict with LensInfo ***

The reason for this can be found in how the lens shows in the LensEXIF tag:

24.0 mm f/248.0

In reality, the lens is a Summilux-M 1:1.4/24 ASPH.

Lots of (different) XMP data

As mentioned above, the various image files contained different XMP data. Firstly, by way of background, XMP stands for Extensible Metadata Platform (XMP). XMP is an ISO standard, originally created by Adobe, to allow metadata to be embedded in images, PDFs, etc. Unsurprisingly, Adobe's DNG format explicitly supports XMP data, although it's very rarely found in camera generated images. It is very often found in images that have been processed by Adobe Lightroom or Photoshop. Both Lightroom and Photoshop use XMP to store adjustments, etc directly to DNG files. So for example, in a DNG that has been processed through Lightroom, you might find something like:

XMP: crs:Temperature="3500"
XMP: crs:Tint="+24"
XMP: crs:Saturation="0"
XMP: crs:Sharpness="29"

This sets the sharpness setting to 29, etc. For a "normal" raw file, Adobe stores this information in separate "sidecar" files.

Lightroom also leaves a history of what's been done to an image in the XMP, so for example:

XMP:
XMP:
XMP: XMP: stEvt:action="converted"
XMP: stEvt:parameters="from image/dng to image/dng"/>
XMP: XMP: stEvt:action="saved"
XMP: stEvt:instanceID="xmp.iid:691e10f8-fb5a-4bc0-8465-cca0e0319c13"
XMP: stEvt:when="2017-01-11T11:52:07-08:00"
XMP: stEvt:softwareAgent="Adobe Photoshop Lightroom 6.8 (Macintosh)"
XMP: stEvt:changed="/metadata"/>
XMP: XMP: stEvt:action="saved"
XMP: stEvt:instanceID="xmp.iid:e90503e6-47da-4388-943e-4643d6307c35"
XMP: stEvt:when="2017-01-11T16:05:29-08:00"
XMP: stEvt:softwareAgent="Adobe Photoshop Camera Raw 9.8 (Macintosh)"
XMP: stEvt:changed="/metadata"/>
XMP:
XMP:

This (ostensibly) shows that the image in question was converted by Adobe Lightroom 6.8. So it's not something you'd expect to find in a camera generated DNG, unless that DNG had been somehow processed though Lightroom. But in fact:

The Lightroom "fingerprints" above are present in the M10 DNG files from both DPReview, and Leica's own performance proofs.
But the images shot on the demo camera don't have Lightroom fingerprint in them. They do have an XMP placeholder, basically an empty XMP header, but no Lightroom adjustment information, etc

It is also the Lightroom fingerprint that is responsible for version numbers not being present in most files; in the files where the fingerprint is present, the "Software" tag that usually holds the Leica version number has been overwritten with "Adobe Photoshop Lightroom 6.8 (Macintosh)".

Lightroom/Photoshop noise reduction settings vary depending on camera settings

The Lightroom/Photoshop settings will be what's loaded when one of these files is imported into Lightroom for the first time. Note also that these settings can vary depending on camera settings. So, for example, noise reduction setting appear to vary by ISO (or perhaps some other camera setting):

For an 800 ISO image:

XMP: crs:LuminanceSmoothing="0"
XMP: crs:ColorNoiseReduction="25"

But for an 6400 ISO image:

XMP: crs:LuminanceSmoothing="25"
XMP: crs:ColorNoiseReduction="25"

So Lightroom is being told to do stronger noise reduction at higher ISOs. Of course, other Lightroom setting may also vary depending on other camera settings.

And then there's the lens profile.....

Just confuse things a bit further, the XMP data that has the Lightroom "fingerprints" in it also has lens profile information, for example:

XMP: crs:LensProfileSetup="LensDefaults"
XMP: crs:LensProfileName="Adobe (Leica SUMMILUX-M 35 mm f/1.4 ASPH.)"
XMP: crs:LensProfileFilename="Leica Camera AG (Leica SUMMILUX-M 35 mm f1.4 ASPH.) - RAW.lcp"

By way of background, in the world of DNGs and Adobe Lightroom/Photoshop, there are two separate way that lens corrections can be applied:

By opcodes embedded into DNGs. This has been extensively used by Leica for their point and shoot cameras, e.g., the Leica Q, as well as the Leica SL. But never for M class cameras.
Via Adobe lens profiles, applied via Lightroom or Photoshop. Generally these have to be installed on the computer that LR/Photoshop is running on. It's this kind of profile information that's embedded in the XMP - not actual lens correction data, but simply a reference to a file that contains the lens correction data.

However, this is a strange thing to put into an camera generated DNG. You can't really be sure that the named profile will be installed, and having a lens profile in the XMP will probably result in that profile being loaded by default, which not all users will be happy with. Also, it will only have an effect if the user is using Adobe software. But given that the profile matches the lens actually on the camera, this seems to be deliberate on Leica's part.

So what's going on here?

So all of this begs the question of what's going on here. It is the case that the M10's DNGs don't show the usual Leica perfectionism - rather they look as if firmware development for the M10 was rushed, and not much quality control was done. It's possible, although I think unlikely, that both DPReview and Leica posted images that actually had been modified by Lightroom in exactly identical ways. But some elements of what I've discussed here clearly are deliberate. I certainly don't know what's in Leica's head, but here are some hypotheses. Aka., guesses:

Firstly, I think it's very likely that both the full size preview image and the "Lightroom XMP data" are there to support the Leica M App:

The full sized preview image is there to allow the user to see the image at full size, even if at relatively low quality. Bear in mind that even today there are only a handful of apps on the App store, e.g., PhotoRaw, that can reliably do a raw conversion on an iPhone or iPad.
The "Lightroom XMP" data is almost certainly primarily there to enable one of the Leica M App's features - being able to rate images and have those ratings transfer to Lightroom. The technique of "fooling" Lightroom by injecting what looks like a complete set of Lightroom adjustments was used for the same purpose by e.g., Photosmith.
In addition to enabling ratings, the "Lightroom XMP" is also used to setup Lightroom into what Leica seem to consider to be an optimal configuration, with ISO dependent noise reduction, the lens profiles, etc.

Secondly, I think a possible reason why there are some image files with the "Lightroom XMP" information, and some without, may relate to how the files were transferred and/or what camera setting were used. Quite possibly the Lightroom XMP info either is only added if the file is transferred by WIFI/the Leica app but is not added if the image is transferred via an SD card, or is only added if the image is rated. In this regard it's notable that the creation and modification dates in the DNGs are sometimes different, strongly implying that the XMP was injected at some time after capture. Hopefully once the camera becomes available, someone will be able to do some testing and verify exactly what's happening. But there may be other explanations for this as well. Notably there may be different firmware versions in the wild - bear in mind, most files don't have a firmware version code, so there's no way to tell whether we're comparing the same firmware revision.

Why is there a lens profile, noise reduction settings, etc embedded in the Lightroom XMP data? Well, I guess someone though it was a convenience. My view however is that's its likely to cause more trouble than it's worth.

Finally of course, all of these difference may just be bugs in the firmware. However, I think that's unlikely. For example, the fact that the lens profiles match to the lens attached tends to indicate that at least some of this is deliberate. But of course, as soon as a new firmware version comes out, all this may change. And Leica typically introduce new firmware versions within a month or two of a camera's initial introduction.

If and when someone gets hold of a camera long enough to do some tests, I'll update this post.

Updated: I've created a tool to clean up the M10 DNGs. This will remove the full size preview and the adjustments in XMP.

Note: This post has been updated with new information since it was first published.

Recent
Date
Label
Author

All apps now natively support Apple Silicon

Dec 23rd

Major updates for AccuRaw EXR, AccuRaw Monochrome and PhotoRaw

Sep 28th

pcdMagic for Windows in now available on the Windows Store

Jun 1st 2

Leica CL raw file (DNG) analysis

Dec 10th 2

What is color?

Oct 21st

The Leica M10 and dynamic range revisited

May 29th

dcpTool for the Mac is now available

Feb 26th 11

How to remove the lens corrections from any raw image

Feb 19th

Cleaning Leica M10 DNGs

Feb 19th

AccuRaw has become AccuRaw EXR

Feb 18th 1

Fuji GFX 50S compressed raw support is now available

Feb 18th

AccuRaw and PhotoRaw now have Fuji GFX 50S support

Jan 30th

The Leica M10 and Dynamic Range

Jan 29th 2

Leica M10 raw file (DNG) analysis

Jan 21st 6

Non-clipping workflows are important for EXR images

Jan 18th

AccuRaw EXR's highlight control curves

Jan 9th

The AccuRaw EXR beta is now available!

Jan 8th

Olympus OM-D E-M1 II raw support in AccuRaw and PhotoRaw

Nov 6th

Canon EOS 5D Mark IV support in AccuRaw and PhotoRaw

Sep 7th

Fuji compressed raw support for the Fuji X-Pro2 in AccuRaw and PhotoRaw

May 22nd 9

Nikon D5 support in AccuRaw and PhotoRaw

Apr 1st 4

The Leica M262 sensor

Jan 2nd 1

Mini comparative review : Eight email apps for Inbox Zero

Dec 14th

CornerFix - still going strong

Dec 12th

Two great articles from Photosmith's developers

Nov 21st

How much lens correction is there on the new Leica SL?

Oct 29th 5

Leica SL (Typ 601) raw file (DNG) analysis

Oct 21st 2

Affinity Photo is out of beta

Jul 10th

Leica Q (Typ 116) raw file (DNG) analysis

Jun 11th 22

Leica M Monochrom (Type 246) raw file (DNG) analysis

May 3rd 2

NDSR Boston chooses pcdMagic for Kodak Photo CD conversion

Feb 10th

PhotoRaw 4.2.1 is available with lots of new camera support

Feb 6th

The iOS 8 raw+jpeg bug is fixed

Nov 26th

dcpTool and the DNG SDK on Linux

Nov 16th 1

AccuRaw support for the Canon 7D II, SX60 HS, G7 X and Fuji X100T

Nov 13th 2

Mini Comparative Review of AccuRaw, Lightroom, Capture One and RawTherapee

Nov 2nd

Don't update to iOS 8

Oct 9th

Nikon D750 support for AccuRaw and PhotoRaw

Oct 5th

AccuRaw and AccuRaw Monochrome 2.2.1 - Nikon D810, Sony A7S, A77 M2, etc support

Jul 31st

PhotoRaw 4.0.7 - Nikon D810, Sony A7S, A77 M2, etc support

Jul 20th

Leica T (Typ 701) raw file (DNG) analysis

Apr 25th 4

Importing raw images into Lightroom Mobile

Apr 15th 3

Adobe Lightroom Mobile and Lossy DNG

Apr 8th 2

The 64-bit version of PhotoRaw is out

Dec 28th

How fast is the iPad Air for image processing, and does NEON make sense?

Dec 10th

iPad Air, iPhone 5S and 64-bit NEON code

Dec 7th

AccuRaw Monochrome and true monochrome processing

Dec 3rd 2

AccuRaw 30% off for the black Friday weekend

Dec 1st

Lens resolution, software correction and raw converters

Nov 6th

"The AccuRaw conversion does the best job..."

Nov 1st 2

Perfect skin tones with dcpTool, hue twists and camera calibration

Oct 30th 7

PhotoRaw and iOS 7

Sep 18th

Adobe to bring Lightroom-style photo editing to tablets

May 3rd 3

Lightroom 4.4RC and Capture One versus the X-Trans sensor

Mar 2nd 14

X-Pro raw conversions from AccuRaw Beta 5

Dec 27th 31

PhotoRaw makes the New York Times

Dec 6th

dcpTool and "V4" camera profiles

Dec 3rd 3

Demosaicing the Fuji X-Pro1 Part 4

Nov 14th 8

Lightroom 4.3 rc and the Fuji X-Pro

Nov 10th 2

Lightroom 4.2 and the Fuji X-Pro - oh dear.....

Oct 3rd 8

HSV conversion in a CIFilter

Sep 13th

I haven't posted for a while - I've been busy coding. Something new in the works that should be available in a month or two. But I do want to quickly publish something I haven't seen documented before, which is how you can do RGB to HSV conversions in a CIFilter. HSV (Hue, Saturation, Value) representations are useful in a number of image manipulations, and CIFilters are part of Apple's image processing in OS X and iOS, and use a reduced version of the OpenGL shader language. In general, HSV conversion in OpenGL shader language are a problem, because of the conditionals, which shader language aren't good at. Some RGB<->HSV conversions in Open GL shader language have been published before, but all the ones I've seen use functions that aren't available in the subset of the shader language that is available in a CIFilter. This implementation takes some ideas about the use of the "step" function from Ian Taylor's "normal" shader language implementation. The CIFilter below adjusts exposure and saturation by converting to HSV, adjusting S and V, and converting back to RGB. You can test this by just pasting it into Quartz Composer. Of course, if all you want to do is to adjust exposure and saturation, there are easier ways than HSV - this is just a demo! Anyway, without more talk:

A much faster PhotoRaw

Jun 21st

The Fuji X-Pro1, Lightroom and SILKYPIX

Jun 7th 14

CornerFix and a Leica Fotografie International Master Shot

Jun 1st

Lightroom 4.1 and the Fuji X-Pro1 - oh dear.....

May 30th 11

The Leica M Monochrom's lack of DNG compression options

May 28th

The Leica M Monochrom's spectral sensitivity

May 25th

Demosaicing the Fuji X-Pro1 Part 3

May 19th 19

Raw developers and camera manufacturers

May 18th 5

Analysis of the Leica M Monochrom's raw file

May 17th

Jono Slack over on the L-Camera-Forum was kind enough to post a raw DNG from Leica's new M Monochrom, so I a took a bit of a root around inside it. No surprises - it's very similar to a M9 DNG. For those interested, there's a field level dump of the interesting fields below, but the highlights are: 14 Bit data, much the same as an M9, with non-zero black level Uncompressed DNG version 1.0.0.0, so still the original DNG spec, none of the new stuff Camera name specified as "M9 monochrom" The MakerNote with its lens info still seems to be as for the M9; in this case the lens shows as "Apo-Summicron-M 75mm f/2 ASPH", which is consistent with what Jono reported The famous "blue dot" is still there The only thing that might surprise a few people is the "PhotometricInterpretation: LinearRaw" part. But that's actually quite correct - the way the DNG spec works, you can either set that to CFA (aka a Bayer array type camera) or to LinearRaw. And this sure isn't a CFA camera. There is one slight side effect of LinearRaw though. When ACR or Lightroom load a normal raw, they apply a tone curve by default. However, with a LinearRaw, they don't. So, for those intent on comparing a M9 image to a M Monochrom image shot side-by-side, be aware that by default they have different tone curves. I'd guess that once the M Monochrom is shipping ACR and LR will have a built-in M Monochrom camera profile that will probably have a tone curve. The other issue to be aware of with LinearRaw is that most raw development programs don't support it, so until Aperture, Capture One, etc are updated, don't expect M Monochrom DNGs to load in much except Adobe products.

Demosaicing the Fuji X-Pro1 Part 2

May 17th 7

Demosaicing the Fuji X-Pro1 and its X-Trans CMOS sensor

May 16th 17

The Leica M Monochrom sensor

May 15th 5

CornerFix in New Zealand

May 8th 1

Sony NEX-7, CV12 and CornerFix

Apr 10th

What does "raw image support" really mean on the iPad?

Apr 10th

A useful gadget for users of M lenses

Apr 8th

The 2012 iPad - looking good for photographers

Mar 20th

A guide to photo editing apps on the iPad

Mar 12th

iPhoto for the iPad and raw files

Mar 10th

Lightroom's new lossy DNG compression

Jan 11th 2

Installing Linux Mint 12 and Samba shares

Jan 7th 2

Adobe DNG profile editor will updated

Dec 30th 2

Adobe LR 3.6 and V4 profiles - Oh dear......

Dec 17th

CornerFix To The Rescue

Dec 7th

TOP finally gets it....

Nov 24th

A short review of PhotoRaw

Oct 23rd

ETTR for the fourth time........

Aug 4th 3

CornerFix and OS X 10.7 Lion

Jul 6th

So the new Leica M9 firmware is out.......

Jun 29th

Photoshop Touch is coming to PhotoRaw

May 10th

Good comparison of iPad raw image processing apps

May 6th

Samsung BX2450 mini review for imaging

Dec 21st 4

Using CornerFix with the Fuji IS-Pro

Oct 27th

Revised and refreshed Chromasoft website

Oct 18th

Solving the dreaded Gigabyte "Won't boot from USB" problem

Oct 15th 85

Installing Ubuntu 10.10 "Maverick Meerkat"

Oct 14th 3

New version of CornerFix and a new website

Oct 14th

pcdMagic for Windows is out

Oct 14th

dcpTool and the dangers of tables

Oct 10th

View comments

scott kirkpatrickJanuary 21, 2017 at 7:15 AM
Leica has in the past (M8, M9, etc) tended to leave tidying up the EXIF to the very last stage/lowest priority while they work out the operational glitches that the first customers are only too glad to complain about. So I am not surprised to find stuff that looks like it was left over from development in these early files. My concern is that the invocation of lens profile files might be a reference to stuff that only LR can supply from its subscription-updated version, thus locking out other software. And making it trickier for Leica to go back to opcodes that everyone can use.

scott
SandyJanuary 21, 2017 at 7:18 AM
Scott, yes, the lens profiles are an odd decision.
scott kirkpatrickJanuary 21, 2017 at 1:36 PM
Version number appears in the line "software = 1.0.1.0" in Tobers files. But it is obscured by the Adobe overwrite in the Leica downloads and the DPReview files. Maybe 1.0.1.0 is a firmware version ID.
0Replies
sunwebJanuary 21, 2017 at 3:27 PM
Hi Sandy,
Thank You so much for this. You say the color matrix is so different from SL that it means this may be a new sensor. What sorts of sensor design changes would cause this?

Could it simply be a very different curve in the IR cut coverglass?

Any suggestions as further reading on this subject?

TY so much,
Charlie
SandyJanuary 21, 2017 at 8:19 PM
The primary driver of the matrix is the characteristics of the dyes used in the Bayer array. In addition, the effects of IR filters, the actual sensitivity of the "naked" sensor, etc all come into it.

Leica M10 raw file (DNG) analysis

What's in the DNGs

What's not in the DNG

The strange looktable

The broken lens EXIF data

Lots of (different) XMP data

Lightroom/Photoshop noise reduction settings vary depending on camera settings

And then there's the lens profile.....

So what's going on here?

View comments

ChromaSoft

Digital imaging and building the software that makes digital imaging happen