1. Updated: AccuRaw and PhotoRaw now have support for compressed and uncompressed RAFs.

    For those that have been asking me about AccuRaw and Fuji GFX 50S support, I'm pleased to say that the version on the App Store as of today has support, although only for uncompressed RAFs. Compressed support will follow over the next few weeks. But now you can take a close look at GFX raw files using AccuRaw.

    Of course, the new support also applies to both AccuRaw Monochrome, and PhotoRaw. At the same time, the new version also adds support for other new cameras, including the Fuji X100F, the Fuji X-T20, the Nikon D5600, and the Leica M10
    0

    Add a comment

  2. I've been reading various articles, posts, etc on the web that deal with Leica's new M10. Many of those suggest that the improvement in dynamic range from the Leica M240 to the M10 is of the order of 1.5 to 2 stops. But I find that difficult to agree with.

    I would guess the improvement in dynamic range between the M10 and the M240, just by eyeballing the published images, to be closer to 0.5 stops than 1.5-2.

    Which raises an interesting question - why the huge discrepancy, when everybody is looking at the same images?

    What is Dynamic Range?

    To someone like me, with seven years of engineering school, the definition of dynamic range is deceptively simple - for an image it's (in non-mathematical terms) the difference between the lightest and darkest levels in which detail is discernible. Usually, dynamic range is measured in stops, or EVs.

    So the problem is???????????

    Actually, there are two problems. Firstly, there's a major problem of definition, and then there's a minor problem of measurement.

    For simplicity, I'll deal with the minor problem first:

    The minor problem of measurement

    The minor problem of measurement is a fairly simple one - what exactly does "detail is discernible" actually mean?

    By way of example, does that mean you could recognize a black cat against a black background? Or a white cat against a black background? Well, as it happens, there is a standard, although (regrettably) it has nothing to do with cats, black or white. For engineers, the most commonly used definition is that "detail is discernible" means that the signal to noise ratio is one. Now you should note that a cat at a signal to noise ratio of one doesn't actually look much like a cat. Or anything. From the folks at Point Grey:

    Now there are a whole host of additional complications - for example:
    • Color images have red, green and blue channels that saturate at different levels - so does detail vanish when one channel is saturated, or all channels?
    • Images in cameras are digitized, and digitization creates it own noise. Are you including that or not? Also, are you including amplifier noise, etc?
    • Human perceptions of signal to noise ratios aren't too good. Notably, you can use dithering (basically, adding carefully controlled amounts of noise) to make a picture look better. Technically, this increases the perceived signal to noise ratio. So sometimes, when you see a signal to noise ratio quoted, it's "perceptual".
    On the last point about dithering, one of the reasons, although not the most important, why I called Adobe's lossy DNG compression an "engineering abomination" a few years ago was that it clips images to 8 bits, and then uses dithering to make the images look better. 

    So there are good reasons why you could have different people measure the dynamic range of a camera, and they could come to somewhat different answers. By way of example, I'll contrast two different sources of dynamic range measurements for cameras: 
    • DxOMark. DxOMark is well known in the photographic community - basically, loved and hated by equal measure. I have my issues with what DxOMark does - mainly that they're not very transparent about how they get to their measurements, and some measurements don't make a whole lot of sense. But they do apply a consistent set of measurements that at least provide a common basis for comparison that you won't get from the camera manufacturers. Or most camera sites either.
    • Sensorgen.info. Sensorgen.info is the imaging nerd's site - it has information that people that work with image sensors want to see, such as quantum efficiency and saturation capacity.
    Here's a table of what both sites say about the dynamic range of some cameras:


    Firstly, you'll note that DxOMark is consistently 0.6 to 1 stops higher than Sensorgen. Next, those with a technical background will notice something odd - the DxOMark result for the D810 is 14.8. But, for a camera with a 14-bit A/D such as the D810, that's impossible, at least if you're using the standard definition of dynamic range, because the maximum dynamic range achievable would be 14. My guess is that one reason for the difference between the two sites is probably that DxOMark is trying to remove the effects of A/D resolution, and just look at the sensor as an analog device. They may also be trying to compensate for the presence of other sources of noise, e.g., in the amplifiers. That would remove the quantization noise element, and give a consistently higher reading for dynamic range across the board. But if anyone knows exactly how DxOMark do their measurements, I'd like to hear from them. 

    But what is reasonably consistent is the difference between cameras. So, e.g., DxOMark view the M240 as 1.6 stops better than an M9 versus Sensorgen at 1.4 stops. For what it's worth, my experience from when I've had the opportunity to test cameras in an optics lab is that Sensorgen is closer to the readings that I get. Of course, I'm measuring the whole package, not trying to isolate just the sensor.

    So, bottom line - there are legitimate technical reasons why not everyone will agree about exactly what the dynamic range of a given camera is. But those differences tend to be more about what the absolute number is, rather than the relative performance of cameras. So while DxOMark and Sensorgen disagree about whether the dynamic range of an M240 is 12.5 or 13.3, they both agree that it's dynamic range is at least a full stop better than the M9, and at least half a stop worse than a Sony A7R.

    The Big Problem of Definition 

    But the differences in measurement don't explain why my guess of the M10's dynamic range relative to the M240 was so far from many individuals with a great deal of experience of Leica's M cameras. Some additional research led me to believe that many of those individuals were concluding that the M10 was usable at 1.5-2 stops higher ISO than the M240. "Usable" here being defined as being able to produce a print that was commercial usable without extensive post-processing. Based in large part on the 1.5-2 stops higher usable ISO, they were then concluding that the M10 had 1.5-2 stops of additional dynamic range. Unfortunately, that's not a conclusion that you can safely draw.

    The reason why an increase in usable ISO doesn't necessarily imply the same increase in dynamic range goes back to the definition of how dynamic range is measured, which is based on whether detail is discernible within noise. But how "usable" an image is in the view of a photographer takes a lot more than just noise into account - it's an assessment of the entirety of image quality. In the specific case of the M240 vs. the M10 the culprit is probably banding. Now banding does contribute to noise, but perhaps less than many might think. Banding is usually lighter and darker bands across the image, usually numbers of pixels wide.  But within those bands, detail is still discernible. It's actually only on the transition between bands, the edges, that banding makes discerning detail more difficult, and contributes to noise. So the effect of banding on a noise measurement, and hence on dynamic range, is less than you might guess. But banding, even light banding, can quickly make an image unusable.

    This then is the problem of definition - there's a general assumption that improvements in dynamic range and improvements in high ISO performance are effectively the same thing. That's not the case - for a "perfectly designed camera", a camera where the only source of image quality imperfection was simple sensor noise, that would be true. But in real cameras, quite frequently there are other sources of defects in image quality. As soon as those other sources come into play, then the direct equivalence between dynamic range and improved high ISO performance is no longer valid.

    Conclusion

    So, in summary, here's the explanation:  It's probably quite reasonable to say the the M10 has a 1.5-2 stop advantage over the M240 at higher ISO. But that doesn't mean that the M10's dynamic range is 1.5-2 stops better than the M240, simply because the M240's practical usability to a photographer is mostly limited at higher ISO by banding, not by simple noise performance.

    So what should you take from all this? Given the issues with dynamic range as applied to cameras above, my best recommendation is that, unless you are actually in an optics lab, just avoiding mentioning the phrase "dynamic range" is probably the best way to go.

    Note: this post has been updated since it was first published.

    Also Note: There's an update here, with the actual measured M10 dynamic range.
    2

    View comments

  3. As usual when new Leica cameras come out, I took a quick look inside some DNGs from one of Leica's new M10 cameras. Usually, there's not much to see with Leica DNGs - they are typically text-book vanilla DNGs. But with the Leica M10, things are bit more interesting.

    The first thing to note is that I obtained sample DNGs from three sources; these three sources will become important later. The sources were:
    1. From the DPReview site.
    2. The "performance proofs" from the official Leica site.
    3. Some shots from a demo camera, shot by "Tobers".

    What's in the DNGs

    Firstly, let's talk about what's in the files:
    1. There is no version number for the camera firmware in the majority of files. In those where a version number does exist, it is 1.0.1.0. Having a firmware version number isn't a requirement, but is certainly considered to be best practice. All previous Leica cameras did have a firmware version number.
    2. The camera name shows as "LEICA M10"
    3. The image data is 14-bit.
    4. All of the files use lossless JPEG compression 
    5. The DNG version is 1.4, with a "backward version" of 1.3. This is the same as most current generation Leica cameras.
    6. In the DNGs I looked at, there are no "opcode" lens corrections, as seen, for example, in the Leica Q. 
    7. In all files, there is a embedded named DNG camera profile, or at least parts of one. The profile is named "LEICA M10              ". Note the large number of trailing spaces. 
    8. Unusually, the DNGs all contain 3 different JPEG preview images in addition to the main raw image, one of 1440x960, one of 160x120, and finally a full sized preview of 5952x3968. Having the full sized preview is particularly odd, as it takes up a lot of space. In the approximately 25-30 MB files I've seen, the full size preview typically takes up about 1.8 MB. I discuss the possible reasons for this later.
    9. The color matrix doesn't appear very similar to that from the Leica SL. So the likelihood is that speculation that the sensor is the same as that in the SL is probably incorrect. 
    10. The profile has a "Looktable". Looktables allow color transforms. For more info see my blog posts on the subject. However, it's a very strange looktable - see below.
    11. There is EXIF data for the lens, which is new. However, it's broken - see below.
    12. All the files had XMP data in them, but what that was there varied - see below.

    What's not in the DNG

    Some users will be disappointed that there's not an estimated as-shot aperture (or f-stop) in the DNG. Leica have flipped-flopped on this - back in the days of the M8, the estimated aperture wasn't written to the DNG, but could be calculated from the "blue dot" data in the makernotes. I set up CornerFix to read that information, and add the estimated aperture to the DNG, and later Adobe added the ability to read the "blue dot" information to Lightroom as well. In later models, Leica relented, and added the estimated aperture to the DNG. But now it seems to have gone again. And no, the "blue dot" information is no more, so CornerFix won't help you. The information is probably hiding in the makernotes somewhere, but it will take some detective work to root out.

    The strange looktable

    ProfileLookTableDims: Hues = 3, Sats = 2, Vals = 1
    ProfileLookTableData:
    h [ 0] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
    h [ 0] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
    h [ 1] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
    h [ 1] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
    h [ 2] s [ 0]: h= 0.0000 s=1.0000 v=1.0000
    h [ 2] s [ 1]: h= 0.0000 s=1.0000 v=1.0000
    ProfileLookTableEncoding: Linear
      This is the looktable. Perfectly formed and valid, but does nothing at all. So what's it doing there?

    The broken lens EXIF data

    One of things that I do to all new DNGs is to run them through Adobe's DNG validation tool. The M10's threw the following warning:
    *** Warning: Possible MaxApertureValue conflict with LensInfo ***
    The reason for this can be found in how the lens shows in the LensEXIF tag:

    24.0 mm f/248.0
    In reality, the lens is a Summilux-M 1:1.4/24 ASPH.

    Lots of (different) XMP data

    As mentioned above, the various image files contained different XMP data. Firstly, by way of background, XMP stands for Extensible Metadata Platform (XMP). XMP is an ISO standard, originally created by Adobe, to allow metadata to be embedded in images, PDFs, etc. Unsurprisingly, Adobe's DNG format explicitly supports XMP data, although it's very rarely found in camera generated images. It is very often found in images that have been processed by Adobe Lightroom or Photoshop. Both Lightroom and Photoshop use XMP to store adjustments, etc directly to DNG files. So for example, in a DNG that has been processed through Lightroom, you might find something like:
    XMP:    crs:Temperature="3500"
    XMP:    crs:Tint="+24"
    XMP:    crs:Saturation="0"
    XMP:    crs:Sharpness="29"
    This sets the sharpness setting to 29, etc. For a "normal" raw file, Adobe stores this information in separate "sidecar" files.

    Lightroom also leaves a history of what's been done to an image in the XMP, so for example:
    XMP:      
    XMP:    
    XMP:      XMP:       stEvt:action="converted"
    XMP:       stEvt:parameters="from image/dng  to image/dng"/>
    XMP:      XMP:       stEvt:action="saved"
    XMP:       stEvt:instanceID="xmp.iid:691e10f8-fb5a-4bc0-8465-cca0e0319c13"
    XMP:       stEvt:when="2017-01-11T11:52:07-08:00"
    XMP:       stEvt:softwareAgent="Adobe Photoshop Lightroom 6.8 (Macintosh)"
    XMP:       stEvt:changed="/metadata"/>
    XMP:      XMP:       stEvt:action="saved"
    XMP:       stEvt:instanceID="xmp.iid:e90503e6-47da-4388-943e-4643d6307c35"
    XMP:       stEvt:when="2017-01-11T16:05:29-08:00"
    XMP:       stEvt:softwareAgent="Adobe Photoshop Camera Raw 9.8 (Macintosh)"
    XMP:       stEvt:changed="/metadata"/>
    XMP:    

    XMP:    
    This (ostensibly) shows that the image in question was converted by Adobe Lightroom 6.8. So it's not something you'd expect to find in a camera generated DNG, unless that DNG had been somehow processed though Lightroom. But in fact:
    • The Lightroom "fingerprints" above are present in the M10 DNG files from both DPReview, and Leica's own performance proofs.
    • But the images shot on the demo camera don't have Lightroom fingerprint in them. They do have an XMP placeholder, basically an empty XMP header, but no Lightroom adjustment information, etc
    It is also the Lightroom fingerprint that is responsible for version numbers  not being present in most files; in the files where the fingerprint is present, the "Software" tag that usually holds the Leica version number has been overwritten with "Adobe Photoshop Lightroom 6.8 (Macintosh)".

    Lightroom/Photoshop noise reduction settings vary depending on camera settings

    The Lightroom/Photoshop settings will be what's loaded when one of these files is imported into Lightroom for the first time. Note also that these settings can vary depending on camera settings. So, for example, noise reduction setting appear to vary by ISO (or perhaps some other camera setting):

    For an 800 ISO image:
    XMP:    crs:LuminanceSmoothing="0"
    XMP:    crs:ColorNoiseReduction="25"
    But for an 6400 ISO image:
    XMP:    crs:LuminanceSmoothing="25"
    XMP:    crs:ColorNoiseReduction="25"
    So Lightroom is being told to do stronger noise reduction at higher ISOs. Of course, other Lightroom setting may also vary depending on other camera settings.

    And then there's the lens profile.....

    Just confuse things a bit further, the XMP data that has the Lightroom "fingerprints" in it also has lens profile information, for example:

    XMP:    crs:LensProfileSetup="LensDefaults"
    XMP:    crs:LensProfileName="Adobe (Leica SUMMILUX-M 35 mm f/1.4 ASPH.)"
    XMP:    crs:LensProfileFilename="Leica Camera AG (Leica SUMMILUX-M 35 mm f1.4 ASPH.) - RAW.lcp"
    By way of background, in the world of DNGs and Adobe Lightroom/Photoshop, there are two separate way that lens corrections can be applied:
    1. By opcodes embedded into DNGs. This has been extensively used by Leica for their point and shoot cameras, e.g.,  the Leica Q, as well as the Leica SL. But never for M class cameras.
    2. Via Adobe lens profiles, applied via Lightroom or Photoshop. Generally these have to be installed on the computer that LR/Photoshop is running on. It's this kind of profile information that's embedded in the XMP - not actual lens correction data, but simply a reference to a file that contains the lens correction data.
    However, this is a strange thing to put into an camera generated DNG. You can't really be sure that the named profile will be installed, and having a lens profile in the XMP will probably result in that profile being loaded by default, which not all users will be happy with. Also, it will only have an effect if the user is using Adobe software. But given that the profile matches the lens actually on the camera, this seems to be deliberate on Leica's part.

    So what's going on here?

    So all of this begs the question of what's going on here. It is the case that the M10's DNGs don't show the usual Leica perfectionism - rather they look as if firmware development for the M10 was rushed, and not much quality control was done. It's possible, although I think unlikely, that both DPReview and Leica posted images that actually had been modified by Lightroom in exactly identical ways. But some elements of what I've discussed here clearly are deliberate. I certainly don't know what's in Leica's head, but here are some hypotheses. Aka., guesses:

    Firstly, I think it's very likely that both the full size preview image and the "Lightroom XMP data" are there to support the Leica M App:
    • The full sized preview image is there to allow the user to see the image at full size, even if at relatively low quality. Bear in mind that even today there are only a handful of apps on the App store, e.g., PhotoRaw, that can reliably do a raw conversion on an iPhone or iPad. 
    • The "Lightroom XMP" data is almost certainly primarily there to enable one of the Leica M App's features - being able to rate images and have those ratings transfer to Lightroom. The technique of "fooling" Lightroom by injecting what looks like a complete set of Lightroom adjustments was used for the same purpose by e.g., Photosmith. 
    • In addition to enabling ratings, the "Lightroom XMP" is also used to setup Lightroom into what Leica seem to consider to be an optimal configuration, with ISO dependent noise reduction, the lens profiles, etc.
    Secondly, I think a possible reason why there are some image files with the "Lightroom XMP" information, and some without, may relate to how the files were transferred and/or what camera setting were used. Quite possibly the Lightroom XMP info either is only added if the file is transferred by WIFI/the Leica app but is not added if the image is transferred via an SD card, or is only added if the image is rated. In this regard it's notable that the creation and modification dates in the DNGs are sometimes different, strongly implying that the XMP was injected at some time after capture. Hopefully once the camera becomes available, someone will be able to do some testing and verify exactly what's happening. But there may be other explanations for this as well. Notably there may be different firmware versions in the wild - bear in mind, most files don't have a firmware version code, so there's no way to tell whether we're comparing the same firmware revision.

    Why is there a lens profile, noise reduction settings, etc  embedded in the Lightroom XMP data? Well, I guess someone though it was a convenience. My view however is that's its likely to cause more trouble than it's worth.

    Finally of course, all of these difference may just be bugs in the firmware. However, I think that's unlikely. For example, the fact that the lens profiles match to the lens attached tends to indicate that at least some of this is deliberate. But of course, as soon as a new firmware version comes out, all this may change. And Leica typically introduce new firmware versions within a month or two of a camera's initial introduction.

    If and when someone gets hold of a camera long enough to do some tests, I'll update this post.

    Updated: I've created a tool to clean up the M10 DNGs. This will remove the full size preview and the adjustments in XMP.

    Note: This post has been updated with new information since it was first published.

    6

    View comments


  4. In a previous post introducing AccuRaw EXR, the new version of my AccuRaw raw processing software, I mentioned the importance of a "non-clipping workflow" for the promise of the EXR format to be a reality. One of the biggest advantages of the ILM's OpenEXR format is that image highlights aren't clipped, no matter how badly overexposed. This makes manipulating images in multiple tools a lot easier. However, it's not enough to have a image format that doesn't clip highlights - the whole workflow need to avoid clipping highlights.

    The need not to clip highlights is why adding EXR support to AccuRaw was more than just a matter of adding an new export format. The core processing engine, and some of the controls, had to change to avoid clipping.

    The easiest way to show the difference is with a practical example. The image below, figure 1, is a crop of a shot that I took in La Serena, Chile a number of years ago (while sitting out a weather related flight delay).


    Fig. 1 : The original image with no adjustments.
    The second image, figure 2, is the result of deliberately overexposing the image in AccuRaw, in this case by adjusting the "ETTR" slider. The ETTR slider, as AccuRaw users will know, makes it's adjustments right at the beginning of AccuRaw's image processing pipeline. This allows exposure offsets in the original raw image to be compensated for without introducing any non-linearities - using the exposure slider for this risks a non-linear end result, as the exposure adjustment occurs later in the processing pipeline. But in this case, I'm using the slider to show that:
    1. AccuRaw EXR's processing engine is non-clipping, and
    2. AccuRaw EXR's processing engine is highly linear.


    Fig. 2 : The image, 3 stops overexposed.
    The next images are screenshots from Blackmagic Design's Fusion 8 software package. Fusion 8 is a compositing tool that's extensively used in the movie industry. For example, it can be used to composite a still image, taken e.g., by a DSLR, into video footage. In this case, I'm using it because it has full (non-clipping!) support for EXR. Here I should point out that while a fair number of apps have some level of EXR support, not many of them are non-clipping. Apple's Preview, found on all Mac's, for example, will read EXR's, but clips any highlight above 1 when images are opened. (It also doesn't properly support EXR chromaticity tags.)

    Fig. 3 : The original and adjusted image in Fusion 8.
    The first Fusion screenshot, figure 3, simply shows the original image and the overexposed image, with any adjustments in Fusion. For those who know Fusion, the Fusion workflow is configured for linear light operation, with the two image viewers in the top half of the screen configured with a LUT to show an "as will be seen" image.


    Fig. 4 : Image recovery in Fusion 8, using EXR format images, and a non-clipping work flow. The images are effectively indistinguishable.

    The second Fusion screenshot, figure 4, shows the results of using Fusion "BC" tool to recover the overexposed image. As can be seen, the recovered and original image are identical. This shows both AccuRaw's linearity, and that AccuRaw doesn't clip highlights.

    Fig. 5 : Attempted image recovery in Fusion 8, from a JPEG image. The inability to recover highlights in a clipping workflow is clear.
    By way of contrast, the final screenshot, figure 5, shows what happens when exactly the same recovery operation is attempted on a clipped image, in this case a JPEG image, which is inherently a clipping format. The difference is stark.






    0

    Add a comment

  5. Now that the AccuRaw EXR beta is out, one of the questions I'm getting is how the new highlight controls work.

    The control curves are below. Note that these curves are on a linear scale, and 1 is white. So anything above 1 is "blown".

    • The new curves are intended to preserve the value of blown highlights in floating point EXR files, so as to allow highlight manipulation, even recovery, in processing after the image is exported from AccuRaw. So contrary to the highlight controls on a conventional raw developer which focus on compressing all values to less than 1, these curves concentrate on giving a smooth roll off for values above 1.
    • The Knee Low setting controls the point at which the highlight compression begins.
    • The Knee High setting controls how much highlight compression occurs.
    • If Knee High is set to its minimum, 3.5, then the control curve is effectively linear. As Knee High is increased from it's minimum, the curve becomes flatter and flatter at higher values.










    0

    Add a comment

  6. AccuRaw EXR is a new version of AccuRaw with support for HDR (High Dynamic Range) images in OpenEXR format. OpenEXR is a high dynamic-range (HDR) image file format developed by Industrial Light & Magic for use in computer imaging applications that uses floating point data. It's been used extensively in the movie industry, on major feature length movies. 

    But the new version of AccuRaw isn't just the addition of a new output format - AccuRaw EXR builds on AccuRaw's highly linear raw image processing engine with new features specifically for HDR and OpenEXR:


    • End-to-end linear floating point processing.
    • Non-clipping workflow that supports the preservation of "greater than 1" highlights in EXR formats.
    • Multiple EXR formats available - RGB or RGBA, 16- or 32-bit floating point formats

    Most raw image processing applications don't support the export of EXR images. The reason is in the "non-clipping workflow". For HDR and OpenEXR to make sense, a raw developer has to support output to EXR of what in other formats would be blown highlights, highlights with magnitudes much greater than 1. So real EXR output isn't just a matter of an additional export format, it's a also a matter of having a processing engine that won't clip highlights anywhere in the processing chain. Practically, that also means a fully floating point engine.

    So while AccuRaw may look the same, it's changed a lot under the hood. Firstly, the processing engine has changed to be fully floating point. Previously, the engine was hybrid - floating point in critical areas, but integer in some parts. Secondly, AccuRaw now has new "Cinematic" Highlight and Shadow controls. The reason for this is that old controls implicitly assumed a clipping workflow. The new tools enable a smooth log compression curve for blown highlights that's designed to be compatible with how EXR images are processed in typical post-production tools. The new tools are optional however - you can revert to the previous tools in AccuRaw's Preferences if you prefer them.

    The beta can be downloaded here.

    I'll blog further over the next few days about the new controls, and why a non-clipping workflow is important for EXR.

    Oh, and the upgrade will be free to existing AccuRaw users!

    0

    Add a comment

Popular Posts
Blog Archive
About Me
About Me
My Photo
Author of AccuRaw, PhotoRaw, CornerFix, pcdMagic, pcdtojpeg, dcpTool, WinDat Opener and occasional photographer....
Loading