ChromaSoft

Lightroom's new lossy DNG compression

2012-01-11T09:47:00.000-08:00

Adobe's Lightroom 4 beta is out, and among it's other enhancements, it features a new lossy compression scheme for DNG images. LR has always has lossless compression, which could usually reduce image size by up to half. However, it's not been used much, except when you used any of Adobe Camera Raw, Lightroom or the Adobe DNG converter to convert an existing raw image to DNG. The lossless compression scheme hasn't, so far as a I know, ever been used in-camera, presumably because it's quite CPU intensive.

Eric Chan was kind enough to give some details on the new lossy scheme, which I'll summarize here, together with my views. You can read the full exchange on the Adobe forums in this thread.

NOTE: This post is based on Eric's comments - the specification isn't yet available.

What is the new scheme?

The new scheme basically works as follows:

It uses JPEG DCT compression - exactly the same as a normal JPEG image.
The data is 8-bit, same as a normal JPEG.
Data is demosaiced, also the same as a regular JPEG.

So up till this point, a lossy DNG is pretty much the same as a regular JPEG. But there's more:

The clever bits

The new scheme has some clever bits that make a lossy DNG better than any normal JPEG. Those bits are:

What Adobe call "1D perceptual mapping". What that actually is, is a tone curve, just like the sRGB tone curve in JPEG images, but a clever tone curve. 8-Bit pixel values range from 0 to 255. What Adobe are doing is to analyze the image data, and optimize the curve to the image data. So, for example, if your image doesn't have significant highlights, a "normal" JPEG image might only use values from 0 to 210, and not use 211 to 255. The new clever curve will spread the pixel values in the image over the full range of 0-255. Better still, this is done for each color channel individually. And because information on the new curve is in the image, that can by undone as soon as the image is loaded again.
Dithering is used to prevent banding. With 8-bit values, there's the danger of banding, especially if a lot of post processing get done on the image. Dithering, basically adding noise (very carefully controlled and shaped noise!), helps to smooth out any banding.
The image stays in the camera's color space, rather than being compressed into the space defined by the JPEG primaries. Pretty much all modern cameras can represent colors that a standard JPEG image can't. Now there are some downsides to this - a wider color space being represented in 8-bit values can result in color banding, so we're going to have to see whether that turns into an issue or not. I'd say probably not, given the use of dithering, but we'll see over the next few months.
The image remains scene referred. This is important for white balance. JPEG images have a fixed white balance of 6500k, so a white balance transform needs to be applied before they are created. That makes it difficult (not impossible, but less easy to do well) to adjust white balance later than if the data is in its "as shot" form.

Why have Adobe designed it this way?

Back when lossy compression was a just a rumor, I predicted that if lossy compression happened, it would be JPEG based. See this thread on LuLu. It's nice to be right, even if only occasionally(!)

The reason, in my opinion, is that a lot of camera chips have hardware acceleration for JPEG DCT built in to support JPEG out, so doing JPEG DCT lossy compression is "costless" in terms of CPU utilization for many modern cameras.

Analysis

So, where are we on this:

Firstly, to state the obvious, if you use the new lossy compression, your file is no longer raw. The new lossy compression works by first demosiacing the image, then compressing. And that's irreversible. So, if at some time in the future, a better demosaicing algorithm comes out, you won't be able to take advantage of it.
Secondly, the compression scheme is, and I can't put this any other way, an engineering abomination if your starting point was Bayer matrix type raw data, which maybe 99% of raw image data is. Here's why: When the image is demosaiced from the raw Bayer data, basically what you're doing is adding two thirds more data that is actually redundant. What demosaicing does is to interpolate, via an algorithm, two thirds of the pixels in an image. But these interpolated pixels are redundant data - they can be recreated from the original raw data just by recalculating. What is then compressed is the original raw data, plus all of this redundant data. So to get, according to Eric's data in the thread above, about a 50% reduction is data size relative to lossless compression (70% relative to no compression), you're actually firstly multiplying the data size by three, then compressing. So, what you're doing is a factor of ten JPEG compression to get a factor of between two (relative to lossless compression) and three (relative to no compression) data size reduction. All because of the two thirds of data that your compressing is actually redundant. That's just horrible.
The quality of Adobe lossy DNG images are going to be very implementation dependent. In other words, there will be good implementations and bad implementations. There is no single way to do JPEG compression; a good JPEG compression engine will look at the data, and chose the best compression parameters for the image. Similarly, there are good and bad ways to do dithering. Finally, a simple way to implement the "1D perceptual mapping" would just to have a single fixed tone curve. Add together a not so great JPEG compression engine, not so great dithering, and some cutting of corners on the "1D perceptual mapping", and you have a recipe for really bad image quality. I have no doubt that Adobe implementation will be good. But I also suspect that if lossy DNG compression makes it into cameras, there will be some pretty bad implementations.

Conclusion and recommendation

Bottom line, a lossy DNG is a JPEG that's been somewhat tweaked to allow easier image adjustment later on. Which is no bad thing, but let's be clear - lossy DNG is an alternative to JPEG, not an alternative to raw.

If, in some future camera, you get the choice of either a JPEG image, or a lossy DNG image, you should almost certainly choose the lossy DNG. The data compression and sample size are the same, and there are advantages to be had from the perceptual mapping, color space and the scene referred nature of the image, as discussed above. In addition, if you want to convert an existing JPEG into DNG, e.g., because your workflow is standardized around DNG, then lossy DNG may be a good idea. You won't lose much quality, and alternate ways of converting from JPEG to DNG results in far larger files than the original.

However, if you're converting an already existing raw image to DNG, don't use lossy compression! Chances are, you'll get similar compression from lossless compression, you won't risk JPEG compression artifacts, and you'll be able to benefit from better demosaicing algorithms in the future. Even if you didn't get the same compression, my view would be that given the low cost of storage today, the trade-off just wouldn't be worth it.

Updated 13 Jan: after I wrote this post, Eric Chan, who is certainly in a good position to know, responded to a question about using lossy DNG compression as follows: "To be honest, I'm not really comfortable with the idea of lossy compressed DNG for archival storage purposes". Which probably sums it up well.

Installing Linux Mint 12 and Samba shares

2012-01-07T05:01:00.000-08:00

So I decided to change to Mint from Ubuntu for my (fairly minimal) Linux needs because I haven't managed to get used to Unity. It may look cool, but it's a pain to use.

The good news is that installation was easy; none of the issues I had last time I installed Ubuntu. Other than of course the usual Gigabyte motherboard won't boot from USB issue I described (with the solution) in a previous post.

When I started to configure the system however, I did run into a nasty problem getting Mint to see Windows (Samba) shares on the network. I could set up Samba fine, and other machine could see the Mint machine and its shares, but the Mint machine just wouldn't see shares on either my Windows or OS X machines.

Anyway, after several hours of searching forums, here's the answer: you need to add the following line to the smb.conf file on the Mint machine, in the [global] section:

name resolve order = bcast host lmhosts wins

Unless this is in, Samba on Mint doesn't look for other Samba shares, and you never see anything.....

Adobe DNG profile editor will updated

2011-12-30T03:30:00.000-08:00

Some good news from Adobe - in a previous post, I reported that the new Adobe "V4" camera profiles broke the Adobe profile editor, a tool a lot of photographers depend on to get accurate color. Well, Eric Chan, who is on Adobe's Camera Raw team, has gone on record that the profile editor will be updated. See here.

Adobe LR 3.6 and V4 profiles - Oh dear......

2011-12-17T07:10:00.000-08:00

Adobe Lightroom 3.6 and Adobe Camera Raw 6.6 final is now out, with new fourth generation "V4" camera profiles. Unfortunately, it turns out that the V4 profiles break dcpTool. Here's the story:

For a while now, the Adobe folks, notably Eric Chan, have been working away at building better camera profiles. Theses profiles are intended to address a problem with some cameras, when used with earlier generation profiles, that could result in highlight banding. This was especially with some of the high-end Nikons e.g., the D3, D700, D300, etc. Back in January, Eric posted a set of beta profiles, called the "V3" profiles that addressed the banding issue. However, they required that exposure be offset by half a stop in Lightroom/ACR. A bit inconvenient, and not really what you want in a production environment.

With Lightroon 3.6/ACR 6.6 Adobe have introduced a new set of "V4" profiles that sort out the banding issues that older profiles had, but don't require the manual half stop offset. The good news is that most people seem to like the new profiles a lot.

The bad news is that the new profiles break both dcpTool, and Adobe's Profile Editor. The reason for this that the "V4" profiles include some new tags that neither product understands. This was confirmed by Eric in a thread on the Adobe forums. Also, I took a look at the profiles, and found three new tags:

Exif 0xc7a4 : 1
Exif 0xc7a5 : -0.5
Exif 0xc7a6 : 1

The middle tag, rather notably, is exactly the half stop manual offset that the old V3 profiles required, so I'd guess that basically this just automates the old offset procedure.

Digging around, it turns out that the problem with dcpTool and the new profiles was noticed by Vit Novak, but he didn't contact me, and I missed the post in which he mentioned the problem.

Unhappily, the new tags are not yet documented by Adobe, and are not yet part of the DNG SDK. Eric has said that Adobe are "working on" a new DNG specification (presumably to be 1.4) and a new version of the SDK. But no word on an expected date.

I'll update dcpTool once the new specs/SDK come out.

The more interesting question is whether Adobe will update the Profile Editor. Now, a lot a people depend on the Profile Editor, but it's always been a beta release, not production software, and it's not been updated since it was introduced. So I'd not recommend holding your breath for an update.

UPDATE: The profile editor will be updated - see this post.

CornerFix To The Rescue

2011-12-07T03:55:00.001-08:00

Luminous Landscape have been testing the Sony NEX-7, and found that when using the NEX-7 with non-Sony lenses, it's "CornerFix To The Rescue" .

TOP finally gets it....

2011-11-24T02:44:00.001-08:00

Looks like the "mainstream" photographic sites are finally starting to "get it" as regards ETTR. Ctein, one of the better knows columnists on The Online Photographer, has published 'Expose to the Right' is a Bunch of Bull.

Welcome to the new converts to the cause! Better late than never :)

A short review of PhotoRaw

2011-10-23T03:20:00.000-07:00

Here a short review of PhotoRaw: RAW Konverter fürs iPad

"With PhotoRaw I've now found one app that does the job satisfactorily.
The 7.49 EUR each for the app is absolutely value."

It's in German, but easy to translate and view via Google translate.

ETTR for the fourth time........

2011-08-04T07:52:00.000-07:00

Umm, yes, scary but true.

Michael Reichmann has a new article on ETTR. It's called Optimizing Exposure, and you can find it on the Luminous Landscape site. The article itself doesn't have anything new - it's just a rehash of Michael's old arguments, "levels", etc, but it is vintage MR stuff, with references to important and highly competent people that Michael knows and that support what he's saying. Only he never actually mentions their names!!!

There's no point in my going over the ETTR stuff again - those who want a point-by-point commentary on what Michael wrote can look in the LL forum discussion of the article - other have already pointed out all the issues.

But two interesting things did come out of the article. Firstly, Michael makes the point that exposure on modern DSLR's in actually pretty primitive. Divorced from all the ETTR mumbo-jumbo, it's a valid point, and worth taking a look at the article for.

The second thing of interest that came out was an additional potential reason for using ETTR.

Quick recap - in my previous ETTR posts, I showed that ETTR was mostly a waste of time except in very particular circumstances - e.g., to effectively synthesize a lower ISO than your camera comes equipped with.

However, some of the folks in the forum pointed to an article by Emil Martinec of the University of Chicago, "The Consequences of Noise". It's fairly technical, and quite theoretical, but here's a short summary as it applies to ETTR:

Firstly, Emil destroys Michael Michael Reichmann's "levels" argument for ETTR. This is what he says:

"In particular, the idea that the benefit of ETTR comes from the "number of available levels" suggests that the image quality would be one stop worse for the ISO 1600 image than it is for the ISO 3200 image, since by being one stop down from the right edge of the histogram, fully half the available levels are not being used. However, as we have seen, noise is much more than two levels in all exposure zones at these ISO's, so the extra levels used in the ISO 3200 image simply go into digitizing the noise, and are thus of no benefit in improving image quality. In fact, the quality of the two images will be very nearly the same (rather than one stop different)"

Sound familiar? It's the same answer I came to previously.

But then Emil goes on to make the point that that noise in a digital image actually comes from more than just the sensor, it also comes from the electronics around it (the amplifiers, etc). This is known as "read noise". In a modern camera, generally the read noise is quite low, and you're only really worried about the sensor noise. But Emil's argument is that you can minimize the read noise by shooting at higher ISO. Specifically he says that:

"Somewhat counter-intuitively, for fixed aperture/shutter speed, it is best to use the highest possible ISO"

Now that might sound somewhat crazy when you hear it the first time. But think of it this way:

Sensor noise is basically fixed for any combination of aperture and shutter speed (the point I made in the previous posts on ETTR, and the reason why you should just adjust ISO, not mess with ETTR). If the same number of photons hit the sensor in the same time, you're going to have the same sensor noise.
However, because of the gain of the amplifiers in the electronics, effective read noise (the noise from the electronics) is lower at high ISO.

So there you have it - there is another situation in which ETTR will benefit you in addition to the ones I laid out in the previous posts. That situation is the one in which the read noise (the noise from the camera electronics rather than the sensor) is a significant portion of the image noise.

The question, of course, is does that happen in practice? ETTR has a number of downsides, such as the color shifts I showed previously, so you won't want to use ETTR unless there's a real advantage.

The answer to that question unfortunately depends on your camera. Personally, I've never seen a practical shooting situation with a modern camera where read noise was an issue relative to sensor noise. But I can't rule out it happening.

LATE EDIT: So, ok, here's the proof - as predicted by the theory above - courtesy of Guillermo Luijk. For some cameras, going to higher ISO really can reduce noise for the same shutter speed/aperture. However, note that it took four stops of overexposure (ISO 100 to ISO 1600) on an old Canon 350D to get Guillermo's results, and Guillermo specifically says that many cameras don't do this at all. Interesting, but in my view, four stops is a bit impractical.

CornerFix and OS X 10.7 Lion

2011-07-06T12:04:00.000-07:00

For those planning to be on the bleeding edge and upgrading to OS X 10.7 Lion, please be aware that there is a new version of CornerFix, version 1.4.2.0, available. Earlier versions may crash with Lion.
You can update now - the new version is compatible with all version of OS X from 10.4 onwards.

There are also two other upgrades (which apply to the Windows version as well):

Fix for an issue that could result in Lightroom displaying an out-of-date thumbnail if the image had been loaded into Lightroom before processing with CornerFix;
Lens detection for the new Leica lenses.

You can download from the usual place.

Enjoy.

Sandy

So the new Leica M9 firmware is out.......

2011-06-29T07:10:00.000-07:00

.....and, not of course that I would say "I told you so", but well, I told you so. As predicted in this post (yes, all the way back in Feb 2010), Leica have managed to significantly improve "red edge"/"Italian flag syndrome", but not completely eliminate it in all cases, even for modern Leica lenses. Take a look at the various posts on the Leica User Forum for the details.

On predictions, a second prediction I made at the time has also turned out - take a look at this interview with Dr. Andreas Kaufmann. "Q: Which is the best performing Leica product line? Dr.Kaufmann: Definitely the M9."

Unfortunately, the jury is still out on third prediction I made at the time - that the Leica S2, while a fine camera, would be a financial bust for Leica. As it turns out, I was at least partially right - the S2 hasn't significantly penetrated the pro market. However, I'm told that it has racked up significant sales in the amateur market. Given the price of the S2 and a lens or two, that tells you something about how much some amateurs have to spend on their hobby. Are those sales significant enough to pay for the program? Well, we'll find out eventually.

Photoshop Touch is coming to PhotoRaw

2011-05-10T08:34:00.000-07:00

Photoshop Touch is Adobe's latest addition to Photoshop's capabilities - basically, it adds one-touch image transfer from an iPhone or iPad to Photoshop. You can read about it on this page.

Good comparison of iPad raw image processing apps

2011-05-06T07:03:00.000-07:00

Rob Galbraith has a good comparison of the various raw processing apps for the iPhone and iPad (including PhotoRaw) in his article on the iPad 2.

Specifically, he lays out the trade-offs between PhotoRaw and the other guys - which is that PhotRaw takes a long time to import images because it does as much as possible up front, but then provides real-time adjustments to exposure, etc, whereas the other guys import faster, but then don't allow real-time adjustments. First time I've seen a clear discussion of the differences.

Update Dec 2011 - Note that while Rob's article is still interesting, the version of PhotoRaw now available has come a LONG way since Rob wrote his review, addressing several of his concerns such as white balance adjustments, etc. Best thing to do is to try the free version PhotoRaw, PhotoRaw Lite.

Samsung BX2450 mini review for imaging

2010-12-21T06:11:00.000-08:00

I've been looking to get a second 24 inch LCD monitor to connect up to my main Mac system as a secondary screen in a dual monitor setup. While my HP LP2475W will remain the my primary screen, but I wanted to be able to get reference material (manuals, documents that I responding to by e-mail, etc) up on a separate screen. It would also allow using Lightroom in dual screen mode, although I've never really felt the need.

It so happens that I found a very good deal on a Samsung BX2450 from an on-line store, with free next-day delivery. The Samsung BX2450 is a new-generation "green" display with an LED backlight. A quick internet search revealed that the BX2450 got high marks from almost all owners. The only negatives I found was fiddly touch sensitive buttons, and that there is a known incompatibility between the BX25450 and the ATI (now AMD) graphics driver on Windows if you connect the BX2450 up with a HDMI to HDMI cable (the BX2450 doesn't have any DVI inputs, just HDMI). Apparent if you use a DVI to HDMI cable, there isn't a problem. But, for my purposes that was fine - mostly I work on a Mac anyway, and if I do need to connect to one of my Windows boxes, using a DVI cable is fine. So I went ahead and bought the BX2450.

The rest of this post is my experiences in using and calibrating the BX2475. It's written very much from an imaging and photographic perspective. Those interested in gaming performance - response times and the like - should look elsewhere.

First impressions and construction

First impressions were positive - the BX2475 looks good, and has a stand that, while it looks a bit fragile, actually seems to support the display as well as as my (much) more expensive HP unit. The BX2450 also appears to to live up to its green credentials - the power supply is external, and about the size of an iPhone. In fact, the whole display is light in weight - it was easy to carry around, and get in position on the desk.

Initial power up was fine; the BX2450 immediately showed as a second screen under OS X, with the correct resolution, etc.

The bad news: you can't adjust the height of the display. At all. There however is a tilt function.

The second piece of bad news is that all the software is Windows only. Not a single piece of Mac software - not even a display profile.

The "just be aware of" news - the BX2450 appears to essentially be an HDMI TV, rather than a computer monitor, even if you set it to "PC" mode. Now that doesn't make a lot of difference but be aware that there are some funnies: you will see the resolution listed as 1080p, rather than 1920x1080, and the display seems to identify itself as a HDMI TV rather than anything else. So you will see an "Overscan" checkbox in the Mac Display preferences which you won't for a normal monitor. Don't try to switch that off, btw - the display will no longer show a picture(!) This identification issue probably also explains the Windows HDMI issues that others have found.

Calibrating the display

Although the default setting on the display were ok, certainly better than some other displays I've seen, the first thing I looked at was how to calibrate the display. Now I didn't buy the BX2450 to display images; that's what the wide gamut LP2475 is there for. But as a I spend a lot of time looking at images, and specifically looking at the color rendition of images, bad color bothers me. I was also interested in how accurate a display I could get from a low-cost LCD display with LED backlighting. So I got out my trusty X-Rite Eye-One calibration device, and set to work. Here's the quick summary of my experiences:

First thing that I had forgotten was the X-Rite's Eye-One Match 3 software doesn't actually support calibrating to two monitors. Now you'll see a lot a stuff written on the internet that this is because "you can't have two calibrated displays on one display board because there's only one LUT in hardware", etc. Nonsense. Any modern OS will happily let you run two different calibrated displays from one video board. This is just a limitation of the Match 3 software. Solution: temporarily disconnect the primary monitor, and use the secondary as the primary while you calibrate. OS X will automatically keep track of which profile applies to which monitor, btw.
Second thing that I found out was that the touch sensitive buttons on the BX2450 really are as fiddly as claimed.
Third thing I found is that the BX2450 (or maybe the combination of the BX2450 and the Eye-One puck) doesn't like to be calibrated to a D55 (5500K) white point. While you can calibrate to that, the resulting display is truly horrible; the brighter parts of the image are ok, but the darker parts get a massive red color cast - totally unusable. So I ended up calibrating to a D65 white point.
Fourth thing I found out was that setting any of the Red, Green or Blue controls to more than 50 is not a good idea; when you generate a profile you get nasty discontinuities in the gamma curves, indicating that the display becomes non-linear. The process I followed was firstly to set the BX2450 to what Samsung call a "Cool" color tone, which was close to D65. Then I manually adjusted to D65 as measured by the Eye-One using the RGB sliders, taking care that no color control went above 50. For reference, the settings I ended up with were Red:20 Green:20 Blue:50
As with most displays, the default brightness was way to high; I set it to 34 on the slider, which corresponds to a 110 cd/m² setting, my preferred LCD value.

The end result of this was pretty good. It's almost impossible to give any idea of color rendition on the web, but visually comparing the BX2450 to my LP2475, the BX2450 stood up pretty well for images with relatively small color gamuts. Specifically, for sRGB images, which are what almost all JPEG files use for a color space, the calibrated BX2450 looked nearly as good as the LP2475. Interestingly, to my eyes, the difference was more a slight lack of contrast in the shadows, than any major difference in color rendition.

The BX2450's gamut

The problem for the BX2450 of course, and the reason why the LP2475 costs several times more, is gamut - how broad a spectrum of colors can be displayed. A quick look at some raw images rather than JPEGs from my Nikon and Sony cameras showed a very different story; the LP2475 was able to display better reds and greens.

Here are the gamuts of the LP2475, the Samsung BX2475, and the sRGB gamut for reference.

Bottom line - the BX2450 doesn't come anywhere near the gamut of the LP2475, especially as regards greens, but also in reds. Compared to the sRGB gamut, the BX2450 is actually pretty close, but is just a bit less capable on both the greens and reds. Which is consistent with what I found looking at actual images - sRGB images are ok, raw images that have wide color gamuts, not so ok.

Conclusion

The bottom line is the following: if your imaging needs are limited to looking at JPEG images, or web surfing, the BX2450 will serve you fine, and if you have a hardware calibration device, you can get quite acceptable color if you stick to a D65 white point. Also, the BX2450 will be a good choice for those that create websites, or artwork that will primarily be seen on the web - in that case the similarity of the BX2450's gamut to sRGB is actually an advantage.

In the role that I'm using the BX2450, and given what I paid for it, I've very happy - it does exactly what I bought it for, and does better than I had anticipated in terms of color accuracy.

However, if you're a serious photographer that deals with raw images, the BX2450 is not for you, at least as a primary monitor. Firstly, the color gamut isn't wide enough. But the real show stopper is the issues I had trying to calibrate the BX2450 to a non-D65 white point. That may not be the fault of the BX2450 - it may be that my Eye-One calibration device isn't up to dealing with LED backlights. But for whatever reason, if you're serious out photography, you need a monitor that will calibrate reliably with current calibration devices.

Using CornerFix with the Fuji IS-Pro

2010-10-26T21:58:00.000-07:00

David Kennard has posted an interesting article and mini-tutorial on using CornerFix with the Fuji IS-Pro camera. The IS-Pro is a specialist camera that is sensitive to both infrared and ultraviolet light - in fact, rather than the usual four channel Bayer array sensor, the IS-Pro's sensor has eight channels in a 2x4 arrangement, requiring that DNG files be in linear raw format before CornerFix can correct them. What David is doing is using a combination of filters and CornerFix to make the Fuji IS-Pro usable for normal photography as well. David goes through the process step by step, discussing settings in detail, and provides some really good examples. The article is here.

Revised and refreshed Chromasoft website

2010-10-18T04:51:00.000-07:00

Google recently converted my Chromasoft website (as distinct from this blog) from the old Google Pages format to Google Sites, as they are doing for all Google Pages sites. The website is where all the reference material for the various things I blogger about here are. So, for example, there is:

Calibration test image (synthetically created "perfect" versions of the Gretag Macbeth 24-patch test chart);
A spreadsheet with color space information;
More information about dcpTool;
Some papers I wrote on color spaces.

The automatic conversion was reasonable, but messed up formatting a few places. So I've taken the opportunity to refresh the site with a new, cleaner format, and also to make use of some of Google Sites features that weren't available in Google Pages. E.g., the navigation widget, which replaces the previous "hard coded" navigation links.

The site is here: http://sites.google.com/site/chromasoft/

Solving the dreaded Gigabyte "Won't boot from USB" problem

2010-10-15T08:41:00.000-07:00

As mentioned in this post, I just did a fresh install of Ubuntu 10.04 "Maverick Meerkat". I mentioned two fairly major problems I had with the Ubuntu installer in that post. I also had another problem, but this one isn't Ubuntu related.
I had chosen to install from a USB key, but what I had forgotten is that the motherboard in the PC is a Gigabyte motherboard, and it doesn't like booting from USB. This is a common problem with Gigabyte motherboards - just Google for Gigabyte and "boot from USB", and you'll see many variations of "can't boot from USB", "won't boot from USB", "unable to boot from USB", etc, etc. What happens is that the motherboard just completely ignores the USB key; effectively, it's like it isn't there.

Unfortunately, there isn't a solution to be found in any of those posts. Or at least I ran out of patience before I found one, although it's clear that the problem is to some extent USB drive dependent - Gigabyte motherboards seem to like some drives, others not. So rather than continue to go through endless posts that don't offer solutions, I started playing around. What I found was:

With any other drive in the machine, even if the boot from the drive was disabled in BIOS, the Gigabyte motherboard would still ignore the USB key and boot from the other drive;
With all other drives physically unplugged, boot would fail with a "Insert boot disk and press enter" message. The light on the USB key didn't even flicker at any point in the process, indicating that the USB key wasn't even being seen;
However, that was where things got interesting. What I found was that if I then unplugged the USB key, and plugged it in again, then pressed enter, the USB key was recognized, and the machine booted.

Now this is a step in the right direction. If you don't have any other bootable drives in the machine, then you can just follow that procedure. Problem is what if you do have other drives in the machine? In my case, the intention was to install Ubuntu on a dual booted drive, so having a bootable drive in the machine was required, and the there wasn't any way to get to the "Insert boot disk" prompt - the machine just booted from the drive I wanted to install Ubuntu to.

The Solution

Thinking about this, the USB key only being recognized if it was unplugged and then plugged in again suggests that somehow, if the USB key is inserted at the time the machine is powered up, the motherboard gets into a mode where it doesn't recognize the key and it gets ignored until a operating system driver starts up. But later in the boot cycle, the motherboard does seem to be able to recognize it. So what I did was simple:

I started the machine with the USB key unplugged;
Once the machine had got through the first parts of its boot sequence, I plugged the key in, then hit the F12 key to bring up the boot menu. And on the third try, there the USB key was, on the list of hard drives. And it booted fine!

Two thing to note here - what I found worked, for me anyway, was to plug the USB key in just after the BIOS displayed the list of PCI devices, and just as it gave its "Verifying DMI pool" message. Secondly, note that the USB key appears on the list of hard drives.

Now I must admit, plugging the USB key in during the boot sequence is neither elegant or reliable - you'll probably have to try it a few times - but it worked for me.

Installing Ubuntu 10.10 "Maverick Meerkat"

2010-10-14T08:53:00.000-07:00

So I use Linux occasionally for software development stuff e.g., pcdtojpeg will run on Linux. I'd skipped Ubuntu 10.04 because (a) the release got a pretty bad name for reliability and (b) I was busy professionally, and with pcdMagic and CornerFix, so anything on Linux wasn't featuring anyway. But having gotten both pcdMagic for Windows and V1.4.0.0 of CornerFix out the door, I decided that an upgrade was called for. I decided on a fresh install, as Ubuntu's upgrade process can't skip releases, and I wanted to change the drive format to EXT4 anyway. Well, it wasn't an easy process - I ran into two fairly major bugs in Ubuntu's install mechanism:

Ubuntu 10.10 "Maverick Meerkat" install bug 1

Problem one was a "SYSLINUX – Unknown keyword in configuration file" error.What I'd decided to do was to boot from USB. So, as the Ubuntu download page recommends btw, what I did was to use the the Ubuntu "Create bootable USB Key" feature. Only I used it under my existing Ubuntu installation, which was 9.10. There's an Ubuntu bug description here, but in summary the problem is that the version of Syslinux (which is what actually boots the USB key) that old versions of Ubuntu place on the USB key isn't compatible with the configuration file that comes with the 10.10 ISO image. So instant crash. Irritatingly enough, looking at Ubuntu's own bug tracker, this was identified as a bug back in July, but nobody bothered to fix it, although I'd rate it as a showstopper. So far as I can tell, there are two solutions:

Create your Ubuntu 10.10 bootable USB key using Ubuntu 10.10. That's a bit of a catch-22 situation, but it may be an option for people with access to a friend's Ubuntu 10.10 installation or whatever.
Edit the /syslinux/syslinux.cfg on the USB key to remove the "ui" keyword, which is what causes the problem. This is the solution that I used.

Ubuntu 10.10 "Maverick Meerkat" install bug 2

Problem two was a crash in the very early part of the install.The symptoms of this are:

If you're in the graphical install mode (the full screen purple screen), the install just hangs. The little dots keep moving, but nothing happens.
If you hit esc, you get a "getpwuid_r(): failed due to unknown user id (0)" message, as documented in this Ubuntu bug report.Again irritatingly, this bug is known. Even more irritatingly, as the bug report makes clear, the error message is entirely misleading - it can result from any number of underlying problems.
So, what you have to do is to get a real error message. The way to do that is to keep Ubuntu out of the graphical install, and have it give text error messages. You do that by hitting F6 as soon as the purple screen comes up, then editing the command line to remove "quiet" and "splash". Then press enter to have your new command line run.
What I then got was the real error: "Buffer I/O error on device fd0, logical block 0".If you know that fd0 is the floppy disk, this tells you what you need to know. Turns out that although the machine doesn't have a Floppy, I had "Floppy disk" enabled in the BIOS of the motherboard (which is the default), and Ubuntu just can't deal with a non-existent floppy. Disabling the Floppy in BIOS solved the problem.

Comments on Ubuntu later.....

New version of CornerFix and a new website

2010-10-14T08:05:00.000-07:00

There's a new version of CornerFix out, V1.4.0.0. The new release extends CornerFix beyond M8s, M9s and S2s to allow images from just about any camera to be corrected - e.g., Sony NEX, Sigma DP series, etc.

This is prompted by the many requests I've been getting from numbers of people for CornerFix to be made compatible with various new large sensor cameras, especially the Sony NEX series, for which there are now a lot of lens adapters available. However, given the size of the NEX sensor, and the characteristics of, for example, the Voigtländer (Cosina) 15mm f/4.5 lens, the same color vignetting as happens on the Leica M8 becomes inevitable. And of course, there's no Leica style "self coding" option on the Sony NEX.

So rather to continue to respond to requests to update CornerFix to each camera, I decided to make CornerFix more general. Now in fact, CornerFix always was quite general - it was just set up to warn about "Unsupported camera model" if it came across any camera I hadn't tested on. But usually, CornerFix would get it right anyway.

What the new version of CornerFix does is to look at the characteristics of the DNG file itself, and decide whether it can be processed. So it will now work with very nearly any DNG file, regardless of whether the file came directly from a camera, or was converted from the camera manufacturer's raw format with Adobe's DNG Converter. The only restrictions are that the DNG has to have either Bayer sensor or "Linear Raw", aka RGB, data in it. Which covers about 99.9% all cameras out there, btw. So, all you have to do to use CornerFix with a Nikon, Canon or whatever is to convert your raw image to DNG, then use CornerFix as usual.

At the same time, CornerFix also has a new website. The website has detailed instructions on how to use CornerFix with converted DNGs, etc.

Downloads are still from the usual place: CornerFix Files on SourceForge.net

pcdMagic for Windows is out

2010-10-14T03:07:00.000-07:00

So, pcdMagic for Windows is out. pcdMagic converts Kodak Photo CD images into more modern formats such as JPEG and TIFF. But unlike all the other solutions out there, it actually gets the color right.

The original pcdMagic, which was Mac only, shipped back in February. At the time, I said there would never be a Windows version. Well, 9 months later, there is. The Windows version is a quite a bit different to what I built for the Mac, however. What I found with the Mac version was that there were two distinct groups of people using pcdMagic:

First, there were the pro photographers, the high-end "art" print shops and advanced amateurs - folks that know their photography, and are using pcdMagic because it's really the only Photo CD conversion solution available that actually gets the color reproduction right. Probably a third of the users of pcdMagic for the Mac fall into that group, judging by the organization names. These are people that can talk about ProPhoto color spaces, etc.

But there's also another group of users - folks who just want to get their images back the they remember them. These users neither know or care about color profiles, but they do know that the images as converted by the various free packages on the Web just look wrong. pcdMagic for Windows is built to make life easier for them, while still providing all the color profiles, etc in the background. So pcdMagic for Windows is built as a drag-and-drop application. All you have to do is to drag-and-drop any Photo CD file on the window, and it gets converted. Batch operations - just drag-and-drop a group of files. All the color profiles, etc, etc are done in the background. But pcdMagic for Windows is still a seriously powerful piece of software; although its a lot simpler to use than the Mac version, the only features it's missing relative to the Mac version are sharpening (easy to do later if you want), and the option to convert to DNG.

For those interested in the technicalities, pcdMagic for Windows is a C# and WPF application. Reason being that WPF brings some useful color management capabilities to the table.

dcpTool and the dangers of tables

2010-10-10T09:13:00.000-07:00

There's a good article by Marco Noldin on his blog where he discusses an issue relating to DNG color profiles with hue twists that I haven't touched on in this blog. I've pointed out the issues that you can get when trying to recover images that were for whatever reason badly exposed, or had their expose adjusted for purposes of ETTR. However, Marco points out an additional risk - in some profiles the HSL tables are quite coarse, and as a result can cause posterization. dcpTool allows him to provide a really good before-and-after example, first showing posterization with a the standard profile, and then showing no posterization using the same profile just edited via dcpTool to remove the table.

The post is in Italian, but Google Translate does a pretty good job, so it's well worth the read, as is his companion article here.

A CornerFix tutorial

2010-09-23T23:44:00.000-07:00

Jeff Hapeman has posted a good tutorial on using CornerFix, going into the detail of how he uses CornerFix in conjunction with the Leica M9 and Voigtländer 12mm and 15mm wide-angle lenses. Good reading for anyone just getting started with CornerFix.

You can find the tutorial here:
Jeff Hapeman's CornerFix tutorial

ETTR for the third time

2010-08-27T20:59:00.000-07:00

Two new articles on ETTR have recently been pointed out to me:

There's a long discussion on ETTR on Photo.net, here.
And also, probably in response to the discussion, an article by Jeff Schewe.

In both discussions Jeff puts up a spirited defense of ETTR. For those interested in the pro and cons, both are well worth the read. Jeff's position and my position aren't actually as far part as some of the posters in the Photo.net article seem to believe - as mentioned in my original blog post, there is a place for ETTR - it's just that the circumstances in which ETTR is useful are a lot narrower than most ETTR proponents believe. The root cause of that is that a lot of ETTR proponents don't actually understand ETTR. To quote from Jeff: "ETTR can aid you along those lines...if you understand how to use it (some apparently don't)".

I do have a bit of a problem with Jeff's article however, where he provides examples of ETTR apparently providing significant noise benefits. The problem is that he doesn't compare against is the simple option - just changing the camera's ISO setting. Pretty good bet all the improvement in noise performance he shows can be duplicated, a lot more simply, just by decreasing ISO.

"Expose to the right" revisited

2010-07-14T02:33:00.000-07:00

I wrote about "expose to the right" (ETTR) last September in this post, showing that while ETTR is useful in some narrow circumstances, in most situations the practical workflow complications vastly outweigh the benefits. Predictably, there have been a wide variety of reactions to the post, ranging from very positive to highly negative.

What has been interesting about all the responses is two things:

Firstly, almost a year later the post still gets attention.
But secondly, not one of the "ETTR is great - you just don't understand it" type forum posts/blog entries/etc that I've seen or had pointed out to me have ever shown (or linked to) a single actual practical example of ETTR showing any visible benefit beyond what I identified in my original post. Which rather proves my point....

Disapointing news on the Adobe lens profiles

2010-05-04T05:11:00.000-07:00

Adobe recently announced that there would be lens correction capabilities in Camera Raw 6 and Lightroom 3, including for vignetting. I had hoped that meant that it would be possible to use these new capabilities to correct for the red edges and red corners that afflict the Leica M8 and M9. Now of course CornerFix does correct for that now, but I had hoped that rather than the current process where users have to process each file though CornerFix, I could just modify CornerFix to initially generate an Adobe lens profile, and users could then just use the profile, all within Lightroom.

However, it seems that what I had hoped isn't going to happen. The bad news for M8/M9 users is on page 9 of the Adobe Camera Model technical report:

"As part of the vignette model calibration process, u ,v , f , f ,α ,α ,α are the set
of model parameters that need to be estimated. Note that these model parameters
are identical for all color channels."

In other words, the Adobe lens correction system does not have the capability to correct for the M8/M9's color dependent vignetting (aka red corners or red edges).

I'm seriously disappointed - it's not as if Adobe aren't aware of the red edges/red corners issue.

A good discussion on dcpTool

2010-04-06T10:37:00.000-07:00

There's a good discussion around dcpTool and untwisted/invariate profiles on Tom Lester's blog.

Well worth the read, and the examples are good, as well as the later discussion.

Windows 7 forgets userid and password for Mac OS X share - Solved

2010-04-03T21:53:00.000-07:00

I've been upgrading to the retail version of Windows 7 on my primary Windows machine, rather than the RC version, and found one major irritation - despite ticking the "Remember my credentials" tickbox when connecting to my Mac's shared folders, the user id and password simply weren't being remembered - I had to re-enter them after every reboot. Very irritating.

I tried entering the info direct into Windows Credential Manager - same result - a "Login failure" message on the first boot, and the credentials erased.

After a bunch of fruitless searching on the Web, and trying various obscure settings changes, it dawned on me that what was being stored in the Credentials Manager as a user id was actually WindowsServer\UserId, rather than just UserId that I had entered. Tie that to the "Login failure", and the solution is obvious. What is actually happening is not that Windows 7 is forgetting the user id and password, but that it is storing the wrong server name, then trying to log in with the wrong server name, failing, and erasing the user id/password pair as invalid. At a guess, I'd say this tied to the current Windows "theology" of single id across the network.

Solution: Rather than just enter your Mac user id on its own, enter the name of the Mac server as well as MacServer\UserId. E.g., IMAC\sandy. Windows then doesn't store the "WindowsServer", and magically everything works. The name of the Mac server is just the name as displayed in Windows Explorer.

Synchronize iCal and Address Book to Gmail calendar and contacts

2010-03-27T08:33:00.000-07:00

Over the past few weeks I've slowly been moving my entire e-mail/contact/calendar to a Gmail based environment. The process has been "interesting", so I thought I'd briefly document what I discovered about syncing contacts and calendar from a Mac to Google. (If you just want "the answer" without the reasons why most of what I tried didn't work, then jump to bottom of this post, btw)

The reason for all this is that I'm increasingly using multiple devices to access e-mail - my desktop computers, my laptop, and iPod Touch. The e-mail part of this was simple; I just use Gmail in IMAP mode on all the desktops and laptops, and everything pretty much just works. The iPod I sync with a combination of Gmail's exchange server connection (for my most used account, because that way I get push e-mail), and IMAP for the other accounts. Also, I no longer need to worry about archiving e-mails; everything is automatically archived to "All Mail" on Gmail. So that all worked great, and was painless to set up - just follow the Gmail instructions.

But then life got "interesting" - I also wanted to sync contacts and calendar to my Mac desktop and laptop. That got complicated and painful. Here's the story and the solution(s):

Syncing Gmail Contacts with Address Book
In theory, this is simple - all you need to do is go to Address Book - Accounts - Account Information, and enable "Synchronize with Google", provide your Gmail account information on the "Configure..." dialog, and you're done. Well, not. I soon found out that not all of my contacts were syncing. A great deal of digging around showed that I had two problems:

Gmail can't deal with contacts that are "the same". "The same" seems to be defined by name/e-mail address/telephone number. Now that was a problem for me, as I have copies of corporate address books as Groups in Address Book, as well as some of the same contacts as personal contacts in other Groups. No problem for Address book, but a big problem for Gmail's contact list. So the duplicates needed to be purged.
Gmail also has problems with first names/last names/company names. Historically, I'd been sloppy back when my contact list was on Outlook, and many companies were not tagged as companies, and had their company names in the "name" field of the address cards. Gmail doesn't like that either. So I needed to do a fairly major contact clean up, tagging all companies as "Company", and ensuring that the company name was in the correct field.

The good news is that once I had done all this (which was probably a good thing anyway, as I now have a clean address book), Address Book and Gmail synced just fine.

Syncing Gmail Calendar with iCal
Again, in theory, this is simple - all you need to do is go to iCal - Accounts and add a account and you're done. See here for detailed instructions. However, again, not so simple in practice. I soon found myself dealing with the dreaded "Calendar could not be found" error. This issue has been documented in a number of locations e.g., here and here, and a number of solutions suggested, mostly involving things that in effect reset the account by changing settings. Sadly, as has been many other people's experience, none of the suggested solutions worked for me. So, I did a bit of digging around in the details of the error messages, and some experiments with syncing first an empty calendar, then a calendar with various items in it. What I found was:

In my case at least, the "Calendar could not be found" errors had nothing to do with account settings - I could sync empty calendars, and very simple calendars easily.
However, once I tried to sync legacy calendar items, problems cropped up. What was happening was that the Google calendar just couldn't deal with some items, and was then returning "Calendar could not be found" errors. Not a very informative error message on Google's part.
More digging showed that the problem calendar items were recurring appointments - Google calendar seemed to be able to deal with pretty much any single event ok (although it sometimes simplified alarms), but some recurring events just caused it to blow up.

So, the solution was simple, but painful - rebuild all the recurring events in the calendar. That fixed the problem, and calendar sync worked.

Problem sorted, you might think. Unfortunately not - it was a day later that a show stopper turned up, in the shape of an e-mailed meeting invite. Now I run multiple e-mail accounts - two for professional purposes, one for my open source software, and a personal account. Turns out that iCal just won't allow you to place an e-mailed invite in a synced Gmail calendar. Given that I need to be able to receive invites on multiple accounts but put them into a single calendar, the iCal Google sync solution just doesn't work for me. You can work round this problem by exporting the invite to a file, then importing it into the Gmail calendar, but that's just too much work.

Alternate ways forward
Well, clearly a simple Gmail sync just wasn't going to work for me. So I started looking for alternate solutions. Here's a brief list of what I tried, and didn't work for me:

Exchange sync to Gmail. Gmail can do a exchange type sync - that's what I use to sync to my iPod Touch. So I thought I could use that to sync to iCal. No. Turns out that a normal (free) Gmail account's Exchange sync is only to a mobile device.
Exchange sync to Gmail enterprise. For $60/year plus the cost of a domain registration, you can get proper exchange sync. This might be a solution, but it wasn't at all clear that it would solve my multiple account issue, and it involved setting up domains, etc.
MobileMe. Apple's sync service - tempting, but $99 per year, and means yet another account.
Hosted Exchange servers - also a possibility, but good ones are $120/year.
Nuevasync, which is free, and comes recommended by e.g., the folks at TechCrunch. However, it turn out that Nuevasync, like Google's Exchange sync on standard accounts, works only with mobile devices.

The Answer - what did work for me
What I ended up with is a product called Spanning Sync. Basically, it just works. It syncs contacts and calendars, and those calendars are just regular iCal type calendars, so you can happily put invites from multiple different e-mail accounts in one calendar. Best of all, they have tech support that works - I did have one minor glitch, caused by the endless number of previous sync attempts with other solutions, but Spanning Sync's tech support responded to my query inside of 15 minutes, and their solution worked first time. Try to get that kind of service from certain other software companies I could mention.

I've now been on Spanning Sync for three weeks, and it's been flawless.

They have a trial version if you want to test the product, and the full version costs either $25 per year or $65 forever. However, if you use this discount code, you will save $5 off either:

Spanning Sync discount code: WMW883

New firmware for the Leica M9, but there's a catch.....

2010-03-13T06:17:00.000-08:00

Well, it looks like there will be new firmware for the M9 on Monday that will at least make the "red edges" problem that I have blogged about previously better. The release notes say that the following lenses have been tweaked:

M 18mm/f3.8 ASPH (11649)
M 21mm/f2.8 ASPH. (11135)
M 24mm/f3.8 ASPH. (11648)
M 28mm/f2 ASPH. (11604)
M 28mm/f2.8 ASPH (11606)

The 18mm especially can do with the help. But I doubt, for the reasons described in the previous post, that there will be a 100% solution.

Problem is, there's a catch. Actually, two catches.

Catch 1: updated vignetting corrections mean that folks that have been using CornerFix on images taken with coded lenses and lens detection enabled will have to redo their profiles to match the new firmware. And remember to keep both the old and new profiles, and use the appropriate profile depending on whether the image was taken with the old or the new firmware.

Catch 2: At least for those same CornerFix users working with coded lenses and lens detection enabled, there's also an unpleasant surprise in the release notes:

"New ISO setting-related vignetting correction: strong correction at ISO 160, lowest ISO correction at 2500, interpolated for intermediate ISO steps."

Most likely this is because on some of the wides, fully correcting the corners visibly amplifies noise. However, what that means is that if you take images with lens detection enabled and want to use CornerFix later to get rid of any residual red edges, you will need to have a CornerFix profile for every ISO setting. Not good.

I've always recommended shooting with lens detection off if you plan on using CornerFix, but in practice shooting coded and letting CornerFix take care of any residuals has worked fine. I know a lot of people shot with their lenses coded, and then only used CornerFix to clean up their "keepers". That's probably not such a good idea anymore.

Will Leica fix the M9's red edges? Part 2

2010-02-17T00:53:00.000-08:00

In a previous post I talked about Leica's delay in delivering revised firmware for the M9. Subsequent to that post, there have been some conversations both on the Leica User Forum and off of it. Here's a summary of the new information:

Firstly, a "usually very well informed source" tell me that the reason, or at least a part of the reason, for the delay to the firmware is an entirely unrelated bug in some new functionality. Apparently this bug has been intermittent, and hard to track down, but has now been fixed.

Separately, Daniel Kennedy ("thrice" on the LUF) is quoting Stephan Daniel as giving a delivery date in late march. The post is here.

Finally, I think there's now a growing consensus the the root cause of the red edges is probably the sensor itself, rather than over-correction in the firmware, or lens issues, although it seems clear that lens de-centering makes the red edges worse. My guess, but it is only a guess, is that there is some kind of a problem with the registration between the microlenses, the Bayer array and the photosites themselves - remember that each of those layers is in effect a separate process step.

But whatever the root cause is, just knowing (or assuming) that it is a sensor problem tells us that the problem won't be able to be completely solved in firmware. Some people are probably assuming that because CornerFix can effectively 100% correct the problem (given a good reference image), Leica will also be able to 100% correct the problem in firmware. Not the case, unfortunately. CornerFix solves the problem for the combination of a specific lens and camera, while Leica's firmware will have to solve the problem for any combination of lens and camera. Realistically, if this is a sensor issue, Leica is going to be able to improve situation, but not solve it entirely. So probably for a lot of users, hopefully the majority, the new firmware will reduce the problem to the point that it effectively is solved. But probably there will still be some users that have to resort the CornerFix.

Two questions come up. The first is whether Leica will replace the sensors in the cameras that are particularly badly affected. Predicting the future is a notoriously chancy business, but my guess is that they will try to avoid sensor replacements at all costs - "recalling" even a small number of M9s, financial impact aside, would be a blow to their ambitions in the pro market with the S2.

The second question is whether there will be a M9.2 with a revised sensor. I would guess probably yes. I think that Leica would like the M9 platform to have as long life as possible. When will that happen? Well, redoing a sensor will take a minimum of three months, then it has to be tested, etc, etc. So I'd think late this year, early 2010.

And then course there's the "out of left field" possibility - remember that the X1's sensor isn't from Kodak. "Mysteriously" the X1 sensor happens to have essentially exactly the same characteristics as Nikon's D300 sensor. So an M10 with a D3x sensor? Don't count it out.......but that would require a very substantial rewrite of the M9 firmware, probably not something Leica would like to do now. So unless Kodak can't or won't fix their current sensor, I'd expect an M9.2 with a revised Kodak sensor in the not to distant future.

pcdMagic is now available

2010-02-15T07:52:00.000-08:00

pcdMagic is an OS X application for converting Kodak Photo CD image files. Kodak abandonned the Photo CD format many years ago, and since then most solutions for converting the files have been just terrible - blown highlights, bad color, etc

Last year I published pcdtojpeg, which performs basic conversion from the command line and fixes the blown highlights, etc. But pcdMagic is a different animal entirely:

No blown highlights - most Photo CD conversion software blows highlights. pcdMagic just doesn't.
Specific color profiles by film type and scanner model - e.g., Kodachome scanned by a Kodak 4000 series scanner versus color negative on a Kodak 2000 series scanner.
Image sharpening that's specific to Photo CD images.
Adaptive interpolation - conventional interpolation can result in unsightly artifacts in your images; pcdMagic's adaptive interpolation avoids this.
Output to JPEG, TIFF or DNG. pcdMagic converts to your choice of output format, even archival level DNG files. And of course, properly color managed, so you get exactly the right color rendition when you view converted files in other applications.
Converts any Photo CD file. pcdMagic handles all Photo CD files at maximum resolution, even 4096 × 6144 "64Base" Photo CD Pro files and the rare "Class three compression" files that other conversion program can't convert at full resolution.

If you have PCD format files and you want to see them looking the way they did when you took the photo, you need pcdMagic. pcdMagic doesn't convert hundreds or thousands of different image formats - it converts just one format, the Kodak Photo CD format, exactly right, giving you your images back, just like you remember them.

You can get the details here: pcdMagic web site

Will Leica fix the M9's red edges?

2010-02-15T07:41:00.000-08:00

In some posts dating back to last year I discussed the Leica M9's by now notorious red edges problem, starting with this post. At the time, I thought, and commented to that effect on the Leica User Forum, that while Leica wouldn't be able to 100% solve the problem given manufacturing variations in both lenses and the M9, they would be able to make substantial improvements by tweaking the M9's firmware corrections. However, there's been a deafening silence out of Solms, and the silence is growing more deafening by the day - no firmware update, no statement about what the issue actually is.

What's concerning is that its now been quite a while. CornerFix V1.3.0.0, which was capable of correcting the red edges in most cases, shipped in early Novemeber last year. CornerFixV1.3.0.4, which fixed some early bugs, and extended "most cases" to "pretty much all cases", shipped in late November. So its now nearly three months since a version of CornerFix that unambiguously nails the problem shipped, but Leica haven't got anything out of the door.

The question is, why haven't Leica shipped a new firmware version? This is actually a major issue, and it's not as if it just impacts 50 year old lenses, or lenses from other manufacturers - it impacts the vast majority of Leica's own 18mm lens, which was introduced in past few years, as well as a good number of examples of other lenses, e.g., the 21 mm and the WATE.

Well, I can only speculate, although it's somewhat informed speculation in the sense that I know exactly how CornerFix solves the problem. It seems to me that there are only a limited number of possible explanations:

Explanation 1: Leica's just busy with the S2, etc. Well, maybe. The S2 is probably Leica's view of its future. But the M9 is today's cash cow. Neglecting getting it right seems like a big risk, even if production hasn't yet caught up with demand.

Explanation 2: The red edges just can't be fixed much better than they are now. This is the "doomsday scenario", and not a pleasant one, and certainly not one that I had considered up till recently. But there is some evidence that supports it:

The Kodak DCS PRO 14n suffered from what was known as the "Italian Flag" problem (green on the left, red on the right!) with some lenses. That was never solved in firmware - it took a next generation Kodak DCS with a revised sensor to solve the problem. And the M9 uses a Kodak sensor.....
Stephan Daniel referred in a recent interview to vignetting correction as being "more of an art than a science", which discourages me. I'd be a lot more confident if this was being described as an engineering problem. In my opinion, Leica should be viewing this as a matter of science, and statistical variations in production. Trying to make corrections by "feel" seems unlikely to result in an optimal solution, and will maybe result in no solution at all.
I'm starting to have a lot of trouble believing that Leica didn't know about the red edges before the M9 shipped, which in turn implies that they had ample opportunity to find a fix, if they could. What we know today, and didn't know three months ago, is that nearly all 18mm lenses show red edges. In fact, every one that I'm aware of does, and I know of at least one M9 user that tried all three that his local dealer had available. And when I specifically asked in a post on the Leica User Forum a few days ago whether anyone knew of an 18mm that didn't have red edges on the M9, nobody responded. So to believe that Leica didn't know about this, we need to believe that either the 18mm wasn't tested for vignetting, or the test was so hopeless incompetent as to miss the problem.

Explanation 3: Red edges are a lot more difficult to fix in firmware than anyone thought. This is a more hopeful scenario, and fortunately one that we have some reason to believe might be true. The reason is the following. If you consider what computations CornerFix has to do in order to correct an image as a proxy for what the M9 firmware might have to do, the answer is very different for the M8 and M9:

For the M8, a simple symmetrical radial devignette, centered on the physical center of the sensor is sufficient for CornerFix to correct any lens, up to and including the CV12 and CV15. That amounts to what is in effect a "half of a radius computation" and a look-up for every pixel.
For the M9, an offset asymmetrical correction is required. What that amounts to is an (a) offset computation, (b) a full radius calculation, (c) multiple look-ups, and (d) a complex interpolation process for each pixel.

Now Leica may be able to do something clever to simplify that process - they have the advantage of presumably knowing in detail what the root cause of the problem is, and thus having a mathematical description of what's going on, which I don't. However, two things are near certain - firstly, the M9's vignetting correction code will need a total rewrite, and secondly it will absorb far more processing power than the M8's algorithm, which also appears to be the current M9 algorithm. However, we know that the M9's processor is already heavily loaded, so unless some really efficient code gets written, the M9 may be slowed to a crawl. And writing really efficient code takes time. And there's a complication - the original code was written by Jenoptik for the M8, so its likely they will have to create any new code, or a new set of programmers will have to understand the old code.

What happens if the M9's processor just can's handle the load, no matter how efficient the code is? Well, there a solution to that, but a solution that will be very unpopular in many quarters. Leica could simply give up on doing the corrections in-camera, and embed correction op-codes into the DNG image, using the DNG 1.3 specification. That means no processing the image pixel by pixel. However, there are two problems with that. That means the in-camera JPEGs aren't fully corrected, and that DNGs can only be processed in Adobe Lightroom or ACR. No other raw developer implements DNG 1.3 opcodes. Which will drive the Capture One enthusiast crazy, and could seriously cut into the "rich snapshot" market. And given the somewhat strained relationship between Phase One and Leica at the moment, it's not clear that Phase One would go to the trouble of writing correction code.

So we live in hope.......

UPDATE: Part 2 here.

Building a DMG installer for the Mac the simple way

2010-02-15T05:09:00.000-08:00

A very common way of providing a semi-automated method of installing Mac OS X programs is a DMG that gives the user a easy way to install your program by just dragging the application into a shortcut to the Applications folder. Below is the KeyChainDD version. The advantage of this is that once the user downloads the DMG, it will automatically open, and what to do is graphically pretty obvious.

The problem is that actually building such a DMG isn't too obvious. Notably, it's very difficult to get the DMG configured in such a way that it always opens to the right size, and showing its icons, not a list of files. There a number of methods on the web showing various ways of doing this, but most of them are very complex. This is my (fairly) simple way, without using any exotic tools:

This assumes you have an application, and a background (in e.g., PNG form) that you want displayed when your DMG is mounted. I used InkScape to build this one, based on the FireFox one.
Create a new image using Disk Utility, leaving all the options at their defaults, sized such that it will contain all your files.
Create a folder inside the mounted DMG and name it “background”. Drop the background PNG into this.
Hide the background folder (aka rename with a leading .) - I use X-Folders to this, but you can do it from the command line as well.
Drop your app into the root of the DMG.
From the View menu of Finder, choose Show View Options, and customize the view to show icons only, no sideview, etc. Also switch off the toolbar in the view menu.
Set the window background to picture, and use Cmd-Shift-G in combination with the full path (e.g. /Volumes/MyDisk/.background) to set your background PNG as the background for the view.
Move the app icon into the right place versus the background image, changing the font size and the dimensions of the icons.
Now just add a shortcut to the Application Folder, place it inside the DMG, and then position the shortcut.
Do a final resize on the Window such that it is the same size of your background. Your DMG should now look as you expect.
Eject the DMG, then go to Disk Utilities, select the DMG and choose “convert“, then “compression” and save again. Now the DMG is compressed (as small as possible) and made unwritable. Note that's its a really good idea to save to a different name, so that you can use the uncompressed DMG as a template for future releases, and not have to go through this whole process again.
That's it

NOTE: Unless the DMG is compressed, it will revert to the default format on opening. But in order to edit it, you have to convert it back to read/write, else changes won't stick. Keep track of what state its in, because there's no obvious way to tell.....

If you need to create a new DMG for a subsequent release, you can just delete the old application from the uncompressed DNG, and then copy the new version of the application to the DMG. Warning: you MUST use either command line, or something like X-Folders to delete the file, rather than move it to the trash as the Mac OS X finder will do. Using Finder will create a trash folder inside the DMG. Once you have a trash in a DMG, it's nearly impossible to get rid of, and you'll have to rebuild the whole DMG.

Interfacing to GIT on Sourceforge

2010-02-15T03:43:00.000-08:00

For KeyChainDD, which is hosted on SourceForge, I'm using GIT as a repository for source code, rather than distributing the source code via distribution files, as I usually do. This post briefly documents the process I used to get GIT on SourceForge working.

Note that this assumes that you have GIT installed on your system, and will be using the standard GIT command line, and are in your project directory.

Here are the steps:

Step 1: Before creating any commits, need to identify yourself, using your SourceForge identity. The easiest way to do so is to make sure the following lines appear in a file named .gitconfig in your home directory:

[user]
name = Your Name Comes Here
email = you@users.sourceforge.net

Step 2: Tell GIT what files not to transfer into the repository - by default, every thing gets transferred. The easiest way to do this is to create a file named .gitignore in your home directory. In my case, this has the lines:

# excludes for xCode and VC++
build/
Unused/
Site/
ScreenShots/

Step 3. Initialize GIT - Then you actually need to create a local repository (note that the -a in the commit isn't really necessary)

git init
git add .
git status
git commit -a -m "KeyChainDD 0.9.0.0 Initial commit"

Step 4. Check that you now have a local repository by asking for a log:

git log

Step 5. Push your local repository to the SourceForge GIT repository:

git remote add origin ssh://sandymcg@keychaindd.git.sourceforge.net/gitroot/keychaindd
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
git push origin master

Step 6. Create an archive (optional):

git archive --format=zip --prefix=KeyChainDD/ HEAD | gzip >KeyChainDD_0_9_0_0.zip

Vignetting Correction Issues on the Leica M9 Part II

2009-10-20T08:46:00.000-07:00

This is a follow-up to my previous post on vignetting corrections with the M9, here. In that post I said that I was working on a new version of CornerFix that would address the issues that I had identified. That new version is out now, as version 1.3.0.0, and can be downloaded from Sourceforge here.

To recap, the previous post showed images of the Tim Ashley's 18mm f/3.8 as corrected by the M9's in-camera correction firmware. The result was a red tint on one edge. I also mentioned that the then current version of CornerFix couldn't do much with the problem, because V1.2 of CornerFix couldn't deal with optical decentering.

However, V1.3.0.0 can. Here's the result, first correcting the 18mm f/3.8 for an original image for which coding was disabled (so no in-camera corrections):

18mm f/3.8 uncoded with CornerFix 1.3.0.0 correction

And now if we use CornerFix to correct the nasty in-camera corrections that gave a red edge:

18mm f/3.8 coded as "Auto" with additional CornerFix 1.3.0.0 correction

Both of those are (IMHO) pretty good - they show no visible signs of red edges anywhere. However, I do recommend that if you are going to use CornerFix, you shoot with lens detection disabled, aka with the lens uncoded. The reason is that if you have a camera/lens combination that shows the decentering issue (and every M9 image I've analyzed regardless of lens shows it) and use your lens coded, a interference pattern can form between the actual vignetting and the in-camera correction. Think about two circles, on top of each other, the same size, but with their centers slightly offset as shown below:

Interference between offset circles

Where the circles come together, you get e.g., over-correction, and where they are far apart, under-correction, in a complex two dimensional pattern. And that pattern is unfortunately nearly impossible to correct; there is no way to tell the difference between what should be there and what shouldn't. That may or may not occur in your case, but the easy way to avoid it is not to use in-camera corrections when you're planning to use CornerFix.

Another happy dcpTool user.....

2009-10-20T01:20:00.000-07:00

Thomas Lester has posted on how he's using dcpTool here, describing it as a "fantastic tool". He shows some some great example images of how the hue twists in some of the Adobe camera profiles can result in really unnatural skin tones, and how to avoid the problem by using dcpTool.

As a pro lifestyle photographer it's not surprising that he's serious about good skin tone, but what is surprising is that the camera is a Nikon D3. That's surprising because the D3 is generally considered to have pretty good color right out of the box, and not need much work in post, unlike some of the Canons.

Vignetting Correction Issues on the Leica M9

2009-10-13T08:30:00.000-07:00

There's been a lot discussion on the Leica User Forum about problems with vignetting correction on the new M9. These have shown up in a number of different images, but the image below of a white diffuser shot with a coded Leica 18mm f/3.8, which should be entirely uniform edge to edge, shows the problem well:

18mm f/3.8 with "Auto" in-camera correction

Basically, there is a red tint on the side - mostly the left hand side - of the image despite, or perhaps because of, the in-camera correction. As CornerFix hasn't been able to correct some these images while it was usually able to correct M8 images with red corners, I've been trying to work out what the issue is, with the help of several people on the forum that have sent me images - Carl Bretteville, Eric Calderwood, Jonothan Slack, Tim Ashley and others - thanks to all of them!

Background

The background to this is that on any M-mount digital camera, you have a fundamental optical problem. On any digital camera you have to have an IR (infra red) filter to prevent color shifts due to IR contamination - all modern sensors are very sensitive to IR. However, because the M-mount has such a short lens to sensor distance, the light has to be at a considerable angle to reach the corners of the sensor. In turn that means that light that goes to the corners of the sensor passes through the IR filter at a sharp angle. That means that it passes through "more" of the filter, and reds get cut, resulting in cyan drift - blue corners. The M8 and M9 compensate for this by coding lenses, allowing the camera to boost reds (and greens, slightly) to compensate for the impact of the IR filter on visible light. CornerFix does the same thing, but as a post processor.

What's happening with the red edges above is that the cyan drift is somehow being over-compensated, red being boosted too much, and as a result you get red edges.

There are a few questions here - why exactly is this happening, what can be done about it, and most puzzling - why does this seem to happen on the left edge of images only?

The 18mm lens uncoded

To find what is happening with the in-camera correction, I did some analysis on the raw data in images shot of either a white diffuser or white walls. All of the images and charts in this post for a Leica 18mm f/3.8, but I've also looked at a number of other lenses, including a Zeiss 25mm, and some CV12's. All of the 18mm images are from Tim Ashley, btw - Tim shot these through a high-end diffuser, so the illumination is very even, which makes the charts clearer.

The first image is from the 18mm with lens detection disabled, so no lens specific in-camera correction. This is the image with the lens uncoded:

18mm f/3.8 with no in-camera correction

The chart below shows the the vignetting of the different colors (red, green1, green2, blue, because this is raw Bayer matrix data) across the horizontal center of the image, with the horizontal axis in pixels, and the vertical axis in stops of vignetting. The chart shows dots for pixels of the Bayer array, and best fit polynomials for each Bayer color to make things clearer :

18mm f/3.8 with no in-camera correction

You can tell a few things from this plot:

The luminance vignetting (effectively the blue curve) is about 1.3 stops at the edges. Note that because this cut is across the center of the image, corner vignetting is more than you see here; if the cut was the diagonal, you would see over two stops.
The chroma vignetting (the difference between the blue and red curves), the thing that gives the cyan corners, is about 0.2 of a stop. It's cyan because green is also impacted by the IR filter.
Most important, the lens is somewhat asymmetrical - vignetting on the left is greater than on the right. Note that both the luma AND chroma vignetting are greater. This is probably caused by the optical center of the lens not being aligned with the optical center of the sensor. Note that some asymmetry exists in almost all lens; I've routinely seen similar (and greater) asymmetry in other lenses from Leica, as well as Zeiss and Nikon lenses.

To correct for vignetting, what we would want is:

To correct chroma vignetting only, we'd want the red, green and blue lines to coincide. What that would mean would be that there would still be "normal" luma vignetting, but no color drift.
To correct luma vignetting as well, we would want all the lines to be flat.

One thing to be aware of is that even on an uncoded lens the M9 may apply some degree of luma vignetting correction, although there's no way to tell for sure. However, this isn't relevant to our discussion here.

The 18mm with in-camera correction

The second chart shows what happens with the Auto lens detection (the corresponding actual image is the one at the start of this post):

18mm f/3.8 with "Auto" in-camera correction

What is shows is that on the right hand side, the correction of the color vignetting is near perfect (all the lines coincide), but on the left, the in-camera algorithm is over correcting red by about 0.1 of a stop. That's what gives the left edge of the actual image its red tint, so the red edge is not a figment of anyone's imagination. It's also interesting to note that Leica is not correcting very much at all of the luma vignetting.

Coding as a WATE at 16mm

It's also instructive to look at what happens if the code the 18mm as a 16mm WATE. This is the image you get:

18mm f/3.8 with WATE 16mm in-camera correction

The chart is as follows:

18mm f/3.8 with WATE 16mm in-camera correction

Two things are interesting here:

Although this image is technicaly not as well corrected as the correct 18mm "Auto" coding, the WATE coding looks better than the 18mm coding. The reason is that while the WATE coded version is less well corrected, it's not over corrected, and its the red edges and corners that make the 18mm correction look so bad.
The second thing to note is that, if you look closely at the left hand side of the chart, you'll see that the red and green curve actually cross. Although its difficult to see in the image, that actually results in sort of a rainbow effect, with the image alternating from cyan under correction to red over correction. This is because the shapes of the correction curves for the WATE and the 18mm are actually different. This is important because many Leica users tend to assume that that the difference between in-camera corrections between lenses is just the "strength" of the correction. That's not actually true; the entire shape of the curve can be different for different lenses. The result is that using an in-camera correction for a lens it wasn't designed for is more difficult that many assume. Given that the M9 needs considerably more correction than the M8 did, this is likely to be a significant issue for users that have become accustomed to being able to use in-camera corrections for non-Leica lenses; that approach may no longer be nearly as effective.

So where does the asymmetry come from?

Bottom line is that I would say that what's happening here is that (as I had suggested in previous posts on the forum) asymmetry is going to be real problem for correcting cyan drift on the M9. It's not really clear what the root cause of the asymmetry is - there are a number of possibilities:

Lens asymmetry. Asymmetry in the lens itself (simplistically, the mechanical and optical centers being different) is by far the most likely reason for what we're seeing, and is certainly a large part of the problem. However, that doesn't explain why all the over correction examples I've seen to date have been red on the left, not the right. You would expect that the direction of lens asymmetry would be random, not all in one direction. It is possible that this is some kind of a artifact of the ways lenses are calibrated in the factory, but it is strange, and the bias to the left wasn't apparent on M8s.
Camera asymmetry. Inevitably, the sensor is not always exactly aligned with the mechanical center of the lens mount. If this was the case for the M9 as a matter of design, or due to some quirk of the manufacturing process, it would explain the left bias. However, sensor to lens mount alignment is usually trivially easy to get right compared to the optical alignment of lenses, and in order to account for the degree of left bias, a substantial asymmetry would be required. It is thus unlikely that camera asymmetry would account for the variation.
Asymmetry in the in-camera correction process. I did perform a test by subtracting the corrected image from the uncorrected image, which gives an estimate of the Leica correction curve, as shown below. Surprisingly, the Leica in-camera correction does appear to be very slightly asymmetric, but in the wrong direction! However, it is only asymmetric to a very small extent, and given that the measurement, made by subtracting two different images, is inherently imprecise, it's likely that the correction is actually symmetrical. The "lines of dots", btw, are as result of compression.

Leica in-camera corrections for 18mm f/3.8 (see text)

Can CornerFix fix this?

In a word, no, not using the current version. The current version of CornerFix will do as well as the Leica in-camera corrections, but no better.

The reason for this is that the current version of CornerFix is explicitly designed to correct symmetric vignetting. Substantial change - already underway - is going to be required to allow it correct for the kind of asymmetry found here.

Conclusion

As a results of these tests, it is fairly clear that the "red edges" are as a result of variations in centering between the lens, the sensor, and the in-camera correction algorithm. This issue showed itself on the M8, but wasn't significant enough to cause major problems. However, on the M9 this clearly is able to cause visible image degradation. Based on the available evidence, we can't say what the root cause of this de-centering is. However, we can reach a number of conclusions:

What I think that Leica probably need to do to fix the immediate in-camera correction problem with the 18mm is just to tune back the 18mm's red correction by 0.1 of a stop, or some value based on production variation with the lens. That will leave the right a bit under compensated, but under compensation is a lot less visually disturbing than over compensation, as shown by the WATE example above.
CornerFix will need to be enhanced to be able to deal with asymmetric vignetting - as mentioned above, I've started that process.

There are still a few things I don't understand however. The major one is why its always "red on the left", as discussed above. The second one is to what extent this issue varies with aperture setting. There is evidence that it does change significantly, getting worse with increasing aperture, but I don't as yet have enough data across enough lenses to be sure. The variation might be with either real aperture, because the aperture blades themselves are contributing to the vignetting, or at least changing the effective optical center of the lens, or with the M9's estimated aperture, because the correction that Leica are applying varies with estimated aperture. Although, supposedly Stephan Daniel said in an interview that the M9 correction is not variable with aperture.

Camera profile and reference images for the Leica M9

2009-09-28T23:46:00.000-07:00

I've posted a camera profile and some reference images for the M9 on the ChromaSoft website. They're at the bottom of this page.

The most important file is the DNG camera profile, generated from a real image of a real GM24 chart using Adobe's Profile Editor. It can can be used in Lightroom and Adobe Capture Raw to improve color rendering of M9 images. It's actually quite close to the embedded Leica profile, but tames reds a bit.

There are also two synthetic calibration images. The first of these is a synthetic GretagMacBeth chart that was created by using an existing image as a template, but then replacing all of the image data with new synthetic data calculated from the l*a*b* data of the GretagMacBeth chart using the color profile embedded into the original DNG file by Leica. Thus the file represents what a GretagMacBeth 24 patch chart would look like if photographed by a "perfect" M9. The second image is grey-level step wedge chart.

Please note that you cannot use the GM24 synthetic image to create a camera profile - as the synthetic image was created from the generic embedded profile, you will just get back to the same generic profile; the synthetic images are intended to calibrate your workflow, not your camera!

Why "Expose to the Right" is just plain wrong

2009-09-27T07:40:00.000-07:00

A recent discussion over on the Leica User Forum got me to finally finish writing this. The discussion in question was largely around how to best use Capture One, but it demonstrated (again!) the almost unthinking acceptance in various parts of the photographic community that "expose to the right" (ETTR for short) is the right way to set exposure on digital cameras. In fact in some corners of the web - and I won't point to them here - ETTR is practically religion.

What I'll be showing in this post is that ETTR is at best wildly over-hyped, and at worst will give you a less satisfactory end result than just exposing normally. I'll be doing that two ways. Firstly, by showing some practical examples of images using ETTR versus images normally exposed, but secondly showing that even in theory, ETTR is flawed under most conditions.

What is ETTR?

What ETTR says is in essence that the best way to set exposure on a digital camera is to place the highlights on the right-hand side of the histogram, hence the "expose to the right" name. This is in contrast to the "classic" exposure techniques which involve either an average exposure, which effectively places the mid-tones of the scene in the middle of the camera's range, or variants of the zone system, which says that you as the photographer need to take a conscious decision about where to expose particular tones in the scene.

ETTR was popularized by (and maybe invented by, depending on who you believe) Michael Reichmann, the publisher of the The Luminous Landscape, in this article. Michael credits the original idea to a comment by Thomas Knoll, the chief architect of Adobe's Camera Raw. What the comment amounted to is that because camera sensors work in a linear space, while we see images in a gamma space, you maximize the signal to noise ratio of the sensor by exposing as much as possible "to the right", where there are more A/D converter codes available. (Read the LL article if you want more detail). As you would expect from Thomas, that's 100% right. And thus was ETTR born.

Note that there are a bunch of different variation on what ETTR really is, and religious wars to go with those variations - some variations focuss on exposure adjustments upwards, some downward, and some in both directions:

"Overexposure ETTR" works by overexposing low contrast scenes. What this means is that noise is reduced, and you get the benefit of the greater number of A/D codes to the right. This is the "classic" version of ETTR that the Reichmann article focuses on.
"Underexposure ETTR" works by underexposing on high contrast scenes, placing the highlights on the right, but driving shadows down to the left. This preserves the highlights, but at cost of generating greater noise in the shadows.

What I'll show here applies to ETTR adjustments in either direction, although I'll focus on the "classic ETTR" as described in the Reichmann article referenced above.

So, given I just said "100% right" to the proposition that ETTR maximizes sensor signal to noise ratio, why do I say ETTR is plain wrong? Simple. What most of the proponents of ETTR forget, or perhaps don't understand, is that maximizing the signal to noise ratio of the sensor is absolutely a good thing, but the sensor is only a small part of the image processing chain that gets you from pressing the shutter button to a print. What I'll show below is that ETTR's benefits are actually minimal except in one very specific situation, but that ETTR is actively dangerous to the rest of the image processing chain pretty much all of the time.

The test conditions

Those of you that took a look at the LL article may have noticed a critical point. No images demonstrating the improved end results from ETTR. Just theory, but without any practice. Which should have set alarm bells ringing right there. So, what we're going to do is to look at some images. First, here are the test conditions:

I've used my Canon G10. The G10 is useful in this situation, because being a 14 Mpix small sensor camera, it delivers fairly clean images at low ISO, but really noisy images at high ISO, so we'll be able to look at images at both ends of the spectrum. If ETTR is going to work, it should work at one or the other end of the G10's spectrum of sensitivity.
To ensure consistency, I've used images of a Gretag Macbeth 24-patch chart, shot on a tripod. I've masked the brightest neutral patches off to give a low contrast test image. Shots were in diffuse daylight.
Exposure values were determined such as to ensure that no channel went into saturation - one of the practical problems with ETTR is that it's very easy to blow the red channel, and end up with color shifts because of that alone. However, that's not what we're investigating today, so histograms were carefully monitored.
I've used Capture One 4.8.3 and Lightroom 2.4 for these tests. All setting were default, other than white balance, which I set to Daylight for all images (as shot was 5000K), and whatever adjustment to the exposure slider was required to compensate for the degree of ETTR push or pull I applied. In all cases, "correct post ETTR exposure" was to be close to the theoretical l value of 50.867 (in L*a*b values) for the CC22 patch - the lightest patch I left uncovered. I made no changes to any other setting (contrast, black point, brightness, curves, etc) - all were left at their defaults for the particular program.

A typical test image looked like this:

Sample test image full size

Testing at low ISO

So, let's get to the images. As a reference point for some low ISO tests, we'll use a ISO 200 image. I've selected 200 so that we can compare to a ISO 100 image later, and we will be using Capture One. Further, we'll be looking just at the intersection of the CC4, CC5, CC10 and CC11 patches (the "foliage", "blue flower", "purple" and "blue green" patches) at 100% scale, as shown in the next image."

ISO 200, 1/15 sec, f/4.5 crop - normal exposure

Capture One V4.8.3, G10 Generic profile

So what happens if we "expose to the right" by one stop - aka overexpose the image by one stop (so +1 stops), then bring it back to the same exposure as the first image by adjusting exposure by -1 in Capture One, so getting us back to 0:

ISO 200, 1/8 sec, f/4.5 crop, +1 stop ETTR exposure

Capture One V4.8.3, G10 Generic profile

So, on an immediate glance - well, not much. Let's overlay the two images to get a better idea of the differences. Here what I've done is to overlay the center part reference image with the "ETTR image" - I've offset them slightly so that its easier to see where the ETTR image starts and stops:

Outer: ISO 200, 1/15 sec, f/4.5 crop - normal exposure

Inner: ISO 200, 1/8 sec, f/4.5 crop, +1 stop ETTR exposure

Capture One V4.8.3, G10 Generic profile

Taking a look at the images overlaid you can see that the inner, ETTR exposure really is just a little better; there is less noise, more or less as ETTR promised. (ETTR fanatics should stop reading now). But that's not the end of this. I choose ISO 200 for a reason, let's take a look at a 100 ISO image, superimposed on our better quality ISO 200 ETTR image:

Outer: ISO 200, 1/8sec, f/4.5 crop, +1 stop ETTR exposure

Inner: ISO 100, 1/8sec, f/4.5 crop - normal exposure

Capture One V4.8.3, G10 Generic profile

As you can see, there's no real difference. So, in this example, all the benefits that you can get by one stop of ETTR can also be obtained by just adjusting the ISO setting down by notch.

This isn't some kind of an aberration that's specific to the G10 or these ISO setting either - there's a good theoretical reason. Take a look at the actual exposure values of each image - they're the same - 1/8sec, f/4.5. Now sensors, either CCD or CMOS essentially work by accumulating electrons in the sensor well; the more electrons, the higher the brightness of that pixel. In this case, because the exposure was the same, the number of electrons was the same. But noise in sensors, counted in terms of number of electrons, is relatively fixed. So for the same exposure, regardless of ISO setting, you will tend to get roughly the same amount of visible noise. The reason why noise increases with ISO setting isn't actually because the amount of noise increases on an absolute scale, it is because as you increase ISO setting, the white point goes down, so the same number of noisy electrons become a larger and larger part of what you see. For example, at ISO 200, you might have 10000 image electrons, and 100 noise electrons, while at ISO 100 you would have 20000 image electrons, but still the same 100 noise electrons. Overexpose the ISO 200 image by 1 stop, and you double the electrons to 20000. So you're right back to the noise level of your ISO 100 image. Only you have to adjust the camera, and adjust back in post, all to get exactly the same result as just changing the ISO setting.

High ISO Test

Ok, the low ISO tests suggest we're wasting our time with ETTR. But let's try that at high ISO, on a really noisy image. First, lets compare a normally exposed ISO 1600 image (outer) to a ETTR exposed ISO 1600 image:

Outer: ISO 1600, 1/125sec, f/4.5 crop - normal exposure

Inner: ISO 1600, 1/60sec, f/4.5 crop, +1 stop ETTR exposure

Capture One V4.8.3, G10 Generic profile

Again, as we'd expect, the ETTR exposure shows less noise. However, again, lets also take a look at the ETTR exposure versus just reducing ISO by a notch and exposing normally:

Outer: ISO 1600, 1/60sec, f/4.5 crop, +1 stop ETTR exposure

Inner: ISO 800, 1/60sec, f/4.5 crop - normal exposure

Capture One V4.8.3, G10 Generic profile

Interesting - the normally exposed inner is actually a bit better than the ETTR equivalent. Why? - because the G10 applies its own internal noise reduction algorithms, based on ISO setting, and Canon's engineers, as you might expect, know a few things about their sensors. So here what we have is that the results as delivered by the camera, exposed normally at a lower ISO, are better than using ETTR. In other words, here ETTR gave a worse image than just exposing normally, and letting the camera do its stuff.

The One Situation where ETTR does work

What the tests above establish is that as a general rule, ETTR is no better than just adjusting the ISO setting down, and in some situations, e.g., where the camera does its own noise reduction, is worse than just exposing normally.

But there is one situation where ETTR can help - when you're already at the lowest ISO setting you camera offers. Take a look at the next image overlay:

Outer: ISO 100, 1/8sec, f/4.5 crop - normal exposure

Inner: ISO 100, 1/4sec, f/4.5 crop, +1 stop ETTR exposure

Capture One V4.8.3, G10 Generic profile

While the G10 has a ISO 100 setting it doesn't have a ISO 50 setting. (It does a ISO 80 setting, but I'm ignoring that as it's too close to 100 to make much of a difference.) So what ETTR is doing here is allowing us to synthesize a lower ISO setting, and hence a better noise performance, than the camera actually has. The disadvantage of course is that the camera's dynamic range is reduced by one stop, but if you have a low contrast image, that might be a price worth paying. But that's not the only disadvantage of ETTR:

The real problem with ETTR - color and tone curve shifts

We've established that outside of one situation - the situation where you're already at the lowest ISO setting you camera has - ETTR offers no practical advantage, and in some situations such as high ISO, may be an active disadvantage as regards noise performance. But now we get to the unpleasant part - color shifts. I already mentioned in the "test conditions" section that when using ETTR it is easy to blow the red channel, and get color shifts as a result. However, what I'll show in this section is that even without blowing a channel, you can still get color and tone curve shifts. Just to emphasize again - none of the images in this article have blown channels.

Firstly, take a look over the previous sets of overlaid images. If you take a close look, and some of you with sharp eyes may have noticed this before, there are some subtle color shifts, especially in the lower right green-blue patch.

However, the issue becomes a lot clearer when you look at how Lightroom responds to ETTR. In this case, we'll use a 2 stop ETTR to make the difference clearer:

Outer: ISO 100, 1/15sec, f/4.5 crop, normal exposure

Inner: ISO 100, 1/4sec, f/4.5 crop, +2 stops ETTR exposure

Lightroom 2.4, Adobe Standard profile

Notice how much more difference there is between the inner and outer parts of the image. To some extent, that's because of the 2 stop difference in noise. But also take a look at two things:

The color difference in the lower right green patch;
The difference in brightness between the border areas - the normal exposure is darker than the ETTR exposure; in fact, of all the overlay images, this is only overlaid image where you can see a distinct difference in the border.

And the difference isn't just random - when you look at the numbers, a pattern emerges:

What you can see is that for each individual ETTR value (each setting of the exposure slider, -1, 0 and +1), the values are consistent, but the L value for the border area, and the b value for the green patch change as ETTR changes between those values. In other words, the color and tone curve is consistent between the ISO 100 and ISO 200 images, but inconsistent between the images with different ETTR values.

So what's happening here? Two things. Let's go back to the theory - Firstly, all raw developer programs apply a tone curve to raw images. In Lightroom, this is just called "Brightness", in Capture One it is explicitly called a "Film Standard" curve, and in Aperture it is called "Raw Boost". However, in Capture One, the curve is applied before the tone curve, while in Adobe products such as Lightroom, the curve is applied after. So Lightroom shows a shift in brightness with changes in Exposure setting, Capture One doesn't.

The second thing that's happening is "hue twists". Hue twists are deliberate changes to colors in an image to give a more pleasing result. So in current versions of Lightroom, you can choose between a number of different profiles, e.g., "Adobe Standard", "Camera Neutral", "Camera Portrait", etc. Each of these is designed to give better colors under specific circumstances. So, for example, a landscape profile will take a light sky blue, and make it a darker "more blue sky" sort of a blue.

So, when you offset exposures by using ETTR, what you are doing at the same time is to completely upset a whole bunch of carefully calculated tweaks to make your images look better. For example, that slightly overexposed blue sky is now way overexposed, and as a result the profile will tweak it to somewhere entirely not like a blue sky. The result is an image with a sky that looks just slightly not right. And you get to spend a lot of time in post trying to sort out subtle color and tone shifts that aren't obvious, but just make your image look slightly wrong.

For those interested in more details on tone curves and hue twists, I've blogged on them in more detail here and here.

Making ETTR work

So, ok, in practice ETTR is only useful under one very specific circumstance - when you want a lower ISO than your camera has, and you're willing to sacrifice both dynamic range and color reproduction to get improved noise performance. But ETTR does have theoretical advantages. Is there any way to get the advantages without also getting the disadvantages?

The answer is yes, but only partially, and at a price. The price is the cost of a "Gen-2" Nikon DSLR, and Nikon's NX2 software. The way this works is as follows. Newer Nikons, e.g., the D3x, have something called Active D-Lighting. In its normal variants, D-Lighting is just some optimization in software, but when set to "Active", D-Lighting actually automatically does ETTR for you. However, I say partial because it only does an "under exposure" ETTR for you; in other words it preserves highlights in high contrast scenes, but doesn't increase exposure in low contrast scenes. The clever bit however is that the camera then encodes the D-Lighting data into the raw image. Then NX2, Nikon's raw developer, can correctly adjust exposure prior to applying any tone curves or hue twists, and without you having to play with sliders. So no tone curve or color shifts. Magic. There's only one problem - this works only with NX2; Nikon have not disclosed to any other raw developer producer how the D-Lighting data in the raw file is encoded, so you only get the magic if you use a Nikon camera and NX2.

All those extra A/D converter codes

I've given a bunch of examples of how the only visible improvement that ETTR gives is as a result of lowered noise, but what about the argument in the original Reichmann article that the advantage was in more A/D codes. Well, I can't prove a negative. But the evidence on this is pretty overwhelming:

There is no sign of any visible improvement from additional codes in any of the test images shown here.
In the Nikon community, many DSLRs can shoot either compressed or uncompressed raws. The compressed raws have 683 codes versus the 4096 to 16384 that uncompressed Nikon raws have. Ever since Nikon cameras came out that could shoot both compressed and uncompressed, people have been trying to show a visible difference. They never have, at least without doing really heroic post processing, things like 4 stops of shadow recovery. And bear in mind, ETTR requires careful exposure; it's easier to get the exposure right in the conventional sense than it is to apply ETTR without blowing channels.
In the Leica community the M8 compresses down from 16384 codes to only 256 codes. Similarly, there has been a lot of testing done to try to demonstrate a loss in image quality from that compression. Including by me - see here. Likewise, nobody has succeeded in showing any difference under normal conditions.

So, unless and until someone can show me an image, normally processed from a real camera that shows a visible advantage that can't be duplicated by switching to a lower ISO, I don't see any evidence to suggest that the theoretical "more levels" advantage translates into any kind of a practical image quality advantage.

And in conclusion.......

Here's my conclusions:

There is no advantage to image quality from ETTR that can't be duplicated by selecting a lower ISO, if a lower ISO setting is available. In some situations, such as where there is in-camera noise reduction, ETTR actually increases noise. That's what the practical tests show, and the theory of the case confirms the practical results to be correct.
The only situation where there is an advantage to ETTR is if you're already at the lowest ISO setting your camera, and you use ETTR to synthesize a lower ISO. However, given the noise performance of most modern cameras, that advantage is often very small. The test I did here - a small sensor high pixel count camera - is the best possible scenario for seeing an improvement. Using a modern DSLR, the improvement would be marginal at best.
Any kind of ETTR brings significant disadvantages in the shape of color and tone curve shifts that will have to be repaired in post processing. While these shifts are small, they are easily the equivalent in effect of changing profiles. So, in effect, ETTR negates the advantages that modern raw developers such as Lightroom bring with them.

Bottom line - ETTR offers improved image quality in only one specific situation - where you can use a lower ISO setting than your camera has. In all other situations, ETTR will only ever decrease image quality.

Update : see my later post here as well, as well as the subsequent two ETTR posts, the last one of which adds another situation in which ETTR may be useful. For some cameras, if you're willing to overexpose by four (yes four!) stops.

dcpTool in the news.....

2009-08-13T07:54:00.000-07:00

dcpTool, and the whole Adobe hue twist story been getting some attention on various forums, primarily with respect to skin tone rendition with the Canon 5DII - at least some people have found that untwisted and/or invariate profiles are giving them better colors than the usual Adobe profiles.

There's a long thread on dpreview here.

And there's also a thread on DCHome here.

The DCHome thread has some example images showing different renderings by TK Chan, who, judging by his gallery, is a seriously talented photographer.

Unfortunately, Sourceforge's statistics system is bust (again), otherwise I'd be able to quote some statistics as to how many hits/attention resulted.

pcdtojpeg in use....

2009-07-17T13:17:00.000-07:00

One of the pleasures of writing imaging software is that you occasionally get to see some really stunning images that were processed through your software.

David Ryan has just published some images that I think are just great. According to David, they were digitized back in the early 90's have been languishing as a pile of CD's ever since. David's now used pcdtojpeg to convert them from Kodak Photo CD format, and put them up as part of his gallery - you can see the collection on David's site here: http://www.davidryanphotography.com. In particular, I think that David's "San Francisco Doors" collection is just great - the use of color and light to turn objects that are quite mundane into something extraordinary is just amazing. And also a pretty good testimonial to the quality levels that pcdtojpeg can deliver(!)

You can also see what David has to say about pcdtojpeg on his blog.

Extracting thumbnails from DNG files

2009-06-27T00:29:00.000-07:00

So I tested V1.0.0.1 of CornerFix on Windows 7, and guess what - it broke. Specifically, while images were converted, corrected, etc, they weren't displayed.

A little digging around showed that Microsoft, in their wisdom, have changed the behavior of the .Net 2 Picturebox control. Previously, under both XP and Vista, if the Picturebox control came across a file that it couldn't decode, but could extract a thumbnail from, it would display the thumbnail. Entirely logical and useful behavior, and what CornerFix depended on. That has changed in Windows 7 - all you get now is that Picturebox throws an exception, and displays a little cross icon.

I posted a query on the .Net part of the MSDN forum, and of course got the helpful suggestion (from a Microsoft employee) that as this was "a Windows 7 problem", I should post on the general Windows 7 forum. Very useful. Let's do anything we can to avoid actually solving the problem.

So given that Microsoft weren't going to offer any work-around, I got to thinking about how to solve this one myself. My first thought was to generate a JPEG from the thumbnail image data in the DNG file. A bit of work showed that while that was possible, it was going to be pretty messy. However, a better thought occurred to me. A DNG file is actually just a TIFF with additional tags, and a sub-IFD structure. In fact, what a DNG consists of is a single IFD which contains the thumbnail as well as the main image in raw form as a sub-IFD. Now the thumbnail is actually a perfectly valid TIFF image, it's just tagged via the kcNewSubFileType tag as "1" to show its a thumnail. In fact, the TIFF/EP spec requires that "In TIFF/EP files, the 0th IFD should be an image that can be read by a baseline TIFF 6.0 reader." So to get a valid TIFF file, which Picturebox can display, all that's required is to extract the thumbnail in the 0th IFD, and relabel it as the main image.

So, that's what I did. Now CornerFix uses Adobe's DNG SDK, so in theory I could have used that, but a quick look showed that the SDK isn't designed to extract thumbnails. In fact, it pretty much ignores them. So I decided to code a completely separate thumbnail extractor in C++, on the basis that it would be useful as a standalone product.

Because TIFF is a complex format, it turned out to be a more complex piece of code than I'd hoped, but I got it done, and its now part of CornerFix V1.1.0.0, which shipped a few days ago.

For those interested in using it, its just a single C++ file - tiffThumb.cpp - and its associated tiffThumb.h file, with no dependencies on anything else. The API is simple - it takes a file name, and return a memory buffer with the TIFF thumbnail file in it. I've tested it on both Windows and OS X. The documentation is in the tiffThumb.h file.

The file can be downloaded as part of the CornerFix source code, here.

keychainDD and pcdtojpeg are out.........

2009-06-02T08:19:00.000-07:00

So I've been lazy about blogging, and productive about writing software. Since I last blogged about software releases, I've released not one, but two new open source applications, keychainDD and pcdtojpeg.

keychainDD
keychainDD is an OS keychain manager that delivers very secure drag&drop capability to the OS X Keychain management system. Why I wrote it was that OS X has a very robust password management system in the Keychain system, but in my view, there were two deficiencies:

No drag&drop - you can have Safari (or whatever else) autofill your passwords on web pages, but I've never been very comfortable with that - I'm always suspicious that even a badly spoofed website may be good enough to get any autofill program to cough up my passwords. And copying via cut&paste is vulnerable to keyloggers (yes, there are keyloggers for OS X). So keychainDD only allows you to explicitly drag&drop passwords. Not only is drag&drop a lot more secure on OS X than cut&paste, but the way that keychainDD implements drag&drop is in a very secure way, btw.
No support for "Memorable Information". A lot of financial sites that I use now require what is effectively a second password, usually one that you have to enter a few selected characters via an on-screen menu or keyboard. This is a protection against keyloggers, and in my view a very good thing. But you have to remeber that information, and be able to count characters to get the 5th character, the 3rd, etc. keychainDD supports memorable information, and has a neat character-by-character tray type display that means you don't have to write down your memorable information to count characters.

So, anyway, that's keyChainDD. It's website is here, and you can download it here.

pcdtojpeg
pcdtojpeg converts Kodak Photo CD (PCD) files into JPEG files. I wrote it because basically every other solution out there for doing any kind of PCD conversion just sucks. Either they blow highlights, get colors wrong, only convert at very low resolution, or just don't work all. I won't mention any names here, but pretty much every other solution out there that I tried doesn't work. And don't just believe me, take a look at Ted Felix's PCD site.

pcdtojpeg gets the color right, won't blow highlights, and runs under any of OS X, Windows or linux.

pcdtojpeg also isn't just a monolithic application, it's PCD decoder comes in the form of a proper C++ library that other programs can use.

Acknowledgments: Although pcdtojpeg shares no code with Hadmut Danisch's hpcdtoppm, and can decode image information that hpctoppm can't, pcdtojpeg would not have been possible without the work that Hadmut did in reverse engineering the format in the early 1990’s in order to write hpcdtoppm. For those interested, hpcdtoppm converts PCD images into ppm format images.

pcdtojpeg's website is here, and you can download it here.

CG Pipelines and CIContext Bugs Part 2

2009-05-31T07:31:00.000-07:00

So, after a long break due to work pressures, back to the CG Pipelines and CIContext Bugs issue.

What I ended up doing was to modify my application to allow me to build a test image, and then assign any profile that I wanted to that image, and to any of the subsequent stages. So the points at which I could set a profile were:

The test image - the input image for the pipeline;
The working space of the CIContext that I use for processing;
The output space of the CIContext that I use for processing;
The rendering space that render a final image into.

I also provided the ability to either create images via createCGImage or by a drawImage/CGBitmapContextCreateImage sequence. Finally, I built a CIKernel that either clips input RGB values to a predetermined value, or sets RGB values to a predetermined value. This CIKernel allowed me to find out what the actual working space of the filter was.

Armed with this collection of things to test, I went through pretty much every possible combination of settings that I could find. The results of this were interesting (and, as you will see, somewhat confusing):

On entry to the processing pipeline, RGB values are clipped to the working space of the pipeline, if you set a working space. In other words, the maximum values for R, G and B are set to 1.0 in whatever the working space of the pipeline is. The working space is set by the kCIContextWorkingColorSpace option of the contextWithCGContext method.
As is claimed by several references on the web, the default working space for a CIFilter is kCGColorSpaceGenericRGBLinear. Default in the sense of what you get if you don't specify a working space via kCIContextWorkingColorSpace. That is, Apple's Generic RGB, set to a gamma of 1. However, there's a twist to that - there is no clipping of the input image on entry to the processing pipeline if you don't set working space.
So far as I could tell, the output space, as set by the kCIContextWorkingColorSpace option has no effect at all.
Contrary to what has been stated on the Web in a few places, you can set the working space to a space with a gamma not equal to one, and the pipeline and your CIFilter will still operate correctly. Mostly. The "mostly" is that if you try to have two different pipelines in the same program, with working spaces set to different gammas, Core Image will get confused, and sometimes give you the right primaries, but the wrong gamma.
There is, as far as I can tell, never any clipping other than on the input to the pipeline - so its entirely possible, if you're using float values, to get outputs that are well out of gamut of the working space. You need to clip those, or using integer values, which will be automatically clipped.
Using a drawImage/CGBitmapContextCreateImage sequence to generate an image rather than createCGImage is not reliable for large images - at some point, drawImage/CGBitmapContextCreateImage just seems to run out of memory (or something). Worse, it fails silently - you just get a blank image.
Rendering intents are honored, but only on the input side. In other words, if your input image has a profile with a rendering intent, that rendering intent will be used when converting to the processing pipeline's working space. However, setting the working space (or output space or rendering space) to have an intent doesn't work.

The upshot of all of this - if your needs for a CIFilter processing pipeline are simple, then you're probably better off just NOT setting the working space. That way, there's not clipping anywhere in the pipeline, and as long as you render the image correctly, all will be well. It's very probable that Apple specifically designed the "default color space doesn't clip" behavior quite deliberately to allow reasonably simple image processing applications to just do their stuff without getting worried about color spaces.

If however you are in the kind of a world where your CIFilter actually needs to be color space aware, then you should set the space to something with a gamma of 1, and it should not have rendering intents (BToA tags or AToB tags). Based on the results above, while non-1.0 gammas work under some conditions, they don't always work.

A final note: The ICC (international Color Consortium) have some very useful profiles, the "Probe Profiles" on their website. These allow you to unambiguously find out whether rendering intents are being honored.

CG Pipelines and CIContext Bugs

2009-03-08T07:30:00.000-07:00

So, I've been starting to build a simple soft-proofing application for Mac OS X. the concept being something simple that could take a TIFF or JPEG as input, then show what the rendering would be, and where any out-of-gamut colors were for any given output profile. E.g., a printer or whatever.

I decided to this via Core Image, for reasons of performance. Core image allows you to write CIFilter's that run on the graphics hardware of your PC, and so can be very, very quick. And, after about a week of frustration, I nearly had a CG processing pipeline up and working to my satisfaction.

There's just one thing that I absolutely couldn't work out, which is CGContextDrawImage's handing of out-of-gamut colors. Here's what the basic processing pipeline is:

I'm creating a CGImage in a wide gamut space, in floating point format.
I then create a floating point bitmap CGContext, and in turn create a CIContext from that.
I then render a new CGImage, using CGContext DrawImage and then CGBitmapContextCreateImage.

So, all very simple. Not.

What I found was the following:

If the CGContext that I rendered the new image to was in floating point format, colors were correctly converted to e.g, the sRGB space, but components are left out of gamut - e.g., <0.913567>. "Correctly" here meaning that the values are exactly the same as what you would get using the ColorSync Utility's calculator.
If the CGContext that I rendered the new image to is in int format, the components are clipped (obviously), but otherwise the values are identical to the floating point case. This occurred regardless of what the rendering intent was set to - there is not even one decimal points change in the values with rendering intent changes.

What I expected was that for both the floating point and int formats, the rendering process would use the rendering intent to adjust colors into gamut, and these adjustments would vary with intent, but as far as I could see, the rendering intent is just ignored, and the only gamut adjustment that occurs is hard clipping.

So, the first thing that I did was to post a query on the Apple development lists. And got only one response, which was unfortunately useless, as the individual in question hadn't actually read my post.

So I ended up spending the next week testing an enormous number of different options of how CIFilters and rendering actually works inside of a CIImage context. The results are surprising. At least they were to me.

More later......

Visualizing DNG Camera Profiles Part 3

2009-02-20T06:50:00.000-08:00

In this post, I'll look at the renderings of the various Adobe DNG profiles on a profile by profile basis.

If you haven't taken a look at Part 1 of this, I'd strongly suggest you do - it's there that I set the scene for these charts, and talk about how to interpret them. Part two is here.

As a reminder, what we're looking at here is the color rendition embedded in the various default Adobe camera profiles for the Canon 5DII. We're looking at that for colors that approximate to the key color patches on a Gretag Macbeth 24 patch color chart. Also, remember that the various "camera" profiles are intended to mimic the camera settings, not necessarily to be what Adobe might consider to be pleasing color renditions.

Firstly, the Adobe Standard profile:

The Adobe standard is really very "neutral" it stays quite close to being a conventional "conversion matrix" based color rendition, similar to the previous generations of Adobe profiles. There are very few hue twists to be seen, and only some color value compression in on red and blue.

Now the Camera Standard Profile:

Camera Standard is bit further from a "plain vanilla" rendering. Here we see more aggressive compression of the red and blue, extending to magenta as well. Also green and yellow have "twists" to them.

The Camera Portrait profile:

Camera Portrait is, as you might expect, really quite aggressive with its changes to the conventional "plain vanilla" camera rendering. Red, magenta and cyan are heavily twisted, and blue is again compressed, as is green.

The Camera Neutral Profile:

The Camera Neutral profile is an interesting one. Both red and magenta a quite close to vanilla for a while, but them abruptly flatted out in value almost completely. Blue and cyan are compressed, and green twists quite substantially.

The Camera Landscape profile:

Camera Landscape is, as we might expect, heavily "compressed", and has some quite substantial twists. Green, red, blue are both compressed and twisted, as is yellow, not a color that other profiles have compressed very much.

And finally, the Camera Faithful profile:

Camera Faithful turns out to be very similar, perhaps unsurprisingly, to Camera Neutral, only quite subtle differences - e.g., in magenta, separating them. But theses are differences subtle enough that I doubt that it would be possible to tell the difference in practice - differences in the tone curves of the two profiles are more likely to be visible than these differences.

So what are commonalities?

Well, there's a fairly common pattern of the the primaries (red, blue, green) being compressed, and for red to be twisted towards magenta, and blue towards magenta
The skin tones tend to be left alone, probably because human vision is quite sensitive to any variation here. Similarly, yellow is mostly left alone, perhaps because its quite close to skin tones. Camera Landscape is the exception to yellows being fairly unprocessed, perhaps in the interest of more attractive greens

Visualizing DNG Camera Profiles Part 2

2009-02-19T08:56:00.001-08:00

In this post, I'll look at the renderings of the various Adobe DNG profiles on a color by color basis.

If you haven't taken a look at Part 1 of this, I'd strongly suggest you do - it's there that I set the scene for these charts, and talk about how to interpret them.

As a reminder, what we're looking at here is the color rendition embedded in the various default Adobe camera profiles for the Canon 5DII. We're looking at that for colors that approximate to the key color patches on a Gretag Macbeth 24 patch color chart. Also, remember that the various "camera" profiles are intended to mimic the camera settings, not necessarily to be what Adobe might consider to be pleasing color renditions.

First up, this is the chart for the three main Gretag patches - CC13 (blue), CC14 (green) and CC15 (red)

What you see here is a few things:

The Adobe Standard profile, is, in almost all cases, closest to the reference color
Reds are easily the most "manipulated" color. This is interesting in light of the fairly frequent complaints about "Adobe Red" with previous version of ACR and Lightroom; it may well be that the fairly aggressive reductions in red are a fix for this. Even the Adobe Standard profile reduces the magnitude of red quite substantially.
The largest reduction in red, and the biggest hue shift for red is in the Camera Portrait setting. Reasonable, given that red faces are a frequent complaint.
Green is mostly fairly "untwisted, with the exception of the Camera Portrait setting, which drift across towards yellow quite significantly.
Green is also fairly compressed at the upper end - Camera Landscape flattens the rendition quite significantly, presumably to keep the greens in foliage either from getting to light and washed out, or too dark.
Blues are a mixed bag. The Camera Faithful, Camera Neutral and Camera Standard profiles reduce blue intensities considerably. The Camera Landscape setting however allows significantly higher intensity, but flattens out aggressively. Presumably this is to give really blue skies, as opposed to somewhat over-exposed white-blue skies you often see.

Next up, we'll take a look that the yellow (CC16), magenta (CC17) and cyan (CC18) patches:

What we see is:

Yellow is very close to reference in almost all cases. Some profiles reduce intensity, but there are few dramatic hue shifts. One reason for this is probably that yellow is quite close to the various skin tones, and significant hue changes would probably be undesirable.
For magenta, the biggest variation is in the Camera Portrait profile, with intensities being compressed into a relatively small, quite high, range. This again is probably to maintain skin tones regardless of exposure. Camera Landscape compresses magenta down quite significantly.
In cyan, the big variation is in Camera Portrait. Again, almost certainly to preserve skin tones. Similar to the case for blue, the Landscape Profile seems to try to preserve sky tones.

Finally, the two skin tones, light skin (CC2) and dark skin (CC1). What's interesting here is that there is very little hue shift. Presumable because human color perception is very sensitive to skin tones - even small variations are regarded as unpleasant.

In part 3, I'll show the charts on a by profile basis, rather than a by color basis.

Fixing the CIAnnotation Sample

2009-02-18T09:36:00.000-08:00

A complete digression, but for anyone struggling with the fact that the Apple CIAnnotation sample doesn't work under Leopard (OS X 10.5), the biggest problem seems to be in the CITextLayer.m module, where the graphics context is not restored. Note there are other issues, but at least after this mod it no longer just crashed in autorelease.......

The code is:

- (void)renderText
{
// release the old image
[textImage release];
textImage = nil;
//First line to change
NSGraphicsContext *cContext = [NSGraphicsContext currentContext];

// render the text objects into the context
CGContextClearRect(bitmapContext,layerRect);
NSGraphicsContext *graphicsContext = [[NSGraphicsContext graphicsContextWithGraphicsPort:bitmapContext flipped:NO] retain];

[NSGraphicsContext setCurrentContext:graphicsContext];
[textObjects makeObjectsPerformSelector:@selector(renderIntoCurrentContext)];

if(layer)
{
// we can render using the CGLayer
textImage = [[CIImage alloc] initWithCGLayer: layer];
} else {
// create a CIImage from the bitmap context by the way of creating a CGImage - this causes a copy !
CGImageRef bitmapImage = nil;

bitmapImage = CGBitmapContextCreateImage(bitmapContext);
if(!bitmapImage)
return;
textImage = [[CIImage alloc] initWithCGImage: bitmapImage];
CGImageRelease(bitmapImage); // now retained by the CIImage
}
[invertFilter setValue:textImage forKey:@"inputImage"];
[blurFilter setValue:[invertFilter valueForKey:@"outputImage"] forKey:@"inputImage"];
[compositeFilter setValue:textImage forKey:@"inputImage"];
[compositeFilter setValue:[blurFilter valueForKey:@"outputImage"] forKey:@"inputBackgroundImage"];
// Second line to change
[NSGraphicsContext setCurrentContext:cContext];
}

Visualizing DNG Camera Profiles Part 1

2009-02-18T04:35:00.000-08:00

In previous posts, I've talked about DNG Camera Profiles, and the "hue twists" that Adobe put them in order to get "pleasing" color. "Pleasing" in this context has, as far as I can gather from Adobe's various comments on the subjects, at least two aspects to it. The first aspect is "color like the JPEGs from my camera" request that comes up on forums regularly. On the more technically sophisticated forums, this comes up as "the same color as NX2" for Nikon and "the same color as DPP" for Canon. This is the reason for the "Camera Landscape", "camera Standard", etc profiles that Adobe now supply. the second aspect is just to get colors that people like. This seems to be where the "Adobe standard" profile comes in.
Jumping right in at the deep end, here is the visualization of all of the Adobe supplied profiles for the Canon 5DII:

Ok, so what IS that? Here's the basics:

In order to make this more accessible, I've built the diagram using Apple's "Grapher" program, that's supplied with all Macs. You can find the original gcx files on the ChromaSoft site here. If you have a Mac, I very much suggest you download those files, and play with them - they allow you look at individual profiles, individual colors, rotate and tilt the graph, etc. Far better than looking at static images.
What's included is data for each of the supplied profiles for the Canon 5DII - Adobe Standard, Camera Neutral, Camera Portrait, Camera Landscape, Camera Faithful and Camera Standard.
For each of those profiles, I've plotted the before and after colors for the hue and saturation values associated with eight of the most important patches on a Gretag Macbeth 24-patch color checker card - blue, green, red, yellow, magenta, cyan, light skin, dark skin.

The Color Space
The plots are in the coordinate space that DNG Camera Profiles use - an HSV, Prophoto gamma 1 space. HSV coordinates are cylindrical. So, if you imagine a cylindrical jar with a color poibt in it, the hue is the angle from the center of the jar, the satuaration is the distance from the center, and the value (loosely, the intensity) is the height from the base of the jar to the color point. The next chart shows how blue actually looks in this space.

Each of the little spheres is a color point. On the base of the cylinder, they are black, because V is zero. They then get brighter as height up the cylinder increases. Saturation is zero in the center of the cylinder, and increases as you go towards the edge. So, the top center is white, and the top edge is a pure color. Finally, as you rotate around the center, the hue changes.

Important note on colors. Some things you need to know as you look at the charts:

You might notice that the little spheres above look pretty bright, even as you get to the middle of the cylinder. That's because of the gamma 1.
On all of my plots, there's the disk at the bottom that shows the color spectrum. Firstly, the disk's at the "wrong" end of the cylinder. The real disk would be black, because at the base, V is zero. A disk like that, in full color, belongs at the top. Secondly, the disk doesn't show saturation. A real disk would have those colors at the edge, then get progressively less saturated towards the middle, as the blue sphere's do. Unfortunately, Grapher can't deal with varying saturation(!)
The colors are approximate. Grapher isn't color managed, and can't deal with any more than one color in a point set.

So, bottom line, the disk is there only to give a rough feel for in what direction hue twists occur, NOT to indicate actual colors.

So, having that behind us, the next graph shows the color changes for just one color (the green patch), in just one profile (Camera Standard). The line with the boxes is the original color, and the line with the spheres is the new "post profile" colors.

To make that (perhaps) a bit clearer here's the same data, just with the before and after points joined.

What you see is the new colors "twisting" round in front of the "normal" colors. So what's happening in this case is:

At low V, the hue is biased off towards blue
As V increases, the hue moves back to wards the reference norm, and becomes more saturated than the norm
Also as V increases, the V of the resulting color is less - a less bright color
Finally, the new color twists back, and desaturates relative to the norm.

In the next post, I'll show this for each of the colors in turn, across all profiles.

dcpTool Version 1.1 is out!

2009-02-14T03:51:00.000-08:00

dcpTool V1.1 is out - a little sooner than I'd been expecting to update it.

Get it here: http://dcptool.sourceforge.net/

What I've done, in response to requests from folks on the Adobe forum, is to add the capability to modify DNG Camera Profiles to either change, or entirely remove the "hue twists" that I described in a previous post.

The previous modifications that I had made to the profiles was simply to move the "LookTable" to the "HueSatDelta" table - this means that any hue twists are processed before the exposure controls, so the problem that changing the exposure control changes colors goes away. However, some further tests have made it clear that profiles don't always respond well to simply being moved from point to point in the processing pipeline. So, in addition to the ability to change the processing pipeline order, I've also added the ability to effectively freeze, or "untwist" a profile - so the processing pipeline remains the same, but there is no change in tint when you adjust exposure.

This "untwisting" is my recomendation as to how to go forward if hue twists are giving you a problem; in my tests so far, color reproduction is very similar to the original profile, but the color shift problem is entirely eliminated.

There's more detail on exactly what dcpTool does to untwist profiles here.

The Adobe "Hue Twist"

2009-02-10T04:53:00.000-08:00

There's been a conversation on the Adobe forums about some unexpected changes in tint when the recover and exposure controls are used in Adobe Camera Raw (ACR), and presumably in Lightroom as well. With input from Eric Chan of Adobe, what started out as a discussion of a possible bug in the ACR quickly moved towards the understanding that the unexpected shift in tint was by design, and part of Adobe's new generation of color profiles. DNG Color Profiles allow for a complex set of what are in effect look-up tables that can translate from a color as photographed to another color as displayed. What's more, these look-up tables provide for the ability to make these color shifts dependent on the intensity. So two pixels with the same tint (a hue and saturation combination) but different intensities (values), can translate to pixels with different tints. This is what Adobe's new generation profiles do.

The chart below shows before (outside) and after (inner) patches for the Adobe Standard profile for the Canon EOS 5D Mark II, for the same tint of yellow/orange discussed in the posts on the Adobe forum (ProPhoto 244,161,65), varied across the V scale. Note that shows only a portion of the scale; the sRGB encoding needed for display on a web page limits the gamut significantly.

Note to Safari users: These images won't display correctly in Safari on a wide gamut monitor - unfortunatly, Blogger strips the sRGB tags, and Safari defaults non-tagged images to monitor space. FireFox will work fine.

The patches have V values of 0.0667, 0.1333, 0.2667, 0.4000, 0.5333 and 0.6667 in the ProPhoto gamma 1 space that DCP files use.

The next chart shows just the tint impact as V is changed. In other words, the V is held constant, to more clearly show the tint impact in isolation.

Calling what's done a "twist" is probably appropriate, looking at the numbers for the tint in question:

What's happening is that the hue adjust is effectively rotating in HSV space, through an angle of -7.11 degree to +19 degrees as V changes, effectively following a twisting trajectory in HSV space.

The final thing that's worth noting is that in the Canon EOS 5D Mark II profile, the twist is implemented as a "LookTable". A LookTable is explicitly specified as something that is applied after exposure adjustments. But it would be possible to implement the same table as a HueSatDelta table, which is implemented before exposure adjustments. This would avoid the unexpected tint change that started the original thread on the Adobe forum.

In conclusion, the way in which Adobe have implemented their new generation profiles makes them vulnerable to tint shifts as a result of exposure adjustments. In most situations, that probably won't be an issue. However, if these tint shifts are a problem, for example if you need to do aggressive exposure recovery, falling back to the older ACRxx profiles that don't have hue twists in them may be a useful option.

Late Addition: Also take a look at my later post about visualizing camera profiles with hue twists in them here.

Problems Uploading Web Project Content to SourceForge

2009-02-09T10:12:00.000-08:00

As mentioned in a previous post, getting dcpTool out the door was a little bit of a problem.

dcpTool is hosted on SourceForge. Previously I've uploaded project web content without any problem, for CornerFix and tnefDD, using SFTP via Cyberduck, which is a neat OS X FTP client. However, since then, SourceForge have done a system migration. And while I could easily use CyberDuck to upload to the SourceForge file management system, no way could I access the web site area. Not of course that I was helped by SourceForge's woefully inadequate documentation.

But to cut a long story short, it appears that the problem is actually more of a Cyberduck problem than a SourceForge problem. SourceForge have implemented their project web area access as an "authenticate against" structure. So, under command line sftp, you need:

sftp sandymcg,dcptool@web.sourceforge.net:htdocs/

The comma syntax being the issue. Normally, you don't need the ",dcptool"

Well, anyway, that works for the command line sftp client. It doesn't for Cyberduck. All you get are "Login failure" errors.

So, if you're uploading project web content to SourceForge - be very careful of which client you use....

BTW, I have registered a bug with the Cyberduck folks.

Update:

This is solved. Turns out that both Cyberduck and FileZilla automatically add the "@web.sourceforge.net", so if you want to log in, you need to use "sandymcg,dcptool", without the "@web...". Not doubt blindingly obvious to those that deal with sftp clients everyday, but not all obvious if you don't. Anyway, the Sourceforge documentation has now been improved to make it clear that the user id doesn't have the server address.

dcpTool is out!!

2009-02-09T09:57:00.000-08:00

dcpTool did indeed ship out yesterday:

dcpTool is a compiler/decompiler for DNG camera profiles (DCP files), as used by Photoshop, Camera Raw and Lightroom. dcpTool can decompile binary format DCP files into an XML format for editing, and then compile the XML format file (editable with a text editor) back into a binary DCP file, as well as extract embedded profiles from image files. It runs on Windows and OS X command lines.

The web site, which also has some background material on DNG Camera Profiles and an example of a "decompiled" one, is here: http://dcptool.sourceforge.net/

Health warning: dcpTool is a command line utility - if you're not comfortable with command line stuff, dcpTool probably won't be of any interest to you.

BTW, "DNG Camera Profile" is probably not a good name. DNG Camera Profiles are used when importing any raw file into Photoshop or Lightroom, not just DNG format images.

There's also a discussion on the Adobe forums.

New Version Of CornerFix

2009-02-08T07:46:00.000-08:00

The new version of CornerFix (V1.0.0.0) is out. This has:

1. Lens definitions for new Leica lenses
2. Fixes some minor memory leaks in the Mac version
3. Updates the Windows version to VC 2008 - that brings with it some advantages as regards how Vista does its UAC. Technically, CornerFix is no longer in a virtualised sandbox, but is a fully fledged Vista application. But its unlikely that any users will notice the difference.

Next steps for CornerFix are likely to be an upgrade to V1.2 of the DNG spec, and also an upgrade to enable operation with almost any DNG image file. Actually, right now CornerFix does work with most DNG files, although it will spit out warnings. But It doesn't work with files that have a black level pattern - which includes some Canon and Olympus cameras.

But the priorities are to get out dcpTool, and keychainDD. dcpTool will be going out today, hopefully. Watch this space.....

Lightroom, Aperture and Capture One Mini-Review Part 6

2008-03-11T05:14:00.000-07:00

In my previous post on the subject of the color responses of Aperture, Lightroom and Capture, I mentioned that once I adjusted for exposure and contrast, the rendition that Capture One gave became very bright. The image that I was using was taken with a Nikon D80, and is not particularly badly exposed. Below is the rendering that Aperture gives completely unadjusted:

Aperture 2 test chart, unadjusted

And now the image that Aperture 2 gives when the adjusted. The process that I followed was to adjust the exposure and contrast setting of each program to get the “White” and “Black” patches to exactly the correct values. In both Lightroom and Aperture, this gives a normal looking image - ceratinly more exposed than the original, to be expected given the underexposed "grey" look to the white patch, but generally correct.

Aperture 2 test chart, adjusted for contrast and exposure

The Capture One test chart however is very different; as we might have expected from the numeric results in my previous post, it’s very bright, to the point that the image is becoming washed out. For example, the “Light Skin” patch is effectively no longer skin colored – it’s closer to white.

Capture One V4 test chart, adjusted for contrast and exposure

A numeric view confirms the different brightness behavior of the three programs. The three charts below compare the expected versus actual values for the five neutral toned patches on the D80 test image above. Both Aperture and Lightroom hold the deviation from expected to within 6 units. However, Capture One deviates from its unadjusted tone curve by 21 units, a very substantial difference. Some difference in tone curve is probably unavoidable in practice – at the end of the day, any adjust made to exposure effectively impacts on the tone curve of the raw developer in question. But its clear that Capture One’s exposure and contrast controls interact with the tone curve to an extent not seen in the other software.

A few questions come up here; firstly, is there a better way to make exposure adjustments, and secondly, is this a problem, or just a quirk? The answer to the first question is that while Capture One experts may be able to point out a way to make basic exposure adjustments that doesn’t interact with the Capture one tone curve, when I used the shadow and highlight controls, the only obvious alternatives:

I was unable to get both the white and black patch to simultaneously be at the correct value, and
Using those controls to adjust basic exposure would rather beg the question of what the role of the exposure and contrast controls should be.

To answer the second question, I attempted to use each program’s brightness control to get the “standard” tone curve for each program after making the adjustments to exposure and contrast. In the case of Aperture and Lightroom, this was quite easy, and I could “dial in” the value of each of the neutral patches to within a unit or two quite easily, using a combination of exposure, contrast and brightness adjustments. For Capture One however, it proved impossible to get close to having all the neutral patches at near to their correct values. The settings required were such that the exposure, contrast and brightness controls ended up at the extremes of their ranges. Given that what I was doing was making fairly minor (less than 1 stop) exposure adjustments, this seems to me to be a fundamental flaw in the way that Phase One have implemented their exposure controls. Now it may be that the forthcoming pro version of Capture One will function differently, but it seems unlikely that so basic a part of a raw developer would change between the standard and pro version.

Lightroom, Aperture and Capture One Mini-Review Part 5

2008-02-23T05:19:00.000-08:00

In this post, I’ll look at the color response of Capture One, Lightroom and Aperture against an image of an actual GretagMacbeth test chart, as cteated on a Nikon D80, rather than the Leica M8 I used in the previous post. As was the case last time, I first adjust the contrast and exposure setting on each program to exactly match expected values of the lightest and darkest monochrome patches on the GretagMacbeth chart. This exactly matches the exposure of the real images to the effective exposure of the synthetic image. As was the case for the synthetic images, all the test results are on a 0 to 100 scale, and represent the difference between the expected value as derived from the color values of the GretagMacbeth chart and the actual values measured. So, for example, if the red bar of the “Cyan patch” shows a value of -5, that means that the actual measured value of the R component of the RGB values as read out by the software in question was 5 units less that the theoretical value.

In the case of Lightroom, relative to the deviations in the M8 actual image from the last post, we see less negative deviations overall, indicating more saturated colors, and a significantly more saturated red patch. However, overall the picture is relatively similar to that of the M8. This starts to imply that differences in color rendering really are more to do with the raw conversion software, and less to do with different cameras.

Aperture shows a somewhat different picture. The 1.1 rendering of the M8 showed some large positive spikes in the blue component of several patches, especially the yellow patch. This doesn’t appear on the D80 rendering. However, the red component is the cyan patch is quite negative. The 2.0 rendering shows considerable change relative to the previous V1.1. Firstly, most of negative deviations have gone – the largest negative deviation anywhere is in the red component of the cyan patch, but even this is well down from the previous value. Overall, the 2.0 D80 rendering appears significantly better controlled than the previous version.

Capture One is an interesting case. At a first glance, it appears that the rendering is simply a considerable distance away from the theoretical values, almost all color components appear to be greater than the theoretical values would indicate. The peak deviations are over 20 units, in sharp contrast to Capture One’s rendering on the M8 image, in which the deviations are of the order of 10 units. However, a closer look shows that what has actually occurred is that the pattern of deviations has remained very much the same, but that their magnitude has grown, and been offset in a positive direction. What this amounts to is that the image is considerably brighter overall. This is a strange result, and one that I’ll come back to in my next post.

For the moment ignoring the issue of the Capture One brightness, we can draw two conclusions at this point:

Most of the color variation that we see appears to be due to variations in the calibration of the raw converters, rather than variations between camera brands. The Lightroom M8 and D80 color renderings look a more alike than, for example, the color renderings of the M8 using Lightroom and Capture One.
Aperture 2’s color rendering appears to have been significantly improved, at least in a technical sense, relative to the previous versions of Aperture.

Lightroom, Aperture and Capture One Mini-Review Part 4

2008-02-17T09:00:00.000-08:00

In the previous part of this mini-review, I looked at the color response of Capture One, Lightroom and Aperture against a synthetically generated GretagMacbeth test chart. In this post, I’ll look at the response of the same programs against an image of an actual GretagMacbeth test chart. The process that I will follow for the actual image is a little different to that for the synthetic image. In the case of the synthetic image, I made no adjustment whatsoever to the image – the readings are exactly as they appear when the image is imported into each program. For the real image however, I first adjust the contrast and exposure setting on each program to exactly match expected values of the lightest and darkest monochrome patches on the GretagMacbeth chart. This exactly matches the exposure of the real images to the effective exposure of the synthetic image. As was the case for the synthetic images, all the test results are on a 0 to 100 scale, and represent the difference between the expected value as derived from the color values of the GretagMacbeth chart and the actual values measured. So, for example, if the red bar of the “Cyan patch” shows a value of -5, that means that the actual measured value of the R component of the RGB values as read out by the software in question was 5 units less that the theoretical value.

Relative to the synthetic image, Lightroom was, as expected, very close to the theoretical values for the GretagMacbeth chart. Versus a real image however, it shows significant differences, most noticeably in the red patch, where it has significantly more blue and green than might be expected. At first sight, this is a somewhat counter-intuitive result, as while the greater levels of green and blue indicate a more saturated color than the theoretical representation; Lightroom in general has a reputation for excessive red. It’s only in the green patch that there is significant excess red. This would imply that when the complaint of Lightroom’s “excessive red” is made, it is probably more of a complaint about the saturation of reds in the image, rather than an excess of the red color component.

Aperture shows no clear pattern of greater or lesser overall saturation, but does show two interesting characteristics. Firstly, the green components are still very much less than are the case for Lightroom, but at the same time the absolute variation from the theoretical value is far less – versus the synthetic image, the variation was -15.3, but against the actual it is only -6.3. This suggests that the Aperture calibration for the green components in a real image is probably better than Lightroom’s, even though the Lightroom’s better matches the synthetic image . Secondly however there are significant variations in the blue component, especially in the 1.1 profile. The newer 2.0 and DNG profiles show color rendition that is a lot closer to expected values that the previous version. This is consistent with Apple’s statements that the raw conversion subsystem has been substantially revised and improved in the new version. Overall, the actual M8 inages converted with the Aperture 2.0 profile is a better match to theoretical values than either the previous version of Aperture, or Lightroom.

Turning to Capture One, the most significant feature of the charts is the absence of “negative spikes” – while both Lightroom and Aperture have at least some color patches where at least one color is significantly less than the theoretical value, Capture One is relativley better controlled in this respect – only in the yellow patch is there a significant negative deviation. In addition, this control of negative peaks isn’t at the expense of spikes in the positive direct; no spike exceeds 12 units. It’s also interesting to note that in the three primary patches, the red component is within three units of the theoretical value in the red patch, and the green in the green patch and the blue in the blue patch are similarly well controlled. Thus, while Aperture is overall closer to the theoretical values, Capture One is perhaps “closer where it counts”.

In my next post, I’ll take a brief look at color rendering for the same three programs against an actual image from a Nikon D80, so as to get a feeling for whether the patterns here are M8 specific, or relate more to the programs in question.

Lightroom, Aperture and Capture One Mini-Review Part 3

2008-02-16T05:35:00.000-08:00

At long last, here’s the comparison of color rendering promised several weeks ago – between work and the display board in my main PC failing, this has taken longer than I’d expected. This post compares the color rendering of Lightroom, Aperture and Capture One versus a synthetic test image. That image was created by taking the raw image from a Leica M8, which is in DNG format, and then replacing the contents of the image with a synthetic version of a GretagMacbeth 24-patch color chart. This can be done because DNG format files contain all the color calibration information that’s required to go from the Camera’s raw image space to a real image in the two ColorMatrix matrices. So the synthetic image is built by taking the l*a*b* color values for the GretagMacbeth test chart, and reversing the calibration matrixes in the Leica DNG file. This, btw, is being done using a modified version of CornerFix – I’m currently debating whether to include the synthetic image creation functionality in the next official CornerFix release.

The synthetic test image is what a “perfect” M8 would show. But “perfect” here means an M8 that matches Leica’s calibration matrixes. However, there is no one single best calibration for a real camera. Pretty much all camera calibration is done via a three by three matrix. Using that, you can dial in any three particular colors exactly. So, for example, you can get the red, blue and green patch on the GretagMacbeth chart down to the last decimal point. If sensors were perfect, that calibration would also mean that every other patch would also be calibrated. However, in a real sensor, there are a whole lot of imperfections – among other things, the filters in the Bayer matrix aren’t ideal, so colors bleed between each other, and the sensitivity of the sensor itself varies with the frequency of the light striking it. So, even if you dial three patches in perfectly, the others will be out. So practically, what camera manufacturers and raw developer software writers have to do is to find a calibration that is a compromise across a whole range of colors. However, because people are more sensitive to certain colors being out (e.g., skin tone, foliage, etc) that compromise is often weighted in favor of the sensitive colors.

The M8 test images can be found here: http://chromasoft.googlepages.com/referenceimages

The charts below show the difference between the theoretical color values that we should see for a selection of six of the more important color patches, and what we actually get. So, for example, if the red bar of the “Cyan patch” shows a value of -5, that means that the actual measured value of the R component of the RGB values as read out by the software in question was 5 units less that the theoretical value as shown in the spreadsheet I discussed in the last post. In all cases, the scale is 0 to 100.

First up is Lightroom. It shows minimal deviations from the theoretical values – all the values are within 3 units. But this shouldn’t come as a surprise – Lightroom internally uses the exact same color model as the DNG file, and we know that Lightroom uses exactly the same color calibration as the Leica DNG’s have embedded into them. The minor deviations that we seeing are really just slight imperfections in the tone curve and in the color temperature interpolation process that Lightroom uses.

Next up is Aperture. There are three Aperture graphs, the first for Aperture V1.5.4. In addition I also have graphs for Aperture 2.0, which came out a few days ago. Aperture 2.0 provides four “Raw Fine Tuning” settings, “1.0”, “1.1”, “2.0” and “2.0 DNG”. I checked, and color rendering from the old 1.5.4 and what you get by setting “1.1” in 2.0 are indeed identical. Firstly, all of the Aperture settings have lot less green in the red patch than Lightroom, and less red in the blue and cyan patches. The 2.0 results are not much different to the 1.5 results; a little bit less red in the blue patch, a bit less green in the red patch, but far less blue in the yellow patch.

The “2.0 DNG” setting is more interesting. There doesn’t seem to be much documentation on what it does – the Apple aperture site itself is silent on the subject, and various third party sites have words to the effect of “changes to the image using the 2.0 DNG converter are made based on the DNG specification of the file”. This implies that rather than using the Aperture color conversion parameters, setting the DNG mode will give you the colors as set by the ColorMatrix values embedded in the DNG. As it turns out however, that’s just not the case – if it were, we’d see values that looked like Lightroom, but what we see are just some subtle changes to the “2.0” profile. Although visible if you change the setting on the fly, the change is actually more subtle than the change between 1.5 and 2.0.

Finally, there is Capture One. During the course of this process, Capture One 4.0.1 came out; the results shown here are for 4.0.1, but they are identical to those for 4.0; as far as I can tell, no changes have been made to color rendering between versions. Capture One provides two profiles, one Generic, and one UVIR, designed to match to the M8’s color rendering when mounted with a UVIR filter. While the differences between these two are there, they are quite subtle. Overall however, there are significant differences to the rendering of either Lightroom or Aperture. Capture One shows less red for most patches, especially the red patch, but more red in the cyan patch. Finally, there is generally somewhat less saturation for most colors. This is broadly consistent with most people’s views on Capture One’s rendering as being “less red” than Lightroom.

In the next post I'll show the same charts for actual rather than synthetic images.

Lightroom, Aperture and Capture One Mini-Review Part 2

2008-01-24T10:53:00.000-08:00

In part 1 of this series, I promised to show the tone curves for the various raw developers that I'm looking at. Here they are:

The Lightroom curves, for various settings of brightness and contrast - brightness has by far the most pronounced impact the image. In Lightroom, to get to a linear curve, you need to do three things - set brightness to zero, set contrast to zero, and select "Tone Curve - Flat" from the presets.

Then the Aperture curve showing the effect of the Aperture boost setting; the effect is essentially the same as the Lightroom brightness setting. To get a linear curve from Aperture, all you have to do is to set boost to zero.

Finally, the Capture One V4 setting; Capture One is a bit different to Lightroom and Aperture. Where Lightroom and Aperture have slider settings that are non-zero by default (brightness and boost respectively), on Capture One all settings default to zero. However, also by default, Capture One loads a "Film Standard" tone curve, which has a very similar effect to the other two program's non-zero settings. To get rid of the curve, all you need to do is to select the "Linear Response" Curve setting.

The last set of curves show a comparison of the default curve for each program, all referred back to a sRGB/2.2 gamma curve to make them comparable. While all the curves are about the same shape, there's a distinct difference in the "aggressiveness" of each curve. Lightroom/ACR adds the most brightness in the mid tones, and Capture One the least, and Aperture's about in the middle. We'll come back to this issue later in this series, but the next step is to use these tone curves to allow us to calibrate colors from the GretagMacbeth test chart.

And actually using the tone curves to calibrate colors is quite easy. All that's involved is the following two steps:

Convert the l*a*b* color values for the GretagMacBeth patches to RGB in the color space of the program in question
Use the tone curve to adjust the RGB values in accordance with the curve

Once we've done this, we have RGB values that are what should be displayed for each program, if the color calibration is correct. I've done this for the three programs I'm testing - a spreadsheet with the values is posted here: http://chromasoft.googlepages.com/calibrationspreadsheets.

In the next post in this series, I'll take a look at how each program compares.

Lightroom, Aperture and Capture One Mini-Review Part 1

2008-01-24T10:51:00.000-08:00

Just before the Christmas break, I decided to spend some time over the holidays comparing Adobe’s Photoshop Lightroom, Apple’s Aperture and Phase One’s Capture One raw developer/digital asset management products. By way of background I’ve been a Lightroom user since the earlier betas, and really like Lightroom’s workflow management. But I’ve never been happy with Lightroom’s color rendition, but have also not had the time to really dig into why I wasn’t getting what I wanted. But by just before Christmas, I was sufficiently frustrated that, despite what I like about Lightroom, I was in “there’s got to be a better way than this” mode – ready for change.

Now one person’s great color rendition is another person’s nightmare, so there isn’t any point in trying to just play with the sliders till something nice comes out – at least not for me. Also, on some products there eighteen sliders, all of which interact. I accept that there may be people out there that can look at an image, move a few sliders, and get what they want, but that isn’t me. So what I’ve chosen to do is to look at color rendition in a more “scientific” way; and ask three questions:

How close is the default rendition of each product to a 24-patch GretagMacBeth color checker?
How easy is it, using each product to calibrate the rendition to as exactly as possible match the theoretical values of the GretagMacBeth test chart, as printed on the instruction sheet that comes with the chart?
How usable is the calibration that I’ve created – is it easy to transfer to other images, how sensitive is it to changes in exposure settings, etc.

At least, while calibrating to to a GretagMacBeth chart doesn't mean I'll have "good" color, at least it's a consistent starting point. And yes, accepted that questions 1 and 2 are fairly simple and objective questions, but that question 3 starts to get more subjective.

As perhaps I might have expected this turned out to be a far more complex process than I’d thought, and eventually pulled me into comparing the three products quite a bit more broadly, for example, as regards the performance of their Bayer interpolation engines.

For the record, the software versions used for this comparison were:

Lightroom 1.3.1 Camera Raw 4.3.1
Aperture 1.5.4 and 2.0
Capture One 4.0.14154.14152 and 4.0.1.14900.14887

Now the first issue that comes up when trying to do this is a simple one - given that, for example, the skin patch on a GetagMacBeth cart has the l*a*b* color values of (65.711, 18.13, 17.81), what does that actually mean in terms of the RGB values that we should expect to get from the cursor read out in each program. A little of experimentation will show that this isn't a simple question.

Actually there are two issues with trying to calibrate imaging software - in what units is the readout, and secondly what adjustments are being applied to the image. Typically, when a raw developer loads a TIFF file, it does so without adjustment, but usually applies some kind of tone curve when loading a raw file.

The readout units for the programs that I've calibrated are:

Lightroom: Melissa RGB - Melissa RGB is the combination of the ProPhoto primaries and the sRBG gamma curve, Also known as "bastard RGB", as it's the bastard child of ProPhoto and sRGB.
Aperture: Wide Gamut. (Note: this was correct at the time this post was originally written. But see the comments this year below - its now Adobe RGB. However, the color rendition information is still correct)
Capture One: Capture One uses whatever color space is set as its output space, so you can set it to any ICM profile you have. For a bunch of ICM profile you can use, see my ICM Profiles page.

The adjustments made by default are more complex - generally, each raw developer has its own tone curve, and also its own default brighness settings.

This is the graph of the ACR 3 default tone curve, as extracted from the Adobe DNG toolkit - it shows the flatting at the top and bottom of the curve, and also the default brightness setting

What I've done to get real tone curves from the packages I'm looking at is to use the monochrome stepwedge reference image (shown above, and available in DNG format on my Reference images page) to work out what the tone curve is.

In part 2, I'll show those curves.

ChromaSoft web pages are up

2008-01-18T10:23:00.000-08:00

At long last, I've done something I've had on my to-do list for a long time now, which is to create a web space where I can put various files that might be useful to other people. It's at http://chromasoft.googlepages.com/. The first thing that I've posted there are two papers I wrote several years ago. The first goes into the mathematics of color spaces as used by ICC profiles, and color conversion between color spaces. Pretty much any serious imaging software package will use color space descriptions like this, either implicitly, or explicitly as is the case for Phase One's Capture One product.

Although the papers are largely written from the point of view of color conversion on monitors, for display purposes, all the maths are exactly the same as for cameras. The second paper shows how the shape of the CIE color space can be modeled in three dimensions.These two documents were originally created in MathCad, which is a mathematical modeling package from PTC. It allows the document to contain live mathematical equations, so that you can check that your maths actually works, and foots to real answers.

Both of these papers are either available in Adobe PDF form, or as MathCad documents. The MathCad documents are live, so allow real calculations to be made. However, they require that you have MathCad.

ACR, Lightroom and DNG color

2008-01-16T05:06:00.000-08:00

Well, I learned something new this morning, thanks to a question that Baxter Bradford asked over on the LUF. What he asked was (in effect) whether the various Adobe raw products (Adobe Camera Raw, Lightroom) would pick up on a changed camera profiles in a DNG file. This was in regard to the Leica M8, which changed its camera calibration data after it's IR sensitivity problems were discovered.

The way color works in a DNG files is that there are two pairs of color matrices:

ColorMatrix1 and ColorMatrix2. These two provide color calibration t two different color temperatures; in order to set an intermediate temperature, a linear interpolation is used.
CameraCalibration1 and CameraCalibration2. These are used to provide color calibration that is specific to the individual camera, rather than to the camera model.

The color temperature adjusted ColorMatrix and CameraCalibration matrices are multiplied together to get an overall color conversion matrix. In most DNGs the CameraClalibration matrices are not used (set to an identity matrix, technically) - the only DNGs that I've seen using these are for an Olympus E-3.

Up till this morning, I would have automatically responded to Baxter's question to the effect that that ACR and LR read the color matrices in the DNG, and since Leica has modified that post IR filters, ACR/LR's color calibration will in effect have changed to match the IR filter adjusted sensitivity. However, I've never actually checked that. So today I did, by overriding the ColorMatrix's in a test DNG, which should give weird color. And it made no difference, which was quite unexpected. Then I also overrode the EXIF camera name to "unknown", and then, guess what, weird color, as expected. After a bit more digging what I found is:

If ACR/Lightroom recognizes the camera name in the DNG, it ignores the ColorMatrix matrices, but still uses the CameraCalibration matrices.
If ACR/Lightroom does not recognize the camera name, it uses both the ColorMatrix matrices and the CameraCalibration matrices as contained in the DNG.

So the bottom line is, even if ACR/LR are reading a DNG, if either program sees a camera name they recognize, you will get an Adobe Camera Raw color calibration, not what's in the DNG. Only if they don't recognize the camera name will they use the DNG values. However, ACR/Lightroom always honor the CameraCalibration matrices.

To confirm this, I took a look inside the LR/ACR code, and "Leica M8 digital Camera" is indeed listed in there.

CornerFix

2008-01-15T06:20:00.000-08:00

As part of my journey into digital imaging, I found myself writing CornerFix, which can be found on http://sourceforge.net/projects/cornerfix/. The image is a screen shot of the Mac version.

CornerFix corrects for color dependent vignetting in digital images, which shows as cyan colored corners, as in the image on the left hand side of CornerFix's main window in the screen shot. The image in the screen shot comes from an M8 with a CV12 lens and IR filter on it. All digital cameras are some extent subject to this; current generation sensors are highly IR sensitive, so there needs to be a IR filter somewhere. But the combination of sensors and IR filters also cuts into the red part of the spectrum, and do so in a way that depends on the angle through which the light bends as it travels to the sensor. So red gets cut more in the corners. Most DSLR's do this a bit - take a picture of a white wall and you will probably see it, although many cameras correct internally to a greater or lesser extent.

Leica's M8 has a particular problem in this regard. Historically, one of the advantages that rangefinder cameras had was that the back of a rangefinder lens can be a lot closer to the film surface than is the case for a film SLR - the SLR needs space for the mirror, which a rangefinder doesn't have. So in the film world, rangefinder lenses had a lot more design freedom that SLR lenses - wide-angle lenses that on a SLR had to be reverse telephoto designs could be normal designs on a rangefinder. Ironically, in the digital world, this turns into a disadvantage. Because the back of the lens can be a lot closer to the film, the angle is more acute, and you get a worse cyan corner problem than a DSLR would.

M8's can correct for this problem themselves, but there are two issues for users - firstly, your lenses must be coded, which means that it must be a Leica lens, either new enough to have been coded when it was manufactured, or one that has been sent to the factory to be upgraded. So anyone with a non-Leica lens or one that can't be upgraded is out of luck, unless they somehow code the lens themselves. The second problem is that the M8's correction is "one size fits all"; it's designed for average situations, and sometimes doesn't do well in unusual lighting.

CornerFix allows the cyan corners problem to be corrected for any lens, in more or less any lighting situation, by post processing the camera's image file. The right hand side of the main window in the screen shot shows the corrected image. CornerFix is available for the Mac and for Windows. It's free and open source, released under the GPL. Note however that image must be a DNG formatted file - CornerFix doesn't work with TIFF or JPEG files.

For those interested in the technical details, the core of Cornerfix, which is common to both the Mac and PC versions, is written in "pure" C++, and uses the Adobe DNG toolkit to decode files. The GUI of the Windows version is written in C++/.Net, and the GUI of the Mac version in Cocoa. For those REALLY interested in the technical details, as CornerFix is GPL, you can download all the source code from site above.

What's this all about?

2008-01-15T05:17:00.000-08:00

Over the past year or so, I've been involving myself more and more in the world of digital imaging. Photography isn't new to me - I was using a rangefinder while in school, developing and printing my own work. Neither is technology new to me - once upon a time, I co-founded (and ran all the R&D and engineering functions) a start-up that built data acquisition systems. Which is essentially what a digital camera is - a sensor, analog to digital converters, and a way of storing and displaying what you get. And I've been programming on and off since then - not as a core part of what I do, but to support what I do. Things like simulations of various parts of the global financial system.

What I've found is that digital imaging is in a fascinating space; it's only just - say over the past 2-3 years - got out of what I call the "Bear" phase. Not bear in the financial markets sense, but in the "People look at a dancing bear not because of how well it dances, but because it dances at all". The technology has now reached the point that it's mostly better than film. But, there's still lots of innovation coming down the track. Up until recently, digital cameras concentrated on just being more convenient than, and as good as, film. So digital imaging was focussed on being "just like film" only better. That's what's changing now. Things like sensor resolution, the "more megapixels race" are close to done; in the high end DSLR format, we're up against the limits of lenses and basic physics. So now what's starting to happen is things like the "d-lighting" on Nikon's new generation DSLR's. Something that has no analog in the film world. Similarly, some of the capabilities being built raw developers have no analog in the darkroom - e.g., Lightroom's "smart vibrance", a vibrance control that can recognize skin tones, and leave them untouched while colors around the skin can be made brighter.

Along the way, I also wrote CornerFix, which I talk about more in another post. That re-involved me in Windows C++ programming as well as involved me for the first time in programming for the Mac, which was a journey in of itself. So, while I'm active on some of the photography forums - e.g., the LUF (www.l-camera-forum.com). - this is where I talk about some of the deep technology issues, some of the really broad "where are we going" stuff, or about some of the programming that I do. So that's what this about - a mix a deep imaging technology, what's happening to the photography market, and stuff about Windows and Mac software development.