Well, I wasn't expecting to come back to the topic of Fuji, the X-Pro1 and its X-Trans sensor. However, I have been putting a lot of work into the suppression of artifacts when demosaicing. A lot more work than I had intended to, but that's another story. This is for a new product that I hope to release in a few weeks time (several months later than I'd hoped). But I did stumble into a better understanding of the nature of the chroma smearing (or watercolor effect, as it has also become known). The previous posts about Demosaicing the Fuji X-Pro1 are here, here and here.
In previous posts, I compared renderings from Adobe Camera Raw, SILKYPIX and Fuji's in-camera JPEG processing, as well as DCRAW and RPP. Finally, I compared those renderings to renderings from PhotoRaw, both in its "retail" configuration, and in modified form with post demosaic filtering. Practically, DCRAW and RPP were pretty much outclassed -- they use VNG algorithms that generate substantial zipper effects.
In post three, I hypothesized that the chroma smearing effect that you see very visibly in the ACR conversion, and to a lesser extent in the SILKYPIX conversion, was due to filtering, possibly mean filtering post demosaic. I now think that I was probably wrong, or at least partially wrong - the effect is due to filtering, but not mean filtering post demosaic. Rather, it's as a result of filtering during the demosaic process itself.
PhotoRaw, ACR and SILKYPIX
Just for reference, here are the various contenders from previous posts:
PhotoRaw 3.5.4, 400% crop
PhotoRaw does pretty well, but has those pesky artifacts on the paperclips.....
Adobe Camera Raw V7.1 beta
ACR beta 7.1 - Lots of chroma smearing, and the letters are quite desaturated.
SILKYPIX conversion
SILKYPIX - some chroma smearing, saturation down, resolution appears slightly lower than PhotoRaw or ACR
PhotoRaw 3.5.4 plus mean filter, 5 pass, 400% crop
The "PhotoRaw plus mean filter" showed a similar pattern of chroma smearing to SILKYPIX and ACR, but some parts didn't quite match. For both SILKYPIX and ACR, the chroma smearing is more pronounced in specific areas, e.g., in between the upper parts of the "Y", and inside the "A".
Multipass Demosaic Algorithms
As mentioned in the introduction, I've been working on artifact suppression for another product. Now the easiest way to suppress artifacts is not to generate them in the first place, so one of the things I looked at are demosaicing methods that have inherently low levels of chroma artifacts. There are a number of demosaicing algorithms that I'll call "multipass", for lack of a better term. The best known of these is LMMSE (see reference 1 below). LMMSE works, in simplified form, by first performing an initial demosaic in both vertical and horizontal directions, then filtering in the chroma domain, then using the filtered chromas to perform what is in effect a second higher quality demosaic. In this case, higher quality in the sense of fewer visible chroma errors, as the second demosaic is partially based on filtered chroma.
The results of using such a algorithm in a beta version of the new product are shown below. The first crop shows the algorithm using a filter width as it would be for a conventional two by two Bayer array. The second uses exactly the same algorithm, but with double the filter width. Note that unlike the crops above, these two are unsharpened, due to limitations of the beta.
"New Product" beta, multipass demosaic, single filter width, 400% crop, unsharpened
"New Product" beta, multipass demosaic, double filter width, 400% crop, unsharpened
Several thing stand out here (note - you may want to double-click on the crops to get a better view):
- The multipass algorithm shows fewer artifacts than the "AHD class" algorithm, even with a "normal Bayer" filter width.
- However, with a wider filter two things happen. Firstly, artifacts are reduced, as we'd expect - the X-Trans sensor has sparse colored photo sites, so its logical that we'd have to filter more heavily to get this particular algorithm to perform. Secondly, the crop starts to show the characteristic chroma smear pattern that the SILKYPIX conversion shows, as well as the slight loss of resolution and saturation that the SILKYPIX conversion shows. In fact, the conversion starts to look uncannily like SILKYPIX, a lot more so that was the case for the mean filter I originally suspected was involved.
- Extrapolating, you can easily see how, with an ever wider filter, ACR would generate the rendering it does.
Conclusion
It's highly likely that both SILKYPIX and ACR use some form of multipass algorithm, and that the chroma smearing is as a result of the need to adapt the algorithm to the sparse colored photo sites in the X-Trans sensor. ACR appears to be using a very wide filter, probably indication the whatever methods are being used in the initial and secondary demosaic are not well suited to the X-Trans sensor. Note that it's highly unlikely that either ACR or SILKYPIX are using LMMSE in anywhere near it's text book form (neither is the "New Product" beta used above, btw). The text book form depends on using color data to improve the initial green demosaic; given the X-Trans sensors relatively large number of green photo sites and low number of red or blue photo sites, that would not work well.
References:
1. L. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square estimation, IEEE Trans. on Image Processing, vol. 14, pp. 2167-2178, Dec. 2005.
View comments