1. In my series of posts about demosaicing the Fuji X-Pro1's X-Trans sensor, I pointed out considerable chroma smearing in the the raw conversion from a beta version of Adobe Camera Raw. I also clearly pointed out that a beta is, well, a beta, and that hopefully the Camera Raw team would improve the situation in a final release. But I also pointed out that I very much doubted whether Adobe would have allowed DPReview access to a beta that wasn't a good representation of the final product.

    Lightroom 4.1 final is now out, with support for the Fuji X-Pro1, so we can expect a final version of ACR with X-Pro1 support very soon as well. (For those that don't know, ACR and Lightroom share a common raw conversion engine, and releases track closely.)

    Sadly, the Lightroom final release shows exactly the same chroma smearing as the ACR beta. A crop of the comparable area shown in the original posts is below.

    BTW, Lightroom does an excellent job in many other respects - the diagonal edges of the paperclips are just amazingly clean - I very much doubt that any other raw converter could match them. But the chroma is a mess!!

    Turns out that X-Trans sensor really wasn't a good idea.

    Updated 7 June 2012 - also see this post

    Lightroom 4.1 final, Fuji X-Pro1
    11

    View comments

  2. There have been a few questions about why the Leica M Monchrom doesn't have the compression options for DNG files that the M8 and the M9 have.

    I certainly can't claim to know what drove Leica's design decisions, but I can make a informed guess as to what drove the lack of compression options:

    In the Leica M8 and M9, DNG compression is done by using a tone curve (somewhat like an sRGB tone curve) to reduce the data width from 14 bits to 8 bits. On the M9, the compression is almost always invisible, unless you do some extreme processing and then look really hard.

    The usual problems with images that are level compressed is that they show banding. Some people think that the reason why M8 and M9 images don't show banding (except under extreme circumstances) is that the tone curve that Leica uses is super clever. Not so - it's a standard power-of-two curve. Actually, the main reason for the lack of banding is that the Bayer demosaicing operation that happens to the data after its been decompressed tends to smooth out any banding. What happens is that because you're interpolating pixels, the interpolated pixels tend to end up "between" the bands that resulted from the compression.

    Now on the M Monochom, the crucial difference is that there's no demosaicing step. So no interpolation, and no smoothing. So you're immediately far more at risk of banding.

    So, I'd guess Leica chose to remove the compression options in order to avoid banding in compressed images.
    0

    Add a comment

  3. I previously wrote about the Leica M Monochrom's sensor in this post. In summary, what I said was that I doubted that the sensor was simple a "naked sensor"  - in order to get reasonable spectral sensitivity, there would have to be a filter of some sort.

    Leica have now published the spectral sensitivity of the sensor - you can find it here. Leica provide both a graph of spectral sensitivity, and a visualization of the Leica M Monochrom versus Kodak T-Max 400 film. It's not clear why they chose T-Max as the reference - not my idea of a film with benchmark tonality(!)

    Something that is quite important to realize when looking at the visualization is that, although Leica don't explicitly say so, it clearly excludes the effect of the film and the Leica M Monochrom's tone curve (aka, how it responds to light and dark). The visualization is only about spectral response. So if you actually took a picture of the test chart with a M Monochrom and an M7 loaded with T-Max, what you would see would not be the same as the visualization.

    Taking a look at the graphs, it seems pretty clear that there is a filter on the Leica M Monochrom's sensor, as expected. I'd think, by the shape of the curve, that the filter is a simple red/IR filter, as I'd speculated in the original article. It appears to start to cut quite early, at about 550 nm. This is probably to get good IR cut characteristics.

    A simple red/IR filter is a good compromise between spectral response and sensitivity - one of the selling points of the Leica M Monochrom is sensitivity, and a more aggressive filter would reduce that.

    If you compare the "visualization" of the Leica M Monochrom versus Kodak T-Max, a few things stand out:

    • The Leica M Monochrom's sensor and filter combination is overall less sensitive to red than film, in contrast to what we would expect from the sensor alone. You can see that clearly in the red patches of the chart, which are darker for the M Monochrom than for film. 
    • Because the red/IR filter has a fairly gradual cut-off, the M Monochrom isn't going to be at its best with very deep reds. However, that's the case for pretty much all CCD sensors anyway.
    • It's interesting to look at the blue, green, red patches (fourth row, fifth column on the chart). What you see is blue equal, green lighter for the M Monochrom, red darker for the M Monochrom. So in terms of relative sensitivity versus the T-Max, the M Monochrom is slightly biased towards green sensitivity. That's consistent with the classic luminosity equation as discussed in the original post. It implies that the Leica have balanced the camera to be somewhere between B&W films, and the kind of results you'd get by desaturating a color image in a image editor.
    • The skin patches appear reasonably well balanced, which is important. So while the patches differ from the T-Max, the differences are fairly common across the range of skin tones. Skin rendition is probably the thing that most viewers will be the most sensitive to, because humans have very good "mental maps" of what other people's faces "should" look like. However, the variation in red sensitivity does cause some deviations between "caucasian skin" patches. I would be a bit concerned that you might see some patchiness in skin tones from the M Monochrom -  a carefully chosen light green filter may help in such situations. Once some production M Monochrom's are available, hopefully someone will do trials with commercially available filters.
    Overall, the visualization looks like a reasonable compromise. Slightly more even skin tones would be nice, but Leica have to work with the reality of the filter materials that are actually available and are compatible with the manufacturing process, and the need to keep good sensitivity. Slightly more even skin tones at the cost of several stops of sensitivity wouldn't be a good trade-off.

    What I do find encouraging here is that Leica have published this document so quickly after the questions were raised. It's no secret that I was highly unimpressed that none of the "professional reviewers" that Leica invited to the launch and/or gave preproduction cameras to do testing seemed to even realize that in a monochrome sensor, spectral sensitivity is important. But Leica's willingness to answer questions like this quickly with real data is a real positive. Update: I've since found out that Sean Reid, at Reid Reviews did address spectral sensitivity/filtering. My subscription to the site wasn't current when I originally wrote this post so I hadn't yet read his review.
    0

    Add a comment

  4. Yes. Part three. The previous posts about Demosaicing the Fuji X-Pro1 are here and here.

    This post follows on from the previous two by showing how to get demosaicing that is, for practical purposes, as good as SILKYPIX, the best of the  Fuji X-Pro1 raw developers I tested in the previous posts.

    In post one, I noted the chroma smearing that was very evident in the Adobe Camera Beta conversion, and to a lesser extent in the SILKYPIX conversion. At the time, I didn't recognize the cause, and asked for input from readers. I didn't get any input, but not being one to be able let a technical puzzle alone till it's solved, I carried on thinking about it. Finally, something clicked. Attentive readers may also remember that I had mentioned in passing that it would be possible to eliminate the highlight artifacts that PhotoRaw showed by median filtering.

    What clicked was that there are also simpler forms of filtering that can eliminate those kinds of artifacts - specifically, mean filtering. Mean filtering is considered to be better than median filtering at reducing artifacts, but I hadn't previously really considered mean filtering because it's also considered to be quite primitive. Specifically, it smears chroma and reduces saturation. Duh! I had been thinking in terms of ACR and SILKYPIX using one of the very high-tech anti-artifacting algorithms to be found in academic journals, not something that barely gets a mention.

    Mean filtering and median filtering

    Before going further, I should briefly explain mean and median filtering. In simplified form, mean filtering is just simple averaging of a series of colors (or numbers) . Median filtering takes the center value of a series.

    For example, consider a series of five numbers:
    1 2 4 4 5
    The mean of those is  3.2, but the median is 4, because if you order the numbers, the number in the middle is 4.

    In imaging, median filtering is (usually) considered better because you get a color that actually exists in the image. E.g. if you had an area of blue and red pixels, a "mean" would be green. Problem is, green isn't either red or blue, and will stand out. A median will always give you either red or blue, and so will blend in.

    So I did some experiments:

    PhotoRaw, ACR and SILKYPIX

    Just for reference, here are the three contenders from post one:

    PhotoRaw 3.5.4, 400% crop

    PhotoRaw does pretty well, but has those pesky artifacts on the paperclips.....

    Adobe Camera Raw V7.1 beta

    ACR beta 7.1  - Lots of chroma smearing, and the letters are quite desaturated.


    SILKYPIX conversion

    SILKYPIX - some chroma smearing, saturation down, resolution appears slightly lower than PhotoRaw or ACR

    Now for something different......PhotoRaw plus!

    What I did as an experiment is to build two modified version of PhotoRaw 3.5.4:
    1. A version with  3x3 median filtering
    2. A version with  5x5 mean filtering

    Here's the results:

    PhotoRaw 3.5.4 plus median filter, 10 pass, 400% crop

    PhotoRaw 3.5.4 plus mean filter, 5 pass, 400% crop

    What we see is:

    • The median filter cleans up most (but not all) the artifacts, even run 10 times. But it doesn't leave any significant chroma smearing. As expected.
    • The mean filter cleans up all the artifacts, run 5 times. But it has chroma smearing that's very similar to SILKYPIX(!). In fact, at this point PhotoRaw plus a mean filter starts to look nearly indistinguishable from SILKYPIX - a normal user would be unlikely to spot any difference.
    • The remaining difference between SILKYPIX and PhotoRaw plus the mean filter is that SILKYPIX's mean filter appears to only be being applied close to edges. In addition, there appears to be some local saturation adjustment going, probably to counteract the saturation reduction that mean filtering would cause. If you look closely, the pixels right next to the lettering on the SILKYPIX image are smeared very similarly to the PhotoRaw plus image. But a few pixels away, SILKYPIX has gone back to an unfiltered version. The intent here would be to  give SILKYPIX the best of both worlds - artifact suppression, but relatively little chroma smearing away from edges.
    • ACR also appears to be using some form of mean filter, which also seems to be being applied near the edges, probably again with some kind of saturation adjustment causing the odd lightening effect. But ACR is doing so in a very heavy-handed way. On the face of it, it looks rather as if the ACR engine doesn't much like the X-Pro1's sensor pattern, and the camera raw team have just brute force filtered out any artifacts. Hopefully the team will be able to improve this in a final release. (Update - they didn't - see this post)

    Conclusion

    Firstly, the addition of mean filtering to the new PhotoRaw "generalized AHD" engine brings it very close in performance to the Fuji recommended raw developer as regards the X-Pro1. Not quite as good, but "PhotoRaw plus" now rates as the second best (and only by a whisker!) raw converter for the X-Pro1. Also, there's a clear development path (edge sensitive mean filtering rather than mean filtering everywhere) to get it all the way.

    Secondly, it seems very likely that the way that both SILKYPIX and ACR are handling the X-Pro1 sensor is by some form of mean filtering. In the case of SILKYPIX, it seems well tuned to the sensor. ACR, not so much. But SILKYPIX had the advantage of Fuji help prior to the camera's release.

    Finally, there doesn't seem to be any major "secret sauce" to what Fuji/SILKYPIX are doing. In a way, this is somewhat discouraging as it implies that there isn't anything more than what we've seen to be extracted from the sensor.

    The reason why mean filtering, rather than median filtering is being used is probably two things. Firstly, as we've seen, mean filtering is better at removing the artifacts that the X-Trans sensor is prone to. Secondly, mean filtering is a lot faster than median filtering - no sample ordering step.

    The inevitable question is, where does this leave PhotoRaw? Sadly, a version of PhotoRaw using the mean filtering technology above won't be coming to an App Store near you. It's too slow. This is where the difference between a desktop raw processor and a iPad raw processor shows.

    What I will be doing however is looking at whether it is possible to implement an "artifact sensitive" mean filter that only filters where it has to. This would be somewhat similar to what SILKYPIX and Adobe seem to be doing, but would be focussed on improving speed by processing far fewer pixels more than on image quality. Such a version might be fast enough to make the cut into a release version of PhotoRaw.

    UPDATE: It's probably the case that while filtering does cause the chroma smearing, it's not median or mean filtering - see part 4 here.

    19

    View comments

  5. My previous two posts about the Fuji X-Pro1 ( here and here) drew some interesting (and amusing) comments on the web. What those comments did show is that there's a lot of misunderstanding about how the relationship between camera manufacturers and software developers works.

    Again, this is my (quite cynical) viewpoint - take it for what it's worth.

    In short, there's actually far less of a relationship than most people seem to assume.

    Firstly, let's talk about size and scale - In the original article I made a comment about "Maybe tomorrow Fuji will send me an email". That was a joke. Perhaps not a good joke, but a joke that other small developers would have recognized. PhotoRaw sells about 100 times (maybe 1000 times) too few copies for Fuji to even know my name. Or care. Big camera manufacturers do stuff only if there's money on the line. Practically, that means keeping the bigger and more influential web sites (e.g., DPReview) on side with early access to cameras, etc. But not much else. In the time that I've been invoked in the industry as an independent player, either writing open source software such as CornerFix or commercial software such as PhotoRaw, no camera manufacturer has ever contacted me. Ever. At all, about anything. If one did, I'd be stunned.

    Secondly, camera manufacturers don't really like software developers, even big ones. That may sound really strange - camera owners want their cameras to be supported by major players like Adobe and Apple, and so the camera manufacturers should work closely with them, right? Wrong.

    Most camera manufacturers regard software developers essentially as parasites, making money from their hard work in designing and manufacturing cameras. So as far as a manufacturer is concerned, if anyone is going to make money from anything related to their cameras, it should be them. It's no secret that camera manufacturers have always made it hard for independent lens manufacturers to get their lens to work on cameras from the major manufacturers. Software is no different - to a camera manufacturer, software developers are a competitor, not a partner.

    Practically of course, the camera manufacturers know that their customers expect their cameras to work with Lightroom or Aperture. But there's no incentive for the camera manufacturers to make that quick or easy.

    The ideal position for a manufacturer is that their cameras are supported by third party software, but the manufacturer's own software works better. Ever tried to get D-Lighting on a Nikon camera to play nicely with Adobe's ACR or Lightroom? It works a lot better with Nikon's own software. And it's the software developer that will get the blame for any glitches. Fuji will have no problems at all with SILKYPIX working better than ACR - quite the contrary.

    Take a look at the current finger pointing between Adobe and Fuji about information on the X-Pro1. A lot of people are assuming that part of what Fuji would give to Adobe would be information on how to get the best out of the sensor. That's really unlikely. Probably what they would have provided was basic information on the matrix layout, raw file format, calibration information such as color response, maybe technical data on noise, etc. But helpful advice on how to interpolate unevenly spaced pixels built up over the time that they had been developing the sensor? Not likely. There's no incentive to provide anything like that, and Adobe might just be able to use that information elsewhere in ACR or Lightroom. Not what Fuji would want.

    5

    View comments

  6. Jono Slack over on the L-Camera-Forum was kind enough to post a raw DNG from Leica's new M Monochrom, so I a took a bit of a root around inside it. No surprises - it's very similar to a M9 DNG. For those interested, there's a field level dump of the interesting fields below, but the highlights are:
    • 14 Bit data, much the same as an M9, with non-zero black level
    • Uncompressed
    • DNG version 1.0.0.0, so still the original DNG spec, none of the new stuff
    • Camera name specified as "M9 monochrom"
    • The MakerNote with its lens info still seems to be as for the M9; in this case the lens shows as "Apo-Summicron-M 75mm f/2 ASPH", which is consistent with what Jono reported
    • The famous "blue dot" is still there
    The only thing that might surprise a few people is the "PhotometricInterpretation: LinearRaw" part. But that's actually quite correct - the way the DNG spec works, you can either set that to CFA (aka a Bayer array type camera) or to LinearRaw. And this sure isn't a CFA camera.

    There is one slight side effect of LinearRaw though. When ACR or Lightroom load a normal raw, they apply a tone curve by default. However, with a LinearRaw, they don't. So, for those intent on comparing a M9 image to a M Monochrom image shot side-by-side, be aware that by default they have different tone curves.

    I'd guess that once the M Monochrom is shipping ACR and LR will have a built-in M Monochrom camera profile that will probably have a tone curve.

    The other issue to be aware of with LinearRaw is that most raw development programs don't support it, so until Aperture, Capture One, etc are updated, don't expect M Monochrom DNGs to load in much except Adobe products.


    DNG Field dump as per CornerFix
    Software: "0.007"
    SubIFDs: IFD = 424
    ExifIFD: 672
    SelfTimerMode: Off
    DateTimeOriginal: 2012:03:27 16:40:26
    FocalPlaneXResolution: 3700.0000
    FocalPlaneYResolution: 3689.0000
    FocalPlaneResolutionUnit: Inch
    TIFF/EPStandardID: 0.0.0.1
    DNGVersion: 1.0.0.0
    UniqueCameraModel: "M9 monochrom"
    BaselineExposure: -0.50
    BaselineNoise: 1.00
    BaselineSharpness: 1.00
    LinearResponseLimit: 1.00
    MakerNoteSafety: Safe
    NextIFD = 0
    SubIFD 1: Offset = 424, Entries = 20
    NewSubFileType: Main Image
    ImageWidth: 5216
    ImageLength: 3472
    BitsPerSample: 16
    Compression: Uncompressed
    PhotometricInterpretation: LinearRaw
    StripOffsets: Offset = 213504
    SamplesPerPixel: 1
    RowsPerStrip: 3472
    StripByteCounts: Count = 36219904
    XResolution: 300.00
    YResolution: 300.00
    PlanarConfiguration: 1
    ResolutionUnit: Inch
    BlackLevelRepeatDim: Rows = 1, Cols = 1
    BlackLevel: 43.00
    WhiteLevel: 16383
    DefaultCropOrigin: H = 2.00 V = 2.00
    DefaultCropSize: H = 5212.00 V = 3468.00
    AntiAliasStrength: 1.00
    NextIFD = 0
    Exif IFD: Offset = 672, Entries = 20
    ExposureTime: 1/750 sec
    ExposureProgram: Aperture Priority
    ISOSpeedRatings: 320
    ExifVersion: 2.20
    DateTimeDigitized: 2012:03:27 16:40:26
    ShutterSpeedValue: 1/724 sec
    ExposureBiasValue: 0.00
    MaxApertureValue: f/2.0
    MeteringMode: CenterWeightedAverage
    LightSource: Unknown
    Flash: 0
        Flash did not fire
    FocalLength: 75.0 mm
    FileSource: DSC
    SceneType: A directly photographed image
    ExposureMode: Manual exposure
    DigitalZoomRatio: Not used
    FocalLengthIn35mmFilm: 75 mm
    SceneCaptureType: Standard
    ImageUniqueID: <000000000000000000000000000006ff>
    NextIFD = 0
    Leica MakerNote: Offset = 1168, Entries = 4
    .
    .
    .
    .
    .
    .
    Leica MakerNotes
      Lens Id: Apo-Summicron-M 75mm f/2 ASPH.
      Frame Selector Position: 50/75mm frame lines engaged
      M9 Blue Dot Brightness: 428544
      M9 TTL Brightness: 500992
      Leica Estimated Aperture: f/4.0


    0

    Add a comment

  7. This post is a follow-on to my previous post on Demosaicing the  Fuji X-Pro1.

    I was somewhat bemused to find that the original post, which was a largely technical article, was picked up on a number of photography news sites.  Anything with "Demosaicing" in the title should have been a dead give away that the content wasn't really light reading.  Some of the interpretations and comments on the forums were "interesting" to say the least. Especially the the suggestion I was in some way "lazy" was certainly a new take(!) This after going to a lot of trouble support the X-Pro1, resulting in PhotoRaw being the first raw developer to provide non-beta X-Pro1 support (other than SILKYPIX, which had pre-release help from Fuji).

    However, interesting comments on web the aren't the point of this post. A reader pointed me to betas of both DCRaw and RPP that support the X-Pro1, so I took a look at both.

    The betas are here:
    • DCRaw "9.13test": //tinyurl.com/6vkeqrq 
    • RPP Version 4.5.0 (1520) Beta: http://tinyurl.com/6t4raxa
    Here are the results - the crops I'll show here are the same as for the original article. Except where noted, all settings are default.

    IMPORTANT NOTE: the RPP and DCRaw images here are from betas, and they are from betas that are almost certainly at a far earlier stage of development than the ACR beta in the previous post. As Adobe provided that beta to DPReview, its a pretty good assumption that Adobe is comfortable with it as a near final product. The two betas in this post are quite possibly real early stage.

    First, for reference, the same PhotoRaw crop as in the previous post

    PhotoRaw 3.5.4 400% crop


    Then DCRaw

    DCRaw "9.13test" 400% crop

    Firstly, note that I've adjusted exposure on the DCRaw image be closer to the others. Most raw developers, including ACR and PhotoRaw, add a tone curve to their images; DCRaw doesn't. The results from DCRaw unfortunately aren't too pretty - the beta version shows zipper effect on straight lines, and also artifacts on the highlights. Resolution also appears lower. Interestingly enough, these are similar issues to the ones I ran into with the the "Attempt 1" VNG interpolation scheme I described trying and abandoning in the previous post. 

    Here it's probably important to put artifacts in some kind of context. No raw processor is perfect, but what needs to be considered is both the severity and frequency of artifacts. So:
    • PhotoRaw displays colored pixels on sharp (one or two) pixel transitions out of saturated highlights. That's a severe artifact, but there aren't that many such areas in any given image. In most images, you won't see such a situation at all.
    • ACR and SILKYPIX display chroma smearing. That's a low severity issue because most users will never notice. But the image is smeared everywhere(!)
    • What DCRaw is showing is zipper effects on straight lines. That's a problem because straight lines are really common, and the artifact is really noticeable.

    The test version of DCRaw came with source code, and a quick look shows, perhaps unsurprisingly, that "9.13test" appears to be using a modified VNG algorithm for X-Pro1 images.


    RPP Half 400% crop

    Next up is RPP. Firstly, there is an obvious difference between RPP on one hand and ACR, PhotoRaw, SILKPIX etc on the other hand as to color rendering. Secondly, the beta uses an interpolation scheme called "Half" by default. Visually this appears not really be interpolation at all - rather it seems to reduce image resolution to the number of "real" pixels. The results are just plain ugly - the less said the better. Fortunately you can also select VNG interpolation.


    RPP VNG 400% crop

    RPP's VNG interpolation does a much better job. But, as is the case for DCRaw, it loses plot quite badly on straight edges, giving zipper effects, and seems not to quite have the resolution of ACR, PhotoRaw and SILKYPIX. Visually, if I had to take a guess, I think that RPP's VNG interpolation has the same core as DCRaw, but does some noise reduction in post-processing afterwards.

    Conclusion


    The conclusion really is the same as the previous post:

    Firstly, Fuji X-Pro1 images are a pig to interpolate, and don't appear to show better resolution than a conventional sensor, even when using Fuji's official raw developer, SILKYPIX.

    Secondly, raw developers are a trade-off - none are perfect, and all work inside of constraints on memory size and processor speed.

    Update: part three shows how a modified form  of PhotoRaw gets very, very close to the performance of SILKYPIX.

    Finally, if you're interested in my (somewhat cynical) view on the relationship between camera manufacturers and software developers, you can read this post
    7

    View comments

  8. Update: There's now a part 2 to this post and also a part three

    Looks like it's the time for oddball sensors. Or for me to write about them anyway. I've just finished updating PhotoRaw for the Fuji X-Pro1, and I thought it was worthwhile to document the journey, and what it means for the X-Pro1's X-Trans sensor. Specifically, whether it will deliver on the claims that Fuji has made for it.

    BTW, the new version of PhotoRaw will be out early next week, assuming there are no glitches on Apple's side.

    This is my personal view-point. It's based on spending a lot of time writing code, and more time pixel peeping X-Pro1 images than any sane person ought to do, but it's still just my personal view.

    If you happen to live anywhere a lab where people develop software for decoding raw images hang out, you've probably been hearing a whole lot of swearing and cursing recently. You've probably also been hearing a lot of jokes with punchlines about "there's a reason why Fuji is a four letter word starting with f". Here's why:

    Fuji has introduced the X-Pro1 a while ago, with a novel "X-Trans" sensor. A conventional bayer sensor is a two by two rangement, but the X-Trans has a six by six layout, intended to mimic the random grain structure of film. The layout is in the image below.

    Fuji has a long history of innovative sensor design, for example the Super-CCD sensors with a mix of high and low sensitivity cells, as well as sensors with the Bayer array rotated at 45 degrees, etc. However, none of these have made a lasting impact on the market, with the major manufacturers resolutely sticking to conventional Bayer arrays.

    The advantages of the X-Trans sensor

    The big advantage of the X-Trans layout is moire suppression - because the layout isn't as regular as the conventional layout, moire (nasty colored pixels on sharp edges) isn't nearly as much of a problem as it would otherwise be. Fuji claim that moire is eliminated - that's probably optimistic, but I'd guess that moire would be suppressed by three to four times. That's a LOT. In turn, this means that while a sensor with a conventional layout needs an anti-aliasing (AA) filter to avoid moire, a camera built with the X-Trans sensor doesn't need an AA filter, or at least can use one that's a lot weaker. An AA filter is basically a filter that slightly blurs the image. No AA filter (or a weaker AA filter) means less blurring, and higher effective resolution.

    End result is that the the Fuji X-Pro1, which has a 16 MP sensor, should out resolve cameras with larger full-frame sensors.

    This is all sound theory, and there have been a fair number of rave reviews of the X-Pro1 in the press. So why all the the unhappy developers? And does the sensor really deliver?

    The problems

    The downside with the X-Trans sensor is that it's a bit of a pig to demosaic. Fuji openly say that it will take much more processing power, and that the X-Pro1 has a beefed processor to cope. Demosaicing involves interpolating pixels that are missing from the sensor array - e.g., in the case of a green pixel, the red and blue pixels need to to interpolated.

    The easiest way to see the challenge that the X-Trans sensor poses is to take a look at the blue pixel in the first column of the diagram above. To interpolate effectively, the ideal situation is to have pixels that you can interpolate from close by. In addition, the best kind of pixels to interpolate from have their centers aligned - in other words, you can draw a line between a pair of pixels that passes through the center of of the pixel you want to interpolate. So, e.g., let's say you want to interpolate the red value of that blue pixel. Vertically, the closest other red pixel is three down, and three up, aka, a gap tof two pixels. In the conventional array, its two up and two down, a gap of one. Take a look at where the closes same color pixel, another blue pixel, is and it get worse - it's six pixels away vertically, versus two for the conventional bayer array. If you take a look at green pixels, in the conventional bayer array, any green pixel is between 4 other green pixels at a 45 degree angle, and has a gap of one in any direction vertically or horizontally. In the X-Trans sensor, the spacing is uneven - one pixel in one direction, two in another.

    In a real demosaicing engine, things are more complex than what I laid out in the previous paragraph - a modern engine uses information from all the pixels near it, not just one color, and also takes decisions on the fly about what pixels to use to get the best result. But the the basic rule is still that close, aligned pixel give better results, and the X-Trans layout is short on close, aligned pixels.

    To insult to injury, many interpolation implementations optimize based on the sensor pattern repeating on a power of two - two, four or eight. That allows you do bit masks rather than division/remainder operations. The X-Trans sensor repeats on a six pixel basis, not a power of two.

    Finally, Fuji sprang the X-Pro1 on the market without providing pre-launch information (or indeed post-launch information!) to raw developers. Take a look on forums frequented by Adobe staff, and those close to Adobe, and you'll find a number of negative comments abouts Fuji's handling of the X-Pro1's launch.

    Bottom line is, the X-Trans sensor trades off freedom from moire for a considerably more complex interpolation problem. And Fuji openly admit this. The question is whether the tradeoff is worth it - will the combination of better sensor level resolution because the AA filter isn't required anymore actually translate into a real advantage once the image is demosaiced. Which brings us to PhotoRaw's demosaicing engine.

    Implementing X-Trans interpolation in PhotoRaw

    Previous generations of PhotoRaw mainly used a modified AHD interpolation engine. It wasn't limited to two by two patterns, but it did make some assumptions about where it could find pixels. I say "mainly used" because not every sensor went through the AHD engine - some special cases were processed separately. Unhappily, the X-Trans sensor breaks the old engine's assumptions about pixel locations.

    So PhotoRaw needed a new way of processing X-Trans images.

    Attempt one - modified VNG: My first try at a solution was to find a quick and easy fix. Looking at the X-Trans sensor, it looked like it would be well suited to a gradients-type approach. The uneven pixel distances could be handled via weighting without much change to the underlying algorithm. Broadly speaking, calculating a gradient can be done similarly regardless of pixel distance. So I modified a VNG interpolation algorithm - VNG is threshold-based Variable Number of Gradients interpolation. The good news was, the solution worked, and worked a lot better than bilinear interpolation, which is the fallback when all else fails. The bad news was, it didn't work too well. Specifically, it fell over badly on large sharp transitions - in fact, it looked little better than bilinear interpolation there - blocky edges, etc.

    Attempt two - generalized AHD: When the easy approach didn't work well enough, I had two choices. Either continue working the VNG solution, or try something new. I chose to try to implement an AHD (Adaptive Homogeneity-Directed interpolation) solution instead. This was for a number of reasons. Firstly, I know AHD better than I know VNG, so there would be less work on understanding how to modify the algorithm. Secondly, PhotoRaw already has a well proven AHD interpolator that's multithreaded, etc, etc. And I mean really multithreaded as in multiple threads working on one image, as opposed to thread safe. Finally, I prefer AHD to other interpolation methods - I've tried most of the others, and my experience is that AHD is more stable.

    What I ended up implementing is a generalized AHD engine. It's table driven, so the way that it works is that initially PhotoRaw analyses the sensor pattern, and sets up a number of tables that describe how to interpolate the sensor. The advantage of this is that the analysis algorithm can be quite complex, and not necessarily efficient, because it only runs once. Then the multi-threaded core can just run through the tables very efficiently. The way its implemented, the analysis algorithm is general - it can analyze pretty much any sensor pattern and come up with a optimized set of interpolation tables. In essence, the analysis portion looks for optimal pixel combinations, how close the pixels are, whether the pixels are aligned, etc and choses the best combinations it can.

    And that's what will be running Fuji X-Pro1 conversions in future. It won't be running the majority of conversion however - the way PhotoRaw now operates is that it uses a hard coded, very efficient, engine for two by two patterns (95% plus of cameras), and goes to the general engine for more complex patterns.

    The results

    Here are some crops of the end results versus Adobe Camera Raw (ACR), SILKYPIX and the Fuji in-camera JPEG renditions of a sample image. The SILKYPIX conversion is from Ario Arioldi (thanks Ario!!!)

    << UPDATE: Part two of this post has results for beta versions of DCRaw and RPP as well, and this post shows results for Adobe's Lightroom 4.1 final >>


    Important note: these are 400% crops, and they are of an area on an image that I know will be really troublesome with raw converters - so bear in mind that we're looking at imperfections you probably won't ever see in practice. Also, the ACR crops below are a beta version of ACR.

    Also bear in mind that PhotoRaw runs on the iPhone or iPad - it operates in an environment with about one twentieth the memory of a desktop machine, and several orders of magnitude less processing power than either a desktop machine, or the dedicated image processing chip in the Fuji X-Pro1.

    The original image, and the ACR conversion is from DPReview's preview of the Fuji X-Pro1. You can check out the originals there.

    All of these have default sharpening, etc, etc :

    PhotoRaw 3.5.4, 400% crop

    This is PhotoRaw - good resolution, clean colors, very clean and well defined edges. However, some pixel noise in transitions from highlights, as seen in the reflections from the paperclips. That could be dealt with, e.g., by median filtering. The problems however is that a median filter, or similar algorithms, would add a lot more processing, and PhotoRaw really doesn't have scope for more processing. Maybe when the iPad 10 comes out.


    Adobe Camera Raw V7.1 beta

    ACR beta 7.1 is a bit different. At first glance, it's doing better than PhotoRaw. The paperclips are entirely clean - really impressively so. However, take a look at the "Product of Italy" text, especially the insides of the "A" and "Y". Lots of chroma smearing, and the letters are quite desaturated. This is an understandable trade-off by Adobe - ACR runs on the desktop, and so can implement complex filtering, and the resulting chroma smearing is less noticeable than PhotoRaw's pixel noise.


    Fuji X-Pro1 in-camera JPEG

    Thirdly, the Fuji in-camera JPEG - no problems with the paperclips, and while there is still some chroma smearing, it's less than the ACR beta. But the in-camera JPEG loses a little resolution to both the PhotoRaw and ACR rendition. E.g., take a look at the marks between the letters - in the PhotoRaw and ACR versions they're clearer.



    SILKYPIX conversion

    Finally, SILKYPIX. SILKYPIX is currently the only officially supported raw converter for the X-Pro1. Again, no problems with the paperclips, and while there is still some chroma smearing, it's less again than the in-camera JPEG, and a lot less than ACR. Resolutions appears slightly lower than PhotoRaw or ACR, but the difference is slight, and may be due to SILKPIX sharpening less by default. Really, the only issue here is that slight remaining chroma smear on the "A" and the "Y", and a slight loss of saturation in the red of the lettering.

    If any readers know exactly what processing is being used by both Adobe, SILKYPIX and Fuji that gives that chroma smearing effect, I'd be interested in knowing. Update: this mystery is solved in part three.

    Conclusions

    I mentioned above that Fuji X-Pro1 images are a pig to interpolate. They are. Whatever you may think of PhotoRaw, ACR is a very,very good raw converter, and here it's having trouble with the Fuji X-Pro1 images. That may be because it's a beta, but the hard truth is that any interpolation engine is going to have trouble with pixels that are spread out, not aligned, and unevenly spaced. SILKYPIX, the official raw developer for the X-Pro1, does a much more credible job, but still shows slight chroma smearing.

    It's worth going to the DPReview site and using their image comparison tool to compare the ACR conversion for the Nikon D7000 versus the Fuji X-Pro1 conversion, and the SILKYPIX conversion shown here. The Nikon and the Fuji X-Pro1 have virtually the same number of pixels (16.2 M versus 16.3 M), and approximately the same sized sensor. The Nikon D7000 conversion doesn't have the chroma smearing you see above.

    So my conclusion is, sorry to say, that the Fuji X-Pro1 X-Trans sensor doesn't deliver the Fuji promise of outperforming similarly sized sensors. In fact, it underperforms similar DX sensored cameras - with the official SILKPIX raw developer, the underperformance is too slight to be noticeable under normal circumstances, but is still there if you look closely.

    This is not to say that the Fuji X-Pro1 is a bad camera - far from it - the camera has great lenses, a really attractive viewfinder and design, and in many ways the sensor is great, with low noise and clean data, in the class of the recent Sony and Nikon sensors. If you use it with the official raw converter, it's within a whisker of the competition. But in my opinion, it would have been a better camera with a conventional two-by-two sensor layout.

    Maybe tomorrow Fuji will send me an email that explains how to interpolate Fuji X-Pro1 images such that they deliver on the promise. And I'm certainly interested to see how a final version of ACR performs, and also how Dave Coffin's DCRaw handles the Fuji X-Pro1, once Dave updates DCRaw to handle it. (Update: part 2 of this post has results from a DCRaw beta.) But right now, I don't see the performance that's been claimed for the X-Pro1. (Second update: part three shows how a modified form  of PhotoRaw gets very, very close to the performance of SILKYPIX.)

    NOTE: This post has been updated with SILKYPIX images since it was first published.

    Part two of this post is here

    Part three of this post is here

    Finally, if you're interested in my (somewhat cynical) view on the relationship between camera manufacturers and software developers, you can read this post
    17

    View comments

  9. Leica's just announced the M Monochrom - essentially a Leica M9 with a monochrome sensor, although apparently calling it a "M9 Monochrom" is bad. Very bad!

    Now the M Monochrom isn't for me - I do a lot of monochrome photography, but the way I work is  by using channel mixing to get the same effect as the red, yellow, etc filters that I used to use back when I first learned photography using my father's old rangefinder camera. Nostalgia is great, but I don't feel that the gain in resolution would justify going back to physical filters.

    However, the M Monochrom still fascinates me - it's very much a purist's camera, and those that know me will probably confirm that I have a large purist streak.

    There does seem to be quite a bit of confusion around the M Monochrom's sensor. Leica haven't been very forthcoming on the topic, and none of the initial reviews tell us much.  The assumption seems to be that the sensor is an M9 sensor with the Bayer array removed. That may be the case, but I doubt it's that simple.

    You could have a "naked" sensor, but there are at least two reasons why there probably would be a filter on the M Monochrom's sensor:
    1. Varying sensitivity to wavelength of the sensor itself (usually corrected for by the matrix in raw processing of a CFA sensor, but there's no matrix here). 
    2. Also human's perception of luminance differs by color. 
    By way of comparison, here are the spectral sensitivities of good old TRI-X film and a monochome CCD sensor (not the M Monochrom one, a similar one).

    TRI-X film

    Monochrome CCD sensor

    Now, note that the TRI-X curve is in log scale, and the CCD is in a linear scale. But it's still pretty clear that there is a huge difference in response. The CCD rises to a peak at about 650 nm, which is deep red. TRI-X has a "brick wall" cut off at just about 650 nm. In addition, TRI-X responds deep in the ultra-violet, down to 300 nm (which is why UV filters were a good idea on film cameras, btw).

    The M Monochrom's sensor may respond differently to the one above, but the point remains. If you don't filter to get the sensor output to conform to what us humans perceive as luminance, the images will appear "muddy" or "wrong". So you would probably need a single color filter (versus a 4 color bayer filter) on a mono sensor. It should certainly pass more light in total than a bayer array, but not 100%. This would be a design trade-off for the manufacturer - more sensitivity versus a more natural color response. You would also hopefully have a IR filter, else you'd end up with out-of-focus IR contamination blurring your otherwise beautiful high-res image.

    In fact, somewhat ironically, the most important characteristic of a monochrome sensor is probably its color response. You can't adjust the response in post as you can with a regular sensor, so you really need it to be built right.

    So, I certainly hope that the M Monochrom has a filter of some sort. Looking at the curves, I would think that Leica would want a red/infra-red cut filter, and probably a green cut as well if they wanted to match something like TRI-X. If they wanted something that matched a theoretical luminance equation, which is green heavy, they might only use a red/IR filter.

    Hopefully Leica have tuned the M Monochrom to be similar to some of the classic B&W films. Certainly, after the M8 IR debacle, and the M9 red edge debacle, one would hope that they're sensitive to special response as an issue. Slightly scary that many of the sample images look a bit grey and muddy - like the filter isn't too great.

    Sad that this aspect was entirely missed in the reviews. This need for a filter, BTW, was probably what Leica were alluding to when they talked about it being the M Monochrom being more than just a M9 without the bayer array.

    Note to Leica: I get it that you want to be loyal to reviewers that supported you back when all that Leica had to sell was the M7. I also get that the PR department wants reviewers that are prepared to stick to the company line. But could you at least invite one reviewer that is digital imaging savvy enough to be able to at least ask the right questions? Please? It's a bit frustrating to have to read through pages about sensor resolution (yes, no bayer filters, better resolution. Got it!!!), but never get to any useful information.

    Update: Above I made reference to "theoretical luminance equation, which is green heavy, they might only use a red/IR filter". I should probably make that a little less cryptic - the classic formula is
     Y = 0.2126 R + 0.7152 G + 0.0722 B 
    In other words 71% of luma is green, rather than a flat response. That is approximately what you would get in a conventional color to mono conversion in LR or C1. But a bare sensor would, I'd suggest, have too much red sensitivity to do a good job no matter what view you take on what the ideal should be.

    Second Update: Leica have published the M Monochrom's actual spectral sensitivity - see this post.


    5

    View comments

  10. There's a good article, complete with a step-by-step guide, on using CornerFix with the 15mm Voigtlander Super Wide Heliar f/4.5 Asph. Well worth the read. The photographs are pretty good as well.

    Check it out here.
    1

    View comments

Popular Posts
Blog Archive
About Me
About Me
My Photo
Author of AccuRaw, PhotoRaw, CornerFix, pcdMagic, pcdtojpeg, dcpTool, WinDat Opener and occasional photographer....
Loading