Perfect Sharpening

20 February 2010

A Proposal for Perfect Edge Sharpening Using COTS Equipment - Software Implementation Required
Sean Baker, 20 February 2010

Summary
It is possible, through software implementation, to use existing, COTS equipment to create for any digital imaging combination a 'perfect' recreation of scene sharpness, accounting in one step for lens imperfections, diffraction, low-pass filtering, and demosaicing. Image data truly lost by the imaging system is not proposed to be recoverable, but restoration of all that which was originally transmitted is. Suggestions for implementation are provided; specific code and algorithms are not.

Background
In the digital photography world, it is common for photographers working in certain fields to photograph color charts so as to later ensure that when colors in the scene are rendered in the final digital image match their color in life, controlling for variance in sensor color sensitivity, color casts in lighting equipment & modifiers employed, environmental reflections, etc. The most common such tool is a Gretag MacBeth ColorChecker - a card consisting of 24 swatches of known, factory-inspected color which are then used by software such as XRite's own application, Adobe's DNG Profiles Editor, or various scripting solutions for Adobe Photoshop to render color perfectly after its RAW capture.

It is also known that digital imaging systems by their nature have a certain degree of 'softness' to their RAW files as a result of optical imperfection in lenses, low pass filters employed to prevent moire patterns, the Bayer array itself, etc. This softness leads to many experts in the field recommending a 'Capture sharpening' step to post-processing, so that all subsequent actions taken are based on the most accurate version of the scene available.

These two facts present an opportunity for one to combine them and create a reconstruction of the imaging system's imperfections, allowing a theoretically 'perfect' sharpening during post production.

Concept, Implementation
By using a frame-filling, or nearly frame-filling shot of a color checker, the edges of the color swatches can be used in determining both the horizontal as well as the vertical components of the system Point Spread Function (PSF) across all parts of the image area. The advantage here over other point or line based systems is that correction can be made for each part of the frame and not generalized over the whole (consider that the MTF of a lens always changes, generally degrading radially from the image center). As each component edge of the CC card is sampled, the edge width is measured and stored in a correlation table along with its image position. The user at this point might also be given the option to provide additional CC samplings to provide additional data points. Upon completion of the table, a fitting function (likely - but not necessarily - a 2d order polynomial) is generated from the table and used to calculate horizontal and vertical PSF components at each pixel position in the image. These fitting functions ought be savable and transmissible in the same manner as presets in Adobe Lightroom, Curves in Adobe Photoshop, etc.

Sharpening is now performed to restore scene sharpness. In the actual software implementation, it is desirable that the user be given a familiar 'Strength' setting, allowing him / her to use more or less of the sharpening than 'ideal', as well to include at least a rudimentary noise reduction function to exclude such pixels from the sharpening process. [It should be noted that for the NR mentioned, if something more advanced than a median filter is to be employed, it would be ideal again to sample from the known darker and / or bluer portions of the CC chart so as to determine the 'baseline' noise present.]

Recommendation, Closing
As someone who values image sharpness - or at least the option of having it - I would pay good money to have a product such as this in my workflow. As competition within the digital postproduction realm increases, companies like Adobe would do well to internalize such a feature for future revisions of its products (namely - LR, PS) to maintain its position as the 'gold standard'. As a user, if you feel this would be a valuable addition to your workflow, I strongly suggestion you contact Adobe and let them know your thoughts. They'll only be able to devote the time if there is a genuine clamoring for it, so it's to us to make enough noise to be heard.

Afterward
It's worth noting that the idea outlined above will still be imperfect (and why a 'strength' control becomes so important). It will not account for very low frequency contrast differences across the imaging area. As well, because the focus distance for a frame-filling color checker is likely to be considerably different than the subject distance in the actual scene, the optical system will move and its MTF slightly altered in the process. This is unavoidable, but also presents another possibility to be explored. By using an interface similar to that of PictureCode's Noise Ninja, wherein problem noise areas are selected and controlled for by the software, the user might select those edges within a scene which are in focus and which ought to be 'perfectly' sharp (natural edges, etc.). By correlating these across the scene in the same manner as described above [correlation table, fitting function, etc.] the scene might be better restored than by any conventional method.

Also, the horizontal and vertical components might also be allowed to be supplemented either in the manner described just above, or by allowing considerable rotation of the CC during its capture, allowing for calculation of more axes to a PSF. While this will considerably increase the level of computation involved, for some users it may be worth the time & effort.

Update:
Thanks to Bob Freund over at MM, I've been reminded that I should have been explicit in stating that all of the above measurement and sharpening can and should only be performed after the correction of chromatic aberration, as its presence could confuse or wholly ruin any measure of the actual PSF. An algorithm could be developed to account for, and overcome its presence, but better to just correct it in the first place - after all, who cares this much about sharpening but not correcting CA?
-