Fake Foveon Adventure

Disclaimer

Please note that I do not have a background in color science. This article has been written for fun and is based on pop-level digital photography understanding. As such, the results discussed here may vary and are likely to be worse compared to those obtained using modern RAW processors.

Fake Foveon Adventure

Today, I played with the concept of achieving Foveon-like image - using standard Bayer CFA sensor. 

Foveon sensors are unique, they capture color information differently - with each pixel gathering a full spectrum of visible light, though there are some limitations. Developed by Sigma Corporation, this technology has a cult following because of its impressive image quality. I'm anticipating their release of a 35mm full-frame sensor using this technology.

FOVEON     / CC BY-SA 3.0 Dicklyon & Anoneditor / Wikipedia

In contrast, traditional digital cameras use a Bayer pattern filter on top of monochromatic sensor, which provides somewhat "complete" information regarding color-related brightness - but only for the color of the actual color filter at the top of the sensel.  CFA (color filter array) at the top of monochromatic sensor - a mosaic of R G G B (or similar) pattern that is trying to capture information required to reconstruct image that digital camera/sensor is seeing. 

Bayer CFA array over monochromatic sensor: / CC BY-SA 3.0 Cburnett / Wikipedia

After capturing light, CCD or CMOS sensors convert the number of photons into digital values through an Analog-to-Digital Conversion (ADC). This results in values that roughly represent the brightness projected onto each camera sensel.  Data from all pixels is converted through multiple steps to reconstruct the full image. The color RGB value at each pixel is estimated (interpolated). This process takes advantage of the fact that the human eye is more sensitive to brightness than to color, allowing some approximations.

For instance, if you have a Green pixel, it means this pixel captured only Green light. To obtain full RGB data for that pixel, you consider the values of surrounding pixels. This process is repeated for all pixels. Afterward, Unsharp Mask (USM) is applied, along with various coefficients to adjust the white balance, and so forth.

So what we can do to skip USM and guessed color values at each pixel?  To have *fake* Foveon sensor we can use GRGB array [2x2 blocks] and just average this data block into one pixel (this operation have several limitations that I will skip today). In the case of my camera, the Sony A7R, which has 36 megapixels, the resulting image will be 9 megapixels due to the processing method used. Although this is relatively low resolution, I am curious to see if the image will exhibit qualities similar to those produced by Foveon.

As you might notice it is still different process, as Foveon can capture full spectrum in one area, we are averaging something from wider space (from 4 pixels) - so this is prone to errors, like aliasing or wrong color.

To see the results for yourself:

1. Download supplied RAW file
2. Download/Install dcraw
3. Install Python3
4. Download Python script

Execution steps:

In dirctory of ARW file execute 

dcraw -d -4 DSC01801.ARW

The result will be a DSC01801.pgm file (Linear 16-bit, to avoid posterization, in Document mode). Document mode is a special feature provided by dcraw that outputs a flat file containing values for each pixel without any color information. At the Python script level, we will apply the "color logic."

Now run:

pip3 install numpy

pip3 install opencv-python

python3 interpolate_new.py

The Result:

path_to_output_rgb_image.png

Resulting image can be loaded into any "raw" editing software just to adjust white balance and overall tonality - I've applied DXO data from CIE-50 measurement to reconstruct colours. If you want to use your RAW file you have to modify python script with proper values.

DXO A7R color conversion matrix 
https://www.dxomark.com/Cameras/Sony/A7R---Measurements

Have fun!

Document mode output - this is what sensor sees, without colouring each pixel or interpolating colour data.

Regular RAW conversion + scale down (to match Foveon) and then blown up x2 to check detail levels

Fake Foveon conversion (upscaled x2 for detail inspection), colours reconstructed using DXO data.

Results

It seems that the entire exercise was more of a learning experience; the differences were minimal at best. I'll try again another day with noisy, high-ISO images, or perhaps with a different subject or a different camera to see if that makes a significant difference.