Supplementary Material for IEEE Transactions on Image Processing

Authors

David Alleysson, Sabine Süsstrunk and Jeanny Hérault

Summary

Matlab code is available upon request.

(Note that sometimes you have to click twice on a link to bring the pictures up. We are trying to fix the bug)

There is an analogy between single-chip color cameras and the human visual system in that these two systems acquire only one limited wavelength sensitivity band per spatial location. We have exploited this analogy, defining a model that characterizes a one-color per spatial position image as a coding into luminance and chrominance of the corresponding three-colors per spatial position image. Luminance is defined with full spatial resolution while chrominance contains sub-sampled opponent colors. Moreover, luminance and chrominance follow a particular arrangement in the Fourier domain, allowing for demosaicing by spatial frequency filtering. This model shows that visual artifacts after demosaicing are due to aliasing between luminance and chrominance and could be solved using a pre-processing filter. This approach also gives new insights for the representation of single-color per spatial location images and enables formal and controllable procedures to design demosaicing algorithms that perform well compared to concurrent approaches, as demonstrated by experiments.

Bayer CFA
Figure 1: Decomposition of the Bayer CFA into three sub-sampled lattices
Lighthouse Bilinear
(a) (b)
Original Low-Pass Bilinear
(c) (d)
Figure 2: Example of a bilinear interpolation computed with convolution filters. (a) Original image: the frequency spectrum of the red (and green and blue) occupies most of the Fourier spectrum. (b) Reconstructed image with bilinear interpolation. (c) Low pass filtering applied to the original image to reduce the size of the Fourier spectrum of the image. (d) The reconstruction of image (c) after mosaicing according to the Bayer CFA and demosaicing by bilinear interpolation shows no artifacts.
Lighthouse Luminance Chrominance
(a) (b) (c)
Bayer CFA Image Multiplexed and sub-sampled Chominance Sub-sampled Chrominance
(d) (e) (f)
Figure 3: Decomposition of (a) a color image into (b) luminance and (c) chrominance as defined in eq. 5. (d) A CFA image with a single chromatic sensitivity per spatial location according to the Bayer CFA arrangement (the image appears greenish because of the double number of green pixels in the CFA). When subtracting luminance (b) from the CFA image (d), the resulting image is (e), which corresponds to a multiplexed version of sub-sampled chrominance. Selecting pixels in front of each color sample of the Bayer CFA (i.e. de-multiplexing opponent colors) results in image (f). Also, sub-sampling image (c) results exactly in image (f). Thus, image (c) can be recovered from (f) using interpolation.
CFA in Fourier
(a)
Luminance filter Amplitude Spectrum
(b) (c)
Figure 4: (a) Fourier representation of a CFA image. The energy of the luminance is concentrated in the center of the image. The energy of chrominance is located on the borders. (b) The luminance filter used in the simulations discussed in Section V. (c) Representation of the transfer function of the luminance estimation filter.
Excessive blurring Grid Effect
(a) (b)
Watercolor False Color
(c) (d)
Figure 5: The four artifacts visible after demosaicing: (a) Excessive blurring (b) Grid effect (c) Watercolor (d) False color.
Lighthouse result Sails result Statue result
Window Result
Figure 6: Results of the frequency selection demosaicing algorithm on four images. The same luminance filter (Figure 4-b) is used for the reconstruction.

Supplementary Figures

lighthouse Freqsel lighthouse Alternating projection1 lighthouse AP2
ZLFS ZLAP1 ZLAP2
ZLFS ZLAP1-1 ZLAP2-2
demosaicing by frequency selection alternate projection with bilinear interpolation alternate projection with template matching
sails freq sel sails AP1 sails AP2
ZSFS ZSAP1 ZSAP2
demosaicing by frequency selection alternate projection with bilinear interpolation alternate projection with template matching
window AP1
ZWFS ZWAP1
demosaicing by frequency selection alternate projection with bilinear interpolation
window BL2
ZWAP2
alternate projection with template matching
Figure S1: Comparison of demosaicing by frequency selection and two implementations of the alternating projection algorithm [1]. In the first implementation, we use bilinear interpolation as initialization, one level of decomposition, and eight iterations. In the second implementation, we use a template matching method as initialization, one level of decomposition, and eight iteration. For demosaicing by frequency selection, we use for all images the Filter given in Figure 4 (a).
CFAGB FS CFA GB
ZFSCFAGB ZFSCFAGB2
Figure S2: An image mosaiced with a Bayer CFA containing two times more blue than red and green pixels, demosaiced by frequency selection.
CFARG FS CFA RG
ZFSCFARG ZFSCFARG2
Figure S3: An image mosaiced with a Bayer CFA containing two times more red than blue and green pixels, demosaiced by frequency selection.
Bayer CFA R<->G B<->G
Lighthouse 33.88 34.43 34.34
Sails 35.72 36.28 36.59
Statue 37.70 37.92 37.86
Window 35.27 35.47 35.87

Table 1: Comparison of CPSNR obtained with the Bayer CFA, a modified Bayer CFA where the red and green pixels are exchanged, and a CFA where the blue and green pixels are exchanged. An optimal filter is computed for each image and each CFA with respect to CPSNR.

  Bilinear Hue Based Gradient Based Alternating projection 1 Alternating projection 2 Frequency Selection
Lighthouse 25.44 27.04 31.5 32.33 35.26 33.75
Time 1 1.70 49.59 28.34 32.64 2.30
Sails 28.43 29.91 34.37 34.89 36.92 35.72
Time 1 1.65 48.21 28.34 32.47 2.30
Statue 28.36 29.66 32.78 35.86 37.65 37.53
Time 1 1.34 48.21 29.75 33.94 2.41
Window 27.96 29.60 31.46 34.76 35.40 34.79
Time 1 1.31 48.87 27.99 32.08 2.31

Table 2: Comparison of CPSNR and computation time between different demosaicing algorithms. The same frequency selection filter (Figure 4-b) is used for all images.

 

[1] B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, Color plane interpolation using alternating projections, IEEE Transactions on Image Processing, vol. 11, no 9, Sept. 2002, pp. 997-1013