In this paper, we deal with the demosaicing problem when the Bayer pattern is used. We propose a fast heuristic algorithm, consisting of three parts.
In the first one, we initialize the green channel by means of an edge-directed and weighted average technique. In the second part, the red and blue channels are updated, thanks to an equality constraint on the second derivatives. The third part consists of a constant-hue-based interpolation.
We show experimentally how the proposed algorithm gives in mean better reconstructions than more computationally expensive algorithms.
The demosaicing problem is related to the acquisition of RGB color images by means of CCD digital cameras. In the RGB model, each pixel of a digital color image is associated to a triple of numbers, which indicate the light intensity of the red, green and blue channel, respectively. However, most cameras use a single sensor, associated with a color filter that allows only the measure at each pixel of the reflectance of the scene at one of the three colors, according to a given scheme or pattern, called Color Filter Array (CFA). For this reason, at each pixel, the other two missing colors should be estimated. Different CFA’s are proposed for the acquisition [1C. Bai, Z. Lin, J. Yu, Y.W. Chen, and Y.W. Chen, "Penrose demosaicking", IEEE Trans. Image Process., vol. 24, no. 5, pp. 1672-1684.[http://dx.doi.org/10.1109/TIP.2015.2409569] [PMID: 25769160] -3K. Hirakawa, and P. Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity", Image Processing (ICIP), pp. 81-4.[http://dx.doi.org/10.1109/ICIP.2007.4379097] ]. The most common is the Bayer pattern [4BE Bayer, " Color imaging array. ", Patent US. 3971065, .]. In this scheme, the numbers of pixels in which the green color is sampled are double with respect to those associated with the red and blue channels, because of the higher sensibility of the human eye to the green wavelengths. If we decompose the acquired image into three channels, we obtain three downsampled grayscale images, so that demosaicing could be interpreted as interpolating grayscale images from sparse data. In most cameras, demosaicing is a part of the processing required to obtain a visible images. The camera’s built-in-firmware is substantially based on fast local interpolation algorithms.
The heuristic approaches, which do not try to solve an optimization problem defined in mathematical terms, are widely used in the literature. These methods, in general, are very fast. Our proposed technique is of heuristic kind. In general, the heuristic techniques consist of filtering operations, which are formulated by means of suitable observations on color images. The nonadaptive algorithms, among which bilinear and bicubic interpolation, yield satisfactory results in smooth regions of an image, but they can fail in textured or edge areas. Edge-directed interpolation is an adaptive approach, where, by ana-lyzing the area around each pixel, we choose the possible interpolation direction. In practice, the interpolation direction is chosen to avoid interpolating across the edges. In [5R.H. Hibbard, "Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients", Patent US. 5382976, .], for each pixel the horizontal and vertical gradients are compared with a constant threshold. If the gradient in one direction is greater than the threshold, then interpolation is not performed along this direction. Some other direct interpolation methods use larger neighborhoods by examining different color channels. In [6CA Laroche, and MA Prescott, "Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients", Patent US. 537332, .], to determine the edges of the green channels, the red and blue channels are employed. On the other hand, to determine the edges of the red and blue channels, some discrete derivation operators of the second order are used, while in [7R. Kakarala, and Z. Baharav, "Adaptive demosaicking with the principal vector method", IEEE Trans. Consum. Electron., vol. 48, pp. 932-937.[http://dx.doi.org/10.1109/TCE.2003.1196423] ], to determine the edges in the various channels, a suitable Jacobian operator is applied. In [8A. Buades, B. Coll, J.M. Morel, and C. Sbert, "Self-similarity driven color demosaicking", IEEE Trans. Image Process., vol. 18, no. 6, pp. 1192-1202.[http://dx.doi.org/10.1109/TIP.2009.2017171] [PMID: 19403366] ], local homogeneity is used as an indicator to choose horizontally or vertically interpolated intensities. Thanks to homogeneity-directed interpolation, the luminance and chrominance values have to be similar in a suitable neighborhood. In demosaicing it is often assumed that the differences or the ratios of the intensity values in different channels are locally constant [5R.H. Hibbard, "Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients", Patent US. 5382976, ., 6CA Laroche, and MA Prescott, "Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients", Patent US. 537332, ., 9J.E. Adams, "Interactions between color plane interpolation and other image processing functions in electronic photography", Proc. SPIE, vol. 2416, pp. 144-151.[http://dx.doi.org/10.1117/12.204825] -14J.A. Weldy, "Optimized design for a single-sensor color electronic camera system", Proc. SPIE, vol. 1071, pp. 300-307.[http://dx.doi.org/10.1117/12.952531] ]. In [11R. Kimmel, "Demosaicing: image reconstruction from color CCD samples", IEEE Trans. Image Process., vol. 8, no. 9, pp. 1221-1228.[http://dx.doi.org/10.1109/83.784434] [PMID: 18267539] ] the probability of having an edge in a certain direction is determined and used to find the weights relative to the weighted average employed as an interpolation operator. In this algorithm, the color channels are updated iteratively according to the constant color ratio condition. In [15W. Lu, and Y.P. Tan, "Color filter array demosaicking: New method and performance measures", IEEE Trans. Image Process., vol. 12, no. 10, pp. 1194-1210.[http://dx.doi.org/10.1109/TIP.2003.816004] [PMID: 18237887] ] a similar algorithm is proposed, where -size neighborhoods are employed to find the edges of the green channel, and -size neighborhoods are used to determine the edges of the red and blue channels. An analogous algorithm is defined in [16X. Wu, W.K. Choi, and P. Bao, "Color restoration from digital camera data by pattern matching", Proc. SPIE, vol. 3018, pp. 12-17.[http://dx.doi.org/10.1117/12.271585] ], where the interpolation can be done also in the diagonal direction, while in [17P.S. Tsai, T. Acharya, and A.K. Ray, "Adaptive fuzzy color interpolation", J. Electron. Imaging, vol. 11, pp. 293-305.[http://dx.doi.org/10.1117/1.1479702] ] the weighted directional interpolation is used by means of a fuzzy membership assignment. A second order operator is employed as a correction term. [18J.E. Adams, and J.F. Hamilton, "Design of practical color filter array interpolation algorithms for digital cameras", Proc. SPIE, vol. 3028, pp. 117-125.[http://dx.doi.org/10.1117/12.270338] ]
To have more accurate results, several techniques, which use iterative methods, are proposed. However, they have a higher computational cost with respect to the heuristic techniques. One of well-known techniques is the algorithm of Alternate Projections (AP) [19B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, "Color plane interpolation using alternating projections", IEEE Trans. Image Process., vol. 11, no. 9, pp. 997-1013.[http://dx.doi.org/10.1109/TIP.2002.801121] [PMID: 18249722] ], which uses the strong correlation between the high frequences of the three colored components, by projecting alternately the estimated image in a constraint of observation and in a constraint which imposes similarity between the red and green edges and between the blue and green edges, until a fixed point is found. Another widely used technique is regularization [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] , 21D. Menon, and G. Calvagno, "Regularization approaches to demosaicking", IEEE Trans. Image Process., vol. 18, no. 10, pp. 2209-2220.[http://dx.doi.org/10.1109/TIP.2009.2025092] [PMID: 19527958] ]. The algorithm is based on interpolation in a residual domain [22D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, "Beyond color difference: Residual interpolation for color image demosaicking", IEEE Trans. Image Process., vol. 25, no. 3, pp. 1288-1300.[PMID: 26780794] ]. The residuals are the differences between the observed and estimated pixel values which minimize a Laplacian energy.
The algorithm here presented consists of three steps. The first two ones are initialization steps, while the third one is an iterative steps. In the first one, the missing valued in the green component are determined, in particular a weighted average-type technique is used. The weights are determined in an edge-directed approach, in which we consider also the possible edges in the red and blue components. In the second step, we determine the missing values in the red and blue components. In this case we use two alternative techniques, according to the position of the involved pixel in the Bayer pattern. In the first technique, the missing value is determined by imposing that the second derivative of the intensity value of the red/blue channel is equal to the second derivative of the intensity values of the green channel. This is done according to the proposed approaches in the AP algorithm and the regularization algorithm given in [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ]. In particular, a constraint is imposed, to make the derivatives of all channels similar as soon as possible [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ]. At the third step, all values of the three channels are recursively updated, by means of a constant-hue-based technique. In particular, we assume the constant color difference. The technique we propose at this step is similar to that used by W. T. Freeman [10W. T. Freeman, "Median filter for reconstructing missing color samples", Patent US. 4774565, .]. Indeed, even here a median filter is employed, in order to correct small spurious imperfections. We repeat iteratively the third step. However, to avoid increasing excessively the computational cost, we experimentally estimate that only four iterations are necessary to obtain an accurate demosaicing. We call our technique as Local Edge Preserving (LEP) algorithm.
The paper is structured as follows. In Section 2 we give a mathematical formulation of the demosaicing problem. In Section 3 we describe the initialization of the proposed algorithm, which consists of the two first steps aforementioned. In Section 4 we give the third iterative step of our algorithm, highlighting the differences with the Freeman filter. In Section 5 our experimental results are presented. This section consists of two parts. In the first one, we determine the best detection function which can be used in order to evaluate the edges. In the second one, we compare our algorithm with some other techniques recently proposed in the literature and we show how the LEP method gives in mean more accurate reconstructions than the other considered algorithms.
An RGB (Red-Green-Blue) color image with height n and width m is a vector of the type
where are the red, green and blue channels according to the lexicographic order, respectively. We consider the problem of acquisition of data from a digital camera, and call it "mosaicing problem". Given an ideal image , the acquired or "mosaiced image" is defined by
where and is a linear operator defined by setting
is the null matrix, and are diagonal matrices whose principal entries, if we use the Bayer pattern Fig. 1, are given by
where the symbol indicates that i-j is even. The corresponding demosaicing problem is the associated inverse problem, that is to determine the ideal color image x, knowing the mosaiced image and the linear operator M. An inverse problem is said to be well-posed (in the sense of Hadamard) if and only if the solution exists, is unique and stable with respect to data variation. A not well-posed problem is said to be ill-posed [23J. Hadamard, Lectures on cauchy’s Problem in linear partial differential equations. Yale., University Press: New Haven, .]. Note that the demosaicing problem is ill-posed, since the matrix M in (2) is singular, as it is readily seen, and so there are infinitely many solutions.
Fig. (1) Bayer pattern. |
In the initialization phase we proceed as follows: first we initialize the green channel, since in this channel we have more data than in the other ones, and successively we update the other two.
We refer to a clique as a pair of adjacent pixels. Every missing value of the green channel is initialized by a weighted mean of the known green values in its neighborhood. The weights of the considered mean take into account possible discontinuities in a set of adjacent cliques. We consider cliques both in the blue and in the red channel, since it is well-known that there is a correlation between the discontinuities in the various channels related to edges, such as object borders and textures [11R. Kimmel, "Demosaicing: image reconstruction from color CCD samples", IEEE Trans. Image Process., vol. 8, no. 9, pp. 1221-1228.[http://dx.doi.org/10.1109/83.784434] [PMID: 18267539] ]. Here we distinguish three cases: the first one is when we have the value of the green light intensity on a pixel; the second one is when we the blue value of the involved pixel is known, that is when i and j are both odd; the third one is when the red value on the considered pixel is known, namely when i and j are both even.
The first approximation g^{(0)} of the green ideal image g is given by
Note that, in the first case, we keep the value we already have. In the second case, we do a weighted mean of the intensity values taken on the adjacent pixels where the green value is known. The weights t_{1}, t_{2}, t_{3}, t_{4} of the mean are computed by using the green and the blue channels. We define
(1) |
In particular, we get
(2) |
where k = 1, 2, 3, 4, ϕ is a suitable positive decreasing detection function and τ_{k} is defined by
and the pixels e_{k}, f_{k}, p_{k}and q_{k} are as in (1). When the differences between the green values on the pixels e_{k} and f_{k} (see the yellow arc in Fig. (2) for k = 1), and p_{k} (see the cyan arc in Fig. (2) for k = 1), and between the blue values on the pixels (i, j)and q_{k} (see the brown arc in Fig. (2) for k = 1) are small enough, then we can assume that there are no discontinuities between the pixels e_{k} and (i, j) (see the red line in Fig. (2) for k = 1). So, in the calculus of the green value on the pixel (i, j), we give a large weight t_{1} to the green value in the position e_{k}. Thus, when the value τ_{k} is small, the probability of having a discontinuity between the pixels (i, j) and e_{k} in the green channel is large, and vice versa. The computation of τ_{k} is illustrated in Fig. (2). For k = 1, 2, 3 the computation of τ_{k} can be described by an appropriately rotated similar figure.
Fig. (2) Computation of for τ_{k} for k = 1. |
Even in the third case, we compute the weighted mean of the intensity values taken on the adjacent pixels where the green value is known. The weights t_{4+k}, k = 1, 2, 3, 4 of the mean are computed by using the green and the red channels. In particular, t_{4+k} = ϕ (τ_{4+k}), where
and e_{k}, f_{k}, p_{k} and q_{k} are as in (1). We argue analogously as in the computation of the weight t_{k}, k = 1, 2, 3, 4, where the role of the blue channel is played by the red component.
Here we distinguish four cases: the first one is when we already know the red value of a pixel; the second one is when we know the red values in the two adjacent pixels in the same column, that is i is odd and j is even (Fig. 3a); the third one is when we know the red values in the two adjacent pixels in the same row, namely i is even and j is odd (Fig. 3b); the fourth one is when we know the red values of the pixels adjacent in the corners of the involved pixel, that is i and j are both odd (Fig. 3c). In the second and in the third case we equalize the second derivatives of the red and the green channels previously computed. In the last case we use the computed values of the red channel to determine the weights of a suitable mean.
So, we define the initial estimate r^{(0)} of the red ideal image r by
Note that, in the first case, we keep the value which we already have. In the second case, we pose that the finite difference of the second order in the vertical direction of the red channel coincides with that of the green channel, which we have already initialized, namely
(3) |
Since we know , we can deduce the value of from (3).
Fig. (3) Different cases in the initialization of the red channel. |
In the third case, we impose that the finite difference of the second order in the horizontal direction of the red channel coincides with that of the green channel, just already initialized, that is
(4) |
By proceeding analogously as above, we obtain the value of from (4).
In the fourth case, we do a weighted mean of the intensity values taken on the adjacent pixels where the red value has just been computed. The weights t_{8+k}, k = 1, 2, 3, 4, of the mean are calculated by using the just initialized red channel and the observed blue channel. In particular, t_{8+k} is given by ϕ(τ_{8+k}), k = 1, 2, 3, 4, where ϕ is the detection function used in initializing the green channel, and
where e_{k}, f_{k}, p_{k}and q_{k} are as in (1). When the differences between the red values on the pixels e_{k} and f_{k}, e_{k} and p_{k}, and between the blue values on the pixels (i, j) and q_{k}, are sufficiently small, then we can suppose that there are no edges between the pixels (i, j) and e_{k}. So, in the calculus of the red value on the pixel (i, j), we have a large weight t_{8+k}, k = 1, 2, 3, 4, in correspondence with the red value in the position e_{k}.
Also in this setting, we distinguish four cases: the first one is given when we know the blue value of a pixel; the second one is when we know the blue values in the two adjacent pixels in the same column, that is i is even and j is odd (Fig. 4a); the third one is when we know the blue values in the two adjacent pixels in the same row, namely i is odd and j is even (Fig. 4b); the fourth one is when we know the blue values of the pixels adjacent in the corners of the involved pixel, that is i and j are both even (Fig. 4c). In the second and third cases we equalize the second derivatives of the blue and the green channels previously calculated. In the last case we use the computed values of the blue channel to determine the weights of a suitable mean.
Fig. (4) Different cases in the initialization of the blue channel. |
Thus, we define the estimate b^{(0)} of the blue ideal image b by
Note that, in the first case, we keep the value which we already have.
In the second case, analogously as before, we impose
(5) |
As we know , we derive the value of from (5).
In the third case, similarly as above, we get
(6) |
By arguing as in the previous section, we deduce the value of from (6).
In the fourth case, we do a weighted mean of the intensity values of the adjacent pixels where the blue value has just been computed. The weights t_{12+k}, k = 1, 2, 3, 4, of the mean are calculated by using the observed red channel and the just initialized blue channel.
Analogously as before, we obtain t_{12+k} = ϕ (τ_{12+k}), where
where e_{k}, f_{k}, p_{k}and q_{k} are as in (1).
A classical filter, often used to solve the demosaicing problem, is the Freeman filter also [10W. T. Freeman, "Median filter for reconstructing missing color samples", Patent US. 4774565, .]. The phase described in this section is a suitable modification of this filter. The Freeman filter performes the initialization phase by means of the bilinear filter, which works as follows. When the value of a certain color of a pixel is not available, such a value is computed by the arithmetic mean of the values of that color, which are assumed in the neighborhood of this pixel, that is the bilinear estimation is given as
Moreover, the following values are defined, by means of the median of the color differences of the channels red-green and blue-green [10W. T. Freeman, "Median filter for reconstructing missing color samples", Patent US. 4774565, .]:
where
(7) |
with . The median turns out to be very useful to correctly preserve the edges which are in the images. Indeed, the median filter is often used to restore images corrupted by salt-and-pepper noise, namely by the noise present only in a few pixels not adjacent each other.
In the Freeman filter it is assumed that the color differences are constant in a suitable subarea. Thus, the Freeman estimation is defined as follows:
In this paper we modify the Freeman filter as follows.
From the initial estimation we define the following variables:
where . So, we define the estimates for s = 1, 2,... as follows:
We pose our final estimate as
We saw experimentally that a good approximation is given by . We call the technique associated to this estimate as Local Edge Preserving (LEP) algorithm.
In this section we present the experimental results obtained from the implementation of the proposed algorithm, which was tested for the Bayer CFA on the set of Kodak sample images [24"Kodak lossless true color image suiter", Available on: http:// r0k. us/ graphics/ kodak/], of size, shown in Fig. (5). This dataset represents the typical benchmark images used in the literature to compare the various demosaicing algorithms. These high quality images have been acquired as raw images, in order to minimize the compression. We have implemented our algorithm in the C language on an Ubuntu operating system by means of a computer having a processor of speed of 3.40 GHz.
To define a specific LEP method, we assume that the radius t of the neighborhood of the median filter in the equation (7) is equal to, and we experimentally choose the detection function in (2). In particular, the tested functions are
Observe that the detection functions ϕ_{j}, j = 1,...,6 are decreasing and continuous. Moreover, we get ϕ_{j}(0) = 2 and .
In Table 1 there are the errors of the LEP algorithm in terms of Mean Square Error MSE, also [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ] in reconstructing the images of the Kodak set as the detection function varies. The values in bold are related to the best reconstruction of a specific image. In the last line there are the means of the MSE obtained in the reconstruction of the Kodak sample images, as the detection function varies. Note that the best result can be obtained by different detection functions, but, if one takes the means, then the minimal error corresponds to the detection function ϕ_{4}. To evaluate whether the function ϕ_{4} is actually the best detection function, we proceed as follows. For each sample image we give five points to the detection function which allows to obtain an estimate with the minimal error; four points to the detection function which obtain the second best minimal error; three points in correspondence with the third minimal error, and so on.
Fig. (5) Kodak image set. |
In Table 2 there are the results obtained by the all detection functions on the single images, and in the last line there is the global score. Observe that, even in this case, the highest score is obtained by the detection function ϕ_{4}.
From now on, we use the detection function ϕ_{4} in the LEP algorithm. In Fig. (6a) the reconstruction of Image 02 is shown. If we evaluate the results visually, it is very difficult to perceive the errors present during the reconstruction. Thus, in Fig. (6b) we present the image of errors, which is given by
Fig. (6) LEP reconstruction of Figure 02. |
where is the estimate obtained by the LEP algorithm, x is the ideal image and e is the column vector belonging to , whose entries are equal to one. Again, it is difficult to note visually the errors of the algorithm. So, in Fig. (6c) we show the image of the enlarged errors, that is
where it is possible to see in detail the errors of the algorithm.
Since most algorithms existing in the literature do not allow to see easily the errors related to the reconstructions, because they seem to be perfect, then, to compare our algorithm with some of those proposed in the literature, we use the table of the errors in the reconstruction of the Kodak dataset. In Tables 3 and 4, we compare the LEP method with the original Freeman filter [10W. T. Freeman, "Median filter for reconstructing missing color samples", Patent US. 4774565, .] and with some other recently published algorithms [8A. Buades, B. Coll, J.M. Morel, and C. Sbert, "Self-similarity driven color demosaicking", IEEE Trans. Image Process., vol. 18, no. 6, pp. 1192-1202.[http://dx.doi.org/10.1109/TIP.2009.2017171] [PMID: 19403366] , 12S.C. Pei, and I.K. Tam, "Effective color interpolation in CCD color filter arrays using signal correlation", IEEE Trans. Circ. Syst. Video Tech., vol. 13, pp. 503-513.[http://dx.doi.org/10.1109/TCSVT.2003.813422] , 19B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, "Color plane interpolation using alternating projections", IEEE Trans. Image Process., vol. 11, no. 9, pp. 997-1013.[http://dx.doi.org/10.1109/TIP.2002.801121] [PMID: 18249722] , 22D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, "Beyond color difference: Residual interpolation for color image demosaicking", IEEE Trans. Image Process., vol. 25, no. 3, pp. 1288-1300.[PMID: 26780794] , 26S. Ferradans, M. Bertalmío, and V. Caselles, "Geometry-based demosaicking", IEEE Trans. Image Process., vol. 18, no. 3, pp. 665-670.[http://dx.doi.org/10.1109/TIP.2008.2010204] [PMID: 19188122] -33X. Wu, and N. Zhang, "Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)", IEEE Trans. Image Process., vol. 13, no. 9, pp. 1263-1274.[http://dx.doi.org/10.1109/TIP.2004.832920] [PMID: 15449587] ]. Although the proposed algorithm gives the best reconstruction of two images, the total mean of the errors obtained with the LEP algorithm is the smallest of the selected methods.
In the literature there exist several other algorithms, for instance that proposed [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ], which is one of the best performed algorithms, obtaining a MSE mean equal to 6.11. However, in order to reach this goal, the needed mean computation time is equal to 27 minutes and 4 seconds, while the mean computation time for the LEP algorithm is equal to 0.16 seconds. The aim of the LEP algorithm is to obtain good results in a very short period of time. This method can be used as an initialization algorithm for the technique proposed [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ], obtaining meaningful reductions of its computational cost.
In Fig. (7a) the reconstruction of Image 08 by LEP is presented. Its MSE, between the original Image 08 is about 19.99, obtained in a computational time of 0.16 seconds. The relative enlarged error image is presented in Fig. (7b). In Fig. (7c) the reconstruction of Image 08 by the algorithm proposed [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ] is illustrated. Its MSE between the original Image 08 is about 12.33, obtained in a computational time of 27 minutes and 54.74 seconds. The relative enlarged error image is given in Fig. (7d). From the enlarged error images it is possible to notice how the algorithm proposed specially refines the reconstruction of the buildings on the left part of the image [20I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] ], however it does not allow an immediate processing of the image.
Fig. (7) Reconstruction of Image 08. |
We investigated the demosaicing problem and proposed a heuristic technique, in order to obtain a very fast algorithm. In particular, we proposed an algorithm consisting of three steps. In the first one, the green channel was updated by means of an edge-directed and weighted average technique. In the second one, the red and blue channels were updated, by using also the constraint of equality of the second derivatives in the various channels. In the third step, we proposed an iterative algorithm, assuming the constant color difference. this choice allowed to obtain accurate reconstructions in very short calculation times. Moreover, similarly as in the Freeman technique, in this phase we employed a median filter. We fixed a maximum number of iterative steps as four, in order to obtain low computational costs. We called our algorithm Local Edge Preserving (LEP).
The choice to impose an edge-preserving reconstruction, to impose a correlation between the channels on the derivatives of the second order, and to impose that the difference between the channels is constant, was the result of an extensive experimentation. Such an experimentation showed how this choice allowed to obtain accurate reconstructions in very short calculation times. Thus the proposed technique turns out to be very competitive when compared with some other methods known in the literature also [8A. Buades, B. Coll, J.M. Morel, and C. Sbert, "Self-similarity driven color demosaicking", IEEE Trans. Image Process., vol. 18, no. 6, pp. 1192-1202.[http://dx.doi.org/10.1109/TIP.2009.2017171] [PMID: 19403366] , 12S.C. Pei, and I.K. Tam, "Effective color interpolation in CCD color filter arrays using signal correlation", IEEE Trans. Circ. Syst. Video Tech., vol. 13, pp. 503-513.[http://dx.doi.org/10.1109/TCSVT.2003.813422] , 19B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, "Color plane interpolation using alternating projections", IEEE Trans. Image Process., vol. 11, no. 9, pp. 997-1013.[http://dx.doi.org/10.1109/TIP.2002.801121] [PMID: 18249722] , 22D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, "Beyond color difference: Residual interpolation for color image demosaicking", IEEE Trans. Image Process., vol. 25, no. 3, pp. 1288-1300.[PMID: 26780794] , 26S. Ferradans, M. Bertalmío, and V. Caselles, "Geometry-based demosaicking", IEEE Trans. Image Process., vol. 18, no. 3, pp. 665-670.[http://dx.doi.org/10.1109/TIP.2008.2010204] [PMID: 19188122] -33X. Wu, and N. Zhang, "Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)", IEEE Trans. Image Process., vol. 13, no. 9, pp. 1263-1274.[http://dx.doi.org/10.1109/TIP.2004.832920] [PMID: 15449587] ].
Not applicable.
The authors declare no conflict of interest, financial or otherwise.
The first author was partially supported by the G. N. A. M. P. A. (the Italian National Group of Mathematical Analysis, Probability and Applications). The second and the third author were partially supported by the G. N. C. S. (the Italian National Group for the Scientific Computations).
This research was partially supported by the projects “Ricerca di Base 2017” (Metodi di Teoria dell'Approssimazione e di Analisi Reale per problemi di approssimazione ed applicazioni) and “Ricerca di Base 2018” (Metodi di Teoria dell'Approssimazione, Analisi Reale, Analisi Nonlineare e loro applicazioni).
[1] | C. Bai, Z. Lin, J. Yu, Y.W. Chen, and Y.W. Chen, "Penrose demosaicking", IEEE Trans. Image Process., vol. 24, no. 5, pp. 1672-1684.[http://dx.doi.org/10.1109/TIP.2015.2409569] [PMID: 25769160] |
[2] | J.F. Hamilton, and J.T. Compton, " Processing color and panchromatic pixels", Patent US 0024879, . |
[3] | K. Hirakawa, and P. Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity", Image Processing (ICIP), pp. 81-4.[http://dx.doi.org/10.1109/ICIP.2007.4379097] |
[4] | BE Bayer, " Color imaging array. ", Patent US. 3971065, . |
[5] | R.H. Hibbard, "Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients", Patent US. 5382976, . |
[6] | CA Laroche, and MA Prescott, "Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients", Patent US. 537332, . |
[7] | R. Kakarala, and Z. Baharav, "Adaptive demosaicking with the principal vector method", IEEE Trans. Consum. Electron., vol. 48, pp. 932-937.[http://dx.doi.org/10.1109/TCE.2003.1196423] |
[8] | A. Buades, B. Coll, J.M. Morel, and C. Sbert, "Self-similarity driven color demosaicking", IEEE Trans. Image Process., vol. 18, no. 6, pp. 1192-1202.[http://dx.doi.org/10.1109/TIP.2009.2017171] [PMID: 19403366] |
[9] | J.E. Adams, "Interactions between color plane interpolation and other image processing functions in electronic photography", Proc. SPIE, vol. 2416, pp. 144-151.[http://dx.doi.org/10.1117/12.204825] |
[10] | W. T. Freeman, "Median filter for reconstructing missing color samples", Patent US. 4774565, . |
[11] | R. Kimmel, "Demosaicing: image reconstruction from color CCD samples", IEEE Trans. Image Process., vol. 8, no. 9, pp. 1221-1228.[http://dx.doi.org/10.1109/83.784434] [PMID: 18267539] |
[12] | S.C. Pei, and I.K. Tam, "Effective color interpolation in CCD color filter arrays using signal correlation", IEEE Trans. Circ. Syst. Video Tech., vol. 13, pp. 503-513.[http://dx.doi.org/10.1109/TCSVT.2003.813422] |
[13] | B. Tao, I. Tastl, T. Cooper, M. Blasgen, and E. Edwards, "Demosaicing using human visual properties and wavelet interpolation filtering", Proceedings of Color Imaging Conference: Color Science, Systems, Applications, pp. 252-6. |
[14] | J.A. Weldy, "Optimized design for a single-sensor color electronic camera system", Proc. SPIE, vol. 1071, pp. 300-307.[http://dx.doi.org/10.1117/12.952531] |
[15] | W. Lu, and Y.P. Tan, "Color filter array demosaicking: New method and performance measures", IEEE Trans. Image Process., vol. 12, no. 10, pp. 1194-1210.[http://dx.doi.org/10.1109/TIP.2003.816004] [PMID: 18237887] |
[16] | X. Wu, W.K. Choi, and P. Bao, "Color restoration from digital camera data by pattern matching", Proc. SPIE, vol. 3018, pp. 12-17.[http://dx.doi.org/10.1117/12.271585] |
[17] | P.S. Tsai, T. Acharya, and A.K. Ray, "Adaptive fuzzy color interpolation", J. Electron. Imaging, vol. 11, pp. 293-305.[http://dx.doi.org/10.1117/1.1479702] |
[18] | J.E. Adams, and J.F. Hamilton, "Design of practical color filter array interpolation algorithms for digital cameras", Proc. SPIE, vol. 3028, pp. 117-125.[http://dx.doi.org/10.1117/12.270338] |
[19] | B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, "Color plane interpolation using alternating projections", IEEE Trans. Image Process., vol. 11, no. 9, pp. 997-1013.[http://dx.doi.org/10.1109/TIP.2002.801121] [PMID: 18249722] |
[20] | I. Gerace, F. Martinelli, and A. Tonazzini, "Demosaicing of noisy color images through edge-preserving regularization", Proceeding of 2014 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), pp. 1-5.[http://dx.doi.org/10.1109/IWCIM.2014.7008795] |
[21] | D. Menon, and G. Calvagno, "Regularization approaches to demosaicking", IEEE Trans. Image Process., vol. 18, no. 10, pp. 2209-2220.[http://dx.doi.org/10.1109/TIP.2009.2025092] [PMID: 19527958] |
[22] | D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, "Beyond color difference: Residual interpolation for color image demosaicking", IEEE Trans. Image Process., vol. 25, no. 3, pp. 1288-1300.[PMID: 26780794] |
[23] | J. Hadamard, Lectures on cauchy’s Problem in linear partial differential equations. Yale., University Press: New Haven, . |
[24] | "Kodak lossless true color image suiter", Available on: http:// r0k. us/ graphics/ kodak/ |
[25] | O. Hosam, "Side-informed image watermarking scheme based on dither modulation in the frequency domain", Open Signal Process. J., vol. 5, pp. 1-6.[http://dx.doi.org/10.2174/1876825301305010001] |
[26] | S. Ferradans, M. Bertalmío, and V. Caselles, "Geometry-based demosaicking", IEEE Trans. Image Process., vol. 18, no. 3, pp. 665-670.[http://dx.doi.org/10.1109/TIP.2008.2010204] [PMID: 19188122] |
[27] | JF Hamilton, and JE Adams, "Adaptive color plane interpolation in single sensor color electronic camera. ", Patent US. 5629734, . |
[28] | K. Hirakawa, and T.W. Parks, "Adaptive homogeneity-directed demosaicing algorithm", IEEE Trans. Image Process., vol. 14, no. 3, pp. 360-369.[http://dx.doi.org/10.1109/TIP.2004.838691] [PMID: 15762333] |
[29] | X. Li, "Demosaicing by successive approximation", IEEE Trans. Image Process., vol. 14, no. 3, pp. 370-379.[http://dx.doi.org/10.1109/TIP.2004.840683] [PMID: 15762334] |
[30] | R. Lukac, and K.N. Plataniotis, "Data-adaptive filters for demosaicing: A framework", IEEE Trans. Consum. Electron., vol. 51, pp. 560-570.[http://dx.doi.org/10.1109/TCE.2005.1468002] |
[31] | D.D. Muresan, and T.W. Parks, "Demosaicing using optimal recovery", IEEE Trans. Image Process., vol. 14, no. 2, pp. 267-278.[http://dx.doi.org/10.1109/TIP.2004.838697] [PMID: 15700531] |
[32] | T. Sakamoto, C. Nakanishi, and T. Hase, "Software pixel interpolation for digital style camera suitable for a 32-bit MCU", IEEE Trans. Consum. Electron., vol. 44, pp. 1342-1352.[http://dx.doi.org/10.1109/30.735836] |
[33] | X. Wu, and N. Zhang, "Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)", IEEE Trans. Image Process., vol. 13, no. 9, pp. 1263-1274.[http://dx.doi.org/10.1109/TIP.2004.832920] [PMID: 15449587] |
[34] | A.A. Moghadam, M. Aghagolzadeh, M. Kumar, and H. Radha, "Compressive framework for demosaicing of natural images", IEEE Trans. Image Process., vol. 22, no. 6, pp. 2356-2371.[http://dx.doi.org/10.1109/TIP.2013.2244215] [PMID: 23380854] |