Sie sind auf Seite 1von 9

Color Filter Array Interpolation

General Description
Most digital cameras capture imagery using a single sensor overlaid with a color filter array (CFA). Today the most commonly used CFA pattern is Bayer pattern. Thus in the captured CFA image, each pixel contains only one of the three primary colors. This Color Filter Array Interpolation core interpolates the missing color components for every pixel. The Color Filter Array Interpolation core produces the high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation methods. blurring and the generation of false color also called color aliasing. Several algorithms have been developed to improve these artifacts. A variety of simple interpolation methods, such as Pixel Replication, Nearest Neighbor Interpolation, Bilinear Interpolation and Bi-cubic Interpolation have been widely used for CFA demosaicing. But these simple algorithms produce low quality images. More complicated algorithms like the edge-directed interpolation have generated better quality image than simple interpolation methods. But these algorithms still generate the artifacts. Some algorithms have been developed to improve these problems. These algorithms often require huge computation power, so it is impossible to be implemented in real time system. Our Color Filter Array Interpolation core produces high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation method. The Color Filter Array Interpolation core uses 5*5 pixels. A Bayer CFA image consists of 50% green, 25% red, and 25% blue samples. The G pixels interpolation is more important in generating high quality images than the others signal. High quality R or B signals are interpolated using these G signals. More computations are used to interpolate the G signals than R or B signals. Prior to interpolation of R or B, interpolation of G is performed first. Interpolation of the G signal is performed by considering the signal level of G-R or G-B and the change of signal edges. And G-R or G-B signals are interpolated by a simple operation. A total of seven line memories are used to perform the Color Filter Array Interpolation. Although relatively a small number of line memories are used, interpolated image of high quality is achieved.

Features
RGB and CMY Bayer image sensor support 5x5 pixels interpolation High quality interpolation 8, 10, and, 12 bit input and output precision Support for Xilinx, Altera FPGA and ASIC

Applications
Video Surveillance Industrial Imaging Machine Vision Digital Still Camera Mobile Phone Camera

Overview
Today, most consumer digital cameras capture an image with a single image sensor chip to minimize cost and size of the camera. Today the most commonly used CFA pattern is Bayer pattern, a schematic of which is shown in Fig. 1. To retain color information, a color filter array (CFA) is placed before the image sensor. As a result, there is only one color (red, green or blue) available at each spatial location. But for viewing, editing and printing, three colors per pixel (red, green, and blue) are necessary. The Color Filter Array Interpolation or Demosaicing refers to the algorithms that allow recreating a three-color per pixel image from a one color per pixel image. The simplest way for demosaicing is the bilinear interpolation. But the interpolated image shows artifacts inherent to demosaicing:

Fig.1. Bayer CFA pattern

Color Filter Array Interpolation

Input and output signals


Table 1 contains the port descriptions of the Color CFA Interpolation core. Table 1. Port Descriptions Port Name clock reset act_sync Port Width 1 1 1 Direction
IN IN IN Clock signal Reset signal, active high Input active sync signal, high level signal during active pixel area, refer to Fig.2, Fig.3,Fig.4 and Fig.5 Line identification signal 0: R/G pixel line in Bayer CFA patterns 1: G/B pixel line in Bayer CFA patterns Pixel identification signal 0: R pixel position in R/G pixel line or G pixel position in G/B pixel line 1: G pixel position in R/G pixel line or B pixel position In G/B pixel line Bayer CFA data input signals Output active sync signal, high level signal during active pixel area of Red/Green/Blue signal output. Refer to Fig.2, Fig.3,Fig.4 and Fig.5 Red signals Green signals Blue signals

Description

lid

IN

pid

IN

datain act_sync_out Red Green Blue

WIDTH 1 WIDTH WIDTH WIDTH

IN OUT OUT OUT OUT

Active pixel area

Optical black area

Fig.2. Active pixel area and optical black area in Image Sensor

Color Filter Array Interpolation

Block diagram
Fig.3 is the block diagram of the Color Filter Array Interpolation core. Prior to interpolation of R or B, interpolation G of is performed first. Interpolation of the G signal is performed by considering the signal level of G-R or G-B and the change of signal edges. And G-R or G-B signals are interpolated by a simple operation. Red, Green and Blue signals are produced by subtracting the G signals from the interpolated G-R/G-B signals.
clock, reset act_sync lid, pid

to each block Delay matting act_sync_out

Bayer CFA datain

4 line memory

G Interpolation

G-R/G-B

2 line memory

G-R and G-B Interpolation

Red RGB Out Green Blue

1 line Memory

Fig.3. Color Filter Array Interpolation Block Diagram

Color Filter Array Interpolation

Timing diagram
Fig.4 shows an example of the Active pixel area and optical black area in VGA size image sensor. The horizontal size of the active pixel area is 659 and the vertical size of the active pixel area is 494. Fig.5a shows the timing of act_sync. Fig.5b shows the timing of act_sync_out. The Color Filter Array Interpolation core uses 7 line memories. The 6 line memories of all the 7 line memories made the time delays of 6 horizontal syncs. One other line memory is just used for delay matching. In addition, because the interpolation is processed using the 6 line memory, the invalid data in the duration of 6 horizontal syncs is produced. The valid data starts from the 7th horizontal line. The valid data in the case using VGA sensor of Fig.4 is from input 4th to 491th of the original image. Fig.5c shows the detailed timing of act_sync in one horizontal sync period and Fig.5d shows the detailed timing of act_sync_out in one horizontal sync period. And some timing delay and invalid data is generated by the interpolation in horizontal direction and pipeline processing for reducing timing delay. Invalid data in the duration of 6 pixels is produced. The 3 pixels from the first pixel in horizontal sync of act_sync_out have invalid data and the 3 pixels from the last pixel in horizontal sync of act_sync_out have invalid data. So the valid data starts from the 4th pixel and ends to the last pixel number-3(656th pixel number) in the case using VGA sensor of Fig.4. Total delays in horizontal direction have 18 clock delays including the interpolation processing delay and the pipeline processing delay.
659

active pixel area

494

optical black area

Fig.4. Active pixel area and optical black area in VGA size Image Sensor

(a)

490

491

492

493

494

(b)

489

490

491

6 lines invalid data

Valid data

(c)

6 5 9

6 5 6

6 5 7

6 5 8

6 5 9

(d)

6 5 4

6 5 5

6 5 6

6 5 3

6 5 4

6 5 5

6 5 6

18 clock delay

3 pixels invalid data

Valid data

3 pixels invalid data

Fig.5a. The act_sync timing diagram Fig.5b. The act_sync_out timing diagram Fig.5c. The detailed act_sync timing diagram in one horizontal sync period Fig.5d. The detailed act_sync_out timing diagram in one horizontal sync period Fig.6 shows the timing relationship between lid, pid, act_sync signal and Bayer CFA pixels.

Color Filter Array Interpolation

When lid and pid signal level is 0, Red pixel should be input into the datain pin. And when lid signal level is 0 and pid signal level is 1, Green pixel should be input into the datain pin. When lid signal level is 1 and pid signal level is 0, Green pixel should be input into the datain pin. When lid signal level is 1 and pid signal level is 1, Blue pixel should be input into the datain pin. The lid signal should be changed in the 0 level period of act_sync.

act_sync

lid

pid

R G R G R G R G R G

G B G B G B G B G

R G R

Fig.6. Timing Relationship between lid, pid, act_sync signal and Bayer CFA pixels

Color Filter Array Interpolation

Image Qualities
Fig. 7 shows our test images. Table 2 shows the performance comparison of Bilinear Interpolation and our Interpolation core. The Color Filter Array Interpolation core produces the high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation methods. Fig.9. shows the original images and the images obtained by bilinear interpolation and our proposed interpolation of a cropped region from test images

Fig.7. Test images (referred to as Image 1 to Image 24, enumerated from left-to-right and top-to-bottom). source: http://r0k.us/graphics/kodak/ Table 2: Performance ComparisonPSNR results (in dB) of the red, green and blue color planes are listed in the 1st, 2nd and 3rd rows of each image.
Image 1 Bilinear 25.37 29.55 25.29 32.49 36.28 32.32 33.91 37.06 33.48 32.96 36.51 32.71 26.06 29.14 25.66 27.00 30.98 26.69 32.61 36.18 32.58 22.64 27.27 22.42 31.25 35.57 31.43 31.17 35.30 31.80 28.39 32.10 28.10 32.45 36.66 32.39 Proposed 34.78 36.50 34.96 40.18 41.94 40.59 39.92 43.70 41.99 40.83 42.09 38.29 34.87 37.73 36.92 35.01 37.52 35.74 40.28 43.78 42.94 32.49 35.11 33.28 41.41 42.88 40.40 40.85 43.24 40.47 36.97 38.70 37.37 40.38 43.50 41.34 Image 13 Bilinear 23.05 26.35 23.01 28.63 31.94 28.24 32.41 35.62 32.16 30.35 34.58 30.21 30.84 34.43 31.52 26.94 30.66 27.44 26.89 31.68 26.74 30.76 34.33 30.77 27.57 31.49 27.60 29.24 33.26 29.81 33.89 37.86 34.20 25.42 29.47 26.46 29.26 33.09 29.29 Proposed 30.54 32.41 31.38 34.99 38.62 37.50 38.82 40.71 38.97 38.35 40.89 38.79 39.14 40.83 39.85 35.08 36.60 35.15 38.27 39.83 37.66 37.97 41.62 40.68 35.68 38.15 36.86 36.25 39.30 37.69 40.92 44.21 43.50 31.72 35.18 34.25 37.32 39.79 38.19

14

15

16

17

18

19

20

21

10

22

11

23

12

24

Average

Color Filter Array Interpolation

(Original)

(Bilinear Interpolation)

(Our proposed Interpolation)

Fig.9. The original images and the images obtained by bilinear interpolation and our proposed interpolation of a cropped region from test images

Color Filter Array Interpolation

FPGA Device Utilization and Core Performance


Table 2: Resource Utilization and Target Speed for Virtex-4 xc4vlx15-12sf363 Maximum Number of horizontal size
1024 1024 1024 1024 1024 1024 2048 2048 2048 2048 2048 2048

Input Width

Logic Optimization type


Area Speed Area Speed Area Speed Area Speed Area Speed Area Speed

Maximum Clock Frequency (MHz)


106 129 87 126 71 125 105 126 89 127 72 127

FFs

LUTs

Slices

Xtreme DSPs

block RAMs

8 8 10 10 12 12 8 8 10 10 12 12

1003 2487 1034 3437 1155 4463 1008 2492 1039 3442 1160 4468

2609 2076 4132 2837 4929 3667 2608 2079 4133 2840 4932 3670

1717 2104 2573 2860 3074 3623 1718 2109 2575 2864 3076 3627

4 4 4 4 4 4 4 4 4 4 4 4

7 7 7 7 7 7 7 7 14 14 14 14

Free Core samples for simulation and Xilinx FPGA implementation


We provide the structural verilog code of core and test bench file for simulation and edif file for FPGA implementation. You can download the following file. Please first read Reame.txt. This core can be also used for an interpolation of Max 1024 horizontal pixel size and 8bit input data width.

http://asicfpga.com/site_upgrade/asicfpga/pds/isp_pds_files/CFA_int_768by512.zip

You can also download core samples for other image size. But we provide only structural verilog code and edif. You need to modify the sample files (test bench and other source files in CFA_int_768by512.zip for simulation. You can evaluate this core for an interpolation of under Max. 2048 horizontal pixel size and 8/10/12 bit input data width. http://asicfpga.com/site_upgrade/asicfpga/pds/isp_pds_files/CFA_int_2048_12bit.zip

These core samples that we provide can just be used for evaluation. These cores are the time limited version. So these cores are operated just for one hour at 10 MHz. These cores are optimized by logic size rather than logic speed. So if you want to use more high speed core or special image size core, please contact us. We can make easily the requested core by our interpolation core compiler. So we can send this core to you within 2 business days.

Color Filter Array Interpolation

Free Core samples for simulation and Other FPGA implementation


The core sample for ALTERA will be prepared as soon as possible. If you need the core sample for other FPGA, Please contact us.

The CFA Interpolation software for window OS


We provide our interpolation software for window OS. You can use this software to test the performance of our interpolation algorithm before you evaluate the core samples for simulation and FPGA implementation. Our interpolation window software is included in CFA_int_768by512.zip. This program uses a bitmap file. This program is very simple. So you could easily use this program without especially learning how to use.

Gate Level License version


You can get the following services when you purchase the core after evaluation. Gate level netlist for FPGA/ASIC Functional simulation support Full functionality in with no time outs

RTL source Level License version


You can get the following services when you purchase the core after evaluation. Verilog RTL source files Functional simulation support Full functionality in with no time outs

Contact
If you have a question and request, please email it to info@asicfpga.com

Das könnte Ihnen auch gefallen