Beruflich Dokumente
Kultur Dokumente
General Description
Most digital cameras capture imagery using a single sensor overlaid with a color filter array (CFA). Today the most commonly used CFA pattern is Bayer pattern. Thus in the captured CFA image, each pixel contains only one of the three primary colors. This Color Filter Array Interpolation core interpolates the missing color components for every pixel. The Color Filter Array Interpolation core produces the high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation methods. blurring and the generation of false color also called color aliasing. Several algorithms have been developed to improve these artifacts. A variety of simple interpolation methods, such as Pixel Replication, Nearest Neighbor Interpolation, Bilinear Interpolation and Bi-cubic Interpolation have been widely used for CFA demosaicing. But these simple algorithms produce low quality images. More complicated algorithms like the edge-directed interpolation have generated better quality image than simple interpolation methods. But these algorithms still generate the artifacts. Some algorithms have been developed to improve these problems. These algorithms often require huge computation power, so it is impossible to be implemented in real time system. Our Color Filter Array Interpolation core produces high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation method. The Color Filter Array Interpolation core uses 5*5 pixels. A Bayer CFA image consists of 50% green, 25% red, and 25% blue samples. The G pixels interpolation is more important in generating high quality images than the others signal. High quality R or B signals are interpolated using these G signals. More computations are used to interpolate the G signals than R or B signals. Prior to interpolation of R or B, interpolation of G is performed first. Interpolation of the G signal is performed by considering the signal level of G-R or G-B and the change of signal edges. And G-R or G-B signals are interpolated by a simple operation. A total of seven line memories are used to perform the Color Filter Array Interpolation. Although relatively a small number of line memories are used, interpolated image of high quality is achieved.
Features
RGB and CMY Bayer image sensor support 5x5 pixels interpolation High quality interpolation 8, 10, and, 12 bit input and output precision Support for Xilinx, Altera FPGA and ASIC
Applications
Video Surveillance Industrial Imaging Machine Vision Digital Still Camera Mobile Phone Camera
Overview
Today, most consumer digital cameras capture an image with a single image sensor chip to minimize cost and size of the camera. Today the most commonly used CFA pattern is Bayer pattern, a schematic of which is shown in Fig. 1. To retain color information, a color filter array (CFA) is placed before the image sensor. As a result, there is only one color (red, green or blue) available at each spatial location. But for viewing, editing and printing, three colors per pixel (red, green, and blue) are necessary. The Color Filter Array Interpolation or Demosaicing refers to the algorithms that allow recreating a three-color per pixel image from a one color per pixel image. The simplest way for demosaicing is the bilinear interpolation. But the interpolated image shows artifacts inherent to demosaicing:
Description
lid
IN
pid
IN
Fig.2. Active pixel area and optical black area in Image Sensor
Block diagram
Fig.3 is the block diagram of the Color Filter Array Interpolation core. Prior to interpolation of R or B, interpolation G of is performed first. Interpolation of the G signal is performed by considering the signal level of G-R or G-B and the change of signal edges. And G-R or G-B signals are interpolated by a simple operation. Red, Green and Blue signals are produced by subtracting the G signals from the interpolated G-R/G-B signals.
clock, reset act_sync lid, pid
4 line memory
G Interpolation
G-R/G-B
2 line memory
1 line Memory
Timing diagram
Fig.4 shows an example of the Active pixel area and optical black area in VGA size image sensor. The horizontal size of the active pixel area is 659 and the vertical size of the active pixel area is 494. Fig.5a shows the timing of act_sync. Fig.5b shows the timing of act_sync_out. The Color Filter Array Interpolation core uses 7 line memories. The 6 line memories of all the 7 line memories made the time delays of 6 horizontal syncs. One other line memory is just used for delay matching. In addition, because the interpolation is processed using the 6 line memory, the invalid data in the duration of 6 horizontal syncs is produced. The valid data starts from the 7th horizontal line. The valid data in the case using VGA sensor of Fig.4 is from input 4th to 491th of the original image. Fig.5c shows the detailed timing of act_sync in one horizontal sync period and Fig.5d shows the detailed timing of act_sync_out in one horizontal sync period. And some timing delay and invalid data is generated by the interpolation in horizontal direction and pipeline processing for reducing timing delay. Invalid data in the duration of 6 pixels is produced. The 3 pixels from the first pixel in horizontal sync of act_sync_out have invalid data and the 3 pixels from the last pixel in horizontal sync of act_sync_out have invalid data. So the valid data starts from the 4th pixel and ends to the last pixel number-3(656th pixel number) in the case using VGA sensor of Fig.4. Total delays in horizontal direction have 18 clock delays including the interpolation processing delay and the pipeline processing delay.
659
494
Fig.4. Active pixel area and optical black area in VGA size Image Sensor
(a)
490
491
492
493
494
(b)
489
490
491
Valid data
(c)
6 5 9
6 5 6
6 5 7
6 5 8
6 5 9
(d)
6 5 4
6 5 5
6 5 6
6 5 3
6 5 4
6 5 5
6 5 6
18 clock delay
Valid data
Fig.5a. The act_sync timing diagram Fig.5b. The act_sync_out timing diagram Fig.5c. The detailed act_sync timing diagram in one horizontal sync period Fig.5d. The detailed act_sync_out timing diagram in one horizontal sync period Fig.6 shows the timing relationship between lid, pid, act_sync signal and Bayer CFA pixels.
When lid and pid signal level is 0, Red pixel should be input into the datain pin. And when lid signal level is 0 and pid signal level is 1, Green pixel should be input into the datain pin. When lid signal level is 1 and pid signal level is 0, Green pixel should be input into the datain pin. When lid signal level is 1 and pid signal level is 1, Blue pixel should be input into the datain pin. The lid signal should be changed in the 0 level period of act_sync.
act_sync
lid
pid
R G R G R G R G R G
G B G B G B G B G
R G R
Fig.6. Timing Relationship between lid, pid, act_sync signal and Bayer CFA pixels
Image Qualities
Fig. 7 shows our test images. Table 2 shows the performance comparison of Bilinear Interpolation and our Interpolation core. The Color Filter Array Interpolation core produces the high quality images that the non-real time software interpolation methods can achieve. Despite of the high quality image, the smaller size logic gates are used than other interpolation methods. Fig.9. shows the original images and the images obtained by bilinear interpolation and our proposed interpolation of a cropped region from test images
Fig.7. Test images (referred to as Image 1 to Image 24, enumerated from left-to-right and top-to-bottom). source: http://r0k.us/graphics/kodak/ Table 2: Performance ComparisonPSNR results (in dB) of the red, green and blue color planes are listed in the 1st, 2nd and 3rd rows of each image.
Image 1 Bilinear 25.37 29.55 25.29 32.49 36.28 32.32 33.91 37.06 33.48 32.96 36.51 32.71 26.06 29.14 25.66 27.00 30.98 26.69 32.61 36.18 32.58 22.64 27.27 22.42 31.25 35.57 31.43 31.17 35.30 31.80 28.39 32.10 28.10 32.45 36.66 32.39 Proposed 34.78 36.50 34.96 40.18 41.94 40.59 39.92 43.70 41.99 40.83 42.09 38.29 34.87 37.73 36.92 35.01 37.52 35.74 40.28 43.78 42.94 32.49 35.11 33.28 41.41 42.88 40.40 40.85 43.24 40.47 36.97 38.70 37.37 40.38 43.50 41.34 Image 13 Bilinear 23.05 26.35 23.01 28.63 31.94 28.24 32.41 35.62 32.16 30.35 34.58 30.21 30.84 34.43 31.52 26.94 30.66 27.44 26.89 31.68 26.74 30.76 34.33 30.77 27.57 31.49 27.60 29.24 33.26 29.81 33.89 37.86 34.20 25.42 29.47 26.46 29.26 33.09 29.29 Proposed 30.54 32.41 31.38 34.99 38.62 37.50 38.82 40.71 38.97 38.35 40.89 38.79 39.14 40.83 39.85 35.08 36.60 35.15 38.27 39.83 37.66 37.97 41.62 40.68 35.68 38.15 36.86 36.25 39.30 37.69 40.92 44.21 43.50 31.72 35.18 34.25 37.32 39.79 38.19
14
15
16
17
18
19
20
21
10
22
11
23
12
24
Average
(Original)
(Bilinear Interpolation)
Fig.9. The original images and the images obtained by bilinear interpolation and our proposed interpolation of a cropped region from test images
Input Width
FFs
LUTs
Slices
Xtreme DSPs
block RAMs
8 8 10 10 12 12 8 8 10 10 12 12
1003 2487 1034 3437 1155 4463 1008 2492 1039 3442 1160 4468
2609 2076 4132 2837 4929 3667 2608 2079 4133 2840 4932 3670
1717 2104 2573 2860 3074 3623 1718 2109 2575 2864 3076 3627
4 4 4 4 4 4 4 4 4 4 4 4
7 7 7 7 7 7 7 7 14 14 14 14
http://asicfpga.com/site_upgrade/asicfpga/pds/isp_pds_files/CFA_int_768by512.zip
You can also download core samples for other image size. But we provide only structural verilog code and edif. You need to modify the sample files (test bench and other source files in CFA_int_768by512.zip for simulation. You can evaluate this core for an interpolation of under Max. 2048 horizontal pixel size and 8/10/12 bit input data width. http://asicfpga.com/site_upgrade/asicfpga/pds/isp_pds_files/CFA_int_2048_12bit.zip
These core samples that we provide can just be used for evaluation. These cores are the time limited version. So these cores are operated just for one hour at 10 MHz. These cores are optimized by logic size rather than logic speed. So if you want to use more high speed core or special image size core, please contact us. We can make easily the requested core by our interpolation core compiler. So we can send this core to you within 2 business days.
Contact
If you have a question and request, please email it to info@asicfpga.com