Hardware Implementation of Demosaicing Algorithm

This week I was assigned to draw the schematic for the demosaicing algorithm. The algorithm is separated into two parts, which is bilinear interpolation in the boundary of image, and PPG interpolation for the rest of image pixels.

The Overview of Demosaicing Schematic

The hardware implementation of the demosaicing requires the storing of pixels information in buffers (RAM), processing, and finally output in RGB form. As I was finding the way to store the buffer, I found that there is actually an existing CFA demosaicing IP core from Xilinx, but they did not specify the demosaicing algorithm or method. Nevertheless, it requires purchase and will only provide the core in encrypted RTL form.. However, this IP core documentation do helps me a lot in understanding what and why the core is designed in such way, and the latency of current IP core. After reading the Xilinx core documentation on the usage of the core, I made a draft on the flow of the schematic.

1. Buffer : Storing the ADC output (12/14/16-bits) into BRAM blocks. At least 5 lines of Image would need to be stored into the BRAM before starting of any interpolation. 6 lines of BRAM would be needed so that the interpolation and the storing process would not be interfered.
2. Green Pixel Interpolation : Interpolate the green pixels and store 5 lines of green pixels into BRAM blocks.
3. Blue & Red pixel Interpolation at Green pixel : Interpolate the missing blue and red pixels at green pixel. (Require neighbor green pixels value)
4. Blue/Red pixel Interpolation at Red/Blue pixel : Interpolate the missing blue and red pixels at red and blue pixels respectively. (Require neighbor green pixels value)
5. Output each RGB pixel in a fixed data rate

In the progress of drawing the schematic, I had read a lot of documentation from Xilinx. The hard part to draw the schematic is actually on the Block RAM (BRAM) and the different Bayer pattern. Previously, my supervisor had discussed with me about the storing of the pixels into BRAM, by shifting the BRAM contents into the next BRAM using Read/Write operation. Meanwhile, another was to implement the BRAM buffer storage using multiplexers. As I try to draw these schematics, I found out that any of these methods require a state machine to control the Read/Write operation of Data into the BRAM.

In order to make the Demosaicing module able to interpolate all 4 Bayer Pattern of RGGB, GRBG, GBRG and BGGR, the module should be provided the information of the first red pixel coordination, as in (0,1), (1,0), (0,1) and (1,1) for each Bayer pattern respectively. Currently, I am working on the schematic of the interpolation for all 4 Bayer pattern.

Hopefully by next week, I would be able to finish the full schematic of the whole demosaicing module.