1

IMAGE FUSION ALGORITHMS FOR HDR IMAGING

DINESH R ATHUL S MANU ELDHO JITHIN KARUNAKARAN ARJUN M

What is HDR?
High Dynamic Range Can store large variation of intensity in an HDR image. Photographs captured using ordinary cameras are SDR images. Low range of intensity variation can be depicted.

SDR Standard Dynamic Range

Why HDR?
Clouds are visible.
But the rest of the part is dark.

Why HDR?
Cloud is washed out.
So an SDR image cant include all the details in a scene. Human eyes can perceive 10-3 to 10+5 cd/m2. An SDR images dynamic range is about 102.

How to make an HDR image?

SDR Image SDR 1 Image1 SDR Image .2 . . . . . . SDR Image m

Combine

HDR Image

How to display HDR image

Normal devices cannot display luminance with high dynamic range.
So an HDR image has to be remapped to an SDR image. This is called tone mapping.

Tone mapping should ensure that the maximum details in the HDR image is made available in the resultant SDR image.

How to display HDR image

Scene

Exposure SDR Image 1 1

Exposure 2

Exposure m

SDR . Image .2 . . . . . SDR Image m

Combi ne

HDR Image

Tone Mapping

Final viewed image

How to combine?
Using Camera response function
Gradient method Probabilistic Exposure method

Using Camera Response Function

Scene
Exposure time 1

SDR Image 1 Luminance value * Exposure Time = Exposure

Pixel value in SDR Image is a function of exposure at that pixel. The function is called the Camera Response Function (CRF). CRF is obtained from SDR images at different exposures.
SDR Image 1

10

Mimicked HDR

So another idea is to generate an SDR image directly from the input images. This SDR image is constructed in such a way that the maximum information is contained it. Also called mimicked HDR image.

11

Probabilistic Exposure Fusion [1]

Two stages: 1. Scene detail extraction

2. SDR image synthesis

Scene Detail Extraction

Visible Contrast Visible Gradient

13

Finding Visible Contrast

Consider a small square sized area within the image of size (m)*(m) with pixel (i,j) as its centre.
Find the mean of the luminance of the pixels in that square. Contrast of pixel (i,j) is the absolute difference between the luminance of the pixel (i,j) and the mean of the luminance of the pixels in the square. To have a better human like perception we multiply the contrast with Weber coefficients. If the resultant contrast is greater than the predefined threshold then the visible contrast of the pixel (i,j) is updated with this value. Else it is considered to be zero.

14

Visible Contrast
Visible contrast for each pixel.
Consider a window with pixel (i,j) as centre. Visible difference between magnitude of pixel and average luminance of window.

15

Finding visible contrast

v(i,j)=T(c(IH,mean(i,j))IH(i,j)) c(IH,mean(i,j))IH(i,j) v(i,j) : Visible contrast at pixel (i,j) T(x)=1, x>=0.2 and T(x)=0, x<0.2 c( ) : Weber coefficients IH,mean(i,j) : Luminance of pixel (i,j) IH(i,j) : magnitude of (IH(i,j) - IH,mean(i,j))

16

Finding luminance uI(i,j)

We have to find the visible contrast for the same pixel in all the images.
Find the maximum visible contrast for the pixel (i,j). Update uI(i,j) with the luminance of the pixel that gives the maximum visible contrast. Also update the chrominance of our resultant image with the chrominance of the pixels giving maximum visible contrast.

17

Visible Contrast

Image with uI as the luminance

18

Using uI(i,j) as the luminance we can synthesize an SDR image.

Image will have luminance which give maximum visible contrast. But there will be halos in the image due to gradient inconsistency.

19

Sample images

20
Image 1 Image 2 Image 3

SDR image synthesized from uI(i,j)

21

22

Visible Gradient
For all pixels (i,j) in the window

Gradients in x & y direction

VMAX = max(x_grad,y_grad) If VMAX > Threshold VISIBLE_GRAD = VISIBLE_GRAD + VMAX

VISIBLE_GRAD is the visible gradient for (i,j).

23

Visible Gradient

Fig. Visible Gradient Extraction [1]

24

Scene Gradient
For all pixels (i,j) in the window

Find VISIBLE_GRAD for all images.

Find the maximum value. Gradient of the corresponding pixel is scene gradient.

25

SDR Image Synthesis

For a given scene IH, the corresponding SDR image IS should preserve the visible contrasts presented in IH and suppress the gradient reversal between IH and I S.
IS should be selected in such a way that it maximizes p(IS/ IH).

26

SDR Image Synthesis

Fig 2: Probability model [1]

27

Probability model

28

Image Synthesis

29

SDR Image Synthesis

Fig. Luminance evaluation [1]

Simplifying the above equation we have IS(x+1,y)+IS(x-1,y)+IS(x,y+1)+IS(x,y-1)(4+C2/S2)IS(x,y) = Gx(x,y) - Gx(x-1,y) + Gy(x,y) - Gy(x,y-1) C2/S2uI(x,y)

30

SDR Image Synthesis

Solving the equation for all pixels (x,y) in the image we can generate the luminance values of the resultant SDR image.

Sample images

31
Image 2 Image 3

Image 1

Output images Window size=10 (c/s)2=1 Window size=10 (c/s) 2=0.008

32
Window size=10 (c/s) 2=0.08

Output images

33
Window size=36 (c/s) 2=0.08 Window size=36 (c/s) 2=0.008 Window size=6 (c/s) 2=0.08

Gradient + Gaussian smoothing [2]

34

Divide the images into smaller segments Find the magnitude of gradient of each pixel in the segment in x and y directions. Find the maximum of the x gradient and y gradient for each pixel in the segment. Find the value for all pixels in the segment and add up. This would be the amount of detail MD in the segment of the particular input image.

35

Detail Extraction

Similarly find MD for the same segment in the other input images. One with the maximum detail is considered for the resultant image processing. Repeat this for all the segments. The image obtained just using this will have sharp transitions on the borders of the segments.

36

Gaussian Smoothing

So to avoid the sharp transition a 2 D Gaussian curve can be used. The 2 D Gaussian is used to smoothen the luminance of the pixels in a segment. Previously obtained luminance value at pixel (i,j) is multiplied with the Gaussian function value at the pixel to obtain the luminance of the resultant image.

37

Using Photomatix PRO [3]

Using Gradient method [2]

38

FUTURE WORK
We used gradient method and probabilistic exposure fusion technique to fuse SDR images to obtain HDR image.
One of the chief limitation of all these fusion technique is its requirement that the camera and scene be still between different exposures If there are movements between different exposures the HDR output image would be blurred with double images and sometimes in worst case multiple images.

39

So two main problem with fusion techniques are

Misalignment: camera motion that results in misalignment of images that cause the combined HDR image look blurry. Ghosting: moving objects in scene while capturing images ,will appear in different locations of the combined HDR image causing double images

40

41

42

IMAGE REGISTRATION
Its practically impossible to not have motion between different exposures.
Image registration for mitigating effects of ghosting due to misalignment and moving object in scene So far no algorithms can completely undo the effects of ghosting though MTB (Median Threshold Bitmap by Ward) method is an efficient solution We first going to implement MTB method and later hoping to come up with our own better algorithms.

43

Ghost mitigation techniques involves two process.

Ghost detection Ghost removal

44

Ghost detection techniques: based on motion detection in exposure sequence.

We can identify two types of motion in dynamic scene Moving objects in static background.eg moving cars and people Moving back ground with static or dynamic objects eg; windblown leaves or waves

METHODS FOR MOTION DETECTION

45

Variance based ghost detection

Entropy based ghost detection Prediction based ghost detection Pixel order relation Multilevel based ghost detection Bitmap based ghost detection

46

GHOST REMOVAL TECHNIQUES

Keeping a single occurrence of moving object:
In this method we are going to apply multiple exposure fusion in ghost free region while selecting single reference exposure in ghost affected area Removing all moving objects: It would be desirable to have no moving objects in the final HDR. Here the simple approach is to discard exposures that are affected by ghosting in each location.

47

48

49

Depending on users interest moving object could be either completely removed or kept at a fixed position
High contrast movements (moving object with different back ground)could be correctly detected Low contrast movements are difficult detect and remove.

50

We are hoping to come up with our own techniques for image fusion and image registration by next semester
We will code these algorithms in c language and produce a real product for HDR image production

51

References
[1] Mingli Song, Dacheng Tao, Chun Chen,
Jiajun Bu, JieboLuo and Chengqi Zhang, Probabilistic exposure fusion", in IEEE Transactions on Image Processing, vol. 21, no. 1, pp. 341-357, January 2012. [2] Andras Rovid, Annamaria R, Varkonyi-Koczy , Takeshi Hashimoto, Szilveszter Balogh, Yoshifumi Shimodaira, Gradient Based Synthesized Multiple Exposure Time HDR Image, in Instrumentation and Measurement Technology Conference Proceedings, pp. 1-6, 2007 [3] www.hdrsoft.com