Lections on Image analysis, OpenCV

7. Working with camera.

Background and motion analysis

http://people.rit.edu/andpph/photofile-misc/strobe-motion-ta-08.jpg

www.uralvision.blogspot.com perevalovds@gmail.com
USU / IMM Fall 2010
Working with the camera
Getting frames from the camera
VideoCapture capture; // A class to work with cameras and videos
capture.open (0); // Open the first chamber (numbered from 0)
// But if you specify a string VideoCapture capture.open ("myfile.avi"); - will read the avi-file

if (! capture.isOpened ()) { // If the camera does not complete the work

cout <<"no camera" <<endl;
return -1;
}

Mat image, smoothed, edges;

for (;;) {
capture>> image; // get the frame

cvtColor (image, smoothed, CV_BGR2GRAY);

GaussianBlur (smoothed, smoothed, Size (7,7), 1.5, 1.5);
Canny (smoothed, edges, 0, 30, 3);
imshow ("image", image);
imshow ("edges", edges);
if (waitKey (30)> = 0) break; // wait for keypress 30 ms, if pressed - exit
}

// Destructor capture himself off the camera

return 0;
Getting frames from the camera
Camera calibration
Consider a simple calibration when a camera looks at the side of the plane. (There are
more sophisticated calibration methods, which allow to eliminate the distortion at the
edges of images from the camera connected with the inaccuracy of optics).

Original image Image with the selected calibration Calibration results:

points image is aligned
(Angles of the red 4-gon)
Camera Calibration
1. To calibrate the need to specify the coordinates of several points on the image and
indicate to what point they should go.

Must be at least 4 points to search for perspective projection. In this case any point should
not lie on one line.

Note: the coordinates are real numbers, which improves the accuracy of the result.
You can use the sub-pixel methods that calculate the position of objects with an accuracy
higher than pixel.

2. Then, using the getPerspectiveTransform calculate the optimal matrix of perspective

transformation (a method of least squares type, which helps to use a lot of points taken
from the error).

3. Finally, this matrix is passed to the functionwarpPerspective for calibrated images.

Camera Calibration
const int K = 4, // we use 4 points.
cv:: Point2f src [K]; // coordinates of points on the image
cv:: Point2f dst [K]; // coordinates of the resulting points

src [0] = cv:: Point2f (..., ...); // coordinates of the corners of the image
src [1] = cv:: Point2f (..., ...); // should go clockwise from top left corner
src [2] = cv:: Point2f (..., ...); // - in accordance with dst, see below
src [3] = cv:: Point2f (..., ...);

dst [0] = cv:: Point2f (0, 0) // the resulting point; w and h- The size of the result
dst [1] = cv:: Point2f (w, 0);
dst [2] = cv:: Point2f (w, h);
dst [3] = cv:: Point2f (0, h);

Mat transform = getPerspectiveTransform(Src, dst); // compute the matrix transforms.

warpPerspective(Image, imageResult, transform, cv:: Size (w, h), INTER_LINEAR);

// Get from the input image image result - imageResult
Methods for analysis
of background
Methods for analysis of background

Problem - to calculate the pixels are the background and the

background.
1. Learning the background

Idea: store the image background and then consider the difference between shots with the
camera from the stored images.

A more advanced method - "code book", he remembers the brightness of the background
in each pixel for a while, that allows to deal with objects such as opening doors.

There are also more complex algorithms.

Problems:
- Rapid changes in lighting - should be specifically
consider
- The shadows of objects are often perceived as
no background, which is usually undesirable, so
also be specially taken into account (through the color analysis).
2. Adaptive learning

Simple method, where the background is averaging of the last N frames

const float k = 0.01;

back = (1-k) * back + k * image;
a low pass filter

A few seconds later the object came into the picture becomes the background. For some
tasks it is good.

Plus - no need to specifically construct a situation where there is no one in the frame for
storing the background.

There are more sophisticated algorithms automatically calculate the background with the
help of several accounting staff.
3.Using data on the depth

1. Stereo camera
2. Camera with the calculation of flight time (based on a laser rangefinder, time of filght
cameras)
3. Kinect

Provide information on the distance to each pixel, allowing long-discarded objects by

implementing a threshold depth of processing.
Motion Analysis
The difference between two frames -
the algorithm "Motion Detector"
1. Let image1, image2 - two consecutive frames from the camera, single-channel.
2. Building a module of their difference
Mat diff;
absdiff (image1, image2, diff);
3. Perform threshold processing
Mat bin;// Here we will find the pixels that correspond to areas of motion
threshold (diff, bin, 100 / * threshold * /, 255, CV_THRESH_BINARY);
The difference between two frames -
the algorithm
"Motion Detector"
This method allows you to find the area to which there was a
movement.

For the analysis directions movement of such objects - the

notion of Optical Flow.
What is the optical flow
Optical Flow(Optical flow, optic flow) - is a vector field of apparent motion of objects,
surfaces and edges in a visual scene, caused by relative motion between the observer
(eye camera) and the stage.

Note: it seems that the point

of the camera moves down,
but maloteksturirovannoy of
flow is not - as the local
pixels in the neighborhood
do not change

http://www.ultimategraphics.co.jp/jp/images/stories/ultimate/BCC/optical.jpg
The main application of optical flow

1. To determine the direction in which the moving objects in the

frame.

2. The production of films - for a smooth morphing between

successive frames, or between the shots taken adjacent cells
(the most characteristic is used in the movie "The Matrix").

3. In principle can be applied to stereo vision - to determine the

distance to an object by analyzing the optical flow of personnel
coming from the two chambers.
Methods of calculating the optical flow
Input: two frames. Output - vector field (fx (x, y), fy (x, y))
- Vector of geometric shift pixels from the first frame on the second.

(I) Block ("Naive" methods)

For each point searched shift that minimizes the difference in the local window.

(II) Differential (Most used)

Estimation of the derivatives of x, y, t.
1. Lucas-Kanade - Very fast.
2.Farneback- Good enough, but slow
3.CLG - Very high quality but not yet implemented in OpenCV
4. Pyramidal Lucas-Kanade, calculated only on "points of interest"
5. Horn-Schunk - not very resistant to noise

(III) based on discrete optimization (Intensive)

The solution is constructed using the methods of min-cut, max-flow, linear programming
and belief propagation.
Methods of calculating the optical flow

1. Lucas-Kanade
Local method, which uses the Taylor expansion in time t in the neighborhood of each pixel
independently.
Dignity: Quickly calculated.
Shortcomings: "Aperture problem" - in areas with a uniform texture, works poorly due to
the fact that it is local.
Realized only in the C-version of OpenCV,cvCalcOpticalFlowLK.

Flow superimposed on the image.

Note that the flow inside the palm is not found.

Flow drawn in increments of 10 pixels, the

function line. (Note: The values of the flux
smoothed over the neighborhood for a more
stable result.)
Methods of calculating the optical flow

2. Farneback
Local method used to calculate an approximation of the image by a polynomial function.

Dignity: Significantly more resistant to the problems of the aperture.

Shortcomings: Runs noticeably slower Lucas-Kanade.

Methods of calculating the optical flow

void calcOpticalFlowFarneback(
const Mat & prevImg, // first frame, an 8-bit single channel image
const Mat & nextImg, // second frame, the type and size as a prevImg
Mat & flow, // resulting flow will be of the type CV_32FC2
double pyrScale, // <1, the scale of the construction of the pyramid.0.5
int levels, // Number of levels of the pyramid5
int winsize, // window averaging. The more - the result
// More diffuse, but also more resistant to noise5
int iterations, // Number of iterations at each level of the pyramid 3
int polyN, // window size to calculate the approximation
// Polynomial7
double polySigma, // parameter is Gaussian for smoothing derivatives
// In the construction of approximation1.5
int flags // flags
//OPTFLOW_USE_INITIAL_FLOW- Do not use!
//OPTFLOW_FARNEBACK_GAUSSIAN- Use a Gaussian for the averaging
// Gives a more accurate result at the expense of quality.
)
Methods of calculating the optical flow

3. CLG

"Combining Local and Global"

The method of combining Lucas-Kanade and Horn-Schunk

Now (2010), he and his modification is considered one of the

best quality / speed.

Not yet implemented in OpenCV.

Difficult task (for exam):

find a realization of CLG in C / C + +, compile, and run on a pair
of test images.