OpenCV – introduction

Mašinska vizija, 2017.

OpenCV -History
• OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at
real-time computer vision.
• Intel team from Nizhny Novgorod created the library
• Vadim Pisarevsky – the largest single contributor to the library
• Gary Bradski – in 1999. launched OpenCV with the hopes of accelerating computer vision and artificial
intelligence by providing a solid infrastructure for everyone working in the field.
• OpenCV has received much of its support over the years from Intel and Google, but especially
from Itseez (recently acquired by Intel), which did the bulk of the early development work.
• Intel also donated the built-in performance primitives code, which provides for seamless speedup
on Intel architectures.
• open-source BSD liscence
• imposes minimal restrictions on the redistribution of covered software
• allows Linking, Distribution, Modification, Private use and Subliscencing
• requires that the code under BSD must be licensed under BSD if redistributed in source code format
Nizhny Novgorod
 According to one well-known story, in 1966, Marvin
Minsky at MIT asked his undergraduate student Gerald
Jay Sussman to “spend the summer linking a camera to
Why OpenCV? a computer and getting the computer to describe what
it saw” – That is how computer/machine vision started!

• OpenCV was designed for computational efficiency and with a strong focus on real-time
applications. It is written in optimized C++ and can take advantage of multicore
processors. => IT IS FAST
• The library is written in C and C++ and runs under Linux, Windows, and Mac OS X. There
is active development on interfaces for Python, Java, MATLAB, and other languages,
including porting the library to Android and iOS for mobile applications. => GOOD
PORTABILITY
• One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that
helps people build fairly sophisticated vision applications quickly. The OpenCV library
contains over 500 functions that span many areas in vision, including factory product
inspection, medical imaging, security, user interface, camera calibration, stereo vision,
and robotics. => DIVERSE USAGE and provides good starting points in each area
• Functions are well documented, but they lack in example diversity.
• OpenCV is not great if you just want to manipulate images.
• OpenCV is great for teaching the computer how to see something!
Learning OpenCV x.x – A.Kaehler, G. Bradski
• The intent of this book is tutorial
• Supplemental material (code examples, exercises, etc.) is available for download
at https://github.com/oreillymedia/Learning-OpenCV-3_examples.
• The purpose of this book is to:
• Comprehensively document OpenCV by detailing what function calling conventions really
mean and how to use them correctly
• Give the reader an intuitive understanding of how the vision algorithms work
• Give the reader some sense of what algorithm to use and when to use it
• Give the reader a boost in implementing computer vision and machine learning algorithms by
providing many working code examples to start from
• Suggest ways to fix some of the more advanced routines when something goes wrong
 OpenCV Block Diagram
• OpenCV is built in layers. At the top is the OS
under which OpenCV operates.
• Next comes the language bindings and sample
applications.
• Below that is the contributed code in
opencv_contrib, which contains mostly higher-
level functionality.
• After that is the core of OpenCV, and at the
bottom are the various hardware
optimizations in the hardware acceleration
layer (HAL).
OpenCV contrib
• Face Recognition
• Object Tracking
• DNN – Deep Neural Networks – Using Caffe framework models
• SFM – Structure From Motion
• Structured Light
• Multi-camera Calibration
• Stereo Matching
• ...
Installing OpenCV 3.2
• Using prebuilt OpenCV binaries [Win]
• tutorial from docs.opencv.org - it is made for v2.4 but works for all newer
• In order to use them in Visual Studio include files and libraries need to be included/linked
• libraries “as is”, if we want some additional function we need to rebuild sources
• NuGet
• nuget.org
• package manager for the Microsoft development platform
• works only for VS14 and older
• very simple and fast OpenCV installation, but not flexible
• recommendation: install opencv3.1 package
• Building OpenCV from source
• same tutorial from above
• rather complicated on Windows, linux provides much easier interface
• download OCV source code -> install Cmake
• install additional libraries and download opencv-contrib
• use cmake to configure and generate build, then VS to compile
Installing OpenCV
• Using prebuilt OpenCV binaries [Win]
• Sourceforge page -> 3.2.0 -> -vc14.exe
• run the .exe, select path for extraction (suggested E:/MS1MV/opencv)
• set the environment variable: OPENCV_DIR
E:\MS1MV\opencv\build\x64\vc14
• add bin folder to system path: %OPENCV_DIR%\bin
• lib – static libraries
• dll – dynamic link libraries – loaded only on demand during runtime
Building applications with OCV inside the
Microsoft Visual Studio
• To build an application with OpenCV you need to do two things:
• Tell to the compiler how the OpenCV library looks. You do this by showing it
the header files.
• Tell to the linker from where to get the functions or data structures of
OpenCV, when they are needed.
• VS -> New Project - > Win32 Console Application -> give name -> OK -
> Next -> Empty Project -> Finish
• View -> Other Windows -> Property Manager
• open Property Manager view and select Debug|x64 -> Add New
Project Property Sheet
Building applications with OCV inside the
Microsoft Visual Studio
• created Property Sheet -> Properties
• OpenCV include directory (*.h and *.hpp files)
• C++ -> General -> Additional Include Directories =
$(OPENCV_DIR)\..\..\include (make sure to press Enter and Apply)
• Libraries – we want to link them
• static (*.lib)
• Linker -> General -> Additional Library Directories -> $(OPENCV_DIR)\lib
• Linker -> Input -> Additional Dependencies -> we need to write down all debug libraries
(when creating release property page, then release libraries).
• dynamic (*.dll)
• program will look for the .dll’s during runtime and if we have correctly added their path
to path environment variable it should successfully find them
Run some code!
• Create new .cpp file in Source Files
• Copy the code
#include <opencv2/opencv.hpp> //Include file for every supported OpenCV function

int main(int argc, char** argv) {

cv::Mat img = cv::imread(argv[1], -1); // Image name is expected as command line argument
if (img.empty()) return -1; // always check if the image was successfully read!!!
// create display window
cv::namedWindow("Image", cv::WINDOW_AUTOSIZE);
cv::imshow("Image", img); // show image; the window is selected through is unique name
cv::waitKey(0); // to keep the window showing we use waitKey(ms delay), 0 is forever
cv::destroyWindow("Example1"); // after any button is pressed close the window and finish
return 0;
}
Documentation
• http://docs.opencv.org/master/
• tutorials
• introduction: installation and hello world
• core module: matrix manipulation, file I/O, random generator…
• imgproc: smoothing, sharpening, edge detection, common image processing
• highgui: manipulate windows displays and add trackbars and other UI
• …
• cheatsheet (OCV 2.4!) http://docs.opencv.org/3.0-last-
rst/opencv_cheatsheet.pdf
• High level overview of the library
• OpenCV 3.x Modules ppt
First program – Display a picture - DONE
• OpenCV provides utilities for reading from a wide array of image file
types, as well as from video and cameras. These utilities are part of a
toolkit called HighGUI, which is included in the OpenCV package. We
used some of these utilities to create a simple program that opens an
image and displays it on the screen.
• Display grayscale version of the image – cv::cvtColor (convert color)
• cvtColor is not a part of highgui -> look for appropriate module and include it
• which color systems are supported?
• in OpenCV input images are in BGR color space!!!
Simple transformation
• Perform image blurring with Gaussian filter 5x5, sigma = 5.
• cv::GaussianBlur(cv::InputArray src, cv::OutputArray dst, Size ksize,
double sigmaX, double sigmaY=0.0, int borderType =
cv::BORDER_DEFAULT)

• Sobel edge detection on a grayscale image -> Save image (imwrite)

• where are the edges?
• what’s the data type?
• how does imshow() work?
Input from a Camera -> cv::VideoCapture
• The first thing we need is the cv::VideoCapture object. This object contains the
information needed for reading frames from a camera or video file.
• Depending on the source, we use one of three different calls to create a
cv::VideoCapture object:
• cv::VideoCapture::VideoCapture( const string& filename); // video filename
• cv::VideoCapture::VideoCapture(int device); // camera (device) ID
• cv::VideoCapture::VideoCapture(); // default constructor
• If the open is successful and we are able to start reading frames, the member function
cv::VideoCapture::isOpened() will return true.
• A lot of people don’t always check these sorts of things, assuming that nothing will go
wrong. Don’t do that here. The returned value of cv::VideoCapture::isOpened() will be
false if for some reason the file could not be opened (e.g., if the file does not exist), but
that is not the only possible cause. The constructed object will also not be ready to be
used if the codec with which the video is compressed is not known. Because of the many
issues surrounding codecs (legal as well as technical), this is not as rare of an occurrence
as one might hope.
Reading the camera
• cv::VideoCapture::read(cv::OutputArray image);
• cv::VideoCapture::operator>>(cv::Mat& image);
• cv::VidedoCapture::grab(void) -> copies raw data from camera quickly to
computer, and then when we want to use it, we must decode it
• cv::VideoCapture::retrieve(cv::OutputArray image, int channel=0)
• read() preforms grabbing and retrieving together
• Where is grab-retrieve useful?
• The most common situation arises when there are multiple cameras (e.g., with
stereo imaging). In this case, it is important to have frames that are separated in time
by the minimum amount possible (ideally they would be simultaneous for stereo
imaging). Therefore, it makes the most sense to first grab all the frames and then
come back and decode them after you have them all safely in your buffers.
Video metadata
• cv::VideoCapture::get() and cv::VideoCapture::set()
• Video files contain not only the video frames themselves, but also important
metadata, which can be essential for handling the files correctly. When a video
file is opened, that information is copied into the cv::VideoCapture object’s
internal data area. It is very common to want to read that information from the
cv::VideoCapture object, and sometimes also useful to write to that data area
ourselves.
• Some metadata:
• video duration
• frame size
• bit rate mode
• color cpace
• chroma subsampling
• …
Input from a Camera
• Open file read_camera.cpp and copy the code (make sure that you
don’t have two main functions)
• Function expects camera filename as input, so make sure you have no
command line arguments
• Run the code, see what happens!
• Instead of simple streaming, show edges!
• we tried Sobel for image, now try Canny
Getting to know OpenCV data types
• OpenCV has many data types, which are designed to make the
representation and handling of important computer vision concepts
relatively easy and intuitive.
• At the same time, many algorithm developers require a set of
relatively powerful primitives that can be generalized or extended for
their particular needs.
• This library attempts to address both of these needs through the use
of templates for fundamental data types, and specializations of those
templates that make everyday operations easier.
OpenCV data types
1. Basic data types - are those that are assembled directly from C++
primitives (int, float, etc.). These types include simple vectors and
matrices, as well as representations of simple geometric concepts like
points, rectangles, sizes, and the like. cv::Matx
2. Helper objects - These objects represent more abstract concepts such as
the range objects used for slicing, e.g. cv::Range
3. Large array types - These are objects whose fundamental purpose is to
contain arrays or other assemblies of primitives or, more often, the basic
data types. The star example of this category is the cv::Mat class, which
is used to represent arbitrary-dimensional arrays containing arbitrary
basic elements.
In addition to these types, OpenCV also makes heavy use of the Standard
Template Library (STL). OpenCV particularly relies on the vector class, and
many OpenCV library functions now have vector template objects in their
argument lists.
cv::Matx – fixed matrix class
• matrix dimension known at compile time
• all memory for their data is allocated on the stack, which means that
they allocate and clean up quickly
• Operations on them are fast, and there are specially optimized
implementations for small matrices (2 × 2, 3 × 3, etc.)
cv::Vec<> - also fixed
• Used through aliases (typedefs) for common instantiations of the cv::Vec<>
template.
• They have names like cv::Vec2i, cv::Vec3i, and cv::Vec4d (for a two-element
integer vector, a three element integer vector, or a four-element double-
precision floating-point vector, respectively).
• In general, anything of the form cv::Vec{2,3,4,6}{b,w,s,i,f,d} is valid for any
combination of two to four dimensions and the six data types.
• Can be accessed by a vector index -> myvec[0], myvec[1],…
• In the proper sense of C++ inheritance, it is correct to say that the fixed
vector template cv::Vec<> is a cv::Matx<> whose number of columns is
one.
cv::Point<> - also fixed
• The point classes, which are containers for two or three
values of one of the primitive types.
• cv::Point2i, cv::Point2f
• members are accessed by named variables -> mypoint.x, mypoint.y

• Similar to cv::Point<> is cv::Size which allows storing 2D object size

and also has two members: width and height.
• Mostly used for defining mask/matrix size
cv::Range – helper object
• two elements: start and end
• Ranges are inclusive of their start value, but not inclusive of their end
value
• cv::Range(int start, int end) -> [start, end)
cv::InputArray and cv::OutputArray
• Many OpenCV functions take arrays as arguments and return arrays as
return values, but in OpenCV, there are many kinds of arrays. We have
already seen that OpenCV supports
• small array types
(cv::Scalar, cv::Vec, cv::Matx) and STL’s std::vector<>
• large array types (cv::Mat and cv::SparseMat).
• In order to keep the interface from becoming onerously complicated (and
repetitive), OpenCV defines the types cv::InputArray and cv::OutputArray.
• In effect, these types mean “any of the above” with respect to the many
array forms supported by the library.
• The primary difference between cv::InputArray and cv::OutputArray is that
the former is assumed to be const (i.e., read only).
cv::Mat
• Could be considered the epicenter of the entire C++ implementation of the
OpenCV library.
• The overwhelming majority of functions in the OpenCV library are
members of the cv::Mat class, take a cv::Mat as an argument, or return
cv::Mat as a return value; quite a few are or do all three.
• Objects such as images are specialized uses of the cv::Mat class, but such
specific use does not require a different class or type.
• The cv::Mat class is used to represent dense arrays of any number of
dimensions. In this context, dense means that for every entry in the array,
there is a data value stored in memory corresponding to that entry, even if
that entry is zero.
• Most of the used types have their own Mat constructor. -> open docs
cv::Mat
• dims - number of dimensions
• rows & cols - number of rows and columns. If dims>2, (rows, cols) =(-1,-1)
• size – Size(cols, rows)
• *data - pointer to the data
• step – defines data layout of the matrix (regarded as an array)
• This latter member allows cv::Mat to behave very much like a smart pointer for the data
contained in data. The data array is laid out such that the address of an element whose indices
are (for 2D array) i and j:
&(mat_i,j) = mat.data + mat.step[0]*i + mat.step[1]*j
• This means that 2-dimensional matrices are stored row-by-row, 3-dimensional matrices are
stored plane-by-plane, and so on.
• Each element of the data in a cv::Mat can itself be either a single number or multiple numbers.
Creating array/matrix
• You can create an array simply by instantiating a variable of type cv::Mat. An array
created in this manner has no size and no data type.
• You can, however, later ask it to allocate data by using a member function such as
create().
• One variation of create() takes as arguments a number of rows, a number of columns,
and a type, and configures the array to represent a two-dimensional object.
• The type of an array determines what kind of elements it has.
• Valid types in this context specify both the fundamental type of element as well as the
number of channels. All such types are defined in the library header, and have the form
CV_{8U,16S,16U,32S,32F,64F}C{1,2,3}.
• For example, CV_32FC3 would imply a 32-bit floating-point three-channel array.
• If you prefer, you can also specify these things when you first allocate the matrix. There
are many constructors for cv::Mat, one of which takes the same arguments as create()
 THE MOST IMPORTANT PARAGRAPH IN THE
BOOK
• It is critical to understand that the data in an array is not attached rigidly to the
array object.
• The cv::Mat object is really a header for a data area, which — in principle — is an
entirely separate thing.
• For example, it is possible to assign one matrix n to another matrix m (i.e., m=n).
In this case several things happen:
• the data pointer inside of m will be changed to point to the same data as n.
• the data pointed to previously by the data element of m (if any) will be deallocated.
• the reference counter for the data area that they both now share will be incremented.
• the members of m that characterize its data (such as rows, cols, and flags) will be updated to
accurately describe the data now pointed to by data in m.
• This all results in a very convenient behavior, in which arrays can be assigned to
one another, and the work necessary to do this takes place automatically behind
the scenes to give the correct result.
Accessing Array Elements Individually
• By location -> member function at<>()
• The way this function works is that you specialize the at<>() template to the
type of data that the matrix contains, then access that element using the row
and column locations of the data you want.

• cv::DataType<>
Accessing Array Elements Individually
• Through iteration
• use the iterator mechanism built into cv::Mat
• cv::MatConstIterator<> -> The cv::Mat methods begin() and end() return objects of
this type.

• This method of iteration is convenient because the iterators are smart enough to
handle the continuous packing and noncontinuous packing cases automatically, as ell
as handling any number of dimensions in the array.
Accessing Array Elements by Block
• member functions of the cv::Mat class
• row()
• col()
• rowRange() -> rowRange(i0, i1) == rowRange(cv::Range(i0,i1))
• colRange()
• m(cv::Range(i0,i1), cv::Range(j0,j1)) -> Array corresponding to the
subrectangle of matrix m with one corner at i0, j0 and the opposite corner at
(i1-1, j1-1)
• m( cv::Rect(i0,i1,w,h) ); Array corresponding to the subrectangle of matrix m
with one corner at i0, j0 and the opposite corner at (i0+w-1, j0+h-1)
Back to Sobel
• Perform Sobel edge detection step by step by accessing each
individual pixel and its neighborhood.
• S = [-1, -2, -1;
0, 0, 0;
1, 2, 1]; //horizontal edges
Algebra and cv::Mat
• Operations available for matrix expressions
• m0 + m1, m0 – m1;
• m0 + s; m0 – s; s + m0, s – m1;
• -m0;
• s * m0; m0 * s;
• m0.mul( m1 ); m0/m1; // per element
• m0 * m1; // matrix multiplication
• m0>m1; m0>=m1; m0==m1; m0<=m1; m0<m1; // per element comparison
• m0&m1; m0|m1; m0^m1; ~m0;
m0&s; s&m0; m0|s; s|m0; m0^s; s^m0 ; //bitwise logical operations
More Things an Array Can Do
• m1 = m0.clone();
• m0.copyTo( m1 );
• m0.copyTo( m1, mask ); //only entries indicated in the array mask are
copied
• m0.convertTo(m1, type, scale, offset); // Convert elements of m0 to
type (e.g., CV_32F) and write to m1 after scaling by scale (default 1.0)
and adding offset (default 0.0)
• m0.reshape( chan, rows);
• m0.empty();
Final goal – act like cats
1. Detect red circle in the image
2. Extract it from the background
3. extract its center position
4. Display the path of the circle
through the given image set

Images and template code given:

• track_red_circle.cpp
• \fotke
Hints and advices
• Use only one image for start!
• decide upon color space you want to use (RGB, Lab, HSV)
• debugging is not so simple, show your image between each image
processing step, use multiple windows
• want to extract a range of values -> use cv::inRange(src, lowerb, upperb,
dst)
• Image is of bad quality? Maybe use some blurring to make objects
uniform?
• Object has holes or not-so-round shape? Morphological operations!
• I detect several objects? bwlabel (connectedComponents)
• Thresholding is boring? Try using Hough circle transform!
Time performance of accessing single
elements
• DEBUG RELEASE
Reference
[1] http://docs.opencv.org/master/– OpenCV 3.2.0-dev Documentation
[2] Adrian Kaehler and Gary Bradski. 2016. Learning OpenCV 3:
Computer Vision in C++ with the OpenCV Library (1st ed.). O'Reilly
Media, Inc..
[3] Kenneth Dawson-Howe. 2014. A Practical Introduction to Computer
Vision with OpenCV (1st ed.). Wiley Publishing.