1 .. _cameraCalibrationOpenCV:
3 Camera calibration With OpenCV
4 ******************************
6 Cameras have been around for a long-long time. However, with the introduction of the cheap *pinhole* cameras in the late 20th century, they became a common occurrence in our everyday life. Unfortunately, this cheapness comes with its price: significant distortion. Luckily, these are constants and with a calibration and some remapping we can correct this. Furthermore, with calibration you may also determine the relation between the camera's natural units (pixels) and the real world units (for example millimeters).
11 For the distortion OpenCV takes into account the radial and tangential factors. For the radial factor one uses the following formula:
15 x_{corrected} = x( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6) \\
16 y_{corrected} = y( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6)
18 So for an old pixel point at :math:`(x,y)` coordinates in the input image, its position on the corrected output image will be :math:`(x_{corrected} y_{corrected})`. The presence of the radial distortion manifests in form of the "barrel" or "fish-eye" effect.
20 Tangential distortion occurs because the image taking lenses are not perfectly parallel to the imaging plane. It can be corrected via the formulas:
24 x_{corrected} = x + [ 2p_1xy + p_2(r^2+2x^2)] \\
25 y_{corrected} = y + [ p_1(r^2+ 2y^2)+ 2p_2xy]
27 So we have five distortion parameters which in OpenCV are presented as one row matrix with 5 columns:
31 Distortion_{coefficients}=(k_1 \hspace{10pt} k_2 \hspace{10pt} p_1 \hspace{10pt} p_2 \hspace{10pt} k_3)
33 Now for the unit conversion we use the following formula:
37 \left [ \begin{matrix} x \\ y \\ w \end{matrix} \right ] = \left [ \begin{matrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{matrix} \right ] \left [ \begin{matrix} X \\ Y \\ Z \end{matrix} \right ]
39 Here the presence of :math:`w` is explained by the use of homography coordinate system (and :math:`w=Z`). The unknown parameters are :math:`f_x` and :math:`f_y` (camera focal lengths) and :math:`(c_x, c_y)` which are the optical centers expressed in pixels coordinates. If for both axes a common focal length is used with a given :math:`a` aspect ratio (usually 1), then :math:`f_y=f_x*a` and in the upper formula we will have a single focal length :math:`f`. The matrix containing these four parameters is referred to as the *camera matrix*. While the distortion coefficients are the same regardless of the camera resolutions used, these should be scaled along with the current resolution from the calibrated resolution.
41 The process of determining these two matrices is the calibration. Calculation of these parameters is done through basic geometrical equations. The equations used depend on the chosen calibrating objects. Currently OpenCV supports three types of objects for calibration:
43 .. container:: enumeratevisibleitemswithsquare
45 + Classical black-white chessboard
46 + Symmetrical circle pattern
47 + Asymmetrical circle pattern
49 Basically, you need to take snapshots of these patterns with your camera and let OpenCV find them. Each found pattern results in a new equation. To solve the equation you need at least a predetermined number of pattern snapshots to form a well-posed equation system. This number is higher for the chessboard pattern and less for the circle ones. For example, in theory the chessboard pattern requires at least two snapshots. However, in practice we have a good amount of noise present in our input images, so for good results you will probably need at least 10 good snapshots of the input pattern in different positions.
54 The sample application will:
56 .. container:: enumeratevisibleitemswithsquare
58 + Determine the distortion matrix
59 + Determine the camera matrix
60 + Take input from Camera, Video and Image file list
61 + Read configuration from XML/YAML file
62 + Save the results into XML/YAML file
63 + Calculate re-projection error
68 You may also find the source code in the :file:`samples/cpp/tutorial_code/calib3d/camera_calibration/` folder of the OpenCV source library or :download:`download it from here <../../../../samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp>`. The program has a single argument: the name of its configuration file. If none is given then it will try to open the one named "default.xml". :download:`Here's a sample configuration file <../../../../samples/cpp/tutorial_code/calib3d/camera_calibration/in_VID5.xml>` in XML format. In the configuration file you may choose to use camera as an input, a video file or an image list. If you opt for the last one, you will need to create a configuration file where you enumerate the images to use. Here's :download:`an example of this <../../../../samples/cpp/tutorial_code/calib3d/camera_calibration/VID5.xml>`. The important part to remember is that the images need to be specified using the absolute path or the relative one from your application's working directory. You may find all this in the samples directory mentioned above.
70 The application starts up with reading the settings from the configuration file. Although, this is an important part of it, it has nothing to do with the subject of this tutorial: *camera calibration*. Therefore, I've chosen not to post the code for that part here. Technical background on how to do this you can find in the :ref:`fileInputOutputXMLYAML` tutorial.
75 1. **Read the settings.**
80 const string inputSettingsFile = argc > 1 ? argv[1] : "default.xml";
81 FileStorage fs(inputSettingsFile, FileStorage::READ); // Read the settings
84 cout << "Could not open the configuration file: \"" << inputSettingsFile << "\"" << endl;
88 fs.release(); // close Settings file
92 cout << "Invalid input detected. Application stopping. " << endl;
96 For this I've used simple OpenCV class input operation. After reading the file I've an additional post-processing function that checks validity of the input. Only if all inputs are good then *goodInput* variable will be true.
98 #. **Get next input, if it fails or we have enough of them - calibrate**. After this we have a big loop where we do the following operations: get the next image from the image list, camera or video file. If this fails or we have enough images then we run the calibration process. In case of image we step out of the loop and otherwise the remaining frames will be undistorted (if the option is set) via changing from *DETECTION* mode to the *CALIBRATED* one.
105 bool blinkOutput = false;
107 view = s.nextImage();
109 //----- If no more image, or got enough, then stop calibration and show result -------------
110 if( mode == CAPTURING && imagePoints.size() >= (unsigned)s.nrFrames )
112 if( runCalibrationAndSave(s, imageSize, cameraMatrix, distCoeffs, imagePoints))
117 if(view.empty()) // If no more images then run calibration, save and stop loop.
119 if( imagePoints.size() > 0 )
120 runCalibrationAndSave(s, imageSize, cameraMatrix, distCoeffs, imagePoints);
122 imageSize = view.size(); // Format input image.
123 if( s.flipVertical ) flip( view, view, 0 );
126 For some cameras we may need to flip the input image. Here we do this too.
128 #. **Find the pattern in the current input**. The formation of the equations I mentioned above aims to finding major patterns in the input: in case of the chessboard this are corners of the squares and for the circles, well, the circles themselves. The position of these will form the result which will be written into the *pointBuf* vector.
132 vector<Point2f> pointBuf;
135 switch( s.calibrationPattern ) // Find feature points on the input format
137 case Settings::CHESSBOARD:
138 found = findChessboardCorners( view, s.boardSize, pointBuf,
139 CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FAST_CHECK | CV_CALIB_CB_NORMALIZE_IMAGE);
141 case Settings::CIRCLES_GRID:
142 found = findCirclesGrid( view, s.boardSize, pointBuf );
144 case Settings::ASYMMETRIC_CIRCLES_GRID:
145 found = findCirclesGrid( view, s.boardSize, pointBuf, CALIB_CB_ASYMMETRIC_GRID );
149 Depending on the type of the input pattern you use either the :calib3d:`findChessboardCorners <findchessboardcorners>` or the :calib3d:`findCirclesGrid <findcirclesgrid>` function. For both of them you pass the current image and the size of the board and you'll get the positions of the patterns. Furthermore, they return a boolean variable which states if the pattern was found in the input (we only need to take into account those images where this is true!).
151 Then again in case of cameras we only take camera images when an input delay time is passed. This is done in order to allow user moving the chessboard around and getting different images. Similar images result in similar equations, and similar equations at the calibration step will form an ill-posed problem, so the calibration will fail. For square images the positions of the corners are only approximate. We may improve this by calling the :feature2d:`cornerSubPix <cornersubpix>` function. It will produce better calibration result. After this we add a valid inputs result to the *imagePoints* vector to collect all of the equations into a single container. Finally, for visualization feedback purposes we will draw the found points on the input image using :calib3d:`findChessboardCorners <drawchessboardcorners>` function.
155 if ( found) // If done with success,
157 // improve the found corners' coordinate accuracy for chessboard
158 if( s.calibrationPattern == Settings::CHESSBOARD)
161 cvtColor(view, viewGray, CV_BGR2GRAY);
162 cornerSubPix( viewGray, pointBuf, Size(11,11),
163 Size(-1,-1), TermCriteria( CV_TERMCRIT_EPS+CV_TERMCRIT_ITER, 30, 0.1 ));
166 if( mode == CAPTURING && // For camera only take new samples after delay time
167 (!s.inputCapture.isOpened() || clock() - prevTimestamp > s.delay*1e-3*CLOCKS_PER_SEC) )
169 imagePoints.push_back(pointBuf);
170 prevTimestamp = clock();
171 blinkOutput = s.inputCapture.isOpened();
175 drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
178 #. **Show state and result to the user, plus command line control of the application**. This part shows text output on the image.
182 //----------------------------- Output Text ------------------------------------------------
183 string msg = (mode == CAPTURING) ? "100/100" :
184 mode == CALIBRATED ? "Calibrated" : "Press 'g' to start";
186 Size textSize = getTextSize(msg, 1, 1, 1, &baseLine);
187 Point textOrigin(view.cols - 2*textSize.width - 10, view.rows - 2*baseLine - 10);
189 if( mode == CAPTURING )
191 if(s.showUndistorsed)
192 msg = format( "%d/%d Undist", (int)imagePoints.size(), s.nrFrames );
194 msg = format( "%d/%d", (int)imagePoints.size(), s.nrFrames );
197 putText( view, msg, textOrigin, 1, 1, mode == CALIBRATED ? GREEN : RED);
200 bitwise_not(view, view);
202 If we ran calibration and got camera's matrix with the distortion coefficients we may want to correct the image using :imgproc_geometric:`undistort <undistort>` function:
206 //------------------------- Video capture output undistorted ------------------------------
207 if( mode == CALIBRATED && s.showUndistorsed )
209 Mat temp = view.clone();
210 undistort(temp, view, cameraMatrix, distCoeffs);
212 //------------------------------ Show image and check for input commands -------------------
213 imshow("Image View", view);
215 Then we wait for an input key and if this is *u* we toggle the distortion removal, if it is *g* we start again the detection process, and finally for the *ESC* key we quit the application:
219 char key = waitKey(s.inputCapture.isOpened() ? 50 : s.delay);
223 if( key == 'u' && mode == CALIBRATED )
224 s.showUndistorsed = !s.showUndistorsed;
226 if( s.inputCapture.isOpened() && key == 'g' )
232 #. **Show the distortion removal for the images too**. When you work with an image list it is not possible to remove the distortion inside the loop. Therefore, you must do this after the loop. Taking advantage of this now I'll expand the :imgproc_geometric:`undistort <undistort>` function, which is in fact first calls :imgproc_geometric:`initUndistortRectifyMap <initundistortrectifymap>` to find transformation matrices and then performs transformation using :imgproc_geometric:`remap <remap>` function. Because, after successful calibration map calculation needs to be done only once, by using this expanded form you may speed up your application:
236 if( s.inputType == Settings::IMAGE_LIST && s.showUndistorsed )
238 Mat view, rview, map1, map2;
239 initUndistortRectifyMap(cameraMatrix, distCoeffs, Mat(),
240 getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0),
241 imageSize, CV_16SC2, map1, map2);
243 for(int i = 0; i < (int)s.imageList.size(); i++ )
245 view = imread(s.imageList[i], 1);
248 remap(view, rview, map1, map2, INTER_LINEAR);
249 imshow("Image View", rview);
251 if( c == ESC_KEY || c == 'q' || c == 'Q' )
256 The calibration and save
257 ========================
259 Because the calibration needs to be done only once per camera, it makes sense to save it after a successful calibration. This way later on you can just load these values into your program. Due to this we first make the calibration, and if it succeeds we save the result into an OpenCV style XML or YAML file, depending on the extension you give in the configuration file.
261 Therefore in the first function we just split up these two processes. Because we want to save many of the calibration variables we'll create these variables here and pass on both of them to the calibration and saving function. Again, I'll not show the saving part as that has little in common with the calibration. Explore the source file in order to find out how and what:
266 bool runCalibrationAndSave(Settings& s, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs,vector<vector<Point2f> > imagePoints )
268 vector<Mat> rvecs, tvecs;
269 vector<float> reprojErrs;
270 double totalAvgErr = 0;
272 bool ok = runCalibration(s,imageSize, cameraMatrix, distCoeffs, imagePoints, rvecs, tvecs,
273 reprojErrs, totalAvgErr);
274 cout << (ok ? "Calibration succeeded" : "Calibration failed")
275 << ". avg re projection error = " << totalAvgErr ;
277 if( ok ) // save only if the calibration was done with success
278 saveCameraParams( s, imageSize, cameraMatrix, distCoeffs, rvecs ,tvecs, reprojErrs,
279 imagePoints, totalAvgErr);
283 We do the calibration with the help of the :calib3d:`calibrateCamera <calibratecamera>` function. It has the following parameters:
285 .. container:: enumeratevisibleitemswithsquare
287 + The object points. This is a vector of *Point3f* vector that for each input image describes how should the pattern look. If we have a planar pattern (like a chessboard) then we can simply set all Z coordinates to zero. This is a collection of the points where these important points are present. Because, we use a single pattern for all the input images we can calculate this just once and multiply it for all the other input views. We calculate the corner points with the *calcBoardCornerPositions* function as:
291 void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Point3f>& corners,
292 Settings::Pattern patternType /*= Settings::CHESSBOARD*/)
298 case Settings::CHESSBOARD:
299 case Settings::CIRCLES_GRID:
300 for( int i = 0; i < boardSize.height; ++i )
301 for( int j = 0; j < boardSize.width; ++j )
302 corners.push_back(Point3f(float( j*squareSize ), float( i*squareSize ), 0));
305 case Settings::ASYMMETRIC_CIRCLES_GRID:
306 for( int i = 0; i < boardSize.height; i++ )
307 for( int j = 0; j < boardSize.width; j++ )
308 corners.push_back(Point3f(float((2*j + i % 2)*squareSize), float(i*squareSize), 0));
313 And then multiply it as:
317 vector<vector<Point3f> > objectPoints(1);
318 calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern);
319 objectPoints.resize(imagePoints.size(),objectPoints[0]);
321 + The image points. This is a vector of *Point2f* vector which for each input image contains coordinates of the important points (corners for chessboard and centers of the circles for the circle pattern). We have already collected this from :calib3d:`findChessboardCorners <findchessboardcorners>` or :calib3d:`findCirclesGrid <findcirclesgrid>` function. We just need to pass it on.
323 + The size of the image acquired from the camera, video file or the images.
325 + The camera matrix. If we used the fixed aspect ratio option we need to set the :math:`f_x` to zero:
329 cameraMatrix = Mat::eye(3, 3, CV_64F);
330 if( s.flag & CV_CALIB_FIX_ASPECT_RATIO )
331 cameraMatrix.at<double>(0,0) = 1.0;
333 + The distortion coefficient matrix. Initialize with zero.
337 distCoeffs = Mat::zeros(8, 1, CV_64F);
339 + For all the views the function will calculate rotation and translation vectors which transform the object points (given in the model coordinate space) to the image points (given in the world coordinate space). The 7-th and 8-th parameters are the output vector of matrices containing in the i-th position the rotation and translation vector for the i-th object point to the i-th image point.
341 + The final argument is the flag. You need to specify here options like fix the aspect ratio for the focal length, assume zero tangential distortion or to fix the principal point.
345 double rms = calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix,
346 distCoeffs, rvecs, tvecs, s.flag|CV_CALIB_FIX_K4|CV_CALIB_FIX_K5);
348 + The function returns the average re-projection error. This number gives a good estimation of precision of the found parameters. This should be as close to zero as possible. Given the intrinsic, distortion, rotation and translation matrices we may calculate the error for one view by using the :calib3d:`projectPoints <projectpoints>` to first transform the object point to image point. Then we calculate the absolute norm between what we got with our transformation and the corner/circle finding algorithm. To find the average error we calculate the arithmetical mean of the errors calculated for all the calibration images.
352 double computeReprojectionErrors( const vector<vector<Point3f> >& objectPoints,
353 const vector<vector<Point2f> >& imagePoints,
354 const vector<Mat>& rvecs, const vector<Mat>& tvecs,
355 const Mat& cameraMatrix , const Mat& distCoeffs,
356 vector<float>& perViewErrors)
358 vector<Point2f> imagePoints2;
359 int i, totalPoints = 0;
360 double totalErr = 0, err;
361 perViewErrors.resize(objectPoints.size());
363 for( i = 0; i < (int)objectPoints.size(); ++i )
365 projectPoints( Mat(objectPoints[i]), rvecs[i], tvecs[i], cameraMatrix, // project
366 distCoeffs, imagePoints2);
367 err = norm(Mat(imagePoints[i]), Mat(imagePoints2), CV_L2); // difference
369 int n = (int)objectPoints[i].size();
370 perViewErrors[i] = (float) std::sqrt(err*err/n); // save for this view
371 totalErr += err*err; // sum it up
375 return std::sqrt(totalErr/totalPoints); // calculate the arithmetical mean
381 Let there be :download:`this input chessboard pattern <../../../pattern.png>` which has a size of 9 X 6. I've used an AXIS IP camera to create a couple of snapshots of the board and saved it into VID5 directory. I've put this inside the :file:`images/CameraCalibration` folder of my working directory and created the following :file:`VID5.XML` file that describes which images to use:
385 <?xml version="1.0"?>
388 images/CameraCalibration/VID5/xx1.jpg
389 images/CameraCalibration/VID5/xx2.jpg
390 images/CameraCalibration/VID5/xx3.jpg
391 images/CameraCalibration/VID5/xx4.jpg
392 images/CameraCalibration/VID5/xx5.jpg
393 images/CameraCalibration/VID5/xx6.jpg
394 images/CameraCalibration/VID5/xx7.jpg
395 images/CameraCalibration/VID5/xx8.jpg
399 Then passed :file:`images/CameraCalibration/VID5/VID5.XML` as an input in the configuration file. Here's a chessboard pattern found during the runtime of the application:
401 .. image:: images/fileListImage.jpg
402 :alt: A found chessboard
405 After applying the distortion removal we get:
407 .. image:: images/fileListImageUnDist.jpg
408 :alt: Distortion removal for File List
411 The same works for :download:`this asymmetrical circle pattern <../../../acircles_pattern.png>` by setting the input width to 4 and height to 11. This time I've used a live camera feed by specifying its ID ("1") for the input. Here's, how a detected pattern should look:
413 .. image:: images/asymetricalPattern.jpg
414 :alt: Asymmetrical circle detection
417 In both cases in the specified output XML/YAML file you'll find the camera and distortion coefficients matrices:
421 <Camera_Matrix type_id="opencv-matrix">
426 6.5746697944293521e+002 0. 3.1950000000000000e+002 0.
427 6.5746697944293521e+002 2.3950000000000000e+002 0. 0. 1.</data></Camera_Matrix>
428 <Distortion_Coefficients type_id="opencv-matrix">
433 -4.1802327176423804e-001 5.0715244063187526e-001 0. 0.
434 -5.7843597214487474e-001</data></Distortion_Coefficients>
436 Add these values as constants to your program, call the :imgproc_geometric:`initUndistortRectifyMap <initundistortrectifymap>` and the :imgproc_geometric:`remap <remap>` function to remove distortion and enjoy distortion free inputs for cheap and low quality cameras.
438 You may observe a runtime instance of this on the `YouTube here <https://www.youtube.com/watch?v=ViPN810E0SU>`_.
443 <iframe title=" Camera calibration With OpenCV - Chessboard or asymmetrical circle pattern." width="560" height="349" src="http://www.youtube.com/embed/ViPN810E0SU?rel=0&loop=1" frameborder="0" allowfullscreen align="middle"></iframe>