Lens Calibration

In order to accurately calibrate the two cameras, DepthKit needs to understand the subtleties of the camera lenses – imperfect manufacturing and lens distortion mean that every lens will be slightly different. To sense these values for both the Kinect and color camera we capture and analyze images of a checkerboard seen from both cameras.

The calibration process begins by connecting your Kinect to your PC (or Bootcamp'd Mac) and loading up DepthKit Capture. If properly connected, you should see a live preview of the Kinect in the window on the left.

Set your Working Directory to a convenient place on your hard drive. All your footage will be stored here. Change it by clicking the text at the top of the DepthKitCapture window. The default directory is depthframes which is located inside of your application folder. Observe that the software creates a '_calibration' folder for you inside the project directory. The calibration files are automatically saved as you go and will get overwritten if you repeat calibration steps.

Select Calibrate Lenses tab, the left-most tab in the capture application. It is split in half vertically; your depth camera stream should appear on the left and the right pane should be empty to start. If you don't see the depth camera view, see troubleshooting page.

Note for Depth-only captures: You only need to self-calibrate the Depth camera.

Self-Calibrate Kinect Lens

Note the FoV's are symmetrical, and that the principal point is at the center of the depth camera's fixed 640x480px frame.

Note the FoV's are symmetrical, and that the principal point is at the center of the depth camera's fixed 640x480px frame.

Aim your rig at an area of space which has plenty of information visible to the Kinect - it needs variable depth in the scene. Do not aim it at a flat wall, aim it at a busy room full of things in the near, middle and background. Hit the Self Calibrate Depth Camera button at the bottom of the left-hand pane. This will automatically analyze your sample and once complete should display results similar to the results at right.

Calibrate Video Camera Lens

To capture the video camera's lens properties it takes a bit more effort and patience. Note that the DepthKit does not currently do camera control of any kind, so you will be operating your camera as you usually do and ingesting the footage. Choose camera settings that provide you with as much depth of field as possible - f5.6 or above is best and keep your ISO low if possible. Set your resolution to the desired final resolution for your project and your framerate to around 24p. 

Set your camera rig up on a tripod and point it at an open stretch of space you can move around in.

Place your checkerboard on a stand (or wall) in front of the camera, at a distance from the camera so that it occupies approximately 1/4 of the frame. If you need a link to download the checkerboards again, you can find the A3 checkerboard here and the A4 checkerboard here.

Be careful not to overexpose the image. It's better to under expose it by a stop or two than to blow out the white areas on the checkerboard.

Be careful not to overexpose the image. It's better to under expose it by a stop or two than to blow out the white areas on the checkerboard.

Adjust the tripod head so that the checkerboard is in the top-left quadrant. Don't worry if the checkerboard is not exactly horizontal or vertical, but do ensure that the entire checkerboard is in the frame, including the white border around the outside black squares. Make sure the board is well exposed (err on the dark side,) evenly lit across the surface, and that the camera is focused on the checkerboard for every sample. Refocus and confirm that all the corners are crisp for every sample.

Record a 1-3 second video of the checkerboard, mindful of keeping the camera very still. Repeat the process four times at this distance from the checkerboard, filling all four quadrants of the frame, top-left, top-right, bottom-left, bottom, right.

Repeat this process at a distance where the checkerboard occupies around 1/9th of the frame, refocus and take three rows of three images -- top-left, top-middle, top-right, middle-left etc. Take care to refocus every clip as the focus distance to the checkerboard changes as you til the camera.

Once complete, make a new folder called "RGBIntrinsics" inside the "_calibration" folder inside of the working directory you set in the first step. Download the clips you just took into the newly created "RGBIntrinsics" folder.

Set the Square Size (cm) of the checkerboard inside the application. For reference, use 3.38 if you have used A3 sized checkerboard and 2.54 if you used the A4 sized board. If yours is a different size, measure one square precisely and use that width.

Drag the "RGBIntrinsics" folder onto the 'Capture Lenses' tab's right-hand window (which should be empty). This should automatically start the calibration process. You may need to wait for a few seconds while this takes place; the application selects the middle frame from each video, converts it into a black and white PNG which is stored in the _calibration directory in your working folder.

Note: Older versions of the tool required dragging the clips themselves. This is no longer required. You can just drag the folder itself. Note as well that the calibration doesn't read subfolders, so all your video files should be directly in the "RGBIntrinsics" folder, not in a folder inside of that folder.

good_intrinsics.png

Once the analysis is complete, the software will display a 'Total Error' figure below the checkerboard images. This is the average error across all the calibration images - think of it as an overall grade for calibration. You can also view the individual error margins for each image by scrubbing the mouse from left to right across the calibration window. A Total Error of less than 0.2 is desirable.

If your calibration has resulted in a higher Total Error, scrub through your image set by moving your mouse left & right over the window and look for any outlier images which have an Error of higher than 0.3. Note the filename of any outliers. You can redo the calibration at any time by dragging the videos onto the window pane again - this time excluding the clips with high error numbers. This should improve your Total Error.

If nearly all of your images have high error, you will need to re-shoot them. Before you do this, look for elements in your environment which could have caused the error. Is there light streaking across your checkerboard? Were you not careful about focusing? Check the troubleshooting section for more reasons why you may be getting high error.

Congratulations, you've now sensed a mathematical model of your lenses. You can use this same intrinsic calibration the next time you shoot with the same lens/camera combination. Using this internal calibration we can next determine the external relationship between the two cameras.