Please enable javascript in your browser to view this site!

DepthKit Tutorials

Welcome to the DepthKit tutorials page! Here you'll find resources for everything DepthKit, instructing you on how to successfully execute a shoot from start to finish.

This is an evolving page, so if you see anything that needs to be fixed or have any suggestions for something that is missing, feel free to contact us or send us a note in Slack.

 

PRE-PRODUCTION

debugcrop.png

Equipment

Filming with DepthKit requires a few more things than just filming with a regular 2D video camera. Once you've gathered all the parts though, you're good to go! DepthKit also doesn't use any hard-to-find equipment, so you can likely find everything you need for a shoot by simply visiting any big electronics retailer.

To use DepthKit you'll need a camera, a Kinect, a hardware mount and a suitable computer. We've got a list on Amazon of recommend parts for you if you're looking for an easy buy & build solution.

Computer

Computer requirements for DepthKit Capture
For the on set capture stage, you'll need a computer with a solid state hard drive running Windows 8 / 8.1+ that meets the minimum requirements for running the Kinect. You can use the Kinect Configuration Verifier tool to confirm that your computer meets the requirements.

You can use either a dedicated Windows PC that meets these requirements, or run Bootcamp on an OSX machine with Windows 8/8.1+ installed. We've got a custom pc spec'd out here for you to purchase and build if you're starting from scratch.

Make sure you have enough space in your partition or enough space on the drive that you are writing to though, as DepthKit "footage" can quickly eat up space. We recommend at least 320GB on an SSD drive. DepthKit footage writes at a high bitrate, so we reccommend testing your connection to your storage medium to make sure it's fast (hence the SSD). We recommend AJA's tool for testing, which you can download for free here.

Though the footage takes up more space when it's recording (it is compressed down after recording a raw file), you can expect DepthKit to have a data/time ratio of 762MB/min.

If using a PC, you'll also need Quicktime installed. You can download that here.

Computer requirements for DepthKit Visualize
For the visualization stage you can use either Windows or OS X with a non-integrated GPU. Does not support Intel HD graphics cards (typical of new Macbook Air/Pro) or graphics cards made before 2013. Visualize also requires Quicktime, which can be downloaded from here.

Sensor

A Kinect for Xbox One with a Kinect Adapter for Windows or Kinect for Windows v2 (which includes the Adapter.) We recommend the Kinect for Xbox One and the Windows Adapter, as the Kinect for Windows v2 has been discontinued.

Camera

A high resolution camera – the higher resolution the better – but 1080p is sufficient. We have used everything from a GoPro to an Arri Alexa. The biggest thing to keep in mind here is that your mounting needs will highly depend on the shape of your camera. We also consider some version of live-preview a necessity here, as you'll need to visually align the cameras during the calibration stage.

For most use-cases, we reccommend a full frame DSLR like the Canon 5D MKII/III with a wide angle prime lens. See more below.

Lens
We recommend a wide angle prime lens for filming with the DepthKit so that the field of view of the camera is close to that of the Kinect (very wide!). We recommend as wide as a 12mm lens on a typical full-frame camera if possible but it will still function with more common lenses. The closer you can match the Kinect's FOV, the better.

If shooting with a full-frame DSLR, we reccommend the Canon or Samyang 14mm prime lens.

 

Mount
A hardware mount or other method to securely bind the Kinect to the camera. We sell a simple mount kit that can be used to attach and adjust your sensor relative to your camera. 

Checkerboard

Print out an A3 calibration checkerboard PDF in black and white on matte paper. You can also choose to instead print an A4 calibration board, but as its squares are smaller, we recommend the A3. Glue or otherwise mount it to something flat and rigid like wood or foamcore. This can be done easily at most print shops – if you do it at home look out for bubbling or warping. It helps to attach a bracket or some way to put it to a stand or the wall.

NOTE: Be careful when you print out your checkerboard that you print it on a non glossy surface with non-glossy ink. Laser printers work the best for this, and photo printers (inkjet) do not. If your ink is too glossy, there is a chance you won't be able to properly calibrate your camera.

 

Planning Your Shoot

Filming with DepthKit is unusual in some ways but it has the benefit of not disrupting the conventional set dynamics. The DP still operates the camera and the DepthKit team can film alongside. To understand the intricacies & conventions of filming with DepthKit Capture on set see notes on filming.

Lighting

The technology that makes the Kinect work relies on the near-infrared spectrum for illumination. If you use lights containing a lot of infrared content it can disrupt the Kinect, degrading your depth capture. 

 

Unsuitable Light Types

  • Tungsten
  • Incandescent
  • Halogen
  • Direct sunlight

Recommended Light Types

  • Fluorescent tube lights
  • HMIs
  • LED Light Panels
  • Indirect sunlight

A rule of thumb is that any lights that are hot to the touch (including our sun) generally introduce some near-infrared pollution, making them not ideal for scanning. But we always recommend that you test your lights!

When considering how to light your scene for DepthKit we recommend considering your final design and designing your lighting for that context. For example, if I intend to place my subject so that they're illuminated from the left by a window in VR, then on set it's best if I put a large softbox to the left of my subject to mimic that light.

Sensor Specifications

Range
The Kinect has a range of around .5m and 4.5m (1'8"-14'6".) We also find that based on the brightness of the IR illuminators on the Kinect V2 that when objects are close to the near range they're often too bright for it to sense. The quality of the depth data falls off as you step away from the sensor, so the general recommendation is that you fill your depth frame with your subject whenever possible to get the optimal resolution.

Resolution
The Kinect V2 has a depth resolution of 512 x 424 pixels with a field of view (FoV) of 70.6° x 60° resulting in an average of around 7 x 7 depth pixels per degree. Due to the use time-of-flight as the core mechanism for depth capture, each pixel in the depth image contains a real measured depth value (z-coordinate). Read a more thorough breakdown.

Field of View
The depth camera on the Kinect V2 has a FoV of 70.6° x 60° and the color camera has a resolution of 1920 x 1080px with a FoV of 84.1° x 53.8°, resulting in an average of about 22 x 20 color pixels per degree. There is a great interactive Kinect 1 vs Kinect V2 FoV comparison tool here.

Framerate
We recommend setting the framerate on your camera to your local standard around 24p. The Kinect shoots at a variable framerate around 30fps and DepthKit conforms it to the framerate of the footage provided. We recommend shooting at 23.976 fps / 24p but it will work at any framerate below 30fps.

On-set Material/Garment Constraints

Avoid the use of transparent or reflective materials! This means glasses, mirrors, windows, shiny leather, waxed jeans, etc.

The Kinect relies on cameras and a small infrared emitter to sense depth, so it is susceptible to many of the same optical illusions that trick our eyes. It will struggle to resolve transparent surfaces like glasses or windows. It will also represent reflective surfaces in strange ways – for example if I film something in a mirror, the object will appear to be inside the mirror. It occasionally will struggle with certain fabrics or materials – leather or some kinds of waxed or otherwise treated jeans can appear to be missing. On critical shoots, always test your materials in the Kinect before shooting. 

When choosing patterns or colors in clothing, adhere to all the same constraints that govern a typical film shoot – small repeating patterns can cause moiré effects and if garments are too bright or too dark they will indeed appear bright or dark respectively when filmed with DepthKit.

 

Sensor Mounting

The Kinect must be mounted to and maintain the same relationship to the color camera during the whole shooting process, so make sure you have a strong mounting solution.

In order to calibrate and film with DepthKit you'll need to attach a Kinect firmly to your camera in a way that allows you to adjust the angle of view. We have designed a set of hardware mounts that allow you to attach the leading sensors to a camera – please purchase at left.

Mount kits include one aluminum base and a set of interchangeable arms that can accommodate the Asus/PrimeSense, Kinect V1 or Kinect V2 depending on the mount you buy. The sensor will need to be mounted to the camera for the duration of your shoot – if you change the relationship between camera and sensor you'll have to re-calibrate.

Note: we typically only ship mounts on Mondays. Don't forget to allow for ample time in customs for international orders. If you're on a tight timeframe or have any questions about mounts please contact us directly.

 

Choosing a space to shoot/Type of shot

When planning a DepthKit shoot, it's important to understand the environment you're shooting in before arriving.

DepthKit works best in locations with diffused or controlled lighting. Have a discussion with the team you plan to work with about the location and your requirements for a space. If possible, head to the space ahead of time and run a test shoot with DepthKit to see how it responds to the lighting and space.

On shots that are critical to DepthKit, it's also important to have a clear dialogue with the person running the color camera if it isn't you. A DepthKit shot requires a much wider lens than normal, so be sure to manage expectations and understand what is needed for a given shot from DepthKit's perspective.

 
 

PRODUCTION

lisbeth_wide.png
 

A DepthKit production is very similar to a standard production and can easily fit inside the flow of a shoot. The key difference is that there is a period of calibration that needs to be done before you can roll with DepthKit, so be sure to set an allotted amount of time prior to shooting where you properly calibrate and prepare the rig for filming. We recommend 30 minutes - 1 hour depending on how well versed you are in running a DepthKit shoot.

For a full overview of the process, check out the video below. You can also scroll down and find video documentation of each step individually.


 

Lens Calibration

In order to accurately calibrate the two cameras, DepthKit needs to understand the subtleties of the camera lenses – imperfect manufacturing and lens distortion mean that every lens will be slightly different. To sense these values for both the Kinect and color camera we capture and analyze images of a checkerboard seen from both cameras.

The calibration process begins by connecting your Kinect to your PC (or Bootcamp'd Mac) and loading up DepthKit Capture. If properly connected, you should see a live preview of the Kinect in the window on the left.

Set your Working Directory to a convenient place on your hard drive. All your footage will be stored here. Change it by clicking the text at the top of the DepthKitCapture window. The default directory is depthframes which is located inside of your application folder. Observe that the software creates a '_calibration' folder for you inside the project directory. The calibration files are automatically saved as you go and will get overwritten if you repeat calibration steps.

Select Calibrate Lenses tab, the left-most tab in the capture application. It is split in half vertically; your depth camera stream should appear on the left and the right pane should be empty to start. If you don't see the depth camera view, see troubleshooting page.

Note for Depth-only captures: You only need to self-calibrate the Depth camera.

Self-Calibrate Kinect Lens

Note the FoV's are symmetrical, and that the principal point is at the center of the depth camera's fixed 640x480px frame.

Note the FoV's are symmetrical, and that the principal point is at the center of the depth camera's fixed 640x480px frame.

Aim your rig at an area of space which has plenty of information visible to the Kinect - it needs variable depth in the scene. Do not aim it at a flat wall, aim it at a busy room full of things in the near, middle and background. Hit the Self Calibrate Depth Camera button at the bottom of the left-hand pane. This will automatically analyze your sample and once complete should display results similar to the results at right.

Calibrate Video Camera Lens

To capture the video camera's lens properties it takes a bit more effort and patience. Note that the DepthKit does not currently do camera control of any kind, so you will be operating your camera as you usually do and ingesting the footage. Choose camera settings that provide you with as much depth of field as possible - f5.6 or above is best and keep your ISO low if possible. Set your resolution to the desired final resolution for your project and your framerate to around 24p. 

Set your camera rig up on a tripod and point it at an open stretch of space you can move around in.

Place your checkerboard on a stand (or wall) in front of the camera, at a distance from the camera so that it occupies approximately 1/4 of the frame. If you need a link to download the checkerboards again, you can find the A3 checkerboard here and the A4 checkerboard here.

Be careful not to overexpose the image. It's better to under expose it by a stop or two than to blow out the white areas on the checkerboard.

Be careful not to overexpose the image. It's better to under expose it by a stop or two than to blow out the white areas on the checkerboard.

Adjust the tripod head so that the checkerboard is in the top-left quadrant. Don't worry if the checkerboard is not exactly horizontal or vertical, but do ensure that the entire checkerboard is in the frame, including the white border around the outside black squares. Make sure the board is well exposed (err on the dark side,) evenly lit across the surface, and that the camera is focused on the checkerboard for every sample. Refocus and confirm that all the corners are crisp for every sample.

Record a 1-3 second video of the checkerboard, mindful of keeping the camera very still. Repeat the process four times at this distance from the checkerboard, filling all four quadrants of the frame, top-left, top-right, bottom-left, bottom, right.

Repeat this process at a distance where the checkerboard occupies around 1/9th of the frame, refocus and take three rows of three images -- top-left, top-middle, top-right, middle-left etc. Take care to refocus every clip as the focus distance to the checkerboard changes as you til the camera.

Once complete, make a new folder called "RGBIntrinsics" inside the "_calibration" folder inside of the working directory you set in the first step. Download the clips you just took into the newly created "RGBIntrinsics" folder.

Set the Square Size (cm) of the checkerboard inside the application. For reference, use 3.38 if you have used A3 sized checkerboard and 2.54 if you used the A4 sized board. If yours is a different size, measure one square precisely and use that width.

Drag the "RGBIntrinsics" folder onto the 'Capture Lenses' tab's right-hand window (which should be empty). This should automatically start the calibration process. You may need to wait for a few seconds while this takes place; the application selects the middle frame from each video, converts it into a black and white PNG which is stored in the _calibration directory in your working folder.

Note: Older versions of the tool required dragging the clips themselves. This is no longer required. You can just drag the folder itself. Note as well that the calibration doesn't read subfolders, so all your video files should be directly in the "RGBIntrinsics" folder, not in a folder inside of that folder.

good_intrinsics.png

Once the analysis is complete, the software will display a 'Total Error' figure below the checkerboard images. This is the average error across all the calibration images - think of it as an overall grade for calibration. You can also view the individual error margins for each image by scrubbing the mouse from left to right across the calibration window. A Total Error of less than 0.2 is desirable.

If your calibration has resulted in a higher Total Error, scrub through your image set by moving your mouse left & right over the window and look for any outlier images which have an Error of higher than 0.3. Note the filename of any outliers. You can redo the calibration at any time by dragging the videos onto the window pane again - this time excluding the clips with high error numbers. This should improve your Total Error.

If nearly all of your images have high error, you will need to re-shoot them. Before you do this, look for elements in your environment which could have caused the error. Is there light streaking across your checkerboard? Were you not careful about focusing? Check the troubleshooting section for more reasons why you may be getting high error.

Congratulations, you've now sensed a mathematical model of your lenses. You can use this same intrinsic calibration the next time you shoot with the same lens/camera combination. Using this internal calibration we can next determine the external relationship between the two cameras.

DepthKit Calibration

After the color camera and depth camera are calibrated, you need to calibrate the relationship between the two of them in order to properly project the color information from the video camera onto the depth. To do this, we need to determine the physical position of the two cameras relative to one another. This allows us to combine the two data streams into one 3D scene.

Adjust Kinect Angle 

The DepthKit Calibration process captures the position & rotation of the two cameras. Before calibrating, check that the fields of view & alignment between the two cameras match.

Compare the live preview on your camera with the Kinect image in the capture app to make sure that the video camera and the Kinect are well aligned. To do this, we recommend picking an object in the room and aligning it to the center of the live-preview color image, then adjust the Kinect so that the object can be seen in it's preview mode in a location that matches the color camera's preview. 

To adjust the Kinect with our mount, loosen the adjustment screw on the side of the mount and tilt the Kinect until the center point of both the video camera and Kinect views match. When they're matched, tighten the screw down and make sure that everything is secure. If they don't match perfectly it will still work but if either your video camera or Kinect FoV are significantly wider than one-another, then you may have 'wasted pixels' in the areas that are not shared between both views and hence won't be able to be seen in DepthKit.

If you are using a custom mount, know that once you align the cameras you should not adjust them again at any point during the shoot or the calibration will be off. When using a custom mount, follow the above instructions to align the two views (depth and color) to have the same field of view, and be sure to "lock in" the camera alignments in the way that your mount allows.

Calibrate Correspondence

Now that the camera views are matched, we'll take corresponding images of the checkerboard from the Depth & Infrared views on the Kinect and video camera to determine where they sit relative to each other. It is important to make sure that no screws are loose or anything is wobbly in the mount at this stage. If the relationship between the camera and depth sensor change after we create this calibration, you'll have to repeat this step.

Navigate to the second tab in the Capture application, labeled Calibrate Correspondence. 

This is an example of a checkerboard that's too close to the Kinect such that the Kinect is over-illuminating the surface. In the depth image (left) the black areas represent regions without valid depth samples. Meanwhile, in the IR image (right) we don't see the red grid pattern that's needed for calibration. In this case either rotate the checkerboard away from the Kinect to reduce glare or move it back until the entire surface is all white/grey at left and the red checkerboard dots appear at right.

First, place the checkerboard 2.5'-3' in front of the camera and move it backwards until there are no black spots on the top left window. Any pixels that appear black are not being sensed. Typically that is caused by being too close to the Kinect and as a result the IR illumination is too bright.

Without moving the camera or the checkerboard, you'll next need to capture one image each of (L-R) Depth, IR, then a short video with the video camera to place in the right-hand slot:

  1. Hover your mouse over the left-hand box, double check that the surface of the checkerboard has no black spots on it and sample it by clicking inside the box.
  2. Without moving anything, hover over the middle box and make sure that you can see a grid of small red dots. If you can, click the box to sample the IR camera. 
  3. Without moving anything, check your focus on the video camera to ensure that the checkerboard is in focus and take a short (1-3 sec) clip with the video camera.

Move the checkerboard back 9", rotate it off axis a little bit, move it slightly off-center and repeat the steps above. Repeat these steps until you've filled all four rows. The goal is to sample a variety of areas around the frame with checkerboards at varying angles and varying depth.

 

You'll next need to ingest the color video you just recorded into the application itself.

 

Once you've captured all four sets, download the video clips from the camera and drop them into a new folder in the working directory you set before. One at a time, drag the video files into the empty  rectangular tiles in the application next to the corresponding depth and IR thumbnails taken from the device. Visually confirm that you haven't mixed them up. Note that if you don't see a clearly defined checkerboard in the viewer on the right, your ink may be too glossy and you need to print with a less glossy ink. This is a known issue.

With four sets of three images complete, click 'Generate RGB/Depth Correspondence'. If you get an error it means the algorithm was unable to find an exact fit for the selected checkerboard pairs. Try hitting 'ignore' to exclude one of the image sets - 'bad apples' may be throwing off the model. Typically it's advisable to start by discarding the furthest or most obliquely angled set and clicking 'regenerate.'

You can navigate in the window at right with the following controls:

Hotkey   Function
W        Move Forward
S        Move Backward
A        Move Left
D        Move Right
E        Move Up
C        Move Down
Q        Rotate Counterclockwise
R        Rotate Clockwise

Just like before, excluding images may help in this situation. Click 'Ignore' on all but one of the images, and attempt to Generate RGB/Depth Correspondence again. When you find an image that allows the process to complete successfully, try combining it with other images. There is some randomness in the algorithm, so it helps to try the same combinations a few times just to see if it 'guesses' a better starting place.

By pressing the left and right arrows you can cycle through previews of the four checkerboard calibration sets. If it's correct, you'll see the checkerboard image data pixels (in black and white) mapped cleanly onto the depth model of the same image.

You'll also see corresponding colored dots floating in space corresponding to the checkerboard depth planes. Some dots are likely to be missing from the grid pattern, as they were removed as outliers while generating the calibration. An ideal calibration will contain dots from at least three different sets of colors. By cycling through all tests checkerboards sets, the checkerboard image should be visibly well aligned to the depth data.

One note about the Kinect V2: currently for some reason we've been seeing far fewer matching dots on this final view but it turns out that the calibration is more than satisfactory. If you have at least a few dots from most if not all the checkerboards in the scene, we recommend saving the calibration and shooting a test clip to visually assess whether the calibration is good or not. 

 

Once you're here, congratulations! You're all calibrated and ready to go, with the hardest part of using DepthKit behind you.

Capture

After going through the calibration process, you're good to go to shoot! Running DepthKit on set is simple and can easily sit alongside another productions workflow. There are a few things to keep in mind though, which we've outline below.

Navigate to the Record tab in the Capture application.

If connected properly, you will be able to see a preview of your depth camera on the left. Your takes will show on the right side in the format 'TAKE_MO_DD_HH_MI_SS', with the numbers corresponding to the time of capture (there will be none before you've captured your first take).

NOTE: You are just "running" the depth camera. Unless you are also in charge of running the color camera, you'll only be performing operation for the depth sensor, managing when it's capturing and when it isn't. If you're working with a Director of Photography, communicate this with them so they know they can operate the color camera as usual once you've calibrated everything.

Notes on Filming

Data Constraints
It is worth reiterating here that we highly recommend you use a computer with an internal SSD that you're writing to that can handle the image bandwidth of DepthKit recordings. You should also expect files to take up space at a rate of 1GB/minute of capture, so plan your harddrive space accordingly.

Set your clocks
After the shoot when you're correlating the depth & color takes it will help you a lot if you have properly synchronized the clocks on the camera and the computer.

You're tethered
When planning your shoot, be aware that your camera operator will not be able to move further from the computer than the depth camera's USB cable will allow. This distance can be extended with a specific active USB extender if needed. Beyond ~16ft you'll need a specialized active USB 3.0 extender. Always test your extenders before your shoot.

Camera control
DepthKit controls your depth sensor but not your camera. You'll be controlling the video camera as you normally would and all the standard best practices apply; keep your subject in focus, expose your scene properly and make sure that you're using proper settings to stop motion and keep the important things in focus. 

Synchronizing (Clapping)
Each take requires a visual signal to synchronize the depth and color. Have someone ready to stand in front of the DepthKit rig and clap at the beginning of the each take. Ideally use a clapper, but using your hands can work fine. The principle is the same as synchronizing external audio recording, do it at the beginning of the take after both the Kinect and video camera are rolling.

Be sure to also clap/sync in a space that has a fair amount of space behind an in front of it, as you'll need to be able sync the depth data later on and if it's too close to another object depth-wise it can be hard to discern the proper sync point.

Rolling
We typically follow this convention on set to avoid confusion:

  1. Video DP announces "Rolling color!" and starts recording on the video camera.
  2. Data DP announces "Rolling depth!" and hits the spacebar in the Record tab of the Capture application – alternatively hit the 'Toggle Record' button. The depth camera preview window should display a red border with a flashing Record button to show that recording is taking place.
  3. Clap in front of both cameras for sync.
  4. Make three dimensional magic.
  5. Hit space bar again to stop recording.
When this bar fills up with red it will start dropping frames. This can happen if you have slow write speed on your disk (not an SSD,) your HD is full or issues with connectivity to the Kinect. If this happens, stop shooting and fix the issue. 

When this bar fills up with red it will start dropping frames. This can happen if you have slow write speed on your disk (not an SSD,) your HD is full or issues with connectivity to the Kinect. If this happens, stop shooting and fix the issue. 

When you stop recording you should see your TAKE appear in the right side of the Record tab. If you're recording and you see the 'Toggle Record' button starting to fill with red, it means that some images are failing to save to disk. This usually occurs when the hard drive is nearly full. If this happens, stop recording as soon as possible and wait for the red bar to go down, ensuring that all the footage is written to disk. If you're recording to an external drive, make sure it's an SSD with FireWire, USB 3.0, or Thunderbolt (but we don't generally recommend recording to an external drive.)

A Note on Compression
As soon as you finish recording, the application will begin compressing the raw depth data in the background. The progress of this process is shown as a gray bar overlaid on the TAKE description on the top right. You can close and reopen the app and it will pick the conversion up again. The files will compress down to about half their size, so if you're nearing the end of the shoot but are running out of harddrive space, wait for some of the prior recorded takes to finish compressing to free up some space on your drive.

Previewing Recorded Takes
It's possible to preview the recorded depth data inside of the capture application once a take has finished compressing. In the playback tab, select a Take and hit space bar to play the timeline. Use the same controls from the calibration step to navigate the point cloud as your capture plays back.

 

Post-Production

Screen Shot 2016-06-27 at 2.52.18 PM.png

Footage Preparation

After a shoot, you'll need to copy the video files into the working directory. For each file you recorded, find the corresponding TAKE_ folder. Having the camera's clock set correctly is helpful so you can match the time stamps between the footage and the folder names. Within the take directory, put the video file into the TAKE_/color folder.

A sample Workspace folder with a single TAKE inside of it.

The working folder contains a folder called _calibration and folders for each of the TAKES you captured in the previous step. In order for a TAKE to be valid, the /color folder must contain a video.

Visualize doesn't support all video formats and codecs, so we recommend that you transcode your files to a standard high resolution format. If you're on PC, we recommend you encode to a high-bitrate Motion JPEG or h264 (~210000 kbps). If you're on OS X, we recommend using Apple ProRes (APCN). For your convenience we provide an Adobe Media Encoder preset here.

The Visualizer allows for the option of having a proxy video; a smaller version of your video that makes visualization fast without sacrificing quality at export. To create an offline video we recommend using Handbrake to create a low res proxy at 640x360 (assuming you shot in 16:9 aspect ratio. If not make sure to export the correct picture size.) We supply a preset that makes this process easier, download here. Add _small.mov to the end of the filename and put in the Color folder. This is the clip that DepthKitVisualize will load for preview, it will be swapped out automatically for the large file when rendering.

Once your files are in place inside their color folders, you're ready to Visualize. It you're having problems with this step, check out the Troubleshooting section.


Visualize

The next stage is to merge the depth and color and either render video or export your footage for either realtime or VFX tools. Find hotkeys here.

Workflow

The workflow in Visualize varies depending on your chosen export target, but the general workflow has you using the takes you recorded during the capture stage to generate compositions, or "comps," that can be exported to your target tool. Syncing is done inside of the comps so you can temporally align the depth data with the color data — a process that needs to be done because it's likely you didn't start rolling the color camera at the exact same time you rolled the depth.

Every take can have any number of compositions, so we usually recommend syncing a take with a "Sync" comp, then using that as your base comp which you can then duplicate inside Visualize to then start working with your footage in a new comp. This way you do not have to continually sync your footage for every new comp you choose to make.

The following tutorials will take you through the process of navigating Visualize as well as syncing your footage to get it ready to export to different platforms.

Interface

The DepthKit Visualize Interface

The main DepthKit Visualize interface has four main components. The File manager at the top, the Settings on the left, the Scene Viewport in the middle (where you see your footage), and the Timeline at the bottom.

The File Manager gives you the ability to:

  • Change your workspace by clicking the current workspace path.
  • Save you current composition by clicking "Save Comp"..
  • Change which comp or TAKE you're working on by clicking the TAKE's name.
  • Make a new comp using the settings of the current comp by clicking "Copy to New."

 

The Scene Viewport allows you to view DepthKit footage in 3D and set camera positions for image sequence renders.

  • Move the camera in 3D space with W,A,S,D.
  • Rotate the camera to the left and right with Q and R respectively.
  • Use E and C to boom the camera up and down.
  • Click and drag the mouse to change the direction the camera is facing. 

 

The Settings menu give you access to the bulk of the editing/rendering settings for Visualize. You can move the settings panel around by clicking and dragging on the top bar.

Sync Settings

  • Temporal Alignment: Click this to jump to the mode where you synchronize the depth and color tracks to each other.
  • Set Color-Depth Time: When in the Temporal Alignment mode, click this to lock in playback on both tracks to a single time code. For more on this, read the "Synchronize" section below. Do not press this button multiple times, if you do select the extraneous keyframes and press Delete/Backspace.

3D Camera Settings

  • Camera Speed: Set how fast you move through the scene with the W,A,S,D keys.
  • Camera Roll Speed: Set how fast the camera rotates with the Q and R keys.
  • Reset Camera: Reset the 3D camera back to it's starting point. Useful if you get lost in the scene or if you can't find the subject.
  • Lock to track: When keyframing camera positions for the image sequence export, click this setting (shortcut Shift+L) to make sure the camera moves along the keyframed path. For more on this, see the section on image sequence exporting.
  • Set Camera Point: Use this (shortcut Shift+T) to set a keyframe for a camera's location and orientation on the Camera track in the Camera tab. These points determine how a camera will move for an image sequence export. For more on this, see the section on image sequence exporting below.
  • Mirror: Use this to horizontally flip the image in the display window.

Depth Data Settings

  • Freeze Frame Mode: Allows you to stop motion at the playhead. Using this mode allows you to create camera movements or effects around on a single frame from your capture.
  • Fill Holes: Toggles hole filling, plugging in holes in your capture data. This setting can create artifacts if used too liberally. See Hole Kernel Size & Hole Fill Iterations below – ideally use a balance between small kernel size and a moderate number of iterations (eg: 1, 3.)
  • Hole Kernel Size: The size of the 'plug' used to fill holes.
  • Hole Fill Iterations: The number of attempts using the size of the Kernel. 
  • Subtract Background: Allows the user to subtract a background plate from the scene based on one chosen with the "Capture Background" option below. See more in the "Background Subtraction" section below.
  • Capture Background allows the user to state that the current previewed frame is the blank background that can be subtracted.
  • Background Threshold slider controls how "strong" the subtraction is.

Rendering Settings

Note: These settings mainly control effects that will only be rendered if you choose to do an image sequence export. Rending .OBJs, .ABCs, or Combined Per Pixel will export geometry only and will not export these effects.

  • Draw Point Grid: Draw an array of points on a grid that match up to the DepthKit images vertices. In other words, render a gridded pointcloud. You can control this effect in the Timeline section labeled "Rendering."
  • Draw Points Random: Draw a random array of points to the DepthKit mesh allowing you to break up the uniformity of the point grid. You can control this effect in the Timeline section labeled "Rendering.
  • Draw Wireframe: Draw a dense wireframe that represents the whole DepthKit mesh. You can control this effect in the Timeline section labeled "Rendering."
  • Draw Vertical/Horizontal Lines: Draw horizontal and/or vertical lines across the mesh. This is like a simplified Wireframe view. You can control these effects in the Timeline section labeled "Rendering".
  • Draw Mesh: Draw the colored Mesh, textured directly from the color camera.
  • Self Occlude: Make it so the DepthKit footage isn't "transparent." When you're looking at the model as a wireframe view or a point cloud this setting makes foreground elements opaque.
  • Depth of Field: Render Depth of Field effects. You can control settings in the Depth of Field tab in the Timeline.
  • Wave Distort: Turn on a "Wave" effect that makes the DepthKit footage look like a wave is running through it. You can control this effect in the Timeline section labeled "Waves."
  • Warp Distort: Turn on a warp effect that warps the mesh in space. You can control this effect in the Timeline section labeled "Warp."
  • Frame Width/Height: Set how wide/tall the frame should render to be. This will also update the window size in the scene window.
  • Render 720p/1080p: Use the 1080p or 720p render size preset for rendering image sequences. For more on this, see the section on image sequence exporting below.
  • obj Files/Textures: Check these boxes if you want to render an OBJ sequence (with or without textures) of your DepthKit sequence. Note that that any effects or camera moves you've made in Visualize will not render. Read more about this is the section below about export for VFX tools.
  • abc Files/Textures: Check these boxes if you want to render an Alembic file (with or without textures) of your DepthKit sequence. Note that that any effects or camera moves you've made in Visualize will not render. Read more about this in the section below about exporting for VFX tools. 
  • Combined Per Pixel: Check this setting if you want to render for importing into Unity. This will render at the size of the original depth map. For more on this, look at the "Exporting for Unity" section below.
  • Combined 720p/1080p: This format is not currently supported in the Unity workflow, it's designed for users who design custom import workflows. This will render at a preset size of either 1080p or 720p with a depth image on the right. For more on this, look at the "Exporting for Unity" section below.

 

The Timeline is how you navigate your footage temporally, add camera keyframes and keyframe the values presented in the settings menu.

Using the Timeline

  • Navigation is done by simply clicking on a tabbed heading.
  • To add a keyframe to a track, simply click the space in a given track and a keyframe should appear.
  • To delete a keyframes, click a keyframe and then press "Delete" on your keyboard.
  • To adjust keyframes, you can either click and drag, or use the arrow keys on the keyboard for more fine-tuned adjustment.
  • For further Timeline hotkeys see Hotkeys Glossary.
The striped green bar appears on hover to indicate that section can be resized. In the above image, the section under "Simplify X" is being hovered.

The striped green bar appears on hover to indicate that section can be resized. In the above image, the section under "Simplify X" is being hovered.

Note if you are low on screen space, you can mouse to the bottom green bar of any section of the timeline (it should be highlighted with green slashes when you do) and click and drag it up to "hide" that section of the timeline. To access it again you just drag the corresponding bar down to reveal it.

Timeline Section Overviews

  • Camera: This is where you can navigate footage temporally after it has been synced. You can also set keyframes for camera positions in the Scene Viewport here by either clicking "Set Camera Point" in the Settings or by using the shortcut Shift+T. The Camera dampen track smooths the camera's motion between keyframed sections a given amount.
  • Geometry: Here you can adjust how much geomery is visible in the scene as well as how it is depicted. For more on this, check out the "Trimming Geomerty" section below.
  • Clipping: This is where you can isolate your subject in your composition by clipping the edges of the geometry, as well as clipping the depth image in general. For more on this, check out the "Trimming Geometry" section below.
  • Rotation: Here you can keyframe discrete camera rotations that will be used in playback and rendered if you are doing an image sequence export.
  • Rendering: Here you can find the settings for how the pointcloud, wireframe, and mesh are rendered if you are doing an image sequence export. For these to work, you need to have at least one of the pointcloud, wireframe, or mesh settings selected from the settings menu.
    • Point Alpha controls how transparent the points are when either "Draw Points Random" or "Draw Point Grid" are selected from the settings menu.
    • Point Size controls how big or small the points are when either "Draw Points Random" or "Draw Point Grid" are selected from the settings menu.
    • Random Point Amount controls how many points are generated when "Draw Points Random" is selected from the settings menu.
    • Wireframe Alpha controls how transparent the wireframe is when "Draw Wireframe" is selected in the settings menu.
    • Mesh Alpha controls how transparent the mesh is when "Draw Mesh" is selected in the settings menu.
  • Scan Lines: When either "Draw Vertical Lines" or "Draw Horizontal Lines" is selected from the settings menu, you can can edit their respective properties here.
    • Horizontal/Vertical Scanline Alpha controls how transparent the respective lines are.
    • Horizontal/Vertical Scanline Thickness controls how thick the respective lines are.  
  • Waves: When "Wave Distort" is selected from the settings menu, you can control its respective settings here.
    • X/Y Sin Amplitude controls how "large" the waves are.
    • X/Y Sin Speed controls how fast the waves move across the mesh.
    • X/Y Sin Frequency controls how "many" waves there are at a given time.
  • Warp: When "Warp Distort" is selected from the settings menu, you can control its respective settings here.
    • Warp Amplitude controls the "strength" of the warp.
    • Warp Density controls how rigid or fluid the warp should be. Higher values mimic fluids, lower values mimic solids.
    • Warp Speed controls how the mesh is warped by the above warp settings.
    • Warp Stretch X/Y/Z controls how much the warp should warp the mesh in a given dimension.
  • Color: Here you can set a background color as well as key global color fade setting for your points of mesh.
    • Fade Amount controls how much you fade from one fade color to another. Note that no Fade Amount means that no Fade Color is applied.
    • Fade Color is the color you are trying to fade to. Note that this fades everything to that color.
    • Background Color allows you to set a background color. This is also useful in syncing as it lets you more easily isolate depth and color streams. Note though that the color is rendered out if you do an image sequence export.
  • Depth of Field: When "Depth of Field" is selected from the Settings menu, you can control its effects here. This simulates a normal cameras Depth of Field effect.
    • DOF Distance can be thought of as where the camera is focusing in space. 
    • DOF Range is the amount of space that should be "in focus." Manipulating this is like manipulating a camera's aperture.
    • DOF Blur is how much blur should be given to an "out of focus" object.
  • Synchronize: This is where you can synchronize your depth and color streams. For more on this, see the "Synchronize" section below.
  • Texture Alignment: This is where you can better align your color stream to your depth data to offset any calibration errors that happened during the capture stage. For more on this, see the "Texture Alignment" section below.

 

Synchronize

The first step when working with any DepthKit TAKE is to sync the color video to the depth data stream so that the two will play back in unison, with the color properly matching to the depth.

 

Creating the Sync Comp

Launch DepthKit Visualize and point it to the top-level folder that contains your TAKE folders.

Note: Point to the top level folder that contains your TAKE folders, not to a specific take. If you point to a specific take, Visualize will tell you that there are "No valid takes."

When you open a valid Workspace, you should see something that looks like this. Your valid takes will be listed on the left, and after clicking one, you should see "Create New Composition From This Scene" listed next the the TAKE like in the above image.

When you open a valid Workspace, you should see something that looks like this. Your valid takes will be listed on the left, and after clicking one, you should see "Create New Composition From This Scene" listed next the the TAKE like in the above image.

Once you point to the proper directory, you should see the list of TAKEs that you recorded. Only takes with videos of a recognized format inside the /color folder will show up, so if you don't see one of your TAKEs, make sure you've added the video to the folder. Select the TAKE you'd like to visualize and click 'Create new Composition with this Scene.' If you have issues with this stage see troubleshooting below.

FOR DEPTH ONLY CAPTURES:
You'll need to generate black video in either a video editing program or FFMPEG that is the same length of your captured depth sequence. If you do not have a video file in in the "/color" folder, Visualize will not recognize your captures as a valid Take.

 

Syncing the Depth to the Color

Assuming it's your first time loading this scene you'll need to synchronize the depth & color streams. On the timeline at the bottom you should see two rows of thumbnails: color & depth. Scrub through them by clicking on the line of thumbnails and dragging left or right. You'll see previews of the depth data and the RGB video as thumbnails on the top right. With your mouse over the respective timelines can also use the arrow keys until you find the synch moment in both Depth and Color (usually a clap).

Confirm that Temporal Alignment mode is selected at top left and that there are no existing keyframes in the bottom timeline. Click & drag (or left/right arrow keys) in the depth & color timelines until you've identified the synch frame. Then press Set Color-Depth Time at top left.

Confirm that Temporal Alignment mode is selected at top left and that there are no existing keyframes in the bottom timeline. Click & drag (or left/right arrow keys) in the depth & color timelines until you've identified the synch frame. Then press Set Color-Depth Time at top left.

Once you have the precise moment of the sync selected in both preview windows, press Set Color-Depth Time in the Settings panel. Selecting it should automatically toggle off Temporal Alignment mode. Now if you scrub through the timeline by clicking on the bar with the yellow tickers in the "Synchronize" tab, you should see the combined DepthKit footage playing back in synch.

 

Correcting Sync Errors/Restarting Sync

If you make a mistake and find that the synch is incorrect, you can always navigate back to the Synchronize tab, select the blue sync marker in the timeline and press DELETE to start over. It's NEVER necessary to set multiple synch points, so avoid duplicates by delete any preexisting markers.

After you think you have a good sync, you can fly around in the scene (see camera Hotkeys) to check how good your sync is. Remember, if you get lost you can use the "Reset Camera" button at the top of the Settings menu. You can also play back in this mode (Spacebar) and watch your footage and see if the sync seems off.

Texture Alignment (Fine-tuning Calibration)

After syncing, it's usually necessary to tweak the color texture alignment in how it falls on the depth data, even if you had a good calibration during the Calibration step.

Navigate to the Texture Alignment tab on the far right to adjust the placement of the color on the depth. To fix any alignment errors, you'll need to subtly tweak the values of the given sections:

The Texture Alignment panel

The Texture Alignment panel

X/Y Texture Rotate will rotate the texture "projector" so the effect will be more noticeable at at a distance.

X/Y Texture Translate will shift the texture "projector," meaning that that the change will be equivalent in the near ground and the background. 

X/Y Texture Scale will scale the projected texture, stretching it out in the corresponding direction to better cover a given depth map.

If you find that you've offset your textures too much and need to start over, remember that you can delete any keyframe by clicking it and pressing "Delete" on your keyboard.

After you tweak these values, it's always worth playing back your footage and making sure the tweaking you just did still properly aligns the footage further down the timeline. It's very easy for you to fine tune a specific shot, but then find in playback that it ruins the rest of the footage. Instead of trying to find the perfect alignment for any given frame, we recommend finding a single good in-between alignment that plays best for the duration of the shot and not try to do a lot of individual keyframes for the shot's full duration.


Making other Compositions

After you properly sync and align your first take, you can duplicate it to a new take that you can freely work in without fear of messing up your sync. To duplicate a take, click "Copy To New" at the top of the screen and name your take something descriptive that gets at what you want to do with it.

Remember, each Take can have any number of compositions, so feel free to experiment. Composition let you render the scene in a different way, with a different in and out point, changing camera perspectives and different rendering styles.

Trimming Geometry

Right after syncing the first thing you'll likely want to do is trim out some unnecessary object/information in your scene. This can be done inside of Visualize and help your to export a relatively clean model to the program you're targeting.


Background Subtraction

Visualize supports the ability to intelligently subtract the background of the scene with the "Background Subtraction" toggles in the Settings menu. To subtract the background, click "Capture Background" on a blank shot where your subject isn't in the frame. To subtract the background, click "Subtract Background". The program will then cull the background from the scene based on the "plate" that you instructed it to use. Use "Background Threshold" to control how "strong" the subtraction is. Note: This is an experimental feature. 

The Geometry Tab

The Geometry Tab

Visualize also allows you to trim out geometry behind or in front of your subject by using the options inside of the Geometry tab.

Z Threshold Max trims out geometry starting from the rear of the scene, meaning it's usually the first place you'll want to go if you find that you're seeing objects in the background you don't want to be present in your export.

Z Threshold Min trims out geometry starting from the front of the scene. This is used if something comes between you and your subject and you need to remove it from your scene.

Simplify X/Y allows you to create more contrast between the positive and negative space in your scene by increasing or decreasing the resolution of the underlying geometry.

 

Removing "Streaks"

The Clipping Tab

The Clipping Tab

When the texture is projected onto the depth, it's very likely that it isn't always a perfect match due to a variety of factors. This means that in literal "edge cases", you will often have parts of a texture "streak" towards the back of the scene. Some people find this "look" desirable, but if you're looking to remove it, navigate to the Clipping tab.

Edge Clip is used to trim the edges of anything in the scene, allow you to reduce the amount of "streaking" that occurs for a given object.

Top/Right/Bottom/Left clip allow you to clip the DepthKit image in the stated direction any given amount. This a good solution if you're looking to isolate a given object for export.


Using Visualize FOR EXPORT

In addition to the ability to "edit" recorded DepthKit footage, Visualize allows users the ability to export to a number of formats that allow for further editing/exporting.

Current supported export formats:

  • PNG image sequence with or without transparency (to be combined into a video).
  • Textured .OBJ file sequence
  • Textured Alembic (.ABC) sequence
  • Combined custom per pixel video file (for Unity importing)

How to Render from Visualize

When you render out of Visualize, you have two "choices" about what will be rendered: How much will be rendered and what format will be rendered.

Designate a whole comp to be rendered
By default, DepthKit will render the whole comp. 

Designate a segment of a comp to be rendered
On the "Camera" tab of the Timeline, use Shift+T to set a keyframe in the track. Use Shift+T again else where on the track to set another keyframe. When the comp is staged for render, the render will start at the earliest keyframe it finds in your camera track, aka the "In" frame. Rendering will stop at the last keyframe it finds in the track, aka the "Out" keyframe.

You can set up more than two keyframes in a comp (helpful for complex camera moves), but when rendering, DepthKit will always render between the earliest keyframe and the last keyframe.

 

Rendering in Visualize

Rendering with these selected settings would export a 1080p .png sequence, along with the Combined Per Pixel render necessary for Unity integration.

Rendering with these selected settings would export a 1080p .png sequence, along with the Combined Per Pixel render necessary for Unity integration.

In the "Export" setting of the "Settings" menu, select the export option of your choosing. Note that you can select multiple options to render multiple types of files with a given export.

To export with the selected export settings, click on the composition name in the "File Manager" to go back to the menu where you are able to view all of your Takes and their corresponding comps. Notice the small 'R' (Render) button next to each comp.

To stage a composition for rendering, click the "R" next to the comp you'd like to render. When you click the "R," you will see the composition added to the render queue on the right side of the screen. You can remove a comp from the queue by clicking the small [x] next to it. Note: If you make changes to that comp you will have to re-add the comp to the queue by clicking the Take in the render queue and re-adding the comp.

Once you've selected all the compositions you wish to render, click the "Start Render Batch >>" button and sit back and relax as all the compositions you've queued up begin to render. Be careful not to press the spacebar - it cancels your render.

Renders are saved into the _RenderBin folder in the main workspace folder (next to _calibration). Each rendered folder is stamped with the scene name it was rendered from, as well as the composition name and the date of export, which prevents renders from ever being overwritten.

 

Visualize Export Options

Export as Image Sequence (Video)

Choosing to export  as an image sequence lets you leverage the bulk of the effects possible inside of Visualize by rending them out as image frames.

Visualize exports an image sequence by default, and as such most functionality within in the tool is built for this type of export. Exporting for video works by rendering a given composition from the perspective chosen in the camera track (what you'll see in "Lock to Track" mode in Settings) into a series of PNG frames.

Frames render at 1920x1080 by default but can be changed on the right-hand side of the Settings in the Visualize app by changing the Frame Width & Frame Height Sliders. There are presets for 720P & 1080P built in below the sliders.

The image sequences will be rendered out with the black background as transparent using an Alpha channel for compositing. However, if "Draw DoF" is checked it will remain black or if you've changed the background color it will not render with an alpha channel.

Designing Shots for Export

Camera Moves

What's a video without camera moves? To create a camera move, navigate to the "Camera" tab of the Timeline, and then scrub to a place in the timeline you'd like to start your move. Adjust the camera in the Scene Viewport to reflect the angle/orientation you'd like for the camera to have at that point. 

Once you find a position you like, either press "Set camera point" on the GUI or press SHIFT+T to create a keyframe at that time and location. You'll see the keyframe you created represented by a blue marker appearing on the Camera timeline. You can then scrub to a different place on the camera track, move the camera to a new location, and press SHIFT+T again to create another keyframe. In doing this, you're setting "In" and "Out" points for your camera move, declaring where the camera should start, and where the camera should end.

To see the actual camera move you just designed, select "Lock to Track" (or press SHIFT+L) on the Settings panel, then press Space to play. You'll see the camera traveling along a path your just described with your two key frames.

Other Keyframe Actions

  • You can click and drag the camera points to change their location in time.
  • You can delete them with the Delete or backspace key.
  • Right-clicking on a keyframe allows you to select different tweening methods between keyframes.
  • By clicking on the triangles to the left or right of the marker and pressing the up or down arrows you can have the camera movement ease in or out represented by a curved arc, or cut (blocky steps.)

 

Depth of Field

Example of Depth of Field effect

Example of Depth of Field effect

The Depth of Field effect can be convincing for simulating a camera language. To get started, select "Draw DOF" from the Settings menu. Next, navigate to the "Depth of Field" tab in the Timeline. You can adjust the Depth of field setting by setting keyframes and values in the tracks in this tab.

Depth Distance in the Depth of Field tab allows you to change the focus plane is in relation to the camera - think of this like turning the focus ring on your camera.

Depth Range changes how large your focal plane actually is. Think of this like adjusting the aperture.

Depth Blur controls how much blur should be applied to out of focus objects. 

Depth of field keyframes matched to camera moves can simulate racking or tracking focus. The white rectangle on the top right acts as a focus-assist by marking the area in focus in white and previewing the focus range.

 

A Few Tips

  • The rendering styles are screen-blended onto one another. So you can get a lot of mileage out of combining points, wireframes and meshes together & varying opacities.
  • If you don't feel like you can get the aesthetic you want, you can duplicate comps and render out multiple passes allowing you to blend in an external compositing tool like After Effects.

Exporting Geometry for Visual Effects Tools

 
 

In order to export textured OBJ files toggle Export .obj Files on the left-hand panel then render as usual. This will render out a sequential series of .OBJ files with .MTL files and matching .PNG frames to the _RenderBin/ folder.

It will using your first camera keyframe as the in point and the last keyframe as the out point for the render. Keep in mind that there is no camera tracking or perspective information embedded in these sequences as they are simply textured 3D files, but it does still use your first and last camera points as in & out points for the render. 

Follow the same procedure for exporting Alembic (.ABC) files. They will render out a single geometry file that includes all your frames, accompanied by a series of PNG textures.

Exporting For AND Working IN Unity (+ Hololens)

 
Working with DepthKit in Unity is a two part process. First, you must properly prepare and export your footage from Visualize and then import it into Unity using our supplied Unity package. You can download the Unity plugin below.

You can follow along with the videos below for an overview of this process, or simply scroll past them to read the text documentation.

The password for the tutorial videos is "dkunitytutorial". Use the button below to download the tutorial package.


Preparing Your Footage In Visualize

When exporting for Unity, it's important to know what exactly will be exported, as it effects how your capture is rendered in Unity. The export setting, Combined-Per-Pixel, packs all of the data from your scene into a single conformed video file. This means that any extraneous detail you have not gotten rid of will also be exported, often decreasing the overall quality of your export.

To get the most out of your data, you'll need to adhere to the recommended workflow for preparing your footage for passthrough:

1. Identify the subject of your export. Figure out what exactly you are trying to export for Unity playback and get a feel for what needs to stay in the scene and what needs to go.

2. Switch your export setting to "Combined-Per-Pixel". You'll see a preview of what your export looks like. This preview will also guide you through the following steps, as you'll be able to see how your changes affect the geometry.

3. Subtract the background & foreground in the Geometry tab. By subtracting the background and foreground, you'll be able to easily isolate the target of your export. You'll want to clean away the foreground and background of you subject so that you can't see anything in front of or behind them.

You should also be careful to not cut off parts of your image. It's very easy to naively subtract the foreground and background and consider it "good", but be sure to track through your clip and make sure your subject doesn't step outside your target range for any reason. A good subtraction at the start of your clip doesn't necessarily mean a good one all the way thorough. If you run into places where your plane is too tight, adjust and key the subtraction accordingly at that moment to better fit your image.

Notice here how a rainbow of color exists across the subject, with no visible background. This is good! This is what you should try to get your exports to look like, as it ensures you're packing the maximal amount of depth information into your exports.

Notice here how a rainbow of color exists across the subject, with no visible background. This is good! This is what you should try to get your exports to look like, as it ensures you're packing the maximal amount of depth information into your exports.

You can see here how the subject is almost completely a single color. This is bad! This means that the whole body exists at one depth, meaning your export will be almost flat.

You can see here how the subject is almost completely a single color. This is bad! This means that the whole body exists at one depth, meaning your export will be almost flat.

It's important to try and identify the tightest subtraction possible without clipping into your subject, as a tighter "hold" on the capture will generate a more accurate and data-rich model. To identify a good "hold", look at the export preview and try to generate a strong color gradient across your depth image vs. having it just be a single color. Different colors mean different depths, so try to squeeze the whole spectrum into any data you export.

Note: Depending on your delivery platform, it may be more less lenient on the color encoding of the depth. While depth that is "red" is acceptable in some cases, if you find that, once in Unity, your clips are clipping off depth that you see as visible in Visualize's exported video, try being a little more lenient, making your depth not push too far into red and consider orange your boundary.

4. Clip off the sides or bottom as needed in the Clipping tab. As a final step, clip the sides of your image to better isolate and target your subject. This doesn't affect the final resolution of the export (you're clips won't change aspect), but it does help to make sure you're keeping as tight a frame as possible.

Once you've done all these steps you can queue your video for rendering, and export out like any other type. Like the movie sequence export, you'll need to convert your frames to a video sequence that will be read into Unity. To do this, we recommend Adapter, which is an easy to use GUI-based interface around FFMPEG and doesn't require any coding. As an alternative, you can use raw command line FFMPEG, Premiere, After Effects, or Quicktime Pro.

 

Working in Unity

DepthKit provides easy integration into Unity, and does so through support for both native players and 3rd party ones. Read on to find out how to best implement Unity integration for your project.

Once you're prepared your video, you're ready to get started in Unity. A DepthKit Unity project requires a few things:

  1. The DepthKit Unity package, downloadable below.
  2. A Combined Per Pixel export of a clip that has been exported from Visualize, along with it's corresponding meta.txt file and poster frame (usually the first exported .png from Visualize for the corresponding clip export).
  3. Depending on your targeted delivery platform, you may need a 3rd party media player. See the chart below to figure out which media player best fits your needs:

Notes on Unity Movie Players

As you can see above, Unity's native MovieTexture doesn't play on mobile. While SpriteSequences will play, they will make your final package delivery very large, so aren't wholly recommended. They are a good fallback though if you have no other option.

AVProVideo and MPMP are good options if you desire more functionality or flexibility from your video players. Both allow you to track to specific points in your video for playback, and AVPro even allows you stream from a server!

When you import a player, be sure to not place it's contents in one of Unity's "Protected Folders," as this will most often cause the player to break. Just import it as is, and you should be fine.

If you're interested in having us support another player, please reach out!

 

Getting Started in Unity 

After you've determined if you need to download another media player or not based on your needs, start a new Unity project, then import our package and any movie player you may need by going to Assets > Import Package > Custom Package. You should see a list of components you want to import. You'll need all of the package's contents, so go ahead and import everything.

Once the package is imported, you're ready to go! Navigate to Assets > DepthKit > Prefabs, and drag in the your platforms corresponding prefab into your scene. If you're targeting a standalone build, use the standalone prefab, and if you're targeting mobile, use the mobile prefab. When you select it in the scene hierarchy, you should see something like what's on the right in the inspector.

After you've dragged in the prefab, select the player type you'd like to use in the "Player Type" dropdown. Depending on your current platform as well as your currently imported media players, your options for player backend will vary. However, once you import a 3rd party media player, you should see it available in the "Player Type" dropdown.

Depending on what you select, you'll see some helper text appear below to guide you on how to best setup the Clip. For a brief overview of the Clip components in general:

Notice how selecting different players populates the Inspector with the corresponding backend.

Notice how selecting different players populates the Inspector with the corresponding backend.

Player Type is the type of player driving the playback of DepthKit. This defaults to Movie Texture for Standalone builds and Sprite Sequence for Mobile builds, as these are what is supported by default in Unity. You can change this to be any of the currently supported players in your Unity project. Upon changing the Player Type, you'll see the components below swap out to reveal the corresponding player type (assuming you have it imported). You shouldn't need to interact with the added components. Only pay attention to Clip.

FOR ALL PLAYERS

Meta Data File is the meta file that is exported with your clip from Visualize.

Poster is a png that represents the static, editor only preview pose for your clip. This is most often provided by just using the first frame exported from Visualize.

Auto Load states whether or not this video should load upon entering Play mode. Loading a video can take some time depending on what player you are using, so you may want to uncheck this and trigger load/play from elsewhere in a script unless you want the video to start as soon as you press play.

Auto Play states whether or not the video should play as soon as you enter Play mode. Note that upon checking this setting you can opt for a delay time that sets how much you want to offset the start of playing. Note also that for a video to autoplay, it must also auto load.

FOR MOVIETEXTURE

Movie is the MovieTexture corresponding to your film. You can simply drag on an imported movie onto the field. Note: We've had some issues with users not seeing their clips as MovieTextures in their projects. If you're on Windows, you may need to download Quicktime (available here). If the asset still doesn't appear as a MovieTexture once downloading Quicktime, try right-clicking on it and selecting "Reimport." If this still doesn't work, try converting your video to .ogv (Theora) and then reimporting.

FOR MPMP/AVPRO

Video Path is the path to the video. You can simply use the filename for this section, provided that your video file is in the top level of Assets > StreamingAssets. Note that for MPMP, the file has to be in StreamingAssets. For more advanced functionality, see the documentation for MPMP or AVProVideo.

FOR AVPRO

File Location states where the specified file is relative to a provided path. We recommend putting your video in StreamingAssets.

FOR SPRITE SEQUENCE (Plugin v before 0.1.7)

Clip Animation is the animation clip you want to play for your sprite sequence. For more information on this, see below.

FOR NATIVE VIDEO CLIP (Plugin v0.1.7+)

The DepthKit Unit Plugin supports the new native VideoClip playback backend in Unity 5.6 (currently in beta). If you import the package you'll see the ability to select VideoClip in the player type dropdown. To use this option, move an exported video into your Unity project folder (no need to put it into streamingassets like other players) and in the import settings select VideoClip. You can then drop it on the Clip in the VideoClip section and should experience playback like normal. NOTE This method makes native Unity playback work on all platforms without needing an external player!

For more info on VideoClip, check out the Unity documentation here: https://docs.google.com/document/d/128X3Wr3xWDK-XTEhwnGkK4-ndqZAezijsZgQY43q0vM/edit


To easily get up and running, simply drag on the corresponding files into their sections on the Clip inspector (or fill out the necessary fields) and press play. 

At the most basic level, you can simply drag in your movie onto the MovieTexture field, followed by the meta file into the "Meta Data File" field, and then the poster frame in the field after. To see playback, ensure Auto Play and Auto Load are checked, then enter Play Mode. Ta-da!

Working with the API

It's likely that you may want to control how DepthKit plays in your scene, and the best way to do this is through the Clip class. To work with the Clip, we provide a function, GetPlayer(), that allows you to call the public methods of a given player. The most useful ones are below:

  • StartVideoLoad() will initiate a load for the video. This must be called before a video can play.
  • bool GetLoadingStatus() will let you know if a video is done loading.
  • Play() will play a video, if it has been loaded.
  • Pause() will pause a video.
  • Stop() will stop the video and reset its time to zero.
  • SeekTo(double time) will seek to the seconds provided in the parameter (note that this doesn't work for MovieTextures).
  • IEnumerator LoadAndPlay() will load a video and then play it once it is done loading. This method nicely bundles loading and playback, but because there is no way to know how long a load will take, when the video actually starts is unpredictable and hence not recommended for people who need playback to start at a specific time.

We also provide a number of events that can be used to listen for player events and join your own functionality to their callbacks. These events are as follows:

  • PlaybackStarted is called when playback on the clip has begun.
  • PlaybackPaused is called when playback is paused.
  • PlaybackStopped is called when playback is stopped.
  • LoadingStarted is called when the loading of a clip has begun.
  • LoadingFinished is called once the loading of a clip has finished.

For an example of how both the event/delegate pattern works with DepthKit, as well as how to call Player functions, check out the script in the plugin called APIExample.cs, as it gives a concise overview of working with DepthKit clips from code.

Working with Sprite Animations

The sprite animation workflow is slightly different than the video workflow as it doesn't use an actual video. Instead, you'll import all the raw frames exported from Visualize into your Unity project. Once imported, select them all and then look at the inspector.

In the "Texture Type" dropdown, change it to Sprite and click apply. The sprites will then reimport. To build an animation, select all the sprites again and then drag them onto your scene. You should be prompted with a window asking you to save and name the new Animation. Once you do, drag the Animation from the project to the corresponding section in the clip inspector, and you're good to go!

General Notes

  • As stated before, switching out players will cause the components on the DepthKit object to shift. Do not be alarmed, this is totally normal.
  • You should not try to switch out player backends at runtime. Behavior will be erratic at best.
  • For platform specific questions (like setting up Android or iOS builds), see the corresponding documentation on Unity's docs.
  • We've found that, if using MPMP, using the VLC backend will make the video not play in Windows builds. To change this, go to the MPMP preferences in Unity and deselect the "Use VLC Backend on Windows" option.
  • The Android builds are tested against a minimum API level of 17.
  • If you remove a 3rd party player after setting it up with DepthKit, you'll need to change to player directives to account for the change. Inside Edit > Project Settings > Player, find your delivery platform and inside the "Player Defines", delete either "DK_USING_AVPRO" or "DK_USING_MPMP" depending on the player your removed.
 

When you uncheck the "Auto Graphics API for Windows" box you should see the Graphics API selection menu. Clicking the minus button with on of the APIs selected will remove that option.

When you uncheck the "Auto Graphics API for Windows" box you should see the Graphics API selection menu. Clicking the minus button with on of the APIs selected will remove that option.

Exporting for Hololens
Assuming you have everything setup for Hololens development in Unity (See a great set of resources on Unity's website here, the key to getting DepthKit working in your Hololens project is to disable the DX9 fallback renderer in Unity. To do that, see the steps below.

After unchecking the box you should see only a single option for the Graphics API.

After unchecking the box you should see only a single option for the Graphics API.

  1. Navigate to the Player Settings (Edit>Project Settings>Player) and click "Other Settings".
  2. Make sure the "Auto Graphics API for Windows" checkbox is unchecked.
  3. You should see two renderers there, "Direct3D11" and "Direct3D9". Select "Direct3D9" so that it highlights and click the little minus sign button below it to remove it as an option.
  4. You should now only see the "Direct3D11" renderer option available.
  5. After disabling the "Direct3D9" option, you can build for Hololens like you would for a normal Hololens project, but with your DepthKit footage now properly rendering!

Exporting FOR THE WEB IN ThreeJS

 
 

A tutorial on how to play DepthKit captures in three.js. Download the necessary files from the DepthKitJS repo here

  1. Export your footage using "combined color per-pixel"
  2. Modify and run the included FFMPEG script create_webm.sh
  3. Upload to /files/
  4. Modify Min & Max depths in the index.html
  5. Try it out in the browser
  6. Modify and make your own!
 

TROUBLESHOOTING

colorgrad.jpg

Troubleshooting

My application will not launch or it crashes.

  • Maybe your system doesn't support the toolkit. You'll need a computer with accelerated graphics no more than 4 or 5 years old.
  • We have also had recent issues with brand new mac laptops with integrated graphics cards. 
  • Make sure you have not moved the data/ folder in relationship to the DepthKitCapture or DepthKitVisualize applications.

My depth camera feed is not showing up in the Calibrate Lenses tab.

  • Try disconnecting and reconnecting your depth camera, then restarting the application.
  • Occasionally fully restarting the computer has helped with sensors that fail to pick up.
  • Confirm that you're using the correct version of the app for your sensor – beta is designed to be used with the Kinect v2 on a Windows machine.
  • If you're using a USB extender, remove it and plug the sensor in directly.

My depth camera's FoV/principal point don't match the figures given in the video. Or I can't calibrate the Depth Camera.

  • If they are only slightly different, don't worry!
  • Try the process again from scratch by disconnecting devices and relaunching the application.
  • Make sure there is information in the scene that falls within the range visible to the sensor (~3'-12').
  • Check that there is variation in the scene; information in the foreground, middle and background.

I'm getting high error numbers on the calibrate lenses page.

  • High error may occur when the checkerboard is too close or too far away.
  • Make sure there is no glare on the checkerboard from lights. Even one hot spot from a nearby light could throw off the reading.
  • Make sure that the exposure is set properly on the camera - if the whites of the checkerboards are too bright sometimes you need to shoot it again at a slightly darker exposure.
  • Make sure you refocused on the checkerboard at every stage and that your aperture is no too low, making the depth of field shallow.

While calibrating, my correspondences are not detecting or are way off.

  • Confirm that you have a low error on the first stage.
  • Make sure that your set up is totally stable when capturing the correspondence images. A wobbling checkerboard or camera tripod will throw things off.
  • Often this will happen if there is not enough ambient IR light, get closer to a window
  • Conversely if there is too much or uneven lighting it'll also blow out your checkerboard
  • Try different depths from the camera, being wary moving it too close or too far.

My recordings are dropping frames.

  • Close any other apps.
  • Make sure you are plugged into wall power
  • If writing to an external hard drive make sure it's USB 3.0 or FireWire 800.
  • This can be caused by using a USB extender for the Kinect that is too long without an inline repeater. Remove the USB extender or use an active USB 3.0 extender.

The application crashes when I try to load movie files from the HD camera.

  • Make sure your video files are compatible with the toolkit, H.264 or ProRes MOV's.
  • Make sure you have the latest version of Quicktime installed. If you just ran a fresh install of Windows, it's likely that you don't have Quicktime. Download the latest version here.
  • Make sure you are running the application from a location where you have write access

I'm having some errors with needing Quicktime.

  • Modern installations and fresh installs of Windows don't natively package Quicktime 7, so you'll need to download the latest version of codec here.

My video doesn't show up in the Visualizer (either the thumbnails or synch preview) or otherwise acts strangely, or the console shows an error referring to "static framerate."

This is usually caused by issues with your source videos. Generally there's no issue with simply dropping your files straight into the /color folders, but occasionally you'll need to transcode the videos. We recommend transcoding your full resolution videos to Apple ProRes and your low resolution proxies using Quicktime>MJPEG-A. Replace the videos in the .../color folder with your transcoded videos, close the Visualizer and relaunch it. If you're looking for a tool to transcode your videos you can use Adapter (free), MPEG Streamclip (free) or Quicktime Pro (paid). We also offer an Adobe Media Encoder preset you can download here.

My texture alignment is way off.

  • Try using the X & Y Rotate, Shift or Scale in the Texture Alignment tab to accommodate for slight inaccuracies.
  • You may want to recalibrate, which is fine to do after you've shot as long as the relationship between camera and sensor is still fixed.

My time alignment/synch is way off.

  • Delete any markers on the 3rd "Alignment" track on the timeline to start over.
  • Look for a very decisive temporal moment to align to, meaning that there is a specific fast movement which allows for precisely selecting a depth and video frame from close times. Because the cameras have different frame rates and clocks, the alignment will never be perfect but should not drift over time.

The preview in my Capture application seems to stutter a bit, freezing for brief moments every few seconds.

  • This is usually caused by using a USB extender for the Kinect that is too long without an inline repeater. If this is happening, either remove the USB extender or use an active USB 3.0 extender.

I'm unable to drag in my clips for the Calibrate RGB Intrinsics step or the Calibrate Correspondence step.

  • There is an issue where launching DepthKit Capture "As an administrator" will make it so dropping in clips isn't possible. If you're running into this error, try starting DepthKit normally, without selecting "Run as administrator."
  • You also need to make sure your clips are the proper codec — XAVC, .R3D, and other file formats will likely not work. You may need to transcode your calibration videos to h264 in order to import them.

Visualize will not let me track/seek depth frames, and the frame number is stuck at 0 in the depth track.

  • If you have verified that the depth frames are indeed in the /depth folder, there is likely a problem with your drive permissions and you'll need to change the permissions to allow you to both read and write from the drive. Otherwise move your Take and calibration file to your main drive and try again.

My renders stop always stop at a given frame, no matter what my in/out frames are.

  • DepthKit Visualize has some problems playing back MJPEG codec videos, and if your video is of this type it's worth trying to rewrap it as h264/mp4.
  • If you try to rewrap your video and it still doesn't work, you may have a problem with the timecode encoding in your video. Try to re-render your video with fresh timecode, which should then fix the issue.

I get an error that states "The program can’t start because MSVCP110.dll is missing from your computer"

 


 

Hotkeys

DepthKit Capture

Hotkey          Function
Space Start/stop playback in Playback Tab
Space Start/stop record in Capture Tab
W Move camera forward
A Move camera back
S Move camera left
D Move camera right
Q Rotate left via Kinect's perspective
R Rotate right via Kinect's perspective
E Move camera up
C Move camera down

DepthKit Visualize

Hotkey                  Function
Space Start/stop playback
W Move camera forward
A Move camera back
S Move camera left
D Move camera right
Q Rotate Kinect perspective left
R Rotate Kinect perspective right
E Move camera up
C Move camera down
F Toggle full screen mode
Shift+T Set camera keyframe
Shift+L Lock camera to track
Cmd+Z Undo
Shift+Cmd+Z Redo
Delete Delete all selected keyframes
Cmd+A Select all keyframes in track
Hotkey                  Function
Cmd+C Copy selected keyframe(s)
Cmd+C Paste selected keyframe(s)
Cmd+X Cut selected keyframe(s)
Move keyframe small increment up in timeline
Move keyframe small increment down in timeline
Move keyframe back one frame in timeline
Move keyframe forward one frame in timeline
Shift+▲ Move keyframe large increment up in timeline
Shift+▼ Move keyframe large increment down in timeline
Shift+◀ Move keyframe back 5 frames in timeline
Shift+► Move keyframe forward 5 frames in timeline
1-9 Switch to tabs 1-9
Alt+E Expand focused timeline track
Alt+C Collapse all timeline tracks
Alt+Shift+C Collapse all timeline tracks

Note: On OSX the COMMAND (⌘) key is used while on Windows the CTRL key is used.


Known Issues

My Capture app crashes when I attempt to the Calibrate Lens step using extra high resolution (3.5k, 4k) videos.

This happens when you try the Calibrate Lens step using the old method of calibration. The program now allows you to drag the folder containing the RGB videos instead of the videos themselves, which should fix the problem.


  1. Drag the first five video clips from the camera & drag them into the window.
  2. Find the /_rgbCalibration directory inside .../MediaBin/_calibration/ where the Capture Application saves the converted PNGs.
  3. Copy the PNGs generated for the first five videos into another directory.
  4. Repeat steps 1-3 in sets of five videos until you've processed all the calibration videos.
  5. Gather all the PNGS from the external folder and copy them back into the folder in _calibration. 
  6. Relaunch the app.

The program is unable to find the checkerboard in the IR image in the "Calibrate Correspondence" section.

It's very likely that you printed your checkerboard with an ink that is too glossy. This can be the case if you used a high-end photo printer or something like a Canon Pixima printer. You should try to print instead with a laster printer, or even just photocopy what you initially printed and use the copy instead of the original, as that should make the checkerboard more receptive to IR light.

The camera is aimlessly flying off into the ether without input from me.

This often happens if you've held ALT while pressing one of the movement keys (W,A,S,D etc.) Either hold alt again and press the same key or reload the comp.  

My Kinect is unable to self-calibrate

DepthKit maintains a relatively narrow margin of error when it comes to calibrating the Kinect, so if a Kinect calibrates outside of that range it will fail to calibrate. We're working on a new version that lets you calibrate with the caveat that the Kinect isn't as well-calibrated as it could be, but for now you'll need to either buy a new Kinect or download a prefabricated calibration like the one here. Once you download the file, close the Capture application, and then place that file in you matrices folder. When you open the Capture app again, it should say that your camera has been properly calibrated.


Frequently Asked Questions


Features

Q: Does DepthKit support multi-camera shoots like the footage in Blackout?
A: Not yet. Our strategy is that, in building Blackout, we'll be concurrently working on the tool and won't release it before it's ready.

Q: Does DepthKit currently include the filtering techniques used in Giant?
A: The public version does not. We're still working on those techniques but we can selectively offer it as a service.

Q: When will you support [new depth sensor]?
A: We're constantly testing and working with new sensors. Currently we're excited about Orbbec, ZED, Intel RealSense, PMD, Tango etc. We're not currently making any specific promises about when we'll add support for new sensors. If you've encountered a sensor that you're excited about (or make), please get in touch!

Capabilities

Q: Does DepthKit work with X software product or Y workflow?
A: We're going to continue supporting common export formats (.OBJ, .ABC) and we'll be publishing tutorials as we go, but we are focusing on a particular workflow first: DepthKit > Unity for VR/AR. We encourage you to test and share workflows as you experiment with DepthKit footage in your favorite environments. If you are an expert in a particular toolset please get in touch with suggestions for favorite import formats or other conventions.

Q: What framerate does DepthKit shoot?
A: We recommend filming at/around 24p. The Kinect shoots at a variable framerate around 30fps and DepthKit down-samples to the framerate of the footage provided. We recommend shooting at 23.976 fps / 24p.

Q: What is the range of DepthKit? 
A: Currently we're focused on the Kinect V2 which has a range of around .5m and 4.5m (1'8"-14'6".)

Q: What is the Kinect V2 depth resolution?
A: The new Kinect has a depth resolution of 512 x 424 pixels with a FoV of 70.6° x 60° resulting in an average of around 7 x 7 pixels per degree. Due to the use time-of-flight as the core mechanism for depth capture, each pixel in the depth image contains a real measured depth value. See Roland Smeenk's thorough breakdown for more information.

Q: What is the optimal range for shooting with DepthKit?
A: The quality of the depth data falls off as you move away from the sensor, so the general recommendation is that you fill your depth frame with your subject whenever possible.

Q: Do I need to be plugged in to the wall to run DepthKit? 
A: With the Kinect V2 - which is the best sensor that we're supporting right now - the answer is yes. However the legacy versions of the DepthKit support the Asus/Primesense sensors that run off USB power. Alternatively we've had users make mobile rigs that include a battery and an AC/DC inverter that work great for running a Kinect.

Q: Do I need an external camera to shoot with DepthKit if I only want depth data sequences? 
A: No, you do not. For more information, contact us or ask a question about this in the DepthKit Beta Slack channel.

Q: What is the field of view (FoV) Kinect V2?
A: The Depth camera on the Kinect V2 has an FoV of  FoV of 70.6° x 60° and the color camera has a color resolution of 1920 x 1080px and a FoV of 84.1° x 53.8° resulting in an average of about 22 x 20 pixels per degree. There is a great interactive Kinect 1 vs Kinect V2 FoV comparison tool here.

Have a question about the beta program?