The next stage is to merge the depth and color and either render video or export your footage for either realtime or VFX tools. Find hotkeys here.
The workflow in Visualize varies depending on your chosen export target, but the general workflow has you using the takes you recorded during the capture stage to generate compositions, or "comps," that can be exported to your target tool. Syncing is done inside of the comps so you can temporally align the depth data with the color data — a process that needs to be done because it's likely you didn't start rolling the color camera at the exact same time you rolled the depth.
Every take can have any number of compositions, so we usually recommend syncing a take with a "Sync" comp, then using that as your base comp which you can then duplicate inside Visualize to then start working with your footage in a new comp. This way you do not have to continually sync your footage for every new comp you choose to make.
The following tutorials will take you through the process of navigating Visualize as well as syncing your footage to get it ready to export to different platforms.
The main DepthKit Visualize interface has four main components. The File manager at the top, the Settings on the left, the Scene Viewport in the middle (where you see your footage), and the Timeline at the bottom.
The File Manager gives you the ability to:
- Change your workspace by clicking the current workspace path.
- Save you current composition by clicking "Save Comp"..
- Change which comp or TAKE you're working on by clicking the TAKE's name.
- Make a new comp using the settings of the current comp by clicking "Copy to New."
The Scene Viewport allows you to view DepthKit footage in 3D and set camera positions for image sequence renders.
- Move the camera in 3D space with W,A,S,D.
- Rotate the camera to the left and right with Q and R respectively.
- Use E and C to boom the camera up and down.
- Click and drag the mouse to change the direction the camera is facing.
The Settings menu give you access to the bulk of the editing/rendering settings for Visualize. You can move the settings panel around by clicking and dragging on the top bar.
- Temporal Alignment: Click this to jump to the mode where you synchronize the depth and color tracks to each other.
- Set Color-Depth Time: When in the Temporal Alignment mode, click this to lock in playback on both tracks to a single time code. For more on this, read the "Synchronize" section below. Do not press this button multiple times, if you do select the extraneous keyframes and press Delete/Backspace.
3D Camera Settings
- Camera Speed: Set how fast you move through the scene with the W,A,S,D keys.
- Camera Roll Speed: Set how fast the camera rotates with the Q and R keys.
- Reset Camera: Reset the 3D camera back to it's starting point. Useful if you get lost in the scene or if you can't find the subject.
- Lock to track: When keyframing camera positions for the image sequence export, click this setting (shortcut Shift+L) to make sure the camera moves along the keyframed path. For more on this, see the section on image sequence exporting.
- Set Camera Point: Use this (shortcut Shift+T) to set a keyframe for a camera's location and orientation on the Camera track in the Camera tab. These points determine how a camera will move for an image sequence export. For more on this, see the section on image sequence exporting below.
- Mirror: Use this to horizontally flip the image in the display window.
Depth Data Settings
- Freeze Frame Mode: Allows you to stop motion at the playhead. Using this mode allows you to create camera movements or effects around on a single frame from your capture.
- Fill Holes: Toggles hole filling, plugging in holes in your capture data. This setting can create artifacts if used too liberally. See Hole Kernel Size & Hole Fill Iterations below – ideally use a balance between small kernel size and a moderate number of iterations (eg: 1, 3.)
- Hole Kernel Size: The size of the 'plug' used to fill holes.
- Hole Fill Iterations: The number of attempts using the size of the Kernel.
- Subtract Background: Allows the user to subtract a background plate from the scene based on one chosen with the "Capture Background" option below. See more in the "Background Subtraction" section below.
- Capture Background allows the user to state that the current previewed frame is the blank background that can be subtracted.
- Background Threshold slider controls how "strong" the subtraction is.
Note: These settings mainly control effects that will only be rendered if you choose to do an image sequence export. Rending .OBJs, .ABCs, or Combined Per Pixel will export geometry only and will not export these effects.
- Draw Point Grid: Draw an array of points on a grid that match up to the DepthKit images vertices. In other words, render a gridded pointcloud. You can control this effect in the Timeline section labeled "Rendering."
- Draw Points Random: Draw a random array of points to the DepthKit mesh allowing you to break up the uniformity of the point grid. You can control this effect in the Timeline section labeled "Rendering.
- Draw Wireframe: Draw a dense wireframe that represents the whole DepthKit mesh. You can control this effect in the Timeline section labeled "Rendering."
- Draw Vertical/Horizontal Lines: Draw horizontal and/or vertical lines across the mesh. This is like a simplified Wireframe view. You can control these effects in the Timeline section labeled "Rendering".
- Draw Mesh: Draw the colored Mesh, textured directly from the color camera.
- Self Occlude: Make it so the DepthKit footage isn't "transparent." When you're looking at the model as a wireframe view or a point cloud this setting makes foreground elements opaque.
- Depth of Field: Render Depth of Field effects. You can control settings in the Depth of Field tab in the Timeline.
- Wave Distort: Turn on a "Wave" effect that makes the DepthKit footage look like a wave is running through it. You can control this effect in the Timeline section labeled "Waves."
- Warp Distort: Turn on a warp effect that warps the mesh in space. You can control this effect in the Timeline section labeled "Warp."
- Frame Width/Height: Set how wide/tall the frame should render to be. This will also update the window size in the scene window.
- Render 720p/1080p: Use the 1080p or 720p render size preset for rendering image sequences. For more on this, see the section on image sequence exporting below.
- obj Files/Textures: Check these boxes if you want to render an OBJ sequence (with or without textures) of your DepthKit sequence. Note that that any effects or camera moves you've made in Visualize will not render. Read more about this is the section below about export for VFX tools.
- abc Files/Textures: Check these boxes if you want to render an Alembic file (with or without textures) of your DepthKit sequence. Note that that any effects or camera moves you've made in Visualize will not render. Read more about this in the section below about exporting for VFX tools.
- Combined Per Pixel: Check this setting if you want to render for importing into Unity. This will render at the size of the original depth map. For more on this, look at the "Exporting for Unity" section below.
- Combined 720p/1080p: This format is not currently supported in the Unity workflow, it's designed for users who design custom import workflows. This will render at a preset size of either 1080p or 720p with a depth image on the right. For more on this, look at the "Exporting for Unity" section below.
The Timeline is how you navigate your footage temporally, add camera keyframes and keyframe the values presented in the settings menu.
Using the Timeline
- Navigation is done by simply clicking on a tabbed heading.
- To add a keyframe to a track, simply click the space in a given track and a keyframe should appear.
- To delete a keyframes, click a keyframe and then press "Delete" on your keyboard.
- To adjust keyframes, you can either click and drag, or use the arrow keys on the keyboard for more fine-tuned adjustment.
- For further Timeline hotkeys see Hotkeys Glossary.
Note if you are low on screen space, you can mouse to the bottom green bar of any section of the timeline (it should be highlighted with green slashes when you do) and click and drag it up to "hide" that section of the timeline. To access it again you just drag the corresponding bar down to reveal it.
Timeline Section Overviews
- Camera: This is where you can navigate footage temporally after it has been synced. You can also set keyframes for camera positions in the Scene Viewport here by either clicking "Set Camera Point" in the Settings or by using the shortcut Shift+T. The Camera dampen track smooths the camera's motion between keyframed sections a given amount.
- Geometry: Here you can adjust how much geomery is visible in the scene as well as how it is depicted. For more on this, check out the "Trimming Geomerty" section below.
- Clipping: This is where you can isolate your subject in your composition by clipping the edges of the geometry, as well as clipping the depth image in general. For more on this, check out the "Trimming Geometry" section below.
- Rotation: Here you can keyframe discrete camera rotations that will be used in playback and rendered if you are doing an image sequence export.
- Rendering: Here you can find the settings for how the pointcloud, wireframe, and mesh are rendered if you are doing an image sequence export. For these to work, you need to have at least one of the pointcloud, wireframe, or mesh settings selected from the settings menu.
- Point Alpha controls how transparent the points are when either "Draw Points Random" or "Draw Point Grid" are selected from the settings menu.
- Point Size controls how big or small the points are when either "Draw Points Random" or "Draw Point Grid" are selected from the settings menu.
- Random Point Amount controls how many points are generated when "Draw Points Random" is selected from the settings menu.
- Wireframe Alpha controls how transparent the wireframe is when "Draw Wireframe" is selected in the settings menu.
- Mesh Alpha controls how transparent the mesh is when "Draw Mesh" is selected in the settings menu.
- Scan Lines: When either "Draw Vertical Lines" or "Draw Horizontal Lines" is selected from the settings menu, you can can edit their respective properties here.
- Horizontal/Vertical Scanline Alpha controls how transparent the respective lines are.
- Horizontal/Vertical Scanline Thickness controls how thick the respective lines are.
- Waves: When "Wave Distort" is selected from the settings menu, you can control its respective settings here.
- X/Y Sin Amplitude controls how "large" the waves are.
- X/Y Sin Speed controls how fast the waves move across the mesh.
- X/Y Sin Frequency controls how "many" waves there are at a given time.
- Warp: When "Warp Distort" is selected from the settings menu, you can control its respective settings here.
- Warp Amplitude controls the "strength" of the warp.
- Warp Density controls how rigid or fluid the warp should be. Higher values mimic fluids, lower values mimic solids.
- Warp Speed controls how the mesh is warped by the above warp settings.
- Warp Stretch X/Y/Z controls how much the warp should warp the mesh in a given dimension.
- Color: Here you can set a background color as well as key global color fade setting for your points of mesh.
- Fade Amount controls how much you fade from one fade color to another. Note that no Fade Amount means that no Fade Color is applied.
- Fade Color is the color you are trying to fade to. Note that this fades everything to that color.
- Background Color allows you to set a background color. This is also useful in syncing as it lets you more easily isolate depth and color streams. Note though that the color is rendered out if you do an image sequence export.
- Depth of Field: When "Depth of Field" is selected from the Settings menu, you can control its effects here. This simulates a normal cameras Depth of Field effect.
- DOF Distance can be thought of as where the camera is focusing in space.
- DOF Range is the amount of space that should be "in focus." Manipulating this is like manipulating a camera's aperture.
- DOF Blur is how much blur should be given to an "out of focus" object.
- Synchronize: This is where you can synchronize your depth and color streams. For more on this, see the "Synchronize" section below.
- Texture Alignment: This is where you can better align your color stream to your depth data to offset any calibration errors that happened during the capture stage. For more on this, see the "Texture Alignment" section below.
The first step when working with any DepthKit TAKE is to sync the color video to the depth data stream so that the two will play back in unison, with the color properly matching to the depth.
Creating the Sync Comp
Launch DepthKit Visualize and point it to the top-level folder that contains your TAKE folders.
Note: Point to the top level folder that contains your TAKE folders, not to a specific take. If you point to a specific take, Visualize will tell you that there are "No valid takes."
Once you point to the proper directory, you should see the list of TAKEs that you recorded. Only takes with videos of a recognized format inside the /color folder will show up, so if you don't see one of your TAKEs, make sure you've added the video to the folder. Select the TAKE you'd like to visualize and click 'Create new Composition with this Scene.' If you have issues with this stage see troubleshooting below.
FOR DEPTH ONLY CAPTURES:
You'll need to generate black video in either a video editing program or FFMPEG that is the same length of your captured depth sequence. If you do not have a video file in in the "/color" folder, Visualize will not recognize your captures as a valid Take.
Syncing the Depth to the Color
Assuming it's your first time loading this scene you'll need to synchronize the depth & color streams. On the timeline at the bottom you should see two rows of thumbnails: color & depth. Scrub through them by clicking on the line of thumbnails and dragging left or right. You'll see previews of the depth data and the RGB video as thumbnails on the top right. With your mouse over the respective timelines can also use the arrow keys until you find the synch moment in both Depth and Color (usually a clap).
Once you have the precise moment of the sync selected in both preview windows, press Set Color-Depth Time in the Settings panel. Selecting it should automatically toggle off Temporal Alignment mode. Now if you scrub through the timeline by clicking on the bar with the yellow tickers in the "Synchronize" tab, you should see the combined DepthKit footage playing back in synch.
Correcting Sync Errors/Restarting Sync
If you make a mistake and find that the synch is incorrect, you can always navigate back to the Synchronize tab, select the blue sync marker in the timeline and press DELETE to start over. It's NEVER necessary to set multiple synch points, so avoid duplicates by delete any preexisting markers.
After you think you have a good sync, you can fly around in the scene (see camera Hotkeys) to check how good your sync is. Remember, if you get lost you can use the "Reset Camera" button at the top of the Settings menu. You can also play back in this mode (Spacebar) and watch your footage and see if the sync seems off.
Texture Alignment (Fine-tuning Calibration)
After syncing, it's usually necessary to tweak the color texture alignment in how it falls on the depth data, even if you had a good calibration during the Calibration step.
Navigate to the Texture Alignment tab on the far right to adjust the placement of the color on the depth. To fix any alignment errors, you'll need to subtly tweak the values of the given sections:
X/Y Texture Rotate will rotate the texture "projector" so the effect will be more noticeable at at a distance.
X/Y Texture Translate will shift the texture "projector," meaning that that the change will be equivalent in the near ground and the background.
X/Y Texture Scale will scale the projected texture, stretching it out in the corresponding direction to better cover a given depth map.
If you find that you've offset your textures too much and need to start over, remember that you can delete any keyframe by clicking it and pressing "Delete" on your keyboard.
After you tweak these values, it's always worth playing back your footage and making sure the tweaking you just did still properly aligns the footage further down the timeline. It's very easy for you to fine tune a specific shot, but then find in playback that it ruins the rest of the footage. Instead of trying to find the perfect alignment for any given frame, we recommend finding a single good in-between alignment that plays best for the duration of the shot and not try to do a lot of individual keyframes for the shot's full duration.
Making other Compositions
After you properly sync and align your first take, you can duplicate it to a new take that you can freely work in without fear of messing up your sync. To duplicate a take, click "Copy To New" at the top of the screen and name your take something descriptive that gets at what you want to do with it.
Remember, each Take can have any number of compositions, so feel free to experiment. Composition let you render the scene in a different way, with a different in and out point, changing camera perspectives and different rendering styles.
Right after syncing the first thing you'll likely want to do is trim out some unnecessary object/information in your scene. This can be done inside of Visualize and help your to export a relatively clean model to the program you're targeting.
Visualize supports the ability to intelligently subtract the background of the scene with the "Background Subtraction" toggles in the Settings menu. To subtract the background, click "Capture Background" on a blank shot where your subject isn't in the frame. To subtract the background, click "Subtract Background". The program will then cull the background from the scene based on the "plate" that you instructed it to use. Use "Background Threshold" to control how "strong" the subtraction is. Note: This is an experimental feature.
Visualize also allows you to trim out geometry behind or in front of your subject by using the options inside of the Geometry tab.
Z Threshold Max trims out geometry starting from the rear of the scene, meaning it's usually the first place you'll want to go if you find that you're seeing objects in the background you don't want to be present in your export.
Z Threshold Min trims out geometry starting from the front of the scene. This is used if something comes between you and your subject and you need to remove it from your scene.
Simplify X/Y allows you to create more contrast between the positive and negative space in your scene by increasing or decreasing the resolution of the underlying geometry.
When the texture is projected onto the depth, it's very likely that it isn't always a perfect match due to a variety of factors. This means that in literal "edge cases", you will often have parts of a texture "streak" towards the back of the scene. Some people find this "look" desirable, but if you're looking to remove it, navigate to the Clipping tab.
Edge Clip is used to trim the edges of anything in the scene, allow you to reduce the amount of "streaking" that occurs for a given object.
Top/Right/Bottom/Left clip allow you to clip the DepthKit image in the stated direction any given amount. This a good solution if you're looking to isolate a given object for export.