Depth Cameras and 3D Position in Robot Learning: Evidence and Practice

Depth cameras can improve robot policies by making distance and spatial relationships easier to represent. The strongest evidence concerns a complete pipeline: measured or learned geometry, consistent coordinate frames, an appropriate model, and a controller that can execute its predictions. A depth channel alone is not a dependable upgrade.

This research feature reviews representative papers from 2023–2026 and implementation material checked on 30 October 2026. Results below are the authors' reports, not Dentry replications. The selection is a focused reading guide, not a systematic review or a ranking across benchmarks.

What “position information” means

Several different signals are often grouped under one label:

• Scene geometry: a depth map provides per-pixel depth; a calibrated camera model turns pixels and depth into three-dimensional points.
• Coordinate transforms: the camera's pose relates those points to the robot base or a map. A wrist camera's transform changes as the arm moves.
• Robot state: joint positions, gripper opening or the current end-effector pose describe the robot itself. Different policies use different subsets.
• Model position encoding: coordinates or relative offsets tell an attention model how scene features and candidate actions relate. This is a representation choice, not another sensor.

Our interpretation is that keeping these distinctions explicit makes experiments easier to diagnose: an accurate depth image cannot compensate for a wrong camera-to-robot transform.

DP3: a point cloud helps more than an extra image channel

In DP3's 72-task simulation comparison, average success is 74.4%, versus 59.8% for RGB Diffusion Policy: a difference of 14.6 percentage points using the reported rounded values. DP3 conditions its action generator on a compact point-cloud representation and robot poses. [1]

Its six-task representation ablation is especially informative: point clouds reach 78.3%, RGB 40.7%, depth images 32.0%, and RGB-D images 34.7%. Simply adding depth to an image-based input did not reproduce the point-cloud result. Encoders also differ, so this tests the representation pipeline rather than isolating a sensor's causal effect. [1]

For a software team, the useful question is how geometry survives preprocessing and encoding. DP3's official repository supplies code and data links for investigating that question. [7]

3D Diffuser Actor: relative position also matters

On the paper's RLBench ablation, a 2D variant reaches 47.0%, a 3D variant with absolute attention reaches 71.3%, and the full model reaches 81.3%. The full model improves on those variants by 34.3 and 10.0 percentage points, respectively. The 2D variant also pools features within each view, so this is evidence about representation and fusion together. [2]

Scene features, the current end-effector location and candidate actions interact in a shared spatial representation. In the real robot setup, a motion planner executes predicted poses. This is a useful separation between learning a target and making a robot reach it. [2]

RVT-2: precise placement needs local geometric detail

RVT-2 refines a coarse target using a closer virtual view of the relevant point-cloud region. Its RLBench result is 81.4%, compared with RVT's 62.9%. Both already use three-dimensional observations; the 18.5-point difference is not a depth-camera versus RGB-camera comparison. [3]

For three real precision tasks, including peg and plug insertion, the reported average rises from 33.3% to 53.3%. Each task has only ten demonstrations and ten evaluation trials. Those results support a research prototype, not a claim of production-ready insertion. The authors also discuss force feedback as a way to address remaining contact errors. Code and models are available from the official RVT repository. [3] [8]

Generalization can improve while familiar-task results do not

The iDP3 paper provides an important qualification. In a new-scene pick-and-place test, iDP3 succeeds in 9 of 10 trials, compared with 2 of 10 for RGB Diffusion Policy with a fine-tuned R3M encoder. However, the RGB baseline performs better in a separate, small-data training-setting experiment. Those experiments use different evaluation protocols and should not be merged into one headline success rate. [4]

iDP3 combines camera-frame geometry from a head-mounted RealSense L515 LiDAR depth camera with joint positions. Its demonstrated humanoid manipulation setup has a supported, stationary lower body; these results do not establish autonomous walking and household service. The practical lesson is to test scene changes separately from familiar-task accuracy. [4]

GeoAlign: depth can be useful during training without a depth camera at rollout

GeoAlign, a 2026 study, uses RGB-D supervision in geometry post-training. At execution time, its policy consumes RGB, language and robot state, without measured or predicted depth maps as policy inputs. This makes it a distinct deployment option from point-cloud policies that require depth observations at every step. [5]

Across eight real ALOHA tasks and 160 trials, the reported RGB-only baseline succeeds 104 times, or 65.0%; GeoAlign succeeds 126 times, reported as 78.8%. The difference is approximately 13.8 percentage points. The system also uses an additional 580,960-frame RGB-D dataset for geometry post-training, so the improvement cannot be credited to architecture alone. As checked for this article, the project still lists code as coming soon. [5] [13]

This is a research direction for teams with existing RGB robot datasets. It is not yet a ready-to-install replacement for an available depth-based policy.

RISE: hardware and data collection change the outcome

RISE uses a single-view point cloud and predicts actions in the camera frame; its policy does not add separate proprioceptive inputs. Its practical discussion reports difficulty reproducing DP3 under its own camera and demonstration setup. Depth noise, workspace choices and demonstration trajectories all affect transfer. [6]

Its official repository provides calibration and real-robot evaluation guidance. The code and data are offered under CC BY-NC-SA 4.0, so commercial use requires checking the applicable permission rather than assuming every public research repository permits it. [9]

Applications beyond the learned policy

For tabletop manipulation, the most direct uses are locating grasp targets, moving objects to spatially specified locations, and refining placement near a socket or receptacle. The studies above demonstrate examples of these capabilities, with different robots and evaluation conditions.

Navigation can benefit without retraining a large action model. NVIDIA's nvblox takes depth observations, camera parameters and a pose transform, integrates a three-dimensional map, and derives obstacle-distance information for path planning. This is a geometry-and-planning component, not evidence that a language model becomes more capable merely by receiving depth. [10]

HomeRobot illustrates the broader integration problem: its open-vocabulary mobile manipulation benchmark combines finding objects, navigating, grasping and placing. Its original real-world baselines reached 20% success. That is a historical system result, not a current ceiling or an isolated depth ablation. [11]

A practical evaluation plan

The following is our proposed engineering workflow, rather than an experiment already performed by Dentry.

1. Choose a bounded task. Start with pick-and-place on a fixed work surface, and define success before collecting demonstrations. Record partial completion separately from complete-task success.
2. Establish a usable geometry baseline. Check depth at the actual working range and on the actual materials. Inspect missing pixels, RGB-depth alignment, timestamps, camera intrinsics and the camera-to-robot transform. Vendor tuning guidance is a useful starting point, not a substitute for measuring the installation. [12]
3. Log synchronized observations and actions. Preserve raw RGB, depth, calibration, robot state and timestamps. Keep the coordinate frame and units explicit; a camera-frame target and a robot-base-frame target are not interchangeable.
4. Compare controlled variants. Use the same demonstrations, training budget, task distribution and controller where possible. Compare RGB plus state, RGB-D plus state, and a point-cloud representation plus state. State clearly when an encoder or pretrained model also changes.
5. Test changes that matter outside the training set. Move objects, change backgrounds and lighting, alter viewpoint, introduce occlusion, and evaluate different object instances. For insertion or compliant manipulation, test the contact-control strategy as well as visual localization.
6. Measure the whole loop. Report complete-task success with trial counts and uncertainty, placement error, interventions, collisions, inference latency and time to completion. Keep motion limits, collision checking and an emergency stop in the execution layer.

A useful first deployment candidate is a reproducible tabletop policy with a calibrated RGB-D pipeline and an existing controller. A mobile platform may instead get immediate value from a depth-based map. RGB-only runtime methods such as GeoAlign are worth watching when adding sensors is difficult, but training data and release readiness remain part of the cost.

The decision should follow the failure being addressed: poor spatial localization, discarded geometric detail, distribution changes, or contact execution. Measure that failure first, then select the sensor and representation that improve it.