2.1 Scene registration and geometry
Transform native RGB-D measurements into the policy world frame, fit the table plane, and apply the same surface and mounting transforms to training and deployment. The measured table height/orientation and 20 mm flange-to-hand offset determine object reset height and collision geometry.
Native D455 depth→Policy world frame→RANSAC + plane refinement→Aligned training scene
Plane estimation objective and calibration scope +
Native depth points are deprojected and transformed through the camera's factory depth-to-color calibration and the retained camera-to-policy-world transform. A RANSAC fit rejects outliers; orthogonal least squares refines the table plane using the inliers:
piW is a depth point in the policy world frame and 𝓘 is the RANSAC inlier set. Fixed spatial image tiles are held out from fitting. The 30-frame capture gives a 1.064° plane tilt and 2.343 mm point-to-plane fit RMS.
The plane determines table height, roll and pitch. The 20 mm flange-to-hand spacer is represented in the robot geometry and deployment kinematics; object reset height follows the table surface. The table footprint and placement can be varied during training. Plane residual is not absolute robot-to-scene alignment accuracy, and this step reuses rather than independently recalibrates the camera extrinsic transform.
Depth processing and plane fit ↗ ·
Training geometry integration ↗
2.2 Controller and plant identification
Record desired TCP poses, joint motion, commanded motor torque and joint torque sensor signals. Identify link/payload and drive effects separately, then evaluate the complete controller-in-loop response on a recording excluded from fitting.
CONTROLLER MODEL
Cartesian impedance reference acceleration
Md, Bd and Kd are the desired Cartesian inertia, damping and stiffness. Pose error e contains translation and rotation-vector error; velocity and external wrench are filtered estimates.
Structural form. The replica also preserves the deployed controller's native unit conversions, Jacobian mapping, torque compensation, friction observer and limits.
PLANT MODEL
Rigid-body and drive dynamics with external load
B captures reflected drive inertia. h includes Coriolis and gravity terms; drive friction depends on velocity and joint-sensor load. External torque enters the plant, allowing motion to respond to contact.
01Link and payload regression
Fit payload, link friction and sensor bias to joint torque sensor (JTS) data using linear least squares. Regularize wrist center-of-mass corrections and project payload inertia to physically valid values.
02Drive identification
Use delayed commanded motor torque minus JTS to identify reflected inertia and directional, load-dependent friction. Estimate breakaway from stick-to-slip transitions; identify 2 ms state and 6 ms actuation delays, reproduced at the simulation step resolution.
03Held-out trajectory evaluation
Feed desired TCP poses into the replica and generate its subsequent motion. Evaluate a held-out recording excluded from fitting, then run the identified model in the learning environment.
Identification objective and implementation details +
For each joint, bounded least squares separates reflected inertia, load-dependent friction and a velocity lookup table:
dj(t) = τtarget_motor,j(t − 6 ms) − τJTS,j(t), using the commanded motor torque before the actuation delay. The basis Hj uses separate positive/negative piecewise-linear velocity tables with 18 speed knots. The fit bounds reflected inertia to [0, 10] and the load coefficient to [0, 0.5]. Detected external-load intervals are excluded from link/payload fitting and free-motion evaluation.
The controller uses e = [pd − p; Log(RdRT)] and a low-pass-filtered API wrench after offset subtraction. Identification is performed component by component; full closed-loop trajectories are a subsequent validation.
The learning integration disables the generic PhysX arm position servo and applies the replicated controller and identified drive model. This is the concrete change from the baseline arm actuation used in this adaptation.
Drive identification source ↗ ·
Controller and simulation integration ↗