RA-L Submission · 2026

From Perception to Assistance: Open‑Vocabulary Shared Autonomy for Robotic Manipulation

Murilo Vinicius da Silva, Ricardo de Godoy, Juliano Negri, Gustavo Jose Giardini Lahr, Ranulfo Bezerra, Marcelo Becker
University of São Paulo

A vision-based interface maps wrist motion and hand gestures to end-effector references with no wearables or calibration. A free-form text prompt is grounded by a vision-language model and tracked across the robot's cameras as a 3D grasp frame. A cuRobo MPPI controller tracks every command against a live nvblox ESDF, and an attractive potential field corrects the reference onto the target inside a 0.4 m activation radius.

Video Code </> Paper
The shared-autonomy framework in operation during an industrial valve manipulation task.

The operator's free motion and gestures drive Spot's arm; the assisted end-effector converges onto the language-grounded target "wheel valve" while avoiding the surrounding structure.

01. AbstractCalibration-free teleoperation, language-grounded targets, collision-free execution

Teleoperating a robotic manipulator in industrial environments demands precision that camera-based interfaces alone struggle to deliver. The operator must align the end-effector with a target in clutter, under limited depth perception, and without colliding with the surrounding structures. This paper presents a shared-autonomy framework that assists the operator throughout this process.

A single RGB-D camera captures the operator's arm motion and hand gestures without wearables, fiducials, or a calibration stage. The intended target is specified by a free-form text prompt, grounded by a vision-language model in the robot's gripper camera, and tracked across its onboard cameras by a promptable video-segmentation model, resulting in a grasp frame continuously separated from the obstacle map. Every commanded motion is executed by a GPU-accelerated model-predictive controller that enforces self- and environment-collision avoidance against an online volumetric reconstruction, while a potential field corrects the operator's reference toward the grounded target during the final approach. An autonomous mode can be gesture-triggered to complete the grasp on the same target without a separate perception pipeline. The framework is validated on a quadruped mobile manipulator. The interface achieves a positional RMSE of 59 mm relative to motion-capture ground truth, and the controller keeps the arm at least 18 cm from obstacles while the operator deliberately commands the arm into them by 6 cm. In an industrial valve manipulation and a pick-and-place task, the full framework succeeded in all trials, while ablating either the collision or the assistance module produced failures through complementary mechanisms, and autonomous execution succeeded in four of five trials per task.

59 mm
end-to-end tracking RMSE against motion-capture ground truth
≥ 18 cm
obstacle clearance while the operator commands 6 cm into the obstacle
10/10
full-framework success across both contact-rich tasks

02. DemonstrationSee it run

Industrial valve manipulation · pick-and-place · collision-avoidance stress test · autonomous execution

03. MethodOne perception, four layers of assistance

The framework composes four concurrent subsystems around a single perception of the scene: the operator interface, open-vocabulary grounding, volumetric mapping with collision-aware model-predictive control, and the assistance layer that bridges teleoperation and autonomy.

Framework overview diagram.

Framework overview. Operator tracking (MediaPipe on a ZED 2i) produces the end-effector reference; Qwen3-VL + SAM 2 ground and track the target; nvblox builds the obstacle map; cuRobo's MPPI MPC executes every motion, teleoperated or autonomous, under the same collision constraints.

Track the operator

Wrist motion maps to an end-effector reference through an online-estimated workspace scale; the palm normal sets gripper roll, and hand gestures command the gripper and mode switches. No wearables, no fiducials, no calibration.

MediaPipe · ZED 2i

Ground the prompt

A free-form prompt ("wheel valve", "drill") is grounded by Qwen3-VL in the gripper camera and tracked across three onboard cameras by SAM 2 streaming predictors, yielding a world-latched grasp frame kept out of the obstacle map.

Qwen3-VL · SAM 2

Plan collision-free

nvblox fuses the onboard stereo depth into a TSDF/ESDF, and cuRobo's MPPI-based MPC tracks the reference at 50 Hz while enforcing self- and environment-collision constraints against the live map.

nvblox · cuRobo · 50 Hz

Assist the approach

Inside a 0.4 m radius, an attractive potential field bends the operator's reference onto the grasp frame while the operator retains authority. A two-finger gesture delegates the grasp to the autonomous mode , on the same target, through the same planner.

potential field · autonomy
Open-vocabulary grounding of the text prompt in the gripper camera.

Open-vocabulary grounding. The prompt is grounded in the gripper camera and propagated to the onboard cameras, producing a grasp frame continuously separated from the static obstacle map.

Simulation teleoperation demo in Isaac Sim.

Simulation stack. The same pipeline drives Spot in Isaac Sim, with the arm under cuRobo MPC and the legs under a learned locomotion policy.

04. ResultsCollision avoidance and potential-field assistance suppress complementary failure modes

Validation proceeds in three stages: interface accuracy against motion capture, collision-aware shared control under deliberate operator error, and the full framework on two contact-rich industrial tasks with an ablation over its two assistance layers.

Collision-aware shared control experiment.

Collision-aware shared control. Left: reconstructed workspace with the commanded (orange), executed wrist (red), and executed hand (green) trajectories. Right: signed distance of each to the nearest obstacle. The operator deliberately drives the command 6 cm into the obstacle; the executed wrist and hand never come closer than 19 cm and 18 cm.

Wrist and gripper motion along each axis against motion-capture ground truth.

Teleoperation tracking accuracy. Operator wrist command vs. executed gripper motion along each axis, measured by OptiTrack. The full pipeline achieves an RMSE of 59 mm; deviations behave as lag during fast reversals rather than drift, preserving precision in the slow final approach.

ConditionValvePick & place
Teleoperation only0/51/5
+ Collision avoidance (C)2/52/5
+ Potential field (PF)3/54/5
Full framework (C + PF)5/55/5
Autonomous execution4/54/5

Ablation. Five trials per condition; any contact with the structure ends a trial as failure. Collision avoidance alone still misses grasps; assistance alone still contacts the structure. Together, every trial succeeds.

05. Acknowledgments

This work was supported by Petróleo Brasileiro S/A – Petrobras, using resources from the ANP R&D clause, in partnership with the University of São Paulo (USP) and the Fundação de Apoio à Física e à Química (FAFQ), under Cooperation Agreements #2023/00016-6 and #2023/00013-7.

06. CitationBibTeX

@misc{dasilva2026perceptionassistanceopenvocabularyshared,
      title={From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation},
      author={Murilo Vinicius da Silva and Ricardo V. Godoy and Juliano Negri and Gustavo J. G. Lahr and Ranulfo Bezerra and Marcelo Becker},
      year={2026},
      eprint={2607.17323},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.17323},
}