Dual-Stream Safety for Robot-to-Human Handover: How I Made a VLA Robot Hesitate

This is the technical companion to my master's dissertation, Safe Tool Handover in Human-Robot Collaboration. If you'd rather have the human story , the year, the struggle, the family who carried it , that's in a separate post. You can also read or download the full dissertation.

Here I want to walk through the actual engineering: the problem, the architecture, the experiment, and the result that surprised me.

The silent-confidence problem

Vision-Language-Action (VLA) models are the reason robots suddenly feel capable. OpenVLA, Physical Intelligence's pi-zero, and their siblings take a camera image plus a natural-language instruction , "hand the screwdriver to the person" , and emit motor actions end to end. No explicit reprogramming for new objects. It's genuinely impressive.

The failure mode is the catch. When a VLA model hits an input outside its training distribution, it doesn't degrade into uncertainty. It stays confident and acts on a wrong belief. In robotics this is sometimes called silent failure, and it's the precise property that makes these models dangerous in close collaboration. A robot working alone can fail loudly and cheaply. A robot mid-handover, with a hand reaching toward its gripper, cannot.

So the question of the dissertation became: can a runtime safety monitor catch these failures fast enough, without being so twitchy that it blocks the handovers that were fine?

Why tool handover is the right test case

Of all the collaborative tasks I could have studied, handover is the most demanding, which is exactly why it's the most revealing. A handover forces a robot and a human to share one object, at one point in space, inside a tight time window. Both parties have to commit to complementary actions before either is certain the other is ready , the giver has to release exactly as the receiver grasps. Get the timing wrong and you either drop the tool or never let go.

That makes it a stress test for two completely different things at once: the robot's own competence, and its read of the human. Which is the whole reason one monitor isn't enough.

Two streams, because failures come from two places

The core argument is an ablation. There are two independent ways a handover can go wrong, and they live on opposite sides of the interaction.

Stream One watches the robot. It's modeled on the SAFE line of work , read the policy's own internal state (in full OpenVLA, the hidden-layer activations) at each timestep, push it through a small classifier trained only on successful rollouts, and emit a calibrated confidence score. Below a threshold, the robot slows or halts. This stream catches failures that corrupt what the robot perceives or how it acts: a shifted camera, bad lighting, an unfamiliar tool shape.

Stream Two watches the human. It takes the partial hand trajectory and predicts the reach intent and target before the reach completes. This is the only way to catch a failure that never shows up in the robot's own state , a human approaching at the wrong angle, or reaching in before the robot has even presented the tool.

The fusion layer is deliberately rule-based, not learned. A learned fuser would need failure data to train on, and you can't collect that ethically near a person. So fusion is a transparent threshold controller with four graduated responses:

• both streams calm → run at full speed
• one stream moderately worried → slow to half speed
• either stream strongly worried → yield the workspace
• both alarmed at once → halt and request confirmation

Rule-based keeps the whole thing auditable, which matters enormously if you ever want a safety body to certify it. You can explain every intervention trial by trial.

The experiment

Four conditions, identical in everything except which streams are active:

1. Unmonitored , raw OpenVLA, the floor any monitor has to beat
2. Confidence-only , Stream One alone
3. Intent-only , Stream Two alone
4. Dual-stream , both, fused

Everything ran in PyBullet , a deterministic, headless physics sim , on a 7-DOF Franka Emika Panda with a parameterised kinematic human arm reaching along a minimum-jerk trajectory. Three tools (screwdriver, mallet, pencil) spanned the dimensions that matter for safe handover: orientation for sharp objects, timing for heavy ones, precision for small ones.

Against each condition I threw four out-of-distribution failure groups , camera shift, lighting variation, novel tool geometry, and unexpected human approach , plus a nominal control group to measure false positives. Fifty trials per cell, 1,000 trials total, zero dropouts, fully reproducible from a fixed base seed.

Crucially, the criterion for success was pre-registered , fixed in writing before any data existed. The dual-stream system would only be judged to have won if it beat its baselines on a three-part test. No moving the goalposts after seeing the numbers.

The thresholds for both streams were calibrated on successful nominal trials only, using conformal prediction (Angelopoulos & Bates), then frozen. So the system was never tuned on its own test data.

The headline result

Table 4.1 , Failure scenarios, 200 trials per condition.

The unmonitored baseline completed every failure trial, and roughly a quarter of those completions were unsafe , the tool released into the hand when the grasp had actually failed. Both conditions that consult the confidence stream drove unsafe completions to zero. The dual-stream system recorded the highest intervention rate at 81.5%.

That alone might suggest Stream Two is redundant. It isn't , and this is the result the whole design exists to surface.

Where the second stream earns its place

Break the intervention rate down by failure type and the picture splits cleanly in two.

Figure 4.1 , Safety intervention rate by failure scenario. Robot-side failures are saturated; the human-side approach is the only place the second stream changes the outcome.

On the three robot-side failures , camera shift, lighting, novel geometry , the confidence stream is essentially perfect (98–100%). Adding the intent stream changes nothing there, because those failures are saturated already.

On the single human-side failure , the unexpected approach trajectory , everything inverts. The confidence stream catches only 8%, because that failure never shows up in the robot's own internal state. The intent stream lifts it to 18%, and the full dual-stream system to 28%.

That's the contribution in one figure: each stream covers a failure family the other can't, and only the combination addresses both. Watch only the robot, or only the human, and a whole class of failure walks straight through.

The honest weak point

I'm not going to oversell 28%. That human-side number is the weakest result in the dissertation, and naming it is part of the work. The intent stream uses a geometric extrapolator on an idealised minimum-jerk reach , a deliberate simplification. The single change most likely to raise that number is replacing it with the trained transformer the design actually calls for, which is the first item on the future-work list.

What I can defend: the intent recogniser hits 99.5% prediction accuracy by the halfway point of the reach, clearing the lead-time threshold the literature sets as the minimum useful.

Figure 4.3 , Intent prediction accuracy reaches 99.5% by the halfway point of the reach.

So the recogniser is accurate; the bottleneck is the geometric intent model feeding the controller, not the prediction timing.

Did safety cost fluency?

The fear with any safety monitor is that it makes the robot useless , too cautious to be worth it. The nominal control says it didn't.

Table 4.2 , Nominal control, 50 trials per condition.

The full system's false-positive cost sits within four points of the unmonitored baseline, and confidence-only matches the baseline almost exactly on handover timing. Monitoring the robot added no measurable delay to a handover that was going to succeed. Safety and fluency were not traded against each other.

Figure 4.4 , Outcome breakdown by condition. The confidence-using conditions show no missed catches and a 1% false-alarm share.

What this does and doesn't claim

Honesty about scope is the whole point of a pre-registered study. This is simulation-only evidence. It does not deliver a deployment-ready safety system, it does not validate a physical robot near a real worker, and it does not solve the sim-to-real or cross-worker generalisation problems.

What it does establish is narrower and firmer: under controlled conditions, a dual-stream architecture covers a failure family that neither stream covers alone; monitoring the robot adds no measurable delay to a successful handover; and a transparent, rule-based fusion can hold its false-positive cost to a few points while eliminating unsafe completions entirely.

The roadmap from here, in order: integrate full OpenVLA so Stream One reads genuine activations; replace the geometric intent extrapolator with the trained transformer; then move from idealised to recorded human motion, and finally to hardware on a real Panda , carrying over the calibrated thresholds and fusion logic this simulation stage was built to produce.

The structural lesson

Strip away the specific numbers and one conclusion remains, and it's the reason a single handover task is worth studying at all:

A robot is not safe or unsafe in the abstract. It is safe or unsafe doing a particular task next to a particular person, and the only way to know which is to monitor both the robot and the person and to act on what the monitoring says.

That's the regulatory direction of travel too , certifying a collaborative application, not a machine in isolation. Watching only one side leaves a failure family uncovered, and the cost of watching both is small enough that there's no good reason not to.

The code, configuration, seed list, and analysis scripts are openly available, and the full dissertation is here to read or download. If you want the story of building it rather than the architecture, that's over here.