A good line cook doesn’t just follow a recipe. They can feel when a process is going off the rails: the grasp that didn’t quite close, the tongs slipping off a heavy meat patty, or two slices of cheese stuck together. They fix these situations before they lead to mistakes.
When an AI model learns exclusively from flawless human demonstrations (i.e., successful demonstrations), it will struggle the moment reality deviates from that perfect script. In our previous blog about the data flywheel, we improved our burger assembly robot by folding its own successes back into training: 4.5% more data, all of it clean runs executed by the robot policy itself. Success jumped from 75% to 91.3% in single burger-making evaluations.
This blog post explores a related idea: learning from mistakes, both the robot’s own mistakes and moments where a human needs to step in and take over (i.e., human interventions or human-in-the-loop (HITL)). Those takeovers turn out to be dense, nearly-free supervision and far cheaper than manually labeling mistakes. We used them to train ValueFormer, a small critic that observes the same camera streams as the policy and evaluates the policy’s performance at each step. We then put this critic to work in two places: as a live safety monitor that flags failures, and as a critic loop that re-weights the policy’s own training data according to the critic’s scores.
We evaluate on consecutive burger assembly: two burgers in a row, no scene reset in between. Single-burger assembly (scene reset first) is easier and reliably achievable for our vision-language-action model (VLA), so the consecutive setting is our real test. It stress tests the policy because performance is highly sensitive to scene staging, hard to quantify, and difficult to reproduce, since ingredients are deformable and the scene evolves over time. The policy’s own actions compound this (e.g., a patty tips over when the robot picks another, or the cheese stack gets pushed), making the second burger assembly task substantially harder (a natural form of hard-example mining). ValueFormer increased successful burger assembly from 70% to 85%.
The gap between doing and knowing
The Food Foundation Model (FFM) is a flow-matching VLA policy in the π₀ family. It can do genuine bi-manual manipulation: two arms, six ingredient bins, six ordered stages, a full sandwich in under a minute. What it cannot do is evaluate its own performance.
This limitation is a direct consequence of how we trained our robot. The policy learns entirely from behavior cloning (BC), imitating expert demonstrations frame by frame. It acts like a line cook who reproduces a chef’s motions perfectly but never tastes the result (i.e., lack of reward or feedback). It doesn’t get a signal on whether the dish turned out well. There is no built-in reward, no score, and no notion of progress. This is one of the well-established limitations of BC [1].
As a result, entirely different situations look exactly the same to the system’s internal logic. Whether the robot is struggling to grasp a piece of lettuce, smoothly picking up a second patty by mistake, or perfectly on track to finish the meal, it outputs joint commands with the exact same confidence. It executes mistakes just as “flawlessly” as successes, meaning the core policy has no calibrated sense of whether it is winning or losing.
The critic behavior cloning never had
Many of today’s most successful reinforcement learning (RL) methods train two models together: an actor (the policy) and a critic. Think of the critic as an internal scorekeeper that continuously asks, “Am I still on track to succeed?” The critic estimates how promising the current situation is, and the policy uses that feedback to gradually learn which decisions lead to better outcomes.
BC learns only the policy. Because it is trained purely by imitating expert demonstrations, there is no reward signal from which a critic can learn. As a result, there is no built-in notion of progress or internal score telling the robot whether it is getting closer to or farther from completing the task.
One might ask: why not just use RL? In principle, we could. In practice, RL for robotic food manipulation would require either enormous amounts of real-world experience or a highly realistic simulator. Simulating deformable, granular food ingredients accurately enough remains an open challenge, leaving a large sim-to-real gap.
This means we need to train a distinct value function alongside the main policy: our standalone critic model, ValueFormer (see the diagram below). Bringing this critic online allows us to leverage a proven RL architecture: the actor-critic method, in which the critic’s real-time evaluations are folded back in to improve the actor’s performance.
ValueFormer is deliberately tiny: a compact causal transformer (two layers, four heads, d_model=256, about 3.5M trainable parameters) sitting on the frozen DINOv3 backbone (ViT-L/16 at 518 px) that the policy already runs. For each frame, it encodes six views: the three cameras (overhead, left wrist, right wrist) plus three region-of-interest (ROI) crops into a 6144-dim descriptor. It then adds (not concatenates) the 14-dim bi-manual joint state and a time feature. By using addition instead of concatenation, we prevent the high-dimensional visual embeddings from mathematically overpowering the much smaller joint signals. Including these close-up wrist camera views is important for the robot’s responsiveness. A failed grasp is visible in the wrist feed seconds before the overhead camera registers the downstream consequence of the mistake. Its context window is 8 seconds of history (16 samples at 2 Hz). The context window size is determined by a hyper-parameter sweep on a validation set.
ValueFormer is designed to be lightweight and practical. It is policy-agnostic, meaning we can snap it onto our existing AI without rebuilding or retraining the robot’s core system (it only shares the same visual feed). It does still require task-specific labels and training.
Our critic is also fast to train: under three minutes on an NVIDIA RTX 5090 GPU using a dataset of about 1,400 sandwich-making runs. The one-time data prep is more work-intensive. It takes about 90 minutes. Once the data is ready, creating the critic is almost instant. We’ll discuss deployment inference speed in more detail in a later section.
Why a causal transformer
Two choices in this small model do a lot of the work. Both relate to time.
First, ValueFormer is a transformer over history, not a per-frame classifier. Success and failure aren’t things we can determine from a single snapshot; they reveal themselves over a sequence of frames (e.g., a grasp that didn’t quite close, an arm drifting toward the wrong bin, or reaching for a second patty in a burger that already has one). Each failure begins with a trajectory going wrong, and none of them is obvious in any one image. So at every frame, the critic reads the last 6-8 seconds of the run and asks, “Given how the last few seconds have gone, how am I doing right now?” This temporal context is exactly what lets it tell apart the three look-alike runs from earlier ones (the stall, the second patty, the clean run) that a frame-by-frame model cannot.
Second, ValueFormer is a causal transformer: a masked decoder (right in the figure below) rather than a bi-directional encoder (left in the figure below). A “frame” in this context is the single fused per-frame embedding built above (i.e., the six views, joint state, and time for one timestep, combined into one token=1x256-dim), so the sequence the model attends over is time, one token per timestep. Attention lets each such frame pull in information from the others: the frame doing the looking is the query, and the frames it can look at are the keys. The two designs differ only in regard to which keys each query may see:
- A bi-directional encoder (BERT-style) lets every frame attend to every other frame in the window, including future ones. It is the natural choice if we already hold the whole clip and just want the best labels after the fact.
- A causal, masked decoder (GPT-style) uses a triangular attention mask: frame “t” may attend to frames up to and including “t,” but not beyond. Each prediction sees only the past.
We use the causal pattern because of deployment. The monitor (i.e., ValueFormer) runs live on the robot (a forward pass every half-second, next to the policy), so in the moment it scores a frame, the future has not happened yet. Bi-directional training (i.e., encoder-type training) on full clips allows models to cheat by peeking at future outcomes and relying on data unavailable in real time. A causal mask solves this by limiting frames to past data only, ensuring predictions remain honest and trustworthy during a live operation.
ValueFormer predicts two per-frame signals off the shared backbone that pull in opposite directions on purpose:
- V_mc: the smooth Monte Carlo (MC) value (i.e., the critic). Represented by the green line in Figure 1 in [0,1] (e.g., 0.6 or 0.9), higher is better. It tracks gradual progress toward a finished burger.
- V_bin: the sharp per-frame mistake detector (i.e., the safety filter). We report it as a failure probability (1 − V_bin). Represented by the orange line in Figure 1. Lower is better. The line increases immediately when a mistake begins.
A critic wants a smooth, slowly-varying target so it can measure whether things are trending up or down. A safety filter wants a sharp step at the exact frame a mistake starts occurring, so an operator has a clear moment to act.
The label is a design choice, too
The model is not the only place where design effort is justified; the architecture design of the causal, two-head critic matters, too. It’s tempting to stop here, simply pour all of our data into the network, and treat the training target as something fixed. When we hold the model architecture fixed and vary only the label, the labeling choice matters a lot. A continuous labeling scheme produced 2-3x better results in value prediction mean absolute error (MAE: 0.024) than the other labeling variants we tried (outcome-scaled: 0.058, cliff: 0.047, C-linear: 0.058, and C-late: 0.048). More details will be published in an auxiliary paper. From this observation, we conclude that the labeling choice deserves the same care as the model design.
The naive MC label marks every frame of a failed episode as 0 (red solid line in the left chart below). In practice, a failed run looks identical to a successful one until the mistake happens, and labeling that clean prefix a “failure” teaches the model that ordinary (good) observations mean failure. At evaluation, that leads to misreading perfectly good runs, as predictions start oscillating near zero.
The fix is a stage-aware, success-then-decay label. On a failed episode, the pre-failure frames follow the same rising success curve as on a good run. The value only decays smoothly after the failure stage. This allows partial credit: a run that successfully placed four out of six ingredients before failing keeps a higher tail than a run that failed at the first step. The post-failure decay is exponential, not linear, because a linear label would flatten the downstream advantage into a constant that a later filter couldn’t read.
Free supervision from human takeovers
The critic is only as good as its labels (upper bound), and hand-labeling requires significant time and labor effort. Therefore, we draw on two label sources of very different sizes:
- 88 hand-marked failures. We labeled the failure moments on our set of 88 failed episodes in under two hours (about a minute per episode). This method is accurate but labor-intensive and difficult to scale. One label example is “total episode duration: 75.5s, failure occurred: 47.1, reason: couldn’t find patty.”
- 333 human-takeover segments across 129 episodes (105 episodes for training, 24 episodes held out for validation). Whenever an operator takes over a struggling run, the recording already carries a per-frame intervention binary flag, so each takeover is a free, dense label: a “going wrong starts here, fixed by here” dip-and-recovery, exactly the good → bad → good gradient a critic needs, and which a terminal pass/fail label never captures.
Takeovers may lag the moment things actually go wrong (i.e., the operator reacts too late), so we correct for that timing before training. The payoff is leverage: two hours of hand-labeling, plus supervision the fleet already generates for free. In addition, there are short frozen moments when the human operator takes over control that we need to detect and compensate for (i.e., frame dropping for images and linear interpolation for joint states) in order to avoid learning that behavior.
Impact: folding the critic back into the policy
As the critic can assess whether a run is improving or worsening, we can use its output to improve the policy itself. The idea is to double down on the moments where the critic says things are going well during training and discount the moments where it says things are going wrong. This is a single, targeted improvement step, done offline on the data we already have rather than letting the robot experiment with live burger assembly tasks.
We try several ways of turning takeovers into a training signal. All of them kept the expert’s data at full weight and differed only in how they treat the autonomous rollout data:
- Intervention Weighted Regression (IWR) (the baseline): trust the takeover flag alone (a higher constant weight for takeover), with no critic.
- vf-mask (valueFormer-mask): let the critic switch off the data it judges is failing (about 1 in 10), a smarter replacement for a fixed cut-off window.
- vf-awr (valueFormer Advantage Weighted Regression): let the critic weight each data point by how much better or worse things got, easing off only where it matters.
How it did on the robot
We ran two on-hardware A/B tests against the same control (C vs. T1 and C vs. T2):
- C: flag-only IWR (the baseline)
- T1: vf-mask
- T2: vf-awr
All three share the same training set (985K frames) and the same high weight during takeover. The only difference is how each of them treats the autonomous frames of episodes with human interventions (takeover): IWR keeps them as is, vf-mask zeroes the ones the critic flags as failing, and vf-awr weights them by the critic’s advantage (i.e., scoring indicator; how much better or worse at the moment). So each test isolates the critic-derived weight on identical data.
The two critic variants (vf-mask and vf-awr) reached the same headline completion with slightly different approaches:
vf-mask is the superior of the two: it cuts the number of catastrophic subtasks from 7 to 2, clears redundant repeat-picks entirely (from four to zero), and is the only variant whose second burger came out better than its first (the completion figure’s right panel shows it gaining, while every other variant, the control included, slipped on the harder second). vf-awr got the highest quality instead: the best mean subtask score of any variant (0.96, against the IWR control’s 0.91), with the same clean zero on repeat-picks and the same drop in catastrophic subtask. Both reach 85% completion at a median duration within seconds of the control (1:21 and 1:22 vs. 1:15). This apparent slowdown is mainly due to the IWR control failing fast, whereas the two critic-in-the-loop variants learn to avoid that quick but incorrect motion. They thus keep correctly working a hard target until they succeed.
Two different critic-derived weights (a hard on/off gate and a smooth continuous advantage) got the same 85% completion and the same zero on repeated picks (e.g., two patties), each clearly outperforming the no-critic baseline.
How it keeps pace with the policy
The progress monitor is only valid and useful if it keeps pace with the policy instead of slowing it down. Both live on the same RTX 5090 (32 GB): Pi0.5 on a JAX server, ValueFormer as a PyTorch server queried at 2 Hz. It is advisory and fail-safe, so the only real risk is the reverse: the critic starving the policy’s 2.5 Hz loop (400 ms), which has a ~100 ms prefetch budget before motion hitches.
Under the concurrent load (i.e., ValueFormer alongside the VLA policy), ValueFormer’s inference time stretches toward 400ms, mainly due to the frozen encoder’s feature extraction: six DINOv3 ViT-L/16 passes run sequentially at batch 1 in FP32. That collision with the policy eventually stalls motion, so we fixed the encode path: batch the six views on-GPU and adjust precision.
With this optimization, one inference goes from 350 ms to ~85 ms (bf16), a 4x speed improvement: there is subtle regression in AUROC (from 0.943 to 0.942), but detection rate is maintained (0.85).
Discussions and limitations
These are early, promising results because each version was tested on only 20 burger assembly tasks. A larger, back-to-back re-test is the next step with independent scorers. We will also be upfront about one surprise: the continuous-advantage version did better than we expected, matching the best result rather than trailing as we had predicted.
On safety, the monitor is an assistant, not an autopilot. It raises a flag for a person to see. It does not yet pause or take over the robot on its own, and it needs more tuning before it could. Most importantly, ValueFormer does not replace the person. It is a helper that tells someone (an operator or a supervisor watching a whole fleet) when to look into a potential issue.
What’s next
Our team will work on three things next. We will push the self-improvement loop one step further, feeding the critic’s judgment to the policy more directly. We will aslo firm up the evidence with more back-to-back re-tests, and keep tuning the safety monitor until it can be trusted to pause or hand off the robot on its own, not just alert a human.
The bigger prize is the same one that our data flywheel points is directed at: our robots already work in food manufacturing facilities every day, generating what we believe is the world’s largest dataset of real-world food manipulation. Every human takeover in that stream is a free, dense label. The flywheel learns from the robot’s wins, and ValueFormer learns from mistakes and from moments a person stepped in. Applied to our fleet of robots, the two loops run together: the robot keeps more of what works, notices sooner when it doesn’t, and gets better at both.
If you’re thinking about physical AI in commercial kitchens or want to follow our work, get in touch with our team.
Contributors
Inkyu Sa, Luis Rayas, Konstantin Stulov, Tina Kao, Nick LaBounty, Krishna Teja
References
[1] Florence, P., Lynch, C., Zeng, A., et al. (2021). Implicit Behavioral Cloning. arXiv. https://doi.org/10.48550/arxiv.2109.00137
.jpg)
