Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
426 lines (382 loc) · 22.4 KB
/
Copy pathindex.html
File metadata and controls
426 lines (382 loc) · 22.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Foresight: Iterative Reasoning About Clues that Matter for Navigation</title>
<meta name="description"
content="Foresight: iterative plan-critique refinement for open-world mapless navigation from sparse language instructions.">
<meta name="keywords"
content="vision-language models, robot navigation, test-time reasoning, reinforcement learning from human feedback, motion planning">
<meta property="og:type" content="website">
<meta property="og:title" content="Foresight: Iterative Reasoning About Clues that Matter for Navigation">
<meta property="og:description"
content="Foresight: iterative plan-critique refinement for open-world mapless navigation from sparse language instructions.">
<meta property="og:image" content="./static/images/mainfigure.jpg">
<link rel="icon" type="image/x-icon" href="./static/images/favicon.ico">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link
href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&family=Source+Serif+4:opsz,wght@8..60,400;8..60,500;8..60,600;8..60,700&display=swap"
rel="stylesheet">
<link rel="stylesheet" href="./static/css/index.css">
</head>
<body>
<header class="titleblock">
<div class="wrap">
<h1>
Foresight
<span class="sub">Iterative Reasoning About Clues that Matter for Navigation</span>
</h1>
<div class="venue award"><span class="ico">★</span> Best Paper Award<span class="sep">•</span><span class="keepcase">CoRL-SISREN 2026</span></div>
<p class="authors">
<a href="https://www.arthurkzhang.com" target="_blank" rel="noopener">Arthur Zhang</a><sup>1</sup>,
<a href="https://carl-qi.github.io/" target="_blank" rel="noopener">Carl Qi</a><sup>1</sup>,
<a href="https://snowbawler.github.io" target="_blank" rel="noopener">Donne Su</a><sup>1</sup>,
<a href="https://xymeng.github.io/xymeng/" target="_blank" rel="noopener">Xiangyun Meng</a><sup>2</sup>,
<a href="https://amyzhang.github.io/" target="_blank" rel="noopener">Amy Zhang</a><sup>1</sup>,
<a href="https://www.joydeepb.com/" target="_blank" rel="noopener">Joydeep Biswas</a><sup>1</sup>
</p>
<p class="affil"><sup>1</sup>The University of Texas at Austin<span class="sep">•</span><sup>2</sup>Field AI
</p>
<div class="logos">
<img src="./static/images/logos.png?v=2"
alt="Logos of the Autonomous Mobile Robotics Laboratory, the MIDI Lab, Field AI, and The University of Texas at Austin">
</div>
<nav class="btnrow">
<a class="btn" href="./static/pdfs/ForesightPreprint.pdf"><span class="ico">☰</span> Paper</a>
<a class="btn outline" href="#deployment"><span class="ico">▶</span> Videos</a>
<a class="btn outline" href="https://github.com/ut-amrl/foresight_public" target="_blank" rel="noopener"><span
class="ico">❮❯</span> Code</a>
<span class="btn pending"><span class="ico">▤</span> Dataset (coming soon)</span>
</nav>
</div>
</header>
<!-- ===================== HERO ===================== -->
<section class="hero">
<div class="wrap-wide">
<figure class="frame" style="margin:0 auto">
<video autoplay muted playsinline controls loop preload="auto"
poster="./static/posters/websitemainfigure_v2.jpg">
<source src="./static/videos/websitemainfigure_v2.mp4?v=2" type="video/mp4">
Your browser does not support the video tag.
</video>
</figure>
<figcaption>Foresight iteratively proposes and critiques image-space motion plans with a finetuned
vision-language model, refining navigation behavior at test time to follow sparse language instructions in
open-world environments.</figcaption>
</div>
</section>
<!-- ===================== ABSTRACT ===================== -->
<section id="abstract">
<div class="wrap">
<p class="section-kicker">Abstract</p>
<h2>Discovering which cues matter, then planning using them.</h2>
<p>Open-world mapless navigation from sparse language instructions requires inferring which environmental cues
matter — the ramp, sign, or detour that reveals where to go. Prior works draw cues from closed-set factor
categories or identify them before motion planning, missing plan-dependent cues. Pretrained Vision-Language
Models (VLMs) can discover such cues, but require adaptation to act on them.</p>
<p>We present <strong>Foresight</strong>, a test-time framework in which a finetuned VLM alternates between
proposing image-space motion plans and critiquing them against the language goal and visual context,
conditioning each plan on prior critiques to refine its motion before execution. A reward model learned from
human feedback post-trains the VLM with reinforcement learning inside this loop, aligning it with open-set
behavior preferences. Across offline evaluations and 6 real-world environments, Foresight improves average task
success by <strong>37%</strong> and reduces interventions per mission by <strong>52%</strong> relative to
state-of-the-art test-time reasoning and foundation-model baselines, while running in real-time on a Jetson AGX
Orin.</p>
</div>
</section>
<!-- ===================== APPROACH ===================== -->
<section id="approach">
<div class="wrap">
<p class="section-kicker">Approach</p>
<h2>Test-time reasoning via iterative plan-critique refinement.</h2>
<p>Foresight formulates motion planning as a test-time reasoning loop. Conditioned on the image observation and
language instruction, a finetuned VLM proposes an image-space motion plan and then critiques it with respect to
the language goal and visual context. Subsequent plans are conditioned on prior critiques, enabling the model to
surface <em class="term">plan-dependent cues</em> — ramps, signs, detours — and iteratively refine
its motion before execution. A lightweight policy converts the refined image-space plan into executable
trajectories.</p>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<a href="./static/images/mainfigure.jpg">
<picture>
<source srcset="./static/images/mainfigure.webp" type="image/webp">
<img src="./static/images/mainfigure.jpg" alt="Foresight architecture overview" loading="lazy">
</picture>
</a>
</div>
<figcaption><strong>Figure 1.</strong> The Foresight loop. A single finetuned VLM alternates between proposing
an image-space motion plan and critiquing it against the instruction and visual context, with each new plan
conditioned on the critiques that preceded it.</figcaption>
</figure>
</div>
</section>
<!-- ===================== TRAINING ===================== -->
<section id="training">
<div class="wrap">
<p class="section-kicker">Training</p>
<h2>Supervised finetuning, then reinforcement learning with human preferences.</h2>
<p>To adapt pretrained VLMs for iterative refinement, we propose a simple two-stage training recipe. We first
apply
supervised finetuning to teach the VLM the structure of the plan-critique loop. We then learn a reward model
from human preference annotations to judge motion plan quality, and use it to post-train the VLM with
reinforcement learning, aligning critiques and refinements with open-set behavior preferences that are
difficult to capture with hand-designed objectives alone.</p>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<video muted playsinline controls loop preload="metadata" data-autoplay
poster="./static/posters/trainingrecipe.jpg">
<source src="./static/videos/trainingrecipe.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
<figcaption><strong>Figure 2.</strong> The two-stage training recipe, from supervised finetuning of the
plan-critique structure through reward-model-driven reinforcement learning.</figcaption>
</figure>
</div>
</section>
<!-- ===================== DOMAINS ===================== -->
<section id="domains">
<div class="wrap">
<p class="section-kicker">Domains</p>
<h2>Training, evaluation, and deployment contain distinct clues and locations.</h2>
<p>Training data, the offline benchmark, and the real-robot deployments are drawn from distinct geographic
regions, so no test evaluation site appears in training. Instructions are collected the way an operator would
phrase them: <em class="term">“go to the nearest entrance”</em>, <em class="term">“use
the wheelchair friendly path”</em>, which leaves the relevant cue
unstated and forces the model to determine what to look for.</p>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<a href="./static/images/trainingdataset-satellite.jpg">
<picture>
<source srcset="./static/images/trainingdataset-satellite.webp" type="image/webp">
<img src="./static/images/trainingdataset-satellite.jpg"
alt="Satellite map with disjoint training, offline evaluation, and online evaluation regions, alongside example training instructions and held-out test episodes with plan critiques."
loading="lazy">
</picture>
</a>
</div>
<figcaption><strong>Figure 3.</strong> <em>Left:</em> training, offline evaluation, and online evaluation
routes occupy disjoint parts of the map; the six deployment sites are labelled. <em>Right:</em> training
episodes pair an observation with a sparse instruction, while held-out test episodes additionally show the
critique that redirects the initial plan toward the correct cue.</figcaption>
</figure>
</div>
</section>
<!-- ===================== DEPLOYMENT ===================== -->
<section id="deployment">
<div class="wrap">
<p class="section-kicker">Real-robot deployment</p>
<h2>Closed-loop missions across six unseen environments.</h2>
<p>We deploy Foresight on a real robot and follow sparse language instructions in six environments held out from
training, grouped into three families: detour rerouting, structural clues, and sign understanding. Each site is
run against the same baselines, and an operator intervenes whenever the robot commits to an unrecoverable
plan.</p>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<a href="./static/images/testinglocations.jpg">
<picture>
<source srcset="./static/images/testinglocations.webp" type="image/webp">
<img src="./static/images/testinglocations.jpg"
alt="Six real-world deployment scenes grouped into detour rerouting, structural clues, and sign understanding, each with the executed route overlaid."
loading="lazy">
</picture>
</a>
</div>
<figcaption><strong>Figure 4.</strong> The six deployment sites with executed routes overlaid. Detour rerouting
(blue) blocks the direct path; structural clues (red) hide the route in ramps, stairs, and handrails; sign
understanding (purple) requires reading posted text to pick a direction.</figcaption>
</figure>
</div>
<div class="wrap">
<div class="headline">
<div class="h">
<div class="v">37%</div>
<div class="k">higher average task success than state-of-the-art baselines.</div>
</div>
<div class="h">
<div class="v">53%</div>
<div class="k">fewer interventions per waypoint than the strongest baseline.</div>
</div>
<div class="h">
<div class="v">6</div>
<div class="k">unseen environments, spanning three families of navigation cue.</div>
</div>
</div>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<a href="./static/images/robotexperiments.jpg">
<picture>
<source srcset="./static/images/robotexperiments.webp" type="image/webp">
<img src="./static/images/robotexperiments.jpg"
alt="Left: average interventions per waypoint by cue category. Right: real-world success rate against intervention budget per waypoint."
loading="lazy">
</picture>
</a>
</div>
<figcaption><strong>Figure 5.</strong> <em>Left:</em> average interventions per waypoint, overall and split by
cue family. <em>Right:</em> success rate as a function of the intervention budget an operator is willing to
spend per waypoint.</figcaption>
</figure>
</div>
<div class="wrap">
<p>Both post-training steps matter on hardware, but refinement matters more. <em>Foresight, no refine</em> lands
alongside the LeLaN and Alpamayo baselines, showing that a finetuned planner that never critiques itself behaves
much
like a feed-forward one. While <em>Foresight, no RL</em> outperforms all other non-refinement baselines. The
margin is widest on structural clues, where the relevant evidence only becomes salient once the critic
identifies the clue.</p>
<p>The success-rate curve provides a more deployment-centric view. With no interventions allowed, Foresight
already completes more than half of the missions while every external baseline finishes under a fifth of them.
Granting a budget of a single intervention per waypoint pushes Foresight close to saturation without closing the
gap for the baselines, which still need multiple attempts to generate a valid plan.</p>
</div>
<div class="wrap-wide">
<div class="video-block">
<h3>Long-horizon navigation mission</h3>
<div class="vframe">
<video muted playsinline controls loop preload="metadata" data-autoplay
poster="./static/posters/foresight_longhorizon.jpg">
<source src="./static/videos/foresight_longhorizon.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</div>
<div class="video-block">
<h3>Head-to-head comparisons</h3>
<p class="lede">Each clip runs LeLaN, Alpamayo, and Foresight on the same instruction and environment.</p>
<div class="qual-switcher" id="qualSwitcher">
<div class="qual-tabs" role="tablist" aria-label="Qualitative comparison categories">
<button class="qual-tab is-active" type="button" role="tab" aria-selected="true"
data-src="./static/videos/detour1-compound.mp4" data-poster="./static/posters/detour1-compound.jpg"
data-caption="Detour rerouting tasks require inferring instruction-aligned alternative routes when the current path is blocked.">
Detour Rerouting
</button>
<button class="qual-tab" type="button" role="tab" aria-selected="false"
data-src="./static/videos/sign2-compound.mp4" data-poster="./static/posters/sign2-compound.jpg"
data-caption="Sign following tasks require interpreting textual signs to reach the goal.">
Sign Following
</button>
<button class="qual-tab" type="button" role="tab" aria-selected="false"
data-src="./static/videos/structural1-compound.mp4"
data-poster="./static/posters/structural1-compound.jpg"
data-caption="Structural clue tasks require understanding structures like walkways, badge signs, or door mats that reveal the intended route.">
Structural Clues
</button>
</div>
<div class="vframe">
<video id="qualVideo" muted playsinline controls loop preload="metadata" data-autoplay
poster="./static/posters/detour1-compound.jpg">
<source id="qualVideoSource" src="./static/videos/detour1-compound.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
<p class="qual-caption" id="qualCaption">Detour rerouting tasks require inferring instruction-aligned
alternative routes when the current path is blocked.</p>
</div>
</div>
</div>
</section>
<!-- ===================== OFFLINE BENCHMARK ===================== -->
<section id="benchmark">
<div class="wrap">
<p class="section-kicker">Offline benchmark</p>
<h2>Where the gains come from, compared against baselines and ablations.</h2>
<p>Offline, we score predicted image-space plans against human-demonstrated ones with the Hausdorff distance at 4
unique urban environments, which lets us isolate each design decision for larger models infeasible to run on the
robot. We sweep the number of refinement iterations and compare against both general-purpose VLMs and robot
foundation models,
then ablate the reward design and the training configuration of the planner and the critic.</p>
</div>
<div class="wrap-wide">
<figure class="fig-single wide">
<div class="imgwrap">
<a href="./static/images/combined_ablations_hausdorff.jpg">
<picture>
<source srcset="./static/images/combined_ablations_hausdorff.webp" type="image/webp">
<img src="./static/images/combined_ablations_hausdorff.jpg"
alt="Three boxplot panels of Hausdorff distance against refinement iteration: baseline comparisons, reward design ablations, and motion planner and critic ablations."
loading="lazy">
</picture>
</a>
</div>
<figcaption><strong>Figure 6.</strong> Hausdorff distance to the reference plan against refinement iteration;
lower is better. <em>Left:</em> comparison against general-purpose and robot foundation model baselines.
<em>Middle:</em> reward design ablation. <em>Right:</em> which of the planner and the critic must be adapted.
</figcaption>
</figure>
</div>
<div class="wrap">
<h3>Refinement only pays off once the model is trained to critique.</h3>
<p>In the baseline panel, the general-purpose VLMs sit flat across refinement iterations as extra reasoning
steps do not help a model that cannot tell a good plan from a bad one. Alpamayo, being feed-forward,
has no refinement to give past the initial reasoning step. Foresight acheives the lowest-error method by
improving from iteration 0 to 1, with most of the benefit realized by the first critique. Its
error distribution is also visibly tighter, which is what thins out the tail of bad plans that force
interventions on hardware.</p>
<h3>Both reward terms and modules, are necessary.</h3>
<p>Neither reward term alone reproduces the full model: training against either in isolation leaves a clearly
higher median at both iteration counts, and only their combination shifts the whole distribution down. The
planner/critic ablation makes a related point, but asymmetrically. A zero-shot planner (<em>ZS/SFT</em>) is
catastrophic, with a median error several times that of any adapted variant — a good critic cannot rescue
plans that were never plausible. A zero-shot critic (<em>SFT/ZS</em>) is far less damaging and lands close to
finetuning both (<em>SFT/SFT</em>), which is why reinforcement learning on top of <em>SFT/SFT</em>, rather than
supervised finetuning alone, is what produces the final margin.</p>
</div>
</section>
<!-- ===================== LIMITATIONS ===================== -->
<section id="limitations">
<div class="wrap">
<p class="section-kicker">Limitations</p>
<h2>What iterative refinement does not solve yet.</h2>
<p>While Foresight makes significant strides toward scalable mapless navigation, several challenges remain:</p>
<ul class="notes">
<li><strong>Credit assignment in multi-step refinement.</strong> Outcome-level supervision makes it hard to
attribute gains to better critiques, critique-conditioned planning, or critique following. Process-level
supervision could provide denser signal and reduce spurious correlations.</li>
<li><strong>Memory and multi-view understanding.</strong> Navigation plans can degrade when relevant cues are
sparse or absent from the local observation history. Retrieval-augmented memory and co-training on multi-view
reasoning could improve cue grounding in these scenarios.</li>
<li><strong>Static-environment focus.</strong> Our experiments primarily target static settings. Dynamic
environments demand faster policies that reason over temporal relationships, which we leave for future
work.</li>
</ul>
</div>
</section>
<!-- ===================== CITATION ===================== -->
<section id="citation">
<div class="wrap">
<p class="section-kicker">Citation</p>
<h2>If Foresight is useful in your work, please cite us.</h2>
<div class="bib">
<button class="copybtn" data-target="bibtex" type="button">Copy</button>
<pre id="bibtex">@article{zhang2026foresight,
title = {Foresight: Iterative Reasoning About Clues that Matter for Navigation},
author = {Zhang, Arthur and Qi, Carl and Su, Donne and Meng, Xiangyun and Zhang, Amy and Biswas, Joydeep},
journal = {arXiv preprint arXiv:2606.12550},
year = {2026},
}</pre>
</div>
</div>
</section>
<footer>
<div class="wrap">
© 2026 the authors, <a href="https://amrl.cs.utexas.edu/">Autonomous Mobile Robotics Laboratory</a> at The
University of Texas at Austin. Page content is released under
<a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>.
</div>
</footer>
<script src="./static/js/main.js"></script>
</body>
</html>