Latest / Embodied AI 101 / R2SGen: Can Constraint Solving Improve Real-to-Sim Driving Scenarios?
Transcript
- R2SGen: Can Constraint Solving Improve Real-to-Sim Driving Scenarios?. Real-to-Sim Generation: Synthesizing Scenario Programs from Real-World Data via Constraint Solving • Peishan Huang, Wenmeng Zhang, Yusen Chen, Zhenbang Chen • National University of Defense Technology • PACMPL / OOPSLA • 2026. Can we turn a real traffic snapshot into a reusable simulation program without asking a language model to invent its geometry?. R2SGen makes a credible case for doing that with structured search and constraint solving. My take is that this is a useful contribution to scenario authoring, not evidence that the sim-to-real problem has been solved. The paper does not evaluate end-to-end policy transfer. Its value is better understood as making a particular part of a simulation pipeline more explicit, inspectable, and controllable. That distinction matters. A scenario can faithfully encode supplied measurements while leaving unanswered whether its variations stay on the road, whether its rendered images resemble camera data, or whether training on those images improves a deployed system. For a working roboticist, the interesting question is therefore narrower: where should a verified program synthesizer sit between recorded data and a simulator, and what guarantees survive that journey?. What the Paper Did. R2SGen performs training-free synthesis from typed two-dimensional positions and headings to static Scenic programs. Declaration-order search uses cached dReal queries. Relational, relative-numeric, and absolute descriptions receive penalties of zero, one, and two. Prefix pruning retains each declared subset’s cheapest partial program; depth-first search also uses branch-and-bound. Let’s unpack that compact description with a running example of our own. Imagine a delivery van, a taxi, and a motorcycle observed near the ego vehicle. We already have their measured locations and orientations. The task is not to recognize them in pixels. It is to find an economical relational explanation of their arrangement. One explanation could list every coordinate independently. Another could place the van relative to ego, the taxi behind the van, and the motorcycle beside the taxi. Both might describe the observation. But they offer different handles for editing and resampling. The second explanation lets an engineer ask, “What happens if this gap increases?” without manually recomputing every downstream coordinate. That is the appeal of a program: it exposes dependencies that a flat coordinate list leaves implicit. Scenic already provides the underlying idea of distributions over scenes, together with explicit constraints. Fremont and colleagues’ original Scenic work also investigated detector training and targeted corner-case testing. R2SGen should therefore not be credited with discovering that executable scenario descriptions can help machine learning. The interesting problem is automating the construction of those descriptions. Now consider the scoring rule. Under this rubric, an expression such as “behind another vehicle” is preferred to a numeric displacement, which is preferred to an absolute placement. This is an intentional inductive bias: spend fewer points when the program uses the vocabulary the designer considers meaningful. But “cost” here is not compute cost, annotation cost, or collision risk. It is a preference over representations. That distinction creates a useful thought experiment. Suppose the taxi’s most convenient geometric reference is the motorcycle, even though the motorcycle has little relevance to the driving event we care about. A low-scoring program could still be an awkward explanation for a human investigator. Minimizing a grammar-based objective does not automatically recover the event’s important relationships. The genuinely interesting computational issue is declaration order. If the taxi is described relative to the van, the van must already be available as a reference. Choosing what to declare next changes which inexpensive descriptions become possible. Searching every possible order quickly becomes wasteful. Here is why the prefix abstraction helps. Suppose one partial program introduces the van and then the taxi. Another introduces the taxi and then the van. Both have now made the same two objects available. If future statement costs depend only on that available set, the more expensive prefix has no advantage for completing the program. We can discard it without needing to explore all its descendants. Viewed this way, the central optimization is not mysterious: identify what information about the past can actually affect the future. Declaration history is larger than that necessary state. The set of already-declared entities can be sufficient for the current objective. Branch-and-bound supplies a complementary argument. Once we have a complete candidate costing, say, six points, a partial candidate that has already accumulated six cannot improve on it if every additional penalty is nonnegative. We can stop exploring that branch. These two ideas attack different sources of wasted work. Prefix abstraction removes redundant histories. Branch-and-bound removes histories that cannot beat an incumbent solution. Neither makes the problem automatically cheap at arbitrary scale. Our own counting argument makes that clear: a set of 24 entities has more than sixteen million subsets. Replacing permutations with subsets is an enormous improvement, but an exponential representation can still be expensive. There is also important prior context. Edward Kim and colleagues’ Querying Labelled Data with Scenario Programs for Sim-to-Real Validation already used decomposed SMT reasoning to determine whether recorded scenes matched a supplied Scenic program. That work incrementally checked program components rather than relying exclusively on one large constraint formula. R2SGen reverses the practical direction: instead of starting with a program and looking for matching observations, it starts with an observation and searches for a program. My assessment is that the novelty is strongest in this synthesis formulation and its search-state reduction—not in the general idea of applying SMT to scene geometry. The headline evaluation covers 20 nuScenes scenes with 5–24 entities. R2SGen achieves 100% consistency and average program cost 5.7; stronger ScenicNL configurations also reach 100% consistency, averaging 58.0. So the comparison is not simply “the solver is correct and every language model is wrong.” The more interesting result is the difference between reproducing supplied geometry and finding a relational representation under the chosen objective. What a Consistency Guarantee Actually Buys. Here, consistency means that the supplied scene belongs to the program’s represented scene space. That is a useful property, but let’s distinguish two very different questions. First: can this program represent the observation?. Second: are the other scenes it generates useful and realistic?. The first does not settle the second. Imagine observing two vehicles beside one another on a straight road. A relational program might preserve their side-by-side arrangement while changing their separation. Unless road containment is separately enforced, some variations could put a vehicle beyond the road boundary. The relation remains true; the scenario becomes unsuitable. This is not an allegation that a particular R2SGen output does that. It is a counterexample to the broader inference that grounding one observation guarantees the realism of every variation. A simulator or Scenic requirement might reject unsuitable samples. But then rejection rate becomes part of the deployment story. If most proposed variations are rejected, the usable distribution and generation throughput may differ substantially from what the initial program suggests. There is a second issue: equally acceptable programs can induce different correlations. Consider three vehicles in a queue. In one program, every vehicle’s position varies relative to ego. In another, each vehicle varies relative to the vehicle immediately ahead. In the chain representation, moving the first vehicle moves the references for vehicles farther back. In the ego-anchored representation, those movements need not propagate. Both programs can contain the same recorded snapshot. They need not generate the same family of traffic configurations. This is why I would not equate a minimum-cost program with an identified traffic distribution. The optimization can select a compact representation without establishing that its dependency structure captures the right variation. The reported positional and heading tolerances are 10 and 5 degrees; distance and heading interval widths are 5 meters and 5 degrees. Those choices illustrate an important distinction between constructing variation and learning variation. If I choose a neighborhood around an observed state, I have designed an augmentation policy. I have not necessarily estimated how traffic varies in that neighborhood. For perception debugging, a designed neighborhood may be exactly what we want. For estimating real-world event frequencies, it is not enough. The numerical guarantee also deserves careful wording. The paper’s proofs assume an ideal SMT oracle. The official dReal documentation distinguishes an exact unsatisfiability result from a delta-satisfiable result, which concerns a numerically perturbed formula. That is a principled solver contract, but it is not identical to unrestricted exact satisfiability. My practical response would be to separate three tolerances: uncertainty in the input measurements, slack deliberately allowed by the scene relation, and numerical tolerance introduced by the solver. These serve different purposes. Combining them into one informal notion of “close enough” would make failures difficult to diagnose. Finally, there is a language-interface question. Scenic’s placement specifiers account for object dimensions. A relation between measured object centers therefore needs to be translated carefully into a relation involving actual simulated objects. For two aligned, four-meter-long vehicles, a ten-meter center distance is not a ten-meter bumper gap. Both quantities are perfectly reasonable, but confusing them changes the scene. I would want conformance tests at that boundary: feed a known arrangement through the synthesizer, instantiate the resulting program, and check the intended geometric contract. I am not claiming the implementation mishandles this. I am identifying a validation step that compilation alone cannot replace. Are the Language-Model Comparisons Fair?. The LLM results allow five attempts and provide the same label information. The tested backbones include GPT-4o, GPT-5.4, and Gemini-3.1-Pro. ChatScene uses a few-shot adaptation; ScenicNL uses chat-compatible prompting with compiler feedback. That protocol answers a reasonable question: given the necessary spatial facts, which system produces the desired program?. But it is not the native task of every baseline. ScenicNL was developed to turn natural-language descriptions, including autonomous-vehicle crash reports, into probabilistic scenario programs. Its motivation includes uncertainty in those reports and the difficulty of generating a relatively low-resource language. Its approach combines multiple prompting strategies with compiler and simulator components. That is different from receiving a complete inventory of object poses and minimizing a predefined structural penalty. If a crash report says that a vehicle turned across a cyclist’s path, the difficult work includes interpreting the event, choosing relevant agents, and deciding how to represent missing information. Exact-coordinate reconstruction deliberately removes much of that ambiguity. ChatScene has another emphasis. Its original work decomposes traffic descriptions into components such as locations and behaviors, retrieves corresponding Scenic knowledge, and executes scenarios in CARLA. It also evaluates generated scenarios through adversarial training of driving agents. Its central objective is safety-critical scenario generation, not reconstructing a supplied snapshot with the lowest relational score. Those differences do not invalidate the R2SGen comparison. They limit its interpretation. I would accept the claim that a specialized symbolic backend is better suited to a precisely specified reconstruction-and-abstraction task. I would not generalize that into a claim that the backend replaces language-based scenario interfaces or adversarial scenario generators. The best-of-five protocol also matters operationally. “At least one acceptable result” is not equivalent to “the first result is dependable.” An interactive authoring tool can sometimes tolerate retries. A large unattended generation job needs a reliable way to identify the successful attempt. For practitioners, that suggests reporting success, verification effort, and latency together. A generator that eventually succeeds but requires expert selection has a different deployment cost from one whose output can be accepted automatically. There is another baseline I would want before interpreting the program-cost difference too broadly: a deterministic coordinate converter. Under the stated penalty rule, simply expressing absolute positions and headings in the ego frame can reduce each relevant penalty from two to one. That conversion need not discover a meaningful traffic pattern. Yet it improves the score. This does not explain away the full reported difference. It shows why the metric needs comparators that are deliberately designed to exploit its structure. I would add a greedy relational baseline: precompute feasible relationships, repeatedly choose a cheap declaration, and stop without proving global optimality. Then compare both methods on useful sample yield, human editing effort, and downstream performance. That experiment could reveal two very different outcomes. Perhaps exact optimization finds substantially better scenario templates. Or perhaps it mostly improves a surrogate score after a simple heuristic has already produced everything an engineer needs. Both outcomes would be useful. The present cost comparison does not distinguish them. The Search Ablation Is More Convincing Than the Broad Realism Claim. For the 12-entity task, adding prefix abstraction to A* reduces runtime from 336.55 to 5.83 seconds. Only BFS with prefix abstraction completes the 24-entity task: 2,515.8 seconds and 6,129.6 megabytes. Timings are five-run averages without error bars. These numbers support a concrete engineering conclusion. The first difference is roughly a 58-fold speedup. Without a reported variance estimate, I would avoid calling it statistically significant in a formal sense. But it is not the sort of marginal timing difference that immediately suggests ordinary run-to-run noise. It also isolates an informative design choice: improve the representation of repeated states, not merely the order in which those states are visited. That lesson travels well beyond scenario generation. If two search histories expose the same future decisions and differ only in accumulated cost, retaining both may be the wrong computational abstraction. The largest result, however, translates to approximately 42 minutes. That can be acceptable for offline template construction. It would be a very different proposition if someone presented it as interactive scene generation or an online autonomy component. The relevant throughput question is how often synthesis must run. If one expensive synthesis produces a template that generates many valuable tests, the cost may amortize well. If every recorded frame requires a fresh optimization, it may dominate the pipeline. I would also ask for time to the first valid program, time to a useful incumbent, and time to certify optimality. These are not interchangeable milestones. An engineer may prefer a good template after seconds to a slightly better one after much longer. There is a reporting detail to resolve before reproducing the memory boundary: the setup specifies 16-gigabyte containers, while the table labels out-of-memory above 8 gigabytes. That could reflect an intentional algorithm-level cutoff inside a larger container. It need not be a contradiction. But the stopping rule and accounting method should be explicit. Finally, I would want a geometric-oracle ablation. For fixed poses, many candidate relations can be checked with direct coordinate transforms, distances, and angular differences. Interval endpoints can also be constructed under simple width rules. That suggests an alternative implementation using ordinary geometric computations while leaving the discrete search unchanged. This is my proposed comparison, not an additional result from the paper. It would separate the value of the search abstraction from the value of using SMT for this particular collection of geometric checks. SMT may become more compelling when constraints become richer. That is a good reason to preserve the interface—not a reason to skip the simpler comparator. Does It Generalize Beyond Trusted Scene Labels?. The paper’s 50-image frontend experiment reports recall of 0.449 for its best-performing, domain-aligned PGD detector. That number changes how I would position an image-to-program deployment. A structured scene extracted from an image is an estimate, not a specification handed down by nature. If an upstream detector misses a vehicle, a downstream solver can produce a perfectly consistent explanation of an incomplete world. Imagine a parked van partly hiding a motorcycle. The detector reports only the van. The synthesizer finds an elegant program describing the van’s relation to ego. Every downstream consistency check passes. Yet the object most relevant to the failure investigation is absent. No amount of exact reasoning about the supplied inventory can recover an entity that never entered the problem. This is why formal guarantees should identify the data assumptions on which they depend. Pose uncertainty creates a related issue. Suppose a vehicle sits near the boundary between two qualitative relations. A small heading error could change which reference yields the preferred description. The resulting program might change discretely even though the underlying measurement changed only slightly. I would therefore test structural stability: perturb the input within a plausible uncertainty region and observe whether the selected dependency graph remains useful. A backend can be numerically consistent and still be operationally brittle if tiny perception changes produce very different templates. There are also several meanings of “generalization” that should not be collapsed. One is geometric reuse: move or rotate an arrangement while preserving selected relationships. Another is distributional generalization: produce configurations representative of situations not used to construct the generator. A third is task transfer: improve a perception or control system on held-out real-world data. My reading is that these should be separate acceptance tests. Passing the first does not provide a shortcut to the other two. TrafficGen offers useful context for the second question. Its autoregressive encoder-decoder places vehicles using traffic context and map information, then generates trajectories through a motion-prediction component and repeated rollout. That approach is aimed at learning traffic distributions rather than extracting a compact explanation of one observation. I would not rank those approaches on a single leaderboard without first specifying the job. For reproducing and editing a particular observed arrangement, a symbolic program can be attractive. For sampling a population of traffic situations, a learned generator addresses a different requirement. For closed-loop testing, agent behavior becomes another essential component. A hybrid is plausible: a learned model proposes variation, and explicit constraints screen or condition those proposals. But that is a proposed architecture, not evidence that all these capabilities arrive automatically with R2SGen. Reproducibility: An Algorithm Description Is Not Yet a Reproduction. The authors’ publication page links an artifact. As of this review on October 8, 2026, I could not retrieve the linked Zenodo contents. I therefore cannot verify its implementation, dependency versions, license, or reproduction scripts. That is a limitation of this review’s access—not a claim that no artifact exists. My assessment is that a competent team could implement the central search idea from the description. Reproducing the exact measurements would require a more complete execution contract. I would want the precise scene identifiers and preprocessing, including how object coordinates are transformed into the synthesis frame. I would also want an explicit definition of which entities are included and how the ego reference is represented. For the numerical layer, I would pin solver versions and precision, then build small tests around angular wraparound, coincident positions, boundary cases, and nearly satisfied relations. For the generated programs, I would pin the Scenic environment and any simulator model definitions. A change in object dimensions or defaults can matter even when the surface syntax remains unchanged. For the language-model comparisons, I would preserve prompts, model identifiers, generation settings, retry histories, and the rules used to accept a result. Otherwise a reproduction risks becoming a new experiment with superficially similar inputs. I would also preserve intermediate outputs. A final success flag tells us much less than the chosen declaration order, reference relationships, solver decisions, and emitted program. Those traces would make disagreements between implementations far easier to resolve. This is where the program-synthesis framing has practical promise: the intermediate representation is something an engineer can inspect, not just a generated image whose failure must be inferred after rendering. What This Means for Practitioners. My recommendation would be to evaluate R2SGen as an offline scenario-template compiler. Start with a recorded situation that matters: a perception miss, an unusual arrangement, or a regression case. Establish a trustworthy object inventory and coordinate frame. Then ask the synthesizer for an editable relational description. Do not immediately turn the output into a large training set. First, inspect whether the chosen references preserve the phenomenon you actually care about. If the target issue is an occlusion, a low-cost arrangement that allows the occluder to drift away may be less useful than a higher-cost program that explicitly preserves visibility conditions. Next, compose the template with independent constraints for the intended simulator. Depending on the application, those might include road containment, valid object footprints, visibility, or a carefully chosen range of overlaps. For safety testing, the objective is not necessarily to exclude collisions; it is to distinguish meaningful hazardous events from malformed initialization. Then sample and measure. I would track how many samples compile, how many instantiate, how many satisfy the external validity checks, and how many preserve the target phenomenon. Those are different failure stages. Collapsing them into a single “generation success” rate would hide where engineering effort is needed. Only after that would I run the downstream learning experiment. A useful comparison would hold the rendering budget, assets, training schedule, and evaluation set fixed. One condition would replay recorded geometry. Another would use simple coordinate perturbations. A third would use relational templates. A fourth, if justified, would add more sophisticated learned variation. The question would be whether relational synthesis contributes something beyond having more images. For a perception task, I would look at the targeted failure slice as well as aggregate metrics. An augmentation method can help one rare configuration while degrading others. For a control task, I would also require closed-loop evaluation rather than treating plausible initial placement as a substitute for plausible interaction. Three follow-up studies would most change my assessment. The first is a direct comparison against cheap geometry-based synthesis. Keep the output language and scoring rule identical, but compare exact search with greedy selection and an arithmetic oracle. That would reveal what the expensive machinery buys. The second is an uncertainty study. Perturb labels, remove entities, and vary confidence. Measure not only whether synthesis succeeds, but whether the resulting template remains stable and useful. This would connect the clean specification problem to realistic perception pipelines. The third is a downstream transfer study with independent evaluation. Test whether synthesized variations improve performance on real cases beyond those used to construct the templates. That is the experiment needed to turn a scenario-authoring result into a claim about the sim-to-real gap. There is also a promising optimization opportunity in the distinction between obtaining a good program and proving it optimal. I would explore stopping rules based on a practical budget, retaining useful incumbents, and allowing a human or learned proposal mechanism to suggest promising orders. Such changes should be evaluated against their effect on template quality, not only the final cost. For applications outside driving, I would transfer the architectural idea before transferring the actual relation vocabulary. A warehouse scene, a tabletop arrangement, and an articulated manipulation task require different geometric and physical contracts. A successful abstraction for planar traffic placement does not automatically encode support, contact, or reachability. The strongest takeaway is therefore quite specific. R2SGen points toward a useful division of labor: search decides how to describe a scene; geometric reasoning checks candidate descriptions; a program exposes the resulting structure for editing and sampling. Its search-state abstraction is worth studying independently of the broader simulation claims. But a guarantee is only as broad as its specification. Realistic sensing, meaningful variation, plausible behavior, and improved robotic performance each require their own evidence. For now, I would adopt the idea as a way to make scenario construction more auditable—and insist on measuring everything that happens after the program is written.