Preprint

FZJ-2026-04959

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

Krieger, L.FZJ* ; Zhao, X.FZJ* ; Cao, Z. ; Wang, Q.FZJ* ; Scharr, H.FZJ* ; Assent, I.FZJ*

2026

This record in other databases:  

Please use a persistent id in citations: doi:10.34734/FZJ-2026-04959

Abstract: Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.

Note: accepted at RLxF @ ICML 2026

---

Contributing Institute(s):

1. Datenanalyse und Maschinenlernen (IAS-8)

Research Program(s):

1. 5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511) (POF4-511)

Appears in the scientific report 2026

---

Database coverage:

---