Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

Why Hybrid Offline-Online RL?

Vision-language-action (VLA) policies can be trained offline from fixed demonstration datasets or online through environment interaction. In practice, online RL often produces stronger final policies and better out-of-distribution (OOD) behavior, but at substantially higher cost. Offline SFT remains cheaper and easier to deploy, yet struggles under distribution shift.

Recent work on RL4VLA showed that PPO fine-tuning of OpenVLA LoRA adapters can improve OOD performance relative to SFT, but requires much more expensive online training. This raises a practical question: in parameter-efficient VLA adaptation, can offline supervision be used not only for initialization, but as a structured signal inside the RL objective?

We study two guided PPO variants and one natural baseline:

• RefKL — PPO regularized toward a frozen SFT reference policy via a KL penalty.
• DataBC — PPO augmented with behavior cloning on the offline demonstration dataset.
• PPO SFT-init — standard PPO initialized from the SFT LoRA without explicit guidance during RL.

Our central finding is that offline guidance acts as an effective optimization prior: guided methods improve over standard PPO at the same online budget, and RefKL reaches nearly the same operating point as a much longer PPO run while using far fewer environment interactions, without sacrificing OOD capability.

Method

All methods operate on the same pretrained OpenVLA-warmup backbone. During training the base VLA is frozen and only a LoRA adapter is optimized. Methods differ solely in their objectives and the data used for optimization.

Problem Formulation

We model control as a partially observable Markov decision process (POMDP). At step t, the agent receives an observation o_t = (I_t, l), where I_t is the current RGB image and l is a natural-language instruction fixed within an episode. The latent state x_t ∈ X is not observed directly.

The policy outputs tokenized actions u_t ∈ U, which the environment wrapper maps to executed robot commands a_t ∈ A. Episodes have fixed horizon T = 80. A trajectory is τ = (o₀, u₀, a₀, r₀, …, o_T). The reward is a sparse difference-based signal: the agent is rewarded for increases in task progress, not for remaining in an already achieved state.

Notation

Policy Parameterization

Let φ denote the frozen OpenVLA-warmup backbone. The trainable policy is π_θ(a|o) = f(φ, Δ_θ; o), where Δ_θ is a low-rank LoRA module (rank 32) attached to the policy network. For RL, we use the RL4VLA value-head modification so SFT and RL share the same backbone and action interface.

Supervised Fine-Tuning (SFT)

We first train a policy on offline demonstrations by maximizing the likelihood of expert actions. The resulting LoRA checkpoint is used either as a frozen reference π_ref (RefKL) or as initialization for PPO SFT-init.

Proximal Policy Optimization (PPO)

PPO is our baseline online RL algorithm. At each iteration we collect on-policy rollouts, estimate advantages with the value head, and optimize the clipped surrogate objective.

Here Â_t is the advantage estimate and ε is the PPO clipping parameter. Standard PPO uses zero-initialized LoRA; PPO SFT-init starts from θ_SFT instead.

SFT-Guided PPO

For hybrid offline-online training we supplement the PPO objective with an auxiliary loss. We evaluate two variants: regularization toward a frozen SFT reference policy (RefKL) and behavior cloning on the offline dataset (DataBC). In both cases, π_ref is the SFT model trained on the same demonstrations as D_off.

RefKL: Reference-Policy Guidance

RefKL penalizes KL divergence between the current policy and the frozen reference on the RL minibatch B. This preserves SFT action preferences while still allowing on-policy adaptation.

DataBC: Offline Dataset Guidance

DataBC augments PPO with a behavior-cloning loss on offline demonstrations sampled from D_off. This directly injects expert transitions into RL optimization.

Regularization Weight Scheduling

The auxiliary losses are weighted by a time-dependent coefficient β(t). Rather than keeping β fixed, we use a curriculum that gradually removes offline guidance:

In our experiments, t₁ = 100,000, t₂ = 300,000, and β₀ = 0.6. The auxiliary term is held constant for the first 100k steps, linearly annealed over the next 200k steps, and removed entirely thereafter. The same schedule is used for both RefKL and DataBC.

Ablation experiments show that stronger initial guidance improves both IND and OOD success, and that the curriculum outperforms a constant-β schedule.

Results

At an equal budget of 1M environment steps, both guided variants outperform plain PPO. RefKL reaches IND 0.93 and average OOD 0.77, matching the final 2M-step PPO checkpoint (IND 0.92, OOD 0.77).

Main benchmark results on RL4VLA. RefKL at 1M steps matches 2M-step PPO on average OOD success while using half the online interaction budget.

Comparison at Equal 1M-Step Budget

At the same online training budget, both guided variants outperform plain PPO on IND and all OOD categories. PPO initialized directly from the SFT LoRA starts from a much stronger checkpoint but improves surprisingly slowly and remains below both guided methods by 1M steps. Notably, SFT-init is relatively strong on vision and language OOD but weaker on action OOD, suggesting that fitting the supervised distribution more strongly can make execution-level adaptation under RL alone more difficult.

Beta Ablation (RefKL)

We vary the initial regularization strength β₀ under the curriculum schedule and compare against a constant-β variant. Stronger auxiliary supervision consistently improves both IND and OOD success; the curriculum schedule also works better than keeping β fixed throughout training.

Benchmark & Evaluation

All experiments follow the RL4VLA evaluation protocol. Each checkpoint is evaluated with three evaluation seeds. We report 64 in-distribution episodes and 960 out-of-distribution episodes in total (64 episodes for each of 15 OOD environments). OOD settings are grouped into vision, language, and action shifts; reported success rates are averaged across evaluation and training seeds.

RL4VLA evaluation environments. Top: an in-distribution scene used during training. Bottom: examples of out-of-distribution scenes with vision, language, and action shifts.

@misc{poyarkov2026offline,
  title={Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models},
  author={Poyarkov, Dmitriy and Staroverov, Aleksei and Panov, Aleksandr I.},
  howpublished={ICML 2026 Workshop on Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning},
  year={2026}
}