Offline RL Русский

Chapter 3: Off-Policy Evaluation (OPE)

"You cannot run the new policy in production to see how good it is — so you must estimate its value from the data you already have."

The Problem

In Chapter 2 we defined the offline RL setting: a fixed dataset $\mathcal{D}$ of transitions $(s, a, r, s')$ collected by a behavior policy $\pi_\beta$. We train a new policy $\pi$ (e.g. via CQL, IQL, or any other method in this book) on $\mathcal{D}$. The critical question before deployment:

What is $J(\pi) = \mathbb{E}_{\tau \sim \pi}\bigl[\sum_t \gamma^t r_t\bigr]$ — the true expected return of $\pi$?

We cannot answer this by running $\pi$ in the real environment. That would be deployment, not evaluation. In industrial settings, deploying an untested policy can be unsafe or costly. We need to estimate $J(\pi)$ using only the offline dataset $\mathcal{D}$ — which was generated by $\pi_\beta$, not by $\pi$. This is Off-Policy Evaluation (OPE).

OPE is the sibling of offline RL: offline RL learns a policy from $\mathcal{D}$; OPE evaluates a candidate policy from the same $\mathcal{D}$. In practice you use both: train several policies (e.g. CQL with different $\alpha$), use OPE to rank them or to select the best one, and only then consider deploying the top candidate (possibly after a final safety check).

Why Not Just Use the Learned Q-Function?

Chapter 2 already introduced $\hat{J}(\hat\pi) = \mathbb{E}\_{s,a \sim d^{\hat\pi}}[Q\_\theta(s,a)]$ — the return predicted by the learned Q-function under the greedy policy. We argued that $\hat{J}$ can be wildly optimistic when $Q_\theta$ overestimates OOD actions. So using the training Q-network as an evaluator is biased and unreliable for the very policies we care about (policies that may choose OOD actions).

OPE provides dedicated estimators of $J(\pi)$ that do not assume the Q-function is correct. They use the dataset directly, with explicit treatment of the distribution shift between $\pi_\beta$ and $\pi$.

Three Families of Estimators

1. Importance Sampling (IS)

The return of a trajectory $\tau = (s_0, a_0, r_0, s_1, a_1, \ldots)$ under $\pi$ is $G(\tau) = \sum_{t\ge 0} \gamma^t r_t$. We have trajectories drawn from $\pi_\beta$. The identity:

$$\mathbb{E}_{\tau \sim \pi}[G(\tau)] = \mathbb{E}_{\tau \sim \pi_\beta}\left[ \frac{\rho(\tau)}{\rho_\beta(\tau)} \, G(\tau) \right]$$

where $\rho(\tau)$ is the probability of $\tau$ under $\pi$ and $\rho_\beta(\tau)$ under $\pi_\beta$. The ratio $\prod_{t} \frac{\pi(a_t|s_t)}{\pi_\beta(a_t|s_t)}$ is the trajectory importance weight. So we can estimate $J(\pi)$ by averaging weighted returns over trajectories in $\mathcal{D}$:

$$\hat{J}_{IS} = \frac{1}{n}\sum_{i=1}^n \left( \prod_{t=0}^{T_i-1} \frac{\pi(a_t^{(i)}|s_t^{(i)})}{\pi_\beta(a_t^{(i)}|s_t^{(i)})} \right) G^{(i)}$$

Pros: Unbiased if $\pi_\beta$ has full support (every action with positive probability under $\pi$ is also taken with positive probability under $\pi_\beta$).
Cons: Variance explodes when $\pi$ and $\pi_\beta$ differ a lot — the weight product can be huge. In practice, per-step or per-decision IS and weight clipping are used to reduce variance at the cost of bias.

2. Direct Method (DM)

Learn a model of the reward and dynamics (or just the Q-function) from $\mathcal{D}$, then estimate $J(\pi)$ by simulating $\pi$ in the model or by averaging the learned $Q(s,a)$ under $\pi$. Because $J(\pi)$ is the expected return from the initial state distribution, the correct DM estimator is the average over initial states $s_0^{(i)}$ (one per trajectory in $\mathcal{D}$, or from a held-out set):

$$\hat{J}_{DM} = \frac{1}{n}\sum_{i=1}^n \hat{Q}(s_0^{(i)}, \pi(s_0^{(i)}))$$

Averaging $\hat{Q}(s, \pi(s))$ over all states $s$ in $\mathcal{D}$ would give the average value over the visited state distribution, not the expected episode return; it is sometimes used as a cheap proxy when initial states are not explicitly tracked, but it is not equal to $J(\pi)$ unless the state distribution in $\mathcal{D}$ matches the initial distribution. Fitted Q Evaluation (FQE) is the standard way to obtain $\hat{Q}$ for a given policy $\pi$; we detail it below.

Pros: Low variance; uses the whole dataset.
Cons: Biased when the model or $\hat{Q}$ is wrong — and in offline RL it often is (extrapolation error). DM/FQE can be overly optimistic if $\hat{Q}$ overestimates for $\pi$'s actions.

Fitted Q Evaluation (FQE) in detail

FQE (Le et al., 2019, Batch Policy Learning under Constraints; formalized for OPE there) fits a Q-function that approximates the value of the target policy $\pi$ using only off-policy data from $\mathcal{D}$. Unlike Fitted Q Iteration (FQI), which alternates between policy improvement and evaluation, FQE evaluates a fixed policy: the Bellman target uses $\pi$ at the next state, not $\max_a$ or the behavior policy.

Bellman target for $\pi$: The true $Q^\pi$ satisfies $Q^\pi(s,a) = r + \gamma \mathbb{E}_{s'}\bigl[ Q^\pi(s', \pi(s')) \bigr]$. So we regress $\hat{Q}$ to the target $y = r + \gamma \hat{Q}(s', \pi(s'))$ on transitions $(s, a, r, s') \in \mathcal{D}$. In practice $\hat{Q}$ on the right-hand side is the current iterate (or a target network), and we minimize TD error over the dataset:

$$\mathcal{L}_{FQE} = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[ \left( r + \gamma \hat{Q}_{tgt}(s', \pi(s')) - \hat{Q}(s, a) \right)^2 \right]$$

We can do a single regression (one backward pass over $\mathcal{D}$ with a fixed target $\hat{Q}\_{tgt}$) or iterate: update $\hat{Q}\_{tgt} \leftarrow \hat{Q}$ every $K$ steps. For evaluation we only need $\hat{Q}(s, \pi(s))$ at states (and optionally at dataset $(s,a)$ for DR), so we often use the same $\hat{Q}$ for both the bootstrap and the regression (one-step TD) or a lagged target to stabilize training.

A minimal training step and the J estimate look like this (policy $\pi$ is a callable, e.g. a neural network):

# FQE update: TD target uses π at next state (not behavior action)
with torch.no_grad():
    a_next = policy_fn(next_states)           # π(s')
    q_next = Q_tgt(next_states, a_next)
    td_target = rewards + gamma * (1 - dones) * q_next
q = Q(states, actions)
loss = F.mse_loss(q, td_target)
# ... backward, optimizer step, then soft update Q_tgt ← Q

# FQE estimate: (1/n) Σ Q(s, π(s)) over *initial* states for J(π); or over dataset states as proxy
def estimate_J_FQE(Q, policy_fn, states):
    with torch.no_grad():
        a = policy_fn(states)
        return Q(states, a).mean().item()

Why FQE is useful: No importance weights — no need to know $\pi_\beta$ or form ratios. Works naturally for continuous actions: we only need to query $\hat{Q}(s', \pi(s'))$ at next states, which is a forward pass. The estimate $\hat{J}_{FQE} = \frac{1}{n}\sum_i \hat{Q}(s_0^{(i)}, \pi(s_0^{(i)}))$ (or average over a held-out set of initial states) is cheap to compute once $\hat{Q}$ is trained, so FQE scales well when comparing many candidate policies: train one FQE model per $\pi$, or reuse a single model if policies are similar.

Bias: $\hat{Q}$ is trained on the same distribution as the data — state-action pairs that appear in $\mathcal{D}$. When we evaluate $\pi$, we plug in $\pi(s)$ and $\pi(s')$, which may be out-of-distribution relative to the actions in $\mathcal{D}$. So the same extrapolation error from Chapter 2 applies: $\hat{Q}$ can be overoptimistic (or under) for OOD actions. FQE is therefore most reliable when $\pi$ is close to $\pi_\beta$ or when the state coverage of $\mathcal{D}$ is rich enough that $\pi(s)$ often falls near observed actions.

📄 Code: fqe.py — FQE on the same toy env as in later chapters: train BC, fit $Q^\pi$ with the FQE TD loss, report $\hat{J}_{FQE}$ and compare with rollout return.

3. Doubly Robust (DR)

Combines IS and DM so that the estimator is unbiased if either (a) the importance weights are correct, or (b) the value model $\hat{Q}$ is correct:

$$\hat{J}_{DR} = \frac{1}{n}\sum_{i=1}^n \left[ \hat{Q}(s_0^{(i)}, \pi(s_0^{(i)})) + \sum_{t\ge 0} \gamma^t \rho_{0:t}^{(i)} \left( r_t^{(i)} + \gamma \hat{Q}(s_{t+1}^{(i)}, \pi(s_{t+1}^{(i)})) - \hat{Q}(s_t^{(i)}, a_t^{(i)}) \right) \right]$$

where $\rho_{0:t}$ is the importance weight up to step $t$. If $\hat{Q} = Q^\pi$, the trailing terms have expectation zero. If $\pi_\beta = \pi$, the importance weights are 1, but DR does not reduce sample-wise to DM: the TD-residual correction terms remain. Under a correct value model those terms have zero expectation; with on-policy data they can telescope toward a Monte Carlo return-style estimate. DR thus interpolates between model-based evaluation and trajectory correction, rather than simply becoming DM. In practice DR often has lower variance than IS and lower bias than DM alone. Caveat: on long horizons the cumulative weight $\rho_{0:t}$ can still explode or vanish as $t$ grows, so classical DR still suffers from high variance on long trajectories; variants such as Marginalized IS (state-marginal importance weights) are used to mitigate this.

Practical Recommendations

Setting Prefer
$\pi$ close to $\pi_\beta$ IS (or weighted IS with clipping)
Large action space / $\pi$ far from $\pi_\beta$ FQE or DM (accept some bias)
Need a single number to compare many policies FQE is fast; rank by $\hat{J}_{FQE}$ then validate top few with DR or IS if feasible
Safety-critical Use multiple estimators; if they disagree strongly, treat as unreliable and do not deploy

Variance reduction: For IS, use per-decision IS, weight capping, or normalized weights. For DR, use a well-tuned $\hat{Q}$ (e.g. trained with TD on $\mathcal{D}$) to reduce the variance of the correction term.

Confidence intervals: Bootstrap or concentration bounds (e.g. Chebyshev) can give intervals around $\hat{J}$; wide intervals indicate high uncertainty and should discourage deployment without further validation.

Limitations

Summary

Estimator Bias Variance When to use
IS Low (if support) High $\pi \approx \pi_\beta$
DM / FQE High (model error) Low Fast ranking, accept bias
DR Low if Q or weights correct Medium Default choice when feasible

Off-Policy Evaluation answers: from offline data only, how good is this policy? It does not replace careful deployment and monitoring, but it allows you to compare and select policies before any real-world rollout. The next chapter starts the algorithmic pipeline: Conservative Q-Learning (CQL) — the first method we use to train a policy from $\mathcal{D}$.

References