Deep reinforcement learning / robustness

Domain-shift quantification.

Can we measure whether a policy will suit an upcoming environment before a failure at deployment? That is the question behind this dissertation project.

Interactive experiment map

Measure behaviour under controlled shifts.

Vary dynamics and observations separately, collect the policy’s behaviour, then use the meta-model to estimate suitability.

Case study

Predicting whether a policy will transfer

01 / Problem

Familiar simulations can hide failures

A policy can look reliable in a familiar simulation, then lose performance abruptly after a small change in dynamics or observation.

02 / System

Learn from controlled shifts

Generate environments with controlled shifts and collect policy behaviour and returns. A second-order model then links the shift signals to policy suitability.

03 / Artifact

Experiment code and dissertation

The public repository brings together the DomainShift experiment code, tests, paper material, and associated dissertation.

Experiment detail

Four stages of the experiment

  1. Train a source policy

    Establish how the agent behaves in the environment used for learning.

  2. Parameterize shifts

    Alter dynamics and observations separately so it is clear which part of the environment changed.

  3. Collect rollouts

    For each controlled change, measure trajectories, state visitation, and returns.

  4. Estimate suitability

    Use a meta-model to predict whether the policy is compatible with the upcoming domain.

Evidence boundary

A dissertation research framework.

The project sets out the case for intrinsic shift quantification and implements it as a research framework. This is dissertation research; it does not give a universal guarantee of out-of-domain policy safety.