Visual Causal-chain Bookmarking

[MSc Dissertation] Retrieval-weighted credit assignment for reinforcement learning from delayed reward. VCBM concentrates learning signal on the causally relevant decision points in a trajectory instead of spreading it uniformly, via two instantiations: statistical causal discovery with Welford value estimation in a tabular MDP (beating a RUDDER return-redistribution baseline — 95.07% correct-decision rate in a mean of 1,532 vs. RUDDER's 3,327 episodes across 20 seeds), and retrieval-weighted advantage blending for an LLM agent trained with LOOP inside the AppWorld benchmark.

RLDissertation
# overview

Problem to result

Who had the problem, what I built, the hardest part, and what it achieved.

01 · The problem

When a reward arrives many steps after the decision that caused it, reinforcement learning gives every action the same credit, so agents learn slowly and unreliably.

02 · What I built

VCBM, a method that bookmarks the decision points that actually decide an episode's outcome and concentrates the learning signal on them.

03 · The hardest part, and how I solved it

Scaling it to an LLM agent in AppWorld, where discovering causal structure is intractable. I replaced causal discovery with a structural heuristic (state-changing tool calls) and spread credit by retrieving similar moments from episodic memory, blended with LOOP's advantage.

04 · The results

95.07% correct decisions in a mean of 1,532 episodes vs RUDDER's 3,327 (p < 0.001, 20 seeds); AppWorld task success raised from 0.354 to 0.458 over plain LOOP.

# graphs

Performance Graphs

Visualisations of model performance and results across experiments.

Training on the University of Manchester CSF3 cluster — 2x RTX A800 (80 GB VRAM each)

AppWorld · single seed · VCBM ≈ estimate

Return per 1k output tokensbetter ↖︎00.10.20.30.40.50k0.5k1k1.5k2kAvg output tokens per episode (thousands)ML-LOOPVCBM

ML-LOOP vs VCBM — AppWorld 7B Results

# simulation

Live Simulation Output

Simulated console execution.

Outputs
Training Run — Iteration 194–200
$_▋