Visual Causal-chain Bookmarking
[MSc Dissertation] Retrieval-weighted credit assignment for reinforcement learning from delayed reward. VCBM concentrates learning signal on the causally relevant decision points in a trajectory instead of spreading it uniformly, via two instantiations: statistical causal discovery with Welford value estimation in a tabular MDP (beating a RUDDER return-redistribution baseline — 95.07% correct-decision rate in a mean of 1,532 vs. RUDDER's 3,327 episodes across 20 seeds), and retrieval-weighted advantage blending for an LLM agent trained with LOOP inside the AppWorld benchmark.
Problem to result
Who had the problem, what I built, the hardest part, and what it achieved.
When a reward arrives many steps after the decision that caused it, reinforcement learning gives every action the same credit, so agents learn slowly and unreliably.
VCBM, a method that bookmarks the decision points that actually decide an episode's outcome and concentrates the learning signal on them.
Scaling it to an LLM agent in AppWorld, where discovering causal structure is intractable. I replaced causal discovery with a structural heuristic (state-changing tool calls) and spread credit by retrieving similar moments from episodic memory, blended with LOOP's advantage.
95.07% correct decisions in a mean of 1,532 episodes vs RUDDER's 3,327 (p < 0.001, 20 seeds); AppWorld task success raised from 0.354 to 0.458 over plain LOOP.
Performance Graphs
Visualisations of model performance and results across experiments.
Training on the University of Manchester CSF3 cluster — 2x RTX A800 (80 GB VRAM each)
AppWorld · single seed · VCBM ≈ estimate
ML-LOOP vs VCBM — AppWorld 7B Results
Live Simulation Output
Simulated console execution.
Source
The preprint and code for this project.