How to Teach LLM Agents to Remember Context and Reproduce Scientific Papers

Recently came across the PaperGuru benchmark, where the authors decided to dig into an unpleasant problem with autonomous agents. Context windows in modern models have grown to a million tokens. Yet multi-day tasks where an agent needs to explore repositories, read dozens of papers, and write working code still fall apart.

Usually, everything comes down to memory. Vector databases find text chunks by cosine similarity, but completely miss relationships over time. If a paper has been updated or a library has become outdated, a standard RAG will happily mix an outdated fragment into the prompt. In the PaperGuru-Benchmark repository, researchers from the AutoTrustAI team released a long-term memory architecture with data lifecycle awareness (Lifecycle-Aware Memory, or LAM), along with results from testing it on complex benchmarks.

What's Wrong with Standard RAG

When an agent writes a large literature review or reproduces code from a PDF paper, flat embedding search stumbles on basic things.

First, information becomes outdated. If a method was disproven in a more recent paper, a flat database doesn't know that.

Second, the needed evidence often lies not in the chunk that resembles the query by keywords, but two links away in the citation graph.

Third, as the archive grows, search costs increase, and the agent starts drowning in noise.

The authors formulated four rules for working with memory:

1. Content versioning. The system tracks edits, deprecations, and paper retractions.
2. Multi-hop structural relevance. Search goes through the relationship graph, not just vector similarity.
3. Bounded query cost with infinite archive growth.
4. Evidence tracing. Each agent claim is tied to a specific source.

Capital Chunk Memory Architecture

Instead of slicing text into uniform chunks and dumping them into Chroma or Pinecone, the PaperGuru architecture splits memory into two layers. The first layer is called chunk heads. These are compact headers with metadata for each artifact, used for fast routing. The second layer, chunk contents, stores raw text and is loaded lazily only when actually needed.

The router relies on a temporal artifact graph. The graph holds two types of relationships: structural (for example, cites, implements, benchmarked-on) and causal (deprecated-by, retracted-by, superseded-by).

The generation pipeline consists of four steps:

• Search: fast search for matching artifact headers in the archive.
• Extract: pulling out needed fragments and assembling so-called evidence cards.
• Reason: a generation and critique cycle where the model drafts and checks the logic.
• Verify: final validation with source reference checking.

What the Tests Show

The authors tested the system on two tough benchmarks: PaperBench from OpenAI and SurveyBench.

PaperBench evaluates a model's ability to take an ML paper PDF and write a working repository with experiment reproduction. The human baseline (an ML PhD student with a 48-hour budget) is 41%.

PaperGuru showed an average result of 66.05% on 23 papers, beating all published baseline solutions. The previous best result from other agents held at 35.74%.

On 19 out of 20 papers with known baselines, the new memory architecture showed significant improvement. For example, on reproducing the classifier-free guidance paper, the result grew by 68%. The only dip occurred on the PINN task (-4.47%), where the original baseline used manual domain-specific heuristics.

On SurveyBench, which evaluates the quality of writing large scientific reviews, the system scored 94.66% on content quality under a judge based on Claude Opus.

It's worth looking at the Richness metric. It counts not subjective language model ratings, but the actual presence of compiled graphs, tables, working code, and correct citations in the generated material.

Here, PaperGuru scored 43.76%, while half of the competing approaches scored zero, generating bare text with no structure.

What's in the Repository

The repository is about 350 MB and contains lots of practical materials:

• Full benchmark infrastructure with reproducible evaluation pipelines
• Pre-computed embeddings and graph structures for all benchmark papers
• Baseline implementations of LAM architecture components
• Evaluation scripts and visualization tools
• Ready-to-use submissions for all 23 PaperBench papers

All graphs from the README can be rebuilt locally. The assets/figures/ folder contains a data.json file with all metrics and a build script:

python scripts/rebuild_graphs.py

Who Should Study This Project

If you're building agent systems that work with large codebases or complex technical documentation, this repository provides excellent food for thought. The idea of splitting memory into lightweight headers and a causal relationship graph easily transfers to corporate knowledge bases.

Ready submissions in the PaperBench/submissions/ folder will be useful for those testing their own code generation pipelines from papers. There you can see how to structure the reproduction of complex ML pipelines, when the model needs to output not just a single script, but a working project tree with dependencies and tests.