Naive RAG Outperforms Complex Agentic Memory on Conversation Benchmarks

The paper studies whether complex agentic memory systems are necessary for long-term conversation understanding. It evaluates Raw-RAG, a naive baseline using dense retrieval and reranking over raw history. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among compared memory systems in six of seven benchmark-model settings, outperforming the strongest compared agentic memory system by 0.13-5.53 points. Evidence analysis attributes its competitiveness to higher coverage of necessary evidence, and the authors argue raw information preservation matters.

Key takeaways

• Raw-RAG ranks first in six of seven benchmark-model settings against compared memory systems.
• It outperforms the strongest compared agentic memory system by 0.13-5.53 points.
• Higher coverage of necessary evidence can offset its lower evidence utilization.
• Even complete access to transformed memory stores leaves an accuracy gap versus raw evidence.

Why it matters

The results challenge assumptions that complex agentic memory operations are required for conversation tasks, suggesting naive RAG should be treated as an essential baseline when evaluating memory systems.

Read with care

Findings are limited to the evaluated conversation tasks and benchmarks; source claims not independently checked.