Clarify relationship between "LongMemEval" results and MemoryAgentBench's reconstructed longmemeval_s subset

[junkuanliu]

Hi, thanks for releasing this benchmark suite — really useful unification of the memory systems.

I'm trying to reproduce the "LongMemEval" numbers in the paper, but the only data path I can find in the code is benchmark/memoryagentbench/hf_datasets.py pulling ai-hyz/MemoryAgentBench — there's no standalone LongMemEval loader, and the context chunking, prompts, and gold answers all come from that MemoryAgentBench split rather than the original LongMemEval release.
Could you clarify whether the reported numbers are entirely from this MemoryAgentBench-reconstructed subset, whether the workload should instead be labeled "MemoryAgentBench (longmemeval_s*)"?

[Beliefuture]

@junkuanliu Hi! Thanks for your interest and question in our work.

Yes. The current released version of our benchmark (v1.0) utilized the MemoryAgentBench (longmemeval_s*) to keep the suite clean and easy-to-use. The differences between the original LongMemEval and the motivation is stated below.

Although task types like information extraction (IE) or multisession reasoning are included, most tasks can be reformulated as single-retrieval problems requiring agents to retrieve the correct segments spanning a long multi-turn conversation. We reformulated chat history into five long dialogues (∼355K tokens) with 300 questions.

However, we are working and will continuously incorporate more datasets and methods (and other features) in our subsequent release of this benchmark suite. Please stay tuned and we welcome any collaborations from the community if possible :)