When an agent's history hits the context wall, a model compacts it. We do not grade how the summary reads. We measure whether the same agent can still finish the task after the rewind.
Long-horizon agents accumulate actions, observations, failed attempts and partial progress until the working context nears its window. Modern harnesses then ask an LM to compact that history into something shorter. CompAct-Bench turns this into a controlled experiment: start from a trajectory the executor already solves, cut it, let a compactor rewrite the past at 10% of its length, and check whether the same executor can still finish.
Long-horizon agents increasingly rely on context compaction as growing interaction histories approach model context limits. Most existing systems delegate this process to an LM that converts the accumulated history into a shorter working context, yet the compaction capability of current models remains poorly understood. To this end, we introduce CompAct-Bench, a 200-task process-level benchmark spanning information seeking, software engineering, and workspace automation through BrowseComp, SWE-bench, and an internal CompanyBench suite. For each instance, an evaluated LM compacts the history at an intermediate boundary, and a fixed executor resumes from the corresponding environment state under a matched action budget. We evaluate twelve frontier LMs as compactors. Although every retained task is solved in all three pre-compaction screening runs, average post-compaction accuracy falls to 56.7%, and the best model reaches only 68.5%. Failure analysis reveals two pathways: omissions lead to repeated work and budget exhaustion, while state distortion can turn tentative evidence into conclusions and trigger premature termination. By making working-context compaction measurable as an execution-critical model capability, CompAct-Bench provides a foundation for building long-horizon agents that remain reliable across context-window boundaries.
Tasks come from three agent settings with very different context scales, from 8.8K tokens per SWE-bench trajectory to 122.8K on CompanyBench. Each retained task passed all three screening runs (pass3) and has at least eight interaction rounds.
| Domain | Harness | Executor | Evaluation |
|---|---|---|---|
| Search | search_agent | Claude Opus 4.8 | Gemini 3.5 Flash judge |
| Coding | swe_agent | Claude Opus 4.8 | Repository tests |
| Workspace | companybench_agent | GPT-5.5 Codex | Native evaluator |
Each row shows the median and IQR of ractual = ℓ(Mb) / ℓ(Hb) for one compactor, the shaded band is the tolerance [0.097, 0.103], and the right column gives the share of contexts inside it. Output length is controlled, so accuracy differences reflect what is kept, not how much.
Strict Avg@3 post-compaction accuracy at a 10% retention target. Overall scores span 46.0% to 68.5%; the 95% bootstrap intervals of the strongest and weakest compactors do not overlap. Sort any column, filter by boundary, open a row for its stage profile.
Averaged over compactors, Search falls from 52.9% at b = 25% to 34.6% at 75%, while Coding rises from 64.0% to 86.1%. Workspace peaks in the middle. Hover a model to trace it across all three panels.
Later boundaries always hand the compactor a longer history, yet Search gets worse and Coding gets better. A repository keeps edits on disk; search evidence lives only in the conversation and must survive inside Mb.
With no history at all, continuing from the reconstructed boundary state beats starting over by up to +56.3 pts. Mb still adds on top of it at 50% and 75%.
Without Hb, search continuations collapse to 7.4% at the late boundary. Mb buys a steady +26–27 pts: the evidence lives nowhere else.
Failed continuations from all twelve compactors were audited unit by unit against Hb. Omission makes the agent redo what it already did. Distortion makes it believe the wrong state, and stop.
Critical omission raises repeated work by +18.2 pts, progress/state loss by +13.8. The executor reopens solved questions and re-runs finished tools until the T − b actions are gone.
Premature commitment appears in 56.7% of early failures vs. 46.0% of budget-exhausted ones. Constraint/goal distortion lifts unverified commitment by +10.5 pts.
Five trajectory-grounded cases from the paper, set up as an investigation board. The left column is what happened (Hb), the middle is what the compactor wrote down (Mb), the right is what the executor did next. Click the card you think is responsible.
You are Cφ. Pick fragments of the agent's history, choose how to phrase them, and stay inside the 10% capacity. Then the executor resumes from your Mb with a fixed action budget. Shorter phrasings are cheaper, and sometimes they quietly change what the agent believes.
Six questions about how you take notes when you are interrupted. We match your answers against the Search / Coding / Workspace profiles of the twelve compactors in Table 2. Entirely unscientific, numerically honest.
@article{chen2026compactbench,
title = {CompAct-Bench: A Benchmark for Working-Context
Compaction in Long-Horizon Agent Tasks},
author = {Chen, Jiayi and Chen, Lizhe and Chen, Yujia and
Dou, Shihan and Zhang, Ming and Ge, Chendi and
Ni, Shiguang and Gui, Tao and Zhang, Qi and
Huang, Xuanjing and Pan, Maxm},
journal = {arXiv preprint},
year = {2026}
}