Skip to content

Issue 78: Automatic In-Turn Context Compaction

Objective

Automatically compact an active agent/tool loop before its next model request becomes unsafe, without requiring another user message or using provider-specific compaction APIs.

The implementation uses Keen's existing semantic compaction approach for every supported client:

  • OpenAI-compatible Chat Completions
  • OpenAI Responses
  • OpenAI Codex
  • Anthropic
  • Google AI / Genkit
  • Bedrock

Automatic compaction is an internal checkpoint. The generated summary is never rendered as agent output; the agent continues its same tool loop after the checkpoint.

Confirmed Product Behavior

Area Decision
Proactive trigger Immediately before the next model request, when the estimated input is at least 90% of the effective input budget.
Effective input budget context window - requested output reserve - safety margin.
Trigger boundary Between model/tool-loop iterations only: after all tool results have been added, before the next model request. Never while streaming a model response or in the middle of a tool batch.
Forced recovery If Keen's preflight reducer or the provider reports a context-window error, compact and retry the failed sampling request once.
Retained task Preserve the most recent user message verbatim.
Summary scope Summarize all history, including user/assistant activity and meaningful tool results, while instructing the compactor not to restate the retained user message verbatim.
Replacement context Replace conversational history with one synthetic user message containing the compacted checkpoint and the verbatim last user message. The original agent system prompt remains outside this replacement.
Compaction output Private. Do not emit summary chunks, reasoning, tool events, or a normal assistant message for the compaction request.
REPL progress UI Replace the current spinner text with Compacting..., then restore the ordinary agent loading text when compaction ends.
REPL completion UI Display transient Context compacted automatically. using the existing notification timeout; do not persist/replay that transient notice.
Headless Enable automatic compaction for keen run as well as interactive REPL sessions. Do not emit compaction summary text in headless text or JSON output.
Cancellation Esc during automatic compaction cancels only the compaction child request. The original loop history has not changed, so no input is lost. Ctrl+C continues to cancel the entire parent agent turn.
Proactive cancellation/failure Restore the normal loader and continue the unchanged loop once. Suppress another proactive attempt until history changes.
Forced-recovery cancellation/failure Stop the agent turn with an actionable context-limit error; do not resend the known-oversized request.
Timeout Do not impose a fixed automatic-compaction timeout. The user can cancel an unusually slow compaction with Esc; parent context cancellation and shutdown still propagate normally.

Replacement-message format

The automatic replacement is exactly one synthetic llm.RoleUser message:

<compacted_context>
The prior conversation context was compacted automatically to fit the
model's context window. The following summary preserves relevant goals,
constraints, progress, discoveries, tool results, and pending work.

[summary]
</compacted_context>

<last_user_message>
The following is the most recent user message. Treat it as the current
task and its requirements as authoritative.

[verbatim latest user message]
</last_user_message>

This wrapper is generated by Keen, not by the compaction model. The model generates only [summary].

Current Architecture Constraints

  1. The active agent loop is owned by each provider client, not by the REPL:
  2. internal/llm/openai.go
  3. internal/llm/openai_responses.go
  4. internal/llm/openai_codex.go
  5. internal/llm/anthropic.go
  6. internal/llm/genkit.go

  7. internal/llm/bedrock.go

  8. AppState receives the normal assistant message only after a whole agent turn ends. Starting the existing /compact UI flow while a provider loop is active would compact stale app state and leave the provider's native history unchanged.

  9. Existing manual compaction is already provider-neutral:

  10. request creation: internal/cli/repl/appstate/state.go:180
  11. compaction prompt: internal/llm/systemprompt.go:106

  12. manual replacement: internal/cli/repl/appstate/state.go:269

  13. Existing request-time reduction only removes oldest oversized tool outputs. It is a guardrail, not semantic compaction:

  14. internal/llm/context_reducer.go

  15. Existing HistoricalToolActivity stores compact historical tool metadata, but not raw tool output. Auto compaction needs an in-memory-only rich representation of completed in-flight tool calls/results so the compactor can summarize the work that caused the current loop to grow.

Design

1. Shared auto-compaction primitives in internal/llm

Create a shared automatic-compaction implementation, rather than six provider-specific summarizers.

Add an internal helper (new internal/llm/auto_compaction.go) responsible for:

  1. Defining the 90% threshold policy against an effective budget.
  2. Locating the most recent RoleUser message.
  3. Building the automatic compaction request:
  4. dedicated auto-compaction system prompt;
  5. full non-system canonical history, including the latest user task and in-flight rich tool records;
  6. a final instruction explaining that the latest user message will be retained verbatim outside the summary.
  7. Calling the same LLMClient.StreamChat implementation with:
  8. no tool registry;
  9. OneShot: true;
  10. an explicit DisableAutoCompaction: true option to prevent recursive compaction;
  11. the active session ID for normal request metadata/caching where supported.
  12. Privately collecting only normal text chunks into a summary; do not forward nested events to the parent agent stream.
  13. Validating a non-empty summary and building the synthetic replacement message above.

The helper must keep the original history untouched until collection and validation succeed. It returns either:

  • a valid replacement history ([]Message containing the one synthetic user message), or
  • an error/cancellation without mutating the caller's working history.

2. Stream protocol for lifecycle and cancellation

Extend internal/llm/message.go with dedicated automatic-compaction stream events, for example:

StreamEventTypeAutoCompactionStarted
StreamEventTypeAutoCompactionApplied
StreamEventTypeAutoCompactionCancelled
StreamEventTypeAutoCompactionFailed

Add a structured payload to StreamEvent containing:

  • a cancellation callback on started;
  • replacement []Message on applied;
  • optional compaction request usage for accounting;
  • an error on failed.

Do not send the generated summary as chunk or reasoning_chunk on the parent event stream.

The provider loop creates a child context.WithCancel(ctx) only for the private compaction request. It emits started before collecting the summary. The REPL can call that child cancel function without cancelling the original parent agent context.

3. Canonical in-flight history with raw tool results

Extend the in-memory historical-tool representation in internal/llm/message.go / internal/llm/message_format.go so an auto-compaction snapshot can replay actual current-turn tool results into the compaction request.

Recommended shape:

  • preserve existing persisted fields (Tool, Input, Status, ExitCode);
  • add unexported-from-session/JSON transient fields such as HasRawOutput and RawOutput;
  • update clone helpers to deep-clone transient raw values;
  • make historicalToolResult serialize transient raw output when present, otherwise retain existing compact status/exit-code behavior.

For every completed tool iteration, each provider client should append an in-flight assistant llm.Message to a local canonical history. It contains:

  • final/textual assistant content for that model iteration;
  • tool call inputs;
  • raw tool result or structured tool-error payload;
  • successful/error status.

Use the normal generic provider conversion functions when sending this history to the compaction request. This avoids inflated provider-native JSON transcripts, preserves valid assistant-tool-result sequencing, and keeps the solution provider-neutral.

Transient raw results are only kept for the currently running provider loop. Do not begin persisting them in completed normal assistant history/session records.

4. Effective-budget and context-error policy

Refactor internal/llm/context_reducer.go so all request sizing uses an explicit effective budget based on the output capacity actually requested by that client.

  1. Keep the current safety margin policy: 
    max(4096 tokens, 5% of context window)
  2. Replace the implicit universal 8,192 output reserve with a budget function that accepts an output reserve.
  3. Supply the reserve per client/request:
  4. Anthropic and Bedrock: the 64k value currently requested by the client.
  5. OpenAI Responses, OpenAI Codex, OpenAI-compatible, and Genkit: the actual configured/requested completion reserve; retain 8,192 only where that is the request policy.
  6. Make existing tool-result reducers consume this same effective-budget calculation.
  7. Add shouldAutoCompact(estimatedInput, effectiveBudget) with an integer-safe 90% threshold.

Replace the string-only context overflow with a typed/sentinel error, e.g. ErrContextWindowExceeded, while retaining useful provider detail through wrapping.

Classify provider context-window failures before normal transient retry handling. The classifier should cover Keen's local post-tool-pruning overflow and recognizable provider 400/context-length failures. Context errors must not consume the normal network retry budget before automatic recovery is tried.

5. Integrate at provider-loop checkpoints

For each provider StreamChat loop:

  1. Start with a cloned canonical generic history from the incoming messages.
  2. Preserve the ordinary system prompt/instructions separately; it is never replaced by the automatic summary.
  3. After a model iteration returns tool calls and every tool result has been appended to both native and canonical history, reach the next-loop checkpoint.
  4. Before running the existing native tool-result reducer for the next request:
  5. estimate the unreduced native input;
  6. if at least 90% of the effective budget, run proactive automatic compaction;
  7. on success, replace local native request state by converting [original system/instructions + synthetic replacement message], clear incompatible pending native state, update canonical history to the same replacement, then continue the loop;
  8. on proactive cancellation/failure, leave all native and canonical state unchanged, suppress another proactive attempt until the history advances, and issue the original next request.
  9. If the existing reducer cannot make the request fit, force automatic compaction against the unreduced canonical snapshot. On success, rebuild state and retry once. On failure/cancellation, end the turn safely.
  10. If a provider returns a classified context-window error, emit a normal retry/rewind signal for any partial model output, force automatic compaction, and retry that sampling request once.
  11. If the post-compaction retry still exceeds the window, terminate with the context error. Never recursively compact or repeatedly retry unchanged context.

Provider-specific reset points:

Client Native loop history to rebuild after successful compaction
OpenAI-compatible oaiMessages in internal/llm/openai.go
OpenAI Responses input in internal/llm/openai_responses.go
OpenAI Codex input while preserving separate instructions in internal/llm/openai_codex.go
Anthropic msgParams while preserving systemBlocks in internal/llm/anthropic.go
Genkit aiMessages in internal/llm/genkit.go
Bedrock msgParams while preserving system in internal/llm/bedrock.go

The current pendingState implementations are native, process-local recovery state. On successful automatic compaction, clear the matching pending state because its tool call/result IDs point to discarded native history. A compaction snapshot must include any meaningful active-loop work before that reset; do not silently re-inject stale pending records after the replacement.

6. Prompt and request refactor

Update internal/llm/systemprompt.go:

  1. Strengthen the shared compactionPrompt to explicitly preserve:
  2. goals and non-negotiable instructions;
  3. current progress, completed actions, and remaining work;
  4. relevant files, commands, errors, and important tool discoveries/results;
  5. the immediate next action needed to continue an active agent loop.
  6. Keep the existing structured headings so manual /compact output remains familiar.
  7. Add BuildAutoCompactionPrompt() or an equivalent prompt extension stating:
  8. this is an internal agent checkpoint;
  9. the most recent user message is retained verbatim by Keen;
  10. do not reproduce that user message verbatim;
  11. output only the structured summary, with no preamble.

Extract common request construction from AppState.buildCompactionRequest into internal/llm so manual and automatic paths share the same core prompt and tools-disabled request behavior:

  • manual /compact: current history plus optional user focus hint;
  • auto compaction: canonical active-loop history, no focus hint, original system messages excluded because the normal agent system prompt is retained separately.

Keep AppState.StreamCompact as the manual command API, but make it call the shared builder.

7. REPL lifecycle, UI, persistence, and cancellation

Refactor compactionState in internal/cli/repl/repl.go from the current manual-only boolean into a mode/phase, for example:

type compactionMode uint8

const (
    compactionNone compactionMode = iota
    compactionManual
    compactionAutomatic
)

Manual /compact retains its existing behavior and visible streamed summary.

Add Tea message routing in internal/cli/repl/stream_msgs.go and internal/cli/repl/handlers.go for the automatic lifecycle events.

On auto_compaction_started

  • set automatic compaction mode;
  • store the child cancellation callback;
  • set m.loading.text and StreamHandler loading text to Compacting...;
  • retain the existing active parent stream, spinner, elapsed timer, and event channel;
  • do not start a second visible StreamHandler.

On Esc during automatic compaction

  • call only the auto-compaction child cancel function;
  • do not call m.stream.cancel;
  • do not interrupt/reset the active stream handler;
  • allow ordinary typing and queueing behavior to continue while the checkpoint is pending.

Ctrl+C retains its current full-turn cancellation behavior; parent cancellation also cancels the child request through context inheritance.

On auto_compaction_applied

Checkpoint the visible pre-compaction portion of the active agent turn before replacing app state:

  1. Render/persist its current stream segments as an assistant-turn session event so session replay retains all user-visible pre-checkpoint agent/tool activity.
  2. Do not append that checkpoint assistant message to AppState, since its content is represented by the replacement summary.
  3. Add a StreamHandler.Checkpoint-style method that renders/returns current content and clears response/segments while retaining the active event channel and stream state.
  4. Replace AppState history with the replacement message carried by the event.
  5. Append a KindCompactionApplied session event with the replacement messages and an empty status/transcript. This preserves conversation projection while avoiding a durable/replayed completion line.
  6. Start a fresh turn-memory accumulator for post-compaction agent activity.
  7. Clear stale lastUsage context metrics; the resumed model request will shortly publish the new low-context usage.
  8. restore a normal agent loading string;
  9. show Context compacted automatically. through showNotification, using the existing two-second notification expiration flow.

This checkpoint/reset is required to prevent handleLLMDone from appending pre-compaction assistant content a second time after the compacted replacement history.

On automatic cancellation/failure

  • restore the normal agent loading text and clear automatic mode/cancel state;
  • never alter AppState, session projection, provider local history, or stream contents;
  • for proactive compaction, allow the loop to continue unchanged;
  • for forced recovery, let the provider loop emit its terminal actionable context-limit failure.

The existing manual handleCompactionDone / handleCompactionError paths must continue to apply only when the mode is compactionManual.

8. Headless lifecycle

Update internal/cli/repl/headless_run.go to consume the same automatic lifecycle events.

On successful automatic compaction:

  1. checkpoint current handler segments to the session as an assistant-turn event;
  2. replace headless AppState history with the event replacement messages;
  3. append the empty-status CompactionApplied event;
  4. reset handler content and turn-memory accumulator for resumed work;
  5. preserve pre-checkpoint user-visible agent text in a separate accumulated result buffer, so final keen run output does not lose agent text merely because the conversation state was compacted.

Do not write Compacting..., the private summary, or the transient REPL completion notification to headless text/JSON output.

On forced compaction cancellation/failure, return the terminal context-limit error after persisting available partial agent activity, consistent with existing incomplete/error behavior.

9. Session projection and replay

No new session event kind is required. Reuse KindCompactionApplied in internal/session/event.go because it already persists replacement Messages and session.BuildConversation already replaces projected history.

For auto compaction:

  • persist the replacement messages;
  • leave Status empty, because completion feedback is transient rather than replayed;
  • persist visible agent work immediately before the checkpoint as an ordinary assistant turn;
  • rely on session.BuildConversation to discard that checkpoint message from future model context when the following compaction event is projected;
  • retain it in the transcript event stream so REPL session replay still shows what the agent visibly did.

Manual compaction remains unchanged and may retain its existing visible status/transcript behavior.

Implementation Steps

  1. Introduce common policy, errors, and event types.
  2. Update internal/llm/context_reducer.go with explicit per-request effective-budget helpers, a 90% predicate, and ErrContextWindowExceeded.
  3. Update internal/llm/message.go / internal/llm/client.go with auto-compaction lifecycle payloads and DisableAutoCompaction stream option.

  4. Add provider context-error normalization before retry handling.

  5. Build the shared silent compactor.

  6. Add internal/llm/auto_compaction.go.
  7. Extract request construction from internal/cli/repl/appstate/state.go.
  8. Add automatic replacement-message construction and last-user validation.
  9. Update internal/llm/systemprompt.go with strengthened base and auto-specific instructions.

  10. Ensure nested compaction uses no tools, one-shot state, and disabled automatic compaction.

  11. Represent active-loop tool results in canonical history.

  12. Extend cloned transient tool-result data in internal/llm/message.go and provider formatting in internal/llm/message_format.go.
  13. Adapt each client’s tool execution path to return/capture rich per-tool activity only for the active loop.

  14. Append a canonical generic assistant/tool message after each completed tool batch.

  15. Integrate checkpoint/rebase behavior in every provider loop.

  16. Add the same pre-request checkpoint around the current reduction call in:
    • internal/llm/openai.go
    • internal/llm/openai_responses.go
    • internal/llm/openai_codex.go
    • internal/llm/anthropic.go
    • internal/llm/genkit.go
    • internal/llm/bedrock.go
  17. Rebuild the respective native history after a successful replacement.
  18. Implement proactive suppression after cancellation/failure and one forced-recovery retry.

  19. Keep current tool-result pruning after automatic compaction as a final guardrail.

  20. Handle automatic lifecycle events in the REPL.

  21. Add automatic compaction mode, event routing, loader overrides, and Esc child-cancel behavior.
  22. Add the stream-handler checkpoint operation.
  23. Persist checkpoint transcript + compaction replacement correctly without duplicate AppState messages.

  24. Show the transient success notification and restore normal loading state.

  25. Handle events in headless runs.

  26. Add the matching checkpoint, replacement, session persistence, and output-prefix behavior to RunHeadless.

  27. Keep all compaction output private.

  28. Preserve manual /compact.

  29. Verify its current command, streaming summary, cancellation, and Context compacted. persistence/replay behavior continue to work after the mode/request-builder refactor.

Test Plan

internal/llm

  • Effective budget tests:
  • output reserve and safety margin are applied correctly;
  • 90% triggers exactly at the intended boundary;
  • no trigger below the boundary;
  • per-provider 64k/8k reserves are honored.
  • Context error tests:
  • local post-pruning exhaustion wraps ErrContextWindowExceeded;
  • recognizable provider context errors normalize to the sentinel;
  • context overflow is not retried as a generic transient failure.
  • Auto compaction request tests:
  • system messages are excluded from summarized history;
  • tools are disabled;
  • nested calls are OneShot and DisableAutoCompaction;
  • latest user task is retained verbatim in the synthetic wrapper;
  • empty summary/no last user fails transactionally;
  • automatic prompt includes active-loop preservation and no-verbatim-restatement rules.
  • In-flight tool-history tests:
  • raw output is available to the compaction request during a loop;
  • transient output is cloned safely;
  • completed normal session/app history does not begin persisting raw tool output.
  • Provider loop tests for all six clients using current fake stream factories:
  • normal tool response -> auto compaction -> resumed model request;
  • compaction summary is absent from parent chunks/reasoning;
  • started and applied events occur in order;
  • rebuilt request contains the synthetic checkpoint and does not contain discarded native history;
  • proactive cancellation leaves original request history intact and resumes it once;
  • forced local/provider overflow compacts and retries once;
  • a second overflow after recovery fails terminally instead of looping.

internal/cli/repl

  • Stream-event routing maps all automatic lifecycle events without treating them as manual /compact completion.
  • Start event changes only the existing loader text to Compacting....
  • Esc cancels the child auto-compaction callback but not stream.cancel; Ctrl+C still cancels the parent stream.
  • Applied event:
  • does not render summary text;
  • checkpoints pre-compaction segments to session output;
  • replaces AppState with one correctly wrapped synthetic user message;
  • resets current stream content/turn memory without disconnecting the event channel;
  • does not duplicate pre-compaction assistant content on final done;
  • shows and expires Context compacted automatically. through notification state.
  • Proactive cancellation/failure restores normal loader and preserves original state.
  • Manual /compact tests continue to pass unchanged.

Headless/session tests

  • A headless auto-compaction event updates the session projection and continues to final output without printing the internal summary.
  • Headless final output retains pre-checkpoint visible agent text plus post-checkpoint text in the correct order.
  • Reloaded session projects the replacement message for subsequent model calls while replaying pre-checkpoint visible agent activity.
  • Auto compaction has no replayed completion line; manual compaction retains existing replay behavior.

Final verification

Run after implementation:

gofmt -w <modified-go-files>
go mod tidy
go test -race ./...

Non-Goals

  • Do not implement provider-native Anthropic/OpenAI compaction APIs in this issue.
  • Do not automatically compact merely because the REPL displayed the existing /compact suggestion.
  • Do not use a fixed timeout for automatic compaction.
  • Do not expose or persist the private compaction summary as a normal terminal assistant response.
  • Do not change the manual /compact output contract beyond prompt-quality improvements shared with automatic compaction.