jev-gc  ·  v0.1.0

Real-time context garbage collection for LLM agents

jev-gc watches the OpenTelemetry spans your agent already emits and decides, while it is still running, what stays in the next prompt, what shrinks to a pointer, and what gets archived — and it can give any of it back.

• Python ≥ 3.10
• MIT
• 120 tests
• GitHub

Why this exists

A growing context window is a correctness problem, not just a billing one.

An agent that runs for twenty turns carries everything it has ever seen into turn twenty-one. The failed API call from turn three, the directory listing that turned out to be the wrong directory, the retrieved document that looked relevant and wasn't — all of it is still there, still being read by the model, still being paid for.

The usual answers are to summarize the history periodically or truncate it by age. Both discard the same thing: the judgment about which items still matter. A stack trace from turn three may be the single most important item in the window at turn twenty; a verbose directory listing from turn nineteen may already be worthless.

jev-gc makes that judgment continuously, as spans arrive. Anything decidable by a rule — is it pinned, did it error, is it recent, did it produce no output — is decided by a rule. Only the genuinely ambiguous remainder goes to Jev, a fast typed-classification model that answers a fixed question set rather than reasoning in prose.

Install

Not yet on PyPI — install from source.

git clone https://github.com/adigulalkari/Jev_GC
cd Jev_GC
pip install -e .

Optional extras, installable together or separately:

pip install -e ".[dev,langgraph]"

You will also need a Jev API key from the TypeSafe dashboard. Without one, jev-gc still runs — every Jev call fails open and the deterministic rules do all the work — but you are then using a recency filter, not a relevance filter.

Quickstart

Two ways in, depending on whether you already emit OpenTelemetry spans.

If your agent is already instrumented

This is the intended path. Attach jev-gc to your tracer provider and every span it exports flows through the pipeline automatically — you never construct a span record yourself.

from opentelemetry.sdk.trace import TracerProvider
from jevgc import JevGC

provider = TracerProvider()
gc = JevGC.from_config("jevgc.yaml")     # reads ${JEV_API_KEY} from the env
gc.attach_to_tracer_provider(provider)

# ... your agent runs; spans stream in ...

await gc.wait_all()                       # let in-flight spans finish scoring

prompt_context = gc.build_context(
    task="Reconcile invoice #4521 across ERP and warehouse",
    budget_tokens=8000,
)

build_context is what you splice into your next prompt. Call it once per turn, immediately before you call the model. The task string matters: it is the question Jev scores each span's relevance against, so it should describe what the agent is trying to do right now, not what it was doing at the start of the session.

If you are not emitting spans

Build SpanRecord objects yourself and hand them over. This is the documented escape hatch, and it is what the bundled example and the test suite use.

from jevgc.models import SpanRecord, SpanStatus

record = SpanRecord(
    span_id="tool-call-7",
    trace_id="session-1",
    name="query_erp_table",
    status=SpanStatus.OK,
    start_time_unix_ns=t0,
    end_time_unix_ns=t1,
    output_preview="ERP total: $4521.00",
    output_token_count=200,
    turn_index=3,
)

decision = await gc.observe(record)
print(decision.tier, decision.treatment, decision.reason)

Every decision carries a human-readable reason. When you are tuning thresholds, that field — not the tier — is the thing to read.

Telling jev-gc about your turns

gc.advance_turn()               # once per agent turn; drives the recency rules
gc.pin("system-prompt-span")    # never evict: system prompt, hard constraints, open TODOs
gc.mark_referenced("span-42")   # the agent just used this again — restore and keep it close

Configuration

One YAML file, environment variables, or kwargs.

JevGC.from_config("jevgc.yaml") loads the file and interpolates ${VAR} placeholders from the environment. Every value below is a default, so the only required key is the API key.

jev:
  api_key: ${JEV_API_KEY}              # required; SecretStr, never logged or repr'd
  base_url: https://api.typesafe.ai/v1
  timeout_seconds: 2.0
  max_retries: 2

prefilter:
  keep_last_n_turns: 3                 # always HOT, regardless of score
  drop_zero_output_after_turns: 10     # pure code, never reaches Jev

scorer:
  batch_max_size: 50                   # flush a batch at N spans...
  batch_max_wait_ms: 200               # ...or after T ms, whichever comes first

policy:
  relevance_keep_threshold: 0.35       # below this, demote out of the prompt
  min_confidence_to_drop: 0.6          # below this, fail open and keep
  error_default_treatment: keep_error_summary_only

store:
  backend: memory                      # memory | sqlite
  sqlite_path: null

archive:
  max_content_bytes: 8388608           # 8 MiB ceiling on retained evicted content

telemetry:
  emit_self_metrics: true              # mirror counters to OpenTelemetry metrics

The same settings are readable from the environment with a JEVGC_ prefix and __ as the nesting delimiter — JEVGC_JEV__API_KEY, JEVGC_POLICY__RELEVANCE_KEEP_THRESHOLD — or passed directly as a dict via JevGCConfig.from_dict({...}), which is what the tests do.

The two knobs that matter most

relevance_keep_threshold sets how aggressive eviction is. Raise it and more spans leave the prompt; lower it and you keep more and save less. min_confidence_to_drop is the safety catch: when Jev is less certain than this, the item stays regardless of its score. Config load is the one place jev-gc fails fast — a missing API key raises ConfigurationError at startup rather than at the first eviction.

How it decides

Three tiers, and a path back from the bottom two.

The deterministic rules, in order

First match wins. Four of these six outcomes never cost a Jev call at all, which is what keeps the common case cheap.

An errored span matches rule 2 and is therefore never dropped — but it is still routed to Jev's error track, because only its payload size is negotiable, never its presence. A 4 KB stack trace can become a one-line error type; it cannot become nothing.

Fail open, deliberately

When Jev times out, errors, or answers below min_confidence_to_drop, the item stays at its current tier. The asymmetry is the point: a false keep costs a few tokens, while a false drop breaks a reasoning chain several turns later with no error message and no trace.

API reference

JevGC is the only class most users touch.

Types

Exceptions: JevGCError is the base; ConfigurationError, JevAPIError, JevTimeoutError and JevRateLimitError derive from it. Only ConfigurationError is raised at startup; the Jev errors are caught internally and turned into fail-open decisions, so you will see them in logs rather than in your call stack.

Recovering evicted context

The part that is easy to get wrong, and was.

An early version of this had a circular flaw worth describing, because it is the natural way to build this and it does not work. Evicted spans were reachable by span_id — but an agent that can no longer see a span has no way to learn its id. Discovery required already knowing the answer.

Worse, a compressed span's stored text is already a pointer. Promoting its tier back to HOT restored the pointer, not the content. The tier moved and nothing was actually recovered.

Both halves are fixed by the same mechanism. Every span leaving HOT is archived as an immutable snapshot, and a keywords-only index of what is evicted stays cheap enough to include in every prompt. The agent searches it by what it half-remembers, gets a span_id back, and asks for the content:

hits = gc.search_cold("archive released max_content_bytes")
# [ColdIndexEntry(span_id='read-archive-budget', tier=COLD, keywords=(...))]

text = gc.rehydrate(hits[0].span_id)     # full content, restored to HOT
gc.build_context(task=..., budget_tokens=900)   # now contains it again

The snapshot is immutable on purpose. rehydrate() resolves to what jev-gc observed at eviction time, never to a live re-read of the original source. If it re-read, "restore this span" could hand back content that never existed when the eviction decision was made, and every audit of that decision would be reading different evidence than the decision saw.

Measuring false eviction

The failure that leaves no trace.

A bad eviction does not raise. The agent simply answers slightly worse, several turns later, for reasons nothing in the pipeline can observe. Something moved to WARM at turn three looked irrelevant at turn three — and was, until the task shifted.

So every tier transition is logged, and jev-gc ships a shadow replay. Run the same task twice, once with jev-gc evicting and once with full context, then hand back both final answers:

findings = gc.analyze_regret(
    baseline_output=full_context_run_answer,
    evicted_output=gc_run_answer,
)

for f in findings:
    print(f.span_id, f.regret_score, f.eviction_reason, f.matched_keywords)

Each finding names an evicted span whose distinctive terms appear in the full-context answer and are absent from the GC'd one, scored by the fraction of that span's keywords that went missing. Spans that were evicted and later rehydrated are excluded — that is the recovery case, not a regret.

A worked example

One session, every decision, with the reasons.

scripts/worked_example.py runs a multi-turn bug investigation against this repository — real git grep output, real file reads, a real FileNotFoundError — and prints every decision. It is deterministic: no network, scripted Jev answers, byte-identical on re-run. Excerpts follow; run the script for all 301 lines.

Three spans, showing three different paths through the pipeline:

  span_id   grep-principle-5   (span name: grep_codebase, status OK)
  tool      git grep -n -i -- "principle 5" src/jevgc
  content   src/jevgc/gc.py:116:        # *presence* is guaranteed. But per SPEC.md §2 principl...
            [3 line(s), 285 chars]
  decision  tier=HOT   treatment=include_full             used_jev=True
            relevance=0.91   confidence=0.93
  reason    above_relevance_threshold treatment=include_full
  why       ambiguous -> Jev. Scored high: this is where the rule is written down.

  span_id   read-archive-budget   (span name: read_file, status OK)
  tool      read_file src/jevgc/archive.py lines 207-217
  decision  tier=COLD  treatment=drop                     used_jev=True
            relevance=0.07   confidence=0.88
  reason    below_relevance_threshold treatment=drop
  why       dead end: a different thing called a budget (the archive's memory ceiling, not the
            prompt's). Scored low + DROP -> COLD.

  span_id   read-missing-module   (span name: read_file, status ERROR)
  tool      read_file src/jevgc/nonexistent_module.py (really raised)
  decision  tier=WARM  treatment=keep_error_summary_only  used_jev=True
  reason    error_treatment=keep_error_summary_only
  why       errored -> prefilter rule 2 KEEP, but still routed to Jev's error track: only the
            payload size is negotiable, never the presence.

Then the dead end from turn 1 turns out to matter after all. The agent cannot name it — it was evicted — so it searches for what it half-remembers:

  BEFORE
    'max_content_bytes' present in the assembled context? False
    tier of read-archive-budget                           cold
    assembled context                                     259 tokens

  gc.search_cold('archive released max_content_bytes discoverability')
    1. span_id=read-archive-budget  tier=cold  turn=2

  gc.rehydrate('read-archive-budget') -> 770 chars, restored verbatim

  AFTER
    'max_content_bytes' present in the assembled context? True
    tier of read-archive-budget                           hot
    assembled context                                     452 tokens (+193)
    cold index                                            3 entries

And the regret pass on an equivalent session where that span was not recovered, including the null control:

  gc.analyze_regret(min_keyword_hits=2) -> 1 finding(s)

    1. span_id=read-archive-budget  evicted_to=cold  turn=2  regret_score=0.33
       eviction reason:  below_relevance_threshold treatment=drop
       matched keywords: keywords, released, max_content_bytes, discoverability

  Control -- the same session with the two answers made identical: 0 finding(s).

For this eight-span session: 486 tokens of verbatim content stored as 279, of which the 900-token budget drew 259 into the prompt. The errored span survived compression, nothing was irrecoverable, and the one eviction that cost an answer could be named afterwards.

Measured cost

What the decision layer costs before you wire it into a hot loop.

The worry with a per-span decision layer is that it stops being free once a session gets long. The sweep below runs the same synthetic session shape at four sizes a hundredfold apart, five times each, reporting the median — a single run on a laptop cannot distinguish a real trend from scheduling noise.

Per-span cost rises 1.14× while span count rises 100×. That is a small but real increase, not noise: the observed range at 10 spans (81.9–90.2 µs) does not overlap the range at 1,000 (94.3–107.4 µs). Call it flat for capacity planning, but do not call it constant.

build_context() and search_cold() grow roughly linearly, which is expected and correct — the first sorts every candidate and greedily fills a budget, the second scans the whole keyword index. Both are called once per turn rather than once per span, and both stay in the low milliseconds at a thousand spans.

Memory

At a thousand spans the archive holds 208.7 KB — 2.2% of the 9.44 MB the process grew. The rest is the live per-span object graph, one ContextItem and GCDecision per span. Retaining evicted content is not what costs you memory; tracking spans at all is.

Note that released is zero at every point: the 8 MiB archive ceiling never fires at these sizes, so this sweep exercises the budget's accounting but not its eviction path.

Tiers and tokens

"Saved" is the plain net, full_content − rendered, and it can go negative: a pointer sometimes costs more than the short content it replaces. Clamping those cases to zero would sum the wins and drop the losses.

What this does not prove

Read this before citing any number above.

• Nothing here measures Jev's judgment quality. Every number on this page was produced against FakeJevClient with scripted answers. That isolates jev-gc's own pipeline cost, which is the right thing to measure for "what does this library cost me" — but the eviction decisions being timed are stipulated, not earned. How well Jev actually separates relevant from irrelevant context is a separate question this page does not answer.
• The token-savings figure is a property of the fixture. The benchmark generates spans in fixed proportions chosen to exercise every decision path. A real workload with a different shape will save a different amount. 46.8% is not a promise.
• The overhead numbers are an upper bound. The benchmark sets the batch window and the recency window to zero, because otherwise nearly every span short-circuits on a deterministic rule and the measurement captures nothing. Real configurations do strictly less work than this.
• One machine. Five repeats on one laptop, not a distribution across hardware.
• No network latency. Real Jev calls cross a network. Batching exists precisely to amortize that, and none of it appears in these timings.
• The archive's eviction path is untested at scale. released_count is zero at every sweep point; unit tests cover the budget, the benchmark does not.
• The regret analysis has never run against a real pair of agent runs. The mechanism is implemented and tested; the claim that it surfaces genuine false evictions in production is untested.

Reproduce it

git clone https://github.com/adigulalkari/Jev_GC
cd Jev_GC
pip install -e ".[dev]"

python scripts/worked_example.py                        # the session above
python scripts/benchmark_overhead.py --repeat 5 --json  # the tables above
pytest                                                   # 120 tests

The benchmark's raw output is committed at results/benchmark.json and written up in results/report.md, so a re-run can be diffed against what this page cites. The worked example is deterministic and should be byte-identical on any machine at the same commit.

Design rationale, including why each judgment is delegated to Jev or kept in code, is in SPEC.md.