2026-06-10 · Agent work tracker

Roadmap Implementation Tracker

Living work tracker for autonomous agents. Pick tasks from here, implement in assigned scope, update status in ROADMAP_IMPLEMENTATION_TRACKER.md.

LoCoMo: benchmarks/results/locomo10_condensate_v53_fair.json · comparative report

Deprecated: benchmarks/loco_benchmark_work_tracker.md (LOC-001–017 only)

[x] Completed [/] In progress [ ] Pending

wsl -e bash -lc "cd /mnt/c/LocalProjects/Condensates && <command>"

Parallel agent timeline (rough estimate)

Estimates assume WSL + Docker benchmarks, phase gates respected (do not parallelize Phase 4 with Phase 2), and fair full runs as the main wall-clock bottleneck (~4–8 hours each on one machine).

LoCoMo recovery path (Phases 0–4) - critical path

Mode	Calendar time	Notes
1 serial agent	4–6 weeks	One LOC per session; 3–4 fair runs through pipeline
2 parallel agents	2.5–4 weeks	Agent A: retrieve/router; Agent B: CI/scripts/condenser; merge at phase gates
3–4 parallel agents	2–3 weeks	+ dedicated benchmark runner; diminishing returns on same repo/files

Per-phase effort (engineering days)

Phase	Serial	2 agents	Parallelizable tasks
0 - Gates	1–2 d	0.5–1 d	LOC-027 ∥ LOC-029
1 - Collateral	3–5 d	2–3 d	028d ablation ∥ 028a/b
2 - Evidence	7–10 d	5–7 d	LOC-025 condenser ∥ LOC-030/031 router
3 - Multi-hop	7–10 d	5–7 d	LOC-026 partial ∥ LOC-011
4 - Adversarial	7–10 d	5–8 d	Mostly sequential; mini-gate each pattern
Fair runs + report	2–3 d	2–3 d	Wall-clock bound; not parallel on one host

Milestone targets vs LOC-020 (95% GTM)

~87–88% overall after Phase 2 - realistic with 2 agents in ~2 weeks from Phase 0 start
~90–92% overall after Phase 4 - +2–3 weeks more; adversarial 70–75%
≥95% GTM gate - not fully scoped in Phases 0–4; budget +4–8 weeks iteration (graph, single-hop polish, ingest) beyond Phase 4

Full tracker (all pending tranches)

~120+ pending items across Tranches 1–7 (MCP, enterprise, cloud, multimodal). With 4 parallel agents and LoCoMo as P0: LoCoMo recovery ~3 weeks, remaining roadmap 6–12+ months depending on scope cuts.

LoCoMo-10 v5.3 fair - score history

Canonical: locomo10_condensate_v53_fair.json (2026-06-10). Peak: 85.4%. Baseline: locomo10_condensate_v53_fair_prev.json.

Metric	Peak	Current	Industry
Overall retrieval	~85.4%	83.6%	92.5%
Tokens / query	~2,100	1,647	6,956
Native answer	-	72.6%	-

Type	Peak	Current	Δ
Open-domain	96.4%	92.5%	−3.9
Temporal	97.2%	95.6%	−1.6
Single-hop	89.0%	84.0%	−5.0
Multi-hop	84.4%	81.2%	−3.2
Adversarial	54.7%	58.3%	+3.6

Prev → current: +46 / −82 (−36 net). GTM gate: ≥95% overall at <7k tokens (LOC-020). Current −11.4 pts from gate. Do not change the harness to improve scores.

LoCoMo recovery plan

Pick one task per session. Fair full only after phase batch passes LOC-027.

flowchart TD
    P0[Phase 0: Regression gates] --> P1[Phase 1: Stop adversarial collateral]
    P1 --> P2[Phase 2: Evidence fidelity]
    P2 --> P3[Phase 3: Multi-hop graph]
    P3 --> P4[Phase 4: Surgical adversarial to 75%]
    P4 --> Fair[Full fair run v53]
    P0 --> Mini[locomo_mini after each PR]
    P2 --> Mini
    P3 --> Mini

Milestone	Overall	Adversarial	Single-hop	Multi-hop	Open-domain
After Phase 1	~84.5%	~57–58%	~86%	~81%	~93%
After Phase 2	~87–88%	~58–60%	~90%	~82%	~95%
After Phase 3	~88–89%	~60%	~90%	~88%	~95%
After Phase 4	~90–92%	~70–75%	~90%	~88%	~95%
GTM LOC-020	≥95%	-	-	-	-

Phase 0 - Measurement discipline (1–2 days)

Never ship retrieval changes without category regression detection.

[ ] LOC-027 CI blocking gate on locomo_mini.json
[ ] LOC-029 Extend compare_fair_runs.py loss tagging
[ ] Category slice validation: conv-26, conv-41, conv-49

Phase 1 - Stop adversarial collateral (3–5 days)

[ ] LOC-028a Trap-filter safety valve
[ ] LOC-028b Split reranking tiers
[ ] LOC-028c Audit 24 single-hop losses
[ ] LOC-028d Ablation RETRIEVE_SWAP_TRAP_FILTER=0

Phase 2 - Evidence fidelity (1–2 weeks)

[/] LOC-024 Source-turn hydration
[ ] LOC-030 Hydration trigger expansion
[ ] LOC-025 Observation condensation fidelity
[ ] LOC-031 Re-rank hydrated turns above observations

Phase 3 - Multi-hop graph (1–2 weeks)

[ ] LOC-011 Mandatory 2-hop retrieval + counterfactual decomposition
[ ] LOC-026 Multi-evidence merge

Phase 4 - Adversarial to 75%+ (1–2 weeks)

[x] LOC-018a Entity-swap recall path
[x] LOC-018b Narrow swap-trap filter
[ ] LOC-018c Counterfactual path polish
[ ] LOC-018d Undetected possessive bucket
[ ] LOC-018e Redact trap clause vs empty context

Phase 5 - Fair-run variance (ongoing)

[ ] LOC-032 Double fair runs; report mean ± range
[ ] Freeze condenser version in bench env
[ ] Resume only for infra failures

GTM & parity

[ ] LOC-019 Single-hop ≥90%
[ ] LOC-020 Overall ≥95% at <7k tokens
[ ] LOC-015 Production retrieve defaults (blocked on LOC-020)
[ ] LongMemEval first run
[ ] LOC-023 SDK bulk_episodes / session retrieve

Done: LOC-016, LOC-021, LOC-022, LOC-018a, LOC-018b

make test-benchmarks
make test-locomo-v53-fair
make test-locomo-report

Tranche 1: Frontend & UI

Scope: frontend/

[x] Break App.jsx into modules; React Router; Zustand/Jotai; error boundaries
[ ] MCP Apps (SEP-1865) graph visualization
[ ] Agent memory presentation via MCP
[ ] Memory editing through MCP App UI

Tranche 2: CI/CD, Testing & Evaluation

Scope: tests, .github, benchmarks

[x] Phase 1: pytest, integration, coverage, SAST, lint, Docker CI (all complete)
[x] LoCoMo harness LOC-001–017 subset; ContradictionBench; baselines; LOC-016 weekly mini
[ ] Run LongMemEval
[ ] Publish static research page (condensate.io/research)

See LoCoMo recovery for active LOC-011–032 work.

Tranche 3: Core Backend & Security

[x] Phase 1 complete (API keys, rate limit, logging, health, config)
[ ] Phase 4: cross-agent memory scopes, RBAC, team merge
[ ] Phase 5: audit trail, policy-as-code, SOC2, GDPR erasure, OTel, Grafana (16 items)

Tranche 4: Database, Graphs & Architecture

[x] Phase 1: Alembic, migrations CI, vector dim fix
[ ] Phase 3: taint tracking, forgetting engine, contradiction detection (13 items)
[ ] Phase 5: sharding, replicas, PgBouncer, edge SQLite mode

Tranche 5: Protocol & Intelligence Engine

[ ] MCP JSON-RPC, Streamable HTTP, OAuth 2.1, A2A (11 items)
[/] Multi-hop query planning (LOC-011)
[/] Dynamic context assembly (LOC-015/020)
[x] Token counting per query

Tranche 6: SDKs & Integrations

[ ] CMEF export, condensate CLI, VS Code extension
[ ] LangGraph, CrewAI, AutoGen, LlamaIndex integrations
[ ] AST codebase ingestion, multimodal pipeline

Tranche 7: Cloud Infrastructure

[ ] Multi-tenant K8s SaaS, billing, free/pro/enterprise tiers

Generated from ROADMAP_IMPLEMENTATION_TRACKER.md · source markdown