Internal benchmark · June 25, 2026

Kaptain vs Codex Direct vs Codex + CodeGraph

The clean same-task result shows Kaptain preparing KodeGraph context before Codex runs: same gold-file recall, 78.1% fewer total Codex tokens, and zero Codex-side tool calls.

78.1%

Total token reduction

11,438 total Codex tokens with Kaptain-prepared KodeGraph context versus 52,241 for Direct Codex.

4.6×

Fewer total tokens

Kaptain used about one fifth of Direct Codex's total token budget on the exact lookup.

1.000

Same recall

Both clean June 25 runs found the gold file: server/kaptain/context.py.

Codex tool calls

Kaptain supplied the answer-bearing context up front.

Supported internal claim: on the June 25 exact-lookup task, Kaptain-prepared KodeGraph context cut Codex token use while preserving answer quality.

What the three conditions mean

Condition	What ran	What it tests	Validity
Kaptain + KodeGraph context	Kaptain calls KodeGraph first, then supplies compact context to Codex.	Kaptain as the context orchestrator.	Clean for June 25 small exact lookup.
Codex Direct	Fresh `codex exec` with ordinary read-only shell lookup.	Baseline Codex spending its own budget to discover the file.	Clean for June 25 small exact lookup.
Codex + CodeGraph MCP	Codex calls an external CodeGraph MCP surface itself.	External tool-surface comparison.	Valid only for the June 21 equal-recall medium row.

Clean same-task benchmark

Task: Find where the symbol assemble_context is defined and briefly state what file owns it.

Metric	Codex Direct	Kaptain + KodeGraph	Difference
Success	yes	yes	tie
Gold-file recall	1.000	1.000	tie
Total Codex tokens	52,241	11,438	-40,803
Uncached input tokens	19,444	9,377	-10,067
Output tokens	413	141	-272
Codex tool calls	2	0	-2
Wall time	13.1 s	5.5 s	-7.6 s

Kaptain resolved kodegraph_search plus kodegraph_node first. Codex then answered from supplied context instead of spending model turns on shell exploration.

Valid CodeGraph comparison row

The June 21 three-way run has one valid equal-recall head-to-head row: medium_approval_gate. Both CodeGraph MCP and KodeGraph MCP found the gold files with 1.000 recall.

Metric	Codex + CodeGraph MCP	Codex + KodeGraph MCP	Result
Success	yes	yes	tie
Gold-file recall	1.000	1.000	tie
Total tokens	1,013,030	636,086	37.2% lower with KodeGraph
Uncached input tokens	179,117	124,910	30.3% lower with KodeGraph
Output tokens	4,345	4,936	CodeGraph lower
Tool calls	40	41	near tie
Wall time	143.5 s	163.8 s	CodeGraph faster in this row

Use this row as a token-efficiency comparison, not a latency claim. It supports KodeGraph's context-efficiency advantage at equal recall.

Three-way internal view

clean row

Kaptain + KodeGraph

11,438 total tokens on the clean June 25 exact lookup. Kaptain prepared the working set and Codex made zero tool calls.

clean row

Codex Direct

52,241 total tokens on the same June 25 task. Codex solved it, but spent more budget discovering context.

separate valid row

Codex + CodeGraph MCP

1,013,030 total tokens on the valid June 21 medium row. CodeGraph reached 1.000 recall but cost more tokens than KodeGraph in that row.

The product direction is to prefer Kaptain-prepared context where Kaptain can resolve the working set, keep KodeGraph available as an interactive tool surface, and remeasure raw Codex self-calling tool paths after cancellation behavior is fixed.

Diagnostic boundaries

Codex + KodeGraph MCP self-call on June 25: the row used 81,836 total tokens and succeeded only after shell fallback because kodegraph_search and kodegraph_node were cancelled by the Codex MCP layer. Direct MCP health was verified separately, so this is not a KodeGraph retrieval failure.

June 21 direct rows: those rows used a temporary CODEX_HOME, and Codex's Linux sandbox helper could not initialize: bwrap: execvp codex-linux-sandbox: No such file or directory. Do not use them as a public Direct Codex baseline.

June 21 small and large graph rows: some correct files were described but not placed in strict answer_files JSON, so the scorer counted 0 recall. The next harness should split retrieval success from answer-format success.

Engineering follow-up

Rerun the three-way harness with a non-temporary CODEX_HOME.
Fix or avoid Codex MCP tool-call cancellation for KodeGraph and CodeGraph self-call rows.
Score retrieval success separately from strict JSON answer formatting.
Add one clean same-task row covering Kaptain-prepared context, Codex Direct, and Codex + CodeGraph MCP.

Sources: codex_direct_vs_kaptain_kodegraph_precontext.small.latest.json, codex-3way-20260621-0134.raw.json, and codex-3way-20260621-0134.md.