Terminal X GPT-6.1 Sol: What 88.6% Agreement Means
GPT-6.1 Sol review results; GPT-6 Luna handles volume.
Terminal X GPT-6.1 Sol matched a senior reviewer in 88.6% of 500 judgments, OpenAI reported October 20.
For teams assessing AI review systems, the useful result is the change in disagreements and the kinds of errors caught. Those questions are more actionable than treating the headline percentage as a general accuracy score.
Terminal X GPT-6.1 Sol: read the agreement result
The official case reports 86.4% for its unnamed comparator.
Calculated from the rounded rates; underlying judgment labels are unpublished.
| Implied counts | Sol | Comparator |
|---|---|---|
| Agreements | 443 | 432 |
| Disagreements | 57 | 68 |
Pairing the judgments is more informative than treating two rounded rates as a complete comparison.
Neither calculation tells us whether the two systems failed on the same individual cases. For example, a model could fix many comparator errors while introducing new ones elsewhere. A buyer needs the paired case-level outcomes to identify those swaps. Aggregate agreement alone cannot show which mandate, document type, or decision stage benefited.
Use the result to ask for a replay on your own reviewed examples. Do not convert agreement with one reviewer into a forecast of market performance or a promise that an investment recommendation is correct.
Mandate drift is a separate error measure
A count of flagged answers needs a denominator: how many answers actually violated the fund’s rules?
For a review queue, count both missed departures and compliant answers wrongly flagged. Catching more exceptions can help, but an excessive false-alarm load can consume the reviewer time the system was intended to save. Keep those two error types in separate columns rather than combining them into a single reassuring score.
A strong financial narrative can still recommend the wrong type of transaction for a particular fund. The practical acceptance question is therefore whether the answer follows that desk’s stated mandate, not whether the evaluator finds it persuasive.
A rubric needs evidence for each failed check
Its 54 pass/fail checks feed corrections and reasons into the firm’s Memory.
OpenAI’s evaluation guidance recommends choosing evaluation data to match the objective and using human feedback to calibrate automated scoring. It also describes isolating underlying model interactions in multistep workflows.
For a small pilot, our suggested review record would preserve the criterion, the cited passage, the verdict, the proposed correction and the human decision. This is an illustrative record design, not Terminal X’s published schema.
Suppose a memo quotes a valuation multiple accurately but uses a company outside the approved comparison set. The arithmetic check can pass while the comparable-selection check fails. Keeping those judgments separate lets an analyst replace the source without rewriting every otherwise valid part of the memo.
Before accepting a correction as a reusable instruction, decide whether it expresses a lasting policy or an exception for that deal. Store the relevant policy version with the example. Otherwise, a later evaluation may reward yesterday’s instruction after the investment team has changed its requirements.
Model routing and cost require their own comparison
GPT-6.1 Sol handles judgment; GPT-6 Luna handles volume. Reported query cost was roughly 30% below the unnamed comparator at similar speed.
Use the GPT-6.1 Sol reference and GPT-6 Luna model page when writing the routing test.
For an adoption decision, compare the cost of an accepted, reviewed output. Include human corrections and the proportion of tasks that require another run. Our Asana workflow-cost report examines a different workload and explains why model changes and application changes should not be collapsed into one savings claim.
What to ask for before a production pilot
OpenAI’s grader documentation explains how reference answers support string checks, text similarity and model-based grading. Both that documentation and the evaluation guide carry product-deprecation notices; their general testing concepts are not an instruction to start a new hosted Evals deployment.
Ask for the rubric definitions and a sample of disagreements before asking for another headline benchmark. Have an investment reviewer decide which disagreements would actually change the memo, escalate the case, or prevent an out-of-mandate recommendation from reaching the next stage.
Mark each verdict as no change, source replacement, mandate correction or escalation. A writing fix can be checked by its author; a mandate change needs the designated approver. Grouping these outcomes helps the desk compare reviewer workload without treating every correction as equally consequential. For each escalation, name a decision owner and attach the disputed passage. The queue entry should specify the decision needed, so that a reviewer can resolve the exception without reconstructing its context.
Keep one set of cases separate from the examples used to adjust the rubric. Then compare a proposed revision on that untouched set. Record the source documents and policy version so that an apparent improvement can be traced to the change being tested rather than to a different evidence pack.
The versioned-workflow idea connects to our V7 institutional-memory case. A useful pilot should test that particular review job: whether the evaluator finds consequential departures, explains them with evidence, and reduces the work needed to reach a human-approved decision.