GPT-6 Software Maintenance: Sol’s Cost Test
OpenAI compares GPT-6.1 Sol and GPT-6 Astra on repository maintenance, exposing a tradeoff between API expenditure and reviewed cleanup findings.
GPT-6 software maintenance has a new cost comparison. OpenAI’s October 14 report puts GPT-6.1 Sol Ultra at 92% of GPT-6 Astra Ultra’s reviewed stale-experiment findings for 14% of its API expenditure. The decision is whether that tradeoff suits a particular repository audit.
GPT-6 software maintenance: what the experiment measured
Rodrigo Roim’s OpenAI maintenance report compares the same prompt across subdirectories in one repository, spanning more than 6,000 files and a million lines.
| Reported configuration | Findings | API expenditure |
|---|---|---|
| Astra Ultra | 142 | $109.06 |
| Sol 6.1 Ultra | 131 | $15.12 |
The totals include research subagents and automatic follow-up turns at Standard API pricing. Ultra is the experiment’s effort label, not an Ultrafast billing claim. A reported discrepancy involved constant flags used by other flags: Astra found cases Sol missed.
Count overlap before calling it coverage
Our interpretation is to compare the actual finding sets before choosing a configuration. Consider an invented pilot where two runs each propose ten cleanups. Seven proposals match; each run contributes three different proposals. The counts are equal, but the reviewer now has thirteen distinct candidates, not ten interchangeable answers. Equal totals would hide that difference.
Give each candidate a stable identity based on the affected object and proposed change. A file path alone is insufficient when one file contains several independent flags. Conversely, the same flag mentioned in three files should not automatically become three successes. Deduplication determines what the comparison means before any cost division is useful.
For this proposed pilot, record matched candidates, candidates unique to each run, rejected candidates and unresolved cases. Then ask a repository owner to investigate the disagreements. A model’s longer list is evidence of more proposals, not a complete inventory of everything removable. Without an independently established inventory, do not label the count ratio as recall.
This distinction changes the buying decision. A team seeking a cheap discovery queue can accept different omissions from a team signing off a complete migration. Define which job the audit serves before deciding that a cheaper list is sufficient.
Follow the dependency, not just a constant value
Here is our hypothetical maintenance case, not a reproduction of OpenAI’s repository. A flag called Premium Preview is permanently true, and a second flag called Checkout Trial uses it as an eligibility condition. A background job also reads the first flag’s name from configuration. A search limited to direct source-code references could produce an incomplete removal plan.
The reviewer needs separate answers about value, consumers and replacement behavior. A constant value establishes what one condition returns today. It does not identify every consumer or establish what a replacement should do when a consumer still expects the flag to exist.
For this example, our proposed triage record links the configuration entry, the dependent condition and the background job’s lookup. The owner chooses whether to simplify an expression, retain a compatibility value, or reject the cleanup. Those outcomes are different from deleting both flags because one currently returns true.
A focused follow-up would request the missing consumer evidence, leaving the original candidate record intact. If the second investigation changes the decision, preserve both the first rationale and the corrected dependency map. That makes the miss useful for a later comparison rather than quietly replacing a failed proposal with a successful one.
Price the completed workflow you intend to operate
OpenAI’s model-selection guidance recommends comparing models on the same inputs and retaining the lightest setting that meets the required quality bar. It also distinguishes work that needs a quick response from work that can run overnight. Apply both dimensions to maintenance: an interactive incident investigation and an unattended candidate-discovery pass need not have the same operating target.
Our suggested decision sheet gives each configuration its own run cost, elapsed time, accepted candidates and investigation backlog. If a cheaper first pass needs a stronger follow-up, measure that combined policy separately. Do not assume that a cascade recreates the stronger run’s findings or that two independently reported totals describe its price.
The GPT-6.1 Sol API reference prices input, cached input, cache writes and output separately. A token estimate therefore needs those categories, not just the number of files scanned. The pricing calculator can support a token-cost estimate; it cannot infer how many investigations or reviewer hours a repository will require.
Keep an operating-cost column alongside API expenditure if your team values review time. For an illustrative choice, a lower API bill can still be unattractive if its unresolved cases consume the maintainer’s available review window. That is an economic criterion to measure, not a reason to assume one model always wins. Use the Sol reference and Astra reference for model facts rather than converting this experiment into a family-wide price change.
Turn disagreements into the next evaluation
OpenAI’s evaluation guidance calls for task-specific criteria, representative examples and human feedback to calibrate scoring. A maintenance pilot can implement that process with local records; it does not require adopting a hosted evaluation product.
Our proposed acceptance record separates a plausible discovery from an approved code change. For the fictional flag above, discovery means identifying the object and its suspected redundancy. Approval requires an owner’s decision and evidence about the dependent behavior. A merged patch and an observed operational improvement would be later outcomes, recorded separately.
Keep rejected and unresolved dependency examples in the next comparison set. Otherwise, repeated easy successes can conceal whether the original omission was addressed. Freeze the repository snapshot and review criteria for a comparison, then make a deliberate new comparison when either changes.
The earlier Asana browser-cost report concerns request-history and caching policies. This maintenance experiment raises a different decision: how much dependency investigation a lower-cost audit leaves for the team. Select a workflow from its reviewed candidates and remaining work, not from an isolated price or finding count.