MultiLangBench: GPT-6 Results and COBOL Migration
MultiLangBench reports results for GPT-6 Astra, GPT-6 Luna and GPT-6.1 Sol, alongside a separate COBOL migration evaluation.
OpenAI’s MultiLangBench report measures GPT-6 database building and COBOL migration, reporting stronger results on both.
MultiLangBench results and the separate COBOL test
| Evaluation | Model and effort | Reported result |
|---|---|---|
| SQLite-compatible engine, 20 languages | GPT-6 Astra, max | 87.4% average test pass rate; GPT-5.6 Sol: 26.8% |
| Same engine evaluation | GPT-6 Luna, max | 40.3%; estimated $0.77 per sample versus $3.48 for GPT-5.6 Sol |
| COBOL migration | GPT-6.1 Sol, low | 86.6% versus GPT-5.6 Sol’s 5.4% at the same low effort; 61% of its API cost |
| COBOL migration | GPT-6.1 Sol, higher effort | 97.9% accuracy |
These are OpenAI’s measurements. Read the research report.
Use the task contract, not a cross-task ranking, to decide which pilot to build.
Make the effort setting part of the comparison
A benchmark label should become an explicit configuration in a local comparison, not an informal instruction to “think harder.” OpenAI’s reasoning documentation identifies reasoning effort as a request parameter and explains that reasoning tokens consume context and are billed as output. A response’s visible length therefore cannot stand in for the model work charged to that request.
The GPT-6.1 Sol model documentation supplies the exact identifier, gpt-6.1-sol. It supports low through max effort, defaults to medium, and does not support none or minimal. Tool calling requires Responses; Chat Completions is available without tools. Record those choices alongside the result so that another developer can reproduce the configuration.
For a proposed comparison, change one setting at a time. Keep the repository revision, instructions and execution allowance fixed while comparing low with a higher effort. If the higher setting also receives extra files or a longer session, the outcome no longer isolates effort. Keep that expanded-context run as a separate experiment instead of merging its score with the first pair.
Our GPT-6.1 Sol launch report covers the model’s release and availability.
Build a migration acceptance packet before asking for a rewrite
The following is a suggested evaluation design, not an integration or benchmark run performed here. Pick one bounded legacy component whose owner can explain its inputs and outputs. Preserve the original executable and a fixed input collection. Write down which files, exit states and externally consumed records constitute its contract before generating a replacement.
For an illustrative batch-export component, the packet could distinguish a valid record, an absent record and a malformed record. Specify whether an empty export is an accepted result or an error. Include the output encoding and record boundaries if another program consumes the file. These details turn “looks equivalent” into questions a reviewer can answer without interpreting a model’s explanation of its own work.
Keep the expected outputs outside the material available to the coding session. A developer should be able to run the old component and the candidate against the same inputs, then inspect each disagreement. When an owner decides that a difference is intentional, record the approved contract change separately. Otherwise a modernization exercise can quietly become a requirements change.
OpenAI’s evaluation guidance recommends task-specific objectives, datasets and metrics, with logging and human calibration. Applied here, that means evaluating the artifact produced by the coding session, not grading the fluency of its migration explanation. The site’s debugging prompt can help structure a failing case; it is not a COBOL converter or a substitute for the acceptance packet.
Separate translation failures from execution failures
A failed comparison needs a useful destination. Keep compilation failures distinct from a program that runs and produces a wrong result. Preserve the command, environment and first diagnostic for the former; preserve the input and output difference for the latter. This proposed triage makes the next review concrete without assuming that all failures need a larger model.
Also decide what happens when a run produces no usable artifact. Excluding unfinished attempts from the denominator would answer a different question from “how often does this workflow deliver an accepted replacement?” Report the accepted fraction of all assigned tasks, then separately summarize the reasons for failure. Keep developer repair work visible rather than counting a manually corrected output as an unassisted model success.
For a first pilot, select a component whose behavior can be checked without connecting the replacement to a live transaction path. A successful offline comparison can support the next engineering decision. It does not itself authorize changing deployment, data access or production ownership.
Compare the cost of accepted work
Use an evaluation ledger with a row for every attempt: configuration, completed artifact, acceptance result, model usage and developer intervention. Calculate the model spend across all attempts before dividing by the number accepted. Keep engineering time in a separate column unless the team has agreed how to value it.
This denominator can change the practical comparison. A cheaper attempt that frequently needs another run may not produce a cheaper accepted artifact. Conversely, escalating every task can spend more without changing the acceptance outcome. The ledger should reveal which happened in this pilot, rather than assume either outcome from a headline rate.
Keep the general pricing reference separate from the research’s measured workload costs. This News report does not change subscription allowances, API rates or the site’s calculators. The next useful result is a reproducible acceptance comparison on a component the team actually maintains.