GPT-6 Astra Testing at Perplexity: APIs and Connectors
Perplexity describes testing workflows with simulated services. Explore the reported method and an original connector-testing example.
GPT-6 Astra testing at Perplexity includes generating a small program that simulates external services and checks an application’s full workflow, according to OpenAI’s September 14, 2026 customer story. Below is the reported approach, followed by an original example for developers testing an API or connector integration.
GPT-6 Astra testing at Perplexity: the reported approach
Perplexity cofounder Johnny Ho describes asking Astra to produce realistic responses resembling those from a language model API or connector. The application can then be tested against those substitute services. He reports needing to check on these workflows less frequently than with earlier models. OpenAI also describes using Astra for communications, software changes and production monitoring. These are attributed customer observations; the official Perplexity story does not publish test code or a measured reduction in testing time.
Turn the idea into a connector test
The following example is our suggested exercise, not Perplexity’s implementation or a benchmark we have run. Imagine a support dashboard that retrieves a document through a connector, sends its contents to a summarizer and displays the result. A useful first test follows that whole journey instead of checking only that the summarizer returns text.
Start by writing down the contract: the connector returns a document identifier, title and body; the summarizer returns a short summary; the dashboard displays either that summary or a recoverable error. Specify what should happen when any part is missing. If the expected behavior is undecided, record the decision before asking a model to generate assertions.
Next, give Astra that contract and a small, sanitized example response. Ask it to propose substitute responses and identify which dashboard behavior each one should exercise. Review the proposed responses against the actual integration format. A plausible JSON object with the wrong field names tests an invented interface.
For teams starting from a bug report, our reproduction planning prompt helps separate the observed symptom, expected behavior and competing explanations before a fix is attempted.
An example matrix for API and connector responses
Use the following rows as starting cases for this hypothetical dashboard. Replace the proposed expectations with your product’s agreed behavior. Keep the real response schema in the fixtures so that a failed test points to a meaningful integration problem.
| Substitute response | Proposed dashboard behavior | Evidence to inspect |
|---|---|---|
| A complete document and valid summary | Display the summary beside the correct document title | Document identifier, displayed title and summary |
| A valid response containing no documents | Show an empty state without retaining an older summary | Empty-state message and absence of stale content |
| A connector timeout followed by a successful retry | Explain the failure, then show the recovered result | First error, retry action and final visible state |
| A document with a missing required body | Stop before summarization and show a clear error | Validation result and absence of a summarizer request |
| A malformed summarizer response | Show a recoverable result error | Received fixture and rendered error message |
| Two documents requested in quick succession | Keep each result attached to its own document | Request identifiers and final document/result pairing |
For each row, save the input fixture, expected result, observed result and the assertion that decides pass or fail. Keep the fixture stable while debugging. If the model regenerates its simulated response on every run, a changed result becomes harder to attribute to the application change you intended to test.
The retry row deserves a complete sequence: trigger the failed request, activate Retry and inspect the recovered dashboard. Merely verifying that a retry function was called leaves the user’s final screen unchecked. Our separate frontend QA exercise demonstrates this distinction with a search-and-clear interaction and recorded browser evidence.
Check the generated test before trusting a pass
For this exercise, ask Astra to explain how each assertion could fail. Inspect whether the test reaches the application code or simply compares two values that the test itself created. Confirm that the simulated connector replaces only the intended dependency and that the dashboard still performs its normal parsing and state updates.
Then make a deliberate, temporary error in an isolated copy of the application: for example, display a previous document’s summary after the new request finishes. The relevant assertion should fail. Restore the application code and rerun the same test. This proposed check helps expose a test that passes without observing the behavior named in its description.
Keep two results separate in your report. A controlled substitute response can exercise application logic predictably. A test against the real service checks whether the integration still matches that service. Passing the first does not establish the second. Record which dependencies were simulated so a reviewer can interpret the result without reading the entire test program.
Evaluate the generated answers separately
If your application also generates summaries or answers, assess their quality in addition to the surrounding request flow. OpenAI’s evaluation best practices recommend defining an objective, collecting representative examples, choosing evaluation criteria, comparing runs and expanding the evaluation set as new cases appear. The guidance includes ordinary, edge and adversarial cases, with human judgment used to calibrate automated scoring.
Applied to our dashboard example, one check could verify that a displayed summary refers to the correct document. A separate content evaluation could ask whether the summary preserves a crucial exception in that document. A successful HTTP response answers neither question by itself. Decide those criteria before comparing model-generated outputs.
The evaluation guidance is useful independently of a particular evaluation service. The linked page currently flags the Evals platform’s deprecation, so it should not be read as a recommendation to start a new dependency on that platform.
What developers can take from this case
The Perplexity report gives developers a concrete integration-testing idea to investigate. It does not establish universal reliability or announce Astra availability for Perplexity subscribers. Start with one workflow whose expected outcome your team can explain, and expand from demonstrated failures or gaps.
For a related customer example, our Cognition and Devin news brief covers Astra’s role in testing software and showing results. This page focuses on simulated API and connector responses; the linked brief covers a separate company’s reported use.