The testing process stages complete interactions between a simulated user and an agent, documenting every exchanged message and tool call into a single trajectory. Following the session, an LLM judge assesses the entire recorded conversation to determine performance. Beyond full agents armed with code search or web fetching capabilities, the platform tests raw inference calls directed to standalone models, Dedicated Inference deployments, or an Inference Router. Configuration relies on JSONL scenario sets, which require only a description field while supporting optional constraints like personas and turn caps.
Key points
Multi-turn evaluations catch consecutive dependency failures that traditional single prom…: Multi-turn evaluations catch consecutive dependency failures that traditional single prompt-and-response assessments regularly miss.
Optional stopping criteria instruct the simulated user to continue interactions until spe…: Optional stopping criteria instruct the simulated user to continue interactions until specific task conditions or turn limits are met.
Testing accommodates both complex agents using external tools like web fetching and simpl…: Testing accommodates both complex agents using external tools like web fetching and simpler, direct inference endpoints.
Sources checked by our analysis pipeline: 1
By Venkat Ranabothu
Ask a coding agent which Python version to upgrade to from 3.11, and it picks one from memory. It cites a few changes from that version’s release notes and calls the upgrade straightforward. A single-turn evaluation asking “Is this a reasonable recommendation?” would likely pass it.
An experienced human engineer works through it over several exchanges:
Each step builds on the previous one’s findings, and none of that is visible in a single prompt and response.
That is the gap Simulations - a capability of DigitalOcean Evaluations now in Public Preview - is built to close. Instead of scoring a single response, Simulations runs a full conversation between a simulated user and your agent, records every message and tool call in a trajectory, and has an LLM judge the whole thing. The launch announcement covers what Simulations does. Here’s how it works under the hood.
We’ll follow one question through every stage: which Python version should a service on 3.11 upgrade to? You’ll see how to write the scenario, simulate the user, run the agent, score the…
The Toronto-headquartered artificial intelligence developer is securing massive sovereign and private backing to expand its foundational model operations. This major capital injection establishes the company as one of the most valuable tech entities in the country. Strong domestic state support highlights strategic national efforts to maintain indigenous control over critical artificial intelligence technologies.
Growing public debate over large-scale technology infrastructure has officially reached the ballot box as local communities challenge these facilities. Brookings scholar Darrell West reports that resistance to the artificial intelligence boom is crossing traditional political lines. Voters across party affiliations are increasingly questioning the actual local advantages of host status while challenging broader technological initiatives.
Operating around the clock, an autonomous engineering agent built on the Nerve open-source runtime identifies root causes, compiles software, and resolves continuous integration failures. The system has expanded into fixing correctness and performance bugs across major database components, leaving human maintainers primarily to conduct code reviews. Its persistent memory architecture enables continuous learning from prior feedback to independently tackle intricate software investigations.