Public benchmarks do not tell you how a model does on your work
The evaluation that would actually decide your model choice runs against your production data: support transcripts, claims, clinical notes, contracts, transaction histories. That is exactly the data you cannot paste into a vendor’s API to find out.
- Sending real records to a model vendor for evaluation is often the thing your policy exists to prevent.
- Sanitized or synthetic evaluation sets measure the sanitized case, not the one you care about.
- Public benchmarks leak into training data over time, so a strong score may be memorization rather than capability.
- Vendors, meanwhile, cannot hand you model weights to evaluate locally.
The model meets the data inside the escrow agent
Neither side has to give up what it cannot give up. The evaluation runs once, under terms both parties approved, and only the agreed measurements leave.
Agreed metrics only
Both parties fix the evaluation set, the scoring method, and the exact metrics released before anything runs.
No prompts or records retained
Your evaluation data is not exposed to the vendor and is not available for training, retention, or inspection.
Scores, not transcripts
What comes back is the measurement (accuracy, error rates, cost and latency profiles) not the underlying inputs and outputs.
Compare models on a benchmark only you hold
Because the evaluation set never leaves escrow, it stays genuinely held out. The same benchmark can be re-run against a new model or a new version months later and still mean something; all while ensuring it has not been published, scraped, or trained on.
- Score several candidate models against one private evaluation set under identical conditions.
- Re-run the same benchmark on each new model release to see whether an upgrade is real.
- Measure quality alongside cost and latency, so the trade-off is visible in one place.
- Keep the benchmark uncontaminated, because it is never disclosed to the models being scored.
Judge what the agent did, not just what it answered
Agents act by calling tools, reading sources, and combining data. Evaluating one means watching the whole trajectory, which is precisely the part that is too sensitive to hand over.
- Run agents against real tasks and real data inside escrow.
- Score task completion, tool-use correctness, and failure modes over the full trajectory.
- Check whether an agent accessed or revealed more than it was permitted to.
- Release only the aggregate evaluation, never the trajectories themselves.
This is the same data-aware authorization described in programmable authorization for shared data, applied to evaluation.
Both sides of the evaluation get something they could not get alone
Model buyers
Choose a model on evidence from your own workload instead of a leaderboard, without moving regulated or contractually restricted data to a vendor.
Model and agent vendors
Prove performance on a prospect’s real data without ever receiving it, and without asking them to run an evaluation you cannot verify.