Your Eval Gate Runs on a Platform That Is Being Shut Down

Eval platform deprecation is not an outage you can retry. When a vendor retires the hosted eval dashboard, API, and graders your release gate calls, the gate does not fail, it stops existing.

Viral RuparelNo. 93 of 9311 min read

Two panel figure contrasting a release gate wired directly into a hosted eval API that is being retired against a gate backed by eval cases, graders, and thresholds stored as code in the repository with the hosted platform as a swappable runner.
Figure. Hosted gate versus a portable eval harness

The platform you built your gate on has a shutdown date

A team I was helping ran their release gate through a hosted eval product. Every pull request fired a job that sent fifty prompts to the vendor's eval API, scored the answers with the vendor's managed graders, and blocked the merge if the pass rate slipped below ninety percent. It worked well enough that nobody looked at it for months. Then the provider posted a deprecation notice. The eval dashboard, the API, and the graders would go read only on a date a few weeks out, and be gone after that. The gate was not failing. It was scheduled to stop existing.

This is not a hypothetical. On 30 November 2026, OpenAI's Evals dashboard, API, and graders go read only for eval workflows, with Promptfoo named as the migration path. The same date retires Agent Builder, and the /v1/prompts reusable prompt objects, with the blunt instruction to move prompt content into your application code. Three hosted primitives that teams quietly built pipelines on, pulled on one day. If any of those sat on the critical path between a commit and a deploy, that path breaks on a Monday for reasons that have nothing to do with your model or your code.

A model getting retired is a problem you already know how to frame. A platform getting retired is the same shape of problem pointed at your tooling instead of your weights, and it is the one teams tend not to plan for.

What is actually being retired

It helps to separate three things that a hosted eval platform bundles together. The first is the eval logic: the cases you send, the assertions you check, the grader prompts, and the thresholds that decide pass or fail. The second is the runner: the thing that executes those cases against a model and collects outputs. The third is the store: the run history, the trend lines, the dashboard you open after a bad release.

When the platform is deprecated, the runner and the store go away, and so does any eval logic that lived only inside them. That last clause is where the damage hides. If your assertions were clicked together in a web UI, if your grader was a managed rubric you configured in the vendor's account, if your thresholds were a setting rather than a file, then the deprecation does not just remove a service. It removes the definition of what passing means. You cannot re-point a gate at a new backend if the only copy of the gate's rules left with the old one.

This is the same lesson that a CI eval gate teaches on day one, now with the stakes made obvious. The gate is only as durable as the least portable thing it depends on. If that thing is a vendor product with a lifecycle page, your gate inherits the shutdown date.

A sunset is not an outage, and retries will not save you

When the eval API starts returning errors, the instinct from reliability work is to reach for the usual tools. Add a timeout, trip a circuit breaker, fail over to a backup. Those tools assume the dependency comes back. A brief eval API hiccup is worth a retry. A scheduled removal is not, because there is nothing on the other side of the retry. The endpoint is gone on purpose.

This is the same distinction that separates a provider outage from a model shutdown date. One is a reliability event you ride out at runtime. The other is a migration you run before a calendar date, and the only question is whether you start it on your schedule or the vendor's. A platform sunset belongs firmly in the second category. The move is to make your eval harness portable while the old platform still works, so that switching backends is a configuration change you test, not a rebuild you do under a deadline with the gate already dark.

Portability here means one specific thing: the eval logic lives in your repository as files, and the hosted platform is one interchangeable way to execute those files, not the place they are defined.

Keep the eval set and graders as code

Start by pulling the definition of a passing release out of the platform and into version control. The cases, the assertions, the grader prompts, and the thresholds become plain files that sit next to the code they gate and change through pull requests like anything else. Promptfoo's configuration is one shape for this, and it is not a coincidence that it is the migration target the vendor itself points to: evals as a config file are evals you can move.

# evals/support-router.yaml, lives in the repo, reviewed like any change
prompts:
  - file://prompts/support_router.txt    # the prompt is a file, not a vendor object
providers:
  - id: openai:gpt-5.2
  - id: anthropic:claude-opus-4-8         # a second backend, same cases
tests:
  - vars: { message: "I was charged twice this month" }
    assert:
      - type: equals
        value: billing                    # deterministic grader, no model call
      - type: llm-rubric                   # judge grader, prompt kept in the repo
        value: Names the double charge and offers a refund path
thresholds:
  pass_rate: 0.9                           # the gate lives next to the cases it gates

Two details in there matter more than they look. The prompt is referenced as a file, not a vendor prompt object, so the /v1/prompts deprecation cannot touch it. And the judge grader's rubric is written into the config rather than selected from a managed menu, so when you change backends the grader goes with you. The graders you lean on for anything subjective are your own LLM-as-judge rubrics, and they are far too load bearing to leave in someone else's account.

Make the platform a runner you can swap

With the logic in files, the hosted platform becomes a backend behind an interface you own. Your harness defines the contract, how a case is scored and how the suite rolls up into one number, and a given platform is one implementation of that contract. A local, in process backend that runs the same assertions is both your escape hatch and your test that the contract is real.

# evalrunner.py: your harness owns the contract, the backend is replaceable
from dataclasses import dataclass

@dataclass
class Case:
    vars: dict
    assertions: list

class EvalBackend:
    def score(self, case: Case, output: str) -> bool: ...

# a hosted platform is ONE implementation, not the source of truth
class HostedBackend(EvalBackend): ...
class LocalBackend(EvalBackend):           # runs the same assertions in process
    def score(self, case, output):
        return all(a.check(output) for a in case.assertions)

def run(cases, model, backend: EvalBackend) -> float:
    passed = sum(backend.score(c, model(c.vars)) for c in cases)
    return passed / len(cases)             # one number the CI gate reads

The gate itself then runs in your CI, where you can see it, rather than in a dashboard you rent. The CI step calls the harness, reads the single pass rate, and exits non-zero below the bar. The run output is an artifact you store in your own system, which is also what keeps your history alive when the vendor's store goes read only.

# .github/workflows/eval-gate.yml, the gate runs in your CI, not a vendor UI
- name: eval gate
  run: |
    promptfoo eval -c evals/support-router.yaml --no-cache
    promptfoo eval -c evals/support-router.yaml --assert   # exit non-zero below the bar
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
# publish the json result as a build artifact: the history is yours, not rented

Once the gate reads a number your own harness produced, the hosted platform is optional. You can keep using it for its dashboard and triage, point it at a second vendor, or run everything locally for a week while you migrate. None of those choices touch the definition of what passing means, because that definition never left your repository. This is also the groundwork for a living eval set and for online evals on canaries and shadows, both of which assume the eval logic is something you can run anywhere, not a product you are a tenant of.

The parts that bite

Grader parity is the first thing to check, and it is easy to get wrong. A managed grader and a rubric you move into your repo are rarely scored identically, because the judge model, its version, and its prompt all differ. Run both for a window on the same cases and compare pass rates before you cut over, or you will trade a dead gate for a gate that quietly passes things the old one caught. A judge grader that changes answer between backends is not a portable grader yet.

Run history is the second. The trend lines in a hosted dashboard feel like yours until the read only date, after which you cannot export them. If a regression investigation or an audit ever needs the last six months of eval runs, pull that data out as plain records before the cutoff, not after. The dashboard is rented. The exported data is owned.

Prompts and agent definitions are the third. The /v1/prompts and Agent Builder deprecations land on the same date as the eval one, and for the same reason: anything configured in a vendor console rather than written in your code is on a lifecycle you do not control. If your eval prompts, your production prompts, or your agent graphs live in a hosted builder, moving them into files is part of the same migration, not a separate project. The register of what your system depends on, including which SaaS primitives sit on critical paths, is worth keeping the way you would keep an agent and service registry, so a deprecation notice maps to a known blast radius instead of a scramble.

The last cost is honest to name: running your own harness means you now own the runner. You pay for the compute, you keep the dependencies current, and you maintain the backend adapters. That is a real cost. It is also much smaller than discovering, three weeks before a shutdown date, that the thing deciding whether you can ship is a product with an end of life.

The takeaway

A hosted eval platform is a convenience, and conveniences are fine right up until they become load bearing. The failure is not using one. The failure is letting it quietly become the only place your eval cases, graders, and thresholds exist, so that a routine deprecation notice turns into a dead release gate. The fix is cheap and boring: keep the definition of a passing release as files in your repository, make the platform a backend you can swap, and keep a local runner that proves the contract is real. Then a sunset is a backend change you test on a Tuesday, not an outage you discover on a Monday.

The same move protects you from the next three deprecations too, because the pattern generalizes past evals to any hosted primitive your pipeline leans on. If you are not sure which parts of your release path live in a vendor console with a lifecycle page, that audit is a good half day, and often an uncomfortable one. If you want a second set of eyes on where your gate actually runs and what it would take to make it portable, I am happy to take a look. No pitch, just the map.

ShareXLinkedInviralruparel.com/blog/eval-platform-deprecation-portable-harness

Questions this essay answers

01What is eval platform deprecation?

Eval platform deprecation is when the hosted product you use to run and score model evaluations, the dashboard, the API, and the managed graders, is scheduled to stop working on a set date. The evaluations still run until then, after which the endpoints return errors and the dashboard goes read only or disappears. It is different from a model deprecation, which retires the model you are testing. Here the tool that decides whether a release passes is the thing being removed, so your release gate can break even though your model and your code are untouched.

02How is a platform sunset different from an outage?

An outage is temporary and the service returns, so retries, timeouts, and failover are the right tools. A platform sunset is permanent and scheduled. The eval API you call is not coming back, so no amount of retrying restores it. The fix is portability decided before the date: your eval cases, graders, and thresholds live in your repository, and the hosted platform is one interchangeable backend rather than the system of record. Treat the two as separate failure modes, because the tooling that protects you from a brief outage does nothing for a scheduled removal.

03Should I keep my eval cases and graders in version control?

Yes. The eval dataset, the assertions, the grader prompts, and the pass thresholds are part of your application and belong in the repository beside the code they gate. Keeping them as plain files makes every change reviewable in a pull request, lets you run the same suite on any backend, and means a platform sunset cannot take your test logic with it. A hosted dashboard is useful for triage and history, but it should render what your files define, not be the only place that definition exists.

04What do I lose when a hosted eval platform is retired?

Three things tend to disappear that teams assumed they owned: the stored run history and trend lines, any grader definitions that lived only in the vendor account, and prompts or agent configurations held in the vendor's prompt store or builder. Export the run history as data before the read only date, move grader logic into files you control, and pull prompt and agent definitions into code. What you keep in your repository survives the sunset. What lived only in the platform leaves with it.

Working on this in production?I help teams make agents like this reliable. Thirty minutes, no pitch.
Viral Ruparel

Generative AI consultant and full-stack architect. Ex-IBM, former Director of Engineering at a YC W20 startup. Writes about what breaks when agents meet reality.