Bottom line: You cannot confirm model degradation with a single question. Keep the model, parameters, and test set fixed; repeat samples at different times; then compare protocol fields, token usage, dynamic capability probes, streaming, tool calls, and real business outcomes.
Start by saving a known-good baseline, then retest with the same model ID, parameters, and test set. If several critical capabilities degrade repeatedly after rate limits, context truncation, client configuration, and network instability have been ruled out, classify it as suspected capability degradation. If the response model, provider announcements, or routing metadata changes directly, investigate a routing change separately even if quality appears stable. Gateways can rewrite metadata, so these signals still do not prove the underlying model identity.
Open the LLM API gateway checker, you can first generate a sub-report, and then supplement the business question set according to the method in this article.
How do you check for LLM capability degradation?
First fix the model ID, prompts, parameters and business test set, and save one working result as a baseline. Compare model declarations, token usage, dynamic probes, SSE, tool calls and task completion under the same conditions. Record a suspected capability degradation only when essential capabilities keep worsening across multiple runs and time periods after rate limits, context truncation, network issues and client differences have been ruled out. One low score, a change in writing style or a model's self-description cannot prove degradation on its own.
This typeModel quality testingIt is suitable for answering "whether the current API works as declared", but not suitable for proving the identity of the underlying model based on a single result. If you need to compare multiple portals, please use the same model, parameters and question set, and save the request time and masking request ID of each report.
Define capability degradation first
A perceived drop in quality can come from several places: a routing change, gateway-added system instructions, context truncation, incompatible tool calls, filtered request parameters, peak-time throttling, or normal output variance. Black-box tests can detect a mismatch between observed behavior and documented claims, but they cannot identify the internal cause from symptoms alone.
Therefore, it is recommended to split the problem into three levels:
| Hierarchy | Questions that need to be answered | main evidence |
|---|---|---|
| Protocol layer | Does the API return a standard structure as declared? | status code,model、usage, streaming events |
| Capability layer | Whether key tasks are completed stably | Dynamic questions, tool calls, structured output, business question sets |
| Time dimension | Whether continuous drift occurs in the same configuration | Multi-period reporting, latency, error rate, token and result changes |
Repeatable detection steps
1. Fixed request conditions
Record Base URL, model ID,temperature, maximum output length, system prompts and test time. When comparing two API gateways, except for the API Key and address, the remaining conditions should be consistent. Otherwise the difference in results cannot be attributed.
2. Check the structured signal first
Prioritize checking authentication, HTTP status, response model,request ID,usage and Token arithmetic. Structured field exceptions do not necessarily mean that the model will be replaced, but they can quickly expose gateway compatibility, billing transparency or routing declaration issues.
Special attention is required:/models Not opening or returning 401, 403, or 429 may just be permissions and rate limiting strategies, and cannot be used as evidence of fraud alone.
3. Re-verify key capabilities
Cover at least the following three categories:
- Use server-generated dynamic probes to reduce the chance of answers being prepared for a fixed test.
- SSE streaming output, checking event format, delta content, and end tag.
- Tool call, check if the tool name, JSON parameters and random nonce are returned unchanged.
If your business relies on code, long text, retrieval, or structured JSON, you should also include your own desensitized samples. General dynamic questions can only check part of the ability and cannot replace production tasks.
4. Repeat the test at different times
It is recommended to run once at low peak and once at peak, and retest the next day. A failure may come from network, rate limiting or upstream fluctuations; multiple key capabilities continue to decline and can be repeated under the same conditions, which is worth upgrading to a high-risk signal.
What phenomena deserve special investigation?
Triage the symptom before attributing failures to model degradation
| phenomenon | Check first | Next action |
|---|---|---|
| The request returns 401 | Temporary key, account permissions, and authentication headers | Troubleshoot keys and authentication |
| The request returns 429 | Balance, request or token limits, and concurrency | Troubleshoot quotas and rate limits |
| Non-streaming responses work, but SSE disconnects | Stream events, timeouts, and proxy buffering | Troubleshoot 502 errors and stream disconnects |
| The requested model differs from the response model | Provider alias mappings and routing documentation | What the model field can and cannot prove |
| No HTTP errors, but business task results keep deteriorating | Matching context, parameters, test sets, and repeated baseline runs | Retest using the same comparison template |
Do not retry indefinitely to raise the test pass rate. Fix configuration for authentication errors; reduce concurrency and follow the provider's backoff rules for rate limits. Repeated requests can increase costs and compromise comparability.
- Request model and response
modelLong-term inconsistency, and the service provider cannot interpret the alias mapping. usagePersistently missing or apparently inconsistent Token arithmetic.- Usable tool calls, streaming output, or structured results continue to fail.
- The constraint compliance rate of the same business problem set dropped significantly, while the request parameters did not change.
- Systematic differences in model claims, capability results, or output structures between peak and off-peak periods.
These signals indicate that "investigation is needed", but they do not mean that fraud or intelligence has been proven. Gateway upgrades, model version changes, and client differences may also produce similar phenomena.
Don’t just look at the total score
The comprehensive score in the detection tool measures the compatibility of the protocol with key capabilities, not the model intelligence score or the manufacturer certification score. You should first check whether the failed item affects your business, and then check whether the results of multiple rounds are consistent.
For example, a failed tool call may not matter for plain text summarization, but it can block an agent workflow. A self-reported knowledge cutoff and output style are weak signals; use them only for longitudinal comparison, never as standalone proof.
Recommended comparison records to save
time, base_url, requested_model, response_model,
status, latency_ms, input_tokens, output_tokens,
dynamic_probe, sse, tool_call, business_suite, notesDo not log full API keys, private user data, or prompts that have not been sanitized. When an anomaly occurs, rerun the same test set and give the provider the request time and a redacted request ID. That is much easier to investigate than a vague report that the model has become worse.
Next step
Run it once firstStandard Model Testing, read againHow to interpret test reports. If the main exception is that the request model is inconsistent with the response model, continue to checkModel impersonation and route substitution checks.