A Per-Model Smoke Test: One Script That Confirms Every Endpoint and Model Still Answers

Why You Need a Per-Model Smoke Test

APIs can break in subtle ways: a model may be deprecated, an endpoint may move, or a transient error may cause failures. Without regular checks, you might only discover issues when users report them. A smoke test—a minimal request to each endpoint/model combo—acts as an early warning system.

Designing the Script

Keep it simple and fast:

• List all endpoints and models you want to monitor. This could be static or fetched dynamically from an API if available.
• Send a minimal request to each, such as a short prompt with a low token limit. The goal is to verify connectivity and basic response, not to assess quality.
• Check the response for expected structure (e.g., HTTP 200, valid JSON, presence of a completion).
• Record the result (success/failure, latency) for alerting and trend analysis.

Handling Authentication and Costs

Use a dedicated API key with minimal privileges. Since most providers charge per request, keep the test prompts extremely short to minimize cost. If your aggregator supports it, consider using a separate key with spending limits.

Scheduling and Execution

Run the script on a schedule—every few minutes or hourly, depending on your needs. Options include:

• Cron jobs on a server
• Cloud functions with timer triggers
• CI/CD pipelines (e.g., scheduled GitHub Actions)

Ensure the script runs from a reliable environment with network access.

Alerting and Monitoring

When a failure occurs, you need to know. Integrate with your preferred alerting system:

• Send a message to Slack, Discord, or email.
• Log failures to a monitoring service (e.g., Sentry, Datadog) for dashboards and history.
• Avoid alert fatigue: only alert on repeated failures or set thresholds.

Interpreting Results

A single failure could be a transient network issue. Look for patterns:

• Consistent failure for one model: possibly deprecated or moved.
• Failures across all models: likely your network or the aggregator's API is down.
• Increased latency: could indicate load or degradation.

Best Practices

• Keep tests independent: one failure shouldn't stop others.
• Set timeouts: avoid hanging on unresponsive endpoints.
• Version your script: track changes as you add or remove models.
• Review regularly: update the model list as new models are added or old ones retired.

Integrating with Your Aggregator

If you use an aggregator like the one described, you can leverage a single API key to test multiple models. The smoke test helps ensure that the aggregator's routing is working and that each upstream provider is reachable. This is especially useful when you rely on many models from different providers.

Conclusion

A per-model smoke test is a small investment that pays off by catching issues early. By automating checks and alerts, you can maintain a reliable service and focus on building features instead of firefighting.