Health-Checking Contributed Vendor Keys Before They Enter Rotation on the Aggregator

Why Health-Check Contributed Keys

On a multi-model aggregator, contributed vendor keys expand capacity and model availability. But a faulty key can cause failed requests, latency spikes, or security gaps. Health-checking keys before they enter rotation ensures only reliable keys serve traffic.

Key Validation Checklist

Before any key is added to the rotation pool, verify:

• Authentication: The key must successfully authenticate with the vendor's API endpoint.
• Permissions: The key should have the necessary scopes for the models it will serve. Over-privileged keys are a risk; under-privileged keys cause errors.
• Quota and Rate Limits: Confirm the key has sufficient quota and that rate limits align with expected traffic. Do not assume; test with a small request.
• Model Access: Ensure the key can access the specific models it is intended to serve. Some keys are restricted to certain model families.
• Billing Status: Verify the key's billing account is active and in good standing.

Testing Procedure

Run these tests in a staging environment before production rotation:

1. Basic Connectivity: Send a minimal request (e.g., a short prompt) to each model the key will serve. Expect a successful response.
2. Error Handling: Simulate common errors (invalid model, exceeded quota) to ensure the key returns expected error codes and that the aggregator handles them gracefully.
3. Latency Check: Measure response times over several requests. Compare against the aggregator's latency thresholds for rotation.
4. Concurrency Test: Send parallel requests to confirm the key handles concurrent load without failures.
5. Security Scan: Ensure the key is transmitted securely and stored encrypted. Check for any leakage in logs or error messages.

Integrating into Rotation

Only after passing all checks should a key enter rotation. Implement a phased rollout:

• Start with a small percentage of traffic.
• Monitor error rates, latency, and quota consumption.
• Gradually increase traffic if metrics remain healthy.

Contributor Compensation

Contributors whose keys pass health checks and serve traffic are credited in USDC. The credit is based on the official price of usage: contributors receive official price × 1.1 (or × 1.2 for premium keys). This incentivizes reliable, high-quality keys.

Automating Health Checks

Build automated health checks into your key onboarding pipeline:

• Schedule periodic re-checks to detect quota exhaustion or permission changes.
• Use canary requests to validate keys before each rotation cycle.
• Log results and alert on failures.

Security Considerations

• Never log full API keys; mask them in all outputs.
• Store keys in a secure vault with access controls.
• Rotate keys regularly and revoke compromised keys immediately.
• Conduct health checks in isolated environments to prevent data leakage.

Conclusion

Health-checking contributed keys is essential for a reliable aggregator. By following a structured validation and testing process, you ensure only healthy keys enter rotation, maintain service quality, and fairly compensate contributors.