One Key, Six Models: A Capability Probe Script to Run Right After You Get Your Key

Why probe capabilities before you build

When you have one API key that can call multiple models, it's tempting to pick a favorite and start coding. But models differ in which features they support: streaming, tool calling (function calling), JSON mode, and vision. If you assume a feature works and it doesn't, you'll waste time debugging.

A capability probe is a small script that sends the same tiny set of requests to each model and records what happens. It gives you a factual snapshot of each model's behavior with your key, so you can design around real constraints.

What to probe

Focus on four capabilities that affect architecture:

• Streaming – Does the model return incremental chunks, or only a full response?
• Tool use – Can the model accept a tool definition and return a structured tool call?
• JSON mode – Does the model reliably return valid JSON when asked?
• Vision – Can the model accept an image input (e.g., a small base64-encoded image) and describe it?

These are the features most likely to vary across backends. Other differences (context length, latency, cost) are model-specific but less likely to break your code.

Designing the probe script

Keep the script minimal and repeatable. It should:

1. Define a short list of model identifiers (the exact names your provider uses).
2. For each model, run a small set of test cases – one per capability.
3. Capture the raw response (or error) and a simple pass/fail judgment.
4. Output a table or JSON summary you can save and share.

Use the same prompt for all models to keep the comparison fair. For example, a trivial prompt like "Reply with the word 'ok'" works for streaming; for JSON mode, ask for a specific JSON object; for tools, define a dummy tool; for vision, include a tiny image.

Example probe script (Python)

Below is a conceptual script. It assumes your provider offers an OpenAI-compatible chat completions endpoint. Adjust the base URL, model names, and payloads to match your provider's API.

import json
import base64
import requests

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.your-provider.com/v1/chat/completions"

MODELS = ["model-a", "model-b", "model-c", "model-d", "model-e", "model-f"]

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

# A 1x1 transparent PNG as base64 for the vision test
TINY_IMAGE_B64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="

def post(payload, stream=False):
    resp = requests.post(BASE_URL, headers=HEADERS, json=payload, stream=stream, timeout=30)
    resp.raise_for_status()
    return resp

def test_streaming(model):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": "Say 'ok'."}],
        "stream": True,
    }
    try:
        resp = post(payload, stream=True)
        # Check for at least one SSE data line
        for line in resp.iter_lines():
            if line and line.startswith(b"data: "):
                return {"capability": "streaming", "result": "supported"}
        return {"capability": "streaming", "result": "no stream chunks"}
    except Exception as e:
        return {"capability": "streaming", "result": f"error: {e}"}

def test_tools(model):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": "What's the weather in Paris? Use the tool."}],
        "tools": [{
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }],
        "tool_choice": "auto",
    }
    try:
        resp = post(payload)
        data = resp.json()
        message = data["choices"][0]["message"]
        if message.get("tool_calls"):
            return {"capability": "tools", "result": "supported"}
        return {"capability": "tools", "result": "no tool_calls in response"}
    except Exception as e:
        return {"capability": "tools", "result": f"error: {e}"}

def test_json_mode(model):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": "Return a JSON object with key 'status' and value 'ok'. Only output JSON."}],
        "response_format": {"type": "json_object"},
    }
    try:
        resp = post(payload)
        data = resp.json()
        content = data["choices"][0]["message"]["content"]
        parsed = json.loads(content)
        if parsed.get("status") == "ok":
            return {"capability": "json_mode", "result": "supported"}
        return {"capability": "json_mode", "result": "valid JSON but wrong content"}
    except Exception as e:
        return {"capability": "json_mode", "result": f"error: {e}"}

def test_vision(model):
    payload = {
        "model": model,
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{TINY_IMAGE_B64}"}},
            ],
        }],
    }
    try:
        resp = post(payload)
        data = resp.json()
        content = data["choices"][0]["message"]["content"]
        return {"capability": "vision", "result": "supported", "response": content[:50]}
    except Exception as e:
        return {"capability": "vision", "result": f"error: {e}"}

def main():
    results = {}
    for model in MODELS:
        results[model] = []
        results[model].append(test_streaming(model))
        results[model].append(test_tools(model))
        results[model].append(test_json_mode(model))
        results[model].append(test_vision(model))
    print(json.dumps(results, indent=2))

if __name__ == "__main__":
    main()

This script is intentionally simple. It doesn't handle retries or rate limits, and it treats any error as "not supported." For a quick probe, that's usually enough.

Running the probe

1. Save the script as probe.py and fill in your API key, base URL, and the model names you want to test.
2. Run it: python probe.py.
3. Capture the JSON output. You can redirect it to a file: python probe.py > capabilities.json.

Run the script once and keep the result as a reference. If you later add models or suspect a change, run it again.

Interpreting results

• A "supported" result means the model returned a response consistent with the capability.
• An "error" result could mean the model doesn't support the feature, the request format is wrong, or there was a transient issue. Check the error message.
• Some models might return a valid response but not in the expected format (e.g., JSON mode returns text with extra explanation). That's still useful information.

Remember that a single probe is a snapshot, not a guarantee. Models can behave differently with different prompts, and providers may update backends. Use the probe as a starting point, not a contract.

Adapting the probe to your provider

The script assumes an OpenAI-compatible API, which many aggregators offer. If your provider uses a different schema, adjust the payloads accordingly. The key idea is to send the same minimal requests to each model and record the outcomes.

If your provider charges per token, these probes cost very little because the prompts are tiny. You pay the provider's rate (which may include a markup over official prices, e.g., 1.3x). The cost is negligible.

Next steps

Once you have your capability matrix, you can:

• Choose models that support the features you need.
• Add fallbacks for capabilities that are missing.
• Document the matrix for your team.

This probe takes minutes to run and can save hours of debugging later. It's a small investment that makes multi-model development more predictable.