Multimodal Input Formats on One Endpoint: A Compatibility Matrix for Vision Models

Why Multimodal Input Compatibility Matters

When you call vision models through a single API endpoint—such as an aggregator that provides one key for Claude, GPT, DeepSeek, Qwen, GLM, and Kimi—you might assume the request format is identical. In reality, each model family accepts images in different ways. Some expect base64-encoded strings, others require URLs, and some support binary uploads or special image tokens. Sending the wrong format leads to errors, wasted tokens, and failed batch jobs.

A per-model compatibility matrix helps you:

• Choose the correct encoding before sending a request.
• Avoid trial-and-error when switching models.
• Design batch pipelines that work across multiple models.

Common Image Input Formats

Vision models typically accept one or more of these input styles:

• Base64 data URI: The image is encoded as a base64 string and prefixed with data:image/jpeg;base64, or similar.
• Direct URL: The model fetches the image from a publicly accessible URL.
• Binary file upload: The image is sent as a file part in a multipart request.
• Image token placeholder: Some APIs use a special token (e.g., <image>) that is replaced server-side.
• Inline bytes with MIME type: The raw bytes are passed along with a MIME type field, without base64 encoding.

Not all models support all formats. For example, a model might accept base64 but not URLs, or it might require a specific key name like image_url versus image.

Why Models Differ

Differences arise from:

• API design choices by the model provider (OpenAI-style vs. Anthropic-style vs. custom).
• Security policies: Some providers block external URL fetching to prevent SSRF or data leakage.
• Performance optimizations: Base64 may be preferred for small images, while URLs are better for large files.
• Tokenization: How the image is converted into tokens affects cost and context length.

When using an aggregator, it is crucial to know that the aggregator may normalize some formats but still pass through model-specific requirements. Always check the provider's documentation for each model.

How to Build Your Compatibility Matrix

Follow these steps to create a matrix that fits your workflow.

1. List the Vision Models You Use

Identify the specific model names or families you plan to call (e.g., Claude 3, GPT-4o, Qwen-VL, GLM-4V). Group them by provider if helpful.

2. Enumerate Input Formats to Test

For each model, test the following formats (or those listed in its docs):

• Base64 with data: prefix
• Base64 without prefix (raw string)
• Public URL
• Binary upload (if supported by the endpoint)
• Token placeholder (if applicable)

3. Run Minimal Test Requests

Send a small, fixed image to each model using each format. Record the HTTP status, error messages, and any required fields. Use a script to automate this, but keep the test image tiny to minimize cost.

4. Document Results in a Table

Create a table like this (fill with your findings):

| Model | Base64 (data URI) | Base64 (raw) | URL | Binary | Token |
|------------|-------------------|--------------|-----|--------|-------|
| Model A | ✅ | ❌ | ✅ | ❌ | ❌ |
| Model B | ✅ | ✅ | ❌ | ✅ | ❌ |
| Model C | ❌ | ❌ | ✅ | ❌ | ✅ |

Note any additional requirements, such as specific MIME types or size limits.

5. Handle Aggregator Normalization

Some aggregators accept a unified format and translate it to the model's native format. Test whether your aggregator does this. If it does, your matrix might simplify to a single column: "supported by aggregator." If not, you must format per model.

Practical Tips for Batch Jobs

• Group requests by input format: If you must send different formats, split batches into homogeneous groups to reduce errors.
• Validate images before sending: Check size, format, and accessibility of URLs.
• Use fallbacks: If a model rejects one format, retry with another if supported.
• Monitor token usage: Some formats may consume more tokens. Base64 images are often larger in bytes than a URL, but tokenization depends on the model.
• Cache converted images: If you frequently switch between base64 and URL, cache the converted forms to save time.

Example: Converting an Image for Multiple Models

Suppose you have a JPEG image. You can prepare it in several ways:

• Base64 data URI: data:image/jpeg;base64,/9j/4AAQSk...
• Raw base64: /9j/4AAQSk...
• URL: https://example.com/image.jpg

Store these variants and reference them when constructing requests. For batch jobs, select the variant that matches each model's requirement.

Using the Aggregator's API Key

With a single API key, you can call any supported model. The key works across models, but the request body must still conform to each model's expected format. The aggregator may provide helper libraries or documentation that abstracts some differences—check those first.

Cost Considerations

On this platform, you pay the official price × 1.3 for usage. Key contributors are credited official price × 1.1 (premium × 1.2) in USDC. This applies uniformly across models, so your choice of model and input format should be based on compatibility and performance, not cost differences.

Summary

Building a compatibility matrix for image input formats prevents guesswork and batch failures. Start by testing the formats each vision model accepts, document the results, and design your pipeline to group requests by format. Always refer to the latest model documentation, as formats can change. With a single API key and a clear matrix, you can efficiently route images to the right model.