Sending Image Input as base64 vs. URL on the Aggregator: Which Backends Accept Which Encoding

Why image encoding matters on an aggregator

When you call a single OpenAI-compatible endpoint that fronts many models, you write one request shape but the aggregator forwards it to whichever backend you selected. Image inputs are one of the places where backends diverge most. Two encodings dominate:

• Base64 data URI — the image bytes are embedded in the JSON request as data:image/png;base64,....
• External URL — the request contains a plain https://... link and the backend fetches the image itself.

Understanding who accepts what, and what each costs you in payload size, saves you from requests that pass locally but fail on a particular model.

How base64 data URIs behave

A data URI carries the full image inside the request body. That has clear consequences:

• Self-contained. No second fetch, no dependency on the backend being able to reach your host. Works with private or temporary images you upload inline.
• Bigger requests. Base64 inflates binary size by roughly a third, and the whole image travels inside JSON, so request bodies grow quickly with resolution.
• Broad compatibility. Most multimodal chat APIs, including OpenAI-style ones, accept image_url content parts whose value is a data URI. This is the safest common denominator across backends.
• Size ceilings. Aggregators and backends impose their own maximum request size. Very large base64 images can hit those limits even when the model itself would accept the image.

How external URLs behave

A URL sends only a short string; the backend downloads the pixels server-side.

• Tiny requests. Payload size stays small regardless of image dimensions, which helps when you send many images or high-resolution ones.
• Fetch dependency. The backend must be able to reach the URL. Private networks, signed links that expire quickly, auth-protected hosts, or hosts that block server-side fetchers will fail.
• Backend variance. URL fetching support is less uniform than data URIs. Some backends fetch reliably, others may reject the URL form, follow redirects differently, or have their own content-type expectations. Treat URL support as something to verify per model rather than assume.
• Caching and reuse. A stable public URL can be fetched repeatedly; whether any caching happens is backend-specific and not something you should rely on for cost control.

Which backends accept which encoding

The short answer: base64 data URIs are the most portable choice, and URL support should be treated as model-dependent.

• Claude, GPT, and other mainstream multimodal models generally accept image content parts with either a data URI or a URL when served through an OpenAI-compatible interface, but the effective support is decided by the backend adapter the aggregator uses, not by the model name alone.
• DeepSeek, Qwen, GLM, Kimi and similar families vary by generation: some are text-only, some accept images. Among those that accept images, the data-URI path is the more consistently supported form.
• Vision capability itself varies. Before debating encoding, confirm the specific model you selected takes image input at all. Sending an image to a text-only model should be expected to fail regardless of encoding.

Because support moves as backends evolve, test the exact model you plan to use with a small image in both forms and record what works. Keep that note next to your model choice in code.

Payload size, latency, and token cost

Encoding choice affects three different budgets, and they are not the same thing:

• Network payload. Base64 makes your upload larger; URLs keep the upload tiny but move the download to the backend. For large images or batch jobs, URLs reduce what you send.
• Latency. A URL adds a server-side fetch step, which can be slower or faster than uploading inline depending on the image and the network path. Base64 removes that fetch but increases transfer time on your side.
• Token accounting. Image tokens are generally derived from the image's dimensions or resolved detail settings, not from whether you sent a URL or a data URI. The encoding changes the transport, not usually the billed image tokens. Confirm per-model behavior rather than assuming, since resizing and detail rules differ.

On this aggregator, billing is a straightforward multiplier: you pay the official price × 1.3 for the tokens the model reports. Contributors who help supply capacity are credited official price × 1.1 (premium × 1.2) in USDC. So the lever for image cost is the image's resolution and detail settings, not the encoding wrapper.

Practical guidance

• Default to base64 data URIs for portability across backends, especially when switching models often.
• Use URLs when images are large, public, and stable, and when you have confirmed the specific model fetches them.
• Downscale before sending. Resolution drives image tokens more than encoding does; smaller images cut both payload and billed tokens.
• Respect size limits. If a base64 request is rejected for size, resizing usually fixes it before switching encodings would.
• Keep a per-model compatibility note in your codebase: model, encoding tested, and result. This is the cheapest way to avoid regressions as backends change.
• Handle failures gracefully. When an image request fails, check encoding support, vision capability, and payload size in that order.

Quick comparison

• Base64 data URI: most compatible, self-contained, larger uploads, subject to request-size limits.
• External URL: small uploads, backend-dependent fetch, may fail on private or expiring links, support varies by backend.
• Token cost: driven mainly by image dimensions and detail settings, largely independent of encoding.

Pick the encoding that the target backend reliably accepts, then optimize the image itself for cost.