Sending Images Through the OpenAI-Compatible Chat Endpoint on One Key: Base64 vs URL Differences Per Backend

Why image input is not uniform across backends

An OpenAI-compatible chat endpoint gives you a single request shape for many vision-capable models. The compatibility layer is a convenience, not a guarantee of identical behavior. When one key fans out to several backends (for example Claude, GPT, DeepSeek, Qwen, GLM, or Kimi families), the same JSON body can take different code paths inside each provider.

The two ways to attach an image are:

• Inline base64 — you embed the encoded bytes directly in the message content.
• Remote URL — you pass a link and let the backend fetch the image.

These are not interchangeable in practice. Size limits, MIME detection, fetch permissions, and failure behavior differ per backend.

How each input form is represented

In the OpenAI chat schema, a user message with an image looks like a content array containing a text part and an image part. The image part typically has a type and either a URL (which may itself be a data: URI carrying base64) or an explicit base64 payload depending on the dialect the backend accepts.

{
  "model": "<model-id>",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Describe this chart." },
        { "type": "image_url", "image_url": { "url": "https://example.com/chart.png" } }
      ]
    }
  ]
}

For inline data, the same field usually carries a data URI:

{ "type": "image_url", "image_url": { "url": "data:image/png;base64,<...>" } }

Some backends also accept a dedicated base64 field. Check which dialect a given model documents before assuming the data URI form is parsed.

Base64 inline: predictable but heavy

Inline base64 removes every network dependency after the request leaves your client. The backend never has to fetch anything, so there is no risk of a firewall, signed-URL expiry, or redirect breaking the call.

Trade-offs to plan for:

• Payload inflation. Base64 increases byte size relative to the raw image (roughly a third larger). A large image therefore consumes more of your request budget and more of the model's context window.
• Request body limits. Reverse proxies and API gateways often cap request body size. A base64 image that is fine on one backend may be rejected before the request is even routed.
• Token cost. Image inputs are billed as tokens. Large images cost more, so resize before encoding.
• Logging exposure. Base64 blobs bloat logs and may contain sensitive content; scrub them if you log raw requests.

Encode at a reasonable resolution and re-encode to a compact format before sending. Prefer sending only what the task needs.

Remote URL: light requests, more failure modes

Passing a URL keeps the request small. The backend (or an intermediate fetcher) retrieves the image. This is convenient when images already live in object storage or a CDN.

What can go wrong:

• Accessibility. The URL must be publicly reachable from the backend's network. Private buckets, localhost, and VPN-only hosts will fail.
• Expiry. Signed URLs can expire between request creation and fetch. Use long enough validity for retries.
• Content type. Servers that return application/octet-stream or omit Content-Type may be mishandled by backends that rely on the header rather than sniffing bytes.
• Redirects and auth. Some backends do not follow redirects or send credentials. A URL that works in a browser may not work in the backend.
• Fetch policy. Some providers restrict which hosts they will fetch from.

MIME sniffing differences

Backends differ in how they decide what an image is:

• Some trust the Content-Type header or the data URI's media type and will reject a mismatch.
• Some sniff the leading bytes (magic numbers) and ignore the declared type, which is more forgiving.
• Some require an explicit media type and fail if it is missing.

Practical rules that work across backends:

• Always set an accurate Content-Type on hosted images (image/png, image/jpeg, image/webp, image/gif as supported).
• Always include the correct media type in data URIs (data:image/png;base64,...).
• Avoid renaming files to a different extension than their real format; a .png that is really a JPEG trips header-trusting backends.
• When an image fails on one backend but works on another, test the MIME path first before assuming the model lacks vision.

Silent drops and degraded responses

A common pitfall is a backend that accepts the request but ignores the image. You get a fluent text answer that never mentions the picture. Causes include:

• The model in that family is text-only, and the image part is discarded rather than rejected.
• The image part uses a field name or type the backend does not parse.
• The remote URL fetch failed and the backend continued with text only.
• The image exceeded a size limit and was dropped instead of erroring.

Because these failures are silent, validation must be on your side. Do not infer success from a 200 status.

A request strategy that survives backend differences

1. Pick the smallest adequate image. Downscale and compress; both input forms benefit.
2. Prefer base64 for small, sensitive, or ephemeral images where you control the bytes fully and want no external fetch.
3. Prefer URLs for large assets already hosted publicly with stable, non-expiring links and correct content types.
4. Send one image per message initially. Multi-image support varies; add more only after confirming per backend.
5. Assert on the response. Include a question whose answer requires the image (a count, a label, a value). If the model cannot answer, treat it as a possible silent drop.
6. Log the image mode and backend. When output quality varies, you can correlate it with inline versus URL input.
7. Have a fallback path. If a backend drops a remote URL, retry the same request with base64 (or vice versa) before switching models.

Cost and billing reality

Images consume tokens, and your spend tracks usage. With one key, you pay the official price multiplied by 1.3 for calls. Contributors who bring usage into the platform are credited at the official price multiplied by 1.1 (or 1.2 for premium routes) back in USDC. That means image-heavy workloads should be sized deliberately: blurry 4K screenshots cost more tokens than a cropped region that still answers the question. Top-ups are made in USDC on Base with no KYC, so keeping a small balance for experimentation is straightforward.

Quick checklist

• Correct media type in data URIs and on hosted images.
• Image small enough for the tightest backend you target.
• Public, non-expiring URL if using remote input.
• One image per message until multi-image support is confirmed.
• A response assertion that proves the image was seen.
• A documented fallback between base64 and URL.

Treat base64 and URL as two transports with different failure modes, not two spellings of the same thing. Verify per backend, assert on output, and keep the payload lean.