Which Input Formats Each Vendor Actually Accepts: Text, Image URLs, Base64 and File Parts

Why Input Formats Matter

When you call an LLM through an aggregator that routes to multiple backends, the same logical request can succeed on one model and silently ignore half your content on another. Vendors do not share a single wire format for multimodal messages: some accept a plain string, others require a content array, and the supported part types differ. This matters when you switch models or build fallbacks, because a request that works on one backend may lose images or files on another without an error.

This guide walks through the content array shapes commonly accepted by major vendors (as of the time of writing) and highlights where parts may silently drop. It is conceptual, not a substitute for official documentation, and avoids vendor-specific claims that may change. Always test with your chosen model and inspect the response for completeness.

The Two Main Shapes: String vs Array

Most APIs accept either:

• A plain string as the message content. This is the simplest form and supports text-only messages. If you include no other parts, this shape works everywhere.
• A content array with typed parts (e.g., {"type":"text","text":"..."}). This enables multimodal inputs. Not all vendors accept arrays, and those that do may only recognise a subset of part types.

If a vendor does not support arrays, sending one may result in an error or be treated as a single string. Some backends might silently convert the array to a string representation, which can confuse the model.

Image URLs

Many vendors accept image parts where the image is specified as a URL. The typical part looks like:

{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}

Support varies:

• Some vendors fetch the URL server-side and include the image in the prompt.
• Others require the URL to be publicly accessible and may time out on slow hosts.
• A few may not support URL-based images at all, only base64.

If the vendor does not support URL images, the part may be ignored silently or cause an error. Silent drops are more common when the part type is unknown but the vendor still tries to process the message as text.

Base64-Encoded Images

An alternative to URLs is embedding the image data directly as base64. The part may look like:

{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}

or a dedicated image part with a data field. Support for base64 is often more consistent than URL images because there is no external dependency. However:

• Some vendors impose size limits on base64 payloads.
• The data URL prefix may need to be exact; malformed headers can cause drops.
• Not all models that accept text and images support base64; some may only accept URLs.

Again, without an explicit error, a dropped base64 part looks like a successful call but with the image missing.

File Parts

Some vendors support file parts for documents (PDF, text, etc.). The part type might be file or document, often with a URL or base64 data. Support for file parts is less universal than images:

• Only a subset of models accept file inputs.
• When accepted, the file may be converted to text or processed natively, depending on the vendor.
• If unsupported, the part is typically ignored, and the model responds as if the file were not there.

Because file handling is resource-intensive, vendors may also have stricter limits on size and number of files.

How to Avoid Silent Drops

To ensure your multimodal requests are interpreted correctly across backends:

• Check the model's capabilities before sending images or files. The aggregator's model list or documentation should indicate which input types are accepted.
• Use a consistent shape that your target models support. If you need portability, prefer base64 images over URLs, as base64 avoids network issues.
• Validate responses when you expect visual understanding. A simple test question about the image content can reveal if the image was processed.
• Fall back gracefully: if a model does not support a part type, either skip it or route to a model that does. The aggregator may not automatically transform parts between vendors.

Testing Strategy

When building with an aggregator that fronts multiple vendors:

1. Create a minimal test message with one text part and one image part (base64 or URL).
2. Send it to each model you plan to use.
3. Inspect the response to see if the model acknowledges the image (e.g., by describing it).
4. If the image is ignored, check the vendor's documentation for the correct part shape.
5. Document which shapes work for each model so you can construct requests accordingly.

This empirical approach is more reliable than assuming a universal format.

Conclusion

Input format support is fragmented across LLM vendors. While text is universally accepted, images and files require care. Silent drops are a real risk when part types are unsupported. By understanding the common shapes, testing with your chosen models, and validating responses, you can build robust multimodal applications that work across backends. Always refer to the official documentation for the most current details, as capabilities evolve.