Sending Audio Clips Through One Aggregator Key: Which Backends Accept Them and What the Payload Looks Like

Why Audio Input Is Tricky with Aggregators

Sending audio to an LLM through an aggregator seems simple: just add an audio field to your request. But backends differ in support, and some silently ignore audio, returning text-only responses. This guide shows you how to structure requests, identify which backends accept audio, and validate outputs before relying on transcripts.

The OpenAI-Compatible Request Shape for Audio

Most aggregators expose an OpenAI-compatible chat completions endpoint. To send audio, you include an audio object in the message content. The exact schema varies, but a common pattern is:

{
  "model": "backend-model-name",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Transcribe this audio." },
        { "type": "audio", "audio": { "data": "<base64-encoded-audio>", "format": "wav" } }
      ]
    }
  ]
}

Key points:

• Base64 encoding: Audio is typically base64-encoded and placed in the data field.
• Format hint: Specify the audio format (e.g., wav, mp3) to help backends decode correctly.
• Multimodal content: Some backends require audio to be part of a content array alongside text; others accept a separate audio field at the top level. Always check the backend's documentation.

If the backend expects a different structure, you may need to adjust the payload. The aggregator passes your request through, so the schema must match what the backend supports.

Which Backends Accept Audio?

Not all models process audio. Support depends on the underlying model and the backend's API implementation. Here’s a conceptual breakdown:

• Audio-native models: Some models are designed for audio input (e.g., speech-to-text or multimodal models). These generally accept audio in the OpenAI-compatible format.
• Text-only models: Most LLMs are text-only. They will either reject audio requests or ignore the audio content, returning a text response based on the text prompt alone.
• Silent dropping: Some backends may accept the request without error but discard audio content, producing a response that seems plausible but ignores the audio. This is the most dangerous case.

Always verify support by checking the backend's documentation or testing with a known audio clip.

Validating Before Trusting the Transcript

Before you rely on transcripts in production, validate that the backend actually processed the audio. Here’s a practical approach:

1. Send a controlled audio clip with a known transcript (e.g., a short phrase).
2. Inspect the response for a transcript field or text that matches the expected content.
3. Check for audio-specific response fields (e.g., audio or transcript in the response).
4. Compare multiple backends using the same clip to see which ones return consistent results.
5. Implement a fallback: If the response lacks a transcript or seems unrelated, treat it as a failure and retry with a known audio-capable backend.

A simple validation step: include a text prompt like "Transcribe the audio exactly." If the response doesn't contain a transcription, the audio was likely ignored.

Handling Errors and Silent Failures

Audio requests can fail in several ways:

• Explicit errors: Backend returns a 4xx error indicating unsupported media type. This is easy to catch.
• Silent dropping: Backend returns 200 OK but ignores audio. Detect this by validating the response as above.
• Partial support: Some backends accept audio only for certain models. Check the model's capabilities.

Use try/catch blocks and response validation to handle all cases.

Payment and Cost Considerations

With this aggregator, you pay the official price × 1.3 for each request. If you contribute keys, you're credited official price × 1.1 (or × 1.2 for premium) in USDC. Audio requests may consume more tokens or be priced differently by the backend, so monitor usage.

Practical Tips

• Test with short clips to minimize cost during validation.
• Cache transcripts if you process the same audio repeatedly.
• Use explicit format hints to avoid decoding errors.
• Document which backends work for your audio use case.

By following these steps, you can confidently send audio through the aggregator and trust the transcripts you receive.