Sampling Frames From a Video and Sending Them Through One Key: Which Vision Backends Accept How Many Images

Why frame sampling matters for video summaries

When you send video frames to a vision model, you are effectively converting a temporal signal into a set of still images. The model does not watch the video; it reasons over the frames you provide. If you sample too few frames, you miss key changes. If you sample too many, you hit image count or token limits and the backend may silently drop frames, truncate context, or return an incomplete summary.

A reliable video-summary job needs three things:

• A deliberate frame sampling strategy tuned to the video's content and length.
• Awareness of per-request image count limits and image payload size limits.
• Understanding of how each backend accounts for image tokens, so you can budget context correctly.

Sampling frames from a video

Frame sampling is the process of selecting which frames to send. There is no universal correct rate; it depends on how much motion and detail the summary must capture.

Common approaches:

• Fixed interval sampling – extract one frame every N seconds. Simple and predictable, but may over-sample static scenes and under-sample fast action.
• Scene-change detection – extract a frame when the visual content changes significantly. Reduces redundant frames and focuses on transitions.
• Keyframe extraction – use the video's own keyframes (I-frames) as candidates, then subsample if needed.
• Adaptive sampling – combine fixed interval with scene detection, then cap the total number of frames to a budget you define.

For a summary job, start with a frame budget: decide the maximum number of images you will send in one request or across a batch. Then choose a sampling method that fits within that budget. For example, if you can send at most 20 images per request, sample 20 frames evenly across the video's duration, or use scene detection and then select the 20 most representative frames.

Always record the sampling parameters (interval, scene threshold, max frames) alongside the summary so results are reproducible.

Image payload limits: size, count, and format

Every vision backend imposes limits on the images you send. These limits fall into three categories:

1. Per-image size – maximum file size or pixel dimensions (width × height). Exceeding this may cause the request to fail or the image to be rejected.
2. Per-request image count – maximum number of images in a single request. Exceeding this may cause an error, or the backend may silently ignore extra images.
3. Total payload size – maximum combined size of all images and text in the request. This is often tied to the context window.

Because these limits vary by backend, a job that works with one model may fail or truncate with another. When you route through a single API key that fans out to multiple backends, you must normalize your images before sending:

• Resize frames to a reasonable resolution. Many vision models downsample large images internally, so sending very high resolution wastes bandwidth and may hit size limits without improving quality.
• Compress images (e.g., JPEG quality) to reduce payload size while preserving enough detail for the summary.
• Enforce a hard cap on the number of frames per request based on the most restrictive backend you plan to use, or implement per-backend caps.
• Validate the total request size before sending, and split into multiple requests if needed.

Per-backend image token accounting

Vision models do not count images as a fixed number of tokens. Instead, they convert each image into a number of tokens based on its resolution and the backend's internal tiling or patch strategy. This is the most common source of silent truncation: you budget context for text, but images consume far more tokens than expected.

General patterns to understand:

• Token cost scales with image area. A larger image produces more visual tokens. Some backends tile the image into fixed-size patches and count tokens per patch.
• There is often a base cost per image plus additional tokens for resolution.
• Some backends cap the number of tokens per image, so extremely large images do not cost unbounded tokens—but they may be downsampled, losing detail.
• Token accounting differs between backends. The same image may cost different token counts on different models.

When you send multiple frames, the total image token cost is the sum across all frames. If that sum plus your text prompt exceeds the model's context window, the backend may truncate the oldest or newest frames, or return an error. Silent truncation is dangerous because your summary may be based on an incomplete set of frames without any warning.

To avoid this:

• Estimate image token cost per frame for each backend you use. Many providers document the formula (e.g., tokens = base + (width × height) / patch_area). If not documented, measure empirically by sending a single image and inspecting usage metadata.
• Calculate total image tokens = frames × tokens per frame.
• Ensure total image tokens + prompt tokens + expected output tokens ≤ context window. Leave a safety margin.
• If the total exceeds the limit, reduce the number of frames, reduce resolution, or split the video into segments and summarize each segment separately, then combine summaries.

Routing through one key without silent truncation

A single API key that fronts multiple backends simplifies authentication but does not eliminate backend differences. To keep video-summary jobs reliable:

• Profile each backend you might use. Record its max images per request, max image dimensions, and image token formula (or measured token cost).
• Set a conservative global frame cap that satisfies all backends, or implement per-backend caps and fallback logic.
• Validate before sending. Check image count, per-image size, total payload size, and estimated total tokens against the target backend's limits.
• Request usage metadata in the response and log it. Compare estimated vs. actual image tokens to catch mismatches.
• Detect truncation. If the backend returns a finish reason indicating length or if the number of frames processed is less than sent, treat the result as incomplete and retry with fewer frames or a larger-context backend.
• Segment long videos. For videos that would exceed limits even with aggressive sampling, split into chunks, summarize each chunk, and then summarize the summaries.

Practical checklist

Before running a video-summary job through one key:

• Decide the summary granularity and choose a sampling method.
• Resize and compress frames to a consistent, moderate resolution.
• Determine the target backend's image count limit and image token formula.
• Compute total image tokens and confirm it fits the context window with margin.
• Cap frames to the most restrictive limit in your backend pool, or route to a backend that can handle your frame count.
• Log sampling parameters and usage metadata for reproducibility.
• Implement fallback: if a request fails or truncates, reduce frames or switch backend.

By treating frame sampling, payload limits, and token accounting as first-class concerns, you can route video-summary jobs through a single key without silently losing frames or getting incomplete summaries.