Estimating Embedding Batch Cost Before You Commit USDC: Token Counting and Backend Choice on One Aggregator Key

Why Estimate Before You Top Up

Embedding jobs scale with corpus size, and cost scales with tokens. A corpus that looks small as files can be large as tokens, especially with code, JSON, or non-English text. Because you pay in USDC up front, a short planning pass prevents over-topping-up or under-funding a job mid-run.

This worksheet is conceptual: it covers the steps, the variables that matter, and how to validate before committing budget. Actual prices, limits, and model availability live on the pricing page of the aggregator and the providers' own documentation.

Step 1: Count Tokens, Not Files or Characters

Embedding billing is per input token. Characters, words, and bytes are unreliable proxies. Plan to tokenize with the tokenizer that matches your target backend, since different model families segment text differently (code and non-Latin scripts diverge the most).

• Sample first. Tokenize a representative slice of the corpus rather than the whole thing, then extrapolate by document type.
• Separate the corpus into strata. Natural language, source code, structured data (JSON, logs), and multilingual text can have very different tokens-per-character ratios.
• Include preprocessing effects. Normalization, whitespace cleanup, and metadata stripping change token counts; decide what you send before you count.
• Watch truncation. If a backend truncates inputs beyond a maximum length, count only the tokens that will actually be embedded, or you will overestimate.

A simple worksheet row per stratum: number of items, average tokens per item after chunking, and total tokens. Sum the rows for the corpus total.

Step 2: Decide Chunk Size Before Pricing

Chunking changes both the token total and the number of requests.

• Smaller chunks mean more items and more request overhead, but each embedding covers less context.
• Larger chunks mean fewer requests and fewer tokens spent on repeated prefixes, but risk diluting meaning in retrieval use cases.
• Overlap between chunks adds tokens. Account for overlap explicitly in the estimate rather than discovering it later.

Compute total tokens as items × average chunk tokens × (1 + overlap factor). Keep the chunk plan fixed while you compare backends so the comparison is apples-to-apples.

Step 3: Compare Backends at the 1.3x Rate

On this aggregator you call many providers with one key. Your cost is the provider's official price multiplied by 1.3, paid in USDC. So the estimate is:

• Look up each candidate backend's official embedding price per token (or per million tokens) on the provider's pricing page.
• Multiply by 1.3 to get your effective per-token cost on the aggregator.
• Multiply by your corpus token total from Steps 1-2.

Do this per backend, because embedding prices differ substantially across model families, and cheap-per-token options are not always interchangeable in quality or dimensionality. If a backend offers tiered pricing by volume, note that tiers are the provider's, not the aggregator's; your 1.3x applies to the official price you would otherwise pay.

Record for each backend: official price, effective price (×1.3), estimated job cost, embedding dimensions, and maximum input length. That table is your decision record.

Step 4: Account for Re-runs and Shrinkage

First-pass estimates assume a clean run. Budget headroom for:

• Failed or retried requests.
• Re-embedding a subset after a chunking or preprocessing change.
• Verification passes where you sample and re-embed documents to check consistency.

A modest buffer, expressed as a percentage of the estimate, is easier to reason about than an arbitrary USDC figure.

Step 5: Pilot Before Committing

Before spending the full budget, run a small pilot batch end to end: tokenize and embed a slice covering every stratum, confirm the actual token counts match your extrapolation, and confirm the backend's output shape (dimensions, normalization) fits your downstream store. Compare measured cost of the pilot against the predicted cost of that slice; if they diverge, fix the model before scaling.

How Contributor Credits Fit In

If you also contribute keys or capacity to the aggregator, credits are issued at the official price × 1.1 (× 1.2 for premium) in USDC. That is separate from what you pay as a caller (× 1.3) and does not change the per-token cost math above. Treat credits as a settlement detail, not as a discount on the embedding estimate.

A Compact Checklist

• Tokenize a representative sample per stratum with the target tokenizer.
• Fix the chunking and overlap plan; compute items × tokens × overlap factor.
• Build a backend table: official price, ×1.3 effective price, dimensions, max input length, estimated cost.
• Add a re-run buffer.
• Run a pilot slice and reconcile predicted vs. measured tokens and cost.
• Top up USDC only after the pilot reconciles; keep the decision table for the next job.

Repeat the worksheet whenever the corpus mix or chunk plan changes, since both move the token total more than price changes do.