Building a Small RAG Pipeline on One Key: Embeddings, Retrieval and Answer Generation Split Across Backends

Why Split a RAG Pipeline Across Backends?

A minimal RAG pipeline has three stages: embed chunks, retrieve relevant chunks, and generate an answer. With a multi-model API aggregator, you can assign each stage to a different model backend without juggling multiple API keys. This lets you optimize for cost, latency, or quality per stage.

Common splits:

• Cheap, fast embeddings for indexing (e.g., a smaller embedding model)
• A dedicated retrieval scoring step (could be the same or a different model)
• A stronger generation model for the final answer

All calls go through one key, so billing and usage tracking stay centralized.

Embedding Stage: Choosing a Backend

The embedding model you choose fixes two critical properties:

• Vector dimension (e.g., 768, 1024, 1536) – must be consistent across all chunks and queries.
• Distance metric (cosine, dot product, Euclidean) – affects how similarity is computed.

On an aggregator, you can pick any supported embedding model. But if you later switch models or mix embeddings from different providers, the vectors will be incompatible. Always store the model name alongside each vector, and re-embed the entire corpus if you change models.

Retrieval Stage: Scoring and Ranking

Retrieval typically uses approximate nearest neighbor (ANN) search over your stored vectors. The scoring assumption is that similar vectors have higher cosine similarity (or lower Euclidean distance). This holds only if:

• All vectors come from the same embedding model and version.
• The same normalization (e.g., L2) is applied consistently.

If you embed queries with model A and documents with model B, their vector spaces are unrelated, and similarity scores become meaningless. Even using the same model but different preprocessing (e.g., truncation length) can degrade results.

You can also use a separate model for re-ranking retrieved chunks. That step is optional and adds latency, but can improve relevance when the initial retrieval is noisy.

Generation Stage: Answer Synthesis

Once you have the top-k chunks, you feed them into a generation model along with the user query. The generation model can be from a different family (e.g., embeddings from one provider, generation from another) because it receives text, not vectors.

Key considerations:

• Context window: Ensure the concatenated chunks fit within the model's limit. Truncate or summarize if needed.
• Instruction format: Different models expect different prompt styles. Test carefully.
• Cost: Generation typically dominates cost. Use a cheaper model for simple queries, or route based on query complexity.

Where Assumptions Break

The biggest risks when splitting backends:

• Vector dimension mismatch: Attempting to compare vectors of different sizes throws errors or silently fails in some vector databases.
• Scoring inconsistency: If your vector DB uses dot product but your embedding model expects cosine, ranking may invert (favoring longer vectors or opposite directions). Normalize vectors at index time if unsure.
• Model updates: Embedding models get retrained. A version bump can shift the vector space, breaking existing indexes. Pin model versions and monitor for changes.
• Tokenization differences: Some models handle whitespace or multilingual text differently, affecting embedding quality.

Practical Checklist

• Pick one embedding model and stick with it for both indexing and querying.
• Store the model name and dimension in your vector DB metadata.
• Normalize vectors if using cosine similarity with dot-product indexes.
• Re-embed the corpus when upgrading the embedding model.
• Test retrieval quality before adding a re-ranker or switching generation models.
• Use the aggregator's unified key to simplify credential management, but log which model handled each stage for debugging.

Conclusion

Splitting a RAG pipeline across backends is feasible with a single API key, but it demands discipline around embedding consistency. Treat the embedding model as a fixed component: changing it invalidates your index. With that constraint, you can mix and match models for retrieval and generation to balance cost and quality.

Note: Users pay official price × 1.3 for API calls; key contributors are credited official price × 1.1 (premium × 1.2) in USDC. This does not change the technical steps above—just the billing model.