Which Sub-Processors Sit Between Your API Key and the Model Vendor?

The Request Path: An Overview

When you call a model through an API aggregator, your request does not go directly to the model vendor. Instead, it passes through a chain of intermediaries. Each tier in this chain may have different levels of access to your data. Understanding this path helps you assess where your prompt content is visible and where only metadata is exposed.

Tier 1: Your Application to the Aggregator

The first hop is from your application to the aggregator's API endpoint. At this stage, the aggregator receives the full prompt, including any system messages, user input, and parameters. This is the point where the aggregator must handle your data according to its privacy policy.

• Visibility: Full prompt content and metadata (API key, model choice, timestamps).
• Key consideration: The aggregator acts as a data processor. It needs to route your request, so it must see the prompt to forward it.

Tier 2: Aggregator's Internal Processing

Inside the aggregator, the request may pass through several sub-components:

• Authentication and authorization: Verifies your API key and checks permissions. Sees metadata only.
• Rate limiting and quota management: Tracks usage. Sees metadata only.
• Request routing: Determines which model vendor to send the request to. May see prompt content if routing decisions depend on content, but typically only model name and parameters.
• Logging and monitoring: May log requests for debugging or analytics. This is a critical point where prompt content could be stored. Check the aggregator's logging policy.

Tier 3: Aggregator to Model Vendor

The aggregator forwards your request to the model vendor's API. At this stage, the model vendor receives the full prompt. The aggregator may add its own authentication and possibly modify the request (e.g., to enforce usage policies).

• Visibility: The model vendor sees the full prompt content and metadata.
• Key consideration: The model vendor's privacy policy governs how they handle your data. The aggregator's policy should disclose which vendors are used.

Tier 4: Model Vendor's Infrastructure

Within the model vendor, the request may traverse internal systems:

• API gateway: Routes the request to the appropriate model. Sees metadata and possibly prompt content.
• Inference servers: Execute the model. These servers process the prompt content to generate a response.
• Logging and abuse monitoring: Vendors often log requests for safety and improvement. Prompt content may be retained for a period.

Tier 5: Response Path Back

The response travels back through the same tiers in reverse. Each tier may log or inspect the response, but typically the response is not modified except for formatting.

Third-Party Sub-Processors

Beyond the core tiers, aggregators may use third-party sub-processors for various functions:

• Cloud hosting providers: Host the aggregator's infrastructure. They may have access to data at rest and in transit.
• CDN and DDoS protection: Handle incoming traffic. They typically see metadata (IP addresses, headers) but not prompt content if TLS is terminated at the aggregator.
• Payment processors: Handle billing. They see payment metadata, not prompt content.
• Analytics services: May receive anonymized usage data. Check if they have access to prompt content.

Each sub-processor introduces another party that could potentially access your data. Reputable aggregators list their sub-processors and their roles.

Metadata vs. Content: What Each Tier Sees

| Tier | Prompt Content | Metadata |
|------|----------------|----------|
| Your app to aggregator | Yes | Yes |
| Aggregator auth/rate limiting | No | Yes |
| Aggregator routing | Sometimes | Yes |
| Aggregator logging | Possibly | Yes |
| Model vendor API | Yes | Yes |
| Model vendor inference | Yes | Yes |
| Model vendor logging | Possibly | Yes |
| Cloud hosting | At rest/in transit | Yes |
| CDN/DDoS | No (if TLS terminates upstream) | Yes |
| Payment processor | No | Yes |
| Analytics | Anonymized only | Yes |

Implications for Privacy and Security

• Trust boundaries: Each tier is a trust boundary. You must trust the aggregator and the model vendor to handle your data appropriately.
• Data minimization: Some aggregators offer features like prompt logging opt-out or zero-retention policies. Understand what is available.
• Encryption: TLS protects data in transit between tiers. Data at rest should be encrypted by the aggregator and vendor.
• Compliance: If you have regulatory requirements (e.g., GDPR, HIPAA), ensure all sub-processors are compliant.

Questions to Ask Your Aggregator

• Which model vendors do you route to?
• Do you log prompt content? For how long?
• What sub-processors do you use, and what is their role?
• Can I opt out of logging?
• How do you handle data deletion requests?

By understanding the request path and the role of each sub-processor, you can make informed decisions about using an aggregator for sensitive workloads.