AI Runtime Architecture and Evals

Runtime boundary

Codapult keeps provider calls behind one server-side gateway. Product features call the gateway or the full pipeline; they do not create provider clients in route handlers or components.

Client UI / server action
        │
        ▼
API route: auth → rate limit → Zod request validation → organization quota
        │
        ▼
AI pipeline: budget → RAG context → input guardrails → tools → gateway
        │
        ▼
Model catalogue → provider adapter → Vercel AI SDK
        │
        ├── response / stream
        ├── output guardrails
        └── usage, cost, latency, and request log

The route owns access control and request shape. The pipeline owns product policy. The gateway owns model selection, retries, fallback, AI SDK calls, usage normalization, and cost calculation.

Agents and batch jobs use the same pipeline as chat. Agents resolve their versioned prompt and organization-scoped tools before entering it; batch resolves its prompt version once and runs each item through the pipeline. API entry points apply the user rate limit and organization AI quota before work is scheduled.

End-to-end request flow

The same runtime boundary applies the important controls in a deliberate order. The exact path for an interactive request is:

request
  → auth / organization scope
  → quota and budget checks
  → model allow-list and routing
  → input guardrails
  → tools and optional RAG context
  → model generation or stream
  → output evaluation
  → usage and cost recording
  → organization-scoped request log

Tool executions also emit compact activity events (ai.tool.executed or ai.tool.failed) with the organization, agent, tool, handler type, and duration. Arguments and results are deliberately excluded from these events; sensitive tools should add their own domain audit event when the business action itself must be reconstructable.

This is a shared execution path, not a promise that every AI feature has identical UX. Chat streams responses, agents can run multi-step tools, and batch jobs process asynchronously; they converge on the gateway and its provider, policy, and metering boundaries.

Runtime guarantees and boundaries

• Prompt versioning: prompts are stored with versions and the selected prompt ID can be attached to a gateway request and its usage log. Agent execution resolves the current configured version before calling the gateway.
• Tool permissions: tools are resolved only when enabled and available to the organization. Sensitive tools still need authorization inside their handler; registering a tool is not a substitute for domain permission checks.
• Tenant isolation: RAG retrieval, guardrail loading, tool lookup, request logs, quotas, and agent lookup accept the active organization scope. Do not derive that scope from arbitrary browser input.
• Fallback semantics: retries happen within a model attempt; the configured fallback chain is considered after a classified error, rate limit, or timeout, and every fallback model must pass the configured allow-list.
• Evaluation and testing: deterministic evals cover request validation, model allow-lists, guardrail behavior, and cost accounting. Provider-backed quality evals remain application-specific and should be versioned with their datasets.
• PII redaction: non-streaming responses can be blocked or redacted before they are returned. Streaming output is evaluated after completion because already-delivered tokens cannot be retracted. The gateway accepts requireCompleteResponse: true to force the complete-response path even if a caller also requests streaming; configure output guardrails on that route when pre-delivery redaction is required.
• Streaming and cancellation: streaming is implemented through the AI SDK and the chat route now propagates the request AbortSignal to the provider call. Batch cancellation still changes the persisted job state; it is not the same as cancelling every already-started batch item.
• Audit trail: request logs include a compact summary of agent tool calls. The current aiRequestLog is still an operational log, not a complete immutable audit trail of every tool side effect; add domain audit events around sensitive handlers when that evidence is required.

Capability coverage

This table is the contract boundary for the current AI foundation. “Application-specific” means the extension point is present, but the buyer must provide the product policy or domain implementation.

Source map

Generate and stream paths

Use runPipeline for a complete non-streaming request. It performs the budget check, loads organization guardrails, optionally adds RAG context, resolves organization tools, calls the gateway, evaluates the output, and records usage.

Use runStreamPipeline for interactive chat. It performs the request-side policy checks before streaming and runs output evaluation in the completion callback for persisted analytics.

Streaming cannot retract tokens already delivered to a client. If output policy must block or redact content before the user sees it, send requireCompleteResponse: true together with the selected output guardrails. The gateway then uses the non-streaming path. A server-owned sensitive route should set this policy itself rather than relying on browser code to choose it. Buffering is an application-specific alternative when a delayed streaming-like UX is required.

{
  "messages": [{ "role": "user", "content": "Summarize this record" }],
  "stream": true,
  "requireCompleteResponse": true,
  "guardrails": ["customer-output-pii"]
}

requireCompleteResponse controls delivery timing; it does not create a guardrail. The referenced organization-scoped output rule must exist and use the desired block or redact action.

Extension rules

• Add a model to src/lib/ai/models.ts before using it in configuration, UI, or pricing.
• Add provider credentials to the server environment schema; never send them to the client.
• Add domain tools through the tool registry with a Zod-compatible parameter schema and an explicit organization scope.
• Keep tenant context in PipelineContext; do not resolve organization ownership from browser-provided data alone.
• Keep prompts versioned when the response is product-critical, and record the prompt ID with usage logs.
• Bound maxSteps, maxTokens, request size, RAG chunks, retries, and budgets.
• Treat model output and tool arguments as untrusted input. Apply validation, authorization, and guardrails before side effects.

Runtime evals

The built-in evals are deterministic contract checks around the provider boundary. They do not call an external model or spend tokens. They cover:

• request schema rejection;
• model catalogue and allow-list behavior;
• input blocking and output PII redaction;
• additive token-cost accounting.

Run them independently from the full suite:

pnpm test:ai-evals

Provider-backed quality evals belong to the product that owns the prompts and acceptance criteria. Keep those datasets versioned, redact user data, pin the model and temperature, and record latency, token usage, cost, and pass/fail reasons. Do not make external model calls part of the default unit-test suite.