AI Features

Codapult provides a shared AI application layer built on the Vercel AI SDK. Chat, agents, batch jobs, playground experiments, tools, guardrails, metering, model routing, and retrieval use the same gateway instead of implementing provider calls independently.

Guides

• Streaming Chat — chat UI, model adapters, streaming route, and tool calls.
• RAG and Semantic Search — indexing, embeddings, vector store, and retrieval.
• AI Runtime Architecture and Evals — request flow, extension boundaries, and deterministic runtime contracts.
• Quotas and Memory — conversation persistence, organization request quotas, cost budgets, and rate limits.
• AI Agents and IDEs — tools, agents, and the gateway contract for AI-powered development.

Architecture

src/lib/ai/
├── models.ts         # Client-safe model options (id, label, provider)
├── providers.ts      # getModel() — resolves modelId → LanguageModel
├── embeddings/       # Embedding adapter (OpenAI / Ollama)
├── vector-store/     # Vector store adapter (SQLite / memory)
├── rag.ts            # RAG pipeline (index → chunk → embed → store → retrieve)
├── conversations.ts  # Conversation/message CRUD
├── gateway/          # Model routing, retries, shared pipeline, and providers
├── prompts/          # Versioned prompts and A/B testing
├── tools/            # Tool definitions, built-ins, and agents
├── guardrails/       # Input/output policy evaluation
├── metering/         # Token usage, cost, and budget analytics
└── batch/             # Asynchronous batch processing

Chat Endpoint

POST /api/ai/chat accepts a JSON body with a messages array and an optional model selector. This built-in surface is streaming by design:

{
  "messages": [{ "role": "user", "content": "How do I deploy?" }],
  "modelId": "gpt-5.6-terra"
}

The endpoint follows the standard API route pattern: auth check → rate limiting (30 requests per 60 seconds per user) → org quota check → Zod validation → RAG context injection → streaming response.

For server-to-server use, POST /api/ai/gateway supports both complete and streaming responses. Set requireCompleteResponse: true when an output guardrail must finish before any response is delivered. This forces the complete-response path even if stream: true is also present; it does not create a guardrail, so the request must reference a configured organization-scoped output rule.

Available Models

Models are defined in src/lib/ai/models.ts. To add a model, add its complete definition there, including provider, SDK model ID, capabilities, and pricing. Unknown model IDs are rejected; provider detection by naming convention is not used.

Custom OpenAI-compatible endpoint

For a self-hosted or third-party OpenAI-compatible endpoint, keep the model definition in src/lib/ai/models.ts with provider: 'custom'. Configure only the endpoint and secret through environment variables:

CUSTOM_AI_BASE_URL="https://llm.example.com/v1"
CUSTOM_AI_API_KEY="..."
AI_DEFAULT_MODEL="your-model-id"

CUSTOM_AI_BASE_URL and CUSTOM_AI_API_KEY do not register arbitrary models. The selected model ID must still exist in models.ts and, when configured, in AI_ALLOWED_MODELS.

Configuration

Runtime AI settings live in environment variables and are exposed server-side through env.ai:

AI_DEFAULT_MODEL = 'gpt-5.6-terra';
AI_ALLOWED_MODELS = 'gpt-5.6,gpt-5.6-terra,gpt-5.6-luna,claude-sonnet-5,gemini-3.7-flash';
AI_MAX_RETRIES = '2';
AI_DEFAULT_TEMPERATURE = '0.7';
AI_DEFAULT_MAX_TOKENS = '4096';
AI_DEFAULT_TOP_P = '1';
ENABLE_AI_RAG = 'false';
AI_RAG_MAX_CHUNKS_PER_QUERY = '5';
AI_RAG_MIN_SCORE = '0.4';

The system instruction is a source-controlled constant in src/lib/ai/system-prompt.ts. Keep product behavior there; secrets and deployment controls belong in the environment.

Tool Use

Chat supports function calling via the Vercel AI SDK. Tools are defined in /api/ai/chat/route.ts. The example below is illustrative — replace it with your own domain-specific tools:

import { z } from 'zod';
import { tool } from 'ai';

const chatTools = {
  lookupOrder: tool({
    description: 'Look up an order by ID',
    inputSchema: z.object({ orderId: z.string() }),
    execute: async ({ orderId }) => {
      // Replace with your own business logic
      const order = await db.select().from(orders).where(eq(orders.id, orderId)).limit(1);
      return order[0] ?? { error: 'Not found' };
    },
  }),
};

Multi-step tool invocations are enabled with maxSteps: 3. To add a new tool, define it in chatTools with a Zod parameters schema and an execute function.

Organization Quotas

AI access is tracked per organization. Each plan defines an aiChat request quota with a configured time window in src/lib/org-quotas.ts. The quota is checked before chat, gateway, and batch work through checkOrgQuota(). This request quota is separate from token-cost and budget analytics; configure daily or monthly cost budgets through the AI metering settings.

Chat Memory

Conversation history is persisted in the database via src/lib/ai/conversations.ts:

The only connected Chat UI component (src/components/ai/ChatUI) connects to these endpoints and renders the production chat interface with model selection, conversation switching, streaming responses, and tool-call progress/results. Use ChatUI directly or compose its supporting AI components for a specialized surface. Optional agent, debug, and message-cost props are integration points for callers that already have that metadata—they are not populated automatically by the basic chat route.

RAG Pipeline

The RAG (Retrieval-Augmented Generation) pipeline lets the AI chat reference your domain-specific content — blog posts, help docs, feature requests, or any custom text.

How It Works

1. Index — content is chunked (800 chars, 150 overlap), embedded, and stored in the vector store
2. Retrieve — user queries are embedded and matched against stored vectors by cosine similarity
3. Augment — matching chunks are injected into the system prompt with source citations

Indexing Content

Use the indexDocument function or the admin API:

import { indexDocument } from '@/lib/ai/rag';

await indexDocument({
  sourceType: 'help',
  sourceId: 'getting-started',
  title: 'Getting Started Guide',
  content: markdownContent,
});

For large content, use the rag-index background job:

import { enqueue } from '@/lib/jobs';

await enqueue('rag-index', {
  sourceType: 'blog',
  sourceId: 'post-123',
  title: 'My Blog Post',
  content: markdownContent,
});

Admin Indexing API

POST /api/ai/index is admin-only — requires role: "admin" session. Use it to manage the knowledge base from the admin panel or via scripts. It supports three actions:

Embedding Providers

Embeddings use the adapter pattern, switched via the EMBEDDING_PROVIDER env var:

Ollama enables fully self-hosted embeddings — no external API calls. The default Ollama model is nomic-embed-text.

Vector Store

Vector storage uses the adapter pattern, switched via VECTOR_STORE_PROVIDER (accessed as env.ai.vectorStoreProvider in server code):

Source Types

Indexed content is categorized by source type:

Environment Variables

Removing the Module

The AI core is a removable module that contains the shared gateway, chat, agents, batch processing, playground, tools, guardrails, metering, and RAG. Chat and RAG can be disabled independently with ENABLE_AI_CHAT="false" and ENABLE_AI_RAG="false", but both require ENABLE_AI_CORE="true". RAG is opt-in because indexing incurs embedding cost. Use the setup wizard (npx @codapult/cli setup) to remove the AI bundle, or use the feature flags for reversible runtime changes. See the Modules documentation for manual removal steps.