Build and run customer support AI assistants

Answer customer questions from your own documentation, with passages retrieved by vector search and an answer generated by a model on Azion.

Last updated

View as Markdown Agent setup

A support or product team wants an assistant that answers customers from the company’s own documentation, help center, or product data, instead of from a model’s general knowledge. A model alone answers from what it learned in training, so it cannot know a return policy or a plan limit that only your documents state. This page sets up the whole assistant on Azion: a function splits the documents stored in Object Storage, embeds each passage with a model on AI Inference, and stores the passages in SQL Database for vector search. A second function embeds each question, retrieves the closest passages, and asks a model on AI Inference to answer from them and cite them. The result is measured by answer latency, the share of answers that cite retrieved sources, and answer accuracy on a test set.

This use case does not cover AI features that are not conversational. For those, refer to Add AI features to existing applications.

Prerequisites

  • An application and a workload that serve your domain, with Application Accelerator turned on, which the Run Function behavior requires. To create them, refer to Applications quickstart.
  • SQL Database enabled on the account. The product is in Preview and is not enabled by default, so request access through Technical Support.
  • An Object Storage bucket with workloads_access set to read_only, holding the documents as UTF-8 text or Markdown files. To create the bucket and upload the files, refer to Object Storage quickstart.
  • A personal token with the Edit SQL Database permission, for the database calls. To create one, refer to Personal tokens.
  • The Azion CLI, installed and authorized, to store the environment variables.
  • The values of your own setup. This page uses support-docs for the bucket, returns-policy.md for one document in it, support-kb for the database, /admin/ingest and /api/ask for the two paths, and www.example.com for the domain. Replace each value with yours in every step.

Required products

The assistant needsWhich meansProductDocumented in
The company’s documents in one place the assistant reads fromA bucket that the ingestion function reads with the azion:storage moduleObject StorageObject Storage API
Passages that can be found by meaning, not by keywordA table with a vector column and a vector index, queried with vector_top_kSQL DatabaseVector search
A vector for every passage and every question, and an answer written from the passagesAn embedding model and a chat model called with Azion.AI.runAI InferenceEmbed documents into a vector table with AI Inference
Code that splits, embeds, retrieves, and assembles the promptTwo functions, each run by a rule on its own pathFunctionsFunctions quickstart
Rules that run a function on a pathThe Run Function behavior, which requires Application Accelerator on the applicationApplication AcceleratorHow Functions works

Reference architecture

This page builds the Retrieval-augmented assistant on platform-hosted models: the documents, the vectors, the retrieval, and both models stay on Azion.

The diagram carries two flows that meet at the database. The ingestion flow, at the top, runs when a document changes: it turns a document into passages and vectors, and writes them to the passages table. The request flow runs on every question: the assistant function embeds the question, retrieves passages by vector distance, and asks the chat model to answer from them. Both models run on AI Inference, so neither flow leaves Azion. A failed model call ends inside the function, and the function decides what the customer receives.

Dataflow

  1. An operator sends POST /admin/ingest with a document key, and the ingestion function reads that document from the support-docs bucket in Object Storage.
  2. The function splits the document into passages, embeds all of them in one call to the embedding model on AI Inference, and writes them to the passages table through the Azion API, because the runtime connection to a database is read-only.
  3. A customer sends a question to POST /api/ask. The application’s rule runs the assistant function, which embeds the question with the same embedding model.
  4. The assistant function asks the vector index of passages for the four passages nearest to the question, through a read replica of the database.
  5. The function sends the passages, the earlier turns of the conversation, and the question to the chat model on AI Inference, which answers and cites the passages it used.
  6. The function returns the answer and the list of its sources to the customer. A model call that fails ends inside the function, so the failure flow stays inside Azion.

Components

  • Functions: runs ingestion and retrieval. The support-ingest function splits and embeds documents, and the support-assistant function embeds the question, retrieves passages, assembles the prompt, and calls the chat model. Both models are reached with Azion.AI.run, so the code names no host and holds no model credential.
  • AI Inference: runs the embedding model, Qwen/Qwen3-Embedding-4B, that turns passages and questions into vectors, and the chat model, Qwen/Qwen3-30B-A3B-Instruct-2507-FP8, that writes the answer. Both run on Azion’s infrastructure, which keeps the request flow and its failures inside Azion.
  • SQL Database: stores the passages, their source documents, and their vectors in one table, passages, in the support-kb database. The assistant reads them through a read replica, and ingestion writes them through the Azion API.
  • Vector Search: the Feature of SQL Database that ranks passages by vector distance. The passages_idx vector index answers vector_top_k without comparing the question with every row, so retrieval reads a fixed number of passages however large the table grows.
  • Object Storage: holds the source documents in the support-docs bucket. The ingestion function reads them by key, so the documents stay the single copy that the passages are derived from.
  • application: the Platform Resource that receives the questions and serves the front end. Its rules decide which path runs which function, and they run before the function does.

Other designs for this use case

  • Retrieval-augmented assistant over third-party LLMs: for teams committed to a model provider such as OpenAI or Anthropic. A function on Azion still retrieves the passages from SQL Database, but it sends them with the question to the provider’s API and keeps conversation state between turns, so generation crosses to an external provider and adds provider keys, provider latency, and fallback decisions to the request flow.

Configure the passages table

The passages table holds each passage, the document it came from, and its vector. The vector column declares F32_BLOB(1024) because the functions request 1,024-dimension vectors from Qwen/Qwen3-Embedding-4B, one of the five widths the model returns. The column and the request must state the same number, and a narrower vector stores less per passage. The index uses the cosine metric, and the table carries INTEGER PRIMARY KEY AUTOINCREMENT, because a vector index requires a ROWID or a single-column primary key.

To create the database, send its name to the SQL Database API:

Shell
01curl --request POST \
02 --url https://api.azion.com/v4/workspace/sql/databases \
03 --header 'Accept: application/json' \
04 --header 'Authorization: Token [TOKEN VALUE]' \
05 --header 'Content-Type: application/json' \
06 --data '{"name":"support-kb"}'

The API answers 202 with the new database. Keep its id, which the next call and the ingestion function use:

JSON
01{
02 "state": "pending",
03 "data": {
04 "id": <database-id>,
05 "name": "support-kb",
06 "status": "creating",
07 "active": true,
08 ...
09 }
10}

Provisioning takes about 15 seconds. Send GET /v4/workspace/sql/databases/<database-id> until status reads created. Then create the table and its index:

Shell
01curl --request POST \
02 --url https://api.azion.com/v4/workspace/sql/databases/<database-id>/query \
03 --header 'Accept: application/json' \
04 --header 'Authorization: Token [TOKEN VALUE]' \
05 --header 'Content-Type: application/json' \
06 --data {"statements":[
07 "CREATE TABLE passages (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL, content TEXT NOT NULL, embedding F32_BLOB(1024));",
08 "CREATE INDEX passages_idx ON passages (libsql_vector_idx(embedding, '\''metric=cosine'\''));"
09 ]}

The API answers 200 with "state": "executed" and one entry in data per statement. A statement that fails still answers 200, with error in place of results in its entry, so read both entries before you continue.

The support-kb database holds an empty passages table and the passages_idx vector index. The index also adds a shadow table, passages_idx_shadow, that a table listing shows beside passages.

Configure the ingestion function

The ingestion function turns one document into rows of passages. It reads, splits, and embeds the document as Embed documents into a vector table with AI Inference describes, and writes the rows as Write rows to SQL Database from a function describes, with the assistant’s values:

  • Path: POST /admin/ingest?key=<object-key>, one document per request. The route writes to the database, so it refuses a request without the ingestion secret.
  • Bucket: support-docs, read from DOCS_BUCKET.
  • Passages: runs of paragraphs up to 1,500 characters. A short passage points the answer at one section of a document, and four of them still fit the prompt with room for the conversation. A single paragraph longer than that stays one passage, which the embedding model’s 32k-token context reads whole.
  • Passages per document: at most 99, so the DELETE for the document’s old rows and one INSERT per passage fit one call of 100 statements. A longer document is refused with 413, so split it into two files.
  • Vectors: Qwen/Qwen3-Embedding-4B at 1,024 dimensions, the width of the embedding column.
  • Write credentials: SQL_DATABASE_ID and AZION_TOKEN.

To store the four values the function reads, run these commands with the Azion CLI. A key that contains token or secret is stored as a secret by default:

Shell
01azion create variables --key "AZION_TOKEN" --value "[TOKEN VALUE]"
02azion create variables --key "INGEST_SECRET" --value "<ingest-secret>"
03azion create variables --key "SQL_DATABASE_ID" --value "<database-id>" --secret false
04azion create variables --key "DOCS_BUCKET" --value "support-docs" --secret false

Create the variables before the function: a function that is already running does not read a changed value until it is redeployed.

Create a function named support-ingest with this code:

JavaScript
01import Storage from "azion:storage";
02
03const EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-4B";
04const DIMENSIONS = 1024;
05const MAX_PASSAGE = 1500;
06const MAX_PASSAGES = 99;
07
08function split(text) {
09 const passages = [];
10 let current = "";
11 for (const paragraph of text.split(/\n\s*\n/)) {
12 const p = paragraph.trim();
13 if (!p) continue;
14 if (current && current.length + p.length > MAX_PASSAGE) {
15 passages.push(current);
16 current = "";
17 }
18 current = current ? current + "\n\n" + p : p;
19 }
20 if (current) passages.push(current);
21 return passages;
22}
23
24const quote = (value) => "'" + value.replaceAll("'", "''") + "'";
25
26export default {
27 async fetch(request, env, ctx) {
28 if (request.method !== "POST") {
29 return new Response("Method not allowed", { status: 405 });
30 }
31 if (request.headers.get("Authorization") !== `Bearer ${Azion.env.get("INGEST_SECRET")}`) {
32 return new Response("Unauthorized", { status: 401 });
33 }
34 const key = new URL(request.url).searchParams.get("key");
35 if (!key) {
36 return Response.json({ error: "key is required" }, { status: 400 });
37 }
38
39 const object = await new Storage(Azion.env.get("DOCS_BUCKET")).get(key);
40 const passages = split(new TextDecoder().decode(await object.arrayBuffer()));
41 if (passages.length === 0 || passages.length > MAX_PASSAGES) {
42 return Response.json({ error: `${passages.length} passages; split the document` }, { status: 413 });
43 }
44
45 const embedded = await Azion.AI.run(EMBEDDING_MODEL, {
46 "input": passages,
47 "encoding_format": "float",
48 "dimensions": DIMENSIONS
49 });
50
51 const statements = [`DELETE FROM passages WHERE source = ${quote(key)};`];
52 for (const item of embedded.data) {
53 statements.push(
54 `INSERT INTO passages (source, content, embedding) VALUES (${quote(key)}, ${quote(passages[item.index])}, vector('[${item.embedding.join(",")}]'));`
55 );
56 }
57
58 const result = await fetch(
59 `https://api.azion.com/v4/workspace/sql/databases/${Azion.env.get("SQL_DATABASE_ID")}/query`,
60 {
61 method: "POST",
62 headers: {
63 "Accept": "application/json",
64 "Authorization": `Token ${Azion.env.get("AZION_TOKEN")}`,
65 "Content-Type": "application/json"
66 },
67 body: JSON.stringify({ statements })
68 }
69 );
70 const body = await result.json();
71 const failed = (body.data ?? []).find((entry) => entry.error);
72 if (!result.ok || failed) {
73 return Response.json({ error: failed?.error ?? `SQL API answered ${result.status}` }, { status: 502 });
74 }
75 return Response.json({ key, passages: passages.length });
76 },
77};

Run the function on its path with these values, following Functions quickstart:

  • Function instance: support-ingest, with no Args.
  • Rule: a Request Phase rule named support - ingest, with the criterion ${uri} starts with /admin/ingest and the Run Function behavior selecting the support-ingest instance.

A POST to /admin/ingest with the secret and a document key replaces that document’s passages in passages, and answers with the number it stored. The DELETE makes a second ingestion of the same key replace its rows instead of duplicating them.

Configure the assistant function

The assistant function answers one question per request. It embeds the question with the same model and width as the passages, because a query vector is comparable only with vectors that model produced. It retrieves the four nearest passages: four passages of up to 1,500 characters ground the answer in more than one section, and keep the prompt far below the 64k-token context of the chat model.

Three more decisions are in the code:

  • The query vector is written into the SQL text. The runtime refuses a JavaScript string as a parameter value, so the vector goes inside vector('[...]'), as the vector search reference does. It holds only numbers the embedding model returned.
  • The client carries the conversation. The request body holds history, the earlier user and assistant messages, and the function keeps the last six, which is three turns. The function stores nothing between requests, and a bounded history keeps the prompt from growing with each turn.
  • max_tokens is 800. The function waits while the model generates, so a cap on the answer’s length is also a cap on the time a customer waits.

The function writes one log line per answer, recording whether the answer cites a passage. That line is the source of the citation metric in Measuring results.

To store the database name the function opens, run:

azion create variables --key "SQL_DATABASE_NAME" --value "support-kb" --secret false

Create a function named support-assistant with this code:

JavaScript
01const EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-4B";
02const CHAT_MODEL = "Qwen/Qwen3-30B-A3B-Instruct-2507-FP8";
03const DIMENSIONS = 1024;
04const TOP_K = 4;
05
06const SYSTEM_PROMPT =
07 "You answer customer questions using only the numbered passages in the user message. " +
08 "Cite every passage you use as [n]. If the passages do not contain the answer, " +
09 "say that you do not know and suggest contacting support.";
10
11async function retrieve(question) {
12 const embedded = await Azion.AI.run(EMBEDDING_MODEL, {
13 "input": question,
14 "encoding_format": "float",
15 "dimensions": DIMENSIONS
16 });
17 const vector = embedded.data[0].embedding.join(",");
18
19 const { Database } = Azion.Sql;
20 const connection = await Database.open(Azion.env.get("SQL_DATABASE_NAME"));
21 const rows = await connection.query(
22 `SELECT passages.source, passages.content FROM vector_top_k('passages_idx', vector('[${vector}]'), ${TOP_K}) JOIN passages ON passages.rowid = id;`
23 );
24
25 const passages = [];
26 let row = await rows.next();
27 while (row) {
28 passages.push({ source: row.getString(0), content: row.getString(1) });
29 row = await rows.next();
30 }
31 return passages;
32}
33
34export default {
35 async fetch(request, env, ctx) {
36 if (request.method !== "POST") {
37 return new Response("Method not allowed", { status: 405 });
38 }
39 const { question, history = [] } = await request.json();
40 if (typeof question !== "string" || question.trim() === "") {
41 return Response.json({ error: "question is required" }, { status: 400 });
42 }
43
44 const passages = await retrieve(question);
45 const context = passages
46 .map((p, i) => `[${i + 1}] (${p.source})\n${p.content}`)
47 .join("\n\n");
48
49 const modelResponse = await Azion.AI.run(CHAT_MODEL, {
50 "stream": false,
51 "max_tokens": 800,
52 "messages": [
53 { "role": "system", "content": SYSTEM_PROMPT },
54 ...history.slice(-6),
55 { "role": "user", "content": `Passages:\n${context}\n\nQuestion: ${question}` }
56 ]
57 });
58
59 const answer = modelResponse?.choices?.[0]?.message?.content ?? "";
60 const cited = /\[\d+\]/.test(answer);
61 console.log(JSON.stringify({ event: "answer", cited, passages: passages.length }));
62
63 return Response.json({
64 answer,
65 sources: passages.map((p, i) => ({ n: i + 1, source: p.source }))
66 });
67 },
68};

Run the function on its path with these values, following Functions quickstart:

  • Function instance: support-assistant, with no Args.
  • Rule: a Request Phase rule named support - ask, with the criterion ${uri} is equal /api/ask and the Run Function behavior selecting the support-assistant instance.

A POST to /api/ask with a question answers with the model’s answer, which cites passages as [n], and with the document each cited passage came from. A new rule takes a few minutes to propagate.

Verify the setup

  • A document becomes passages. Ingest the example document:

    Shell
    01curl -X POST 'https://www.example.com/admin/ingest?key=returns-policy.md' \
    02 -H 'Authorization: Bearer <ingest-secret>'

    The function answers with the key and the number of passages it stored, such as {"key":"returns-policy.md","passages":3}. A request without the Authorization header answers 401.

  • The passages carry vectors. Count the rows of the document through the SQL Database API:

    Shell
    01curl --request POST \
    02 --url https://api.azion.com/v4/workspace/sql/databases/<database-id>/query \
    03 --header 'Accept: application/json' \
    04 --header 'Authorization: Token [TOKEN VALUE]' \
    05 --header 'Content-Type: application/json' \
    06 --data '{"statements":["SELECT COUNT(*) FROM passages WHERE source = '\''returns-policy.md'\'' AND embedding IS NOT NULL;"]}'

    The single entry in data carries results, whose one row holds the same number the ingestion returned.

  • An answer comes from the documents and cites them. Ask a question the document answers:

    Shell
    01curl -X POST https://www.example.com/api/ask \
    02 -H 'Content-Type: application/json' \
    03 -d '{"question":"How many days do I have to return an item?"}'

    The response carries answer, with at least one [n] citation, and sources, whose entries name returns-policy.md.

  • A question outside the documents is not answered from training. Ask a question no document covers, such as What is the capital of France?. The answer says that the assistant does not know, and it carries no citation.

When a request answers 404 or the default page of the application, the rule may still be propagating. When it persists after a few minutes, read the function’s log lines in Real-Time Events, under the Functions Console data source.

Measuring results

MetricWhere to read itWhat working looks like
Answer latencyThe Request Time of requests to /api/ask, in the HTTP Requests data source of Real-Time EventsStable as the number of documents grows, because each answer reads four passages whatever the size of the table
Share of answers that cite retrieved sourcesThe "event":"answer" lines the assistant function writes, in the Functions Console data source of Real-Time Events: the lines with "cited":true over all of themClose to every answer. A drop points at a question the documents do not cover, or at passages that no longer match the documents
Answer accuracy on a test setA fixed list of questions with known answers from your documents, sent to /api/ask after every change to the documents, the prompt, or the modelsThe share of correct answers holds or rises from one run to the next

Best practices

  • Embed questions and passages with the same model and width. Vector distance compares two vectors only when one model produced both at one width. Changing the model or dimensions means a new column declaration and a full re-ingestion, so keep both values in one constant shared by the two functions.
  • Re-ingest a document when it changes. The DELETE before the INSERT statements replaces a document’s passages, so the table never answers from a superseded version. A retired document needs its own DELETE FROM passages WHERE source = '<key>';.
  • Check every statement entry as well as the status. The SQL Database API answers 200 even when a statement fails, with error in that statement’s entry. The ingestion function reads every entry, and any script that writes passages should do the same. For the error shapes, refer to SQL Database best practices.
  • Read the model’s text with optional chaining. The function reads modelResponse?.choices?.[0]?.message?.content, so a response without one of those levels yields an empty answer instead of an exception. For the reasoning, refer to AI Inference best practices.

Guides in this use case

Embed documents into a vector table with AI InferenceReads, splits, and embeds each document, the pattern the ingestion function follows.Write rows to SQL Database from a functionWrites the passages through the Azion API and checks every statement entry.