> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ooneex.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Retrieval-Augmented Generation toolkit with vector database integration

The `@ooneex/rag` package is a Retrieval-Augmented Generation toolkit. It turns your documents into searchable knowledge: convert PDFs into clean text chunks, embed them with OpenAI models, store them in a [LanceDB](https://lancedb.com) vector database, and retrieve the most relevant passages with hybrid search.

You define a typed vector database class, open a table, add records, and search — every record carries an `id`, the searchable `text`, and a `metadata` object of your own typed fields.

## Why this package

* **Hybrid retrieval.** Full-text and vector search run together and are merged with RRF reranking, so you get both keyword precision and semantic recall.
* **Typed end to end.** Your `metadata` shape drives the types for records, filters, and selected fields.
* **PDF in, chunks out.** The `Convertor` parses PDFs into heading-aware chunks with page metadata, ready to embed.
* **Local-first storage.** LanceDB stores vectors on disk — no separate database server to run.
* **Composable filters.** Combine field conditions with `AND`, `OR`, and `NOT` to narrow results.
* **Container-friendly.** Register databases with a decorator and resolve them from the DI container.

## The building blocks

| Block               | What it is                                                                               | Page                                       |
| ------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------ |
| **Vector Database** | A typed class describing where data lives, which embedding model to use, and the schema. | [Vector Database](/ai/rag/vector-database) |
| **Vector Table**    | The handle you add records to, look them up, and index.                                  | [Vector Table](/ai/rag/vector-table)       |
| **Convertor**       | Turns PDFs into structured, embeddable chunks.                                           | [Convertor](/ai/rag/convertor)             |
| **Embeddings**      | The OpenAI models that turn `text` into vectors.                                         | [Embeddings](/ai/rag/embeddings)           |
| **Search**          | Hybrid full-text + vector search with RRF reranking.                                     | [Search](/ai/rag/search)                   |
| **Filtering**       | Composable conditions to scope queries.                                                  | [Filtering](/ai/rag/filtering)             |

## Installation

```bash theme={null}
bun add @ooneex/rag
```

Embeddings are generated with OpenAI, so set your API key in the environment:

```bash theme={null}
OPENAI_API_KEY=sk-...
```

## The record shape

Every row in a table has the same three top-level fields:

```typescript theme={null}
{
  id: string;        // your identifier
  text: string;      // the content that gets embedded and full-text indexed
  metadata: { ... }; // your custom, typed fields used for filtering and selection
}
```

You describe `metadata` with a `DataType` and the database carries it through everywhere — added records, search results, filters, and selected columns are all typed against it.

## End-to-end example

```typescript theme={null}
import { AbstractVectorDatabase } from "@ooneex/rag";
import type {
  EmbeddingModelType,
  EmbeddingProviderType,
  FieldValueType,
} from "@ooneex/rag";
import { Utf8 } from "apache-arrow";

type ArticleData = {
  metadata: {
    title: string;
    category: string;
  };
};

class ArticleVectorDatabase extends AbstractVectorDatabase<ArticleData> {
  public getDatabaseUri = (): string => "./data/articles.lance";

  public getEmbeddingModel = (): {
    provider: EmbeddingProviderType;
    model: EmbeddingModelType["model"];
  } => ({ provider: "openai", model: "text-embedding-3-small" });

  public getSchema = (): { [K in keyof ArticleData]: FieldValueType } => ({
    metadata: new Utf8(),
  });
}

// Connect and open a table (created with indexes on first open).
const db = new ArticleVectorDatabase();
await db.connect();
const table = await db.open("articles");

// Add records.
await table.add([
  {
    id: "1",
    text: "An introduction to retrieval-augmented generation systems.",
    metadata: { title: "RAG Intro", category: "AI" },
  },
  {
    id: "2",
    text: "How vector databases store and search embeddings.",
    metadata: { title: "Vector DBs", category: "Database" },
  },
]);

// Search — hybrid full-text + vector, with an optional filter.
const results = await table.search("retrieval augmented generation", {
  limit: 5,
  select: ["title", "category"],
  filter: { field: "category", op: "=", value: "AI" },
});
```

## How retrieval works

When you call `search()`, the table:

1. Runs a **vector** search over the embedded `text` and a **full-text** search over the same column.
2. Merges both result sets with an **RRF reranker** (Reciprocal Rank Fusion).
3. Applies your `filter` and `select`, then returns the top `limit` records, typed against your `metadata`.

This hybrid approach catches results that pure keyword search misses (semantic matches) and results that pure vector search ranks poorly (exact terms, names, codes).

<Note>
  Tables are created with three indexes on first `open()` — a btree index on `id`, a full-text (FTS) index on `text`, and an IVF-PQ vector index on `vector` — so search is fast out of the box. See [Vector Database](/ai/rag/vector-database) for details.
</Note>