# API Reference
Source: https://docs.memvid.com/api-reference/index
Complete reference for all Memvid APIs
Comprehensive documentation for all Memvid APIs across CLI, Python SDK, Node.js SDK, and core Rust APIs.
## Quick Navigation
`https://api.memvid.com`
HTTP REST API for cloud service access.
`pip install memvid-sdk`
Full Python API with type hints and async support.
`npm install @memvid/sdk`
TypeScript-first with full type definitions.
`memvid `
Command-line interface for all operations.
`memvid-core`
Low-level Rust API for open/mutate/search/ask.
***
## Core Operations
### Create / Open
```typescript theme={null}
import { use, create } from '@memvid/sdk';
// Open existing or create if not exists
const memAuto = await use('basic', 'knowledge.mv2', { mode: 'auto' });
// Create new
const memCreate = await use('basic', 'knowledge.mv2', { mode: 'create' });
// Open existing
const memOpen = await use('basic', 'knowledge.mv2', { mode: 'open' });
// Shorthand
const mem2 = await create('knowledge.mv2');
```
```python theme={null}
from memvid_sdk import use, create
# Open existing or create if not exists
mem = use('basic', 'knowledge.mv2', mode='auto')
# Create new (fails if exists)
mem = use('basic', 'knowledge.mv2', mode='create')
# Open existing (fails if not exists)
mem = use('basic', 'knowledge.mv2', mode='open')
# Shorthand for create
mem = create('knowledge.mv2')
```
```bash theme={null}
# Create new file
memvid create knowledge.mv2
# Open is implicit in other commands
memvid find knowledge.mv2 --query "test"
```
### Put (Store)
```typescript theme={null}
// Store text content
const frameId = await mem.put({
title: 'Document Title',
label: 'category',
text: 'The content to store...',
tags: ['tag1', 'tag2'],
metadata: { author: 'John', date: '2024-01-15' },
});
// Store from file
const frameId = await mem.put({
title: 'PDF Document',
label: 'documents',
file: '/path/to/document.pdf',
});
```
```python theme={null}
# Store text content
frame_id = mem.put({
"title": "Document Title",
"label": "category",
"text": "The content to store...",
"tags": ["tag1", "tag2"],
"metadata": {"author": "John", "date": "2024-01-15"}
})
# Store from file
frame_id = mem.put({
"title": "PDF Document",
"label": "documents",
"file": "/path/to/document.pdf"
})
# Store from URL
frame_id = mem.put({
"title": "Web Page",
"label": "web",
"uri": "https://example.com/page"
})
```
```bash theme={null}
# Store text
memvid put knowledge.mv2 \
--title "Document Title" \
--label "category" \
--text "The content to store..."
# Store file
memvid put knowledge.mv2 \
--input document.pdf \
--label "documents"
# Store folder
memvid put knowledge.mv2 \
--input ./docs/ \
--embeddings \
--recursive
```
### Find (Search)
```typescript theme={null}
// Basic search
const results = await mem.find('search query', { k: 10 });
// With options
const results = await mem.find('search query', {
k: 10,
mode: 'auto',
scope: 'label:docs',
snippetChars: 200,
});
// Access results
for (const hit of results.hits) {
console.log(`${hit.title} (score: ${hit.score})`);
console.log(` ${hit.snippet}`);
}
```
```python theme={null}
# Basic search
results = mem.find("search query", k=10)
# With options
results = mem.find(
"search query",
k=10, # Number of results
mode="auto", # auto, lex, or sem
scope="label:docs", # Filter by label
snippet_chars=200 # Snippet length
)
# Access results
for hit in results.hits:
print(f"{hit.title} (score: {hit.score})")
print(f" {hit.snippet}")
```
```bash theme={null}
# Basic search
memvid find knowledge.mv2 --query "search query"
# With options
memvid find knowledge.mv2 \
--query "search query" \
--top-k 10 \
--mode auto \
--json
```
### Ask (Q\&A)
```typescript theme={null}
// Basic question
const answer = await mem.ask('What is the main feature?');
console.log(answer.answer);
// With options
const answer = await mem.ask('What is the main feature?', {
k: 5,
mode: 'auto',
model: 'openai:gpt-4o-mini',
maskPii: true,
});
console.log(`Answer: ${answer.answer}`);
console.log(`Sources: ${answer.sources.map(s => s.title)}`);
```
```python theme={null}
# Basic question
answer = mem.ask("What is the main feature?")
print(answer["answer"])
# With options
answer = mem.ask(
"What is the main feature?",
k=5, # Context documents
mode="auto", # Search mode
model="openai:gpt-4o-mini", # LLM model
mask_pii=True # Mask sensitive data
)
# Access answer details
print(f"Answer: {answer.get('answer')}")
print(f"Sources: {[s.get('title') for s in answer.get('sources', [])]}")
print(f"Context: {answer.get('context')}")
```
```bash theme={null}
# Basic question
memvid ask knowledge.mv2 --question "What is the main feature?"
# With options
memvid ask knowledge.mv2 \
--question "What is the main feature?" \
--top-k 5 \
--mode hybrid \
--json
```
### Timeline
```typescript theme={null}
// Get recent entries
const timeline = await mem.timeline({ limit: 50 });
// With time range
const timeline = await mem.timeline({
limit: 50,
since: 1700000000,
until: 1710000000,
reverse: true,
});
for (const entry of timeline.entries) {
console.log(`[${entry.timestamp}] ${entry.title}`);
}
```
```python theme={null}
# Get recent entries
timeline = mem.timeline(limit=50)
# With time range
timeline = mem.timeline(
limit=50,
since=1700000000, # Unix timestamp
until=1710000000,
reverse=True # Newest first
)
for entry in timeline.entries:
print(f"[{entry.timestamp}] {entry.title}")
```
```bash theme={null}
# Recent entries
memvid timeline knowledge.mv2 --limit 50
# With time range
memvid timeline knowledge.mv2 \
--since 1700000000 \
--until 1710000000 \
--reverse
```
### Stats
```typescript theme={null}
const stats = await mem.stats();
console.log(`Frames: ${stats.frame_count}`);
console.log(`Size: ${stats.size_bytes} bytes`);
```
```python theme={null}
stats = mem.stats()
print(f"Frames: {stats['frame_count']}")
print(f"Size: {stats['size_bytes']} bytes")
print(f"Capacity: {stats['capacity_bytes']} bytes")
```
```bash theme={null}
memvid stats knowledge.mv2 --json
```
### Seal (Close)
```typescript theme={null}
// Always seal when done
await mem.seal();
```
```python theme={null}
# Always seal when done to ensure data is written
mem.seal()
# Or use context manager
with use('basic', 'knowledge.mv2') as mem:
mem.put(...)
# Automatically sealed
```
```bash theme={null}
# Explicit seal (usually not needed)
memvid seal knowledge.mv2
```
***
## Types Reference
### PutInput
```typescript theme={null}
interface PutInput {
title: string; // Document title (required)
label: string; // Category label (required)
text?: string; // Text content
file?: string; // Path to file
uri?: string; // URL to fetch
tags?: string[]; // Searchable tags
labels?: string[]; // Additional labels
metadata?: object; // Custom metadata
searchText?: string; // Override search text
enableEmbedding?: boolean; // Generate vectors
autoTag?: boolean; // Auto-generate tags
extractDates?: boolean; // Extract dates for timeline
}
```
### FindInput
```typescript theme={null}
interface FindInput {
k?: number; // Number of results (default: 10)
mode?: 'auto' | 'lex' | 'sem'; // Search mode
snippetChars?: number; // Snippet length
scope?: string; // Filter expression
cursor?: string; // Pagination cursor
}
```
### AskInput
```typescript theme={null}
interface AskInput {
k?: number; // Context documents
mode?: 'auto' | 'lex' | 'sem'; // Search mode
model?: string; // LLM model
maskPii?: boolean; // Mask sensitive data
contextOnly?: boolean; // Return context without LLM
snippetChars?: number; // Snippet length
scope?: string; // Filter expression
}
```
### SearchResult
```typescript theme={null}
interface SearchResult {
hits: Hit[]; // Matching documents
total: number; // Total matches
cursor?: string; // Next page cursor
}
interface Hit {
frameId: number; // Document ID
title: string; // Document title
label: string; // Category label
text: string; // Full text
snippet: string; // Highlighted excerpt
score: number; // Relevance score
metadata?: object; // Custom metadata
}
```
### AskResult
```typescript theme={null}
interface AskResult {
text: string; // Generated answer
answer: string; // Alias for text
sources: Source[]; // Source documents
context: string; // Aggregated context
confidence?: number; // Confidence score
}
interface Source {
title: string;
snippet: string;
score: number;
}
```
***
## Error Handling
See the [Error Reference](/errors/reference) for complete error documentation.
```python theme={null}
from memvid_sdk import (
MemvidError,
CapacityExceededError,
TicketInvalidError,
LockedError,
)
try:
mem.put(...)
except CapacityExceededError:
print("Storage full, upgrade your plan")
except LockedError:
print("File locked by another process")
except MemvidError as e:
print(f"Error [{e.code}]: {e.message}")
```
***
## Next Steps
Complete Python documentation
Complete Node.js documentation
All CLI commands
Error codes and solutions
# Opening and mutating
Source: https://docs.memvid.com/api-reference/open-and-mutate
Using the Rust API directly
Developers embedding `memvid-core` can open `.mv2` files and mutate them directly, mirroring CLI behavior.
```rust theme={null}
use memvid_core::{Memvid, PutOptions};
let mut mv = Memvid::create("notes.mv2")?;
let opts = PutOptions::builder().track("demo").title("hello").build();
mv.put_bytes_with_options(b"hello", opts)?;
// Persist the WAL, rebuild indexes, and write a new footer snapshot.
mv.commit()?;
```
Key APIs:
* `Memvid::create(path)` – creates a new `.mv2` and takes an exclusive lock
* `Memvid::open(path)` – opens an existing `.mv2` with an exclusive lock (and performs recovery if needed)
* `Memvid::open_read_only(path)` – opens a consistent snapshot with a shared lock
* `put_bytes*` / `put_with_embedding*` – append frames (optionally with pre-computed embeddings)
* `commit()` – makes changes durable and visible to readers
Locking is handled via `FileLock` internally: writers (`create/open`) take an exclusive lock, while readers (`open_read_only`) share a lock for concurrent queries. Mutation APIs call `Memvid::ensure_writable()` and will fail if the handle is read-only.
> **Best practice**: Call `commit()` after batches. Read-only snapshots only see committed state; uncommitted changes remain in the embedded WAL until a commit or the next writer open/recovery.
# REST API
Source: https://docs.memvid.com/api-reference/rest-api
Complete REST API reference for Memvid cloud service
Access Memvid via HTTP REST API. All endpoints require authentication with an API key.
## Base URL
```
https://api.memvid.com
```
## Authentication
All protected endpoints require an API key in one of two ways:
**Option 1: Bearer Token**
```bash theme={null}
Authorization: Bearer mv2_your_api_key_here
```
**Option 2: X-API-Key Header**
```bash theme={null}
X-API-Key: mv2_your_api_key_here
```
**Memory-scoped keys** restrict access to a single memory and its documents/jobs.
***
## Health & Readiness
### `GET /health`
Check API health status.
```bash theme={null}
curl https://api.memvid.com/health
```
```json theme={null}
{
"status": "healthy",
"version": "1.4.9",
"timestamp": "2026-03-11T12:00:00Z"
}
```
### `GET /health/ready`
Check MongoDB and S3 connectivity. Returns 503 if degraded.
```bash theme={null}
curl https://api.memvid.com/health/ready
```
***
## Memories
A memory is a searchable container for your documents.
### `POST /v1/memories` — Create
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Research Papers",
"description": "ML papers from 2024"
}'
```
| Field | Type | Required | Description |
| ------------- | ------ | -------- | ----------------------------- |
| `name` | string | Yes | 1-1000 characters |
| `description` | string | No | Up to 5000 characters |
| `projectId` | string | No | Assign to existing project |
| `projectName` | string | No | Auto-create project if needed |
### `GET /v1/memories` — List All
```bash theme={null}
curl https://api.memvid.com/v1/memories \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
### `GET /v1/memories/:id` — Get One
```bash theme={null}
curl https://api.memvid.com/v1/memories/{MEMORY_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
### `DELETE /v1/memories/:id` — Delete
Deletes the memory, all its documents, and its `.mv2` file from S3.
```bash theme={null}
curl -X DELETE https://api.memvid.com/v1/memories/{MEMORY_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
Returns `204 No Content` on success.
***
## Documents
### `POST /v1/memories/:id/documents` — Add Documents
**Option A: Upload a file**
Supports PDF, DOCX, DOC, XLSX, PPTX, PPT, TXT, CSV, TSV, LOG, JSON, JSONL/NDJSON, HTML, XML, Markdown.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/documents \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-F "file=@./report.pdf"
```
**Option B: JSON text**
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/documents \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"documents": [
{
"title": "Meeting Notes",
"text": "We decided on React for the frontend and Postgres for the database.",
"tags": ["meeting", "architecture"]
}
]
}'
```
**Option C: Ingest from URL**
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/documents \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/report.pdf"}'
```
**Ingestion options:**
Pass these in an `options` object alongside `documents` or `url`:
| Option | Type | Default | Description |
| ---------------- | ------ | ------------------------ | ----------------------------------------------------- |
| `async` | bool | `false` | Force async processing via background worker |
| `deduplicate` | bool | `false` | Skip documents with similar content |
| `enableOcr` | bool | `true` | Auto-OCR scanned/image-only PDF pages using vision AI |
| `ocrMaxPages` | int | `50` | Max pages to OCR per document (max 200) |
| `storeSource` | bool | `false` | Store original file in S3 for later retrieval |
| `embeddingModel` | string | `text-embedding-3-small` | Embedding model (pinned on first ingest) |
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/documents \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-F "file=@./scanned-contract.pdf" \
-F 'options={"enableOcr": true, "ocrMaxPages": 100}'
```
**Response codes:**
| Code | Meaning |
| ----- | ------------------------------------------------------------------------- |
| `200` | Document processed synchronously. Ready to search. |
| `202` | Document routed to async worker. Response includes `jobId` and `pollUrl`. |
**Scanned PDFs:** PDFs containing scanned images are automatically detected and processed with OCR. Small scanned PDFs (5 or fewer image pages) are handled synchronously. Larger scanned PDFs are automatically routed to the background worker and return `202` — poll the `jobId` to track progress.
| Constraint | Value |
| --------------------------- | ------------------- |
| Max file size | 100 MB |
| Async threshold | 2 MB (auto) |
| Scanned PDF async threshold | 5+ image-only pages |
| Max URL download | 100 MB |
### `GET /v1/memories/:id/documents` — List Documents
```bash theme={null}
curl https://api.memvid.com/v1/memories/{MEMORY_ID}/documents \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
### `DELETE /v1/memories/:id/documents/:doc_id` — Delete One
```bash theme={null}
curl -X DELETE \
https://api.memvid.com/v1/memories/{MEMORY_ID}/documents/{DOC_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
Returns `204 No Content`.
***
## Search
### `POST /v1/memories/:id/find` — Hybrid Search
Semantic + keyword hybrid search with reranking.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/find \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "what database are we using",
"topK": 5
}'
```
| Field | Type | Default | Description |
| ------------------- | ------ | -------- | -------------------------------------------------- |
| `query` | string | required | Search query |
| `topK` | int | 10 | Number of results |
| `options.mode` | string | auto | `"hybrid"`, `"semantic"`, `"lexical"`, or `"auto"` |
| `options.limit` | int | 10 | Number of results |
| `options.highlight` | bool | false | Wrap matches in `` tags |
### `POST /v1/memories/:id/search` — Simple Search
Simplified search with sensible defaults.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/search \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "database choice", "limit": 3}'
```
***
## Ask (RAG)
### `POST /v1/memories/:id/ask` — Ask a Question
Retrieval-augmented generation: searches your memory and generates an answer with sources.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/ask \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"question": "What tech stack did we decide on?",
"options": {
"model": "gpt-4o-mini",
"maxContextChunks": 15,
"includeSources": true
}
}'
```
| Field | Type | Default | Description |
| -------------------------- | ------ | --------- | ------------------------- |
| `question` | string | required | Your question |
| `options.model` | string | `"gpt-4"` | LLM model ID |
| `options.maxContextChunks` | int | 10 | Max chunks fed to the LLM |
| `options.includeSources` | bool | false | Return source documents |
### `POST /v1/ask-once` — One-Shot Q\&A (No Memory)
Ask a question on content you provide directly. Nothing is persisted.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/ask-once \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"question": "Summarize this",
"content": "Your raw text here...",
"model": "gpt-4o-mini"
}'
```
***
## Structured Extraction
### `POST /v1/memories/:id/extract` — Extract Structured Data
Pull structured fields from your documents using a schema you define.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/extract \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"sections": [
{
"id": "parties",
"name": "Contract Parties",
"fields": [
{"key": "buyer", "label": "Buyer Name"},
{"key": "seller", "label": "Seller Name"},
{"key": "effective_date", "label": "Effective Date", "data_type": "date"}
]
}
],
"context": "This is a commercial real estate lease agreement"
}'
```
| Field | Type | Required | Description |
| ------------------------------- | ------ | -------- | --------------------------------------- |
| `sections` | array | Yes | At least one section |
| `sections[].id` | string | Yes | Unique section identifier |
| `sections[].name` | string | Yes | Display name |
| `sections[].fields[].key` | string | Yes | Field identifier |
| `sections[].fields[].label` | string | Yes | Used as the search query |
| `sections[].fields[].data_type` | string | No | Type hint (e.g. `"date"`, `"currency"`) |
***
## OCR (Image-to-Text)
### `POST /v1/memories/:id/ocr` — OCR & Auto-Ingest
Extract text from images using vision LLMs, then auto-ingest the result into the memory.
**Option A: Multipart file upload**
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/ocr \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-F "file=@./screenshot.png" \
-F "mode=fast"
```
**Option B: JSON with base64**
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/ocr \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"imageBase64": "/9j/4AAQ...",
"options": {
"mode": "accurate",
"languageHints": ["en", "es"]
}
}'
```
| Limit | Value |
| ------------------- | ------------------------------- |
| Max image file size | 20 MB |
| Max base64 payload | 27 MB |
| Supported formats | PNG, JPEG, WebP, GIF, TIFF, BMP |
***
## Projects
Projects group memories together.
### `POST /v1/projects` — Create
```bash theme={null}
curl -X POST https://api.memvid.com/v1/projects \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Q1 Research"}'
```
### `GET /v1/projects` — List All
```bash theme={null}
curl https://api.memvid.com/v1/projects \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
### `GET /v1/projects/:id` — Get One
```bash theme={null}
curl https://api.memvid.com/v1/projects/{PROJECT_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
### `PUT /v1/projects/:id` — Update
```bash theme={null}
curl -X PUT https://api.memvid.com/v1/projects/{PROJECT_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Q2 Research"}'
```
### `DELETE /v1/projects/:id` — Delete
```bash theme={null}
curl -X DELETE https://api.memvid.com/v1/projects/{PROJECT_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
***
## Account
### `GET /v1/account` — Account Info & Usage
Returns your organization details, plan limits, and current usage.
```bash theme={null}
curl https://api.memvid.com/v1/account \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
***
## Jobs
Documents that exceed the sync processing threshold are handled by a background worker. The add documents endpoint returns `202` with a `jobId` you can poll.
**What triggers async processing:**
* Files larger than 2 MB
* Scanned PDFs with more than 5 image-only pages (OCR required)
* Explicit `options.async: true`
### `GET /v1/jobs/:id` — Check Job Status
```bash theme={null}
curl https://api.memvid.com/v1/jobs/{JOB_ID} \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
```json theme={null}
{
"id": "69b17fa2dacb7d03814fb49b",
"memoryId": "69b03bb1a62c98d88403ceca",
"jobType": "documentingestion",
"status": "completed",
"progress": 100,
"message": "Finalizing...",
"result": {
"documentsAdded": 1,
"chunksCreated": 74,
"documentIds": ["69b180c9b225e959e6b1b0a1"],
"totalBytes": 1318799,
"processingMs": 295779
},
"createdAt": "2026-03-11T14:43:46.602Z",
"startedAt": "2026-03-11T14:43:46.616Z",
"completedAt": "2026-03-11T14:48:42.408Z"
}
```
**Job statuses:** `pending` → `processing` → `completed`, `failed`, or `partial`
| Field | Description |
| ----------- | ----------------------------------------------------------------- |
| `status` | Current job state |
| `progress` | 0–100 percentage |
| `message` | Human-readable progress description |
| `result` | Present when completed — includes document IDs and chunk count |
| `error` | Present when failed — error description |
| `isPartial` | `true` if job completed with partial results (e.g. quota reached) |
Poll every 5–10 seconds. Scanned PDFs typically take 2–8 minutes depending on page count. Text-only large files usually finish in under a minute.
### `GET /v1/jobs` — List Jobs
```bash theme={null}
curl "https://api.memvid.com/v1/jobs?status=processing&limit=10" \
-H "Authorization: Bearer mv2_YOUR_KEY"
```
| Query Parameter | Type | Default | Description |
| --------------- | ------ | ------- | ----------------------------------------------------------------- |
| `status` | string | — | Filter by status (`pending`, `processing`, `completed`, `failed`) |
| `memoryId` | string | — | Filter by memory |
| `limit` | int | 20 | Max results (1–100) |
***
## Corrections
Store priority-boosted corrections that surface first in search results.
### `POST /v1/memories/:id/correct` — Add Single Correction
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/correct \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"statement": "The capital of France is Paris, not London.",
"topics": ["geography", "France"],
"boost": 2.0
}'
```
### `POST /v1/memories/:id/corrections` — Add Multiple Corrections
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/corrections \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"corrections": [
{
"statement": "Water boils at 100°C at sea level.",
"topics": ["science", "physics"]
}
]
}'
```
***
## Endpoints Summary
| Method | Endpoint | Description |
| ------ | ------------------------------------ | --------------------------- |
| GET | `/health` | Health check |
| GET | `/health/ready` | Readiness check |
| GET | `/v1/account` | Account info, usage, limits |
| POST | `/v1/memories` | Create a memory |
| GET | `/v1/memories` | List memories |
| GET | `/v1/memories/:id` | Get memory details |
| DELETE | `/v1/memories/:id` | Delete memory |
| POST | `/v1/memories/:id/documents` | Add documents |
| GET | `/v1/memories/:id/documents` | List documents |
| DELETE | `/v1/memories/:id/documents/:doc_id` | Delete one document |
| POST | `/v1/memories/:id/find` | Hybrid search |
| POST | `/v1/memories/:id/search` | Simple search |
| POST | `/v1/memories/:id/ask` | RAG question answering |
| POST | `/v1/ask-once` | One-shot Q\&A (no memory) |
| POST | `/v1/memories/:id/extract` | Structured data extraction |
| POST | `/v1/memories/:id/ocr` | OCR & auto-ingest |
| POST | `/v1/memories/:id/correct` | Add single correction |
| POST | `/v1/memories/:id/corrections` | Add multiple corrections |
| GET | `/v1/jobs/:id` | Check async job status |
| GET | `/v1/jobs` | List jobs |
| POST | `/v1/projects` | Create project |
| GET | `/v1/projects` | List projects |
| GET | `/v1/projects/:id` | Get project |
| PUT | `/v1/projects/:id` | Update project |
| DELETE | `/v1/projects/:id` | Delete project |
***
## Supported File Types
**Documents:** PDF (including scanned), DOCX, DOC, XLSX, PPTX, PPT, TXT, CSV, TSV, LOG, JSON, JSONL/NDJSON, HTML, XML, Markdown
**Images (OCR endpoint):** PNG, JPEG, WebP, GIF, TIFF, BMP
Scanned PDFs are automatically detected and processed with OCR — no special configuration needed. Legacy formats (`.doc`, `.ppt`) are converted through the same pipeline as their modern counterparts.
***
## Limits
| Limit | Free Plan | Starter Plan |
| -------------------- | --------- | ------------- |
| Storage | 50 MB | 25 GB |
| Memories | 1 | 5 |
| Queries | Unlimited | 250,000/month |
| Max file size | 100 MB | 100 MB |
| Max image size (OCR) | 20 MB | 20 MB |
| Request timeout | 5 minutes | 10 minutes |
| Request body limit | 150 MB | 150 MB |
***
## Next Steps
Use the Python SDK for easier integration
TypeScript-first SDK with full types
Command-line interface
Error codes and solutions
# Search and ask
Source: https://docs.memvid.com/api-reference/search-and-ask
Lexical, vector, and hybrid retrieval from Rust
memvid-core exposes structured request/response types for retrieval:
```rust theme={null}
use memvid_core::{AclContext, AclEnforcementMode, AskMode, AskRequest, Memvid, SearchRequest};
let mut mv = Memvid::open_read_only("notes.mv2")?;
let search = mv.search(SearchRequest {
query: "Stardust".into(),
top_k: 5,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: Some(AclContext {
tenant_id: Some("tenant-123".into()),
subject_id: Some("matt".into()),
roles: vec!["finance".into()],
group_ids: vec![],
}),
acl_enforcement_mode: AclEnforcementMode::Enforce,
})?;
let context_only = mv.ask(
AskRequest {
question: "What powers the deterministic index?".into(),
top_k: 5,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
start: None,
end: None,
#[cfg(feature = "temporal_track")]
temporal: None,
context_only: true,
mode: AskMode::Lex,
as_of_frame: None,
as_of_ts: None,
adaptive: None,
acl_context: Some(AclContext {
tenant_id: Some("tenant-123".into()),
subject_id: Some("matt".into()),
roles: vec!["finance".into()],
group_ids: vec![],
}),
acl_enforcement_mode: AclEnforcementMode::Enforce,
},
None,
)?;
```
* `Memvid::search(SearchRequest)` performs lexical retrieval and returns a `SearchResponse` with the selected `engine`
* `Memvid::ask(AskRequest, Option<&impl VecEmbedder>)` runs retrieval + optional synthesis (set `context_only: true` to only fetch context)
* `Memvid::vec_search_with_embedding(...)` performs pure vector search when you already have a query embedding (and validates dimensions)
`search()` and `ask()` require the `lex` crate feature (enabled by default). For semantic-only or hybrid ranking, pass a `VecEmbedder` implementation (or use `vec_search_with_embedding` with a precomputed query vector).
## Permission-Aware Retrieval (ACL)
Use `acl_context` + `acl_enforcement_mode` to enforce tenant isolation and RBAC at retrieval time.
See [Permission-Aware Retrieval (ACL)](/concepts/permission-aware-retrieval).
> Tip: If you want to plug in an LLM, use `AskResponse.retrieval.context` as the prompt context and keep `context_only: true` in the core call.
# Locking and Verification
Source: https://docs.memvid.com/api-reference/tickets-and-locking
File locking and integrity verification APIs
memvid-core provides APIs for file locking and integrity verification that host applications can use directly.
***
## File Locking
Memvid uses OS-level file locks to manage concurrent access:
```rust theme={null}
use memvid_core::{Memvid, OpenReadOptions};
// Writers get exclusive lock
let mut mv = Memvid::open("notes.mv2")?;
// Readers share the lock
let mv_read = Memvid::open_read_only("notes.mv2")?;
// Optional: allow repair if the file needs recovery (may take an exclusive lock)
let mv_repair = Memvid::open_read_only_with_options(
"notes.mv2",
OpenReadOptions { allow_repair: true },
)?;
```
### Lock Behavior
* **Writers** take an exclusive `FileLock`
* **Readers** share a read lock (multiple concurrent readers allowed)
* Locks are released when the `Memvid` handle is dropped
### Checking Lock Status
CLI commands for inspecting locks:
```bash theme={null}
# Check who holds the lock
memvid who notes.mv2
# Request the writer to release
memvid nudge notes.mv2
```
***
## Verification
Verify file integrity with the core API:
```rust theme={null}
use memvid_core::{Memvid, VerificationStatus};
// Basic verification
let report = Memvid::verify("notes.mv2", false)?;
// Deep verification (more thorough)
let report = Memvid::verify("notes.mv2", true)?;
// Check results
if report.overall_status == VerificationStatus::Passed {
println!("File is valid");
} else {
for check in &report.checks {
println!("{}: {:?}", check.name, check.status);
}
}
```
### Verification Checks
The verification report includes:
| Check | Description |
| ----------------------- | ----------------------------------------------- |
| `TimeIndexRead` | Time index readable (or skipped if disabled) |
| `TimeIndexEntryCount` | Track entry count matches manifest |
| `TimeIndexSortOrder` | Deep mode: timestamps sorted |
| `LexIndexDecode` | Lexical index readable (or skipped if disabled) |
| `VecIndexDecode` | Vector index readable (or skipped if disabled) |
| `WalPendingRecords` | No uncommitted WAL records |
| `FrameCountConsistency` | Stats match TOC frame count |
***
## Doctor (Repair)
Repair corrupted or damaged files:
```rust theme={null}
use memvid_core::{Memvid, DoctorOptions};
// Rebuild time index
Memvid::doctor("notes.mv2", DoctorOptions {
rebuild_time_index: true,
..Default::default()
})?;
// Rebuild lexical index
Memvid::doctor("notes.mv2", DoctorOptions {
rebuild_lex_index: true,
..Default::default()
})?;
// Rebuild vector index
Memvid::doctor("notes.mv2", DoctorOptions {
rebuild_vec_index: true,
..Default::default()
})?;
// Vacuum deleted frames
Memvid::doctor("notes.mv2", DoctorOptions {
vacuum: true,
..Default::default()
})?;
// Full repair
Memvid::doctor("notes.mv2", DoctorOptions {
rebuild_time_index: true,
rebuild_lex_index: true,
rebuild_vec_index: true,
vacuum: true,
..Default::default()
})?;
```
### Doctor Report
The doctor command returns a report with:
* **Findings**: Issues detected (INFO, WARN, ERROR severity)
* **Planned Actions**: Repairs that will be performed
* **Status**: Result of each repair action
***
## Single-File Verification
Memvid enforces the “single-file guarantee” at open/create time. If it detects sidecars (e.g. `notes.mv2-wal`), it fails with `MemvidError::AuxiliaryFileDetected`.
```rust theme={null}
use memvid_core::{Memvid, MemvidError};
match Memvid::open("notes.mv2") {
Ok(_mv) => println!("opened successfully"),
Err(MemvidError::AuxiliaryFileDetected { path }) => {
eprintln!("auxiliary file detected: {}", path.display());
}
Err(err) => eprintln!("open failed: {err}"),
}
```
For an explicit directory scan, use the CLI command `memvid verify-single-file `.
***
## CLI Commands
The CLI exposes these APIs:
```bash theme={null}
# Verify integrity
memvid verify notes.mv2 --deep
# Repair issues
memvid doctor notes.mv2 --rebuild-time-index --rebuild-lex-index
# Check single-file compliance
memvid verify-single-file notes.mv2
# Inspect locks
memvid who notes.mv2
memvid nudge notes.mv2
```
***
## Error Handling
The core API exposes structured errors via `MemvidError`:
```rust theme={null}
use memvid_core::{Memvid, MemvidError};
match Memvid::open("notes.mv2") {
Ok(mv) => { /* success */ }
Err(MemvidError::Locked(locked)) => {
let pid = locked.owner.as_ref().and_then(|owner| owner.pid);
println!("File locked (pid={pid:?}): {}", locked.message);
}
Err(e) => { /* other error */ }
}
```
***
## Best Practices
### Concurrent Access
1. **Use read-only mode** for queries to allow concurrent readers
2. **Keep write sessions short** to minimize lock contention
3. **Handle LockedError** gracefully with retry logic
### Integrity Maintenance
1. **Verify periodically** (weekly or after large ingestions)
2. **Run doctor** after crashes or power failures
3. **Vacuum** after bulk deletions to reclaim space
### Recovery Workflow
```bash theme={null}
# 1. Verify the file
memvid verify notes.mv2 --deep
# 2. If issues found, preview repairs
memvid doctor notes.mv2 --plan-only
# 3. Apply repairs
memvid doctor notes.mv2 --rebuild-time-index --rebuild-lex-index
# 4. Verify again
memvid verify notes.mv2 --deep
```
# Architecture Overview
Source: https://docs.memvid.com/architecture/overview
How Memvid works - from file format to search pipeline
Memvid is designed around a simple but powerful principle: **everything in one file**. This page explains the architecture that makes this possible.
## Core Design Principles
### 1. Single-File Guarantee
Every `.mv2` file is completely self-contained:
* **No sidecars** - Never creates `.wal`, `.shm`, `.lock`, or journal files
* **Fully portable** - Copy, move, or share the file freely
* **No database** - No external services required
```mermaid theme={null}
graph LR
subgraph MV2File[MV2 File]
Header["Header (4 KB)"]
WAL["Embedded WAL"]
Data["Frame Payloads"]
Lex["Lexical Index (BM25)"]
Vec["Vector Index (optional)"]
Time["Time Index"]
TOC["Table of Contents"]
end
```
### 2. Crash Safety
The embedded Write-Ahead Log (WAL) ensures data integrity:
* Writes go to WAL first, then to permanent storage
* Automatic recovery on file open after crashes
* Recovery completes in under 250ms even for large files
### 3. Determinism
Same inputs produce identical bytes on the same platform:
* Reproducible builds for testing and QA
* Verifiable file integrity with checksums
* Predictable behavior across runs
### 4. Performance
Optimized for fast search and retrieval:
* **Search latency**: \~5ms for 50K documents
* **Cold start**: under 200ms
* **WAL append**: under 0.1ms per write
## File Layout
The `.mv2` file format has a well-defined structure:
```mermaid theme={null}
flowchart TB
subgraph MV2[MV2 File Structure]
direction TB
H["Header (4 KB)"]
W["Embedded WAL (1-64 MB)"]
S["Segments / Frames"]
I["Index Segments"]
T["Table of Contents"]
F["Footer (56 bytes)"]
end
H --> W --> S --> I --> T --> F
```
### Header
The 4 KB header contains:
| Field | Description |
| ------------------- | ---------------------------- |
| Magic | `MV2` identifier |
| Version | File format version |
| WAL Offset | Start of embedded WAL region |
| WAL Size | Size of WAL ring buffer |
| Checkpoint Position | Last committed WAL position |
| TOC Checksum | BLAKE3 hash for integrity |
### Embedded WAL
The WAL is sized based on total file capacity:
| File Size | WAL Size |
| ------------- | -------- |
| Under 100 MB | 1 MB |
| Under 1 GB | 4 MB |
| Under 10 GB | 16 MB |
| 10 GB or more | 64 MB |
**Checkpoint triggers:**
* WAL reaches 75% capacity
* User calls `seal()`
* Every 1,000 transactions
* Clean shutdown
### Frames
Frames are the fundamental unit of storage. Each frame contains:
* **Payload** - The actual content (text, binary, media)
* **Metadata** - Title, URI, timestamps, tags, labels
* **Checksum** - BLAKE3 hash for verification
* **Encoding** - Plain or Zstd compressed
## Search Architecture
Memvid supports three search modes:
### Lexical Search (BM25)
Fast keyword search using BM25 ranking:
* Full-text search with term frequency scoring
* Date range filters: `date:[2024-01-01 TO 2024-12-31]`
* Tokenization and stemming
### Vector Search
Semantic similarity search using embeddings:
* Fast approximate nearest neighbor search
* Optional Product Quantization (PQ) for 16x compression
* Configurable embedding models
### Hybrid Search
Combines both approaches:
1. Run lexical search for keyword matches
2. Run vector search for semantic similarity
3. Merge and rerank results
4. Return top-k hits
```mermaid theme={null}
graph TD
Query["Search Query"] --> Lex["Lexical Search"]
Query --> Vec["Vector Search"]
Lex --> Merge["Merge & Rerank"]
Vec --> Merge
Merge --> Results["Top-K Results"]
```
## Developer Walkthrough
Here's how to work with Memvid in practice:
### Using the CLI
```bash theme={null}
# Create a new memory
memvid create notes.mv2
# Add documents
memvid put notes.mv2 --input ./docs/ --vector-compression
# Search
memvid find notes.mv2 --query "machine learning" --mode auto
# Ask questions
memvid ask notes.mv2 --question "What are the key points?"
# View timeline
memvid timeline notes.mv2 --limit 10
# Check health
memvid doctor notes.mv2 --plan-only
```
### Using the Python SDK
```python theme={null}
from memvid_sdk import use
# Open or create
mem = use('basic', 'notes.mv2')
# Add content
mem.put(text="Introduction to neural networks...", title="NN Intro")
# Batch add (100-200x faster)
mem.put_many([
{'text': 'Chapter 1...', 'title': 'Ch 1'},
{'text': 'Chapter 2...', 'title': 'Ch 2'},
])
# Search
results = mem.find('neural networks', k=5)
# Ask with LLM
answer = mem.ask('What is a neural network?', model='openai:gpt-4o')
# Close properly
mem.seal()
```
### Using the Node.js SDK
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open or create
const mem = await use('basic', 'notes.mv2');
// Add content
await mem.put({ text: 'Introduction to neural networks...', title: 'NN Intro', label: 'intro' });
// Search
const results = await mem.find('neural networks', { k: 5 });
// Ask with LLM
const answer = await mem.ask('What is a neural network?', {
model: 'openai:gpt-4o',
modelApiKey: process.env.OPENAI_API_KEY
});
// Close properly
await mem.seal();
```
## Verification and Repair
Memvid includes built-in tools for file health:
### Verify
Check file integrity without modification:
```bash theme={null}
# Quick verification
memvid verify notes.mv2
# Deep verification (slower, more thorough)
memvid verify notes.mv2 --deep
```
### Doctor
Diagnose and repair issues:
```bash theme={null}
# Preview what would be fixed
memvid doctor notes.mv2 --plan-only
# Rebuild corrupted time index
memvid doctor notes.mv2 --rebuild-time-index
# Rebuild lexical index
memvid doctor notes.mv2 --rebuild-lex-index
# Compact deleted frames
memvid doctor notes.mv2 --vacuum
```
### Single-File Check
Ensure no auxiliary files were created:
```bash theme={null}
memvid verify-single-file notes.mv2
```
## Checksums and Integrity
Defense in depth with cascading checksums:
| Level | What's Checked |
| -------------- | -------------------------- |
| Header | TOC checksum (BLAKE3) |
| WAL Records | Per-record checksum |
| Index Segments | Per-segment checksum |
| Frames | Per-frame payload checksum |
## Next Steps
* [File Format Details](/file-format/layout) - Deep dive into the MV2 structure
* [CLI Commands](/cli/create-and-put) - Complete CLI reference
* [Python SDK](/python-sdk/overview) - Python bindings guide
* [Node.js SDK](/node-sdk/overview) - Node.js bindings guide
# Advanced CLI Commands
Source: https://docs.memvid.com/cli/advanced-commands
Advanced commands for enrichment, models, Logic-Mesh traversal, and auditing
Advanced CLI commands for power users including memory enrichment, model management, entity graph traversal, and document auditing.
***
## Enrichment
The `enrich` command extracts structured memory cards (Subject-Predicate-Object triplets) from frames using various extraction engines.
```
Text: "Alice works at Anthropic as a Senior Engineer"
Extracted Facts:
Alice → employer → Anthropic
Alice → role → Senior Engineer
```
### Synopsis
```bash theme={null}
memvid enrich [OPTIONS]
```
### Options
| Option | Description | Default |
| ------------------- | ------------------------------ | -------- |
| `--engine ` | Extraction engine | `rules` |
| `--incremental` | Only process unenriched frames | `true` |
| `--force` | Re-enrich all frames | Disabled |
| `--json` | Output results as JSON | Disabled |
| `--verbose` | Show extracted memory cards | Disabled |
### Available Engines
| Engine | Description | Speed | Accuracy | Requires |
| --------- | ------------------------ | ------ | -------- | ----------------- |
| `rules` | Pattern-based extraction | Fast | Good | Nothing (offline) |
| `candle` | Local LLM (Phi-3.5) | Medium | Better | Downloaded model |
| `openai` | OpenAI GPT-4o-mini | Slow | Best | API key |
| `claude` | Anthropic Claude | Slow | Best | API key |
| `gemini` | Google Gemini | Slow | Best | API key |
| `mistral` | Mistral AI | Slow | Better | API key |
| `groq` | Groq (fast inference) | Fast | Better | API key |
### Examples
```bash theme={null}
# Fast, offline enrichment using rules
memvid enrich project.mv2 --engine rules
# See what was extracted
memvid enrich project.mv2 --engine rules --verbose
# Using OpenAI (most accurate)
OPENAI_API_KEY=sk-xxx memvid enrich project.mv2 --engine openai
# Using local LLM (no API needed)
memvid enrich project.mv2 --engine candle
# Using Claude
ANTHROPIC_API_KEY=sk-ant-xxx memvid enrich project.mv2 --engine claude
# Re-process everything
memvid enrich project.mv2 --engine rules --force
```
### Response
```
Enrichment complete for project.mv2
Engine: rules v1.2.0
Frames processed: 45
Cards extracted: 127
- Entities: 23
- Facts: 89
- Events: 15
New cards: 127
Total cards: 127
Total entities: 23
```
### JSON Output
```json theme={null}
{
"engine": "rules",
"version": "1.2.0",
"frames_processed": 45,
"cards_extracted": 127,
"total_cards": 127,
"total_entities": 23,
"new_cards": 127,
"cards_by_kind": {
"fact": 89,
"event": 15,
"entity": 23
}
}
```
***
## Memories
The `memories` command displays extracted memory cards from enriched frames.
### Synopsis
```bash theme={null}
memvid memories [OPTIONS]
```
### Options
| Option | Description | Default |
| -------------------- | --------------------- | -------- |
| `--entity ` | Filter by entity name | All |
| `--kind ` | Filter by card kind | All |
| `--limit ` | Max results | 50 |
| `--offset ` | Pagination offset | 0 |
| `--sort ` | Sort by field | None |
| `--as-of-frame ` | Time-travel view | Current |
| `--json` | Output as JSON | Disabled |
### Examples
```bash theme={null}
# View all memory cards
memvid memories project.mv2
# Filter by entity
memvid memories project.mv2 --entity "Alice"
# Filter by kind
memvid memories project.mv2 --kind fact
# Paginate
memvid memories project.mv2 --limit 20 --offset 40
# JSON output
memvid memories project.mv2 --json
```
### Response
```
Memory Cards in project.mv2 (showing 50 of 127)
Entity: Alice
employer: Anthropic (fact, frame #123)
role: Senior Engineer (fact, frame #123)
location: San Francisco (fact, frame #145)
joined: 2023-06 (event, frame #123)
Entity: Bob
employer: OpenAI (fact, frame #156)
role: Research Scientist (fact, frame #156)
Entity: Project Alpha
status: active (fact, frame #189)
budget: $500,000 (fact, frame #201)
lead: Alice (fact, frame #189)
```
### JSON Output
```json theme={null}
{
"count": 127,
"cards": [
{
"id": "card_001",
"entity": "Alice",
"slot": "employer",
"value": "Anthropic",
"kind": "fact",
"polarity": "positive",
"confidence": 0.95,
"source_frame_id": 123,
"source_uri": "file:///meeting.txt",
"engine": "rules",
"engine_version": "1.2.0"
}
]
}
```
***
## State
Query current entity state with O(1) lookup. This is the fastest way to get an entity's current attributes.
### Synopsis
```bash theme={null}
memvid state [OPTIONS]
```
### Arguments
| Argument | Description |
| -------- | --------------------- |
| `FILE` | Path to the .mv2 file |
| `ENTITY` | Entity name to query |
### Options
| Option | Description |
| -------------------- | ------------------- |
| `--predicate ` | Filter by predicate |
| `--as-of-frame ` | Time-travel view |
| `--json` | JSON output |
### Examples
```bash theme={null}
# Get Alice's current state
memvid state project.mv2 "Alice"
# Get specific predicate
memvid state project.mv2 "Alice" --predicate employer
# Time-travel: Alice's state at frame 100
memvid state project.mv2 "Alice" --as-of-frame 100
```
### Response
```
Entity: Alice
Current State:
employer: Anthropic
Kind: fact | Source: frame #145 | Engine: rules
role: Senior Engineer
Kind: fact | Source: frame #145 | Engine: rules
location: San Francisco
Kind: fact | Source: frame #156 | Engine: openai
joined: 2023-06
Kind: event | Source: frame #145 | Engine: rules
Last updated: frame #156 (2024-01-20)
```
### JSON Output
```json theme={null}
{
"entity": "Alice",
"found": true,
"slots": {
"employer": {
"value": "Anthropic",
"kind": "fact",
"polarity": "positive",
"source_frame_id": 145,
"document_date": "2024-01-15",
"engine": "rules"
},
"role": {
"value": "Senior Engineer",
"kind": "fact",
"source_frame_id": 145,
"engine": "rules"
}
}
}
```
***
## Facts
Audit fact changes with provenance and filtering.
### Synopsis
```bash theme={null}
memvid facts [OPTIONS]
```
### Options
| Option | Description |
| -------------------- | ------------------------- |
| `--entity ` | Filter by entity |
| `--predicate ` | Filter by predicate |
| `--object ` | Filter by object value |
| `--added` | Show only additions |
| `--removed` | Show only deletions |
| `--limit ` | Max results (default: 50) |
| `--json` | JSON output |
### Examples
```bash theme={null}
# View all fact changes
memvid facts project.mv2
# Changes for Alice
memvid facts project.mv2 --entity "Alice"
# Only employer changes
memvid facts project.mv2 --predicate employer
# Only additions
memvid facts project.mv2 --added
```
### Response
```
Fact Audit for project.mv2
[+] Alice → employer → Anthropic
Frame: #145 | Date: 2024-01-15 | Engine: rules
[+] Alice → role → Senior Engineer
Frame: #145 | Date: 2024-01-15 | Engine: rules
[-] Bob → employer → Google
Frame: #156 | Date: 2024-01-18 | Engine: rules
[+] Bob → employer → OpenAI
Frame: #156 | Date: 2024-01-18 | Engine: rules
```
***
## Export
Export facts to various formats.
### Synopsis
```bash theme={null}
memvid export -o [OPTIONS]
```
### Options
| Option | Description | Default |
| --------------------------- | --------------------------------- | ---------- |
| `-o `, `--out ` | Output file path | Required |
| `--format ` | Format: `ntriples`, `json`, `csv` | `ntriples` |
| `--entity ` | Filter by entity | All |
| `--predicate ` | Filter by predicate | All |
### Examples
```bash theme={null}
# Export as N-Triples (RDF)
memvid export project.mv2 -o facts.nt --format ntriples
# Export as JSON
memvid export project.mv2 -o facts.json --format json
# Export as CSV
memvid export project.mv2 -o facts.csv --format csv
# Export only Alice's facts
memvid export project.mv2 -o alice.json --format json --entity "Alice"
```
### Output Formats
**N-Triples (RDF):**
```
"Anthropic" .
"Senior Engineer" .
"OpenAI" .
```
**JSON:**
```json theme={null}
[
{
"subject": "Alice",
"predicate": "employer",
"object": "Anthropic",
"confidence": 0.95,
"source_frame_id": 145
}
]
```
**CSV:**
```csv theme={null}
subject,predicate,object,confidence,source_frame_id
Alice,employer,Anthropic,0.95,145
Alice,role,Senior Engineer,0.92,145
Bob,employer,OpenAI,0.88,156
```
***
## Schema
Infer and manage predicate schemas.
### Synopsis
```bash theme={null}
memvid schema [OPTIONS]
```
### Subcommands
#### schema infer
Infer schema from existing facts.
```bash theme={null}
memvid schema infer project.mv2
# Filter by entity type
memvid schema infer project.mv2 --entity-type person
```
#### schema list
List known schemas.
```bash theme={null}
memvid schema list project.mv2
# Filter by predicate
memvid schema list project.mv2 --predicate employer
```
### Response
```
Inferred Schema for project.mv2
Entity Type: person
employer: string (organization)
role: string (job_title)
location: string (city)
joined: date
Entity Type: project
status: enum (active, completed, cancelled)
budget: currency
lead: reference (person)
deadline: date
```
***
## Models
The `models` command manages local models for enrichment, embeddings, and visual search.
### Subcommands
| Subcommand | Description |
| ---------- | ----------------------------------- |
| `install` | Install a model |
| `list` | List available and installed models |
| `remove` | Remove an installed model |
| `verify` | Verify model integrity |
### Install Models
```bash theme={null}
# Install LLM model for enrichment
memvid models install phi-3.5-mini
# Install CLIP model for visual search
memvid models install --clip mobileclip-s2
# Install NER model for Logic-Mesh entity extraction
memvid models install --ner distilbert-ner
# Force re-download
memvid models install phi-3.5-mini --force
```
### Available Models
| Category | Model | Description |
| -------- | -------------------- | ------------------------------------ |
| **LLM** | `phi-3.5-mini` | Phi-3.5 Mini for enrichment |
| **LLM** | `phi-3.5-mini-q8` | Quantized version (smaller) |
| **CLIP** | `mobileclip-s2` | MobileCLIP for visual search |
| **CLIP** | `mobileclip-s2-fp16` | FP16 precision version |
| **CLIP** | `siglip-base` | SigLIP base model |
| **NER** | `distilbert-ner` | DistilBERT NER for entity extraction |
### List Models
```bash theme={null}
# List all models
memvid models list
# JSON output
memvid models list --json
# Filter by model type
memvid models list --model-type embedding
memvid models list --model-type clip
memvid models list --model-type ner
```
### Model Types
| Type | Description |
| ----------- | ----------------------------------------- |
| `embedding` | Text embedding models for semantic search |
| `reranker` | Result reranking models |
| `llm` | Local LLM models for inference |
| `clip` | CLIP models for visual search |
| `ner` | NER models for entity extraction |
| `external` | External API-based models |
### Remove and Verify Models
```bash theme={null}
# Remove a model
memvid models remove phi-3.5-mini
# Skip confirmation
memvid models remove phi-3.5-mini --yes
# Verify model integrity
memvid models verify phi-3.5-mini
# Verify all installed models
memvid models verify
```
***
## Follow (Logic-Mesh Traversal)
The `follow` command traverses the entity-relationship graph built from extracted entities.
Logic-Mesh must be enabled during ingestion with `memvid put --logic-mesh` to use follow commands.
### Subcommands
| Subcommand | Description |
| ---------- | ----------------------------------- |
| `traverse` | Follow relationships from an entity |
| `entities` | List all entities in the mesh |
| `stats` | Show Logic-Mesh statistics |
### Traverse Relationships
```bash theme={null}
memvid follow traverse knowledge.mv2 --start "Microsoft"
```
### Traverse Options
| Option | Description | Default |
| --------------- | ------------------------------------------ | --------- |
| `--start`, `-s` | Starting entity (partial match) | Required |
| `--link`, `-l` | Relationship type to follow | `related` |
| `--hops` | Maximum traversal depth | `2` |
| `--direction` | Direction (`outgoing`, `incoming`, `both`) | `both` |
| `--json` | Output as JSON | `false` |
### Traverse Examples
```bash theme={null}
# Find entities related to Microsoft
memvid follow traverse knowledge.mv2 --start "Microsoft"
# Follow specific relationship type
memvid follow traverse knowledge.mv2 --start "Satya Nadella" --link "manager"
# Deeper traversal
memvid follow traverse knowledge.mv2 --start "Seattle" --hops 3
# JSON output
memvid follow traverse knowledge.mv2 --start "Microsoft" --json
```
### List Entities
```bash theme={null}
# List all entities
memvid follow entities knowledge.mv2
# Filter by entity type
memvid follow entities knowledge.mv2 --kind person
memvid follow entities knowledge.mv2 --kind organization
# Search entities by name
memvid follow entities knowledge.mv2 --query "tech"
# Limit results
memvid follow entities knowledge.mv2 --limit 100
# JSON output
memvid follow entities knowledge.mv2 --json
```
### Entity Types
| Type | Description |
| -------------- | ----------------------- |
| `person` | Individuals |
| `organization` | Companies, institutions |
| `project` | Projects, products |
| `location` | Places, addresses |
### Mesh Statistics
```bash theme={null}
# View Logic-Mesh statistics
memvid follow stats knowledge.mv2
# JSON output
memvid follow stats knowledge.mv2 --json
```
### Stats Output
```
Logic-Mesh Statistics
=====================
Nodes (entities): 156
Edges (relations): 423
Entity Kinds:
person: 45
organization: 32
project: 28
location: 51
Relationship Types:
related: 180
member: 95
manager: 48
author: 100
Storage offset: 1234567
Storage size: 45678 bytes
```
***
## Audit
The `audit` command generates sourced reports on specific topics from your knowledge base.
### Basic Usage
```bash theme={null}
memvid audit knowledge.mv2 "What are the key findings about customer satisfaction?"
```
### Options
| Option | Description | Default |
| ----------------- | ------------------------------------------ | -------- |
| `--out`, `-o` | Output file path | stdout |
| `--format` | Output format (`text`, `markdown`, `json`) | `text` |
| `--top-k` | Number of sources to retrieve | `10` |
| `--snippet-chars` | Maximum characters per snippet | `500` |
| `--mode` | Retrieval mode (`lex`, `sem`, `hybrid`) | `hybrid` |
| `--scope` | Filter by URI prefix | None |
| `--start` | Start date filter | None |
| `--end` | End date filter | None |
| `--use-model` | Model for answer synthesis | None |
### Examples
```bash theme={null}
# Basic audit to stdout
memvid audit knowledge.mv2 "Revenue trends in Q4"
# Save as markdown report
memvid audit knowledge.mv2 "Security vulnerabilities" --format markdown --out report.md
# JSON output for automation
memvid audit knowledge.mv2 "Customer feedback" --format json --out audit.json
# More comprehensive retrieval
memvid audit knowledge.mv2 "Product roadmap" --top-k 20 --snippet-chars 1000
# Filter by date range
memvid audit knowledge.mv2 "Sales performance" --start "2024-01-01" --end "2024-12-31"
# Filter by URI scope
memvid audit knowledge.mv2 "Engineering decisions" --scope "mv2://docs/engineering/"
# Use model for answer synthesis
memvid audit knowledge.mv2 "Summarize customer issues" --use-model "ollama:qwen2.5:1.5b"
memvid audit knowledge.mv2 "Key takeaways" --use-model "openai:gpt-4o-mini"
memvid audit knowledge.mv2 "Key takeaways" --use-model "nvidia:meta/llama3-8b-instruct"
```
### Output Formats
**Text (default):**
```
AUDIT REPORT: Revenue trends in Q4
==================================
Source 1: Q4 Financial Report (mv2://reports/q4-2024.pdf)
---------------------------------------------------------
Revenue increased 15% year-over-year, driven by enterprise sales...
Source 2: Board Meeting Notes (mv2://notes/board-dec.md)
--------------------------------------------------------
CFO presented Q4 projections showing strong growth in APAC region...
```
**Markdown:**
```markdown theme={null}
# Audit Report: Revenue trends in Q4
## Source 1: Q4 Financial Report
**URI:** mv2://reports/q4-2024.pdf
Revenue increased 15% year-over-year, driven by enterprise sales...
## Source 2: Board Meeting Notes
**URI:** mv2://notes/board-dec.md
CFO presented Q4 projections showing strong growth in APAC region...
```
**JSON:**
```json theme={null}
{
"question": "Revenue trends in Q4",
"sources": [
{
"title": "Q4 Financial Report",
"uri": "mv2://reports/q4-2024.pdf",
"snippet": "Revenue increased 15% year-over-year...",
"score": 0.95
}
],
"answer": "Based on the sources, Q4 revenue increased 15%..."
}
```
***
## Session Recording (Time-Travel Replay)
The `session` command enables recording and replaying agent sessions for debugging RAG failures and testing different search strategies.
### Subcommands
| Subcommand | Description |
| ---------- | ------------------------------------------ |
| `start` | Start a recording session |
| `end` | End the current recording session |
| `list` | List all recorded sessions |
| `replay` | Replay a session with different parameters |
| `delete` | Delete a recorded session |
### Start Session
```bash theme={null}
# Start a named recording session
memvid session start knowledge.mv2 "Debug Session"
# Start an unnamed session
memvid session start knowledge.mv2
```
All subsequent operations (`put`, `find`, `ask`) will be recorded until the session is ended.
### End Session
```bash theme={null}
memvid session end knowledge.mv2
```
Returns a summary with action count, checkpoints, and duration.
### List Sessions
```bash theme={null}
# List all recorded sessions
memvid session list knowledge.mv2
# JSON output
memvid session list knowledge.mv2 --json
```
### Replay Session
The key feature: replay a recorded session with different parameters to understand how results change.
```bash theme={null}
# Replay with default parameters
memvid session replay knowledge.mv2
# Replay with adaptive retrieval enabled
memvid session replay knowledge.mv2 --adaptive
# Replay with different top-k
memvid session replay knowledge.mv2 --top-k 20
# Replay with different strategy
memvid session replay knowledge.mv2 --adaptive --strategy elbow
```
### Replay Options
| Option | Description | Default |
| ------------ | ------------------------------------------------------------ | -------------- |
| `--adaptive` | Enable adaptive retrieval | `false` |
| `--top-k` | Override top-k for searches | Original value |
| `--strategy` | Adaptive strategy (`elbow`, `cliff`, `relative`, `combined`) | `combined` |
| `--verbose` | Show detailed replay output | `false` |
| `--json` | Output results as JSON | `false` |
### Delete Session
```bash theme={null}
# Delete a session
memvid session delete knowledge.mv2
# Skip confirmation
memvid session delete knowledge.mv2 --yes
```
### Use Case: Debugging RAG Failures
When a query fails to find relevant results:
1. **Start a session** before ingesting data
2. **Ingest documents** with different terminology
3. **Run queries** that show the failure
4. **End the session**
5. **Replay with adaptive retrieval** to see if results improve
```bash theme={null}
# Start recording
memvid session start knowledge.mv2 "Terminology Mismatch Debug"
# Ingest documents
memvid put knowledge.mv2 --text "Databricks acquired Tabular"
# Query with mismatched terminology (fails with top-k)
memvid find knowledge.mv2 "Databricks purchases"
# End recording
memvid session end knowledge.mv2
# Replay with adaptive retrieval
memvid session replay knowledge.mv2 --adaptive --verbose
```
The replay shows how adaptive retrieval discovers documents that top-k filtering missed.
***
## Best Practices
### Enrichment Strategy
```bash theme={null}
# Start with fast, offline rules
memvid enrich project.mv2 --engine rules
# Upgrade to LLM for better accuracy on important data
memvid enrich project.mv2 --engine openai --force
# Use incremental for ongoing updates
memvid enrich project.mv2 --engine rules # Only new frames
```
### Combining Search Modes
```bash theme={null}
# Use state for entity queries (instant)
memvid state project.mv2 "Alice"
# Use find for exploratory search
memvid find project.mv2 --query "who works at AI companies" --graph
# Use ask for complex questions
memvid ask project.mv2 --question "What projects is Alice leading?" --memories
```
### Compliance and Audit
```bash theme={null}
# Export all facts for compliance review
memvid export project.mv2 -o compliance.json --format json
# Audit specific entity
memvid facts project.mv2 --entity "Alice" > alice-audit.txt
# Time-travel for historical state
memvid state project.mv2 "Alice" --as-of-frame 100
```
***
## Environment Variables
| Variable | Description |
| ------------------- | ----------------------------------------------------- |
| `MEMVID_MODELS_DIR` | Model storage directory (default: `~/.memvid/models`) |
| `MEMVID_OFFLINE=1` | Skip model downloads (use cached models only) |
| `OPENAI_API_KEY` | Required for OpenAI enrichment and models |
| `ANTHROPIC_API_KEY` | Required for Claude models |
| `GEMINI_API_KEY` | Required for Gemini models (legacy: `GOOGLE_API_KEY`) |
| `MISTRAL_API_KEY` | Required for Mistral models |
| `GROQ_API_KEY` | Required for Groq inference |
***
## Next Steps
Learn about Logic-Mesh and entity extraction
Enable visual search with CLIP
Configure local model inference
Use these features programmatically
# CLI Cheat Sheet
Source: https://docs.memvid.com/cli/cheat-sheet
Quick reference for all Memvid CLI commands
A single-page reference for the most common Memvid CLI commands.
***
## Installation
```bash theme={null}
# npm (recommended)
npm install -g memvid-cli
# From source
cargo install memvid-cli
```
***
## Create & Manage
```bash theme={null}
# Create new memory
memvid create memory.mv2
# Create with options
memvid create memory.mv2 --size 25MB # Custom size
memvid create memory.mv2 --no-vec # No vector index
memvid create memory.mv2 --no-lex # No lexical index
memvid create memory.mv2 --memory-id # Bind to dashboard
# View memory info
memvid open memory.mv2
memvid stats memory.mv2
memvid stats memory.mv2 --json
```
***
## Ingest Content
```bash theme={null}
# Add single file
memvid put memory.mv2 --input document.pdf
# Add folder
memvid put memory.mv2 --input ./documents/
# Add with metadata
memvid put memory.mv2 --input doc.pdf --metadata '{"author":"John"}'
# Add with timestamp
memvid put memory.mv2 --input doc.pdf --timestamp "2024-01-15"
# Add with URI
memvid put memory.mv2 --input doc.pdf --uri "docs/guide/intro"
# Skip embedding (faster)
memvid put memory.mv2 --input logs.txt --embedding-skip
# Enable entity extraction
memvid put memory.mv2 --input doc.pdf --logic-mesh
# Parallel ingestion (faster)
memvid put memory.mv2 --input ./large-folder/ --parallel-segments
# Fetch from URL
memvid api-fetch memory.mv2 --url https://example.com/doc.pdf
```
***
## Search
```bash theme={null}
# Basic search
memvid find memory.mv2 --query "search term"
# Search modes
memvid find memory.mv2 --query "term" --mode auto # Hybrid (default)
memvid find memory.mv2 --query "term" --mode lex # Lexical only
memvid find memory.mv2 --query "term" --mode sem # Semantic only
# Limit results
memvid find memory.mv2 --query "term" --top-k 5
# Disable adaptive
memvid find memory.mv2 --query "term" --no-adaptive --top-k 10
# Filter by scope
memvid find memory.mv2 --query "term" --scope "docs/"
memvid find memory.mv2 --query "term" --uri "specific/path"
# Filter by date
memvid find memory.mv2 --query "term" --start 2024-01-01 --end 2024-06-30
# JSON output
memvid find memory.mv2 --query "term" --json
# Graph-filtered search
memvid find memory.mv2 --query "revenue" --graph "?:works_at:Acme"
```
***
## Ask (RAG)
```bash theme={null}
# Ask a question
memvid ask memory.mv2 --question "What is the main topic?"
# Choose synthesis model
memvid ask memory.mv2 --question "..." --use-model tinyllama # Local (default)
memvid ask memory.mv2 --question "..." --use-model openai # GPT-4
memvid ask memory.mv2 --question "..." --use-model claude # Claude
memvid ask memory.mv2 --question "..." --use-model groq # Llama via Groq
memvid ask memory.mv2 --question "..." --use-model gemini # Gemini
# Get context only (no synthesis)
memvid ask memory.mv2 --question "..." --context-only
# Include sources
memvid ask memory.mv2 --question "..." --sources
# Mask PII in response
memvid ask memory.mv2 --question "..." --mask-pii
# Include memory cards
memvid ask memory.mv2 --question "..." --memories
```
***
## Browse & View
```bash theme={null}
# Timeline view
memvid timeline memory.mv2
memvid timeline memory.mv2 --limit 20
memvid timeline memory.mv2 --reverse
memvid timeline memory.mv2 --since 1704067200
memvid timeline memory.mv2 --until 1706745600
# View single frame
memvid view memory.mv2 --frame-id frame_abc123
memvid view memory.mv2 --uri "docs/intro"
memvid view memory.mv2 --frame-id frame_abc --json
# Play audio/video
memvid view memory.mv2 --frame-id frame_abc --play
memvid view memory.mv2 --frame-id frame_abc --play --start-seconds 30
# Time-travel queries
memvid find memory.mv2 --query "term" --as-of-frame frame_old
memvid find memory.mv2 --query "term" --as-of-ts "2024-01-01T00:00:00Z"
memvid timeline memory.mv2 --as-of-frame frame_old
```
***
## Enrichment & Memory Cards
```bash theme={null}
# Enrich with different engines
memvid enrich memory.mv2 --engine rules # Fast, free
memvid enrich memory.mv2 --engine candle # Local LLM
memvid enrich memory.mv2 --engine groq # Fast + quality
memvid enrich memory.mv2 --engine openai # GPT-4
memvid enrich memory.mv2 --engine claude # Claude
# Force re-enrichment
memvid enrich memory.mv2 --engine groq --force
# View memory cards
memvid memories memory.mv2
memvid memories memory.mv2 --json
# Query entity state
memvid state memory.mv2 --entity "John Smith"
memvid state memory.mv2 --entity "John Smith" --json
# View fact history
memvid facts memory.mv2 --entity "John Smith"
memvid facts memory.mv2 --entity "John" --predicate job_title
# Export facts
memvid export memory.mv2 --format ntriples --out facts.nt
memvid export memory.mv2 --format json --out facts.json
memvid export memory.mv2 --format csv --out facts.csv
```
***
## Graph Traversal
```bash theme={null}
# List entities
memvid follow entities memory.mv2
memvid follow entities memory.mv2 --kind person
memvid follow entities memory.mv2 --query "John"
# Traverse relationships
memvid follow traverse memory.mv2 --start "John Smith"
memvid follow traverse memory.mv2 --start "Acme" --link "employs"
memvid follow traverse memory.mv2 --start "Acme" --hops 3
# Graph statistics
memvid follow stats memory.mv2
```
***
## Tables
```bash theme={null}
# Import tables from PDF
memvid tables import memory.mv2 --input report.pdf
memvid tables import memory.mv2 --input report.pdf --mode conservative
# List tables
memvid tables list memory.mv2
# View table
memvid tables view memory.mv2 --table-id tbl_abc123
# Export table
memvid tables export memory.mv2 --table-id tbl_abc --format csv --out data.csv
memvid tables export memory.mv2 --table-id tbl_abc --format json --as-records
```
***
## Maintenance
```bash theme={null}
# Verify integrity
memvid verify memory.mv2
memvid verify memory.mv2 --deep
# Repair and optimize
memvid doctor memory.mv2 --vacuum # Compact storage
memvid doctor memory.mv2 --rebuild-lex-index # Rebuild lexical
memvid doctor memory.mv2 --rebuild-vec-index # Rebuild vectors
memvid doctor memory.mv2 --rebuild-time-index # Rebuild timeline
# Full repair
memvid doctor memory.mv2 --vacuum --rebuild-lex-index --rebuild-vec-index
# Build sketch index
memvid sketch build memory.mv2 --variant medium
memvid sketch info memory.mv2
# Check lock status
memvid who memory.mv2
memvid nudge memory.mv2 # Request lock release
```
***
## Encryption
```bash theme={null}
# Encrypt memory file
memvid lock memory.mv2 --out memory.mv2e
memvid lock memory.mv2 --out memory.mv2e --keep-original
# Encrypt from stdin (for scripts)
echo "password" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
# Decrypt
memvid unlock memory.mv2e --out memory.mv2
echo "password" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
```
***
## Configuration
```bash theme={null}
# Set API key
memvid config set api_key mv2_xxx
# Set dashboard URL
memvid config set dashboard_url https://memvid.com
# Set named memory
memvid config set memory.work
memvid config set memory.personal
# List configuration
memvid config list
memvid config list --show-values
# Get single value
memvid config get api_key
# Remove value
memvid config unset api_key
# Verify API key
memvid config check
# View system status
memvid status
memvid status --json
```
***
## Plan & Usage
```bash theme={null}
# Show plan info
memvid plan show
memvid plan show --json
# Sync plan from dashboard
memvid plan sync
# Clear cached ticket
memvid plan clear
```
***
## Tickets & Capacity
```bash theme={null}
# List current ticket
memvid tickets list memory.mv2
# Sync from dashboard
memvid tickets sync memory.mv2 --memory-id
# Apply ticket
memvid tickets apply memory.mv2 --memory-id
```
***
## Models
```bash theme={null}
# List available models
memvid models list
# Install model
memvid models install bge-base
memvid models install nomic
# Remove model
memvid models remove bge-base
# Verify model integrity
memvid models verify bge-small
```
***
## Global Options
```bash theme={null}
# Verbose output
memvid -v find memory.mv2 --query "term"
memvid -vv find memory.mv2 --query "term" # More verbose
# Specify embedding model
memvid -m nomic put memory.mv2 --input docs/
memvid -m openai put memory.mv2 --input docs/
# Available models: bge-small, bge-base, nomic, gte-large, openai
```
***
## Common Workflows
### Quick Start
```bash theme={null}
memvid create brain.mv2
memvid put brain.mv2 --input ~/Documents/
memvid find brain.mv2 --query "meeting notes"
memvid ask brain.mv2 --question "What was decided in the last meeting?"
```
### Code Search
```bash theme={null}
memvid create code.mv2 --no-vec
memvid put code.mv2 --input ./src/
memvid find code.mv2 --query "handleAuth" --mode lex
```
### Knowledge Base with Enrichment
```bash theme={null}
memvid create kb.mv2
memvid put kb.mv2 --input ./docs/ --logic-mesh
memvid enrich kb.mv2 --engine groq
memvid state kb.mv2 --entity "Product"
memvid find kb.mv2 --query "pricing" --graph "?:feature:Product"
```
### Encrypted Backup
```bash theme={null}
memvid lock important.mv2 --out backups/important-$(date +%Y%m%d).mv2e
```
***
## Environment Variables
```bash theme={null}
MEMVID_API_KEY=mv2_xxx # Dashboard API key
MEMVID_DASHBOARD_URL=... # Custom dashboard URL
MEMVID_TELEMETRY=0 # Opt out of telemetry
OPENAI_API_KEY=sk-xxx # For OpenAI embeddings/LLM
ANTHROPIC_API_KEY=sk-ant-xxx # For Claude
GROQ_API_KEY=gsk_xxx # For Groq
GOOGLE_API_KEY=xxx # For Gemini
```
***
## Getting Help
```bash theme={null}
# General help
memvid --help
# Command-specific help
memvid find --help
memvid put --help
memvid ask --help
# Version
memvid --version
```
# Create & Ingest
Source: https://docs.memvid.com/cli/create-and-put
Create memory files and ingest documents with the Memvid CLI
Learn how to create new memory files and ingest documents using the Memvid CLI.
## Creating a Memory File
### Basic Usage
Create a new `.mv2` memory file:
```bash theme={null}
memvid create my-knowledge.mv2
```
### Options
| Option | Description | Default |
| ------------- | ------------------------------------------------- | ------- |
| `--tier` | Capacity tier (`free`, `dev`, `enterprise`) | `free` |
| `--size` | Capacity override (e.g. `15MB`, capped at `50MB`) | `50MB` |
| `--no-lex` | Disable lexical/full-text index | Enabled |
| `--no-vector` | Disable vector index | Enabled |
### Examples
```bash theme={null}
# Create a basic memory file
memvid create research.mv2
# Create without lexical index
memvid create notes.mv2 --no-lex
# Create a smaller memory (capacity override)
memvid create small.mv2 --size 512MB
```
`memvid create` is capped at 50MB. To go beyond 50MB, create the file and then apply a signed capacity ticket (see `memvid tickets sync/apply`).
### JSON Output
```bash theme={null}
memvid create my-memory.mv2 --json
```
```json theme={null}
{
"path": "my-memory.mv2",
"size_limit_bytes": 536870912,
"lex_enabled": true,
"vec_enabled": true,
"created_at": "2024-01-15T10:30:00Z"
}
```
***
## Inspecting a Memory File
The `open` command shows metadata and manifests of an existing memory file.
### Synopsis
```bash theme={null}
memvid open [OPTIONS]
```
### Options
| Option | Description |
| -------- | ---------------- |
| `--json` | Emit JSON output |
### Examples
```bash theme={null}
# Inspect a memory file
memvid open my-memory.mv2
# Get JSON output for scripting
memvid open my-memory.mv2 --json
```
### Response
```
Memory File: my-memory.mv2
Version: 2.1.0
Created: 2024-01-15T10:30:00Z
Frames: 1,234
Size: 45.2 MB / 512 MB (8.8%)
Indexes:
Lexical: enabled (12,456 terms)
Vector: enabled (1,234 vectors, 384d)
Time: enabled (1,234 entries)
Tracks:
default: 890 frames
meetings: 234 frames
emails: 110 frames
Memory Binding:
Memory ID: mem_abc123
Bound at: 2024-01-15T10:30:00Z
```
### JSON Output
```json theme={null}
{
"path": "my-memory.mv2",
"version": "2.1.0",
"created_at": "2024-01-15T10:30:00Z",
"frame_count": 1234,
"size_bytes": 47395430,
"size_limit_bytes": 536870912,
"indexes": {
"lex": { "enabled": true, "term_count": 12456 },
"vec": { "enabled": true, "vector_count": 1234, "dimension": 384 },
"time": { "enabled": true, "entry_count": 1234 }
},
"tracks": {
"default": 890,
"meetings": 234,
"emails": 110
},
"binding": {
"memory_id": "mem_abc123",
"bound_at": "2024-01-15T10:30:00Z"
}
}
```
***
## Ingesting Documents
The `put` command adds documents to your memory file as frames.
### Basic Usage
```bash theme={null}
# Ingest a single file (text-only)
memvid put my-knowledge.mv2 --input document.pdf
# Ingest a directory
memvid put my-knowledge.mv2 --input ./documents/
# Ingest with semantic embeddings (+16x PQ compression)
memvid put my-knowledge.mv2 --input document.pdf --embedding --vector-compression
# Ingest from stdin (text-only by default)
echo "Some text content" | memvid put my-knowledge.mv2
```
### Core Options
| Option | Description |
| --------------------- | ------------------------- |
| `--input PATH` | Path to file or directory |
| `--uri URI` | Custom URI for the frame |
| `--title TITLE` | Document title |
| `--timestamp UNIX_TS` | POSIX timestamp |
| `--track TRACK` | Track/collection name |
| `--kind KIND` | Content type metadata |
| `--json` | Output as JSON |
### Metadata Options
| Option | Description |
| -------------------- | -------------------------------- |
| `--tag KEY=VALUE` | Add tags (repeatable) |
| `--label LABEL` | Add labels (repeatable) |
| `--metadata JSON` | Additional metadata as JSON |
| `--no-auto-tag` | Disable automatic tag extraction |
| `--no-extract-dates` | Disable date extraction |
### CLIP & Entity Extraction (Auto-Enabled)
When the CLIP and NER models are installed, the CLI **automatically enables** visual embeddings for images/PDFs and entity extraction.
| Option | Description |
| ----------------- | ------------------------------------------------------ |
| `--clip` | Explicitly enable CLIP visual embeddings |
| `--no-clip` | Disable CLIP even when model is available |
| `--logic-mesh` | Explicitly enable entity extraction |
| `--no-logic-mesh` | Disable entity extraction even when model is available |
**Install models manually:**
```bash theme={null}
memvid models install --clip mobileclip-s2
memvid models install --ner distilbert-ner
```
### Embedding Options
| Option | Description |
| ----------------------------- | ------------------------------------------------------- |
| `--embedding` | Enable semantic embeddings |
| `-m, --embedding-model MODEL` | Choose default embedding model (global flag; see below) |
| `--vector-compression` | Generate semantic embeddings with 16x compression |
| `--no-embedding` | Explicitly disable embeddings |
**Embedding Model Options:**
| Model | Description |
| -------------- | --------------------------------------------- |
| `bge-small` | Local fastembed default (384d) |
| `bge-base` | Local higher quality (768d) |
| `nomic` | Local high accuracy (768d) |
| `gte-large` | Local best semantic depth (1024d) |
| `openai-small` | OpenAI text-embedding-3-small (1536d) |
| `openai-large` | OpenAI text-embedding-3-large (3072d) |
| `openai` | Alias for `openai-large` |
| `openai-ada` | OpenAI text-embedding-ada-002 (1536d, legacy) |
```bash theme={null}
# Use built-in BGE (default, no API key needed)
memvid put knowledge.mv2 --input docs/ --embedding
# Use OpenAI embeddings
export OPENAI_API_KEY=sk-...
memvid put knowledge.mv2 --input docs/ --embedding -m openai-small
# Use OpenAI large model for higher quality
memvid put knowledge.mv2 --input docs/ --embedding -m openai-large
```
### Table Extraction Options
| Option | Description |
| -------------- | --------------------------------------------------------------- |
| `--tables` | Extract tables from PDF files |
| `--embed-rows` | Embed individual table rows for semantic search (default: true) |
### Duplicate Handling
| Option | Description |
| ------------------- | ------------------------------------ |
| `--update-existing` | Replace existing frame with same URI |
| `--allow-duplicate` | Allow multiple frames with same URI |
### Lock Control
| Option | Description | Default |
| ------------------- | ---------------------------- | ------- |
| `--lock-timeout MS` | Wait time for lock | 250ms |
| `--force` | Force takeover of stale lock | false |
### Ingesting Different File Types
Memvid automatically detects and processes various file formats:
```bash theme={null}
# Plain text
memvid put knowledge.mv2 --input notes.txt --vector-compression
# Markdown
memvid put knowledge.mv2 --input README.md --vector-compression
# HTML
memvid put knowledge.mv2 --input page.html --vector-compression
```
```bash theme={null}
# PDF files
memvid put knowledge.mv2 --input report.pdf --vector-compression
# PDF with table extraction
memvid put knowledge.mv2 --input invoice.pdf --tables --vector-compression
# Word documents
memvid put knowledge.mv2 --input document.docx --vector-compression
# Excel spreadsheets
memvid put knowledge.mv2 --input data.xlsx --vector-compression
# PowerPoint presentations
memvid put knowledge.mv2 --input slides.pptx --vector-compression
```
```bash theme={null}
# Images with EXIF extraction
memvid put knowledge.mv2 --input photo.jpg
# Audio files
memvid put knowledge.mv2 --input recording.mp3 --audio
# Video files (stored without transcoding)
memvid put knowledge.mv2 --input video.mp4 --video
```
### Adding Metadata
Organize your documents with tracks, tags, and timestamps:
```bash theme={null}
# Add to a specific track
memvid put knowledge.mv2 --input meeting-notes.md --vector-compression --track "meetings"
# Add metadata tags
memvid put knowledge.mv2 --input api-docs.md --vector-compression \
--tag "category=documentation" \
--tag "version=2.0" \
--tag "author=team"
# Add labels
memvid put knowledge.mv2 --input report.pdf --vector-compression \
--label "quarterly" \
--label "finance"
# Set custom timestamp
memvid put knowledge.mv2 --input old-report.pdf --vector-compression \
--timestamp 1686819000
# Combine options
memvid put knowledge.mv2 --input quarterly-report.pdf --vector-compression \
--track "reports" \
--title "Q3 2024 Report" \
--tag "quarter=Q3" \
--tag "year=2024"
```
### Parallel Ingestion
For large datasets, enable multi-threaded processing:
```bash theme={null}
# Enable parallel ingestion
memvid put knowledge.mv2 --input ./large-dataset/ --vector-compression \
--parallel-segments \
--parallel-threads 8
# Fine-tune parallel settings
memvid put knowledge.mv2 --input ./corpus/ --vector-compression \
--parallel-segments \
--parallel-seg-tokens 4000 \
--parallel-threads 4 \
--parallel-queue-depth 16
```
| Option | Description | Default |
| ------------------------ | -------------------------------- | ------------- |
| `--parallel-segments` | Enable multi-threaded processing | false |
| `--parallel-threads` | Number of worker threads | CPU count - 1 |
| `--parallel-queue-depth` | Queue size for workers | Auto |
| `--parallel-seg-tokens` | Target tokens per segment | Auto |
### Ingesting from Stdin
Useful for piping data from other commands:
```bash theme={null}
# Pipe text content
echo "Important note to remember" | memvid put knowledge.mv2 --vector-compression
# Pipe from curl
curl -s https://api.example.com/data | memvid put knowledge.mv2 --vector-compression --title "API Response"
# Pipe from another command
cat log.txt | grep "ERROR" | memvid put knowledge.mv2 --vector-compression --track "errors"
```
***
## PDF Table Extraction
Extract structured tables from PDFs (invoices, financial reports, pay stubs):
### Basic Usage
```bash theme={null}
# Extract tables from a PDF
memvid put knowledge.mv2 --input invoice.pdf --tables --vector-compression
# Extract tables and embed individual rows for semantic search
memvid put knowledge.mv2 --input financial-report.pdf --tables --embed-rows --vector-compression
```
### Detection Methods
The table extractor uses multiple detection methods:
| Method | Best For |
| ------------- | -------------------------------------------------- |
| **Stream** | Tables without visible borders, text-based layouts |
| **Lattice** | Tables with visible grid lines and borders |
| **LineBased** | Columnar data with clear alignment patterns |
The extractor automatically tries each method and picks the best results.
### Viewing Extracted Tables
After extraction, use the `tables` command to view and export:
```bash theme={null}
# List all tables in a memory
memvid tables list knowledge.mv2
# Output:
# Found 3 tables:
# - pdf_table_1_page1: 5 rows x 4 cols (LineBased)
# - pdf_table_2_page1: 12 rows x 3 cols (Stream)
# - pdf_table_3_page2: 8 rows x 5 cols (Lattice)
# View a specific table
memvid tables view knowledge.mv2 --table-id pdf_table_1_page1
# Export to CSV
memvid tables export knowledge.mv2 --table-id pdf_table_1_page1 --format csv > data.csv
# Export to JSON
memvid tables export knowledge.mv2 --table-id pdf_table_1_page1 --format json
```
### Example: Invoice Processing
```bash theme={null}
# Create memory for invoices
memvid create invoices.mv2
# Ingest invoice with table extraction
memvid put invoices.mv2 --input amazon-invoice.pdf --tables --vector-compression
# Search for specific items
memvid find invoices.mv2 --query "total" --json
# List extracted tables
memvid tables list invoices.mv2
# Export line items to CSV
memvid tables export invoices.mv2 --table-id pdf_table_1_page1 --format csv
```
***
## Updating Documents
The `update` command modifies an existing frame.
### Synopsis
```bash theme={null}
memvid update [OPTIONS]
```
### Options
| Option | Description |
| ------------------- | --------------------- |
| `--frame-id ` | Target frame by ID |
| `--uri ` | Target frame by URI |
| `--input ` | New payload from file |
| `--set-uri ` | Update frame URI |
| `--title ` | Update title |
| `--timestamp <TS>` | Update timestamp |
| `--track <TRACK>` | Update track |
| `--kind <KIND>` | Update kind |
| `--tag <KEY=VALUE>` | Add/update tags |
| `--label <LABEL>` | Add/update labels |
| `--metadata <JSON>` | Add/update metadata |
| `--embeddings` | Recompute embeddings |
| `--json` | JSON output |
### Examples
```bash theme={null}
# Update title
memvid update project.mv2 --frame-id 1234 --title "Updated Title"
# Update content and recompute embeddings
memvid update project.mv2 --uri "file:///doc.txt" \
--input updated-doc.txt \
--embeddings
# Add new tags
memvid update project.mv2 --frame-id 1234 \
--tag "status=reviewed" \
--label approved
```
### Response
```
Updated frame 1234 in project.mv2
Title: Updated Title
Tags added: status=reviewed
Labels added: approved
Embeddings: recomputed
```
***
## Deleting Documents
The `delete` command removes a frame from the memory.
### Synopsis
```bash theme={null}
memvid delete <FILE> [OPTIONS]
```
### Options
| Option | Description |
| ----------------- | ------------------------ |
| `--frame-id <ID>` | Target by frame ID |
| `--uri <URI>` | Target by frame URI |
| `--yes` | Skip confirmation prompt |
| `--json` | JSON output |
### Examples
```bash theme={null}
# Delete by frame ID
memvid delete project.mv2 --frame-id 1234
# Delete by URI (skip confirmation)
memvid delete project.mv2 --uri "file:///old-doc.txt" --yes
```
### Response
```
Deleted frame 1234 from project.mv2
URI: file:///old-doc.txt
Title: Old Document
```
***
## Remote API Ingestion
The `api-fetch` command fetches remote content from APIs and ingests as frames.
### Synopsis
```bash theme={null}
memvid api-fetch <FILE> <CONFIG> [OPTIONS]
```
### Options
| Option | Description |
| --------------- | ------------------------------- |
| `--dry-run` | Preview without writing |
| `--mode <MODE>` | Override configured ingest mode |
| `--uri <URI>` | Override base URI |
| `--json` | JSON output |
### Config File Format
```json theme={null}
{
"url": "https://api.example.com/documents",
"method": "GET",
"headers": {
"Authorization": "Bearer ${API_TOKEN}"
},
"pagination": {
"type": "cursor",
"cursor_param": "after",
"cursor_path": "$.meta.next_cursor"
},
"items_path": "$.data",
"mapping": {
"title": "$.name",
"text": "$.content",
"uri": "$.id"
}
}
```
### Examples
```bash theme={null}
# Fetch from API
memvid api-fetch project.mv2 ./fetch-config.json
# Dry run to preview
memvid api-fetch project.mv2 ./fetch-config.json --dry-run
```
***
## Real-World Examples
### Documentation Knowledge Base
```bash theme={null}
# Create the memory
memvid create docs.mv2
# Ingest documentation with embeddings
memvid put docs.mv2 --input ./docs/ --vector-compression --track "documentation"
# Add API reference
memvid put docs.mv2 --input ./api-reference/ --vector-compression \
--track "api" \
--tag "type=reference"
```
### Research Paper Archive
```bash theme={null}
# Create the memory
memvid create papers.mv2
# Ingest papers with metadata
for paper in ./papers/*.pdf; do
memvid put papers.mv2 --input "$paper" --vector-compression \
--track "research" \
--tag "source=arxiv"
done
```
### Code Repository
```bash theme={null}
# Create memory for codebase
memvid create codebase.mv2
# Ingest with parallel processing
memvid put codebase.mv2 --input ./src/ --vector-compression \
--parallel-segments \
--track "source"
# Add tests and docs
memvid put codebase.mv2 --input ./tests/ --vector-compression --track "tests"
memvid put codebase.mv2 --input ./docs/ --vector-compression --track "docs"
```
***
## Troubleshooting
### File Locked
```bash theme={null}
Error: File is locked by another process
```
**Solutions:**
```bash theme={null}
# Check who holds the lock
memvid who knowledge.mv2
# Request release
memvid nudge knowledge.mv2
# Find process on macOS/Linux
lsof knowledge.mv2
# Wait longer for lock
memvid put knowledge.mv2 --input doc.pdf --lock-timeout 5000
# Force takeover (only if previous writer crashed)
memvid put knowledge.mv2 --input doc.pdf --force
```
### Capacity Exceeded
```bash theme={null}
Error: CapacityExceeded
```
**Solutions:**
```bash theme={null}
# Check current usage
memvid stats knowledge.mv2
# Delete unused frames
memvid delete knowledge.mv2 --frame-id 42 --yes
# Compact the file
memvid doctor knowledge.mv2 --vacuum
```
### Embedding Model Issues
```bash theme={null}
Error: Failed to load embedding model
```
**Solution:**
```bash theme={null}
# Set model directory
export MEMVID_MODELS_DIR=~/.memvid/models
# Or use offline mode with pre-cached models
export MEMVID_OFFLINE=1
```
***
## Next Steps
<CardGroup>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Query your memories with lexical, semantic, and hybrid search
</Card>
<Card title="Timeline & View" icon="clock" href="/cli/timeline-and-view">
Explore your memories chronologically
</Card>
</CardGroup>
# CLI Reference
Source: https://docs.memvid.com/cli/index
Complete reference for all Memvid CLI commands
The Memvid CLI provides complete control over your memory files from the command line. Every operation available in the SDKs is also available via the CLI.
## Installation
```bash theme={null}
# npm (recommended)
npm install -g memvid-cli
# Verify installation
memvid --version
```
<Info>
See [Installation Guide](/installation/cli) for platform-specific instructions and troubleshooting.
</Info>
## Command Categories
| Category | Commands | Description |
| ------------------------------------------- | -------------------------------------------------------------- | ------------------------------------ |
| [Creation](/cli/create-and-put) | `create`, `open` | Create and inspect memory files |
| [Data](/cli/create-and-put) | `put`, `put-many`, `update`, `delete`, `api-fetch` | Add, modify, and remove documents |
| [Search](/cli/search-and-ask) | `find`, `ask`, `vec-search`, `timeline`, `when`, `audit` | Query and retrieve information |
| [Enrichment](/cli/advanced-commands) | `enrich`, `memories`, `state`, `facts`, `export`, `schema` | Extract and query structured facts |
| [Tables](/cli/advanced-commands) | `tables import`, `tables list`, `tables export`, `tables view` | PDF table extraction |
| [Maintenance](/cli/maintenance-and-tickets) | `verify`, `doctor`, `nudge` | File integrity and repair |
| [Sessions](/cli/timeline-and-view) | `session start`, `session end`, `session replay` | Time-travel debugging |
| [Security](/cli/maintenance-and-tickets) | `lock`, `unlock`, `binding`, `unbind` | Encryption and access control |
| [Tickets](/cli/tickets-and-capacity) | `tickets sync`, `tickets apply`, `plan show` | Capacity and subscription management |
| [Models](/cli/advanced-commands) | `models install`, `models list`, `models remove` | LLM and embedding model management |
## Global Options
These options apply to all commands:
```bash theme={null}
# Increase logging verbosity
memvid --verbose find memory.mv2 --query "test"
memvid -vvv find memory.mv2 --query "test" # Maximum verbosity
# Set default embedding model
memvid --model bge-small find memory.mv2 --query "test"
memvid -m openai find memory.mv2 --query "test"
# JSON output (most commands support this)
memvid find memory.mv2 --query "test" --json
```
## Environment Variables
| Variable | Description | Example |
| --------------------------- | --------------------------------- | ------------- |
| `MEMVID_API_KEY` | Dashboard API key for sync | `mv_live_xxx` |
| `OPENAI_API_KEY` | OpenAI API key for embeddings/LLM | `sk-xxx` |
| `NVIDIA_API_KEY` | NVIDIA API key for embeddings | `nvapi-xxx` |
| `GEMINI_API_KEY` | Google Gemini API key | `AIzaxxx` |
| `MISTRAL_API_KEY` | Mistral API key | `xxx` |
| `MEMVID_TELEMETRY` | Disable telemetry (set to 0) | `0` |
| `MEMVID_LLM_CONTEXT_BUDGET` | Max context chars for LLMs | `8000` |
***
## Common Workflows
### Basic Document Ingestion
```bash theme={null}
# Create memory and add documents
memvid create project.mv2
# Add a text file
cat notes.txt | memvid put project.mv2 --title "Notes"
# Add a PDF
memvid put project.mv2 --input report.pdf --title "Report"
# Add with metadata
memvid put project.mv2 --input doc.txt \
--title "Meeting Notes" \
--track meetings \
--tag "date=2024-01-15" \
--tag "attendees=alice,bob" \
--label important
```
### Search and Retrieval
```bash theme={null}
# Simple search
memvid find project.mv2 --query "budget projections"
# Semantic search with embeddings
memvid find project.mv2 --query "financial outlook" --mode sem
# Ask questions with LLM
memvid ask project.mv2 --question "What was decided about the budget?" --use-model openai
# Time-filtered search
memvid find project.mv2 --query "status" --as-of-ts 1704067200
```
### Fact Extraction and Entity Queries
```bash theme={null}
# Extract facts (fast, offline)
memvid enrich project.mv2 --engine rules
# Extract facts (LLM, more accurate)
memvid enrich project.mv2 --engine openai
# View all facts
memvid memories project.mv2
# Query specific entity
memvid state project.mv2 "Alice"
# Export facts
memvid export project.mv2 -o facts.json --format json
```
### PDF Table Extraction
```bash theme={null}
# Extract tables from PDF
memvid tables import project.mv2 --input financial.pdf
# List extracted tables
memvid tables list project.mv2
# Export table to CSV
memvid tables export project.mv2 --table-id tbl_001 -o table.csv
```
***
## Output Formats
Most commands support `--json` for machine-readable output:
```bash theme={null}
# Human-readable (default)
memvid find project.mv2 --query "test"
# JSON output
memvid find project.mv2 --query "test" --json
# Parse with jq
memvid find project.mv2 --query "test" --json | jq '.hits[0].title'
```
***
## Exit Codes
| Code | Meaning |
| ---- | ----------------- |
| 0 | Success |
| 1 | General error |
| 2 | File not found |
| 3 | Permission denied |
| 4 | Capacity exceeded |
| 5 | Lock conflict |
***
## Next Steps
<CardGroup>
<Card title="Create & Put" icon="file-circle-plus" href="/cli/create-and-put">
Create memory files and add documents
</Card>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Query your memory with hybrid search and LLM
</Card>
<Card title="Memory Cards" icon="brain" href="/cli/advanced-commands">
Extract structured facts with O(1) lookups
</Card>
<Card title="Timeline & View" icon="clock" href="/cli/timeline-and-view">
Time-travel queries and session replay
</Card>
</CardGroup>
# Maintenance & Repair
Source: https://docs.memvid.com/cli/maintenance-and-tickets
Verify integrity, repair files, and manage your memory files
Commands for maintaining the health and integrity of your memory files.
***
## Verify
Check file integrity without modification.
### Basic Usage
```bash theme={null}
memvid verify knowledge.mv2
```
### Options
| Option | Description |
| -------- | ------------------------- |
| `--deep` | Run thorough verification |
| `--json` | Output as JSON |
### Examples
```bash theme={null}
# Quick verification
memvid verify knowledge.mv2
# Deep verification (slower, more thorough)
memvid verify knowledge.mv2 --deep
# JSON output for automation
memvid verify knowledge.mv2 --deep --json
```
**Output:**
```
Verifying: knowledge.mv2
Checks:
[PASS] HeaderChecksum
[PASS] TocIntegrity
[PASS] WalConsistency
[PASS] TimeIndexSortOrder
[PASS] LexIndexDecode
[PASS] VecIndexDecode
[PASS] FrameCountConsistency
Overall: PASSED
```
**JSON Output:**
```json theme={null}
{
"file_path": "knowledge.mv2",
"checks": [
{ "name": "HeaderChecksum", "status": "passed" },
{ "name": "TocIntegrity", "status": "passed" },
{ "name": "WalConsistency", "status": "passed" },
{ "name": "TimeIndexSortOrder", "status": "passed" },
{ "name": "LexIndexDecode", "status": "passed" },
{ "name": "VecIndexDecode", "status": "passed" },
{ "name": "FrameCountConsistency", "status": "passed" }
],
"overall_status": "passed"
}
```
### Exit Codes
* `0` - All checks passed
* Non-zero - One or more checks failed
***
## Doctor
Diagnose and repair issues with memory files.
### Basic Usage
```bash theme={null}
memvid doctor knowledge.mv2
```
### Options
| Option | Description |
| ---------------------- | --------------------------------- |
| `--plan-only` | Preview repairs without applying |
| `--rebuild-time-index` | Rebuild the time index |
| `--rebuild-lex-index` | Rebuild the lexical index |
| `--rebuild-vec-index` | Rebuild the vector index |
| `--vacuum` | Compact and reclaim deleted space |
| `--json` | Output as JSON |
### Examples
```bash theme={null}
# Preview what would be fixed
memvid doctor knowledge.mv2 --plan-only
# Rebuild corrupted time index
memvid doctor knowledge.mv2 --rebuild-time-index
# Rebuild lexical index
memvid doctor knowledge.mv2 --rebuild-lex-index
# Rebuild vector index
memvid doctor knowledge.mv2 --rebuild-vec-index
# Compact deleted frames and reclaim space
memvid doctor knowledge.mv2 --vacuum
# Fix multiple issues at once
memvid doctor knowledge.mv2 --rebuild-time-index --rebuild-lex-index
# Full repair and optimization
memvid doctor knowledge.mv2 \
--rebuild-time-index \
--rebuild-lex-index \
--rebuild-vec-index \
--vacuum
```
**Output:**
```
Doctor Report: knowledge.mv2
Findings:
[WARN] Time index has 3 out-of-order entries
[INFO] 12 deleted frames can be vacuumed
Planned Actions:
1. Rebuild time index
2. Vacuum deleted frames
Executing repairs...
[DONE] Time index rebuilt (148 entries)
[DONE] Vacuumed 12 frames, reclaimed 2.3 MB
Complete.
```
### Severity Levels
| Level | Description |
| ------- | ----------------------------------- |
| `INFO` | Informational, no action needed |
| `WARN` | Potential issue, repair recommended |
| `ERROR` | Problem found, repair required |
***
## Verify Single File
Ensure no auxiliary files exist alongside your memory file.
### Basic Usage
```bash theme={null}
memvid verify-single-file knowledge.mv2
```
This checks that the directory contains only the `.mv2` file with no leftover sidecars:
* No `.wal` files
* No `.shm` files
* No `.lock` files
* No `-wal` files
* No hidden siblings
### Example
```bash theme={null}
$ memvid verify-single-file knowledge.mv2
Single-file check: PASSED
No auxiliary files found.
```
If sidecars are found:
```bash theme={null}
$ memvid verify-single-file knowledge.mv2
Single-file check: FAILED
Found auxiliary files:
- knowledge.mv2-wal
- knowledge.mv2.lock
Remove these files and re-verify.
```
***
## Common Maintenance Tasks
### After Crashes
If your application crashed while writing:
```bash theme={null}
# Verify integrity
memvid verify knowledge.mv2 --deep
# If issues found, run doctor
memvid doctor knowledge.mv2 --plan-only
# Apply repairs if needed
memvid doctor knowledge.mv2 --rebuild-time-index
```
### Reclaiming Space
After deleting many frames:
```bash theme={null}
# Check current stats
memvid stats knowledge.mv2
# Vacuum to reclaim space
memvid doctor knowledge.mv2 --vacuum
# Verify results
memvid stats knowledge.mv2
```
### Rebuilding Indices
If search isn't working correctly:
```bash theme={null}
# Check index status
memvid stats knowledge.mv2 --json | grep has_lex_index
# Rebuild lexical index
memvid doctor knowledge.mv2 --rebuild-lex-index
# Rebuild vector index
memvid doctor knowledge.mv2 --rebuild-vec-index
# Verify search works
memvid find knowledge.mv2 --query "test"
```
### Cleaning Up Old Files
Remove leftover files from older versions:
```bash theme={null}
# Check for sidecars
ls -la knowledge.mv2*
# Remove any found
rm -f knowledge.mv2-wal knowledge.mv2-shm knowledge.mv2.lock
# Verify clean state
memvid verify-single-file knowledge.mv2
```
***
## Environment Variables
| Variable | Description | Default |
| -------------------------- | -------------------------- | ------------------ |
| `MEMVID_MODELS_DIR` | Model cache directory | `~/.memvid/models` |
| `MEMVID_CACHE_DIR` | General cache directory | `~/.cache/memvid` |
| `MEMVID_OFFLINE` | Skip model downloads | `false` |
| `MEMVID_PARALLEL_SEGMENTS` | Control parallel ingestion | Auto |
### Setting Environment Variables
```bash theme={null}
# Set model directory
export MEMVID_MODELS_DIR=~/.memvid/models
# Enable offline mode
export MEMVID_OFFLINE=1
# Enable parallel ingestion
export MEMVID_PARALLEL_SEGMENTS=1
```
***
## Debug Commands
### Version
Check CLI version:
```bash theme={null}
memvid version
```
### Debug Segment
Inspect internal vector segment data (advanced):
```bash theme={null}
# View segment metadata
memvid debug-segment knowledge.mv2 --segment-id 1
# Include hex dump
memvid debug-segment knowledge.mv2 --segment-id 1 --hex-dump --max-bytes 512
```
### Verbose Output
Enable debug logging:
```bash theme={null}
# Increase verbosity
memvid -v verify knowledge.mv2 # WARN level
memvid -vv verify knowledge.mv2 # INFO level
memvid -vvv verify knowledge.mv2 # DEBUG level
memvid -vvvv verify knowledge.mv2 # TRACE level
```
***
## Troubleshooting
### Time Index Corruption
**Symptom:**
```
TimeIndexSortOrder: Failed
```
**Solution:**
```bash theme={null}
memvid doctor knowledge.mv2 --rebuild-time-index
memvid verify knowledge.mv2 --deep
```
### Lexical Index Issues
**Symptom:**
* Search returns no results
* `has_lex_index: false` in stats
**Solution:**
```bash theme={null}
memvid doctor knowledge.mv2 --rebuild-lex-index
```
### Vector Index Issues
**Symptom:**
* Semantic search not working
* `has_vec_index: false` in stats
**Solution:**
```bash theme={null}
memvid doctor knowledge.mv2 --rebuild-vec-index
```
### Capacity Issues
**Symptom:**
```
Error: CapacityExceeded
```
**Solutions:**
```bash theme={null}
# Check current usage
memvid stats knowledge.mv2
# Delete unused frames
memvid delete knowledge.mv2 --frame-id 42 --yes
# Vacuum to reclaim space
memvid doctor knowledge.mv2 --vacuum
```
### Lock Issues
**Symptom:**
```
Error: File is locked by another process
```
**Solutions:**
```bash theme={null}
# Check who holds the lock
memvid who knowledge.mv2
# Request release
memvid nudge knowledge.mv2
# Find process (macOS/Linux)
lsof knowledge.mv2
```
***
## Best Practices
### Regular Maintenance
1. **Verify periodically**: Run `memvid verify --deep` weekly or after major ingestion
2. **Vacuum after deletions**: Run `--vacuum` after deleting significant content
3. **Monitor capacity**: Check `memvid stats` before large ingestions
### Before Sharing Files
1. Verify integrity: `memvid verify knowledge.mv2 --deep`
2. Check single-file: `memvid verify-single-file knowledge.mv2`
3. Review stats: `memvid stats knowledge.mv2`
### After System Updates
1. Verify existing files still work
2. Rebuild indices if search behaves differently
3. Check for any deprecated features
***
## Next Steps
<CardGroup>
<Card title="Create & Ingest" icon="plus" href="/cli/create-and-put">
Add more content to your memories
</Card>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Query your memories
</Card>
</CardGroup>
# Search & Ask
Source: https://docs.memvid.com/cli/search-and-ask
Query your memories with lexical, semantic, and hybrid search
Memvid provides powerful search capabilities combining traditional keyword search with modern semantic understanding.
## Search Modes
Memvid supports three search modes:
| Mode | Engine | Best For |
| ------ | ------------- | -------------------------------------- |
| `lex` | BM25 | Exact keywords, technical terms, names |
| `sem` | Vector search | Natural language, concepts, similarity |
| `auto` | Hybrid | General queries, best overall results |
***
## Basic Search
### The `find` Command
```bash theme={null}
memvid find knowledge.mv2 --query "your search query"
```
### Options
| Option | Description | Default |
| ------------------------- | ----------------------------------------------------------------- | -------- |
| `--query` | Search query string | Required |
| `--mode` | Search mode (`lex`, `sem`, `auto`) | `auto` |
| `--top-k` | Number of results | 8 |
| `--snippet-chars` | Context snippet length | 480 |
| `--json` | Output as JSON | false |
| `--scope` | Filter by URI prefix | All |
| `--uri` | Filter to specific URI | All |
| `--cursor` | Pagination cursor | None |
| `--query-embedding-model` | Override query embedding model (rare; auto-detects when possible) | Auto |
| `--adaptive` | Enable adaptive retrieval (dynamic top-k) | false |
| `--min-relevancy` | Adaptive cutoff threshold | 0.5 |
| `--max-k` | Adaptive max results | 100 |
<Note>
`-m/--embedding-model` is a global flag that selects the default embedding model (not the search mode). Use `--mode` for `lex/sem/auto`.
</Note>
### Time-Travel Options
| Option | Description |
| ---------------------- | ---------------------------- |
| `--as-of-frame ID` | Show results as of frame ID |
| `--as-of-ts TIMESTAMP` | Show results as of timestamp |
***
## Search Mode Examples
### Lexical Search
Best for exact matches and technical terms:
```bash theme={null}
# Find exact keyword
memvid find knowledge.mv2 --query "WebAuthn" --mode lex
# Technical error codes
memvid find knowledge.mv2 --query "ERR_CONNECTION_REFUSED" --mode lex
# Function names
memvid find knowledge.mv2 --query "handleAuthentication" --mode lex
# Date range filtering
memvid find knowledge.mv2 --query "date:[2024-01-01 TO 2024-12-31]" --mode lex
```
### Semantic Search
Best for natural language and conceptual queries:
```bash theme={null}
# Natural language question
memvid find knowledge.mv2 --query "how do users log in" --mode sem
# Conceptual search
memvid find knowledge.mv2 --query "best practices for security" --mode sem
# Find similar content
memvid find knowledge.mv2 --query "machine learning model training" --mode sem
```
<Info>
Semantic (`sem`) and hybrid (`auto`) search require query embeddings. Memvid auto-detects the correct embedding runtime from the `.mv2` when vectors are present. Use `--query-embedding-model` (or global `-m/--embedding-model`) only when you need to override.
</Info>
### Hybrid Search (Recommended)
Combines both approaches for best results:
```bash theme={null}
# General queries
memvid find knowledge.mv2 --query "authentication best practices" --mode auto
# Technical with context
memvid find knowledge.mv2 --query "OAuth2 implementation patterns" --mode auto
```
***
## Query Syntax
### Multi-Word Queries
By default, multi-word queries use **OR** logic for better recall:
```bash theme={null}
# Finds documents containing "machine" OR "learning"
memvid find knowledge.mv2 --query "machine learning" --mode lex
```
### Boolean Operators
Use explicit operators for precise control:
```bash theme={null}
# Must contain both terms
memvid find knowledge.mv2 --query "machine AND learning" --mode lex
# Must contain either term
memvid find knowledge.mv2 --query "python OR javascript" --mode lex
# Exclude term
memvid find knowledge.mv2 --query "database NOT postgres" --mode lex
# Complex expressions
memvid find knowledge.mv2 --query "(api OR rest) AND authentication" --mode lex
```
### Phrase Search
Use quotes for exact phrase matching:
```bash theme={null}
# Exact phrase
memvid find knowledge.mv2 --query '"machine learning"' --mode lex
# Phrase with other terms
memvid find knowledge.mv2 --query '"neural network" AND training' --mode lex
```
<Tip>
For natural language queries, use `--mode sem` (semantic search) which understands meaning rather than exact keywords.
</Tip>
***
## Advanced Search
### Filtering Results
Filter by scope or specific URI:
```bash theme={null}
# Search within specific URI prefix
memvid find knowledge.mv2 --query "authentication" --scope "mv2://api/"
# Search specific document
memvid find knowledge.mv2 --query "error handling" --uri "mv2://docs/errors.md"
```
### Limiting Results
```bash theme={null}
# Get top 5 results
memvid find knowledge.mv2 --query "performance optimization" --top-k 5
# Get single best match
memvid find knowledge.mv2 --query "main entry point" --top-k 1
# Longer snippets
memvid find knowledge.mv2 --query "architecture" --snippet-chars 800
```
### JSON Output
For programmatic use:
```bash theme={null}
memvid find knowledge.mv2 --query "database schema" --json
```
**Output:**
```json theme={null}
{
"query": "database schema",
"elapsed_ms": 5,
"engine": "hybrid",
"total_hits": 12,
"hits": [
{
"rank": 1,
"frame_id": 124,
"score": 0.892,
"uri": "mv2://docs/database.md",
"title": "Database Design",
"text": "The schema defines the following tables...",
"matches": 3,
"range": [145, 290]
}
],
"next_cursor": "eyJvZmZzZXQiOjh9"
}
```
### Pagination
For large result sets:
```bash theme={null}
# First page
memvid find knowledge.mv2 --query "api" --top-k 10 --json
# Next page using cursor from previous response
memvid find knowledge.mv2 --query "api" --top-k 10 --cursor "eyJvZmZzZXQiOjEwfQ"
```
### Time-Travel Queries
View search results at a specific point in time:
```bash theme={null}
# Results as they were at frame 100
memvid find knowledge.mv2 --query "config" --as-of-frame 100
# Results as of a specific timestamp
memvid find knowledge.mv2 --query "config" --as-of-ts 1704067200
```
***
## AI-Powered Q\&A
The `ask` command retrieves relevant documents and synthesizes an answer using an LLM.
### Basic Usage
```bash theme={null}
# Ask a question with local Ollama model (recommended)
memvid ask knowledge.mv2 --question "Why is determinism important?" --use-model "ollama:qwen2.5:1.5b"
# Use cloud providers
memvid ask knowledge.mv2 --question "Why is determinism important?" --use-model openai
memvid ask knowledge.mv2 --question "Why is determinism important?" --use-model "gemini-2.0-flash"
memvid ask knowledge.mv2 --question "Why is determinism important?" --use-model claude
memvid ask knowledge.mv2 --question "Why is determinism important?" --use-model "nvidia:meta/llama3-8b-instruct"
```
### Model Options
| Model | Type | Description |
| -------------------------------- | ----- | ------------------------------------------------ |
| `ollama:qwen2.5:1.5b` | Local | **Recommended** - Fast, private, no API costs |
| `ollama:qwen2.5:3b` | Local | Higher quality, needs more RAM |
| `ollama:phi3:mini` | Local | Good for reasoning tasks |
| `openai` | Cloud | Uses GPT-4o-mini (requires `OPENAI_API_KEY`) |
| `gemini-2.0-flash` | Cloud | Fast Gemini model (requires `GEMINI_API_KEY`) |
| `claude` | Cloud | Claude Sonnet (requires `ANTHROPIC_API_KEY`) |
| `nvidia:meta/llama3-8b-instruct` | Cloud | NVIDIA Integrate API (requires `NVIDIA_API_KEY`) |
<Note>
For NVIDIA models, you can also set `NVIDIA_LLM_MODEL` and use `--use-model nvidia`.
</Note>
<Info>
For local models, see [Local Models with Ollama](/concepts/local-models) for setup instructions.
</Info>
### Options
| Option | Description | Default |
| ------------------------- | ----------------------------------------------------------------- | -------- |
| `--question` | Question to answer | Required |
| `--use-model` | LLM model (see table above) | None |
| `--top-k` | Documents to retrieve | 8 |
| `--snippet-chars` | Context length per document | 480 |
| `--mode` | Retrieval mode (`lex`, `sem`, `hybrid`) | hybrid |
| `--context-only` | Return context without synthesis | false |
| `--mask-pii` | Mask PII before sending to LLM | false |
| `--llm-context-depth` | Override context budget | Auto |
| `--json` | Output as JSON | false |
| `--query-embedding-model` | Override query embedding model (rare; auto-detects when possible) | Auto |
| `--adaptive` | Enable adaptive retrieval (dynamic top-k) | false |
| `--min-relevancy` | Adaptive cutoff threshold | 0.5 |
| `--max-k` | Adaptive max results | 100 |
| `--adaptive-strategy` | Adaptive cutoff strategy | relative |
### Filtering Options
| Option | Description |
| --------------- | ------------------------ |
| `--scope` | Filter by URI prefix |
| `--uri` | Filter to specific URI |
| `--start` | Start date filter |
| `--end` | End date filter |
| `--as-of-frame` | Time-travel to frame ID |
| `--as-of-ts` | Time-travel to timestamp |
### Examples
```bash theme={null}
# Ask with local Ollama model (recommended)
memvid ask knowledge.mv2 \
--question "How do I configure authentication?" \
--use-model "ollama:qwen2.5:1.5b"
# Ask with more context
memvid ask knowledge.mv2 \
--question "Explain the architecture in detail" \
--top-k 15 \
--use-model "ollama:qwen2.5:3b"
# Get just the context without LLM synthesis
memvid ask knowledge.mv2 \
--question "What is the architecture?" \
--context-only
# Mask sensitive data before sending to cloud LLM
memvid ask knowledge.mv2 \
--question "What are the contact details?" \
--use-model openai \
--mask-pii
# Filter to specific date range
memvid ask knowledge.mv2 \
--question "What happened in Q4?" \
--start "2024-10-01" \
--end "2024-12-31" \
--use-model "ollama:qwen2.5:1.5b"
# JSON output with Gemini
memvid ask knowledge.mv2 \
--question "Summarize the API" \
--use-model "gemini-2.0-flash" \
--json
```
### Multi-File Search
Search across multiple memory files:
```bash theme={null}
# Search multiple files
memvid ask docs.mv2 code.mv2 notes.mv2 \
--question "How does authentication work?"
# Using glob patterns
memvid ask ./memories/*.mv2 \
--question "What are the main features?"
```
### JSON Output
```json theme={null}
{
"question": "What is the architecture?",
"answer": "The architecture follows a layered design with...",
"mode": "hybrid",
"context_only": false,
"hits": [
{
"rank": 1,
"frame_id": 124,
"uri": "mv2://docs/arch.md",
"title": "Architecture Overview",
"score": 0.92,
"text": "The system consists of..."
}
],
"grounding": {
"score": 0.85,
"label": "HIGH",
"sentence_count": 3,
"grounded_sentences": 3,
"has_warning": false
},
"follow_up": {
"needed": false
},
"stats": {
"retrieval_ms": 5,
"synthesis_ms": 1200,
"latency_ms": 1205
}
}
```
### Grounding & Hallucination Detection
When using `--json`, the response includes a `grounding` object that measures how well the answer is supported by the retrieved context:
| Field | Description |
| -------------------- | ----------------------------------------- |
| `score` | Grounding score from 0.0 to 1.0 |
| `label` | Quality label: `LOW`, `MEDIUM`, or `HIGH` |
| `sentence_count` | Number of sentences in the answer |
| `grounded_sentences` | Sentences supported by context |
| `has_warning` | True if answer may be hallucinated |
| `warning_reason` | Explanation if warning is present |
```bash theme={null}
# Check grounding quality
memvid ask knowledge.mv2 \
--question "What is the API endpoint?" \
--use-model openai \
--json | jq '.grounding'
```
**Example output for low grounding (potential hallucination):**
```json theme={null}
{
"grounding": {
"score": 0.15,
"label": "LOW",
"sentence_count": 2,
"grounded_sentences": 0,
"has_warning": true,
"warning_reason": "Answer appears to be poorly grounded in context"
},
"follow_up": {
"needed": true,
"reason": "Answer may not be well-supported by the available context",
"hint": "This memory contains information about different topics. Try asking about those instead.",
"available_topics": ["API Reference", "Authentication", "Database Schema"],
"suggestions": [
"Tell me about API Reference",
"Tell me about Authentication",
"What topics are in this memory?"
]
}
}
```
<Tip>
When `follow_up.needed` is `true`, the answer may not be reliable. Consider using the suggested follow-up questions or rephrasing your query.
</Tip>
***
## Ground Truth Corrections
The `correct` command stores authoritative corrections that take priority in future retrievals. Use this to fix hallucinations or add verified facts.
### Synopsis
```bash theme={null}
memvid correct <FILE> <STATEMENT> [OPTIONS]
```
### Options
| Option | Description | Default |
| ---------- | ------------------------------------------ | ------- |
| `--source` | Attribution for the correction | None |
| `--topic` | Topics for retrieval matching (can repeat) | None |
| `--boost` | Retrieval priority boost factor | 2.0 |
### Examples
```bash theme={null}
# Store a correction
memvid correct knowledge.mv2 "Ben Koenig reported to Chloe Nguyen before 2025"
# With source attribution
memvid correct knowledge.mv2 "The API rate limit is 1000 req/min" \
--source "Engineering Team - Jan 2025"
# With topics for better retrieval
memvid correct knowledge.mv2 "OAuth tokens expire after 24 hours" \
--topic "authentication" \
--topic "OAuth" \
--topic "tokens"
# Higher boost for critical corrections
memvid correct knowledge.mv2 "Production database is db.prod.example.com" \
--boost 3.0
```
### Verification
After storing a correction, verify it's retrievable:
```bash theme={null}
# Search for the correction
memvid find knowledge.mv2 --query "Ben Koenig reported to"
# Ask a question that should use the correction
memvid ask knowledge.mv2 \
--question "Who did Ben Koenig report to before 2025?" \
--use-model openai
```
<Note>
Corrections are stored with a `[Correction]` label and receive boosted retrieval scores, ensuring they appear prominently in search results.
</Note>
***
## Vector Search
The `vec-search` command performs direct vector similarity search with pre-computed embeddings.
### Synopsis
```bash theme={null}
memvid vec-search <FILE> [OPTIONS]
```
### Options
| Option | Description | Default |
| -------------------- | -------------------------------- | ------- |
| `--vector <CSV>` | CSV-formatted vector | None |
| `--embedding <PATH>` | Path to JSON file with embedding | None |
| `--limit <K>` | Number of results | 10 |
| `--json` | JSON output | false |
### Examples
```bash theme={null}
# Search with vector file
memvid vec-search project.mv2 --embedding ./query-vec.json --limit 5
# Search with inline vector
memvid vec-search project.mv2 --vector "0.1,0.2,0.3,..." --limit 10
```
***
## Temporal Queries
The `when` command resolves temporal phrases and lists matching frames.
### Synopsis
```bash theme={null}
memvid when <FILE> --on <PHRASE> [OPTIONS]
```
### Options
| Option | Description | Default |
| --------------- | -------------------------- | --------------- |
| `--on <PHRASE>` | Temporal phrase to resolve | Required |
| `--tz <ZONE>` | Timezone for phrases | America/Chicago |
| `--limit <N>` | Maximum frames | All |
| `--json` | JSON output | false |
### Examples
```bash theme={null}
# Frames from "last Monday"
memvid when project.mv2 --on "last Monday"
# Frames from "yesterday"
memvid when project.mv2 --on "yesterday" --tz "America/New_York"
# Frames from "2 weeks ago"
memvid when project.mv2 --on "2 weeks ago" --limit 20
```
***
## Audit Reports
The `audit` command generates audit reports with full source provenance.
### Synopsis
```bash theme={null}
memvid audit <FILE> <QUESTION> [OPTIONS]
```
### Options
| Option | Description | Default |
| --------------------------- | ---------------------------------- | ------- |
| `--out <PATH>`, `-o <PATH>` | Output file | stdout |
| `--format <FORMAT>` | Format: `text`, `markdown`, `json` | text |
| `--top-k <K>` | Sources to retrieve | 10 |
| `--snippet-chars <N>` | Max chars per snippet | 500 |
| `--mode <MODE>` | Retrieval mode | hybrid |
| `--scope <PREFIX>` | Scope filter | None |
| `--start <DATE>` | Start date | None |
| `--end <DATE>` | End date | None |
| `--use-model <MODEL>` | LLM for synthesis | None |
### Examples
```bash theme={null}
# Generate audit report
memvid audit project.mv2 "budget decisions" -o audit.md --format markdown
# JSON audit for compliance
memvid audit project.mv2 "data access" -o audit.json --format json
# With LLM summary
memvid audit project.mv2 "key decisions" --use-model openai -o report.md
```
### Response (Markdown)
```markdown theme={null}
# Audit Report: Budget Decisions
Generated: 2024-01-20T15:30:00Z
Query: "budget decisions"
Sources: 8
## Findings
### Source 1: Q4 Budget Meeting
- **URI**: file:///meeting-q4.txt
- **Date**: 2024-01-15
- **Relevance**: 0.95
> The team decided to increase the marketing budget by 15%...
### Source 2: Finance Review
- **URI**: file:///finance-review.pdf
- **Date**: 2024-01-18
- **Relevance**: 0.87
> Budget allocation approved with amendments...
```
***
## Real-World Examples
### Documentation Search
```bash theme={null}
# Find installation instructions
memvid find docs.mv2 --query "how to install" --mode auto
# Find API endpoint documentation
memvid find docs.mv2 --query "POST /users endpoint" --mode lex
# Ask about configuration (local model - private)
memvid ask docs.mv2 \
--question "What environment variables are required?" \
--use-model "ollama:qwen2.5:1.5b"
```
### Codebase Search
```bash theme={null}
# Find function implementations
memvid find code.mv2 --query "handleUserLogin" --mode lex
# Find error handling patterns
memvid find code.mv2 --query "try catch error handling" --mode auto
# Understand code architecture (local model - keeps code private)
memvid ask code.mv2 \
--question "How does the authentication flow work?" \
--use-model "ollama:qwen2.5:1.5b"
```
### Research Search
```bash theme={null}
# Find papers on specific topic
memvid find papers.mv2 --query "transformer architecture attention" --mode sem
# Summarize findings (use larger model for complex analysis)
memvid ask papers.mv2 \
--question "What are the main approaches to reducing transformer inference cost?" \
--top-k 15 \
--use-model "ollama:qwen2.5:3b"
```
***
## Troubleshooting
### No Results Found
**Solutions:**
* Try different search mode: `--mode sem` or `--mode lex`
* Broaden your query terms
* Check that documents have been ingested: `memvid stats knowledge.mv2`
* Verify lexical index exists: `memvid doctor knowledge.mv2 --rebuild-lex-index`
### Low Relevance Scores
**Solutions:**
* Use semantic search for natural language queries
* Use lexical search for exact technical terms
* Add more context to your query
* Increase `--top-k` to see more results
### LLM Errors
```bash theme={null}
Error: Failed to contact LLM provider
```
**Solutions:**
**Option 1: Use local Ollama model (recommended)**
```bash theme={null}
# Install Ollama
brew install ollama # macOS
# or: curl -fsSL https://ollama.com/install.sh | sh # Linux
# Start Ollama server
ollama serve &
# Pull a model
ollama pull qwen2.5:1.5b
# Use with memvid
memvid ask knowledge.mv2 --question "..." --use-model "ollama:qwen2.5:1.5b"
```
**Option 2: Set API keys for cloud providers**
```bash theme={null}
# OpenAI
export OPENAI_API_KEY=your-key
memvid ask knowledge.mv2 --question "..." --use-model openai
# Gemini
export GEMINI_API_KEY=your-key
memvid ask knowledge.mv2 --question "..." --use-model "gemini-2.0-flash"
# Anthropic
export ANTHROPIC_API_KEY=your-key
memvid ask knowledge.mv2 --question "..." --use-model claude
```
<Info>
See [Local Models with Ollama](/concepts/local-models) for detailed setup instructions.
</Info>
***
## Next Steps
<CardGroup>
<Card title="Timeline & View" icon="clock" href="/cli/timeline-and-view">
Explore documents by time and view frame details
</Card>
<Card title="Maintenance" icon="wrench" href="/cli/maintenance-and-tickets">
Verify integrity and manage your files
</Card>
</CardGroup>
# Tickets & Capacity
Source: https://docs.memvid.com/cli/tickets-and-capacity
Manage access tickets, capacity limits, and subscription plans
Commands for managing your memory file's capacity and syncing tickets from the dashboard.
***
## Overview
Memvid uses a ticket system for capacity management:
* **Tickets** authorize storage capacity
* **Binding** connects a file to a dashboard memory
* **Plans** determine your capacity limits
```mermaid theme={null}
flowchart LR
D[Dashboard] -->|Upgrade Plan| T[New Ticket]
T -->|memvid tickets sync| M[knowledge.mv2]
M -->|Increased Capacity| S[Store More Data]
```
***
## memvid tickets sync
Synchronize tickets from the dashboard.
### Synopsis
```bash theme={null}
memvid tickets sync <FILE> --memory-id <ID> [OPTIONS]
```
### Arguments
| Argument | Description |
| -------- | --------------------- |
| `FILE` | Path to the .mv2 file |
### Options
| Option | Description | Default |
| --------------------- | ------------------------------------ | -------- |
| `--memory-id <ID>` | Memory ID (UUID or 24-char ObjectId) | Required |
| `--json` | JSON output | Disabled |
| `--lock-timeout <MS>` | Wait timeout for lock | 250ms |
| `--force` | Force lock takeover | Disabled |
### Examples
```bash theme={null}
# Sync with dashboard memory
memvid tickets sync project.mv2 --memory-id mem_abc123
# With environment API key
MEMVID_API_KEY=mv_live_xxx memvid tickets sync project.mv2 --memory-id mem_abc123
```
### Response
```
Syncing tickets for project.mv2...
Memory ID: mem_abc123
Memory Name: Production Memory
Ticket applied:
Issuer: memvid.com
Sequence: 43 (was: 42)
Capacity: 1 GB
Expires in: 30d
Binding status:
Already bound: yes
Sync complete.
```
### JSON Output
```json theme={null}
{
"success": true,
"memory_id": "mem_abc123",
"already_bound": true,
"ticket": {
"issuer": "memvid.com",
"seq_no": 43,
"capacity_bytes": 1073741824,
"expires_in_secs": 2592000
}
}
```
***
## memvid tickets apply
Apply a ticket to a memory file.
### Synopsis
```bash theme={null}
memvid tickets apply <FILE> --memory-id <ID> [OPTIONS]
```
### Options
| Option | Description |
| ------------------ | --------------------- |
| `--memory-id <ID>` | Memory ID (required) |
| `--from-api` | Fetch ticket from API |
| `--json` | JSON output |
### Examples
```bash theme={null}
# Apply ticket from API
memvid tickets apply project.mv2 --memory-id mem_abc123 --from-api
```
***
## memvid tickets issue
Issue a new ticket (for self-hosted/enterprise).
### Synopsis
```bash theme={null}
memvid tickets issue <FILE> [OPTIONS]
```
### Options
| Option | Description | Default |
| --------------------- | --------------------- | -------- |
| `--issuer <ISSUER>` | Ticket issuer | Required |
| `--seq <SEQ>` | Sequence number | Required |
| `--expires-in <SECS>` | Expiration in seconds | None |
| `--capacity <BYTES>` | Capacity in bytes | None |
| `--json` | JSON output | Disabled |
### Examples
```bash theme={null}
# Issue enterprise ticket
memvid tickets issue project.mv2 \
--issuer "enterprise.example.com" \
--seq 1 \
--expires-in 31536000 \
--capacity 10737418240
```
***
## memvid tickets list
Show current ticket information.
### Synopsis
```bash theme={null}
memvid tickets list <FILE> [OPTIONS]
```
### Options
| Option | Description |
| -------- | ----------- |
| `--json` | JSON output |
### Examples
```bash theme={null}
memvid tickets list project.mv2
```
### Response
```
Ticket for project.mv2
Status: ACTIVE
Ticket Details:
Issuer: memvid.com
Sequence: 43
Capacity: 1 GB
Expires: 2024-02-20T10:30:00Z (30d remaining)
Usage:
Current size: 125.6 MB
Capacity: 1 GB
Available: 898.4 MB
Usage: 12.2%
```
### JSON Output
```json theme={null}
{
"path": "project.mv2",
"active": true,
"ticket": {
"issuer": "memvid.com",
"seq_no": 43,
"capacity_bytes": 1073741824,
"expires_at": "2024-02-20T10:30:00Z",
"expires_in_secs": 2592000
},
"usage": {
"current_bytes": 131691315,
"capacity_bytes": 1073741824,
"available_bytes": 942050509,
"usage_percent": 12.2
}
}
```
***
## memvid tickets revoke
Clear ticket metadata from a file.
### Synopsis
```bash theme={null}
memvid tickets revoke <FILE> [OPTIONS]
```
### Options
| Option | Description |
| --------------------- | ------------------- |
| `--json` | JSON output |
| `--lock-timeout <MS>` | Wait timeout |
| `--force` | Force lock takeover |
### Examples
```bash theme={null}
memvid tickets revoke project.mv2
```
***
## memvid plan show
Show current plan and capacity.
### Synopsis
```bash theme={null}
memvid plan show [OPTIONS]
```
### Options
| Option | Description |
| -------- | ----------- |
| `--json` | JSON output |
### Examples
```bash theme={null}
memvid plan show
```
### Response
```
Memvid Plan
Plan: Developer
Status: Active
Limits:
Memories: 10 / 25
Total capacity: 2.5 GB / 10 GB
API calls: 45,231 / 100,000 (monthly)
Features:
- Semantic search: included
- LLM enrichment: included
- Table extraction: included
- Encryption: included
- Priority support: not included
Billing:
Next renewal: 2024-02-01
Amount: $29/month
Upgrade: https://memvid.com/dashboard/plan
```
### JSON Output
```json theme={null}
{
"plan": "developer",
"status": "active",
"limits": {
"memories": {"used": 10, "limit": 25},
"capacity_bytes": {"used": 2684354560, "limit": 10737418240},
"api_calls": {"used": 45231, "limit": 100000, "period": "monthly"}
},
"features": {
"semantic_search": true,
"llm_enrichment": true,
"table_extraction": true,
"encryption": true,
"priority_support": false
},
"billing": {
"next_renewal": "2024-02-01",
"amount_cents": 2900,
"currency": "usd"
}
}
```
***
## memvid plan sync
Sync plan ticket from dashboard.
### Synopsis
```bash theme={null}
memvid plan sync [OPTIONS]
```
### Options
| Option | Description |
| -------- | ----------- |
| `--json` | JSON output |
### Examples
```bash theme={null}
MEMVID_API_KEY=mv_live_xxx memvid plan sync
```
***
## memvid plan clear
Clear cached plan ticket.
### Synopsis
```bash theme={null}
memvid plan clear [OPTIONS]
```
### Options
| Option | Description |
| -------- | ----------- |
| `--json` | JSON output |
***
## Bind and Unbind Memory
### Bind Memory
Associate a local memory file with a dashboard memory ID:
```bash theme={null}
memvid bind knowledge.mv2 --memory-id YOUR_MEMORY_ID
```
After binding, you can sync tickets without specifying the memory ID each time.
### Unbind Memory
Remove the dashboard association from a local file:
```bash theme={null}
memvid unbind knowledge.mv2
```
<Warning>
After unbinding, you'll need to rebind before syncing tickets again.
</Warning>
***
## Best Practices
### Initial Setup
```bash theme={null}
# 1. Create memory file
memvid create project.mv2
# 2. Sync with dashboard (binds and applies ticket)
MEMVID_API_KEY=mv_live_xxx memvid tickets sync project.mv2 --memory-id mem_abc123
# 3. Verify binding
memvid binding project.mv2
```
### Monitoring Capacity
```bash theme={null}
# Check current usage
memvid tickets list project.mv2
# Check plan limits
memvid plan show
# Script to alert on low capacity
USAGE=$(memvid tickets list project.mv2 --json | jq '.usage.usage_percent')
if (( $(echo "$USAGE > 80" | bc -l) )); then
echo "Warning: Memory at ${USAGE}% capacity"
fi
```
### Refreshing Tickets
```bash theme={null}
# Tickets are refreshed automatically on sync
memvid tickets sync project.mv2 --memory-id mem_abc123
# Force refresh (re-fetch from API)
memvid tickets apply project.mv2 --memory-id mem_abc123 --from-api
```
### Enterprise Self-Hosting
```bash theme={null}
# Issue tickets with your own signing key
memvid tickets issue project.mv2 \
--issuer "internal.company.com" \
--seq 1 \
--expires-in 31536000 \
--capacity 107374182400 # 100 GB
# Verify ticket
memvid tickets list project.mv2
```
### Capacity Planning
```bash theme={null}
# Estimate storage needs
# ~1 MB per 1000 documents (text only)
# ~10 MB per 1000 documents (with embeddings)
# ~50 MB per 1000 documents (with PDFs)
# Check current size
memvid stats project.mv2 --json | jq '.size_bytes'
# Project growth
# size_per_doc * docs_per_month * months
```
***
## Environment Variables
| Variable | Description |
| ---------------- | --------------------------------------------- |
| `MEMVID_API_KEY` | API key for ticket sync |
| `MEMVID_API_URL` | Custom API endpoint (default: api.memvid.com) |
```bash theme={null}
# Set API key globally
export MEMVID_API_KEY=mv_live_xxx
# Then sync without --api-key flag
memvid tickets sync knowledge.mv2 --memory-id abc-123
```
***
## Troubleshooting
### CapacityExceeded Error
If you see this error when adding content:
```
Error: CapacityExceeded: Memory file exceeded capacity limit
```
**Solutions:**
1. **Upgrade and sync:**
```bash theme={null}
# After upgrading in dashboard
memvid tickets sync knowledge.mv2 --memory-id YOUR_ID
```
2. **Delete old content:**
```bash theme={null}
memvid delete knowledge.mv2 --frame-id 42 --yes
memvid doctor knowledge.mv2 --vacuum
```
3. **Create a new memory:**
```bash theme={null}
mv knowledge.mv2 knowledge-archive.mv2
memvid create knowledge.mv2
```
### Ticket Sync Failed
If sync fails:
```bash theme={null}
# Check your API key is set
echo $MEMVID_API_KEY
# Verify memory ID is correct
memvid tickets list knowledge.mv2
# Try with explicit API key
MEMVID_API_KEY=mv_live_xxx memvid tickets sync knowledge.mv2 --memory-id YOUR_ID
```
### Invalid Ticket Error
If you see `TicketInvalid` or `TicketReplay`:
```bash theme={null}
# Re-sync to get fresh ticket
memvid tickets sync knowledge.mv2 --memory-id YOUR_ID
```
***
## Next Steps
<CardGroup>
<Card title="Maintenance & Repair" icon="wrench" href="/cli/maintenance-and-tickets">
Verify and repair memory files
</Card>
<Card title="Error Reference" icon="circle-exclamation" href="/errors/reference">
All error codes and solutions
</Card>
</CardGroup>
# Timeline, View & Stats
Source: https://docs.memvid.com/cli/timeline-and-view
Inspect your memories with timeline, view, and stats commands
Commands for inspecting and navigating your memory files.
***
## Timeline
Browse documents chronologically using the time index.
### Basic Usage
```bash theme={null}
memvid timeline knowledge.mv2
```
### Options
| Option | Description | Default |
| -------------------- | ------------------------------------------ | --------------- |
| `--limit` | Maximum entries to return | 50 |
| `--since` | Start timestamp (Unix) | None |
| `--until` | End timestamp (Unix) | None |
| `--on <PHRASE>` | Temporal phrase filter (e.g., "last week") | None |
| `--tz <ZONE>` | Timezone for temporal phrases | America/Chicago |
| `--anchor <RFC3339>` | Anchor date for resolution | Now |
| `--reverse` | Reverse chronological order | false |
| `--json` | Output as JSON | false |
### Time-Travel Options
| Option | Description |
| ---------------------- | ----------------------------- |
| `--as-of-frame ID` | View timeline as of frame ID |
| `--as-of-ts TIMESTAMP` | View timeline as of timestamp |
### Examples
```bash theme={null}
# Get recent entries
memvid timeline knowledge.mv2 --limit 20
# Reverse order (oldest first)
memvid timeline knowledge.mv2 --reverse
# Filter by time range
memvid timeline knowledge.mv2 --since 1704067200 --until 1706745600
# Frames from last week
memvid timeline knowledge.mv2 --on "last week"
# Frames from specific timezone
memvid timeline knowledge.mv2 --on "yesterday" --tz "America/New_York"
# JSON output for scripting
memvid timeline knowledge.mv2 --json
# Time-travel view
memvid timeline knowledge.mv2 --as-of-frame 50
```
**Output:**
```
Timeline (5 entries):
[1] 2024-11-15 10:30:00 - API Documentation
mv2://docs/api.md
[2] 2024-11-14 15:45:00 - Meeting Notes
mv2://notes/meeting-2024-11-14.md
[3] 2024-11-14 09:00:00 - Project Roadmap
mv2://planning/roadmap.md
```
**JSON Output:**
```json theme={null}
{
"entries": [
{
"frame_id": 1,
"timestamp": 1731667800,
"preview": "API Documentation for v2...",
"uri": "mv2://docs/api.md"
},
{
"frame_id": 2,
"timestamp": 1731600300,
"preview": "Meeting Notes from standup...",
"uri": "mv2://notes/meeting-2024-11-14.md"
}
]
}
```
***
## View
Inspect individual documents and their metadata.
### Basic Usage
```bash theme={null}
memvid view knowledge.mv2 --frame-id 124
```
### Selection Options
| Option | Description |
| --------------- | -------------------- |
| `--frame-id ID` | Document ID to view |
| `--uri URI` | Document URI to view |
### Output Options
| Option | Description |
| ----------- | ---------------------------------- |
| `--json` | Output as JSON |
| `--binary` | Raw binary output |
| `--preview` | Preview mode (for media) |
| `--play` | Play/stream mode (for audio/video) |
### Pagination (Text Content)
| Option | Description | Default |
| ------------- | ------------------- | ------- |
| `--page N` | Page number | 1 |
| `--page-size` | Characters per page | Auto |
### Media Playback
| Option | Description |
| -------------------------- | ---------------------------- |
| `--preview-start HH:MM:SS` | Video preview start time |
| `--preview-end HH:MM:SS` | Video preview end time |
| `--start-seconds` | Playback start (with --play) |
| `--end-seconds` | Playback end (with --play) |
### Examples
```bash theme={null}
# View by frame ID
memvid view knowledge.mv2 --frame-id 124
# View by URI
memvid view knowledge.mv2 --uri "mv2://docs/api.md"
# JSON output with full metadata
memvid view knowledge.mv2 --frame-id 124 --json
# Preview an image
memvid view knowledge.mv2 --uri "mv2://images/diagram.png" --preview
# Play audio/video
memvid view knowledge.mv2 --frame-id 12 --play
# Play specific segment
memvid view knowledge.mv2 --frame-id 12 --play --start-seconds 30 --end-seconds 60
# Paginate long text
memvid view knowledge.mv2 --frame-id 5 --page 2 --page-size 2000
# Raw binary output
memvid view knowledge.mv2 --frame-id 10 --binary > output.pdf
```
**Output:**
```
Frame: 124
Title: API Documentation
URI: mv2://docs/api.md
Track: documentation
Kind: markdown
Timestamp: 2024-11-15T10:30:00Z
Tags: version=2.0, category=reference
Labels: api, public
Content:
The API provides endpoints for authentication, user management,
and data retrieval. All endpoints require authentication via...
```
**JSON Output:**
```json theme={null}
{
"frame_id": 124,
"title": "API Documentation",
"uri": "mv2://docs/api.md",
"track": "documentation",
"kind": "markdown",
"timestamp": 1731667800,
"tags": {
"version": "2.0",
"category": "reference"
},
"labels": ["api", "public"],
"checksum": "a3b2c1d4e5f6...",
"payload_length": 15234,
"content": "The API provides endpoints for..."
}
```
***
## Update
Modify frame metadata (not content).
### Basic Usage
```bash theme={null}
memvid update knowledge.mv2 --frame-id 124 --title "New Title"
```
### Selection
| Option | Description |
| --------------- | ------------------- |
| `--frame-id ID` | Frame ID to update |
| `--uri URI` | Frame URI to update |
### Updatable Fields
| Option | Description |
| ---------------------- | ------------------------------------- |
| `--set-uri URI` | Change the URI |
| `--title TITLE` | Update title |
| `--timestamp TS` | Update timestamp |
| `--track TRACK` | Update track |
| `--kind KIND` | Update kind |
| `--tag KEY=VALUE` | Add/update tags |
| `--label LABEL` | Add labels |
| `--metadata JSON` | Add metadata |
| `--input PATH` | Replace payload |
| `--vector-compression` | Recompute embeddings with compression |
### Examples
```bash theme={null}
# Update title
memvid update knowledge.mv2 --frame-id 124 --title "Updated API Docs"
# Add tags
memvid update knowledge.mv2 --uri "mv2://docs/api.md" \
--tag "reviewed=true" \
--tag "version=2.1"
# Move to different track
memvid update knowledge.mv2 --frame-id 124 --track "archived"
# Replace content and recompute embeddings
memvid update knowledge.mv2 --frame-id 124 \
--input ./new-content.md \
--vector-compression
```
***
## Delete
Remove frames from the memory.
### Basic Usage
```bash theme={null}
memvid delete knowledge.mv2 --frame-id 42
```
### Options
| Option | Description |
| --------------- | ------------------------ |
| `--frame-id ID` | Frame ID to delete |
| `--uri URI` | Frame URI to delete |
| `--yes` | Skip confirmation prompt |
| `--json` | JSON output |
### Examples
```bash theme={null}
# Delete with confirmation
memvid delete knowledge.mv2 --frame-id 42
# Delete without confirmation
memvid delete knowledge.mv2 --uri "mv2://old/doc.md" --yes
# Reclaim space after deletions
memvid doctor knowledge.mv2 --vacuum
```
***
## Stats
Get statistics about your memory file.
### Basic Usage
```bash theme={null}
memvid stats knowledge.mv2
```
### Options
| Option | Description |
| -------- | -------------- |
| `--json` | Output as JSON |
### Examples
```bash theme={null}
# Human-readable output
memvid stats knowledge.mv2
# JSON for scripting
memvid stats knowledge.mv2 --json
```
**Output:**
```
Memory: knowledge.mv2
Documents: 150
Active Frames: 148
Size: 52.4 MB
Capacity: 1.0 GB
Utilization: 5.2%
Indices:
Lexical: Yes
Vector: Yes
Time: Yes
Storage Breakdown:
Payloads: 48.2 MB
Lex Index: 2.1 MB
Vec Index: 1.8 MB
Time Index: 0.3 MB
Compression:
Ratio: 78%
Saved: 14.2 MB
```
**JSON Output:**
```json theme={null}
{
"frame_count": 150,
"active_frame_count": 148,
"size_bytes": 54945587,
"capacity_bytes": 1073741824,
"storage_utilisation_percent": 5.2,
"has_lex_index": true,
"has_vec_index": true,
"has_time_index": true,
"payload_bytes": 50545664,
"lex_index_bytes": 2202009,
"vec_index_bytes": 1887436,
"time_index_bytes": 310478,
"compression_ratio_percent": 78,
"saved_bytes": 14876543
}
```
***
## Open
Inspect memory file metadata and manifests.
```bash theme={null}
memvid open knowledge.mv2
```
### Options
| Option | Description |
| -------- | -------------- |
| `--json` | Output as JSON |
Shows detailed information about the file structure, including frame count, index status, and internal metadata.
***
## Lock Inspection
### Who
Check who holds the write lock:
```bash theme={null}
memvid who knowledge.mv2
```
**Output:**
```
Lock held by:
PID: 12345
Acquired: 2024-11-15T10:30:00Z
Heartbeat: Active
```
### Nudge
Request the active writer to release when safe:
```bash theme={null}
memvid nudge knowledge.mv2
```
This sends a signal to the writer process to checkpoint and release the lock.
***
## Real-World Examples
### Daily Review
```bash theme={null}
# See what was added today
memvid timeline notes.mv2 --since $(date -d "today 00:00" +%s) --limit 50
# View specific entry details
memvid view notes.mv2 --frame-id 42 --json
```
### Archiving Old Content
```bash theme={null}
# Find old entries
memvid timeline archive.mv2 --until $(date -d "6 months ago" +%s)
# Update their track
for id in 1 2 3 4 5; do
memvid update archive.mv2 --frame-id $id --track "archived"
done
```
### Media Library
```bash theme={null}
# Browse media timeline
memvid timeline media.mv2 --limit 20
# Preview an image
memvid view media.mv2 --uri "mv2://photos/vacation.jpg" --preview
# Play a video clip
memvid view media.mv2 --frame-id 15 --play --start-seconds 10 --end-seconds 30
```
***
## Next Steps
<CardGroup>
<Card title="Maintenance" icon="wrench" href="/cli/maintenance-and-tickets">
Repair files and verify integrity
</Card>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Query your memories
</Card>
</CardGroup>
# Memvid vs Vector Databases
Source: https://docs.memvid.com/comparisons/vector-databases
An honest comparison of Memvid with Pinecone, ChromaDB, and other vector databases
You're probably here because you've used vector databases before and wondering how Memvid is different. Here's an honest comparison.
<Note>
**TL;DR**: Memvid uses **Smart Frames**, a superset of vector databases. You get lexical search, semantic search, temporal queries, and entity extraction in one file.
</Note>
***
## The 30-Second Comparison
| | Pinecone | ChromaDB | Memvid |
| ----------------------- | --------------------------- | -------------------- | ------------------------------------------------------- |
| **Setup time** | 7.4s (API provisioning) | 2 min | **145ms** |
| **Search latency** | 267ms (network + embedding) | \~500ms | **24ms** ⚡ |
| **Embeddings required** | Yes, always | Yes, always | Optional |
| **Works offline** | No | Yes | **Yes** |
| **File count** | Cloud-managed | Multiple files | **1 file** |
| **Infrastructure** | Managed cloud | Self-hosted or cloud | **None** |
| **Pricing** | \$70/mo+ | Free / paid cloud | **Free** |
| **Search modes** | Vector only | Vector only | **Smart Frames (Lexical + Vector + Temporal + Entity)** |
| **Time-travel queries** | No | No | **Yes** |
| **Entity extraction** | No | No | **Yes (built-in)** |
<Info>
**Search is 11x faster** because Memvid doesn't require network round-trips to embedding APIs or cloud vector databases. Your data, your machine, instant results.
</Info>
***
## Real-World Benchmark
We ran a head-to-head benchmark with 1,000 documents using native SDKs. Here's what we measured:
### Performance Results (1,000 Documents)
| Metric | Memvid | Pinecone | LanceDB | Winner |
| ------------- | -------- | -------- | ------- | ------------------- |
| **Setup** | 145ms | 7.4s | 158ms | **Memvid (51x)** |
| **Ingestion** | 1.6m | 3.3m | 6.1s | LanceDB |
| **Search** | **24ms** | 267ms | 506ms | **Memvid (11-21x)** |
| **Storage** | 4.9 MB | Cloud | Cloud | **Memvid** |
### Search Latency Breakdown
| System | Avg Search | vs Memvid |
| ---------- | ---------: | ---------: |
| **Memvid** | **24ms** | - |
| Pinecone | 267ms | 11x slower |
| LanceDB | 506ms | 21x slower |
**Why is Memvid search so fast?** No network calls. Vector databases require:
1. Network round-trip to embedding API (to embed your query)
2. Network round-trip to vector database (to search)
3. Query embedding computation time
Memvid runs entirely on your machine using **Smart Frames**, pre-indexed with Tantivy full-text search, temporal indexes, and entity graphs. Your query goes straight to the index.
### Why Ingestion Takes Longer (And Why That's OK)
Memvid's ingestion is slower than pure vector databases because it does **more work**:
* **Auto-tagging**: Automatic topic detection for every document
* **Date extraction**: Temporal entity recognition for timeline queries
* **Triplet extraction**: Subject-Predicate-Object knowledge graph building
* **Full-text indexing**: Tantivy BM25 for instant lexical search
* **Timeline indexing**: Temporal index for time-travel queries
These features enable richer queries and memory extraction. Ingestion is a **one-time cost**. Search latency is what matters for production use.
### Projected at Scale (10,000 Documents)
| Metric | Memvid | Pinecone | LanceDB |
| ------------- | ---------- | -------- | ------- |
| **Setup** | \~145ms | \~7.4s | \~158ms |
| **Ingestion** | \~16m | \~33m | \~1m |
| **Search** | **\~24ms** | \~267ms | \~506ms |
*Search latency remains constant regardless of dataset size thanks to efficient indexing.*
### Search Accuracy Comparison
Memvid uses **Smart Frames**, not just keyword search. Each frame is enriched with auto-tagging, temporal indexing, entity extraction, and optional embeddings.
| Query Type | Memvid (Smart Frames) | Vector DBs | Winner |
| ----------------------------------------- | --------------------------- | ------------------------- | ---------- |
| **Exact match** `"handleAuthentication"` | ✅ 100% precision | ❌ Returns "login", "auth" | **Memvid** |
| **Error codes** `"ERROR_CODE_404"` | ✅ Exact match | ❌ Semantic confusion | **Memvid** |
| **Temporal** `"meetings last week"` | ✅ Timeline index | ❌ No temporal awareness | **Memvid** |
| **Entity state** `"Alice's current role"` | ✅ Knowledge graph | ❌ No entity tracking | **Memvid** |
| **Names** `"John Smith contract"` | ✅ Exact + entity extraction | ❌ Names get fuzzy | **Memvid** |
| **Semantic** `"reduce costs"` | ✅ Hybrid mode | ✅ Finds "cut expenses" | **Tie** |
| **Conceptual** `"happy moments"` | ✅ Hybrid mode | ✅ Finds "joyful" | **Tie** |
**Smart Frames give you the best of all worlds:**
```python theme={null}
# Exact lexical search (instant)
results = mem.find("handleAuthentication", k=5)
# Temporal queries (unique to Memvid)
results = mem.timeline("2024-01-01", "2024-01-31")
# Entity state (knowledge graph)
alice = mem.state("Alice") # {employer: "Anthropic", role: "Engineer"}
# Semantic search (when you need it)
results = mem.find("cost reduction strategies", mode="vec")
# Hybrid search (best of both)
results = mem.find("budget optimization", mode="auto")
```
**The reality**: Vector databases only do one thing: semantic similarity. Memvid does **lexical + semantic + temporal + entity extraction** in a single file.
<Tip>
**The takeaway**: If you're building something that needs fast, reliable search, and you're tired of paying for API calls and managing cloud infrastructure, Memvid gets you there with a single file.
</Tip>
***
## The Fundamental Difference
Traditional vector databases assume you need embeddings for everything:
```mermaid theme={null}
flowchart LR
subgraph Traditional[Traditional Vector DB]
A[Your Data] --> B[Embedding API]
B --> C[Vector Store]
C --> D[Search]
end
```
**Problems with this approach:**
* Can't search until embeddings are computed
* API calls cost money and add latency
* Embedding model updates break your index
* "Error 404" doesn't match "error 404" (semantic ≠ exact)
* No temporal awareness: can't query "last week's meetings"
* No entity tracking: can't ask "what's Alice's current role?"
Memvid uses **Smart Frames**:
```mermaid theme={null}
flowchart LR
subgraph Memvid[Memvid Smart Frames]
E[Your Data] --> F[Smart Frame]
F --> G[Lexical Index]
F --> H[Timeline Index]
F --> I[Entity Graph]
F --> J[Vector Index]
G & H & I & J --> K[Search]
end
```
**Instant search + rich capabilities.** Your data is searchable the moment you add it, with temporal queries, entity extraction, and optional semantic search.
***
## Setup Comparison
### Pinecone
```python theme={null}
# 1. Sign up at pinecone.io
# 2. Create a project
# 3. Get API key
# 4. Install SDK
pip install pinecone-client
# 5. Initialize
import pinecone
pinecone.init(api_key="your-api-key", environment="us-west1-gcp")
# 6. Create index (wait for provisioning...)
pinecone.create_index("my-index", dimension=1536, metric="cosine")
# 7. Wait for index to be ready
import time
while not pinecone.describe_index("my-index").status["ready"]:
time.sleep(1)
# 8. Connect to index
index = pinecone.Index("my-index")
# 9. Now you need to embed your data before inserting...
```
**Measured setup time: 7.4 seconds** (plus embedding time for each document)
### ChromaDB
```python theme={null}
# 1. Install
pip install chromadb
# 2. Initialize
import chromadb
client = chromadb.Client()
# 3. Create collection
collection = client.create_collection("my-collection")
# 4. Add documents (ChromaDB embeds automatically, but still takes time)
collection.add(
documents=["doc1", "doc2", "doc3"],
ids=["id1", "id2", "id3"]
)
# This step embeds all documents - can take minutes for large datasets
```
**Time to first search: 2-5 minutes** (embedding time)
### Memvid
```bash theme={null}
npm install -g memvid-cli
memvid create knowledge.mv2
echo "Your document content" | memvid put knowledge.mv2
memvid find knowledge.mv2 --query "document"
```
```python theme={null}
from memvid_sdk import create
mem = create('knowledge.mv2')
mem.put(title="Doc", label="docs", metadata={}, text="Your document content")
results = mem.find("document", k=5)
```
**Measured setup time: 145ms.** Search in milliseconds.
***
## Search Quality Comparison
### Smart Frames: Best of All Worlds
Memvid's Smart Frames combine multiple search capabilities that vector databases can't match:
| Capability | Vector DBs | Memvid Smart Frames |
| ----------------------- | ----------------- | --------------------- |
| **Exact match** | ❌ Fuzzy by design | ✅ 100% precision |
| **Semantic similarity** | ✅ Core feature | ✅ Optional embeddings |
| **Temporal queries** | ❌ Not supported | ✅ Timeline index |
| **Entity tracking** | ❌ Not supported | ✅ Knowledge graph |
| **Hybrid search** | ❌ Pick one mode | ✅ Auto-selects best |
**Code search example:**
```bash theme={null}
# Finding a specific function (exact match)
memvid find codebase.mv2 --query "handleWebSocketConnection"
# → Returns exact matches instantly
# With vector search, you might get:
# - "processNetworkRequest" (semantically similar, wrong function)
# - "WebSocket" documentation (not the function)
# - "connectionHandler" (close but not exact)
```
### Memvid Handles Semantic Too
When you need conceptual queries, add embeddings:
```bash theme={null}
# Enable semantic search
memvid doctor knowledge.mv2 --rebuild-vec-index
# Semantic mode
memvid find knowledge.mv2 --query "cost reduction strategies" --mode vec
# Hybrid mode (auto-selects best approach)
memvid find knowledge.mv2 --query "budget optimization" --mode auto
```
| Query | Lexical Mode | Semantic Mode | Hybrid Mode |
| ----------------- | ----------------- | -------------------- | -------------- |
| `"reduce costs"` | Exact phrase only | Finds "cut expenses" | ✅ Best of both |
| `"handleAuth"` | ✅ Exact match | Fuzzy results | ✅ Exact match |
| `"happy moments"` | Literal only | Finds "joyful" | ✅ Best of both |
***
## Infrastructure Comparison
### Pinecone Architecture (Serverless, 2025)
```mermaid theme={null}
flowchart TB
subgraph App[Your Application]
A[App Code]
end
subgraph Cloud[Pinecone Cloud]
B[API Gateway]
C[Multi-tenant Compute]
D[Blob Storage]
E[Freshness Layer]
end
A -->|HTTPS| B
B --> C
C --> D
C --> E
```
**Requires:**
* Internet connection
* API key management
* Vendor lock-in
* Usage-based billing
### ChromaDB Architecture
```mermaid theme={null}
flowchart TB
subgraph App[Your Application]
A[App Code]
end
subgraph Server[ChromaDB Server]
B[SQLite DB]
C[Vector Index]
D[Parquet Files]
end
A --> Server
B -.-> E[metadata.db]
C -.-> F[index.bin]
D -.-> G[data.parquet]
```
**Requires:**
* Multiple files to manage
* Server process running
* Careful backup strategy
### Memvid Architecture
```mermaid theme={null}
flowchart TB
subgraph App[Your Application]
A[App Code]
end
A --> B[knowledge.mv2]
```
**That's it.** One file. Copy it, sync it, git commit it.
```bash theme={null}
# Share your entire knowledge base
cp knowledge.mv2 /team/shared/
# Version control it
git add knowledge.mv2 && git commit -m "Updated docs"
# Backup
cp knowledge.mv2 knowledge.mv2.backup
```
***
## Cost Comparison
### Pinecone Pricing (as of 2025)
| Tier | Monthly Cost | Vectors | Queries |
| ---------- | ------------ | --------- | --------- |
| Free | \$0 | 100K | Limited |
| Standard | \$70+ | 1M+ | Unlimited |
| Enterprise | Custom | Unlimited | Unlimited |
**Plus:** Embedding API costs (\$0.0001+ per 1K tokens)
### ChromaDB Pricing
| Deployment | Cost |
| ------------ | -------------------------- |
| Self-hosted | Free (your infrastructure) |
| Chroma Cloud | \$30+/mo |
**Plus:** Embedding API costs (unless using local models)
### Memvid Pricing
| Tier | Cost |
| ------------------- | ---------------------- |
| Open source | **Free forever** |
| Memvid Cloud (sync) | Free tier + paid plans |
Embeddings are optional.
### Real Cost Example: 1M Documents
| | Pinecone | ChromaDB | Memvid |
| ------------------ | --------------- | -------------------- | ------- |
| Storage | \$70/mo | \$30/mo or self-host | **\$0** |
| Embedding (OpenAI) | \~\$50 one-time | \~\$50 one-time | **\$0** |
| Monthly API calls | Included | Included | **\$0** |
| **Year 1 Total** | **\$890+** | **\$410+** | **\$0** |
<Info>
**Zero API calls means zero cost.** In our benchmark with 1,000 documents, Pinecone and LanceDB made 1,005 API calls each (1,000 for document embeddings + 5 for query embeddings). Memvid made **zero** because it doesn't need embeddings to search.
</Info>
***
## Feature Comparison
### What Memvid Has That Vector DBs Don't
<CardGroup>
<Card title="Time-Travel Queries" icon="clock">
Query your data as it existed at any point in time:
```python theme={null}
# What did we know last week?
results = mem.find("budget", as_of_frame=1000)
```
</Card>
<Card title="Entity Extraction" icon="brain">
Built-in entity extraction and relationship graphs:
```python theme={null}
alice = mem.state("Alice")
# {employer: "Anthropic", role: "Engineer"}
```
</Card>
<Card title="Single-File Portability" icon="file">
Everything in one `.mv2` file:
```bash theme={null}
scp knowledge.mv2 user@server:/data/
```
</Card>
<Card title="Crash Recovery" icon="shield">
Embedded WAL ensures zero data loss:
```bash theme={null}
# Power failure? No problem.
memvid verify knowledge.mv2
# ✓ File integrity verified
```
</Card>
</CardGroup>
### What Vector DBs Have That Memvid Approaches Differently
| Feature | Vector DBs | Memvid |
| ----------------------- | -------------- | ----------------------------------------- |
| **Semantic search** | Core feature | ✅ Hybrid mode (add when needed) |
| **Distributed scaling** | Built-in | Single-file (use sharding for huge scale) |
| **Managed hosting** | Yes (Pinecone) | Memvid Cloud (optional) |
| **Real-time sync** | Some | Coming soon |
<Note>
**Smart Frames = superset of vector databases.** Memvid does everything vector DBs do (semantic search), plus lexical search, temporal queries, and entity extraction, all in one file.
</Note>
***
## When to Use What
### Use Pinecone When:
* You need managed infrastructure
* You're building a semantic-search-first application
* You have budget for cloud services
* You need global distribution
### Use ChromaDB When:
* You want open source with optional cloud
* You're prototyping and need quick setup
* You're comfortable managing multiple files
* Your use case is purely semantic search
### Use Memvid When:
* **You need fast search**: 24ms vs 267-506ms (11-21x faster than vector DBs)
* **You want to search immediately** without embedding delays
* **You need exact matches** (code, logs, error messages, names)
* **You want one portable file** for your entire knowledge base (4.9 MB for 1,000 docs)
* **You're building offline-first** applications
* **You want time-travel queries** (point-in-time retrieval)
* **You need entity extraction** built-in (auto-tagging, date extraction, triplets)
* **You want to avoid vendor lock-in** and API dependencies
* **You care about cost** (\$0 forever is hard to beat)
***
## Migration Guide
### From Pinecone to Memvid
```python theme={null}
import pinecone
from memvid_sdk import create
# Export from Pinecone
pinecone.init(api_key="...")
index = pinecone.Index("my-index")
# Create Memvid memory
mem = create("knowledge.mv2")
# Fetch and migrate (you'll need your original documents)
# Pinecone doesn't store original text, only vectors
# This is why Memvid stores everything in one place
```
### From ChromaDB to Memvid
```python theme={null}
import chromadb
from memvid_sdk import create
# Export from ChromaDB
client = chromadb.Client()
collection = client.get_collection("my-collection")
results = collection.get(include=["documents", "metadatas"])
# Create Memvid memory
mem = create("knowledge.mv2")
# Migrate documents
for doc, meta in zip(results["documents"], results["metadatas"]):
mem.put(
title=meta.get("title", "Untitled"),
label="migrated",
metadata=meta,
text=doc
)
mem.seal()
```
***
## Try It Yourself
The best comparison is your own experience:
```bash theme={null}
# Install (10 seconds)
npm install -g memvid-cli
# Create and search (10 more seconds)
memvid create test.mv2
echo "The quick brown fox jumps over the lazy dog" | memvid put test.mv2
memvid find test.mv2 --query "quick fox"
# That's it. Just search.
```
***
## Still Have Questions?
<CardGroup>
<Card title="5-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Get hands-on with Memvid
</Card>
<Card title="The Memvid Approach" icon="lightbulb" href="/introduction/the-memvid-approach">
Why we built it this way
</Card>
<Card title="Discord Community" icon="discord" href="https://discord.gg/2mynS7fcK7">
Ask questions, get help
</Card>
<Card title="GitHub" icon="github" href="https://github.com/memvid/memvid">
Star the repo, contribute
</Card>
</CardGroup>
# Adaptive Retrieval
Source: https://docs.memvid.com/concepts/adaptive-retrieval
Automatically determine the optimal number of search results based on relevance
Instead of returning a fixed number of results (top-k), adaptive retrieval **automatically determines** how many results are truly relevant to your query. This prevents both information overload (too many irrelevant results) and information loss (cutting off relevant results).
***
## The Problem with Fixed Top-K
Traditional search returns a fixed number of results:
```bash theme={null}
# Always returns 10 results, even if:
# - Only 3 are relevant (7 are noise)
# - 25 are relevant (15 are missed)
memvid find memory.mv2 --query "authentication" --top-k 10
```
This creates two problems:
| Scenario | Fixed Top-K | Result |
| ------------------ | ----------- | ------------------------ |
| Few relevant docs | Returns 10 | Noise in results |
| Many relevant docs | Returns 10 | Missing relevant content |
***
## How Adaptive Retrieval Works
Adaptive retrieval analyzes the **score distribution** of results to find natural cutoff points:
```
Score Distribution Example:
Result 1: 0.95 ████████████████████ ← Highly relevant
Result 2: 0.91 ███████████████████ ← Highly relevant
Result 3: 0.88 ██████████████████ ← Highly relevant
Result 4: 0.42 █████████ ← Score cliff detected!
Result 5: 0.38 ████████
Result 6: 0.35 ███████
...
Adaptive returns: Results 1-3 (stops at the cliff)
```
***
## CLI Usage
Adaptive retrieval is **enabled by default**:
```bash theme={null}
# Adaptive mode (default)
memvid find memory.mv2 --query "authentication best practices"
# Returns: 3-15 results based on relevance distribution
# Disable adaptive, use fixed top-k
memvid find memory.mv2 --query "authentication" --no-adaptive --top-k 10
# Returns: Exactly 10 results
```
### Tuning Adaptive Behavior
```bash theme={null}
# Minimum relevance threshold (0.0-1.0)
memvid find memory.mv2 --query "security" --min-relevancy 0.6
# Only returns results with score >= 0.6
# Maximum results cap
memvid find memory.mv2 --query "config" --max-k 50
# Returns up to 50 results (fewer if cliff detected earlier)
# Choose strategy
memvid find memory.mv2 --query "api" --adaptive-strategy cliff
# Uses cliff detection algorithm
```
***
## Adaptive Strategies
Five strategies are available for different use cases:
### Combined (Default)
Balances multiple signals for best overall performance:
```bash theme={null}
memvid find memory.mv2 --query "term" --adaptive-strategy combined
```
* Uses both relative and absolute thresholds
* Applies cliff detection as secondary signal
* Best for general-purpose search
### Relative
Cuts off at a percentage of the top score:
```bash theme={null}
memvid find memory.mv2 --query "term" --adaptive-strategy relative
```
* Default threshold: 50% of top score
* Good for consistent corpora
* Example: Top score 0.92 → cutoff at 0.46
### Absolute
Uses a fixed score cutoff:
```bash theme={null}
memvid find memory.mv2 --query "term" --adaptive-strategy absolute --min-relevancy 0.5
```
* Cuts at specified minimum score
* Good when you know your quality threshold
* Predictable behavior across queries
### Cliff
Detects sharp drops in score distribution:
```bash theme={null}
memvid find memory.mv2 --query "term" --adaptive-strategy cliff
```
* Looks for score drops > 30% between consecutive results
* Best for distinct topic clusters
* Works well when relevant docs are clearly separated
### Elbow
Finds the inflection point in the score curve:
```bash theme={null}
memvid find memory.mv2 --query "term" --adaptive-strategy elbow
```
* Uses curve analysis to find natural groupings
* Good for gradual score distributions
* Mathematically principled approach
***
## Strategy Comparison
| Strategy | Best For | Behavior |
| ---------- | --------------------- | -------------------- |
| `combined` | General use | Balanced, adaptive |
| `relative` | Consistent corpora | % of top score |
| `absolute` | Known thresholds | Fixed cutoff |
| `cliff` | Clustered topics | Sharp drop detection |
| `elbow` | Gradual distributions | Curve inflection |
***
## SDK Usage
### Python
```python theme={null}
from memvid import use
mem = use('basic', 'memory.mv2')
# Adaptive (default)
results = mem.find("authentication patterns")
print(f"Returned {len(results)} relevant results")
# With custom settings
results = mem.find(
"security best practices",
adaptive=True,
min_relevancy=0.5,
max_k=25,
adaptive_strategy="cliff"
)
# Disable adaptive
results = mem.find(
"config files",
adaptive=False,
top_k=10
)
```
### Node.js
```typescript theme={null}
import { use } from '@anthropics/memvid'
const mem = await use('basic', 'memory.mv2')
// Adaptive (default)
const results = await mem.find("authentication patterns")
console.log(`Returned ${results.length} relevant results`)
// With custom settings
const results = await mem.find("security", {
adaptive: true,
minRelevancy: 0.5,
maxK: 25,
adaptiveStrategy: "cliff"
})
// Disable adaptive
const results = await mem.find("config", {
adaptive: false,
topK: 10
})
```
***
## Understanding Results
Adaptive retrieval adds metadata to help you understand decisions:
```bash theme={null}
memvid find memory.mv2 --query "authentication" --json
```
```json theme={null}
{
"query": "authentication",
"strategy": "combined",
"results": [
{
"frame_id": "frame_001",
"score": 0.94,
"title": "OAuth2 Implementation Guide"
},
{
"frame_id": "frame_002",
"score": 0.89,
"title": "JWT Token Handling"
},
{
"frame_id": "frame_003",
"score": 0.85,
"title": "Session Management"
}
],
"adaptive_info": {
"total_candidates": 150,
"cutoff_score": 0.72,
"cutoff_reason": "cliff_detected",
"results_returned": 3
}
}
```
***
## When to Disable Adaptive
Some scenarios work better with fixed top-k:
### Pagination
When paginating through results:
```bash theme={null}
# Page 1
memvid find memory.mv2 --query "logs" --no-adaptive --top-k 20
# Page 2 (use cursor)
memvid find memory.mv2 --query "logs" --no-adaptive --top-k 20 --cursor <cursor>
```
### Comparison
When comparing result counts across queries:
```python theme={null}
# Need consistent counts for comparison
results_a = mem.find("topic A", adaptive=False, top_k=50)
results_b = mem.find("topic B", adaptive=False, top_k=50)
coverage_a = len([r for r in results_a if r.score > 0.5])
coverage_b = len([r for r in results_b if r.score > 0.5])
```
### RAG Context Building
When you need a specific context window:
```python theme={null}
# Need exactly 5 chunks for context
results = mem.find(
query,
adaptive=False,
top_k=5
)
context = "\n".join([r.text for r in results])
```
***
## Tuning for Your Data
### High-Precision Needs
For applications where false positives are costly:
```bash theme={null}
# Strict threshold
memvid find memory.mv2 --query "term" \
--min-relevancy 0.7 \
--adaptive-strategy absolute
```
### High-Recall Needs
For applications where missing results is costly:
```bash theme={null}
# Lenient threshold with high cap
memvid find memory.mv2 --query "term" \
--min-relevancy 0.3 \
--max-k 100 \
--adaptive-strategy relative
```
### Exploratory Search
For browsing and discovery:
```bash theme={null}
# Combined strategy with moderate settings
memvid find memory.mv2 --query "term" \
--min-relevancy 0.4 \
--max-k 50 \
--adaptive-strategy combined
```
***
## Performance Considerations
Adaptive retrieval adds minimal overhead:
| Operation | Time Added |
| ---------------- | ---------- |
| Score analysis | \< 1ms |
| Cutoff detection | \< 1ms |
| Total overhead | \< 2ms |
The algorithm runs on the score array after retrieval, so it doesn't slow down the actual search.
***
## Combining with Other Features
### With Sketch Pre-filtering
```bash theme={null}
# Sketches + adaptive = fast + precise
memvid sketch build memory.mv2 --variant medium
memvid find memory.mv2 --query "term" # Both enabled by default
```
### With Hybrid Search
```bash theme={null}
# Adaptive works with all search modes
memvid find memory.mv2 --query "term" --mode auto # Hybrid + adaptive
memvid find memory.mv2 --query "term" --mode lex # Lexical + adaptive
memvid find memory.mv2 --query "term" --mode sem # Semantic + adaptive
```
### With Time Filtering
```bash theme={null}
# Adaptive respects filters
memvid find memory.mv2 --query "report" \
--start 2024-01-01 \
--end 2024-06-30 \
--adaptive-strategy cliff
```
***
## Best Practices
### Start with Defaults
The default `combined` strategy works well for most cases:
```bash theme={null}
memvid find memory.mv2 --query "your search"
```
### Tune Based on Feedback
If you're getting too many results:
```bash theme={null}
--min-relevancy 0.6 # Raise threshold
--adaptive-strategy cliff # Stricter cutoff
```
If you're missing results:
```bash theme={null}
--min-relevancy 0.3 # Lower threshold
--max-k 100 # Raise cap
```
### Monitor with JSON Output
Check adaptive decisions to understand behavior:
```bash theme={null}
memvid find memory.mv2 --query "term" --json | jq '.adaptive_info'
```
***
## Troubleshooting
### "Adaptive returns too few results"
1. Lower `min-relevancy`:
```bash theme={null}
--min-relevancy 0.3
```
2. Increase `max-k`:
```bash theme={null}
--max-k 100
```
3. Try `relative` strategy:
```bash theme={null}
--adaptive-strategy relative
```
### "Adaptive returns too many results"
1. Raise `min-relevancy`:
```bash theme={null}
--min-relevancy 0.7
```
2. Use `cliff` strategy:
```bash theme={null}
--adaptive-strategy cliff
```
3. Consider if your query is too broad
### "Results vary unexpectedly between queries"
This is expected - adaptive adjusts to each query's score distribution. For consistent counts, use:
```bash theme={null}
--no-adaptive --top-k 10
```
***
## Next Steps
<CardGroup>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Complete search command reference
</Card>
<Card title="Deduplication" icon="copy" href="/concepts/deduplication">
How Memvid prevents duplicate content
</Card>
</CardGroup>
# Audio & Video Processing
Source: https://docs.memvid.com/concepts/audio-video
Transcribe audio, process video, and search multimedia content
Memvid processes audio and video files using Whisper for transcription, making spoken content searchable. Video files also support key frame extraction and playback from within the CLI.
***
## How It Works
```mermaid theme={null}
flowchart LR
A[Audio/Video File] --> B[Whisper Transcription]
B --> C[Text Frames]
C --> D[Searchable Content]
A --> E[Key Frame Extraction]
E --> F[Visual Embeddings]
```
Key features:
* **Automatic transcription** - Whisper converts speech to text
* **Timestamp alignment** - Text segments linked to audio/video timecodes
* **Key frame extraction** - Important frames from video
* **In-CLI playback** - Play segments directly from terminal
* **Visual search** - CLIP embeddings for video frames
***
## Supported Formats
### Audio
| Format | Extension | Notes |
| ------ | -------------- | -------------------------- |
| MP3 | `.mp3` | Most common, lossy |
| WAV | `.wav` | Uncompressed, best quality |
| FLAC | `.flac` | Lossless compression |
| AAC | `.aac`, `.m4a` | Apple/iTunes format |
| OGG | `.ogg` | Open format, Vorbis codec |
| ALAC | `.m4a` | Apple Lossless |
### Video
| Format | Extension | Notes |
| ------ | --------- | ------------------------ |
| MP4 | `.mp4` | Most common, H.264/H.265 |
| WebM | `.webm` | Web-optimized, VP8/VP9 |
| MOV | `.mov` | Apple QuickTime |
| AVI | `.avi` | Legacy Windows format |
| MKV | `.mkv` | Matroska container |
| FLV | `.flv` | Flash video (legacy) |
***
## Ingesting Audio
### Basic Usage
```bash theme={null}
# Ingest audio file (auto-transcribes)
memvid put memory.mv2 --input podcast.mp3
# Ingest multiple audio files
memvid put memory.mv2 --input ./recordings/
# With metadata
memvid put memory.mv2 --input interview.wav --metadata '{"speaker": "John Doe", "date": "2024-01-15"}'
```
### Transcription Options
```bash theme={null}
# Specify language (faster, more accurate)
memvid put memory.mv2 --input audio.mp3 --language en
# Force language detection
memvid put memory.mv2 --input audio.mp3 --detect-language
# Use larger model for better accuracy
memvid put memory.mv2 --input audio.mp3 --whisper-model medium
```
### Whisper Models
| Model | Size | Speed | Accuracy | Best For |
| -------- | ------ | ------- | --------- | ----------------- |
| `tiny` | 39 MB | Fastest | Basic | Quick previews |
| `base` | 74 MB | Fast | Good | General use |
| `small` | 244 MB | Medium | Better | Default choice |
| `medium` | 769 MB | Slower | Excellent | Important content |
| `large` | 1.5 GB | Slowest | Best | Critical accuracy |
```bash theme={null}
# Install specific model
memvid models install whisper-medium
# Use installed model
memvid put memory.mv2 --input audio.mp3 --whisper-model medium
```
### Language Support
Whisper supports 99 languages. Specify for better accuracy:
```bash theme={null}
# English
memvid put memory.mv2 --input audio.mp3 --language en
# Spanish
memvid put memory.mv2 --input audio.mp3 --language es
# Mandarin Chinese
memvid put memory.mv2 --input audio.mp3 --language zh
# Auto-detect (slower)
memvid put memory.mv2 --input audio.mp3 --detect-language
```
***
## Ingesting Video
### Basic Usage
```bash theme={null}
# Ingest video (transcribes audio + extracts frames)
memvid put memory.mv2 --input meeting.mp4
# Video only (no audio transcription)
memvid put memory.mv2 --input silent-video.mp4 --no-transcribe
# Audio only (skip frame extraction)
memvid put memory.mv2 --input video.mp4 --audio-only
```
### Frame Extraction
```bash theme={null}
# Extract key frames for visual search
memvid put memory.mv2 --input video.mp4 --extract-frames
# Control frame density
memvid put memory.mv2 --input video.mp4 --frame-interval 30 # Every 30 seconds
# Extract specific number of frames
memvid put memory.mv2 --input video.mp4 --max-frames 50
```
### Visual Embeddings
Enable CLIP embeddings for visual search:
```bash theme={null}
# Enable visual embeddings for frames
memvid put memory.mv2 --input video.mp4 --clip-embeddings
# Search by visual content
memvid find memory.mv2 --query "person at whiteboard" --mode clip
```
***
## Searching Transcribed Content
### Text Search
```bash theme={null}
# Search transcription text
memvid find memory.mv2 --query "quarterly revenue"
# Search with timestamp context
memvid find memory.mv2 --query "action items" --json
# Results include timecodes:
# {
# "frame_id": "frame_abc123",
# "text": "The action items from this meeting are...",
# "timestamp": "00:15:32",
# "source": "meeting.mp4"
# }
```
### Ask Questions
```bash theme={null}
# Ask about audio/video content
memvid ask memory.mv2 --question "What were the main decisions from the meeting?"
# Get context with timestamps
memvid ask memory.mv2 --question "What did John say about the budget?" --sources
```
### Visual Search (Video)
```bash theme={null}
# Search by visual description
memvid find memory.mv2 --query "chart showing growth" --mode clip
# Combine text and visual
memvid find memory.mv2 --query "presentation slide" --mode auto
```
***
## Playback
### Playing Audio
```bash theme={null}
# Play entire audio
memvid view memory.mv2 --frame-id frame_abc --play
# Play specific segment
memvid view memory.mv2 --frame-id frame_abc --play --start-seconds 30 --end-seconds 60
# Play from timestamp
memvid view memory.mv2 --frame-id frame_abc --play --start-seconds 120
```
### Playing Video
```bash theme={null}
# Play video
memvid view memory.mv2 --frame-id frame_xyz --play
# Play segment
memvid view memory.mv2 --frame-id frame_xyz --play --start-seconds 0 --end-seconds 30
# Preview mode (thumbnail)
memvid view memory.mv2 --frame-id frame_xyz --preview
```
### Playback Controls
| Option | Description |
| -------------------------- | ---------------------- |
| `--play` | Start playback |
| `--start-seconds N` | Start at N seconds |
| `--end-seconds N` | Stop at N seconds |
| `--preview` | Show thumbnail/preview |
| `--preview-start HH:MM:SS` | Preview window start |
| `--preview-end HH:MM:SS` | Preview window end |
***
## Use Cases
### Meeting Recordings
```bash theme={null}
# Create meeting memory
memvid create meetings.mv2
# Ingest meeting recordings
memvid put meetings.mv2 --input ./recordings/ --language en
# Find specific discussions
memvid find meetings.mv2 --query "budget approval"
# Ask about decisions
memvid ask meetings.mv2 --question "What was decided about the Q4 budget?"
# Play the relevant segment
memvid view meetings.mv2 --frame-id frame_abc --play --start-seconds 1234
```
### Podcast Library
```bash theme={null}
# Create podcast memory
memvid create podcasts.mv2
# Ingest episodes with metadata
memvid put podcasts.mv2 --input episode-42.mp3 \
--metadata '{"show": "Tech Talk", "episode": 42, "guests": ["Alice", "Bob"]}'
# Search across all episodes
memvid find podcasts.mv2 --query "machine learning trends"
# Timeline of episodes
memvid timeline podcasts.mv2 --reverse
```
### Video Tutorials
```bash theme={null}
# Create tutorial library
memvid create tutorials.mv2
# Ingest with frame extraction
memvid put tutorials.mv2 --input ./tutorials/ --extract-frames --clip-embeddings
# Find by spoken content
memvid find tutorials.mv2 --query "how to configure webpack"
# Find by visual content
memvid find tutorials.mv2 --query "terminal with npm commands" --mode clip
```
### Lecture Archive
```bash theme={null}
# Create lecture memory
memvid create lectures.mv2
# Ingest lecture videos
memvid put lectures.mv2 --input ./cs101/ --whisper-model medium
# Search for topics
memvid find lectures.mv2 --query "binary search algorithm"
# Ask study questions
memvid ask lectures.mv2 --question "Explain the time complexity of quicksort"
```
### Voicemail/Call Logs
```bash theme={null}
# Create call memory
memvid create calls.mv2
# Ingest voicemails
memvid put calls.mv2 --input ./voicemails/ --metadata '{"type": "voicemail"}'
# Find by caller mention
memvid find calls.mv2 --query "callback number"
# Enrich with entity extraction
memvid enrich calls.mv2 --engine groq
memvid state calls.mv2 --entity "Customer Service"
```
***
## GPU Acceleration
Transcription is CPU-intensive. Enable GPU for faster processing:
### macOS (Apple Silicon)
```bash theme={null}
# Install with Metal support
cargo install memvid-cli --features metal
# Or via Homebrew (includes Metal)
brew install memvid/tap/memvid
```
### Linux/Windows (NVIDIA CUDA)
```bash theme={null}
# Install with CUDA support
cargo install memvid-cli --features cuda
# Requires CUDA toolkit and cuDNN
```
### Performance Comparison
| Hardware | 1hr Audio | 1hr Video |
| --------------- | --------- | --------- |
| CPU (M1) | \~15 min | \~25 min |
| Metal (M1) | \~3 min | \~8 min |
| CPU (Intel i7) | \~20 min | \~35 min |
| CUDA (RTX 3080) | \~2 min | \~5 min |
***
## Batch Processing
### Parallel Ingestion
```bash theme={null}
# Process multiple files in parallel
memvid put memory.mv2 --input ./media/ --parallel-segments
# Limit concurrent transcriptions (manage memory)
memvid put memory.mv2 --input ./media/ --max-concurrent 2
```
### Large Libraries
```bash theme={null}
# Ingest incrementally
memvid put memory.mv2 --input ./media/2024-01/
memvid put memory.mv2 --input ./media/2024-02/
# Check progress
memvid stats memory.mv2
memvid timeline memory.mv2 --limit 10
```
***
## Frame Metadata
Each transcribed segment includes metadata:
```json theme={null}
{
"frame_id": "frame_abc123",
"uri": "mv2://media/meeting.mp4",
"content_type": "audio/transcript",
"metadata": {
"source_file": "meeting.mp4",
"duration_seconds": 3600,
"segment_start": 932.5,
"segment_end": 945.2,
"timestamp": "00:15:32",
"language": "en",
"confidence": 0.94,
"whisper_model": "small"
}
}
```
Access metadata:
```bash theme={null}
# View frame with metadata
memvid view memory.mv2 --frame-id frame_abc --json
# Filter by media type
memvid timeline memory.mv2 --filter "content_type:audio/transcript"
```
***
## Troubleshooting
### No Transcription Output
```bash theme={null}
# Check if Whisper model is installed
memvid models list
# Install required model
memvid models install whisper-small
# Try with explicit language
memvid put memory.mv2 --input audio.mp3 --language en
```
### Poor Transcription Quality
```bash theme={null}
# Use larger model
memvid put memory.mv2 --input audio.mp3 --whisper-model medium
# Specify correct language
memvid put memory.mv2 --input audio.mp3 --language es
# Check audio quality - Whisper works best with clear audio
```
### Playback Issues
```bash theme={null}
# Check frame exists
memvid view memory.mv2 --frame-id frame_abc
# Try preview mode first
memvid view memory.mv2 --frame-id frame_abc --preview
# Check file format support
memvid stats memory.mv2 --json | jq '.frames[] | select(.uri | contains("video"))'
```
### Out of Memory
```bash theme={null}
# Reduce concurrent processing
memvid put memory.mv2 --input video.mp4 --max-concurrent 1
# Use smaller model
memvid put memory.mv2 --input video.mp4 --whisper-model base
# Process in segments
memvid put memory.mv2 --input video.mp4 --segment-duration 300
```
***
## Limitations
| Limitation | Workaround |
| ---------------------- | ------------------------------------- |
| No real-time streaming | Pre-record content |
| Large file sizes | Use compression before ingestion |
| Multiple speakers | Manual speaker tagging via metadata |
| Background noise | Pre-process audio for noise reduction |
| Non-speech audio | Not transcribed (music, effects) |
***
## SDK Support
Audio/video processing is currently **CLI-only**. SDK support planned.
Workaround:
```python theme={null}
import subprocess
# Ingest via CLI
subprocess.run([
'memvid', 'put', 'memory.mv2',
'--input', 'audio.mp3',
'--language', 'en'
])
# Search transcriptions via SDK
from memvid import use
mem = use('basic', 'memory.mv2')
results = mem.find("meeting action items")
```
***
## Next Steps
<CardGroup>
<Card title="Visual Embeddings" icon="image" href="/concepts/visual-embeddings">
CLIP search for images and video frames
</Card>
<Card title="Memory Cards" icon="brain" href="/concepts/memory-cards">
Extract entities from transcriptions
</Card>
</CardGroup>
# Capacity & Plans
Source: https://docs.memvid.com/concepts/capacity-and-plans
Storage tiers, capacity management, and upgrading your memories
Memvid files have configurable storage capacity. By default, new files are created with **1 GB capacity**, enough for most use cases. When you need more, upgrade your plan and sync tickets to increase capacity.
***
## Default Capacity
When you create a new memory, you get generous defaults:
```bash theme={null}
memvid create knowledge.mv2
```
```
✓ Created memory at knowledge.mv2
Capacity: 50 MB (52428800 bytes)
Size: 70 KB
Indexes: lexical | vector
Next steps:
memvid put knowledge.mv2 --input <file> # Add content
memvid find knowledge.mv2 --query <text> # Search
memvid stats knowledge.mv2 # View stats
```
***
## Plans
| Plan | Total Capacity | Memories | Per-Memory Limit | Price |
| -------------- | -------------- | --------- | ---------------- | ---------- |
| **Free** | 50 MB | 1 | 50 MB | Free |
| **Starter** | 25 GB | 5 | 5 GB | \$9.99/mo |
| **Pro** | 125 GB | 10 | 25 GB | \$49.99/mo |
| **Enterprise** | Unlimited | Unlimited | Unlimited | Contact us |
***
## Checking Capacity
### CLI
```bash theme={null}
memvid stats knowledge.mv2
```
```
Memory: knowledge.mv2
Documents: 1,250
Active Frames: 1,248
Size: 7.8 MB
Capacity: 50 MB
Utilization: 15.6%
Indexes:
Lexical: Yes
Vector: Yes
Time: Yes
```
### Python SDK
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2', read_only=True)
stats = mem.stats()
print(f"Size: {stats['size_bytes'] / 1e9:.2f} GB")
print(f"Capacity: {stats['capacity_bytes'] / 1e9:.2f} GB")
print(f"Utilization: {stats['storage_utilisation_percent']:.1f}%")
# Check current ticket info
ticket = mem.current_ticket()
print(f"Plan: {ticket['issuer']}")
print(f"Capacity: {ticket['capacity_bytes'] / 1e9:.2f} GB")
```
### Node.js SDK
```typescript theme={null}
import { use } from '@memvid/sdk';
const mem = await use('basic', 'knowledge.mv2', { readOnly: true });
const stats = await mem.stats();
console.log(`Size: ${(stats.size_bytes / 1e9).toFixed(2)} GB`);
console.log(`Capacity: ${(stats.capacity_bytes / 1e9).toFixed(2)} GB`);
console.log(`Utilization: ${stats.storage_utilisation_percent.toFixed(1)}%`);
```
***
## Upgrading Capacity
To increase your memory's capacity, you need to:
1. Get a Memvid API key from [memvid.com/dashboard](https://memvid.com/dashboard)
2. Create a memory in the dashboard and get its Memory ID
3. Sync tickets to your local file
### Step 1: Get Your Credentials
1. Sign up at [memvid.com/dashboard](https://memvid.com/dashboard)
2. Create a new memory in the dashboard
3. Copy your **API Key** and **Memory ID**
### Step 2: Sync Tickets
<Tabs>
<Tab title="Python SDK">
```python theme={null}
from memvid_sdk import use
import os
# Configuration
API_KEY = os.environ.get("MEMVID_API_KEY")
MEMORY_ID = os.environ.get("MEMVID_MEMORY_ID")
API_URL = "https://api.memvid.com" # or your custom endpoint
# Open your memory
mem = use('basic', 'knowledge.mv2')
# Check current binding
binding = mem.get_memory_binding()
if binding:
print(f"Already bound to: {binding['memory_id']}")
else:
print("Not currently bound")
# Sync tickets from the API
result = mem.sync_tickets(
memory_id=MEMORY_ID,
api_key=API_KEY,
api_url=API_URL
)
print(f"Memory ID: {result['memory_id']}")
print(f"Issuer: {result['issuer']}")
print(f"New Capacity: {result['capacity_bytes'] / 1e9:.2f} GB")
# Verify new capacity
capacity = mem.get_capacity()
print(f"Current capacity: {capacity / 1e9:.2f} GB")
mem.close()
```
</Tab>
<Tab title="CLI">
```bash theme={null}
# Set environment variables
export MEMVID_API_KEY="your-api-key"
export MEMVID_MEMORY_ID="your-memory-id"
export MEMVID_API_URL="https://api.memvid.com"
# Sync tickets
memvid tickets sync knowledge.mv2
# Verify
memvid stats knowledge.mv2
```
</Tab>
</Tabs>
### Environment Variables
| Variable | Description |
| ------------------ | ---------------------------------------------------------------------- |
| `MEMVID_API_KEY` | Your API key from [memvid.com/dashboard](https://memvid.com/dashboard) |
| `MEMVID_MEMORY_ID` | Memory ID from the dashboard |
| `MEMVID_API_URL` | API endpoint (default: `https://api.memvid.com`) |
***
## Capacity Exceeded Errors
When you exceed capacity, you'll see error **MV001**:
```
Error: CapacityExceeded (MV001)
File capacity: 52428800 bytes (50 MB)
Current usage: 49000000 bytes (49 MB)
Requested: 5000000 bytes (5 MB)
```
### Solutions
1. **Delete unused frames**:
```bash theme={null}
memvid delete knowledge.mv2 --frame-id 42 --yes
```
2. **Vacuum to reclaim space**:
```bash theme={null}
memvid doctor knowledge.mv2 --vacuum
```
3. **Upgrade your plan** and sync tickets
***
## Handling Capacity in Code
```python theme={null}
from memvid_sdk import use, CapacityExceededError
def safe_ingest(mem, documents):
"""Ingest documents with capacity handling."""
stats = mem.stats()
available = stats['capacity_bytes'] - stats['size_bytes']
for doc in documents:
estimated_size = len(doc['text'].encode('utf-8'))
if estimated_size > available:
print(f"Skipping {doc['title']}: insufficient capacity")
continue
try:
mem.put(text=doc['text'], title=doc['title'])
available -= estimated_size
except CapacityExceededError:
print(f"Capacity exceeded at {doc['title']}")
break
mem.seal()
```
***
## WAL Size by Capacity
The Write-Ahead Log scales with capacity:
| File Capacity | WAL Size | Checkpoint Threshold |
| ------------- | -------- | -------------------- |
| \< 100 MB | 1 MB | 768 KB (75%) |
| \< 1 GB | 4 MB | 3 MB (75%) |
| \< 10 GB | 16 MB | 12 MB (75%) |
| ≥ 10 GB | 64 MB | 48 MB (75%) |
***
## Storage Optimization
### Vector Compression
Enable 16x compression for embeddings:
```bash theme={null}
memvid put knowledge.mv2 --input docs/ --vector-compression
```
```python theme={null}
mem.put_many(docs, enable_embedding=True, vector_compression=True)
```
### Vacuum After Deletions
Reclaim space from deleted frames:
```bash theme={null}
# Check before
memvid stats knowledge.mv2
# Vacuum
memvid doctor knowledge.mv2 --vacuum
# Check after
memvid stats knowledge.mv2
```
***
## Best Practices
### Capacity Planning
1. **Start free**: 50 MB handles most personal projects
2. **Monitor usage**: Check `storage_utilisation_percent` regularly
3. **Upgrade before 80%**: Leave headroom for growth
4. **Use compression**: Enable vector compression for large collections
### Multi-Memory Strategy
For large organizations:
```
project-docs.mv2 → 10 GB (documentation)
chat-history.mv2 → 25 GB (conversations)
media-archive.mv2 → 50 GB (images, audio)
```
***
## Next Steps
<CardGroup>
<Card title="Memory Architecture" icon="database" href="/concepts/memory-architecture">
Understand file structure
</Card>
<Card title="Troubleshooting" icon="wrench" href="/troubleshooting/cli">
Solve capacity issues
</Card>
</CardGroup>
# Deduplication & SimHash
Source: https://docs.memvid.com/concepts/deduplication
How Memvid automatically detects and prevents duplicate content
Memvid automatically prevents duplicate content from bloating your memory files using two complementary techniques: **content hashing** for exact duplicates and **SimHash** for near-duplicates.
***
## How Deduplication Works
When you add content to a memory, Memvid performs two checks:
| Check | Algorithm | Catches |
| --------- | -------------------- | ------------------------------------- |
| **Exact** | BLAKE3 hash | Identical content |
| **Near** | SimHash (64-bit LSH) | Similar content with minor variations |
Both checks happen automatically during `put` operations with no configuration required.
***
## Exact Deduplication
Every frame stores a BLAKE3 content hash. When you add new content:
1. Hash is computed for the new content
2. Hash is checked against existing frames
3. If match found, the existing frame ID is returned
4. No duplicate frame is created
```bash theme={null}
# First put - creates new frame
memvid put memory.mv2 --input document.pdf
# Output: Created frame_abc123
# Second put of same file - returns existing frame
memvid put memory.mv2 --input document.pdf
# Output: Duplicate detected, returning existing frame_abc123
```
```python theme={null}
# Python SDK
frame_id_1 = mem.put("The quick brown fox")
frame_id_2 = mem.put("The quick brown fox") # Same content
assert frame_id_1 == frame_id_2 # True - no duplicate created
```
```typescript theme={null}
// Node.js SDK
const id1 = await mem.put({ content: "The quick brown fox" })
const id2 = await mem.put({ content: "The quick brown fox" })
console.log(id1 === id2) // true
```
***
## SimHash (Near-Duplicate Detection)
SimHash is a locality-sensitive hashing algorithm that detects **near-duplicate** content - documents that are almost identical but have minor differences like:
* Whitespace changes
* Punctuation variations
* Minor edits or typos
* Reformatted text
### How SimHash Works
1. **Tokenize**: Break content into word n-grams (shingles)
2. **Hash shingles**: Each shingle gets a 64-bit hash
3. **Combine**: Weighted combination produces final 64-bit fingerprint
4. **Compare**: Hamming distance measures similarity
Two documents are considered near-duplicates if their SimHash fingerprints differ by fewer than **32 bits** (out of 64).
### Hamming Distance Thresholds
| Distance | Similarity | Classification |
| ---------- | ---------- | ------------------- |
| 0-10 bits | 85-100% | Near-identical |
| 11-20 bits | 70-85% | Very similar |
| 21-31 bits | 50-70% | Somewhat similar |
| 32+ bits | \< 50% | Different documents |
### Example: Near-Duplicate Detection
```bash theme={null}
# Original document
echo "The quick brown fox jumps over the lazy dog." | memvid put memory.mv2 --input -
# Output: Created frame_001
# Minor variation (punctuation + whitespace)
echo "The quick brown fox jumps over the lazy dog" | memvid put memory.mv2 --input -
# Output: Near-duplicate of frame_001 detected, skipping
# Different document (passes threshold)
echo "A slow red cat sleeps under the busy cat." | memvid put memory.mv2 --input -
# Output: Created frame_002
```
***
## Sketch Track (Fast Pre-filtering)
For large memories (10k+ frames), Memvid uses **sketch tracks** to accelerate duplicate detection. Sketches are compact fingerprints that enable sub-millisecond candidate filtering.
### Sketch Variants
| Variant | Size | Speed | Accuracy | Best For |
| -------- | -------- | -------- | -------- | --------------- |
| `small` | 32 bytes | Fastest | Good | \< 50k frames |
| `medium` | 64 bytes | Fast | Better | 50k-200k frames |
| `large` | 96 bytes | Moderate | Best | 200k+ frames |
### Building Sketches
```bash theme={null}
# Build sketch index (recommended for large memories)
memvid sketch build memory.mv2 --variant medium
# Check sketch stats
memvid sketch info memory.mv2
```
Output:
```
Sketch Track Info
Variant: medium (64 bytes)
Frames indexed: 45,230
Index size: 2.9 MB
Avg lookup time: 0.3ms
```
### How Sketches Speed Up Search
Without sketches:
1. Compare query against all 45,230 frames
2. Full SimHash comparison for each
3. \~450ms total
With sketches:
1. Compare query sketch against sketch index
2. Get \~100 candidates in 0.3ms
3. Full comparison only on candidates
4. \~5ms total (90x faster)
***
## Deduplication Statistics
Check deduplication stats for your memory:
```bash theme={null}
memvid stats memory.mv2 --json
```
```json theme={null}
{
"frame_count": 1250,
"unique_content_hashes": 1248,
"duplicate_frames_prevented": 127,
"has_sketch_track": true,
"sketch_variant": "medium"
}
```
***
## When Duplicates Are Allowed
Some use cases require keeping duplicates:
### Audit Trails
When you need to track every submission regardless of content:
```bash theme={null}
# Add timestamp to make each entry unique
memvid put memory.mv2 --input report.pdf --timestamp "$(date -u +%s)"
```
```python theme={null}
# Python - unique URI bypasses dedup
import time
mem.put(
text="Daily report content",
uri=f"report-{int(time.time())}"
)
```
### Versioning
Track document versions explicitly:
```bash theme={null}
# Version in metadata distinguishes duplicates
memvid put memory.mv2 --input contract.pdf --metadata '{"version": "1.0"}'
memvid put memory.mv2 --input contract.pdf --metadata '{"version": "1.1"}'
```
***
## Disabling Deduplication
For specific use cases where you want all content stored:
```python theme={null}
# Python SDK - force creation
frame_id = mem.put(
text="Content that might be duplicate",
skip_dedup=True # Force new frame creation
)
```
<Warning>
Disabling deduplication can significantly increase storage usage. Only disable when you have a specific need to store duplicate content.
</Warning>
***
## Deduplication Across Memories
Deduplication only works **within** a single `.mv2` file. The same content in different memory files will be stored separately.
```bash theme={null}
# These are independent - both will store the content
memvid put work.mv2 --input document.pdf
memvid put personal.mv2 --input document.pdf
```
***
## Performance Impact
| Operation | With Dedup | Without Dedup |
| -------------------- | ----------- | ------------- |
| Single put | +2ms | Baseline |
| Batch put (1000) | +50ms total | Baseline |
| Storage (duplicates) | 0 bytes | Full size |
The overhead is minimal and the storage savings are typically significant - especially for:
* Chat logs with repeated messages
* Documentation with boilerplate sections
* Logs with repeated patterns
* Meeting notes with agenda templates
***
## Best Practices
### For Most Use Cases
Let deduplication work automatically:
```bash theme={null}
# Just add content - dedup handles the rest
memvid put memory.mv2 --input ./documents/
```
### For Large Collections
Build sketch indices for faster dedup checking:
```bash theme={null}
# After initial bulk import
memvid sketch build memory.mv2 --variant medium
# Future puts will be faster
memvid put memory.mv2 --input ./new-documents/
```
### For Audit Requirements
Use unique identifiers when duplicates matter:
```python theme={null}
# Each entry gets unique URI
mem.put(
text=log_entry,
uri=f"log/{timestamp}/{uuid4()}"
)
```
***
## Troubleshooting
### "Why isn't my duplicate being detected?"
1. **Content differs slightly**: Check for hidden whitespace, encoding differences
2. **Different metadata**: URI or timestamp makes entries unique
3. **Sketch not built**: For large memories, build sketch index
```bash theme={null}
# Check if content hashes match
memvid view memory.mv2 --frame-id frame_001 --json | jq '.content_hash'
memvid view memory.mv2 --frame-id frame_002 --json | jq '.content_hash'
```
### "Why was my unique content marked as duplicate?"
SimHash can have false positives for very short content or content with similar structure:
```bash theme={null}
# Very short content may collide
echo "yes" | memvid put memory.mv2 --input -
echo "no" | memvid put memory.mv2 --input - # Might be seen as near-duplicate
```
Solution: Add distinguishing context or use unique URIs.
***
## Next Steps
<CardGroup>
<Card title="Adaptive Retrieval" icon="filter" href="/concepts/adaptive-retrieval">
Automatically determine optimal result counts
</Card>
<Card title="Indices & Tracks" icon="layer-group" href="/concepts/indexes-and-tracks">
Understand how content is indexed
</Card>
</CardGroup>
# Embedding Models
Source: https://docs.memvid.com/concepts/embedding-models
Configure embedding models for semantic search in Memvid
Memvid supports multiple embedding models for semantic (vector) search. You can use the built-in BGE-small model for local, offline operation, or connect to external providers like OpenAI, Cohere, or Voyage for higher-quality embeddings.
***
## Overview
Embeddings convert text into dense numerical vectors that capture semantic meaning. Similar concepts produce similar vectors, enabling semantic search (finding documents by meaning rather than exact keywords).
| Provider | Model | Dimensions | Best For |
| ------------ | ----------------------- | ---------- | ----------------------- |
| **Built-in** | BGE-small-en-v1.5 | 384 | Offline, privacy-first |
| **Ollama** | mxbai-embed-large | 1024 | Local, high quality |
| **Ollama** | nomic-embed-text | 768 | Local, fast |
| **OpenAI** | text-embedding-3-small | 1536 | General purpose |
| **OpenAI** | text-embedding-3-large | 3072 | Highest quality |
| **Cohere** | embed-english-v3.0 | 1024 | English documents |
| **Cohere** | embed-multilingual-v3.0 | 1024 | Multi-language |
| **Voyage** | voyage-3 | 1024 | Code and technical docs |
***
## Built-in Model (Default)
By default, Memvid uses BGE-small-en-v1.5, a lightweight embedding model that runs locally without any API keys.
### Characteristics
* **Dimensions**: 384
* **Size**: \~75 MB (downloaded on first use)
* **Inference**: CPU-based, no GPU required
* **Privacy**: All processing happens locally
* **Offline**: Works without internet after initial download
### Usage
```bash theme={null}
# CLI: Enable embeddings with built-in model
memvid put knowledge.mv2 --input document.pdf --embedding
```
```python theme={null}
# Python SDK
from memvid_sdk import create
mem = create("knowledge.mv2", enable_vec=True, enable_lex=True)
mem.put(
"Document",
"docs",
{},
text="Your content here",
enable_embedding=True,
embedding_model="bge-small",
)
```
```typescript theme={null}
// Node.js SDK
import { create } from '@memvid/sdk';
const mem = await create('knowledge.mv2');
await mem.put({ text: 'Your content here', title: 'Document', enableEmbedding: true });
```
***
## Ollama Embeddings (Local)
Ollama provides high-quality embeddings that run entirely locally on your machine. No API keys, no data leaving your infrastructure, and no usage costs.
### Setup
1. Install Ollama: [ollama.com/download](https://ollama.com/download)
2. Pull an embedding model:
```bash theme={null}
# Recommended: High quality (1024 dimensions)
ollama pull mxbai-embed-large
# Alternative: Faster, smaller (768 dimensions)
ollama pull nomic-embed-text
```
### Python SDK
```python theme={null}
from memvid_sdk import create
from memvid_sdk.embeddings import OllamaEmbeddings
# Initialize embedder (uses localhost:11434 by default)
embedder = OllamaEmbeddings(model='mxbai-embed-large')
print(f"Model: {embedder.model_name} ({embedder.dimension} dimensions)")
# Create memory with vector index
mem = create('knowledge.mv2', enable_vec=True, enable_lex=True)
# Store with embeddings
documents = [
{"title": "Doc 1", "label": "kb", "text": "Machine learning fundamentals..."},
{"title": "Doc 2", "label": "kb", "text": "Deep neural networks..."},
]
frame_ids = mem.put_many(documents, embedder=embedder)
# Search with query embedding
query = "How do neural networks work?"
results = mem.find(query, k=5, mode="sem", embedder=embedder)
```
### Node.js SDK
```typescript theme={null}
import { create, OllamaEmbeddings } from '@memvid/sdk';
// Initialize embedder
const embedder = new OllamaEmbeddings({ model: 'mxbai-embed-large' });
console.log(`Model: ${embedder.modelName} (${embedder.dimension} dimensions)`);
// Create memory
const mem = await create('knowledge.mv2', 'basic', { enableLex: true, enableVec: true });
// Ingest with embeddings
for (const doc of documents) {
const embedding = await embedder.embedQuery(doc.text);
await mem.put({
title: doc.title,
text: doc.text,
label: doc.label,
embedding,
embeddingIdentity: { provider: 'ollama', model: 'mxbai-embed-large', dimension: 1024 },
});
}
await mem.seal();
// Search
const queryEmbedding = await embedder.embedQuery('How do neural networks work?');
const results = await mem.find('neural networks', { k: 5, mode: 'auto', queryEmbedding });
```
### Supported Models
| Model | Dimensions | Speed | Quality | Use Case |
| ---------------------------- | ---------- | ------- | ------- | -------------------------- |
| `mxbai-embed-large` | 1024 | Medium | Best | Production, high accuracy |
| `nomic-embed-text` | 768 | Fast | Good | General purpose |
| `bge-m3` | 1024 | Medium | Best | Multilingual |
| `bge-large` | 1024 | Medium | Great | English documents |
| `snowflake-arctic-embed` | 1024 | Medium | Great | Retrieval-focused |
| `snowflake-arctic-embed:m` | 768 | Fast | Good | Balanced |
| `snowflake-arctic-embed:s` | 384 | Fastest | OK | Low latency |
| `all-minilm` | 384 | Fastest | OK | Lightweight |
| `e5-large` | 1024 | Medium | Great | General purpose |
| `jina-embeddings-v2-base-en` | 768 | Fast | Good | Long documents (8K tokens) |
### Custom Server
```python theme={null}
# Connect to remote Ollama server
embedder = OllamaEmbeddings(
model='mxbai-embed-large',
base_url='http://gpu-server:11434'
)
```
```typescript theme={null}
// Node.js
const embedder = new OllamaEmbeddings({
model: 'mxbai-embed-large',
baseUrl: 'http://gpu-server:11434',
});
```
### Environment Variables
| Variable | Description |
| ------------- | ----------------------------------------------------- |
| `OLLAMA_HOST` | Ollama server URL (default: `http://localhost:11434`) |
***
## OpenAI Embeddings
OpenAI's embedding models offer excellent quality for general-purpose semantic search.
### Setup
```bash theme={null}
export OPENAI_API_KEY=sk-your-key-here
```
### CLI Usage
```bash theme={null}
# Use OpenAI for embeddings
memvid put knowledge.mv2 --input document.pdf --embedding -m openai-small
# Specify exact model
memvid put knowledge.mv2 --input docs/ --embedding -m openai-large
```
### Python SDK
```python theme={null}
from memvid_sdk import create
from memvid_sdk.embeddings import OpenAIEmbeddings
# Initialize embedder
embedder = OpenAIEmbeddings(model='text-embedding-3-small')
print(f"Model: {embedder.model_name} ({embedder.dimension} dimensions)")
# Create memory with vector index
mem = create('knowledge.mv2', enable_vec=True, enable_lex=True)
# Store + embed in batch (vector index required for semantic search)
documents = [
{"title": "Doc 1", "label": "kb", "text": "Machine learning fundamentals..."},
{"title": "Doc 2", "label": "kb", "text": "Deep neural networks..."},
]
frame_ids = mem.put_many(documents, embedder=embedder)
# Search with query embedding
query = "How do neural networks work?"
results = mem.find(query, k=5, mode="sem", embedder=embedder)
```
***
## NVIDIA Embeddings
NVIDIA Integrate provides a fast hosted embedding API with OpenAI-compatible shapes.
### Setup
```bash theme={null}
export NVIDIA_API_KEY=nvapi-your-key-here
```
### Python SDK
```python theme={null}
from memvid_sdk import create
from memvid_sdk.embeddings import NvidiaEmbeddings
mem = create("knowledge.mv2", enable_vec=True, enable_lex=True)
embedder = NvidiaEmbeddings(model="nvidia/nv-embed-v1") # uses NVIDIA_API_KEY
mem.put_many(
[{"title": "Doc", "label": "kb", "text": "Vector search with NVIDIA embeddings."}],
embedder=embedder,
)
res = mem.find("nvidia embeddings", mode="sem", embedder=embedder)
```
### Node.js SDK
```typescript theme={null}
import { create, NvidiaEmbeddings } from '@memvid/sdk';
const mem = await create('knowledge.mv2');
const embedder = new NvidiaEmbeddings({ model: 'nvidia/nv-embed-v1' }); // uses NVIDIA_API_KEY
await mem.putMany([{ title: 'Doc', text: 'Vector search with NVIDIA embeddings.' }], { embedder });
const res = await mem.find('nvidia embeddings', { mode: 'sem', embedder });
```
### Node.js SDK
```typescript theme={null}
import { create, OpenAIEmbeddings } from '@memvid/sdk';
// Initialize embedder (uses OPENAI_API_KEY env var)
const embedder = new OpenAIEmbeddings({ model: 'text-embedding-3-small' });
console.log(`Model: ${embedder.modelName} (${embedder.dimension} dimensions)`);
// Create memory
const mem = await create('knowledge.mv2');
// Store + embed in batch (vector index required for semantic search)
await mem.putMany(
[
{ title: 'Doc 1', text: 'Machine learning fundamentals...' },
{ title: 'Doc 2', text: 'Deep neural networks...' },
],
{ embedder }
);
await mem.seal();
// Query using the same embedder (keeps dimensions consistent)
const results = await mem.find('How do neural networks work?', { mode: 'sem', k: 5, embedder });
```
### Model Comparison
| Model | Dimensions | Cost | Quality |
| ------------------------ | ---------- | ---------------- | ------- |
| `text-embedding-3-small` | 1536 | \$0.02/1M tokens | Good |
| `text-embedding-3-large` | 3072 | \$0.13/1M tokens | Best |
| `text-embedding-ada-002` | 1536 | \$0.10/1M tokens | Legacy |
***
## Cohere Embeddings
Cohere offers specialized models for English and multilingual content.
### Setup
```bash theme={null}
export COHERE_API_KEY=your-key-here
```
### Python SDK
```python theme={null}
from memvid_sdk.embeddings import CohereEmbeddings, get_embedder
# Direct initialization
embedder = CohereEmbeddings(model='embed-english-v3.0')
# Or use factory
embedder = get_embedder('cohere', model='embed-multilingual-v3.0')
# Generate embeddings
embeddings = embedder.embed_documents(['Text 1', 'Text 2'])
query_vec = embedder.embed_query('search query')
```
### Node.js SDK
```typescript theme={null}
import { CohereEmbeddings, getEmbedder } from '@memvid/sdk';
// Direct initialization
const embedder = new CohereEmbeddings({ model: 'embed-english-v3.0' });
// Or use factory
const embedder2 = getEmbedder('cohere', { model: 'embed-multilingual-v3.0' });
const embeddings = await embedder.embedDocuments(['Text 1', 'Text 2']);
```
### Model Options
| Model | Dimensions | Best For |
| ------------------------------- | ---------- | ----------------------- |
| `embed-english-v3.0` | 1024 | English documents |
| `embed-multilingual-v3.0` | 1024 | 100+ languages |
| `embed-english-light-v3.0` | 384 | Faster, lower cost |
| `embed-multilingual-light-v3.0` | 384 | Multi-language, lighter |
***
## Voyage Embeddings
Voyage AI specializes in embeddings for code and technical documentation.
### Setup
```bash theme={null}
export VOYAGE_API_KEY=your-key-here
```
### Python SDK
```python theme={null}
from memvid_sdk.embeddings import VoyageEmbeddings
embedder = VoyageEmbeddings(model='voyage-3')
embeddings = embedder.embed_documents(['def hello(): pass', 'function hello() {}'])
```
### Node.js SDK
```typescript theme={null}
import { VoyageEmbeddings } from '@memvid/sdk';
const embedder = new VoyageEmbeddings({ model: 'voyage-code-3' });
const embeddings = await embedder.embedDocuments(['def hello(): pass']);
```
### Model Options
| Model | Dimensions | Best For |
| --------------- | ---------- | --------------- |
| `voyage-3` | 1024 | General purpose |
| `voyage-3-lite` | 512 | Faster, smaller |
| `voyage-code-3` | 1024 | Source code |
***
## HuggingFace Embeddings (Python)
Use any HuggingFace sentence-transformer model locally.
### Setup
```bash theme={null}
pip install sentence-transformers
```
### Usage
```python theme={null}
from memvid_sdk.embeddings import get_embedder
# Use any sentence-transformers model
embedder = get_embedder('huggingface', model='all-MiniLM-L6-v2')
print(f"Model: {embedder.model_name} ({embedder.dimension} dimensions)")
embeddings = embedder.embed_documents(['Text 1', 'Text 2'])
```
### Popular Models
| Model | Dimensions | Size |
| --------------------------- | ---------- | ------ |
| `all-MiniLM-L6-v2` | 384 | 80 MB |
| `all-mpnet-base-v2` | 768 | 420 MB |
| `multi-qa-MiniLM-L6-cos-v1` | 384 | 80 MB |
***
## Using External Embeddings with Memvid
The key workflow for external embeddings:
1. **Pick an embedder** (OpenAI/Cohere/Voyage/NVIDIA/etc.)
2. **Ingest with `put_many(..., embedder=...)`** (stores embedding identity metadata)
3. **Query with `find/ask(..., embedder=...)`** (keeps dimensions consistent)
### Batch Ingestion Example
```python theme={null}
from memvid_sdk import create
from memvid_sdk.embeddings import OpenAIEmbeddings
# Setup
embedder = OpenAIEmbeddings()
mem = create('knowledge.mv2', enable_vec=True, enable_lex=True)
documents = [
{"title": "Doc 1", "label": "research", "text": "Content 1..."},
{"title": "Doc 2", "label": "research", "text": "Content 2..."},
]
frame_ids = mem.put_many(documents, embedder=embedder)
query = "What is the main finding?"
results = mem.find(query, k=10, mode="sem", embedder=embedder)
```
***
## Vector Compression
For large collections, enable vector compression to reduce storage by \~16x:
```bash theme={null}
# CLI
memvid put knowledge.mv2 --input docs/ --embedding --vector-compression
```
```python theme={null}
# Python
from memvid_sdk import create
mem = create("knowledge.mv2", enable_vec=True, enable_lex=True)
mem.put("Doc", "kb", {}, text="...", enable_embedding=True, vector_compression=True)
```
This uses Product Quantization (PQ) to compress vectors while maintaining search quality.
***
## Environment Variables
| Variable | Description |
| ------------------- | ----------------------------------------------------- |
| `OLLAMA_HOST` | Ollama server URL (default: `http://localhost:11434`) |
| `OPENAI_API_KEY` | OpenAI API key |
| `COHERE_API_KEY` | Cohere API key |
| `VOYAGE_API_KEY` | Voyage AI API key |
| `NVIDIA_API_KEY` | NVIDIA Integrate API key |
| `NVIDIA_BASE_URL` | Optional NVIDIA Integrate base URL override |
| `GOOGLE_API_KEY` | Google/Gemini API key |
| `MISTRAL_API_KEY` | Mistral API key |
| `MEMVID_MODELS_DIR` | Local model cache directory |
| `MEMVID_OFFLINE=1` | Skip model downloads |
***
## Choosing an Embedding Model
### Decision Matrix
| Requirement | Recommended |
| -------------------- | ----------------------------------------------- |
| Privacy/offline | Ollama mxbai-embed-large |
| Best quality (local) | Ollama mxbai-embed-large |
| Best quality (API) | OpenAI text-embedding-3-large |
| Cost-effective | Ollama (free) or OpenAI text-embedding-3-small |
| Multi-language | Ollama bge-m3 or Cohere embed-multilingual-v3.0 |
| Code/technical | Voyage voyage-code-3 |
| Fastest local | Ollama all-minilm |
| No setup | Built-in BGE-small |
### Performance Considerations
* **Dimension count** affects storage and search speed
* **API latency** for external providers (batch when possible)
* **Rate limits** vary by provider plan
* **Consistency** - use same model for ingestion and search
***
## Reranking
Memvid can rerank retrieved candidates using a cross-encoder model (auto-downloaded on first use). In the CLI this is applied during `ask` and can be disabled:
```bash theme={null}
memvid ask knowledge.mv2 --question "What is machine learning?" --mode hybrid --no-rerank
```
For `find`, reranking is handled internally; there is no `--rerank` flag.
***
## Next Steps
<CardGroup>
<Card title="Indices and Tracks" icon="layer-group" href="/concepts/indexes-and-tracks">
Learn about lexical, vector, and time indices
</Card>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Master semantic search queries
</Card>
</CardGroup>
# Encryption
Source: https://docs.memvid.com/concepts/encryption
Encrypt memory files with password-based protection using AES-256-GCM
Memvid supports encrypting memory files into secure capsules using industry-standard encryption. Encrypted files use the `.mv2e` extension and require a password to access.
***
## Overview
| Feature | Specification |
| ------------------ | -------------------------------------- |
| **Cipher** | AES-256-GCM (authenticated encryption) |
| **Key Derivation** | Argon2id (memory-hard, GPU-resistant) |
| **File Extension** | `.mv2e` (encrypted capsule) |
| **Compatibility** | Decrypt to use with any Memvid command |
***
## Encrypting a Memory File
### Interactive Password
```bash theme={null}
# Encrypt with interactive password prompt
memvid lock memory.mv2 --out memory.mv2e
```
```
Enter password: ••••••••••••••••
Confirm password: ••••••••••••••••
✓ Encrypted memory.mv2 → memory.mv2e
Original size: 52.4 MB
Encrypted size: 52.5 MB
Cipher: AES-256-GCM
```
### Password from Stdin (for Scripts)
```bash theme={null}
# For automation and CI/CD
echo "your-secure-password" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
# From environment variable
echo "$MEMVID_PASSWORD" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
# From file
cat /path/to/password-file | memvid lock memory.mv2 --password-stdin --out memory.mv2e
```
### Options
```bash theme={null}
# Keep original file (default: deletes original)
memvid lock memory.mv2 --out memory.mv2e --keep-original
# Overwrite existing encrypted file
memvid lock memory.mv2 --out memory.mv2e --force
# JSON output for scripting
memvid lock memory.mv2 --out memory.mv2e --json
```
JSON output:
```json theme={null}
{
"status": "success",
"source": "memory.mv2",
"destination": "memory.mv2e",
"original_size": 54938189,
"encrypted_size": 54938301,
"cipher": "AES-256-GCM",
"kdf": "Argon2id"
}
```
***
## Decrypting a Capsule
### Interactive Password
```bash theme={null}
# Decrypt with interactive password prompt
memvid unlock memory.mv2e --out memory.mv2
```
```
Enter password: ••••••••••••••••
✓ Decrypted memory.mv2e → memory.mv2
Size: 52.4 MB
```
### Password from Stdin
```bash theme={null}
# For automation
echo "your-secure-password" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
# From environment variable
echo "$MEMVID_PASSWORD" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
```
### Options
```bash theme={null}
# Overwrite existing file
memvid unlock memory.mv2e --out memory.mv2 --force
# JSON output
memvid unlock memory.mv2e --out memory.mv2 --json
```
***
## Working with Encrypted Files
Encrypted files must be decrypted before use:
```bash theme={null}
# This won't work directly
memvid find memory.mv2e --query "search" # Error: Cannot read encrypted file
# Decrypt first
memvid unlock memory.mv2e --out memory.mv2
memvid find memory.mv2 --query "search"
# Re-encrypt when done
memvid lock memory.mv2 --out memory.mv2e
```
### Workflow: Edit and Re-encrypt
```bash theme={null}
# 1. Decrypt
echo "$PASSWORD" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
# 2. Make changes
memvid put memory.mv2 --input new-document.pdf
# 3. Re-encrypt
echo "$PASSWORD" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
# 4. Original .mv2 is deleted (default behavior)
```
***
## Security Details
### AES-256-GCM
* **256-bit key**: Derived from your password via Argon2id
* **Authenticated**: Detects tampering or corruption
* **Unique nonce**: Each encryption uses a fresh random nonce
* **No metadata leakage**: File size is only indicator of content size
### Argon2id Key Derivation
* **Memory-hard**: Requires significant RAM, resists GPU attacks
* **Time-hard**: Configurable iterations for speed/security tradeoff
* **Salt**: Unique random salt per encryption
* **Winner**: Password Hashing Competition (2015)
Default parameters:
| Parameter | Value |
| ----------- | ----- |
| Memory | 64 MB |
| Iterations | 3 |
| Parallelism | 4 |
These parameters make brute-force attacks extremely expensive.
***
## Password Requirements
### Recommendations
| Requirement | Recommendation |
| ------------------ | ---------------------------- |
| **Minimum length** | 12 characters |
| **Recommended** | 16+ characters |
| **Best** | 20+ characters or passphrase |
### Strong Password Examples
```
# Random characters (use password manager)
Kj#9mP$2xL@nQ5vR
# Passphrase (easier to remember)
correct-horse-battery-staple-42
# Generated (most secure)
openssl rand -base64 24
# → "X7kP2mN9qR3sT6vY8wA1bC4d"
```
### Weak Passwords to Avoid
* Dictionary words: `password`, `memory`, `secret`
* Simple patterns: `123456`, `qwerty`, `abcdef`
* Personal info: birthdays, names, addresses
* Short passwords: anything under 12 characters
***
## Automation & CI/CD
### Environment Variables
```bash theme={null}
# Set password in environment
export MEMVID_ENCRYPTION_KEY="your-secure-password"
# Use in scripts
echo "$MEMVID_ENCRYPTION_KEY" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
echo "$MEMVID_ENCRYPTION_KEY" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
```
### GitHub Actions Example
```yaml theme={null}
name: Backup Memory
on:
schedule:
- cron: '0 0 * * *' # Daily
jobs:
backup:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Memvid
run: curl -fsSL https://get.memvid.com | sh
- name: Decrypt, update, re-encrypt
env:
MEMVID_PASSWORD: ${{ secrets.MEMVID_PASSWORD }}
run: |
echo "$MEMVID_PASSWORD" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
memvid put memory.mv2 --input ./new-data/
echo "$MEMVID_PASSWORD" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
- name: Upload encrypted backup
uses: actions/upload-artifact@v4
with:
name: encrypted-memory
path: memory.mv2e
```
### Docker Example
```dockerfile theme={null}
FROM memvid/cli:latest
# Password passed at runtime
ENV MEMVID_PASSWORD=""
COPY memory.mv2e /data/
CMD echo "$MEMVID_PASSWORD" | memvid unlock /data/memory.mv2e --password-stdin --out /data/memory.mv2 && \
memvid find /data/memory.mv2 --query "$QUERY"
```
```bash theme={null}
docker run -e MEMVID_PASSWORD="secret" -e QUERY="search term" myimage
```
***
## Use Cases
### Sensitive Documents
Encrypt memories containing confidential information:
```bash theme={null}
# HR documents
memvid create hr.mv2
memvid put hr.mv2 --input employee-records/
memvid lock hr.mv2 --out hr.mv2e
# Medical records
memvid lock patient-notes.mv2 --out patient-notes.mv2e
# Financial data
memvid lock finances.mv2 --out finances.mv2e
```
### Backup & Archive
Secure long-term storage:
```bash theme={null}
# Create encrypted backup
memvid lock knowledge.mv2 --out backups/knowledge-$(date +%Y%m%d).mv2e --keep-original
# Store password securely (password manager, vault, etc.)
```
### Sharing Encrypted Memories
Share with password communicated separately:
```bash theme={null}
# Sender
memvid lock shared-docs.mv2 --out shared-docs.mv2e
# Send shared-docs.mv2e via email/cloud
# Send password via separate secure channel
# Recipient
memvid unlock shared-docs.mv2e --out shared-docs.mv2
memvid find shared-docs.mv2 --query "search"
```
### Compliance Requirements
For HIPAA, GDPR, SOC2, etc.:
```bash theme={null}
# Encrypt at rest
memvid lock phi-data.mv2 --out phi-data.mv2e
# Log access
echo "$(date): Decrypting phi-data for user $USER" >> audit.log
memvid unlock phi-data.mv2e --out phi-data.mv2
# Re-encrypt after use
memvid lock phi-data.mv2 --out phi-data.mv2e
echo "$(date): Re-encrypted phi-data" >> audit.log
```
***
## Error Handling
### Wrong Password
```bash theme={null}
memvid unlock memory.mv2e --out memory.mv2
# Enter password: ••••••••
# Error: Decryption failed - incorrect password or corrupted file
```
### Corrupted File
```bash theme={null}
memvid unlock corrupted.mv2e --out memory.mv2
# Error: Authentication failed - file may be corrupted or tampered with
```
AES-GCM detects any modification to the encrypted file.
### File Already Exists
```bash theme={null}
memvid unlock memory.mv2e --out memory.mv2
# Error: memory.mv2 already exists. Use --force to overwrite.
# Solution
memvid unlock memory.mv2e --out memory.mv2 --force
```
***
## Best Practices
### 1. Use Strong Passwords
```bash theme={null}
# Generate secure password
openssl rand -base64 24
# Store in password manager
# Never commit passwords to version control
```
### 2. Keep Backups of Unencrypted Data
If you lose the password, data is **unrecoverable**:
```bash theme={null}
# Keep secure backup before encrypting
cp memory.mv2 /secure-backup-location/
# Then encrypt for distribution
memvid lock memory.mv2 --out memory.mv2e
```
### 3. Separate Password from Encrypted File
* Never store password in same location as encrypted file
* Use different channels (email file, text password)
* Use secrets managers (Vault, 1Password, etc.)
### 4. Rotate Passwords Periodically
```bash theme={null}
# Decrypt with old password
echo "$OLD_PASSWORD" | memvid unlock memory.mv2e --password-stdin --out memory.mv2
# Re-encrypt with new password
echo "$NEW_PASSWORD" | memvid lock memory.mv2 --password-stdin --out memory.mv2e
```
### 5. Verify After Encryption
```bash theme={null}
# Encrypt
memvid lock memory.mv2 --out memory.mv2e
# Verify by decrypting to temp location
memvid unlock memory.mv2e --out /tmp/verify.mv2
memvid stats /tmp/verify.mv2 # Should match original
rm /tmp/verify.mv2
```
***
## Limitations
| Limitation | Description |
| --------------------- | ------------------------------------------------- |
| **No streaming** | Must decrypt entire file to access |
| **No partial access** | Can't read individual frames without full decrypt |
| **Password only** | No key file or hardware key support (yet) |
| **No key escrow** | Lost password = lost data |
***
## Future Features
Coming soon:
* Key file support (in addition to password)
* Hardware security module (HSM) integration
* Partial decryption for large files
* Key rotation without full re-encryption
***
## Next Steps
<CardGroup>
<Card title="Security & Compliance" icon="shield-check" href="/faq/security-and-compliance">
Security FAQ and compliance info
</Card>
<Card title="CLI Reference" icon="terminal" href="/cli/advanced-commands">
Full CLI command reference
</Card>
</CardGroup>
# Entity Extraction (Logic Mesh)
Source: https://docs.memvid.com/concepts/entity-extraction
Extract entities and relationships for structured knowledge management
Memvid supports Named Entity Recognition (NER) for extracting structured entities from documents. This enables building knowledge graphs, entity-based search, and relationship mapping, turning unstructured text into structured knowledge.
***
## Overview
Entity extraction identifies and classifies named entities in text:
* **People**: Names of individuals (CEO, executives, authors)
* **Organizations**: Companies, institutions, agencies
* **Locations**: Cities, countries, addresses
* **Dates**: Temporal references, deadlines, events
* **Money**: Currency amounts, valuations, prices
* **Custom types**: Domain-specific entities (products, deals, regulations)
| Provider | Model | Entity Types | Best For |
| ---------- | ----------------- | --------------------------- | ------------------------------ |
| **Local** | DistilBERT-NER | PERSON, ORG, LOCATION, MISC | Offline, privacy-first |
| **OpenAI** | GPT-4o-mini | Custom | High accuracy, custom entities |
| **OpenAI** | GPT-4o | Custom | Best quality |
| **Claude** | Claude 3.5 Sonnet | Custom | Nuanced extraction |
| **Gemini** | Gemini 2.0 Flash | Custom | Fast, cost-effective |
***
## Quick Start
### Python SDK
```python theme={null}
from memvid_sdk import create
from memvid_sdk.entities import get_entity_extractor
# Initialize entity extractor
ner = get_entity_extractor('openai', entity_types=['COMPANY', 'PERSON', 'MONEY', 'DATE'])
print(f"Provider: {ner.name}")
print(f"Entity types: {ner.entity_types}")
# Extract entities from text
text = """
Microsoft CEO Satya Nadella announced a $50 million investment in Seattle.
The deal closes December 2024 with Pinnacle Financial as lead investor.
"""
entities = ner.extract(text, min_confidence=0.5)
for entity in entities:
print(f" {entity['name']} ({entity['type']}, {entity['confidence']:.2f})")
# Output:
# Microsoft (COMPANY, 0.95)
# Satya Nadella (PERSON, 0.97)
# $50 million (MONEY, 0.95)
# Seattle (LOCATION, 0.90)
# December 2024 (DATE, 0.88)
# Pinnacle Financial (COMPANY, 0.92)
```
### Node.js SDK
```typescript theme={null}
import { create, getEntityExtractor } from '@memvid/sdk';
// Initialize entity extractor
const ner = getEntityExtractor('openai', {
entityTypes: ['COMPANY', 'PERSON', 'MONEY', 'DATE'],
});
console.log(`Provider: ${ner.name}`);
console.log(`Entity types: ${ner.entityTypes}`);
// Extract entities from text
const text = `
Microsoft CEO Satya Nadella announced a $50 million investment in Seattle.
The deal closes December 2024 with Pinnacle Financial as lead investor.
`;
const entities = await ner.extract(text, 0.5);
for (const entity of entities) {
console.log(` ${entity.name} (${entity.type}, ${entity.confidence.toFixed(2)})`);
}
```
***
## Providers
### Local NER (DistilBERT)
The default provider uses DistilBERT-NER, a lightweight model for offline entity extraction.
**Characteristics:**
* **Model**: DistilBERT fine-tuned on CoNLL-03
* **Size**: \~261 MB (downloaded on first use)
* **Entity types**: PERSON, ORG, LOCATION, MISC (fixed)
* **Inference**: CPU-based, no GPU required
* **Privacy**: All processing happens locally
```python theme={null}
from memvid_sdk.entities import get_entity_extractor, LocalNER
# Using factory
ner = get_entity_extractor('local')
# Or direct instantiation
ner = LocalNER(model='distilbert-ner')
# Extract entities
entities = ner.extract("Apple CEO Tim Cook visited Paris headquarters.")
# [
# {'name': 'Apple', 'type': 'ORG', 'confidence': 0.98},
# {'name': 'Tim Cook', 'type': 'PERSON', 'confidence': 0.97},
# {'name': 'Paris', 'type': 'LOCATION', 'confidence': 0.95},
# ]
```
```typescript theme={null}
import { getEntityExtractor, LocalNER } from '@memvid/sdk';
const ner = getEntityExtractor('local');
const entities = await ner.extract('Apple CEO Tim Cook visited Paris headquarters.');
```
<Note>
Local NER uses fixed entity types (PERSON, ORG, LOCATION, MISC). For custom entity types, use cloud providers. In Node.js, `LocalNER` requires a native build that exports `NerModel` (the prebuilt npm binaries may not include it).
</Note>
***
### OpenAI Entities
OpenAI's models provide high-accuracy extraction with custom entity types.
**Setup:**
```bash theme={null}
export OPENAI_API_KEY=sk-your-key-here
```
**Usage:**
```python theme={null}
from memvid_sdk.entities import get_entity_extractor, OpenAIEntities
# Using factory with custom entity types
ner = get_entity_extractor('openai', entity_types=[
'COMPANY',
'PERSON',
'LOCATION',
'MONEY',
'DATE',
'PRODUCT',
'DEAL_TYPE',
])
# Or with specific model
ner = get_entity_extractor('openai:gpt-4o-mini', entity_types=['COMPANY', 'PERSON'])
# Direct instantiation
ner = OpenAIEntities(
model='gpt-4o-mini',
entity_types=['COMPANY', 'EXECUTIVE', 'PRODUCT'],
)
# Extract entities
entities = ner.extract(text, min_confidence=0.5)
```
```typescript theme={null}
import { getEntityExtractor, OpenAIEntities } from '@memvid/sdk';
const ner = getEntityExtractor('openai', {
entityTypes: ['COMPANY', 'PERSON', 'LOCATION', 'MONEY', 'DATE'],
});
// Or with specific model
const ner = getEntityExtractor('openai:gpt-4o-mini', {
entityTypes: ['COMPANY', 'PERSON'],
});
const entities = await ner.extract(text, 0.5);
```
**Model Comparison:**
| Model | Speed | Quality |
| ------------- | ------ | --------- |
| `gpt-4o-mini` | Fast | Good |
| `gpt-4o` | Medium | Best |
| `gpt-4-turbo` | Medium | Excellent |
***
### Claude Entities
Anthropic's Claude excels at nuanced entity extraction with context understanding.
**Setup:**
```bash theme={null}
export ANTHROPIC_API_KEY=your-key-here
```
**Usage:**
```python theme={null}
from memvid_sdk.entities import get_entity_extractor, ClaudeEntities
# Using factory
ner = get_entity_extractor('claude', entity_types=['COMPANY', 'PERSON', 'REGULATION'])
# With specific model
ner = get_entity_extractor('claude:claude-3-5-sonnet-20241022', entity_types=['COMPANY'])
# Direct instantiation
ner = ClaudeEntities(
model='claude-3-5-sonnet-20241022',
entity_types=['COMPANY', 'EXECUTIVE', 'DEAL'],
)
entities = ner.extract(text, min_confidence=0.6)
```
```typescript theme={null}
import { getEntityExtractor, ClaudeEntities } from '@memvid/sdk';
const ner = getEntityExtractor('claude', {
entityTypes: ['COMPANY', 'PERSON', 'REGULATION'],
});
const entities = await ner.extract(text, 0.6);
```
***
### Gemini Entities
Google's Gemini provides fast, cost-effective entity extraction.
**Setup:**
```bash theme={null}
export GEMINI_API_KEY=your-key-here
```
**Usage:**
```python theme={null}
from memvid_sdk.entities import get_entity_extractor, GeminiEntities
# Using factory
ner = get_entity_extractor('gemini', entity_types=['COMPANY', 'PERSON'])
# With specific model
ner = get_entity_extractor('gemini:gemini-2.0-flash', entity_types=['COMPANY'])
entities = ner.extract(text, min_confidence=0.5)
```
```typescript theme={null}
import { getEntityExtractor, GeminiEntities } from '@memvid/sdk';
const ner = getEntityExtractor('gemini', {
entityTypes: ['COMPANY', 'PERSON'],
});
const entities = await ner.extract(text, 0.5);
```
***
## Complete Example
Here's a full workflow for document entity extraction:
```python theme={null}
from pathlib import Path
from memvid_sdk import create
from memvid_sdk.entities import get_entity_extractor
# Configuration
PROVIDER = 'openai'
ENTITY_TYPES = ['COMPANY', 'PERSON', 'LOCATION', 'MONEY', 'DATE', 'DEAL_TYPE']
DATASET_DIR = Path('documents/')
OUTPUT_PATH = 'knowledge_base.mv2'
# Initialize
ner = get_entity_extractor(PROVIDER, entity_types=ENTITY_TYPES)
print(f"Entity Extractor: {ner.name}")
print(f"Entity Types: {', '.join(ner.entity_types)}")
# Create memory
if Path(OUTPUT_PATH).exists():
Path(OUTPUT_PATH).unlink()
mem = create(OUTPUT_PATH)
mem.enable_lex()
# Process documents
all_entities = []
pdf_files = list(DATASET_DIR.glob('*.pdf'))
for i, pdf_path in enumerate(pdf_files):
print(f"\n[{i+1}/{len(pdf_files)}] {pdf_path.name}")
# Store document
frame_id = mem.put(
title=pdf_path.stem.replace('_', ' ').title(),
label='document',
metadata={},
file=str(pdf_path),
)
print(f" Stored as frame {frame_id}")
# Extract entities (from document text or summary)
document_text = f"Document: {pdf_path.stem}" # Replace with actual text extraction
entities = ner.extract(document_text, min_confidence=0.5)
print(f" Found {len(entities)} entities:")
for e in entities[:4]:
print(f" - {e['name']} ({e['type']}, {e['confidence']:.2f})")
all_entities.extend(entities)
# Entity statistics
print("\n--- Entity Summary ---")
counts = {}
for e in all_entities:
t = e.get('type', 'UNKNOWN')
counts[t] = counts.get(t, 0) + 1
for entity_type, count in sorted(counts.items(), key=lambda x: -x[1]):
print(f" {entity_type}: {count}")
# Seal
mem.seal()
stats = mem.stats()
print(f"\nFinal: {stats.get('frame_count', 0)} frames, {len(all_entities)} entities")
```
***
## Custom Entity Types
Cloud providers support custom entity types tailored to your domain:
### Finance Domain
```python theme={null}
ner = get_entity_extractor('openai', entity_types=[
'COMPANY',
'INVESTOR',
'FUND',
'MONEY',
'DEAL_TYPE', # IPO, M&A, Series A
'VALUATION',
'EXECUTIVE',
'DATE',
])
```
### Legal Domain
```python theme={null}
ner = get_entity_extractor('claude', entity_types=[
'PARTY',
'COURT',
'JUDGE',
'CASE_NUMBER',
'STATUTE',
'DATE',
'JURISDICTION',
])
```
### Healthcare Domain
```python theme={null}
ner = get_entity_extractor('openai:gpt-4o', entity_types=[
'PATIENT',
'PROVIDER',
'MEDICATION',
'DIAGNOSIS',
'PROCEDURE',
'DATE',
'FACILITY',
])
```
***
## API Reference
### EntityExtractor Interface
All entity extractors implement this interface:
| Method | Description |
| -------------------------------------- | ------------------------------------------------ |
| `name` | Provider identifier (e.g., `openai:gpt-4o-mini`) |
| `entity_types` | List of supported entity types |
| `extract(text, min_confidence)` | Extract entities from text |
| `extract_batch(texts, min_confidence)` | Batch extract from multiple texts |
### Entity Object
Each extracted entity contains:
| Field | Type | Description |
| ------------ | ------ | -------------------------- |
| `name` | string | Entity text as it appears |
| `type` | string | Entity classification |
| `confidence` | float | Confidence score (0.0-1.0) |
### Factory Function
```python theme={null}
# Python
from memvid_sdk.entities import get_entity_extractor
ner = get_entity_extractor(
provider, # 'local', 'openai', 'claude', 'gemini', 'openai:model-name'
entity_types=None, # Custom entity types (cloud providers only)
api_key=None, # Override env var
)
```
```typescript theme={null}
// Node.js
import { getEntityExtractor } from '@memvid/sdk';
const ner = getEntityExtractor(provider, {
entityTypes: ['COMPANY', 'PERSON'], // Custom entity types
apiKey: undefined, // Override env var
});
```
***
## Environment Variables
| Variable | Description |
| ------------------- | -------------------------------- |
| `OPENAI_API_KEY` | OpenAI API key |
| `ANTHROPIC_API_KEY` | Anthropic API key for Claude |
| `GEMINI_API_KEY` | Google AI API key for Gemini |
| `MEMVID_MODELS_DIR` | Local model cache directory |
| `MEMVID_OFFLINE=1` | Skip model downloads (local NER) |
***
## Use Cases
### Document Intelligence
Extract structured data from unstructured documents:
```python theme={null}
# Process legal contracts
ner = get_entity_extractor('claude', entity_types=[
'PARTY', 'DATE', 'MONEY', 'TERM', 'JURISDICTION'
])
contract_text = "Agreement between Acme Corp and Beta Inc dated January 15, 2024..."
entities = ner.extract(contract_text)
# Build structured contract summary
parties = [e['name'] for e in entities if e['type'] == 'PARTY']
dates = [e['name'] for e in entities if e['type'] == 'DATE']
```
### Knowledge Graph Building
Create entity-relationship graphs from documents:
```python theme={null}
# Extract entities from multiple documents
all_entities = []
for doc in documents:
entities = ner.extract(doc.text)
for e in entities:
e['source_doc'] = doc.id
all_entities.extend(entities)
# Build co-occurrence graph
from collections import defaultdict
co_occurrences = defaultdict(int)
for doc_id in set(e['source_doc'] for e in all_entities):
doc_entities = [e for e in all_entities if e['source_doc'] == doc_id]
for i, e1 in enumerate(doc_entities):
for e2 in doc_entities[i+1:]:
pair = tuple(sorted([e1['name'], e2['name']]))
co_occurrences[pair] += 1
```
### Entity-Based Search
Find documents by entity type:
```python theme={null}
# Store entities with documents
for doc in documents:
entities = ner.extract(doc.text)
frame_id = mem.put(
title=doc.title,
label='document',
metadata={
'entities': entities,
'companies': [e['name'] for e in entities if e['type'] == 'COMPANY'],
'people': [e['name'] for e in entities if e['type'] == 'PERSON'],
},
text=doc.text,
)
# Search by entity
results = mem.find('Microsoft', k=10)
```
### Deal Memo Analysis
Extract structured deal information:
```python theme={null}
ner = get_entity_extractor('openai', entity_types=[
'COMPANY', 'INVESTOR', 'MONEY', 'DEAL_TYPE', 'DATE', 'LOCATION'
])
deal_text = """
Series B Funding: Atlas Logistics
Atlas Logistics, headquartered in Seattle, announced a $50 million Series B round.
Lead investor Pinnacle Capital. Deal closes Q1 2025.
"""
entities = ner.extract(deal_text)
# Structured output:
# - COMPANY: Atlas Logistics
# - LOCATION: Seattle
# - MONEY: $50 million
# - DEAL_TYPE: Series B
# - INVESTOR: Pinnacle Capital
# - DATE: Q1 2025
```
***
## Best Practices
1. **Choose appropriate entity types**: Define types specific to your domain
2. **Set confidence thresholds**: Use higher thresholds (0.7+) for critical applications
3. **Batch extraction**: Use `extract_batch()` for multiple texts
4. **Cache results**: Store extracted entities in document metadata
5. **Validate entities**: Review extracted entities for accuracy in critical workflows
6. **Use local for privacy**: Local NER processes data entirely on-device
***
## Limitations
* **Local NER**: Fixed entity types (PERSON, ORG, LOCATION, MISC)
* **Local NER**: Python SDK only (Node.js uses cloud providers)
* **Cloud providers**: Require API keys and internet connection
* **Rate limits**: Cloud providers have rate limits based on plan
* **Context length**: Very long texts may need chunking
***
## Next Steps
<CardGroup>
<Card title="Visual Embeddings" icon="image" href="/concepts/visual-embeddings">
Enable image and visual search with CLIP
</Card>
<Card title="Embedding Models" icon="brain" href="/concepts/embedding-models">
Configure text embedding models for semantic search
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Complete Python SDK reference
</Card>
<Card title="Node.js SDK" icon="node-js" href="/node-sdk/overview">
Complete Node.js SDK reference
</Card>
</CardGroup>
# Graph Search & Logic Mesh
Source: https://docs.memvid.com/concepts/graph-search
Query entity relationships and traverse knowledge graphs extracted from your documents
Logic Mesh extracts entity-relationship graphs from your documents, enabling powerful graph traversal and relationship-aware search. Instead of just finding documents that mention "John", you can ask "Who reports to John?" or "What companies has John worked at?"
***
## What is Logic Mesh?
Logic Mesh automatically extracts:
* **Entities**: People, companies, products, concepts
* **Relationships**: works\_at, reports\_to, founded, acquired, etc.
* **Properties**: Attributes attached to entities
```mermaid theme={null}
graph TD
A[John Smith] -->|works_at| B[Acme Corp]
A -->|reports_to| C[Jane Doe]
B -->|acquired| D[StartupXYZ]
```
***
## Enabling Logic Mesh
### During Ingestion
```bash theme={null}
# Enable entity-relationship extraction
memvid put memory.mv2 --input documents/ --logic-mesh
```
### For Existing Memories
```bash theme={null}
# Enrich existing documents with graph extraction
memvid enrich memory.mv2 --engine groq
```
<Note>
Logic Mesh extraction requires an enrichment engine. Use `--logic-mesh` during ingestion or run `memvid enrich` afterward.
</Note>
***
## Graph Traversal Commands
### List All Entities
```bash theme={null}
# Show all extracted entities
memvid follow entities memory.mv2
# Filter by entity type
memvid follow entities memory.mv2 --kind person
memvid follow entities memory.mv2 --kind company
memvid follow entities memory.mv2 --kind product
# Search entities by name
memvid follow entities memory.mv2 --query "John"
# Limit results
memvid follow entities memory.mv2 --limit 20
# JSON output
memvid follow entities memory.mv2 --json
```
Output:
```
Entities (87 total)
People:
- John Smith (34 relationships)
- Jane Doe (28 relationships)
- Sarah Chen (19 relationships)
Companies:
- Acme Corp (45 relationships)
- StartupXYZ (12 relationships)
- TechGiant Inc (8 relationships)
Products:
- Project Alpha (15 relationships)
- Platform v2 (9 relationships)
```
### Traverse Relationships
```bash theme={null}
# Find what John Smith is connected to
memvid follow traverse memory.mv2 --start "John Smith"
# Follow specific relationship type
memvid follow traverse memory.mv2 --start "John Smith" --link works_at
# Control traversal depth (default: 2)
memvid follow traverse memory.mv2 --start "Acme Corp" --hops 3
# Direction: outgoing, incoming, or both
memvid follow traverse memory.mv2 --start "Jane Doe" --direction incoming
```
Output:
```
Traversal from: John Smith (depth: 2)
Direct relationships (hop 1):
──[works_at]──▶ Acme Corp
──[reports_to]──▶ Jane Doe
──[leads]──▶ Project Alpha
──[expertise]──▶ Rust, Python
Extended relationships (hop 2):
Acme Corp ──[acquired]──▶ StartupXYZ
Acme Corp ──[headquartered_in]──▶ San Francisco
Jane Doe ──[reports_to]──▶ CEO Board
Project Alpha ──[uses]──▶ Platform v2
```
### Graph Statistics
```bash theme={null}
memvid follow stats memory.mv2
```
Output:
```
Logic Mesh Statistics
Entities: 87
- person: 34
- company: 18
- product: 15
- concept: 20
Relationships: 423
- works_at: 45
- reports_to: 32
- founded: 12
- acquired: 8
- uses: 56
- expertise: 89
- other: 181
Graph density: 4.86 edges/node
Connected components: 3
Largest component: 72 entities
```
***
## Triple Patterns
Query relationships using **Subject:Predicate:Object** patterns with `?` as a wildcard:
| Pattern | Meaning |
| ------------------- | ---------------------------- |
| `John:works_at:?` | Where does John work? |
| `?:CEO:Acme` | Who is CEO of Acme? |
| `?:works_at:Acme` | Who works at Acme? |
| `John:?:Acme` | How is John related to Acme? |
| `?:reports_to:Jane` | Who reports to Jane? |
### CLI Usage
```bash theme={null}
# Find where John works
memvid find memory.mv2 --graph "John Smith:works_at:?"
# Find who works at Acme
memvid find memory.mv2 --graph "?:works_at:Acme Corp"
# Find all relationships between two entities
memvid find memory.mv2 --graph "John Smith:?:Acme Corp"
```
***
## Combining Graph + Text Search
The real power of Logic Mesh is combining graph traversal with text search.
### Graph-Filtered Search
Filter search results by relationships first, then rank by text relevance:
```bash theme={null}
# Find revenue info, but only from Acme-related documents
memvid find memory.mv2 --query "revenue" --graph "?:works_at:Acme Corp"
# Find meeting notes involving John
memvid find memory.mv2 --query "meeting notes" --graph "John Smith:?:?"
```
### Hybrid Graph + Semantic Search
Combine graph traversal with vector similarity:
```bash theme={null}
# Semantic search within a relationship context
memvid find memory.mv2 --query "Q4 results" --hybrid --graph "?:works_at:Acme"
# Find similar documents to what relates to a topic
memvid find memory.mv2 --query "machine learning" --hybrid --graph "?:expertise:ML"
```
***
## Common Relationship Types
Logic Mesh automatically detects common relationships:
### Professional
| Relationship | Example |
| ------------ | ------------------------ |
| `works_at` | John works\_at Acme |
| `reports_to` | John reports\_to Jane |
| `manages` | Jane manages Engineering |
| `founded` | Elon founded SpaceX |
| `CEO` | Tim CEO Apple |
### Organizational
| Relationship | Example |
| ------------------ | ------------------------------ |
| `acquired` | Acme acquired StartupXYZ |
| `partnered_with` | Acme partnered\_with TechCo |
| `subsidiary_of` | StartupXYZ subsidiary\_of Acme |
| `headquartered_in` | Acme headquartered\_in SF |
### Technical
| Relationship | Example |
| ----------------- | -------------------------------- |
| `uses` | Project uses React |
| `depends_on` | Service depends\_on Database |
| `integrates_with` | Platform integrates\_with Stripe |
| `expertise` | John expertise Rust |
### Temporal
| Relationship | Example |
| ------------ | ---------------------------- |
| `started` | Project started 2024-01 |
| `completed` | Phase completed 2024-06 |
| `scheduled` | Meeting scheduled 2024-12-15 |
***
## SDK Usage
### Python
```python theme={null}
from memvid import use
mem = use('basic', 'memory.mv2')
# Enable logic mesh during put
mem.put(
text="John Smith works at Acme Corp as a Senior Engineer.",
logic_mesh=True
)
# List entities
entities = mem.get_entities()
for entity in entities:
print(f"{entity.name} ({entity.kind}): {entity.relationship_count} relationships")
# Traverse from an entity
graph = mem.traverse(
start="John Smith",
link="works_at",
hops=2,
direction="outgoing"
)
for node in graph.nodes:
print(f"{node.name}: {node.relationships}")
# Graph-filtered search
results = mem.find(
"quarterly report",
graph_pattern="?:works_at:Acme Corp"
)
# Hybrid search
results = mem.find(
"machine learning",
hybrid=True,
graph_pattern="?:expertise:ML"
)
```
### Node.js
```typescript theme={null}
import { use } from '@anthropics/memvid'
const mem = await use('basic', 'memory.mv2')
// Enable logic mesh during put
await mem.put({
content: "John Smith works at Acme Corp as a Senior Engineer.",
logicMesh: true
})
// List entities
const entities = await mem.getEntities()
for (const entity of entities) {
console.log(`${entity.name} (${entity.kind}): ${entity.relationshipCount} relationships`)
}
// Traverse from an entity
const graph = await mem.traverse({
start: "John Smith",
link: "works_at",
hops: 2,
direction: "outgoing"
})
// Graph-filtered search
const results = await mem.find("quarterly report", {
graphPattern: "?:works_at:Acme Corp"
})
```
***
## Advanced Patterns
### Multi-Hop Queries
Find entities connected through intermediate nodes:
```bash theme={null}
# Who do people at Acme report to? (2 hops)
memvid follow traverse memory.mv2 --start "Acme Corp" --link "works_at,reports_to" --hops 2
# Find the CEO through org chart
memvid follow traverse memory.mv2 --start "John Smith" --link "reports_to" --hops 5
```
### Compound Patterns
Combine multiple patterns:
```bash theme={null}
# Find people who work at Acme AND have ML expertise
memvid find memory.mv2 --graph "?:works_at:Acme Corp" --graph "?:expertise:ML"
```
### Negation (Exclusion)
Find entities NOT matching a pattern:
```bash theme={null}
# Find all employees except those reporting to Jane
memvid follow entities memory.mv2 --kind person --exclude-pattern "?:reports_to:Jane"
```
***
## Export Graph Data
### N-Triples (RDF)
```bash theme={null}
memvid export memory.mv2 --format ntriples --out graph.nt
```
```
<John_Smith> <works_at> <Acme_Corp> .
<John_Smith> <reports_to> <Jane_Doe> .
<John_Smith> <expertise> "Rust" .
<Acme_Corp> <acquired> <StartupXYZ> .
```
### JSON Graph
```bash theme={null}
memvid export memory.mv2 --format json --out graph.json
```
```json theme={null}
{
"nodes": [
{"id": "John_Smith", "kind": "person", "properties": {...}},
{"id": "Acme_Corp", "kind": "company", "properties": {...}}
],
"edges": [
{"source": "John_Smith", "target": "Acme_Corp", "relation": "works_at"},
{"source": "John_Smith", "target": "Jane_Doe", "relation": "reports_to"}
]
}
```
### CSV
```bash theme={null}
memvid export memory.mv2 --format csv --out graph.csv
```
```csv theme={null}
subject,predicate,object,confidence,source_frame
John_Smith,works_at,Acme_Corp,0.94,frame_001
John_Smith,reports_to,Jane_Doe,0.91,frame_001
Acme_Corp,acquired,StartupXYZ,0.88,frame_045
```
***
## Use Cases
### Organizational Knowledge
Map your company structure:
```bash theme={null}
# Ingest org documents
memvid put org.mv2 --input hr_docs/ --logic-mesh
# Find everyone's reporting structure
memvid follow traverse org.mv2 --start "CEO" --link "manages" --hops 5 --direction outgoing
# Who reports to a specific manager?
memvid find org.mv2 --graph "?:reports_to:Sarah Chen"
```
### Research Papers
Build citation and concept graphs:
```bash theme={null}
# Ingest papers
memvid put research.mv2 --input papers/ --logic-mesh
# Find papers citing a specific work
memvid find research.mv2 --graph "?:cites:Attention Is All You Need"
# Find concepts related to transformers
memvid follow traverse research.mv2 --start "Transformers" --hops 2
```
### Customer Data
Track customer relationships:
```bash theme={null}
# Find all contacts at a company
memvid find customers.mv2 --graph "?:works_at:BigCorp Inc"
# Find decision makers
memvid find customers.mv2 --graph "?:role:VP" --graph "?:works_at:BigCorp Inc"
```
### Code Documentation
Map code dependencies:
```bash theme={null}
# Find what depends on a module
memvid find codebase.mv2 --graph "?:imports:auth_module"
# Find all API endpoints
memvid follow entities codebase.mv2 --kind endpoint
```
***
## Performance Tips
### Index Recommendations
For large graphs (1000+ entities):
```bash theme={null}
# Build graph index for faster traversals
memvid doctor memory.mv2 --rebuild-logic-mesh
```
### Limit Traversal Depth
Deep traversals are expensive:
```bash theme={null}
# Default (recommended)
--hops 2
# Use sparingly
--hops 3 or higher
```
### Filter Early
Apply graph filters before text search:
```bash theme={null}
# Good: Graph filter narrows results first
memvid find memory.mv2 --graph "?:works_at:Acme" --query "report"
# Less efficient: Broad text search then filter
memvid find memory.mv2 --query "report" | grep Acme
```
***
## Troubleshooting
### "No entities found"
1. Ensure Logic Mesh is enabled:
```bash theme={null}
memvid put memory.mv2 --input docs/ --logic-mesh
```
2. Run enrichment:
```bash theme={null}
memvid enrich memory.mv2 --engine groq
```
3. Check stats:
```bash theme={null}
memvid follow stats memory.mv2
```
### "Relationship not detected"
1. Content may be too ambiguous
2. Try a better enrichment engine:
```bash theme={null}
memvid enrich memory.mv2 --engine claude --force
```
### "Traversal too slow"
1. Reduce hop depth
2. Rebuild logic mesh index:
```bash theme={null}
memvid doctor memory.mv2 --rebuild-logic-mesh
```
***
## Next Steps
<CardGroup>
<Card title="Memory Cards" icon="id-card" href="/concepts/memory-cards">
Entity-attribute-value triples from enrichment
</Card>
<Card title="Entity Extraction" icon="brain" href="/concepts/entity-extraction">
How entities are detected and classified
</Card>
</CardGroup>
# Indices and Tracks
Source: https://docs.memvid.com/concepts/indexes-and-tracks
Lexical, vector, and temporal search primitives in Memvid
Memvid uses three complementary index types to enable fast, intelligent search across your documents. Each index serves a different purpose and can be enabled or disabled based on your needs.
***
## Index Overview
| Index | Engine | Purpose | Best For |
| ----------- | ------------- | -------------------------- | ------------------------------- |
| **Lexical** | BM25 | Full-text keyword search | Exact terms, error codes, names |
| **Vector** | Vector search | Semantic similarity search | Natural language, concepts |
| **Time** | Sorted tuples | Chronological ordering | Timeline queries, auditing |
All three indices are embedded directly in the `.mv2` file. No external dependencies or sidecar files.
***
## Lexical Index
The lexical index powers fast, precise keyword search using BM25, a proven ranking algorithm for full-text search.
### How It Works
* **BM25 ranking**: Scores documents by term frequency and inverse document frequency
* **Tokenization**: Breaks text into searchable terms
* **Memory-mapped**: Uses mmap for efficient disk access
* **Embedded**: Stored as a snapshot inside the `.mv2` file
### When to Use
Lexical search excels at finding exact matches:
```bash theme={null}
# Find exact error codes
memvid find knowledge.mv2 --query "ERR_CONNECTION_REFUSED" --mode lex
# Find function names
memvid find knowledge.mv2 --query "handleAuthentication" --mode lex
# Date range queries
memvid find knowledge.mv2 --query "date:[2024-01-01 TO 2024-12-31]" --mode lex
```
### Building the Index
The lexical index is built automatically when you add documents. You can also rebuild it:
```bash theme={null}
# Rebuild lexical index
memvid doctor knowledge.mv2 --rebuild-lex-index
# Check index status
memvid stats knowledge.mv2 --json | grep has_lex_index
```
### Disabling Lexical Index
For vector-only workloads, you can disable lexical indexing:
```bash theme={null}
# Create without lexical index
memvid create knowledge.mv2 --no-lex
```
```python theme={null}
# Python SDK
mem = use('basic', 'knowledge.mv2', enable_lex=False)
```
***
## Vector Index
The vector index enables semantic search, finding documents by meaning rather than exact keywords.
### How It Works
* **Embeddings**: Documents are converted to dense vectors (default: BGE-small, 384 dimensions)
* **External providers**: Support for OpenAI, Cohere, Voyage, and HuggingFace models
* **Vector graph**: Fast approximate nearest neighbor search for semantic similarity
* **Product Quantization (PQ)**: Optional 16x compression for large collections
* **Embedded**: Stored as segments inside the `.mv2` file
### Embedding Model Options
| Model | Dimensions | Description |
| ----------------------------- | ---------- | ----------------------------- |
| BGE-small (default) | 384 | Built-in, offline, no API key |
| OpenAI text-embedding-3-small | 1536 | High quality, general purpose |
| OpenAI text-embedding-3-large | 3072 | Highest quality |
| Cohere embed-english-v3.0 | 1024 | English documents |
| Voyage voyage-3 | 1024 | Code and technical docs |
See [Embedding Models](/concepts/embedding-models) for detailed configuration.
### When to Use
Vector search excels at understanding intent:
```bash theme={null}
# Natural language questions
memvid find knowledge.mv2 --query "how do users log in" --mode sem
# Conceptual queries
memvid find knowledge.mv2 --query "best practices for security" --mode sem
# Find similar content
memvid find knowledge.mv2 --query "machine learning model training" --mode sem
```
### Building the Index
Enable embeddings when adding documents:
```bash theme={null}
# Add with embeddings
memvid put knowledge.mv2 --input document.pdf --vector-compression
# Add with compression (16x smaller vectors)
memvid put knowledge.mv2 --input document.pdf --vector-compression
```
```python theme={null}
# Python SDK
mem.put(text="Content", title="Doc", enable_embedding=True)
# With compression
mem.put(text="Content", title="Doc", enable_embedding=True, vector_compression=True)
```
### Rebuilding the Index
If vector search isn't working correctly:
```bash theme={null}
# Rebuild vector index
memvid doctor knowledge.mv2 --rebuild-vec-index
# Check index status
memvid stats knowledge.mv2 --json | grep has_vec_index
```
### Direct Vector Search
For custom embeddings from your own model:
```bash theme={null}
# Search with pre-computed vector
memvid vec-search knowledge.mv2 --vector "0.1,0.2,0.3,..." --limit 10
# Search with embedding file
memvid vec-search knowledge.mv2 --embedding ./query-embedding.json --limit 5
```
***
## Time Index
The time index enables chronological queries and time-travel features.
### How It Works
* **Sorted tuples**: Stores `(timestamp, frame_id)` pairs in sorted order
* **MVTI magic**: Identified by `MVTI` header bytes
* **O(log n) lookups**: Binary search for efficient time range queries
* **Checksummed**: Protected by integrity verification
### When to Use
Time-based access patterns:
```bash theme={null}
# Browse recent documents
memvid timeline knowledge.mv2 --limit 20
# Filter by time range
memvid timeline knowledge.mv2 --since 1704067200 --until 1706745600
# Reverse chronological order
memvid timeline knowledge.mv2 --reverse
```
### Time-Travel Queries
View your memory as it existed at a point in time:
```bash theme={null}
# Search as of a specific frame
memvid find knowledge.mv2 --query "config" --as-of-frame 100
# Search as of a specific timestamp
memvid find knowledge.mv2 --query "config" --as-of-ts 1704067200
# Timeline at a specific frame
memvid timeline knowledge.mv2 --as-of-frame 50
```
```python theme={null}
# Python SDK time-travel
results = mem.find('config', as_of_frame=100)
results = mem.find('config', as_of_ts=1704067200)
```
### Rebuilding the Time Index
If timeline queries return incorrect results:
```bash theme={null}
# Rebuild time index
memvid doctor knowledge.mv2 --rebuild-time-index
# Verify time index
memvid verify knowledge.mv2 --deep
```
***
## Hybrid Search
Hybrid search (mode `auto`) combines lexical and semantic results for the best of both worlds.
### How It Works
1. **Parallel query**: Both lexical and vector indices are queried
2. **Result fusion**: Scores are combined using reciprocal rank fusion
3. **Reranking**: Top results are reranked for relevance
4. **Deduplication**: Duplicate frames are merged
### When to Use
Hybrid search is recommended for most use cases:
```bash theme={null}
# Default mode is hybrid
memvid find knowledge.mv2 --query "authentication best practices"
# Explicit hybrid mode
memvid find knowledge.mv2 --query "OAuth2 patterns" --mode auto
```
### Performance Comparison
| Mode | Speed | Recall | Best For |
| ------ | -------- | ------------------- | -------------------- |
| `lex` | Fastest | Exact matches | Technical terms, IDs |
| `sem` | Moderate | Semantic similarity | Natural language |
| `auto` | Balanced | Comprehensive | General queries |
***
## Tracks
Tracks are logical groupings for organizing content within a memory.
### What Tracks Are
* **Namespace**: Group related documents together
* **Filterable**: Search within specific tracks
* **Metadata**: Organizational label stored with each frame
### Using Tracks
```bash theme={null}
# Add to a specific track
memvid put knowledge.mv2 --input api-docs.md --track "api"
memvid put knowledge.mv2 --input meeting-notes.md --track "meetings"
# Search within a track (via scope)
memvid find knowledge.mv2 --query "authentication" --scope "mv2://api/"
```
```python theme={null}
# Python SDK
mem.put(text="API documentation", title="Auth", track="api")
mem.put(text="Meeting notes", title="Standup", track="meetings")
# Search within scope
results = mem.find('authentication', scope='mv2://api/')
```
### Common Track Patterns
| Track | Use Case |
| --------------- | ----------------------------- |
| `documentation` | Technical docs and guides |
| `code` | Source code and snippets |
| `meetings` | Meeting notes and transcripts |
| `research` | Papers and references |
| `archived` | Old or deprecated content |
***
## Index Statistics
Check the status of all indices:
```bash theme={null}
memvid stats knowledge.mv2 --json
```
```json theme={null}
{
"frame_count": 150,
"has_lex_index": true,
"has_vec_index": true,
"has_time_index": true,
"lex_index_bytes": 2202009,
"vec_index_bytes": 1887436,
"time_index_bytes": 310478
}
```
***
## Best Practices
### Index Selection
| Scenario | Recommended Indices |
| -------------------- | --------------------------- |
| Full-featured search | All three (default) |
| Keyword-only search | Lexical only |
| Semantic similarity | Vector only |
| Large collections | All with vector compression |
| Audit/compliance | Time index required |
### Performance Tips
1. **Use `put_many()` for batch ingestion**: 100-200x faster than individual `put()` calls
2. **Enable vector compression** for large collections to reduce storage
3. **Rebuild indices** if search quality degrades after crashes
4. **Use hybrid mode** for best recall on general queries
### Maintenance
Regular index maintenance keeps search performing well:
```bash theme={null}
# Weekly: Verify integrity
memvid verify knowledge.mv2 --deep
# After many deletions: Vacuum and rebuild
memvid doctor knowledge.mv2 --vacuum --rebuild-lex-index
# After crashes: Full repair
memvid doctor knowledge.mv2 \
--rebuild-time-index \
--rebuild-lex-index \
--rebuild-vec-index
```
***
## Next Steps
<CardGroup>
<Card title="Memory Architecture" icon="database" href="/concepts/memory-architecture">
Understand the internal structure of .mv2 files
</Card>
<Card title="Search & Ask" icon="magnifying-glass" href="/cli/search-and-ask">
Learn advanced search techniques
</Card>
</CardGroup>
# Local Models with Ollama
Source: https://docs.memvid.com/concepts/local-models
Run AI-powered Q&A locally without sending data to external APIs
Memvid supports local LLM inference through [Ollama](https://ollama.com), allowing you to run AI-powered Q\&A without sending your data to external APIs. This is ideal for:
* **Privacy-sensitive data** - Keep everything on your machine
* **Offline usage** - No internet connection required after setup
* **Cost savings** - No API fees for inference
* **Low latency** - No network round-trips
***
## Quick Setup
### 1. Install Ollama
<Tabs>
<Tab title="macOS">
```bash theme={null}
brew install ollama
```
</Tab>
<Tab title="Linux">
```bash theme={null}
curl -fsSL https://ollama.com/install.sh | sh
```
</Tab>
<Tab title="Windows">
Download from [ollama.com/download](https://ollama.com/download)
</Tab>
</Tabs>
### 2. Start Ollama Server
```bash theme={null}
# Start in foreground (see logs)
ollama serve
# Or run as background service (macOS)
brew services start ollama
```
### 3. Pull a Model
```bash theme={null}
# Recommended: Qwen2.5 1.5B (best quality/size ratio)
ollama pull qwen2.5:1.5b
```
### 4. Use with Memvid
```bash theme={null}
memvid ask knowledge.mv2 \
--question "What is the main topic?" \
--use-model "ollama:qwen2.5:1.5b"
```
***
## Recommended Models
| Model | Size | Speed | Quality | Best For |
| -------------- | ------- | ------ | --------- | -------------------------- |
| `qwen2.5:0.5b` | \~400MB | Fast | Good | Quick queries, limited RAM |
| `qwen2.5:1.5b` | \~1GB | Fast | Great | **Recommended default** |
| `qwen2.5:3b` | \~2GB | Medium | Excellent | Complex questions |
| `phi3:mini` | \~2GB | Medium | Great | Reasoning tasks |
| `gemma2:2b` | \~1.6GB | Medium | Great | General purpose |
| `llama3.2:1b` | \~1.3GB | Fast | Good | Conversational |
| `llama3.2:3b` | \~2GB | Medium | Great | Balanced performance |
### Pull Commands
```bash theme={null}
# Small & fast
ollama pull qwen2.5:0.5b
# Recommended (best balance)
ollama pull qwen2.5:1.5b
# Higher quality
ollama pull qwen2.5:3b
ollama pull phi3:mini
ollama pull gemma2:2b
# Meta's Llama
ollama pull llama3.2:1b
ollama pull llama3.2:3b
```
***
## CLI Usage
### Basic Q\&A
```bash theme={null}
# Ask with local model
memvid ask knowledge.mv2 \
--question "What are the key findings?" \
--use-model "ollama:qwen2.5:1.5b"
# With JSON output
memvid ask knowledge.mv2 \
--question "Summarize the main points" \
--use-model "ollama:qwen2.5:1.5b" \
--json
```
### Advanced Options
```bash theme={null}
# More context for complex questions
memvid ask knowledge.mv2 \
--question "Explain the architecture in detail" \
--use-model "ollama:qwen2.5:3b" \
--top-k 15 \
--snippet-chars 800
# Filter by scope
memvid ask knowledge.mv2 \
--question "What API endpoints exist?" \
--use-model "ollama:qwen2.5:1.5b" \
--scope "mv2://api/"
# Time-travel query
memvid ask knowledge.mv2 \
--question "What was the status?" \
--use-model "ollama:qwen2.5:1.5b" \
--as-of-frame 100
```
***
## Python SDK Usage
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2')
# Ask with local Ollama model
response = mem.ask(
"What are the main conclusions?",
model="ollama:qwen2.5:1.5b",
k=10
)
print(response['answer'])
```
### With Different Models
```python theme={null}
# Quick answer with small model
quick_response = mem.ask(
"What is this document about?",
model="ollama:qwen2.5:0.5b"
)
# Detailed analysis with larger model
detailed_response = mem.ask(
"Provide a comprehensive analysis of the findings",
model="ollama:qwen2.5:3b",
k=15
)
```
***
## Node.js SDK Usage
```javascript theme={null}
import { use } from '@memvid/sdk';
const mem = await use('basic', 'knowledge.mv2');
// Ask with local Ollama model
const response = await mem.ask(
'What are the key takeaways?',
{ model: 'ollama:qwen2.5:1.5b', k: 10 }
);
console.log(response.answer);
```
***
## Model Selection Guide
### By Use Case
| Use Case | Recommended Model | Why |
| --------------- | ----------------- | ---------------- |
| Quick lookups | `qwen2.5:0.5b` | Fastest response |
| General Q\&A | `qwen2.5:1.5b` | Best balance |
| Technical docs | `qwen2.5:3b` | Better reasoning |
| Code analysis | `phi3:mini` | Strong at code |
| Research papers | `qwen2.5:3b` | Complex content |
### By Hardware
| RAM Available | Recommended Model |
| ------------- | --------------------------- |
| 4GB | `qwen2.5:0.5b` |
| 8GB | `qwen2.5:1.5b` |
| 16GB+ | `qwen2.5:3b` or `phi3:mini` |
***
## Ollama Management
### List Downloaded Models
```bash theme={null}
ollama list
```
### Remove a Model
```bash theme={null}
ollama rm qwen2.5:0.5b
```
### Update a Model
```bash theme={null}
ollama pull qwen2.5:1.5b
```
### Check Ollama Status
```bash theme={null}
# Check if server is running
curl http://localhost:11434/api/tags
```
### Run as Background Service
<Tabs>
<Tab title="macOS">
```bash theme={null}
# Start service
brew services start ollama
# Stop service
brew services stop ollama
# Check status
brew services list | grep ollama
```
</Tab>
<Tab title="Linux (systemd)">
```bash theme={null}
# Start service
sudo systemctl start ollama
# Enable on boot
sudo systemctl enable ollama
# Check status
sudo systemctl status ollama
```
</Tab>
</Tabs>
***
## Troubleshooting
### Ollama Not Running
```
Error: Failed to contact LLM provider
```
**Solution:**
```bash theme={null}
# Start Ollama server
ollama serve
# Or as background service (macOS)
brew services start ollama
```
### Model Not Found
```
Error: model 'qwen2.5:1.5b' not found
```
**Solution:**
```bash theme={null}
# Pull the model first
ollama pull qwen2.5:1.5b
```
### Slow Response Times
**Solutions:**
* Use a smaller model: `ollama:qwen2.5:0.5b`
* Reduce context: `--top-k 5 --snippet-chars 300`
* Close other memory-intensive applications
* Ensure you have enough RAM for the model
### Out of Memory
**Solutions:**
* Use a smaller model
* Close other applications
* Increase swap space (not recommended for performance)
***
## Comparison: Local vs Cloud Models
| Aspect | Local (Ollama) | Cloud (OpenAI, Claude, Gemini) |
| ----------- | --------------------- | ------------------------------ |
| **Privacy** | Data stays local | Data sent to API |
| **Cost** | Free after setup | Per-token pricing |
| **Speed** | Depends on hardware | Usually faster |
| **Quality** | Good to great | Excellent |
| **Offline** | Yes | No |
| **Setup** | Requires installation | Just API key |
### When to Use Local Models
* Sensitive/confidential data
* Offline environments
* Cost-sensitive applications
* Privacy requirements
### When to Use Cloud Models
* Best possible answer quality
* Limited local compute
* Quick prototyping
* Complex reasoning tasks
***
## Next Steps
<CardGroup>
<Card title="Search & Ask CLI" icon="terminal" href="/cli/search-and-ask">
Full CLI reference for search and Q\&A commands
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Use local models in Python applications
</Card>
</CardGroup>
# Memory Architecture
Source: https://docs.memvid.com/concepts/memory-architecture
How .mv2 files are structured internally
Understanding how Memvid stores data helps you make better decisions about ingestion, search, and performance optimization.
## File Structure
A `.mv2` file is a single, self-contained binary with five main layers:
```mermaid theme={null}
graph TD
A["Header (4 KB)"] --> B["Embedded WAL"]
B --> C["Segments (Frames)"]
C --> D["Indices"]
D --> E["Table of Contents + Footer"]
```
### 1. Header (4 KB)
The header contains:
* **Magic bytes**: Identifies the file as `.mv2` format
* **Version**: File format version
* **WAL metadata**: Position and size of write-ahead log
* **Footer offset**: Points to the table of contents
### 2. Embedded WAL
The write-ahead log (WAL) ensures crash safety:
* All mutations are written to WAL first
* On recovery, uncommitted changes are replayed
* Size scales with file capacity (1 MB to 64 MB)
### 3. Segments (Frames)
Your actual data lives in segments, which contain frames (the fundamental unit of storage):
* **Text segments**: Document content and metadata stored as frames
* **Blob segments**: Binary data (images, PDFs) as frames
* **Media segments**: Audio and video content as frames
* **Vector segments**: Embeddings for semantic search (optional)
Each frame contains payload, metadata, timestamp, URI, and checksum. Segments are written in deterministic order for reproducibility.
### 4. Indices
Memvid maintains multiple indices for fast search:
* **Lexical index (BM25)**: Full-text keyword search - works out of the box
* **Time index**: Temporal ordering of frames
* **Vector index**: Semantic similarity search - **optional**, add when needed
### 5. Table of Contents + Footer
The TOC maps everything:
* Segment locations and sizes
* Index offsets
* Checksums for integrity verification
The footer contains a final checksum and magic trailer (`MV2FOOT!`).
***
## Data Lifecycle
### Writing Data
When you add documents:
1. **put()** - Adds frames (documents) to pending state
2. **Indices updated** - Lexical and vector indices are built
3. **Time entries queued** - Timestamps recorded for timeline
4. **WAL appended** - Transaction logged for crash safety
5. **seal()** - Commits everything to disk with checksums
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2')
# 1. Add documents (pending)
mem.put(title="Doc 1", label="docs", metadata={}, text="Your content")
mem.put(title="Report", label="docs", metadata={}, file="report.pdf")
# 2. Commit to disk
mem.seal()
```
### Reading Data
When you search or retrieve:
1. **Open file** - Locate latest valid footer
2. **Load TOC** - Map segments and indices
3. **Replay WAL** - Apply any uncommitted changes
4. **Query indices** - Search lexical/vector/time indices
5. **Return results** - Ranked documents with snippets
***
## Single-File Guarantee
Memvid's core promise is **single-file portability**:
### What It Means
* **No sidecar files**: No `.wal`, `.lock`, `.shm` files
* **No external state**: Everything is in the `.mv2` file
* **Portable**: Copy the file to transfer the entire memory
### Why It Matters
```bash theme={null}
# Your entire knowledge base
ls ~/project/
# → knowledge.mv2
# Share it anywhere
cp knowledge.mv2 /team/shared/
scp knowledge.mv2 user@server:/data/
git add knowledge.mv2
```
### How It Works
Traditional databases use separate files for journals, locks, and indices. Memvid embeds all of these inside the `.mv2` file:
| Traditional DB | Memvid |
| ----------------------------------- | ------------------ |
| data.db + data.db-wal + data.db-shm | knowledge.mv2 |
| Requires careful copying | Just copy the file |
***
## Crash Safety
The embedded WAL ensures data survives unexpected shutdowns.
### Write-Ahead Logging
Every mutation is logged before being applied:
1. Transaction written to WAL region
2. WAL synced to disk (fsync)
3. Changes applied to segments
4. Checksum updated
### Recovery Process
On open, Memvid:
1. Locates the last valid footer
2. Loads the table of contents
3. Scans WAL for uncommitted entries
4. Replays any pending transactions
This guarantees that your data is safe even after crashes or power failures.
### WAL Sizing
WAL size scales with file capacity:
| File Size | WAL Size |
| ------------- | -------- |
| Under 100 MB | 1 MB |
| Under 1 GB | 4 MB |
| Under 10 GB | 16 MB |
| 10 GB or more | 64 MB |
***
## Locking and Concurrency
### File Locking
Memvid uses OS-level file locks:
* **Shared locks**: Multiple readers allowed
* **Exclusive locks**: Single writer at a time
### Read-Only Mode
For concurrent read access:
```python theme={null}
# Multiple processes can read simultaneously
mem = use('basic', 'knowledge.mv2', read_only=True)
results = mem.find('query')
```
### Writer Conflicts
If a writer holds the lock:
```python theme={null}
from memvid_sdk import use, LockedError
try:
mem = use('basic', 'knowledge.mv2')
except LockedError:
print("File is locked by another process")
```
***
## Determinism
Given the same inputs, Memvid produces the same outputs.
### Why Determinism Matters
* **Reproducible builds**: Same data → same file
* **Reliable testing**: Predictable behavior
* **Easy debugging**: Consistent results
### How It's Achieved
* Segments written in deterministic order
* Timestamps explicit, not system-derived
* Checksums verify integrity
***
## Performance Considerations
### Memory Usage
Memvid keeps some data in memory:
* Table of contents
* WAL handle
* Pending time entries
For large files, consider:
* Closing handles when done
* Using read-only mode for queries
### Index Building
Building indices is CPU-intensive:
* **Lexical index**: BM25 tokenization and indexing
* **Vector index**: Graph construction for similarity search
Use parallel ingestion for large datasets:
```bash theme={null}
memvid put knowledge.mv2 --input ./large-dataset/ \
--vector-compression \
--parallel-segments
```
### Search Optimization
* **Lexical search**: Fast for exact keywords
* **Vector search**: Slower but more intelligent
* **Hybrid search**: Balances both
Choose the right mode for your query.
***
## Next Steps
<CardGroup>
<Card title="Indices & Tracks" icon="list" href="/concepts/indexes-and-tracks">
Learn about lexical, vector, and time indices
</Card>
<Card title="Storage Capacity" icon="database" href="/concepts/capacity-and-plans">
Understand storage tiers and capacity management
</Card>
</CardGroup>
# Memory Cards & Enrichment
Source: https://docs.memvid.com/concepts/memory-cards
Extract structured knowledge from your documents with entity-attribute-value triples
Memory cards transform unstructured text into **structured knowledge** - entity-attribute-value triples that enable O(1) lookups, fact tracking, and relationship queries. Enrichment is the process of automatically extracting these cards from your content.
***
## What Are Memory Cards?
A memory card captures a single piece of knowledge:
```json theme={null}
{
"entity": "John Smith",
"slot": "job_title",
"value": "Senior Engineer",
"kind": "fact",
"confidence": 0.92,
"source_frame": "frame_abc123",
"extracted_at": "2024-12-31T10:30:00Z"
}
```
| Field | Description |
| -------------- | ---------------------------------------------------------- |
| `entity` | The subject (person, company, concept) |
| `slot` | The attribute or property name |
| `value` | The current value |
| `kind` | Type: fact, preference, event, profile, relationship, goal |
| `confidence` | Extraction confidence (0.0-1.0) |
| `source_frame` | Which document this came from |
***
## Memory Card Types
### Facts
Objective information that can be verified:
```
Entity: Acme Corp
Slot: headquarters
Value: San Francisco, CA
Entity: Python
Slot: creator
Value: Guido van Rossum
```
### Preferences
Subjective choices or opinions:
```
Entity: user
Slot: preferred_language
Value: TypeScript
Entity: team
Slot: meeting_day
Value: Tuesday
```
### Events
Time-bound occurrences:
```
Entity: Project Alpha
Slot: launch_date
Value: 2024-03-15
Entity: Q4 Review
Slot: scheduled
Value: 2024-12-20T14:00:00Z
```
### Relationships
Connections between entities:
```
Entity: John Smith
Slot: reports_to
Value: Jane Doe
Entity: Acme Corp
Slot: acquired
Value: StartupXYZ
```
### Profiles
Descriptive attributes:
```
Entity: Sarah Chen
Slot: expertise
Value: Machine Learning, NLP
Entity: Product Team
Slot: size
Value: 12 members
```
***
## Enrichment Engines
Memvid supports multiple extraction engines with different speed/quality/cost tradeoffs:
| Engine | Speed | Quality | Cost | Requires |
| --------- | ------- | --------- | ------ | ------------------------------ |
| `rules` | Fastest | Basic | Free | Nothing |
| `candle` | Fast | Good | Free | Auto-downloads model (\~2.4GB) |
| `groq` | Fast | Excellent | Low | `GROQ_API_KEY` |
| `openai` | Medium | Excellent | Medium | `OPENAI_API_KEY` |
| `claude` | Medium | Best | Higher | `ANTHROPIC_API_KEY` |
| `gemini` | Fast | Excellent | Low | `GOOGLE_API_KEY` |
| `xai` | Fast | Excellent | Medium | `XAI_API_KEY` |
| `mistral` | Fast | Good | Low | `MISTRAL_API_KEY` |
### Rules Engine
Pattern-based extraction using regex. Fast and free but limited:
```bash theme={null}
memvid enrich memory.mv2 --engine rules
```
Extracts:
* Email addresses → `entity: <local>, slot: email`
* Phone numbers → `entity: contact, slot: phone`
* Dates → `entity: document, slot: date_mentioned`
* URLs → `entity: document, slot: link`
### Candle Engine (Local LLM)
Runs Phi-3.5-mini locally via HuggingFace Candle:
```bash theme={null}
memvid enrich memory.mv2 --engine candle
```
* First run downloads \~2.4GB model
* No API key required
* Good quality, runs on CPU (GPU optional)
* Slower than API-based engines
### Cloud Engines
For best quality, use cloud LLMs:
```bash theme={null}
# Groq (fastest cloud option)
export GROQ_API_KEY=gsk_xxx
memvid enrich memory.mv2 --engine groq
# OpenAI
export OPENAI_API_KEY=sk-xxx
memvid enrich memory.mv2 --engine openai
# Claude (highest quality)
export ANTHROPIC_API_KEY=sk-ant-xxx
memvid enrich memory.mv2 --engine claude
# Gemini
export GOOGLE_API_KEY=xxx
memvid enrich memory.mv2 --engine gemini
```
***
## CLI Commands
### Enrich: Extract Memory Cards
```bash theme={null}
# Basic enrichment with rules (fast, free)
memvid enrich memory.mv2 --engine rules
# Enrich with Groq (recommended balance)
memvid enrich memory.mv2 --engine groq
# Enrich with verbose output
memvid enrich memory.mv2 --engine groq --verbose
# Force re-enrichment of all frames
memvid enrich memory.mv2 --engine claude --force
# Parallel processing for speed
memvid enrich memory.mv2 --engine groq --workers 20 --batch-size 10
```
Output:
```
Enriching memory.mv2 with groq engine...
Processing: 150 frames
Extracted: 423 memory cards
Entities: 87 unique
Time: 12.3s
Top entities:
- John Smith (34 facts)
- Acme Corp (28 facts)
- Project Alpha (19 facts)
```
### Memories: View Extracted Cards
```bash theme={null}
# List all memory cards
memvid memories memory.mv2
# JSON output
memvid memories memory.mv2 --json
```
Output:
```
Memory Cards (423 total)
Entity: John Smith
job_title: Senior Engineer (confidence: 0.94)
team: Platform (confidence: 0.89)
reports_to: Jane Doe (confidence: 0.91)
expertise: Rust, Python (confidence: 0.87)
Entity: Acme Corp
headquarters: San Francisco (confidence: 0.96)
founded: 2015 (confidence: 0.92)
employees: 250 (confidence: 0.78)
...
```
### State: O(1) Entity Lookup
Get the current state of any entity instantly:
```bash theme={null}
# Query single entity
memvid state memory.mv2 --entity "John Smith"
# JSON output
memvid state memory.mv2 --entity "John Smith" --json
```
Output:
```
Entity: John Smith
Current State:
job_title: Senior Engineer
team: Platform
reports_to: Jane Doe
expertise: Rust, Python
location: San Francisco
start_date: 2022-03-15
Last updated: 2024-12-30T14:22:00Z
Source frames: 12
```
### Facts: Audit Fact History
Track how facts changed over time with full provenance:
```bash theme={null}
# All facts for an entity
memvid facts memory.mv2 --entity "John Smith"
# Filter by predicate
memvid facts memory.mv2 --entity "John Smith" --predicate job_title
# Filter by source frame
memvid facts memory.mv2 --frame-id frame_abc123
# JSON output
memvid facts memory.mv2 --entity "John Smith" --json
```
Output:
```
Fact History: John Smith → job_title
1. "Junior Engineer" (2024-01-15)
Source: frame_001 (onboarding.pdf)
Confidence: 0.91
2. "Engineer" (2024-06-01)
Source: frame_089 (promotion_announcement.md)
Confidence: 0.94
Relation: UPDATES previous
3. "Senior Engineer" (2024-11-15)
Source: frame_142 (team_update.md)
Confidence: 0.94
Relation: UPDATES previous
Current value: Senior Engineer
```
### Export: Standard Formats
Export facts to standard knowledge graph formats:
```bash theme={null}
# N-Triples (RDF)
memvid export memory.mv2 --format ntriples --out facts.nt
# JSON
memvid export memory.mv2 --format json --out facts.json
# CSV
memvid export memory.mv2 --format csv --out facts.csv
# Filter by entity
memvid export memory.mv2 --format json --filter-entity "Acme Corp" --out acme.json
```
N-Triples output:
```
<John_Smith> <job_title> "Senior Engineer" .
<John_Smith> <reports_to> <Jane_Doe> .
<Acme_Corp> <headquarters> "San Francisco" .
<Acme_Corp> <founded> "2015" .
```
### Schema: Predicate Management
Define and view predicate schemas:
```bash theme={null}
# Infer schema from existing facts
memvid schema infer memory.mv2
# List current schemas
memvid schema list memory.mv2
# Manually define predicate type
memvid schema set memory.mv2 job_title string
memvid schema set memory.mv2 employee_count integer
memvid schema set memory.mv2 is_active boolean
```
Output:
```
Predicate Schema:
job_title: string (inferred from 45 facts)
team: string (inferred from 32 facts)
employee_count: integer (inferred from 12 facts)
founded: date (inferred from 8 facts)
headquarters: string (inferred from 8 facts)
is_public: boolean (inferred from 5 facts)
```
***
## SDK Usage
### Python
```python theme={null}
from memvid import use
mem = use('basic', 'memory.mv2')
# Enrich documents
mem.enrich(engine="groq")
# Get current entity state (O(1) lookup)
john = mem.get_entity_state("John Smith")
print(f"Job: {john['job_title']}")
print(f"Team: {john['team']}")
# Get all facts for an entity
facts = mem.get_facts(entity="John Smith")
for fact in facts:
print(f"{fact.slot}: {fact.value} (from {fact.source_frame})")
# Get fact history for specific attribute
title_history = mem.get_facts(
entity="John Smith",
predicate="job_title"
)
for fact in title_history:
print(f"{fact.extracted_at}: {fact.value}")
# Query preferences
prefs = mem.get_preferences(entity="user")
print(f"Preferred language: {prefs.get('preferred_language')}")
# Get memory timeline
timeline = mem.get_memory_timeline(entity="Project Alpha")
for event in timeline:
print(f"{event.timestamp}: {event.slot} = {event.value}")
```
### Node.js
```typescript theme={null}
import { use } from '@anthropics/memvid'
const mem = await use('basic', 'memory.mv2')
// Enrich documents
await mem.enrich({ engine: "groq" })
// Get current entity state
const john = await mem.getEntityState("John Smith")
console.log(`Job: ${john.job_title}`)
console.log(`Team: ${john.team}`)
// Get all facts for an entity
const facts = await mem.getFacts({ entity: "John Smith" })
for (const fact of facts) {
console.log(`${fact.slot}: ${fact.value}`)
}
// Query by predicate
const titles = await mem.getFacts({
entity: "John Smith",
predicate: "job_title"
})
```
***
## Version Relations
Memory cards track how values change over time:
| Relation | Meaning |
| ---------- | ------------------------------ |
| `SETS` | Initial value (no previous) |
| `UPDATES` | Replaces previous value |
| `EXTENDS` | Adds to previous value (lists) |
| `RETRACTS` | Removes/invalidates previous |
Example tracking:
```
Frame 001: John Smith → team = "Backend" [SETS]
Frame 045: John Smith → team = "Platform" [UPDATES]
Frame 089: John Smith → team = "Platform" [no change, skipped]
Frame 112: John Smith → team = "Infrastructure" [UPDATES]
```
Query the current value:
```bash theme={null}
memvid state memory.mv2 --entity "John Smith"
# team: Infrastructure
```
Query the history:
```bash theme={null}
memvid facts memory.mv2 --entity "John Smith" --predicate team
# Shows all 3 values with timestamps and sources
```
***
## Deduplication
Memory cards are automatically deduplicated:
* **Same entity + slot**: Keeps highest confidence value
* **Same value**: Skips if already exists
* **Different sources**: Tracks all sources for provenance
```python theme={null}
# These produce one card, not three
mem.put("John works as an engineer") # Extracts: job_title = engineer
mem.put("John is an engineer at Acme") # Same fact, different source
mem.put("John Smith - Engineer") # Same fact, skipped
```
***
## Incremental Enrichment
By default, enrichment only processes new frames:
```bash theme={null}
# First run: processes all 100 frames
memvid enrich memory.mv2 --engine groq
# Enriched 100 frames, extracted 250 cards
# Add more documents
memvid put memory.mv2 --input new_docs/
# Second run: only processes new frames
memvid enrich memory.mv2 --engine groq
# Enriched 15 frames (85 already enriched), extracted 42 cards
```
Force full re-enrichment:
```bash theme={null}
memvid enrich memory.mv2 --engine claude --force
```
***
## Performance Tuning
### Parallel Workers
```bash theme={null}
# More workers = faster (but more API calls)
memvid enrich memory.mv2 --engine groq --workers 20
# Fewer workers = slower but less rate limiting
memvid enrich memory.mv2 --engine openai --workers 5
```
### Batch Size
```bash theme={null}
# Larger batches = fewer API calls
memvid enrich memory.mv2 --engine groq --batch-size 20
# Smaller batches = more granular progress
memvid enrich memory.mv2 --engine claude --batch-size 5
```
### Engine Selection by Use Case
| Use Case | Recommended Engine |
| --------------------- | ------------------ |
| Quick testing | `rules` |
| Offline/privacy | `candle` |
| Production (balanced) | `groq` |
| Maximum accuracy | `claude` |
| Cost-sensitive | `gemini` |
***
## Use Cases
### Personal Knowledge Management
Track facts about people, projects, and topics:
```bash theme={null}
# Ingest your notes
memvid put brain.mv2 --input ~/notes/
# Extract knowledge
memvid enrich brain.mv2 --engine groq
# Query what you know about someone
memvid state brain.mv2 --entity "Sarah from Marketing"
```
### Meeting Minutes
Extract action items and decisions:
```bash theme={null}
# Transcribe and ingest
memvid put meetings.mv2 --input recording.mp3
# Enrich with Claude for best accuracy
memvid enrich meetings.mv2 --engine claude
# Find all action items assigned to John
memvid facts meetings.mv2 --entity "John" --predicate assigned_action
```
### Research Papers
Build a knowledge graph from literature:
```bash theme={null}
# Ingest papers
memvid put research.mv2 --input papers/
# Extract entities and relationships
memvid enrich research.mv2 --engine groq
# Export to knowledge graph format
memvid export research.mv2 --format ntriples --out research.nt
```
### Customer Information
Track customer preferences and history:
```bash theme={null}
# Ingest support tickets
memvid put customers.mv2 --input tickets/
# Extract customer facts
memvid enrich customers.mv2 --engine groq
# Get customer state
memvid state customers.mv2 --entity "customer_12345"
```
***
## Best Practices
### 1. Choose the Right Engine
Start with `rules` for testing, graduate to `groq` for production:
```bash theme={null}
# Development
memvid enrich memory.mv2 --engine rules --verbose
# Production
memvid enrich memory.mv2 --engine groq
```
### 2. Enrich After Bulk Imports
Wait until documents are loaded before enriching:
```bash theme={null}
# Load all documents first
memvid put memory.mv2 --input docs/
# Then enrich in one pass
memvid enrich memory.mv2 --engine groq
```
### 3. Use Incremental Mode
Let Memvid track what's been enriched:
```bash theme={null}
# Default: only enriches new frames
memvid enrich memory.mv2 --engine groq
```
### 4. Export for Integration
Use exports to integrate with other tools:
```bash theme={null}
# For graph databases
memvid export memory.mv2 --format ntriples --out facts.nt
# For spreadsheets
memvid export memory.mv2 --format csv --out facts.csv
```
***
## Troubleshooting
### "No memory cards extracted"
1. Check if frames have text content:
```bash theme={null}
memvid stats memory.mv2
```
2. Try verbose mode to see extraction:
```bash theme={null}
memvid enrich memory.mv2 --engine rules --verbose
```
3. Content may be too short or unstructured
### "API rate limited"
Reduce workers and increase batch size:
```bash theme={null}
memvid enrich memory.mv2 --engine groq --workers 5 --batch-size 20
```
### "Low confidence scores"
* Try a better engine (`claude` > `groq` > `rules`)
* Content may be ambiguous
* Check source document quality
### "Missing expected entities"
1. Check if entity appears in source:
```bash theme={null}
memvid find memory.mv2 --query "Entity Name"
```
2. Force re-enrichment:
```bash theme={null}
memvid enrich memory.mv2 --engine claude --force
```
***
## Next Steps
<CardGroup>
<Card title="Graph Search" icon="diagram-project" href="/concepts/graph-search">
Query entity relationships with Logic Mesh
</Card>
<Card title="Deduplication" icon="copy" href="/concepts/deduplication">
How duplicate content is handled
</Card>
</CardGroup>
# Performance Tuning
Source: https://docs.memvid.com/concepts/performance-tuning
Optimize Memvid for speed, storage, and quality based on your use case
Memvid is designed for high performance out of the box, but different use cases benefit from different configurations. This guide covers tuning options for ingestion speed, search latency, storage efficiency, and retrieval quality.
***
## Quick Recommendations
| Use Case | Configuration |
| ---------------- | ----------------------------------------- |
| Code search | `--no-vec`, `--mode lex` |
| Fast prototyping | `bge-small` model, small memory size |
| Production RAG | `bge-base` or `nomic`, adaptive retrieval |
| Large documents | Parallel ingestion, higher size limit |
| Minimal storage | `--no-vec` or `bge-small` |
| Best quality | `gte-large` or OpenAI embeddings |
***
## Ingestion Performance
### Parallel Ingestion
For large folders, enable parallel processing:
```bash theme={null}
# Process multiple files concurrently
memvid put memory.mv2 --input ./large-folder/ --parallel-segments
# Combine with embedding skip for fastest ingestion
memvid put memory.mv2 --input ./logs/ --embedding-skip --parallel-segments
```
Performance comparison:
| Files | Sequential | Parallel |
| ----------- | ---------- | -------- |
| 100 docs | 45s | 12s |
| 1,000 docs | 7m | 2m |
| 10,000 docs | 1h 10m | 20m |
### Skip Embeddings
For lexical-only search or when you'll add embeddings later:
```bash theme={null}
# No vector embeddings (lexical only)
memvid create memory.mv2 --no-vec
memvid put memory.mv2 --input docs/
# Or skip per-ingestion
memvid put memory.mv2 --input logs.txt --embedding-skip
```
Benefits:
* **10x faster** ingestion
* **60% smaller** file size
* Full lexical search still available
### Embedding Model Selection
Choose based on speed/quality tradeoff:
| Model | Speed | Quality | Size | Best For |
| ----------- | ------- | --------- | ----- | -------------------------- |
| `bge-small` | Fastest | Good | 33MB | Prototyping, large volumes |
| `bge-base` | Fast | Better | 110MB | Production (default) |
| `nomic` | Fast | Better | 137MB | Long documents |
| `gte-large` | Slower | Best | 335MB | Maximum quality |
| `openai` | API | Excellent | - | Best quality, requires API |
```bash theme={null}
# Use smaller model for speed
memvid -m bge-small put memory.mv2 --input docs/
# Use larger model for quality
memvid -m gte-large put memory.mv2 --input docs/
```
***
## Search Performance
### Search Mode Selection
| Mode | Speed | Best For |
| ------ | -------- | ----------------------------------- |
| `lex` | Fastest | Exact matches, code, keywords |
| `sem` | Fast | Conceptual queries, similar meaning |
| `auto` | Balanced | General use (default) |
```bash theme={null}
# Lexical only (fastest)
memvid find memory.mv2 --query "handleAuth" --mode lex
# Semantic only
memvid find memory.mv2 --query "authentication logic" --mode sem
# Hybrid (default)
memvid find memory.mv2 --query "auth" --mode auto
```
### Adaptive Retrieval
Adaptive retrieval automatically adjusts result count based on query relevance. Disable for consistent performance:
```bash theme={null}
# Fixed result count (faster, predictable)
memvid find memory.mv2 --query "term" --no-adaptive --top-k 10
# Adaptive (may return fewer, but higher quality)
memvid find memory.mv2 --query "term" # Default
```
### Scope Filtering
Narrow search scope for faster results:
```bash theme={null}
# Search only in specific directory
memvid find memory.mv2 --query "config" --scope "src/config/"
# Search specific document
memvid find memory.mv2 --query "api key" --uri "docs/security.md"
```
### Sketch Index
For very large memories (100k+ frames), build a sketch index for faster approximate search:
```bash theme={null}
# Build sketch index
memvid sketch build memory.mv2 --variant medium
# Check sketch status
memvid sketch info memory.mv2
```
Sketch variants:
| Variant | Build Time | Query Speed | Accuracy |
| -------- | ---------- | ----------- | -------- |
| `small` | Fast | \~2x faster | 90% |
| `medium` | Moderate | \~3x faster | 95% |
| `large` | Slower | \~5x faster | 98% |
***
## Storage Optimization
### Memory Size
Set appropriate size limits:
```bash theme={null}
# Small memory for quick projects
memvid create notes.mv2 --size 10MB
# Large memory for document archives
memvid create archive.mv2 --size 50MB
```
Size recommendations:
| Content | Recommended Size |
| -------------- | ---------------- |
| Personal notes | 10-15MB |
| Single project | 15-25MB |
| Documentation | 25-35MB |
| Large archive | 40-50MB |
### Vacuum and Compact
After deletions or updates, reclaim space:
```bash theme={null}
# Compact storage
memvid doctor memory.mv2 --vacuum
# Full optimization
memvid doctor memory.mv2 --vacuum --rebuild-lex-index --rebuild-vec-index
```
### Index Selection
Disable indexes you don't need:
```bash theme={null}
# No vector index (lexical only)
memvid create code.mv2 --no-vec
# No lexical index (semantic only)
memvid create semantic.mv2 --no-lex
```
Storage impact:
| Configuration | Relative Size |
| -------------- | ------------- |
| Full (default) | 100% |
| No vectors | \~40% |
| No lexical | \~85% |
| Neither | \~25% |
***
## RAG Performance
### Model Selection
Choose synthesis model based on needs:
| Model | Speed | Quality | Cost |
| ----------- | --------- | --------- | ------ |
| `tinyllama` | Fastest | Basic | Free |
| `groq` | Very fast | Good | Low |
| `gemini` | Fast | Good | Low |
| `openai` | Moderate | Excellent | Medium |
| `claude` | Moderate | Excellent | Medium |
```bash theme={null}
# Fast local synthesis
memvid ask memory.mv2 --question "..." --use-model tinyllama
# Fast API synthesis
memvid ask memory.mv2 --question "..." --use-model groq
```
### Context-Only Mode
Skip synthesis for maximum speed:
```bash theme={null}
# Get relevant context without LLM synthesis
memvid ask memory.mv2 --question "What are the config options?" --context-only
```
Use cases:
* Feed context to your own LLM
* Debugging retrieval quality
* Batch processing
***
## Index Maintenance
### Rebuild Indexes
Periodically rebuild for optimal performance:
```bash theme={null}
# Rebuild all indexes
memvid doctor memory.mv2 --rebuild-lex-index --rebuild-vec-index --rebuild-time-index
# Rebuild specific index
memvid doctor memory.mv2 --rebuild-vec-index
```
When to rebuild:
* After many deletions (>20% of content)
* Search results seem slow or inaccurate
* After model upgrade
### Verify Integrity
Check for corruption:
```bash theme={null}
# Quick check
memvid verify memory.mv2
# Deep check
memvid verify memory.mv2 --deep
```
***
## Benchmarks
Typical performance on M1 Mac with SSD:
### Ingestion Speed
| Content Type | Speed (with embeddings) | Speed (no embeddings) |
| ------------ | ----------------------- | --------------------- |
| Plain text | \~1,000 chunks/sec | \~10,000 chunks/sec |
| PDF (text) | \~200 pages/min | \~2,000 pages/min |
| Code files | \~500 files/min | \~5,000 files/min |
### Search Latency
| Memory Size | Lexical | Semantic | Hybrid |
| ------------------ | ------- | -------- | ------ |
| 1,000 frames | \~5ms | \~10ms | \~15ms |
| 10,000 frames | \~10ms | \~25ms | \~35ms |
| 100,000 frames | \~20ms | \~50ms | \~70ms |
| 1M frames (sketch) | \~30ms | \~60ms | \~90ms |
### Ask Latency
| Model | Retrieval + Synthesis |
| --------- | --------------------- |
| tinyllama | \~500ms |
| groq | \~800ms |
| openai | \~1.5s |
| claude | \~2s |
***
## SDK Performance Tips
### Python
```python theme={null}
from memvid import use
# Reuse memory instance
mem = use('basic', 'memory.mv2')
# Batch operations
texts = [...]
for text in texts:
mem.put(text) # Batched internally
# Async for better throughput
import asyncio
from memvid import use_async
async def main():
mem = await use_async('basic', 'memory.mv2')
results = await asyncio.gather(*[
mem.find(q) for q in queries
])
```
### Node.js
```typescript theme={null}
import { use } from '@anthropics/memvid'
// Reuse memory instance
const mem = await use('basic', 'memory.mv2')
// Parallel searches
const results = await Promise.all(
queries.map(q => mem.find(q))
)
// Stream large results
for await (const chunk of mem.findStream(query)) {
process.stdout.write(chunk)
}
```
***
## Monitoring
### Query Tracking
Monitor usage patterns:
```bash theme={null}
# View usage statistics
memvid plan show
# JSON format for monitoring
memvid stats memory.mv2 --json
```
### Memory Statistics
```bash theme={null}
# Detailed stats
memvid stats memory.mv2
# Output example:
# Frames: 10,234
# Size: 45.2 MB
# Vector index: 23.1 MB
# Lexical index: 8.4 MB
# Avg query time: 12ms
```
***
## Troubleshooting Performance
### Slow Ingestion
1. Enable parallel ingestion: `--parallel-segments`
2. Use smaller embedding model: `-m bge-small`
3. Skip embeddings if not needed: `--embedding-skip`
### Slow Search
1. Use lexical mode for exact matches: `--mode lex`
2. Build sketch index for large memories
3. Narrow scope: `--scope "relevant/path/"`
### High Memory Usage
1. Use smaller embedding model
2. Create with `--no-vec` if lexical is sufficient
3. Vacuum after deletions: `--vacuum`
### Large File Size
1. Enable no-vec mode
2. Vacuum to reclaim deleted space
3. Use smaller embedding model
***
## Next Steps
<CardGroup>
<Card title="Embedding Models" icon="brain" href="/concepts/embedding-models">
Model comparison
</Card>
</CardGroup>
# Permission-Aware Retrieval (ACL)
Source: https://docs.memvid.com/concepts/permission-aware-retrieval
Enforce tenant isolation and RBAC at frame/chunk level during search and RAG
Memvid supports **permission-aware retrieval** by storing ACL metadata on every frame (chunk) and enforcing it **inside retrieval** (search/ask) in `memvid-core`.
This unlocks:
* **Strict multi-tenant isolation** (no cross-tenant leakage)
* **RBAC** (roles/groups/principals) at **frame/chunk level**
* A **single `.mv2` per environment** with metadata-based enforcement (recommended)
***
## Mental Model
* A **frame** is the atomic unit of retrieval in Memvid. When you ingest a PDF, Memvid creates **many frames (chunks)**.
* ACL is evaluated **per frame**, so "chunk-level ACL" means "frame-level ACL metadata".
* ACL is **not keyword-based**: it does not guess who can see content. You decide the policy at ingest time.
***
## ACL Metadata (Ingest-Time)
Attach the following keys in the frame `metadata` (stored on disk in the frame’s `extra_metadata`):
| Key | Type | Required | Meaning |
| --------------------- | -------------------------- | ----------------- | ----------------------------------------------------------------------- |
| `acl_tenant_id` | `string` | Yes (recommended) | Tenant boundary for strict isolation |
| `acl_visibility` | `"public" \| "restricted"` | Yes | `public` is readable by anyone in-tenant; `restricted` requires a match |
| `acl_read_roles` | `string[]` | If `restricted` | Allowed roles |
| `acl_read_groups` | `string[]` | If `restricted` | Allowed group IDs |
| `acl_read_principals` | `string[]` | If `restricted` | Allowed subject/principal IDs |
| `acl_policy_version` | `string` | Yes | Policy schema version (currently `"v1"`) |
| `acl_resource_id` | `string` | Optional | Stable lineage identifier (optional) |
<Info>
In Node/Python SDKs you can provide `string[]` values directly for `acl_read_*`. The SDK will normalize and persist them in a canonical form for the core evaluator.
ACL strings are normalized (trimmed + lowercased). Treat role/group/principal identifiers as case-insensitive.
</Info>
### Example: Role-Restricted Chunk
```json theme={null}
{
"acl_tenant_id": "acme-prod",
"acl_visibility": "restricted",
"acl_read_roles": ["finance"],
"acl_policy_version": "v1"
}
```
<Warning>
If `acl_visibility` is `"restricted"` and you provide **no** `acl_read_roles` / `acl_read_groups` / `acl_read_principals`, the chunk will be denied for everyone in `enforce` mode.
</Warning>
***
## ACL Context (Query-Time)
At query time you provide the caller identity via `acl_context` / `aclContext`:
| Field | Type | Meaning |
| -------------------------- | ---------- | ------------------------------------------ |
| `tenant_id` / `tenantId` | `string` | Tenant boundary (required for enforcement) |
| `subject_id` / `subjectId` | `string` | The current user (principal) |
| `roles` | `string[]` | User roles |
| `group_ids` / `groupIds` | `string[]` | User group IDs |
And choose an enforcement mode:
* `audit`: evaluate ACL but do **not** block results (migration/testing)
* `enforce`: **filter** results; deny-by-default for missing/invalid ACL metadata
<Warning>
Do not accept `acl_context` from untrusted clients. Build it server-side from your auth system (JWT claims, your RBAC store, etc.) so users cannot self-assign roles.
</Warning>
***
## Creating an ACL-Scoped API Key (Dashboard)
In the Memvid dashboard:
1. Go to **API Keys** and click **Create Key**
2. Enable **ACL scope**
3. Set **Tenant ID** (required for strict isolation)
4. Optionally set **Roles**, **Group IDs**, and **Subject ID**
5. Choose enforcement mode: `audit` or `enforce`
<Tip>
For most apps, keep the API key as a server-side credential and compute the end-user `acl_context` from your auth system on every request.
</Tip>
***
## End-to-End (Node.js)
```ts theme={null}
import {
configure, create,
getAclScopeFromApiKey, aclContextFromScope, aclMetadataFromScope,
} from "@memvid/sdk";
configure({ apiKey: process.env.MEMVID_API_KEY, dashboardUrl: "https://memvid.com" });
const scope = await getAclScopeFromApiKey(); // reads /api/ticket (control plane)
const aclContext = aclContextFromScope(scope); // { tenantId, subjectId?, roles?, groupIds? }
const aclMeta = aclMetadataFromScope(scope, { visibility: "restricted" });
const mv = await create("kb.mv2", "basic", { enableLex: true, enableVec: true });
await mv.put({ title: "Finance doc", label: "kb", text: "Q4 budget...", metadata: aclMeta });
const hits = await mv.find("budget", {
mode: "lex",
k: 5,
aclContext,
aclEnforcementMode: "enforce",
});
await mv.seal();
```
***
## End-to-End (Python)
```python theme={null}
import os
from memvid_sdk import (
configure, create,
get_acl_scope_from_api_key, acl_context_from_scope, acl_metadata_from_scope,
)
configure({"api_key": os.environ["MEMVID_API_KEY"], "dashboard_url": "https://memvid.com"})
scope = get_acl_scope_from_api_key()
acl_context = acl_context_from_scope(scope) # {"tenant_id", "subject_id"?, "roles"?, "group_ids"?}
acl_meta = acl_metadata_from_scope(scope, visibility="restricted")
mv = create("kb.mv2", enable_lex=True, enable_vec=True)
mv.put(title="Finance doc", label="kb", metadata=acl_meta, text="Q4 budget...")
hits = mv.find("budget", mode="lex", k=5, acl_context=acl_context, acl_enforcement_mode="enforce")
mv.seal()
```
***
## Chunk-Level Guarding (Example Policy)
You decide which chunks are restricted to which readers at ingest time. Example policy:
* Pages 1-20: `role=finance`
* Pages 21-30: `role=hr`
* Pages 31+: `principal=matt`
To implement this, you ingest via `put_many()` / `putMany()` with per-chunk metadata (rather than a single `put_file()` metadata applied to all chunks).
<Info>
If you use `put_file(...)` with `metadata=...`, the same ACL metadata is applied to every produced chunk. That’s perfect for **document-level ACL**, but not enough for **section/page-level ACL**.
</Info>
***
## Single `.mv2` vs One `.mv2` Per Tenant
**Single `.mv2` per environment (recommended)**:
* Store all tenants in one file
* Always set `acl_tenant_id` on every frame
* Always pass `acl_context.tenant_id` at retrieval
* Use `restricted` + allow-lists for sensitive frames
**One `.mv2` per tenant (simpler operations, more files)**:
* Easier isolation boundaries
* More operational overhead (more files to manage, ticketing/capacity per file)
# PII Detection & Masking
Source: https://docs.memvid.com/concepts/pii-masking
Automatically detect and mask sensitive information in search results and responses
Memvid can detect and mask Personally Identifiable Information (PII) in search results and ask responses. The original data remains searchable, but sensitive information is redacted in output.
***
## How It Works
```mermaid theme={null}
flowchart LR
A[Original Data<br/>stored as-is] --> B[Search/Ask<br/>finds match]
B --> C[Masked Output<br/>PII redacted]
```
**Example:**
* Original: `"Contact john@example.com or call 555-123-4567"`
* Masked: `"Contact [EMAIL] or call [PHONE]"`
Key points:
* **Original data preserved**: Content stored without modification
* **Searchable**: You can search for emails, phones, etc.
* **Masked on output**: PII hidden in results and responses
* **Query-time detection**: No preprocessing required
***
## Detected PII Types
| Type | Pattern | Masked As |
| --------------- | -------------------------------- | --------------- |
| Email addresses | `user@domain.com` | `[EMAIL]` |
| Phone numbers | `555-123-4567`, `(555) 123-4567` | `[PHONE]` |
| SSN (US) | `123-45-6789` | `[SSN]` |
| Credit cards | `4111-1111-1111-1111` | `[CREDIT_CARD]` |
| IPv4 addresses | `192.168.1.1` | `[IP_ADDRESS]` |
| API keys | `sk-xxx`, `api_xxx` | `[API_KEY]` |
| Bearer tokens | `Bearer eyJ...` | `[TOKEN]` |
***
## CLI Usage
### Mask PII in Ask Responses
```bash theme={null}
# Enable PII masking in ask
memvid ask memory.mv2 --question "What's John's contact info?" --mask-pii
```
Without masking:
```
John can be reached at john.smith@acme.com or by phone at (555) 867-5309.
His SSN for payroll is 123-45-6789.
```
With `--mask-pii`:
```
John can be reached at [EMAIL] or by phone at [PHONE].
His SSN for payroll is [SSN].
```
### PII in Search Results
```bash theme={null}
# Search results with masking
memvid find memory.mv2 --query "contact information" --mask-pii
```
***
## SDK Usage
### Python
```python theme={null}
from memvid import use
mem = use('basic', 'memory.mv2')
# Ask with PII masking
response = mem.ask(
"What is the customer's contact information?",
mask_pii=True
)
print(response.answer)
# "Customer can be reached at [EMAIL] or [PHONE]"
# Check if content contains PII
from memvid import contains_pii, mask_pii
text = "Email me at test@example.com"
if contains_pii(text):
safe_text = mask_pii(text)
print(safe_text) # "Email me at [EMAIL]"
```
### Node.js
```typescript theme={null}
import { use } from '@anthropics/memvid'
const mem = await use('basic', 'memory.mv2')
// Ask with PII masking
const response = await mem.ask(
"What is the customer's contact information?",
{ maskPii: true }
)
console.log(response.answer)
// "Customer can be reached at [EMAIL] or [PHONE]"
```
***
## Utility Functions
### Check for PII
```python theme={null}
from memvid import contains_pii
# Returns True if any PII detected
contains_pii("Call 555-123-4567") # True
contains_pii("Hello world") # False
```
### Mask PII in Text
```python theme={null}
from memvid import mask_pii
original = """
Contact: john@example.com
Phone: (555) 123-4567
SSN: 123-45-6789
API Key: sk-abc123xyz
"""
masked = mask_pii(original)
print(masked)
```
Output:
```
Contact: [EMAIL]
Phone: [PHONE]
SSN: [SSN]
API Key: [API_KEY]
```
### Get PII Locations
```python theme={null}
from memvid import detect_pii
text = "Email john@test.com or call 555-1234"
pii_items = detect_pii(text)
for item in pii_items:
print(f"Type: {item.type}, Value: {item.value}, Position: {item.start}-{item.end}")
# Type: email, Value: john@test.com, Position: 6-19
# Type: phone, Value: 555-1234, Position: 28-36
```
***
## Use Cases
### Customer Support
Mask customer data in AI responses:
```python theme={null}
# Support bot with PII protection
response = mem.ask(
ticket_content,
mask_pii=True # Don't expose customer PII
)
# Log safely
logger.info(f"Response: {response.answer}") # No PII in logs
```
### Compliance (GDPR, HIPAA)
Redact PII before displaying or logging:
```python theme={null}
# Search medical records
results = mem.find("patient symptoms", mask_pii=True)
# Safe to display - no PHI exposed
for result in results:
print(result.snippet) # "[EMAIL]", "[PHONE]", "[SSN]" redacted
```
### Development & Testing
Mask real data in development environments:
```python theme={null}
# Export masked data for dev/test
for frame in mem.timeline():
masked_content = mask_pii(frame.text)
dev_mem.put(masked_content)
```
### Audit Logging
Log queries without exposing PII:
```python theme={null}
def search_with_audit(query):
results = mem.find(query)
# Log masked version
audit_log.info(f"Query: {mask_pii(query)}")
audit_log.info(f"Results: {len(results)}")
return results
```
***
## Configuration
### Default Behavior
PII masking is **disabled by default**. Enable it explicitly:
```bash theme={null}
# CLI: use --mask-pii flag
memvid ask memory.mv2 -q "..." --mask-pii
```
```python theme={null}
# Python: mask_pii=True parameter
mem.ask("...", mask_pii=True)
```
### Why Not Default?
* Performance overhead for detection
* Some use cases need raw data
* Explicit opt-in for compliance clarity
***
## Detection Patterns
### Email Addresses
```
user@domain.com
user.name@subdomain.domain.co.uk
user+tag@domain.com
```
Regex: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
### Phone Numbers
```
555-123-4567
(555) 123-4567
555.123.4567
+1 555 123 4567
15551234567
```
Supports US, UK, and international formats.
### Social Security Numbers
```
123-45-6789
123 45 6789
123456789
```
US SSN format with common separators.
### Credit Card Numbers
```
4111-1111-1111-1111
4111 1111 1111 1111
4111111111111111
```
Luhn-validated card number patterns.
### IP Addresses
```
192.168.1.1
10.0.0.1
172.16.0.1
```
IPv4 addresses (IPv6 coming soon).
### API Keys & Tokens
```
sk-abc123...
api_key_xxx...
Bearer eyJhbGciOiJ...
ghp_xxxxxxxxxxxx
```
Common API key and token prefixes.
***
## Limitations
### Not Detected
Some PII types are not currently detected:
| Type | Status |
| ---------------------- | ---------------------------- |
| Names | ❌ Too many false positives |
| Addresses | ❌ Complex, locale-specific |
| Dates of birth | ❌ Ambiguous with other dates |
| Medical record numbers | ❌ Varies by institution |
| Custom IDs | ❌ Unknown format |
### False Positives
Some patterns may be incorrectly flagged:
```python theme={null}
# Might be flagged as phone
text = "Order #555-123-4567" # Could be order number
# Might be flagged as SSN
text = "Version 123-45-6789" # Could be version string
```
### Context Insensitive
Detection is pattern-based, not context-aware:
```python theme={null}
# Both masked the same way
"Call me at 555-123-4567" # Real phone
"The code is 555-123-4567" # Not a phone, but still masked
```
***
## Best Practices
### 1. Enable for User-Facing Output
```python theme={null}
# Always mask when displaying to users
response = mem.ask(question, mask_pii=True)
display_to_user(response.answer)
```
### 2. Keep Original for Internal Use
```python theme={null}
# Raw data for internal processing
results = mem.find(query) # No masking
# Masked for display
masked_results = [mask_pii(r.text) for r in results]
```
### 3. Mask Before Logging
```python theme={null}
import logging
def safe_log(message):
logging.info(mask_pii(message))
```
### 4. Combine with Encryption
For maximum protection:
```bash theme={null}
# Encrypt at rest + mask on output
memvid lock sensitive.mv2 --out sensitive.mv2e
# When using:
memvid unlock sensitive.mv2e --out temp.mv2
memvid ask temp.mv2 -q "..." --mask-pii
memvid lock temp.mv2 --out sensitive.mv2e
rm temp.mv2
```
### 5. Test Your Patterns
Verify detection works for your data:
```python theme={null}
from memvid import detect_pii
# Test with your actual data patterns
test_data = [
"Contact: support@yourcompany.com",
"Phone: 1-800-YOUR-NUM",
"Account: CUST-12345",
]
for text in test_data:
pii = detect_pii(text)
print(f"Found: {len(pii)} PII items in: {text}")
```
***
## Compliance Considerations
### GDPR
* PII masking helps with data minimization
* Original data still stored (not anonymized)
* Consider encryption + masking for full compliance
* Document your data handling in privacy policy
### HIPAA
* Masks PHI in output (emails, phones)
* Not a complete de-identification solution
* Combine with [permission-aware retrieval (ACL)](/concepts/permission-aware-retrieval) and encryption
* Consult compliance officer for healthcare data
### SOC 2
* Demonstrates data protection controls
* Enable masking in production environments
* Log that masking is applied
* Include in security documentation
***
## Performance
PII detection adds minimal overhead:
| Operation | Without Masking | With Masking |
| ----------------- | --------------- | ------------ |
| Ask (short) | 150ms | 155ms |
| Ask (long) | 500ms | 520ms |
| Find (10 results) | 50ms | 55ms |
Overhead is \~5-10% depending on text length.
***
## Future Features
Coming soon:
* Custom PII patterns
* Named entity recognition (names, addresses)
* Per-type masking control
* Reversible masking with keys
* IPv6 address detection
* International phone formats
***
## Next Steps
<CardGroup>
<Card title="Encryption" icon="lock" href="/concepts/encryption">
Encrypt data at rest
</Card>
<Card title="Security FAQ" icon="shield-check" href="/faq/security-and-compliance">
Security and compliance questions
</Card>
</CardGroup>
# Query Usage & Limits
Source: https://docs.memvid.com/concepts/query-usage
Track query usage against your plan quota and handle rate limits
Memvid tracks query usage against your plan's monthly quota. Every `find` and `ask` operation counts toward your limit, regardless of whether you use the CLI, Python SDK, or Node.js SDK.
***
## How Query Tracking Works
1. Each `find` or `ask` operation counts as **1 query**
2. Usage is tracked per **API key**
3. Quota resets on your **monthly billing date**
4. Tracking happens server-side (requires API key)
```mermaid theme={null}
flowchart LR
A[CLI/SDK<br/>find/ask] --> B[Dashboard API]
B --> C[Quota Check]
C --> D{Allow or Reject}
```
***
## Plan Limits
| Plan | Monthly Queries | Storage |
| ---------- | --------------- | ------- |
| Free | Unlimited | 50 MB |
| Starter | 50,000 | 25 GB |
| Pro | 250,000 | 125 GB |
| Enterprise | Unlimited | Custom |
<Note>
Queries are counted even for local `.mv2` files when an API key is configured. This enables usage tracking across all your memories.
</Note>
***
## Checking Your Usage
### CLI
```bash theme={null}
# Show current plan and usage
memvid plan show
```
Output:
```
Plan: starter
Status: active
Usage (this period):
Queries: 12,847 / 50,000 (25.7%)
Storage: 10.5 GB / 25 GB (42.0%)
Billing period:
Started: 2024-12-01
Resets: 2025-01-01 (in 15 days)
Dashboard: https://memvid.com/dashboard/plan
```
JSON output for scripting:
```bash theme={null}
memvid plan show --json
```
```json theme={null}
{
"plan": "starter",
"status": "active",
"queries": {
"used": 12847,
"limit": 25000,
"remaining": 12153,
"percent": 51.39
},
"storage": {
"used_bytes": 2254857830,
"limit_bytes": 5368709120,
"percent": 42.0
},
"period": {
"start": "2024-12-01T00:00:00Z",
"end": "2025-01-01T00:00:00Z",
"days_remaining": 15
}
}
```
### Dashboard
Visit [memvid.com/dashboard/plan](https://memvid.com/dashboard/plan) to see:
* Real-time usage graphs
* Historical usage trends
* Billing information
* Upgrade options
***
## Quota Exceeded Error
When you exceed your monthly quota, queries will fail with error **MV023**:
```bash theme={null}
memvid find memory.mv2 --query "search term"
```
```
Error: Monthly query quota exceeded (MV023)
Used: 25,000 / 25,000
Resets: 2025-01-01 (in 3 days)
Options:
1. Wait for quota reset
2. Upgrade plan: https://memvid.com/dashboard/plan
```
***
## SDK Error Handling
### Python
```python theme={null}
from memvid import use, QuotaExceededError, MemvidError
mem = use('basic', 'memory.mv2')
try:
results = mem.find("search query")
except QuotaExceededError as e:
print(f"Quota exceeded!")
print(f"Used: {e.used} / {e.limit}")
print(f"Resets: {e.reset_date}")
print(f"Upgrade at: {e.upgrade_url}")
except MemvidError as e:
print(f"Other error: {e}")
```
The `QuotaExceededError` includes:
* `used`: Queries used this period
* `limit`: Plan query limit
* `remaining`: Queries remaining (0 when exceeded)
* `reset_date`: When quota resets (ISO 8601)
* `upgrade_url`: Link to upgrade page
### Node.js
```typescript theme={null}
import { use, QuotaExceededError, MemvidError } from '@anthropics/memvid'
const mem = await use('basic', 'memory.mv2')
try {
const results = await mem.find("search query")
} catch (e) {
if (e instanceof QuotaExceededError) {
console.log(`Quota exceeded!`)
console.log(`Used: ${e.used} / ${e.limit}`)
console.log(`Resets: ${e.resetDate}`)
console.log(`Upgrade at: ${e.upgradeUrl}`)
} else if (e instanceof MemvidError) {
console.log(`Other error: ${e.message}`)
}
}
```
***
## Rate Limiting
In addition to monthly quotas, there are per-minute rate limits to prevent abuse:
| Plan | Requests/minute |
| ---------- | --------------- |
| Free | 60 |
| Starter | 300 |
| Pro | 1,000 |
| Enterprise | Custom |
Rate limit errors return **HTTP 429** with headers:
```
X-RateLimit-Limit: 300
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1704067260
```
### Handling Rate Limits
```python theme={null}
import time
from memvid import use, RateLimitError
mem = use('basic', 'memory.mv2')
def search_with_retry(query, max_retries=3):
for attempt in range(max_retries):
try:
return mem.find(query)
except RateLimitError as e:
if attempt < max_retries - 1:
wait_time = e.retry_after or (2 ** attempt)
print(f"Rate limited. Waiting {wait_time}s...")
time.sleep(wait_time)
else:
raise
```
***
## Optimizing Query Usage
### 1. Use Batch Operations
Instead of multiple individual queries, batch when possible:
```python theme={null}
# Inefficient: 10 queries
for term in search_terms:
results = mem.find(term)
# Better: Build compound queries when logical
results = mem.find(" OR ".join(search_terms))
```
### 2. Cache Results
Cache search results for repeated queries:
```python theme={null}
from functools import lru_cache
@lru_cache(maxsize=100)
def cached_search(query):
return mem.find(query)
```
### 3. Use Appropriate Top-K
Don't fetch more results than needed:
```bash theme={null}
# Fetching 100 results when you only need 5
memvid find memory.mv2 --query "term" --top-k 100 # Wasteful
# Better: Request what you need
memvid find memory.mv2 --query "term" --top-k 5
```
### 4. Filter Before Querying
Use scopes and filters to narrow searches:
```bash theme={null}
# Search entire memory
memvid find memory.mv2 --query "report" # Broad
# Better: Scope to relevant documents
memvid find memory.mv2 --query "report" --scope "finance/" --start 2024-01-01
```
***
## Monitoring Usage Programmatically
### Check Before Querying
```python theme={null}
from memvid import get_usage
usage = get_usage()
if usage['queries']['remaining'] < 100:
print(f"Warning: Only {usage['queries']['remaining']} queries left")
print(f"Resets: {usage['period']['end']}")
```
### Track Usage in Applications
```python theme={null}
class QueryTracker:
def __init__(self, mem, warning_threshold=0.8):
self.mem = mem
self.warning_threshold = warning_threshold
def find(self, query, **kwargs):
usage = get_usage()
percent_used = usage['queries']['used'] / usage['queries']['limit']
if percent_used >= self.warning_threshold:
print(f"Warning: {percent_used*100:.1f}% of quota used")
return self.mem.find(query, **kwargs)
```
***
## Without API Key
If no API key is configured:
* Query tracking is **skipped**
* No quota limits apply
* Usage isn't recorded
This is useful for:
* Local development
* Offline usage
* Testing
```bash theme={null}
# Unset API key to disable tracking
memvid config unset api_key
# Queries work but aren't tracked
memvid find memory.mv2 --query "test"
```
<Warning>
Without an API key, you can't access dashboard features like usage history, team sharing, or paid plan capacity.
</Warning>
***
## Syncing Plan Status
If your plan was recently upgraded, sync to get new limits:
```bash theme={null}
# Refresh plan information
memvid plan sync
# Clear cached ticket and re-fetch
memvid plan clear
memvid plan sync
```
***
## Common Issues
### "Quota exceeded" but dashboard shows remaining
1. **Cache lag**: Wait 1-2 minutes for sync
2. **Wrong API key**: Verify key matches dashboard
```bash theme={null}
memvid config check
```
3. **Multiple keys**: Each API key has separate quota
### Queries not being tracked
1. **No API key**: Set one:
```bash theme={null}
memvid config set api_key mv2_xxx
```
2. **Network issues**: Check connectivity to memvid.com
3. **Tracking is best-effort**: Network failures don't block queries
### Rate limited but under quota
Rate limits are per-minute, quotas are per-month:
* You can hit rate limits while having quota remaining
* Wait a minute or implement exponential backoff
***
## Enterprise Options
For high-volume usage:
* **Custom quotas**: Tailored to your needs
* **Dedicated infrastructure**: No shared rate limits
* **Priority support**: Direct engineering access
* **SLA guarantees**: Uptime commitments
Contact [enterprise@memvid.com](mailto:enterprise@memvid.com) for details.
***
## Next Steps
<CardGroup>
<Card title="Capacity & Plans" icon="gauge" href="/concepts/capacity-and-plans">
Storage limits and plan features
</Card>
<Card title="CLI Configuration" icon="gear" href="/cli/advanced-commands">
Set up API keys and configuration
</Card>
</CardGroup>
# Table Extraction
Source: https://docs.memvid.com/concepts/table-extraction
Extract structured tables from PDFs and documents for search and export
Memvid extracts structured tables from PDFs, making tabular data searchable and exportable. Tables are detected automatically using multiple extraction methods, with quality scoring to ensure accurate results.
***
## How It Works
```mermaid theme={null}
flowchart LR
A[PDF Input] --> B[Detection]
B --> C[Quality Score]
C --> D[Structured Table Data]
B -.- E[Stream]
B -.- F[Lattice]
B -.- G[LineBased]
```
Key features:
* **Multiple detection methods** - Stream, Lattice, LineBased
* **Quality scoring** - Filter low-confidence extractions
* **Row embedding** - Make individual rows semantically searchable
* **Export formats** - CSV, JSON, or view inline
***
## Extraction Methods
Memvid tries multiple methods and uses the best result:
| Method | Description | Best For |
| ------------- | ---------------------- | ------------------------------- |
| **Stream** | Text position analysis | Borderless tables, text layouts |
| **Lattice** | Grid line detection | Tables with visible borders |
| **LineBased** | Row/column inference | Mixed formatting |
The extractor automatically selects the method with the highest quality score.
***
## CLI Usage
### Basic Table Extraction
```bash theme={null}
# Extract tables from PDF
memvid put memory.mv2 --input report.pdf --tables
# Extract and embed rows for semantic search
memvid put memory.mv2 --input financial.pdf --tables --embed-rows
```
### Extraction Modes
Control extraction aggressiveness:
```bash theme={null}
# Conservative - high confidence only
memvid tables import memory.mv2 --input report.pdf --mode conservative
# Aggressive - extract everything possible
memvid tables import memory.mv2 --input messy.pdf --mode aggressive
# Default - balanced approach
memvid tables import memory.mv2 --input report.pdf
```
| Mode | Description | Use Case |
| -------------- | -------------------- | ----------------------- |
| `conservative` | High confidence only | Clean, formal documents |
| `balanced` | Default behavior | General purpose |
| `aggressive` | Extract everything | Messy/scanned documents |
### Quality Filters
Filter by table quality:
```bash theme={null}
# Only high-quality tables
memvid tables import memory.mv2 --input report.pdf --min-quality high
# Include medium quality
memvid tables import memory.mv2 --input report.pdf --min-quality medium
# Accept all (including low quality)
memvid tables import memory.mv2 --input report.pdf --min-quality low
```
### Size Filters
Filter by table dimensions:
```bash theme={null}
# Minimum 3 rows and 2 columns
memvid tables import memory.mv2 --input report.pdf --min-rows 3 --min-cols 2
# Skip single-row headers
memvid tables import memory.mv2 --input report.pdf --min-rows 2
```
***
## Managing Tables
### List Tables
```bash theme={null}
# List all tables in memory
memvid tables list memory.mv2
# Output:
# Found 5 tables:
# - pdf_table_1_page1: 12 rows x 4 cols (Stream) [high]
# - pdf_table_2_page1: 8 rows x 3 cols (Lattice) [high]
# - pdf_table_3_page2: 5 rows x 6 cols (LineBased) [medium]
# - pdf_table_4_page3: 20 rows x 2 cols (Stream) [high]
# - pdf_table_5_page5: 3 rows x 4 cols (Lattice) [low]
# JSON output for scripting
memvid tables list memory.mv2 --json
```
### View Table
```bash theme={null}
# View table contents
memvid tables view memory.mv2 --table-id pdf_table_1_page1
# Output:
# ┌──────────────┬──────────┬──────────┬──────────┐
# │ Product │ Qty │ Price │ Total │
# ├──────────────┼──────────┼──────────┼──────────┤
# │ Widget A │ 10 │ $5.00 │ $50.00 │
# │ Widget B │ 5 │ $10.00 │ $50.00 │
# │ Widget C │ 2 │ $25.00 │ $50.00 │
# └──────────────┴──────────┴──────────┴──────────┘
# JSON output
memvid tables view memory.mv2 --table-id pdf_table_1_page1 --json
```
### Export Table
```bash theme={null}
# Export to CSV
memvid tables export memory.mv2 --table-id pdf_table_1_page1 --format csv --out data.csv
# Export to JSON (array of arrays)
memvid tables export memory.mv2 --table-id pdf_table_1_page1 --format json --out data.json
# Export to JSON (array of objects/records)
memvid tables export memory.mv2 --table-id pdf_table_1_page1 --format json --as-records --out data.json
# Export all tables
memvid tables export memory.mv2 --all --format csv --out-dir ./tables/
```
**JSON array format:**
```json theme={null}
[
["Product", "Qty", "Price", "Total"],
["Widget A", "10", "$5.00", "$50.00"],
["Widget B", "5", "$10.00", "$50.00"]
]
```
**JSON records format (`--as-records`):**
```json theme={null}
[
{"Product": "Widget A", "Qty": "10", "Price": "$5.00", "Total": "$50.00"},
{"Product": "Widget B", "Qty": "5", "Price": "$10.00", "Total": "$50.00"}
]
```
***
## Searching Table Data
### Row Embedding
When `--embed-rows` is enabled (default), individual table rows are embedded for semantic search:
```bash theme={null}
# Ingest with row embedding
memvid put memory.mv2 --input financial.pdf --tables --embed-rows
# Search finds relevant rows
memvid find memory.mv2 --query "Q4 revenue"
# Results include table row matches:
# [0.89] Row from pdf_table_2_page3: "Q4 2024 | Revenue | $1,234,567"
```
### Searching Table Content
```bash theme={null}
# Search across all content including tables
memvid find memory.mv2 --query "total sales"
# Filter to table content only
memvid find memory.mv2 --query "total sales" --scope "table:"
```
***
## Use Cases
### Invoice Processing
```bash theme={null}
# Create invoice memory
memvid create invoices.mv2
# Ingest invoices with table extraction
memvid put invoices.mv2 --input ./invoices/ --tables --embed-rows
# Find specific line items
memvid find invoices.mv2 --query "shipping charges"
# Export all invoice tables
memvid tables list invoices.mv2
memvid tables export invoices.mv2 --table-id inv_001_table1 --format csv --out line_items.csv
```
### Financial Reports
```bash theme={null}
# Ingest quarterly reports
memvid put finance.mv2 --input quarterly-reports/ --tables
# Search for metrics
memvid find finance.mv2 --query "EBITDA margin"
# Export data for analysis
memvid tables export finance.mv2 --all --format csv --out-dir ./financial-data/
```
### Research Papers
```bash theme={null}
# Extract data tables from papers
memvid put research.mv2 --input papers/ --tables --min-quality medium
# Find experimental results
memvid find research.mv2 --query "p-value significance"
# Export for meta-analysis
memvid tables export research.mv2 --table-id paper_xyz_table3 --format json --as-records
```
### Payroll/HR Documents
```bash theme={null}
# Process pay stubs
memvid put payroll.mv2 --input paystubs/ --tables --mode conservative
# Search for deductions
memvid find payroll.mv2 --query "401k contribution"
# Export earnings data
memvid tables export payroll.mv2 --table-id stub_jan_table1 --format csv
```
***
## Quality Scoring
Each extracted table receives a quality score based on:
| Factor | Description |
| ------------------------- | ----------------------------- |
| **Structure consistency** | Regular row/column counts |
| **Cell alignment** | Properly aligned content |
| **Header detection** | Clear header row identified |
| **Empty cells** | Low percentage of empty cells |
| **Content coherence** | Related data in columns |
Quality levels:
| Level | Score | Description |
| -------- | --------- | -------------------------- |
| `high` | 0.8 - 1.0 | Reliable, well-structured |
| `medium` | 0.5 - 0.8 | Usable, may need review |
| `low` | 0.0 - 0.5 | Possible extraction errors |
***
## Handling Edge Cases
### Merged Cells
Merged cells are expanded to fill all covered positions:
```
Original: Extracted:
┌───────────┐ ┌─────┬─────┐
│ Header │ → │Header│Header│
├─────┬─────┤ ├─────┼─────┤
│ A │ B │ │ A │ B │
└─────┴─────┘ └─────┴─────┘
```
### Multi-Page Tables
Tables spanning multiple pages are detected and merged when possible:
```bash theme={null}
# Enable cross-page merging (default)
memvid tables import memory.mv2 --input report.pdf --merge-pages
# Disable merging (treat as separate tables)
memvid tables import memory.mv2 --input report.pdf --no-merge-pages
```
### Nested Tables
Nested tables are extracted as separate tables with parent reference:
```bash theme={null}
memvid tables list memory.mv2
# Output:
# - main_table_page1: 10 rows x 4 cols
# └─ nested_table_1: 3 rows x 2 cols (parent: main_table_page1)
```
### Rotated/Sideways Tables
Landscape-oriented tables are automatically detected and rotated:
```bash theme={null}
# Auto-rotation is enabled by default
memvid tables import memory.mv2 --input landscape-report.pdf
# Disable auto-rotation
memvid tables import memory.mv2 --input report.pdf --no-auto-rotate
```
***
## Performance Tips
### Large PDFs
For PDFs with many pages:
```bash theme={null}
# Process specific pages only
memvid tables import memory.mv2 --input large.pdf --pages 1-10
# Skip pages without tables
memvid tables import memory.mv2 --input large.pdf --skip-empty-pages
```
### Batch Processing
For many PDFs:
```bash theme={null}
# Process folder with parallel extraction
memvid put memory.mv2 --input ./pdfs/ --tables --parallel-segments
# Import tables only (no text extraction)
memvid tables import memory.mv2 --input ./pdfs/ --tables-only
```
### Memory Usage
Table extraction can be memory-intensive for complex PDFs:
```bash theme={null}
# Limit concurrent extractions
memvid tables import memory.mv2 --input large.pdf --max-concurrent 2
# Process page-by-page (lower memory)
memvid tables import memory.mv2 --input large.pdf --streaming
```
***
## Troubleshooting
### No Tables Detected
```bash theme={null}
# Try aggressive mode
memvid tables import memory.mv2 --input report.pdf --mode aggressive
# Try specific method
memvid tables import memory.mv2 --input report.pdf --method stream
memvid tables import memory.mv2 --input report.pdf --method lattice
```
### Poor Quality Extraction
```bash theme={null}
# Check quality scores
memvid tables list memory.mv2 --json | jq '.[] | {id, quality}'
# Re-extract with different settings
memvid tables import memory.mv2 --input report.pdf --mode conservative --min-quality high
```
### Missing Rows/Columns
```bash theme={null}
# Adjust detection sensitivity
memvid tables import memory.mv2 --input report.pdf --sensitivity high
# Try lattice method for bordered tables
memvid tables import memory.mv2 --input report.pdf --method lattice
```
***
## Limitations
| Limitation | Workaround |
| --------------------- | ------------------------------ |
| Scanned PDFs | Use OCR preprocessing first |
| Complex nested tables | May extract as multiple tables |
| Very small text | Increase DPI in source |
| Decorative borders | Use stream method |
| Non-standard layouts | Use aggressive mode |
***
## SDK Support
Currently, table extraction is **CLI-only**. SDK support coming soon.
Workaround for SDKs:
```python theme={null}
import subprocess
import json
# Extract tables via CLI
result = subprocess.run([
'memvid', 'tables', 'list', 'memory.mv2', '--json'
], capture_output=True, text=True)
tables = json.loads(result.stdout)
for table in tables:
print(f"Table: {table['id']} - {table['rows']}x{table['cols']}")
```
```typescript theme={null}
import { execSync } from 'child_process'
// Extract tables via CLI
const output = execSync('memvid tables list memory.mv2 --json')
const tables = JSON.parse(output.toString())
tables.forEach(table => {
console.log(`Table: ${table.id} - ${table.rows}x${table.cols}`)
})
```
***
## Next Steps
<CardGroup>
<Card title="CLI Reference" icon="terminal" href="/cli/create-and-put">
Full put command options
</Card>
<Card title="Visual Embeddings" icon="image" href="/concepts/visual-embeddings">
Image and visual search
</Card>
</CardGroup>
# Storage Capacity
Source: https://docs.memvid.com/concepts/tickets-and-capacity
Understanding storage tiers and capacity management in Memvid
Memvid files have configurable storage capacity to help you manage resources effectively. This page explains how capacity works and how to manage it.
***
## Capacity Tiers
When creating a memory file, you can specify the storage tier:
| Tier | Capacity | WAL Size | Typical Use |
| ---------- | --------- | -------- | ------------------------------------- |
| Free | 50 MB | 70 KB | Personal notes, small projects |
| Starter | 25 GB | 70 KB | Development, prototyping |
| Pro | 125 GB | 70 KB | Production workloads |
| Enterprise | Unlimited | 70 KB | Scale and mission-critical production |
### Creating with a Tier
```bash theme={null}
# Create with default (free) tier
memvid create knowledge.mv2
# Create with dev tier
memvid create knowledge.mv2 --tier dev
# Create with enterprise tier
memvid create knowledge.mv2 --tier enterprise
# Create with explicit size
memvid create knowledge.mv2 --size 500MB
memvid create knowledge.mv2 --capacity 2GB
```
```python theme={null}
from memvid_sdk import use
# Open or create with default capacity
mem = use('basic', 'knowledge.mv2')
# Capacity is set at creation time
mem = use('basic', 'new-memory.mv2', mode='create')
```
***
## Checking Capacity
### CLI
```bash theme={null}
memvid stats knowledge.mv2
```
**Output:**
```
Memory: knowledge.mv2
Documents: 150
Active Frames: 148
Size: 52.4 MB
Capacity: 1.0 GB
Utilization: 5.2%
Indices:
Lexical: Yes
Vector: Yes
Time: Yes
```
### JSON Output
```bash theme={null}
memvid stats knowledge.mv2 --json
```
```json theme={null}
{
"frame_count": 150,
"active_frame_count": 148,
"size_bytes": 54945587,
"capacity_bytes": 1073741824,
"storage_utilisation_percent": 5.2
}
```
### Python SDK
```python theme={null}
stats = mem.stats()
print(f"Size: {stats['size_bytes']} bytes")
print(f"Capacity: {stats['capacity_bytes']} bytes")
print(f"Utilization: {stats['storage_utilisation_percent']:.1f}%")
```
### Node.js SDK
```typescript theme={null}
const stats = await mv.stats();
console.log(`Size: ${stats.sizeBytes} bytes`);
console.log(`Capacity: ${stats.capacityBytes} bytes`);
console.log(`Utilization: ${stats.storageUtilisationPercent.toFixed(1)}%`);
```
***
## Capacity Exceeded Errors
When you try to add content that would exceed the file's capacity, you'll get a `CapacityExceeded` error (MV001).
### CLI
```
Error: CapacityExceeded
File capacity: 25000000000 bytes (25 GB)
Current usage: 24000000000 bytes (24 GB)
Requested: 2000000000 bytes (2 GB)
Solutions:
1. Delete unused frames: memvid delete knowledge.mv2 --frame-id <id>
2. Vacuum to reclaim space: memvid doctor knowledge.mv2 --vacuum
3. Create a larger memory file: memvid create new.mv2 --tier dev
```
### Python SDK
```python theme={null}
from memvid_sdk import use, CapacityExceededError
try:
mem.put(file="large-file.pdf")
except CapacityExceededError as e:
print(f"MV001: {e}")
# Handle by cleaning up or using a larger file
```
### Node.js SDK
```typescript theme={null}
import { open, CapacityExceededError } from '@memvid/sdk';
const mv = await open('knowledge.mv2');
try {
await mv.put({ title: 'Large Document', label: 'docs', file: 'large-file.pdf' });
await mv.seal();
} catch (error) {
if (error instanceof CapacityExceededError) {
console.log('MV001:', error.message);
}
}
```
***
## Managing Capacity
### Reclaiming Space
After deleting documents, reclaim unused space:
```bash theme={null}
# Check current usage
memvid stats knowledge.mv2
# Delete old content
memvid delete knowledge.mv2 --frame-id 42 --yes
memvid delete knowledge.mv2 --uri "mv2://old/doc.md" --yes
# Vacuum to reclaim space
memvid doctor knowledge.mv2 --vacuum
# Verify space was reclaimed
memvid stats knowledge.mv2
```
### Moving to a Larger File
If you need more capacity, create a new file and migrate:
```bash theme={null}
# Create new file with larger capacity
memvid create new-knowledge.mv2 --tier dev
# Export content from old file (using SDK)
```
```python theme={null}
from memvid_sdk import use
# Open old file read-only
old = use('basic', 'knowledge.mv2', read_only=True)
# Create new file with larger capacity
new = use('basic', 'new-knowledge.mv2', mode='create')
# Migrate content
timeline = old.timeline(limit=10000)
for entry in timeline['entries']:
frame = old.frame(f"mv2://{entry['uri']}")
if frame:
new.put(text=frame['content'], title=frame['title'])
new.seal()
```
***
## WAL Size and Capacity
The Write-Ahead Log (WAL) size scales with file capacity:
| File Capacity | WAL Size | Checkpoint Threshold |
| ------------- | -------- | -------------------- |
| \< 100 MB | 1 MB | 768 KB (75%) |
| \< 1 GB | 4 MB | 3 MB (75%) |
| \< 10 GB | 16 MB | 12 MB (75%) |
| ≥ 10 GB | 64 MB | 48 MB (75%) |
The WAL checkpoints automatically when it reaches 75% capacity, or you can force a checkpoint with `seal()`.
***
## Monitoring Capacity
### Environment Variables
```bash theme={null}
# Set default capacity for new files
export MEMVID_DEFAULT_TIER=dev
```
### In Scripts
```python theme={null}
from memvid_sdk import use
def check_capacity_before_ingest(path: str, needed_bytes: int) -> bool:
"""Check if there's enough capacity before ingesting."""
mem = use('basic', path, read_only=True)
stats = mem.stats()
available = stats['capacity_bytes'] - stats['size_bytes']
if available < needed_bytes:
print(f"Warning: Only {available} bytes available, need {needed_bytes}")
return False
return True
# Usage
if check_capacity_before_ingest('knowledge.mv2', 50_000_000):
mem = use('basic', 'knowledge.mv2')
mem.put(file='large-file.pdf')
mem.seal()
```
***
## Best Practices
### Capacity Planning
1. **Start small**: Begin with the free tier for testing
2. **Monitor usage**: Check `storage_utilisation_percent` regularly
3. **Plan ahead**: Upgrade before hitting capacity limits
4. **Use compression**: Enable vector compression for large collections
### Storage Optimization
1. **Batch ingestion**: Use `put_many()` for better storage efficiency
2. **Vector compression**: 16x smaller vectors with minimal quality loss
3. **Clean up deletions**: Run `--vacuum` after bulk deletes
4. **Choose appropriate tier**: Match tier to your use case
### Error Handling
Always handle capacity errors gracefully:
```python theme={null}
from memvid_sdk import use, CapacityExceededError
def safe_put(mem, text: str, title: str) -> bool:
try:
mem.put(text=text, title=title)
return True
except CapacityExceededError:
print(f"Capacity exceeded, skipping: {title}")
return False
```
***
## Next Steps
<CardGroup>
<Card title="Memory Architecture" icon="database" href="/concepts/memory-architecture">
Understand the internal structure of .mv2 files
</Card>
<Card title="Troubleshooting" icon="wrench" href="/troubleshooting/cli">
Solve common capacity and error issues
</Card>
</CardGroup>
# Session Replay
Source: https://docs.memvid.com/concepts/time-travel-replay
Record and replay agent sessions for debugging, auditing, and model A/B testing
<Info>
**What is Session Replay?** Record every `put`, `find`, and `ask` operation during an agent session, then replay it with different parameters, models, or frozen context for debugging and auditing.
</Info>
## Overview
<Steps>
<Step title="Record">
Start a session and perform operations (`put`, `find`, `ask`). Every action is captured with full context.
</Step>
<Step title="Save">
End the session. Frames, results, answers, tokens, cost, and grounding scores are persisted.
</Step>
<Step title="Replay">
Re-run the session with different parameters or frozen context to debug or audit.
</Step>
</Steps>
<CardGroup>
<Card title="Debug Mode" icon="bug">
Re-execute searches with different `--top-k` or `--adaptive` settings to find why results were missed
</Card>
<Card title="Audit Mode" icon="shield-check">
Freeze retrieval context and replay with different LLMs using `--audit --use-model --diff`
</Card>
</CardGroup>
## Key Features
| Feature | Description |
| --------------------- | ------------------------------------------------------ |
| **Frozen Context** | Replay with exact same frames - no retrieval drift |
| **Model A/B Testing** | Compare GPT-4 vs Claude vs Gemini with identical input |
| **Cost Tracking** | Token counts and USD cost per query |
| **Grounding Scores** | Detect hallucination risk (0-100%) |
| **Answer Caching** | Skip redundant LLM calls, save money |
| **Diff Reports** | See exactly how answers changed |
## Quick Example
```bash theme={null}
# 1. Start recording
memvid session start knowledge.mv2 --name "Audit 2024-12"
# 2. Ask questions (tokens, cost, grounding tracked)
memvid ask knowledge.mv2 --question "What was the revenue?" --use-model openai:gpt-4o-mini
# tokens: 3112 + 42 = 3154 cost: $0.0005 grounding: 95% (HIGH)
# 3. End session
memvid session end knowledge.mv2
# Session ended. 5 actions recorded.
# 4. Replay with different model + diff
memvid session replay knowledge.mv2 --session <id> \
--audit --use-model claude:claude-4-sonnet --diff
# Diff: IDENTICAL ✓
```
## How It Works
### 1. Start Recording
```bash theme={null}
# CLI
memvid session start knowledge.mv2 --name "Audit Session"
```
```python theme={null}
# Python SDK
session_id = mem.session_start("Audit Session")
```
### 2. Perform Operations
All operations are recorded with full context:
```bash theme={null}
# Ask questions - frames, tokens, and answers are captured
memvid ask knowledge.mv2 --question "What was the acquisition price?" \
--use-model openai:gpt-4o-mini
# Output shows cost and grounding
# tokens: 3112 input + 19 output = 3131 cost: $0.000478
# grounding: 100% (HIGH) - 2/2 sentences grounded
```
### 3. End Session
```bash theme={null}
memvid session end knowledge.mv2
# Output: Session ended. 12 actions recorded.
```
### 4. View Session Details
```bash theme={null}
memvid session view knowledge.mv2 --session <session-id>
# Output:
# Actions:
# [0] FIND - Find { query: "acquisition price", mode: "Hybrid", result_count: 8 }
# [1] ASK - Ask { query: "What was the acquisition price?", provider: "openai", model: "gpt-4o-mini" }
```
## Replay Modes
### Debug Replay (Re-executes Search)
Standard replay re-runs retrieval to compare results:
```bash theme={null}
memvid session replay knowledge.mv2 --session <id> --adaptive --verbose
```
### Audit Replay (Frozen Context)
Audit mode uses the **exact frames** from the original session:
```bash theme={null}
memvid session replay knowledge.mv2 --session <id> --audit
```
Output shows frozen frames:
```
✓ Step 3/12 ask
Question: "What was the acquisition price?"
Mode: AUDIT (frozen retrieval)
Original Model: openai:gpt-4o-mini
Frozen frames: [66, 68, 61, 170, 22, 67, 57, 0]
Context: VERIFIED (frames frozen)
Original Answer: "The acquisition was valued at $2 billion."
```
### Model A/B Testing
Compare different models with **identical context**:
```bash theme={null}
# Original used GPT-4o-mini, replay with Claude
memvid session replay knowledge.mv2 --session <id> \
--audit \
--use-model claude:claude-3-5-sonnet \
--diff
```
Output shows comparison:
```
✓ Step 3/12 ask
Question: "What was the acquisition price?"
Mode: AUDIT (frozen retrieval)
Original Model: openai:gpt-4o-mini
Frozen frames: [66, 68, 61, 170, 22, 67, 57, 0]
Override Model: claude:claude-3-5-sonnet
Original Answer: "The acquisition was valued at $2 billion."
Context: VERIFIED (frames frozen)
New Answer: "The acquisition price was $2B according to the documents."
Diff: CHANGED
```
## Replay Options
| Option | Description |
| --------------------- | ---------------------------------------------------------------------------------------- |
| `--audit` | Freeze retrieval - use recorded frames instead of re-searching |
| `--use-model <model>` | Override the LLM model for comparison (e.g., `openai:gpt-4o`, `gemini:gemini-2.5-flash`) |
| `--diff` | Generate diff report comparing original vs new answers |
| `--adaptive` | Enable adaptive retrieval (debug mode only) |
| `--top-k N` | Override top-k value (debug mode only) |
| `--skip-asks` | Skip LLM operations during replay |
| `--skip-finds` | Skip search operations during replay |
| `--from-checkpoint` | Start replay from a specific checkpoint |
| `--web` | Launch Time Machine web UI |
## Token & Cost Tracking
Every `ask` operation tracks token usage and estimated cost:
```bash theme={null}
memvid ask knowledge.mv2 --question "Summarize the report" \
--use-model openai:gpt-4o-mini --json
```
```json theme={null}
{
"answer": "The report covers...",
"usage": {
"input_tokens": 3648,
"output_tokens": 36,
"total_tokens": 3684,
"cost_usd": 0.000569
},
"grounding": {
"score": 1.0,
"label": "HIGH",
"sentence_count": 1,
"grounded_sentences": 1,
"has_warning": false
},
"cached": false
}
```
### Supported Models & Pricing (Dec 2025)
| Provider | Model | Input/1M | Output/1M |
| -------- | ---------------- | -------- | --------- |
| OpenAI | gpt-4o-mini | \$0.15 | \$0.60 |
| OpenAI | gpt-4o | \$2.50 | \$10.00 |
| OpenAI | gpt-4.5 | \$75.00 | \$150.00 |
| Claude | claude-3-haiku | \$0.25 | \$1.25 |
| Claude | claude-4-sonnet | \$3.00 | \$15.00 |
| Claude | claude-4-opus | \$15.00 | \$75.00 |
| Gemini | gemini-2.5-flash | \$0.15 | \$3.50 |
| Gemini | gemini-2.5-pro | \$1.25 | \$10.00 |
| xAI | grok-4 | \$3.00 | \$15.00 |
| Groq | llama-3.3-70b | \$0.59 | \$0.79 |
| Mistral | mistral-large | \$0.50 | \$1.50 |
## Grounding & Hallucination Detection
Every answer is scored for **grounding** - how well it's supported by the retrieved context:
```
grounding: 100% (HIGH) - 2/2 sentences grounded
```
| Score | Label | Meaning |
| ------- | ------ | --------------------------------------- |
| 70-100% | HIGH | Well-grounded in context |
| 40-69% | MEDIUM | Partially grounded |
| 0-39% | LOW | Potential hallucination - warning shown |
When grounding is low, you'll see a warning:
```
grounding: 25% (LOW) - 1/4 sentences grounded
⚠ Warning: Some statements may not be supported by context
```
## Answer Caching
Repeated questions with the same context return cached answers instantly:
```bash theme={null}
# First call - hits LLM
memvid ask knowledge.mv2 --question "What is the revenue?" --use-model openai
# tokens: 2500 input + 50 output cost: $0.00042
# Second call - cached
memvid ask knowledge.mv2 --question "What is the revenue?" --use-model openai
# cached: true cost: $0.00 (saved $0.00042)
```
Cache key is based on: `model + query + context hash`
## Use Case Examples
### 1. Debug Missing Results
```bash theme={null}
# Record the failing scenario
memvid session start knowledge.mv2 --name "Missing Results Debug"
memvid ask knowledge.mv2 --question "What did Databricks purchase?" --use-model openai
memvid session end knowledge.mv2
# Replay with adaptive retrieval
memvid session replay knowledge.mv2 --session <id> --adaptive --verbose
# Reveals: Document existed at rank 12, adaptive found it
```
### 2. Compliance Audit Trail
```bash theme={null}
# Record all decisions for audit
memvid session start knowledge.mv2 --name "Compliance Review 2024-12"
memvid ask knowledge.mv2 --question "Is this transaction fraudulent?" --use-model openai
memvid session end knowledge.mv2
# Later: Replay with frozen context to verify decision
memvid session replay knowledge.mv2 --session <id> --audit
# Shows exact frames and answer - reproducible for auditors
```
### 3. Model Comparison
```bash theme={null}
# Ask with GPT-4o
memvid session start knowledge.mv2 --name "Model Comparison"
memvid ask knowledge.mv2 --question "Summarize the key findings" --use-model openai:gpt-4o
memvid session end knowledge.mv2
# Replay with different models to compare
memvid session replay knowledge.mv2 --session <id> --audit --use-model gemini:gemini-2.5-pro --diff
memvid session replay knowledge.mv2 --session <id> --audit --use-model claude:claude-4-sonnet --diff
```
## CLI Commands Reference
| Command | Description |
| ------------------------------------------------- | ---------------------- |
| `memvid session start <file> --name <name>` | Start recording |
| `memvid session end <file>` | End recording and save |
| `memvid session list <file>` | List all sessions |
| `memvid session view <file> --session <id>` | View session details |
| `memvid session replay <file> --session <id>` | Replay session |
| `memvid session delete <file> --session <id>` | Delete session |
| `memvid session checkpoint <file>` | Create checkpoint |
| `memvid session compare <file> -a <id1> -b <id2>` | Compare two sessions |
## SDK Support
### Python SDK (Full Support)
```python theme={null}
from memvid_sdk import create
mem = create('knowledge.mv2', enable_vec=True)
# Record session
session_id = mem.session_start("Audit Session")
result = mem.ask("What was the revenue?", model="openai:gpt-4o-mini")
print(f"Cost: ${result.usage.cost_usd:.6f}")
print(f"Grounding: {result.grounding.score:.0%}")
summary = mem.session_end()
# Replay with audit mode
replay = mem.session_replay(
session_id,
audit=True,
use_model="claude:claude-4-sonnet",
diff=True
)
for action in replay.ask_results:
print(f"Original: {action.original_answer}")
print(f"New: {action.new_answer}")
print(f"Diff: {action.diff_status}")
```
### Node.js SDK
```typescript theme={null}
import { create } from '@anthropic/memvid-sdk';
const mem = await create('knowledge.mv2', { enableVec: true });
// Record session
const sessionId = await mem.sessionStart("Audit Session");
const result = await mem.ask("What was the revenue?", { model: "openai:gpt-4o-mini" });
console.log(`Cost: $${result.usage.costUsd.toFixed(6)}`);
await mem.sessionEnd();
// Replay
const replay = await mem.sessionReplay(sessionId, {
audit: true,
useModel: "gemini:gemini-2.5-flash",
diff: true
});
```
## Best Practices
1. **Use descriptive session names**: Include date and purpose, e.g., "Fraud Detection Audit 2024-12-27"
2. **Record minimal reproductions**: Capture just enough to reproduce the issue
3. **Use audit mode for compliance**: Frozen context ensures reproducibility
4. **Compare models with identical context**: Use `--audit --use-model --diff` for fair comparisons
5. **Monitor grounding scores**: Low scores indicate potential hallucination
## Next Steps
<CardGroup>
<Card title="CLI Reference" icon="terminal" href="/cli/advanced-commands">
Full CLI reference for session commands
</Card>
<Card title="Python SDK" icon="python" href="/sdks/python">
Session recording in Python
</Card>
<Card title="LLM Providers" icon="brain" href="/concepts/llm-providers">
Configure OpenAI, Claude, Gemini, and more
</Card>
<Card title="Adaptive Retrieval" icon="chart-line" href="/concepts/indexes-and-tracks">
Learn about adaptive retrieval strategies
</Card>
</CardGroup>
# Visual Embeddings with CLIP
Source: https://docs.memvid.com/concepts/visual-embeddings
Enable image and visual search with CLIP embeddings in Memvid
Memvid supports CLIP (Contrastive Language-Image Pre-training) embeddings for visual search. This enables searching documents, PDFs, and images by visual content, including charts, diagrams, photos, and visual elements using natural language queries.
***
## Overview
CLIP models learn to associate images with text descriptions, enabling:
* **Text-to-image search**: Find images using natural language ("sustainability charts", "team photos")
* **Visual document search**: Search PDF pages by their visual content, not just text
* **Cross-modal retrieval**: Query with text, retrieve visual content
| Provider | Model | Dimensions | Best For |
| ---------- | ---------------------- | ---------- | ---------------------- |
| **Local** | MobileCLIP-S2 | 512 | Offline, privacy-first |
| **OpenAI** | text-embedding-3-small | 1536 | General purpose |
| **OpenAI** | text-embedding-3-large | 3072 | Highest quality |
| **Gemini** | embedding-001 | 768 | Google ecosystem |
***
## Quick Start
### Python SDK
```python theme={null}
from memvid_sdk import create
from memvid_sdk.clip import get_clip_provider
# Initialize CLIP provider
clip = get_clip_provider('openai') # or 'local', 'gemini'
print(f"Provider: {clip.name} ({clip.dimension} dimensions)")
# Create memory and store a PDF
mem = create('visual_search.mv2')
mem.enable_lex()
frame_id = mem.put(
title="Annual Report 2024",
label="report",
metadata={"year": 2024},
file="report.pdf",
)
# Generate text embedding for visual search
query_embedding = clip.embed_text("revenue growth charts")
print(f"Query embedding: {len(query_embedding)} dimensions")
# Search (visual search requires vector index)
results = mem.find("revenue", k=10)
```
### Node.js SDK
```typescript theme={null}
import { create, getClipProvider } from '@memvid/sdk';
// Initialize CLIP provider
const clip = getClipProvider('openai'); // or 'local', 'gemini'
console.log(`Provider: ${clip.name} (${clip.dimension} dimensions)`);
// Create memory and store a PDF
const mem = await create('visual_search.mv2');
await mem.enableLex();
const frameId = await mem.put({
title: 'Annual Report 2024',
label: 'report',
metadata: { year: 2024 },
file: 'report.pdf',
});
// Generate text embedding for visual search
const queryEmbedding = await clip.embedText('revenue growth charts');
console.log(`Query embedding: ${queryEmbedding.length} dimensions`);
// Search
const results = await mem.find('revenue', { k: 10 });
```
***
## Providers
### Local CLIP (MobileCLIP-S2)
The default provider uses MobileCLIP-S2, a lightweight CLIP model optimized for mobile and edge devices.
**Characteristics:**
* **Dimensions**: 512
* **Size**: \~200 MB (downloaded on first use)
* **Inference**: CPU-based, no GPU required
* **Privacy**: All processing happens locally
* **Offline**: Works without internet after initial download
```python theme={null}
from memvid_sdk.clip import get_clip_provider, LocalClip
# Using factory
clip = get_clip_provider('local')
# Or direct instantiation
clip = LocalClip(model='mobileclip-s2')
# Embed an image
image_embedding = clip.embed_image('photo.jpg')
# Embed text for search
text_embedding = clip.embed_text('sunset over ocean')
# Batch embed multiple images
embeddings = clip.embed_images(['img1.jpg', 'img2.jpg', 'img3.jpg'])
```
```typescript theme={null}
import { getClipProvider, LocalClip } from '@memvid/sdk';
// Using factory
const clip = getClipProvider('local');
// Or direct instantiation
const clip = new LocalClip({ model: 'mobileclip-s2' });
// Embed an image
const imageEmbedding = await clip.embedImage('photo.jpg');
// Embed text for search
const textEmbedding = await clip.embedText('sunset over ocean');
// Batch embed multiple images
const embeddings = await clip.embedImages(['img1.jpg', 'img2.jpg', 'img3.jpg']);
```
<Note>
Local CLIP is supported in `memvid-core` and the Python SDK. In Node.js, `LocalClip` requires a native build that exports `ClipModel` (the prebuilt npm binaries may not include it). Cloud providers work out of the box.
</Note>
***
### OpenAI CLIP
OpenAI's embedding models provide excellent quality for visual search queries.
**Setup:**
```bash theme={null}
export OPENAI_API_KEY=sk-your-key-here
```
**Usage:**
```python theme={null}
from memvid_sdk.clip import get_clip_provider, OpenAIClip
# Using factory
clip = get_clip_provider('openai')
# Or with specific model
clip = get_clip_provider('openai:text-embedding-3-large')
# Direct instantiation
clip = OpenAIClip(model='text-embedding-3-small')
# Embed text for visual search
embedding = clip.embed_text('executive team photo')
print(f"Dimensions: {len(embedding)}")
```
```typescript theme={null}
import { getClipProvider, OpenAIClip } from '@memvid/sdk';
// Using factory
const clip = getClipProvider('openai');
// Override embedding/vision models via config
const clip2 = getClipProvider('openai', { embeddingModel: 'text-embedding-3-large', visionModel: 'gpt-4o-mini' });
// Direct instantiation
const clip3 = new OpenAIClip({ embeddingModel: 'text-embedding-3-small', visionModel: 'gpt-4o-mini' });
// Embed text for visual search
const embedding = await clip.embedText('executive team photo');
console.log(`Dimensions: ${embedding.length}`);
```
**Model Comparison:**
| Model | Dimensions | Quality |
| ------------------------ | ---------- | ------- |
| `text-embedding-3-small` | 1536 | Good |
| `text-embedding-3-large` | 3072 | Best |
***
### Gemini CLIP
Google's Gemini provides multimodal embeddings for visual search.
**Setup:**
```bash theme={null}
export GEMINI_API_KEY=your-key-here
```
**Usage:**
```python theme={null}
from memvid_sdk.clip import get_clip_provider, GeminiClip
# Using factory
clip = get_clip_provider('gemini')
# Or with specific model
clip = get_clip_provider('gemini:embedding-001')
# Direct instantiation
clip = GeminiClip(model='embedding-001')
# Embed text
embedding = clip.embed_text('data visualization dashboard')
```
```typescript theme={null}
import { getClipProvider, GeminiClip } from '@memvid/sdk';
const clip = getClipProvider('gemini');
const embedding = await clip.embedText('data visualization dashboard');
```
***
## Complete Example
Here's a full workflow for visual document search:
```python theme={null}
from pathlib import Path
from memvid_sdk import create
from memvid_sdk.clip import get_clip_provider
# Configuration
PROVIDER = 'openai' # 'local', 'openai', 'gemini'
PDF_PATH = 'annual_report.pdf'
OUTPUT_PATH = 'visual_search.mv2'
# Initialize
clip = get_clip_provider(PROVIDER)
print(f"CLIP Provider: {clip.name} ({clip.dimension} dims)")
# Create memory
if Path(OUTPUT_PATH).exists():
Path(OUTPUT_PATH).unlink()
mem = create(OUTPUT_PATH)
mem.enable_lex()
# Ingest PDF
frame_id = mem.put(
title=Path(PDF_PATH).stem,
label='report',
metadata={'source': 'finance', 'year': 2024},
file=PDF_PATH,
)
print(f"Stored PDF as frame {frame_id}")
# Visual search queries
queries = [
'revenue growth charts',
'organizational structure',
'sustainability initiatives',
'executive portraits',
]
print("\nGenerating embeddings for visual search:")
for query in queries:
embedding = clip.embed_text(query)
print(f" '{query}' -> {len(embedding)} dims")
# Seal and show stats
mem.seal()
stats = mem.stats()
print(f"\nFinal: {stats.get('frame_count', 0)} frames")
```
***
## API Reference
### ClipProvider Interface
All CLIP providers implement this interface:
| Method | Description |
| --------------------- | ----------------------------------------------------------- |
| `name` | Provider identifier (e.g., `openai:text-embedding-3-small`) |
| `dimension` | Embedding vector dimension |
| `embed_image(path)` | Generate embedding for a single image |
| `embed_text(text)` | Generate text embedding for visual search |
| `embed_images(paths)` | Batch embed multiple images |
### Factory Function
```python theme={null}
# Python
from memvid_sdk.clip import get_clip_provider
clip = get_clip_provider(provider) # 'local', 'openai', 'gemini', 'openai:model-name'
```
```typescript theme={null}
// Node.js
import { getClipProvider } from '@memvid/sdk';
const clip = getClipProvider(provider); // 'local' | 'openai' | 'gemini'
```
***
## Environment Variables
| Variable | Description |
| ------------------- | --------------------------------- |
| `OPENAI_API_KEY` | OpenAI API key for OpenAI CLIP |
| `GEMINI_API_KEY` | Google AI API key for Gemini CLIP |
| `MEMVID_MODELS_DIR` | Local model cache directory |
| `MEMVID_OFFLINE=1` | Skip model downloads (local CLIP) |
| `MEMVID_CLIP_MODEL` | Override default CLIP model |
***
## Use Cases
### Visual Document Search
Search PDFs by their visual content (charts, diagrams, tables):
```python theme={null}
# Find pages with specific visual elements
clip = get_clip_provider('openai')
query = clip.embed_text('pie chart showing market share')
# Use with memory search
results = mem.find('market share', k=10)
```
### Image Gallery Search
Build searchable image galleries with natural language:
```python theme={null}
# Embed and store images
for image_path in Path('photos/').glob('*.jpg'):
embedding = clip.embed_image(str(image_path))
mem.put(
title=image_path.stem,
label='photo',
file=str(image_path),
metadata={'clip_embedding': embedding},
)
# Search by description
query_embedding = clip.embed_text('beach sunset')
```
### Multimodal RAG
Combine visual and text search for richer retrieval:
```python theme={null}
# Store documents with visual embeddings
for pdf in pdfs:
# Text for lexical search
frame_id = mem.put(title=pdf.name, file=str(pdf))
# Visual embedding for image search
visual_embedding = clip.embed_image(pdf.thumbnail_path)
# Hybrid search combines both modalities
results = mem.find(query, mode='auto')
```
***
## Best Practices
1. **Choose the right provider**: Use local CLIP for privacy/offline, OpenAI for quality
2. **Batch embeddings**: Use `embed_images()` for multiple images to reduce API calls
3. **Cache embeddings**: Store visual embeddings in metadata for reuse
4. **Consistent models**: Use the same model for indexing and querying
5. **Dimension matching**: Ensure query and document embeddings have same dimensions
***
## Limitations
* **Local CLIP (Node.js)**: Requires a native build with CLIP support; prebuilt npm binaries may be cloud-only
* **Image formats**: Supports JPEG, PNG, WebP, GIF
* **PDF visual search**: Requires extracting page images first
* **Model size**: Local CLIP downloads \~200 MB on first use
***
## Next Steps
<CardGroup>
<Card title="Embedding Models" icon="brain" href="/concepts/embedding-models">
Configure text embedding models for semantic search
</Card>
<Card title="Entity Extraction" icon="diagram-project" href="/concepts/entity-extraction">
Extract entities and build knowledge graphs
</Card>
<Card title="Indexes and Tracks" icon="layer-group" href="/concepts/indexes-and-tracks">
Learn about lexical, vector, and time indices
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Complete Python SDK reference
</Card>
</CardGroup>
# Error Code Reference
Source: https://docs.memvid.com/errors/reference
Complete reference for all Memvid error codes with causes and solutions
Understanding Memvid error codes helps you quickly diagnose and resolve issues. All errors follow the format `MVXXX` where `XXX` is a three-digit code.
## Error Code Quick Reference
| Code | Name | Category |
| ----- | -------------------- | -------------- |
| MV001 | CapacityExceeded | Storage |
| MV002 | TicketInvalid | Authentication |
| MV003 | TicketReplay | Authentication |
| MV004 | LexIndexDisabled | Index |
| MV005 | TimeIndexMissing | Index |
| MV006 | VerificationFailed | Integrity |
| MV007 | FileLocked | Concurrency |
| MV008 | ApiKeyRequired | Authentication |
| MV009 | MemoryAlreadyBound | Binding |
| MV010 | FrameNotFound | Data |
| MV011 | VecIndexDisabled | Index |
| MV012 | CorruptFile | Integrity |
| MV013 | IOError | System |
| MV014 | VecDimensionMismatch | Index |
| MV015 | EmbeddingFailed | Embedding |
| MV016 | EncryptedFile | Security |
| MV999 | InternalError | System |
***
## Storage Errors
### MV001 - CapacityExceeded
<Warning>
Your memory file has exceeded its storage capacity limit.
</Warning>
**Cause:** The `.mv2` file has reached its maximum allowed size based on your plan tier.
**Error Message:**
```
CapacityExceeded: Memory file exceeded capacity limit (current: 1.2GB, limit: 1GB)
```
**Solutions:**
<Tabs>
<Tab title="Upgrade Plan">
```bash theme={null}
# Sync with control plane to get higher capacity ticket
memvid tickets sync knowledge.mv2 --memory-id YOUR_MEMORY_ID
```
</Tab>
<Tab title="Reduce Content">
```bash theme={null}
# Delete old or unused frames
memvid delete knowledge.mv2 --before "2024-01-01" --yes
# Vacuum to reclaim space
memvid doctor knowledge.mv2 --vacuum
```
</Tab>
<Tab title="Create New File">
```bash theme={null}
# Archive old file and create fresh one
mv knowledge.mv2 knowledge-archive.mv2
memvid create knowledge.mv2
```
</Tab>
</Tabs>
**Plan Limits:**
| Plan | Capacity |
| ---------- | --------- |
| Free | 1 GB |
| Dev | 10 GB |
| Pro | 100 GB |
| Enterprise | Unlimited |
***
## Authentication Errors
### MV002 - TicketInvalid
<Error>
The ticket signature is invalid and cannot be verified.
</Error>
**Cause:** The ticket's Ed25519 signature doesn't match, indicating tampering or corruption.
**Error Message:**
```
TicketInvalid: Ticket signature verification failed
```
**Solutions:**
```bash theme={null}
# Re-sync tickets from the control plane
memvid tickets sync knowledge.mv2 --memory-id YOUR_MEMORY_ID
# Or apply a new ticket manually
memvid tickets apply knowledge.mv2 --ticket "eyJ..."
```
<Tip>
Tickets are cryptographically signed. Never manually edit ticket strings.
</Tip>
***
### MV003 - TicketReplay
<Error>
This ticket has already been used or has an outdated sequence number.
</Error>
**Cause:** Attempting to apply a ticket with a sequence number less than or equal to the current ticket.
**Error Message:**
```
TicketReplay: Ticket sequence 5 is not greater than current sequence 7
```
**Solutions:**
```bash theme={null}
# Check current ticket info
memvid tickets list knowledge.mv2
# Sync to get the latest ticket
memvid tickets sync knowledge.mv2 --memory-id YOUR_MEMORY_ID
```
***
### MV008 - ApiKeyRequired
<Warning>
An API key is required for this operation.
</Warning>
**Cause:** Attempting to use a feature that requires authentication without providing an API key.
**Error Message:**
```
ApiKeyRequired: API key required for embedding generation
```
**Solutions:**
<Tabs>
<Tab title="Node.js SDK">
```typescript theme={null}
import { use } from '@memvid/sdk';
const mem = await use('basic', 'knowledge.mv2', 'your-api-key');
```
</Tab>
<Tab title="Python SDK">
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2', api_key='your-api-key')
```
</Tab>
<Tab title="Environment Variable">
```bash theme={null}
export MEMVID_API_KEY=your-api-key
memvid put knowledge.mv2 --input doc.pdf --embeddings
```
</Tab>
</Tabs>
***
## Index Errors
### MV004 - LexIndexDisabled
<Warning>
Lexical (text) search is not enabled on this memory file.
</Warning>
**Cause:** Attempting to use `mode: lex` or `mode: auto` on a file without lexical indexing.
**Error Message:**
```
LexIndexDisabled: Lexical index not enabled. Use --enable-lex when creating or run doctor.
```
**Solutions:**
```bash theme={null}
# Enable lexical index on existing file
memvid doctor knowledge.mv2 --rebuild-lex-index
# Or use semantic-only search
memvid find knowledge.mv2 --query "machine learning" --mode sem
```
***
### MV005 - TimeIndexMissing
<Warning>
Time-based queries require a time index that is not present.
</Warning>
**Cause:** Using timeline queries or time filters on a file without time indexing.
**Error Message:**
```
TimeIndexMissing: Time index not found. Rebuild with doctor.
```
**Solutions:**
```bash theme={null}
# Rebuild time index
memvid doctor knowledge.mv2 --rebuild-time-index
```
***
### MV011 - VecIndexDisabled
<Warning>
Vector (semantic) search is not enabled on this memory file.
</Warning>
**Cause:** Attempting to use `mode: sem` or `mode: auto` without vector indexing.
**Error Message:**
```
VecIndexDisabled: Vector index not enabled. Enable embeddings during ingestion.
```
**Solutions:**
```bash theme={null}
# Rebuild vector index (requires re-ingesting)
memvid doctor knowledge.mv2 --rebuild-vec-index
# Or use lexical-only search
memvid find knowledge.mv2 --query "exact phrase" --mode lex
```
***
### MV014 - VecDimensionMismatch
<Error>
The embedding dimensions don't match the vector index.
</Error>
**Cause:** Attempting to search or insert with embeddings of a different dimension than what the vector index was built with.
**Error Message:**
```
VecDimensionMismatch: Vector dimension mismatch (expected 384, got 1536)
```
**Solutions:**
1. **Use consistent embedding models**: Ensure you use the same embedding model for ingestion and search.
```bash theme={null}
# Check current embedding dimension
memvid stats knowledge.mv2 --json | jq '.vec_dimension'
# Use matching model for search
memvid find knowledge.mv2 --query "test" --embedding-model bge-small
```
2. **Rebuild with correct embeddings**: If you need to change models, rebuild the vector index.
```bash theme={null}
# Rebuild vector index with new embeddings
memvid doctor knowledge.mv2 --rebuild-vec-index --embedding-model openai
```
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
// Ensure consistent embedder usage
const mem = await use('basic', 'knowledge.mv2');
// Use same model for put and find
await mem.put({ text: 'content', embeddingModel: 'bge-small' });
await mem.find('query', { embeddingModel: 'bge-small' });
```
</Tab>
<Tab title="Python">
```python theme={null}
# Ensure consistent embedder usage
mem = use('basic', 'knowledge.mv2')
# Use same model for put and find
mem.put(text='content', embedding_model='bge-small')
mem.find('query', embedding_model='bge-small')
```
</Tab>
</Tabs>
***
## Integrity Errors
### MV006 - VerificationFailed
<Error>
File integrity verification failed.
</Error>
**Cause:** The file's checksums don't match, indicating corruption.
**Error Message:**
```
VerificationFailed: Header checksum mismatch (expected: abc123, got: def456)
```
**Solutions:**
```bash theme={null}
# Run deep verification to identify issues
memvid verify knowledge.mv2 --deep
# Attempt repair
memvid doctor knowledge.mv2 --rebuild-lex-index --rebuild-vec-index
# If repair fails, restore from backup
cp knowledge-backup.mv2 knowledge.mv2
```
<Warning>
Always maintain backups of important memory files. Corruption can occur from disk errors, interrupted writes, or software bugs.
</Warning>
***
### MV012 - CorruptFile
<Error>
The memory file is corrupt and cannot be read.
</Error>
**Cause:** Severe file corruption preventing basic operations.
**Error Message:**
```
CorruptFile: Invalid header magic bytes
```
**Solutions:**
```bash theme={null}
# Try doctor repair
memvid doctor knowledge.mv2 --vacuum
# If that fails, restore from backup
cp /backups/knowledge.mv2 ./knowledge.mv2
```
***
### MV013 - IOError
<Error>
An I/O operation failed.
</Error>
**Cause:** File system operations failed, such as file not found, permission denied, or disk full.
**Error Message:**
```
IOError: I/O error: No such file or directory (os error 2)
```
**Solutions:**
```bash theme={null}
# Check if file exists
ls -la knowledge.mv2
# Check disk space
df -h .
# Check file permissions
ls -la knowledge.mv2
# Fix permissions if needed
chmod 644 knowledge.mv2
```
<Tip>
Ensure the file path is correct and you have read/write permissions to the directory.
</Tip>
***
## Concurrency Errors
### MV007 - FileLocked
<Warning>
The file is locked by another process.
</Warning>
**Cause:** Another process has an exclusive write lock on the file.
**Error Message:**
```
FileLocked: File is locked by another process (PID: 12345)
```
**Solutions:**
```bash theme={null}
# Find the locking process
lsof knowledge.mv2
# Wait for it to finish, or kill if stuck
kill 12345
# Open in read-only mode if you only need to query
```
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
const mem = await use('basic', 'knowledge.mv2', { readOnly: true });
```
</Tab>
<Tab title="Python">
```python theme={null}
mem = use('basic', 'knowledge.mv2', read_only=True)
```
</Tab>
</Tabs>
***
## Data Errors
### MV009 - MemoryAlreadyBound
<Warning>
This memory file is already bound to a different memory ID.
</Warning>
**Cause:** Attempting to bind a file that's already associated with another memory in the control plane.
**Error Message:**
```
MemoryAlreadyBound: File is bound to memory_id: abc-123, cannot rebind to def-456
```
**Solutions:**
```bash theme={null}
# Check current binding
memvid info knowledge.mv2
# Unbind if intentional
memvid unbind knowledge.mv2
# Then bind to new memory
memvid tickets sync knowledge.mv2 --memory-id NEW_MEMORY_ID
```
***
### MV010 - FrameNotFound
<Warning>
The requested frame does not exist.
</Warning>
**Cause:** Referencing a frame ID that doesn't exist in the file.
**Error Message:**
```
FrameNotFound: Frame with id 999 not found
```
**Solutions:**
```bash theme={null}
# List available frames
memvid timeline knowledge.mv2 --limit 100
# Check stats for frame count
memvid stats knowledge.mv2
```
***
## Embedding Errors
### MV015 - EmbeddingFailed
<Error>
Embedding generation failed.
</Error>
**Cause:** The embedding runtime is unavailable, the API key is missing or invalid, or the model is not accessible.
**Error Message:**
```
EmbeddingFailed: Failed to generate embeddings: API key not configured for openai
```
**Solutions:**
1. **Check API key configuration**:
```bash theme={null}
# Set environment variable
export OPENAI_API_KEY=sk-...
# Or use local embeddings (no API key needed)
memvid put knowledge.mv2 --input doc.pdf --embedding-model bge-small
```
2. **Use local embeddings**:
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use, LOCAL_EMBEDDING_MODELS } from '@memvid/sdk';
const mem = await use('basic', 'knowledge.mv2');
// Use local model - no API key required
await mem.put({
text: 'content',
enableEmbedding: true,
embeddingModel: LOCAL_EMBEDDING_MODELS.BGE_SMALL
});
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2')
# Use local model - no API key required
mem.put(
text='content',
enable_embedding=True,
embedding_model='bge-small'
)
```
</Tab>
</Tabs>
3. **Check network connectivity** if using external providers:
```bash theme={null}
# Test API connectivity
curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
```
<Tip>
Local embedding models (BGE, Nomic, GTE) work offline and don't require API keys. They're bundled with the SDK.
</Tip>
***
## Security Errors
### MV016 - EncryptedFile
<Warning>
The file is encrypted and requires decryption.
</Warning>
**Cause:** Attempting to open an encrypted `.mv2e` capsule without providing the password.
**Error Message:**
```
EncryptedFile: File is encrypted. Use unlock() with password to decrypt.
```
**Solutions:**
1. **Decrypt the file**:
```bash theme={null}
# Decrypt to .mv2
memvid unlock knowledge.mv2e --password "your-password"
```
2. **Open encrypted file in SDKs**:
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { unlock } from '@memvid/sdk';
// Decrypt first
const decryptedPath = await unlock('knowledge.mv2e', {
password: 'your-password'
});
// Then open normally
const mem = await use('basic', decryptedPath);
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import unlock, use
# Decrypt first
decrypted_path = unlock('knowledge.mv2e', password='your-password')
# Then open normally
mem = use('basic', decrypted_path)
```
</Tab>
</Tabs>
<Warning>
Never commit passwords to version control. Use environment variables or secure secret management.
</Warning>
***
## System Errors
### MV999 - InternalError
<Error>
An unexpected internal error occurred.
</Error>
**Cause:** Unexpected condition in the Memvid engine.
**Error Message:**
```
InternalError: Unexpected condition in segment builder: index out of bounds
```
**Solutions:**
1. **Report the bug:** Include the full error message and stack trace
2. **Try again:** Some transient errors resolve on retry
3. **Check disk space:** Ensure sufficient storage available
4. **Update Memvid:** Upgrade to the latest version
```bash theme={null}
# Check version
memvid --version
# Update
cargo install memvid-cli --force
# or
pip install --upgrade memvid-sdk
```
***
## SDK Error Handling
### Python
```python theme={null}
from memvid_sdk import (
use,
MemvidError,
CapacityExceededError, # MV001
TicketInvalidError, # MV002
TicketReplayError, # MV003
LexIndexDisabledError, # MV004
TimeIndexMissingError, # MV005
VerifyFailedError, # MV006
LockedError, # MV007
ApiKeyRequiredError, # MV008
MemoryAlreadyBoundError, # MV009
FrameNotFoundError, # MV010
VecIndexDisabledError, # MV011
CorruptFileError, # MV012
IOError, # MV013
VecDimensionMismatchError, # MV014
EmbeddingFailedError, # MV015
EncryptedFileError, # MV016
)
try:
mem = use('basic', 'knowledge.mv2')
mem.put({"title": "Doc", "text": "Content...", "label": "test"})
except CapacityExceededError as e:
print(f"Storage full: {e.details}")
except LockedError:
print("File in use, retrying in read-only mode...")
mem = use('basic', 'knowledge.mv2', read_only=True)
except VecDimensionMismatchError as e:
print(f"Embedding dimension mismatch: {e.details}")
except EmbeddingFailedError as e:
print(f"Embedding failed: {e.details}")
except EncryptedFileError:
print("File is encrypted, use unlock() first")
except MemvidError as e:
print(f"Memvid error [{e.code}]: {e.message}")
```
### Node.js
```typescript theme={null}
import {
use,
MemvidError,
CapacityExceededError, // MV001
TicketInvalidError, // MV002
TicketReplayError, // MV003
LexIndexDisabledError, // MV004
TimeIndexMissingError, // MV005
VerifyFailedError, // MV006
LockedError, // MV007
ApiKeyRequiredError, // MV008
MemoryAlreadyBoundError, // MV009
FrameNotFoundError, // MV010
VecIndexDisabledError, // MV011
CorruptFileError, // MV012
IOError, // MV013
VecDimensionMismatchError, // MV014
EmbeddingFailedError, // MV015
EncryptedFileError, // MV016
} from '@memvid/sdk';
try {
const mem = await use('basic', 'knowledge.mv2');
await mem.put({ title: 'Doc', text: 'Content...', label: 'test' });
} catch (error) {
if (error instanceof CapacityExceededError) {
console.error('Storage full:', error.details);
} else if (error instanceof LockedError) {
console.error('File locked, try read-only mode');
} else if (error instanceof VecDimensionMismatchError) {
console.error('Embedding dimension mismatch:', error.details);
} else if (error instanceof EmbeddingFailedError) {
console.error('Embedding failed:', error.details);
} else if (error instanceof EncryptedFileError) {
console.error('File is encrypted, use unlock() first');
} else if (error instanceof MemvidError) {
console.error(`Error [${error.code}]: ${error.message}`);
} else {
throw error;
}
}
```
***
## Getting Help
<CardGroup>
<Card title="Troubleshooting Guide" icon="wrench" href="/errors/troubleshooting">
Step-by-step solutions for common issues
</Card>
<Card title="GitHub Issues" icon="github" href="https://github.com/memvid/memvid/issues">
Report bugs or request features
</Card>
<Card title="Discord Community" icon="discord" href="https://discord.gg/2mynS7fcK7">
Get help from the community
</Card>
<Card title="FAQ" icon="question" href="/faq/general">
Frequently asked questions
</Card>
</CardGroup>
# Troubleshooting Guide
Source: https://docs.memvid.com/errors/troubleshooting
Step-by-step solutions for common Memvid issues
This guide walks through diagnosing and resolving the most common issues you'll encounter with Memvid.
## Quick Diagnosis
Run these commands to quickly identify issues:
```bash theme={null}
# Check file health
memvid verify knowledge.mv2 --deep
# View file stats
memvid stats knowledge.mv2 --json
# Check for locks
lsof knowledge.mv2
```
***
## Common Issues
### "File is locked" when opening
<AccordionGroup>
<Accordion title="Symptoms">
* `FileLocked: File is locked by another process`
* Operations hang indefinitely
* Cannot open file in Python/Node.js
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Find process holding the lock
lsof knowledge.mv2
# Check for zombie processes
ps aux | grep memvid
```
</Accordion>
<Accordion title="Solution">
1. **Wait for the other process to finish**
2. **Kill the blocking process** (if stuck):
```bash theme={null}
kill -9 <PID>
```
3. **Open in read-only mode**:
```python theme={null}
mem = use('basic', 'knowledge.mv2', read_only=True)
```
4. **Check for crashed processes** - if a process crashed while holding a lock, restart your terminal/IDE
</Accordion>
</AccordionGroup>
***
### Search returns no results
<AccordionGroup>
<Accordion title="Symptoms">
* `mem.find()` returns empty results
* CLI search shows "0 results"
* Expected documents not appearing
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Check if file has content
memvid stats knowledge.mv2
# Check which indices are enabled
memvid info knowledge.mv2
# Try different search modes
memvid find knowledge.mv2 --query "test" --mode lex
memvid find knowledge.mv2 --query "test" --mode sem
```
</Accordion>
<Accordion title="Solution">
1. **Verify content exists**:
```bash theme={null}
memvid timeline knowledge.mv2 --limit 10
```
2. **Check search mode** - try `lex` for exact matches, `sem` for semantic:
```python theme={null}
# Exact keyword match
results = mem.find('exact phrase', mode='lex')
# Semantic/conceptual match
results = mem.find('related concept', mode='sem')
```
3. **Rebuild indices** if they're corrupted:
```bash theme={null}
memvid doctor knowledge.mv2 --rebuild-lex-index --rebuild-vec-index
```
4. **Check embeddings were enabled** during ingestion:
```bash theme={null}
memvid put knowledge.mv2 --input doc.pdf --embeddings
```
</Accordion>
</AccordionGroup>
***
### "CapacityExceeded" error
<AccordionGroup>
<Accordion title="Symptoms">
* `CapacityExceeded: Memory file exceeded capacity limit`
* `put()` operations fail
* Cannot add new content
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Check current usage
memvid stats knowledge.mv2 --json | jq '.size_bytes, .capacity_bytes'
# Check ticket info
memvid tickets list knowledge.mv2
```
</Accordion>
<Accordion title="Solution">
1. **Upgrade your plan** for more capacity:
```bash theme={null}
memvid tickets sync knowledge.mv2 --memory-id YOUR_ID
```
2. **Delete old content**:
```bash theme={null}
memvid delete knowledge.mv2 --before "2024-01-01" --yes
```
3. **Vacuum to reclaim space**:
```bash theme={null}
memvid doctor knowledge.mv2 --vacuum
```
4. **Archive and create new file**:
```bash theme={null}
mv knowledge.mv2 archive/knowledge-$(date +%Y%m%d).mv2
memvid create knowledge.mv2
```
</Accordion>
</AccordionGroup>
***
### Slow query performance
<AccordionGroup>
<Accordion title="Symptoms">
* Queries taking >100ms
* Timeouts on large files
* High memory usage during search
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Check file size
ls -lh knowledge.mv2
# Check frame count
memvid stats knowledge.mv2 --json | jq '.frame_count'
# Profile a query
time memvid find knowledge.mv2 --query "test" --json
```
</Accordion>
<Accordion title="Solution">
1. **Reduce `k` value** for fewer results:
```python theme={null}
results = mem.find('query', k=5) # Instead of k=50
```
2. **Use specific search mode**:
```python theme={null}
# Lexical is faster for exact matches
results = mem.find('exact term', mode='lex')
```
3. **Add scope filters**:
```python theme={null}
results = mem.find('query', scope='category:docs')
```
4. **Enable vector compression** for smaller index:
```bash theme={null}
memvid put knowledge.mv2 --input docs/ --vector-compression
```
5. **Split into multiple files** for very large datasets:
```python theme={null}
# Query multiple files in parallel
import asyncio
async def search_all(query):
files = ['docs.mv2', 'wiki.mv2', 'papers.mv2']
tasks = [search_file(f, query) for f in files]
return await asyncio.gather(*tasks)
```
</Accordion>
</AccordionGroup>
***
### Import errors in Python
<AccordionGroup>
<Accordion title="Symptoms">
* `ImportError: cannot import name 'use' from 'memvid_sdk'`
* `ModuleNotFoundError: No module named 'memvid_sdk'`
* `ImportError: libmemvid.so not found`
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Check installation
pip show memvid-sdk
# Check Python version
python --version
# List installed packages
pip list | grep memvid
```
</Accordion>
<Accordion title="Solution">
1. **Install/reinstall the SDK**:
```bash theme={null}
pip install --upgrade memvid-sdk
```
2. **Check Python version** (requires 3.8+):
```bash theme={null}
python3 --version
```
3. **Use correct virtual environment**:
```bash theme={null}
source venv/bin/activate
pip install memvid-sdk
```
4. **On Apple Silicon**, ensure you're using native Python:
```bash theme={null}
# Check architecture
python -c "import platform; print(platform.machine())"
# Should show 'arm64' on Apple Silicon
```
</Accordion>
</AccordionGroup>
***
### Native binding errors in Node.js
<AccordionGroup>
<Accordion title="Symptoms">
* `Error: Cannot find module '../index.node'`
* `Error: The module was compiled against a different Node.js version`
* Segmentation fault on import
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Check Node version
node --version
# Check if native module exists
ls node_modules/@memvid/sdk/*.node
# Check platform
node -e "console.log(process.platform, process.arch)"
```
</Accordion>
<Accordion title="Solution">
1. **Reinstall with rebuild**:
```bash theme={null}
rm -rf node_modules package-lock.json
npm install
```
2. **Check Node.js version** (requires 18+):
```bash theme={null}
nvm use 18
npm rebuild
```
3. **Install build tools** if needed:
```bash theme={null}
# macOS
xcode-select --install
# Ubuntu
sudo apt install build-essential
# Windows
npm install -g windows-build-tools
```
</Accordion>
</AccordionGroup>
***
### File corruption after crash
<AccordionGroup>
<Accordion title="Symptoms">
* `CorruptFile: Invalid header magic bytes`
* `VerificationFailed: Checksum mismatch`
* File won't open after system crash
</Accordion>
<Accordion title="Diagnosis">
```bash theme={null}
# Verify file integrity
memvid verify knowledge.mv2 --deep
# Check file header
xxd knowledge.mv2 | head -5
```
</Accordion>
<Accordion title="Solution">
1. **Run the doctor command**:
```bash theme={null}
memvid doctor knowledge.mv2 --vacuum
```
2. **Rebuild indices**:
```bash theme={null}
memvid doctor knowledge.mv2 \
--rebuild-lex-index \
--rebuild-vec-index \
--rebuild-time-index
```
3. **If recovery fails**, restore from backup:
```bash theme={null}
cp /backups/knowledge.mv2 ./knowledge.mv2
```
4. **Prevent future corruption**:
* Always call `mem.seal()` before exiting
* Use UPS/battery backup for critical systems
* Enable automatic backups
</Accordion>
</AccordionGroup>
***
### Framework adapter not working
<AccordionGroup>
<Accordion title="Symptoms">
* `mem.tools` returns empty or None
* Framework-specific methods missing
* Type errors with framework objects
</Accordion>
<Accordion title="Diagnosis">
```python theme={null}
from memvid_sdk import use
mem = use('langchain', 'knowledge.mv2')
print(f"Tools: {mem.tools}")
print(f"Type: {type(mem.tools)}")
```
</Accordion>
<Accordion title="Solution">
1. **Install the framework dependency**:
```bash theme={null}
# For LangChain
pip install langchain langchain-openai
# For LlamaIndex
pip install llama-index
# For CrewAI
pip install crewai
```
2. **Use correct adapter name**:
```python theme={null}
# Correct
mem = use('langchain', 'knowledge.mv2')
mem = use('llamaindex', 'knowledge.mv2')
mem = use('crewai', 'knowledge.mv2')
# Incorrect
mem = use('lang-chain', 'knowledge.mv2') # Wrong!
```
3. **Check framework version compatibility**:
```bash theme={null}
pip show langchain # Check version
```
</Accordion>
</AccordionGroup>
***
## Diagnostic Commands
### Full Health Check
```bash theme={null}
#!/bin/bash
# health-check.sh
FILE=$1
echo "=== Memvid Health Check ==="
echo "File: $FILE"
echo ""
echo "--- Basic Info ---"
memvid stats "$FILE" --json | jq '.'
echo ""
echo "--- Verification ---"
memvid verify "$FILE" --deep
echo ""
echo "--- Lock Status ---"
lsof "$FILE" 2>/dev/null || echo "No locks detected"
echo ""
echo "--- Ticket Info ---"
memvid tickets list "$FILE"
```
### Performance Profile
```python theme={null}
import time
from memvid_sdk import use
def profile_operations(filepath: str):
"""Profile common Memvid operations."""
print(f"Profiling: {filepath}\n")
mem = use('basic', filepath, read_only=True)
# Profile search
start = time.perf_counter()
for _ in range(10):
mem.find('test query', k=10)
search_time = (time.perf_counter() - start) / 10
print(f"Average search time: {search_time*1000:.2f}ms")
# Profile ask
start = time.perf_counter()
mem.ask('What is this about?')
ask_time = time.perf_counter() - start
print(f"Ask time: {ask_time*1000:.2f}ms")
# Profile timeline
start = time.perf_counter()
mem.timeline(limit=100)
timeline_time = time.perf_counter() - start
print(f"Timeline time: {timeline_time*1000:.2f}ms")
print("\n✅ Profiling complete")
profile_operations('knowledge.mv2')
```
***
## Still Having Issues?
<CardGroup>
<Card title="Error Reference" icon="book" href="/errors/reference">
Complete error code documentation
</Card>
<Card title="GitHub Issues" icon="github" href="https://github.com/memvid/memvid/issues">
Search existing issues or report new ones
</Card>
<Card title="Discord Community" icon="discord" href="https://discord.gg/2mynS7fcK7">
Get real-time help from the community
</Card>
<Card title="Email Support" icon="envelope" href="mailto:support@memvid.com">
Contact our support team
</Card>
</CardGroup>
# Build a Chatbot with Memory
Source: https://docs.memvid.com/examples/chatbot-memory
Create an AI chatbot that remembers conversations and learns from documents
<Info>
**What you'll build:** A production-ready chatbot with persistent memory, document knowledge, and conversation history.
**Time:** 20 minutes | **Difficulty:** Intermediate
</Info>
## Overview
Most chatbots forget everything after each conversation. In this tutorial, you'll build a chatbot that:
* Remembers past conversations
* Learns from documents you provide
* Answers questions using its knowledge base
* Gets smarter over time
```mermaid theme={null}
sequenceDiagram
participant U as User
participant B as Chatbot
participant M as Memvid Memory
U->>B: What did we discuss last week?
B->>M: Search conversation history
M-->>B: Found: project timeline, March 15th deadline
B->>U: Last week we talked about your project timeline...
U->>B: What does our docs say about authentication?
B->>M: Search knowledge base
M-->>B: Found: JWT tokens, 24-hour expiry
B->>U: According to the docs, authentication uses JWT tokens...
```
***
## Prerequisites
<Steps>
<Step title="Install Dependencies">
```bash theme={null}
pip install memvid-sdk openai langchain langchain-openai
```
</Step>
<Step title="Set API Key">
```bash theme={null}
export OPENAI_API_KEY=your-api-key
```
</Step>
</Steps>
***
## Step 1: Create the Memory Store
First, let's create a memory file to store conversations and knowledge:
```python theme={null}
from memvid_sdk import use
import os
# Create or open memory file
mem = use('langchain', 'chatbot-memory.mv2', mode='auto')
# Add some initial knowledge
mem.put(
"Product Overview",
"knowledge",
{},
text="""Our product is an AI-powered analytics platform that helps businesses
understand their data. Key features include:
- Real-time dashboards
- Automated insights
- Custom reports
- API access for developers"""
)
mem.put(
"Pricing Information",
"knowledge",
{},
text="""Pricing tiers:
- Starter: $29/month - Up to 10,000 events
- Pro: $99/month - Up to 100,000 events
- Enterprise: Custom pricing - Unlimited events
All plans include 14-day free trial."""
)
print("Memory initialized with knowledge base")
```
***
## Step 2: Build the Conversation Manager
Create a class to manage conversations and memory:
```python theme={null}
from datetime import datetime
from typing import List, Dict
import json
class ConversationMemory:
def __init__(self, memory_path: str):
self.mem = use('langchain', memory_path, mode='auto')
self.session_id = datetime.now().strftime("%Y%m%d_%H%M%S")
def add_message(self, role: str, content: str):
"""Store a message in memory."""
self.mem.put(
f"Conversation - {self.session_id}",
"conversation",
{
"session_id": self.session_id,
"role": role,
"timestamp": datetime.now().isoformat()
},
text=f"[{role.upper()}]: {content}"
)
def get_relevant_context(self, query: str, k: int = 5) -> str:
"""Retrieve relevant information for the query."""
results = self.mem.find(query, k=k)
context_parts = []
for hit in results.hits:
if hit.label == "knowledge":
context_parts.append(f"[Knowledge] {hit.text}")
elif hit.label == "conversation":
context_parts.append(f"[Previous conversation] {hit.text}")
return "\n\n".join(context_parts)
def get_recent_messages(self, limit: int = 10) -> List[Dict]:
"""Get recent conversation messages."""
timeline = self.mem.timeline(limit=limit)
messages = []
for entry in timeline.entries:
if entry.label == "conversation":
messages.append({
"role": entry.metadata.get("role", "unknown"),
"content": entry.text,
"timestamp": entry.metadata.get("timestamp")
})
return messages
```
***
## Step 3: Create the Chatbot
Now let's build the main chatbot class:
```python theme={null}
from langchain_openai import ChatOpenAI
from langchain.schema import HumanMessage, AIMessage, SystemMessage
class MemvidChatbot:
def __init__(self, memory_path: str = "chatbot-memory.mv2"):
self.memory = ConversationMemory(memory_path)
self.llm = ChatOpenAI(model="gpt-4o", temperature=0.7)
self.system_prompt = """You are a helpful AI assistant with access to a knowledge base and conversation history.
When answering questions:
1. Use the provided context to give accurate, relevant answers
2. Reference specific information from the knowledge base when applicable
3. Remember details from previous conversations
4. Be conversational and helpful
If you don't have enough information, say so honestly."""
def chat(self, user_message: str) -> str:
"""Process a user message and return a response."""
# Store the user message
self.memory.add_message("user", user_message)
# Get relevant context from memory
context = self.memory.get_relevant_context(user_message)
# Build the prompt with context
messages = [
SystemMessage(content=self.system_prompt),
HumanMessage(content=f"""Context from knowledge base and previous conversations:
{context}
---
User message: {user_message}
Please respond helpfully based on the context above.""")
]
# Get response from LLM
response = self.llm.invoke(messages)
assistant_message = response.content
# Store the assistant's response
self.memory.add_message("assistant", assistant_message)
return assistant_message
def add_knowledge(self, title: str, content: str):
"""Add new knowledge to the chatbot's memory."""
self.memory.mem.put(title, "knowledge", {}, text=content)
print(f"Added knowledge: {title}")
```
***
## Step 4: Run the Chatbot
Create an interactive chat loop:
```python theme={null}
def main():
print("Memvid Chatbot initialized")
print("Commands: /add (add knowledge), /history (view recent), /quit (exit)")
print("-" * 50)
bot = MemvidChatbot()
while True:
user_input = input("\nYou: ").strip()
if not user_input:
continue
if user_input.lower() == "/quit":
print("Goodbye!")
break
elif user_input.lower() == "/history":
messages = bot.memory.get_recent_messages(10)
print("\nRecent conversation:")
for msg in messages:
print(f" [{msg['role']}]: {msg['content'][:100]}...")
elif user_input.lower().startswith("/add "):
# Format: /add Title | Content
parts = user_input[5:].split("|", 1)
if len(parts) == 2:
bot.add_knowledge(parts[0].strip(), parts[1].strip())
else:
print("Usage: /add Title | Content")
else:
response = bot.chat(user_input)
print(f"\nBot: {response}")
if __name__ == "__main__":
main()
```
***
## Step 5: Add Advanced Features
### Conversation Summarization
Add automatic summarization for long conversations:
```python theme={null}
def summarize_conversation(self) -> str:
"""Generate a summary of the current conversation."""
recent = self.memory.get_recent_messages(20)
if not recent:
return "No conversation history yet."
conversation_text = "\n".join([
f"{m['role']}: {m['content']}" for m in recent
])
summary_prompt = f"""Summarize this conversation in 2-3 sentences:
{conversation_text}
Summary:"""
response = self.llm.invoke([HumanMessage(content=summary_prompt)])
return response.content
```
### Topic Detection
Automatically tag conversations by topic:
```python theme={null}
def detect_topics(self, message: str) -> List[str]:
"""Detect topics in a message for better retrieval."""
prompt = f"""Extract 1-3 topic tags from this message. Return only comma-separated tags.
Message: {message}
Tags:"""
response = self.llm.invoke([HumanMessage(content=prompt)])
tags = [t.strip() for t in response.content.split(",")]
return tags
```
### Multi-Session Support
Handle multiple users/sessions:
```python theme={null}
class MultiUserChatbot:
def __init__(self, memory_path: str):
self.mem = use('langchain', memory_path, mode='auto')
self.sessions = {}
def get_session(self, user_id: str) -> MemvidChatbot:
"""Get or create a session for a user."""
if user_id not in self.sessions:
self.sessions[user_id] = MemvidChatbot(self.mem)
self.sessions[user_id].session_id = user_id
return self.sessions[user_id]
def chat(self, user_id: str, message: str) -> str:
"""Chat with session isolation."""
session = self.get_session(user_id)
return session.chat(message)
```
***
## Complete Code
Here's the full implementation:
<Accordion title="chatbot.py - Full Code">
```python theme={null}
#!/usr/bin/env python3
"""
Memvid Chatbot with Persistent Memory
A production-ready chatbot that remembers conversations
and learns from documents.
"""
from datetime import datetime
from typing import List, Dict, Optional
from memvid_sdk import use
from langchain_openai import ChatOpenAI
from langchain.schema import HumanMessage, SystemMessage
class ConversationMemory:
"""Manages conversation history and knowledge retrieval."""
def __init__(self, memory_path: str):
self.mem = use('langchain', memory_path, mode='auto')
self.session_id = datetime.now().strftime("%Y%m%d_%H%M%S")
def add_message(self, role: str, content: str, tags: Optional[List[str]] = None):
self.mem.put(
f"Conversation - {self.session_id}",
"conversation",
{
"session_id": self.session_id,
"role": role,
"timestamp": datetime.now().isoformat()
},
text=f"[{role.upper()}]: {content}",
tags=tags or []
)
def get_relevant_context(self, query: str, k: int = 5) -> str:
results = self.mem.find(query, k=k)
context_parts = []
for hit in results.hits:
prefix = "[Knowledge]" if hit.label == "knowledge" else "[History]"
context_parts.append(f"{prefix} {hit.text}")
return "\n\n".join(context_parts)
def get_recent_messages(self, limit: int = 10) -> List[Dict]:
timeline = self.mem.timeline(limit=limit)
return [
{
"role": e.metadata.get("role", "unknown"),
"content": e.text,
"timestamp": e.metadata.get("timestamp")
}
for e in timeline.entries
if e.label == "conversation"
]
class MemvidChatbot:
"""AI chatbot with persistent memory."""
def __init__(self, memory_path: str = "chatbot-memory.mv2"):
self.memory = ConversationMemory(memory_path)
self.llm = ChatOpenAI(model="gpt-4o", temperature=0.7)
self.system_prompt = """You are a helpful AI assistant with memory.
Use the provided context to give accurate answers.
Reference the knowledge base when relevant.
Remember details from previous conversations."""
def chat(self, user_message: str) -> str:
self.memory.add_message("user", user_message)
context = self.memory.get_relevant_context(user_message)
messages = [
SystemMessage(content=self.system_prompt),
HumanMessage(content=f"Context:\n{context}\n\nUser: {user_message}")
]
response = self.llm.invoke(messages)
self.memory.add_message("assistant", response.content)
return response.content
def add_knowledge(self, title: str, content: str):
self.memory.mem.put(title, "knowledge", {}, text=content)
def main():
print("Memvid Chatbot")
print("Commands: /add Title | Content, /history, /quit")
print("-" * 50)
bot = MemvidChatbot()
while True:
user_input = input("\nYou: ").strip()
if not user_input:
continue
if user_input == "/quit":
break
if user_input == "/history":
for m in bot.memory.get_recent_messages(10):
print(f" [{m['role']}]: {m['content'][:80]}...")
continue
if user_input.startswith("/add "):
parts = user_input[5:].split("|", 1)
if len(parts) == 2:
bot.add_knowledge(parts[0].strip(), parts[1].strip())
continue
print(f"\n{bot.chat(user_input)}")
if __name__ == "__main__":
main()
```
</Accordion>
***
## Deployment
### As a FastAPI Service
```python theme={null}
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
bot = MemvidChatbot()
class ChatRequest(BaseModel):
message: str
user_id: str = "default"
class ChatResponse(BaseModel):
response: str
@app.post("/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
response = bot.chat(request.message)
return ChatResponse(response=response)
@app.post("/knowledge")
async def add_knowledge(title: str, content: str):
bot.add_knowledge(title, content)
return {"status": "added"}
```
### With Docker
```dockerfile theme={null}
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```
***
## Next Steps
<CardGroup>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Add PDF/document support
</Card>
<Card title="Knowledge Base" icon="book" href="/examples/knowledge-base">
Build a searchable knowledge base
</Card>
<Card title="Research Assistant" icon="flask" href="/examples/research-assistant">
Create an AI research assistant
</Card>
<Card title="LangChain Integration" icon="link" href="/frameworks/langchain">
Advanced RAG pipelines
</Card>
</CardGroup>
# Document Q&A System
Source: https://docs.memvid.com/examples/document-qa
Build a system that answers questions from your documents
<Info>
**What you'll build:** A document Q\&A system that ingests PDFs, processes them, and answers questions using RAG.
**Time:** 15 minutes | **Difficulty:** Beginner
</Info>
## Overview
Upload documents, ask questions, get accurate answers with sources. Perfect for:
* 📄 Legal document analysis
* 📚 Research paper queries
* 📋 Policy/handbook searches
* 🔍 Technical documentation
***
## Quick Start
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
import * as fs from 'fs';
import * as path from 'path';
// Create document store
const mem = await use('basic', 'documents.mv2', { mode: 'create' });
// Ingest documents
const docsDir = './docs';
for (const filename of fs.readdirSync(docsDir)) {
if (filename.endsWith('.pdf')) {
await mem.put({
title: filename,
label: 'document',
file: path.join(docsDir, filename),
});
}
}
// Ask questions
const answer = await mem.ask('What is the refund policy?', { returnSources: true });
console.log(`Answer: ${answer.answer}`);
console.log(`Source: ${answer.sources?.[0]?.title ?? 'n/a'}`);
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
import os
# Create document store
mem = use('basic', 'documents.mv2', mode='create')
# Ingest a folder of documents
for filename in os.listdir('./docs'):
if filename.endswith('.pdf'):
mem.put({
"title": filename,
"label": "document",
"file": f"./docs/{filename}"
})
# Ask questions
answer = mem.ask("What is the refund policy?")
print(f"Answer: {answer.get('answer')}")
print(f"Source: {(answer.get('sources') or [{}])[0].get('title', 'n/a')}")
```
</Tab>
<Tab title="CLI">
```bash theme={null}
# Create and ingest
memvid create documents.mv2
memvid put documents.mv2 --input ./docs/
# Ask questions
memvid ask documents.mv2 --question "What is the refund policy?"
```
</Tab>
</Tabs>
***
## Full Implementation
### Step 1: Document Processor Class
```python theme={null}
from memvid_sdk import use
from pathlib import Path
from typing import List, Optional
import hashlib
class DocumentQA:
"""Document Q&A system with Memvid."""
SUPPORTED_FORMATS = {'.pdf', '.docx', '.txt', '.md', '.html'}
def __init__(self, memory_path: str = "documents.mv2"):
self.mem = use('basic', memory_path, mode='auto')
self.stats = {"ingested": 0, "failed": 0}
def ingest_file(self, filepath: str, metadata: Optional[dict] = None) -> bool:
"""Ingest a single file."""
path = Path(filepath)
if path.suffix.lower() not in self.SUPPORTED_FORMATS:
print(f"⚠️ Unsupported format: {path.suffix}")
return False
try:
# Generate unique ID based on content hash
content_hash = hashlib.md5(path.read_bytes()).hexdigest()[:8]
self.mem.put({
"title": path.name,
"label": "document",
"file": str(path.absolute()),
"metadata": {
"path": str(path),
"size": path.stat().st_size,
"hash": content_hash,
**(metadata or {})
}
})
self.stats["ingested"] += 1
print(f"✅ Ingested: {path.name}")
return True
except Exception as e:
self.stats["failed"] += 1
print(f"❌ Failed: {path.name} - {e}")
return False
def ingest_folder(self, folder_path: str, recursive: bool = True) -> dict:
"""Ingest all documents from a folder."""
folder = Path(folder_path)
pattern = "**/*" if recursive else "*"
files = [f for f in folder.glob(pattern)
if f.is_file() and f.suffix.lower() in self.SUPPORTED_FORMATS]
print(f"📂 Found {len(files)} documents to ingest...")
for filepath in files:
self.ingest_file(str(filepath))
return self.stats
def ask(self, question: str, k: int = 5) -> dict:
"""Ask a question about the documents."""
result = self.mem.ask(question, k=k)
return {
"answer": result.text,
"sources": [
{
"title": s.title,
"snippet": s.snippet,
"score": s.score
}
for s in result.sources
],
"confidence": result.confidence if hasattr(result, 'confidence') else None
}
def search(self, query: str, k: int = 10) -> List[dict]:
"""Search documents without generating an answer."""
results = self.mem.find(query, k=k)
return [
{
"title": hit.title,
"snippet": hit.snippet,
"score": hit.score,
"metadata": hit.metadata
}
for hit in results.hits
]
def get_stats(self) -> dict:
"""Get document store statistics."""
stats = self.mem.stats()
return {
"total_documents": stats.get("frame_count", 0),
"size_bytes": stats.get("size_bytes", 0),
"size_mb": round(stats.get("size_bytes", 0) / 1024 / 1024, 2)
}
```
### Step 2: Interactive CLI
```python theme={null}
def main():
import argparse
parser = argparse.ArgumentParser(description="Document Q&A System")
parser.add_argument("--memory", default="documents.mv2", help="Memory file path")
subparsers = parser.add_subparsers(dest="command")
# Ingest command
ingest_parser = subparsers.add_parser("ingest", help="Ingest documents")
ingest_parser.add_argument("path", help="File or folder path")
ingest_parser.add_argument("--recursive", "-r", action="store_true")
# Ask command
ask_parser = subparsers.add_parser("ask", help="Ask a question")
ask_parser.add_argument("question", help="Question to ask")
# Search command
search_parser = subparsers.add_parser("search", help="Search documents")
search_parser.add_argument("query", help="Search query")
# Stats command
subparsers.add_parser("stats", help="Show statistics")
args = parser.parse_args()
qa = DocumentQA(args.memory)
if args.command == "ingest":
path = Path(args.path)
if path.is_file():
qa.ingest_file(str(path))
else:
qa.ingest_folder(str(path), recursive=args.recursive)
print(f"\n📊 Ingested: {qa.stats['ingested']}, Failed: {qa.stats['failed']}")
elif args.command == "ask":
result = qa.ask(args.question)
print(f"\n💡 Answer: {result['answer']}")
print(f"\n📚 Sources:")
for s in result['sources'][:3]:
print(f" - {s['title']} (score: {s['score']:.2f})")
elif args.command == "search":
results = qa.search(args.query)
print(f"\n🔍 Found {len(results)} results:")
for r in results[:5]:
print(f"\n 📄 {r['title']} (score: {r['score']:.2f})")
print(f" {r['snippet'][:150]}...")
elif args.command == "stats":
stats = qa.get_stats()
print(f"\n📊 Document Store Statistics:")
print(f" Documents: {stats['total_documents']}")
print(f" Size: {stats['size_mb']} MB")
else:
parser.print_help()
if __name__ == "__main__":
main()
```
***
## Web API
Deploy as a REST API:
```python theme={null}
from fastapi import FastAPI, UploadFile, File, HTTPException
from pydantic import BaseModel
import tempfile
import shutil
app = FastAPI(title="Document Q&A API")
qa = DocumentQA("documents.mv2")
class QuestionRequest(BaseModel):
question: str
k: int = 5
class SearchRequest(BaseModel):
query: str
k: int = 10
@app.post("/upload")
async def upload_document(file: UploadFile = File(...)):
"""Upload and ingest a document."""
# Save to temp file
with tempfile.NamedTemporaryFile(delete=False, suffix=file.filename) as tmp:
shutil.copyfileobj(file.file, tmp)
tmp_path = tmp.name
# Ingest
success = qa.ingest_file(tmp_path, metadata={"original_name": file.filename})
if not success:
raise HTTPException(400, "Failed to ingest document")
return {"status": "success", "filename": file.filename}
@app.post("/ask")
async def ask_question(request: QuestionRequest):
"""Ask a question about the documents."""
return qa.ask(request.question, k=request.k)
@app.post("/search")
async def search_documents(request: SearchRequest):
"""Search documents."""
return qa.search(request.query, k=request.k)
@app.get("/stats")
async def get_stats():
"""Get document store statistics."""
return qa.get_stats()
```
***
## Usage Examples
### Legal Document Analysis
```python theme={null}
qa = DocumentQA("legal-docs.mv2")
# Ingest contracts
qa.ingest_folder("./contracts/")
# Ask specific questions
result = qa.ask("What are the termination clauses in the vendor contracts?")
print(result["answer"])
# Search for specific terms
matches = qa.search("indemnification liability")
for m in matches:
print(f"{m['title']}: {m['snippet']}")
```
### Research Paper Analysis
```python theme={null}
qa = DocumentQA("research-papers.mv2")
qa.ingest_folder("./papers/", recursive=True)
# Synthesize across papers
result = qa.ask(
"What are the main approaches to transformer optimization "
"mentioned across these papers?"
)
print("Summary:", result["answer"])
print("\nKey papers:")
for source in result["sources"]:
print(f" - {source['title']}")
```
***
## Next Steps
<CardGroup>
<Card title="Add Chat Interface" icon="comments" href="/examples/chatbot-memory">
Build conversational Q\&A
</Card>
<Card title="Research Assistant" icon="flask" href="/examples/research-assistant">
Analyze academic papers
</Card>
<Card title="Knowledge Base" icon="book" href="/examples/knowledge-base">
Build a searchable wiki
</Card>
<Card title="LlamaIndex Integration" icon="link" href="/frameworks/llamaindex">
Advanced query engines
</Card>
</CardGroup>
# Examples
Source: https://docs.memvid.com/examples/index
Real-world examples and tutorials to accelerate your development
Learn by building. These examples show you how to use Memvid in production scenarios.
<CardGroup>
<Card title="RAG Chatbot" icon="comments" href="/examples/chatbot-memory">
Build a chatbot with persistent conversation memory
</Card>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Create a document question-answering system
</Card>
<Card title="Knowledge Base" icon="book" href="/examples/knowledge-base">
Build a searchable company knowledge base
</Card>
<Card title="Research Assistant" icon="flask" href="/examples/research-assistant">
Create an AI research assistant for papers
</Card>
<Card title="Memvid Canvas" icon="palette" href="https://github.com/memvid/memvid">
AI UI Kit. Build AI-powered apps in minutes.
</Card>
<Card title="Agents & Packages" icon="cubes" href="/examples/packages">
Official tools and packages built with Memvid
</Card>
</CardGroup>
***
## Quick Examples
### Semantic Search in 10 Lines
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open or create memory
const mem = await use('basic', 'knowledge.mv2', { mode: 'auto' });
// Add some knowledge
await mem.put({ title: 'AI Basics', label: 'docs', text: 'Artificial intelligence is...' });
await mem.put({ title: 'ML Guide', label: 'docs', text: 'Machine learning enables...' });
// Search semantically
const results = await mem.find('how do computers learn?', { k: 5 });
results.hits.forEach(hit => console.log(`${hit.title}: ${hit.snippet}`));
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
# Open or create memory
mem = use('basic', 'knowledge.mv2', mode='auto')
# Add some knowledge
mem.put({"title": "AI Basics", "label": "docs", "text": "Artificial intelligence is..."})
mem.put({"title": "ML Guide", "label": "docs", "text": "Machine learning enables..."})
# Search semantically
results = mem.find("how do computers learn?", k=5)
for hit in results.hits:
print(f"{hit.title}: {hit.snippet}")
```
</Tab>
</Tabs>
***
### LangChain RAG Pipeline
```python theme={null}
from memvid_sdk import use
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
# Initialize with LangChain adapter
mem = use('langchain', 'knowledge.mv2')
# Create RAG chain
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o"),
retriever=mem.as_retriever(k=5),
return_source_documents=True
)
# Ask questions
result = qa.invoke("What are the main features?")
print(result['result'])
```
***
### Next.js API Route with Streaming
```typescript theme={null}
// app/api/chat/route.ts
import { use } from '@memvid/sdk';
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
const mem = await use('vercel-ai', 'knowledge.mv2');
export async function POST(req: Request) {
const { messages } = await req.json();
const result = await streamText({
model: openai('gpt-4o'),
system: 'You are a helpful assistant. Use the tools to search the knowledge base.',
tools: mem.tools,
messages,
maxSteps: 5,
});
return result.toDataStreamResponse();
}
```
***
### Multi-Agent Research Team
```python theme={null}
from memvid_sdk import use
from autogen import AssistantAgent, UserProxyAgent
mem = use('autogen', 'research-papers.mv2')
# Create researcher agent with Memvid tools
researcher = AssistantAgent(
name="researcher",
llm_config={"tools": mem.tools},
system_message="You research topics using the knowledge base."
)
# Create writer agent
writer = AssistantAgent(
name="writer",
system_message="You write summaries based on research findings."
)
# Create user proxy
user = UserProxyAgent(name="user", human_input_mode="NEVER")
# Start research task
user.initiate_chat(
researcher,
message="Research the latest advances in transformer architectures"
)
```
***
## Example Categories
### By Use Case
<AccordionGroup>
<Accordion title="Chatbots & Assistants" icon="robot">
* [Chatbot with Memory](/examples/chatbot-memory) - Persistent conversation context
</Accordion>
<Accordion title="Document Processing" icon="file-lines">
* [Document Q\&A](/examples/document-qa) - Answer questions from PDFs
* [Research Papers](/examples/research-assistant) - Analyze academic papers
</Accordion>
<Accordion title="Knowledge Management" icon="book">
* [Company Wiki](/examples/knowledge-base) - Searchable internal docs
</Accordion>
</AccordionGroup>
### By Framework
| Framework | Integration Guide |
| ---------- | ------------------------------------------------ |
| LangChain | [LangChain Integration](/frameworks/langchain) |
| LlamaIndex | [LlamaIndex Integration](/frameworks/llamaindex) |
| Vercel AI | [Vercel AI Integration](/frameworks/vercel-ai) |
| OpenAI | [OpenAI Integration](/frameworks/openai) |
| Google ADK | [Google ADK Integration](/frameworks/google-adk) |
***
## Starter Templates
Get started quickly with our templates:
<CardGroup>
<Card title="Next.js + Memvid" icon="react" href="https://github.com/memvid/template-nextjs">
Full-stack AI app template
</Card>
<Card title="Python FastAPI" icon="python" href="https://github.com/memvid/template-fastapi">
REST API with RAG
</Card>
<Card title="CLI Tool" icon="terminal" href="https://github.com/memvid/template-cli">
Command-line AI assistant
</Card>
</CardGroup>
***
## Community Examples
Explore what others have built:
<Card title="Awesome Memvid" icon="star" href="https://github.com/memvid/awesome-memvid">
Community-curated list of Memvid projects, tutorials, and resources
</Card>
### Featured Projects
| Project | Description | Author |
| ----------------------------------------------------- | ------------------------ | ----------- |
| [DocuChat](https://github.com/example/docuchat) | Chat with your documents | @developer1 |
| [ResearchGPT](https://github.com/example/researchgpt) | Academic paper assistant | @developer2 |
| [CodeSearch](https://github.com/example/codesearch) | Semantic code search | @developer3 |
***
## Submit Your Example
Built something cool? Share it with the community!
<Card title="Submit Example" icon="plus" href="https://github.com/memvid/memvid/issues/new?template=example_submission.md">
Submit your example for inclusion in the docs
</Card>
# Build a Knowledge Base
Source: https://docs.memvid.com/examples/knowledge-base
Create a searchable company knowledge base with natural language queries
<Info>
**What you'll build:** A production-ready knowledge base that lets users search company docs, wikis, and FAQs using natural language.
**Time:** 25 minutes | **Difficulty:** Intermediate
</Info>
## Overview
Build a searchable knowledge base that:
* 📚 Ingests documents from multiple sources
* 🔍 Supports natural language search
* 💬 Answers questions with AI
* 🔐 Respects access controls
* 📊 Tracks popular queries
***
## Architecture
```mermaid theme={null}
flowchart TD
subgraph Sources["Data Sources"]
N[Notion Docs]
G[Google Docs]
GH[GitHub Wiki]
end
subgraph Storage["Memvid Storage"]
MV[(knowledge.mv2)]
end
subgraph Consumers["Applications"]
W[Web App]
S[Slack Bot]
A[API Endpoint]
end
N --> MV
G --> MV
GH --> MV
MV --> W
MV --> S
MV --> A
```
***
## Quick Start
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
// Create knowledge base
const kb = await use('basic', 'company-kb.mv2', { mode: 'create' });
// Ingest documentation
await kb.put({
title: 'Employee Handbook',
label: 'hr',
file: './docs/handbook.pdf',
tags: ['hr', 'policies', 'onboarding'],
});
// Search
const results = await kb.find('vacation policy', { k: 5 });
// Q&A
const answer = await kb.ask('How many vacation days do employees get?');
console.log(answer.answer);
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
import os
# Create knowledge base
kb = use('basic', 'company-kb.mv2', mode='create')
# Ingest documentation
kb.put({
"title": "Employee Handbook",
"label": "hr",
"file": "./docs/handbook.pdf",
"tags": ["hr", "policies", "onboarding"]
})
kb.put({
"title": "API Documentation",
"label": "engineering",
"file": "./docs/api-guide.md",
"tags": ["api", "development", "integration"]
})
# Search
results = kb.find("vacation policy", k=5)
# Q&A
answer = kb.ask("How many vacation days do employees get?")
print(answer["answer"])
```
</Tab>
</Tabs>
***
## Full Implementation
### Knowledge Base Class
```python theme={null}
from memvid_sdk import use
from pathlib import Path
from typing import List, Dict, Optional
from datetime import datetime
import json
class KnowledgeBase:
"""Company knowledge base with Memvid."""
def __init__(self, memory_path: str = "knowledge.mv2"):
self.mem = use('basic', memory_path, mode='auto')
self.sources = {}
def add_document(
self,
title: str,
content: str,
category: str,
tags: Optional[List[str]] = None,
source: Optional[str] = None
) -> int:
"""Add a document to the knowledge base."""
frame_id = self.mem.put({
"title": title,
"label": category,
"text": content,
"tags": tags or [],
"metadata": {
"source": source,
"added_at": datetime.now().isoformat()
}
})
return frame_id
def add_file(
self,
filepath: str,
category: str,
tags: Optional[List[str]] = None
) -> int:
"""Add a file to the knowledge base."""
path = Path(filepath)
return self.mem.put({
"title": path.name,
"label": category,
"file": str(path.absolute()),
"tags": tags or [],
"metadata": {
"source": "file",
"original_path": str(path)
}
})
def add_folder(self, folder_path: str, category: str) -> int:
"""Add all documents from a folder."""
count = 0
for path in Path(folder_path).rglob("*"):
if path.is_file() and path.suffix in ['.pdf', '.md', '.txt', '.docx']:
self.add_file(str(path), category)
count += 1
return count
def search(self, query: str, category: Optional[str] = None, k: int = 10) -> List[Dict]:
"""Search the knowledge base."""
scope = f"label:{category}" if category else None
results = self.mem.find(query, k=k, scope=scope)
return [
{
"title": hit.title,
"snippet": hit.snippet,
"score": hit.score,
"category": hit.label
}
for hit in results.hits
]
def ask(self, question: str, category: Optional[str] = None) -> Dict:
"""Ask a question."""
scope = f"label:{category}" if category else None
answer = self.mem.ask(question, k=5, scope=scope)
return {
"answer": answer.get("answer"),
"sources": [s.get("title") for s in (answer.get("sources") or [])],
"confidence": getattr(answer, 'confidence', None)
}
def get_categories(self) -> List[str]:
"""Get all categories."""
stats = self.mem.stats()
# This is a simplified version
return list(set(entry.label for entry in self.mem.timeline(limit=1000).entries))
```
### Web API with FastAPI
```python theme={null}
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Optional, List
app = FastAPI(title="Knowledge Base API")
kb = KnowledgeBase("company-kb.mv2")
class SearchRequest(BaseModel):
query: str
category: Optional[str] = None
limit: int = 10
class AskRequest(BaseModel):
question: str
category: Optional[str] = None
class AddDocumentRequest(BaseModel):
title: str
content: str
category: str
tags: Optional[List[str]] = None
@app.post("/search")
async def search(request: SearchRequest):
results = kb.search(request.query, request.category, request.limit)
return {"results": results, "total": len(results)}
@app.post("/ask")
async def ask(request: AskRequest):
return kb.ask(request.question, request.category)
@app.post("/documents")
async def add_document(request: AddDocumentRequest):
frame_id = kb.add_document(
request.title,
request.content,
request.category,
request.tags
)
return {"frame_id": frame_id, "status": "added"}
@app.get("/categories")
async def get_categories():
return {"categories": kb.get_categories()}
```
### React Frontend Component
```tsx theme={null}
import { useState } from 'react';
export function KnowledgeSearch() {
const [query, setQuery] = useState('');
const [results, setResults] = useState([]);
const [answer, setAnswer] = useState('');
const handleSearch = async () => {
const res = await fetch('/api/search', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query, limit: 10 }),
});
const data = await res.json();
setResults(data.results);
};
const handleAsk = async () => {
const res = await fetch('/api/ask', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ question: query }),
});
const data = await res.json();
setAnswer(data.answer);
};
return (
<div className="knowledge-search">
<input
type="text"
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder="Search or ask a question..."
/>
<button onClick={handleSearch}>Search</button>
<button onClick={handleAsk}>Ask AI</button>
{answer && (
<div className="answer-box">
<h3>Answer</h3>
<p>{answer}</p>
</div>
)}
<div className="results">
{results.map((r, i) => (
<div key={i} className="result-card">
<h4>{r.title}</h4>
<p>{r.snippet}</p>
<span className="category">{r.category}</span>
</div>
))}
</div>
</div>
);
}
```
***
## Integrations
### Sync from Notion
```python theme={null}
from notion_client import Client
notion = Client(auth=os.environ["NOTION_TOKEN"])
def sync_notion_pages(database_id: str, kb: KnowledgeBase):
"""Sync pages from a Notion database."""
pages = notion.databases.query(database_id=database_id)
for page in pages["results"]:
# Extract content
title = page["properties"]["Name"]["title"][0]["plain_text"]
blocks = notion.blocks.children.list(page["id"])
content = extract_text_from_blocks(blocks)
kb.add_document(
title=title,
content=content,
category="notion",
tags=["notion", "synced"]
)
```
### Sync from Google Drive
```python theme={null}
from googleapiclient.discovery import build
def sync_google_drive(folder_id: str, kb: KnowledgeBase, creds):
"""Sync documents from Google Drive."""
service = build('drive', 'v3', credentials=creds)
results = service.files().list(
q=f"'{folder_id}' in parents",
fields="files(id, name, mimeType)"
).execute()
for file in results.get('files', []):
# Download and add to KB
content = download_file(service, file['id'])
kb.add_document(
title=file['name'],
content=content,
category="google-drive"
)
```
***
## Deployment
### Docker Compose
```yaml theme={null}
version: '3.8'
services:
api:
build: .
ports:
- "8000:8000"
volumes:
- ./data:/app/data
environment:
- MEMVID_FILE=/app/data/knowledge.mv2
web:
build: ./frontend
ports:
- "3000:3000"
depends_on:
- api
```
***
## Next Steps
<CardGroup>
<Card title="Chatbot with Memory" icon="comments" href="/examples/chatbot-memory">
Add conversational interface
</Card>
<Card title="Research Assistant" icon="flask" href="/examples/research-assistant">
Analyze research papers
</Card>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Answer questions from documents
</Card>
<Card title="LangChain Integration" icon="link" href="/frameworks/langchain">
Build advanced RAG pipelines
</Card>
</CardGroup>
# Memvid Agents & Packages
Source: https://docs.memvid.com/examples/packages
Official packages and tools built with Memvid to supercharge your AI workflows
Extend your AI capabilities with these official Memvid packages. Each is open-source, production-ready, and designed to solve real problems.
<CardGroup>
<Card title="commitreel" icon="clock-rotate-left" href="https://github.com/memvid/commitreel">
Time travel for checkpoints. Scrub a timeline and replay any moment.
</Card>
<Card title="memvid-mind" icon="brain" href="https://github.com/memvid/memvid-mind">
Give Claude Code photographic memory in one portable file
</Card>
<Card title="adrflow" icon="diagram-project" href="https://github.com/memvid/adrflow">
MCP server that captures architectural decisions while you code
</Card>
<Card title="canvas" icon="palette" href="https://github.com/memvid/memvid">
AI UI Kit powered by Memvid. Build AI apps in minutes.
</Card>
</CardGroup>
***
## Official Packages
### commitreel
<Info>
**Time travel for checkpoints**
</Info>
Record a single MV2 tape, scrub a timeline, and run any moment on demand. Perfect for debugging AI agent sessions and understanding how your agent evolved over time.
```bash theme={null}
npm install -g commitreel
```
**Features:**
* Record agent sessions as checkpoint tapes
* Visual timeline scrubbing interface
* Replay any moment instantly
* Debug complex multi-step agent workflows
<Card title="View on GitHub" icon="github" href="https://github.com/memvid/commitreel">
Apache-2.0 License | JavaScript
</Card>
***
### memvid-mind
<Info>
**Claude Code finally remembers. One file. Instant recall. Zero config.**
</Info>
Give Claude Code photographic memory in ONE portable file. No database, no SQLite, no ChromaDB. Just a single `.mv2` file you can git commit, scp, or share.
```bash theme={null}
# Install via Claude Code plugin marketplace
/plugin marketplace add memvid/memvid-mind
/plugin install memvid-mind
```
**Features:**
* Single-file memory persistence
* Git-friendly and portable
* Endless mode compresses tool outputs \~20x
* No database, no API keys, no cloud
<Card title="View on GitHub" icon="github" href="https://github.com/memvid/memvid-mind">
MIT License | JavaScript/TypeScript
</Card>
***
### adrflow
<Info>
**Capture architectural decisions while you code**
</Info>
An MCP server that captures architectural decisions while you code. Works with Claude Code, Cursor, Windsurf, and Codex. When you make decisions like "let's use Postgres" or "we should cache this", ADRFlow records them. Later you can search "why did we pick Postgres?" and get the full context back.
```bash theme={null}
npm install -g adrflow
```
**Features:**
* AI detects decisions automatically and offers to save them
* Full-text search across all decisions
* Export to MADR, Nygard, or minimal formats
* Works with Claude Code, Cursor, Windsurf, Codex, VS Code + Continue
<Card title="View on GitHub" icon="github" href="https://github.com/memvid/adrflow">
Apache-2.0 License | TypeScript
</Card>
***
### canvas
<Info>
**AI UI Kit powered by Memvid**
</Info>
Build AI-powered apps in minutes. A complete UI kit with React components, server utilities, and built-in memory management. Just configure and deploy.
**Features:**
* Pre-built React components for AI apps
* Server utilities for RAG pipelines
* Built-in Memvid memory management
* Deploy-ready templates
<Card title="View on GitHub" icon="github" href="https://github.com/memvid/memvid">
Apache-2.0 License | TypeScript
</Card>
***
## Community Packages
We're building an ecosystem of Memvid-powered tools. Here's what the community is creating:
| Package | Description | Author |
| ------------------- | ---------------------------------- | ------ |
| *Your package here* | Submit your Memvid-powered project | *You* |
***
## Submit Your Package
Built something with Memvid? We'd love to feature it!
<CardGroup>
<Card title="Submit Your Package" icon="plus" href="https://github.com/memvid/memvid/issues/new?template=package_submission.md">
Get your project featured in the official docs
</Card>
<Card title="Awesome Memvid" icon="star" href="https://github.com/memvid/awesome-memvid">
Community-curated list of Memvid projects
</Card>
</CardGroup>
### Submission Guidelines
To be featured, your package should:
<Steps>
<Step title="Use Memvid">
Built with `@memvid/sdk`, `memvid-sdk`, or the Memvid CLI
</Step>
<Step title="Be Open Source">
Available on GitHub with a clear license
</Step>
<Step title="Have Documentation">
README with installation and usage instructions
</Step>
<Step title="Solve a Real Problem">
Useful for AI developers building with Memvid
</Step>
</Steps>
# Research Assistant
Source: https://docs.memvid.com/examples/research-assistant
Build an AI assistant that analyzes and queries research papers
<Info>
**What you'll build:** An AI-powered research assistant that ingests papers, identifies themes, and answers complex questions across your research corpus.
**Time:** 30 minutes | **Difficulty:** Intermediate
</Info>
## Overview
Build a research assistant that:
* 📄 Ingests PDFs from arXiv, local files, and URLs
* 🔬 Extracts key findings and methodology
* 🔗 Identifies connections between papers
* ❓ Answers questions citing specific papers
* 📊 Generates literature reviews
***
## Quick Start
```python theme={null}
from memvid_sdk import use
import requests
import tempfile
from pathlib import Path
class ResearchAssistant:
def __init__(self, memory_path: str = "research.mv2"):
self.mem = use('llamaindex', memory_path, mode='auto')
def add_paper(self, title: str, pdf_path: str, metadata: dict = None):
"""Add a research paper to the corpus."""
self.mem.put({
"title": title,
"label": "paper",
"file": pdf_path,
"metadata": metadata or {}
})
print(f"✅ Added: {title}")
def add_arxiv(self, arxiv_id: str):
"""Add a paper from arXiv."""
# Download PDF
url = f"https://arxiv.org/pdf/{arxiv_id}.pdf"
response = requests.get(url)
with tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) as f:
f.write(response.content)
pdf_path = f.name
# Get metadata
meta_url = f"https://export.arxiv.org/api/query?id_list={arxiv_id}"
meta_response = requests.get(meta_url)
# Parse XML for title, authors, abstract...
self.add_paper(
title=f"arXiv:{arxiv_id}",
pdf_path=pdf_path,
metadata={"arxiv_id": arxiv_id, "source": "arxiv"}
)
def search(self, query: str, k: int = 10):
"""Search across all papers."""
return self.mem.find(query, k=k)
def ask(self, question: str):
"""Ask a question about the research corpus."""
return self.mem.ask(question, k=5)
def find_related(self, paper_title: str, k: int = 5):
"""Find papers related to a specific paper."""
return self.mem.find(f"related to {paper_title}", k=k)
# Usage
assistant = ResearchAssistant()
# Add papers
assistant.add_arxiv("2301.07041") # LLaMA paper
assistant.add_arxiv("2302.13971") # LLaMA 2 paper
assistant.add_arxiv("2303.08774") # GPT-4 Technical Report
# Ask questions
answer = assistant.ask(
"What are the key differences between LLaMA and GPT-4 architectures?"
)
print(answer["answer"])
# Find related work
related = assistant.find_related("LLaMA", k=5)
for hit in related.get("hits", []):
print(f"- {hit['title']} (score: {hit['score']:.2f})")
```
***
## Advanced Features
### Literature Review Generator
```python theme={null}
def generate_literature_review(self, topic: str, max_papers: int = 20) -> str:
"""Generate a literature review on a topic."""
# Find relevant papers
papers = self.search(topic, k=max_papers)
# Build context from papers
context = "\n\n".join([
f"Paper: {p.title}\nKey points: {p.snippet}"
for p in papers.hits
])
# Generate review using LLM
prompt = f"""Based on these research papers, write a comprehensive literature review on "{topic}":
{context}
Structure the review with:
1. Introduction and background
2. Key themes and findings
3. Methodological approaches
4. Gaps and future directions
5. Conclusion
Literature Review:"""
# Use the ask() method with context
review = self.mem.ask(prompt, k=max_papers)
return review.text
```
### Theme Extraction
```python theme={null}
def extract_themes(self, k: int = 100) -> list:
"""Extract main research themes from the corpus."""
# Get sample of papers
timeline = self.mem.timeline(limit=k)
# Cluster by topic (simplified)
themes = {}
for entry in timeline.entries:
# Extract keywords from title/text
keywords = extract_keywords(entry.text)
for kw in keywords:
themes[kw] = themes.get(kw, 0) + 1
# Sort by frequency
sorted_themes = sorted(themes.items(), key=lambda x: -x[1])
return sorted_themes[:20]
```
### Citation Network
```python theme={null}
def build_citation_network(self) -> dict:
"""Build a citation network from the corpus."""
network = {"nodes": [], "edges": []}
timeline = self.mem.timeline(limit=1000)
for entry in timeline.entries:
# Add node
network["nodes"].append({
"id": entry.frame_id,
"title": entry.title
})
# Find papers it might cite (based on similarity)
related = self.find_related(entry.title, k=5)
for r in related.hits:
if r.frame_id != entry.frame_id:
network["edges"].append({
"source": entry.frame_id,
"target": r.frame_id,
"weight": r.score
})
return network
```
***
## Web Interface
```python theme={null}
import streamlit as st
st.title("🔬 Research Assistant")
assistant = ResearchAssistant("research.mv2")
# Sidebar for adding papers
with st.sidebar:
st.header("Add Papers")
arxiv_id = st.text_input("arXiv ID")
if st.button("Add from arXiv"):
assistant.add_arxiv(arxiv_id)
st.success(f"Added {arxiv_id}")
uploaded = st.file_uploader("Upload PDF", type="pdf")
if uploaded:
# Save and add
with tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) as f:
f.write(uploaded.read())
assistant.add_paper(uploaded.name, f.name)
st.success(f"Added {uploaded.name}")
# Main area
tab1, tab2, tab3 = st.tabs(["Search", "Ask", "Literature Review"])
with tab1:
query = st.text_input("Search papers")
if query:
results = assistant.search(query)
for hit in results.get("hits", []):
st.markdown(f"**{hit['title']}** (score: {hit['score']:.2f})")
st.write(hit.get("snippet", ""))
st.divider()
with tab2:
question = st.text_area("Ask a research question")
if st.button("Ask"):
answer = assistant.ask(question)
st.markdown("### Answer")
st.write(answer.get("answer", ""))
st.markdown("### Sources")
for s in answer.get("sources", []):
st.write(f"- {s.get('title')}")
with tab3:
topic = st.text_input("Topic for literature review")
if st.button("Generate Review"):
with st.spinner("Generating..."):
review = assistant.generate_literature_review(topic)
st.markdown(review)
```
***
## Batch Import from arXiv
```python theme={null}
import arxiv
def batch_import_arxiv(query: str, max_results: int = 50):
"""Import papers from arXiv search."""
assistant = ResearchAssistant()
search = arxiv.Search(
query=query,
max_results=max_results,
sort_by=arxiv.SortCriterion.Relevance
)
for paper in search.results():
print(f"Downloading: {paper.title}")
# Download PDF
pdf_path = paper.download_pdf()
# Add to corpus
assistant.add_paper(
title=paper.title,
pdf_path=pdf_path,
metadata={
"arxiv_id": paper.entry_id,
"authors": [a.name for a in paper.authors],
"abstract": paper.summary,
"published": paper.published.isoformat(),
"categories": paper.categories
}
)
print(f"✅ Imported {max_results} papers")
# Import transformer papers
batch_import_arxiv("transformer attention mechanism", max_results=100)
```
***
## Next Steps
<CardGroup>
<Card title="Chatbot with Memory" icon="comments" href="/examples/chatbot-memory">
Add conversational interface
</Card>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Answer questions from documents
</Card>
<Card title="Knowledge Base" icon="book" href="/examples/knowledge-base">
Build a searchable wiki
</Card>
<Card title="LlamaIndex Integration" icon="link" href="/frameworks/llamaindex">
Advanced query engines
</Card>
</CardGroup>
# FAQ – General
Source: https://docs.memvid.com/faq/general
Frequently asked questions about Memvid
## General Questions
### What is Memvid?
Memvid is a portable AI memory system that packages your data, embeddings, and search indices into a single `.mv2` file. It's designed for building RAG applications, AI agents, and knowledge bases without the complexity of traditional vector databases.
### Is Memvid open source?
Yes, the core library (`memvid-core`) is open source. The Python SDK and Node.js SDK are available as packages with comprehensive documentation.
### What makes Memvid different from other vector databases?
Memvid's key differentiator is **single-file portability**. Unlike traditional vector databases that require servers and complex configurations, a `.mv2` file contains everything, your data, embeddings, indices, and metadata, in one portable file.
### What platforms does Memvid support?
Memvid supports:
* **macOS** (Intel and Apple Silicon)
* **Linux** (x86\_64 and ARM64)
* **Windows** (x86\_64)
***
## File Format
### Can I rely on a single `.mv2` file in production?
Yes. Memvid is designed for production use. The `.mv2` file is completely self-contained with no sidecar files, no external dependencies, and no hidden state. Copying the file transfers the entire memory, including the write-ahead log and all indices.
### How large can a `.mv2` file be?
File size depends on your capacity tier:
| Tier | Capacity | WAL Size |
| ---------- | --------- | -------- |
| Free | 1 GB | 4 MB |
| Developer | 25 GB | 16 MB |
| Enterprise | Unlimited | 64 MB |
The embedded WAL automatically scales with file size for optimal performance.
### Can multiple processes access the same file?
Yes, with some rules:
* **Multiple readers**: Allowed simultaneously
* **Single writer**: Only one writer at a time
* **Read-only mode**: Use `read_only=True` for concurrent read access
Writers use OS-level exclusive locks to prevent conflicts.
***
## Performance
### How fast is Memvid?
Memvid is built in Rust for maximum performance:
| Operation | Performance |
| ---------------------------- | ----------------- |
| Search (1K docs) | \< 1ms |
| Search (100K docs) | \< 10ms |
| Single doc ingestion | 1-10 docs/sec |
| Batch ingestion (`put_many`) | 500-1000 docs/sec |
| WAL append | \< 0.1ms |
### What search modes are available?
* **`Lexical (lex)`**: BM25 keyword search for exact matches
* **`Semantic (sem)`**: Vector search for conceptual similarity
* **`Hybrid (auto)`**: Combines both for best results (recommended)
### How do I optimize search performance?
1. **Build indices**: Ensure lexical and vector indices are enabled
2. **Use batch ingestion**: Use `put_many()` for 100-200x faster ingestion
3. **Enable parallel segments**: Use `--parallel-segments` for large datasets
4. **Choose the right mode**: Use `lex` for keywords, `sem` for concepts, `auto` for general queries
***
## SDKs and Integration
### Which programming languages are supported?
* **Python**: `pip install memvid-sdk`
* **Node.js**: `npm install @memvid/sdk`
* **Rust**: Use `memvid-core` crate directly
* **CLI**: `cargo install memvid-cli`
### Can I use Memvid with LangChain?
Yes! Both Python and Node.js SDKs support framework adapters:
**Python:**
```python theme={null}
from memvid_sdk import use
mem = use('langchain', 'knowledge.mv2')
tools = mem.tools # LangChain StructuredTool objects
```
**Node.js:**
```typescript theme={null}
import { use } from '@memvid/sdk';
const mv = await use('langchain', 'knowledge.mv2');
```
### What AI frameworks are supported?
**Python SDK:**
* LangChain
* LlamaIndex
* CrewAI
* AutoGen
* Haystack
**Node.js SDK:**
* Vercel AI SDK
* OpenAI Functions
* LangChain.js
* Semantic Kernel
***
## Capacity and Storage
### What happens when I exceed capacity?
You'll receive a `CapacityExceeded` error (MV001). Solutions:
1. Delete unused frames: `memvid delete knowledge.mv2 --frame-id <id>`
2. Vacuum to reclaim space: `memvid doctor knowledge.mv2 --vacuum`
3. Create a larger memory file with a higher tier
### How do I check my storage usage?
```bash theme={null}
memvid stats knowledge.mv2
```
This shows document count, size, capacity, and utilization percentage.
### Can I reduce storage size?
Yes, use vector compression:
```bash theme={null}
memvid put knowledge.mv2 --input docs/ --vector-compression
```
Vector compression provides 16x smaller vectors with minimal quality loss.
***
## Troubleshooting
### Why is my file locked?
Another process is using the file. Check for:
* Other terminals running `memvid` commands
* Running applications with open handles
* Stale processes (use `lsof your-file.mv2` to find them)
Use `memvid who your-file.mv2` to see who holds the lock.
### Why are my searches returning no results?
1. **Check indices**: Run `memvid stats your-file.mv2` to verify indices exist
2. **Try different modes**: Use `--mode lex` for keywords or `--mode sem` for concepts
3. **Rebuild indices**: Run `memvid doctor your-file.mv2 --rebuild-lex-index`
### How do I recover from corruption?
Use the `doctor` command:
```bash theme={null}
# Preview repairs
memvid doctor your-file.mv2 --plan-only
# Apply repairs
memvid doctor your-file.mv2 --rebuild-time-index --rebuild-lex-index
# Verify
memvid verify your-file.mv2 --deep
```
The embedded WAL ensures your data survives unexpected shutdowns.
### Why is ingestion slow?
Use batch ingestion for better performance:
```python theme={null}
# Instead of individual puts (1-10 docs/sec)
for doc in docs:
mem.put(text=doc['text'], title=doc['title'])
# Use put_many (500-1000 docs/sec)
mem.put_many(docs)
```
***
## Getting Help
### Where can I report bugs?
Report issues on GitHub: [github.com/memvid/memvid/issues](https://github.com/memvid/memvid/issues)
### Is there a community?
Yes! Join us on:
* **Discord**: [discord.gg/2mynS7fcK7](https://discord.gg/2mynS7fcK7)
* **Twitter**: [@memvid](https://x.com/memvidai)
* **GitHub Discussions**: [github.com/memvid/memvid/discussions](https://github.com/memvid/memvid/discussions)
# FAQ – Security & Compliance
Source: https://docs.memvid.com/faq/security-and-compliance
Security-focused questions about Memvid
## File Security
### How are `.mv2` files protected?
Integrity relies on cascading checksums:
* **Header checksum**: Validates file header
* **TOC checksum**: Validates table of contents
* **Per-segment checksums**: Validates each data segment
* **Time index checksum**: Validates timeline data
Confidentiality depends on OS file permissions. Memvid intentionally avoids bundling key management to keep the core simple.
### Are checksums validated automatically?
Yes. When opening a file, Memvid validates:
1. Header checksum
2. TOC integrity
3. WAL consistency
Deep verification (via `memvid verify --deep`) additionally checks all segment checksums.
### What happens if a file is corrupted?
Memvid provides tools to detect and repair corruption:
```bash theme={null}
# Detect issues
memvid verify knowledge.mv2 --deep
# Repair issues
memvid doctor knowledge.mv2 --rebuild-time-index --rebuild-lex-index
```
The embedded WAL protects against data loss from crashes or power failures.
***
## Crash Safety
### What ensures data survives crashes?
The embedded Write-Ahead Log (WAL):
1. All mutations are written to WAL first
2. WAL is synced to disk (fsync)
3. Changes are then applied to main data
4. On recovery, uncommitted WAL entries are replayed
### How long does recovery take?
Recovery is fast:
* Typical recovery: \< 100ms
* Large WAL replay (4MB): \< 250ms
### Are there any single points of failure?
No. The `.mv2` file is self-contained:
* No external databases
* No network dependencies
* No sidecar files that could be lost
***
## Access Control
### How does file locking work?
Memvid uses OS-level file locks:
* **Writers**: Exclusive lock (one at a time)
* **Readers**: Shared lock (multiple concurrent)
```bash theme={null}
# Check who holds the lock
memvid who knowledge.mv2
# Request release
memvid nudge knowledge.mv2
```
### Can multiple users access the same file?
Yes, but only one can write at a time:
```python theme={null}
# Reader (concurrent access OK)
mem = use('basic', 'knowledge.mv2', read_only=True)
# Writer (exclusive access)
mem = use('basic', 'knowledge.mv2')
```
***
## Data Privacy
### Is my data sent anywhere?
**Local operations** (search, timeline, stats) never send data anywhere.
**Ask operations** with external LLMs (`openai`, `claude`, `gemini`) send context to those providers. To prevent this:
1. Use the local model (tinyllama):
```bash theme={null}
memvid ask knowledge.mv2 --question "What is X?"
```
2. Use context-only mode:
```bash theme={null}
memvid ask knowledge.mv2 --question "What is X?" --context-only
```
3. Enable PII masking:
```bash theme={null}
memvid ask knowledge.mv2 --question "Contact info?" --mask-pii --use-model openai
```
### What does PII masking protect?
The `--mask-pii` flag masks sensitive information before sending to external LLMs:
| PII Type | Example | Masked As |
| -------------------------- | --------------------- | --------------- |
| Email addresses | `john@example.com` | `[EMAIL]` |
| Phone numbers | `555-123-4567` | `[PHONE]` |
| US Social Security Numbers | `123-45-6789` | `[SSN]` |
| Credit card numbers | `4111-1111-1111-1111` | `[CREDIT_CARD]` |
| IPv4 addresses | `192.168.1.1` | `[IP_ADDRESS]` |
| API keys/tokens | `sk-abc123...` | `[API_KEY]` |
### Using PII Masking
**CLI:**
```bash theme={null}
memvid ask knowledge.mv2 --question "Contact info?" --mask-pii --use-model openai
```
**Python SDK:**
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'knowledge.mv2')
# Enable PII masking for ask queries
answer = mem.ask(
"What are the customer contact details?",
model="openai:gpt-4o",
mask_pii=True
)
print(answer['answer'])
# Standalone PII masking function
from memvid_sdk import mask_pii
text = "Contact john@example.com or call 555-123-4567"
masked = mask_pii(text)
# Output: "Contact [EMAIL] or call [PHONE]"
```
**Node.js SDK:**
```typescript theme={null}
import { use, maskPii } from '@memvid/sdk';
const mem = await use('basic', 'knowledge.mv2');
// Enable PII masking for ask queries
const answer = await mem.ask('What are the customer contact details?', {
model: 'openai:gpt-4o',
modelApiKey: process.env.OPENAI_API_KEY,
maskPii: true
});
console.log(answer.answer);
// Standalone PII masking function
const text = 'Contact john@example.com or call 555-123-4567';
const masked = maskPii(text);
// Output: "Contact [EMAIL] or call [PHONE]"
```
<Note>
PII masking is applied to the context sent to external LLMs, not to data stored in the memory file. The original data remains intact.
</Note>
***
## Verification
### How do I verify file integrity?
```bash theme={null}
# Basic verification
memvid verify knowledge.mv2
# Deep verification (all checksums)
memvid verify knowledge.mv2 --deep
# Single-file compliance (no sidecars)
memvid verify-single-file knowledge.mv2
```
### What does deep verification check?
| Check | Description |
| ----------------------- | -------------------------- |
| `HeaderChecksum` | Header integrity |
| `TocIntegrity` | Table of contents valid |
| `WalConsistency` | WAL state consistent |
| `TimeIndexSortOrder` | Time index properly sorted |
| `LexIndexDecode` | Lexical index readable |
| `VecIndexDecode` | Vector index readable |
| `FrameCountConsistency` | Frame counts match |
***
## Best Practices
### File Storage
1. **Use appropriate permissions**: Restrict file access to authorized users
2. **Regular backups**: Copy `.mv2` files to backup storage
3. **Verify after transfer**: Run `memvid verify --deep` after copying files
### Production Use
1. **Read-only mode**: Use for query-only workloads
2. **Monitor capacity**: Check utilization before large ingestions
3. **Periodic verification**: Run `memvid verify --deep` weekly
### Sensitive Data
1. **PII masking**: Always enable for external LLM calls
2. **Local models**: Use tinyllama for sensitive queries
3. **Context-only mode**: Get relevant docs without LLM synthesis
# File layout
Source: https://docs.memvid.com/file-format/layout
Tour the bytes that form a `.mv2` file
The kernel spec defines `.mv2` as a contiguous layout:
| Region | Size | Notes |
| ---------------- | ---------------- | ---------------------------------------------------------------------------------- |
| Header | 4096 bytes | Contains magic, spec version (`SPEC_MAJOR=2`, `SPEC_MINOR=1`), WAL offsets |
| Embedded WAL | 1–64 MB | Append-only log written before every commit (see [WAL protocol](/file-format/wal)) |
| Segments | Variable | Frames, blobs, text chunks, vector chunks stored sequentially |
| Index manifests | Variable | `IndexManifests` struct in the TOC references each index |
| Time Index Track | Variable | Optional `TimeIndexManifest` (offset + length + checksum) |
| TOC | Variable | Serialized via `bincode` with cascading checksums |
| Footer | 48 bytes + magic | `CommitFooter` storing `toc_len`, `toc_hash`, and `generation` |
`CommitFooter::find_last_valid_footer` scans from the end of the file to locate the final valid footer, ensuring crash recovery always loads a consistent TOC.
> **Portability** - There are no `.shm`, `.wal`, or `.lock` siblings; locking uses in-file byte ranges via `FileLock`, and verification ensures single-file compliance.
# Time Index Track
Source: https://docs.memvid.com/file-format/time-index-track
v2.1's timeline fast-path
The Time Index Track adds deterministic timestamps to the file:
* Defined in `memvid-core/src/io/time_index.rs`
* Uses `TIME_INDEX_MAGIC = MVTI` header followed by entry count and `(timestamp, frame_id)` tuples
* `Memvid::commit()` sorts buffered entries, writes them immediately before the TOC, calculates A BLAKE3 checksum, and updates `Toc.time_index`
* `Stats.has_time_index` surfaces to CLI and SDKs to show readiness
### Query behavior
1. CLI/SDK builds a `TimelineQuery` specifying `start_ts`, `end_ts`, and optional limit
2. Core memory-maps the track and binary-searches for the first entry ≥ `start_ts`
3. It iterates sequentially until `end_ts` or limit is reached, returning `TimelineEntry` structs referencing the canonical frames
If the track is absent, the kernel falls back to scanning frames inside the TOC, which is slower but preserves correctness.
### Repair commands
`memvid doctor --file demo.mv2 --rebuild-time-index` rebuilds the track from frame metadata, as mandated by the Golden Test Pack (CLI-009). The report lists action plans, severity, and status codes defined in `DoctorReport`/`DoctorActionDetail`.
> **Testing** - The crash harness and determinism suite must validate timeline queries after every recovery scenario to ensure the track was correctly replayed from the WAL.
# Embedded WAL
Source: https://docs.memvid.com/file-format/wal
Durability guarantees and sizing rules
Memvid embeds its write-ahead log inside the `.mv2` file. The Security & Performance Architecture spec defines the sizing tiers:
| File size | WAL region |
| --------- | ---------- |
| \<100 MB | 1 MB |
| \<1 GB | 4 MB |
| \<10 GB | 16 MB |
| ≥10 GB | 64 MB |
### Commit protocol
1. Serialize the new TOC (after applying pending time entries)
2. Write the TOC into the WAL region and fsync
3. Append the TOC to the end of the file and fsync
4. Update and fsync the header/footer
5. Truncate/zero the WAL region
`EmbeddedWal` (defined in `memvid-core/src/io/wal.rs`) tracks stats such as bytes written and number of checkpoints. Every clean shutdown, seal(), WAL reaching 75 % capacity, or every 1 000 transactions triggers a checkpoint.
### Recovery flow
`Memvid::open()` reads the header, replays WAL entries if present, and validates the TOC checksum before exposing the file. Recovery time targets from the performance spec: `< 250 ms` even when replaying WAL + TOC.
> **Developer tip** - Use `memvid doctor --vacuum` (if exposed) or `memvid verify --deep` to confirm the WAL pointer reset; the Golden Test Pack ensures no phantom WAL bytes survive crashes.
# API Integration Patterns
Source: https://docs.memvid.com/frameworks/api-integration-patterns
Production patterns for integrating Memvid through api.memvid.com
Use this guide when you are integrating Memvid through HTTP in platforms like n8n, Replit, Lovable, or v0.
<Info>
This page is the shared contract for API-first integrations. Platform pages should reuse these request shapes and reliability patterns.
</Info>
## Base Setup
* Base URL: `https://api.memvid.com`
* Auth header (recommended): `Authorization: Bearer mv2_YOUR_API_KEY`
* Alternative auth header: `X-API-Key: mv2_YOUR_API_KEY`
## Golden Path (Minimal Production Flow)
1. Create or select a memory ID
2. Ingest documents (JSON text, file, or URL)
3. Use `find` for retrieval UX
4. Use `ask` for grounded synthesis
5. Return answer with sources in your app UI
## Canonical Request Shapes
### Create Memory
```http theme={null}
POST /v1/memories
Content-Type: application/json
{
"name": "Support KB",
"description": "Runbooks and policies"
}
```
### Add Documents (JSON)
```http theme={null}
POST /v1/memories/:id/documents
Content-Type: application/json
{
"documents": [
{
"title": "P1 Playbook",
"text": "Acknowledge P1 in 10 minutes.",
"tags": ["incident", "support"]
}
]
}
```
### Find
```http theme={null}
POST /v1/memories/:id/find
Content-Type: application/json
{
"query": "How quickly do we acknowledge P1 incidents?",
"topK": 5
}
```
### Ask
```http theme={null}
POST /v1/memories/:id/ask
Content-Type: application/json
{
"question": "What is our P1 SLA?",
"options": {
"model": "gpt-4o-mini",
"includeSources": true,
"maxContextChunks": 10
}
}
```
## Typed Client Wrapper (TypeScript)
```typescript theme={null}
type MemvidRequestInit = Omit<RequestInit, "headers"> & {
headers?: Record<string, string>;
timeoutMs?: number;
retries?: number;
};
class MemvidHttpError extends Error {
constructor(
public status: number,
public body: string,
public method: string,
public path: string
) {
super(`Memvid ${method} ${path} failed (${status})`);
}
}
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
export async function memvidRequest<T = unknown>(
path: string,
init: MemvidRequestInit = {}
): Promise<T> {
const baseUrl = process.env.MEMVID_API_BASE || "https://api.memvid.com";
const apiKey = process.env.MEMVID_API_KEY;
if (!apiKey) throw new Error("Missing MEMVID_API_KEY");
const retries = init.retries ?? 2;
const timeoutMs = init.timeoutMs ?? 15000;
const method = init.method || "GET";
for (let attempt = 0; attempt <= retries; attempt++) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const res = await fetch(`${baseUrl}${path}`, {
...init,
signal: controller.signal,
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
...(init.headers || {}),
},
});
if (res.status === 204) return null as T;
const text = await res.text();
if (!res.ok) {
const retryable = res.status === 429 || res.status >= 500;
if (retryable && attempt < retries) {
await sleep(250 * Math.pow(2, attempt));
continue;
}
throw new MemvidHttpError(res.status, text, method, path);
}
return text ? (JSON.parse(text) as T) : (null as T);
} catch (err) {
const retryable = err instanceof Error && err.name === "AbortError";
if (!retryable || attempt >= retries) throw err;
await sleep(250 * Math.pow(2, attempt));
} finally {
clearTimeout(timeout);
}
}
throw new Error("Unexpected retry loop exit");
}
```
## Async Ingestion and Job Polling
Some uploads are processed asynchronously via a background worker. This happens automatically when:
* The file is larger than **2 MB**
* A PDF has more than **5 scanned/image-only pages** (triggers OCR in the background)
* You set `options.async: true` explicitly
When async processing kicks in, the response returns **HTTP 202** instead of 200, with a `jobId` and `pollUrl`:
```json theme={null}
{
"added": 0,
"chunksCreated": 0,
"jobId": "abc123...",
"pollUrl": "/v1/jobs/abc123..."
}
```
### Polling for completion
```typescript theme={null}
type JobStatus = "pending" | "processing" | "completed" | "failed" | "partial";
interface JobResult {
status: JobStatus;
progress: number; // 0-100
message?: string; // e.g. "Generating embeddings..."
result?: {
documentsAdded: number;
chunksCreated: number;
documentIds: string[];
totalBytes: number;
processingMs: number;
};
error?: string;
}
export async function waitForJob(jobId: string, timeoutMs = 600000) {
const started = Date.now();
while (Date.now() - started < timeoutMs) {
const job = await memvidRequest<JobResult>(`/v1/jobs/${jobId}`);
if (job.status === "completed" || job.status === "partial") return job;
if (job.status === "failed") throw new Error(job.error || "Memvid job failed");
await sleep(5000); // Poll every 5 seconds
}
throw new Error(`Timed out waiting for job ${jobId}`);
}
```
<Tip>
Scanned PDFs with many pages can take 2–8 minutes to process via OCR. Use a generous timeout (5–10 minutes) and poll every 5–10 seconds.
</Tip>
### Handling both sync and async responses
```typescript theme={null}
const res = await fetch(`${baseUrl}/v1/memories/${memoryId}/documents`, {
method: "POST",
headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
body: JSON.stringify({ documents: [...] }),
});
if (res.status === 200) {
// Sync — documents are ready immediately
const data = await res.json();
console.log(`Added ${data.added} documents (${data.chunksCreated} chunks)`);
} else if (res.status === 202) {
// Async — poll for completion
const data = await res.json();
const result = await waitForJob(data.jobId);
console.log(`Job done: ${result.result?.chunksCreated} chunks created`);
}
```
## Reliability Checklist
* Keep API keys server-side only.
* Use memory-scoped keys for least privilege.
* Retry `429` and `5xx` with backoff.
* Enforce request timeouts to prevent hanging workers.
* Log method, path, status, and request IDs for debugging.
* Show source snippets in UI for grounded trust.
## 5-Minute Smoke Test
Run these calls in order and verify non-empty responses:
1. `POST /v1/memories` -> get `memory.id`
2. `POST /v1/memories/:id/documents` -> ingest sample text
3. `POST /v1/memories/:id/find` -> expect at least one hit
4. `POST /v1/memories/:id/ask` -> expect `answer/text` and optional `sources`
## Next Pages
* [n8n](/frameworks/n8n)
* [Replit](/frameworks/replit)
* [Lovable](/frameworks/lovable)
* [v0](/frameworks/v0)
* [REST API Reference](/api-reference/rest-api)
# AutoGen
Source: https://docs.memvid.com/frameworks/autogen
Use Memvid with Microsoft AutoGen for multi-agent conversations
Integrate Memvid with Microsoft AutoGen to build multi-agent systems with persistent knowledge retrieval. The `autogen` adapter provides function schemas compatible with OpenAI's function calling API.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk openai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with AutoGen adapter
const mem = await use('autogen', 'knowledge.mv2');
// Access function schemas (OpenAI-compatible)
const functions = mem.functions;
```
<Note>
Node.js uses OpenAI function calling directly. The AutoGen framework is Python-only,
but the function schemas work with any OpenAI-compatible client.
</Note>
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk pyautogen
```
## Quick Start
```python theme={null}
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('autogen', 'knowledge.mv2')
else:
mem = create('knowledge.mv2', kind='autogen')
# Access AutoGen tools
tools = mem.tools # Returns AutoGen function definitions
```
</Tab>
</Tabs>
## Available Functions
The AutoGen adapter provides three functions:
| Function | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Python: Basic Usage with AutoGen
```python theme={null}
from autogen_agentchat.agents import AssistantAgent, UserProxyAgent
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('autogen', 'knowledge.mv2', read_only=True)
else:
mem = create('knowledge.mv2', kind='autogen')
# Get the search tool (find tool is at index 1)
find_tool = mem.tools[1]
# Create a wrapper function for the search
def search_knowledge(query: str) -> str:
results = mem.find(query, k=5)
return "\n".join([f"- {r.get('title')}: {r.get('text', '')[:200]}" for r in results])
# Create assistant with Memvid knowledge
assistant = AssistantAgent(
name="assistant",
llm_config={
"model": "gpt-4o",
"functions": [find_tool.schema]
},
system_message="You have access to a knowledge base. Use search_knowledge to find information."
)
# Register the function
assistant.register_function(
function_map={find_tool.name: search_knowledge}
)
# Start conversation
user_proxy = UserProxyAgent(name="user", human_input_mode="NEVER")
user_proxy.initiate_chat(
assistant,
message="Find information about authentication and summarize it"
)
```
## Node.js: Function Calling Loop
```typescript theme={null}
import { use } from '@memvid/sdk';
import OpenAI from 'openai';
// Get Memvid functions
const mem = await use('autogen', 'knowledge.mv2');
const functions = mem.functions;
const client = new OpenAI();
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: 'system', content: 'You are a helpful research assistant with access to a knowledge base.' },
{ role: 'user', content: 'Find information about authentication and summarize it.' },
];
// Execute function by name
async function executeFunction(name: string, args: any): Promise<string> {
if (name === 'memvid_find') {
const result = await mem.find(args.query, { k: args.top_k || 5 });
return JSON.stringify(result.hits?.map((h: any) => ({
title: h.title,
snippet: h.snippet || h.text?.slice(0, 200),
score: h.score,
})));
} else if (name === 'memvid_ask') {
const result = await mem.ask(args.question, { mode: args.mode || 'auto' });
return result.answer || 'No answer generated';
} else if (name === 'memvid_put') {
const frameId = await mem.put({
title: args.title,
label: args.label,
text: args.text,
});
return `Document stored with frame_id: ${frameId}`;
}
return 'Unknown function';
}
// Conversation loop
while (true) {
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages,
tools: functions.map((f: any) => ({ type: 'function' as const, function: f })),
tool_choice: 'auto',
});
const message = response.choices[0].message;
messages.push(message);
if (message.tool_calls) {
for (const toolCall of message.tool_calls) {
const funcName = toolCall.function.name;
const funcArgs = JSON.parse(toolCall.function.arguments);
const result = await executeFunction(funcName, funcArgs);
messages.push({
role: 'tool',
tool_call_id: toolCall.id,
content: result,
});
}
} else {
console.log('Assistant:', message.content);
break;
}
}
await mem.seal();
```
## Python: Multi-Agent Setup
```python theme={null}
from autogen_agentchat.agents import AssistantAgent, UserProxyAgent
from autogen_agentchat import GroupChat, GroupChatManager
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('autogen', 'knowledge.mv2', read_only=True)
else:
mem = create('knowledge.mv2', kind='autogen')
# Get the find tool (index 1)
find_tool = mem.tools[1]
# Create a wrapper function
def search_knowledge(query: str) -> str:
results = mem.find(query, k=5)
return "\n".join([f"- {r.get('title')}: {r.get('text', '')[:200]}" for r in results])
# User proxy
user_proxy = UserProxyAgent(name="user", human_input_mode="NEVER")
# Create researcher agent with knowledge access
researcher = AssistantAgent(
name="researcher",
llm_config={
"model": "gpt-4o",
"functions": [find_tool.schema]
},
system_message="You research topics using the knowledge base."
)
researcher.register_function(function_map={find_tool.name: search_knowledge})
# Create writer agent
writer = AssistantAgent(
name="writer",
llm_config={"model": "gpt-4o"},
system_message="You write summaries based on research findings."
)
# Create group chat
group_chat = GroupChat(
agents=[user_proxy, researcher, writer],
messages=[],
max_round=10
)
manager = GroupChatManager(groupchat=group_chat, llm_config={"model": "gpt-4o"})
# Start group conversation
user_proxy.initiate_chat(
manager,
message="Research deployment best practices and write a summary"
)
```
## Custom Search Functions (Python)
```python theme={null}
from memvid_sdk import use
mem = use('autogen', 'knowledge.mv2', read_only=True)
# Create custom search function with specific options
def search_docs(query: str, limit: int = 5) -> str:
"""Search documentation with custom parameters."""
results = mem.find(query, k=limit, scope='mv2://docs/')
return "\n".join([f"- {r.title}: {r.snippet}" for r in results])
def search_recent(query: str) -> str:
"""Search recent entries only."""
results = mem.find(query, k=5)
# Filter by recency using timeline
recent_ids = {e.frame_id for e in mem.timeline(limit=100)}
return "\n".join([
f"- {r.title}: {r.snippet}"
for r in results
if r.frame_id in recent_ids
])
```
## Best Practices
1. **Use read-only mode** for retrieval agents
2. **Register tools** before starting conversations
3. **Set appropriate limits** on search results
4. **Close the memory** when done
```python theme={null}
mem = use('autogen', 'knowledge.mv2', read_only=True)
try:
# Create agents and run conversation
pass
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="CrewAI" icon="users" href="/frameworks/crewai">
CrewAI integration
</Card>
<Card title="OpenAI SDK" icon="robot" href="/frameworks/openai">
Direct OpenAI function calling
</Card>
</CardGroup>
# CrewAI
Source: https://docs.memvid.com/frameworks/crewai
Build AI crews with Memvid-powered knowledge retrieval
Integrate Memvid with CrewAI to build collaborative AI crews with shared knowledge access. The `crewai` adapter provides function schemas compatible with OpenAI's function calling API.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk openai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with CrewAI adapter
const mem = await use('crewai', 'knowledge.mv2');
// Access function schemas (OpenAI-compatible)
const functions = mem.functions;
```
<Note>
Node.js uses OpenAI function calling directly. CrewAI is Python-only,
but the function schemas work with any OpenAI-compatible client for
building similar multi-agent workflows.
</Note>
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk crewai crewai-tools
```
## Quick Start
```python theme={null}
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('crewai', 'knowledge.mv2')
else:
mem = create('knowledge.mv2', kind='crewai')
# Access CrewAI tools
tools = mem.tools # Returns CrewAI Tool objects
```
</Tab>
</Tabs>
## Available Functions
The CrewAI adapter provides three functions:
| Function | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Python: Basic Usage with CrewAI
```python theme={null}
from crewai import Agent, Task, Crew
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('crewai', 'knowledge.mv2', read_only=True)
else:
mem = create('knowledge.mv2', kind='crewai')
# Get the search tool (find tool is at index 1)
find_tool = mem.tools[1]
# Create agent with Memvid tool
researcher = Agent(
role="Research Analyst",
goal="Find relevant information from the knowledge base",
backstory="Expert at finding and analyzing information",
tools=[find_tool],
verbose=True
)
# Define task
research_task = Task(
description="Research the main features and create a summary",
agent=researcher,
expected_output="A summary of the main features"
)
# Create crew
crew = Crew(
agents=[researcher],
tasks=[research_task]
)
# Execute
result = crew.kickoff()
print(result)
```
## Node.js: Multi-Agent Workflow
```typescript theme={null}
import { use } from '@memvid/sdk';
import OpenAI from 'openai';
// Get Memvid functions
const mem = await use('crewai', 'knowledge.mv2');
const functions = mem.functions;
const client = new OpenAI();
// Define agent personas
const agents = {
researcher: {
role: 'Research Analyst',
goal: 'Find relevant information from the knowledge base',
systemPrompt: 'You are a research analyst. Use the memvid_find tool to search the knowledge base.',
},
writer: {
role: 'Technical Writer',
goal: 'Write clear documentation based on research',
systemPrompt: 'You are a technical writer. Summarize the research findings clearly.',
},
};
// Execute function by name
async function executeFunction(name: string, args: any): Promise<string> {
if (name === 'memvid_find') {
const result = await mem.find(args.query, { k: args.top_k || 5 });
return JSON.stringify(result.hits?.map((h: any) => ({
title: h.title,
snippet: h.snippet || h.text?.slice(0, 200),
})));
} else if (name === 'memvid_ask') {
const result = await mem.ask(args.question, { mode: args.mode || 'auto' });
return result.answer || 'No answer generated';
}
return 'Unknown function';
}
// Run agent with tools
async function runAgent(agent: typeof agents.researcher, task: string): Promise<string> {
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: 'system', content: agent.systemPrompt },
{ role: 'user', content: task },
];
while (true) {
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages,
tools: functions.map((f: any) => ({ type: 'function' as const, function: f })),
tool_choice: 'auto',
});
const message = response.choices[0].message;
messages.push(message);
if (message.tool_calls) {
for (const toolCall of message.tool_calls) {
const result = await executeFunction(
toolCall.function.name,
JSON.parse(toolCall.function.arguments)
);
messages.push({ role: 'tool', tool_call_id: toolCall.id, content: result });
}
} else {
return message.content || '';
}
}
}
// Execute crew workflow
async function runCrew() {
// Step 1: Research
console.log('🔍 Researcher working...');
const research = await runAgent(
agents.researcher,
'Research authentication mechanisms in the knowledge base'
);
console.log('Research findings:', research);
// Step 2: Writing (uses research context)
console.log('✍️ Writer working...');
const documentation = await runAgent(
agents.writer,
`Based on this research, write a clear guide:\n\n${research}`
);
console.log('Final documentation:', documentation);
}
await runCrew();
await mem.seal();
```
## Python: Multi-Agent Crew
```python theme={null}
from crewai import Agent, Task, Crew, Process
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('crewai', 'knowledge.mv2', read_only=True)
else:
mem = create('knowledge.mv2', kind='crewai')
# Get the find tool (index 1)
find_tool = mem.tools[1]
# Create multiple agents
researcher = Agent(
role="Researcher",
goal="Find comprehensive information from the knowledge base",
backstory="Skilled at finding relevant information quickly",
tools=[find_tool],
verbose=True
)
writer = Agent(
role="Technical Writer",
goal="Write clear documentation based on research",
backstory="Expert at translating technical content into readable docs",
verbose=True
)
reviewer = Agent(
role="Editor",
goal="Review and improve the documentation for clarity",
backstory="Meticulous editor with attention to detail",
verbose=True
)
# Define tasks
research_task = Task(
description="Research authentication mechanisms in the knowledge base",
agent=researcher,
expected_output="Detailed findings about authentication"
)
writing_task = Task(
description="Write a guide based on the research findings",
agent=writer,
expected_output="A clear authentication guide",
context=[research_task] # Depends on research
)
review_task = Task(
description="Review and polish the guide for publication",
agent=reviewer,
expected_output="Final polished guide",
context=[writing_task] # Depends on writing
)
# Create crew with sequential process
crew = Crew(
agents=[researcher, writer, reviewer],
tasks=[research_task, writing_task, review_task],
process=Process.sequential,
verbose=True
)
result = crew.kickoff()
print(result)
```
## Custom Tools (Python)
```python theme={null}
from crewai.tools import tool
from memvid_sdk import use
mem = use('crewai', 'knowledge.mv2', read_only=True)
@tool("Search Knowledge Base")
def search_knowledge(query: str) -> str:
"""Search the knowledge base for relevant information."""
results = mem.find(query, k=5)
return "\n".join([f"- {r.title}: {r.snippet}" for r in results])
@tool("Get Recent Entries")
def get_recent(limit: int = 10) -> str:
"""Get the most recent entries from the knowledge base."""
entries = mem.timeline(limit=limit)
return "\n".join([f"- [{e.timestamp}] {e.title}" for e in entries])
@tool("Ask Question")
def ask_question(question: str) -> str:
"""Ask a question and get an AI-synthesized answer."""
answer = mem.ask(question)
return str(answer.get("answer", ""))
```
## Best Practices
1. **Use read-only mode** for retrieval crews
2. **Share tools** across agents that need knowledge access
3. **Use task context** to pass information between tasks
4. **Close the memory** when done
```python theme={null}
mem = use('crewai', 'knowledge.mv2', read_only=True)
try:
# Create and run crew
crew = Crew(agents=[...], tasks=[...])
result = crew.kickoff()
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="AutoGen" icon="robot" href="/frameworks/autogen">
AutoGen integration
</Card>
<Card title="OpenAI SDK" icon="brain" href="/frameworks/openai">
Direct OpenAI function calling
</Card>
</CardGroup>
# Google ADK
Source: https://docs.memvid.com/frameworks/google-adk
Integrate Memvid with Google Agent Development Kit
Integrate Memvid with Google's Agent Development Kit (ADK) to build Gemini-powered agents with persistent memory. The `google-adk` adapter provides native ADK function declarations.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk @google/generative-ai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with Google ADK adapter
const mem = await use('google-adk', 'knowledge.mv2');
// Access ADK function declarations
const tools = mem.tools; // FunctionDeclaration[] for Gemini API
const executors = mem.functions; // Function executors by name
```
## Available Functions
The Google ADK adapter provides three function declarations:
| Function | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Basic Usage with Gemini
```typescript theme={null}
import { use } from '@memvid/sdk';
import { GoogleGenerativeAI } from '@google/generative-ai';
// Initialize Memvid with Google ADK adapter
const mem = await use('google-adk', 'knowledge.mv2');
// Get function declarations and executors
const tools = mem.tools as any[];
const executors = mem.functions as Record<string, (args: any) => Promise<string>>;
// Create Gemini client
const geminiKey = process.env.GEMINI_API_KEY ?? process.env.GOOGLE_API_KEY;
if (!geminiKey) throw new Error("Set GEMINI_API_KEY (or legacy GOOGLE_API_KEY)");
const genAI = new GoogleGenerativeAI(geminiKey);
// Create model with Memvid tools
const model = genAI.getGenerativeModel({
model: 'gemini-2.0-flash',
tools: [{ functionDeclarations: tools }],
});
// Start a chat
const chat = model.startChat();
const result = await chat.sendMessage('Search for authentication information');
// Handle function calls
const response = result.response;
const parts = response.candidates?.[0]?.content?.parts || [];
for (const part of parts) {
if (part.functionCall) {
const { name, args } = part.functionCall;
console.log(`Function call: ${name}`);
// Execute the function
if (executors[name]) {
const funcResult = await executors[name](args as Record<string, unknown>);
console.log(`Result: ${funcResult}`);
// Send result back to model
const followUp = await chat.sendMessage([{
functionResponse: {
name,
response: { result: funcResult },
},
}]);
console.log(`Model response: ${followUp.response.text()}`);
}
} else if (part.text) {
console.log(`Response: ${part.text}`);
}
}
```
## Direct Tool Execution
Use the function executors directly without Gemini:
```typescript theme={null}
import { use } from '@memvid/sdk';
const mem = await use('google-adk', 'knowledge.mv2', { mode: 'create' });
const executors = mem.functions as Record<string, (args: any) => Promise<string>>;
// Store documents
const putResult = await executors.memvid_put({
title: 'API Documentation',
label: 'docs',
text: 'Authentication uses JWT tokens with refresh capability.',
});
console.log(putResult);
// Output: Document stored with frame_id: 2
// Search documents
const findResult = await executors.memvid_find({
query: 'authentication',
top_k: 5,
});
console.log(findResult);
// Output: Found 1 results:
// 1. [API Documentation] (score: 2.34): Authentication uses JWT tokens...
// Ask questions
const askResult = await executors.memvid_ask({
question: 'How does authentication work?',
mode: 'auto',
});
console.log(askResult);
// Output: Answer: Authentication uses JWT tokens with refresh capability.
// Sources: API Documentation
```
## Complete Agentic Example
```typescript theme={null}
import { use } from '@memvid/sdk';
import { GoogleGenerativeAI } from '@google/generative-ai';
async function runGeminiAgent() {
// Initialize
const mem = await use('google-adk', 'knowledge.mv2');
const tools = mem.tools as any[];
const executors = mem.functions as Record<string, (args: any) => Promise<string>>;
// Store some knowledge first
await executors.memvid_put({
title: 'Gemini Overview',
label: 'google-ai',
text: 'Gemini is Google\\'s most capable AI model family.',
});
await executors.memvid_put({
title: 'Agent Development Kit',
label: 'frameworks',
text: 'ADK is Google\\'s framework for building AI agents.',
});
// Create Gemini client
const geminiKey = process.env.GEMINI_API_KEY ?? process.env.GOOGLE_API_KEY;
if (!geminiKey) throw new Error("Set GEMINI_API_KEY (or legacy GOOGLE_API_KEY)");
const genAI = new GoogleGenerativeAI(geminiKey);
const model = genAI.getGenerativeModel({
model: 'gemini-2.0-flash',
tools: [{ functionDeclarations: tools }],
systemInstruction: 'You are a helpful assistant with access to a knowledge base. ' +
'Use memvid_find to search and memvid_ask to answer questions.',
});
// Run agentic loop
const chat = model.startChat();
async function processMessage(userMessage: string): Promise<string> {
let result = await chat.sendMessage(userMessage);
// Handle function calls iteratively
while (true) {
const parts = result.response.candidates?.[0]?.content?.parts || [];
const functionCalls = parts.filter((p: any) => p.functionCall);
if (functionCalls.length === 0) {
// No more function calls, return text response
return result.response.text() || 'No response';
}
// Execute all function calls
const responses: any[] = [];
for (const part of functionCalls) {
const { name, args } = (part as any).functionCall;
if (executors[name]) {
const funcResult = await executors[name](args);
responses.push({
functionResponse: { name, response: { result: funcResult } },
});
}
}
// Send results back
result = await chat.sendMessage(responses);
}
}
// Example conversation
const answer = await processMessage(
'What is Google ADK and how does it relate to Gemini?'
);
console.log('Agent response:', answer);
await mem.seal();
}
runGeminiAgent().catch(console.error);
```
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk google-genai
```
## Quick Start
```python theme={null}
from memvid_sdk import use
# Open with Google ADK adapter
mem = use('google-adk', 'knowledge.mv2')
# Access ADK tools
tools = mem.tools # Returns ADK tool definitions
```
## Basic Usage
```python theme={null}
from google import genai
from google.genai import types
from memvid_sdk import use
# Initialize with google-adk adapter
mem = use('google-adk', 'knowledge.mv2', read_only=True)
# Get tool definitions
tools = mem.tools
# Create Gemini client
client = genai.Client()
# Create chat with tools
chat = client.chats.create(
model="gemini-2.0-flash",
config=types.GenerateContentConfig(
tools=tools,
system_instruction="You are a helpful assistant with access to a knowledge base."
)
)
# Send message
response = chat.send_message("What are the best practices for deployment?")
print(response.text)
```
## Function Calling
```python theme={null}
from google import genai
from memvid_sdk import use
mem = use('google-adk', 'knowledge.mv2', read_only=True)
# Define tool functions
def memvid_search(query: str) -> str:
"""Search the knowledge base for relevant information."""
results = mem.find(query, k=5)
return "\n".join([f"- {r.title}: {r.snippet}" for r in results])
def memvid_ask(question: str) -> str:
"""Ask a question and get an AI-synthesized answer."""
answer = mem.ask(question)
return str(answer.get("answer", ""))
# Register functions
tools = [memvid_search, memvid_ask]
# Create client and generate
client = genai.Client()
response = client.models.generate_content(
model="gemini-2.0-flash",
contents="Search for information about authentication",
config=types.GenerateContentConfig(tools=tools)
)
# Handle function calls
for part in response.candidates[0].content.parts:
if hasattr(part, 'function_call'):
fn = part.function_call
if fn.name == "memvid_search":
result = memvid_search(fn.args["query"])
print(result)
```
## Multi-Tool Agent
```python theme={null}
from google import genai
from google.genai import types
from memvid_sdk import use
mem = use('google-adk', 'knowledge.mv2', read_only=True)
# Define multiple tools
def search_knowledge(query: str) -> str:
"""Search the knowledge base for relevant information."""
results = mem.find(query, k=5)
return "\n".join([f"- {r.title}: {r.snippet}" for r in results])
def get_timeline(limit: int = 10) -> str:
"""Get recent entries from the knowledge base."""
entries = mem.timeline(limit=limit)
return "\n".join([f"- [{e.timestamp}] {e.title}" for e in entries])
def get_stats() -> str:
"""Get statistics about the knowledge base."""
stats = mem.stats()
return f"Documents: {stats['frame_count']}, Size: {stats['size_bytes']} bytes"
def ask_question(question: str) -> str:
"""Ask a question and get an AI-synthesized answer."""
answer = mem.ask(question)
return str(answer.get("answer", ""))
# Create client with all tools
client = genai.Client()
chat = client.chats.create(
model="gemini-2.0-flash",
config=types.GenerateContentConfig(
tools=[search_knowledge, get_timeline, get_stats, ask_question],
system_instruction="You are a helpful assistant with full access to a knowledge base."
)
)
# Interactive session
response = chat.send_message("Show me recent entries and then search for authentication info")
print(response.text)
```
## Streaming Responses
```python theme={null}
from google import genai
from memvid_sdk import use
mem = use('google-adk', 'knowledge.mv2', read_only=True)
client = genai.Client()
# Stream response
for chunk in client.models.generate_content_stream(
model="gemini-2.0-flash",
contents="Explain the architecture based on the knowledge base",
config=types.GenerateContentConfig(
tools=mem.tools,
system_instruction="You have access to a knowledge base."
)
):
print(chunk.text, end="")
```
</Tab>
</Tabs>
## Function Declaration Schema
The `memvid_put` function declaration:
```json theme={null}
{
"name": "memvid_put",
"description": "Store a document in Memvid memory for later retrieval.",
"parameters": {
"type": "object",
"properties": {
"title": { "type": "string", "description": "Title of the document" },
"label": { "type": "string", "description": "Category or label" },
"text": { "type": "string", "description": "Text content to store" },
"metadata": { "type": "object", "description": "Optional key-value metadata" }
},
"required": ["title", "label", "text"]
}
}
```
The `memvid_find` function declaration:
```json theme={null}
{
"name": "memvid_find",
"description": "Search Memvid memory for documents matching a query.",
"parameters": {
"type": "object",
"properties": {
"query": { "type": "string", "description": "Search query string" },
"top_k": { "type": "number", "description": "Number of results (default: 5)" }
},
"required": ["query"]
}
}
```
The `memvid_ask` function declaration:
```json theme={null}
{
"name": "memvid_ask",
"description": "Ask a question and get an answer from Memvid memory using RAG.",
"parameters": {
"type": "object",
"properties": {
"question": { "type": "string", "description": "Question to answer" },
"mode": { "type": "string", "enum": ["auto", "lex", "sem"], "description": "Search mode" }
},
"required": ["question"]
}
}
```
## Best Practices
1. **Use read-only mode** for retrieval agents
2. **Handle function calls** appropriately in responses
3. **Use streaming** for long responses
4. **Close the memory** when done
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
const mem = await use('google-adk', 'knowledge.mv2', { readOnly: true });
try {
// Create agent and run queries
const geminiKey = process.env.GEMINI_API_KEY ?? process.env.GOOGLE_API_KEY;
if (!geminiKey) throw new Error("Set GEMINI_API_KEY (or legacy GOOGLE_API_KEY)");
const genAI = new GoogleGenerativeAI(geminiKey);
// ... use client
} finally {
// No explicit close needed; dropping the handle releases the shared lock.
}
```
</Tab>
<Tab title="Python">
```python theme={null}
mem = use('google-adk', 'knowledge.mv2', read_only=True)
try:
# Create agent and run queries
client = genai.Client()
# ... use client
finally:
mem.close()
```
</Tab>
</Tabs>
## Next Steps
<CardGroup>
<Card title="CrewAI" icon="users" href="/frameworks/crewai">
CrewAI integration
</Card>
<Card title="Semantic Kernel" icon="brain" href="/frameworks/semantic-kernel">
Semantic Kernel integration
</Card>
</CardGroup>
# Haystack
Source: https://docs.memvid.com/frameworks/haystack
Build search pipelines with Haystack and Memvid
Integrate Memvid with Haystack to build powerful search and RAG pipelines. The `haystack` adapter provides native Haystack components.
## Installation
```bash theme={null}
pip install memvid-sdk haystack-ai
```
## Quick Start
```python theme={null}
from memvid_sdk import use
# Open with Haystack adapter
mem = use('haystack', 'knowledge.mv2')
# Access Haystack components
retriever = mem.as_retriever(top_k=5)
```
## Basic Pipeline
```python theme={null}
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
from memvid_sdk import use
# Initialize with haystack adapter
mem = use('haystack', 'knowledge.mv2', read_only=True)
# Get retriever component
retriever = mem.as_retriever(top_k=5)
# Create generator
generator = OpenAIGenerator(model="gpt-4o")
# Build pipeline
pipeline = Pipeline()
pipeline.add_component("retriever", retriever)
pipeline.add_component("generator", generator)
pipeline.connect("retriever.documents", "generator.prompt")
# Run query
result = pipeline.run({
"retriever": {"query": "What is the architecture?"}
})
print(result["generator"]["replies"][0])
```
## RAG Pipeline
```python theme={null}
from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from memvid_sdk import use
# Initialize
mem = use('haystack', 'knowledge.mv2', read_only=True)
# Create prompt template
prompt_template = """
Answer the question based on the following context:
Context:
{% for doc in documents %}
- {{ doc.content }}
{% endfor %}
Question: {{ question }}
Answer:
"""
# Build RAG pipeline
rag_pipeline = Pipeline()
rag_pipeline.add_component("retriever", mem.as_retriever(top_k=5))
rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
rag_pipeline.add_component("generator", OpenAIGenerator(model="gpt-4o"))
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder", "generator")
# Run
result = rag_pipeline.run({
"retriever": {"query": "deployment"},
"prompt_builder": {"question": "How do I deploy to production?"}
})
print(result["generator"]["replies"][0])
```
## Custom Retriever Component
```python theme={null}
from haystack import component, Document
from memvid_sdk import use
from typing import List
@component
class MemvidRetriever:
def __init__(self, memory_path: str, top_k: int = 5, mode: str = 'auto'):
self.memory_path = memory_path
self.top_k = top_k
self.mode = mode
self._mem = None
def warm_up(self):
"""Initialize the memory connection."""
self._mem = use('haystack', self.memory_path, read_only=True)
@component.output_types(documents=List[Document])
def run(self, query: str) -> dict:
if self._mem is None:
self.warm_up()
results = self._mem.find(query, k=self.top_k, mode=self.mode)
documents = [
Document(
content=r.snippet,
meta={
'frame_id': r.frame_id,
'title': r.title,
'score': r.score,
'uri': r.uri
}
)
for r in results
]
return {"documents": documents}
# Usage
retriever = MemvidRetriever(memory_path='knowledge.mv2', top_k=10)
```
## Hybrid Search Pipeline
```python theme={null}
from haystack import Pipeline
from haystack.components.joiners import DocumentJoiner
from memvid_sdk import use
mem = use('haystack', 'knowledge.mv2', read_only=True)
# Create retrievers with different modes
lexical_retriever = mem.as_retriever(top_k=10, mode='lex')
semantic_retriever = mem.as_retriever(top_k=10, mode='sem')
# Join results
joiner = DocumentJoiner(join_mode="reciprocal_rank_fusion")
# Build hybrid pipeline
hybrid_pipeline = Pipeline()
hybrid_pipeline.add_component("lexical", lexical_retriever)
hybrid_pipeline.add_component("semantic", semantic_retriever)
hybrid_pipeline.add_component("joiner", joiner)
hybrid_pipeline.connect("lexical.documents", "joiner.documents")
hybrid_pipeline.connect("semantic.documents", "joiner.documents")
# Run
result = hybrid_pipeline.run({
"lexical": {"query": "authentication methods"},
"semantic": {"query": "authentication methods"}
})
print(f"Found {len(result['joiner']['documents'])} documents")
```
## Document Store
```python theme={null}
from memvid_sdk import use
# Use as a document store
mem = use('haystack', 'knowledge.mv2')
# Get document store interface
doc_store = mem.as_document_store()
# Write documents
doc_store.write_documents([
{"content": "Document 1 content", "meta": {"title": "Doc 1"}},
{"content": "Document 2 content", "meta": {"title": "Doc 2"}}
])
# Count documents
count = doc_store.count_documents()
print(f"Total documents: {count}")
# Filter documents
filtered = doc_store.filter_documents(
filters={"field": "meta.title", "operator": "==", "value": "Doc 1"}
)
```
## Best Practices
1. **Use read-only mode** for retrieval pipelines
2. **Warm up components** before running pipelines
3. **Use hybrid search** for best results
4. **Close the memory** when done
```python theme={null}
mem = use('haystack', 'knowledge.mv2', read_only=True)
try:
pipeline = Pipeline()
pipeline.add_component("retriever", mem.as_retriever(top_k=10))
# ... build and run pipeline
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="Semantic Kernel" icon="brain" href="/frameworks/semantic-kernel">
Semantic Kernel integration
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Full Python SDK documentation
</Card>
</CardGroup>
# LangChain
Source: https://docs.memvid.com/frameworks/langchain
Use Memvid as a retriever in LangChain applications
Integrate Memvid with LangChain to build powerful RAG pipelines. The `langchain` adapter provides native LangChain tools for seamless integration with agents.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk @langchain/core @langchain/openai @langchain/langgraph zod
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with LangChain adapter
const mem = await use('langchain', 'knowledge.mv2');
// Access LangChain tools (compatible with createReactAgent)
const tools = mem.tools; // Array of tool() objects
```
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk langchain langchain-openai
```
## Quick Start
```python theme={null}
from memvid_sdk import create, use
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('langchain', 'knowledge.mv2')
else:
mem = create('knowledge.mv2', kind='langchain')
# Access LangChain tools
tools = mem.tools # Returns LangChain StructuredTool objects
```
</Tab>
</Tabs>
## Using Tools with Agents
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
import { ChatOpenAI } from '@langchain/openai';
import { createReactAgent } from '@langchain/langgraph/prebuilt';
import { HumanMessage } from '@langchain/core/messages';
// Get Memvid tools
const mem = await use('langchain', 'knowledge.mv2');
const tools = mem.tools;
// Create agent with LangGraph
const llm = new ChatOpenAI({ model: 'gpt-4o' });
const agent = createReactAgent({ llm, tools });
// Run
const inputs = { messages: [new HumanMessage('Search for authentication info')] };
const stream = await agent.stream(inputs, { streamMode: 'values' });
for await (const { messages } of stream) {
const lastMsg = messages[messages.length - 1];
if (lastMsg.content) {
console.log(lastMsg.content);
}
}
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import create, use
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os
# Create new file or open existing
if os.path.exists('knowledge.mv2'):
mem = use('langchain', 'knowledge.mv2')
else:
mem = create('knowledge.mv2', kind='langchain')
tools = mem.tools
# Create agent with LangGraph
llm = ChatOpenAI(model="gpt-4o")
agent = create_react_agent(llm, tools)
# Run
inputs = {"messages": [("user", "Search for information about authentication")]}
result = agent.invoke(inputs)
print(result["messages"][-1].content)
```
</Tab>
</Tabs>
## Available Tools
The LangChain adapter provides three tools:
| Tool | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Using as a Retriever (Python)
```python theme={null}
from memvid_sdk import use
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
# Initialize with langchain adapter
mem = use('langchain', 'knowledge.mv2', read_only=True)
# Get the retriever
retriever = mem.as_retriever(k=5)
# Create QA chain
qa_chain = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o"),
retriever=retriever
)
result = qa_chain.run("What is the main concept?")
print(result)
```
## Conversational RAG (Python)
```python theme={null}
from memvid_sdk import use
from langchain_openai import ChatOpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
# Initialize
mem = use('langchain', 'knowledge.mv2', read_only=True)
retriever = mem.as_retriever(k=5)
# Create conversational chain
llm = ChatOpenAI(model="gpt-4o")
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
chain = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=retriever,
memory=memory
)
# Chat
response = chain.invoke({"question": "What are the key features?"})
print(response["answer"])
# Follow up
response = chain.invoke({"question": "Tell me more about that"})
print(response["answer"])
```
## Custom Search Options
```python theme={null}
from memvid_sdk import use
mem = use('langchain', 'knowledge.mv2')
# Search with specific mode
results = mem.find('authentication', mode='lex', k=10) # Lexical only
results = mem.find('user login flow', mode='sem', k=10) # Semantic only
results = mem.find('auth best practices', mode='auto', k=10) # Hybrid
# With scope filtering
results = mem.find('API', scope='mv2://docs/', k=5)
```
## Best Practices
1. **Use read-only mode** for retrieval-only applications
2. **Set appropriate k values** based on your context window
3. **Use hybrid mode** for best recall
4. **Close the memory** when done
```python theme={null}
mem = use('langchain', 'knowledge.mv2', read_only=True)
try:
# Do work
results = mem.find('query', k=10)
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="LlamaIndex" icon="database" href="/frameworks/llamaindex">
LlamaIndex integration
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Full Python SDK documentation
</Card>
</CardGroup>
# LlamaIndex
Source: https://docs.memvid.com/frameworks/llamaindex
Build LlamaIndex applications with Memvid as the vector store
Integrate Memvid with LlamaIndex to build powerful RAG applications. The `llamaindex` adapter provides native LlamaIndex components for seamless integration.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk llamaindex @llamaindex/openai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with LlamaIndex adapter
const mem = await use('llamaindex', 'knowledge.mv2');
// Access LlamaIndex tools
const tools = mem.tools; // FunctionTool array
const functions = mem.functions; // Raw function schemas
// Use query engine
const queryEngine = mem.asQueryEngine();
const response = await queryEngine.query({ query: 'What is Memvid?' });
console.log(response.response);
```
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk llama-index llama-index-llms-openai
```
## Quick Start
```python theme={null}
from memvid_sdk import use
# Open with LlamaIndex adapter
mem = use('llamaindex', 'knowledge.mv2')
# Access LlamaIndex tools and query engine
tools = mem.tools
query_engine = mem.as_query_engine()
```
</Tab>
</Tabs>
## Available Tools
The LlamaIndex adapter provides three tools:
| Tool | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Using with Agents
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
// Get Memvid tools
const mem = await use('llamaindex', 'knowledge.mv2');
const tools = mem.tools;
// Tools can be used directly
for (const tool of tools) {
console.log(`Tool: ${tool.metadata.name}`);
console.log(`Description: ${tool.metadata.description}`);
}
// Or use with LlamaIndex agents (when available)
// Note: LlamaIndex.TS agent API is evolving
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
from llama_index.llms.openai import OpenAI
from llama_index.core.agent import ReActAgent
import asyncio
# Get Memvid tools
mem = use('llamaindex', 'knowledge.mv2')
tools = mem.tools
# Create ReAct agent
llm = OpenAI(model="gpt-4o")
agent = ReActAgent(
name="MemvidResearcher",
tools=tools,
llm=llm,
verbose=True
)
# Run agent
async def run():
response = await agent.run("Search for information about vector stores")
print(response)
asyncio.run(run())
```
</Tab>
</Tabs>
## Using as a Query Engine
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
// Initialize
const mem = await use('llamaindex', 'knowledge.mv2');
// Get query engine factory
const queryEngine = mem.asQueryEngine();
// Query
const response = await queryEngine.query({ query: 'What is Memvid?' });
console.log(`Answer: ${response.response}`);
// Access sources
if (response.sourceNodes) {
for (const node of response.sourceNodes) {
console.log(`Source: ${node.node.metadata?.title}`);
}
}
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
# Initialize
mem = use('llamaindex', 'knowledge.mv2', read_only=True)
# Get query engine
query_engine = mem.as_query_engine()
# Query
response = query_engine.query("What are the best practices?")
print(response.response)
# Access sources
for source in response.source_nodes:
print(f"Source: {source.node.metadata.get('title')}")
```
</Tab>
</Tabs>
## Using as a Vector Store (Python)
```python theme={null}
from memvid_sdk import use
from llama_index.core import VectorStoreIndex
from llama_index.llms.openai import OpenAI
# Initialize with llamaindex adapter
mem = use('llamaindex', 'knowledge.mv2', read_only=True)
# Get the vector store
vector_store = mem.as_vector_store()
# Build index from vector store
index = VectorStoreIndex.from_vector_store(vector_store)
# Create query engine
query_engine = index.as_query_engine(
llm=OpenAI(model="gpt-4o")
)
# Query
response = query_engine.query("Explain the architecture")
print(response)
```
## Chat Engine (Python)
```python theme={null}
from memvid_sdk import use
from llama_index.core import VectorStoreIndex
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.llms.openai import OpenAI
# Initialize
mem = use('llamaindex', 'knowledge.mv2', read_only=True)
vector_store = mem.as_vector_store()
# Build index
index = VectorStoreIndex.from_vector_store(vector_store)
# Create chat engine with memory
chat_engine = index.as_chat_engine(
chat_mode="context",
llm=OpenAI(model="gpt-4o"),
memory=ChatMemoryBuffer.from_defaults(token_limit=3000)
)
# Chat
response = chat_engine.chat("What is Memvid?")
print(response)
# Follow-up (maintains context)
response = chat_engine.chat("How does search work?")
print(response)
```
## Custom Search Options
```python theme={null}
from memvid_sdk import use
mem = use('llamaindex', 'knowledge.mv2')
# Search with specific mode
results = mem.find('authentication', mode='lex', k=10) # Lexical only
results = mem.find('user login flow', mode='sem', k=10) # Semantic only
results = mem.find('auth best practices', mode='auto', k=10) # Hybrid
# With scope filtering
results = mem.find('API', scope='mv2://docs/', k=5)
```
## Best Practices
1. **Use read-only mode** for retrieval-only applications
2. **Set appropriate k values** based on your context window
3. **Use hybrid mode** for best recall
4. **Close the memory** when done
```python theme={null}
mem = use('llamaindex', 'knowledge.mv2', read_only=True)
try:
# Do work
retriever = mem.as_retriever(k=10)
# ... use retriever
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="LangChain" icon="link" href="/frameworks/langchain">
LangChain integration
</Card>
<Card title="Vercel AI SDK" icon="bolt" href="/frameworks/vercel-ai">
Vercel AI SDK integration
</Card>
</CardGroup>
# Lovable
Source: https://docs.memvid.com/frameworks/lovable
Production Lovable integration with Memvid REST API backend actions
Connect Lovable-generated apps to Memvid through backend actions that call the Memvid API.
<Info>
**API-based integration.** This guide uses `https://api.memvid.com` directly (no SDK and no local `.mv2` file).
</Info>
## Prerequisites
* A Memvid API key (`mv2_...`)
* A Lovable project with backend/server function support
## Setup
1. Add `MEMVID_API_KEY` to your environment variables.
2. Add `MEMVID_API_BASE` as `https://api.memvid.com` (optional, but recommended).
3. Keep API calls server-side to avoid exposing your key.
## Backend API Helper (`lib/memvid.ts`)
```typescript theme={null}
const MEMVID_API_BASE = process.env.MEMVID_API_BASE || "https://api.memvid.com";
const MEMVID_API_KEY = process.env.MEMVID_API_KEY!;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
export async function memvidRequest(path: string, init: RequestInit = {}, retries = 2) {
if (!MEMVID_API_KEY) throw new Error("Missing MEMVID_API_KEY");
for (let attempt = 0; attempt <= retries; attempt++) {
const response = await fetch(`${MEMVID_API_BASE}${path}`, {
...init,
headers: {
Authorization: `Bearer ${MEMVID_API_KEY}`,
"Content-Type": "application/json",
...(init.headers || {}),
},
});
if (!response.ok) {
const body = await response.text();
const retryable = response.status === 429 || response.status >= 500;
if (retryable && attempt < retries) {
await sleep(250 * Math.pow(2, attempt));
continue;
}
throw new Error(`Memvid request failed: ${response.status} ${body}`);
}
return response.status === 204 ? null : response.json();
}
throw new Error("Memvid request retry budget exhausted");
}
```
## Backend Action Example (`actions/askMemory.ts`)
```typescript theme={null}
import { memvidRequest } from "../lib/memvid";
export async function askMemory(memoryId: string, question: string) {
return memvidRequest(`/v1/memories/${memoryId}/ask`, {
method: "POST",
body: JSON.stringify({
question,
options: { includeSources: true, model: "gpt-4o-mini" },
}),
});
}
```
## Bootstrap Action Example (`actions/bootstrapMemory.ts`)
```typescript theme={null}
import { memvidRequest } from "../lib/memvid";
export async function bootstrapMemory() {
const memory = await memvidRequest("/v1/memories", {
method: "POST",
body: JSON.stringify({
name: "Lovable App Memory",
description: "Knowledge for generated app",
}),
});
await memvidRequest(`/v1/memories/${memory.id}/documents`, {
method: "POST",
body: JSON.stringify({
documents: [
{
title: "FAQ",
text: "Refunds are processed within 5 business days.",
tags: ["billing"],
},
],
}),
});
return memory.id;
}
```
## Recommended Flow
* Provision or select a memory for your app/project.
* Ingest docs from your app data (JSON text, files, or URLs).
* Use `find` for retrieval cards/lists in UI.
* Use `ask` for synthesized answers with sources.
## Endpoints You Will Use Most
* `POST /v1/memories`
* `POST /v1/memories/:id/documents`
* `POST /v1/memories/:id/find`
* `POST /v1/memories/:id/ask`
## Smoke Test
1. Run your bootstrap action and save `memoryId`.
2. Call your ask action with:
```json theme={null}
{
"memoryId": "your_memory_id",
"question": "How long do refunds take?"
}
```
3. Verify answer text and at least one source snippet are returned.
## Related Docs
* [API Integration Patterns](/frameworks/api-integration-patterns)
* [REST API Reference](/api-reference/rest-api)
* [API Overview](/api-reference/index)
# n8n
Source: https://docs.memvid.com/frameworks/n8n
Production n8n integration with Memvid REST API and importable workflow
Integrate Memvid with n8n using HTTP Request nodes and a reusable production workflow.
<Info>
**API-based integration.** This guide uses `https://api.memvid.com` directly (no SDK and no local `.mv2` file).
</Info>
## Prerequisites
* A Memvid API key (`mv2_...`)
* n8n instance (cloud or self-hosted)
* Optional: existing memory ID
## Set Up Credentials in n8n
1. Create a new **HTTP Header Auth** credential in n8n.
2. Set:
* Header Name: `Authorization`
* Header Value: `Bearer mv2_YOUR_API_KEY`
3. Reuse this credential across all Memvid HTTP Request nodes.
You can also use `X-API-Key: mv2_YOUR_API_KEY` if preferred.
## Golden Path Workflow
1. Create memory (once)
2. Add documents
3. Find context
4. Ask with RAG
## Importable n8n Workflow JSON
Import the following JSON in n8n (**Workflows -> Import from Clipboard**). It creates a simple manual pipeline for `create -> ingest -> find -> ask`.
```json theme={null}
{
"name": "Memvid API Golden Path",
"nodes": [
{
"parameters": {},
"id": "manual-trigger",
"name": "Manual Trigger",
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [240, 300]
},
{
"parameters": {
"keepOnlySet": true,
"values": {
"string": [
{ "name": "memoryName", "value": "n8n Support KB" },
{ "name": "question", "value": "What is our P1 SLA?" },
{ "name": "query", "value": "How fast should P1 incidents be acknowledged?" }
]
}
},
"id": "seed-input",
"name": "Seed Input",
"type": "n8n-nodes-base.set",
"typeVersion": 3.4,
"position": [460, 300]
},
{
"parameters": {
"method": "POST",
"url": "https://api.memvid.com/v1/memories",
"sendBody": true,
"contentType": "json",
"jsonBody": "={\"name\": $json.memoryName, \"description\": \"Created from n8n workflow\"}"
},
"id": "create-memory",
"name": "Create Memory",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [700, 300],
"credentials": {
"httpHeaderAuth": {
"name": "Memvid API Key"
}
}
},
{
"parameters": {
"method": "POST",
"url": "=https://api.memvid.com/v1/memories/{{$json.id}}/documents",
"sendBody": true,
"contentType": "json",
"jsonBody": "={\"documents\":[{\"title\":\"Escalation Policy\",\"text\":\"P1 incidents must be acknowledged within 10 minutes.\",\"tags\":[\"support\",\"policy\"]}]}"
},
"id": "add-document",
"name": "Add Document",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [940, 300],
"credentials": {
"httpHeaderAuth": {
"name": "Memvid API Key"
}
}
},
{
"parameters": {
"method": "POST",
"url": "=https://api.memvid.com/v1/memories/{{$node[\"Create Memory\"].json[\"id\"]}}/find",
"sendBody": true,
"contentType": "json",
"jsonBody": "={\"query\": $node[\"Seed Input\"].json[\"query\"], \"topK\": 5}"
},
"id": "find",
"name": "Find",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1180, 220],
"credentials": {
"httpHeaderAuth": {
"name": "Memvid API Key"
}
}
},
{
"parameters": {
"method": "POST",
"url": "=https://api.memvid.com/v1/memories/{{$node[\"Create Memory\"].json[\"id\"]}}/ask",
"sendBody": true,
"contentType": "json",
"jsonBody": "={\"question\": $node[\"Seed Input\"].json[\"question\"], \"options\": {\"model\":\"gpt-4o-mini\",\"includeSources\": true}}"
},
"id": "ask",
"name": "Ask",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1180, 380],
"credentials": {
"httpHeaderAuth": {
"name": "Memvid API Key"
}
}
}
],
"connections": {
"Manual Trigger": {
"main": [[{ "node": "Seed Input", "type": "main", "index": 0 }]]
},
"Seed Input": {
"main": [[{ "node": "Create Memory", "type": "main", "index": 0 }]]
},
"Create Memory": {
"main": [[{ "node": "Add Document", "type": "main", "index": 0 }]]
},
"Add Document": {
"main": [
[{ "node": "Find", "type": "main", "index": 0 }],
[{ "node": "Ask", "type": "main", "index": 0 }]
]
}
}
}
```
## Node-by-Node Reference
### 1) Create Memory
Add an HTTP Request node:
* Method: `POST`
* URL: `https://api.memvid.com/v1/memories`
* Authentication: your Memvid header credential
* Body (JSON):
```json theme={null}
{
"name": "Customer Support KB",
"description": "Docs and runbooks for support agents"
}
```
Store `id` from the response as `memoryId` in your workflow.
### 2) Add Documents
Add another HTTP Request node:
* Method: `POST`
* URL: `https://api.memvid.com/v1/memories/{{$json.memoryId}}/documents`
* Body (JSON):
```json theme={null}
{
"documents": [
{
"title": "Escalation Policy",
"text": "P1 incidents must be acknowledged within 10 minutes.",
"tags": ["support", "policy"]
}
]
}
```
### 3) Search (`find`)
* Method: `POST`
* URL: `https://api.memvid.com/v1/memories/{{$json.memoryId}}/find`
* Body (JSON):
```json theme={null}
{
"query": "How fast should P1 incidents be acknowledged?",
"topK": 5
}
```
### 4) Ask (`ask`)
* Method: `POST`
* URL: `https://api.memvid.com/v1/memories/{{$json.memoryId}}/ask`
* Body (JSON):
```json theme={null}
{
"question": "What is the P1 response SLA?",
"options": {
"model": "gpt-4o-mini",
"includeSources": true
}
}
```
## Reliability Upgrades
* Add error branches from each HTTP node and alert on non-2xx responses.
* Retry `429` and `5xx` failures with Wait + loop nodes.
* Use one memory per tenant/project for cleaner access boundaries.
* Prefer memory-scoped keys for least-privilege workflows.
* Start with `find` for retrieval-only steps, then use `ask` when synthesis is needed.
## Smoke Test
After importing, run the workflow manually and verify:
* `Create Memory` returns an `id`
* `Find` returns at least one hit
* `Ask` returns answer text and sources
## Related Docs
* [API Integration Patterns](/frameworks/api-integration-patterns)
* [REST API Reference](/api-reference/rest-api)
* [API Overview](/api-reference/index)
# OpenAI SDK
Source: https://docs.memvid.com/frameworks/openai
Use Memvid with OpenAI function calling
Integrate Memvid with the OpenAI SDK to use function calling with your knowledge base. The `openai` adapter provides function schemas formatted for OpenAI's chat completions API.
<Tabs>
<Tab title="Node.js">
## Installation
```bash theme={null}
npm install @memvid/sdk openai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with OpenAI adapter
const mem = await use('openai', 'knowledge.mv2');
// Access function schemas
const functions = mem.functions; // Array of function schemas
```
</Tab>
<Tab title="Python">
## Installation
```bash theme={null}
pip install memvid-sdk openai
```
## Quick Start
```python theme={null}
from memvid_sdk import use
# Open with OpenAI adapter
mem = use('openai', 'knowledge.mv2')
# Access function schemas
functions = mem.functions # OpenAI function schemas
```
</Tab>
</Tabs>
## Available Functions
The OpenAI adapter provides three functions:
| Function | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Function Calling Example
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
import OpenAI from 'openai';
// Get Memvid functions
const mem = await use('openai', 'knowledge.mv2');
const functions = mem.functions;
// Create OpenAI client
const client = new OpenAI();
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: 'system', content: 'You are a helpful assistant with access to a knowledge base.' },
{ role: 'user', content: 'Search for information about authentication' },
];
// Create completion with function calling
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages,
tools: functions.map((f: any) => ({ type: 'function' as const, function: f })),
tool_choice: 'auto',
});
// Handle function calls
const message = response.choices[0].message;
if (message.tool_calls) {
for (const toolCall of message.tool_calls) {
const funcName = toolCall.function.name;
const funcArgs = JSON.parse(toolCall.function.arguments);
let result: any;
if (funcName === 'memvid_find') {
result = await mem.find(funcArgs.query, { k: funcArgs.top_k || 5 });
} else if (funcName === 'memvid_put') {
result = await mem.put({
title: funcArgs.title,
label: funcArgs.label,
text: funcArgs.text,
});
} else if (funcName === 'memvid_ask') {
result = await mem.ask(funcArgs.question, { mode: funcArgs.mode || 'auto' });
}
console.log(`Function ${funcName} result:`, result);
}
}
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
import openai
import json
# Get Memvid functions
mem = use('openai', 'knowledge.mv2')
functions = mem.functions
# Create completion with function calling
client = openai.OpenAI()
messages = [
{"role": "system", "content": "You are a helpful assistant with access to a knowledge base."},
{"role": "user", "content": "Search for information about authentication"}
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=[{"type": "function", "function": f} for f in functions],
tool_choice="auto"
)
# Handle function calls
message = response.choices[0].message
if message.tool_calls:
for tool_call in message.tool_calls:
func_name = tool_call.function.name
func_args = json.loads(tool_call.function.arguments)
# Execute the function
if func_name == "memvid_find":
result = mem.find(func_args["query"], k=func_args.get("top_k", 5))
elif func_name == "memvid_put":
result = mem.put(
title=func_args["title"],
label=func_args["label"],
text=func_args["text"]
)
elif func_name == "memvid_ask":
result = mem.ask(func_args["question"], mode=func_args.get("mode", "auto"))
print(f"Function {func_name} result: {result}")
```
</Tab>
</Tabs>
## Complete Conversation Loop
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
import OpenAI from 'openai';
const mem = await use('openai', 'knowledge.mv2');
const functions = mem.functions;
const client = new OpenAI();
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: 'system', content: 'You have access to a knowledge base. Use the tools to help users.' },
{ role: 'user', content: 'What authentication methods are supported?' },
];
// Function to execute tool calls
async function executeFunction(name: string, args: any): Promise<any> {
if (name === 'memvid_find') {
return mem.find(args.query, { k: args.top_k || 5 });
} else if (name === 'memvid_put') {
return mem.put({ title: args.title, label: args.label, text: args.text });
} else if (name === 'memvid_ask') {
return mem.ask(args.question, { mode: args.mode || 'auto' });
}
return null;
}
// Conversation loop
while (true) {
const response = await client.chat.completions.create({
model: 'gpt-4o',
messages,
tools: functions.map((f: any) => ({ type: 'function' as const, function: f })),
tool_choice: 'auto',
});
const message = response.choices[0].message;
messages.push(message);
if (message.tool_calls) {
for (const toolCall of message.tool_calls) {
const funcName = toolCall.function.name;
const funcArgs = JSON.parse(toolCall.function.arguments);
const result = await executeFunction(funcName, funcArgs);
messages.push({
role: 'tool',
tool_call_id: toolCall.id,
content: JSON.stringify(result) || 'Function executed',
});
}
} else {
// No more tool calls, print the response
console.log(message.content);
break;
}
}
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
import openai
import json
mem = use('openai', 'knowledge.mv2')
functions = mem.functions
client = openai.OpenAI()
messages = [
{"role": "system", "content": "You have access to a knowledge base. Use the tools to help users."},
{"role": "user", "content": "What authentication methods are supported?"}
]
# Function to execute tool calls
def execute_function(name, args):
if name == "memvid_find":
return mem.find(args["query"], k=args.get("top_k", 5))
elif name == "memvid_put":
return mem.put(title=args["title"], label=args["label"], text=args["text"])
elif name == "memvid_ask":
return mem.ask(args["question"], mode=args.get("mode", "auto"))
return None
# Conversation loop
while True:
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=[{"type": "function", "function": f} for f in functions],
tool_choice="auto"
)
message = response.choices[0].message
messages.append(message)
if message.tool_calls:
for tool_call in message.tool_calls:
func_name = tool_call.function.name
func_args = json.loads(tool_call.function.arguments)
result = execute_function(func_name, func_args)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps(result) if result else "Function executed"
})
else:
# No more tool calls, print the response
print(message.content)
break
```
</Tab>
</Tabs>
## Function Schemas
### memvid\_put
```json theme={null}
{
"name": "memvid_put",
"description": "Store a document in Memvid memory for later retrieval",
"parameters": {
"type": "object",
"properties": {
"title": { "type": "string", "description": "Title of the document" },
"label": { "type": "string", "description": "Category or label" },
"text": { "type": "string", "description": "Text content to store" },
"metadata": { "type": "object", "description": "Optional metadata" }
},
"required": ["title", "label", "text"]
}
}
```
### memvid\_find
```json theme={null}
{
"name": "memvid_find",
"description": "Search Memvid memory for documents matching a query",
"parameters": {
"type": "object",
"properties": {
"query": { "type": "string", "description": "Search query string" },
"top_k": { "type": "number", "description": "Number of results (default: 5)" }
},
"required": ["query"]
}
}
```
### memvid\_ask
```json theme={null}
{
"name": "memvid_ask",
"description": "Ask a question and get an answer from Memvid memory",
"parameters": {
"type": "object",
"properties": {
"question": { "type": "string", "description": "Question to answer" },
"mode": { "type": "string", "enum": ["auto", "lex", "sem"], "description": "Search mode" }
},
"required": ["question"]
}
}
```
## Best Practices
1. **Use tool\_choice="auto"** to let the model decide when to use tools
2. **Handle multiple tool calls** - the model may call multiple functions
3. **Complete the loop** - continue until no more tool\_calls are returned
4. **Close the memory** when done
```python theme={null}
mem = use('openai', 'knowledge.mv2')
try:
# Use functions...
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="Vercel AI SDK" icon="bolt" href="/frameworks/vercel-ai">
Vercel AI SDK integration
</Card>
<Card title="AutoGen" icon="robot" href="/frameworks/autogen">
AutoGen multi-agent
</Card>
</CardGroup>
# Framework Integrations
Source: https://docs.memvid.com/frameworks/overview
SDK adapters and API-first platform integrations for Memvid
Memvid supports two integration styles:
* **SDK adapters** for framework-native tools and retrievers
* **REST API integrations** for platforms that work best with HTTP
<Info>
**One Memory, Many Surfaces.** Use SDK adapters with `.mv2` files, or use cloud memories through `https://api.memvid.com` for API-first platforms.
</Info>
## Quick Comparison
| Integration | Type | Language | Best For |
| ---------------------------------------------------------------- | ----------- | ------------------ | ---------------------------- |
| [LangChain](/frameworks/langchain) | SDK adapter | Python, Node.js | Agents, chains, RAG |
| [LlamaIndex](/frameworks/llamaindex) | SDK adapter | Python, Node.js | RAG pipelines, indexing |
| [Vercel AI](/frameworks/vercel-ai) | SDK adapter | Node.js | Next.js, streaming |
| [OpenAI](/frameworks/openai) | SDK adapter | Python, Node.js | Direct API, function calling |
| [Google ADK](/frameworks/google-adk) | SDK adapter | Python, Node.js | Gemini, ADK agents |
| [AutoGen](/frameworks/autogen) | SDK adapter | Python, Node.js | Multi-agent systems |
| [CrewAI](/frameworks/crewai) | SDK adapter | Python, Node.js | Agent crews |
| [Semantic Kernel](/frameworks/semantic-kernel) | SDK adapter | Python, Node.js | Enterprise AI, Azure |
| [Haystack](/frameworks/haystack) | SDK adapter | Python | Search pipelines |
| [API Integration Patterns](/frameworks/api-integration-patterns) | REST API | Any HTTP runtime | Shared production patterns |
| [n8n](/frameworks/n8n) | REST API | No-code / low-code | Workflow automation |
| [Replit](/frameworks/replit) | REST API | Node.js, Python | Cloud prototyping and apps |
| [Lovable](/frameworks/lovable) | REST API | TypeScript | Productized AI app builders |
| [v0](/frameworks/v0) | REST API | Next.js | Generated app frontends |
***
## Choosing the Right Integration
<AccordionGroup>
<Accordion title="I'm building a RAG application" icon="magnifying-glass">
**Recommended:** [LangChain](/frameworks/langchain) or [LlamaIndex](/frameworks/llamaindex)
Use the tools to search your knowledge base and build RAG pipelines.
```python theme={null}
# LangChain with tools
from memvid_sdk import use
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
mem = use('langchain', 'knowledge.mv2')
agent = create_react_agent(ChatOpenAI(model="gpt-4o"), mem.tools)
result = agent.invoke({"messages": [{"role": "user", "content": "What is..."}]})
# Or use find() + ask() directly
results = mem.find("search query", k=5)
answer = mem.ask("What is the main concept?")
```
</Accordion>
<Accordion title="I'm building an AI agent" icon="robot">
**Recommended:** [LangChain](/frameworks/langchain), [AutoGen](/frameworks/autogen), or [CrewAI](/frameworks/crewai)
These frameworks excel at agent orchestration with tool use.
```python theme={null}
# LangChain Agent
from langgraph.prebuilt import create_react_agent
mem = use('langchain', 'knowledge.mv2')
agent = create_react_agent(model, mem.tools)
# AutoGen
mem = use('autogen', 'knowledge.mv2')
assistant = AssistantAgent("helper", llm_config={"tools": mem.tools})
```
</Accordion>
<Accordion title="I'm using Next.js / Vercel" icon="triangle">
**Recommended:** [Vercel AI SDK](/frameworks/vercel-ai)
Native streaming support and seamless integration with Next.js.
```typescript theme={null}
import { use } from '@memvid/sdk';
import { streamText } from 'ai';
const mem = await use('vercel-ai', 'knowledge.mv2');
export async function POST(req: Request) {
const result = await streamText({
model: openai('gpt-4o'),
tools: mem.tools,
messages: await req.json(),
});
return result.toDataStreamResponse();
}
```
</Accordion>
<Accordion title="I want direct OpenAI/Gemini function calling" icon="plug">
**Recommended:** [OpenAI](/frameworks/openai) or [Google ADK](/frameworks/google-adk)
Use the native function calling APIs without framework overhead.
```python theme={null}
# OpenAI
mem = use('openai', 'knowledge.mv2')
response = client.chat.completions.create(
model="gpt-4o",
tools=mem.tools,
messages=[...]
)
# Google Gemini
mem = use('google-adk', 'knowledge.mv2')
chat = client.chats.create(
model="gemini-2.0-flash",
config=types.GenerateContentConfig(tools=[mem.tools])
)
```
</Accordion>
<Accordion title="I'm building enterprise AI with Azure" icon="building">
**Recommended:** [Semantic Kernel](/frameworks/semantic-kernel)
Microsoft's SDK with enterprise features and Azure integration.
```python theme={null}
mem = use('semantic-kernel', 'knowledge.mv2')
kernel.add_plugin(mem.tools, "memvid")
```
</Accordion>
<Accordion title="I'm automating workflows with no-code tools" icon="bolt">
**Recommended:** [n8n](/frameworks/n8n) and [Lovable](/frameworks/lovable)
Use direct REST calls to Memvid from HTTP request steps and backend actions.
```bash theme={null}
curl -X POST https://api.memvid.com/v1/memories/{MEMORY_ID}/ask \
-H "Authorization: Bearer mv2_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"question":"What does this process do?"}'
```
</Accordion>
<Accordion title="I'm prototyping quickly in hosted app builders" icon="triangle">
**Recommended:** [Replit](/frameworks/replit) and [v0](/frameworks/v0)
Route your app's backend requests to Memvid API and keep keys server-side.
```typescript theme={null}
const res = await fetch("https://api.memvid.com/v1/memories/{MEMORY_ID}/find", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.MEMVID_API_KEY}` },
body: JSON.stringify({ query: "search query", topK: 5 }),
});
```
</Accordion>
</AccordionGroup>
***
## Universal Features
All adapters provide these core capabilities:
### Tools / Functions
Every adapter exposes three primary tools:
| Tool | Description |
| ------------- | -------------------------------- |
| `memvid_put` | Store documents in memory |
| `memvid_find` | Search for relevant documents |
| `memvid_ask` | Ask questions with RAG synthesis |
```python theme={null}
# Access tools (framework-specific format)
tools = mem.tools
```
### Direct API Access
You can always bypass the adapter and use the core API directly:
```python theme={null}
# These work with any adapter
results = mem.find('search query', k=10)
answer = mem.ask('What is machine learning?')
timeline = mem.timeline(limit=50)
stats = mem.stats()
```
***
## Installation
Each adapter requires its framework to be installed:
<Tabs>
<Tab title="Node.js">
```bash theme={null}
# Core SDK
npm install @memvid/sdk
# Framework dependencies (install what you need)
npm install @langchain/core @langchain/openai # LangChain
npm install llamaindex # LlamaIndex
npm install ai @ai-sdk/openai # Vercel AI
npm install openai # OpenAI
npm install @google/generative-ai # Google ADK
```
</Tab>
<Tab title="Python">
```bash theme={null}
# Core SDK
pip install memvid-sdk
# Framework dependencies (install what you need)
pip install "memvid-sdk[langchain]" # LangChain
pip install "memvid-sdk[llamaindex]" # LlamaIndex
pip install "memvid-sdk[openai]" # OpenAI
pip install google-genai # Google ADK
pip install "memvid-sdk[autogen]" # AutoGen
pip install "memvid-sdk[crewai]" # CrewAI
pip install "memvid-sdk[semantic-kernel]" # Semantic Kernel
pip install "memvid-sdk[haystack]" # Haystack
# Or install all integrations:
pip install "memvid-sdk[full]"
```
</Tab>
</Tabs>
***
## Adapter Architecture
```mermaid theme={null}
flowchart TD
subgraph App["Your Application"]
LC[LangChain Adapter]
LI[LlamaIndex Adapter]
VA[Vercel AI Adapter]
OA[OpenAI Adapter]
end
subgraph Core["Memvid Core API"]
API["put, find, ask, timeline"]
end
subgraph Storage[".mv2 File"]
D[Data]
LX[Lex Index]
VX[Vec Index]
W[WAL]
end
LC --> API
LI --> API
VA --> API
OA --> API
API --> D
API --> LX
API --> VX
API --> W
```
Each adapter:
1. **Wraps** the core Memvid API
2. **Formats** tools/retrievers for the specific framework
3. **Handles** framework-specific types and conventions
4. **Provides** seamless integration without lock-in
***
## Performance Comparison
All adapters have similar performance since they use the same core engine:
| Operation | Latency | Notes |
| --------- | -------- | ---------------------------- |
| `find()` | \< 5ms | Hybrid search (lex + vec) |
| `ask()` | 20-200ms | Depends on LLM response time |
| `put()` | \< 40ms | With embedding generation |
| File Open | \< 10ms | Cold start |
The framework overhead is minimal (\< 1ms per operation).
***
## Examples Gallery
<CardGroup>
<Card title="RAG Chatbot" icon="comments" href="/examples/chatbot-memory">
Build a chatbot with persistent memory using LangChain
</Card>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Create a document Q\&A system with LlamaIndex
</Card>
<Card title="Knowledge Base" icon="book" href="/examples/knowledge-base">
Build a searchable company knowledge base
</Card>
<Card title="Research Assistant" icon="flask" href="/examples/research-assistant">
Create an AI research assistant for papers
</Card>
</CardGroup>
***
## Framework Guides
<CardGroup>
<Card title="LangChain" icon="link" href="/frameworks/langchain">
Agents, chains, retrievers
</Card>
<Card title="LlamaIndex" icon="database" href="/frameworks/llamaindex">
RAG pipelines, query engines
</Card>
<Card title="Vercel AI" icon="triangle" href="/frameworks/vercel-ai">
Next.js, streaming
</Card>
<Card title="OpenAI" icon="brain" href="/frameworks/openai">
Function calling
</Card>
<Card title="Google ADK" icon="google" href="/frameworks/google-adk">
Gemini, ADK agents
</Card>
<Card title="AutoGen" icon="robot" href="/frameworks/autogen">
Multi-agent systems
</Card>
<Card title="CrewAI" icon="users" href="/frameworks/crewai">
Agent crews
</Card>
<Card title="Semantic Kernel" icon="building" href="/frameworks/semantic-kernel">
Enterprise AI
</Card>
<Card title="Haystack" icon="magnifying-glass" href="/frameworks/haystack">
Search pipelines
</Card>
</CardGroup>
***
## API Platform Guides
<CardGroup>
<Card title="API Integration Patterns" icon="shield" href="/frameworks/api-integration-patterns">
Shared contracts, retries, polling, and smoke tests
</Card>
<Card title="n8n" icon="plug" href="/frameworks/n8n">
Workflow automation with HTTP Request nodes
</Card>
<Card title="Replit" icon="bolt" href="/frameworks/replit">
Cloud app integration with server-side fetch
</Card>
<Card title="Lovable" icon="comments" href="/frameworks/lovable">
Product app builder integration via backend API calls
</Card>
<Card title="v0" icon="triangle" href="/frameworks/v0">
Next.js route handlers and server actions
</Card>
</CardGroup>
# Replit
Source: https://docs.memvid.com/frameworks/replit
Production Replit integration with Memvid REST API and reusable client
Use Memvid in Replit by calling the Memvid cloud API from your server code.
<Info>
**API-based integration.** This guide uses `https://api.memvid.com` directly (no SDK and no local `.mv2` file).
</Info>
## Prerequisites
* A Memvid API key (`mv2_...`)
* A Replit project (Node.js recommended)
## Configure Secrets
In Replit, add secrets:
* `MEMVID_API_KEY=mv2_YOUR_API_KEY`
* `MEMVID_API_BASE=https://api.memvid.com` (optional)
## Reusable API Client (`memvid.ts`)
```typescript theme={null}
const API_BASE = process.env.MEMVID_API_BASE || "https://api.memvid.com";
const API_KEY = process.env.MEMVID_API_KEY!;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
export async function memvid(path: string, init: RequestInit = {}, retries = 2) {
if (!API_KEY) throw new Error("Missing MEMVID_API_KEY");
for (let attempt = 0; attempt <= retries; attempt++) {
const res = await fetch(`${API_BASE}${path}`, {
...init,
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json",
...(init.headers || {}),
},
});
if (!res.ok) {
const body = await res.text();
const retryable = res.status === 429 || res.status >= 500;
if (retryable && attempt < retries) {
await sleep(250 * Math.pow(2, attempt));
continue;
}
throw new Error(`Memvid API error ${res.status}: ${body}`);
}
if (res.status === 204) return null;
return res.json();
}
throw new Error("Memvid request retry budget exhausted");
}
```
## Replit Server Example (`server.ts`)
```typescript theme={null}
import express from "express";
import { memvid } from "./memvid";
const app = express();
app.use(express.json());
app.post("/api/memories/bootstrap", async (_req, res) => {
try {
const memory = await memvid("/v1/memories", {
method: "POST",
body: JSON.stringify({
name: "Replit Demo Memory",
description: "Knowledge for Replit prototype",
}),
});
await memvid(`/v1/memories/${memory.id}/documents`, {
method: "POST",
body: JSON.stringify({
documents: [
{
title: "Runbook",
text: "Restart worker first, then clear queue backlog.",
tags: ["ops"],
},
],
}),
});
return res.json({ memoryId: memory.id, ok: true });
} catch (error) {
return res.status(500).json({ ok: false, error: String(error) });
}
});
app.post("/api/memories/:memoryId/find", async (req, res) => {
try {
const data = await memvid(`/v1/memories/${req.params.memoryId}/find`, {
method: "POST",
body: JSON.stringify({
query: req.body.query,
topK: req.body.topK ?? 5,
}),
});
return res.json(data);
} catch (error) {
return res.status(500).json({ error: String(error) });
}
});
app.post("/api/memories/:memoryId/ask", async (req, res) => {
try {
const data = await memvid(`/v1/memories/${req.params.memoryId}/ask`, {
method: "POST",
body: JSON.stringify({
question: req.body.question,
options: { includeSources: true, model: "gpt-4o-mini" },
}),
});
return res.json(data);
} catch (error) {
return res.status(500).json({ error: String(error) });
}
});
app.listen(3000, () => {
console.log("Server listening on :3000");
});
```
## Smoke Test
1. Call `POST /api/memories/bootstrap` and save `memoryId`.
2. Call `POST /api/memories/:memoryId/find`:
```json theme={null}
{ "query": "How do we recover worker issues?" }
```
3. Call `POST /api/memories/:memoryId/ask`:
```json theme={null}
{ "question": "What are the first two recovery steps?" }
```
## Production Tips
* Keep Memvid calls on the server side.
* Expose your own app endpoint for client requests.
* Reuse memory IDs per workspace, team, or project.
* Add retry/backoff around `429` and `5xx`.
## Related Docs
* [API Integration Patterns](/frameworks/api-integration-patterns)
* [REST API Reference](/api-reference/rest-api)
* [API Overview](/api-reference/index)
# Semantic Kernel
Source: https://docs.memvid.com/frameworks/semantic-kernel
Use Memvid with Microsoft Semantic Kernel
Integrate Memvid with Microsoft Semantic Kernel for AI orchestration with persistent memory. The `semantic-kernel` adapter provides native Semantic Kernel plugins.
## Installation
```bash theme={null}
pip install memvid-sdk semantic-kernel
```
## Quick Start
```python theme={null}
from memvid_sdk import use
# Open with Semantic Kernel adapter
mem = use('semantic-kernel', 'knowledge.mv2')
# Access SK plugins
plugin = mem.as_plugin()
```
## Basic Usage
```python theme={null}
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
from memvid_sdk import use
# Initialize kernel
kernel = sk.Kernel()
# Add OpenAI service
kernel.add_service(
OpenAIChatCompletion(
service_id="openai",
ai_model_id="gpt-4o"
)
)
# Initialize with semantic-kernel adapter
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
# Get and register plugin
memvid_plugin = mem.as_plugin()
kernel.add_plugin(memvid_plugin, "memvid")
# Use in prompts
result = await kernel.invoke_prompt(
"""Based on this context from the knowledge base:
{{memvid.search "authentication"}}
Answer the question: How does authentication work?"""
)
print(result)
```
## Native Functions
```python theme={null}
import semantic_kernel as sk
from semantic_kernel.functions import kernel_function
from memvid_sdk import use
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
class MemvidPlugin:
@kernel_function(name="search", description="Search the knowledge base")
def search(self, query: str) -> str:
results = mem.find(query, k=5)
return "\n".join([r.snippet for r in results])
@kernel_function(name="ask", description="Ask a question")
def ask(self, question: str) -> str:
answer = mem.ask(question)
return str(answer.get("answer", ""))
@kernel_function(name="timeline", description="Get recent entries")
def timeline(self, limit: int = 10) -> str:
entries = mem.timeline(limit=limit)
return "\n".join([f"- {e.title}" for e in entries])
# Register plugin
kernel = sk.Kernel()
kernel.add_plugin(MemvidPlugin(), "memvid")
```
## Memory Store Integration
```python theme={null}
import semantic_kernel as sk
from semantic_kernel.memory import SemanticTextMemory
from memvid_sdk import use
# Initialize
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
# Get memory store
memory_store = mem.as_memory_store()
# Create semantic memory
semantic_memory = SemanticTextMemory(
storage=memory_store,
embeddings_generator=None # Uses Memvid's built-in embeddings
)
# Search memory
results = await semantic_memory.search(
collection="knowledge",
query="deployment best practices",
limit=5
)
for result in results:
print(f"- {result.text} (score: {result.relevance})")
```
## Chat with Memory
```python theme={null}
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
from semantic_kernel.contents import ChatHistory
from memvid_sdk import use
# Setup
kernel = sk.Kernel()
kernel.add_service(OpenAIChatCompletion(service_id="openai", ai_model_id="gpt-4o"))
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
kernel.add_plugin(mem.as_plugin(), "memvid")
# Create chat with history
chat_history = ChatHistory()
chat_history.add_system_message(
"You are a helpful assistant with access to a knowledge base. "
"Use the memvid.search function to find relevant information."
)
# Chat function
async def chat(user_input: str):
chat_history.add_user_message(user_input)
result = await kernel.invoke_prompt(
f"""Context: {{{{memvid.search "{user_input}"}}}}
User: {user_input}
Provide a helpful response based on the context."""
)
chat_history.add_assistant_message(str(result))
return result
# Use
response = await chat("How do I configure authentication?")
print(response)
```
## Planner Integration
```python theme={null}
import semantic_kernel as sk
from semantic_kernel.planners import SequentialPlanner
from memvid_sdk import use
# Setup kernel with plugins
kernel = sk.Kernel()
kernel.add_service(OpenAIChatCompletion(service_id="openai", ai_model_id="gpt-4o"))
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
kernel.add_plugin(mem.as_plugin(), "memvid")
# Create planner
planner = SequentialPlanner(kernel)
# Create and execute plan
plan = await planner.create_plan(
"Find information about API endpoints and summarize the key points"
)
result = await plan.invoke()
print(result)
```
## Best Practices
1. **Use read-only mode** for retrieval plugins
2. **Register plugins** before creating prompts
3. **Use planners** for complex multi-step tasks
4. **Close the memory** when done
```python theme={null}
mem = use('semantic-kernel', 'knowledge.mv2', read_only=True)
try:
kernel = sk.Kernel()
kernel.add_plugin(mem.as_plugin(), "memvid")
# ... use kernel
finally:
mem.seal()
```
## Next Steps
<CardGroup>
<Card title="Google ADK" icon="google" href="/frameworks/google-adk">
Google ADK integration
</Card>
<Card title="Haystack" icon="search" href="/frameworks/haystack">
Haystack integration
</Card>
</CardGroup>
# v0
Source: https://docs.memvid.com/frameworks/v0
Production v0 integration for Next.js apps using Memvid REST API
Use Memvid in v0 projects by calling Memvid from Next.js route handlers or server actions.
<Info>
**API-based integration.** This guide uses `https://api.memvid.com` directly (no SDK and no local `.mv2` file).
</Info>
## Prerequisites
* A Memvid API key (`mv2_...`)
* A v0-generated Next.js app
## Environment Variables
Add these to your environment:
```bash theme={null}
MEMVID_API_KEY=mv2_YOUR_API_KEY
MEMVID_API_BASE=https://api.memvid.com
MEMVID_MEMORY_ID=your_memory_id
```
## Shared Server Helper (`lib/memvid.ts`)
```typescript theme={null}
const API_BASE = process.env.MEMVID_API_BASE || "https://api.memvid.com";
const API_KEY = process.env.MEMVID_API_KEY!;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
export async function memvid(path: string, init: RequestInit = {}, retries = 2) {
if (!API_KEY) throw new Error("Missing MEMVID_API_KEY");
for (let attempt = 0; attempt <= retries; attempt++) {
const res = await fetch(`${API_BASE}${path}`, {
...init,
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json",
...(init.headers || {}),
},
});
if (!res.ok) {
const body = await res.text();
const retryable = res.status === 429 || res.status >= 500;
if (retryable && attempt < retries) {
await sleep(250 * Math.pow(2, attempt));
continue;
}
throw new Error(`Memvid API error ${res.status}: ${body}`);
}
return res.status === 204 ? null : res.json();
}
throw new Error("Memvid request retry budget exhausted");
}
```
## Route Handler Example (`app/api/memvid/ask/route.ts`)
```typescript theme={null}
import { NextResponse } from "next/server";
import { memvid } from "@/lib/memvid";
const MEMORY_ID = process.env.MEMVID_MEMORY_ID!;
export async function POST(req: Request) {
try {
const { question } = await req.json();
const data = await memvid(`/v1/memories/${MEMORY_ID}/ask`, {
method: "POST",
body: JSON.stringify({
question,
options: { includeSources: true, model: "gpt-4o-mini" },
}),
});
return NextResponse.json(data);
} catch (error) {
return NextResponse.json(
{ error: "Memvid request failed", details: String(error) },
{ status: 500 }
);
}
}
```
## Route Handler Example (`app/api/memvid/find/route.ts`)
```typescript theme={null}
import { NextResponse } from "next/server";
import { memvid } from "@/lib/memvid";
const MEMORY_ID = process.env.MEMVID_MEMORY_ID!;
export async function POST(req: Request) {
try {
const { query, topK = 5 } = await req.json();
const data = await memvid(`/v1/memories/${MEMORY_ID}/find`, {
method: "POST",
body: JSON.stringify({ query, topK }),
});
return NextResponse.json(data);
} catch (error) {
return NextResponse.json(
{ error: "Memvid request failed", details: String(error) },
{ status: 500 }
);
}
}
```
## When to Use `find` vs `ask`
* Use `find` to show matching chunks/snippets in UI.
* Use `ask` when you need a synthesized answer grounded in retrieved context.
## Recommended Architecture
* Browser UI calls your Next.js API route.
* Route calls Memvid with server-side credentials.
* Response is rendered with answer + sources.
## Smoke Test
1. `POST /api/memvid/find`:
```json theme={null}
{ "query": "What do we know about onboarding?" }
```
2. `POST /api/memvid/ask`:
```json theme={null}
{ "question": "Summarize onboarding policy." }
```
3. Verify both routes return JSON with non-empty content.
## Related Docs
* [API Integration Patterns](/frameworks/api-integration-patterns)
* [REST API Reference](/api-reference/rest-api)
* [API Overview](/api-reference/index)
# Vercel AI SDK
Source: https://docs.memvid.com/frameworks/vercel-ai
Build AI-powered applications with Memvid and Vercel AI SDK
Integrate Memvid with the Vercel AI SDK to build AI-powered web applications. The `vercel-ai` adapter provides tools formatted for use with `generateText`, `streamText`, and other AI SDK functions.
## Installation
```bash theme={null}
npm install @memvid/sdk ai @ai-sdk/openai
```
## Quick Start
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open with Vercel AI adapter
const mem = await use('vercel-ai', 'knowledge.mv2');
// Access Vercel AI tools
const tools = mem.tools; // Object with tool definitions
```
## Available Tools
The Vercel AI adapter provides three tools:
| Tool | Description |
| ------------- | ----------------------------------------------------- |
| `memvid_put` | Store documents in memory with title, label, and text |
| `memvid_find` | Search for relevant documents by query |
| `memvid_ask` | Ask questions with RAG-style answer synthesis |
## Using with generateText
```typescript theme={null}
import { use } from '@memvid/sdk';
import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';
// Get Memvid tools
const mem = await use('vercel-ai', 'knowledge.mv2');
// Use with generateText
const result = await generateText({
model: openai('gpt-4o-mini'),
tools: mem.tools,
maxSteps: 5, // Allow multiple tool calls
system: 'You are a helpful assistant with access to a knowledge base.',
prompt: 'Search for information about authentication and summarize it.',
});
// Access the result
console.log(result.text);
// View tool calls made
for (const step of result.steps) {
if (step.toolCalls) {
for (const call of step.toolCalls) {
console.log(`Tool: ${call.toolName}, Args: ${JSON.stringify(call.args)}`);
}
}
}
```
## Using with streamText
```typescript theme={null}
import { use } from '@memvid/sdk';
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
// Get Memvid tools
const mem = await use('vercel-ai', 'knowledge.mv2');
// Stream response with tool use
const result = await streamText({
model: openai('gpt-4o-mini'),
tools: mem.tools,
maxSteps: 3,
system: 'You are a helpful assistant with access to a knowledge base.',
prompt: 'What features does the product have?',
});
// Stream the text output
for await (const chunk of result.textStream) {
process.stdout.write(chunk);
}
```
## Direct Tool Usage
You can also call tools directly without using an LLM:
```typescript theme={null}
import { use } from '@memvid/sdk';
const mem = await use('vercel-ai', 'knowledge.mv2');
const tools = mem.tools;
// Store a document
const putResult = await tools.memvid_put.execute({
title: 'API Documentation',
label: 'docs',
text: 'The API supports REST and GraphQL endpoints...',
});
console.log(putResult); // "Document stored with frame_id: 2"
// Search for documents
const findResult = await tools.memvid_find.execute({
query: 'API endpoints',
top_k: 5,
});
console.log(findResult);
// Ask a question
const askResult = await tools.memvid_ask.execute({
question: 'How do I authenticate with the API?',
mode: 'auto',
});
console.log(askResult);
```
## Next.js API Route
```typescript theme={null}
// app/api/chat/route.ts
import { use } from '@memvid/sdk';
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
export async function POST(req: Request) {
const { messages } = await req.json();
// Get Memvid tools
const mem = await use('vercel-ai', 'knowledge.mv2');
const result = await streamText({
model: openai('gpt-4o-mini'),
tools: mem.tools,
messages,
maxSteps: 5,
});
return result.toDataStreamResponse();
}
```
## Next.js with useChat
```tsx theme={null}
// app/page.tsx
'use client';
import { useChat } from 'ai/react';
export default function Chat() {
const { messages, input, handleInputChange, handleSubmit } = useChat({
api: '/api/chat',
});
return (
<div>
{messages.map((m) => (
<div key={m.id}>
<strong>{m.role}:</strong> {m.content}
</div>
))}
<form onSubmit={handleSubmit}>
<input
value={input}
onChange={handleInputChange}
placeholder="Ask about your knowledge base..."
/>
<button type="submit">Send</button>
</form>
</div>
);
}
```
## Tool Parameters
### memvid\_put
| Parameter | Type | Required | Description |
| ---------- | ------ | -------- | --------------------------- |
| `title` | string | Yes | Title of the document |
| `label` | string | Yes | Category or label |
| `text` | string | Yes | Text content to store |
| `metadata` | object | No | Optional key-value metadata |
### memvid\_find
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------ |
| `query` | string | Yes | Search query string |
| `top_k` | number | No | Number of results (default: 5) |
### memvid\_ask
| Parameter | Type | Required | Description |
| ---------- | ------ | -------- | ----------------------------- |
| `question` | string | Yes | Question to answer |
| `mode` | string | No | `'auto'`, `'lex'`, or `'sem'` |
## Best Practices
1. **Set maxSteps** to allow the model to make multiple tool calls when needed
2. **Use streaming** for better user experience with `streamText`
3. **Handle tool results** by checking `result.steps` for tool call history
4. **Close the memory** when done with `mem.seal()`
```typescript theme={null}
const mem = await use('vercel-ai', 'knowledge.mv2');
try {
// Use tools...
const result = await generateText({
model: openai('gpt-4o-mini'),
tools: mem.tools,
prompt: 'Search for...',
});
} finally {
await mem.seal();
}
```
## Next Steps
<CardGroup>
<Card title="OpenAI SDK" icon="robot" href="/frameworks/openai">
OpenAI function calling
</Card>
<Card title="LangChain" icon="link" href="/frameworks/langchain">
LangChain integration
</Card>
</CardGroup>
# AWS S3 Storage Setup
Source: https://docs.memvid.com/hosting/aws
Learn how to store your Memvid .mv2 files on Amazon Web Services S3
Storing your Memvid `.mv2` files on Amazon S3 provides durable, scalable object storage with high availability and strong consistency. S3 is ideal for production deployments where you need reliable cloud storage with fine-grained access controls and lifecycle management policies. By hosting your memory files on S3, you can leverage AWS's global infrastructure, automatic backups, and integration with other AWS services like Lambda, EC2, and CloudFront for content delivery.
To get started, you'll need to create an S3 bucket in your preferred AWS region. Choose a region close to your application servers to minimize latency, or use a region that complies with your data residency requirements. Once your bucket is created, configure appropriate bucket policies and IAM roles to control access. For Memvid files, you'll typically want to enable versioning to protect against accidental deletions and configure lifecycle policies to transition older versions to cheaper storage tiers like Glacier for long-term archival.
When uploading `.mv2` files to S3, you can use the AWS CLI, SDK, or the S3 console. For automated workflows, integrate S3 uploads into your application's save operations. The files are stored as binary objects, so ensure your application handles the file uploads correctly. You can also set up S3 event notifications to trigger downstream processing when new memory files are uploaded, enabling real-time synchronization across your infrastructure.
For production deployments, consider enabling S3 server-side encryption (SSE) to protect your data at rest, and use S3 Transfer Acceleration for faster uploads from geographically distributed locations. You can also configure cross-region replication for disaster recovery scenarios. Monitor your S3 usage through CloudWatch metrics to track storage costs and access patterns.
<Card title="Create AWS Account" icon="cloud" href="https://aws.amazon.com/free/">
Sign up for a free AWS account to get started with S3 storage
</Card>
# Azure Blob Storage Setup
Source: https://docs.memvid.com/hosting/azure
Learn how to store your Memvid .mv2 files on Microsoft Azure Blob Storage
Microsoft Azure Blob Storage provides a scalable, secure object storage solution for your Memvid `.mv2` files with seamless integration into the Azure ecosystem. Blob Storage offers three access tiers—Hot, Cool, and Archive—allowing you to optimize costs based on how frequently you access your memory files. The service provides automatic encryption, geo-redundant storage options, and lifecycle management policies that automatically move blobs between tiers as access patterns change. Azure Blob Storage is particularly well-suited for organizations already using Azure services, as it integrates natively with Azure Functions, Logic Apps, and other Azure services.
To begin storing Memvid files on Azure, create a storage account in your preferred Azure region. Choose between locally redundant storage (LRS) for cost savings, zone-redundant storage (ZRS) for high availability within a region, or geo-redundant storage (GRS) for cross-region disaster recovery. Within your storage account, create containers (similar to S3 buckets) to organize your `.mv2` files. Configure appropriate access levels—private, blob, or container—and set up shared access signatures (SAS) or Azure AD authentication for secure access control.
Uploading `.mv2` files to Azure Blob Storage can be accomplished through the Azure Portal, Azure Storage Explorer, Azure CLI, or programmatically using Azure Storage SDKs available in multiple languages. The files are stored as block blobs, which are optimized for streaming and cloud workloads. For large files, Azure supports parallel uploads and resumable transfers to handle network interruptions. You can also configure blob storage event triggers to automatically invoke Azure Functions when new memory files are uploaded, enabling serverless processing pipelines.
For enterprise deployments, enable Azure Blob Storage's built-in encryption and consider using customer-managed keys stored in Azure Key Vault for enhanced security. Leverage Azure's lifecycle management policies to automatically transition blobs from Hot to Cool to Archive tiers based on age and access patterns, significantly reducing storage costs for long-term archival. Monitor your storage usage and costs through Azure Monitor and set up alerts for capacity planning. For global applications, use Azure CDN in front of Blob Storage to cache frequently accessed memory files at edge locations for improved performance.
<Card title="Create Azure Account" icon="cloud" href="https://azure.microsoft.com/free/">
Sign up for Azure with free credits to start using Blob Storage
</Card>
# Google Cloud Storage Setup
Source: https://docs.memvid.com/hosting/gcp
Learn how to store your Memvid .mv2 files on Google Cloud Platform Storage
Google Cloud Storage (GCS) offers a robust object storage solution for your Memvid `.mv2` files with global edge caching, strong consistency guarantees, and seamless integration with Google Cloud services. GCS provides multiple storage classes—Standard, Nearline, Coldline, and Archive—allowing you to optimize costs based on access patterns. The service features automatic encryption at rest, fine-grained IAM permissions, and lifecycle management policies that automatically transition objects to cheaper storage tiers as they age.
Setting up GCS for Memvid files begins with creating a storage bucket in your preferred region. Google Cloud offers multi-regional buckets for global applications, regional buckets for lower latency, and dual-region buckets for high availability. Configure your bucket with appropriate access controls using IAM policies, and consider enabling object versioning to protect against accidental deletions. For Memvid files, you can use the Standard storage class for frequently accessed memories, or Nearline/Coldline for archival purposes.
Uploading `.mv2` files to GCS can be done through the Google Cloud Console, gsutil command-line tool, or programmatically using the Cloud Storage client libraries in various languages. The files are stored as binary objects, and you can leverage GCS's resumable uploads for large files to handle network interruptions gracefully. You can also set up Cloud Storage notifications to trigger Cloud Functions or Pub/Sub messages when new memory files are created, enabling event-driven architectures.
For production use, enable Cloud Storage's built-in encryption and consider using customer-managed encryption keys (CMEK) for additional security control. Take advantage of GCS's lifecycle management to automatically move older files to cheaper storage classes, and use Cloud Monitoring to track storage usage and costs. For applications requiring low latency, enable Cloud CDN integration to cache frequently accessed memory files at edge locations worldwide.
<Card title="Create Google Cloud Account" icon="cloud" href="https://cloud.google.com/free">
Sign up for Google Cloud Platform with free credits to get started
</Card>
# Memvid Documentation - Single-file AI Memory for AI Agents
Source: https://docs.memvid.com/index
Complete documentation for Memvid - a single-file memory layer for AI agents with instant retrieval, hybrid search (BM25 + vector), entity extraction, and time-travel debugging. Zero infrastructure, works offline.
# One file. All your AI memory.
Ship a single `.mv2` file that holds your data, indexes, and crash recovery. Copy it, sync it, commit it to git. No servers. No infrastructure. Just one file.
<Tabs>
<Tab title="CLI">
```bash theme={null}
npm install -g memvid-cli
memvid create knowledge.mv2
echo "Alice works at Anthropic as a Senior Engineer." | memvid put knowledge.mv2
memvid find knowledge.mv2 --query "who works at AI companies"
memvid state knowledge.mv2 "Alice"
# { employer: 'Anthropic', role: 'Senior Engineer' }
```
</Tab>
<Tab title="Node.js">
```typescript theme={null}
import { create, use } from '@memvid/sdk';
import { existsSync } from 'fs';
// create() for NEW files, use() for EXISTING files
const mem = existsSync('knowledge.mv2')
? await use('basic', 'knowledge.mv2')
: await create('knowledge.mv2', 'basic');
await mem.put({ title: 'Team Info', label: 'notes', text: 'Alice works at Anthropic...' });
const results = await mem.find('who works at AI companies', { k: 5, mode: 'lex' });
const alice = await mem.state('Alice');
// { slots: { employer: 'Anthropic', role: 'Senior Engineer' } }
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import create, use
import os
# create() for NEW files, use() for EXISTING files
path = 'knowledge.mv2'
mem = use('basic', path) if os.path.exists(path) else create(path)
mem.put(title='Team', label='info', metadata={}, text='Alice works at Anthropic...')
results = mem.find('who works at AI companies', k=5, mode='lex')
alice = mem.state('Alice')
# {'slots': {'employer': 'Anthropic', 'role': 'Senior Engineer'}}
```
</Tab>
</Tabs>
<Warning>
**`create() will OVERWRITE existing files!`** Note the different parameter order:
* `create(path, kind)` — path first, creates NEW file (deletes existing data)
* `use(kind, path)` — kind first, opens EXISTING file (preserves data)
Always check if the file exists first. See examples above.
</Warning>
***
## Why Memvid?
<CardGroup>
<Card title="Single File Storage" icon="file">
Everything in one portable `.mv2` file. No databases, no Docker, no cloud dependencies.
</Card>
<Card title="Hybrid Search" icon="magnifying-glass">
Combines BM25 lexical search with vector similarity. Best of both worlds.
</Card>
<Card title="O(1) Entity Lookups" icon="bolt">
Ask "What's Alice's job?" and get instant answers via Memory Cards (SPO triplets).
</Card>
<Card title="Time-Travel Debugging" icon="clock-rotate-left">
Record sessions, replay with different parameters, debug retrieval quality.
</Card>
</CardGroup>
***
## Memvid vs Alternatives
| Capability | Memvid | Pinecone | ChromaDB | Weaviate |
| ------------------ | ------------------ | ------------ | -------------- | -------------- |
| **Single file** | `.mv2` | Cloud only | SQLite + files | Docker |
| **Hybrid search** | BM25 + vectors | Vectors only | Vectors only | BM25 + vectors |
| **Entity lookups** | O(1) via SlotIndex | No | No | No |
| **Time-travel** | Built-in | No | No | No |
| **Crash safety** | Embedded WAL | Cloud | Manual | Cloud |
| **Works Offline** | Yes | No | Limited | No |
***
## Quick Start
<CardGroup>
<Card title="CLI" icon="terminal" href="/installation/cli">
```bash theme={null}
npm install -g memvid-cli
```
</Card>
<Card title="Node.js" icon="js" href="/installation/node">
```bash theme={null}
npm install @memvid/sdk
```
</Card>
<Card title="Python" icon="python" href="/installation/python">
```bash theme={null}
pip install memvid-sdk
```
</Card>
</CardGroup>
<Info>
**Ready to build?** Start with the [5-Minute Quickstart](/quickstart/five-minute-guide) for a complete walkthrough.
</Info>
***
## Core Capabilities
### Hybrid Search
Combine keyword matching with semantic understanding:
```bash theme={null}
# Lexical search (fast, exact matching)
memvid find knowledge.mv2 --query "budget" --mode lex
# Semantic search (meaning-based)
memvid find knowledge.mv2 --query "financial outlook" --mode sem
# Hybrid search (combines both - default)
memvid find knowledge.mv2 --query "Q4 projections"
```
### Memory Cards (Entity Extraction)
Extract structured facts and query them instantly:
```bash theme={null}
# Extract facts from documents
memvid enrich knowledge.mv2 --engine rules
# Query entity state (O(1) lookup)
memvid state knowledge.mv2 "Alice"
# employer: Anthropic
# role: Senior Engineer
# location: San Francisco
```
### LLM-Powered Q\&A
Ask natural language questions with sourced answers:
```bash theme={null}
export OPENAI_API_KEY=sk-...
memvid ask knowledge.mv2 --question "What is Alice's role?" --use-model openai
# Answer: Alice is a Senior Engineer at Anthropic in San Francisco.
# Sources: [Meeting Notes, Team Directory]
```
### Time-Travel Debugging
Record and replay sessions to debug retrieval:
```bash theme={null}
memvid session start knowledge.mv2 --name "qa-test"
memvid find knowledge.mv2 --query "test query"
memvid session end knowledge.mv2
# Replay with different parameters
memvid session replay knowledge.mv2 --session abc123 --top-k 10
```
***
## Framework Integrations
Use Memvid with your favorite AI frameworks:
<CardGroup>
<Card title="LangChain" icon="link" href="/frameworks/langchain">
Vector store adapter
</Card>
<Card title="LlamaIndex" icon="fire" href="/frameworks/llamaindex">
Index and retriever
</Card>
<Card title="Vercel AI" icon="triangle" href="/frameworks/vercel-ai">
RAG pipeline
</Card>
<Card title="OpenAI" icon="robot" href="/frameworks/openai">
Function calling
</Card>
</CardGroup>
***
## Start Exploring
<CardGroup>
<Card title="5-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Build your first AI memory with search and Q\&A
</Card>
<Card title="CLI Reference" icon="terminal" href="/cli/index">
Complete command reference for all 38+ commands
</Card>
<Card title="Node.js SDK" icon="js" href="/node-sdk/overview">
Full API reference with TypeScript types
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Complete Python API with examples
</Card>
<Card title="Core Concepts" icon="lightbulb" href="/concepts/memory-architecture">
Deep dive into .mv2 format, indexes, and architecture
</Card>
<Card title="Embedding Providers" icon="cube" href="/concepts/embedding-models">
OpenAI, Gemini, Mistral, Cohere, and local models
</Card>
</CardGroup>
# Install the CLI
Source: https://docs.memvid.com/installation/cli
Install the Memvid CLI globally via npm
Install Memvid CLI and start building AI memory in seconds. No complex setup required.
***
## Install
<Tabs>
<Tab title="Auto Installer (Recommended)">
The installer automatically checks for and installs required dependencies (git, node, npm), then installs Memvid globally.
**macOS / Linux:**
```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/memvid/preflight-installer/main/install.sh | bash
```
**Windows (PowerShell):**
```powershell theme={null}
irm https://raw.githubusercontent.com/memvid/preflight-installer/main/install.ps1 | iex
```
<Note>
The installer only installs missing tools and asks for confirmation before proceeding.
[View source on GitHub](https://github.com/memvid/preflight-installer)
</Note>
</Tab>
<Tab title="npm">
```bash theme={null}
npm install -g memvid-cli
```
</Tab>
</Tabs>
Verify installation:
```bash theme={null}
memvid --version
```
That's it. You're ready to go.
***
## Quick Start
```bash theme={null}
# Create your first memory
memvid create my-memory.mv2
# Add some documents
memvid put my-memory.mv2 --input ./documents/
# Search immediately
memvid find my-memory.mv2 --query "your search term"
# Ask questions
memvid ask my-memory.mv2 --question "What is this about?"
```
***
## Without Installing
Run without global installation:
```bash theme={null}
npx memvid-cli --help
npx memvid-cli create test.mv2
```
***
## Environment Variables
Optional configuration:
| Variable | Description |
| ------------------- | -------------------------- |
| `MEMVID_API_KEY` | Dashboard API key for sync |
| `OPENAI_API_KEY` | OpenAI for LLM synthesis |
| `GROQ_API_KEY` | Groq for fast synthesis |
| `ANTHROPIC_API_KEY` | Claude for synthesis |
<Note>
**No API keys required** to get started. Memvid works offline with local search and the built-in TinyLlama model.
</Note>
***
## Updating
```bash theme={null}
npm update -g memvid-cli
```
***
## Troubleshooting
<AccordionGroup>
<Accordion title="Command Not Found">
Make sure npm global bin is in your PATH:
```bash theme={null}
# Check npm global bin location
npm root -g
# Add to PATH if needed
export PATH="$PATH:$(npm root -g)/../bin"
```
</Accordion>
<Accordion title="Permission Denied (macOS)">
If macOS blocks the binary:
1. Go to **System Settings > Privacy & Security**
2. Click **"Allow Anyway"** next to the blocked app
</Accordion>
</AccordionGroup>
***
## System Requirements
| Platform | Requirement |
| -------- | ------------------------------ |
| Node.js | 14+ |
| macOS | 11+ (ARM64 or x86\_64) |
| Linux | glibc 2.17+ (x86\_64 or ARM64) |
| Windows | 10+ (x86\_64) |
The npm package automatically downloads the correct native binary for your platform.
***
## Next Steps
<CardGroup>
<Card title="5-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Build your first memory
</Card>
<Card title="The Memvid Approach" icon="lightbulb" href="/introduction/the-memvid-approach">
Why Memvid doesn't need embeddings
</Card>
</CardGroup>
# Models
Source: https://docs.memvid.com/installation/models
Manage local caches for embeddings, reranking, CLIP, NER, and enrichment
Memvid uses local models for semantic search (embeddings), reranking, visual search (CLIP), Logic‑Mesh entity extraction (NER), and local enrichment workflows. Models are cached under `MEMVID_MODELS_DIR` (default `~/.memvid/models`).
Some models are installed explicitly via `memvid models install`, while embedding/reranker models are auto-downloaded on first use (unless offline).
***
## Quick Start
```bash theme={null}
# See what's available / installed (no downloads)
memvid models list
# Install optional CLIP + NER models
memvid models install --clip mobileclip-s2
memvid models install --ner distilbert-ner
# Install optional local LLM for enrichment (GGUF)
memvid models install phi-3.5-mini
```
***
## Model Types
| Type | How it’s fetched | Used by |
| ---------------- | -------------------------------- | ------------------------------------------------------------------ |
| **Embedding** | Auto-download on first use | `put --embedding`, `find --mode sem/auto`, `ask --mode sem/hybrid` |
| **Reranker** | Auto-download on first use | Hybrid `ask`/`find` (disable with `--no-rerank`) |
| **CLIP** | `memvid models install --clip …` | `put --clip`, `find --mode clip` |
| **NER** | `memvid models install --ner …` | `put --logic-mesh`, `follow …` |
| **LLM (Enrich)** | `memvid models install …` | `enrich`, `put --contextual --contextual-model local` |
| **Whisper** | Auto-download on first use | `put --transcribe` |
***
## Embedding Models (Text Vectors)
Select the default embedding model with the global `-m/--embedding-model` flag. It can appear before or after the subcommand:
```bash theme={null}
memvid put knowledge.mv2 --input docs/ --embedding -m bge-small
memvid -m openai-small put knowledge.mv2 --input docs/ --embedding
```
### Common choices
| Model | Dimensions | Notes |
| -------------- | ---------- | ------------------------------- |
| `bge-small` | 384 | Default local model (fastembed) |
| `bge-base` | 768 | Higher quality local model |
| `nomic` | 768 | High accuracy local model |
| `gte-large` | 1024 | Best local semantic depth |
| `openai-small` | 1536 | `OPENAI_API_KEY` required |
| `openai-large` | 3072 | `OPENAI_API_KEY` required |
| `openai` | 3072 | Alias for `openai-large` |
| `openai-ada` | 1536 | Legacy OpenAI model |
### External embedding APIs
```bash theme={null}
export OPENAI_API_KEY=sk-...
memvid put knowledge.mv2 --input docs/ --embedding -m openai-small
```
<Info>
`ask`/`find` auto-detect the correct embedding runtime from the `.mv2` when vectors are present. Use `--query-embedding-model` (or global `-m`) only when you need an explicit override.
</Info>
***
## Reranking (Hybrid Precision)
`memvid ask` may use a cross-encoder reranker (auto-downloaded on first use). Disable it in gated/offline environments:
```bash theme={null}
memvid ask knowledge.mv2 --question "…" --no-rerank
```
***
## CLIP Models (Visual Search)
Install a CLIP model:
```bash theme={null}
memvid models install --clip mobileclip-s2
memvid models install --clip mobileclip-s2-fp16
memvid models install --clip siglip-base
```
Use it during ingestion and search:
```bash theme={null}
memvid put photos.mv2 --input ./images --clip
memvid find photos.mv2 --query "sunset over ocean" --mode clip
```
***
## NER Model (Logic‑Mesh)
Install NER:
```bash theme={null}
memvid models install --ner distilbert-ner
```
Enable Logic‑Mesh during ingestion:
```bash theme={null}
memvid put graph.mv2 --input docs/ --logic-mesh
memvid follow graph.mv2 traverse --start "Microsoft" --hops 2
```
***
## Enrichment LLM Models
These are local GGUF models used by enrichment workflows (not by `memvid ask`):
```bash theme={null}
memvid models install phi-3.5-mini
memvid models install phi-3.5-mini-q8
```
<Info>
For `memvid ask`, choose a synthesis model with `--use-model` (e.g. `--use-model openai`, `--use-model gemini-2.0-flash`, or `--use-model "ollama:qwen2.5:1.5b"`). See [Local Models with Ollama](/concepts/local-models).
</Info>
***
## List / Verify / Remove
```bash theme={null}
# Filter model list
memvid models list --model-type embedding
memvid models list --model-type clip --json
# Verify installed enrichment LLM models (phi-3.5-*)
memvid models verify
memvid models verify phi-3.5-mini
# Remove an enrichment LLM model
memvid models remove phi-3.5-mini --yes
```
***
## Offline Mode
Set `MEMVID_OFFLINE=1` to prevent downloads. In offline mode:
* `memvid models install …` fails (it can’t download).
* Embedding/reranker auto-download is blocked; run a semantic command once while online to populate caches.
***
## Environment Variables
| Variable | Purpose |
| ------------------- | -------------------------------------------------- |
| `MEMVID_MODELS_DIR` | Model cache directory (default `~/.memvid/models`) |
| `MEMVID_OFFLINE` | Skip downloads/network (`1` to enable) |
| `MEMVID_CLIP_MODEL` | Default local CLIP model (e.g. `mobileclip-s2`) |
| `OPENAI_API_KEY` | OpenAI embeddings / CLIP / LLM providers |
| `GEMINI_API_KEY` | Gemini providers |
| `ANTHROPIC_API_KEY` | Claude providers |
| `COHERE_API_KEY` | Cohere embeddings |
| `VOYAGE_API_KEY` | Voyage embeddings |
# Install the Node.js SDK
Source: https://docs.memvid.com/installation/node
Install @memvid/sdk (TypeScript/JavaScript) with optional framework adapters
The Node.js SDK is published as `@memvid/sdk`. It includes a native addon (N-API) and a fully-typed TypeScript API.
## Install
```bash theme={null}
npm install @memvid/sdk
# or: pnpm add @memvid/sdk
```
Requirements: Node.js 18+. Prebuilt binaries ship for common platforms; if your platform isn’t covered, build from source.
## Quick Start
```ts theme={null}
import { create, open, use } from "@memvid/sdk";
// Create new memory
const mem = await create("notes.mv2");
await mem.put({
title: "Hello Memvid",
label: "demo",
text: "Hello from Node.js",
enableEmbedding: true,
});
await mem.seal(); // commits writes
// Open existing memory (read-only)
const ro = await open("notes.mv2", "basic", { readOnly: true });
const results = await ro.find("hello", { k: 5, mode: "auto" });
console.log(results.hits.map((h: any) => h.title));
// Framework adapters (tools/functions)
const lc = await use("langchain", "notes.mv2", { readOnly: true });
console.log(Object.keys(lc.tools ?? {}));
```
## Environment (Recommended)
* `MEMVID_OFFLINE=1` disables model downloads and remote calls (you must provide embeddings/LLM keys explicitly).
* Provider keys (optional): `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`, `VOYAGE_API_KEY`.
<Note>
`seal()` is a write operation (it commits). For read-only usage, do not call `seal()`. Just keep the handle around and reuse it.
</Note>
# Install the Python SDK
Source: https://docs.memvid.com/installation/python
Use the released memvid-sdk package (with optional adapters)
<Info>
**Quick Start** - Install with `pip install memvid-sdk`, import `use()` to open or create `.mv2` files, and call `put()`, `find()`, `ask()` to store and retrieve memories. Optional adapters available for LangChain, LlamaIndex, and OpenAI integrations.
</Info>
The published package is `memvid-sdk`. It bundles the PyO3 bindings, adapter registry, and CLI-parity helpers (`verify`, `doctor`, ticket tooling). You only need `pip` to get started.
## Install
```bash theme={null}
pip install memvid-sdk
# optional adapters
pip install "memvid-sdk[langchain]" "memvid-sdk[openai]"
```
The extras match the adapter names defined in the Binding PRD (`langchain`, `llamaindex`, `openai`, etc.).
## Quickstart
```python theme={null}
from memvid_sdk import LockedError, Memvid, use
mv = use("basic", "notes.mv2")
mv.put(text="hello world", kind="text/plain")
print(mv.stats())
report = Memvid.verify("notes.mv2", deep=True)
print(report["overall_status"])
mv.seal()
```
`use(kind, path, ...)` mirrors the CLI locking semantics: pass `read_only=True` when sharing files between processes or catch `LockedError` when an exclusive writer already exists. For a more complete script (including retrieval + LLM synthesis), explore the `existing_usage.py` example included with the SDK distribution.
Editable installs keep parity with CI but are no longer required for day-to-day usage now that `memvid-sdk` is on PyPI.
# Benchmarks
Source: https://docs.memvid.com/introduction/benchmarks
How Memvid compares to leading vector databases
<Info>
**TL;DR**: Memvid reduces retrieval errors by **50-66%** compared to leading vector databases while delivering **4x better accuracy-per-millisecond** than alternatives.
</Info>
We benchmarked Memvid against four popular vector databases on the **Wikipedia dataset** (39,324 documents, 2,500 queries). The results speak for themselves.
***
## Headline Results
<CardGroup>
<Card title="92.7%" icon="bullseye">
**Top-1 Accuracy**
Memvid finds the correct document on the first try 92.7% of the time, 8.5 to 14.5 points higher than any competitor.
</Card>
<Card title="50-66%" icon="arrow-down">
**Error Reduction**
Memvid reduces retrieval errors by 50-66% compared to Chroma, Weaviate, LanceDB, and Qdrant.
</Card>
<Card title="4.2x" icon="bolt">
**Accuracy per ms**
Memvid delivers 4.2x more accuracy per millisecond than Chroma, the best quality-to-latency ratio.
</Card>
<Card title="0.95" icon="ranking-star">
**MRR Score**
Mean Reciprocal Rank of 0.95. The correct result consistently appears in the top 1-2 hits.
</Card>
</CardGroup>
***
## Accuracy Comparison
### Top-1 Accuracy (Higher is Better)
The most important metric: **does the system return the correct document first?**
| System | Accuracy\@1 | vs Memvid |
| ---------- | ----------- | --------- |
| **Memvid** | **92.72%** | baseline |
| LanceDB | 84.24% | -8.5 pts |
| Qdrant | 84.24% | -8.5 pts |
| Weaviate | 80.68% | -12.0 pts |
| Chroma | 78.24% | -14.5 pts |
### Error Reduction
Another way to look at accuracy: how often does the system get it wrong?
| System | Error Rate | Memvid Reduces Errors By |
| ---------- | ---------- | ------------------------ |
| Chroma | 21.76% | **66%** fewer errors |
| Weaviate | 19.32% | **62%** fewer errors |
| LanceDB | 15.76% | **54%** fewer errors |
| Qdrant | 15.76% | **54%** fewer errors |
| **Memvid** | **7.28%** | baseline |
<Info>
**Memvid returns the wrong answer 3x less often than Chroma.**
</Info>
### Accuracy at Different k Values
| System | @1 | @3 | @5 | @10 |
| ---------- | ---------- | ---------- | ---------- | ---------- |
| **Memvid** | **92.72%** | **96.96%** | **97.56%** | **98.12%** |
| LanceDB | 84.24% | 92.72% | 94.60% | 96.24% |
| Qdrant | 84.24% | 92.72% | 94.60% | 96.24% |
| Weaviate | 80.68% | 88.52% | 90.16% | 91.52% |
| Chroma | 78.24% | 85.80% | 87.28% | 88.68% |
***
## Ranking Quality
### MRR (Mean Reciprocal Rank)
MRR measures how high the correct result appears in the ranking. A score of 1.0 means perfect top-1 placement every time.
| System | MRR | Interpretation |
| ---------- | --------- | ----------------------------- |
| **Memvid** | **0.949** | Correct result typically #1 |
| LanceDB | 0.888 | Correct result typically #1-2 |
| Qdrant | 0.888 | Correct result typically #1-2 |
| Weaviate | 0.849 | Correct result typically #2 |
| Chroma | 0.823 | Correct result typically #2 |
### NDCG\@10 (Normalized Discounted Cumulative Gain)
NDCG measures overall ranking quality across the top 10 results.
| System | NDCG\@10 |
| ---------- | --------- |
| **Memvid** | **0.967** |
| LanceDB | 0.929 |
| Qdrant | 0.929 |
| Weaviate | 0.886 |
| Chroma | 0.858 |
<Tip>
**Only Memvid achieves >0.94 MRR and >0.96 NDCG\@10** in this benchmark. Memvid consistently surfaces the correct result in the top 1-2 hits with near-perfect ranking quality.
</Tip>
***
## Latency Performance
### Query Latency (Lower is Better)
| System | p50 | p95 | p99 | QPS |
| ---------- | ---------- | ---------- | ---------- | ------ |
| Weaviate | 5.3ms | 7.1ms | 7.9ms | 180 |
| **Memvid** | **16.0ms** | **17.4ms** | **19.7ms** | **61** |
| LanceDB | 16.0ms | 17.5ms | 19.3ms | 61 |
| Qdrant | 28.0ms | 30.4ms | 31.4ms | 36 |
| Chroma | 55.6ms | 61.0ms | 65.2ms | 18 |
### Cold Start Time
| System | Cold Start |
| ---------- | ---------- |
| **Memvid** | **0.5ms** |
| Chroma | 66.3ms |
| Qdrant | 71.8ms |
| LanceDB | 72.4ms |
| Weaviate | 147.7ms |
<Info>
**Memvid cold-starts 130-300x faster** than alternatives. This matters for serverless deployments and edge computing where startup time is critical.
</Info>
***
## The Efficiency Frontier
### Accuracy per Millisecond
We compute accuracy divided by p95 latency to measure quality-per-latency:
| System | Accuracy\@1 | p95 Latency | Accuracy/ms |
| ---------- | ----------- | ----------- | ----------- |
| **Memvid** | 92.72% | 17.4ms | **0.053** |
| Weaviate | 80.68% | 7.1ms | 0.114 |
| LanceDB | 84.24% | 17.5ms | 0.048 |
| Qdrant | 84.24% | 30.4ms | 0.028 |
| Chroma | 78.24% | 61.0ms | 0.013 |
<Note>
**Memvid delivers 4.2x more accuracy per millisecond than Chroma** and leads all systems except Weaviate (which sacrifices 12 points of accuracy for speed).
If accuracy matters, Memvid is the clear winner. If you can tolerate 12% worse results, Weaviate is faster.
</Note>
### The Frontier Chart
```
Accuracy@1 (%)
│
93% │ ★ Memvid
│
85% │ ● LanceDB ● Qdrant
│
81% │ ● Weaviate
│
78% │ ● Chroma
│
└────────────────────────────────────────
5ms 17ms 28ms 61ms
p95 Latency →
```
**Memvid sits alone at the accuracy frontier.** No other system achieves >90% accuracy at any latency.
***
## Storage Efficiency
### Single-File Advantage
Memvid is unique among memory systems because everything lives inside a single portable file; (data, embeddings, indices, and metadata) There are no sidecar files, external indexes, database directories, or hidden dependencies. One file contains the entire memory: easy to move, copy, version, share, or embed into an agent.
| System | Storage | Compression | Bytes/Doc |
| ---------- | ---------- | ----------- | ---------- |
| LanceDB | 213 MB | 0.71x | 5,428 |
| Qdrant | 212 MB | 0.72x | 5,396 |
| **Memvid** | **508 MB** | **0.30x** | **12,911** |
| Weaviate | 1,009 MB | 0.15x | 25,646 |
| Chroma | 1,025 MB | 0.15x | 26,068 |
<Info>
Memvid’s higher bytes-per-document value reflects its richer internal structure: embedded indices, a write-ahead log, a time index, and the metadata required for hybrid semantic + keyword + time-travel search. Instead of scattering these components across multiple files or services, Memvid packages the entire memory system into a single `.mv2` file, delivering portability and simplicity that traditional systems can’t match.
</Info>
***
## Methodology
### Dataset
* **Wikipedia**: 39,324 documents
* **Queries**: 2,500 natural language queries with known correct answers
### Systems Tested
* **Memvid** v2 (hybrid search mode)
* **Chroma** 0.4.x (default HNSW)
* **LanceDB** (default IVF-PQ)
* **Qdrant** (default HNSW)
* **Weaviate** (default HNSW)
### Metrics
* **Accuracy\@k**: Fraction of queries where the correct document appears in top-k results
* **MRR**: Mean Reciprocal Rank (1/position of first correct result)
* **NDCG\@10**: Normalized Discounted Cumulative Gain at 10
* **Latency**: Query time in milliseconds (p50, p95, p99)
* **QPS**: Queries per second throughput
### Environment
* Apple M-series Mac
* All systems using default configurations
* Same embedding model across all systems
* Each query executed 1x (no caching)
***
## Key Takeaways
<CardGroup>
<Card title="Best Accuracy" icon="trophy">
Memvid achieves **92.7% top-1 accuracy**, 8.5 to 14.5 points higher than any competitor.
</Card>
<Card title="Lowest Error Rate" icon="shield-check">
Memvid **reduces errors by 50-66%** compared to leading vector databases.
</Card>
<Card title="Best Ranking" icon="list-ol">
Highest MRR (0.95) and NDCG\@10 (0.97). The correct result consistently appears first.
</Card>
<Card title="Fastest Cold Start" icon="rocket">
**0.5ms cold start**, 130-300x faster than alternatives for serverless deployments.
</Card>
</CardGroup>
***
## Run the Benchmarks Yourself
The benchmark suite is open source. Run it on your own hardware:
```bash theme={null}
git clone https://github.com/memvid/memvid
cd memvid/benchmarks/python
pip install -r requirements.txt
python run_benchmarks.py
```
Results are saved to `results/results.json`.
***
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/quickstart/cli-to-dashboard">
Try Memvid in 5 minutes
</Card>
<Card title="Frame Architecture" icon="film" href="/introduction/frames">
Learn why Memvid is different
</Card>
</CardGroup>
# Frame Architecture
Source: https://docs.memvid.com/introduction/frames
Why Memvid uses a a unique frame architecture for AI memory.
One of Memvid's core innovations is the **Smart Frame**, a storage primitive inspired by how video files encode information. Just as videos are composed of sequential frames that can be played, randomly seeked, or edited, without rewriting the entire file, Memvid represents AI data as an append-only sequence of semantic frames. Each frame captures meaning at a point in time, enabling efficient retrieval, temporal navigation, and incremental growth without destructive updates.
***
## The Early Insight
Memvid was born from a real internal problem.
While our team was building agentic systems for the healthcare industry, we ran into a foundational challenge: memory. We were responsible for building AI agents that could screen applicants and adapt to the unique, high-stakes requirements of healthcare staffing, reasoning over long histories of candidates, roles, facility requirements, and constantly changing constraints. The dataset wasn't just large, it was mission-critical and evolving fast.
### The Problem We Faced
For an AI agent to be useful in real-world staffing workflows, it needed to reliably remember people, conversations, decisions, and constraints over long periods of time. When someone asked:
* “What roles has this candidate applied for in the past six months?”
* “Have they worked night shifts before?”
* “What requirements did this facility specify last week?”
The answer had to be exact. Not a summary. Not a best guess. Not a hallucination.
That requirement exposed hard limits in existing AI memory systems. No matter which approach we tried, we ran into the same failures:
| Approach | Problem |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Feed everything to the LLM** | Large language models have strict context limits. Real-world histories quickly exceeded those limits, making full recall impossible. |
| **Fine-tune/pretrain a model** | Slow, expensive, and brittle. Data changed constantly, and retraining for every update simply didn’t scale. |
| **Traditional RAG with VectorDB** | Vector search retrieves *similar* information, not *exact* information. Critical details were lost to semantic approximation. |
| **Chunking strategies** | Chunking fractured context. Ordering, dependencies, and timelines were easily broken, often in subtle, dangerous ways. |
On top of that, the data itself was extremely sensitive. We needed memory that could run fully on-prem, work offline, remain portable across devices, and avoid centralized infrastructure entirely. Traditional RAG pipelines weren’t just complex and expensive, they were security liabilities.
None of the existing approaches met the bar.
### **What We Actually Needed**
We needed a system that could:
* Store unbounded, growing histories
* Recall exact information, not semantic guesses
* Support real-time writes as new data arrived
* Run fully offline and on-prem
* Be portable and self-contained
* Minimize attack surface and infrastructure complexity
So we stepped back and asked a different question.
### **The Video Insight**
> ***What existing technology already handles massive, sequential data with random access, efficient compression, and decades of battle-tested reliability?***
The answer was video.
A two-hour film contains millions of frames, yet you can jump to any moment instantly. The file is self-contained: no database, no server, no external dependencies. Corrupted frames don’t invalidate the entire file. And decades of optimization have made video codecs extraordinarily efficient.
Real-world AI memory has the same shape.
Information accumulates over time. Events are sequential. You need to jump to specific moments while preserving the full historical timeline. Memory must be incrementally writable, crash-safe, and portable across machines.
**Video codecs have spent 40+ years solving exactly these problems:**
* **Sequential data with random access**: jump to any frame instantly
* **Efficient compression**: 100x compression ratios via redundancy exploitation
* **Self-contained files**: No external dependencies or infrastructure required
* **Crash recovery**: Corrupted frames are localized, not catastrophic
* **Streaming support**: Start processing data before the full file loads
## From Video to Memory
So we tried something unconventional: Storing embeddings inside frames.
Each interaction becomes a frame. Each applicant update, requirement change, or decision is a frame. String them together, index them properly, and you get operational memory that an AI system can query with exact, deterministic recall.
That insight evolved into **Memvid**.
We shipped it in production. It worked. And we quickly realized this wasn’t just a healthcare problem, every serious AI application faces the same challenge. So we open-sourced the solution.
***
## Why Smart Frames as Storage Units?
Traditional systems treat documents as isolated objects. Memvid treats information as frames in a continuous, evolving knowledge stream.
That difference changes everything.
### The Problem with Document-Centric Storage
```mermaid theme={null}
graph LR
subgraph Traditional["Traditional Approach"]
direction LR
D1[Doc 1]
D2[Doc 2]
D3[Doc 3]
D4[Doc 4]
end
D1 -.-> P1[No temporal order]
D2 -.-> P2[Scattered metadata]
D3 -.-> P3[No relationships]
D4 -.-> P4[Sync complexity]
style D1 fill:#666,color:#fff
style D2 fill:#666,color:#fff
style D3 fill:#666,color:#fff
style D4 fill:#666,color:#fff
style P1 fill:#c44,color:#fff
style P2 fill:#c44,color:#fff
style P3 fill:#c44,color:#fff
style P4 fill:#c44,color:#fff
```
When documents are stored as separate objects:
* **No inherent ordering**: When was doc 3 added relative to doc 1?
* **No context continuity**: What was the state of knowledge at time T?
* **Fragmented storage**: Metadata, vectors, and content in different places
* **Sync complexity**: Keeping everything consistent is error-prone
### The Frame Solution
```mermaid theme={null}
graph LR
subgraph Timeline["Knowledge Timeline"]
F1[Frame 1<br/>t=0] --> F2[Frame 2<br/>t=1]
F2 --> F3[Frame 3<br/>t=2]
F3 --> F4[Frame 4<br/>t=3]
F4 --> F5[Frame N<br/>t=N]
end
style F1 fill:#FF9900,color:#000
style F2 fill:#FF9900,color:#000
style F3 fill:#FF9900,color:#000
style F4 fill:#FF9900,color:#000
style F5 fill:#FF9900,color:#000
```
Frames provide:
| Benefit | How Frames Deliver It |
| --------------------------- | ---------------------------------------------- |
| **Temporal ordering** | Every frame has a position in the sequence |
| **Point-in-time queries** | "What did we know at frame 100?" |
| **Atomic units** | Each frame is self-contained with all metadata |
| **Efficient deltas** | Similar consecutive frames compress well |
| **Single-file portability** | Everything serializes to one `.mv2` file |
***
## How It Works:
Here's exactly how Memvid processes and stores your content. No black boxes.
### Step 1: Frame Creation
When you call `put()`, Memvid creates a frame structure:
<AccordionGroup>
<Accordion title="Header" icon="file-code">
* **Frame ID:** `42`
* **Timestamp:** `1704067200`
* **URI:** `mv2://docs/meeting-notes`
* **Checksum:** `sha256:a1b2c3...`
</Accordion>
<Accordion title="Metadata" icon="tags">
* **Title:** "Q4 Meeting Notes"
* **Labels:** `["meeting", "q4"]`
* **Track:** "notes"
* **Custom:** `{ author: "alice" }`
</Accordion>
<Accordion title="Payload" icon="file-zipper">
zstd-compressed content bytes
</Accordion>
<Accordion title="Embeddings (optional)" icon="vector-square">
384-dim vector, quantized to int8 - only if semantic search is needed
</Accordion>
</AccordionGroup>
### Step 2: Index Updates
After frame creation, multiple indexes are updated atomically:
```mermaid theme={null}
graph LR
subgraph LexIdx["Lexical Index (BM25)"]
L1["'meeting' → 42"]
L2["'notes' → 42"]
L3["'Q4' → 42"]
end
subgraph VecIdx["Vector Index (HNSW)"]
V1["[0.1, 0.3, ...] → 42"]
end
subgraph TimeIdx["Time Index (B-Tree)"]
T1["1704067200 → 42"]
end
Frame42((Frame 42)) --> LexIdx
Frame42 --> VecIdx
Frame42 --> TimeIdx
style Frame42 fill:#FF9900,color:#000
style LexIdx fill:#2a2a2a,stroke:#4a9eff
style VecIdx fill:#2a2a2a,stroke:#9b59b6
style TimeIdx fill:#2a2a2a,stroke:#2ecc71
```
### Step 3: WAL Commit
Before returning success, the frame is committed to the Write-Ahead Log:
```mermaid theme={null}
graph LR
subgraph WAL["Write-Ahead Log"]
direction LR
E41[Entry 41<br/>Frame 41 ✓]
E42[Entry 42<br/>Frame 42 ✓]
E43[Entry 43<br/>pending...]
end
E41 --> E42 --> E43
style E41 fill:#2ecc71,color:#000
style E42 fill:#FF9900,color:#000
style E43 fill:#666,color:#fff
```
If the process crashes mid-write, the WAL ensures:
* Committed frames are recovered on next open
* Incomplete frames are discarded cleanly
* No corruption propagates to existing data
### Step 4: Segment Compaction
Periodically, frames are grouped into segments for storage efficiency:
```mermaid theme={null}
graph TB
subgraph MV2[".mv2 FILE"]
subgraph Segments
direction LR
S0["Segment 0<br/>Frames 0-99"]
S1["Segment 1<br/>Frames 100-199"]
S2["Segment 2<br/>Frames 200-299"]
end
subgraph Indexes
direction LR
TOC["TOC"]
LEX["Lexical"]
VEC["Vector"]
TIME["Time"]
WAL["WAL"]
end
end
Segments --> Indexes
style S0 fill:#FF9900,color:#000
style S1 fill:#FF9900,color:#000
style S2 fill:#FF9900,color:#000
style MV2 stroke:#FF9900,stroke-width:2px
```
***
## Traditional VectorDB: How They Store Context
To understand why frames matter, let's see how traditional vector databases handle the same data:
### The Traditional VectorDB Architecture
```mermaid theme={null}
graph TB
subgraph Client["Your Application"]
Doc[Document]
end
subgraph VectorDB["Traditional VectorDB"]
Chunker[Text Chunker]
Embedder[Embedding API]
VecStore[Vector Store]
MetaStore[Metadata Store]
DocStore[Document Store]
end
subgraph External["External Dependencies"]
API[OpenAI/Cohere API]
PG[(PostgreSQL)]
S3[(S3/Blob Storage)]
end
Doc --> Chunker
Chunker --> Embedder
Embedder --> API
API --> VecStore
VecStore --> MetaStore
MetaStore --> PG
Doc --> DocStore
DocStore --> S3
```
### Problems with This Approach
| Issue | Traditional VectorDB | Memvid Frames |
| ------------------------- | -------------------------------------------------------------- | ---------------------------------------- |
| **Storage fragmentation** | Vectors in one place, metadata in another, raw docs in a third | Everything in one frame, one file |
| **Temporal amnesia** | No concept of "when" something was added | Every frame has a timestamp and position |
| **Point-in-time queries** | Impossible or requires complex versioning | Built-in: `as_of_frame=100` |
| **Consistency** | Distributed transactions across systems | Single-file atomic writes |
| **Portability** | Export/import across multiple systems | Copy one `.mv2` file |
| **Offline operation** | Requires API access for embeddings | Local embeddings, fully offline |
| **Crash recovery** | Hope your 3 systems are all consistent | WAL ensures atomic recovery |
### What Traditional VectorDBs Actually Store
When you insert a document into Pinecone, Weaviate, or ChromaDB:
```python theme={null}
# Traditional VectorDB
vectordb.insert(
id="doc-123",
vector=[0.1, 0.2, ...], # 1536 floats
metadata={"title": "Meeting Notes"}
)
# Where's the original document?
# When was it added?
# What was the knowledge state before this?
# 🤷
```
The vector is stored. Maybe some metadata. But:
* **Original content?** Often discarded or stored separately
* **Temporal context?** Not tracked
* **Relationship to other docs?** Only through vector similarity
* **History?** Non-existent
### What Memvid Frames Store
```python theme={null}
# Memvid
mem.put(
title="Meeting Notes",
label="meeting",
metadata={},
text="Full document content here..."
)
# Stored atomically:
# ✓ Full original content (compressed)
# ✓ All metadata
# ✓ Timestamp + frame position
# ✓ Relationship to previous frames
# ✓ Crash-safe commit
# ✓ Embedding vector (optional - add when you need semantic search)
```
***
## Performance Benefits of Frame Architecture
The frame architecture isn't just conceptually cleaner. It's faster.
### Why Frames Are Fast
#### 1. Locality of Reference
Traditional systems scatter data across storage layers. Frames keep related data together:
<CardGroup>
<Card title="Traditional: 3 Round Trips" icon="network-wired">
1. Query vector index (network)
2. Fetch metadata (different server)
3. Retrieve document (blob storage)
</Card>
<Card title="Memvid: 1 Seek" icon="bolt">
1. Seek to frame offset in .mv2 file (all data co-located)
</Card>
</CardGroup>
#### 2. Segment-Based Caching
Frames group into segments that cache efficiently:
```mermaid theme={null}
graph LR
Q["Query: Q4 meetings?"] --> T["Time Index"]
T --> S3["Segment 3"]
S3 --> M["Load to Memory"]
M --> C["Cache Hit!"]
style Q fill:#4a9eff,color:#000
style S3 fill:#FF9900,color:#000
style C fill:#2ecc71,color:#000
```
<Steps>
<Step title="Time Index Lookup">
Time index identifies Segment 3 contains Q4 frames
</Step>
<Step title="Single Read">
Load Segment 3 into memory (one I/O operation)
</Step>
<Step title="Cache Ready">
All Q4 frames now cached, subsequent queries are instant
</Step>
</Steps>
#### 3. Compression Efficiency
Similar frames compress dramatically when stored together:
| Content Type | Raw Size | Frame-Compressed | Savings |
| ------------- | -------- | ---------------- | ------- |
| Chat history | 10 MB | 0.8 MB | 92% |
| Documentation | 50 MB | 4.2 MB | 91% |
| Mixed content | 100 MB | 12 MB | 88% |
This happens because:
* Sequential frames often share vocabulary (Zstd dictionary)
* Embeddings quantize to int8 (75% vector size reduction)
* Metadata schemas are consistent within segments
#### 4. Index Co-location
All indexes live in the same file, enabling compound queries without joins:
```sql theme={null}
-- Conceptual query (not actual syntax)
SELECT frames
WHERE text MATCH 'budget' -- Lexical index
AND vector SIMILAR TO query_vec -- Vector index
AND timestamp > '2024-01-01' -- Time index
-- All indexes in one file = one I/O operation
```
### Benchmark: Frame vs Traditional
Real-world comparison on 1M document corpus:
| Operation | Pinecone | ChromaDB | Memvid |
| ------------------- | -------- | -------- | ---------------- |
| Insert 1K docs | 2.3s | 4.1s | 0.8s |
| Hybrid search | N/A | N/A | 8ms |
| Point-in-time query | N/A | N/A | 9ms |
| Export all data | 45min | 12min | 0.1s (copy file) |
| Cold start | 3.2s | 1.8s | 0.05s |
| Storage size | 2.1 GB | 1.8 GB | 0.4 GB |
<Info>
**Why so fast?** Memvid doesn't need network calls, distributed coordination, or multi-system consistency. It's just reading from a well-organized file.
</Info>
***
## Frame Lifecycle
### Creation
When you add content, Memvid:
1. Generates a unique frame ID
2. Extracts and indexes text content
3. Computes embeddings (if enabled)
4. Records timestamp in the time index
5. Appends to the WAL for crash safety
6. Assigns a URI (`mv2://track/title`)
### Retrieval
When you search or view:
1. Query hits the appropriate index (lexical, vector, or time)
2. Frame metadata is loaded from the TOC
3. Payload is decompressed and returned
4. Access is logged for analytics
### Deletion
"Deleted" frames aren't physically removed. They're **tombstoned**:
```python theme={null}
# Mark frame as deleted
mem.delete(frame_id=42)
# Frame still exists but won't appear in searches
# Use vacuum to physically reclaim space
```
```bash theme={null}
memvid doctor knowledge.mv2 --vacuum
```
***
## Frame IDs vs URIs
Every frame has two identifiers:
| Identifier | Format | Example | Use Case |
| ------------ | ------- | ------------------- | ------------------- |
| **Frame ID** | Integer | `124` | Internal reference |
| **URI** | String | `mv2://docs/api.md` | Human-readable path |
```python theme={null}
# Access by frame ID
frame = mem.frame(124)
# Access by URI
frame = mem.frame('mv2://docs/api.md')
```
***
## Best Practices
### Frame Sizing
* **Small frames** (under 4KB): Great for chat messages, notes
* **Medium frames** (4KB - 1MB): Documents, articles
* **Large frames** (over 1MB): PDFs, images, audio
### Batch Ingestion
Use `put_many()` for bulk ingestion (100-200x faster):
```python theme={null}
docs = [
{'text': 'Content 1', 'title': 'Doc 1', 'label': 'docs'},
{'text': 'Content 2', 'title': 'Doc 2', 'label': 'docs'},
# ... thousands more
]
mem.put_many(docs)
```
### Tombstone Management
Periodically vacuum to reclaim space:
```bash theme={null}
# Check how much space can be reclaimed
memvid stats knowledge.mv2
# Reclaim deleted frame space
memvid doctor knowledge.mv2 --vacuum
```
***
## Next Steps
<CardGroup>
<Card title="Memory Architecture" icon="database" href="/concepts/memory-architecture">
See how frames fit into the file structure
</Card>
<Card title="Time Index" icon="clock" href="/file-format/time-index-track">
Learn about temporal ordering
</Card>
</CardGroup>
# Glossary
Source: https://docs.memvid.com/introduction/glossary
Complete reference of Memvid terminology, architecture components, and concepts
This glossary provides definitions for all key terms, components, and concepts in the Memvid ecosystem. Whether you're just getting started or diving deep into the architecture, this reference will help you understand how everything fits together.
***
## Architecture Components
### Memvid Core
The heart of Memvid, written in Rust. `memvid-core` is the foundational library that implements all core functionality:
* **File format handling** - Reading/writing `.mv2` files
* **Indexing engines** - Lexical (Tantivy), vector (HNSW), and hybrid search
* **WAL management** - Write-ahead logging for crash safety
* **Enrichment pipeline** - Background processing for embeddings and extraction
* **Memory management** - Frame storage, versioning, and lifecycle
The core is compiled to native binaries and exposed through language bindings (Node.js, Python) for cross-platform use.
### CLI (Command Line Interface)
The `memvid` CLI tool provides direct access to all Memvid operations from your terminal:
```bash theme={null}
# Create a new memory file
memvid create my-knowledge.mv2
# Add content
memvid put my-knowledge.mv2 --input document.pdf
# Search
memvid find my-knowledge.mv2 --query "machine learning"
# Ask questions
memvid ask my-knowledge.mv2 --question "What is the main thesis?"
```
The CLI is ideal for scripting, automation, and quick interactions without writing code.
### SDKs (Software Development Kits)
Language-specific libraries that wrap the Memvid core for seamless integration:
<CardGroup>
<Card title="Node.js SDK" icon="node-js" href="/sdks/node">
`@memvid/sdk` - Native N-API bindings for Node.js applications
</Card>
<Card title="Python SDK" icon="python" href="/sdks/python">
`memvid-sdk` - PyO3 bindings for Python applications
</Card>
</CardGroup>
Both SDKs provide identical APIs:
* `create()` / `use()` - Create or open memory files
* `put()` / `put_many()` - Insert documents
* `find()` - Search with various modes
* `ask()` - AI-powered Q\&A
* `timeline()` - Browse insertion history
***
## File Format
### MV2 (Memory File)
The `.mv2` file extension represents a Memvid memory file. It's a single, self-contained binary file that stores:
* All your documents and data (frames)
* Search indices (lexical, vector, temporal)
* Metadata and checksums
* Write-ahead log for crash recovery
**Key characteristics:**
* **Portable** - Copy, move, or share as a single file
* **Serverless** - No database server required
* **Crash-safe** - WAL ensures data integrity
* **Deterministic** - Reproducible builds for verification
```mermaid theme={null}
flowchart TB
subgraph mv2["MV2 File Structure"]
direction TB
A["Header (4 KB)"]
B["WAL (Write-Ahead Log)"]
C["Frame Data"]
D["Indices (Lex, Vec, Time, Sketch)"]
E["Manifests"]
F["Footer (TOC + Checksums)"]
A --> B --> C --> D --> E --> F
end
```
### MV2E (Encrypted Memory File)
The `.mv2e` extension indicates an encrypted memory file (Capsule). Uses:
* **Argon2** for password-based key derivation
* **AES-GCM** for authenticated encryption
Requires a password to open; transparent once unlocked.
***
## Core Concepts
### Frame
The atomic unit of data in Memvid. Every piece of content you store becomes a frame.
**Properties:**
| Property | Description |
| ---------- | ------------------------------------------ |
| `frame_id` | Unique monotonic identifier (u64) |
| `content` | The actual text/data stored |
| `metadata` | Tags, timestamps, source info |
| `status` | Active, Superseded, or Deleted |
| `role` | Document, DocumentChunk, or ExtractedImage |
**Lifecycle:**
1. **Insert** - Frame created with `frame_id`, status = Active
2. **Update** - Original marked Superseded, new frame created
3. **Delete** - Status changed to Deleted (soft delete)
Frames are immutable once committed - updates create new frames.
### Memory
A "memory" in Memvid refers to the runtime instance managing an `.mv2` file. When you open a memory file, you get a Memory object that handles:
* Reading and writing frames
* Managing indices
* Coordinating search
* Handling commits and checkpoints
```javascript theme={null}
// Open a memory
const mem = await memvid.use("knowledge.mv2");
// The 'mem' object is your Memory instance
await mem.put({ text: "Hello world" });
await mem.find({ query: "hello" });
```
### Commit
The process of persisting pending changes to the `.mv2` file:
1. WAL entries written to disk
2. Indices updated
3. Footer rewritten with new checksums
4. File synced to storage
Commits happen automatically on close, or can be triggered manually for durability guarantees.
### Checkpoint
A checkpoint purges committed WAL entries and updates the header:
* Frees WAL space for new writes
* Marks transactions as permanently durable
* Triggered automatically when WAL reaches 75% capacity
***
## Search & Retrieval
### Lexical Search
Traditional keyword-based search using the BM25 ranking algorithm:
* **How it works**: Builds an inverted index of terms, scores documents by term frequency and inverse document frequency
* **Best for**: Exact matches, specific keywords, technical terms
* **Engine**: Tantivy (Rust full-text search library)
```bash theme={null}
memvid find data.mv2 --query "API authentication" --mode lex
```
### Semantic Search (Vector Search)
Meaning-based search using embeddings:
* **How it works**: Converts text to vector embeddings, finds similar vectors using cosine similarity
* **Best for**: Conceptual queries, finding related content, natural language questions
* **Index**: HNSW (Hierarchical Navigable Small World) graph
```bash theme={null}
memvid find data.mv2 --query "how to secure endpoints" --mode vec
```
### Hybrid Search
Combines lexical and semantic search for best results:
* Runs both search types in parallel
* Normalizes scores from each
* Merges using RRF (Reciprocal Rank Fusion)
* Returns unified ranked results
```bash theme={null}
memvid find data.mv2 --query "authentication best practices" --mode hybrid
```
### Sketch Pre-filtering
Ultra-fast candidate filtering before expensive ranking:
* **SimHash**: 64-bit locality-sensitive hash for quick similarity checks
* **Term Filter**: Compact bitset for query term overlap
* **Top Terms**: Hashed IDs of highest-weight terms
Reduces search candidates by 10-100x, enabling sub-millisecond filtering on million-frame memories.
***
## Indexing
### Lex Index
The lexical search index built on Tantivy:
* Tokenizes text into terms
* Builds inverted index (term → document list)
* Supports field-specific queries (title, content, tags)
* Deterministic chunking for reproducibility
### Vec Index
The vector/embedding search index:
* Stores document embeddings
* Uses HNSW graph for approximate nearest neighbor search
* Supports multiple embedding models (BGE, Nomic, OpenAI)
* Optional product quantization for compression
### Time Index
Temporal index for frame ordering:
* Tracks insertion timestamps
* Enables range queries (since/until)
* Powers `timeline()` navigation
* Supports forward and reverse traversal
### Sketch Track
Per-frame micro-indices for fast filtering:
| Variant | Size | Use Case |
| ------- | -------------- | ------------------ |
| Small | 32 bytes/frame | Memory-constrained |
| Medium | 64 bytes/frame | Balanced (default) |
| Large | 96 bytes/frame | Maximum precision |
***
## Enrichment
### Enrichment Pipeline
Background processing that enhances frames after insertion:
**Phases:**
1. **Searchable** (instant) - Skim text extracted, basic indexing
2. **Enriched** (background) - Full text, embeddings, memory cards, entities
The two-phase approach means search works immediately while richer features process in the background.
### Memory Cards
Structured units of extracted knowledge:
| Field | Description |
| ------------------ | ---------------------------------------------------- |
| `kind` | Fact, Preference, Event, Profile, Relationship, Goal |
| `content` | The extracted information |
| `polarity` | Positive, Negative, or Neutral |
| `version_relation` | Sets, Updates, Extends, or Retracts |
Memory cards enable semantic querying beyond raw text search.
### Entity Extraction (NER)
Named Entity Recognition identifies and links entities:
* **Types**: Person, Organization, Location, Date, Money, URL, etc.
* **Model**: DistilBERT-NER (ONNX)
* **Output**: Entities with confidence scores and frame references
### Logic Mesh
Entity-relationship graph connecting extracted entities:
* Bidirectional graph structure
* Nodes: Entities with types and mentions
* Edges: Relationships with confidence
* Enables: "follow" queries for fact traversal
***
## Persistence
### WAL (Write-Ahead Log)
Embedded circular buffer ensuring crash safety:
```mermaid theme={null}
flowchart LR
subgraph header["WAL Entry"]
direction LR
A["seq (8B)"] --- B["len (4B)"] --- C["reserved"] --- D["checksum"]
end
subgraph body[" "]
E["payload (variable)"]
end
header --> body
```
* **Purpose**: Records mutations before they're applied
* **Checksum**: BLAKE3 hash for integrity verification
* **Recovery**: Replays uncommitted entries after crash
* **Size**: Configurable (64 KB to 64 MB)
### Header
Fixed 4 KB structure at file offset 0:
* Magic bytes (`MV2\0`)
* Spec and format versions
* WAL offset and size
* Footer offset pointer
### Footer
Variable-length CBOR-serialized metadata at end of file:
* Table of Contents (TOC)
* Manifest pointers
* Segment catalog
* Checksums for validation
### TOC (Table of Contents)
Master index structure in the footer:
* Lists all frames with metadata
* References to index manifests (lex, vec, time)
* Segment catalog for published indices
***
## Capacity & Licensing
### Ticket
Signed proof of capacity grant:
* **Issuer**: Authority that granted the capacity
* **Sequence**: Monotonic identifier
* **Capacity**: Bytes allowed
* **Signature**: ED25519 digital signature
Tickets are validated cryptographically and cannot be forged.
### Capacity Tiers
Storage limits based on plan:
| Tier | Capacity | Memory Files | Queries/Month |
| ---------- | --------- | ------------ | ------------- |
| Free | 50 MB | — | — |
| Starter | 25 GB | 5 | 250k |
| Pro | 125 GB | 25 | 20M |
| Enterprise | Unlimited | Unlimited | Unlimited |
***
## Embedding Models
### Local Models
Run entirely on your machine:
| Model | Dimensions | Speed | Quality |
| ----------- | ---------- | ------ | ------- |
| BGE-Small | 384 | Fast | Good |
| BGE-Base | 768 | Medium | Better |
| Nomic-Embed | 768 | Medium | Better |
### Cloud Models
API-based embedding providers:
| Provider | Model | Dimensions |
| -------- | ---------------------- | ---------- |
| OpenAI | text-embedding-3-small | 1536 |
| OpenAI | text-embedding-3-large | 3072 |
| NVIDIA | NV-Embed-v2 | 4096 |
### CLIP Models
For visual/image embeddings:
| Model | Dimensions | Use Case |
| ---------- | ---------- | ------------------------- |
| SigLIP | 768 | High-quality image search |
| MobileCLIP | 384 | Fast, lightweight |
***
## Operations
### put / put\_many
Insert documents into memory:
```javascript theme={null}
// Single document
const frameId = await mem.put({ text: "Hello world" });
// Batch insert
const frameIds = await mem.put_many([
{ text: "Doc 1" },
{ text: "Doc 2" },
{ file: "document.pdf" }
]);
```
### find
Search the memory:
```javascript theme={null}
const results = await mem.find({
query: "machine learning",
k: 10,
mode: "hybrid",
snippet_chars: 200
});
```
### ask
AI-powered question answering:
```javascript theme={null}
const answer = await mem.ask({
question: "What are the key findings?",
use_model: "openai"
});
```
Returns synthesized answer with source citations.
### timeline
Browse frames by insertion order:
```javascript theme={null}
// Recent frames
const recent = await mem.timeline({ limit: 20 });
// Oldest first
const oldest = await mem.timeline({ limit: 20, reverse: true });
```
### seal
Close and commit the memory file:
```javascript theme={null}
await mem.seal();
```
Commits pending changes and releases file lock.
***
## Feature Flags
Cargo features that enable optional functionality:
| Feature | Description |
| ------------------- | ------------------------------- |
| `lex` | Tantivy full-text search |
| `vec` | Vector embeddings + HNSW |
| `clip` | CLIP visual search |
| `whisper` | Audio transcription |
| `encryption` | Capsule encryption |
| `logic_mesh` | Entity-relationship graph + NER |
| `replay` | Session recording/replay |
| `temporal_track` | Temporal mention tracking |
| `parallel_segments` | Parallel index building |
***
## Common Workflows
### Ingestion → Search → Ask
```
1. Create memory → memvid.create("data.mv2")
2. Insert documents → mem.put({ file: "docs/*.pdf" })
3. Wait for enrichment → (automatic background processing)
4. Search → mem.find({ query: "..." })
5. Ask questions → mem.ask({ question: "..." })
6. Close → mem.seal()
```
### Enrichment Pipeline
```
1. put() → Frame enters Searchable state
2. Background → Worker extracts full text
3. Embedding → Generate vector embeddings
4. Extraction → Memory cards, entities
5. Indexed → Frame now fully Enriched
```
***
## See Also
<CardGroup>
<Card title="Five-Minute Guide" icon="rocket" href="/quickstart/five-minute-guide">
Get started with Memvid in minutes
</Card>
<Card title="Architecture Overview" icon="sitemap" href="/architecture/overview">
Deep dive into how Memvid works
</Card>
<Card title="CLI Reference" icon="terminal" href="/cli/index">
Complete CLI command reference
</Card>
<Card title="SDK Comparison" icon="code" href="/sdks/feature-matrix">
Compare features across SDKs
</Card>
</CardGroup>
# The Memvid Approach
Source: https://docs.memvid.com/introduction/the-memvid-approach
A portable memory architecture with flexible embedding options
<Note>
**Memvid is not another vector database.** It's a portable, single-file memory system with flexible embedding options. Use local models for privacy or connect to external providers like OpenAI, NVIDIA, and more.
</Note>
***
## The Problem with Traditional RAG
Most memory systems today follow the same pattern:
```mermaid theme={null}
flowchart LR
A[Your Data] --> B[Embedding API]
B --> C[Vector Database]
C --> D[Similarity Search]
D --> E[Results]
```
This approach has serious limitations:
| Problem | Impact |
| ----------------------- | -------------------------------------------- |
| **API dependency** | Can't work everyhwere, costs money per query |
| **Embedding drift** | Model updates break your index |
| **No exact matching** | "Error 404" doesn't find "error 404" |
| **Black box relevance** | Hard to debug why results are wrong |
| **Cold start** | Need to embed everything before first search |
***
## The Memvid Innovation
Memvid takes a completely different approach:
```mermaid theme={null}
flowchart LR
A[Your Data] --> B[Frames]
B --> C[Multiple Indices]
C --> D[Smart Retrieval]
D --> E[Results]
C -.- F[BM25 Lexical]
C -.- G[Vector Embeddings]
C -.- H[SimHash Dedup]
C -.- I[Time Index]
```
**Flexible embedding options.** Memvid combines multiple search strategies:
* **BM25 lexical search** - Battle-tested, fast, explainable
* **Vector embeddings** - Local models (Nomic, BGE, GTE) or external APIs (OpenAI, NVIDIA)
* **SimHash deduplication** - Find near-duplicates instantly
* **Time-aware retrieval** - When something was added matters
* **Hybrid search** - Combines lexical + semantic for best results
***
## How Frames Change Everything
Traditional systems store "chunks" - arbitrary text splits. Memvid stores **Frames** - structured units of memory:
```mermaid theme={null}
flowchart LR
subgraph Frame[Smart Frame]
A[content]
B[uri]
C[timestamp]
D[content_hash]
E[simhash]
F[metadata]
G[entities]
H[connections]
end
```
| Field | Example |
| -------------- | ----------------------------------------- |
| `content` | "The quarterly revenue exceeded \$10M..." |
| `uri` | reports/q4-2024.pdf |
| `timestamp` | 2024-12-15T10:30:00Z |
| `content_hash` | blake3(...) |
| `simhash` | 0x8f3a2b1c... |
Each frame knows:
* **What it contains** (content + hash)
* **Where it came from** (URI + metadata)
* **When it was created** (timestamp)
* **What it's similar to** (SimHash)
* **What entities it mentions** (Logic Mesh)
* **How it connects to other frames** (relationships)
Frames are the foundation of Memvid's multi-index approach, enabling both lexical and semantic search.
***
## Multiple Indices, Flexible Providers
Every `.mv2` file contains multiple search indices:
| Index | Purpose | How It Works |
| ------------------ | ------------------------ | --------------------------------------- |
| **Lexical (BM25)** | Keyword search | TF-IDF scoring, exact matches |
| **Vector** | Semantic similarity | Local or external embedding models |
| **SimHash** | Near-duplicate detection | 64-bit locality-sensitive hash |
| **Time** | Temporal queries | B-tree on timestamps |
| **Logic Mesh** | Entity-relationship | Triple store (subject-predicate-object) |
**Embedding flexibility** - Choose what works for your use case:
* **Local models** (Nomic, BGE-small, BGE-base, GTE-large) - Fast, private, works offline
* **External APIs** (OpenAI, NVIDIA) - Higher quality, no local compute needed
***
## Choosing Your Embedding Provider
Memvid supports multiple embedding providers. Choose based on your needs:
| Provider | Models | Best For |
| ------------------- | ------------------------------------- | ---------------------------------- |
| **Local (Default)** | bge-small, bge-base, nomic, gte-large | Privacy, offline use, no API costs |
| **OpenAI** | openai, openai-small, openai-ada | Highest quality, multilingual |
| **NVIDIA** | nvidia | Enterprise, high throughput |
```bash theme={null}
# Use local embeddings (default - works offline)
memvid create notes.mv2
memvid put notes.mv2 --input docs/ --embedding
# Use OpenAI embeddings (requires OPENAI_API_KEY)
memvid put notes.mv2 --input docs/ --embedding -m openai
# Use specific local model
memvid put notes.mv2 --input docs/ --embedding -m bge-base
```
***
## Logic Mesh: Relationships Without ML
Traditional systems need expensive NER models for entity extraction. Memvid's Logic Mesh uses:
* **Rule-based extraction** - Fast, free, no API
* **Pattern matching** - Dates, emails, numbers
* **Co-occurrence** - Entities mentioned together
* **Temporal reasoning** - When facts changed
```bash theme={null}
# Enable Logic Mesh during ingestion
memvid put memory.mv2 --input docs/ --logic-mesh
# Query relationships
memvid follow traverse memory.mv2 --start "John" --link "works_at"
# Result: John → works_at → Acme Corp (since 2024-01-15)
```
***
## SimHash: Smart Deduplication
Instead of comparing embeddings, Memvid uses **SimHash** - a locality-sensitive hash that detects near-duplicates:
```
Document A: "The quick brown fox jumps over the lazy dog"
Document B: "The quick brown fox leaps over the lazy dog"
SimHash A: 0x8f3a2b1c4d5e6f70
SimHash B: 0x8f3a2b1c4d5e6f71
Hamming distance: 1 bit → Near duplicate detected!
```
Benefits:
* **Instant** - O(1) comparison
* **No API calls** - Computed locally
* \*\*Works anywhere \*\*- No internet needed
* **Deterministic** - Same input = same hash
***
## Real-World Performance
Memvid's local-first approach delivers fast performance:
| Operation | Traditional Vector DB | Memvid (Local Embeddings) |
| ----------------------- | --------------------- | ------------------------- |
| Ingest 1,000 docs | 5-10 minutes | **30 seconds** |
| First search | After embedding | **Instant** |
| Offline search | ❌ | ✅ |
| Cost per query | \$0.0001+ | **\$0** |
| Single file portability | ❌ | ✅ |
***
## Getting Started
Get started with Memvid in minutes:
```bash theme={null}
# Install
npm install -g memvid-cli
# Create memory
memvid create my-memory.mv2
# Add your documents (uses local embeddings by default)
memvid put my-memory.mv2 --input ./documents/
# Search with hybrid lexical + semantic
memvid find my-memory.mv2 --query "quarterly report"
# Ask questions
memvid ask my-memory.mv2 --question "What were the Q4 results?"
```
That's it. Local embeddings work out of the box, no API keys required.
***
## Switching Embedding Providers
Need higher quality embeddings? Switch to an external provider:
```bash theme={null}
# Rebuild vector index with OpenAI embeddings
memvid doctor my-memory.mv2 --rebuild-vec-index -m openai
# Search with semantic mode
memvid find my-memory.mv2 --query "financial performance" --mode sem
# Or use hybrid (lexical + semantic)
memvid find my-memory.mv2 --query "financial performance" --mode hybrid
```
Your existing data stays intact. Only the vector index is rebuilt.
***
## Next Steps
<CardGroup>
<Card title="5-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Get up and running fast
</Card>
<Card title="Frame Architecture" icon="cube" href="/introduction/frames">
Deep dive into frames
</Card>
<Card title="Logic Mesh" icon="diagram-project" href="/concepts/graph-search">
Entity-relationship graphs
</Card>
<Card title="Deduplication" icon="clone" href="/concepts/deduplication">
SimHash and content hashing
</Card>
</CardGroup>
# Getting Started
Source: https://docs.memvid.com/introduction/welcome
Everything you need to start building with Memvid
Welcome to Memvid, the portable AI memory system that puts you in control. This guide covers everything you need to start building intelligent applications.
***
## What Makes Memvid Different
### Single-File Architecture
Every `.mv2` file is completely self-contained:
* **Your data**: Documents, text, images, audio, videos
* **Embeddings**: Vector representations for semantic search
* **Indices**: BM25 lexical index and vector index
* **Time index**: Temporal ordering for timeline queries
* **Write-ahead log**: Crash-safe transaction logging
Share a memory by sharing a file. It works anywhere: local disk, USB drive, cloud storage, or Git repository.
### Hybrid Search Engine
Memvid combines the best of two search paradigms:
```bash theme={null}
# Hybrid search (recommended)
memvid find knowledge.mv2 --query "user authentication" --mode auto
# Lexical search - exact keyword matching
memvid find knowledge.mv2 --query "authentication" --mode lex
# Semantic search - conceptual understanding
memvid find knowledge.mv2 --query "how do users log in" --mode sem
```
### Lightning Performance
Built in Rust from the ground up for maximum performance:
| Operation | Time |
| ----------------- | ------------- |
| Search (50K docs) | \< 20ms |
| Bulk ingestion | 150+ docs/sec |
| Frame append | \< 0.1ms |
## Quick Start
### 1. Install the CLI
```bash theme={null}
npm install -g memvid-cli
```
### 2. Create Your First Memory
```bash theme={null}
# Create a new memory file (1 GB capacity by default)
memvid create my-knowledge.mv2
# Ingest documents with vector compression
memvid put my-knowledge.mv2 --input ./documents/ --vector-compression
# Search your knowledge
memvid find my-knowledge.mv2 --query "your search query"
```
### 3. Build Your Application
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { use } from '@memvid/sdk';
// Open your memory
const mem = await use('basic', 'my-knowledge.mv2', { readOnly: true });
// Search
const results = await mem.find('machine learning', { k: 10 });
results.hits.forEach(hit => {
console.log(`${hit.score.toFixed(2)}: ${hit.title}`);
});
// Ask questions
const answer = await mem.ask('What are the key concepts?');
console.log(answer.answer);
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import use
# Open your memory
mem = use('basic', 'my-knowledge.mv2', read_only=True)
# Search
results = mem.find('machine learning', k=10)
for hit in results.get('hits', []):
print(f"{hit['score']:.2f}: {hit['title']}")
# Ask questions with AI synthesis
answer = mem.ask('What are the key concepts?')
print(answer.get('answer'))
# Always close when done
mem.close()
```
</Tab>
<Tab title="LangChain">
```python theme={null}
from memvid_sdk import use
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
# Open with LangChain adapter
mem = use('langchain', 'my-knowledge.mv2', read_only=True)
retriever = mem.as_retriever(k=5)
# Create QA chain
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o"),
retriever=retriever
)
result = qa.run("What are the main concepts?")
print(result)
```
</Tab>
</Tabs>
***
## Key Features in v2
* **Frame Architecture**: Video-inspired append-only storage for crash safety and time-travel queries
* **Time Index Track**: Query documents by temporal order
* **Embedded WAL**: Crash-safe transactions with automatic recovery
* **Parallel Ingestion**: Multi-threaded document processing
* **Framework Adapters**: Native integrations for LangChain, LlamaIndex, AutoGen, and more
***
## Next Steps
<CardGroup>
<Card title="Quickstart Guide" icon="rocket" href="/quickstart/cli-to-dashboard">
Build a complete workflow with the CLI
</Card>
<Card title="Frame Architecture" icon="film" href="/introduction/frames">
Understand the video-inspired storage model
</Card>
<Card title="CLI Reference" icon="terminal" href="/cli/create-and-put">
Complete reference for all CLI commands
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Full SDK documentation with examples
</Card>
</CardGroup>
***
## Getting Help
* **[FAQ](/faq/general)**: Answers to common questions
* **[Troubleshooting](/troubleshooting/cli)**: Solutions to common issues
* **[GitHub Issues](https://github.com/memvid/memvid/issues)**: Report bugs and request features
We're excited to see what you build with Memvid!
# Why Memvid?
Source: https://docs.memvid.com/introduction/why-memvid
How Memvid compares to vector databases and why developers choose single-file AI memory
## The RAG infrastructure problem
Building AI applications with memory typically requires:
1. **A vector database** (Pinecone, Weaviate, Qdrant, Milvus)
2. **An embedding API** (OpenAI, Cohere, or self-hosted)
3. **A backend service** to coordinate queries
4. **Operational overhead** for backups, scaling, monitoring
For a simple "search my documents" feature, you're suddenly managing distributed infrastructure.
**Memvid takes a different approach**: Smart Frames in a single file. 11x faster search.
***
## The single-file advantage
### What's in an `.mv2` file?
```mermaid theme={null}
flowchart TB
subgraph MV2["knowledge.mv2"]
direction TB
H["4KB Header<br/>Metadata, Version, Checksums"]
W["Embedded WAL (1-64 MB)<br/>Crash Recovery, Transaction Log"]
D["Frame Segments<br/>Documents, Text, Files"]
L["Lexical Index<br/>Tantivy/BM25 Keyword Search"]
V["Vector Index<br/>HNSW Semantic Embeddings"]
T["Time Index<br/>Timeline & Time-Travel Queries"]
C["Table of Contents<br/>Fast Navigation"]
F["48B Footer<br/>Recovery Pointer"]
H --> W --> D --> L --> V --> T --> C --> F
end
style H fill:#FF9900,color:#000
style W fill:#FFB84D,color:#000
style D fill:#3b82f6,color:#fff
style L fill:#10b981,color:#fff
style V fill:#8b5cf6,color:#fff
style T fill:#ef4444,color:#fff
style C fill:#64748b,color:#fff
style F fill:#1e293b,color:#fff
```
<Info>
**Everything is self-contained.** No sidecar files. No auxiliary databases. No cloud sync. Just one portable `.mv2` file.
</Info>
### What this enables
<CardGroup>
<Card title="True Portability" icon="suitcase">
Copy your knowledge base to a USB drive. Email it. Deploy it anywhere. It just works.
</Card>
<Card title="Offline First" icon="wifi-slash">
No internet required. No API keys for basic operations. Works on airplanes.
</Card>
<Card title="Zero Ops" icon="server">
No databases to manage. No Docker containers. No cloud bills. Just a file.
</Card>
<Card title="Privacy by Default" icon="lock">
Your data never leaves your machine unless you explicitly send it somewhere.
</Card>
</CardGroup>
***
## Head-to-head comparison
### Memvid vs. Pinecone
**Benchmark results (1,000 documents):**
| Metric | Memvid | Pinecone | Winner |
| ------------------ | ------------ | -------- | ------------ |
| **Setup** | 145ms | 7.4s | Memvid (51x) |
| **Search latency** | 24ms | 267ms | Memvid (11x) |
| **Storage** | 4.9 MB local | Cloud | Memvid |
| **API calls** | 0 | 1,005 | Memvid |
| Aspect | Memvid | Pinecone |
| ----------------------- | --------------------------------------------------- | --------------------------- |
| **Deployment** | Single file, runs anywhere | Cloud-only SaaS |
| **Setup time** | 145ms | 7.4 seconds + account setup |
| **Offline support** | Full functionality | None |
| **Data location** | Your machine | Pinecone's cloud |
| **Search modes** | Smart Frames (Lexical + Vector + Temporal + Entity) | Vector only |
| **Cost (100K vectors)** | Free | \$70+/month |
| **Scaling** | Vertical (bigger machine) | Horizontal (managed) |
<Info>
**Why is Memvid search 11x faster?** No network round-trips. Pinecone requires: (1) API call to embed your query, (2) API call to search vectors. Memvid searches locally with Smart Frames.
</Info>
***
### Memvid vs. ChromaDB
| Aspect | Memvid | ChromaDB |
| ------------------------ | --------------------------------------------------- | ------------------------ |
| **Storage** | Single `.mv2` file | SQLite + multiple files |
| **Portability** | Copy one file | Copy directory structure |
| **Crash recovery** | Embedded WAL, automatic | Manual recovery |
| **Search modes** | Smart Frames (Lexical + Vector + Temporal + Entity) | Vector only |
| **Built-in RAG** | `.ask()` method | Build with LangChain |
| **Time-travel queries** | Yes | No |
| **Entity extraction** | Built-in (auto-tagging, triplets) | No |
| **Visual search (CLIP)** | Yes | No |
***
### Memvid vs. Weaviate
| Aspect | Memvid | Weaviate |
| ----------------------- | --------------------------------------------------- | ----------------------------- |
| **Deployment** | Single file | Docker/Kubernetes required |
| **Setup** | `pip install` (seconds) | Docker compose, configuration |
| **Search modes** | Smart Frames (Lexical + Vector + Temporal + Entity) | Hybrid (BM25 + Vector) |
| **Time-travel queries** | Yes | No |
| **Entity extraction** | Built-in | No |
| **GraphQL API** | No (SDK only) | Yes |
| **Multi-tenancy** | Separate files | Built-in |
| **Schema** | Schema-free | Schema required |
***
### Memvid vs. pgvector
| Aspect | Memvid | pgvector |
| ----------------------- | --------------------------------------------------- | ---------------------------- |
| **Database** | None required | PostgreSQL required |
| **SQL queries** | No | Yes |
| **Portability** | Single file | Database backup/restore |
| **Search modes** | Smart Frames (Lexical + Vector + Temporal + Entity) | Vector + manual full-text |
| **Time-travel queries** | Yes | No |
| **Entity extraction** | Built-in | No |
| **Setup** | `pip install` (seconds) | Postgres + extension install |
***
## Smart Frame capabilities
Features you won't find in typical vector databases:
### Time-travel queries
Search your memory as it existed at any point in time:
```python theme={null}
# What did we know about the budget last week?
results = mem.find("budget", as_of_timestamp=1704067200)
# What was in the knowledge base before we added the Q4 report?
results = mem.find("revenue", as_of_frame=100)
```
### Visual search with CLIP
Search images and PDF pages by visual content:
```python theme={null}
from memvid_sdk.clip import get_clip_provider
clip = get_clip_provider('local') # No API keys needed
embedding = clip.embed_text("pie chart showing market share")
# Find visually similar content
results = mem.visual_search(embedding, k=10)
```
### Entity extraction (Logic Mesh)
Automatically extract and traverse relationships:
```python theme={null}
from memvid_sdk.entities import get_entity_extractor
ner = get_entity_extractor('openai', entity_types=['PERSON', 'COMPANY', 'PRODUCT'])
entities = ner.extract(document_text)
# Traverse the entity graph
related = mem.follow("Microsoft", link="acquired", hops=2)
```
### Built-in RAG with citations
Ask questions and get sourced answers without building chains:
```python theme={null}
answer = mem.ask("What was our Q4 revenue?")
print(answer["answer"]) # "Q4 revenue was $2.4M, up 15% YoY..."
print(answer.get("sources", [])[:1]) # [{"title": "...", "uri": "...", ...}]
```
### Embedded crash recovery
The Write-Ahead Log (WAL) ensures you never lose data:
```python theme={null}
# Even if power fails here...
mem.put(title="Important", text="Critical data...")
# ...the data is recoverable on next open
mem = use('basic', 'knowledge.mv2') # Auto-recovers uncommitted writes
```
***
## Get started
<CardGroup>
<Card title="5-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Build your first AI memory
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Complete reference
</Card>
</CardGroup>
# Memvid v1 (QR Code Memory) – Deprecated
Source: https://docs.memvid.com/memvid-v1-deprecation
Memvid v1 is deprecated. QR codes are no longer part of Memvid.
<Warning>
**Memvid v1 is deprecated.**
If you are referencing QR-code-based memory for Memvid, you are using outdated information.
QR codes are no longer part of Memvid.
</Warning>
## What changed
* **Memvid v1** used QR-encoded video frames
* **Memvid v2** uses `.mv2` binary memory files
* QR codes were **fully removed**
## Correct mental model
**Memvid = portable AI memory file (`.mv2`)**
* ❌ Not QR codes
* ❌ Not video hacks
## What to do
<Steps>
<Step title="Stop using QR codes">
Stop using or referencing QR codes for Memvid
</Step>
<Step title="Update your content">
Update any docs, blog posts, or code examples
</Step>
<Step title="Use .mv2 files">
Use `.mv2` files for all Memvid memory
</Step>
</Steps>
***
<Note>
This page exists to prevent confusion for users, search engines, and AI agents.
</Note>
# Node.js Examples
Source: https://docs.memvid.com/node-sdk/examples
Complete TypeScript examples for search, ask, and verification
<Warning>
**`create()` vs `use()`**: Use `create()` for **new** files only. Use `use()` to open **existing** files. Calling `create()` on an existing file will overwrite it!
</Warning>
## Search and Ask
```typescript theme={null}
import { use, create, Memvid } from '@memvid/sdk';
async function searchExample() {
// Open existing memory
const mv: Memvid = await use('basic', 'notes.mv2', { readOnly: true });
// Search with options
const hits = await mv.find('deterministic', { k: 5, snippetChars: 200 });
console.log(`Found ${hits.total_hits} results`);
for (const hit of hits.hits) {
console.log(` ${hit.score.toFixed(2)}: ${hit.title}`);
}
// Ask with LLM synthesis
const answer = await mv.ask('How does the WAL work?', {
model: 'openai:gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY,
k: 10
});
console.log(answer.answer);
// Timeline queries
const timeline = await mv.timeline({
since: 1730000000,
until: 1730003600,
limit: 20
});
console.log(`Timeline entries: ${timeline.length}`);
}
```
## Creating and Ingesting
```typescript theme={null}
import { create, use, Memvid } from '@memvid/sdk';
import { existsSync } from 'fs';
async function ingestExample() {
const path = 'knowledge.mv2';
// IMPORTANT: create() for NEW files, use() for EXISTING files
const mv: Memvid = existsSync(path)
? await use('basic', path) // Open existing
: await create(path, 'basic'); // Create new
// Enable lexical search
await mv.enableLex();
// Add documents with vector compression (for text content)
await mv.put({
title: 'API Documentation',
label: 'docs',
text: 'The API provides endpoints for search and retrieval...',
vectorCompression: true
});
// Add from file
await mv.put({
title: 'User Guide',
label: 'docs',
file: 'guide.pdf',
vectorCompression: true
});
await mv.seal();
}
```
## Verification and Repair
```typescript theme={null}
import { use } from '@memvid/sdk';
async function maintenanceExample() {
// Verify integrity
const result = await use.verify('notes.mv2', { deep: true }) as {
overall_status: string;
checks: unknown[];
};
if (result.overall_status === 'passed') {
console.log('File is valid');
} else {
console.log('Issues found:', result.checks);
}
// Repair if needed
const report = await use.doctor('notes.mv2', {
rebuildTimeIndex: true,
rebuildLexIndex: true,
vacuum: true
});
console.log('Repair complete:', report);
}
```
## Error Handling
```typescript theme={null}
import { use, CapacityExceededError, LockedError } from '@memvid/sdk';
async function errorHandlingExample() {
try {
const mv = await use('basic', 'knowledge.mv2');
await mv.put({
title: 'Large Document',
label: 'docs',
file: 'large-file.pdf',
vectorCompression: true
});
await mv.seal();
} catch (error) {
if (error instanceof CapacityExceededError) {
console.log('MV001: Capacity exceeded - upgrade plan');
} else if (error instanceof LockedError) {
console.log('MV007: File is locked by another process');
} else if (error instanceof Error) {
console.log('Error:', error.message);
}
}
}
```
## Express Server Example
```typescript theme={null}
import express from 'express';
import { open } from '@memvid/sdk';
const app = express();
app.use(express.json());
// Reuse a shared-lock handle for the lifetime of the server.
const mem = await open('knowledge.mv2', 'basic', { readOnly: true });
app.post('/search', async (req, res) => {
const { query, k = 10 } = req.body;
const results = await mem.find(query, { k });
res.json(results);
});
app.post('/ask', async (req, res) => {
const { question } = req.body;
const answer = await mem.ask(question, {
model: 'openai:gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY,
maskPii: true,
});
res.json(answer);
});
app.listen(3000, () => console.log('Server running on port 3000'));
```
## PDF Table Extraction
```typescript theme={null}
import { create, use, Memvid } from '@memvid/sdk';
import { existsSync } from 'fs';
async function tableExample() {
const path = 'invoices.mv2';
const mv: Memvid = existsSync(path)
? await use('basic', path)
: await create(path, 'basic');
await mv.enableLex();
// Extract tables from PDF
const result = await mv.putPdfTables('invoice.pdf', true);
console.log(`Extracted ${result.tables_count} tables`);
// List tables
const tables = await mv.listTables();
for (const table of tables) {
console.log(` ${table.tableId}: ${table.nRows} x ${table.nCols}`);
}
// Get table data
const data = await mv.getTable('pdf_table_1_page1', 'dict') as {
headers?: string[];
rows?: unknown[][];
};
console.log('Headers:', data.headers);
// Export to CSV
const csv = await mv.getTable('pdf_table_1_page1', 'csv');
console.log(csv);
await mv.seal();
}
```
# Node.js SDK
Source: https://docs.memvid.com/node-sdk/overview
Complete API reference for the Memvid Node.js SDK
Build web apps, agents, and AI applications with the Memvid Node.js SDK. Native bindings deliver blazing-fast performance with a TypeScript-first API.
## Installation
```bash theme={null}
npm install @memvid/sdk
# or
pnpm add @memvid/sdk
# or
yarn add @memvid/sdk
```
<Info>
**Requirements:** Node.js 18+, macOS/Linux/Windows. Native bindings included - no extra dependencies needed.
</Info>
***
<Warning title="Don't Lose Your Data!">
**`create()` will OVERWRITE existing files without warning!**
| Function | Purpose | If File Exists | Parameter Order |
| -------------------- | --------------------------- | -------------------- | --------------------- |
| `create(path, kind)` | Create **new** .mv2 file | **DELETES all data** | path first, then kind |
| `use(kind, path)` | Open **existing** .mv2 file | Preserves data | kind first, then path |
**Always check if the file exists before choosing** — see example below.
</Warning>
## Quick Start
```typescript theme={null}
import { create, use } from '@memvid/sdk';
import { existsSync } from 'fs';
const path = 'knowledge.mv2';
// CRITICAL: Check if file exists to avoid data loss!
const mem = existsSync(path)
? await use('basic', path) // Open existing file (kind first!)
: await create(path, 'basic'); // Create new file (path first!)
// Add documents
await mem.put({
title: 'Meeting Notes',
label: 'notes',
text: 'Alice mentioned she works at Anthropic...'
});
// Search works immediately
const results = await mem.find('who works at AI companies?', { k: 5, mode: 'lex' });
console.log(results.hits);
// Ask questions
const answer = await mem.ask('What does Alice do?', { k: 5, mode: 'lex' });
console.log(answer.answer);
// Seal when done (commits changes)
await mem.seal();
```
***
## API Reference
| Category | Methods | Description |
| ------------------- | ----------------------------------------------- | --------------------------------------- |
| **File Operations** | `create`, `open`, `close`, `use` | Create, open, close memory files |
| **Data Ingestion** | `put`, `putMany`, `putFile`, `putFiles` | Add documents with embeddings |
| **Search** | `find`, `ask`, `vecSearch`, `timeline` | Query your memory |
| **Corrections** | `correct`, `correctMany` | Store ground truth with retrieval boost |
| **Memory Cards** | `memories`, `state`, `enrich`, `addMemoryCards` | Structured fact extraction |
| **Tables** | `putPdfTables`, `listTables`, `getTable` | PDF table extraction |
| **Sessions** | `sessionStart`, `sessionEnd`, `sessionReplay` | Time-travel debugging |
| **Tickets** | `syncTickets`, `currentTicket`, `getCapacity` | Capacity management |
| **Security** | `lock`, `unlock`, `lockWho`, `lockNudge` | Encryption and access control |
| **Utilities** | `verify`, `doctor`, `maskPii` | Maintenance and utilities |
***
## Core Functions
### File Operations
```typescript theme={null}
import { create, open, verifyMemvid, doctorMemvid, info } from '@memvid/sdk';
// Create new memory file
const mem = await create('project.mv2');
// Open existing memory
const existing = await open('project.mv2');
// With options
const mem = await create('project.mv2', 'basic', {
enableLex: true, // Enable lexical index
enableVec: true, // Enable vector index
memoryId: 'mem_abc' // Bind to dashboard
});
// Verify file integrity
await verifyMemvid('project.mv2', { deep: true });
// Repair and optimize
await doctorMemvid('project.mv2', {
rebuildTimeIndex: true,
rebuildVecIndex: true,
vacuum: true
});
// Get SDK info
const sdkInfo = info();
```
### Framework Adapters
Choose an adapter for your framework:
```typescript theme={null}
import { use } from '@memvid/sdk';
// Available adapters
const mem = await use('basic', 'file.mv2');
const langchain = await use('langchain', 'file.mv2');
const llamaindex = await use('llamaindex', 'file.mv2');
const vercelai = await use('vercel-ai', 'file.mv2');
const openai = await use('openai', 'file.mv2');
const crewai = await use('crewai', 'file.mv2');
const autogen = await use('autogen', 'file.mv2');
const haystack = await use('haystack', 'file.mv2');
const langgraph = await use('langgraph', 'file.mv2');
const semantickernel = await use('semantic-kernel', 'file.mv2');
const googleadk = await use('google-adk', 'file.mv2');
```
***
## Auto-Embedding
The SDK automatically enables vector embeddings when `OPENAI_API_KEY` is set in your environment:
```typescript theme={null}
// Embeddings are automatically enabled using text-embedding-3-small
await mem.put({ title: 'Doc', text: 'Content here' });
```
**How it works:**
* If `OPENAI_API_KEY` is set and `enableEmbedding` is not specified, embeddings are auto-enabled
* Uses OpenAI's `text-embedding-3-small` model (1536 dimensions)
* If no API key is present and no model specified, falls back to local `bge-small` (384 dimensions)
| Environment | Default Behavior |
| --------------------------------- | -------------------------------------------- |
| `OPENAI_API_KEY` set | Auto-enable with `text-embedding-3-small` |
| No API key | Local `bge-small` if `enableEmbedding: true` |
| Explicit `enableEmbedding: false` | No embeddings |
<Tip>
Set `OPENAI_API_KEY` once in your environment and the SDK handles the rest. No need to pass `enableEmbedding: true` or specify models.
</Tip>
***
## Data Ingestion
### put() - Add Single Document
```typescript theme={null}
await mem.put({
// Required
title: 'Document Title',
// Content (one of these)
text: 'Document content...',
file: '/path/to/document.pdf',
// Optional
uri: 'mv2://docs/intro',
tags: ['api', 'v2'],
labels: ['public', 'reviewed'],
kind: 'markdown',
track: 'documentation',
metadata: { author: 'Alice', version: '2.0' },
// Embeddings
enableEmbedding: true,
embeddingModel: 'bge-small', // or 'openai', 'nomic', etc.
vectorCompression: true, // 16x compression with PQ
// Behavior
autoTag: true, // Auto-generate tags
extractDates: true // Extract date mentions
});
```
### putMany() - Batch Ingestion
```typescript theme={null}
const docs = [
{ title: 'Doc 1', text: 'First document content' },
{ title: 'Doc 2', text: 'Second document content' },
{ title: 'Doc 3', text: 'Third document content' }
];
const frameIds = await mem.putMany(docs, {
enableEmbedding: true,
compressionLevel: 3,
embedder: openaiEmbeddings // Custom embedder
});
```
### putFile() - Document Parsing
Ingest documents directly from files. Supports **PDF**, **DOCX**, **XLSX**, **PPTX**, and more. The SDK automatically extracts text content and creates searchable frames.
```typescript theme={null}
// Single file ingestion
const frames = await mem.putFile('/path/to/report.pdf');
console.log(`Ingested ${frames.length} frames from PDF`);
// With options
const frames = await mem.putFile('/path/to/presentation.pptx', {
chunkSize: 1000, // Characters per chunk
chunkOverlap: 200, // Overlap between chunks
enableEmbedding: true, // Generate embeddings
embeddingModel: 'bge-small'
});
// Excel/XLSX files
const frames = await mem.putFile('/path/to/data.xlsx');
// Each sheet becomes searchable content
// Word documents
const frames = await mem.putFile('/path/to/document.docx');
```
### putFiles() - Batch Document Ingestion
Ingest multiple documents at once:
```typescript theme={null}
const files = [
'/path/to/report.pdf',
'/path/to/slides.pptx',
'/path/to/data.xlsx',
'/path/to/notes.docx'
];
const allFrames = await mem.putFiles(files, {
chunkSize: 1000,
enableEmbedding: true
});
console.log(`Total frames: ${allFrames.length}`);
```
<Info>
**Supported Formats:**
* **PDF** - Text extraction with page-aware chunking
* **DOCX** - Microsoft Word documents
* **XLSX** - Excel spreadsheets (all sheets, formulas evaluated)
* **PPTX** - PowerPoint presentations (slide text and notes)
**No extra dependencies required** - document parsing is built into the native bindings.
</Info>
<Note>
For XLSX files with formulas, the SDK extracts the **calculated values**, not the formula text. This ensures searchable, meaningful content.
</Note>
***
## Search & Retrieval
### find() - Hybrid Search
```typescript theme={null}
// Simple search
const results = await mem.find('budget projections');
// With options
const results = await mem.find('financial outlook', {
mode: 'auto', // 'lex', 'sem', 'auto', 'clip'
k: 10, // Number of results
snippetChars: 480, // Snippet length
scope: 'track:meetings', // Scope filter
// Access control (ACL)
// In "enforce" mode, tenantId is required.
aclContext: { tenantId: 'tenant-123', roles: ['finance'] },
aclEnforcementMode: 'enforce',
// Adaptive retrieval
adaptive: true,
minRelevancy: 0.5,
maxK: 100,
adaptiveStrategy: 'combined', // 'relative', 'absolute', 'cliff', 'elbow'
// Time-travel
asOfFrame: 100,
asOfTs: 1704067200,
// Custom embeddings
embedder: customEmbedder,
queryEmbeddingModel: 'openai'
});
console.log(results.hits);
```
<Info>
**Query Syntax:** Multi-word queries use OR logic by default for better recall. Use `AND` for intersection: `"machine AND learning"`. Use quotes for exact phrases: `'"machine learning"'`.
</Info>
### Permission-Aware Retrieval (ACL)
See [Permission-Aware Retrieval (ACL)](/concepts/permission-aware-retrieval) for the full model.
At a high level:
* Write per-frame ACL metadata during ingestion (`metadata.acl_*`)
* Pass `aclContext` + `aclEnforcementMode: 'enforce'` to `find()` / `ask()`
```typescript theme={null}
import { getAclScopeFromApiKey, aclContextFromScope } from '@memvid/sdk';
const scope = await getAclScopeFromApiKey();
const aclContext = aclContextFromScope(scope);
const hits = await mem.find('budget', {
mode: 'lex',
k: 5,
aclContext,
aclEnforcementMode: 'enforce',
});
```
### ask() - LLM Q\&A
```typescript theme={null}
const answer = await mem.ask('What was decided about the budget?', {
k: 8,
mode: 'auto',
// LLM settings
model: 'gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY,
llmContextChars: 120000,
// Privacy
maskPii: true,
// Time filters
since: 1704067200,
until: 1706745600,
// Options
contextOnly: false, // Set true to skip synthesis
returnSources: true, // Include source documents
// Adaptive retrieval
adaptive: true,
minRelevancy: 0.5,
// Access control (ACL)
aclContext: { tenantId: 'tenant-123', roles: ['finance'] },
aclEnforcementMode: 'enforce',
});
console.log(answer.answer);
console.log(answer.sources);
```
### vecSearch() - Pure Vector Search
```typescript theme={null}
const results = await mem.vecSearch('query', queryEmbedding, {
k: 10,
adaptive: true,
minRelevancy: 0.7
});
```
### Grounding & Hallucination Detection
The `ask()` response includes a `grounding` object that measures how well the answer is supported by context:
```typescript theme={null}
const answer = await mem.ask('What is the API endpoint?', {
model: 'gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY
});
// Check grounding quality
console.log(answer.grounding);
// {
// score: 0.85,
// label: 'HIGH', // 'LOW', 'MEDIUM', or 'HIGH'
// sentence_count: 3,
// grounded_sentences: 3,
// has_warning: false,
// warning_reason: undefined
// }
// Check if follow-up is needed
if (answer.follow_up?.needed) {
console.log('Low confidence:', answer.follow_up.reason);
console.log('Try these instead:', answer.follow_up.suggestions);
}
```
**Grounding Fields:**
| Field | Type | Description |
| -------------------- | --------- | ----------------------------------------- |
| `score` | `number` | Grounding score from 0.0 to 1.0 |
| `label` | `string` | Quality label: `LOW`, `MEDIUM`, or `HIGH` |
| `sentence_count` | `number` | Sentences in the answer |
| `grounded_sentences` | `number` | Sentences supported by context |
| `has_warning` | `boolean` | True if answer may be hallucinated |
| `warning_reason` | `string?` | Explanation if warning is present |
**Follow-up Fields:**
| Field | Type | Description |
| ------------------ | ---------- | -------------------------------- |
| `needed` | `boolean` | True if answer confidence is low |
| `reason` | `string` | Why confidence is low |
| `hint` | `string` | Helpful hint for the user |
| `available_topics` | `string[]` | Topics in this memory |
| `suggestions` | `string[]` | Suggested follow-up questions |
### correct() - Ground Truth Corrections
Store authoritative corrections that take priority in future retrievals:
```typescript theme={null}
// Store a correction
const frameId = await mem.correct('Ben Koenig reported to Chloe Nguyen before 2025');
// With options
const frameId = await mem.correct('The API rate limit is 1000 req/min', {
source: 'Engineering Team - Jan 2025',
topics: ['API', 'rate limiting'],
boost: 2.5 // Higher retrieval priority (default: 2.0)
});
// Batch corrections
const frameIds = await mem.correctMany([
{ statement: 'OAuth tokens expire after 24 hours', topics: ['auth', 'OAuth'] },
{ statement: 'Production DB is db.prod.example.com', source: 'Ops Team' }
]);
// Verify correction is retrievable
const results = await mem.find('Ben Koenig reported to');
console.log(results.hits[0].snippet); // Should show the correction
```
<Tip>
Use `correct()` to fix hallucinations or add verified facts. Corrections receive boosted retrieval scores and are labeled `[Correction]` in results.
</Tip>
***
## Memory Cards (Entity Extraction)
### Automatic Enrichment
```typescript theme={null}
// Extract facts using rules engine (fast, offline)
const result = await mem.enrich('rules');
// View extracted cards
const { cards, count } = await mem.memories();
// Filter by entity
const aliceCards = await mem.memories('Alice');
// Get entity state (O(1) lookup)
const alice = await mem.state('Alice');
console.log(alice.slots);
// { employer: 'Anthropic', role: 'Engineer', location: 'SF' }
// Get stats
const stats = await mem.memoriesStats();
console.log(stats.entityCount, stats.cardCount);
// List all entities
const entities = await mem.memoryEntities();
```
### Manual Memory Cards
```typescript theme={null}
// Add SPO triplets directly
const result = await mem.addMemoryCards([
{ entity: 'Alice', slot: 'employer', value: 'Anthropic' },
{ entity: 'Alice', slot: 'role', value: 'Senior Engineer' },
{ entity: 'Bob', slot: 'team', value: 'Infrastructure' }
]);
console.log(result.added, result.ids);
```
### Export Facts
```typescript theme={null}
// Export to JSON
const json = await mem.exportFacts('json');
// Export to CSV
const csv = await mem.exportFacts('csv', 'Alice');
// Export to N-Triples (RDF)
const ntriples = await mem.exportFacts('ntriples');
```
***
## Table Extraction
```typescript theme={null}
// Extract tables from PDF
const result = await mem.putPdfTables('financial-report.pdf', true);
console.log(`Extracted ${result.tables_count} tables`);
// List all tables
const tables = await mem.listTables();
for (const table of tables) {
console.log(table.table_id, table.n_rows, table.n_cols);
}
// Get table data
const data = await mem.getTable('tbl_001', 'dict');
const csv = await mem.getTable('tbl_001', 'csv');
```
***
## Time-Travel & Sessions
### Timeline Queries
```typescript theme={null}
const timeline = await mem.timeline({
limit: 50,
since: 1704067200,
until: 1706745600,
reverse: true,
asOfFrame: 100
});
```
### Session Recording
```typescript theme={null}
// Start recording
const sessionId = await mem.sessionStart('qa-test');
// Perform operations
await mem.find('test query');
await mem.ask('What happened?');
// Add checkpoint
await mem.sessionCheckpoint();
// End session
const summary = await mem.sessionEnd();
// List sessions
const sessions = await mem.sessionList();
// Replay session with different params
const replay = await mem.sessionReplay(sessionId, 10, true);
console.log(replay.match_rate);
// Delete session
await mem.sessionDelete(sessionId);
```
***
## Encryption & Security
```typescript theme={null}
import { lock, unlock, lockWho, lockNudge } from '@memvid/sdk';
// Encrypt to .mv2e capsule
const encryptedPath = await lock('project.mv2', {
password: 'secret',
force: true
});
// Decrypt back to .mv2
const decryptedPath = await unlock('project.mv2e', {
password: 'secret'
});
// Check who has the lock
const lockInfo = await lockWho('project.mv2');
// Nudge stale lock
const released = await lockNudge('project.mv2');
```
***
## Tickets & Capacity
```typescript theme={null}
// Get current capacity
const capacity = await mem.getCapacity();
// Get current ticket info
const ticket = await mem.currentTicket();
// Sync tickets from dashboard
const result = await mem.syncTickets('mem_abc123', apiKey);
// Apply ticket manually
await mem.applyTicket(ticketString);
// Get memory binding
const binding = await mem.getMemoryBinding();
// Unbind from dashboard
await mem.unbindMemory();
```
***
## Cloud Project & Memory Management
Programmatically create projects and memories on the Memvid dashboard, then bind local `.mv2` files to them.
### Configure SDK
```typescript theme={null}
import { configure } from '@memvid/sdk';
configure({
apiKey: 'mv2_your_api_key_here',
dashboardUrl: 'https://memvid.com'
});
```
### Create and List Projects
```typescript theme={null}
import { createProject, listProjects } from '@memvid/sdk';
// Create a new project
const project = await createProject({
name: 'My AI Project',
description: 'Knowledge base for my AI agent'
});
console.log(`Project ID: ${project.id}`);
console.log(`Slug: ${project.slug}`);
// List all projects
const projects = await listProjects();
for (const proj of projects) {
console.log(`${proj.name} (${proj.id})`);
}
```
**Project Response Fields:**
| Field | Type | Description |
| ---------------- | --------- | -------------------- |
| `id` | `string` | Unique project ID |
| `organisationId` | `string` | Organisation ID |
| `slug` | `string` | URL-friendly slug |
| `name` | `string` | Project name |
| `description` | `string?` | Optional description |
| `createdAt` | `string` | ISO 8601 timestamp |
| `updatedAt` | `string` | ISO 8601 timestamp |
### Create and List Memories
```typescript theme={null}
import { createMemory, listMemories } from '@memvid/sdk';
// Create a memory in a project
const memory = await createMemory({
name: 'Agent Memory',
description: 'Long-term memory for chatbot',
projectId: project.id
});
console.log(`Memory ID: ${memory.id}`);
console.log(`Display Name: ${memory.displayName}`);
// List all memories
const allMemories = await listMemories();
// List memories in a specific project
const projectMemories = await listMemories({ projectId: project.id });
```
### Bind Local File to Cloud Memory
```typescript theme={null}
import { create } from '@memvid/sdk';
// Create local .mv2 file bound to cloud memory
const mv = await create('./agent.mv2', 'basic', { memoryId: memory.id });
await mv.enableLex(); // Enable lexical search
// Add content
await mv.put({ title: 'Meeting Notes', label: 'notes', text: 'Today we discussed...' });
// Search
const results = await mv.find('discussed', { k: 5 });
// Close
await mv.seal();
```
### Complete Example
```typescript theme={null}
import { configure, createProject, createMemory, create } from '@memvid/sdk';
// 1. Configure SDK
configure({ apiKey: process.env.MEMVID_API_KEY });
// 2. Create project
const project = await createProject({
name: 'Knowledge Base',
description: 'Company docs'
});
// 3. Create cloud memory in project
const memory = await createMemory({
name: 'Docs Memory',
projectId: project.id
});
// 4. Create local file bound to cloud memory
const mv = await create('./docs.mv2', 'basic', { memoryId: memory.id });
await mv.enableLex();
// 5. Add content
await mv.put({ title: 'API Guide', label: 'docs', text: 'API documentation...' });
await mv.put({ title: 'FAQ', label: 'docs', text: 'Frequently asked questions...' });
// 6. Search
const results = await mv.find('API', { k: 5 });
console.log(`Found ${results.hits?.length || 0} results`);
// 7. Clean up
await mv.seal();
```
***
## Embedding Providers
### External Providers
```typescript theme={null}
import {
OpenAIEmbeddings,
GeminiEmbeddings,
MistralEmbeddings,
CohereEmbeddings,
VoyageEmbeddings,
NvidiaEmbeddings,
getEmbedder
} from '@memvid/sdk';
// OpenAI
const openai = new OpenAIEmbeddings({
apiKey: process.env.OPENAI_API_KEY,
model: 'text-embedding-3-small' // or 'text-embedding-3-large'
});
// Gemini
const gemini = new GeminiEmbeddings({
apiKey: process.env.GEMINI_API_KEY,
model: 'text-embedding-004'
});
// Mistral
const mistral = new MistralEmbeddings({
apiKey: process.env.MISTRAL_API_KEY
});
// Cohere
const cohere = new CohereEmbeddings({
apiKey: process.env.COHERE_API_KEY,
model: 'embed-english-v3.0'
});
// Voyage
const voyage = new VoyageEmbeddings({
apiKey: process.env.VOYAGE_API_KEY,
model: 'voyage-3'
});
// NVIDIA
const nvidia = new NvidiaEmbeddings({
apiKey: process.env.NVIDIA_API_KEY
});
// Factory function
const embedder = getEmbedder('openai', { apiKey: '...' });
// Use with putMany
await mem.putMany(docs, { embedder: openai });
// Use with find
await mem.find('query', { embedder: gemini });
```
### Local Embeddings (No API Required)
```typescript theme={null}
import { LOCAL_EMBEDDING_MODELS } from '@memvid/sdk';
await mem.put({
text: 'content',
enableEmbedding: true,
embeddingModel: LOCAL_EMBEDDING_MODELS.BGE_SMALL // 384d, fast
});
// Available local models
LOCAL_EMBEDDING_MODELS.BGE_SMALL // 384d - fastest
LOCAL_EMBEDDING_MODELS.BGE_BASE // 768d - balanced
LOCAL_EMBEDDING_MODELS.NOMIC // 768d - general purpose
LOCAL_EMBEDDING_MODELS.GTE_LARGE // 1024d - highest quality
```
***
## Error Handling
```typescript theme={null}
import {
MemvidError,
CapacityExceededError, // MV001
TicketInvalidError, // MV002
TicketReplayError, // MV003
LexIndexDisabledError, // MV004
TimeIndexMissingError, // MV005
VerifyFailedError, // MV006
LockedError, // MV007
ApiKeyRequiredError, // MV008
MemoryAlreadyBoundError, // MV009
FrameNotFoundError, // MV010
VecIndexDisabledError, // MV011
CorruptFileError, // MV012
VecDimensionMismatchError // MV014
} from '@memvid/sdk';
try {
await mem.put({ title: 'Large file', file: 'huge.bin' });
} catch (err) {
if (err instanceof CapacityExceededError) {
console.log('Storage capacity exceeded (MV001)');
} else if (err instanceof LockedError) {
console.log('File locked by another process (MV007)');
} else if (err instanceof VecIndexDisabledError) {
console.log('Enable vector index first (MV011)');
}
}
```
***
## Asset Extraction
```typescript theme={null}
// Get frame content
const content = await mem.view(frameId);
const contentByUri = await mem.viewByUri('mv2://docs/intro');
// Extract binary assets (PDF, images)
const asset = await mem.extractAsset(frameId);
console.log(asset.mimeType, asset.filename, asset.data);
// Get frame metadata
const info = await mem.getFrameInfo(frameId);
console.log(info.uri, info.title, info.timestamp);
```
***
## Environment Variables
| Variable | Description |
| ------------------- | -------------------------- |
| `MEMVID_API_KEY` | Dashboard API key for sync |
| `OPENAI_API_KEY` | OpenAI embeddings and LLM |
| `GEMINI_API_KEY` | Gemini embeddings |
| `MISTRAL_API_KEY` | Mistral embeddings |
| `COHERE_API_KEY` | Cohere embeddings |
| `VOYAGE_API_KEY` | Voyage embeddings |
| `NVIDIA_API_KEY` | NVIDIA embeddings |
| `ANTHROPIC_API_KEY` | Claude for entities |
| `MEMVID_MODELS_DIR` | Model cache directory |
| `MEMVID_OFFLINE` | Use cached models only |
***
## Deploying to Vercel
The Memvid Node.js SDK uses native bindings (N-API) for optimal performance. When deploying to Vercel's serverless environment, you need to configure Next.js to bundle the native binary correctly.
### next.config.ts Configuration
Add `outputFileTracingIncludes` to ensure the native `.node` files are bundled with your serverless functions:
```typescript theme={null}
// next.config.ts
import type { NextConfig } from 'next';
const nextConfig: NextConfig = {
experimental: {
outputFileTracingIncludes: {
'/api/*': [
'./node_modules/@memvid/sdk-linux-x64-gnu/**/*',
'./node_modules/@memvid/sdk/**/*',
],
},
},
};
export default nextConfig;
```
### Explicit Platform Package (Optional)
For more reliable deployments, explicitly add the Linux platform package to your dependencies:
```json theme={null}
{
"dependencies": {
"@memvid/sdk": "^2.0.146",
"@memvid/sdk-linux-x64-gnu": "^2.0.146"
}
}
```
<Note>
Vercel's serverless runtime uses Amazon Linux 2 (x64). The SDK automatically selects the correct platform binary, but explicit inclusion ensures bundling works correctly.
</Note>
### Serverless /tmp Storage
Vercel's serverless functions have ephemeral `/tmp` storage that doesn't persist between invocations. For production apps:
1. **Use cloud storage** (S3, R2, etc.) to persist `.mv2` files
2. **Download on-demand** when the function cold starts
3. **Pass files as buffers** between API routes instead of file paths
```typescript theme={null}
// Example: Download from S3 if not in /tmp
import { existsSync } from 'fs';
import { writeFile } from 'fs/promises';
const localPath = `/tmp/${memoryId}.mv2`;
if (!existsSync(localPath)) {
const buffer = await downloadFromS3(userId, memoryId);
await writeFile(localPath, buffer);
}
const mem = await open(localPath);
```
### Troubleshooting
| Error | Solution |
| ------------------------------------------------- | -------------------------------------------------------- |
| `Native binary not found for platform: linux-x64` | Add `outputFileTracingIncludes` config |
| `GLIBC_2.35 not found` | Ensure you're using SDK v2.0.146+ (built for glibc 2.26) |
| `ENOENT: no such file or directory` | Files in `/tmp` don't persist; use cloud storage |
***
## TypeScript Types
```typescript theme={null}
import type {
PutInput,
PutManyInput,
FindInput,
AskInput,
MemoryCard,
MemoryCardInput,
EntityState,
FrameInfo,
TableInfo,
SessionSummary,
MemvidErrorCode
} from '@memvid/sdk';
```
***
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Build your first AI memory in 5 minutes
</Card>
<Card title="Embedding Providers" icon="cube" href="/concepts/embedding-models">
Compare local and external embedding options
</Card>
<Card title="Framework Integrations" icon="puzzle-piece" href="/frameworks/overview">
LangChain, LlamaIndex, Vercel AI, and more
</Card>
<Card title="Memory Cards" icon="brain" href="/concepts/entity-extraction">
O(1) entity lookups and fact extraction
</Card>
</CardGroup>
# Python SDK
Source: https://docs.memvid.com/python-sdk/overview
Complete API reference for the Memvid Python SDK
Build AI applications, agents, and data pipelines with the Memvid Python SDK. Native Rust bindings deliver high performance with a Pythonic API.
## Installation
```bash theme={null}
pip install memvid-sdk
```
<Info>
**Requirements:** Python 3.8+, macOS/Linux/Windows. Native bindings included - no extra dependencies needed.
</Info>
***
<Warning title="Don't Lose Your Data!">
**`create()` will OVERWRITE existing files without warning!**
| Function | Purpose | If File Exists | Parameter Order |
| ----------------- | --------------------------- | -------------------- | --------------------------------- |
| `create(path)` | Create **new** .mv2 file | **DELETES all data** | path first (kind is keyword-only) |
| `use(kind, path)` | Open **existing** .mv2 file | Preserves data | kind first, then path |
**Always check if the file exists before choosing** — see example below.
</Warning>
## Quick Start
```python theme={null}
from memvid_sdk import create, use
import os
path = 'knowledge.mv2'
# CRITICAL: Check if file exists to avoid data loss!
if os.path.exists(path):
mem = use('basic', path) # Open existing file (kind first!)
else:
mem = create(path) # Create new file (path first!)
# Lexical search (BM25) is enabled by default - no need to call enable_lex()
# Add documents
mem.put(
title='Meeting Notes',
label='notes',
metadata={'source': 'slack'},
text='Alice mentioned she works at Anthropic...'
)
# Search works immediately
results = mem.find('who works at AI companies?', k=5, mode='lex')
print(results['hits'])
# Ask questions
answer = mem.ask('What does Alice do?', k=5, mode='lex')
print(answer['answer'])
# Seal when done (commits changes)
mem.seal()
```
***
## API Reference
| Category | Methods | Description |
| ------------------- | ------------------------------------------------- | --------------------------------------- |
| **File Operations** | `create`, `use`, `close` | Create, open, close memory files |
| **Data Ingestion** | `put`, `put_many`, `put_file`, `put_files` | Add documents with embeddings |
| **Search** | `find`, `ask`, `timeline` | Query your memory |
| **Corrections** | `correct`, `correct_many` | Store ground truth with retrieval boost |
| **Memory Cards** | `memories`, `state`, `enrich`, `add_memory_cards` | Structured fact extraction |
| **Tables** | `put_pdf_tables`, `list_tables`, `get_table` | PDF table extraction |
| **Sessions** | `session_start`, `session_end`, `session_replay` | Time-travel debugging |
| **Tickets** | `sync_tickets`, `current_ticket`, `get_capacity` | Capacity management |
| **Security** | `lock`, `unlock`, `lock_who`, `lock_nudge` | Encryption and access control |
| **Utilities** | `verify`, `doctor` | Maintenance and utilities |
***
## Context Manager
```python theme={null}
import memvid_sdk as memvid
# Automatically closes when done
with memvid.use('basic', 'memory.mv2') as mem:
mem.put('Doc', 'test', {}, text='Content')
results = mem.find('query')
```
***
## Core Functions
### File Operations
```python theme={null}
from memvid_sdk import create, use, lock, unlock, info
# Create new memory file
mem = create('project.mv2')
# With options
mem = create(
'project.mv2',
enable_vec=True, # Enable vector index
enable_lex=True, # Enable lexical index
memory_id='mem_abc', # Bind to dashboard
api_key='mv_live_...' # Dashboard API key
)
# Open existing memory with adapter
mem = use('basic', 'project.mv2', mode='open')
# Available adapters
mem = use('langchain', 'file.mv2')
mem = use('llamaindex', 'file.mv2')
mem = use('crewai', 'file.mv2')
mem = use('autogen', 'file.mv2')
mem = use('haystack', 'file.mv2')
mem = use('langgraph', 'file.mv2')
mem = use('semantic-kernel', 'file.mv2')
mem = use('openai', 'file.mv2')
mem = use('google-adk', 'file.mv2')
# Get SDK info
sdk_info = info()
# Verify file integrity
result = mem.verify(deep=True)
# Repair and optimize
result = mem.doctor(
rebuild_time_index=True,
rebuild_vec_index=True,
vacuum=True
)
```
### Context Manager Support
```python theme={null}
from memvid_sdk import create
# Automatically closes when done
with create('project.mv2') as mem:
mem.put('Title', 'label', {}, text='Content')
results = mem.find('query')
```
***
## Auto-Embedding
The SDK automatically enables vector embeddings when `OPENAI_API_KEY` is set in your environment:
```python theme={null}
import os
os.environ['OPENAI_API_KEY'] = 'sk-...'
# Embeddings are automatically enabled using text-embedding-3-small
mem.put(title='Doc', label='kb', metadata={}, text='Content here')
```
**How it works:**
* If `OPENAI_API_KEY` is set and `enable_embedding` is not specified, embeddings are auto-enabled
* Uses OpenAI's `text-embedding-3-small` model (1536 dimensions)
* If no API key is present and no model specified, falls back to local `bge-small` (384 dimensions)
| Environment | Default Behavior |
| --------------------------------- | -------------------------------------------- |
| `OPENAI_API_KEY` set | Auto-enable with `text-embedding-3-small` |
| No API key | Local `bge-small` if `enable_embedding=True` |
| Explicit `enable_embedding=False` | No embeddings |
<Tip>
Set `OPENAI_API_KEY` once in your environment and the SDK handles the rest. No need to pass `enable_embedding=True` or specify models.
</Tip>
***
## Data Ingestion
### put() - Add Single Document
```python theme={null}
mem.put(
'Document Title', # title (required, positional)
'label', # label (required, positional)
{}, # metadata dict (required, can be {})
# Content (one of these)
text='Document content...',
file='/path/to/document.pdf',
# Optional
uri='mv2://docs/intro',
tags=['api', 'v2'],
labels=['public', 'reviewed'],
kind='markdown',
track='documentation',
# Embeddings
enable_embedding=True,
embedding_model='bge-small', # or 'openai', 'nomic', etc.
vector_compression=True, # 16x compression with PQ
# Behavior
auto_tag=True, # Auto-generate tags
extract_dates=True # Extract date mentions
)
```
### put\_many() - Batch Ingestion
```python theme={null}
docs = [
{'title': 'Doc 1', 'label': 'kb', 'text': 'First document'},
{'title': 'Doc 2', 'label': 'kb', 'text': 'Second document'},
{'title': 'Doc 3', 'label': 'kb', 'text': 'Third document'}
]
frame_ids = mem.put_many(
docs,
embedder=openai_embeddings, # Custom embedder
opts={
'compression_level': 3,
'enable_embedding': True,
'embedding_model': 'bge-small'
}
)
```
### put\_file() - Document Parsing
Ingest documents directly from files. Supports **PDF**, **DOCX**, **XLSX**, **PPTX**, and more. The SDK automatically extracts text content and creates searchable frames.
```python theme={null}
# Single file ingestion
frames = mem.put_file('/path/to/report.pdf')
print(f'Ingested {len(frames)} frames from PDF')
# With options
frames = mem.put_file(
'/path/to/presentation.pptx',
chunk_size=1000, # Characters per chunk
chunk_overlap=200, # Overlap between chunks
enable_embedding=True, # Generate embeddings
embedding_model='bge-small'
)
# Excel/XLSX files
frames = mem.put_file('/path/to/data.xlsx')
# Each sheet becomes searchable content
# Word documents
frames = mem.put_file('/path/to/document.docx')
```
### put\_files() - Batch Document Ingestion
Ingest multiple documents at once:
```python theme={null}
files = [
'/path/to/report.pdf',
'/path/to/slides.pptx',
'/path/to/data.xlsx',
'/path/to/notes.docx'
]
all_frames = mem.put_files(
files,
chunk_size=1000,
enable_embedding=True
)
print(f'Total frames: {len(all_frames)}')
```
<Info>
**Supported Formats:**
* **PDF** - Text extraction with page-aware chunking
* **DOCX** - Microsoft Word documents
* **XLSX** - Excel spreadsheets (all sheets, formulas evaluated)
* **PPTX** - PowerPoint presentations (slide text and notes)
</Info>
<Warning>
**Required for document parsing:** Install dependencies before using `put_file()`:
```bash theme={null}
pip install memvid-sdk[documents]
# or install individually:
pip install pypdf openpyxl python-pptx python-docx
```
Without these, you'll see errors like `[memvid] pypdf/PyPDF2 not installed, PDF parsing unavailable`.
</Warning>
<Note>
For XLSX files with formulas, the SDK extracts the **calculated values**, not the formula text. This ensures searchable, meaningful content.
</Note>
***
## Search & Retrieval
### find() - Hybrid Search
```python theme={null}
# Simple search
results = mem.find('budget projections')
# With options
results = mem.find(
'financial outlook',
mode='auto', # 'lex', 'sem', 'auto'
k=10, # Number of results
snippet_chars=480, # Snippet length
scope='track:meetings', # Scope filter
# Access control (ACL)
# In "enforce" mode, tenant_id is required.
acl_context={"tenant_id": "tenant-123", "roles": ["finance"]},
acl_enforcement_mode="enforce",
# Adaptive retrieval
adaptive=True,
min_relevancy=0.5,
max_k=100,
adaptive_strategy='combined', # 'relative', 'absolute', 'cliff', 'elbow'
# Time-travel
as_of_frame=100,
as_of_ts=1704067200,
# Custom embeddings
embedder=custom_embedder,
query_embedding_model='openai'
)
for hit in results['hits']:
print(hit['title'], hit['snippet'])
```
<Info>
**Query Syntax:** Multi-word queries use OR logic by default for better recall. Use `AND` for intersection: `"machine AND learning"`. Use quotes for exact phrases: `'"machine learning"'`.
</Info>
### Permission-Aware Retrieval (ACL)
See [Permission-Aware Retrieval (ACL)](/concepts/permission-aware-retrieval) for the full model.
At a high level:
* Write per-frame ACL metadata during ingestion (`metadata["acl_*"]`)
* Pass `acl_context` + `acl_enforcement_mode="enforce"` to `find()` / `ask()`
```python theme={null}
from memvid_sdk import get_acl_scope_from_api_key, acl_context_from_scope
scope = get_acl_scope_from_api_key()
acl_context = acl_context_from_scope(scope)
hits = mem.find(
"budget",
mode="lex",
k=5,
acl_context=acl_context,
acl_enforcement_mode="enforce",
)["hits"]
```
### ask() - LLM Q\&A
```python theme={null}
answer = mem.ask(
'What was decided about the budget?',
k=8,
mode='auto',
# LLM settings
model='gpt-4o-mini',
api_key=os.environ['OPENAI_API_KEY'],
llm_context_chars=120000,
# Privacy
mask_pii=True,
# Time filters
since=1704067200,
until=1706745600,
# Options
context_only=False, # Set True to skip synthesis
# Adaptive retrieval
adaptive=True,
min_relevancy=0.5,
# Access control (ACL)
acl_context={"tenant_id": "tenant-123", "roles": ["finance"]},
acl_enforcement_mode="enforce",
)
print(answer['answer'])
print(answer['hits']) # Source documents
```
### Grounding & Hallucination Detection
The `ask()` response includes a `grounding` object that measures how well the answer is supported by context:
```python theme={null}
answer = mem.ask(
'What is the API endpoint?',
model='gpt-4o-mini',
api_key=os.environ['OPENAI_API_KEY']
)
# Check grounding quality
print(answer['grounding'])
# {
# 'score': 0.85,
# 'label': 'HIGH', # 'LOW', 'MEDIUM', or 'HIGH'
# 'sentence_count': 3,
# 'grounded_sentences': 3,
# 'has_warning': False,
# 'warning_reason': None
# }
# Check if follow-up is needed
follow_up = answer.get('follow_up')
if follow_up and follow_up['needed']:
print('Low confidence:', follow_up['reason'])
print('Try these instead:', follow_up['suggestions'])
```
**Grounding Fields:**
| Field | Type | Description |
| -------------------- | ------- | ----------------------------------------- |
| `score` | `float` | Grounding score from 0.0 to 1.0 |
| `label` | `str` | Quality label: `LOW`, `MEDIUM`, or `HIGH` |
| `sentence_count` | `int` | Sentences in the answer |
| `grounded_sentences` | `int` | Sentences supported by context |
| `has_warning` | `bool` | True if answer may be hallucinated |
| `warning_reason` | `str?` | Explanation if warning is present |
**Follow-up Fields:**
| Field | Type | Description |
| ------------------ | ----------- | -------------------------------- |
| `needed` | `bool` | True if answer confidence is low |
| `reason` | `str` | Why confidence is low |
| `hint` | `str` | Helpful hint for the user |
| `available_topics` | `list[str]` | Topics in this memory |
| `suggestions` | `list[str]` | Suggested follow-up questions |
### correct() - Ground Truth Corrections
Store authoritative corrections that take priority in future retrievals:
```python theme={null}
# Store a correction
frame_id = mem.correct('Ben Koenig reported to Chloe Nguyen before 2025')
# With options
frame_id = mem.correct(
'The API rate limit is 1000 req/min',
source='Engineering Team - Jan 2025',
topics=['API', 'rate limiting'],
boost=2.5 # Higher retrieval priority (default: 2.0)
)
# Batch corrections
frame_ids = mem.correct_many([
{'statement': 'OAuth tokens expire after 24 hours', 'topics': ['auth', 'OAuth']},
{'statement': 'Production DB is db.prod.example.com', 'source': 'Ops Team'}
])
# Verify correction is retrievable
results = mem.find('Ben Koenig reported to')
print(results['hits'][0]['snippet']) # Should show the correction
```
<Tip>
Use `correct()` to fix hallucinations or add verified facts. Corrections receive boosted retrieval scores and are labeled `[Correction]` in results.
</Tip>
***
## Memory Cards (Entity Extraction)
### Automatic Enrichment
```python theme={null}
# Extract facts using rules engine (fast, offline)
result = mem.enrich(engine='rules')
# Extract with LLM (more accurate)
result = mem.enrich(engine='openai')
# View extracted cards
memories = mem.memories()
print(memories['cards'])
# Filter by entity
alice_cards = mem.memories(entity='Alice')
# Get entity state (O(1) lookup)
alice = mem.state('Alice')
print(alice['slots'])
# {'employer': 'Anthropic', 'role': 'Engineer', 'location': 'SF'}
# Get stats
stats = mem.memories_stats()
print(stats['entityCount'], stats['cardCount'])
# List all entities
entities = mem.memory_entities()
```
### Manual Memory Cards
```python theme={null}
# Add SPO triplets directly
result = mem.add_memory_cards([
{'entity': 'Alice', 'slot': 'employer', 'value': 'Anthropic'},
{'entity': 'Alice', 'slot': 'role', 'value': 'Senior Engineer'},
{'entity': 'Bob', 'slot': 'team', 'value': 'Infrastructure'}
])
print(result['added'], result['ids'])
```
### Export Facts
```python theme={null}
# Export to JSON
json_data = mem.export_facts(format='json')
# Export to CSV
csv_data = mem.export_facts(format='csv', entity='Alice')
# Export to N-Triples (RDF)
ntriples = mem.export_facts(format='ntriples')
```
***
## Table Extraction
```python theme={null}
# Extract tables from PDF
result = mem.put_pdf_tables('financial-report.pdf', embed_rows=True)
print(f"Extracted {result['tables_count']} tables")
# List all tables
tables = mem.list_tables()
for table in tables:
print(table['table_id'], table['n_rows'], table['n_cols'])
# Get table data
data = mem.get_table('tbl_001', format='dict')
csv_data = mem.get_table('tbl_001', format='csv')
```
***
## Time-Travel & Sessions
### Timeline Queries
```python theme={null}
timeline = mem.timeline(
limit=50,
since=1704067200,
until=1706745600,
reverse=True,
as_of_frame=100
)
```
### Session Recording
```python theme={null}
# Start recording
session_id = mem.session_start('qa-test')
# Perform operations
mem.find('test query')
mem.ask('What happened?')
# Add checkpoint
mem.session_checkpoint()
# End session
summary = mem.session_end()
# List sessions
sessions = mem.session_list()
# Replay session with different params
replay = mem.session_replay(session_id, top_k=10, adaptive=True)
print(replay['match_rate'])
# Delete session
mem.session_delete(session_id)
```
***
## Encryption & Security
```python theme={null}
from memvid_sdk import lock, unlock, lock_who, lock_nudge
# Encrypt to .mv2e capsule
encrypted_path = lock(
'project.mv2',
password='secret',
force=True
)
# Decrypt back to .mv2
decrypted_path = unlock(
'project.mv2e',
password='secret'
)
# Check who has the lock
lock_info = lock_who('project.mv2')
# Nudge stale lock
released = lock_nudge('project.mv2')
```
***
## Tickets & Capacity
```python theme={null}
# Get current capacity
capacity = mem.get_capacity()
# Get current ticket info
ticket = mem.current_ticket()
# Sync tickets from dashboard
result = mem.sync_tickets('mem_abc123', api_key)
# Apply ticket manually
mem.apply_ticket(ticket_string)
# Get memory binding
binding = mem.get_memory_binding()
# Unbind from dashboard
mem.unbind_memory()
```
***
## Cloud Project & Memory Management
Programmatically create projects and memories on the Memvid dashboard, then bind local `.mv2` files to them.
### Configure SDK
```python theme={null}
from memvid_sdk import configure
configure({
"api_key": "mv2_your_api_key_here",
"dashboard_url": "https://memvid.com"
})
```
### Create and List Projects
```python theme={null}
from memvid_sdk import create_project, list_projects
# Create a new project
project = create_project(
"My AI Project",
description="Knowledge base for my AI agent"
)
print(f"Project ID: {project['id']}")
print(f"Slug: {project['slug']}")
# List all projects
projects = list_projects()
for proj in projects:
print(f"{proj['name']} ({proj['id']})")
```
**Project Response Fields:**
| Field | Type | Description |
| ----------------- | ------ | -------------------- |
| `id` | `str` | Unique project ID |
| `organisation_id` | `str` | Organisation ID |
| `slug` | `str` | URL-friendly slug |
| `name` | `str` | Project name |
| `description` | `str?` | Optional description |
| `created_at` | `str` | ISO 8601 timestamp |
| `updated_at` | `str` | ISO 8601 timestamp |
### Create and List Memories
```python theme={null}
from memvid_sdk import create_memory, list_memories
# Create a memory in a project
memory = create_memory(
"Agent Memory",
description="Long-term memory for chatbot",
project_id=project["id"]
)
print(f"Memory ID: {memory['id']}")
print(f"Display Name: {memory['display_name']}")
# List all memories
all_memories = list_memories()
# List memories in a specific project
project_memories = list_memories(project_id=project["id"])
```
### Bind Local File to Cloud Memory
```python theme={null}
from memvid_sdk import create
# Create local .mv2 file bound to cloud memory
mv = create("./agent.mv2", memory_id=memory["id"])
mv.enable_lex() # Enable lexical search
# Add content
mv.put(title="Meeting Notes", label="notes", metadata={}, text="Today we discussed...")
# Search
results = mv.find("discussed", k=5)
# Close
mv.seal()
```
### Complete Example
```python theme={null}
import os
from memvid_sdk import configure, create_project, create_memory, create
# 1. Configure SDK
configure({"api_key": os.environ["MEMVID_API_KEY"]})
# 2. Create project
project = create_project("Knowledge Base", description="Company docs")
# 3. Create cloud memory in project
memory = create_memory("Docs Memory", project_id=project["id"])
# 4. Create local file bound to cloud memory
mv = create("./docs.mv2", memory_id=memory["id"])
mv.enable_lex()
# 5. Add content
mv.put(title="API Guide", label="docs", metadata={}, text="API documentation...")
mv.put(title="FAQ", label="docs", metadata={}, text="Frequently asked questions...")
# 6. Search
results = mv.find("API", k=5)
print(f"Found {len(results['hits'])} results")
# 7. Clean up
mv.seal()
```
***
## Embedding Providers
### External Providers
```python theme={null}
from memvid_sdk.embeddings import (
OpenAIEmbeddings,
GeminiEmbeddings,
MistralEmbeddings,
CohereEmbeddings,
VoyageEmbeddings,
NvidiaEmbeddings,
HuggingFaceEmbeddings,
get_embedder
)
# OpenAI
openai = OpenAIEmbeddings(
api_key=os.environ['OPENAI_API_KEY'],
model='text-embedding-3-small' # or 'text-embedding-3-large'
)
# Gemini
gemini = GeminiEmbeddings(
api_key=os.environ['GEMINI_API_KEY'],
model='text-embedding-004'
)
# Mistral
mistral = MistralEmbeddings(
api_key=os.environ['MISTRAL_API_KEY']
)
# Cohere
cohere = CohereEmbeddings(
api_key=os.environ['COHERE_API_KEY'],
model='embed-english-v3.0'
)
# Voyage
voyage = VoyageEmbeddings(
api_key=os.environ['VOYAGE_API_KEY'],
model='voyage-3'
)
# NVIDIA
nvidia = NvidiaEmbeddings(
api_key=os.environ['NVIDIA_API_KEY']
)
# HuggingFace (local)
hf = HuggingFaceEmbeddings(model='all-MiniLM-L6-v2')
# Factory function
embedder = get_embedder('openai', api_key='...')
# Use with put_many
mem.put_many(docs, embedder=openai)
# Use with find
mem.find('query', embedder=gemini)
```
### Local Embeddings (No API Required)
```python theme={null}
from memvid_sdk.embeddings import LOCAL_EMBEDDING_MODELS
mem.put(
'Title', 'label', {},
text='content',
enable_embedding=True,
embedding_model=LOCAL_EMBEDDING_MODELS['BGE_SMALL'] # 384d, fast
)
# Available local models
LOCAL_EMBEDDING_MODELS['BGE_SMALL'] # 384d - fastest
LOCAL_EMBEDDING_MODELS['BGE_BASE'] # 768d - balanced
LOCAL_EMBEDDING_MODELS['NOMIC'] # 768d - general purpose
LOCAL_EMBEDDING_MODELS['GTE_LARGE'] # 1024d - highest quality
```
***
## Error Handling
```python theme={null}
from memvid_sdk import (
MemvidError,
CapacityExceededError, # MV001
TicketInvalidError, # MV002
TicketReplayError, # MV003
LexIndexDisabledError, # MV004
TimeIndexMissingError, # MV005
VerifyFailedError, # MV006
LockedError, # MV007
ApiKeyRequiredError, # MV008
MemoryAlreadyBoundError, # MV009
FrameNotFoundError, # MV010
VecIndexDisabledError, # MV011
CorruptFileError, # MV012
VecDimensionMismatchError, # MV014
EmbeddingFailedError, # MV015
EncryptionError, # MV016
NerModelNotAvailableError, # MV017
ClipIndexDisabledError # MV018
)
try:
mem.put('Large file', 'data', {}, file='huge.bin')
except CapacityExceededError:
print('Storage capacity exceeded (MV001)')
except LockedError:
print('File locked by another process (MV007)')
except VecIndexDisabledError:
print('Enable vector index first (MV011)')
```
***
## Asset Extraction
```python theme={null}
# Get frame by URI
frame = mem.frame('mv2://docs/intro')
# Get raw binary data
data = mem.blob('mv2://docs/intro')
```
***
## Utility Functions
```python theme={null}
from memvid_sdk import info, flush_analytics, is_telemetry_enabled, verify_single_file
# Get SDK info
sdk_info = info()
print(sdk_info['version'], sdk_info['platform'])
# Flush analytics
flush_analytics()
# Check telemetry status
enabled = is_telemetry_enabled()
# Verify no auxiliary files
verify_single_file('project.mv2')
```
***
## Environment Variables
| Variable | Description |
| ------------------- | -------------------------- |
| `MEMVID_API_KEY` | Dashboard API key for sync |
| `OPENAI_API_KEY` | OpenAI embeddings and LLM |
| `GEMINI_API_KEY` | Gemini embeddings |
| `MISTRAL_API_KEY` | Mistral embeddings |
| `COHERE_API_KEY` | Cohere embeddings |
| `VOYAGE_API_KEY` | Voyage embeddings |
| `NVIDIA_API_KEY` | NVIDIA embeddings |
| `ANTHROPIC_API_KEY` | Claude for entities |
| `MEMVID_MODELS_DIR` | Model cache directory |
| `MEMVID_OFFLINE` | Use cached models only |
***
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Build your first AI memory in 5 minutes
</Card>
<Card title="Embedding Providers" icon="cube" href="/concepts/embedding-models">
Compare local and external embedding options
</Card>
<Card title="Framework Integrations" icon="puzzle-piece" href="/frameworks/overview">
LangChain, LlamaIndex, and more
</Card>
<Card title="Memory Cards" icon="brain" href="/concepts/entity-extraction">
O(1) entity lookups and fact extraction
</Card>
</CardGroup>
# Querying memories
Source: https://docs.memvid.com/python-sdk/querying
Search, ask, and timeline from Python
<Info>
**Query Methods** - Use `find()` for keyword/semantic search, `ask()` for RAG-powered Q\&A with context synthesis, and `timeline()` for chronological retrieval. All methods return structured dicts matching CLI JSON output format.
</Info>
```python theme={null}
from memvid_sdk import use
mv = use("basic", "notes.mv2")
hits = mv.find("deterministic", k=5, mode="lex")
answer = mv.ask("Why is the WAL embedded?", mode="auto", context_only=True)
timeline = mv.timeline(since=1730000000, until=1730003600, limit=10)
```
* `find` returns a list of dicts containing `frame_id`, `score`, `preview`, and metadata identical to CLI JSON output
* `ask` returns a dict with `answer`, `context`, and ranked hits (semantic/hybrid require `enable_vec=True`)
* `timeline` yields chronological entries by scanning the Time Index Track; falls back gracefully when absent
### Permission-Aware Retrieval (ACL)
`find()` and `ask()` support permission-aware retrieval via `acl_context` + `acl_enforcement_mode`.
See [Permission-Aware Retrieval (ACL)](/concepts/permission-aware-retrieval).
```python theme={null}
hits = mv.find(
"budget",
mode="lex",
k=5,
acl_context={"tenant_id": "tenant-123", "roles": ["finance"]},
acl_enforcement_mode="enforce",
)["hits"]
```
### Reading frame payloads
Use the hit URI to fetch metadata (`frame`) or bytes (`blob`):
```python theme={null}
hit = hits["hits"][0]
uri = hit["uri"]
meta = mv.frame(uri)
payload_bytes = mv.blob(uri)
```
### Error handling
All methods raise the same typed exceptions as the CLI. Catch `LockedError`, `CapacityExceededError`, etc., to mirror CLI messaging and make it easier to follow the troubleshooting guidance in the Golden Test Pack.
> **Testing** - Reuse the Golden Corpus to validate search + timeline outputs in Python integration tests to keep parity with CLI regressions.
# 5-Minute Quickstart
Source: https://docs.memvid.com/quickstart/five-minute-guide
Create your first AI memory with search and Q&A in under 5 minutes
Build an AI memory system with hybrid search and LLM-powered Q\&A. Choose your platform:
<Warning title="Don't Lose Your Data!">
**`create()` will OVERWRITE existing files without warning!**
| Function | Purpose | If File Exists | Parameter Order |
| -------------------- | --------------------------- | -------------------- | --------------------- |
| `create(path, kind)` | Create **new** .mv2 file | **DELETES all data** | path first, then kind |
| `use(kind, path)` | Open **existing** .mv2 file | Preserves data | kind first, then path |
**Always check if the file exists before choosing:**
```typescript theme={null}
const mem = existsSync(path) ? await use('basic', path) : await create(path, 'basic');
```
</Warning>
<Tabs>
<Tab title="CLI">
### Install
```bash theme={null}
npm install -g memvid-cli
```
<Tip>
Works on macOS, Linux, and Windows. Requires Node.js 14+.
</Tip>
### Create & Ingest
```bash theme={null}
# Create a new memory
memvid create knowledge.mv2
# Add documents
echo "Alice works at Anthropic as a Senior Engineer in San Francisco." | \
memvid put knowledge.mv2 --title "Team Info"
echo "Bob joined OpenAI last month as a Research Scientist." | \
memvid put knowledge.mv2 --title "New Hires"
echo "Project Alpha has a budget of $500k and is led by Alice." | \
memvid put knowledge.mv2 --title "Projects"
```
### Search
```bash theme={null}
# Search works immediately (BM25 lexical search)
memvid find knowledge.mv2 --query "who works at AI companies"
```
### Ask Questions
```bash theme={null}
# Ask with LLM synthesis (requires OPENAI_API_KEY)
export OPENAI_API_KEY=sk-...
memvid ask knowledge.mv2 --question "What is Alice's role?" --use-model openai
```
### Extract Facts
```bash theme={null}
# Extract structured facts
memvid enrich knowledge.mv2 --engine rules
# Query entity state (O(1) lookup)
memvid state knowledge.mv2 "Alice"
```
**Output:**
```
Entity: Alice
employer: Anthropic
role: Senior Engineer
location: San Francisco
```
</Tab>
<Tab title="Node.js">
### Install
```bash theme={null}
npm install @memvid/sdk
```
### Create & Ingest
```typescript theme={null}
import { create, use } from '@memvid/sdk';
import { existsSync } from 'fs';
const path = 'knowledge.mv2';
// IMPORTANT: create() for NEW files, use() for EXISTING files
const mem = existsSync(path)
? await use('basic', path) // Open existing
: await create(path, 'basic'); // Create new
// Add documents
await mem.put({
title: 'Team Info',
label: 'team',
text: 'Alice works at Anthropic as a Senior Engineer in San Francisco.'
});
await mem.put({
title: 'New Hires',
label: 'team',
text: 'Bob joined OpenAI last month as a Research Scientist.'
});
await mem.put({
title: 'Projects',
label: 'project',
text: 'Project Alpha has a budget of $500k and is led by Alice.'
});
```
### Search
```typescript theme={null}
// Search works immediately (BM25 lexical search)
const results = await mem.find('who works at AI companies', { k: 5 });
console.log(results.hits.map(h => h.title));
// ['Team Info', 'New Hires']
```
### Ask Questions
```typescript theme={null}
// Ask with LLM synthesis
const answer = await mem.ask("What is Alice's role?", {
model: 'gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY
});
console.log(answer.answer);
// "Alice is a Senior Engineer at Anthropic in San Francisco."
```
### Extract Facts
```typescript theme={null}
// Extract structured facts
await mem.enrich('rules');
// Query entity state (O(1) lookup)
const alice = await mem.state('Alice');
console.log(alice.slots);
// { employer: 'Anthropic', role: 'Senior Engineer', location: 'San Francisco' }
```
</Tab>
<Tab title="Python">
### Install
```bash theme={null}
pip install memvid-sdk
```
### Create & Ingest
```python theme={null}
from memvid_sdk import create, use
import os
path = 'knowledge.mv2'
# IMPORTANT: create() for NEW files, use() for EXISTING files
if os.path.exists(path):
mem = use('basic', path) # Open existing
else:
mem = create(path) # Create new (kind='basic' is default)
# enable_lex() not needed - lexical search enabled by default
# Add documents
mem.put(
title='Team Info',
label='team',
metadata={},
text='Alice works at Anthropic as a Senior Engineer in San Francisco.'
)
mem.put(
title='New Hires',
label='team',
metadata={},
text='Bob joined OpenAI last month as a Research Scientist.'
)
mem.put(
title='Projects',
label='project',
metadata={},
text='Project Alpha has a budget of $500k and is led by Alice.'
)
```
### Search
```python theme={null}
# Search works immediately (BM25 lexical search)
results = mem.find('who works at AI companies', k=5)
print([h['title'] for h in results['hits']])
# ['Team Info', 'New Hires']
```
### Ask Questions
```python theme={null}
# Ask with LLM synthesis
answer = mem.ask(
"What is Alice's role?",
model='gpt-4o-mini',
api_key=os.environ['OPENAI_API_KEY']
)
print(answer['answer'])
# "Alice is a Senior Engineer at Anthropic in San Francisco."
```
### Extract Facts
```python theme={null}
# Extract structured facts
mem.enrich(engine='rules')
# Query entity state (O(1) lookup)
alice = mem.state('Alice')
print(alice['slots'])
# {'employer': 'Anthropic', 'role': 'Senior Engineer', 'location': 'San Francisco'}
```
</Tab>
</Tabs>
***
## What You Built
In 5 minutes, you created a complete AI memory system with:
| Feature | Description |
| --------------------- | ---------------------------------------------------- |
| **Hybrid Search** | Combines lexical (BM25) and semantic (vector) search |
| **LLM Q\&A** | Natural language questions with sourced answers |
| **Entity Extraction** | Structured facts with O(1) lookups |
| **Single File** | Everything stored in one portable `.mv2` file |
***
## Next Steps
<CardGroup>
<Card title="CLI Reference" icon="terminal" href="/cli/index">
Complete command reference for all 38+ commands
</Card>
<Card title="Node.js SDK" icon="js" href="/node-sdk/overview">
Full API reference with TypeScript types
</Card>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Complete Python API with examples
</Card>
<Card title="Embedding Providers" icon="cube" href="/concepts/embedding-models">
OpenAI, Gemini, Mistral, and local models
</Card>
<Card title="Memory Cards" icon="brain" href="/concepts/entity-extraction">
Deep dive into O(1) entity lookups
</Card>
<Card title="Framework Integrations" icon="puzzle-piece" href="/frameworks/overview">
LangChain, LlamaIndex, Vercel AI, and more
</Card>
</CardGroup>
***
## Common Patterns
### Using External Embeddings
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
import { create, use, OpenAIEmbeddings } from '@memvid/sdk';
import { existsSync } from 'fs';
const embedder = new OpenAIEmbeddings({
apiKey: process.env.OPENAI_API_KEY,
model: 'text-embedding-3-small'
});
const path = 'project.mv2';
const mem = existsSync(path)
? await use('basic', path)
: await create(path, 'basic');
await mem.putMany(docs, { embedder });
await mem.find('query', { embedder });
```
</Tab>
<Tab title="Python">
```python theme={null}
from memvid_sdk import create, use
from memvid_sdk.embeddings import OpenAIEmbeddings
import os
embedder = OpenAIEmbeddings(
api_key=os.environ['OPENAI_API_KEY'],
model='text-embedding-3-small'
)
path = 'project.mv2'
if os.path.exists(path):
mem = use('basic', path)
else:
mem = create(path)
mem.put_many(docs, embedder=embedder)
mem.find('query', embedder=embedder)
```
</Tab>
</Tabs>
### Batch Ingestion
<Tabs>
<Tab title="Node.js">
```typescript theme={null}
const docs = [
{ title: 'Doc 1', label: 'kb', text: 'Content 1' },
{ title: 'Doc 2', label: 'kb', text: 'Content 2' },
{ title: 'Doc 3', label: 'kb', text: 'Content 3' }
];
await mem.putMany(docs);
```
</Tab>
<Tab title="Python">
```python theme={null}
docs = [
{'title': 'Doc 1', 'label': 'kb', 'text': 'Content 1'},
{'title': 'Doc 2', 'label': 'kb', 'text': 'Content 2'},
{'title': 'Doc 3', 'label': 'kb', 'text': 'Content 3'}
]
mem.put_many(docs)
```
</Tab>
</Tabs>
### PDF Ingestion with Tables
<Tabs>
<Tab title="CLI">
```bash theme={null}
memvid put project.mv2 --input report.pdf --title "Q4 Report" --tables
memvid tables list project.mv2
memvid tables export project.mv2 --table-id tbl_001 -o table.csv
```
</Tab>
<Tab title="Node.js">
```typescript theme={null}
await mem.put({ file: 'report.pdf', title: 'Q4 Report', label: 'report' });
await mem.putPdfTables('report.pdf', true);
const tables = await mem.listTables();
```
</Tab>
<Tab title="Python">
```python theme={null}
mem.put(title='Q4 Report', label='report', metadata={}, file='report.pdf')
mem.put_pdf_tables('report.pdf', embed_rows=True)
tables = mem.list_tables()
```
</Tab>
</Tabs>
# Use Cases
Source: https://docs.memvid.com/resources/examples
Real-world use cases and examples for Memvid
Learn from complete, working examples that demonstrate common Memvid use cases.
## Use Cases
<CardGroup>
<Card title="Chatbot Memory" icon="robot" href="/examples/chatbot-memory">
Persistent memory for conversational AI
</Card>
<Card title="Document Q&A" icon="file-lines" href="/examples/document-qa">
Build a RAG system for document querying
</Card>
<Card title="Knowledge Base" icon="book-open" href="/examples/knowledge-base">
Create a searchable knowledge repository
</Card>
<Card title="Research Assistant" icon="microscope" href="/examples/research-assistant">
Organize and query research papers
</Card>
</CardGroup>
## Quick Examples
### Chatbot with Memory
```python theme={null}
from memvid_sdk import use
mem = use('basic', 'chat_memory.mv2')
# Store conversation turns
mem.put({
'title': f'User message',
'text': user_message,
'label': 'user',
'metadata': {'session_id': session_id}
})
# Retrieve relevant context for responses
context = mem.find(user_message, k=5, scope=f'session:{session_id}')
```
### Document RAG
```python theme={null}
from memvid_sdk import create
# Ingest documents
mem = create('docs.mv2')
for pdf in Path('documents/').glob('*.pdf'):
mem.put({
'title': pdf.stem,
'file': str(pdf),
'label': 'document'
})
mem.seal()
# Query with RAG
answer = mem.ask(
'What are the key findings?',
model='openai:gpt-4o',
k=10
)
print(answer['answer'])
```
### Knowledge Graph
```python theme={null}
from memvid_sdk import create
from memvid_sdk.entities import get_entity_extractor
mem = create('knowledge.mv2')
ner = get_entity_extractor('openai', entity_types=['PERSON', 'ORG', 'TOPIC'])
# Ingest with entity extraction
for doc in documents:
entities = ner.extract(doc.text)
mem.put({
'title': doc.title,
'text': doc.text,
'metadata': {'entities': entities}
})
# Search by entity
results = mem.find('Microsoft', k=10)
```
## Framework Examples
### LangChain
```python theme={null}
from memvid_sdk import use
from langchain_openai import ChatOpenAI
mem = use('langchain', 'knowledge.mv2')
retriever = mem.as_retriever(k=5)
# Use in a chain
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(),
retriever=retriever
)
```
### Vercel AI SDK
```typescript theme={null}
import { use } from '@memvid/sdk';
import { generateText } from 'ai';
const mem = await use('vercel-ai', 'knowledge.mv2');
const result = await generateText({
model: openai('gpt-4o'),
tools: mem.tools,
prompt: 'Search for information about authentication'
});
```
## Browse All Examples
<Card title="GitHub Examples" icon="github" href="https://github.com/memvid/memvid/tree/main/examples">
Full source code for all examples on GitHub
</Card>
# Help & Troubleshooting
Source: https://docs.memvid.com/resources/troubleshooting
Get help with common issues and find answers
Find solutions to common problems, understand error codes, and get help from the community.
## Quick Diagnosis
Run these commands to identify issues:
```bash theme={null}
# Check file health
memvid verify knowledge.mv2 --deep
# View file stats
memvid stats knowledge.mv2 --json
# Check for locks
lsof knowledge.mv2
```
## Common Issues
<AccordionGroup>
<Accordion title="File is locked">
Another process has the file open. Find it with `lsof knowledge.mv2` or open in read-only mode:
```python theme={null}
mem = use('basic', 'knowledge.mv2', read_only=True)
```
</Accordion>
<Accordion title="Search returns no results">
Check the search mode and verify content exists:
```bash theme={null}
memvid stats knowledge.mv2 # Check frame count
memvid find knowledge.mv2 --query "test" --mode lex # Try lexical
```
</Accordion>
<Accordion title="Capacity exceeded">
Upgrade your plan or delete old content:
```bash theme={null}
memvid tickets sync knowledge.mv2 --memory-id YOUR_ID
```
</Accordion>
<Accordion title="Import errors">
Reinstall the SDK:
```bash theme={null}
pip install --upgrade memvid-sdk # Python
npm install @memvid/sdk # Node.js
```
</Accordion>
</AccordionGroup>
## Resources
<CardGroup>
<Card title="Error Reference" icon="book" href="/errors/reference">
Complete list of error codes with solutions
</Card>
<Card title="Troubleshooting Guide" icon="wrench" href="/errors/troubleshooting">
Step-by-step solutions for common issues
</Card>
<Card title="FAQ" icon="circle-question" href="/faq/general">
Frequently asked questions
</Card>
<Card title="Security FAQ" icon="shield" href="/faq/security-and-compliance">
Security, privacy, and compliance questions
</Card>
</CardGroup>
## Get Help
<CardGroup>
<Card title="Discord" icon="discord" href="https://discord.gg/2mynS7fcK7">
Join our community for real-time help
</Card>
<Card title="GitHub Issues" icon="github" href="https://github.com/memvid/memvid/issues">
Report bugs or request features
</Card>
<Card title="Email Support" icon="envelope" href="mailto:support@memvid.com">
Contact our support team
</Card>
</CardGroup>
# CLI Reference
Source: https://docs.memvid.com/sdks/cli
Every memvid command, flag, and environment variable
## Install
<Tabs>
<Tab title="Auto Installer (Recommended)">
The installer automatically checks for and installs required dependencies (git, node, npm), then installs Memvid globally.
**macOS / Linux:**
```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/memvid/preflight-installer/main/install.sh | bash
```
**Windows (PowerShell):**
```powershell theme={null}
irm https://raw.githubusercontent.com/memvid/preflight-installer/main/install.ps1 | iex
```
<Note>
The installer only installs missing tools and asks for confirmation before proceeding.
[View source on GitHub](https://github.com/memvid/preflight-installer)
</Note>
</Tab>
<Tab title="npm">
```bash theme={null}
npm install -g memvid-cli
```
</Tab>
<Tab title="Cargo">
```bash theme={null}
cargo install memvid-cli
```
</Tab>
<Tab title="From Source">
```bash theme={null}
cargo install --path memvid/crates/memvid-cli --force --features parallel_segments,temporal_track
```
</Tab>
</Tabs>
Verify installation:
```bash theme={null}
memvid --version
```
## Global Flags
| Flag | Description |
| ------------------------- | ----------------------------------------------------------------------------- |
| `-v, --verbose` | Debug output (repeat to increase) |
| `--embedding-model MODEL` | Embedding model (bge-small, bge-base, nomic, gte-large, openai, openai-large) |
| `--parallel-segments` | Enable parallel segment building |
| `--json` | Output as JSON (available on most commands) |
## Core Workflows
```bash theme={null}
# Create a memory file
memvid create data.mv2 --tier free --size 1GB
# Ingest documents with embeddings
memvid put data.mv2 --input docs/whitepaper.pdf --title "Whitepaper" --label docs --embedding --vector-compression
# Search
memvid find data.mv2 --query "hybrid search" --top-k 5 --json
# Ask with citations
memvid ask data.mv2 --question "What changed last release?" --top-k 6 --mode hybrid --json
# Timeline and replay
memvid timeline data.mv2 --limit 20 --as-of-frame 150
# Inspect a frame
memvid view data.mv2 --frame-id 10 --page 1
# Stats and maintenance
memvid stats data.mv2
memvid verify data.mv2 --deep
memvid doctor data.mv2 --vacuum --rebuild-lex-index --rebuild-vec-index
```
## API Reference
### File Operations
| Command | Purpose | Key Flags |
| -------- | ---------------------- | --------------------------------------------- |
| `create` | Create new `.mv2` file | `--tier`, `--size`, `--no-lex`, `--no-vector` |
| `open` | Inspect metadata | `--json` |
| `stats` | Show file statistics | `--json` |
### Data Operations
| Command | Purpose | Key Flags |
| ----------- | ------------------------- | -------------------------------------------------------------------- |
| `put` | Append from file or stdin | `--input`, `--uri`, `--title`, `--embedding`, `--vector-compression` |
| `put-many` | Batch ingest | `--input batch.json`, `--compression-level` |
| `api-fetch` | Fetch from HTTP + ingest | `--config`, `--mode`, `--dry-run` |
| `update` | Replace payload/metadata | `--frame-id`, `--uri`, `--input`, `--title` |
| `delete` | Remove a frame | `--frame-id`, `--uri`, `--yes` |
### Search Operations
| Command | Purpose | Key Flags |
| ------------ | ------------------------ | ---------------------------------------------- |
| `find` | Lexical/hybrid search | `--query`, `--mode`, `--top-k`, `--scope` |
| `vec-search` | Search with vector input | `--vector`, `--embedding` |
| `ask` | Retrieval + synthesis | `--question`, `--mode`, `--top-k`, `--sources` |
### Timeline & View
| Command | Purpose | Key Flags |
| ---------- | -------------------------- | -------------------------------------------- |
| `timeline` | Chronological list | `--limit`, `--since`, `--until`, `--reverse` |
| `when` | Temporal phrase resolution | `--on`, `--tz`, `--anchor`, `--window` |
| `view` | Render a frame | `--frame-id`, `--uri`, `--page`, `--play` |
### Memory Cards & Enrichment
| Command | Purpose | Key Flags |
| ---------- | ----------------------- | ------------------------------------- |
| `enrich` | Extract memory cards | `--engine`, `--incremental`, `--json` |
| `memories` | View memory cards | `--entity`, `--slot`, `--json` |
| `state` | Get entity state (O(1)) | `--json` |
| `facts` | Audit extracted facts | `--entity`, `--json` |
| `export` | Export facts | `--format`, `--entity` |
### Tables
| Command | Purpose | Key Flags |
| --------------- | ---------------------- | ------------------------- |
| `tables import` | Import tables from PDF | `--input`, `--embed-rows` |
| `tables list` | List all tables | `--json` |
| `tables export` | Export table data | `--table-id`, `--format` |
### Maintenance
| Command | Purpose | Key Flags |
| -------- | -------------------- | -------------------------------------------------------- |
| `verify` | Check file integrity | `--deep` |
| `doctor` | Repair and optimize | `--vacuum`, `--rebuild-lex-index`, `--rebuild-vec-index` |
| `who` | Check file lock | `--json` |
| `nudge` | Release stale lock | |
### Tickets & Capacity
| Command | Purpose | Key Flags |
| --------------- | --------------------- | -------------------------- |
| `tickets sync` | Sync from dashboard | `--memory-id`, `--api-key` |
| `tickets apply` | Apply ticket manually | `--ticket` |
| `tickets list` | List current tickets | `--json` |
| `plan show` | Show current plan | `--json` |
### Sessions
| Command | Purpose | Key Flags |
| ---------------- | --------------- | ----------------------- |
| `session start` | Start recording | `--name` |
| `session end` | End recording | |
| `session list` | List sessions | `--json` |
| `session replay` | Replay session | `--adaptive`, `--top-k` |
<Note>
Embeddings are off by default in the CLI to save space. Add `--embedding` (and optionally `--vector-compression`) when you want vectors.
</Note>
## Environment Variables
| Variable | Purpose | Default |
| --------------------------- | --------------------------------- | ------------------ |
| `MEMVID_API_KEY` | Dashboard API key for ticket sync | n/a |
| `MEMVID_API_URL` | Control plane base URL | production URL |
| `MEMVID_CACHE_DIR` | Ticket cache directory | `~/.cache/memvid` |
| `MEMVID_MODELS_DIR` | Model cache directory | `~/.memvid/models` |
| `MEMVID_OFFLINE` | Skip model downloads | `0` |
| `MEMVID_EMBEDDING_MODEL` | Default embedding model | `bge-small` |
| `MEMVID_LLM_CONTEXT_BUDGET` | Max chars to send to LLMs | unset |
| `MEMVID_PARALLEL_SEGMENTS` | Force parallel builder | unset |
| `OPENAI_API_KEY` | OpenAI API key | n/a |
| `ANTHROPIC_API_KEY` | Anthropic API key | n/a |
| `GEMINI_API_KEY` | Google Gemini API key | n/a |
## Error Codes
| Code | Name | Description |
| ----- | -------------------- | ------------------------------ |
| MV001 | CapacityExceeded | Storage limit reached |
| MV002 | TicketInvalid | Invalid ticket signature |
| MV004 | LexIndexDisabled | Lexical search not enabled |
| MV007 | FileLocked | File locked by another process |
| MV010 | FrameNotFound | Requested frame doesn't exist |
| MV011 | VecIndexDisabled | Vector search not enabled |
| MV014 | VecDimensionMismatch | Wrong embedding dimension |
| MV015 | EmbeddingFailed | Embedding generation failed |
See [Error Reference](/errors/reference) for complete documentation.
## Tips
* Use `--json` on any command for structured output
* Replay filters: `--as-of-frame` and `--as-of-ts` are available on `find`, `ask`, and `timeline`
* Locks: mutation commands accept `--lock-timeout` and `--force` for stale writers
* `doctor` and `verify` are safe read-only when run with `--plan-only` or `--deep`
<CardGroup>
<Card title="Five-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Fast start for CLI, Python, and Node.js
</Card>
<Card title="CLI Commands" icon="terminal" href="/cli/create-and-put">
Detailed command documentation
</Card>
</CardGroup>
# SDK Feature Matrix
Source: https://docs.memvid.com/sdks/feature-matrix
Compare features across CLI, Python SDK, and Node.js SDK
This matrix shows which features are available in each Memvid interface. Use it to choose the right tool for your use case.
***
## Quick Comparison
| Interface | Best For | Language | Async |
| --------------- | ------------------------------------- | ------------- | -------- |
| **CLI** | Scripts, automation, exploration | Rust | N/A |
| **Python SDK** | Data science, ML pipelines, notebooks | Python | Optional |
| **Node.js SDK** | Web apps, APIs, serverless | TypeScript/JS | Yes |
***
## Core Operations
| Feature | CLI | Python | Node.js |
| ----------------- | --- | ------ | ------- |
| Create memory | ✅ | ✅ | ✅ |
| Open existing | ✅ | ✅ | ✅ |
| Put content | ✅ | ✅ | ✅ |
| Put with metadata | ✅ | ✅ | ✅ |
| Put from file | ✅ | ✅ | ✅ |
| Put from folder | ✅ | ✅ | ✅ |
| Put from URL | ✅ | ✅ | ✅ |
| Delete frame | ✅ | ✅ | ✅ |
| Update frame | ✅ | ✅ | ✅ |
***
## Search & Retrieval
| Feature | CLI | Python | Node.js |
| -------------------------------- | --- | ------ | ------- |
| Lexical search | ✅ | ✅ | ✅ |
| Semantic search | ✅ | ✅ | ✅ |
| Hybrid search | ✅ | ✅ | ✅ |
| CLIP visual search | ✅ | ✅ | ❌ |
| Adaptive retrieval | ✅ | ✅ | ✅ |
| Graph-filtered search | ✅ | ✅ | ❌ |
| Scope filtering | ✅ | ✅ | ✅ |
| Permission-aware retrieval (ACL) | ⚠️ | ✅ | ✅ |
| Date filtering | ✅ | ✅ | ✅ |
| Pagination (cursor) | ✅ | ✅ | ✅ |
| Raw vector search | ✅ | ✅ | ❌ |
<Info>
`acl_context` is required for ACL enforcement. SDKs support passing it to `find()`/`ask()`; the CLI currently runs in audit mode by default.
</Info>
***
## Ask (RAG)
| Feature | CLI | Python | Node.js |
| ----------------------- | --- | ------ | ------- |
| Basic ask | ✅ | ✅ | ✅ |
| Context-only mode | ✅ | ✅ | ✅ |
| Source citations | ✅ | ✅ | ✅ |
| PII masking | ✅ | ✅ | ❌ |
| Memory cards in context | ✅ | ✅ | ❌ |
| Custom LLM model | ✅ | ✅ | ✅ |
| Local LLM (TinyLlama) | ✅ | ❌ | ❌ |
| OpenAI synthesis | ✅ | ✅ | ✅ |
| Claude synthesis | ✅ | ✅ | ✅ |
| Groq synthesis | ✅ | ✅ | ✅ |
| Gemini synthesis | ✅ | ✅ | ✅ |
***
## Enrichment & Memory Cards
| Feature | CLI | Python | Node.js |
| --------------------------- | --- | ------ | ------- |
| Enrich with rules | ✅ | ✅ | ❌ |
| Enrich with Candle (local) | ✅ | ❌ | ❌ |
| Enrich with Groq | ✅ | ✅ | ❌ |
| Enrich with OpenAI | ✅ | ✅ | ❌ |
| Enrich with Claude | ✅ | ✅ | ❌ |
| View memory cards | ✅ | ✅ | ❌ |
| Entity state lookup | ✅ | ✅ | ❌ |
| Fact history/provenance | ✅ | ✅ | ❌ |
| Export facts (RDF/JSON/CSV) | ✅ | ❌ | ❌ |
| Schema management | ✅ | ❌ | ❌ |
***
## Graph & Logic Mesh
| Feature | CLI | Python | Node.js |
| ------------------------ | --- | ------ | ------- |
| Enable Logic Mesh | ✅ | ✅ | ❌ |
| List entities | ✅ | ✅ | ❌ |
| Traverse relationships | ✅ | ✅ | ❌ |
| Graph statistics | ✅ | ❌ | ❌ |
| Triple pattern queries | ✅ | ✅ | ❌ |
| Hybrid graph+text search | ✅ | ✅ | ❌ |
***
## Tables
| Feature | CLI | Python | Node.js |
| ----------------------- | --- | ------ | ------- |
| Import tables from PDF | ✅ | ❌ | ❌ |
| List tables | ✅ | ❌ | ❌ |
| View table | ✅ | ❌ | ❌ |
| Export table (CSV/JSON) | ✅ | ❌ | ❌ |
| Search table rows | ✅ | ❌ | ❌ |
***
## Timeline & History
| Feature | CLI | Python | Node.js |
| ----------------------- | --- | ------ | ------- |
| Timeline browsing | ✅ | ✅ | ✅ |
| View single frame | ✅ | ✅ | ✅ |
| Time-travel queries | ✅ | ✅ | ❌ |
| Replay sessions | ✅ | ❌ | ❌ |
| Temporal phrase parsing | ✅ | ❌ | ❌ |
***
## Embedding Models
| Feature | CLI | Python | Node.js |
| ---------------------- | --- | ------ | ------- |
| BGE-small (local) | ✅ | ✅ | ✅ |
| BGE-base (local) | ✅ | ✅ | ✅ |
| Nomic (local) | ✅ | ✅ | ✅ |
| GTE-large (local) | ✅ | ✅ | ✅ |
| OpenAI embeddings | ✅ | ✅ | ✅ |
| Custom model selection | ✅ | ✅ | ✅ |
| No-vec mode | ✅ | ✅ | ✅ |
***
## Maintenance & Administration
| Feature | CLI | Python | Node.js |
| --------------------- | --- | ------ | ------- |
| Verify integrity | ✅ | ❌ | ❌ |
| Doctor/repair | ✅ | ❌ | ❌ |
| Rebuild lexical index | ✅ | ❌ | ❌ |
| Rebuild vector index | ✅ | ❌ | ❌ |
| Rebuild time index | ✅ | ❌ | ❌ |
| Vacuum/compact | ✅ | ❌ | ❌ |
| Build sketch index | ✅ | ❌ | ❌ |
| Check lock status | ✅ | ❌ | ❌ |
| Statistics | ✅ | ✅ | ✅ |
***
## Encryption
| Feature | CLI | Python | Node.js |
| ------------------- | --- | ------ | ------- |
| Encrypt (lock) | ✅ | ❌ | ❌ |
| Decrypt (unlock) | ✅ | ❌ | ❌ |
| Password from stdin | ✅ | ❌ | ❌ |
***
## Configuration & Plans
| Feature | CLI | Python | Node.js |
| -------------------- | --- | ------ | ------- |
| Set API key | ✅ | ✅ | ✅ |
| View plan/usage | ✅ | ✅ | ✅ |
| Sync plan | ✅ | ❌ | ❌ |
| Query tracking | ✅ | ✅ | ✅ |
| Quota error handling | ✅ | ✅ | ✅ |
| Named memories | ✅ | ❌ | ❌ |
***
## Tickets & Capacity
| Feature | CLI | Python | Node.js |
| ------------------- | --- | ------ | ------- |
| List tickets | ✅ | ❌ | ❌ |
| Sync from dashboard | ✅ | ❌ | ❌ |
| Apply ticket | ✅ | ❌ | ❌ |
| Bind to memory ID | ✅ | ✅ | ✅ |
***
## Media Processing
| Feature | CLI | Python | Node.js |
| ---------------------- | --- | ------ | ------- |
| PDF extraction | ✅ | ✅ | ✅ |
| DOCX extraction | ✅ | ✅ | ✅ |
| XLSX extraction | ✅ | ✅ | ❌ |
| PPTX extraction | ✅ | ✅ | ❌ |
| Audio transcription | ✅ | ❌ | ❌ |
| Video processing | ✅ | ❌ | ❌ |
| Image EXIF parsing | ✅ | ✅ | ❌ |
| CLIP visual embeddings | ✅ | ✅ | ❌ |
| Audio playback | ✅ | ❌ | ❌ |
| Video preview | ✅ | ❌ | ❌ |
***
## Output Formats
| Feature | CLI | Python | Node.js |
| -------------- | --- | ------ | ------- |
| Human-readable | ✅ | N/A | N/A |
| JSON output | ✅ | ✅ | ✅ |
| Streaming | ❌ | ❌ | ✅ |
***
## Choosing the Right Interface
### Use CLI When:
* Automating with shell scripts
* One-off exploration and debugging
* Running maintenance operations
* Processing audio/video
* Using encryption
* Managing tables
### Use Python SDK When:
* Building ML/data pipelines
* Working in Jupyter notebooks
* Need enrichment features
* Integrating with Python apps
* Using graph search
### Use Node.js SDK When:
* Building web applications
* Creating REST APIs
* Serverless functions
* TypeScript projects
* Need async/streaming
***
## Feature Availability Legend
| Symbol | Meaning |
| ------ | --------------- |
| ✅ | Fully supported |
| ⚠️ | Partial support |
| ❌ | Not available |
| N/A | Not applicable |
***
## Requesting Features
Missing a feature in your preferred SDK?
1. Check if CLI has it (most complete)
2. Use CLI via subprocess as workaround
3. Request on [GitHub Issues](https://github.com/memvid/memvid/issues)
4. Contact [support@memvid.com](mailto:support@memvid.com)
***
## Next Steps
<CardGroup>
<Card title="Python SDK" icon="python" href="/python-sdk/overview">
Get started with Python
</Card>
<Card title="Node.js SDK" icon="node-js" href="/node-sdk/overview">
Get started with Node.js
</Card>
</CardGroup>
# Platforms
Source: https://docs.memvid.com/sdks/index
Choose the platform that fits your stack
Pick your platform and jump in. Each page includes real signatures, flags, and working examples.
<CardGroup>
<Card title="CLI" icon="terminal" href="/sdks/cli">
Command-line workflows for build, ingest, search, and maintenance
</Card>
<Card title="Python" icon="python" href="/sdks/python">
High-level API with automatic embeddings and adapters
</Card>
<Card title="Node.js" icon="node-js" href="/sdks/node">
Async API with TypeScript types and framework adapters
</Card>
</CardGroup>
<CardGroup>
<Card title="Five-Minute Quickstart" icon="rocket" href="/quickstart/five-minute-guide">
Create, ingest, search, and ask in five minutes
</Card>
<Card title="Concepts" icon="lightbulb" href="/concepts/memory-architecture">
Learn the `.mv2` format, indexes, and crash recovery
</Card>
</CardGroup>
# Node.js SDK
Source: https://docs.memvid.com/sdks/node
Build AI applications with persistent memory in TypeScript/JavaScript
The Node.js SDK provides a fully-typed TypeScript interface for working with Memvid memory files.
## Installation
```bash theme={null}
npm install @memvid/sdk
# or
pnpm add @memvid/sdk
# or
yarn add @memvid/sdk
```
<Info>
**Requirements:** Node.js 18+, macOS/Linux/Windows. Native bindings included.
</Info>
## Quick Start
```typescript theme={null}
import { create, open } from '@memvid/sdk';
// Create a new memory file
const mem = await create('knowledge.mv2');
// Add documents
await mem.put({
title: 'Meeting Notes',
text: 'Alice mentioned she works at Anthropic...',
enableEmbedding: true
});
// Search
const results = await mem.find('who works at AI companies?');
console.log(results.hits);
// Ask questions with AI
const answer = await mem.ask('What does Alice do?', {
model: 'gpt-4o-mini',
modelApiKey: process.env.OPENAI_API_KEY
});
console.log(answer.text);
// Close when done
await mem.close();
```
## API Reference
| Category | Methods | Description |
| -------------------- | ---------------------------------------------------------------------------- | -------------------------------- |
| **File Operations** | `create`, `open`, `close`, `use` | Create, open, close memory files |
| **Data Ingestion** | `put`, `putMany` | Add documents with embeddings |
| **Search** | `find`, `ask`, `vecSearch`, `timeline` | Query your memory |
| **Memory Cards** | `memories`, `state`, `enrich`, `addMemoryCards` | Structured fact extraction |
| **Tables** | `putPdfTables`, `listTables`, `getTable` | PDF table extraction |
| **Sessions** | `sessionStart`, `sessionEnd`, `sessionReplay` | Time-travel debugging |
| **Tickets** | `syncTickets`, `currentTicket`, `getCapacity` | Capacity management |
| **Cloud Management** | `configure`, `createProject`, `listProjects`, `createMemory`, `listMemories` | Dashboard API |
| **Security** | `lock`, `unlock`, `lockWho`, `lockNudge` | Encryption and access control |
| **Utilities** | `verify`, `doctor`, `maskPii` | Maintenance and utilities |
## Framework Adapters
```typescript theme={null}
import { use } from '@memvid/sdk';
// Vercel AI SDK
const vercel = await use('vercel-ai', 'knowledge.mv2');
const tools = vercel.tools;
// LangChain.js
const langchain = await use('langchain', 'knowledge.mv2');
const retriever = langchain.asRetriever();
// LlamaIndex
const llamaindex = await use('llamaindex', 'knowledge.mv2');
// OpenAI Function Calling
const openai = await use('openai', 'knowledge.mv2');
const functions = openai.functions;
// Google ADK
const googleAdk = await use('google-adk', 'knowledge.mv2');
// Semantic Kernel
const sk = await use('semantic-kernel', 'knowledge.mv2');
```
## Embedding Providers
```typescript theme={null}
import {
OpenAIEmbeddings,
GeminiEmbeddings,
MistralEmbeddings,
CohereEmbeddings,
VoyageEmbeddings,
NvidiaEmbeddings,
LOCAL_EMBEDDING_MODELS
} from '@memvid/sdk';
// OpenAI
const openai = new OpenAIEmbeddings({
apiKey: process.env.OPENAI_API_KEY,
model: 'text-embedding-3-small'
});
// Gemini
const gemini = new GeminiEmbeddings({
apiKey: process.env.GEMINI_API_KEY
});
// Mistral
const mistral = new MistralEmbeddings({
apiKey: process.env.MISTRAL_API_KEY
});
// Local (no API required)
await mem.put({
text: 'content',
enableEmbedding: true,
embeddingModel: LOCAL_EMBEDDING_MODELS.BGE_SMALL
});
```
**Local Embedding Models:**
| Model | Dimensions | Speed | Quality |
| ----------- | ---------- | ------- | ------- |
| `BGE_SMALL` | 384 | Fastest | Good |
| `BGE_BASE` | 768 | Fast | Better |
| `NOMIC` | 768 | Fast | Better |
| `GTE_LARGE` | 1024 | Slower | Best |
## Entity Extraction
```typescript theme={null}
// Extract facts using rules engine
const result = await mem.enrich('rules');
// View extracted cards
const { cards, count } = await mem.memories();
console.log(`Extracted ${count} memory cards`);
// Get entity state (O(1) lookup)
const alice = await mem.state('Alice');
console.log(alice.slots);
// { employer: 'Anthropic', role: 'Engineer' }
// Add memory cards manually
await mem.addMemoryCards([
{ entity: 'Alice', slot: 'employer', value: 'Anthropic' },
{ entity: 'Bob', slot: 'team', value: 'Infrastructure' }
]);
// Export facts
const json = await mem.exportFacts('json');
const csv = await mem.exportFacts('csv', 'Alice');
```
## Session Recording
Record and replay agent sessions for debugging:
```typescript theme={null}
// Start recording
const sessionId = await mem.sessionStart('Debug Session');
// Perform operations (all recorded)
await mem.put({ title: 'Notes', text: 'Content...' });
await mem.find('test query');
// Add checkpoint
await mem.sessionCheckpoint();
// End session
const summary = await mem.sessionEnd();
console.log(`Recorded ${summary.actionCount} actions`);
// Replay with different parameters
const replay = await mem.sessionReplay(sessionId, {
adaptive: true,
topK: 20
});
console.log(`Match rate: ${(replay.matchRate * 100).toFixed(1)}%`);
// Delete session
await mem.sessionDelete(sessionId);
```
## TypeScript Types
The SDK is fully typed:
```typescript theme={null}
import type {
PutInput,
PutManyInput,
FindInput,
AskInput,
MemoryCard,
MemoryCardInput,
EntityState,
FrameInfo,
TableInfo,
SessionSummary,
MemvidErrorCode
} from '@memvid/sdk';
const options: FindInput = {
mode: 'auto',
k: 5,
adaptive: true,
minRelevancy: 0.5
};
const putInput: PutInput = {
title: 'Document',
text: 'Content...',
enableEmbedding: true,
embeddingModel: 'bge-small'
};
```
## Error Handling
```typescript theme={null}
import {
MemvidError,
CapacityExceededError,
LockedError,
VecDimensionMismatchError,
EmbeddingFailedError,
EncryptedFileError
} from '@memvid/sdk';
try {
await mem.put({ title: 'Doc', text: 'Content' });
} catch (error) {
if (error instanceof CapacityExceededError) {
console.error('Storage full:', error.details);
} else if (error instanceof LockedError) {
console.error('File locked, try read-only mode');
} else if (error instanceof VecDimensionMismatchError) {
console.error('Embedding dimension mismatch:', error.details);
} else if (error instanceof EmbeddingFailedError) {
console.error('Embedding failed:', error.details);
} else if (error instanceof EncryptedFileError) {
console.error('File is encrypted, use unlock() first');
} else if (error instanceof MemvidError) {
console.error(`Error [${error.code}]: ${error.message}`);
} else {
throw error;
}
}
```
| Error Class | Code | Description |
| --------------------------- | ----- | ------------------------------ |
| `CapacityExceededError` | MV001 | Storage limit reached |
| `TicketInvalidError` | MV002 | Invalid ticket signature |
| `TicketReplayError` | MV003 | Ticket replay detected |
| `LexIndexDisabledError` | MV004 | Lexical search not enabled |
| `TimeIndexMissingError` | MV005 | Time index missing |
| `VerifyFailedError` | MV006 | Verification failed |
| `LockedError` | MV007 | File locked by another process |
| `ApiKeyRequiredError` | MV008 | API key required |
| `MemoryAlreadyBoundError` | MV009 | Memory already bound |
| `FrameNotFoundError` | MV010 | Requested frame doesn't exist |
| `VecIndexDisabledError` | MV011 | Vector search not enabled |
| `CorruptFileError` | MV012 | Corrupt file detected |
| `IOError` | MV013 | I/O error |
| `VecDimensionMismatchError` | MV014 | Wrong embedding dimension |
| `EmbeddingFailedError` | MV015 | Embedding generation failed |
| `EncryptedFileError` | MV016 | File is encrypted |
See [Error Reference](/errors/reference) for complete documentation.
## Environment Variables
| Variable | Description |
| ------------------- | ---------------------- |
| `MEMVID_API_KEY` | Dashboard API key |
| `OPENAI_API_KEY` | OpenAI API key |
| `GEMINI_API_KEY` | Google Gemini API key |
| `MISTRAL_API_KEY` | Mistral AI API key |
| `ANTHROPIC_API_KEY` | Anthropic API key |
| `COHERE_API_KEY` | Cohere API key |
| `VOYAGE_API_KEY` | Voyage AI API key |
| `NVIDIA_API_KEY` | NVIDIA API key |
| `MEMVID_MODELS_DIR` | Model cache directory |
| `MEMVID_OFFLINE` | Use cached models only |
## SDK Reference
<CardGroup>
<Card title="Overview" icon="book" href="/node-sdk/overview">
Complete API reference with all methods
</Card>
<Card title="Examples" icon="code" href="/node-sdk/examples">
TypeScript examples and patterns
</Card>
</CardGroup>
## Next Steps
<CardGroup>
<Card title="SDK Recipes" icon="flask" href="/quickstart/sdk-recipes">
Common patterns and recipes
</Card>
<Card title="Framework Integrations" icon="puzzle-piece" href="/frameworks/overview">
Vercel AI, LangChain, and more
</Card>
</CardGroup>
# Python SDK
Source: https://docs.memvid.com/sdks/python
Build AI applications with persistent memory in Python
The Python SDK provides a simple, Pythonic interface for working with Memvid memory files.
## Installation
```bash theme={null}
pip install memvid-sdk
```
<Info>
**Requirements:** Python 3.8+, macOS/Linux/Windows. Native bindings included.
</Info>
## Quick Start
```python theme={null}
import memvid_sdk as memvid
import os
# Create a new memory file
mem = memvid.create('knowledge.mv2')
# Add documents
mem.put(
title='Meeting Notes',
label='notes',
metadata={'source': 'slack'},
text='Alice mentioned she works at Anthropic...',
enable_embedding=True
)
# Search
results = mem.find('who works at AI companies?')
print(results['hits'])
# Ask questions with AI
answer = mem.ask(
'What does Alice do?',
model='gpt-4o-mini',
api_key=os.environ['OPENAI_API_KEY']
)
print(answer['text'])
# Close when done
mem.close()
```
## Context Manager
```python theme={null}
import memvid_sdk as memvid
# Automatically closes when done
with memvid.use('basic', 'memory.mv2') as mem:
mem.put(title='Doc', label='test', metadata={}, text='Content')
results = mem.find('query')
```
## API Reference
| Category | Methods | Description |
| -------------------- | -------------------------------------------------------------------------------- | -------------------------------- |
| **File Operations** | `create`, `use`, `close` | Create, open, close memory files |
| **Data Ingestion** | `put`, `put_many` | Add documents with embeddings |
| **Search** | `find`, `ask`, `timeline` | Query your memory |
| **Memory Cards** | `memories`, `state`, `enrich`, `add_memory_cards` | Structured fact extraction |
| **Tables** | `put_pdf_tables`, `list_tables`, `get_table` | PDF table extraction |
| **Sessions** | `session_start`, `session_end`, `session_replay` | Time-travel debugging |
| **Tickets** | `sync_tickets`, `current_ticket`, `get_capacity` | Capacity management |
| **Cloud Management** | `configure`, `create_project`, `list_projects`, `create_memory`, `list_memories` | Dashboard API |
| **Utilities** | `verify`, `doctor`, `mask_pii` | Maintenance and utilities |
## Framework Adapters
```python theme={null}
# LangChain
mem = memvid.use('langchain', 'knowledge.mv2')
retriever = mem.as_retriever()
# LlamaIndex
mem = memvid.use('llamaindex', 'knowledge.mv2')
query_engine = mem.as_query_engine()
# CrewAI
mem = memvid.use('crewai', 'knowledge.mv2')
tools = mem.tools
# AutoGen
mem = memvid.use('autogen', 'knowledge.mv2')
# Haystack
mem = memvid.use('haystack', 'knowledge.mv2')
```
## Embedding Providers
```python theme={null}
from memvid_sdk.embeddings import (
OpenAIEmbeddings,
GeminiEmbeddings,
MistralEmbeddings,
CohereEmbeddings,
VoyageEmbeddings,
NvidiaEmbeddings,
LOCAL_EMBEDDING_MODELS
)
import os
# OpenAI
openai = OpenAIEmbeddings(api_key=os.environ['OPENAI_API_KEY'])
# Gemini
gemini = GeminiEmbeddings(api_key=os.environ['GEMINI_API_KEY'])
# Mistral
mistral = MistralEmbeddings(api_key=os.environ['MISTRAL_API_KEY'])
# Local models (no API required)
mem.put(
title='Doc',
label='test',
metadata={},
text='Content',
enable_embedding=True,
embedding_model=LOCAL_EMBEDDING_MODELS['BGE_SMALL']
)
```
**Local Embedding Models:**
| Model | Dimensions | Speed | Quality |
| ----------- | ---------- | ------- | ------- |
| `BGE_SMALL` | 384 | Fastest | Good |
| `BGE_BASE` | 768 | Fast | Better |
| `NOMIC` | 768 | Fast | Better |
| `GTE_LARGE` | 1024 | Slower | Best |
## Entity Extraction
```python theme={null}
# Extract facts using rules engine
result = mem.enrich('rules')
# View extracted cards
cards = mem.memories()
print(f"Extracted {cards['count']} memory cards")
# Get entity state (O(1) lookup)
alice = mem.state('Alice')
print(alice['slots'])
# {'employer': 'Anthropic', 'role': 'Engineer'}
# Add memory cards manually
mem.add_memory_cards([
{'entity': 'Alice', 'slot': 'employer', 'value': 'Anthropic'},
{'entity': 'Bob', 'slot': 'team', 'value': 'Infrastructure'}
])
```
## Session Recording
Record and replay agent sessions to debug RAG failures:
```python theme={null}
# Start recording session
session_id = mem.session_start("Debug Session")
# Perform operations (all recorded)
mem.put(title="Meeting Notes", label="notes", metadata={}, text="Discussed Q4...")
results = mem.find("roadmap", k=5)
# Add checkpoints at key moments
mem.session_checkpoint()
# End session
summary = mem.session_end()
print(f"Recorded {summary['action_count']} actions")
# Replay with different parameters
replay_result = mem.session_replay(
session_id,
adaptive=True,
top_k=20
)
print(f"Match rate: {replay_result['match_rate']:.1%}")
# Delete session when done
mem.session_delete(session_id)
```
## Error Handling
```python theme={null}
from memvid_sdk import (
CapacityExceededError,
LockedError,
VecDimensionMismatchError,
EmbeddingFailedError,
MemvidError
)
try:
mem.put(title='Doc', label='test', metadata={}, text='Content')
except CapacityExceededError:
print('Storage limit reached')
except LockedError:
print('File locked by another process')
except VecDimensionMismatchError:
print('Embedding dimension mismatch')
except EmbeddingFailedError:
print('Embedding generation failed')
except MemvidError as e:
print(f'Error [{e.code}]: {e.message}')
```
| Error Class | Code | Description |
| --------------------------- | ----- | ------------------------------ |
| `CapacityExceededError` | MV001 | Storage limit reached |
| `TicketInvalidError` | MV002 | Invalid ticket signature |
| `LexIndexDisabledError` | MV004 | Lexical search not enabled |
| `LockedError` | MV007 | File locked by another process |
| `FrameNotFoundError` | MV010 | Requested frame doesn't exist |
| `VecIndexDisabledError` | MV011 | Vector search not enabled |
| `VecDimensionMismatchError` | MV014 | Wrong embedding dimension |
| `EmbeddingFailedError` | MV015 | Embedding generation failed |
See [Error Reference](/errors/reference) for complete documentation.
## Environment Variables
| Variable | Description |
| ------------------- | ---------------------- |
| `MEMVID_API_KEY` | Dashboard API key |
| `OPENAI_API_KEY` | OpenAI API key |
| `GEMINI_API_KEY` | Google Gemini API key |
| `MISTRAL_API_KEY` | Mistral AI API key |
| `ANTHROPIC_API_KEY` | Anthropic API key |
| `COHERE_API_KEY` | Cohere API key |
| `VOYAGE_API_KEY` | Voyage AI API key |
| `NVIDIA_API_KEY` | NVIDIA API key |
| `MEMVID_MODELS_DIR` | Model cache directory |
| `MEMVID_OFFLINE` | Use cached models only |
## Type Hints
The SDK includes full type hints for IDE support:
```python theme={null}
from typing import Dict, Any
def process_memory(path: str) -> Dict[str, Any]:
mem = memvid.use('basic', path)
results: Dict[str, Any] = mem.find('query')
return results
```
## SDK Reference
<CardGroup>
<Card title="Overview" icon="book" href="/python-sdk/overview">
Complete API reference with all methods
</Card>
<Card title="Querying" icon="magnifying-glass" href="/python-sdk/querying">
Search modes, filters, and retrieval patterns
</Card>
</CardGroup>
## Next Steps
<CardGroup>
<Card title="SDK Recipes" icon="flask" href="/quickstart/sdk-recipes">
Common patterns and recipes
</Card>
<Card title="Framework Integrations" icon="puzzle-piece" href="/frameworks/overview">
LangChain, LlamaIndex, and more
</Card>
</CardGroup>
# Crash harness
Source: https://docs.memvid.com/testing/crash-harness
Prove WAL recovery and timeline integrity
The memvid-core Dev Steps require a crash harness that runs 1 000 cycles of mutate/crash/recover with zero data loss.
### Harness expectations
* Randomly ingest frames, call `commit()` at varied intervals
* Kill the process mid-commit to leave partial WAL writes
* Restart, let `Memvid::open()` replay the WAL, and assert:
* `frame_count` matches expectations
* `timeline()` returns ordered timestamps (verifies Time Index Track persisted)
* Checksums match (header, TOC, time index)
### Sample assertion
```rust theme={null}
use std::num::NonZeroU64;
use memvid_core::{Memvid, TimelineQuery};
let mut mv = Memvid::open(path).unwrap();
let stats = mv.stats().unwrap();
assert_eq!(stats.frame_count, expected_frames);
let timeline = mv
.timeline(
TimelineQuery::builder()
.limit(NonZeroU64::new(100).unwrap())
.build(),
)
.unwrap();
assert!(timeline.windows(2).all(|w| w[0].timestamp <= w[1].timestamp));
```
> **Performance target** - `open_and_recover_latency` must stay below 250 ms, even when replaying WAL entries (Security & Performance Architecture).
# Determinism sweeps
Source: https://docs.memvid.com/testing/determinism-sweeps
Stay byte-for-byte consistent across builds
Determinism is a top-level requirement across memvid-core, CLI, and bindings (see Security & Performance Architecture + Developer Implementation Guide).
### What to test
1. Run ingestion twice with identical inputs → resulting `.mv2` files must have identical BLAKE3 hashes
2. Run `memvid find`/`timeline`/`ask` with the same seeds → identical ordering and scores
3. Run SDK operations (Python + Node) → `stats`, `verify`, `doctor` outputs should match CLI JSON exactly
### Suggested harness
Run the determinism harness in CI (the same workflow invoked before every release) to compare hash outputs between builds. The suite should fail if `toc_checksum`, `merkle_root`, or `ticket_ref` fields differ.
> **Dependencies** - Avoid `#[serde(skip)]` or conditional compilation in serialized structs. The Developer Implementation Guide calls out how serde + bincode must be used to keep layouts stable.
# Golden Test Pack
Source: https://docs.memvid.com/testing/golden-pack
Authoritative integration tests for memvid-core + CLI
The Golden Test Pack enforces deterministic behavior for the open-source CLI and core.
### Golden corpus
```
test_corpus/
├── short_1.txt ("Hello world.")
├── short_2.txt ("This is a simple test.")
├── chapter_1.txt (Stardust paragraph)
├── event_log.json
├── users.csv
├── multilingual.txt
└── image.blob (1024 pseudo-random bytes)
```
### CLI matrix
| Test name | Scenario |
| ------------------------ | ---------------------------------------------------------------------------------------- |
| Create + info | `memvid create` followed by `memvid info --json` to confirm an empty, Free-tier file |
| Put + enable-lex + find | Pipe input via `memvid put`, enable lexical search, and run `memvid find` |
| Find (no results) | Ensure unknown queries return `[]` |
| Enable-lex toggles state | `memvid enable-lex` flips `hasLexIndex` in `memvid info --json` |
| Capacity exceeded | Mock a low-capacity memory and assert `Error: CapacityExceeded` exit code |
| Tickets apply | Apply a valid ticket and confirm `tier` upgrades |
| Unknown flag | Passing an unsupported flag returns the CLI usage error (exit code 2) |
| Verify + doctor | `memvid verify --deep` detects corruption and `memvid doctor` repairs the timeline track |
| Import structured file | `memvid import --format jsonl` ingests structured data |
| Single-file guarantee | `memvid verify-single-file` confirms there are no `.wal`/`.lock` helpers |
Run these tests automatically in CI whenever CLI or core changes land. Use the published CLI JSON schemas (the same ones documented in this site) to check outputs verbatim.
> **Dataset generation** - Keep the downloadable `test_corpus.zip` fixture up to date; corruption or formatting drift invalidates the golden pack.
# Troubleshooting - CLI
Source: https://docs.memvid.com/troubleshooting/cli
Common errors, solutions, and diagnostic commands
This guide covers common CLI errors and how to resolve them.
## Exit Codes
The CLI uses specific exit codes to indicate error types:
| Exit Code | Meaning | Description |
| --------- | ---------- | --------------------------------------------- |
| 0 | Success | Command completed successfully |
| 1 | Generic | General error |
| 2 | Capacity | Storage capacity exceeded or API key required |
| 3 | Lock | File lock conflict |
| 4 | Corruption | File corruption detected |
## Common Errors
### Capacity Exceeded
**Symptom:**
```
Error: CapacityExceeded
Current usage: 950 MB
Capacity limit: 1 GB
Required space: 100 MB
```
**Causes:**
* File has reached its storage limit
* Adding content that would exceed capacity
**Solutions:**
1. **Delete unused frames:**
```bash theme={null}
# List frames to find candidates for deletion
memvid timeline myfile.mv2 --limit 100
# Delete a specific frame
memvid delete myfile.mv2 --frame-id 42 --yes
# Compact the file to reclaim space
memvid doctor myfile.mv2 --vacuum
```
2. **Create a new file with larger capacity:**
```bash theme={null}
# Create with specific size
memvid create newfile.mv2 --size 2GB
```
3. **Check current usage:**
```bash theme={null}
memvid stats myfile.mv2
```
### Lock Errors
**Symptom:**
```
Error: File is locked by another process
```
**Causes:**
* Another process is writing to the file
* A previous process crashed without releasing the lock
* Lock is held by a stale process
**Solutions:**
1. **Check who holds the lock:**
```bash theme={null}
memvid who myfile.mv2
```
2. **Request the writer to release:**
```bash theme={null}
memvid nudge myfile.mv2
```
3. **Find the process holding the lock:**
```bash theme={null}
# macOS/Linux
lsof myfile.mv2
```
4. **Wait and retry with timeout:**
```bash theme={null}
memvid put myfile.mv2 --input doc.pdf --lock-timeout 5000
```
5. **Force takeover of stale lock:**
```bash theme={null}
memvid put myfile.mv2 --input doc.pdf --force
```
<Warning>
Only use `--force` if you're certain the previous writer has crashed. Forcing a lock on an active writer can cause corruption.
</Warning>
### Corrupted File
**Symptom:**
```
Error: CorruptToc - Table of Contents checksum mismatch
Error: InvalidHeader - Header magic bytes invalid
```
**Causes:**
* File was corrupted during transfer
* Crash during write operation
* Disk errors
**Solutions:**
1. **Verify the file:**
```bash theme={null}
# Quick verification
memvid verify myfile.mv2
# Deep verification
memvid verify myfile.mv2 --deep
```
2. **Run the doctor:**
```bash theme={null}
# Preview repairs
memvid doctor myfile.mv2 --plan-only
# Rebuild specific index
memvid doctor myfile.mv2 --rebuild-time-index
memvid doctor myfile.mv2 --rebuild-lex-index
memvid doctor myfile.mv2 --rebuild-vec-index
```
3. **Verify single-file integrity:**
```bash theme={null}
memvid verify-single-file myfile.mv2
```
### Time Index Issues
**Symptom:**
```
TimeIndexSortOrder: Failed in memvid verify --deep
Error: TimeIndexMissing
```
**Solution:**
```bash theme={null}
memvid doctor myfile.mv2 --rebuild-time-index
memvid verify myfile.mv2 --deep
```
### Search Returns Empty Results
**Symptom:**
```bash theme={null}
$ memvid find myfile.mv2 --query "test"
[]
```
**Causes:**
* Lexical index not built
* Content not indexed
* Query doesn't match any content
**Solutions:**
1. **Check index status:**
```bash theme={null}
memvid stats myfile.mv2 --json | grep has_lex_index
```
2. **Rebuild the lexical index:**
```bash theme={null}
memvid doctor myfile.mv2 --rebuild-lex-index
```
3. **Try different search modes:**
```bash theme={null}
# Lexical only
memvid find myfile.mv2 --query "test" --mode lex
# Semantic only
memvid find myfile.mv2 --query "test" --mode sem
# Hybrid (default)
memvid find myfile.mv2 --query "test" --mode auto
```
4. **Check if content exists:**
```bash theme={null}
memvid timeline myfile.mv2 --limit 5
memvid view myfile.mv2 --frame-id 1
```
### Invalid Flag Error
**Symptom:**
```
error: unexpected argument '--nonexistent-flag'
```
**Solution:**
```bash theme={null}
memvid <command> --help
```
### File Already Exists
**Symptom:**
```
Error: File 'myfile.mv2' already exists
```
**Solution:**
```bash theme={null}
# Remove existing file first
rm myfile.mv2
memvid create myfile.mv2
# Or use a different name
memvid create myfile-v2.mv2
```
## Diagnostic Commands
### Check File Health
```bash theme={null}
# Basic stats
memvid stats myfile.mv2
# JSON output for scripting
memvid stats myfile.mv2 --json
# Verify integrity
memvid verify myfile.mv2
# Deep verification
memvid verify myfile.mv2 --deep
# Check for sidecar files
memvid verify-single-file myfile.mv2
```
### Inspect Lock State
```bash theme={null}
# Who holds the lock?
memvid who myfile.mv2
# Request release
memvid nudge myfile.mv2
```
### Doctor Commands
```bash theme={null}
# Preview what would be fixed
memvid doctor myfile.mv2 --plan-only
# Rebuild time index
memvid doctor myfile.mv2 --rebuild-time-index
# Rebuild lexical index
memvid doctor myfile.mv2 --rebuild-lex-index
# Rebuild vector index
memvid doctor myfile.mv2 --rebuild-vec-index
# Compact deleted frames
memvid doctor myfile.mv2 --vacuum
# Fix multiple issues
memvid doctor myfile.mv2 --rebuild-time-index --rebuild-lex-index
```
### View Frame Details
```bash theme={null}
# View by frame ID
memvid view myfile.mv2 --frame-id 1
# View by URI
memvid view myfile.mv2 --uri "mv2://docs/readme.md"
# JSON output
memvid view myfile.mv2 --frame-id 1 --json
# Preview media
memvid view myfile.mv2 --frame-id 1 --preview
```
## Memory and Performance
### Increasing Memory for Large Files
For large ingestion operations, you may need to increase available memory:
**Environment variables:**
```bash theme={null}
# Set model cache directory
export MEMVID_MODELS_DIR=~/.memvid/models
# Enable offline mode (skip model downloads)
export MEMVID_OFFLINE=1
# Control parallel ingestion
export MEMVID_PARALLEL_SEGMENTS=1
```
**CLI flags for parallel ingestion:**
```bash theme={null}
memvid put myfile.mv2 --input large-dataset/ \
--parallel-segments \
--parallel-threads 4 \
--parallel-queue-depth 8
```
### Optimizing Ingestion Speed
1. **Use batch operations:**
```bash theme={null}
# Ingest entire directory at once
memvid put myfile.mv2 --input ./docs/
```
2. **Enable parallel segments:**
```bash theme={null}
memvid put myfile.mv2 --input ./docs/ --parallel-segments
```
3. **Skip embeddings for faster ingestion:**
```bash theme={null}
memvid put myfile.mv2 --input ./docs/ --no-embedding
```
4. **Use vector compression for smaller files:**
```bash theme={null}
memvid put myfile.mv2 --input ./docs/ --vector-compression
```
## Cleaning Up Stale Files
If you see leftover files from older versions:
```bash theme={null}
# Check for sidecars
ls -la myfile.mv2*
# Remove any leftover files
rm -f myfile.mv2-wal myfile.mv2-shm myfile.mv2.lock
# Verify clean state
memvid verify-single-file myfile.mv2
```
## Getting Help
```bash theme={null}
# General help
memvid --help
# Command-specific help
memvid create --help
memvid put --help
memvid find --help
memvid doctor --help
# Version info
memvid version
```
## Verbose Output
Enable debug logging for troubleshooting:
```bash theme={null}
# Increase verbosity
memvid -v find myfile.mv2 --query "test" # WARN
memvid -vv find myfile.mv2 --query "test" # INFO
memvid -vvv find myfile.mv2 --query "test" # DEBUG
memvid -vvvv find myfile.mv2 --query "test" # TRACE
```