Glossary

​

Architecture Components

​

Memvid Core

• File format handling - Reading/writing .mv2 files
• Indexing engines - Lexical (Tantivy), vector (HNSW), and hybrid search
• WAL management - Write-ahead logging for crash safety
• Enrichment pipeline - Background processing for embeddings and extraction
• Memory management - Frame storage, versioning, and lifecycle

​

CLI (Command Line Interface)

# Create a new memory file
memvid create my-knowledge.mv2

# Add content
memvid put my-knowledge.mv2 --input document.pdf

# Search
memvid find my-knowledge.mv2 --query "machine learning"

# Ask questions
memvid ask my-knowledge.mv2 --question "What is the main thesis?"

​

SDKs (Software Development Kits)

• create() / use() - Create or open memory files
• put() / put_many() - Insert documents
• find() - Search with various modes
• ask() - AI-powered Q&A
• timeline() - Browse insertion history

​

File Format

​

MV2 (Memory File)

• All your documents and data (frames)
• Search indices (lexical, vector, temporal)
• Metadata and checksums
• Write-ahead log for crash recovery

• Portable - Copy, move, or share as a single file
• Serverless - No database server required
• Crash-safe - WAL ensures data integrity
• Deterministic - Reproducible builds for verification

​

MV2E (Encrypted Memory File)

• Argon2 for password-based key derivation
• AES-GCM for authenticated encryption

​

Core Concepts

​

Frame

1. Insert - Frame created with frame_id, status = Active
2. Update - Original marked Superseded, new frame created
3. Delete - Status changed to Deleted (soft delete)

​

Memory

• Reading and writing frames
• Managing indices
• Coordinating search
• Handling commits and checkpoints

// Open a memory
const mem = await memvid.use("knowledge.mv2");

// The 'mem' object is your Memory instance
await mem.put({ text: "Hello world" });
await mem.find({ query: "hello" });

​

Commit

1. WAL entries written to disk
2. Indices updated
3. Footer rewritten with new checksums
4. File synced to storage

​

Checkpoint

• Frees WAL space for new writes
• Marks transactions as permanently durable
• Triggered automatically when WAL reaches 75% capacity

​

Search & Retrieval

​

Lexical Search

• How it works: Builds an inverted index of terms, scores documents by term frequency and inverse document frequency
• Best for: Exact matches, specific keywords, technical terms
• Engine: Tantivy (Rust full-text search library)

memvid find data.mv2 --query "API authentication" --mode lex

​

Semantic Search (Vector Search)

• How it works: Converts text to vector embeddings, finds similar vectors using cosine similarity
• Best for: Conceptual queries, finding related content, natural language questions
• Index: HNSW (Hierarchical Navigable Small World) graph

memvid find data.mv2 --query "how to secure endpoints" --mode vec

​

Hybrid Search

• Runs both search types in parallel
• Normalizes scores from each
• Merges using RRF (Reciprocal Rank Fusion)
• Returns unified ranked results

memvid find data.mv2 --query "authentication best practices" --mode hybrid

​

Sketch Pre-filtering

• SimHash: 64-bit locality-sensitive hash for quick similarity checks
• Term Filter: Compact bitset for query term overlap
• Top Terms: Hashed IDs of highest-weight terms

​

Indexing

​

Lex Index

• Tokenizes text into terms
• Builds inverted index (term → document list)
• Supports field-specific queries (title, content, tags)
• Deterministic chunking for reproducibility

​

Vec Index

• Stores document embeddings
• Uses HNSW graph for approximate nearest neighbor search
• Supports multiple embedding models (BGE, Nomic, OpenAI)
• Optional product quantization for compression

​

Time Index

• Tracks insertion timestamps
• Enables range queries (since/until)
• Powers timeline() navigation
• Supports forward and reverse traversal

​

Sketch Track

​

Enrichment

​

Enrichment Pipeline

1. Searchable (instant) - Skim text extracted, basic indexing
2. Enriched (background) - Full text, embeddings, memory cards, entities

​

Memory Cards

​

Entity Extraction (NER)

• Types: Person, Organization, Location, Date, Money, URL, etc.
• Model: DistilBERT-NER (ONNX)
• Output: Entities with confidence scores and frame references

​

Logic Mesh

• Bidirectional graph structure
• Nodes: Entities with types and mentions
• Edges: Relationships with confidence
• Enables: “follow” queries for fact traversal

​

Persistence

​

WAL (Write-Ahead Log)

• Purpose: Records mutations before they’re applied
• Checksum: BLAKE3 hash for integrity verification
• Recovery: Replays uncommitted entries after crash
• Size: Configurable (64 KB to 64 MB)

​

Header

• Magic bytes (MV2\0)
• Spec and format versions
• WAL offset and size
• Footer offset pointer

​

Footer

• Table of Contents (TOC)
• Manifest pointers
• Segment catalog
• Checksums for validation

​

TOC (Table of Contents)

• Lists all frames with metadata
• References to index manifests (lex, vec, time)
• Segment catalog for published indices

​

Capacity & Licensing

​

Ticket

• Issuer: Authority that granted the capacity
• Sequence: Monotonic identifier
• Capacity: Bytes allowed
• Signature: ED25519 digital signature

​

Capacity Tiers

​

Embedding Models

​

Local Models

​

Cloud Models

​

CLIP Models

​

Operations

​

put / put_many

// Single document
const frameId = await mem.put({ text: "Hello world" });

// Batch insert
const frameIds = await mem.put_many([
  { text: "Doc 1" },
  { text: "Doc 2" },
  { file: "document.pdf" }
]);

​

find

const results = await mem.find({
  query: "machine learning",
  k: 10,
  mode: "hybrid",
  snippet_chars: 200
});

​

ask

const answer = await mem.ask({
  question: "What are the key findings?",
  use_model: "openai"
});

​

timeline

// Recent frames
const recent = await mem.timeline({ limit: 20 });

// Oldest first
const oldest = await mem.timeline({ limit: 20, reverse: true });

​

seal

await mem.seal();

​

Feature Flags

​

Common Workflows

​

Ingestion → Search → Ask

1. Create memory       → memvid.create("data.mv2")
2. Insert documents    → mem.put({ file: "docs/*.pdf" })
3. Wait for enrichment → (automatic background processing)
4. Search              → mem.find({ query: "..." })
5. Ask questions       → mem.ask({ question: "..." })
6. Close               → mem.seal()

​

Enrichment Pipeline

1. put()           → Frame enters Searchable state
2. Background      → Worker extracts full text
3. Embedding       → Generate vector embeddings
4. Extraction      → Memory cards, entities
5. Indexed         → Frame now fully Enriched

​

See Also