> ## Documentation Index
> Fetch the complete documentation index at: https://docs.memvid.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Memory Architecture

> How .mv2 files are structured internally

Understanding how Memvid stores data helps you make better decisions about ingestion, search, and performance optimization.

## File Structure

A `.mv2` file is a single, self-contained binary with five main layers:

```mermaid theme={null}
graph TD
  A["Header (4 KB)"] --> B["Embedded WAL"]
  B --> C["Segments (Frames)"]
  C --> D["Indices"]
  D --> E["Table of Contents + Footer"]
```

### 1. Header (4 KB)

The header contains:

* **Magic bytes**: Identifies the file as `.mv2` format
* **Version**: File format version
* **WAL metadata**: Position and size of write-ahead log
* **Footer offset**: Points to the table of contents

### 2. Embedded WAL

The write-ahead log (WAL) ensures crash safety:

* All mutations are written to WAL first
* On recovery, uncommitted changes are replayed
* Size scales with file capacity (1 MB to 64 MB)

### 3. Segments (Frames)

Your actual data lives in segments, which contain frames (the fundamental unit of storage):

* **Text segments**: Document content and metadata stored as frames
* **Blob segments**: Binary data (images, PDFs) as frames
* **Media segments**: Audio and video content as frames
* **Vector segments**: Embeddings for semantic search (optional)

Each frame contains payload, metadata, timestamp, URI, and checksum. Segments are written in deterministic order for reproducibility.

### 4. Indices

Memvid maintains multiple indices for fast search:

* **Lexical index (BM25)**: Full-text keyword search - works out of the box
* **Time index**: Temporal ordering of frames
* **Vector index**: Semantic similarity search - **optional**, add when needed

### 5. Table of Contents + Footer

The TOC maps everything:

* Segment locations and sizes
* Index offsets
* Checksums for integrity verification

The footer contains a final checksum and magic trailer (`MV2FOOT!`).

***

## Data Lifecycle

### Writing Data

When you add documents:

1. **put()** - Adds frames (documents) to pending state
2. **Indices updated** - Lexical and vector indices are built
3. **Time entries queued** - Timestamps recorded for timeline
4. **WAL appended** - Transaction logged for crash safety
5. **seal()** - Commits everything to disk with checksums

```python theme={null}
from memvid_sdk import use

mem = use('basic', 'knowledge.mv2')

# 1. Add documents (pending)
mem.put(title="Doc 1", label="docs", metadata={}, text="Your content")
mem.put(title="Report", label="docs", metadata={}, file="report.pdf")

# 2. Commit to disk
mem.seal()
```

### Reading Data

When you search or retrieve:

1. **Open file** - Locate latest valid footer
2. **Load TOC** - Map segments and indices
3. **Replay WAL** - Apply any uncommitted changes
4. **Query indices** - Search lexical/vector/time indices
5. **Return results** - Ranked documents with snippets

***

## Single-File Guarantee

Memvid's core promise is **single-file portability**:

### What It Means

* **No sidecar files**: No `.wal`, `.lock`, `.shm` files
* **No external state**: Everything is in the `.mv2` file
* **Portable**: Copy the file to transfer the entire memory

### Why It Matters

```bash theme={null}
# Your entire knowledge base
ls ~/project/
# → knowledge.mv2

# Share it anywhere
cp knowledge.mv2 /team/shared/
scp knowledge.mv2 user@server:/data/
git add knowledge.mv2
```

### How It Works

Traditional databases use separate files for journals, locks, and indices. Memvid embeds all of these inside the `.mv2` file:

| Traditional DB                      | Memvid             |
| ----------------------------------- | ------------------ |
| data.db + data.db-wal + data.db-shm | knowledge.mv2      |
| Requires careful copying            | Just copy the file |

***

## Crash Safety

The embedded WAL ensures data survives unexpected shutdowns.

### Write-Ahead Logging

Every mutation is logged before being applied:

1. Transaction written to WAL region
2. WAL synced to disk (fsync)
3. Changes applied to segments
4. Checksum updated

### Recovery Process

On open, Memvid:

1. Locates the last valid footer
2. Loads the table of contents
3. Scans WAL for uncommitted entries
4. Replays any pending transactions

This guarantees that your data is safe even after crashes or power failures.

### WAL Sizing

WAL size scales with file capacity:

| File Size     | WAL Size |
| ------------- | -------- |
| Under 100 MB  | 1 MB     |
| Under 1 GB    | 4 MB     |
| Under 10 GB   | 16 MB    |
| 10 GB or more | 64 MB    |

***

## Locking and Concurrency

### File Locking

Memvid uses OS-level file locks:

* **Shared locks**: Multiple readers allowed
* **Exclusive locks**: Single writer at a time

### Read-Only Mode

For concurrent read access:

```python theme={null}
# Multiple processes can read simultaneously
mem = use('basic', 'knowledge.mv2', read_only=True)
results = mem.find('query')
```

### Writer Conflicts

If a writer holds the lock:

```python theme={null}
from memvid_sdk import use, LockedError

try:
    mem = use('basic', 'knowledge.mv2')
except LockedError:
    print("File is locked by another process")
```

***

## Determinism

Given the same inputs, Memvid produces the same outputs.

### Why Determinism Matters

* **Reproducible builds**: Same data → same file
* **Reliable testing**: Predictable behavior
* **Easy debugging**: Consistent results

### How It's Achieved

* Segments written in deterministic order
* Timestamps explicit, not system-derived
* Checksums verify integrity

***

## Performance Considerations

### Memory Usage

Memvid keeps some data in memory:

* Table of contents
* WAL handle
* Pending time entries

For large files, consider:

* Closing handles when done
* Using read-only mode for queries

### Index Building

Building indices is CPU-intensive:

* **Lexical index**: BM25 tokenization and indexing
* **Vector index**: Graph construction for similarity search

Use parallel ingestion for large datasets:

```bash theme={null}
memvid put knowledge.mv2 --input ./large-dataset/ \
  --vector-compression \
  --parallel-segments
```

### Search Optimization

* **Lexical search**: Fast for exact keywords
* **Vector search**: Slower but more intelligent
* **Hybrid search**: Balances both

Choose the right mode for your query.

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Indices & Tracks" icon="list" href="/concepts/indexes-and-tracks">
    Learn about lexical, vector, and time indices
  </Card>

  <Card title="Storage Capacity" icon="database" href="/concepts/capacity-and-plans">
    Understand storage tiers and capacity management
  </Card>
</CardGroup>