> ## Documentation Index
> Fetch the complete documentation index at: https://docs.memvid.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Benchmarks
> How Memvid compares to leading vector databases
**TL;DR**: Memvid reduces retrieval errors by **50-66%** compared to leading vector databases while delivering **4x better accuracy-per-millisecond** than alternatives.
We benchmarked Memvid against four popular vector databases on the **Wikipedia dataset** (39,324 documents, 2,500 queries). The results speak for themselves.
***
## Headline Results
**Top-1 Accuracy**
Memvid finds the correct document on the first try 92.7% of the time, 8.5 to 14.5 points higher than any competitor.
**Error Reduction**
Memvid reduces retrieval errors by 50-66% compared to Chroma, Weaviate, LanceDB, and Qdrant.
**Accuracy per ms**
Memvid delivers 4.2x more accuracy per millisecond than Chroma, the best quality-to-latency ratio.
**MRR Score**
Mean Reciprocal Rank of 0.95. The correct result consistently appears in the top 1-2 hits.
***
## Accuracy Comparison
### Top-1 Accuracy (Higher is Better)
The most important metric: **does the system return the correct document first?**
| System | Accuracy\@1 | vs Memvid |
| ---------- | ----------- | --------- |
| **Memvid** | **92.72%** | baseline |
| LanceDB | 84.24% | -8.5 pts |
| Qdrant | 84.24% | -8.5 pts |
| Weaviate | 80.68% | -12.0 pts |
| Chroma | 78.24% | -14.5 pts |
### Error Reduction
Another way to look at accuracy: how often does the system get it wrong?
| System | Error Rate | Memvid Reduces Errors By |
| ---------- | ---------- | ------------------------ |
| Chroma | 21.76% | **66%** fewer errors |
| Weaviate | 19.32% | **62%** fewer errors |
| LanceDB | 15.76% | **54%** fewer errors |
| Qdrant | 15.76% | **54%** fewer errors |
| **Memvid** | **7.28%** | baseline |
**Memvid returns the wrong answer 3x less often than Chroma.**
### Accuracy at Different k Values
| System | @1 | @3 | @5 | @10 |
| ---------- | ---------- | ---------- | ---------- | ---------- |
| **Memvid** | **92.72%** | **96.96%** | **97.56%** | **98.12%** |
| LanceDB | 84.24% | 92.72% | 94.60% | 96.24% |
| Qdrant | 84.24% | 92.72% | 94.60% | 96.24% |
| Weaviate | 80.68% | 88.52% | 90.16% | 91.52% |
| Chroma | 78.24% | 85.80% | 87.28% | 88.68% |
***
## Ranking Quality
### MRR (Mean Reciprocal Rank)
MRR measures how high the correct result appears in the ranking. A score of 1.0 means perfect top-1 placement every time.
| System | MRR | Interpretation |
| ---------- | --------- | ----------------------------- |
| **Memvid** | **0.949** | Correct result typically #1 |
| LanceDB | 0.888 | Correct result typically #1-2 |
| Qdrant | 0.888 | Correct result typically #1-2 |
| Weaviate | 0.849 | Correct result typically #2 |
| Chroma | 0.823 | Correct result typically #2 |
### NDCG\@10 (Normalized Discounted Cumulative Gain)
NDCG measures overall ranking quality across the top 10 results.
| System | NDCG\@10 |
| ---------- | --------- |
| **Memvid** | **0.967** |
| LanceDB | 0.929 |
| Qdrant | 0.929 |
| Weaviate | 0.886 |
| Chroma | 0.858 |
**Only Memvid achieves >0.94 MRR and >0.96 NDCG\@10** in this benchmark. Memvid consistently surfaces the correct result in the top 1-2 hits with near-perfect ranking quality.
***
## Latency Performance
### Query Latency (Lower is Better)
| System | p50 | p95 | p99 | QPS |
| ---------- | ---------- | ---------- | ---------- | ------ |
| Weaviate | 5.3ms | 7.1ms | 7.9ms | 180 |
| **Memvid** | **16.0ms** | **17.4ms** | **19.7ms** | **61** |
| LanceDB | 16.0ms | 17.5ms | 19.3ms | 61 |
| Qdrant | 28.0ms | 30.4ms | 31.4ms | 36 |
| Chroma | 55.6ms | 61.0ms | 65.2ms | 18 |
### Cold Start Time
| System | Cold Start |
| ---------- | ---------- |
| **Memvid** | **0.5ms** |
| Chroma | 66.3ms |
| Qdrant | 71.8ms |
| LanceDB | 72.4ms |
| Weaviate | 147.7ms |
**Memvid cold-starts 130-300x faster** than alternatives. This matters for serverless deployments and edge computing where startup time is critical.
***
## The Efficiency Frontier
### Accuracy per Millisecond
We compute accuracy divided by p95 latency to measure quality-per-latency:
| System | Accuracy\@1 | p95 Latency | Accuracy/ms |
| ---------- | ----------- | ----------- | ----------- |
| **Memvid** | 92.72% | 17.4ms | **0.053** |
| Weaviate | 80.68% | 7.1ms | 0.114 |
| LanceDB | 84.24% | 17.5ms | 0.048 |
| Qdrant | 84.24% | 30.4ms | 0.028 |
| Chroma | 78.24% | 61.0ms | 0.013 |
**Memvid delivers 4.2x more accuracy per millisecond than Chroma** and leads all systems except Weaviate (which sacrifices 12 points of accuracy for speed).
If accuracy matters, Memvid is the clear winner. If you can tolerate 12% worse results, Weaviate is faster.
### The Frontier Chart
```
Accuracy@1 (%)
│
93% │ ★ Memvid
│
85% │ ● LanceDB ● Qdrant
│
81% │ ● Weaviate
│
78% │ ● Chroma
│
└────────────────────────────────────────
5ms 17ms 28ms 61ms
p95 Latency →
```
**Memvid sits alone at the accuracy frontier.** No other system achieves >90% accuracy at any latency.
***
## Storage Efficiency
### Single-File Advantage
Memvid is unique among memory systems because everything lives inside a single portable file; (data, embeddings, indices, and metadata) There are no sidecar files, external indexes, database directories, or hidden dependencies. One file contains the entire memory: easy to move, copy, version, share, or embed into an agent.
| System | Storage | Compression | Bytes/Doc |
| ---------- | ---------- | ----------- | ---------- |
| LanceDB | 213 MB | 0.71x | 5,428 |
| Qdrant | 212 MB | 0.72x | 5,396 |
| **Memvid** | **508 MB** | **0.30x** | **12,911** |
| Weaviate | 1,009 MB | 0.15x | 25,646 |
| Chroma | 1,025 MB | 0.15x | 26,068 |
Memvid’s higher bytes-per-document value reflects its richer internal structure: embedded indices, a write-ahead log, a time index, and the metadata required for hybrid semantic + keyword + time-travel search. Instead of scattering these components across multiple files or services, Memvid packages the entire memory system into a single `.mv2` file, delivering portability and simplicity that traditional systems can’t match.
***
## Methodology
### Dataset
* **Wikipedia**: 39,324 documents
* **Queries**: 2,500 natural language queries with known correct answers
### Systems Tested
* **Memvid** v2 (hybrid search mode)
* **Chroma** 0.4.x (default HNSW)
* **LanceDB** (default IVF-PQ)
* **Qdrant** (default HNSW)
* **Weaviate** (default HNSW)
### Metrics
* **Accuracy\@k**: Fraction of queries where the correct document appears in top-k results
* **MRR**: Mean Reciprocal Rank (1/position of first correct result)
* **NDCG\@10**: Normalized Discounted Cumulative Gain at 10
* **Latency**: Query time in milliseconds (p50, p95, p99)
* **QPS**: Queries per second throughput
### Environment
* Apple M-series Mac
* All systems using default configurations
* Same embedding model across all systems
* Each query executed 1x (no caching)
***
## Key Takeaways
Memvid achieves **92.7% top-1 accuracy**, 8.5 to 14.5 points higher than any competitor.
Memvid **reduces errors by 50-66%** compared to leading vector databases.
Highest MRR (0.95) and NDCG\@10 (0.97). The correct result consistently appears first.
**0.5ms cold start**, 130-300x faster than alternatives for serverless deployments.
***
## Run the Benchmarks Yourself
The benchmark suite is open source. Run it on your own hardware:
```bash theme={null}
git clone https://github.com/memvid/memvid
cd memvid/benchmarks/python
pip install -r requirements.txt
python run_benchmarks.py
```
Results are saved to `results/results.json`.
***
## Next Steps
Try Memvid in 5 minutes
Learn why Memvid is different