Large File Performance
VisiData uses lazy loading — it only reads what's visible on screen. This is why a 1M-row CSV opens in under a second. But some operations (sorting, frequency tables) require processing all loaded rows. This lesson explains how to stay fast with large files.
Always use --max-rows for initial exploration of large files. Only load the full dataset once you know what columns and filters you need.
Lazy Loading: How VisiData Stays Fast
VisiData reads rows on demand — only the rows currently visible in the terminal window are in memory. Scrolling loads the next batch.
File on disk: 5,000,000 rows
VisiData loads: ~50 rows (one screen height)
Scrolling down: loads next ~50 rows on demand
This means:
- Opening a 1M-row CSV: instant
- Sorting a 1M-row CSV: must read all rows → takes 5–30 seconds
Limit Rows for Exploration (--max-rows)
# Open only the first 100,000 rows for fast exploration
vd --max-rows 100000 ~/large-data/events.csv
# Combined with format override
vd --max-rows 50000 -f csv /var/log/nginx/access.log
Use --max-rows when:
- You want a fast representative sample
- You're testing filters or expressions
- The file is too large for your RAM
Random Sample for Statistical Accuracy
For statistical work, a random sample is better than the first N rows:
# Random 10% sample with shuf
shuf -n 100000 ~/large-data/events.csv | vd -f csv
# Or use head + tail to take a middle slice
tail -n +1000 ~/large-data/events.csv | head -n 50000 | vd -f csv
Reload and Memory (Ctrl+R, options.min_memory_mb)
# Reload the current sheet from disk (clears all in-memory state)
Ctrl+R
Prevent VisiData from using too much RAM:
# ~/.visidatarc
options.min_memory_mb = 100 # pause loading if free RAM drops below 100MB
When triggered, VisiData pauses and asks whether to continue loading.
Profiling Slow Operations (^T)
# While a slow operation is running (e.g., sorting a huge file):
^T
# Opens Threads Sheet — shows running threads and row counts
Threads Sheet columns:
name status rows elapsed
sort-rows running 845230 4.2s
Cancel a thread:
# Move to the thread row
^C # cancel selected thread
Fastest Loading Formats
| Format | Load Speed | Notes |
|---|---|---|
| CSV (plain) | Fast | Streaming, no schema detection |
| JSONL | Fast | One line = one row |
| Parquet | Very fast | Columnar, binary, compressed |
| JSON array | Slow for large files | Full parse required |
| Excel (XLSX) | Slow | XML parsing overhead |
| SQLite | Fast | Lazy table access |
For performance-critical workflows, convert to Parquet or SQLite first:
# Convert large CSV to SQLite via VisiData batch mode
vd -b large.csv -o large.sqlite
# Then open fast:
vd large.sqlite
Performance Options Reference
| Option | Default | Description |
|---|---|---|
--max-rows N | unlimited | Stop loading after N rows |
options.min_memory_mb | 0 (off) | Pause loading if RAM drops below N MB |
--batch | off | Run without TUI (faster for conversion) |
options.encoding | utf-8-sig | Change if file has encoding issues |
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| Sort takes very long | Sorting requires reading all rows | Use --max-rows to limit; sort a sample first |
| VisiData freezes on open | File too large for RAM | Add options.min_memory_mb = 100 to .visidatarc |
Shift+F is slow | Counting all distinct values in a huge column | Use a sample with --max-rows |
| Parquet import error | pyarrow not installed | pip install pyarrow |
| Memory error on frequency table | Too many distinct values × too many rows | Use --max-rows or pre-filter with grep |