Performance and Profiling

VisiData loads entire files into memory by default. For datasets larger than available RAM, or when you need fast sampling, several options control load behavior and resource usage.

Learning Focus

Focus on --max-rows for sampling, lazy-loading strategies, and format-specific performance differences. Use Shift+I to profile data before running expensive operations.

Sample a Large File with --max-rows

Limit rows loaded during initial exploration:

# Load only first 10,000 rows
vd --max-rows 10000 ~/github/practice-folder/visidata/01-loading/01-employees.csv

Status bar shows: 10000/1000000+ rows 5 cols.

# Set permanently in .visidatarc
options.max_rows = 100000

warning

--max-rows loads the first N rows. For a random sample, pre-shuffle:

shuf -n 10000 bigfile.csv | vd -f csv

Pipe Command Output (No File Load)

Piping avoids loading anything into VisiData's memory at all:

# Stream large log — only visible rows are held in memory
tail -n 100000 ~/github/practice-folder/visidata/05-logs/01-access.log | vd -f fixed

# Pipe specific columns from awk to reduce data size
awk '{print $1,$9,$10}' /var/log/nginx/access.log | vd -f fixed

Use SQLite for Lazy Loading

SQLite in VisiData uses lazy loading — only the table you open is read into memory:

vd ~/github/practice-folder/visidata/08-sysadmin/04-inventory.db
# Opens table directory — NO data loaded yet

# Press Enter on 'orders' table
# Only 'orders' rows are loaded now

Profile Before Full Analysis

Use Shift+I (Describe Sheet) instead of building frequency tables on every column:

vd ~/github/practice-folder/visidata/01-loading/01-employees.csv

Shift+I
# Shows: nulls, distinct, min, max, mean for ALL columns
# Runs in background — check status bar for progress

This tells you which columns have nulls, which are constants (distinct=1), and which have outliers — without running any expensive operations.

Format Performance Comparison

Format	Load Speed	Memory	Best For
Parquet	Fastest	Columnar, compressed	Analytics, data pipelines
TSV	Fast	Low overhead	Log data, exports
CSV	Fast	Moderate	Standard data exchange
JSONL	Moderate	One pass streaming	Event logs, API streams
JSON	Moderate	Full parse upfront	API responses, configs
Excel (.xlsx)	Slow	High	Spreadsheet imports
SQLite	Fast (per query)	On-demand	Database exploration

# Prefer TSV over CSV for large files — no quoting overhead
vd data.tsv

# Parquet is fastest for repeated analytics
pip install pyarrow
vd data.parquet

Memory Monitoring

# ~/.visidatarc
options.min_memory_mb = 500   # warn when free RAM < 500 MB

Status bar shows current row count: curr_row/total_rows rows curr_col/total_cols cols.

Column Type Performance

Casting columns (#, %, @) triggers re-rendering of the entire column. For very wide sheets:

• Cast only columns you need to sort or aggregate
• Use Shift+C (Columns Sheet) to bulk-cast multiple columns at once

# Efficient: cast only what you need
# Move to 'salary' column → press % (float)
# Move to 'start_date' column → press @ (date)
# Leave all string columns as-is

Troubleshooting

Problem	Cause	Fix
VisiData hangs on open	File too large	Use --max-rows 100000
Out of memory	Too many rows loaded	Pipe with head -n or use Parquet
Shift+F is slow	Too many distinct values	Use --max-rows to sample first
Type cast is slow	Very wide sheet	Cast column-by-column, not g%

What's Next

• Opening Files and Formats — load CSV, JSON, SQLite, Excel, and more
• Exporting and Batch Mode — saving, batch conversion, pipe output