Piping and Batch Mode

VisiData is not just an interactive TUI — it is also a data conversion engine that can run headlessly in batch mode (-b) for scripted pipelines, and accepts piped input from any shell command.

Learning Focus

Understand when to use VisiData as a CLI conversion tool versus interactive exploration, and how to replay sessions for reproducible batch processing.

Stdin Pipe Input

# Any command output can be piped into VisiData
command | vd -f format

# Examples:
ps aux | vd -f fixed --skip 1
df -h | vd -f fixed --skip 1
netstat -tulnp | vd -f fixed
cat /var/log/nginx/access.log | vd -f txt
curl -s https://example.com/data.csv | vd -f csv

# JSON API output
curl -s https://api.github.com/repos/saulpw/visidata/tags | vd -f json

Stdout Pipe Output

# VisiData can output a sheet to stdout
vd -b input.csv -o -       # output to stdout as TSV
vd -b input.csv -o - | grep "error"

Batch Mode: Data Conversion

Batch mode (-b) runs VisiData without the TUI interface and processes/converts data:

# Convert CSV to JSON
vd -b input.csv -o output.json

# Convert JSONL to CSV
vd -b events.jsonl -o events.csv

# Convert Excel to TSV
vd -b report.xlsx -o report.tsv

# Convert SQLite table to CSV
vd -b site.sqlite -o tables_export.csv

# Convert fixed-width to JSON
ps aux | vd -f fixed --skip 1 -b -o processes.json

CommandLog Replay

VisiData records every action in a CommandLog (.vdj file). You can save and replay sessions for reproducible batch processing.

# Inside VisiData, save the CommandLog:
# Press Shift+D to view the CommandLog sheet
# Press Ctrl+S to save it as a .vdj file

# Replay a saved session in batch mode:
vd --play session.vdj --batch --output result.csv

# Replay interactively (for debugging):
vd --play session.vdj

# Replay with a wait between commands (for demos):
vd --play session.vdj --replay-wait 1

Practical Pipeline Examples

Nginx Log Analysis Pipeline

# Extract fields from nginx access log and export
vd -b /var/log/nginx/access.log -o /tmp/access_analysis.csv

# Or interactively explore then export
vd /var/log/nginx/access.log
# Inside VisiData:
# 1. Set column types as needed
# 2. Filter with s / | (select matching rows)
# 3. Press " to open a filtered sheet
# 4. Press Ctrl+S to save filtered result

Convert WordPress Export

# Export WordPress database table to CSV
vd -b /var/www/html/wp-content/db.sqlite3 -o /tmp/wp_posts.csv

Daily Report Automation

#!/bin/bash
# Run VisiData in batch to convert daily log snapshot
DATE=$(date +%Y%m%d)
vd -b /var/log/nginx/access.log -o /var/www/html/reports/access_${DATE}.csv
echo "Report generated: access_${DATE}.csv"

Multi-File Batch Conversion

# Convert all CSVs in a directory to JSON
for f in /var/www/html/exports/*.csv; do
  base=$(basename "$f" .csv)
  vd -b "$f" -o "/tmp/json/${base}.json"
done

Batch Mode Options Reference

Option	Description
-b / --batch	Run without TUI (batch mode)
-o file / --output file	Save final sheet to file
--play file.vdj	Replay a saved CommandLog
--replay-wait N	Wait N seconds between replayed commands
-i / --interactive	Launch interactively after batch replay
field=value	Replace {field} placeholders in CommandLog

Troubleshooting Matrix

Problem	Cause	Fix
Batch outputs empty file	Pipe closes before VisiData reads	Use explicit file instead of stdin
CommandLog replay fails	Sheet structure changed	Re-record the session
Format not detected in batch	No extension or stdin	Always use -f format in scripts
Encoding error in batch	Non-UTF-8 input	Add --encoding latin1

Best Practices

• Always specify -f in batch scripts — never rely on auto-detection in automation.
• Use CommandLog replay (--play) to document data transformation steps and share with colleagues.
• Pipe vd -b input -o - into other tools (grep, jq, awk) for lightweight ETL pipelines.
• Set --max-rows when sampling large files in batch to avoid long runtimes.

Hands-On Practice

# 1. Convert a CSV to JSON in batch mode
echo "name,age\nAlice,29\nBob,34" > /tmp/people.csv
vd -b /tmp/people.csv -o /tmp/people.json
cat /tmp/people.json

# 2. Pipe ps output and convert to CSV
ps aux | vd -f fixed --skip 1 -b -o /tmp/processes.csv

# 3. Pipe JSON from curl to VisiData interactively
curl -s https://api.github.com/repos/saulpw/visidata/releases | vd -f json

What's Next

• Moving Around Sheets
• Searching and Jumping