Log Analysis Workflow
VisiData turns raw server logs into queryable, filterable, sortable sheets — all without writing SQL or awk pipelines. This lesson covers Nginx access logs, syslog, and JSON application logs through a complete end-to-end workflow.
Learning Focus
Follow each workflow end-to-end. The goal is not memorizing every step — it is building muscle memory for: open → parse fields → type columns → filter → frequency table → export.
Sample Log Data
mkdir -p ~/github/practice-folder/visidata/05-logs
cat > ~/github/practice-folder/visidata/05-logs/01-access.log << 'EOF'
10.0.1.5 - - [15/Jan/2025:00:01:23 +0000] "GET /api/users 200 1234 "curl/7.68"
10.0.1.8 - - [15/Jan/2025:00:01:45 +0000] "POST /api/orders 201 856 "-"
10.0.2.3 - - [15/Jan/2025:00:02:10 +0000] "GET /static/logo.png 200 45678 "-"
10.0.1.5 - - [15/Jan/2025:00:02:33 +0000] "DELETE /api/users/99 404 312 "-"
10.0.3.1 - - [15/Jan/2025:00:03:05 +0000] "GET /api/data 500 98 "-"
10.0.1.8 - - [15/Jan/2025:00:03:22 +0000] "GET /api/users 200 1198 "-"
10.0.3.1 - - [15/Jan/2025:00:04:01 +0000] "GET /api/data 500 102 "-"
EOF
Open Log and Parse Fields
Nginx combined log format is: ip - - [date] "method path protocol" status bytes referer user_agent
vd ~/github/practice-folder/visidata/05-logs/01-access.log
VisiData may open this as a single text column. Parse the fields with regex capture groups using ;:
# Move to the single text column
;
# Enter this regex:
(?P<ip>\S+) \S+ \S+ \[(?P<date>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)
After parsing — named columns appear:
ip date method path status bytes
10.0.1.5 15/Jan/2025:00:01:23 GET /api/users 200 1234
10.0.1.8 15/Jan/2025:00:01:45 POST /api/orders 201 856
10.0.2.3 15/Jan/2025:00:02:10 GET /static/... 200 45678
10.0.1.5 15/Jan/2025:00:02:33 DELETE /api/users.. 404 312
10.0.3.1 15/Jan/2025:00:03:05 GET /api/data 500 98
Cast Column Types
# Move to 'status' column → press # → integer
# Move to 'bytes' column → press # → integer
# Move to 'date' column → press @ → date (auto-detects format)
This enables numeric sorting, aggregation, and date-aware operations.
Analyze Status Codes (Frequency Table)
# Move to 'status' column
Shift+F
Result:
status count percent histogram
200 3 42.9% ████████████
201 1 14.3% ████
404 1 14.3% ████
500 2 28.6% ████████
# Sort by count to see most common codes first
]
# Drill into all 500 errors
# Move to '500' row → press Enter
500 error sheet opens:
ip date method path status bytes
10.0.3.1 15/Jan/2025:00:03:05 GET /api/data 500 98
10.0.3.1 15/Jan/2025:00:04:01 GET /api/data 500 102
IP 10.0.3.1 is hitting /api/data repeatedly and getting 500s.
Find the Busiest IPs
# Press q to return to source sheet
# Move to 'ip' column
Shift+F
Result:
ip count percent
10.0.1.5 2 28.6%
10.0.1.8 2 28.6%
10.0.3.1 2 28.6%
10.0.2.3 1 14.3%
Export 500 Errors to CSV
# From source sheet:
# Select all 500 status rows
|
# Enter: 500
# Open filtered sheet
"
# Save as CSV
Ctrl+S
# Enter: /tmp/nginx_500_errors.csv
Syslog Analysis
vd /var/log/syslog
# Search for all error/warning lines across all columns
g/
# Enter: error|Error|ERROR|warn|WARN
# Select all matching rows
g|
# Enter: error|Error|ERROR
# Filter to error-only sheet
"
# Frequency by hostname
Shift+F
JSON Application Log
vd ~/github/practice-folder/visidata/02-json/01-events.jsonl
# Auto-parsed into columns
# Expand any nested fields: move to nested column → press (
# Filter by error status
|
# Enter: error
# Export error events
"
Ctrl+S
# Enter: /tmp/app_errors.csv
Complete Nginx Log Workflow
vd ~/github/practice-folder/visidata/05-logs/01-access.log
1. ; (regex) → parse into columns: ip, status, bytes, path
2. # on status/bytes → cast to integer
3. @ on date → cast to date type
4. Shift+F on status → HTTP status distribution
5. Enter on '500' → filtered sheet: 500 error rows only
6. Shift+F on ip → which IPs generate the most 500s?
7. Ctrl+S → export error report
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| Log opens as single column | Unknown format | Use ; with a regex to parse fields |
| Regex captures nothing | Regex doesn't match log format | Test with / search first |
| Date sort wrong | Column typed as string | Cast to @ (date) |
| Large log takes long to open | Millions of lines | Use --max-rows 100000 to sample |
| Permission denied | Root-owned log | sudo cat /var/log/syslog | vd |
Hands-On Practice
vd ~/github/practice-folder/visidata/05-logs/01-access.log
# 1. Move to the single text column → press ;
# 2. Enter the parsing regex → see named columns appear
# 3. Cast status to int: # | Cast bytes to int: #
# 4. Move to 'status' → Shift+F → view frequency
# 5. Press Enter on '500' → see 500 error rows
# 6. Press q → back to frequency table
# 7. Press q → back to source
# 8. Move to 'ip' → Shift+F → see busiest IPs