Skip to main content

Log Analysis Workflow

VisiData turns raw server logs into queryable, filterable, sortable sheets — all without writing SQL or awk pipelines. This lesson covers Nginx access logs, syslog, and JSON application logs.

Learning Focus

Follow each workflow end-to-end. The goal is not memorizing every step — it is building muscle memory for: open → type columns → filter → frequency table → export.

Nginx Access Log Analysis​

Open and Structure the Log​

vd /var/log/nginx/access.log

Nginx combined log format has these fields (in order):

ip  -  -  [date]  "method path protocol"  status  bytes  referer  user_agent

VisiData may open this as a single text column. Parse the fields with regex capture groups:

# Move to the single text column
;
# Enter this regex (captures key fields):
(?P<ip>\S+) \S+ \S+ \[(?P<date>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)
# Press Enter → named columns appear: ip, date, method, path, status, bytes

Type the Columns​

# Move to 'status', press #  → integer
# Move to 'bytes', press #   → integer
# Move to 'date', press @    → date

Analyze Status Codes​

# Move to 'status' column
Shift+F
# Frequency table: 200: 85%, 404: 10%, 500: 2%
# Press ] to sort by count descending
# Press Enter on 500 → see all 500 error rows

Find the Busiest IPs​

# Move to 'ip' column
Shift+F
# Frequency table: IPs ranked by request count
# Press ] to sort descending → top attacker IPs at top

Export 500 Errors for Incident Report​

# Select all 500 status rows
# Move to 'status' column
|
# Enter: ^500$

# Open filtered sheet
"

# Save as CSV
Ctrl+S
# Enter: /tmp/nginx_500_errors.csv

Syslog Analysis​

vd /var/log/syslog

# Search for errors
g/
# Enter: error|Error|ERROR

# Select matching rows
g|
# Enter: error|Error|ERROR

# Filter to error-only sheet
"

# Frequency table by hostname (if multiple servers forward to this log)
Shift+F

JSON Application Log Analysis​

# JSON Lines format (one JSON object per line)
vd /var/log/app/events.jsonl

# VisiData auto-parses JSONL into columns
# Expand nested fields
# Move to a column containing nested dicts: press (

Access Log Dashboard Workflow​

A complete end-to-end workflow:

Practical Command Sequence​

# Full nginx log analysis in one session:

vd /var/log/nginx/access.log

# 1. Parse fields
;
# Enter regex: (?P<ip>\S+) \S+ \S+ \[(?P<date>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)

# 2. Type columns
# Move to status → #
# Move to bytes → #

# 3. Status distribution
# Move to status → Shift+F

# 4. Drill into 500s
# Press Enter on '500'

# 5. Find common error paths
# Move to 'path' → Shift+F

# 6. Export
Ctrl+S
# Enter: /tmp/error_paths.csv

# 7. Return to overview
q    (from frequency table)
q    (from error sheet)
q    (from status frequency table)

Troubleshooting Matrix​

ProblemCauseFix
Log opens as single columnUnknown formatUse ; with a regex to parse fields
Regex captures nothingRegex doesn't match log formatTest with / search first
Date sort wrongColumn typed as stringCast to @ (date)
Large log takes long to openMillions of linesUse --max-rows 100000 to sample
Permission deniedRoot-owned logsudo vd /var/log/nginx/access.log or copy first

Best Practices​

  • Always copy logs to /tmp/ before editing to avoid modifying production log files.
  • Save your parsing regex as a macro or CommandLog for reuse across daily log reviews.
  • Use --max-rows 100000 for initial exploration of very large logs, then remove the limit once you know what you are looking for.

Hands-On Practice​

# Generate a sample nginx-style log
cat > /tmp/access.log << 'EOF'
192.168.1.10 - - [12/May/2025:08:01:00 +0000] "GET /index.html HTTP/1.1" 200 1234 "-" "Mozilla/5.0"
192.168.1.11 - - [12/May/2025:08:01:05 +0000] "POST /api/login HTTP/1.1" 200 432 "-" "curl/7.64"
192.168.1.10 - - [12/May/2025:08:01:10 +0000] "GET /missing.html HTTP/1.1" 404 200 "-" "Mozilla/5.0"
192.168.1.12 - - [12/May/2025:08:01:15 +0000] "GET /admin HTTP/1.1" 500 89 "-" "Go-http-client/1.1"
EOF

vd /tmp/access.log

# Parse with regex, type columns, open frequency table on status

What's Next​