json-log-viewer

Introduction

json-log-viewer is a versatile tool designed to simplify the analysis of JSON logs for developers and system administrators. Whether you work in the command line or prefer a browser-based interface, json-log-viewer provides powerful features for filtering, combining, and visualizing log streams.

The primary goal of json-log-viewer is to transform raw JSON logs into a human-readable format, making them significantly easier to interpret and process. With support for advanced filtering, dynamic configuration updates, multi-source input, and streamlined configuration management, this tool offers a comprehensive solution for log analysis across various environments.

You can also try the browser-based version online: DEMO.

Example Output

Below is an example of formatted JSON logs in the command-line interface:

Key Features

Human-Readable Log Output
Effortlessly transform raw JSON logs into a structured, easy-to-read format.
Flexible Interfaces
Use the command-line utility or a browser-based application for visualization.
Multi-Source Input
Combine the outputs of multiple commands into a single unified log stream.
Multiple Input Formats
- JSON (default): Process standard JSON log formats
- Logfmt: Support for key-value pair log formats
- CSV: Parse and analyze CSV-formatted logs with headers
Advanced Filtering
- Apply regular expressions to extract relevant log entries.
- Use SQL-like queries to filter and query JSON fields.
- Filter logs by timestamp ranges.
YAML Configuration
Define input streams and configurations using YAML files.
Dynamic Configuration Updates
Hot-reload filters and configurations without restarting the tool.
stdin Support
Process logs directly from the standard input for real-time analysis.
Integration with k9s
Seamlessly integrate with the k9s Kubernetes CLI tool to visualize logs directly within k9s.
Custom Field Mapping
Configure custom field names mapping to work with non-standard log formats.

Installation

Requirements

Coursier (for installation and dependency management)

Install with Coursier

Install json-log-viewer globally using coursier:

coursier install json-log-viewer --channel https://git.io/JvV0g

After installation, ensure the tool is available:

json-log-viewer --help

Usage

Quick Start

Install json-log-viewer using Coursier.
Pass logs to the tool via stdin:
```
cat log.txt | json-log-viewer
```

Apply filters directly from the command line:

cat log.txt | json-log-viewer --filter "level = 'ERROR'"

Working with Different Log Formats

JSON (Default)

For standard JSON logs:

cat json-logs.txt | json-log-viewer

Logfmt

For logs in logfmt format:

cat logfmt-logs.txt | json-log-viewer --format-in logfmt

CSV

For CSV-formatted logs (requires header row):

cat csv-logs.csv | json-log-viewer --format-in csv

Note: CSV format requires a header row with column names. The tool will map these column names to standard log fields.

SQL Filtering

json-log-viewer supports SQL-like filtering for JSON fields, allowing precise log analysis. You can use comparison and logical operations.

Supported Syntax

Comparison operators:
- =: Equal
- !=: Not equal
- LIKE: Matches with patterns (% for wildcard)
- NOT LIKE: Negates pattern matching
Logical operators:
- AND: Logical AND
- OR: Logical OR
- () for grouping expressions
Literals and Identifiers:
- Identifiers represent JSON keys (e.g., level).
- Literals are strings enclosed in single quotes (e.g., 'ERROR').

Examples

Filter logs where level is 'ERROR':

cat log.txt | json-log-viewer --filter "level = 'ERROR'"

Match logs where message contains the word "timeout":

cat log.txt | json-log-viewer --filter "message LIKE '%timeout%'"

Combine conditions:

cat log.txt | json-log-viewer --filter "level = 'ERROR' AND message LIKE '%timeout%'"

Exclude logs with specific patterns:

cat log.txt | json-log-viewer --filter "message NOT LIKE '%timeout%'"

Group conditions with parentheses:

cat log.txt | json-log-viewer --filter "(level = 'ERROR' OR level = 'WARN') AND message LIKE '%connection%'"

Timestamp Filtering

Filter logs by timestamp range:

cat log.txt | json-log-viewer --timestamp-after 2024-01-01T00:00:00Z --timestamp-before 2024-01-31T23:59:59Z

You can also specify a custom timestamp field:

cat log.txt | json-log-viewer --timestamp-field time

Configuration

json-log-viewer supports defining input streams, filters, and other settings using a YAML configuration file.

YAML Configuration Structure

A configuration file consists of one or more feeds. Each feed represents a log source and can have the following attributes:

name (optional): A descriptive name for the feed.
commands (required): A list of shell commands that produce log output.
inlineInput (optional): Direct input as a string instead of executing commands.
filter (optional): SQL-like filter expression for processing logs (e.g., level = 'ERROR').
formatIn (optional): Input log format. Supported values:
- json (default).
- logfmt.
- csv.
rawInclude and rawExclude (optional): Lists of regular expressions to include or exclude from processing.
excludeFields (optional): List of fields to exclude from output.
fieldNames (optional): Custom mapping for field names, helpful when working with non-standard log formats.

Custom Field Mapping

You can define custom field name mappings either globally or per feed:

# Global field mapping
fieldNames:
  timestamp: "ts"
  level: "severity"
  message: "msg"
  stackTrace: "error"
  loggerName: "logger"
  threadName: "thread"

feeds:
  - name: "application-logs"
    commands:
      - cat log1.txt
    # Feed-specific field mapping (overrides global mapping)
    fieldNames:
      timestamp: "time"
      level: "priority"

Example Configuration File

showEmptyFields: true
feeds:
  - name: "application-1-logs"
    commands:
      - cat log1.txt
    filter: |
      level = 'ERROR' AND message LIKE '%timeout%'
    formatIn: json
    rawInclude:
      - "ERROR"
    rawExclude:
      - "DEBUG"
    excludeFields:
      - "thread_name"
    showEmptyFields: false
  - name: "application-2-logs"
    commands:
      - cat log2.txt
    filter: |
      message NOT LIKE '%heartbeat%'
    formatIn: logfmt
  - name: "csv-logs"
    commands:
      - cat logs.csv
    formatIn: csv
    fieldNames:
      timestamp: "time"
      level: "severity"

Running with a Configuration File

To use a YAML configuration file, pass its path with the --config-file option:

json-log-viewer --config-file json-log-viewer.yml

Command-Line Arguments

json-log-viewer also supports direct configuration via command-line arguments:

Supported Options

--filter: Apply SQL-like filters directly to logs.

cat log.txt | json-log-viewer --filter "level = 'ERROR'"

--config-file: Specify the path to a YAML configuration file.

cat log.txt | json-log-viewer --config-file json-log-viewer.yml

--format-in: Specify the input log format (supported formats: json, logfmt, csv).
```
cat log.txt | json-log-viewer --format-in logfmt
```
--format-out: Specify the output format (supported formats: pretty, raw).
```
cat log.txt | json-log-viewer --format-out raw
```

--timestamp-after and --timestamp-before: Filter logs by a specific time range.

cat log.txt | json-log-viewer --timestamp-after 2024-01-01T00:00:00Z --timestamp-before 2024-01-31T23:59:59Z

--timestamp-field: Specify the field name for timestamps (default: @timestamp).
```
json-log-viewer --timestamp-field time
```
--show-empty-fields: Display fields with empty values (null, empty strings, etc.) in output.
```
cat log.txt | json-log-viewer --show-empty-fields
```

Field Name Options

You can override the default field names to work with non-standard log formats:

--level-field: Override default level field name (default: level).
```
json-log-viewer --level-field severity
```
--message-field: Override default message field name (default: message).
```
json-log-viewer --message-field msg
```
--stack-trace-field: Override default stack trace field name (default: stack_trace).
```
json-log-viewer --stack-trace-field exception
```
--logger-name-field: Override default logger name field name (default: logger_name).
```
json-log-viewer --logger-name-field logger
```
--thread-name-field: Override default thread name field name (default: thread_name).
```
json-log-viewer --thread-name-field thread
```

k9s Plugin

Integrate json-log-viewer with k9s to view formatted JSON logs directly within the k9s interface.

Prerequisites

For the plugin to work correctly, the following must be installed on your system:

coursier - Used to download and run json-log-viewer if it's not installed.
json-log-viewer (optional) - If already installed, the plugin will use it directly; otherwise, it will fall back to using coursier to launch it.

Installation

Add the following to your k9s plugin file (usually located at ~/.k9s/plugins.yaml or, on macOS, check the plugin path with k9s info):

plugins:
  json-log-viewer:
    shortCut: Ctrl-L
    description: "json-log-viewer"
    scopes:
      - pod
      - containers
    command: sh
    background: false
    args:
      - -c
      - |
        if command -v json-log-viewer >/dev/null 2>&1; then
          VIEWER_COMMAND="json-log-viewer"
        else
          VIEWER_COMMAND="coursier launch ru.d10xa:json-log-viewer_3:latest.release"
        fi

        if [ -n "$POD" ]; then
          kubectl logs $POD -n $NAMESPACE --context $CONTEXT -c $NAME -f --tail 500 | $VIEWER_COMMAND; read -p "Press [Enter] to close..."
        else
          kubectl logs $NAME -n $NAMESPACE --context $CONTEXT -f --tail 500 | $VIEWER_COMMAND; read -p "Press [Enter] to close..."
        fi

k9s plugin usage

Install json-log-viewer
Launch k9s
Select a Pod or Container
Press Ctrl+L to view logs formatted by json-log-viewer

Development

This section provides instructions for building and running both the JVM and JavaScript versions of json-log-viewer. It also includes notes for working on the frontend-laminar module.

Prerequisites

Ensure you have the following installed on your system:

Java 11+ (for building the JVM version)
Node.js 23+ (for building the JS version)
sbt (Scala Build Tool)

Building the JVM Version

To build the JVM version of the project, use the command:

sbt stage

This compiles the code and prepares the executable under the jvm/target/universal/stage/bin/ directory.

Running the JVM Version

Run the application with:

cat log.txt | ./json-log-viewer/jvm/target/universal/stage/bin/json-log-viewer

Replace log.txt with the path to your JSON log file. The output will be displayed in a human-readable format.

Building the JavaScript Version

To build the JavaScript version, you can use one of the following options:

Optimized Build: Use the command:
```
sbt fullLinkJS  
```
This generates a production-ready JavaScript file located at:
frontend-laminar/target/scala-3.6.2/frontend-laminar-opt/main.js
Fast Development Build: Use the command:
```
sbt fastLinkJS 
```
This generates a faster, less optimized build located at:
frontend-laminar/target/scala-3.6.2/frontend-laminar-fastopt/main.js

Choose the appropriate option based on your needs:

Use fullLinkJS for production or when you need a fully optimized bundle.
Use fastLinkJS for local development with faster build times.

License

This project is licensed under the MIT License. See the LICENSE file for full details.

Name		Name	Last commit message	Last commit date
Latest commit History 71 Commits
.github/workflows		.github/workflows
frontend-laminar/src		frontend-laminar/src
json-log-viewer		json-log-viewer
make-logs/src/main		make-logs/src/main
project		project
.gitignore		.gitignore
.scalafmt.conf		.scalafmt.conf
LICENSE		LICENSE
README.md		README.md
build.sbt		build.sbt
log-logfmt.txt		log-logfmt.txt
log.txt		log.txt
screenshot.png		screenshot.png
sonatype.sbt		sonatype.sbt

License

d10xa/json-log-viewer

Folders and files

Latest commit

History

Repository files navigation

json-log-viewer

Introduction

Example Output

Key Features

Installation

Requirements

Install with Coursier

Usage

Quick Start

Working with Different Log Formats

JSON (Default)

Logfmt

CSV

SQL Filtering

Supported Syntax

Examples

Timestamp Filtering

Configuration

YAML Configuration Structure

Custom Field Mapping

Example Configuration File

Running with a Configuration File

Command-Line Arguments

Supported Options

Field Name Options

k9s Plugin

Prerequisites

Installation

k9s plugin usage

Development

Prerequisites

Building the JVM Version

Running the JVM Version

Building the JavaScript Version

License

About

Topics

Resources

License

Stars

Watchers

Forks

Languages