Structured logging

Modern software systems generate vast amounts of log data to help teams monitor, troubleshoot, and understand their applications. How we capture and use this data has evolved significantly, from simple text files to structured formats that enable sophisticated analysis and automation.

Logging has been a fundamental component of software systems for decades, but traditionally, it was implemented as text lines with implicit structure embedded in the spacing and formatting. This approach evolved as systems required event recording capabilities but lacked standardization and analytical efficiency.

An example of this type of log message is shown below:

[2025-01-22 12:05:23] [ERROR] [AppServer] Failed to connect to database at db.example.com:5432 - Connection timed out after 30000ms       

While human-readable, these traditional log formats present several challenges. Although they typically contain some semi-structured elements that parsers can be written for, they still have limitations:

• Parsing complexity: Even with semi-structured data, extracting specific information requires custom parsers tailored to each log format
• Inconsistent formats: Different systems implement their own log formats, making cross-system analysis difficult
• Limited searchability: Finding specific events often requires complex regex patterns or string-matching
• Analytical limitations: Correlating events across systems with different formats requires additional transformation steps
• Structural inflexibility: When systems add new fields or change formats, existing parsers often break and require updates

While better than having no logging, this approach transfers the burden of interpretation to the consumer. This leads to the question: why not structure the data at the point of logging?

Structured logging has gained significant adoption, particularly in modern application architectures. This methodology formats logs in a machine-readable structure, predominantly JSON. A JSON log entry incorporates clearly defined fields for timestamp, severity level, message, and relevant metadata. This structured format facilitates programmatic parsing, searching, and analysis.

An example of a structured log message is shown below:

{
  "timestamp": "2025-01-22T12:15:42Z",
  "device": "firewall",
  "action": "BLOCK",
  "direction": "INPUT",
  "interface": "eth0",
  "mac_address": "00:1a:2b:3c:4d:5e",
  "source_ip": "203.0.113.4",
  "destination_ip": "192.168.1.10",
  "protocol": "TCP",
  "source_port": 40125,
  "destination_port": 22,
  "flags": ["SYN"],
  "rule_id": "FW-1234",
  "geo_location": {
    "country": "Unknown",
    "latitude": 0,
    "longitude": 0
  },
  "threat_intel": {
    "known_malicious": false,
    "reputation_score": 35
  }
}

The structured firewall log provides significant advantages over traditional text-based logs. By formatting the security event in JSON, security teams can quickly filter for specific threats (like SSH brute force attempts), correlate events across systems, and build automated responses based on precise field values.

The nested objects for geographical data and threat intelligence provide immediate context without additional lookups, while the clearly labeled fields enable efficient searching and visualization.

This structured approach transforms an opaque text string into actionable security intelligence that can be effectively analyzed, reported on, and integrated with security automation tools.

While traditional logging captures information as text, structured logging provides immediate advantages in both operational efficiency and analytical capabilities. The benefits fall into two main categories: improvements in log processing and enhanced analytical capabilities.

1. Universal Parsing: Log consumers can use standard JSON rather than custom regex-based parsers, significantly reducing implementation complexity.
2. Direct field Access: Each field is directly addressable (e.g., geo_location.latitude or threat_intel.reputation_score), eliminating the need for string manipulation to extract values.
3. Type preservation: Data types are preserved (numbers remain numbers, booleans remain booleans), avoiding type conversion errors common in text log parsing.
4. Schema validation: Logs can be validated against a schema, ensuring consistency and completeness of logged data.

1. Immediate analysis: While traditional logs could eventually be analyzed after extraction and transformation, structured logs are immediately ready for analysis without preprocessing.
2. Efficient querying: Structured data enables precise queries like "show all blocked SSH connections from IPs with reputation scores below 40" without complex text processing.
3. Aggregation capabilities: Statistical operations (counts, averages, percentiles) can be performed directly on numeric fields without extraction steps.
4. Cross-system correlation: Events from different systems using the same structured format can be easily joined and correlated based on common fields.

By structuring logs at the source rather than attempting to extract structure later, organizations gain both immediate operational benefits and long-term analytical capabilities that would otherwise require significant data engineering efforts.

By capturing log data in a structured format from the start, organizations can build powerful solutions that would be difficult or impossible with traditional logging approaches. Here are several key use cases demonstrated by organizations that have implemented structured logging at scale:

Application Performance Monitoring (APM) involves tracking and analyzing software applications' performance and behavior in real-time.

By using structured logging for APM, organizations can capture this telemetry data in a format that's immediately queryable. For example, Tekion's implementation processes 1.2 million records per minute, enabling them to monitor their automotive dealer management software with sub-second query response times. This allows their teams to identify performance bottlenecks quickly, troubleshoot issues, and ensure their applications meet performance objectives.

Structured logging enables sophisticated real-time monitoring and alerting capabilities. When log data is structured, teams can create precise alerts based on specific field values and conditions. trip.com built a self-service platform where teams can monitor their data flows in real-time and create custom alerts, demonstrating how structured logging can support proactive system monitoring at scale.

Structured logging isn't just for technical monitoring - it can provide valuable business insights. When log data is properly structured, teams can analyze business metrics and track key performance indicators efficiently. trip.com leverages its logging platform to monitor critical metrics such as payment completion rates and order statistics, transforming operational data into business intelligence.

Structured logging can support sophisticated monitoring of web crawling operations. Corsearch's implementation tracks search engine operations across major platforms like Google, Yahoo, and YouTube. Their system maintains detailed telemetry of crawl attempts and queue depths, storing over 10 billion rows of historical data in a compressed format spanning four years.

Structured logging enables sophisticated automated response systems by making log data immediately actionable. When logs are structured, systems can analyze real-time patterns and automatically trigger appropriate responses based on specific field values or event combinations.

Prefect demonstrates this capability in its workflow orchestration platform, where it analyzes millions of events daily to automate responses to system issues. The system can automatically trigger database reboots when specific patterns are detected, create incident tickets when failure rates cross thresholds, and send alerts to Slack when workflows run late.

The structured nature of their logs allows them to identify complex patterns and respond to them automatically, reducing the need for manual intervention and improving system reliability.