48.9k
Log management is the discipline of handling log data across its entire lifecycle, from generation and collection through storage, retention, query, and eventual deletion. The volumes make it a first-class engineering problem: ClickHouse's internal logging platform holds over 19 PiB of logs (37 trillion rows) at six months' retention for its AWS regions alone, and NIST maintains a dedicated planning guide for the practice.
TL;DR
- Log management covers a seven-stage lifecycle: generate → collect and ship → parse and structure → store and index → retain and rotate → query → archive or delete.
- NIST SP 800-92 is the canonical reference for planning a log management program.
- Retention is set by the strictest applicable regime: PCI DSS v4.0 requires 12 months of audit log history; HIPAA requires six years for security documentation; SOC 2 auditors expect evidence covering the full audit period.
- The biggest cost levers are structuring logs at the source, filtering at the collector, tiering storage, and columnar compression.
- A SIEM is a downstream, security-focused consumer of managed logs, fed by the same pipeline.
What is log management?
Log management is the practice of collecting, structuring, storing, retaining, and querying log data from every application and infrastructure component under one governed process. It spans the log's full lifecycle, from the line of code that emits it to the policy that deletes it, so logs stay queryable, affordable, and compliant.
It is distinct from log monitoring, which watches the log stream in near real time to detect problems. Monitoring is one consumer of well-managed logs; management is the underlying discipline that keeps the pipeline feeding it healthy. Log aggregation is the centralization stage of the lifecycle, and log analytics is the analytical querying of what the lifecycle stores. NIST formalized the discipline in SP 800-92, the Guide to Computer Security Log Management, which frames the core problem as balancing a limited supply of storage and analyst attention against a continuous, growing supply of log data.
How does the log management lifecycle work?
The log management lifecycle moves every log line through seven stages:
generate → collect/ship → parse/structure → store/index → retain/rotate → query → archive/delete- Generate. Applications, containers, and network devices emit events: syslog messages per RFC 5424, stdout lines from containers, or structured records from logging libraries and OpenTelemetry SDKs.
- Collect and ship. Per-host agents such as the OpenTelemetry Collector (via its filelog receiver) or Fluent Bit, the CNCF-graduated sub-project of Fluentd, tail files and forward events to a central endpoint.
- Parse and structure. Free-form text is parsed into fields: timestamp, severity, service, message. Teams that adopt structured logging at the source skip most of this stage.
- Store and index. Events land in a queryable store, either a search index or a columnar database.
- Retain and rotate. Time-based policies (TTLs,
logrotateon hosts) expire or demote data on schedule. - Query. Engineers search and aggregate the stored logs for debugging, audit, and reporting.
- Archive or delete. Data needed only for compliance moves to cheap object storage; data past its retention obligation is deleted.
Treating these as one governed pipeline, rather than seven ad-hoc scripts, is what separates log management from merely having logs.
How long should you retain logs?
Retain logs as long as the strictest applicable compliance regime requires, and no longer than data-protection rules permit. For most production systems that means 30 to 90 days in fast "hot" storage for debugging, extended to 12 months or more in cheaper tiers where PCI DSS, HIPAA, or audit obligations apply.
| Retention driver | Requirement | Notes |
|---|---|---|
| PCI DSS v4.0 (Req. 10.5.1) | 12 months, most recent 3 months immediately available | Applies to audit logs in cardholder data environments |
| HIPAA (45 CFR §164.316) | 6 years | Covers security documentation, including audit records and activity reviews |
| SOC 2 | Full audit period, typically 12 months | No fixed statutory number; auditors sample logs as evidence |
| GDPR (Art. 5(1)(e)) | No minimum; sets a maximum | Storage limitation: personal data in logs kept no longer than needed |
| Internal debugging | Commonly 7–90 days hot | Driven by incident-investigation windows, not regulation |
The standard implementation is tiered: hot storage holds recent data on fast disks for interactive queries, warm tiers move older data to object storage such as Amazon S3, and cold archives hold compliance data that is rarely read. ClickHouse's LogHouse platform, for example, keeps six months of logs queryable by storing the compressed data directly on S3.
How do you reduce log storage costs?
Log storage costs fall fastest by shrinking data before it is stored and cheapening whatever remains. Four levers do most of the work, and each compounds with the others.
- Structure at the source. Structured, typed fields compress far better than free-form text and remove parsing infrastructure entirely.
- Filter and sample at the collector. OpenTelemetry Collector processors can drop debug-level noise or health-check spam before it ever crosses the network. The cheapest log is the one never shipped.
- Use columnar compression. Repetitive log data compresses dramatically in column-oriented formats: ClickHouse reported a 17x compression ratio on its own logs, storing 19 PiB of raw data as 1.13 PiB with the ZSTD codec.
- Tier by age. Keeping only the hot window on fast disks and everything else on object storage cuts the per-terabyte price of the long tail.
- Set deletion policies. Data past its compliance window is pure cost. Automated TTLs enforce the retention table above without manual cleanup.
Log management vs SIEM: what's the difference?
Log management is the general-purpose discipline of collecting, storing, and querying all log data; a SIEM (security information and event management system) is a security-focused consumer that correlates those logs against threat rules to detect attacks and produce compliance reports. A SIEM sits downstream of log management, not in place of it.
The term SIEM was coined by Gartner analysts Amrit Williams and Mark Nicolett in 2005, and the category's defining features (correlation rules, threat intelligence feeds, alerting on suspicious patterns) all assume a reliable stream of collected, parsed logs to work on. In practice the two share a pipeline: the same agents and aggregation layer feed both the engineering log store and the SIEM, with the SIEM typically receiving a security-relevant subset (authentication events, network flows, audit trails). Teams that conflate them usually overpay, because routing high-volume debug logs through a per-gigabyte-priced SIEM is one of the most common cost failures in the lifecycle.
How ClickStack approaches log management
ClickStack is the ClickHouse observability stack: the ClickHouse database for storage and query, HyperDX as the UI, and a bundled OpenTelemetry collector for ingestion. It covers the lifecycle stages that carry most of the cost and complexity. Collection lands via the OTel collector, structured events store as wide columnar rows, and retention runs as table-level TTL policies.
That makes retention a property of the table itself rather than a cron job. A 30-day expiry policy takes this shape:
ALTER TABLE otel_logs
MODIFY TTL TimestampTime + INTERVAL 30 DAY DELETEThe same TTL mechanism can tier aging data to object storage instead of deleting it. Columnar compression is what makes long retention windows affordable: the same 17x ratio measured on LogHouse means a 12-month PCI-style window costs a fraction of its raw size, and HyperDX queries the full window interactively rather than requiring rehydration from an archive.
FAQ
What is centralized log management?
Centralized log management collects logs from every host, container, and service into one queryable store instead of leaving them scattered across machines. Centralization is what makes cross-service debugging, uniform retention policies, and audit trails possible: one query covers the whole estate, and one policy governs its lifecycle.
How do you implement a log management system?
Start by structuring logs at the source, then deploy a collection agent such as the OpenTelemetry Collector or Fluent Bit on every host, route events to a central store, and define retention policies per compliance requirement. Add query dashboards and alerting last, once the pipeline delivers complete, structured data reliably.
What is the difference between log management and log monitoring?
Log management is the lifecycle discipline that governs log data from generation through deletion. Log monitoring watches the incoming stream in near real time to detect errors and anomalies, and it sits on top of that lifecycle. Weak management undermines monitoring, because alerts are only as good as the pipeline feeding them.
What is the difference between log management and log analytics?
Log management governs the whole lifecycle of log data; log analytics is the querying stage taken further: aggregations, group-bys, and time-series analysis across the stored logs rather than simple keyword search. Analytics quality depends directly on management quality, since unstructured or prematurely deleted data cannot be analyzed.
To see the full lifecycle running on your own logs, try ClickStack, an open-source, OpenTelemetry-native observability stack built on ClickHouse.
