How a log aggregation pipeline handles 1M events per second

The Problem Statement

Interviewer: "Your company runs 2,000 microservices across multiple regions. Each service emits structured logs, and your platform team needs to ingest, transform, and make those logs searchable within 30 seconds. Walk me through how you would design the log aggregation pipeline to handle 1 million events per second."

This question tests four things: your understanding of streaming data pipelines and backpressure, your knowledge of log collection agents and their deployment model, whether you can reason about storage tiering and retention economics, and your ability to design systems that degrade gracefully under load rather than falling over.

Most candidates draw a box called "logs go to Elasticsearch" and stop. Strong candidates cover the full pipeline: agent-based collection, buffering with Kafka, transformation stages, indexing strategies, and tiered storage that balances query speed against cost. The pipeline is the interesting part, not the destination.

Clarifying the Scenario

You: "Before I start, I want to scope this properly."

You: "When you say 1 million events per second, is that sustained or peak? And what is the average log event size?"

Interviewer: "Sustained average is about 500K eps, peak during deployments and incidents goes to 1M+. Average event size is around 1KB."

You: "So roughly 500MB/s sustained, 1GB/s peak. That is about 40TB per day of raw log data. Are we expected to retain all of it?"

Interviewer: "Hot storage for 7 days, warm for 30, cold for a year."

You: "Got it. And when you say 'searchable within 30 seconds,' is that the end-to-end latency from when the log is emitted to when a user can find it in a search?"

Interviewer: "Yes, the ingestion-to-searchable latency."

You: "One more thing: are we building on top of an existing stack like the ELK stack, or designing from scratch? And does the pipeline need to handle transformation, like parsing unstructured logs, enriching with metadata, or filtering sensitive fields?"

Interviewer: "Assume you can pick your stack. And yes, transformation and enrichment are part of the problem."

You: "I will structure my answer in four parts: log collection from services using agents, buffering and backpressure with Kafka, the transformation and enrichment pipeline, and the indexing and tiered storage layer."

My Approach

I break this into five layers, from the edge to the storage backend:

1. Collection layer: Lightweight agents on every host that tail log files, parse them, and forward structured events
2. Buffer layer: Kafka as the central buffer that decouples producers from consumers, absorbs spikes, and provides durability
3. Transformation layer: Stream processors that enrich, filter, redact, and route logs to the right destination
4. Indexing layer: Elasticsearch or ClickHouse that indexes logs for fast full-text and structured queries
5. Storage layer: Tiered storage (hot/warm/cold) that balances query performance against cost

The key insight that separates a good answer from a great one: the pipeline is not a straight line. It is a directed acyclic graph with multiple consumers. The same Kafka topic might feed Elasticsearch for search, a metrics aggregator for log-based alerting, a cold storage writer for compliance, and a sampling pipeline for cost control. Designing for this fan-out from day one is what makes the architecture extensible.

I always think about log pipelines as having two hard problems: backpressure and cost. Backpressure because services cannot stop logging when the pipeline is slow (they will either drop logs or fill up disk). Cost because at 40TB/day, storing everything in a fast search index is prohibitively expensive. Every design decision in the pipeline traces back to one of these two constraints.

The Architecture

Here is the full log aggregation pipeline. Notice the fan-out after Kafka: this is intentional. Kafka acts as the central nervous system, and multiple consumers process the same stream for different purposes.

Here is how the pipeline works end to end:

Step 1: Collection agents tail log files. Every host runs a lightweight log collection agent (Vector, Fluentd, or the Datadog Agent). The agent watches log files or captures stdout from containers. It parses structured logs (JSON) and applies initial formatting to unstructured logs. The agent batches events (typically every 100-500ms) and compresses the batch before sending.

Step 2: Agents push to Kafka. The agents produce to a Kafka cluster that acts as the central buffer. Kafka absorbs burst traffic, provides durability (if a downstream consumer is slow, logs are not lost), and enables fan-out to multiple consumers. At 1M eps, the Kafka cluster needs roughly 12 brokers with 6 partitions per topic for adequate throughput.

Step 3: Transformation pipeline consumes from Kafka. A consumer group reads from Kafka and runs three stages: enrichment (adding region, team, cost-center metadata from a service registry), PII redaction (masking emails, tokens, IP addresses), and routing (sending error logs to the hot tier at 100% and debug logs at 10% sampling).

Step 4: Indexing for search. The routed logs land in Elasticsearch or ClickHouse for full-text and structured search. The hot tier keeps 7 days of data on fast SSDs with full replicas. After 7 days, an Index Lifecycle Management (ILM) policy moves data to the warm tier with reduced replicas.

Step 5: Cold storage for compliance. All logs (regardless of sampling) are written to object storage (S3/GCS) in a columnar format like Parquet. This is the cheapest tier, costing roughly $0.02/GB/month versus $0.10+/GB/month for Elasticsearch. Cold logs can be queried with Athena or BigQuery for compliance investigations.

For your interview: name all five layers in order. Interviewers love seeing that you understand the full lifecycle, not just "logs go to Elasticsearch."

Buffering and Backpressure with Kafka

The single most important component in a log pipeline is the buffer layer. Without it, a spike in log volume (deployments, incidents, retry storms) overwhelms the indexing tier and causes cascading failures. Kafka solves this by decoupling the rate at which logs are produced from the rate at which they are consumed.

Here is how backpressure flows through the pipeline:

The critical insight: Kafka acts as a shock absorber. When Elasticsearch is overwhelmed (bulk indexing returns 429s or times out), the transformation consumers slow down or pause. Kafka happily stores the buffered logs for up to 72 hours. The agents keep producing because Kafka can absorb writes at a much higher rate than Elasticsearch can index.

Log Transformation and Enrichment

Raw logs from services are not ready for indexing. They need three types of processing: enrichment (adding metadata the service does not know about), normalization (making logs from different services queryable with the same fields), and redaction (removing sensitive data before it reaches storage).

Here is the transformation pipeline in detail:

Parsing is the first stage. Structured logs (JSON) are easy: extract fields directly. Unstructured logs require Grok patterns or regex to extract timestamp, level, message, and other fields. Multiline joining handles stack traces and exception blocks that span multiple log lines.

Enrichment adds context the service does not emit. The most valuable enrichment is service metadata from a registry: which team owns this service, what cost center it belongs to, what version is deployed. This enables queries like "show me all error logs from Team Payments in the last hour." GeoIP enrichment resolves IP addresses to regions. Trace correlation links logs to distributed traces using the trace_id field.

PII redaction is critical and non-negotiable. Logs frequently contain sensitive data: email addresses in error messages, API tokens in request/response bodies, IP addresses in access logs. The redaction stage uses configurable regex patterns and (at scale) ML-based detectors to mask sensitive fields before they reach any storage tier. Once PII lands in Elasticsearch, removing it is extremely difficult due to immutable segments.

Sampling and routing is where cost control happens. Not all logs have equal value. Error and warning logs are high-value and go to the hot index at 100%. Debug logs are low-value and get sampled at 10% for the hot index (but 100% goes to cold storage). Audit logs go directly to cold storage at 100% for compliance. This sampling alone can reduce hot storage costs by 60-70%.

Tiered Storage and Retention

At 40TB/day of raw logs, storage cost dominates the pipeline budget. The solution is tiered storage: keep recent, frequently queried logs on fast (expensive) storage, and move older logs to progressively cheaper tiers. This is the same principle S3 uses with its storage classes, but applied to your log index.

Here is how the tiers work:

The total retention strategy looks like this:

• Hot tier (7 days): After sampling, roughly 300K eps reaches Elasticsearch. At 1KB per event compressed to ~200 bytes indexed, that is about 5TB/day indexed. With 7-day retention and 1 replica, the hot tier needs approximately 70TB of SSD.
• Warm tier (30 days): At day 7, Elasticsearch ILM (Index Lifecycle Management) moves indices to warm nodes. These use HDDs, have 0 replicas (or 1 if you need durability), and use force_merge to reduce segment count. Query latency increases but cost drops 60-70%.
• Cold tier (1 year): At day 30, logs are exported to S3/GCS in Parquet format. Full-text search is not available, but columnar queries (Athena, BigQuery, Presto) can scan the data. At $0.023/GB/month for S3 Standard, one year of compressed logs costs roughly the same as one week of hot Elasticsearch storage.

The Tricky Parts

1. Clock skew across services. Log timestamps come from the emitting service's clock. If service clocks drift (common in container environments without NTP), logs appear out of order. When debugging a request flow across 5 services, a 2-second clock skew makes the timeline incomprehensible. The fix: always include a trace_id and let the tracing system provide the authoritative ordering. Use log timestamps for coarse search ("last hour") and trace IDs for precise ordering.

2. Schema evolution and heterogeneous formats. With 2,000 services, you will have JSON logs, plain text logs, logs with different field names for the same concept (e.g., user_id vs userId vs uid), and services that change their log format without notifying the platform team. The pipeline must handle unknown fields gracefully (index them as dynamic fields or drop them), normalize common fields, and not crash when it encounters an unexpected format.

3. Consumer lag during incidents. The worst time for the log pipeline to fall behind is during an incident, which is exactly when it will fall behind because log volume spikes 5-10x. If your Elasticsearch cluster is sized for normal load, it cannot handle incident load. The solution is a combination of headroom (provision for 2-3x normal), dynamic sampling (increase sampling rate for debug logs during spikes), and priority queues (error logs get indexed before debug logs).

4. Reindexing after schema changes. When you add a new enrichment field or change a mapping, you need to reindex historical data. With 35TB in the hot tier, a full reindex takes hours and competes with live indexing for cluster resources. The solution: run the reindex against a separate cluster (or separate indices), then swap aliases once complete. Kafka's 72-hour retention means you can replay recent events through the updated pipeline.

5. Multi-tenancy and noisy neighbors. If one team deploys a chatty service that suddenly emits 10x more logs, they consume a disproportionate share of pipeline capacity. Without per-team quotas, one team's bug can degrade observability for everyone. The fix: per-service rate limits at the agent level, per-team quotas in the routing layer, and chargeback based on log volume so teams have an incentive to log responsibly.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"A log aggregation pipeline at this scale has five layers. At the edge, lightweight agents like Vector or Fluentd run on every host. They tail log files, parse structured and unstructured formats, batch events, and compress them before shipping.

The agents send to Kafka, which is the critical buffer layer. Kafka absorbs traffic spikes, provides 72-hour durability, and decouples producers from consumers. This means if my indexing tier falls behind, logs are not lost. They buffer safely in Kafka until the consumer catches up.

After Kafka, I run a transformation pipeline. Three stages: enrichment (add team, region, service version from a registry), PII redaction (mask emails, tokens, IPs before they reach storage), and routing (error logs go to the hot index at 100%, debug logs at 10% sampling, all logs go to cold archive at 100%).

The indexed logs land in Elasticsearch or ClickHouse for the hot tier, which keeps 7 days on SSDs. After 7 days, ILM moves data to a warm tier on cheaper storage. After 30 days, it exports to S3 in Parquet format for 1-year retention. This tiered approach cuts storage costs by 10x compared to keeping everything in Elasticsearch.

The key design decision is putting Kafka in the middle to provide backpressure and fan-out. Without it, you have no shock absorber when log volume spikes during incidents, which is exactly when you need logs the most."

Interview Cheat Sheet

• Hear "log pipeline" → say "five layers: collection agents, Kafka buffer, transformation, indexing, tiered storage"
• Hear "how do you handle spikes" → say "Kafka absorbs spikes with 72-hour retention, consumers process at their own pace, agent spills to local disk as last resort"
• Hear "what about cost" → say "tiered storage: 7-day hot on SSD, 30-day warm on HDD, 1-year cold in S3 Parquet. Debug logs sampled at 10% for hot tier"
• Hear "what agent do you use" → say "Vector for new deployments (Rust, lower resource usage), Fluentd for existing setups. Both support backpressure, disk buffering, and structured parsing"
• Hear "why Kafka" → say "durability, backpressure, fan-out. Same topic feeds ES, cold storage, alerting. Without Kafka, ES brownout means lost logs"
• Hear "PII in logs" → say "mandatory redaction stage before any storage tier. Regex patterns plus ML-based detection. Once PII reaches ES, removing it is nearly impossible due to immutable segments"
• Hear "how fast can you search" → say "30-second ingestion-to-searchable latency for the hot tier. Cold tier queries via Athena take 10-60 seconds"
• Hear "what about multi-tenancy" → say "per-service rate limits at the agent, per-team quotas in the router, chargeback based on log volume so teams self-regulate"
• Hear "Elasticsearch vs ClickHouse" → say "ES for full-text search and unstructured logs. ClickHouse for high-cardinality structured logs with columnar queries. ClickHouse is 5-10x cheaper for the same query performance on structured data"
• Hear "what about schema changes" → say "Kafka retention enables replay. Reindex to a new cluster, then swap aliases. Dynamic mappings handle unknown fields gracefully"

Test Your Understanding

---

---

---

---

---

---

---

Quick Recap

1. A log aggregation pipeline has five layers: collection agents, Kafka buffer, transformation, indexing, and tiered storage.
2. Kafka is the critical component that provides backpressure, durability, fan-out, and replay capability.
3. Transformation includes parsing, enrichment (team, region, version), PII redaction, and severity-based routing.
4. Sampling debug logs at 10% for the hot tier while sending 100% to cold storage cuts indexing costs by 60-70%.
5. Tiered storage (hot SSD/warm HDD/cold S3) reduces total storage cost by 10x compared to keeping everything in Elasticsearch.
6. PII redaction must happen before any storage tier because removing PII from Elasticsearch after indexing is nearly impossible.
7. Monitor Kafka consumer lag as the single most important health indicator for the entire pipeline.
8. During incidents, degrade gracefully by increasing sampling rates rather than dropping logs entirely.

Related Concepts

• Message Queues and Kafka: Understanding Kafka's partitioning, consumer groups, and retention model is essential for the buffer layer of any log pipeline.
• Elasticsearch Internals: How inverted indices, segments, and merging work explains why bulk indexing has throughput limits and why tiered storage uses ILM policies.
• Distributed Tracing: Log correlation with trace IDs connects log aggregation to the broader observability stack (logs, metrics, traces).
• Stream Processing: The transformation layer is a stream processing job. Understanding frameworks like Kafka Streams or Flink helps design complex enrichment and routing logic.
• Data Tiering and Storage Economics: The hot/warm/cold model applies beyond logs to any system with time-decaying access patterns (metrics, analytics, data warehouses).