How Prometheus collects and stores metrics

The Interview Question

Interviewer: "Your team uses Prometheus to monitor a Kubernetes cluster with 500 microservices. You are seeing high memory usage on the Prometheus server and some queries are timing out. Walk me through how Prometheus stores and queries time-series data internally, and where the bottleneck is likely occurring."

This question tests whether you understand Prometheus beyond "it scrapes metrics and shows graphs." The interviewer wants to hear about the TSDB internals (head block, WAL, compaction), the pull-based scraping model, PromQL's evaluation pipeline, and how cardinality affects performance. If you just say "Prometheus is a pull-based monitoring system," you get partial credit. If you explain how the TSDB stores samples in compressed chunks, how compaction merges blocks, and why high label cardinality causes the memory spike, you nail it.

What to Clarify Before Answering

You: "Before I dive in, let me clarify a few things..."

• "How many active time series are we tracking? 100K and 10M have very different memory profiles."
• "What is the scrape interval? 15 seconds is standard, but 5-second intervals quadruple the ingestion rate."
• "Are we running a single Prometheus instance or a federated setup with Thanos or Cortex?"
• "What retention period is configured? 15 days of local storage vs 90 days significantly changes disk and compaction behavior."
• "Are there any recording rules in place, or is everything evaluated as raw PromQL at query time?"

Why this matters: Prometheus performance is almost entirely determined by three numbers: active time series count (cardinality), scrape interval, and retention period. A candidate who asks for these numbers shows they understand the TSDB's cost model.

The 30-Second Answer

Prometheus uses a pull-based model, scraping HTTP endpoints on target services at a configured interval (typically 15 seconds). Each scrape collects the current value of every metric exposed by the target. These samples are ingested into a custom TSDB that stores data in two stages: a head block (in-memory, for recent data) and persistent blocks (on disk, for older data). The head block uses a Write-Ahead Log (WAL) for crash recovery. Samples are compressed using delta-of-delta encoding for timestamps and XOR compression for float values, achieving approximately 1.37 bytes per sample. When you query with PromQL, the evaluation engine creates an iterator tree that reads from both the head block and on-disk blocks, merging results transparently. Recording rules pre-compute expensive queries on a schedule, and alerting rules feed into the Alertmanager, which handles routing, grouping, silencing, and deduplication before delivering notifications.

The Architecture Overview

The architecture has a clear data flow. The Scrape Manager discovers targets through service discovery (Kubernetes API, Consul, DNS, static files) and scrapes each target's /metrics endpoint on a schedule. Scraped samples pass through relabeling rules that filter, rename, or drop labels before ingestion. The TSDB stores samples in a head block (in-memory) backed by a WAL, with periodic compaction to on-disk blocks.

I find the pull-based design to be the most defining architectural decision. Prometheus pulls metrics from targets rather than targets pushing to Prometheus. This means Prometheus controls the scrape rate, can detect target failures (a failed scrape is informative), and targets do not need to know about the monitoring system. The tradeoff is that Prometheus must be able to reach every target over the network, which complicates monitoring across network boundaries.

The query layer (PromQL engine) and the alerting pipeline run independently from ingestion. Recording rules pre-compute expensive aggregations, and alerting rules evaluate conditions and delegate notification delivery to the separate Alertmanager service.

Pull-Based Scraping: The Collection Model

Prometheus scrapes targets by sending an HTTP GET request to each target's /metrics endpoint. The response is plain text in Prometheus exposition format (or the newer OpenMetrics format), containing the current value of every metric.

# HELP http_requests_total Total HTTP requests processed
# TYPE http_requests_total counter
http_requests_total{method="GET",handler="/api",status="200"} 142857
http_requests_total{method="POST",handler="/api",status="201"} 8234
http_requests_total{method="GET",handler="/health",status="200"} 950123

# HELP http_request_duration_seconds HTTP request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 85234
http_request_duration_seconds_bucket{le="0.5"} 130456
http_request_duration_seconds_bucket{le="1.0"} 141890
http_request_duration_seconds_bucket{le="+Inf"} 142857
http_request_duration_seconds_sum 28571.4
http_request_duration_seconds_count 142857

Each scrape is independent. Prometheus does not maintain a persistent connection to targets. The scrape manager runs one goroutine per target, each sleeping for the scrape interval and then performing the HTTP request, parsing the response, and appending samples to the TSDB.

Why Pull, Not Push?

This is the question every interviewer asks when Prometheus comes up. There are four concrete reasons:

1. Prometheus controls the scrape rate. If a target is slow, Prometheus detects it (scrape duration metric) and does not overwhelm itself. With push, a misbehaving service can flood the monitoring system.
2. Target failure is observable. If a scrape fails, Prometheus records up{job="...",instance="..."}=0. You get a free health check on every target for free. With push, silence could mean "the service is healthy but has nothing to report" or "the service is dead."
3. No client-side configuration. Targets just expose /metrics. They do not need to know the Prometheus server's address, authentication credentials, or retry logic.
4. Service discovery integration. Prometheus discovers targets dynamically through Kubernetes API, Consul, DNS, or file-based discovery. Targets do not need to register themselves.

The tradeoff is network reachability. Prometheus must be able to reach every target. For targets behind NAT or in different networks, you use the Pushgateway (for batch jobs) or remote write to bridge the gap.

Service Discovery: Finding Targets Dynamically

In a dynamic environment like Kubernetes, targets appear and disappear constantly. Prometheus integrates with multiple service discovery mechanisms:

• Kubernetes SD: Watches the Kubernetes API for pods, services, endpoints, and ingresses. Automatically discovers new pods with the right annotations.
• Consul SD: Queries Consul's service catalog for registered services.
• DNS SD: Performs DNS SRV record lookups to discover targets.
• EC2 SD: Queries the AWS API for EC2 instances matching tag filters.
• File SD: Watches a JSON/YAML file on disk that lists targets. Useful for static environments or custom integrations.

In Kubernetes, the most common pattern is using pod annotations to control scraping:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"

Prometheus's Kubernetes SD discovers the pod, reads the annotations via relabeling rules, and adds it to the scrape target list. When the pod is deleted, Prometheus removes it automatically. The up metric transitions to 0, and if you have an alert on up == 0 for 5m, it fires only for unexpected disappearances, not for normal scaling events.

Relabeling: The ETL Pipeline

Between scraping and storage, samples pass through relabeling rules. This is Prometheus's most powerful and most confusing feature. Relabeling rules can:

• Drop targets before scraping (relabel_configs with action: drop)
• Rename labels on scraped metrics (metric_relabel_configs)
• Drop entire time series with high-cardinality labels
• Extract labels from target metadata (Kubernetes pod annotations, Consul tags)

I think of relabeling as Prometheus's ETL pipeline. It is where you control cardinality, normalize label names across services, and filter out metrics you do not need. Getting relabeling right is the single most important operational skill for running Prometheus at scale.

Metric Types: Counter, Gauge, Histogram, Summary

Prometheus has four metric types, and understanding the differences is important for correct PromQL usage:

Counter: A monotonically increasing value that resets to zero on restart. Examples: total HTTP requests, bytes sent. Always use rate() or increase() on counters, never look at raw values.

Gauge: A value that can go up or down. Examples: current memory usage, active connections, temperature. Use raw values, min_over_time(), max_over_time(), or delta().

Histogram: Pre-defined buckets that count observations. A histogram with le buckets like {le="0.1"}, {le="0.5"}, {le="1.0"} tells you how many observations fell below each threshold. Use histogram_quantile() to compute percentiles. Each bucket is a separate time series, so a histogram with 10 buckets across 100 services creates 1,000 series just for that metric.

Summary: Client-side computed quantiles. Unlike histograms, summaries compute percentiles in the application itself and expose them as {quantile="0.99"}. The tradeoff: summaries cannot be aggregated across instances (you cannot average percentiles), while histogram buckets can. I recommend histograms over summaries in almost all cases because they support aggregation.

Native Histograms (Experimental)

Classic Prometheus histograms have a cardinality problem. A histogram with 20 le buckets creates 20 time series per label combination (plus _sum and _count). Across 100 services with 5 endpoints each, that is 20 x 100 x 5 = 10,000 series for a single histogram metric.

Native histograms (introduced in Prometheus 2.40) solve this by storing the entire histogram distribution as a single sample on a single time series. Instead of fixed buckets defined at instrumentation time, native histograms use exponential bucket schemas that automatically adapt to the distribution. The bucket boundaries follow a geometric progression, providing consistent relative resolution across all value ranges.

The advantages are significant:

• ~10x fewer time series for the same metric
• No bucket configuration needed at instrumentation time
• Higher resolution because buckets adapt automatically
• Smaller scrape payloads using Protocol Buffer encoding

The tradeoff is that native histograms require Protocol Buffer scraping (not the text exposition format), and PromQL support is still evolving. I recommend experimenting with native histograms for new deployments, but keeping classic histograms in production until the feature stabilizes fully.

TSDB Internals: How Prometheus Stores Samples

The TSDB (Time Series Database) is the core of Prometheus. It is a custom storage engine optimized for time-series workloads: high write throughput (millions of samples per second), efficient compression, and fast range queries over recent data.

The Write Path

When a sample arrives, two things happen simultaneously:

1. WAL write: The sample is appended to the Write-Ahead Log, a series of segment files on disk. This ensures crash recovery. If Prometheus crashes, it replays the WAL on startup to reconstruct the head block.

2. Head append: The sample is added to the in-memory head block. Prometheus looks up (or creates) the time series using its label set as the key, then appends the sample to the series' active chunk.

The head block contains all samples from approximately the last 2 hours. Each time series has an active chunk that accumulates samples. When a chunk fills up (~120 samples), it is "cut" (finalized) and memory-mapped from disk. This means completed chunks do not consume Go heap memory, only the active chunks and the in-memory index do.

WAL Internals

The Write-Ahead Log is structured as a series of numbered segment files, each up to 128 MB. Segments are written sequentially and never modified after creation. The WAL records three types of entries:

• Series records: New time series (label set to internal ID mapping). These appear when a metric is seen for the first time.
• Sample records: Timestamp-value pairs for existing series. These are the bulk of WAL data.
• Tombstone records: Delete markers for series that should be excluded from queries.

On startup after a crash, Prometheus replays the WAL from the last checkpoint forward, reconstructing the head block. A checkpoint is a compacted snapshot of the WAL that contains only series records (no samples for series that are no longer active). Checkpoints are created periodically to speed up replay. Without checkpoints, startup after a long run would require replaying the entire WAL, which could take minutes.

The WAL is the reason Prometheus can recover from crashes without data loss (for the most recent samples). However, the WAL does not protect against disk failure. If the disk hosting the WAL dies, all non-persisted head block data is lost. For critical deployments, I recommend remote write as a secondary safety net.

Compression: Delta-of-Delta and XOR

Prometheus achieves remarkable compression ratios using two algorithms inspired by Facebook's Gorilla TSDB paper:

Timestamps use delta-of-delta encoding. If samples arrive at regular intervals (e.g., every 15 seconds), the delta between consecutive timestamps is constant (15). The delta-of-delta is zero, which compresses to a single bit. In practice, most scrapes happen within a few milliseconds of the expected time, so the delta-of-delta is very small (a few bits per sample).

Values use XOR compression. Consecutive samples of the same metric often have similar float values. XOR-ing consecutive values produces a number with many leading and trailing zeros. Only the significant bits are stored, along with the count of leading zeros. For slowly changing metrics (like a counter that increments by small amounts), this achieves 1-2 bits per sample. For volatile metrics, it is still significantly better than storing raw 8-byte floats.

The combined compression achieves approximately 1.37 bytes per sample on average, which means 1 million active time series scraped every 15 seconds produces about 5.5 MB per minute, or 8 GB per day. This is why Prometheus can handle millions of time series on a single server.

Compaction: Merging Blocks

Every 2 hours, the head block's completed chunks are written to disk as a new block. Over time, this creates many small 2-hour blocks. Compaction merges adjacent blocks into larger ones, following an exponential scheme:

• Fresh blocks: 2 hours each
• After first compaction: 6 hours (3 blocks merged)
• After second compaction: 18 hours to days
• Maximum block size: ~31 hours (10% of retention period)

Compaction also removes tombstoned (deleted) series, rebuilds the index for efficient querying, and reclaims disk space. The compaction process runs in the background and is the primary source of disk I/O on a Prometheus server.

The Inverted Index

The TSDB's inverted index maps label name-value pairs to sets of series IDs (called "posting lists"). When you query {service="api", status="500"}, the engine:

1. Looks up the posting list for service="api" (e.g., series IDs: 92)
2. Looks up the posting list for status="500" (e.g., series IDs: 200)
3. Intersects the two lists: 38
4. Loads chunks only for series 5 and 38

The posting lists are sorted, so intersection is O(n) using a merge join. This is why label-based queries are fast even with millions of total series: the work scales with matching series, not total series.

The inverted index lives in the head block (in-memory) for recent data and in on-disk block index files for persisted data. Each on-disk block has its own index file, and the querier merges results from all relevant blocks at query time.

Exemplars: Linking Metrics to Traces

Prometheus supports exemplars, which attach trace IDs to specific samples. When a histogram bucket receives a sample, the application can include a trace ID as an exemplar:

http_request_duration_seconds_bucket{le="0.5"} 130456 # {trace_id="abc123"} 0.45 1712937600

This creates a bridge between metrics and traces. Grafana can display exemplar dots on metric graphs that link directly to the corresponding trace in Jaeger or Tempo. I find this invaluable for debugging: you see a latency spike on a graph, click the exemplar dot, and jump straight to the trace that shows exactly what happened.

Exemplars are stored separately from regular samples and have a short retention (typically matching the head block's 2-hour window). They are not compressed or compacted like regular samples.

PromQL Evaluation: How Queries Work

PromQL (Prometheus Query Language) is a functional query language designed for time-series data. Every PromQL query is evaluated by creating a tree of iterators that lazily read and transform data from the TSDB.

Instant vs Range Queries

Prometheus supports two query types:

Instant query: Evaluates at a single timestamp and returns the current value of each matching series. Used for dashboard panels showing "current value."

http_requests_total{service="api", status="200"}
# Returns: the most recent sample for each matching series

Range query: Evaluates over a time range and returns a matrix of samples. Used for graphing and rate calculations.

rate(http_requests_total{service="api"}[5m])
# Returns: per-second rate over the last 5 minutes, for each matching series

rate() vs irate()

This is one of the most common PromQL interview questions. Both calculate per-second rates from counter metrics, but they work differently.

rate(counter[5m]) calculates the average per-second increase over the entire 5-minute window. It uses the first and last data points in the range, dividing the difference by the time span. This smooths out spikes and is the correct choice for alerting and most dashboards.

irate(counter[5m]) calculates the instantaneous rate using only the last two data points in the range. It is much more sensitive to spikes but also much noisier. The [5m] range is only used to find the last two samples (a larger window makes it more resilient to missing scrapes), not for averaging.

Common PromQL Patterns

Beyond rate and irate, several PromQL patterns come up frequently in interviews and production:

Error rate as a ratio:

rate(http_requests_total{status=~"5.."}[5m])
  /
rate(http_requests_total[5m])

This gives you the fraction of requests that returned 5xx errors over the last 5 minutes. This is the foundation for SLO-based alerting.

Histogram percentiles:

histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)

Computes the 99th percentile latency from histogram buckets. Note: rate() is applied to the bucket counters first because histograms are cumulative counters.

Top-K by label:

topk(5, rate(http_requests_total[5m]))

Returns the 5 time series with the highest request rate. Useful for finding hot endpoints or noisy services.

Predicting disk full:

predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600
)

Uses linear regression over the last 6 hours to predict the available disk space 24 hours from now. If the predicted value is negative, the disk will fill up.

Subquery Syntax

PromQL supports subqueries for evaluating a range of instant query results over time:

max_over_time(rate(http_requests_total[5m])[1h:1m])

This computes the rate() over 5-minute windows, evaluated every 1 minute for the last hour, then takes the maximum value. Subqueries are powerful but expensive: the inner expression is evaluated at every step within the outer range.

The Query Evaluation Pipeline

When Prometheus evaluates a PromQL query, it builds an iterator tree:

1. Select series: The inverted index finds all series matching the label selectors. For {service="api", status="200"}, it intersects the posting lists for both labels.
2. Load chunks: For each matching series, load the relevant chunks (from head block or on-disk blocks) that overlap the query time range.
3. Decode samples: Decompress the XOR-encoded values and delta-of-delta timestamps.
4. Apply functions: Functions like rate(), histogram_quantile(), and avg() are applied as iterator transformations. Each function produces a new iterator that reads from the layer below.
5. Aggregate: Aggregation operators (sum by, avg by, topk) group results and compute the final output.

The key performance insight is that the inverted index lookup is O(n) in the number of matching series, not the total number of series. A query for {service="api"} that matches 1,000 series out of 1 million is fast because the posting list for service="api" directly identifies the matching series IDs.

Recording Rules: Pre-Computing Expensive Queries

Recording rules evaluate a PromQL expression at a regular interval and store the result as a new time series. This is essential for two reasons:

1. Dashboard performance: If 10 Grafana users are viewing the same dashboard, each with a panel that computes histogram_quantile(0.99, rate(http_duration_bucket[5m])) over 10,000 series, the TSDB handles the same expensive query 10 times per refresh. A recording rule computes it once and stores the result as a simple gauge that is cheap to query.

2. Alerting stability: Alerting rules that reference recording rules evaluate against pre-computed, stable data instead of computing from raw samples each time. This eliminates timing-dependent query result variations.

groups:
  - name: api_rules
    interval: 30s
    rules:
      - record: api:request_duration:p99_5m
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="api"}[5m]))
      - record: api:error_rate:ratio_5m
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

I consider recording rules non-negotiable for any Prometheus deployment with more than a few dashboards. They transform Prometheus from "expensive queries on every page load" to "cheap lookups of pre-computed aggregations."

Staleness Handling

When a target disappears (pod deleted, service crashed), Prometheus needs to stop returning data for its series. The staleness model works as follows:

After a target fails to scrape, Prometheus inserts a staleness marker (a special NaN value) for all series from that target. PromQL queries treat staleness markers as "no data" and stop including the series in results after 5 minutes of no new samples. This 5-minute staleness window is important: it bridges brief scrape failures without dropping data, but eventually stops returning stale data for truly gone targets.

This staleness model is why you sometimes see a brief "flat line" after a service disappears before it drops from graphs. The last sample continues to be returned for up to 5 minutes.

Handling Counter Resets

Counters in Prometheus are monotonically increasing values that reset to zero when the process restarts. The rate() and increase() functions handle resets automatically by detecting when a counter value decreases and adjusting the calculation.

Here is how it works: if rate() sees samples [100, 105, 110, 0, 8], it detects the drop from 110 to 0 as a reset. It assumes the counter had reached some unknown value before resetting and that 8 represents new growth since the reset. The calculated rate includes both the pre-reset increase (110 - 100 = 10) and the post-reset growth (8), spread over the total time window.

This works well for single resets but can produce misleading results if a counter resets multiple times within the rate window (e.g., a CrashLoopBackOff pod). In that case, widen the rate window or fix the underlying instability.

Alerting Pipeline: From Rules to Notifications

Prometheus evaluates alerting rules on a configurable interval (default: 1 minute). When a rule's PromQL expression returns a non-empty result, Prometheus transitions the alert through states and eventually sends it to the Alertmanager.

The for Duration

The for clause in alerting rules prevents flapping. An alert must remain in the "pending" state for the specified duration before transitioning to "firing." Without for, a brief metric spike could trigger a page that resolves before anyone looks at it.

I recommend a for duration of at least 5 minutes for most alerts. This filters out transient noise while still catching real problems quickly. For critical infrastructure alerts (database down, service unreachable), 1-2 minutes is appropriate.

Alertmanager: Routing, Grouping, and Deduplication

The Alertmanager is a separate service that receives alerts from Prometheus and manages notification delivery. The separation is intentional: it allows multiple Prometheus servers to send alerts to the same Alertmanager cluster, and the Alertmanager handles deduplication across all senders.

Its key features:

Grouping: Alerts with the same labels (e.g., all severity="critical" alerts from the same cluster) are grouped into a single notification. Without grouping, a cascading failure could generate hundreds of individual pages. The group_wait parameter controls how long to wait for additional alerts before sending the first notification (default: 30 seconds). This batching window is important: during a cascade, many alerts fire within seconds of each other, and grouping collects them into one coherent notification.

Deduplication: If multiple Prometheus instances send the same alert (in an HA setup), Alertmanager deduplicates them so you only get one notification. The gossip protocol (built on Hashicorp's memberlist library) synchronizes notification state between Alertmanager peers.

Silences: Time-bounded suppressions for known issues. When you are performing maintenance, create a silence to prevent expected alerts from page engineers.

Inhibition: Automatic suppression based on active alerts. If cluster_down is firing, inhibit all pod_down alerts for that cluster because they are redundant.

Federation and Remote Write: Scaling Beyond One Server

A single Prometheus server can handle 1-10 million active time series. Beyond that, you need to distribute the workload. Prometheus provides two scaling mechanisms.

Hierarchical Federation

A top-level Prometheus scrapes aggregated metrics from leaf Prometheus servers. Each leaf handles a subset of targets (e.g., one per Kubernetes cluster). The top-level server only ingests recording rule outputs and aggregated metrics, keeping its cardinality manageable.

Federation works for simple setups (5-10 clusters, moderate cardinality). It breaks down when you need cross-cluster ad-hoc queries because the top-level server only has pre-aggregated data.

Remote Write

Prometheus can send samples to a remote storage backend in real time via the remote write protocol. Each sample is sent as a Protocol Buffer message over HTTP. The remote write sender buffers samples in a WAL-backed queue and sends them in batches, with configurable retry and backoff.

The remote write path is separate from the local TSDB ingestion. Samples are written to both the local TSDB and the remote write queue simultaneously. If the remote endpoint is temporarily unavailable, the WAL-backed queue holds samples until the endpoint is reachable again (up to the configured capacity).

This is the foundation for modern Prometheus-compatible long-term storage systems:

• Thanos: Adds a sidecar to each Prometheus that uploads blocks to object storage (S3, GCS). A querier component provides a unified query layer across all clusters.
• Cortex / Mimir: A horizontally scalable, multi-tenant TSDB that accepts remote-write from multiple Prometheus instances. Used by Grafana Cloud.
• VictoriaMetrics: A high-performance, single-binary alternative that is API-compatible with Prometheus and handles higher cardinality at lower resource cost.

Thanos Architecture Deep Dive

Thanos is the most commonly discussed scaling solution in interviews. Its architecture adds four components around Prometheus:

1. Sidecar (runs next to each Prometheus): Uploads the 2-hour persisted blocks to object storage. Also proxies queries to the local Prometheus for recent data.
2. Store Gateway: Reads blocks from object storage and serves them to the Querier. Uses an index cache (typically memcached) to avoid reading the full index from remote storage on every query.
3. Querier: The unified query layer. Receives PromQL queries, fans out to all Sidecars and Store Gateways, merges results, and deduplicates overlapping data from HA Prometheus pairs.
4. Compactor: Runs against object storage. Merges small blocks into larger ones, downsamples old data (5-minute and 1-hour resolution) to reduce storage costs, and removes deleted series.

The key insight with Thanos is that local Prometheus servers only need enough storage for the head block plus a few hours of buffer. Everything else lives in object storage (S3, GCS, Azure Blob), which is effectively unlimited and inexpensive. This separates compute (Prometheus scraping and short-term queries) from long-term storage.

Operational Best Practices for Prometheus

Running Prometheus in production requires attention to a few critical operational concerns:

Right-sizing memory: The head block is the primary memory consumer. Each active time series consumes approximately 3.2 KB of memory in the head block (including the inverted index entry and active chunk). Plan for head_series_count * 3.2 KB + 30% overhead for Go runtime. A Prometheus with 1 million active series needs approximately 4 GB of RAM minimum, with 6-8 GB recommended for query headroom.

Disk sizing: Each sample occupies ~1.37 bytes on disk. Calculate daily disk usage as: active_series * (86400 / scrape_interval) * 1.37 bytes. For 1M series at 15-second intervals, that is approximately 8 GB per day. With 15 days retention, plan for 120 GB of disk plus 30% headroom for compaction temporary space.

Monitoring Prometheus itself: Prometheus exposes its own metrics. The most important ones to watch:

Avoiding common pitfalls:

• Do not scrape the same target from multiple Prometheus servers unless you have a deduplication layer (Thanos, Mimir). Duplicate scrapes waste resources.
• Set scrape_timeout lower than scrape_interval. If a scrape takes longer than the interval, Prometheus cannot keep up and queues build.
• Use honor_labels: true cautiously. It lets the target override Prometheus-added labels (like instance and job), which can break service discovery expectations.

What Happens When Things Break

Performance Characteristics

How This Compares to Alternatives

I reach for Prometheus when monitoring Kubernetes workloads with standard cardinality. The ecosystem (Grafana, Alertmanager, exporters) is unmatched. I switch to VictoriaMetrics when cardinality exceeds what a single Prometheus can handle, because it is API-compatible and significantly more memory-efficient. For teams that do not want to operate monitoring infrastructure, Datadog or Grafana Cloud (which uses Mimir underneath) are the correct choice.

Interview Cheat Sheet

• When asked about pull vs push: "Prometheus uses pull-based scraping. The server controls the scrape rate, failed scrapes provide free health checking (up=0), targets need zero monitoring-specific configuration, and service discovery integrates natively. The tradeoff is network reachability requirements."
• When asked about TSDB storage: "Samples go to the head block (in-memory, ~2h) with WAL for crash recovery. Completed chunks are memory-mapped. Every 2h, the head is persisted to a disk block. Compaction merges small blocks into larger ones. Compression uses delta-of-delta for timestamps and XOR for values, achieving ~1.37 bytes per sample."
• When asked about cardinality: "Each unique label combination creates a new time series. Unbounded labels (user IDs, IPs) cause cardinality explosions that OOM the head block. Monitor prometheus_tsdb_head_series, drop high-cardinality labels via relabeling, and send per-user data to tracing/logging systems instead."
• When asked about rate() vs irate(): "rate() averages over the full window (stable, correct for alerting). irate() uses only the last two data points (sensitive to spikes, correct for drill-down dashboards). Never use irate in alerting rules."
• When asked about alerting: "Prometheus evaluates alerting rules (PromQL expressions) on a schedule. Alerts transition through inactive → pending → firing. The for clause prevents flapping. Alertmanager handles routing, grouping, deduplication, silencing, and inhibition. Group alerts by cluster and severity to prevent alert fatigue."
• When asked about scaling: "Single Prometheus handles 1-10M series. For scale, use remote write to Thanos (sidecar + object storage), Mimir (horizontally scalable TSDB), or VictoriaMetrics (efficient single-binary). Federation works for 5-10 clusters with pre-aggregated data."
• When asked about compression: "Gorilla-inspired encoding. Timestamps: delta-of-delta (regular intervals compress to ~1 bit). Values: XOR of consecutive floats, storing only significant bits. Result: ~1.37 bytes per sample. This is why a single server can store millions of series."
• When asked about PromQL performance: "The inverted index maps labels to series IDs. Query cost scales with matching series, not total series. Range queries iterate over compressed chunks. Recording rules pre-compute expensive aggregations to amortize query cost."

Test Your Understanding

Quick Recap

1. Prometheus uses a pull-based scraping model, sending HTTP GET requests to target /metrics endpoints at regular intervals and parsing the exposition format response.
2. The TSDB stores samples in a head block (in-memory, ~2 hours) backed by a WAL for crash recovery, with periodic compaction to on-disk blocks.
3. Compression uses delta-of-delta encoding for timestamps and XOR encoding for float values, achieving ~1.37 bytes per sample.
4. PromQL evaluates queries by building iterator trees that read from the inverted index and compressed chunks, with cost proportional to matching series (not total series).
5. rate() averages over a full window (correct for alerting), while irate() uses only the last two data points (correct for drill-down investigation).
6. Alerting rules transition through inactive, pending, and firing states, with the for clause preventing flapping, before Alertmanager handles routing, grouping, and deduplication.
7. Cardinality (unique label combinations) is the primary scaling constraint. High cardinality labels cause OOM in the head block and slow queries.
8. For retention beyond 15 days or cross-cluster queries, use remote write to Thanos, Mimir, or VictoriaMetrics rather than scaling a single Prometheus vertically.

Related Concepts

• Grafana dashboards: Grafana is the standard visualization layer for Prometheus data, using PromQL queries to power dashboard panels, with alerting capabilities that complement Prometheus's native alerting.
• OpenTelemetry: The OpenTelemetry Collector can receive, process, and export metrics in Prometheus format, bridging Prometheus's pull model with push-based telemetry pipelines.
• Service mesh observability: Istio and Linkerd automatically generate Prometheus metrics for service-to-service communication (request rate, error rate, latency) without application instrumentation.
• eBPF-based monitoring: Tools like Cilium Hubble and Pixie use eBPF to generate metrics without application-level instrumentation, exporting in Prometheus format for scraping.