Weather Service

What is a weather data service?

A weather data service ingests atmospheric readings from distributed sensor networks and third-party data providers, stores billions of time-stamped measurements, and serves current conditions plus short-term forecasts by geographic coordinate. The engineering challenge is not the meteorology: every user query must translate an arbitrary lat/lng into readings from the nearest stations and aggregate them in under 100ms while the ingestion pipeline handles thousands of sensor writes per second in parallel. It tests IoT ingestion, time-series storage, geospatial indexing, and read-heavy caching strategy in a single system.

Functional Requirements

Core Requirements

1. Ingest weather readings from sensor networks or third-party weather APIs at regular intervals.
2. Store current and historical observations per geographic location.
3. Serve current conditions and short-term forecasts by latitude/longitude.
4. Refresh the client UI with updated data at a configurable interval.

Below the Line (out of scope)

• Building numerical weather prediction (NWP) forecasting models.
• Severe weather push alerting.
• Historical data bulk export and analytics.
• User authentication and personalization.

Forecasting model training is out of scope because NWP algorithms (ensemble methods, physics simulations) require specialized ML infrastructure separate from the data serving layer. In practice, you call an external model API (NOAA, Tomorrow.io) and cache the returned forecast. The serving logic is a cache-aside proxy, not a modeling pipeline.

Severe weather alerting could sit beside the write path as a separate consumer: read each new Observation off the Kafka topic, evaluate it against geo-fenced alert cells, and fire a notification if a threshold is crossed. The alert evaluation design mirrors a price alert system. Deliberately deferred because it does not change the ingestion or read path we are designing.

Historical export belongs in a data warehouse (BigQuery, Redshift) fed by a separate Kafka consumer off the same ingestion topic. The export consumer writes to the warehouse; the serving path never touches it. Not a fundamental design change to the read or write path.

User authentication would add a user_id context to the read API and allow personalization such as saved locations. It layers on top of the existing API design without changing the storage or geo-resolution logic.

The hardest part in scope: Translating an arbitrary user coordinate into up-to-date conditions from nearby stations requires a geospatial index that stays fast as the station count grows, and a cache invalidation policy that knows when a reading is stale. Both problems sit in tension: a tighter cache TTL means fresher data but more database reads at scale.

Non-Functional Requirements

Core Requirements

• Scale (writes): 100K weather stations each ingesting one reading every 60 seconds = approximately 1,667 writes/sec sustained. Design for 3x burst headroom to handle thunderstorm events when stations report more frequently.
• Scale (reads): 10M DAU each polling every 5 minutes = approximately 33K reads/sec at even distribution. Expect 5x peak spikes = up to 165K reads/sec during morning weather checks.
• Read latency: Current conditions delivered in under 100ms p99 end-to-end.
• Data freshness: Current conditions stale by no more than 5 minutes. Older than 5 minutes, the app should indicate the last-updated timestamp rather than silently displaying stale data.
• Availability: 99.9% uptime for the read path (current conditions and forecasts). Brief ingestion lag during a partial outage is acceptable; dark screens are not.
• Durability: Historical observations retained for at least 2 years for trend analysis and display.

Below the Line

• Sub-second real-time streaming of sensor readings (pub/sub dashboard use cases).
• Per-sensor raw data export with millisecond timestamps (industrial IoT).

Read/write ratio: Roughly 33K reads vs 1,700 writes peak = approximately 20:1. The system is read-dominant but not as skewed as a URL shortener (1,000:1). The interesting design tension here is that the write path must be durable and throughput-consistent (sensors cannot block waiting for slow writes), while the read path must be fast (geo-query plus aggregation under 100ms). These two paths have conflicting needs that push toward a clean separation via a message queue.

Core Entities

• Station: A geographic measurement source (lat, lng, altitude, station_type). Can be a physical IoT sensor or a virtual aggregation point from a third-party provider. Changes rarely; the table is small and cache-friendly.
• Observation: A single time-stamped reading from one station. Captures temperature_c, humidity_pct, pressure_hpa, wind_speed_kph, wind_direction_deg, and precipitation_mm. The primary hot data; billions of rows accumulate over months.
• Forecast: A set of predicted conditions for a future time window at a location. Sourced from an external NWP API and cached locally. Read-only to this system; the weather service does not generate forecasts.
• WeatherSnapshot: A pre-computed "latest reading" record per station, held in Redis. Avoids live aggregation on every user query. Rebuilt from new Observations as they arrive.

The primary data flow is: raw Observations feed both the durable historical store (TimescaleDB) and the live snapshot cache (Redis). Every user read for current conditions hits the snapshot cache, not the historical store. Schema and indexing decisions are deferred to the deep dives.

API Design

One endpoint per functional requirement, grouped by the requirement it satisfies.

FR 1 (ingest observations):

POST /v1/observations
Content-Type: application/json
Body: {
  station_id: "KNYC",
  readings: [
    {
      observed_at: "2026-04-03T12:00:00Z",
      temperature_c: 18.5,
      humidity_pct: 72,
      pressure_hpa: 1013,
      wind_speed_kph: 14,
      wind_direction_deg: 270,
      precipitation_mm: 0
    }
  ]
}
Response 202: { ingested_count: 1 }

Batch array rather than single-record: sensors often buffer 5-10 readings locally during connectivity gaps and flush in a burst. A batch endpoint handles this without the client making one HTTP call per reading. The 202 (Accepted) response confirms the payload was received and queued; it does not guarantee durable storage, which is handled asynchronously by the Kafka consumer.

FR 2 and FR 4 (current conditions with configurable refresh):

GET /v1/weather/current?lat=40.7128&lng=-74.0060&radius_km=25
Response 200: {
  location: { lat: 40.7128, lng: -74.0060 },
  observed_at: "2026-04-03T12:00:00Z",
  temperature_c: 18.5,
  humidity_pct: 72,
  pressure_hpa: 1013,
  wind_speed_kph: 14,
  wind_direction_deg: 270,
  precipitation_mm: 0,
  nearest_station_id: "KNYC",
  nearest_station_distance_km: 2.4
}
Cache-Control: max-age=300

The Cache-Control: max-age=300 header drives the UI refresh interval without a separate parameter. CDN edges and browsers respect this header and serve cached responses for 5 minutes before re-requesting. The optional radius_km parameter (default 25 km) controls how wide a net to cast for nearby stations; clients in rural areas with sparse sensor coverage can increase this to 100 km without changing anything else in the pipeline.

FR 3 (short-term forecast):

GET /v1/weather/forecast?lat=40.7128&lng=-74.0060&hours=24
Response 200: {
  location: { lat: 40.7128, lng: -74.0060 },
  generated_at: "2026-04-03T12:00:00Z",
  hourly: [
    { hour: "2026-04-03T13:00:00Z", temperature_c: 19.0, precipitation_prob: 0.1, wind_speed_kph: 12 }
  ]
}
Cache-Control: max-age=1800

Forecast data from NWP providers updates at most every 30 minutes, so a max-age=1800 TTL is honest about freshness without over-fetching upstream. The generated_at field tells the client when the underlying model ran, independent of when the cached copy was served.

High-Level Design

1. Ingest weather data from sensor networks and third-party APIs

The write path receives batches of sensor readings from two source types: direct sensor adapters using MQTT or HTTP push, and scheduled pollers that call third-party APIs (NOAA, Tomorrow.io) every few minutes.

Components:

• Adapter Service: Thin normalizers that translate heterogeneous sensor protocols (MQTT, HTTP, CoAP) into a canonical Observation event. One adapter type per source protocol.
• Kafka (raw-observations topic): Decouples ingest acceptance from storage writes. Sensors get a fast 202 ACK; the write consumers work at their own pace. Also absorbs thunderstorm write bursts without backpressure reaching sensors.
• Ingestion Consumer: Reads from Kafka in 5-second batches and bulk-inserts into TimescaleDB. Batch writes convert 8,500 tiny row inserts into a small number of large COPY operations.
• Snapshot Updater: A second Kafka consumer on the same topic that upserts the latest reading per station_id into Redis. This keeps the fast read cache always-warm without touching the TSDB.
• TimescaleDB: The durable historical store for all Observations, auto-partitioned by time.
• Redis (station:{id} hash): Holds the latest Observation JSON for each active station. Sub-millisecond lookup; no TSDB round-trip needed for current conditions.

Request walkthrough:

1. A sensor flushes a batch of buffered readings to the Adapter Service via MQTT/HTTP.
2. The Adapter normalizes the payload to a canonical Observation schema and produces one Kafka message per reading to the raw-observations topic.
3. The Adapter returns HTTP 202 to the sensor immediately (queue accepted, not yet written).
4. The Ingestion Consumer reads a 5-second window of messages, batches them, and bulk-inserts into TimescaleDB.
5. The Snapshot Updater reads the same messages and UPSERTs each station's latest reading into Redis: HSET station:KNYC temperature_c 18.5 humidity_pct 72 ....
6. Both consumers commit their Kafka offsets after a successful write. If either fails, Kafka replays from the last committed offset.

The dual-consumer pattern is the key insight here. One Kafka topic feeds two jobs: a durability writer (TSDB) and a hot-cache writer (Redis). Reads never go near TimescaleDB for current conditions.

2. Store and query current conditions by location

Users query by latitude/longitude, not by station_id. The system must translate a coordinate into the nearest stations and return their aggregated latest readings.

Phase 1 (naive): Bounding-box SQL scan

The simplest approach is a two-column range filter on the stations table.

Components:

• Query Service: Handles GET /v1/weather/current. Runs a bounding-box WHERE clause.
• Stations DB (PostgreSQL): Stores station lat, lng as plain FLOAT columns with separate btree indexes.
• Redis: Returns latest Observation JSON for each matched station_id.

Components walkthrough (naive):

-- Bounding-box scan: no spatial index benefit for two-column ranges
SELECT station_id, lat, lng
FROM stations
WHERE lat BETWEEN :lat - 0.225 AND :lat + 0.225
  AND lng BETWEEN :lng - 0.225 AND :lng + 0.225;

What breaks:

PostgreSQL's planner uses a btree index on one of lat or lng only. The second column requires a row-by-row filter pass. At 100K stations this is acceptable. At 1M stations, a full index scan on one column returns hundreds of thousands of candidates; filtering the second column then happens in memory. The bounding box also over-fetches: the corners of the rectangle are 41% farther than the target radius, so the application must re-filter by true distance, adding code and latency.

Phase 2 (evolved): PostGIS with ST_DWithin and GIST index

PostGIS adds a geography type that stores coordinates on the WGS-84 ellipsoid and a GIST spatial index that partitions space into nested bounding rectangles. ST_DWithin uses the GIST tree to prune candidates to the relevant geographic cell before computing true geodesic distances. The result is an O(log N) spatial lookup rather than a linear scan.

I always recommend PostGIS over hand-rolled geohashing for production systems. It is standard, well-tested, and handles edge cases (antimeridian crossing, polar distortion) that geohash implementations silently get wrong.

Components (evolved):

• Query Service: Runs ST_DWithin + LIMIT 5 ORDER BY distance. Checks a coord-lookup Redis cache before hitting PostGIS.
• PostgreSQL + PostGIS (Stations DB): Station metadata with a GIST-indexed geography column. Typical ST_DWithin query time is under 5ms at 100K stations.
• Coord-lookup cache (Redis): Caches (lat_rounded, lng_rounded) → [station_ids] with a 1-hour TTL. Most users query from a small set of populated locations; this collapses repeat lookups into O(1) Redis reads.
• Redis (observations): Returns latest Observation JSON for each station_id via MGET.

Request walkthrough (evolved):

1. Client sends GET /v1/weather/current?lat=40.7128&lng=-74.0060.
2. Query Service rounds coordinates to 2 decimal places: 40.71, -74.01 (roughly 1 km precision).
3. Query Service checks Redis key coord:40.71:-74.01 for cached station IDs. Cache hit (most requests) returns the list immediately.
4. On cache miss, runs ST_DWithin against PostGIS and stores the result in Redis for 1 hour.
5. Query Service calls MGET station:KNYC station:KLGA ... to fetch the 5 nearest stations' latest readings.
6. Aggregates readings (average temperature, max wind speed, sum precipitation) and returns to the client.

3. Serve short-term forecasts by location

Forecasts come from an external NWP provider. The service does not build its own prediction models; it fetches, caches, and proxies external forecast data.

Components:

• Forecast Proxy Service: Cache-aside pattern. Checks Redis first; calls the upstream NWP API only on a miss.
• Forecast Cache (Redis): Stores forecast responses keyed by forecast:{lat_rounded}:{lng_rounded}:{hours} with a 30-minute TTL.
• External NWP API (NOAA / Tomorrow.io): Priced per request. Calls must be minimized. NWP model output refreshes roughly every hour; calling more often wastes budget without improving accuracy.

Request walkthrough:

1. Client sends GET /v1/weather/forecast?lat=40.7128&lng=-74.0060&hours=24.
2. Forecast Proxy rounds coordinates to 2 decimal places: 40.71, -74.01.
3. Proxy checks Redis key forecast:40.71:-74.01:24. Cache hit returns immediately.
4. On miss, Proxy calls the NWP API for the rounded coordinate.
5. Proxy stores the response under the cache key with TTL = 1,800 seconds.
6. Returns the forecast JSON to the client.

Rounding to 2 decimal places collapses all requests within roughly 1 km of the same point to one cache key. NWP providers already output at 1-5 km grid resolution, so rounding does not reduce accuracy in practice.

4. Refresh the client UI at a configurable interval

Two approaches exist: HTTP polling and server-sent events (SSE). The right choice depends on how real-time the experience needs to be.

Phase 1 (naive): client-side polling with no CDN caching

The simplest approach: the client timer fires every N seconds and calls GET /v1/weather/current. Every user in the same city sends one request per polling interval. At 10M DAU and 5-minute polling, that is 33K requests/sec directly hitting the Query Service.

What breaks at scale:

Most of those 33K requests per second fetch the same data for the same geographic cells. The Query Service does the same Redis lookups and aggregation work millions of times per minute for overlapping areas. It is wasted compute.

Phase 2 (evolved): polling with CDN caching

The Cache-Control: max-age=300 header returned on every /weather/current response tells CDN edges (and browsers) to serve the cached copy for 5 minutes before re-fetching. A city block with 10,000 users polling every 5 minutes collapses to a single origin request every 5 minutes per CDN edge node. The origin sees one request per cache region per TTL, not one per user.

Request walkthrough:

1. Client polls GET /v1/weather/current?lat=40.71&lng=-74.01 at t=0.
2. CDN edge has no cache entry for this coordinate: cache miss, forwards to Query Service.
3. Query Service fetches from Redis, aggregates, returns response with Cache-Control: max-age=300.
4. CDN caches the response. Returns to client.
5. Next 9,999 clients polling the same coordinate within 5 minutes hit the CDN cache. Zero origin requests.
6. At t=300s, CDN cache expires. Next client triggers a fresh origin fetch.

For applications that require sub-minute freshness (large-screen dashboards, weather station operator consoles), use server-sent events (SSE) instead. The client opens one persistent HTTP/2 connection; the server pushes a new event down the stream each time the Snapshot Updater writes a new reading for the stations near this location. SSE adds routing complexity (sticky sessions or a WebSocket proxy), so start with CDN-cached polling and add SSE only if users explicitly need it.

Potential Deep Dives

1. How do we store and query time-series weather observations efficiently?

Three constraints drive the design:

• 100K stations at 1 reading/min generates roughly 52 billion rows over 2 years.
• Most user queries want recent data: current conditions (last 10 minutes) or short trend (last 7 days). Full 2-year scans are rare.
• A 2-year retention policy requires efficient, non-blocking data deletion.

2. How do we resolve weather conditions for an arbitrary lat/lng coordinate?

Three constraints shape the choice:

• The station table is relatively small (100K to 1M rows) but queried on every single user request.
• Queries must return the nearest N stations to an arbitrary coordinate, ordered by distance.
• Results for the same geographic cell repeat across millions of users; aggressive caching is possible.

Final Architecture

The read path for current conditions never touches TimescaleDB during normal operation. Redis holds a warm snapshot of every active station's latest reading. PostGIS (behind a 1-hour coord-lookup cache) handles geo resolution in under 5ms. TimescaleDB stores the durable historical record and serves historical trend queries through pre-aggregated continuous aggregate views. The write path is fully decoupled from reads via Kafka: sensors get a fast 202 ACK while consumers write at their own pace in the background.

When I build this in production, I always put the coord-lookup Redis key as the first cache layer. It eliminates PostGIS for the roughly 80% of queries that repeat the same city-center coordinates, and the 1-hour TTL is safe because stations never move on their own accord.

Interview Cheat Sheet

• State the read/write ratio early: roughly 20:1 for weather (33K reads vs 1,700 writes at peak). Every caching decision downstream traces back to this number.
• The write path uses Kafka to decouple ingest from storage. Sensors get a fast 202 ACK; consumers write to TimescaleDB and Redis at their own pace. This absorbs thunderstorm write bursts without backpressure reaching sensors.
• Use dual Kafka consumers on one topic: an Ingestion Consumer writes durable records to TimescaleDB; a Snapshot Updater keeps Redis always-warm. Two jobs, one event stream, no extra infrastructure.
• Current conditions reads never hit TimescaleDB. Redis holds the latest observation per station (TTL 10 min); the Query Service fetches it with MGET in O(1).
• PostGIS ST_DWithin with a GIST index gives O(log N) spatial lookups. Typical query time under 5ms at 100K stations. No application-side distance re-filtering needed.
• Cache coord-to-station mappings in Redis with a 1-hour TTL. Stations don't move; 1 hour is safe and collapses millions of repeat user queries to a handful of PostGIS reads.
• TimescaleDB hypertable auto-partitions into 7-day chunks. Dropping an old chunk is instant DDL, not a slow DELETE+VACUUM. Retention policy is a one-line add_retention_policy call.
• Continuous aggregates (obs_hourly) pre-aggregate by hour, refreshed every 30 minutes. UI trend queries hit the 10,080-row hourly view instead of 604,800-row raw tables for the same 7-day window.
• Columnstore compression reduces float time-series data by roughly 90%: raw 10 TB becomes approximately 1 TB after 7 days of compression.
• CDN caching on Cache-Control: max-age=300 collapses all users in the same city block to one origin request per 5-minute window. The origin sees one read per geographic cache cell per TTL.
• Round forecast coordinates to 2 decimal places before the Redis cache key. NWP providers output at 1-5 km resolution; exact float coordinates generate spurious cache misses without any accuracy benefit.
• Geohashing is simpler to implement than PostGIS but requires 9-cell neighbor lookups and application-side distance re-filtering to handle cell boundaries correctly. PostGIS handles all of this natively in a single SQL query.