How S3 stores and retrieves objects

The Interview Question

Interviewer: "You mentioned your system stores millions of images in S3. Walk me through what actually happens when you call PUT object on a 500 MB video file, and then a user on the other side of the world calls GET on that same key five seconds later. How does S3 know where to find that object, and how does it guarantee 11 nines of durability without losing a single byte?"

This question reveals whether you understand distributed storage at a mechanical level. The interviewer wants to hear about partition mapping, erasure coding, the write-path durability guarantee, and the consistency model. Saying "S3 replicates across three AZs" earns partial credit. Explaining the partition index, erasure coding fragments, the witness protocol for consistency, and how multipart uploads handle large files is what separates depth from surface knowledge.

What to Clarify Before Answering

You: "Before I dive in, let me clarify the scope..."

• "Are we talking about S3 Standard, or should I cover storage classes like Glacier and Intelligent-Tiering?"
• "Should I walk through the full write path from SDK to durable storage, or focus on the durability mechanism?"
• "Is the interviewer interested in the consistency model? S3 moved to strong read-after-write consistency in December 2020."
• "Should I cover multipart uploads for large objects, or just the simple PUT path?"
• "Are we scoping to a single region, or should I include cross-region replication?"

Why this matters: S3 is a massive system with dozens of internal services. Asking the right scoping questions demonstrates you know the difference between the request routing layer, the storage layer, the metadata indexing layer, and the consistency subsystem. It also shows the interviewer you can structure a complex answer.

The 30-Second Answer

S3 is a distributed object store that maps every bucket + key pair to a partition in a massive distributed index. When you PUT an object, S3 splits it into erasure-coded fragments (data chunks plus parity chunks) and writes those fragments to storage nodes spread across at least three Availability Zones. This erasure coding scheme is what achieves 99.999999999% (11 nines) durability, because S3 can lose multiple fragments and reconstruct the original object. On reads, the partition index resolves the key to the storage locations of those fragments, and S3 assembles them from whichever fragments respond fastest. Since December 2020, all GET requests after a successful PUT return the latest version immediately, with no eventual consistency window.

The Architecture Overview

The architecture has four distinct layers. The frontend receives every HTTP request, validates the SigV4 signature, checks IAM policies, and resolves the bucket and key to a partition. The partition index maintains the mapping from key prefixes to partition IDs, and handles automatic partition splitting when throughput exceeds thresholds. The storage layer is where object data lives, erasure-coded across multiple Availability Zones. The witness system enables strong read-after-write consistency by tracking which writes have completed durably.

I find the partition index the most critical piece to understand. Before 2018, S3 customers had to randomize key prefixes to avoid hot partitions. Now, S3 automatically detects throughput pressure on a partition and splits it, redistributing the key space. This eliminated one of the most common S3 performance pitfalls.

The storage layer uses a concept called "storage cells," which are clusters of storage nodes within each AZ. Each cell manages a pool of physical disks (mostly spinning HDDs for S3 Standard, with SSDs for metadata and hot paths). When a write arrives, the storage layer distributes erasure-coded fragments across cells in different AZs, ensuring that an entire AZ failure cannot cause data loss.

Request Routing and Partition Mapping

Every S3 operation starts with a key lookup. The system must translate bucket + key into a set of physical storage locations. This is the mechanism that makes S3 fast at scale, and it is also the mechanism that was responsible for the infamous "sequential key" performance problems before 2018.

How Partition Mapping Works

S3 maintains a distributed index that maps key prefixes to partitions. When you create a bucket, S3 assigns it a set of partitions based on your region. Each partition handles a range of key prefixes.

// Simplified partition mapping
function resolvePartition(bucket, key):
    partitionIndex = getPartitionIndex(bucket)
    prefix = extractPrefix(key)   // Variable-length prefix
    partition = partitionIndex.lookup(prefix)

    if partition.throughput > SPLIT_THRESHOLD:
        // Automatic partition split (since 2018)
        newPartitions = partition.split()
        partitionIndex.update(newPartitions)

    return partition.storageLocations

When a single partition handles more than approximately 5,500 GET requests per second or 3,500 PUT requests per second, S3 detects the pressure and splits the partition. The split is transparent: the partition index is updated atomically, and in-flight requests to the old partition are redirected.

Partition Split Mechanics

When S3 detects a hot partition, the split process works as follows:

1. The partition index service identifies the key prefix boundary for the split
2. New partitions are created, each assigned to storage nodes in the same AZ configuration
3. The partition index atomically updates the routing table
4. Existing data does not move immediately because the storage locations are preserved
5. New writes go to the appropriate new partition based on the split boundary

The key insight is that partitions are logical, not physical. A partition is a routing concept that points to storage locations. Splitting a partition means updating the index, not moving terabytes of data.

Request Authentication: SigV4 Deep Dive

Every S3 request is authenticated using AWS Signature Version 4 (SigV4). This is worth understanding because it explains why S3 requests are slightly more complex than a simple REST API call, and it also explains some common debugging scenarios.

The SigV4 process works as follows:

1. The SDK builds a canonical request string from the HTTP method, path, query string, headers, and payload hash
2. It creates a "string to sign" that includes the timestamp, region, service name, and a hash of the canonical request
3. It derives a signing key from the secret access key using a chain of HMAC operations (date → region → service → signing)
4. It computes the HMAC-SHA256 signature of the string to sign using the derived key
5. It attaches the signature as the Authorization header or as query parameters (for presigned URLs)

// SigV4 signing process (simplified)
function signRequest(method, path, headers, payload, credentials, region):
    payloadHash = sha256(payload)
    canonicalRequest = method + "\n" + path + "\n" + queryString
                     + "\n" + canonicalHeaders + "\n" + signedHeaders
                     + "\n" + payloadHash

    dateStamp = formatDate(now(), "YYYYMMDD")
    scope = dateStamp + "/" + region + "/s3/aws4_request"
    stringToSign = "AWS4-HMAC-SHA256\n" + timestamp + "\n" + scope
                 + "\n" + sha256(canonicalRequest)

    signingKey = hmac(hmac(hmac(hmac(
        "AWS4" + credentials.secretKey, dateStamp),
        region), "s3"), "aws4_request")

    signature = hmacHex(signingKey, stringToSign)
    return signature

The frontend validates this signature on every request. If the timestamp is more than 15 minutes old, the request is rejected. If the signature does not match, the request is rejected with a 403 Forbidden. This is why clock skew on client machines is one of the most common S3 authentication failures.

Bucket Naming and DNS Resolution

S3 bucket names are globally unique and serve as DNS subdomains. When you create a bucket named my-app-data, S3 registers my-app-data.s3.amazonaws.com and my-app-data.s3.us-east-1.amazonaws.com in DNS.

The DNS resolution path determines which S3 endpoint handles the request. For path-style URLs (s3.amazonaws.com/my-bucket/key), the request hits the global endpoint and is routed to the correct region. For virtual-hosted-style URLs (my-bucket.s3.us-east-1.amazonaws.com/key), the request goes directly to the regional endpoint.

AWS deprecated path-style URLs for new buckets after September 2020. Virtual-hosted-style is now the default, which means bucket names must be DNS-compliant (lowercase, no underscores, 3-63 characters). This is why bucket names cannot contain periods if you want to use HTTPS (the SSL certificate does not match wildcard subdomains with periods).

Erasure Coding: How S3 Achieves 11 Nines

This is the core durability mechanism. S3 does not simply replicate objects three times across three AZs. That would give you roughly 6-8 nines of durability. Instead, S3 uses erasure coding, a mathematical technique that splits each object into data fragments and parity fragments, achieving far higher durability with less storage overhead than full replication.

How Erasure Coding Works

Erasure coding is conceptually similar to RAID but applied at the object level across distributed storage nodes. S3 uses a Reed-Solomon coding scheme. I will walk through the general approach.

Think of it like a book with a special appendix. Imagine you write a story in 6 chapters and then write 3 appendix chapters, where each appendix chapter is a mathematical combination of the original 6. If you lose any 3 chapters (including any mix of story and appendix chapters), you can reconstruct the missing ones from the remaining 6. That is erasure coding.

For a given object, S3:

1. Splits the object data into k data fragments
2. Computes m parity fragments using Reed-Solomon math
3. Distributes all k + m fragments across storage nodes in different AZs
4. Only needs any k out of the k + m fragments to reconstruct the original object

The exact values of k and m are not publicly documented, but based on AWS publications and the 11-nines durability guarantee, the scheme tolerates losing multiple fragments simultaneously. The Reed-Solomon algorithm works over a Galois field, where each parity fragment is a linear combination of all data fragments with different coefficients. This mathematical property guarantees that any k of the k + m fragments contain enough information to solve for the original data.

The Math Behind 11 Nines

The durability calculation works like this. If each individual storage device has an Annual Failure Rate (AFR) of, say, 2%, and you have k + m fragments where you can tolerate losing up to m fragments:

// Simplified durability calculation
P(object_loss) = C(k+m, m+1) * AFR^(m+1)

// With k=6 data fragments, m=3 parity fragments (hypothetical):
// Need to lose 4+ fragments simultaneously:
P(loss) = C(9, 4) * 0.02^4 = 126 * 1.6e-7 ≈ 2e-5 per year

// With more fragments and faster repair:
// P(loss) < 1e-11 per year = 99.999999999% durability

The actual calculation is more complex because S3 actively monitors fragment health and re-creates lost fragments before additional failures occur. This "repair time" is critical: the faster S3 detects a failed drive and re-encodes the missing fragment onto a healthy drive, the smaller the window of vulnerability.

Storage Overhead Comparison

Erasure coding achieves better durability with less storage overhead than simple replication. This is why S3 can offer 11 nines at a cost of $0.023/GB/month. Full 3x replication at the same durability level would require more storage and cost significantly more.

Storage Node Architecture

Understanding what happens at the storage node level is essential for grasping why S3 achieves its performance and durability characteristics. Each Availability Zone contains thousands of storage nodes, grouped into logical units called storage cells.

Inside a Storage Cell

A storage cell is a self-contained cluster of storage nodes that manages a pool of physical disks. Each cell has its own control plane for tracking disk health, managing capacity, and coordinating fragment placement. Cells are the unit of failure isolation within an AZ.

Each storage node in a cell typically manages:

• Dozens of spinning hard drives (HDDs) for bulk object data (8-16 TB each)
• A small set of SSDs for metadata indexing and hot-path caching
• Local memory for buffering writes and caching frequently accessed fragments
• A local journal for crash recovery

// Storage node disk layout (simplified)
StorageNode:
    ssds: [SSD_1, SSD_2]          // Fragment metadata, checksums
    hdds: [HDD_1, HDD_2, ..., HDD_24]  // Erasure-coded fragment data
    writeBuffer: 64 GB RAM        // Buffer incoming fragments
    journal: SSD-backed WAL       // Crash recovery

    function writeFragment(fragmentId, data, checksum):
        journal.append(fragmentId, data)    // WAL first
        hdd = selectDisk(fragmentId)        // Consistent placement
        hdd.write(data)                     // Write to spinning disk
        hdd.fsync()                         // Force to physical media
        ssd.writeMetadata(fragmentId, checksum, offset)
        journal.markComplete(fragmentId)
        return ACK

Disk Selection and Placement

When a fragment arrives at a storage node, the node must decide which physical disk to place it on. S3 uses a placement algorithm that balances capacity, I/O load, and failure domain isolation across disks. The algorithm avoids placing multiple fragments from the same object on the same disk, because a disk failure would then lose multiple fragments simultaneously.

Within each AZ, fragments for a single object are spread across different storage cells. This means that even an entire storage cell failure (dozens of nodes) only loses one or two fragments from any given object, well within the erasure coding tolerance.

Background Integrity Scanning

Storage nodes continuously scan their disks for bit-rot and corruption. This is a background process that reads every fragment, recomputes its checksum, and compares it against the stored checksum on the SSD. If a mismatch is found, the storage node marks the fragment as corrupt and notifies the repair subsystem.

The repair subsystem reads k healthy fragments from other locations, re-encodes the corrupted fragment using Reed-Solomon math, and writes the repaired fragment to a new disk location. This repair loop runs 24/7 and is the engine that sustains the 11-nines durability guarantee over time.

S3 also runs periodic scrub audits where the system verifies that every object has the expected number of healthy fragments. If any object has fewer than the full k + m fragments (e.g., because a disk failed and the repair has not yet completed), the scrub audit elevates the repair priority for that object.

The scrub cycle must complete faster than the expected rate of concurrent failures. If S3 has a Petabyte of data and disks fail at a rate of 2% per year, the scrub system must re-encode missing fragments faster than new failures occur. This is why S3's storage fleet has massive aggregate I/O bandwidth dedicated to background repair operations.

Hardware and Capacity Management

S3's storage fleet is one of the largest collections of spinning hard drives in the world. AWS custom-designs storage servers (code-named internally as "storage sleds") that pack maximum disk density into minimal rack space. Each storage sled contains 24-36 HDDs (currently 16-20 TB each) with a small amount of SSD and RAM for metadata and caching.

When new capacity is needed, AWS deploys additional storage sleds into existing data centers or expands to new data center buildings within each AZ. The partition index and storage cell controllers automatically discover new capacity and begin placing fragments on the new disks. This is how S3 has scaled from terabytes to exabytes without architecture changes.

The capacity planning system monitors per-AZ utilization and triggers hardware procurement months in advance. A fundamental constraint is that all three AZs in a region must have sufficient capacity, because erasure coding requires fragments in each AZ. If one AZ runs low on capacity, S3 cannot accept new writes even if the other two AZs have spare capacity.

The Write Path: From PUT to Durable

When your application calls PUT object, S3 executes a carefully orchestrated write path that ensures durability before returning a 200 OK response. I will walk through each step.

Step-by-Step Write Flow

The critical guarantee: S3 does not return 200 OK until fragments are durable on storage devices across multiple AZs. If any AZ fails to acknowledge the write, S3 retries to alternative storage nodes in that AZ. Only after sufficient fragments are persisted does the frontend return success to the client.

What "Durable on Disk" Means

Each storage node writes the fragment to its local storage device and calls fsync (or the equivalent hardware-level flush) before acknowledging. This means the data has hit the physical storage medium, not just the operating system page cache. The storage devices in S3's fleet are a mix of spinning hard drives (for bulk data) and SSDs (for metadata and hot-tier access).

The write path also generates a checksum (MD5 by default, or SHA-256 if you enable additional checksums). This checksum is stored alongside the fragment metadata and verified on every subsequent read. If a fragment fails its checksum on read, S3 transparently reads from a parity fragment and re-encodes the corrupted one.

Server-Side Encryption

S3 encrypts all objects at rest by default (as of January 2023, SSE-S3 is the default for all new objects). There are three encryption modes:

With SSE-S3, encryption happens at the storage node level after erasure coding. Each fragment is encrypted with a unique data key, and that data key is encrypted with a master key managed by S3. The entire process adds less than 1ms to the write path because modern storage hardware supports AES-256 acceleration.

With SSE-KMS, each object encryption/decryption requires an API call to AWS KMS. At high request rates this can become a bottleneck because KMS has per-region request quotas (typically 5,500-30,000 requests per second depending on key type). S3 mitigates this with bucket keys, which cache a short-lived data key derived from the KMS key. A single KMS call generates a bucket key that encrypts all objects for a period, reducing KMS calls by up to 99%.

Conditional Writes

S3 supports conditional writes using HTTP headers like If-None-Match and If-Match. These allow you to implement pessimistic or optimistic concurrency control for object updates.

For example, to prevent overwriting an existing object, send If-None-Match: * with your PUT request. S3 returns 412 Precondition Failed if the object already exists. To update only if you have the latest version, send If-Match: "etag-value" with the ETag from your most recent GET.

// Optimistic concurrency using ETags
function updateObject(bucket, key, newData):
    response = s3.getObject(bucket, key)
    currentEtag = response.etag

    // Modify data locally
    modifiedData = transform(newData)

    try:
        s3.putObject(bucket, key, modifiedData,
                     headers={"If-Match": currentEtag})
    catch PreconditionFailed:
        // Another writer updated the object, retry
        return updateObject(bucket, key, newData)

This pattern is essential for coordinating writes from multiple producers to the same key without a separate locking service.

Copy Operations and Server-Side Processing

S3 supports server-side COPY operations where data moves between keys (or buckets) without downloading to the client. The COPY operation reads source fragments from storage, optionally re-encodes them with a new encryption key or storage class, and writes the result to the destination key.

For objects up to 5 GB, the COPY is atomic and happens entirely within S3's network. For larger objects, you must use multipart copy, where each part is a range-COPY from the source object:

// Multipart copy for objects > 5 GB
function copyLargeObject(srcBucket, srcKey, dstBucket, dstKey, size):
    uploadId = s3.createMultipartUpload(dstBucket, dstKey)
    partSize = 500 * 1024 * 1024   // 500 MB
    parts = []

    for i in range(0, size, partSize):
        rangeEnd = min(i + partSize - 1, size - 1)
        part = s3.uploadPartCopy(
            dstBucket, dstKey, uploadId, partNumber=len(parts)+1,
            copySource=srcBucket+"/"+srcKey,
            copySourceRange="bytes=" + i + "-" + rangeEnd
        )
        parts.append(part)

    s3.completeMultipartUpload(dstBucket, dstKey, uploadId, parts)

Server-side copy is essential for storage class transitions, cross-region replication (which uses copy internally), and reorganizing data lake prefixes without egress charges. The data never leaves the AWS network, so there are no data transfer costs for same-region copies.

S3 Batch Operations

For operations that affect millions of objects (changing storage class, applying encryption, copying between accounts), S3 Batch Operations provides a managed job engine. You provide a manifest (either an S3 Inventory report or a CSV list of keys), specify the operation, and S3 processes all objects at scale.

Batch Operations can execute copy, invoke Lambda, replace tags, change ACLs, restore from Glacier, or apply Object Lock retention. Each job processes objects in parallel across S3's internal fleet, handling retries and publishing a completion report with per-object status.

I use Batch Operations primarily for two scenarios: retroactive encryption (encrypting all objects with a new KMS key after a compliance audit) and storage class migration (moving an entire prefix from Standard to Standard-IA after analyzing access patterns with S3 Storage Lens).

The Read Path: How GET Requests Are Served

The read path is where S3's architecture really shines. A GET request must locate the correct fragments, read enough of them to reconstruct the object, and return the data to the client, all within milliseconds for typical object sizes.

Read Flow

When a GET request arrives, the frontend resolves the key to a partition, looks up the fragment locations in the partition index, and dispatches parallel read requests to storage nodes. S3 does not need to read all fragments, only k out of k + m. It sends read requests to all fragment locations and returns data as soon as k fragments respond.

This "read from all, use first k" strategy is a well known technique in distributed systems called hedged requests. By sending requests to all fragment locations and using whichever respond first, S3 avoids being bottlenecked by the slowest storage node. If one node is under heavy disk I/O or experiencing a momentary slowdown, the other nodes compensate. This is why S3 GET latency is surprisingly consistent despite running on millions of spinning disks with inherent I/O variability.

// Simplified read path
function getObject(bucket, key, byteRange):
    partition = partitionIndex.lookup(bucket, key)
    fragmentLocations = partition.getFragmentLocations()

    // Check witness for in-flight writes (strong consistency)
    witness.waitForPendingWrites(bucket, key)

    // Read k fragments in parallel (only need k of k+m)
    // Send to ALL locations, use first k responses (hedged reads)
    fragments = parallelRead(fragmentLocations, needed=k)

    // Decode erasure coding
    objectData = reedSolomonDecode(fragments)

    if byteRange:
        return objectData[byteRange.start : byteRange.end]
    return objectData

The hedged read approach has an additional benefit for durability verification. If S3 receives a fragment that fails its checksum during a read, it discards that fragment and waits for the next healthy response. The read succeeds transparently as long as k healthy fragments respond. Meanwhile, the corrupt fragment is flagged for background repair.

Range Reads and Partial Downloads

S3 supports HTTP Range headers, which allow you to download only a portion of an object. This is not just a simple byte-offset operation at the HTTP layer. S3 must figure out which erasure-coded fragments contain the requested byte range, read only those fragments, and decode just the relevant portion.

For a 1 GB object stored as 10 erasure-coded fragments, a range read of bytes 500MB-600MB only needs to read from 1-2 fragments (the ones containing those byte ranges), not all 10. This makes range reads significantly more efficient for large objects.

Range reads are the foundation of several common patterns:

Video streaming: Video players use range requests to fetch segments of a video file on demand. The player requests bytes 0-1MB (header with metadata), then bytes corresponding to the current playback position. S3 serves each range efficiently without reading the full file.

Parallel download: The AWS SDK's download manager splits a large object into range requests and downloads them concurrently. A 10 GB file downloaded with 10 parallel range requests of 1 GB each saturates the network much better than a single sequential GET.

Data sampling: Data science pipelines often read just the first few MB of a Parquet file to inspect the schema and metadata, then read specific column chunks for the data they need. S3 range reads support this without downloading the entire file.

Log tail reading: For log files, you can read the last N bytes using a range request with a suffix specification (Range: bytes=-1000). This lets you implement tail -f style monitoring on S3 objects.

Read Performance

S3 returns the first byte of a standard GET request in 100-200 milliseconds for most objects. This latency comes from three components:

For frequently accessed objects, S3 caches fragment location metadata aggressively. The storage nodes themselves may have the relevant disk blocks in their page cache for hot objects. These optimizations reduce effective read latency for popular content.

Transfer Acceleration

S3 Transfer Acceleration uses CloudFront edge locations as entry points for uploads and downloads. Instead of sending data directly to the S3 regional endpoint, the SDK connects to the nearest edge location, which then routes data over AWS's optimized backbone network to the S3 region.

This matters for cross-continent transfers. An upload from Tokyo to us-east-1 might take 500ms per round trip over the public internet but only 100-200ms through the AWS backbone via a Tokyo edge location. For large files with multipart upload, this acceleration compounds across hundreds of parts.

S3 Express One Zone

For latency-sensitive workloads, AWS introduced S3 Express One Zone in 2023. This storage class uses a single AZ with SSD-backed storage instead of the standard multi-AZ HDD-based architecture. It provides single-digit millisecond latency for GET and PUT operations, roughly 10x faster than S3 Standard.

The tradeoff is clear: S3 Express One Zone sacrifices multi-AZ durability for speed. If the single AZ goes down, your data is unavailable (and potentially lost if the failure is permanent). I use this for ephemeral data like ML training checkpoints, build artifacts, and temporary processing outputs where speed matters more than durability.

Strong Consistency: The Witness Protocol

Before December 2020, S3 was eventually consistent for overwrite PUTs and DELETEs. You could PUT an object, immediately GET it, and receive the old version or a 404. This caused real bugs in data pipelines where a write-then-read pattern assumed the new data was visible.

In December 2020, AWS announced that S3 now provides strong read-after-write consistency for all operations at no additional cost. This was a massive engineering undertaking because the consistency mechanism had to work at S3 scale (trillions of objects, millions of requests per second) without degrading latency.

How the Witness System Works

The witness system is a distributed protocol that tracks in-flight writes. When a write begins, the system registers a "pending write" entry. When reads arrive for the same key, the system checks whether any pending writes exist and, if so, blocks the read until the write completes.

The witness system does not add a consensus protocol for every read. Instead, it uses a lightweight check that only blocks reads when there is an active, concurrent write to the same key. For keys with no in-flight writes (the vast majority), the check is essentially free.

Consistency Guarantees

What Changed Internally

The move to strong consistency required AWS to redesign the metadata path. Previously, S3 metadata updates propagated asynchronously through a cache layer, and reads could hit stale caches. The new architecture ensures that the partition index and witness system are always consulted before returning a response, even when caches are warm.

AWS was able to do this without a performance hit because the witness check is a fast, in-memory operation on the frontend fleet. The witness state for a key only exists during the short window when a write is in progress (typically less than 100ms). The rest of the time, the check returns "no pending writes" immediately.

LIST Consistency and the Challenges It Solved

The most surprising part of the December 2020 announcement was that LIST operations also became strongly consistent. Previously, a newly created object might not appear in LIST results for minutes. This was because LIST operations had to traverse the partition index, and the distributed nature of the index meant different partitions could be at different consistency points.

Making LIST consistent required S3 to ensure that the partition index was fully converged before returning LIST results. AWS solved this by introducing a mandatory consistency barrier in the LIST path that checks all partitions covering the requested prefix for pending index updates. If any partition has uncommitted index entries, the LIST waits for convergence.

This matters for data pipelines that do "list then process" patterns. Before strong LIST consistency, a pipeline that uploaded 1,000 files and then listed the prefix might only see 950 files. Race conditions like this were the source of many subtle data pipeline bugs. Since December 2020, the pipeline always sees all 1,000 files if the LIST happens after all PUTs complete.

Versioning and Lifecycle Management

S3 versioning is a critical feature for data protection, and understanding how it works internally explains some non-obvious behaviors around storage costs and delete operations.

How Versioning Works Internally

When you enable versioning on a bucket, every PUT operation creates a new version of the object instead of overwriting the existing one. Each version gets a unique version ID (a string like 3HL4kqtJvjVBuNhTMOF0R26TXOZ1mldz). The partition index stores all version IDs for a given key, ordered by timestamp.

// Version metadata for key "photo.jpg"
key: "photo.jpg"
versions:
  - versionId: "3HL4kq...", timestamp: 2024-01-15T10:00:00Z, size: 2MB, etag: "abc123"
  - versionId: "7PQ9zr...", timestamp: 2024-01-14T08:30:00Z, size: 1.5MB, etag: "def456"
  - versionId: "null",      timestamp: 2024-01-10T12:00:00Z, size: 1MB, etag: "ghi789"

The "null" version ID exists for objects that were uploaded before versioning was enabled. When you GET an object without specifying a version ID, S3 returns the latest version. When you GET with a specific version ID, S3 returns that exact version.

Delete Markers

When you DELETE a versioned object without specifying a version ID, S3 does not actually remove the data. Instead, it inserts a delete marker, which is a zero-byte object with a special metadata flag. The delete marker becomes the "current" version. Future GET requests without a version ID return 404, but the old versions still exist and are still billed for storage.

This is one of the most common S3 cost surprises. Teams enable versioning, delete objects thinking they are saving storage, and then discover their storage bill keeps growing because all old versions (and delete markers) are retained indefinitely.

To permanently delete a specific version, you must call DELETE with the explicit version ID. This bypasses the delete marker mechanism and actually removes that version's fragments from storage. To delete all versions of an object, you must list all version IDs and delete each one individually.

// Deleting all versions of an object
function permanentlyDelete(bucket, key):
    versions = s3.listObjectVersions(bucket, prefix=key)
    for version in versions.versions:
        s3.deleteObject(bucket, key, versionId=version.versionId)
    for marker in versions.deleteMarkers:
        s3.deleteObject(bucket, key, versionId=marker.versionId)

S3 Object Lock and WORM Compliance

For regulatory compliance (SEC Rule 17a-4, HIPAA, GDPR data retention), S3 Object Lock provides Write Once Read Many (WORM) protection. When Object Lock is enabled, no one, not even the root account, can delete or overwrite the object until the retention period expires.

Object Lock has two modes:

• Governance mode: Users with special IAM permissions (s3:BypassGovernanceRetention) can override the lock. Use this for operational flexibility.
• Compliance mode: No one can override the lock, including the root account. The object is immutable until the retention date passes. Use this for regulatory compliance.

Object Lock works at the version level. Each version can have its own retention period and legal hold flag. A legal hold is a separate mechanism that prevents deletion regardless of the retention period, useful when objects are under legal discovery.

// Setting Object Lock on upload
s3.putObject(bucket, key, data, {
    ObjectLockMode: "COMPLIANCE",
    ObjectLockRetainUntilDate: "2030-01-01T00:00:00Z"
})
// This object cannot be deleted or overwritten until 2030,
// even by the AWS account root user

Lifecycle Rules and Storage Transitions

S3 lifecycle rules automate the movement of objects between storage classes and the deletion of old versions. Internally, lifecycle rules are evaluated by an asynchronous background process that scans the partition index periodically.

How Storage Class Transitions Work Internally

When a lifecycle rule triggers a transition (e.g., moving an object from Standard to Glacier), S3 does not physically move the erasure-coded fragments. Instead, it changes the metadata associated with the object in the partition index to reflect the new storage class. The actual data migration to the target storage tier (which may use different physical hardware) happens asynchronously in the background.

For transitions to Glacier and Glacier Deep Archive, the fragments are eventually moved from the standard HDD/SSD storage tier to a separate, high-density tape or archive storage tier. This is why retrieval from Glacier takes minutes to hours: the system needs to locate the archive media, read the fragments, and re-stage them to the standard storage tier before the object can be served.

Multipart Uploads: How Large Files Work

Any object larger than 100 MB should use multipart upload. Any object larger than 5 GB must use multipart upload because the single PUT limit is 5 GB. Multipart upload allows individual parts up to 5 GB each, with a maximum of 10,000 parts, supporting objects up to 5 TB total.

How Multipart Upload Works Internally

A multipart upload is a three-phase process:

1. Initiate: The client calls CreateMultipartUpload, which returns an UploadId. S3 creates a placeholder in the partition index that tracks the upload state.
2. Upload parts: The client uploads each part with the UploadId and a part number (1-10,000). Each part is erasure-coded and stored independently. S3 returns an ETag for each part.
3. Complete: The client calls CompleteMultipartUpload with the list of part numbers and ETags. S3 stitches the parts together logically (not physically, the fragments stay in place) and creates the final object in the partition index.

The key design insight is that parts are stored independently and never need to be physically combined. The final object is a logical assembly of parts, with the partition index maintaining the mapping from byte ranges to part fragment locations. This is why a completed multipart upload object has a composite ETag (like d41d8cd98f00b204e9800998ecf8427e-7) where the -7 suffix indicates 7 parts.

Abort and Cleanup

If a multipart upload is never completed (the client crashes, the network drops, or the upload is abandoned), the uploaded parts remain in S3 and continue to incur storage charges. This is another common cost trap. S3 does not automatically clean up incomplete multipart uploads.

Always configure a lifecycle rule to abort incomplete multipart uploads after a defined period (typically 7 days):

{
  "Rules": [{
    "ID": "AbortIncompleteMultiparts",
    "Status": "Enabled",
    "AbortIncompleteMultipartUpload": {
      "DaysAfterInitiation": 7
    }
  }]
}

Retry and Resume Semantics

One of the most valuable properties of multipart upload is per-part retry. If part 5 out of 10 fails, the client retries only part 5, not the entire object. The AWS SDK handles this automatically with exponential backoff.

Parts can also be uploaded out of order. The client can upload parts 1, 5, 3, 8, 2 in any sequence, and the CompleteMultipartUpload call specifies the correct ordering. This enables sophisticated upload strategies where a client prioritizes certain byte ranges (e.g., video header data first for progressive streaming).

For resumable uploads across client restarts, the application needs to persist the UploadId and the list of completed part ETags. On restart, it calls ListParts to discover which parts have already been uploaded, then continues with the remaining parts. This is how tools like aws s3 cp --multipart-threshold and the S3 Transfer Acceleration SDK handle unreliable networks.

S3 Event Notifications and Integration Points

S3 is not just a passive object store. It integrates with the broader AWS ecosystem through event notifications that fire on object lifecycle events (create, delete, restore). Understanding this mechanism explains how modern data pipelines use S3 as the central data lake.

Event Delivery Guarantees

Event notifications are delivered at-least-once, which means your consumer must be idempotent. In rare cases (typically during high-throughput bursts), S3 may deliver the same event twice. The event payload includes the object key, version ID, event timestamp, and event type, giving your consumer enough information to deduplicate.

There is also a small but non-zero delay between the object operation completing and the event being delivered. For most operations, this delay is under 1 second, but under extreme load it can stretch to several seconds. Do not design time-critical workflows that assume sub-second event delivery.

Event Notification Architecture

When an object is created or deleted, S3 publishes an event to one or more configured destinations:

• Amazon SQS: For queue-based processing with at-least-once delivery
• Amazon SNS: For fan-out to multiple subscribers
• AWS Lambda: For serverless, event-driven processing
• Amazon EventBridge: For complex filtering and routing rules

Events are emitted asynchronously after the object is durable. There is a small delay (typically 1-5 seconds) between the PUT response and the event delivery. For most use cases this is fine, but if you need guaranteed ordered processing, use EventBridge with SQS FIFO queues.

Event Notification Filtering

S3 supports prefix and suffix filtering on event notifications. You can configure different notification targets for different key patterns:

• s3:ObjectCreated:* on prefix raw/images/ → Lambda function for thumbnail generation
• s3:ObjectCreated:* on prefix raw/videos/ and suffix .mp4 → SQS queue for transcoding pipeline
• s3:ObjectRemoved:* → SNS topic for audit logging

This filtering happens at the S3 level before events are dispatched, so your Lambda functions and SQS consumers only receive relevant events. Without filtering, a high-volume bucket would flood downstream consumers with events they do not care about.

EventBridge provides even finer-grained filtering based on object metadata (size, tags, storage class) and supports content-based routing to different Step Functions state machines. For complex data pipelines with branching logic, EventBridge is the right choice over direct SQS/SNS notifications.

S3 Access Control Model

S3 has four layers of access control that are evaluated in order:

1. Block Public Access: Account-level or bucket-level toggle. When enabled, overrides all other policies to prevent public access.
2. Bucket Policy: JSON IAM policy document attached to the bucket. Controls who can perform which operations on which keys.
3. IAM User/Role Policy: Permissions attached to the AWS principal making the request.
4. ACLs (Legacy): Per-object access control lists. AWS recommends disabling ACLs for new buckets.

The evaluation order matters: Block Public Access is checked first and can veto everything else. Then the effective permission is the intersection of the bucket policy and the IAM policy, with explicit denies always winning.

For presigned URLs (commonly used for direct browser uploads), the URL carries a temporary signature that grants specific permissions to a specific key for a limited duration. The backend generates the URL using its IAM credentials, and the frontend uses it to PUT or GET directly to/from S3 without needing AWS credentials.

This is the standard pattern for user-uploaded content: the client requests a presigned URL from your API, uploads directly to S3 (bypassing your server entirely), and S3 event notifications trigger backend processing. This avoids proxying large file uploads through your application servers, which would consume bandwidth and add latency.

// Presigned URL generation (simplified)
function generatePresignedPut(bucket, key, expiry):
    credential = getIAMCredential()
    timestamp = now()
    stringToSign = "PUT\n" + bucket + "/" + key + "\n" + timestamp
    signature = hmacSHA256(credential.secretKey, stringToSign)
    return "https://" + bucket + ".s3.amazonaws.com/" + key
         + "?X-Amz-Algorithm=AWS4-HMAC-SHA256"
         + "&X-Amz-Credential=" + credential.accessKey
         + "&X-Amz-Date=" + timestamp
         + "&X-Amz-Expires=" + expiry
         + "&X-Amz-Signature=" + signature

Presigned URLs have a maximum expiration of 7 days for IAM users and 36 hours for STS temporary credentials. I recommend short expirations (15-60 minutes) for upload URLs and longer expirations (1-6 hours) for download URLs. If users need persistent access, use CloudFront signed URLs which support custom policies and key pair rotation.

What Happens When Things Break

S3 is designed for failure. Disks fail, nodes crash, entire AZs go offline, and S3 handles all of it transparently. Here are the key failure scenarios and how S3 responds.

Failure Cascade: Disk Failure to Repair

This diagram shows the repair flow when S3 detects a failed disk.

How S3 Detects Corruption Before You Do

S3 does not wait for a read request to discover corrupted data. The integrity scanning system proactively reads every stored fragment and verifies its checksum against the metadata service's record. When a mismatch is detected, the repair process starts immediately, well before any client tries to read that object.

Key facts about the 2017 outage:

• The root cause was a human error in a playbook command that removed too many servers
• The index subsystem tried to restart but was too large to come back online quickly
• AWS had to manually re-provision the index servers, which took several hours
• The S3 health dashboard itself was hosted on S3, so it could not update to show the outage

This proactive scanning is why S3's durability guarantee is a sustained property, not just a point-in-time property. Left unattended, storage media degrades over time (bit-rot). S3's scanning ensures that degradation is caught and repaired continuously, maintaining the full erasure coding redundancy for every object at all times.

The February 2017 S3 Outage

The most famous S3 outage occurred on February 28, 2017, when a routine maintenance operation accidentally removed a larger set of servers than intended from the S3 index subsystem in us-east-1. This took down the partition index, which meant S3 could not resolve any bucket+key lookups. The entire us-east-1 region was affected for about 4 hours.

This outage taught an important lesson: S3's durability and S3's availability are separate guarantees. No data was lost during the outage (11-nines durability held), but the service was unavailable (99.99% availability SLA was broken). The objects were safe on storage nodes. The system just could not find them because the index was down.

The post-mortem revealed a second lesson: the index subsystem was so large that restarting it took hours because it needed to rebuild its in-memory state from durable storage. AWS subsequently redesigned the index subsystem to support faster cold starts and added additional safeguards around maintenance operations.

Cross-Region Replication for DR

For workloads that require availability even during a full region outage, S3 Cross-Region Replication (CRR) asynchronously copies objects to a bucket in a different region. CRR replicates new objects within 15 minutes for most object sizes (with S3 Replication Time Control guaranteeing 99.99% of objects within 15 minutes).

The replication is object-level, not fragment-level. S3 reads the complete object from the source region, transmits it over the AWS backbone, and writes it to the destination region as a new object with its own erasure-coded fragments. This means the destination has full, independent durability.

Replication Metrics and Monitoring

S3 Replication Metrics (available for S3 Replication Time Control) provide real-time visibility into replication status:

• Replication latency: The time from object creation in the source bucket to replication completion in the destination. Typically 5-15 minutes, guaranteed within 15 minutes for 99.99% of objects with RTC.
• Pending replication count: Objects awaiting replication. A growing backlog indicates the replication system is falling behind (usually due to a burst of writes).
• Failed replication count: Objects that failed to replicate (usually due to IAM permission issues or destination bucket policy conflicts).

For disaster recovery planning, I monitor the ReplicationLatency metric in CloudWatch and alert if it exceeds 10 minutes. This gives the operations team advance warning before the 15-minute SLA is breached.

CRR also supports bi-directional replication between two buckets in different regions. This is useful for active-active architectures where both regions need to read and write. However, CRR does not handle conflict resolution. If both regions write to the same key simultaneously, the later replicated write overwrites the earlier one in each region's bucket. Applications using bi-directional CRR must implement their own conflict resolution (e.g., by using unique keys per region or version-aware writes).

Performance Characteristics

Performance Optimization Strategies

Connection reuse: The AWS SDK maintains a connection pool to S3 endpoints. Creating a new HTTPS connection requires a TCP handshake (1 RTT) plus TLS handshake (2 RTTs). Reusing connections saves 50-150ms per request. Always reuse the S3 client object across requests.

Transfer acceleration: For cross-region uploads, S3 Transfer Acceleration routes data through the nearest CloudFront edge location and across the AWS backbone, reducing latency by 2-5x for intercontinental transfers.

Byte-range fetches: For large objects where you only need a portion, use HTTP Range headers. S3 only reads and returns the requested byte range, reducing both network transfer and storage node I/O.

Multipart download: The AWS SDK can download large objects using parallel range-GET requests, assembling the pieces locally. This saturates the network link much better than a single GET because multiple TCP connections run in parallel.

S3 Express One Zone: For latency-critical workloads like ML inference pipelines, S3 Express One Zone provides single-digit millisecond reads from SSD-backed storage in a single AZ. The tradeoff is reduced durability (single-AZ only).

How This Compares to Alternatives

When to Choose What

I reach for S3 when the workload lives primarily in AWS, when I need the deepest integration with other AWS services (Lambda triggers, Athena, EMR), and when the 8 storage classes matter for cost optimization.

I use Google Cloud Storage when the workload is GCP-native because its strong consistency was available years before S3's and BigQuery external table integration is seamless for analytics workloads.

I reach for MinIO when I need an S3-compatible store that runs on-premises, in air-gapped environments, or on Kubernetes clusters where cloud dependency is unacceptable. MinIO's S3 API compatibility means application code does not change when migrating between MinIO and S3.

Azure Blob Storage is the right choice when the rest of the stack is Azure-native. Its tiered pricing is slightly cheaper than S3 for large storage volumes, and Azure CDN integration is straightforward.

S3 vs. Block Storage vs. File Storage

A common interview question is when to use S3 (object storage) versus EBS (block storage) versus EFS (file storage).

I use S3 for anything that can be accessed via HTTP: static websites, application assets, data lake storage, backups, and log archives. I use EBS for database volumes and application state that requires POSIX file system semantics. I use EFS only when multiple EC2 instances need to share the same filesystem, and the workload cannot be restructured to use S3.

S3 as a Data Lake Foundation

S3 has become the de facto data lake storage layer because it decouples storage from compute. Unlike traditional databases where storage and compute are tightly coupled, S3 lets you store data once and query it with multiple engines: Athena (serverless SQL), EMR (Spark, Hive), Redshift Spectrum (data warehouse), and third-party tools like Databricks and Snowflake.

The key architectural principle is "store in open formats, query with any engine." This means:

• Store structured data as Parquet or ORC (columnar formats with built-in compression and predicate pushdown)
• Store semi-structured data as JSON Lines or Avro
• Partition data by common query dimensions (date, region, customer) using Hive-style prefixes like data/year=2024/month=01/day=15/

// Data lake partition structure in S3
s3://my-data-lake/
  raw/                          // Raw ingestion (JSON lines)
    events/year=2024/month=01/
    events/year=2024/month=02/
  processed/                    // Cleaned, transformed (Parquet)
    events/year=2024/month=01/
  aggregated/                   // Pre-computed rollups (Parquet)
    daily_metrics/date=2024-01-15/

When Athena queries SELECT * FROM events WHERE year=2024 AND month=01, it reads the Hive-style partitioning metadata and only scans objects under events/year=2024/month=01/. S3 Select pushes column filtering and row predicates down to the storage layer, so only matching data is transferred. This combination of partition pruning and predicate pushdown makes S3-based data lakes surprisingly efficient.

Interview Cheat Sheet

• When asked about durability: "S3 uses erasure coding, not simple replication. Objects are split into data and parity fragments distributed across at least 3 AZs. S3 can reconstruct the object even if multiple fragments are lost, achieving 99.999999999% durability."
• When asked about the write path: "S3 does not return 200 OK until erasure-coded fragments are durable on physical storage devices across multiple AZs. The frontend generates checksums and the storage nodes call fsync before acknowledging."
• When asked about consistency: "Since December 2020, S3 provides strong read-after-write consistency for all operations, including LIST. A witness system tracks in-flight writes so that any concurrent read blocks until the write is durable."
• When asked about partition performance: "S3 supports 5,500 GET/s and 3,500 PUT/s per prefix. Since 2018, partitions automatically split under load, so sequential key names are no longer a problem for most workloads."
• When asked about large file uploads: "Always use multipart upload for objects over 100 MB. Optimal part size is 64-128 MB. Parts are uploaded in parallel, each independently erasure-coded, and individual parts can be retried without restarting the entire upload."
• When asked about storage classes: "S3 has 8 storage classes ranging from Standard ($0.023/GB) to Glacier Deep Archive ($0.00099/GB). Use Intelligent-Tiering for unpredictable access patterns, it moves objects between tiers automatically."
• When asked about versioning: "Enabling versioning means DELETEs insert a delete marker instead of removing data. Old versions persist and incur storage costs. Always pair versioning with lifecycle rules to expire non-current versions."
• When asked about the 2017 outage: "The us-east-1 outage was caused by removing too many servers from the partition index subsystem during maintenance. No data was lost, but the service was unavailable for 4 hours because the index could not route requests."
• When asked about encryption: "S3 encrypts all objects at rest by default with AES-256 (SSE-S3). For compliance workloads, use SSE-KMS with bucket keys to avoid KMS API throttling at high request rates."
• When asked about event-driven pipelines: "S3 event notifications fire on object create/delete events and can trigger Lambda, SQS, SNS, or EventBridge. Events are at-least-once, so consumers must be idempotent."
• When asked about S3 vs EBS vs EFS: "S3 for HTTP-accessible objects at $0.023/GB, EBS for POSIX block storage attached to EC2 at $0.08/GB, EFS for shared NFS mounts across instances at $0.30/GB. Choose based on access pattern and latency needs."

Test Your Understanding

Quick Recap

1. S3 maps every bucket + key to a partition in a distributed index, and since 2018, partitions automatically split under load to eliminate the old sequential-key bottleneck.
2. Objects are stored using erasure coding (data + parity fragments across 3+ AZs), not simple replication, which achieves 11 nines of durability with roughly 50% storage overhead.
3. The write path does not return 200 OK until erasure-coded fragments are durable on physical storage across multiple AZs, with checksum verification on every write.
4. Strong read-after-write consistency (since December 2020) is implemented via a witness protocol that tracks in-flight writes and blocks concurrent reads only when a write is actively in progress.
5. Multipart upload is mandatory for objects over 5 GB and recommended for anything over 100 MB, with optimal part sizes of 64-128 MB for parallel upload and per-part retry capability.
6. Versioning inserts delete markers instead of removing data, and non-current versions accumulate cost silently unless lifecycle rules expire them.
7. S3 supports 5,500 GET/s and 3,500 PUT/s per prefix, scaling effectively to unlimited aggregate throughput when keys are distributed across prefixes.
8. The 2017 us-east-1 outage demonstrated that S3 durability (no data lost) and S3 availability (service down for 4 hours) are separate guarantees backed by different mechanisms.
9. Storage nodes continuously scan for bit-rot and re-encode corrupted fragments from parity, sustaining the 11-nines guarantee through active repair rather than static redundancy.
10. S3 event notifications enable event-driven architectures where object lifecycle events trigger Lambda, SQS, SNS, or EventBridge for downstream processing.
11. Cross-Region Replication is asynchronous and uses the same write-path mechanisms as normal PUTs in the destination region, meaning replicated objects get full erasure coding and durability guarantees.
12. S3 Transfer Acceleration uses CloudFront edge locations for the initial hop and then travels AWS's backbone network, cutting cross-continent upload latency by 50-500% depending on the client's distance from the destination region.

Related Concepts

• How DynamoDB works internally: DynamoDB shares S3's partition-based routing architecture but optimizes for single-digit millisecond reads on structured data rather than large object storage.
• How Cassandra works internally: Cassandra uses a similar partition-key routing model with consistent hashing, and comparing its replication factor approach to S3's erasure coding illustrates different durability strategies.
• How CDNs cache content: CloudFront's interaction with S3 as an origin demonstrates how edge caching complements object storage for read-heavy workloads.
• How Kafka works internally: Kafka's segment-based storage and replication model provides an interesting contrast to S3's erasure coding approach for append-only data.
• How HDFS works internally: HDFS uses 3x block replication (the approach S3 chose not to use), providing a direct comparison for understanding why erasure coding is more storage-efficient. HDFS 3.0 added erasure coding support as an option, validating the approach S3 pioneered.
• How Lambda works internally: Lambda's tight integration with S3 (event triggers, temporary storage, deployment package storage) makes understanding both systems essential for serverless architecture design. Lambda functions are themselves stored as objects in an internal S3 bucket managed by AWS.