How file upload works at scale

The Problem Statement

Interviewer: "You are building a platform where users upload files, anything from profile photos to multi-gigabyte video files. Walk me through how you would design the file upload system at scale. Why not just POST the file to your API server?"

This question tests three things: your understanding of why proxying binary data through application servers is architecturally wrong at scale, your knowledge of presigned URL patterns for direct-to-storage uploads, and whether you can reason about the full lifecycle of a file from the user's browser to a CDN edge node serving it back to other users.

I like this question because it starts deceptively simple. "Just upload the file to the server and save it." Every junior engineer has done this. The depth comes when you realize that approach falls apart at 100 concurrent uploads, let alone 100,000. Memory exhaustion, connection timeouts, wasted CPU cycles on byte-shuffling, and the inability to resume a failed 2GB upload halfway through. The interviewer wants to see you recognize those failure modes and design around them systematically.

The same patterns show up in Dropbox, Google Drive, YouTube's upload pipeline, Slack file sharing, and every cloud storage product. Master this once and you can apply it everywhere.

The question also naturally extends into security territory: how do you prevent users from uploading malware, how do you validate file types at the byte level, and how do you enforce per-user quotas without race conditions? These are the details that separate a mid-level answer from a senior one.

Every file upload system, whether it handles vacation photos or enterprise documents, ultimately solves the same five problems: getting bytes off the client without proxying, handling partial failures gracefully, scanning for threats, transforming into the right format, and distributing to consumers fast.

Clarifying the Scenario

You: "Before I start designing, a few clarifying questions."

You: "What types and sizes of files are we talking about? Small images under 5MB, or also large videos in the multi-gigabyte range?"

Interviewer: "Both. Assume a mix: most uploads are images under 10MB, but some users upload videos up to 5GB."

You: "Do we need to process files after upload? Thumbnails, virus scanning, transcoding?"

Interviewer: "Yes, all of those. Walk me through the full pipeline."

You: "Should I focus on the upload path first, then cover post-upload processing and serving?"

Interviewer: "That is a good structure. Go ahead."

You: "One more: are we targeting browser uploads, mobile app uploads, or both?"

Interviewer: "Both. But you can focus on the browser case and mention mobile differences where relevant."

You: "Great. I will structure my answer in four parts: why the naive approach fails, the presigned URL pattern for direct-to-storage upload, multipart chunked uploads for large files, and the async post-upload processing pipeline."

Announcing your structure upfront is important. It tells the interviewer you have a plan, and it lets them redirect you if they want to spend more time on a specific area.

My Approach

I break this into five areas:

1. Why proxying through the API server fails: Memory limits, connection timeouts, and wasted compute
2. Presigned URLs for direct upload: Client uploads straight to object storage, bypassing the app server entirely
3. Multipart chunked upload: Splitting large files into 5MB chunks with parallel upload and per-chunk retry
4. Post-upload processing: Virus scanning, thumbnail generation, transcoding as an async event-driven pipeline
5. CDN distribution: Serving processed files from edge nodes closest to the consumer

The core insight is separation of concerns for the data plane versus the control plane. Your API server handles the control plane: authentication, authorization, generating upload URLs, tracking upload status, triggering processing. It never touches the actual file bytes. Object storage handles the data plane: receiving bytes, storing them durably, serving them back. Once you internalize that split, every design choice follows.

Think of it like a valet parking service. The valet (API server) gives you a ticket (presigned URL) and tells you which parking spot to use. You drive your car (file bytes) directly to the spot. The valet never carries your car on their back.

Numbers at a glance

The Architecture

Here is the full file upload pipeline from the user's browser to CDN-served content.

The critical path for the upload itself is steps 1 through 4. The user waits only for presigned URL generation (typically under 50ms) and the actual upload to S3 (bounded by their bandwidth). Steps 5 through 10 happen asynchronously after the upload completes. The user sees a "processing" status until those finish, then the file is available.

I want to emphasize step 4. The browser uploads directly to S3 using the presigned URL. The API server is completely out of the data path. It generated the URL and stepped aside. This is the single most important architectural decision in the entire system.

Deep Dive 1: Presigned URLs and Direct-to-Storage Upload

The presigned URL pattern is the foundation of every modern file upload system. Here is how it works step by step.

The presigned URL contains embedded authentication. When the API server generates it, it signs the URL with its AWS credentials, the target bucket and key, an expiration time, and optional constraints like maximum file size and required content type. The client uses this URL as a regular HTTP PUT endpoint. S3 validates the signature and accepts or rejects the upload without the API server being involved.

For my interview answer, I would say: "The presigned URL is like a short-lived parking pass. It says who issued it, where you can park (bucket + key), when it expires, and how big your car can be. Once issued, the valet is not involved in the actual parking."

Security considerations

The presigned URL constrains what the client can do. I always set these parameters:

• Expiration: 5-15 minutes. Long enough for slow connections, short enough that a leaked URL has limited damage window.
• Content-Type restriction: If the user said they are uploading a JPEG, the presigned URL only accepts image/jpeg. This prevents someone from using an image upload URL to store a malicious executable.
• Size limit: The presigned URL enforces a maximum content-length. A profile photo upload URL caps at 10MB. A video upload URL caps at 5GB.
• Bucket and key: The URL targets a specific key in the raw uploads bucket. The client cannot write to any other location.

Deep Dive 2: Multipart Chunked Upload and Resumability

Small files work great with a single PUT request. But a 2GB video over a flaky mobile connection? That single PUT will fail 80% of the time. This is where multipart upload becomes essential.

The flow works like this: the client requests a multipart upload initiation from the API server. The server calls S3's CreateMultipartUpload API and returns the upload ID plus a set of presigned URLs, one per chunk. The client splits the file into 5MB chunks (the S3 minimum), uploads each chunk in parallel (typically 4-8 concurrent), and tracks which chunks succeeded.

The magic of multipart upload is that each chunk is an independent HTTP request. If chunk 37 fails, you retry only chunk 37. The other 399 chunks that succeeded are safe on S3. This changes the failure model from "all-or-nothing" to "retry the bad parts."

Resumability

Resumability is what separates a good upload system from a great one. When a user's connection drops after uploading 80% of a 2GB file, they should not start over.

The client tracks uploaded chunks locally (in IndexedDB for browsers, or SQLite for mobile). When the connection recovers, it calls the API to get the list of already-uploaded parts (S3 provides this via ListParts), compares with the local tracker, and resumes from the first missing chunk.

The tus protocol formalizes this pattern. It defines a standard HTTP API for resumable uploads with checkpointing, and libraries exist for every platform. I would mention tus in an interview as a known protocol, then explain the underlying mechanism.

Deep Dive 3: Post-Upload Processing Pipeline

Once the raw file lands in S3, the real work begins. Virus scanning, image optimization, video transcoding, metadata extraction, and deduplication all happen in an asynchronous event-driven pipeline.

The pipeline is event-driven and decoupled. Each stage communicates through queues, which gives us independent scaling, retry with backoff, and dead-letter queues for failures. The virus scanner can scale independently of the media processor. A spike in image uploads does not affect video transcoding throughput.

Virus scanning

Every uploaded file must be scanned before it is served to other users. I cannot stress this enough. Skipping virus scanning because "we only accept images" is a security gap. Image files can contain embedded malware, and file extension checks are trivially bypassed.

The scanner runs ClamAV (or a commercial alternative like Sophos) in a Lambda function or container. It downloads the file from S3, scans it in memory, and makes a pass/fail decision. Clean files are moved to the processed bucket. Infected files go to a quarantine bucket with a security alert.

For the interview: "Every file is guilty until proven innocent. The raw upload bucket is a holding cell. Files only graduate to the processed bucket after passing the virus scan."

Content-addressable deduplication

If two users upload the same file, I do not want to store it twice. Before initiating an upload, the client computes a SHA-256 hash of the file and sends it to the API. The API checks if that hash already exists in the metadata database. If it does, the upload is skipped entirely, and the existing file is linked to the new user's record.

This saves storage and processing time. Dropbox and Google Drive both use content-addressable storage for deduplication. The tradeoff is that computing a SHA-256 hash of a 2GB file takes a few seconds on the client, but this is much cheaper than uploading 2GB of redundant data.

File type validation: magic bytes, not extensions

Never trust the file extension. A user can rename malware.exe to cute_photo.jpg. Always validate the actual file content by reading the first few bytes (the "magic bytes" or file signature).

The virus scanner validates magic bytes as part of its scan. But I also check on the API server when generating the presigned URL: the client declares the content type, and the presigned URL constrains it. After upload, the processing pipeline verifies the actual content matches the declared type.

The Tricky Parts

1. Upload progress for presigned URLs: Since the upload goes directly to S3, your API server has no visibility into progress. The client tracks progress using XMLHttpRequest.upload.onprogress events (browser) or URLSession delegate methods (iOS). For multipart uploads, progress is calculated as (completedChunks / totalChunks), which gives a slightly stepped progress bar instead of a smooth one.

2. Rate limiting and quota enforcement: You cannot enforce rate limits at the S3 level (presigned URLs bypass your infrastructure). Instead, enforce them at the presigned URL generation step. If a user has requested 10 upload URLs in the last minute, delay or reject additional requests. Quota checks also happen at URL generation time, but race conditions exist: two concurrent requests might both pass the quota check but together exceed it. I would use optimistic locking on the quota record to handle this.

3. Partial multipart uploads that never complete: A user starts a multipart upload, uploads 8 of 400 chunks, then closes their laptop. Those 8 chunks sit on S3 indefinitely, consuming storage. S3 Lifecycle policies can automatically abort incomplete multipart uploads after a configured duration (typically 7 days). This is a cleanup mechanism that most people forget to configure.

4. Cross-region uploads: If your S3 bucket is in us-east-1 and a user in Tokyo is uploading, latency is high. S3 Transfer Acceleration routes the upload through the nearest CloudFront edge location, which then uses AWS's backbone network to reach the bucket. This can improve upload speed by 50-500% for distant users. The presigned URL just needs the accelerated endpoint.

5. Concurrent uploads from the same user: A user drags 20 files into the upload zone. The client should limit concurrent uploads to 3-4 files to avoid saturating the user's bandwidth. Within each file, chunk uploads are already parallelized, so total concurrent HTTP requests could be 4 files x 6 chunks = 24, which is manageable.

6. CORS configuration for direct upload: Since the browser uploads directly to S3 (a different domain), you need CORS headers on the S3 bucket. This is easy to forget and produces cryptic errors in the browser console. The bucket needs Access-Control-Allow-Origin for your domain, Access-Control-Allow-Methods: PUT, and Access-Control-Expose-Headers: ETag so the client can read the ETag from the response.

7. Idempotent upload completion: The client calls POST /uploads/{id}/complete after the upload finishes. What if the client calls it twice (retry after timeout)? The completion endpoint must be idempotent. If the upload is already marked "complete," return success without re-triggering the processing pipeline. Use the upload ID and a status check to make this safe.

What Most People Get Wrong

How I Would Communicate This in an Interview

Here is how I would actually say this:

"I would never proxy file uploads through the application server. That turns your API tier into a dumb byte-shuffler that burns memory and connections doing zero useful work.

Instead, I use the presigned URL pattern. The client asks the API server for a presigned PUT URL. The API server checks authentication, validates the file type and size against the user's quota, creates a metadata record in the database, and returns a time-limited, constrained presigned URL. The client then uploads directly to S3.

For large files, I use S3's native multipart upload. The client splits the file into 5MB chunks and uploads them in parallel, typically 4-6 concurrent. Each chunk has its own presigned URL. If a chunk fails, only that chunk is retried. Combined with client-side tracking of completed chunks, this gives us full resumability.

After the upload lands in S3, an event notification triggers an async processing pipeline. First, a virus scanner running ClamAV checks the file. Clean files are moved to a processed bucket. Then, depending on the file type, we generate thumbnails, strip EXIF GPS data for privacy, or transcode video into multiple resolutions. Each stage is a separate queue consumer that scales independently.

Processed files are served through CloudFront. For private files, we use signed CloudFront URLs with short expiration.

The key tricky parts are: deduplication using content-addressable SHA-256 hashes, cleanup of abandoned multipart uploads via S3 Lifecycle rules, and rate limiting at the presigned URL generation layer since we cannot rate-limit at S3 directly."

Interview Cheat Sheet

• "Why not proxy through the API?" Say: presigned URLs let the client upload directly to S3, keeping your API server out of the data path entirely; this eliminates memory pressure and connection bottlenecks.
• "How do presigned URLs work?" Say: the API server signs a URL with embedded auth, target key, expiry, and size constraints; S3 validates the signature and accepts the upload without the API server being involved.
• "What about large files?" Say: S3 multipart upload splits the file into 5MB chunks, each with its own presigned URL; chunks upload in parallel, fail independently, and retry individually.
• "How do you resume a failed upload?" Say: client tracks uploaded chunks locally; on reconnect, call S3 ListParts to get completed parts, compare with local tracker, resume from last missing chunk; mention the tus protocol as a standard.
• "What happens after upload?" Say: S3 event triggers an async pipeline via SQS; virus scan first (ClamAV), then media processing (thumbnails, transcoding); each stage is independently scaled with its own queue.
• "How do you prevent malicious uploads?" Say: presigned URL constrains content-type and size; virus scanning catches malware; magic byte validation confirms actual file type; quarantine bucket for infected files.
• "How do you serve files?" Say: processed files go to a separate S3 bucket behind CloudFront CDN; immutable content gets long cache TTLs; private files use signed CloudFront URLs.
• "What about deduplication?" Say: client computes SHA-256 hash before upload; API checks if hash exists; skip upload if duplicate, link to existing file.
• "What about progress tracking?" Say: for single PUT, use XHR upload progress events; for multipart, progress equals completed chunks divided by total chunks.
• "What do most people forget?" Say: S3 Lifecycle rules to abort incomplete multipart uploads after 7 days; rate limiting at presigned URL generation; cross-region upload acceleration.

Test Your Understanding

Q1. Your application generates presigned URLs that expire in 1 hour. A user reports that their upload fails with a 403 error after 45 minutes of uploading a 4GB video. The presigned URL has not expired yet. What is the most likely cause, and how do you fix it?

---

Q2. Two users simultaneously upload the same 500MB video file. Your deduplication logic checks the SHA-256 hash before initiating the upload. Both clients compute the same hash and send it at nearly the same time. Both check the database and find no existing record. Both proceed to upload. How do you prevent the duplicate, and does it matter if you do not?

---

Q3. Your virus scanner Lambda function has a 15-minute execution timeout. A user uploads a 5GB file. ClamAV takes 20 minutes to scan a file this large because it must read every byte. How do you handle this?

---

Q4. Your system uses S3 event notifications to trigger the processing pipeline. A user uploads a file, but the S3 event is lost due to a transient SQS delivery failure. The file sits in the raw bucket unprocessed. How do you detect and recover from this?

---

Q5. Your CDN serves processed files with a 24-hour cache TTL. A user uploads a profile photo, but the old photo is still showing to other users because CDN edge caches have not expired. How do you handle this without purging the entire CDN cache?

---

Q6. A malicious user discovers your presigned URL pattern and writes a script that requests 10,000 upload URLs per minute. Each URL generates a metadata record in your database. They never actually upload any files. How do you defend against this?

---

Q7. Your system strips EXIF GPS data from uploaded images for privacy. A user uploads a JPEG through your system, downloads it, and finds the GPS coordinates are still present. What went wrong?

---

Q8. You are designing the upload system for a platform that operates in China, Europe, and the US. Users in China report upload speeds of 200 Kbps to your us-east-1 S3 bucket, compared to 50 Mbps for US users. How do you fix this without deploying separate upload infrastructure in each region?

Quick Recap

1. Never proxy file bytes through your application server. Use presigned URLs for direct-to-storage upload, keeping the API server on the control plane only.
2. Presigned URLs embed authentication, target location, expiration, and constraints (content-type, max size) so S3 can validate uploads without your server.
3. Large files use S3 multipart upload: split into 5MB chunks, upload in parallel, retry individually, and resume from where you left off after connection failures.
4. Every uploaded file passes through an async processing pipeline: virus scan first, then thumbnails/transcoding/metadata extraction, with separate queues per stage.
5. Virus scanning is non-negotiable. File extensions are trivially spoofed. Validate magic bytes and scan with ClamAV before serving any file to other users.
6. Content-addressable storage (SHA-256 hash) enables deduplication across users, and immutable URLs (including the hash) enable infinite CDN cache TTLs.
7. S3 Lifecycle policies clean up abandoned multipart uploads, and reconciliation jobs catch any events lost between S3 and the processing queue.
8. The entire system separates cleanly: API server for control, S3 for storage, Lambda/ECS for processing, CloudFront for distribution.

Related Concepts

• CDN cache invalidation: How edge caches are refreshed when content changes, and why immutable URLs sidestep the problem entirely.
• Event-driven architecture: The post-upload pipeline uses S3 events, SQS queues, and Lambda consumers, the same pattern used in any async processing system.
• Presigned URLs and temporary credentials: The security model behind S3 presigned URLs uses HMAC signatures with embedded expiration, the same cryptographic pattern as JWTs.
• Video streaming (HLS/DASH): After transcoding, video files are served as segmented streams; the transcoding pipeline described here is the first stage of that path.
• Content-addressable storage: Git, Docker registries, and IPFS all use content hashing for deduplication and integrity verification, the same principle behind upload deduplication.