Valet key pattern

TL;DR

• The valet key pattern gives clients direct, time-limited access to a resource without routing data through your application server.
• Your server generates a pre-signed URL or scoped token, hands it to the client, and steps aside.
• The client uploads or downloads directly to/from the storage service. Your server never handles the file bytes.
• Pre-signed URLs embed a cryptographic signature scoped to one operation, one exact object key, and one expiry time. They cannot be reused or extended.
• The pattern eliminates server-side bandwidth costs, removes a scaling bottleneck, and is the standard approach for direct S3 uploads in modern web and mobile apps.

The Problem With Proxying Uploads and Downloads

The naive design for file uploads: client sends the file to your API server, your server validates it, your server stores it in S3. For downloads, your server fetches from S3 and streams it back. Simple to implement, obvious to reason about.

But every byte passes through your application server twice. A 50 MB video upload consumes 50 MB of inbound bandwidth on your server and another 50 MB outbound from your server to S3. Downloads add the same again in reverse. Your server is paying a full bandwidth tax on every transfer, for work it is not actually doing.

At scale this becomes expensive and fragile. A hundred concurrent 50 MB uploads push 5 GB per minute through your application tier. File transfers are I/O-bound, not CPU-bound, so you cannot simply scale CPU. Your application servers saturate their network interfaces while sitting idle on compute.

The server is not adding value here. It is a very expensive pipe.

One-Line Definition

The valet key pattern issues a short-lived, narrowly-scoped credential that lets a client communicate directly with a backend resource, removing the application server from the data path entirely.

Analogy

A hotel valet takes your car key, drives to the parking garage, and returns a valet stub. That stub lets you retrieve exactly one car (yours), from exactly one garage, within a limited window. The valet cannot drive your car to another city. You cannot use the stub to access someone else's car.

Your application server is the valet. The pre-signed URL is the stub. The client gets exactly one operation, one object, a short window. When the window closes, the credential is worthless.

Solution

Instead of being the data proxy, your server generates a pre-signed URL that embeds time-limited credentials. The client uses that URL to upload or download directly. Your server handles two small JSON API calls; the heavy data transfer happens entirely between the client and storage.

The sequence has three distinct phases. First, your server issues the credential. Second, the client does the transfer entirely without your server. Third, your server records the completion. That middle phase is the entire point.

How a Pre-Signed URL Works

A pre-signed URL is a regular HTTPS URL with authentication embedded in its query parameters. There is no separate credential request at S3 when the client uses it. S3 validates the embedded signature directly.

AWS signs the URL using your IAM credentials (access key + secret key). The signature covers the HTTP method, the exact S3 bucket and object key, an expiry timestamp, and optionally Content-Type and Content-Length. When S3 receives the request, it re-computes the signature from the request components and compares it to the embedded one. If anything was tampered with, the signatures don't match and S3 returns a 403.

import boto3

def generate_upload_url(user_id: str, filename: str, content_type: str, file_size: int) -> dict:
    s3 = boto3.client('s3', region_name='us-east-1')
    object_key = f"users/{user_id}/uploads/{filename}"

    url = s3.generate_presigned_url(
        ClientMethod='put_object',
        Params={
            'Bucket': 'my-user-uploads',
            'Key': object_key,
            'ContentType': content_type,
            'ContentLength': file_size,
        },
        ExpiresIn=300  # 5 minutes
    )
    return {'upload_url': url, 'object_key': object_key, 'expires_in': 300}


def generate_download_url(object_key: str, expiry_seconds: int = 3600) -> str:
    s3 = boto3.client('s3', region_name='us-east-1')
    return s3.generate_presigned_url(
        ClientMethod='get_object',
        Params={'Bucket': 'my-user-uploads', 'Key': object_key},
        ExpiresIn=expiry_seconds
    )

The client uses this URL with a standard HTTP PUT. No AWS SDK needed.

PUT https://my-user-uploads.s3.amazonaws.com/users/123/uploads/avatar.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA...&X-Amz-Date=20260405T...&X-Amz-Expires=300&X-Amz-Signature=abc...
Content-Type: image/jpeg
Content-Length: 1048576
[binary bytes]

The credentials are in the URL, not in a header. You can send this URL to any client (mobile app, browser, third-party system) and they can execute the upload without ever knowing your AWS credentials.

Scoping and Security

The narrower the scope of the credential, the safer the pattern is. I've seen teams generate pre-signed URLs with 24-hour TTLs that could be used for any object in a bucket. That defeats the security model entirely.

Scope by operation. Generate separate URLs for PUT (upload) and GET (download). Never use the same pre-signed URL for both directions.

Scope by exact object key. The URL is bound to one path. users/123/avatar.jpg cannot be used to access users/456/avatar.jpg.

Set a short TTL. Five minutes for uploads is a reasonable default. Long TTLs are a liability: if the URL leaks (in a log, browser history, a forwarded email), it stays usable.

Include ContentType in the signature. If the signature doesn't cover ContentType, a client could upload an HTML file while claiming it's image/jpeg, then use the resulting S3 URL as a phishing page hosted on your domain. With ContentType in the signature, the upload fails unless the client sends exactly the declared type.

File size limits. S3 pre-signed PUT URLs do not enforce a maximum file size. Add a bucket policy condition to reject objects over your limit, or use S3 multipart upload pre-signed URLs that validate each chunk.

Platform Equivalents

All major cloud providers support this pattern with nearly identical semantics.

# Azure Blob Storage: SAS token for a single blob
from azure.storage.blob import generate_blob_sas, BlobSasPermissions
from datetime import datetime, timedelta

def generate_azure_upload_sas(container: str, blob_name: str) -> str:
    sas_token = generate_blob_sas(
        account_name='myaccount',
        container_name=container,
        blob_name=blob_name,
        account_key='my-storage-key',
        permission=BlobSasPermissions(write=True),
        expiry=datetime.utcnow() + timedelta(minutes=5)
    )
    return f"https://myaccount.blob.core.windows.net/{container}/{blob_name}?{sas_token}"

Azure SAS tokens can be scoped to a container or to a single blob. For user-generated content, always scope to a single blob. Container-level SAS tokens are too broad. GCP uses Google Cloud service account keys to sign URLs instead of the access key/secret key model, but the security model is identical.

Post-Upload Confirmation

The biggest operational wrinkle: your application server does not know when the upload completes. The client uploads directly to S3; no request hits your server during the transfer. You need a separate mechanism to confirm completion and trigger downstream processing.

There are three workable approaches:

S3 Event Notifications are best for decoupled pipelines. S3 emits an event to SQS or SNS whenever an object is created. A worker consumes that event, does post-processing (virus scan, image resize, metadata indexing), and marks the upload complete in your database. The downside: there is latency between upload completion and processing (seconds, not milliseconds).

Client Completion Endpoint is best when you need immediate confirmation. After the client finishes the PUT to S3, it calls POST /uploads/{id}/complete. Your server calls HeadObject to verify the file exists, records metadata, and responds. The tradeoff: your server is back in the loop for this one small step.

Polling adds round-trips without solving the real problem. I would not choose it as a primary pattern unless the use case is explicitly batch-oriented.

My recommendation: use S3 event notifications for asynchronous pipelines (video processing, batch document conversion) and the client completion endpoint for interactive uploads where the user waits for confirmation. They are not mutually exclusive.

When to Use

The valet key pattern is the right choice when:

• Clients upload or download large files (images, videos, documents, backups)
• Your storage service supports pre-signed credentials (all major cloud providers do)
• You want to offload bandwidth costs from your application servers
• You are building a mobile app where file transfers happen in the background

It is overkill or inappropriate when:

• Files are very small (under 10 KB) where the extra round-trip for the credential equals or exceeds the bandwidth savings
• Your system uses on-premises storage without pre-signed URL support
• You need to inspect or transform every byte during transfer (DRM watermarking, real-time transcoding) since those use cases require server-side processing

Operations

Monitor upload completion rates. Track the ratio of pre-signed URL generations to confirmed uploads. A large gap means client-side failures: network issues, CORS misconfiguration, or TTL expiry before the upload started.

Audit S3 server access logs. S3 logs every request, including which IP addresses used each pre-signed URL. If you suspect a URL was shared inappropriately, the access logs tell you exactly who used it.

Rotating an IAM key does not invalidate outstanding pre-signed URLs. Pre-signed URLs are signed at generation time. Revoking the IAM key does not retroactively invalidate URLs already generated with that key. If you suspect a key compromise, assume all outstanding pre-signed URLs from that key may be in hostile hands until their TTLs expire. Keep TTLs short for sensitive content.

Set bucket policies as defense in depth. Even with properly scoped pre-signed URLs, add a bucket policy that denies non-HTTPS access and restricts the IAM roles that can generate pre-signed URLs for this bucket.

Interview Cheat Sheet

What it is: A short-lived, narrowly-scoped credential that lets a client communicate directly with a storage service, removing the app server from the data path.

The core tradeoff: Removes server bandwidth costs and scaling pressure, but introduces a completion notification gap and requires careful scope/TTL management.

S3 specifics to name: generate_presigned_url(), method scoping (PUT vs GET), object key binding, ContentType in the signature, 5-minute TTL for uploads.

Three completion options: S3 event notification to SQS (async), client completion endpoint with HeadObject verification (sync), polling (batch only).

Common gotchas to mention:

1. CORS must be configured on the bucket for browser uploads
2. Pre-signed URL TTL does not align with user session revocation
3. S3 PUT pre-signed URLs don't enforce maximum file size by default
4. Rotating an IAM key doesn't invalidate already-issued pre-signed URLs

When NOT to use it: Files under 10 KB, on-premises storage without pre-signed URL support, or use cases requiring byte-level server processing during transfer.

Quick Recap

1. The valet key pattern removes your application server from the data path by issuing a short-lived, narrowly-scoped credential that lets clients upload or download directly to/from storage.
2. AWS S3 pre-signed URLs embed IAM credentials in a signed query string covering one method, one object key, one expiry, and optionally Content-Type and Content-Length. The client needs no AWS credentials of their own.
3. Scope aggressively: separate URLs for PUT and GET, bound to exact object keys, 5-minute TTLs for uploads. Include Content-Type in the signature to block MIME-type manipulation on user-uploaded content.
4. Azure Blob Storage uses SAS tokens; GCP uses Signed URLs. The security model is identical across all major platforms.
5. After direct upload, your server needs a separate completion signal: S3 event notification to SQS for async pipelines, a client-called completion endpoint with HeadObject verification for synchronous UI feedback.
6. Rotating an IAM key does not invalidate already-issued pre-signed URLs. Keep TTLs short for sensitive content and revoke access at the URL generation level, not the key level.