How Google Photos finds all your dog photos instantly
How Google Photos extracts visual embeddings from every photo at upload time, stores them in a vector index, and retrieves semantically similar images using approximate nearest-neighbor search.
The Problem Statement
Interviewer: "You open Google Photos and type 'dog' in the search bar. Within 200 milliseconds, it shows you every photo of your dog across 10 years and 50,000 photos. You never tagged any of them. How does that work?"
This question tests four things: whether you understand how visual content gets converted into searchable representations (embeddings), how similarity search works at scale (approximate nearest-neighbor algorithms), how the system handles incremental updates as new photos arrive, and whether you appreciate the latency constraints of a consumer search product.
I love this question because it crosses the boundary between ML systems and infrastructure. You need to understand both the model that produces embeddings and the data systems that store and query them efficiently.
Clarifying the Scenario
You: "Great question. Let me clarify a few things before I lay out my approach."
You: "When the user types 'dog,' are we talking about text-to-image semantic search, or is there also a label/tag system involved?"
Interviewer: "Assume both. Google Photos has pre-computed labels, but users can also search for things that were never explicitly labeled."
You: "Got it. And should I focus on the search query path, or also the ingestion pipeline that processes photos at upload time?"
Interviewer: "Both. I want to understand the full loop from photo upload to search result."
You: "And scale-wise, are we talking about a single user's library or the global system?"
Interviewer: "Start with a single user's search, but I want to hear how the system handles billions of photos across all users."
You: "OK. I will structure my answer in three parts: the embedding extraction pipeline that runs at upload time, the vector index that enables fast similarity search, and the query path from the user typing 'dog' to results appearing on screen."
My Approach
I break this into five parts:
- Visual embedding extraction: When a photo is uploaded, a vision model (CNN or Vision Transformer) extracts a dense vector (embedding) that captures what is in the image. This happens once per photo, at upload time.
- Embedding storage and indexing: The embedding is stored in a vector database or index structure (like HNSW or ScaNN) that supports fast approximate nearest-neighbor search.
- Label propagation: Some embeddings are close to known concept anchors ("dog," "beach," "birthday"). The system assigns labels to photos whose embeddings are near these anchors.
- Face clustering: A specialized pipeline groups face embeddings into clusters (person A, person B). The user can name a cluster, and all photos in that cluster become searchable by name.
- Query execution: When the user searches for "dog," the system converts the text to an embedding, searches the vector index for nearby photo embeddings, and merges results with label matches.
The Architecture
Here is how the pieces connect. When you upload a photo, the ingestion service stores the original in blob storage and sends it through the ML pipeline. The embedding model (a Vision Transformer like ViT-L/16) produces a 768-dimensional vector that captures the semantic content of the image. Dogs, beaches, text, food, all of it is encoded into this single vector.
That vector goes into a per-user vector index (sharded by user ID). Simultaneously, a label classifier checks the embedding's proximity to known concept anchors and writes labels to the metadata store. The face detector runs separately, producing 128-dimensional face embeddings that feed into an incremental clustering system.
When you search for "dog," the text encoder converts your query into the same 768-dimensional embedding space. The ANN search finds the 100 photos whose embeddings are closest to the "dog" query vector. A re-ranker blends these results with explicit label matches, recency, and photo quality signals to produce the final ranked grid.
For your interview: the key architectural insight is that search happens in embedding space, not in tag space. This is why Google Photos can find photos of things you never explicitly labeled.
Visual Embedding Extraction Pipeline
This is where the magic happens. The embedding model is the foundation of the entire search system. If the embeddings are bad, nothing downstream works.
Google open-sourced ScaNN in 2020. It is the same library that powers Google Photos search, YouTube video deduplication, and Google Lens visual search. In interviews, mentioning ScaNN by name shows you have done your homework. If you prefer, cite HNSW (used by Pinecone, Weaviate) or FAISS (Meta's library) as alternatives.
Approximate Nearest-Neighbor Search at Scale
The vector index is the engine that makes "find all dog photos in 200ms" possible. I think of ANN search as the interview's core differentiator: anyone can talk about embeddings, but explaining how the search actually works at scale separates good answers from great ones.
A common interview mistake is treating vector search as a simple "just use cosine similarity" problem. At scale, the choice of ANN algorithm (HNSW vs IVF-PQ vs ScaNN) determines whether your system can serve queries in 1ms or 100ms. Always mention the algorithm by name and explain why exact search is too slow.
Incremental Face Clustering
Face search is a special case. When you type "Mom" in Google Photos, it finds every photo of your mom, even photos where she is one of five people in a group shot. This requires face detection, face embedding, and clustering.
In an interview, the face clustering discussion is a great place to show maturity. Mentioning that the system asks users to resolve ambiguous merges shows you understand that ML systems have error rates, and the best approach is a human-in-the-loop design rather than trying to achieve 100% accuracy.
The Tricky Parts
-
Text-to-image embedding alignment: The user types "dog" (text), but the photos have image embeddings. For search to work, text and image embeddings must live in the same vector space. This is what CLIP (Contrastive Language-Image Pretraining) achieves: a shared embedding space where "dog" the word and a photo of a dog have high cosine similarity. Without this alignment, text search is limited to pre-computed labels.
-
Embedding model updates: When Google improves its vision model (from ViT-B to ViT-L, or adds new training data), all existing embeddings become stale. The new model produces vectors in a different space. Google must re-process billions of photos to update embeddings, or maintain backward compatibility by training the new model to be compatible with the old embedding space. This is a massive infrastructure challenge.
-
Privacy-preserving face search: Face embeddings are biometric data. Google Photos processes faces on-device for some features and in the cloud for others. The EU's GDPR and Illinois' BIPA have specific rules about face recognition. Google must handle face data differently by jurisdiction, which adds complexity to the pipeline.
-
Multi-object photos: A photo might contain a dog, three people, a birthday cake, and a park. The single 768-dim embedding must capture all of this. The system also generates region-specific embeddings (object detection + per-region embedding) so that searching for "cake" finds this photo even though the cake is a small part of the frame.
-
Deduplication and near-duplicates: Users often upload the same photo multiple times, or burst shots that are nearly identical. The embedding space naturally groups these (near-identical photos have nearly identical embeddings), but the search results should not show 15 copies of the same shot. The re-ranker must detect and collapse near-duplicates.
What Most People Get Wrong
| Mistake | What they say | Why it is wrong | What to say instead |
|---|---|---|---|
| Tag-based thinking | "A classifier labels each photo as dog, cat, etc." | Fixed vocabulary, no nuance, no semantic search. Cannot search "golden retriever puppy in snow." | "A vision model extracts dense embeddings. Search happens in embedding space, not tag space." |
| Ignoring ANN algorithms | "Just compute cosine similarity against all photos" | O(n) per query does not scale to 100K+ photos per user at millions of QPS. | "Use HNSW or ScaNN for O(log n) approximate search with >95% recall." |
| Treating face search as classification | "Train a classifier for each person" | You do not know who is in the library ahead of time. Cannot train a classifier per person. | "Face clustering with incremental assignment. The user names clusters after they form." |
| Forgetting model updates | "Run the model once at upload time and you are done" | When the embedding model is updated, old embeddings are incompatible with new query embeddings. | "Model updates require re-embedding or backward-compatible training." |
| Ignoring latency requirements | "Run the ML model at query time on each photo" | Running inference on 100,000 photos at query time would take minutes. | "All embedding extraction happens at upload time. Query-time work is just vector search (~1ms)." |
How I Would Communicate This in an Interview
Here is how I would actually say this:
"Google Photos search is a two-phase system: offline embedding extraction and online vector search.
At upload time, every photo goes through a Vision Transformer that produces a 768-dimensional embedding vector. This vector captures the semantic content of the image, what objects are in it, the scene, activities, everything. This embedding is stored in a per-user vector index, something like HNSW or Google's ScaNN library.
When the user searches for 'dog,' the system converts that text query into an embedding in the same 768-dimensional space, using a text encoder that was jointly trained with the image encoder (this is the CLIP architecture). Then it does an approximate nearest-neighbor search against the user's photo embeddings. HNSW gives us O(log n) search time, so even for a user with 100,000 photos, the search completes in under a millisecond.
The results are not purely from vector search, though. The system also has pre-computed labels (from the embedding's proximity to concept anchors) stored in a metadata index. The re-ranker blends ANN results with label matches, recency, and photo quality to produce the final ranking.
Face search works differently. A face detector finds faces in each photo, a face embedder produces a 128-dim vector per face, and an incremental clustering algorithm groups faces by identity. The user names the clusters, and from then on, searching 'Mom' returns all photos in that cluster.
The interesting challenge is that when Google updates the vision model, all existing embeddings need to be re-computed to stay compatible with the new query encoder. At Google's scale, that means re-processing billions of photos."
Interview Cheat Sheet
- Trigger: "How does image search work without tags?" Say: "Vision models extract dense embeddings at upload time. Search happens in embedding space using approximate nearest-neighbor algorithms."
- Trigger: "What embedding model?" Say: "Vision Transformer (ViT) for images, jointly trained text encoder for queries. Same architecture as CLIP. 768-dim shared embedding space."
- Trigger: "How fast is the search?" Say: "Sub-millisecond per query using HNSW or ScaNN. O(log n) vs O(n) for brute-force. Pre-computed at upload time, not query time."
- Trigger: "What about face search?" Say: "Separate pipeline: face detection, 128-dim face embedding, incremental clustering. User names clusters. Ambiguous merges surfaced as suggestions."
- Trigger: "How do you handle new concepts?" Say: "Embeddings are concept-agnostic. Any text query maps to the same space. No need to retrain for new search terms."
- Trigger: "What is ScaNN?" Say: "Google's open-source ANN library. Uses partitioning + quantization for 10-100x speedup over brute-force with 95%+ recall."
- Trigger: "How do you handle model updates?" Say: "Re-embed all photos in the background, or train the new model to be backward-compatible with old embeddings. Both are expensive at scale."
- Trigger: "What about multi-object photos?" Say: "Global embedding captures the whole scene. Object detection + per-region embeddings handle searches for small objects in complex photos."
- Trigger: "How is the index organized?" Say: "Per-user shard. Each user's embeddings are in their own HNSW index. Sharding by user keeps index sizes manageable and isolates search latency."
- Trigger: "What about privacy?" Say: "Face embeddings are biometric data. On-device processing where possible. GDPR and BIPA compliance requires jurisdiction-aware data handling."
Test Your Understanding
Quick Recap
- Google Photos extracts a 768-dimensional embedding from every photo at upload time using a Vision Transformer, encoding just the semantic content into a dense vector.
- Embeddings are stored in a per-user vector index (HNSW or ScaNN) that supports O(log n) approximate nearest-neighbor search.
- Text queries are converted to embeddings in the same vector space using a jointly trained text encoder (CLIP architecture).
- Pre-computed labels ("dog," "beach") come from the embedding's proximity to known concept anchors, stored in metadata for fast filtering.
- Face search uses a separate pipeline: face detection, 128-dim face embedding, incremental clustering, and user-assigned cluster names.
- The re-ranker blends ANN results with label matches, recency, quality, and diversity signals to produce the final ranked photo grid.
- Model updates require re-embedding existing photos or backward-compatible training to maintain search quality.
- The system achieves sub-millisecond search latency because all ML inference happens at upload time, and query time is pure vector math.
Related Concepts
- Vector Databases: The index layer (HNSW, ScaNN, FAISS) is core infrastructure for any embedding-based search. Understanding ANN algorithms is essential for recommendation systems, semantic search, and RAG pipelines.
- CLIP and Contrastive Learning: The joint text-image embedding space that makes "type a word, find a photo" possible comes from contrastive learning. This is the foundation of modern multi-modal search.
- Recommendation Systems: Google Photos search and recommendation systems (Netflix, Spotify, TikTok) share the same core: embed items, search in embedding space, re-rank with business signals.
- Face Recognition Systems: Face clustering in Google Photos is the same technology behind facial authentication (FaceID), surveillance systems, and identity verification. Understanding the privacy implications is important for system design discussions.
- CDN and Caching: While this article focuses on ML, the serving layer for photo thumbnails and search results uses the same CDN patterns discussed in other articles.