User feedback flywheel

TL;DR

• Production user data is your most valuable training and eval asset. Most teams don't collect it systematically even though it's sitting right there.
• Implicit signals (edit distance, copy-paste, task completion, session abandonment) outperform explicit ratings because users don't have to do anything extra.
• Sampled production I/O pairs with quality labels become your real-world eval set. This reflects actual user intent distribution better than any synthetic benchmark.
• The core loop: collect, label (auto or human), cluster failures, fix (prompt update or fine-tune), re-eval, deploy, repeat.
• LLM-as-judge can label production data at scale. Spot-check 5-10% with human labels to calibrate it.

---

The problem it solves

You launch an AI feature. It seems to work well in testing. Three months later, a competitor's product feels noticeably better, you're not sure why. The competitor is also building on GPT-4o. The difference isn't the base model. It's the data they've accumulated from real users and the improvements they've made from it.

The best AI products improve the more they're used. But this only happens if you've built the infrastructure to capture, label, and act on user feedback. Without that infrastructure, your product stagnates while competitors build a data advantage that compounds over time.

Most teams don't build this infrastructure at launch because it feels like a "later" problem. By the time they recognize it as critical, they've lost months of data that can never be recovered.

What is it?

The user feedback flywheel is the system that converts user interactions with your AI feature into quality signal, routes that signal back to improve the model, and measures whether the improvement worked. It's a closed loop: usage generates feedback, feedback drives improvement, improvement increases usage.

The flywheel has four stages: signal collection (implicit and explicit), labeling (automated and human), failure analysis (clustering bad outputs by type), and improvement (prompt update, RAG improvement, or fine-tuning).

How it works

Implicit signals

Implicit signals are behavioral data collected without asking users to do anything. They're more reliable than explicit ratings because they reflect real behavior, and you get far more of them.

Edit distance: If the user heavily edits an LLM-generated document, the edit distance is high, indicating low-quality output. If they accept it with minor tweaks, edit distance is low. This is one of the most reliable quality proxies for generative text.

Copy-paste: A user copying LLM-generated text to another application indicates they found it useful. Segment by how much of the response was copied: copied everything (high quality), copied a sentence (partial quality), copied nothing (low quality or irrelevant).

Code execution: If you generate code and the user runs it without modifications, they found it correct. If they run it, get errors, and spend time fixing it, quality was low. GitHub Copilot uses acceptance rate and subsequent edit behavior as primary quality signals.

Task completion: Did the user finish the workflow your AI was helping with? Abandonment after seeing an AI response is a strong negative signal. Completion is a positive signal.

Session behavior: User continues engaging after the AI response (asks follow-ups, clicks on suggestions) = positive. User immediately bounces or starts a new conversation = negative.

Explicit signals

Explicit signals are ratings users provide deliberately: thumbs up/down, "Was this helpful?", "Report an issue." These have much lower collection rates (typically 1-5% of users rate any given output) but provide higher-confidence labels when collected.

Ask for explicit feedback sparingly. Rating prompts shown after every output get ignored. Show the rating UI only at natural completion points (end of a workflow, after a distinct task completion), and for outputs where you have high uncertainty about quality.

Turning signals into eval data

Sampled production I/O pairs with quality labels become your most important eval dataset. Pull a representative sample of recent requests (stratified by user type, query category, time of day), label them with quality scores via your signals, and treat this as your eval benchmark.

This dataset has two advantages over synthetic evals: it reflects the real distribution of queries your users actually ask, and it captures failure modes that your synthetic test cases didn't anticipate.

I've found that teams who build this dataset early gain compounding benefits. Your eval set grows automatically as users interact with the product, so each new prompt or model update is evaluated against an increasingly comprehensive set of real-world inputs.