Progressive disclosure for large files
Instead of loading entire files into context, reveal content progressively: outline first, then expand relevant sections on demand, preserving token budget for reasoning.
TL;DR
- A 5,000-line source file costs 15-20K tokens to load fully. Loading it leaves almost no context budget for reasoning about what's inside.
- Progressive disclosure loads the file in layers: (1) directory listing, (2) file outline (class names, function signatures), (3) specific sections on demand.
- The agent navigates like a human developer: sees the table of contents first, then zooms into relevant chapters. Typical savings: 650 tokens instead of 20,000.
- Three tools enable the pattern:
list_outline(path),read_section(path, start, end), andsearch_file(path, query). - Limitation: some tasks need holistic file understanding (refactoring, migration). Forcing progressive disclosure when the task requires reading everything wastes tool calls instead of tokens.
The Problem It Solves
Your coding agent needs to fix a bug in UserService.java. The file is 3,200 lines long. The naive approach: dump the entire file into context. That costs 12,000 tokens. The agent now has the bug fix context, but it also has 3,150 lines of irrelevant code consuming attention and token budget. If the agent needs to cross-reference two other files, each 2,000+ lines, the context window fills up before the agent can reason about the relationships between them.
This isn't a theoretical problem. Production codebases have files ranging from hundreds to tens of thousands of lines. A monorepo with 500 source files can't fit in any context window. Even with 200K-token models, loading files wholesale is a dead end for any task that touches more than a handful of files.
I've profiled agent token usage on real coding tasks and found that 80-90% of file content loaded into context is never referenced in the agent's reasoning or output. The agent needed 50 lines but paid for 5,000.
What Is It?
Progressive disclosure for large files reveals file content in layers rather than loading everything at once. The agent starts with a high-level view (file names and sizes), drills into a structural outline (function signatures, class hierarchies), and only loads the specific code sections it needs to complete the task.
Think of it as a library's catalog system. You don't photocopy every book in the library to answer a research question. You check the catalog (directory listing), read the table of contents of promising books (file outline), then read the specific chapters that matter (section loading). At each step, you decide whether to go deeper or look elsewhere.
How It Works
The three disclosure levels
Progressive disclosure operates on three distinct levels, each adding more detail while consuming more tokens. The agent starts at Level 1 and descends only as deep as the task requires.
Level 1: Directory tree. The agent sees file names, sizes, and folder structure. This costs 50-200 tokens for a typical project. The agent uses this to identify which files are relevant. For a bug in user profile updates, it immediately knows to focus on UserService.java and User.java, skipping the 30 other files in the directory.
Level 2: File outline. The agent requests the structural outline of a specific file. For code files, this means class names, method signatures, imports, and constants. For documents, this means section headings and subheadings. This costs 200-800 tokens per file, depending on complexity. The agent reads the outline and identifies which methods or sections contain the relevant logic.
Level 3: Section content. The agent requests specific line ranges or named sections. "Show me lines 142-187 of UserService.java" or "Show me the updateProfile method." This costs only as many tokens as the section contains, typically 50-300 tokens for a single method or function.
Outline generation: AST parsing vs. regex
The quality of Level 2 outlines determines how effectively the agent navigates. There are two main approaches to generating outlines from code files.
AST-based parsing (tree-sitter, language-specific parsers) produces accurate, structured outlines. Tree-sitter can parse 50+ languages and extract class hierarchies, method signatures with parameter types, decorators, and docstrings. The output is reliable and handles edge cases like nested classes and multi-line signatures. I recommend tree-sitter for any production implementation. It handles malformed code gracefully (partial parsing) rather than crashing.
Regex-based extraction is simpler to implement but fragile. A regex like def\s+(\w+)\( catches Python function definitions but misses decorators, multi-line signatures, and class methods. For prototyping, regex works. For production, use AST parsing.
For non-code files, the approach varies by format:
- Markdown/docs: Extract heading hierarchy (
#,##,###). - JSON/YAML: Extract top-level keys and their types.
- CSV/data files: Extract column headers and row counts.
- PDFs: Extract section headings and page numbers.
Multi-file navigation and cross-referencing
Real tasks rarely involve a single file. The agent might need to trace a call chain across three files, compare implementations in two modules, or understand how a model class relates to its service and controller layers.
Progressive disclosure handles multi-file navigation by letting the agent outline multiple files cheaply, cross-reference their structures, and then dive into only the specific sections that matter across all files. The workflow looks like:
Continue Reading with Premium
Unlock this article and every other in-depth system design guide on the platform with NotesFromSDE Premium.