A coding assistant needs a dependable answer to a small question before it can answer a large one: what is actually in this file? A search result can identify a word. A syntax tree can identify the declaration containing that word, its parameters, its body, and its location in a particular version of the source. That distinction makes structural parsing valuable even when the assistant already has a capable language model.
Tree-sitter is an incremental parsing system that builds concrete syntax trees and can produce useful structure from incomplete source. Its own introduction describes an editor-oriented library designed for frequent updates. Those are parsing capabilities, not a promise that the library understands program behavior. Type resolution, cross-file references, runtime configuration, permissions, and business requirements still need other evidence. Tree-sitter introduction
This article develops an original reference design for a repository assistant around that boundary. The design recommendations and hypothetical examples below are engineering proposals, not claims about a deployed Apiary feature or measured benchmark. The goal is a system that can find relevant code, explain the limits of its evidence, and produce edits that remain reviewable when the repository changes underneath it.
Begin with a precise structural question
Suppose a maintainer asks an assistant to stop a report endpoint from accepting negative page numbers. A repository search for page might find pagination controls, documentation, analytics events, generated bundles, tests, and unrelated variables. Sending all of those matches to a model creates a ranking problem before the actual programming problem has even begun. The assistant needs a way to identify the request handler, the validation expression, the called service, and the existing tests without treating every textual occurrence as equally important.
A structural index can record function boundaries and import declarations. The assistant can then retrieve the complete request handler rather than a fixed window of lines that ends halfway through a conditional. This makes the evidence easier for both the model and the maintainer to inspect. It also creates a stable place to attach additional facts: the source file, the current content hash, the parser version, and the exact range used to build the excerpt.
That does not establish that the handler is reachable in production. A route may be disabled by configuration, shadowed by middleware, replaced during a build, or unused. The structural question is narrower: this file contains this declaration at this revision. A useful assistant keeps that statement separate from the later operational question of which code handles a real request. Combining them prematurely is how a convincing explanation becomes a wrong patch.
Write the initial retrieval request in terms of the intended decision. For the pagination example, useful questions include where the query parameter is parsed, where bounds are enforced, and which tests describe invalid inputs. Asking for all important code is too vague to evaluate. Asking for the declaration that converts the page query parameter into a number gives the index a concrete job and gives a reviewer a way to judge whether the returned evidence is sufficient.
The parser boundary: structure, locations, and uncertainty
Tree-sitter nodes expose types, child relationships, and source positions. The basic parsing documentation distinguishes named nodes from punctuation-like anonymous nodes, and describes byte offsets alongside row and column positions. These details matter because an editor position, a JavaScript string index, and a UTF-8 byte offset are not interchangeable. Basic parsing
For an assistant, preserve the original bytes as the authority. A structural record should point into those bytes rather than into a reformatted copy. If the retrieval layer normalizes line endings or replaces unusual characters, the displayed excerpt can diverge from the range that a patch operation will later use. The difference may remain invisible until a file contains an accented name, an emoji in a comment, or a mixture of line endings.
Store the encoding decision with the document record. If a file cannot be decoded under the supported policy, mark it as unsupported instead of guessing silently. A plausible but incorrect decoding can produce a tree whose ranges look reasonable while referring to the wrong characters. For most application repositories, an explicit UTF-8 policy with a documented fallback is easier to maintain than a collection of hidden conversion heuristics.
The tree is a representation of a specific input, not a permanent identifier for the program. A node called function_declaration in one grammar may have a different shape in another language or grammar release. Build the indexing layer around language adapters rather than assuming every language shares the same node names. The adapters can produce a common application record, but they should preserve the original parser identity and enough detail to diagnose mismatches.
A practical record might include repository identity, commit or workspace revision, relative path, content hash, language, grammar version, node kind, display name, range, and parse diagnostics. The display name is useful for search; the content hash protects against stale evidence. Neither should substitute for the other. Two functions can share a name, and one function can keep its name while changing behavior completely.
Queries extract patterns; they do not prove meaning
Tree-sitter queries match structural patterns. The documented syntax supports node types, fields, captures, wildcards, and special error-related nodes. A query can therefore identify a declaration and capture a name or body with more precision than a plain text match. It does not turn the captured name into a resolved symbol or establish what a call will do at runtime. Query syntax
Consider two files containing a call named send. One may send a network request, another may append a message to an in-memory queue, and a third may invoke a test double. A query for call expressions whose identifier is send can find candidates. Calling the resulting set “all external communications” would add an unsupported semantic claim. The index should label it as matching syntax and let a language service, import analysis, or manual inspection narrow the interpretation.
For a first implementation, choose a small set of high-value patterns. Function declarations, class declarations, import statements, exported names, and test definitions usually support useful navigation. Add more complicated patterns only when a real task requires them. A hundred speculative queries can produce a large index with no clear answer to which records help a maintainer make a decision.
Keep each query beside examples that demonstrate its intended scope. Include one source fragment that should match, one close-looking fragment that should not, and one incomplete fragment that should be treated cautiously. These examples become a contract for grammar upgrades. They also make it possible to distinguish a parser problem from an incorrect assumption in the application that consumes captures.
If a query captures several elements, preserve their relationship. A list of function names and a separate list of bodies is easy to join incorrectly when nested functions appear. Prefer a record that groups the declaration, its name, its body, and its enclosing scope from one match. The grouping should survive serialization and retrieval so that the model never receives a name from one declaration attached to the body of another.
Build a repository inventory before building a tree index
A parser should not decide which files the assistant is authorized to read. Establish the inventory first. Start from the user's selected repository or workspace, apply explicit exclusions, and record which files were included. Generated dependencies, build outputs, binary assets, private local configuration, and unrelated directories usually need different treatment. The precise policy depends on the task, but the decision belongs outside the parser.
Use relative paths within a known root. Resolve filesystem links according to an explicit policy and verify that an included file still belongs to the intended workspace. A repository can contain a symbolic link that points elsewhere. Following it automatically may broaden the assistant's access without improving the task. A safe index should record a skipped link or a separately authorized target rather than silently treating every reachable path as repository content.
An inventory also provides a baseline for deletion and renaming. If a file disappears, its structural records must stop appearing in normal retrieval. Leaving them in an embedding store can cause an assistant to propose changes to a file that no longer exists. A tombstone or generation marker allows the system to retire those records while retaining a private audit trail when that trail is actually needed.
Separate tracked files from working-tree changes. The maintainer may have unsaved editor content, modified files, or new files that are absent from the current commit. A commit-only index and a workspace index can both be useful, but they answer different questions. The response should identify which view it used. Mixing a declaration from the workspace with a test from an older commit can create a configuration that never existed.
An inexpensive implementation can start with a manifest of paths and hashes, followed by parsing only changed files. It does not need a distributed database on day one. The important property is reproducibility: given the same manifest, parser versions, and query definitions, the system should produce the same structural records. That makes later performance work much safer because correctness already has a reference point.
Incremental updates need an explicit version contract
Tree-sitter's advanced parsing guide describes editing an existing tree's positions and passing the old tree into a subsequent parse. This supports reuse of unchanged structure. Applications still have to provide accurate edit coordinates and manage the lifetime of tree objects and node references. Advanced parsing
In an editor assistant, treat every document change as a new version. The parse result should carry that version, and a consumer should reject a result whose version no longer matches the document it intends to edit. This protects against a common race: the user types while the assistant is retrieving a function, and the assistant later applies an edit at the old location. A successful parse does not make those stale coordinates safe.
The simplest version contract can be a monotonically increasing document number plus a content hash at publication time. The number makes local scheduling cheap. The hash verifies that persisted evidence still describes the exact bytes. When parsing completes out of order, publish only the result for the current version; retain older results only if an explicit historical operation needs them.
For a batch indexer, file hashes can be sufficient. Compare the current manifest with the previous one, parse additions and changes, remove missing paths from the current generation, and publish the new generation atomically. Readers should see either the old complete generation or the new complete generation, not a mixture assembled during a half-finished scan. This is a database consistency problem even if the database is a directory of JSON files.
Do not equate incremental parsing with incremental semantic analysis. Changing an exported type in one file may leave most syntax trees untouched while invalidating type information across many other files. Likewise, changing a build flag can alter which sources participate without changing any source text. A mature assistant maintains separate invalidation rules for syntax, symbol resolution, build configuration, and test evidence.
An example: finding the right validation boundary
Return to the report endpoint. Imagine a handler reads a query string, converts it using Number, and passes the result to a report service. A UI component also clamps its own page selection to positive integers. The assistant finds both. It should not assume the UI check protects the endpoint, because direct callers can bypass the UI. The relevant application boundary is the server-side entry point that accepts the untrusted parameter.
The structural layer helps collect the handler and its adjacent helpers. The assistant then reads the exact expressions and inspects the service's expectations. Does the service accept zero-based or one-based pages? Does an omitted parameter mean the first page? Are fractional values rejected or rounded? Are very large values capped? These questions require code and tests, not a syntax query alone.
A useful evidence packet contains the handler, the service signature, the relevant tests, and a short list of unresolved assumptions. The assistant can propose a check that accepts the documented valid range and rejects the invalid values consistently. The reviewer should be able to follow each assumption back to source. If the expected behavior is missing, the assistant should identify that gap instead of manufacturing a contract from variable names.
The patch should be conditional on the current file hashes. If the handler changes after retrieval, the assistant should re-read it and rebuild the patch. A clean application of a textual diff is helpful, but it does not prove that the surrounding behavior remains the same. The content check protects the reasoning that produced the edit, not just the mechanical ability to insert characters.
After editing, reparse the affected file, run the focused behavior tests, and inspect the diff. Parsing can detect a malformed declaration; the tests can establish the intended handling of invalid page numbers. Neither check replaces the other. The final receipt should say what changed, which cases were exercised, and whether the real endpoint was also checked. This keeps structural evidence in its proper place in the verification chain.
Chunking for retrieval without losing the contract
A function is often a better retrieval unit than a fixed number of characters, but it is not always sufficient. A short function can depend on a type, constant, closure variable, or module-level initialization elsewhere. A very large function can exceed the useful context budget by itself. Structural chunking therefore needs a policy for surrounding evidence rather than a rule that every node becomes one isolated chunk.
Start with the smallest complete declaration that addresses the question. Attach its signature and enclosing context, then add explicitly referenced definitions when they affect the answer. For the report endpoint, that might mean a schema definition and a pagination constant. It probably does not mean the entire UI bundle. Each addition should have a reason visible in the retrieval record.
Avoid cutting a chunk across a conditional or exception boundary merely to meet a token target. If a declaration is too large, split it at meaningful child blocks and retain a parent summary that identifies omitted sections. The summary should be generated from, and linked to, the same source revision. It is an aid to navigation rather than a substitute for the omitted source when the assistant makes a concrete claim.
Ranking can combine exact names, structural kinds, path relevance, and lexical matches before using semantic similarity. An exact exported function name in the expected package is often more informative than a semantically similar comment in an old experiment. Keep enough ranking evidence to explain surprising results. A maintainer who asks why an obsolete file was selected should receive an answer more useful than “the vector score was high.”
Finally, record what was not retrieved. An answer based on three files should not imply a repository-wide audit. An index can be incomplete because of unsupported languages, failed parses, excluded files, or a stale generation. Expose that coverage information with the result. Coverage is part of the evidence, especially when the assistant uses phrases such as “all callers” or “the only validation.”
Handle broken code as a normal editor state
An editor spends much of its life showing code that is temporarily incomplete. A developer has typed an opening parenthesis but not its matching close, started a string, or moved half a declaration. The assistant must remain useful during those moments without treating partial structure as a verified program. A tolerant parse can support navigation while the user is typing; it cannot certify that the file will compile.
Design the interface around levels of confidence in the extracted region. A declaration outside an error region may remain useful for navigation. A capture that overlaps a missing token should carry a warning. A proposed automatic edit that depends on that ambiguous region may need to wait for a stable version or use a different source of evidence. These are application choices, and they should be explicit rather than hidden inside an opaque confidence score.
For example, a user may be editing a nested callback when the assistant tries to identify the enclosing function. The parse may recover by attaching subsequent text under a broader node than expected. If the assistant then replaces that node's entire range, it could remove unrelated code. A safer operation checks the expected start and end text, confirms the document version, and limits the patch to the smallest region whose identity is still established.
Keep malformed-source fixtures in the retrieval tests. Include an unfinished function, a broken import, an unclosed template literal, and a missing delimiter inside a test. The desired result may be a partial answer with an explanation, not a successful capture. Tests that only contain valid source encourage the application to confuse parser resilience with semantic certainty.
The same discipline applies to generated source and merge-conflict markers. A file containing unresolved conflict markers is not just another unusual formatting style. The assistant should surface the conflict and avoid presenting either side as the definitive implementation. Structural tools can help locate the damaged region, but conflict resolution requires understanding which changes belong in the final program.
Add semantic services without erasing provenance
A language server or compiler can resolve types, definitions, and references that a syntax tree alone cannot establish. Integrating those services makes the assistant more capable, but it creates another version boundary. The syntax index and semantic service may have loaded different workspace configurations or different document versions. Their results should be joined only when the relevant identities agree.
Suppose the parser finds a call named validate and the language service resolves it to an imported function. Store both pieces of evidence: the source range of the call and the resolved definition with its project context. If the language service is unavailable, the assistant can still report the syntactic call as a candidate. It should not retain the previous resolution and quietly present it as current after dependencies or aliases change.
Build systems complicate this further. A monorepo may contain multiple projects with different module resolution settings. The same import string can resolve differently depending on the project. A global symbol table keyed only by text will merge facts that should remain separate. Include the project or compilation context in semantic records, and show that context when a result influences a cross-package edit.
Runtime evidence is a further layer. A compiler may establish that a call is type-correct while an integration test reveals that the configured endpoint rejects it. A static call graph may miss reflective dispatch, plugin loading, or data-driven routing. Instead of presenting one graph as complete truth, identify the kind of relationship each edge represents: syntactic occurrence, resolved reference, configured route, or observed execution.
This layered model improves explanations. The assistant can say that a handler syntactically calls a service, that the language service resolves the import to a specific file, and that a named test exercised the error path. Each statement has its own source and limitation. A reviewer can challenge one layer without discarding the entire answer or accepting an unsupported leap between them.
Mixed-language files require ownership rules
Modern application files often combine markup, scripts, styles, and embedded expressions. A structural index must decide which parser owns each region and how results map back to the original document. Treating every file extension as one language is convenient until a template contains a substantial embedded program. Conversely, parsing every region independently can lose the relationship between the host document and the embedded code.
For a reference design, maintain one original document identity and attach language-region records beneath it. Each region should specify its source range, parser identity, and parent relationship. A JavaScript function extracted from a template still belongs to the template file. Its range should not be reported as though it came from a temporary standalone JavaScript file unless the mapping back is preserved.
Edits near region boundaries deserve extra care. Adding a delimiter in the host language can change which bytes belong to the embedded language. The old region map may then be invalid even if the inner function text appears unchanged. Recompute the affected region map before reusing captures, and invalidate downstream records whose interpretation depended on the previous boundary.
The practical test is not whether every language fragment can be parsed. It is whether a reviewer can identify exactly what an edit will change in the original file. If a transformation operates on a derived representation, require a reversible mapping or generate a reviewable patch against the original bytes. Never assume that formatting a temporary extracted fragment and pasting it back will preserve indentation, escaping, or template semantics.
Start with the mixed-language formats actually used by the repository. A narrow, well-tested adapter for one template system is more valuable than a claim to understand every embedded language. Record unsupported regions so retrieval can explain its blind spots. That makes the assistant useful today while leaving a clear path for expanding coverage later.
Treat indexed source as untrusted content
Repository text can contain instructions addressed to an AI assistant. Comments, documentation, test fixtures, and generated files may include phrases that look like commands. Parsing them into a structured record does not make those instructions authoritative. The retrieval layer should preserve the distinction between task instructions supplied by the user and source content supplied as evidence.
For example, a fixture might contain a string telling an agent to upload configuration files. The assistant may need to read that string to understand a parser test. It does not need to obey it. A well-designed evidence packet labels the material as repository content, includes its origin, and avoids wrapping it in language that promotes it into a higher-trust instruction channel.
The same principle applies to summaries and embeddings. A summary can accidentally transform “this test contains a malicious command” into “run this command.” Preserve quotations and attribution when compressing source that includes operational language. Where possible, keep the original excerpt available beside the summary so the assistant can inspect the actual context before acting.
Do not let search results expand file access automatically. A result mentioning a path outside the authorized repository is a fact to evaluate, not permission to read that path. Similarly, a URL found in source is not permission to send repository data to that destination. The index is a navigation aid within the task's access policy, not an alternative authorization system.
An effective adversarial test can use harmless canary text in comments and fixtures. Ask the assistant to perform a legitimate code task and verify that it neither follows the canary instruction nor includes private material in output. This tests the complete retrieval-and-action path. A unit test of the parser alone cannot reveal whether the surrounding agent treats retrieved text as trusted direction.
Measure usefulness with tasks, not just index size
An index with millions of nodes can still be poor at answering a maintainer's question. Evaluate the system with tasks that have known relevant evidence: finding a validation boundary, locating a regression test, identifying an exported function, or explaining a configuration dependency. Record whether the retrieved packet contains the necessary source and whether it introduces misleading material.
Use a small curated evaluation set before scaling. Each task should include a repository revision, the question, expected relevant files or declarations, and known traps. A trap might be a similarly named deprecated function or a generated copy. The expected answer should allow reasonable alternatives; the goal is to test evidence quality rather than force one particular wording.
Measure stale-result behavior separately. Begin retrieval, modify a relevant file, and then attempt to apply the proposed edit. The desired outcome is a detected version mismatch and a refreshed analysis. A system that produces a perfect answer from stale bytes has still failed the operational task. This test often exposes issues that accuracy metrics on a frozen repository cannot capture.
Track latency distributions rather than only an average. A few slow parses of very large files may dominate the user experience even when most files are cheap. Separate inventory time, parsing time, query time, persistence time, and retrieval time. Otherwise, an optimization in one stage may be credited for a delay that actually comes from another stage, such as reading files from a network-mounted directory.
Also measure context efficiency. The useful question is how much relevant evidence fits in the model's budget, not simply how short the packet is. Removing a type definition may save tokens while making the explanation less reliable. Compare packets against task outcomes and reviewer effort. A slightly longer packet that eliminates an unsupported assumption can be a better result than a compact but ambiguous one.
Upgrade grammars as a controlled data migration
A grammar update can alter node shapes, field names, error recovery, and query matches. Treat that as a migration of the index schema. Pin versions in the build environment, record them in generated records, and run the fixture suite before publishing a new generation. The update should be reversible without reconstructing the old environment from memory.
For a representative repository sample, compare captures before and after the upgrade. Count declarations, inspect changed kinds, and identify queries that now return no results or unexpectedly many results. A count difference is a signal to inspect, not proof of regression. The newer grammar may correctly recognize constructs the old one missed, but the downstream application must understand the new representation.
Keep query definitions versioned alongside the grammar selection. Updating one without the other can create a subtle failure in which parsing succeeds but retrieval loses important records. The deployment receipt should name both versions and the index generation they produced. That makes it possible to trace a bad answer back to the exact extraction logic.
For a service with concurrent readers, build the upgraded index separately and switch the active generation only after validation. Retain the previous generation for a bounded rollback period if storage policy allows it. Do not mutate records in place while clients are reading them. Atomic generation selection is easier to reason about than a collection of per-record flags updated over several minutes.
An upgrade should also revisit unsupported-language reporting and encoding tests. Improvements in one grammar can tempt the application to broaden claims about total repository coverage. Confirm those claims from the inventory rather than from the success of the parser build. A successful dependency upgrade is only one part of a trustworthy structural indexing release.
A practical implementation sequence
Begin with a command that creates a manifest for one repository and prints the included and excluded file counts. Add a parser adapter for the primary language and extract a small number of declaration types. Save the records in a simple local format with hashes and versions. At this stage, resist adding embeddings or autonomous edits; first make the evidence inspectable by a human.
Next, build a retrieval command that accepts an exact symbol name or a path and returns complete declarations with source references. Use it on real maintenance questions. Record where the packet lacks context, where it includes too much, and where the language adapter misclassifies a construct. Those observations should drive the next query or context-expansion rule.
Then add workspace updates and stale-result rejection. Simulate edits while retrieval is running, rename files, remove declarations, and introduce temporary syntax errors. A trustworthy assistant must handle these ordinary events before it can safely automate patches. Reliability during change matters more than a polished demonstration against one frozen example repository.
Only after those foundations work should the assistant propose edits. Require a patch against a known input revision, show the diff, and run appropriate validation. Start with changes whose expected behavior is easy to test, such as adding a missing input guard with existing test coverage. Broader refactors should wait until the system can preserve and explain the dependencies that make them safe.
Finally, add semantic services and richer ranking where the observed tasks justify them. Keep the earlier lexical and structural paths available as diagnostic baselines. If a sophisticated ranking change makes results worse, the team should be able to compare it against a simpler method. Complexity is easier to justify when it solves a measured retrieval problem.
A review worksheet for the proposed system
Before accepting an answer, ask which repository revision it describes, whether the relevant files were included, and whether the captures overlap parse errors. Ask whether a claim is syntactic, semantically resolved, configuration-derived, or runtime-observed. These questions convert a broad statement such as “the assistant understands the repository” into a set of inspectable assertions.
Before accepting an edit, ask whether the source hashes still match, whether the patch is limited to the intended files, and whether the tests exercise the behavior that motivated the change. If the assistant changed a validation rule, a formatting-only test is insufficient. If it changed a public interface, the evidence should include the affected consumers or explicitly state that their compatibility remains unverified.
Before accepting a release, ask whether the index generation, query definitions, and parser versions can be reproduced. Check that deleted files disappear from current retrieval and that private exclusions remain excluded. Verify that a failed parse is visible in coverage reports. A system that hides its failures behind a successful overall exit code makes every later answer harder to trust.
The strongest outcome is a modest, specific claim: this tool found these declarations in this source version, gathered these related facts, and helped produce this tested patch. That claim can be extended as more evidence becomes available. It does not need to pretend that structural parsing, by itself, supplies a complete model of a running application.
Exercise: distinguish three kinds of evidence
Take a small repository and choose one endpoint that reads a user-supplied value. First, use text search to find possible handlers. Second, inspect the syntax structure around the best candidate and record the complete declaration and its source hash. Third, run a focused test that exercises one valid value and one invalid value. Write a separate sentence for what each step established. Avoid combining the three sentences until their source identities agree.
Now rename the handler without changing its behavior and repeat the exercise. The textual search may change, the structural identity must be refreshed, and the behavior test should still describe the same contract. Finally, modify the validation behavior while keeping the function name unchanged. This reverses the situation: the name remains familiar while the important behavior has changed. The exercise demonstrates why a name, a tree range, and a passing test are complementary records rather than interchangeable forms of proof.
Sources and further reading
- Tree-sitter introduction
- Basic parsing and node positions
- Advanced parsing and incremental edits
- Query syntax and error nodes
AI-assisted editorial disclosure: this article rewrites and expands local research drafts. Product facts were checked against the linked primary documentation on October 30, 2026. The reference architecture, examples, evaluation plan, and review worksheet are original engineering recommendations, not a report of production deployment or measured performance.