AI disclosure: Drafted with AI assistance for Apiary. The collection, provider responses, and records below are synthetic. The fixture is a local test, not evidence that a particular remote API was scanned.
A repair cooperative runs a nightly inventory of shared tools. The dashboard usually shows about 1,400 items. On Thursday it shows 27. The synchronizer reports success and has helpfully marked everything else missing. Nothing was stolen. The provider returned 27 records, then an empty page with a continuation token, and the importer stopped at the first empty item list.
This is an easy mistake to understand. “No rows” sounds like “no more rows.” But pagination contracts often define the end using something other than the length of the current page. The wrapper can make it worse: it might filter out an entire provider page locally while preserving the next token. The output will look empty even though the upstream collection has barely been touched.
A correct scanner knows whether it saw an empty page, completed the provider's traversal, completed its own processing, or obtained a consistent snapshot. Those are different claims. Keeping them separate can save a business from replacing a working inventory with a partial one.
Read the provider's end-of-list rule
Google's published AIP-158 pagination guidance uses the absence of a next page token as the end of a collection. It allows a service to return fewer results than requested, including zero, without having reached the end. The Google Drive files.list reference likewise describes partial or empty pages before the end. It also surfaces incompleteSearch in applicable responses, which is a separate warning about coverage. This is not a rule for every API: use the actual end condition and incomplete indicators of the service you called.
Do not stop merely because the page returned fewer than your requested limit. Some services cap the page size, enforce a response byte budget, or omit items that your current principal cannot see. Similarly, do not invent a continuation by incrementing a cursor or decoding an opaque token. Send it back unchanged under the provider's documented request rules, and keep filters and scoping fields coherent across pages. If a user switches from one warehouse to another half way through a scan, it is a new scan.
One exception is deliberately requesting only the first page, for example when a search UI wants the first 20 suggestions. That's legitimate, but the output label should say “first page” or “sample.” It should not become the number of tools the business owns. A UI label can do real harm when people treat a partial count as a full ledger.
Separate the provider's page from your projection
Imagine that the co-op uses a file API as its source and only counts machine-readable purchase records. The provider returns a page of photos. The importer drops all the photos and sees zero items. The next provider page contains a spreadsheet with the records everyone expected. If traversal is driven from the filtered list, the scanner quits before it sees what it needs.
A reasonable receipt has at least three counts:
- Provider pages successfully read, including empty ones.
- Raw rows returned in those pages, before deduplication or local filters.
- Distinct rows accepted into the destination after projection and deduplication.
Those counts can explain “1,600 raw rows, 1,412 kept, no deletion pass” without making an operator work backwards from a single number. If relevant, separately count nonmatching types, repeated IDs, schema rejects, and conflicts. A distinct ID with a malformed revision is not necessarily equivalent to an item you deliberately filtered out.
Don't sort or deduplicate by display name. A co-op can own two different angle grinders with the same label. An item may be renamed between requests. Prefer a stable provider ID as the identity, and preserve source revision metadata if the provider supplies it. Whether the name is canonical, formatted for the local screen, or derived from another field should be a projection rule, not how you decide which rows are the same.
A local failure fixture
Start with a pure collector and a synthetic provider. The page shape below assumes the adapter already translated the provider's contract so absence of a continuation is represented by None. It treats repeated continuation or conflicting records as incomplete rather than quietly choosing a winner. The amount of data fits memory because it is a fixture, not a production storage design.
def collect(next_page, max_pages=100):
seen_tokens, by_id = set(), {}
token = None
for page_number in range(1, max_pages + 1):
page = next_page(token)
if page.get("incomplete"):
raise ValueError("provider marked coverage incomplete")
for row in page["rows"]:
key = row["id"]
if key in by_id and by_id[key] != row:
raise ValueError(f"conflicting source row for {key}")
by_id[key] = row
token = page.get("next")
if token is None:
return {"pages": page_number, "rows": by_id}
if token in seen_tokens:
raise ValueError("repeated continuation")
seen_tokens.add(token)
raise ValueError("page bound reached before provider end")
fixture = {
None: {"rows": [{"id": "A", "kind": "tool"}], "next": "p2"},
"p2": {"rows": [], "next": "p3"},
"p3": {"rows": [{"id": "A", "kind": "tool"},
{"id": "B", "kind": "photo"},
{"id": "C", "kind": "tool"}]},
}
result = collect(fixture.__getitem__)
assert result["pages"] == 3
assert list(result["rows"]) == ["A", "B", "C"]
assert [k for k, v in result["rows"].items() if v["kind"] == "tool"] == ["A", "C"]
broken = {None: {"rows": [], "next": "same"},
"same": {"rows": [], "next": "same"}}
try:
collect(broken.__getitem__)
except ValueError as err:
assert str(err) == "repeated continuation"
else:
raise AssertionError("loop must not produce a complete result")
This code doesn't handle network retries, checkpoint persistence, credentials, provider-specific rules, or mutable datasets. It demonstrates the narrow invariant worth keeping: a continuation on an empty page is followed, and an unfinished traversal doesn't return a completed receipt. The duplicate assertion also makes it impossible for a repeated record to quietly inflate a count.
Extend the test suite before expanding real coverage. Force an error after the first page. Verify that no final pointer moves. Force a failure after rows are staged but before the checkpoint is written. Reprocess the page and show the destination remains consistent. Test a final empty page with no token. Unlike an empty page with a token, in this adapter it is a legitimate completion signal. Mock an explicitly incomplete provider response even if it also omits a token; it must still fail the completeness gate.
Reach the end without claiming a stronger snapshot than you have
A complete traversal under a mutable listing may not represent a single instant. Somebody can rename an item, move it, or add a new one while your client is between pages. Some providers promise a stable snapshot across continuation calls. Others don't. The fact that you used an opaque cursor does not by itself establish which behavior applies.
As a useful contrast, Kubernetes API concepts describe a consistent snapshot for paginated list requests continued under the relevant contract; the documentation discusses token expiration and how to respond when a replacement token might change consistency. That is Kubernetes behavior, not a generic feature you can infer from every token. Read the service semantics and describe your result accordingly.
A receipt can record both walk_complete and snapshot_guarantee. A co-op summary could say “completed listing for the tool room; source can change during scan; 1,412 visible records kept.” That is useful without pretending it proves an instantaneous count or every installed item's existence. For a financial or safety-sensitive stocktake, you might require a provider generation, a frozen export, a deliberately quiescent maintenance window, or a reconciliation against an authoritative change feed. Don't upgrade a best-effort listing by writing a stronger label onto it.
Scope matters even for a stable snapshot. A connector authorized only for a shared folder cannot establish the count for the whole company. A “complete scan” means complete within the explicit scope and permissions used to obtain it. Preserve a nonsecret scope label along with the endpoint and filter version. Do not store raw credentials or opaque page tokens in a public receipt or shared spreadsheet.
Checkpoint and stage before making the display current
Store writes in a staging generation keyed by the scan ID. A page is not complete until both its rows and its checkpoint are durable. Depending on your database and job architecture, commit them as one transaction or design the intermediate states for safe replay. Keying ingestion by scan ID plus source item ID lets a re-read of the same page replace an identical staged row without increasing the count.
A carefully ordered scan could work as follows:
- Create a generation and record the scope, rule version, start time, and any provider snapshot identity.
- Fetch a page using the stored continuation and current worker lease.
- Validate its response and stage rows. Persist progress together with the committed page outcome.
- Continue until the provider indicates completion and no additional incompleteness policy applies.
- Validate conflicts, schema rejects, scope, and count changes before promoting the generation.
- Move the active pointer in a single guarded operation and append the completion receipt.
- Treat removals as a separate policy decision rather than an automatic side effect.
If an older worker wakes after a newer scan was promoted, it must not be able to make its old generation current just because its timer finally ended. A compare-and-set on the active pointer or an ordering rule enforced by the data store can guard the race. A single-writer process is easier to reason about but does not remove the need for a durable progress rule if the process can restart.
For large inventories, stage only the fields downstream needs. Store authorized diagnostic details separately and apply retention. A full raw dump of record content may have very different privacy obligations from a count and a list of validated item IDs. Test with synthetic samples. Count errors by category without copying confidential record fields into operational logs.
Treat expired tokens and rate limits as different events
A token can expire or become invalid. A read can be throttled. A network timeout can leave the client unsure whether it received or staged a page. Those are not three spellings of “start over right now.” Follow the documented error policy for the provider, and do not keep retrying the same invalid continuation indefinitely.
If a retryable request failed, preserve the same scope and token when retrying, and let staging handle any re-delivery of the page. Honor the provider's waiting guidance and keep elapsed retry time bounded. If an expired token requires restarting the complete traversal, create a distinct attempt or generation rather than appending new page 1 to old page 13. The output from the abandoned generation remains useful for debugging. It must not become the basis of a claimed complete snapshot.
A daily window matters here. If a scan is still processing at the morning checkpoint, the dashboard can keep yesterday's active count while showing “today's scan running; 19 pages staged.” If it fails, label yesterday's value with its observation time. The operator then knows both what was last confirmed and what is currently unknown. A spectacular drop to 27 should not become a permanent number just because the dashboard refuses to show uncertainty.
Build the same safety into first runs. If there is no previous verified generation, the right dashboard value is “no complete scan yet,” not zero. Keep that vocabulary consistent in exports and notifications. A blank could be ambiguous; an explicit state helps consumers distinguish a genuinely empty room from a job that never finished.
Add tests where collection and acceptance meet
A function-level check can tell you the loop is correct, but it cannot catch an adapter returning the wrong field. Run a fixture through the same interface that starts the real scan. Feed it a first page with two records and a token, a second page that has a token but yields no eligible records, and a third page with one eligible record and no token. At the adapter boundary, record the exact continuation value received, the declared scope, and the order of the calls. At the UI boundary, assert the displayed count and the completion status. Testing only the final count could pass even if the wrong three records happened to match.
One repair-shop example makes the test concrete. A folder is moved into a new shared workspace during the week, and its name still looks the same in the tool-room app. The importer saved a continuation issued for the old workspace. Starting the new scope from that token is invalid, even if the connector exposes it as a reusable string. Include a policy check that compares the checkpoint's scope and filter hash with the current job. The mismatch must fail before making a provider call, and the new scope needs its own generation. Compare the old and new scopes only after each is complete on its own terms.
To make incident triage quick, capture the number of failed page attempts in aggregate, time elapsed, and the last page checkpoint written. Preserve enough to distinguish “finished walking source but destination validation failed” from “couldn't read beyond page 19.” Those states can lead to different operator actions. In neither case should the prior active generation be automatically marked stale at the exact time the attempt starts.
Follow one scan across its separate counts
Give the fictional tool-room example stable identifiers rather than trusting plausible names. Suppose the catalog is scoped to the east workroom and only includes items eligible for check-out this season. In an invented first page, the provider returns three objects: two active clamps and a retired stand that the local projection will exclude. Its continuation points to a second page that contains no objects but still has a token. The third page contains the first active clamp again and a newly visible square. Its response has no continuation. The scan received five item observations across three pages and saw four distinct IDs. Three distinct IDs were eligible. The retired stand has a known observed exclusion. It is not the same as an ID absent from every response.
These are deliberately small choices so an operator can see the relationships without relying on a program to count for them. If the adapter wrongly interprets the empty second page as the end, the dashboard might report two eligible IDs. If it counts observations rather than unique IDs, the scan might report four eligible records. If it quietly ignores the scope or projections, it might also report four, but for a different reason. This is why a fixture should check visited page IDs, final set of accepted stable identities, and disposition for excluded rows. A single integer cannot distinguish the failures.
Write an invariant with explicit numerator and denominator. “Three active tools” means three unique IDs passing the reported projection among eligible observations in that complete traversal. It does not mean the provider globally has only three tools, that the other workspace has none, or that three items were physically present when somebody opened the dashboard. The final receipt should say which room and filter generated the value, whether the traversal was complete, and whether the provider guaranteed a stable snapshot. Do not use a room's friendly label as its identifier when two distinct containers can share that label.
A model of a complete empty result also deserves a fixture with no previous verified records. Suppose page one reports zero records and no continuation under the documented end rule. The local acceptance pipeline completes normally. That is the legitimate zero case, independent of whether a timeout in an earlier practice run produced no data. Prepare two separate result rows that look similar in a table but have different states. The dashboard should render “0 eligible items, scan complete” in one and “unverified, last attempt timed out before completion” in the other. This small test catches formatting that defaults a missing result to a zero long before anyone tries to interpret a worrying decline.
Record a query identity that means the same thing after restart
An opaque token is not a new query. It continues a specific run under conditions supplied by a provider. Save enough of the materialized query to ensure a restarted worker asks the same question: which logical account or authorized context, which endpoint, which stable collection identifier, and which supported filter, ordering, fields, and projection are in effect. Do not assume every API binds tokens to the exact same set of knobs. Treat provider documentation as the contract; learn what your own staging and acceptance layers need as well. A production connector should remember only what is useful and authorized, not raw authorization headers.
Use a synthetic drift scenario. A manager changes the default filter from eligible only to include inactive, while an attempt prepared under the old filter pauses. If the restarted worker reads today's default from configuration and resumes the old token, the staged pages no longer share a defined scope. The stronger behavior is to bind the old attempt to its original declared query, or abandon it with a distinct reason and explicitly launch a new generation. Both the app's requested scope and the provider's effective scope matter. A projection that flips after page four but leaves the provider request alone can still produce an internally incoherent result.
A fingerprint can guard the restart. Serialize the schema version and normalized scope fields in one known order, then hash that representation for comparisons. Keep the structured description alongside the fingerprint, since a mismatch needs to be explainable. If the new query differs only by the order of equivalent filter arguments, decide whether you are confident they are equivalent before canonicalizing it. Blind normalization of case, empty fields, or character encodings can turn genuinely different provider requests into the same local checkpoint key. A reliable fixture can show that an intentional locale or filter change produces a new identity while serialization noise does not.
Avoid using a hash of a secret token as a public proof of pagination. Metadata needed to resume should stay within the protected worker state, with ordinary operations logs carrying a sequence number or opaque receipt ID. Even if a given provider uses nonsecret cursors, an observer should not need the original cursor to know where a scan stopped. A status row such as “restored checkpoint 17; same room and projection; waiting for page 18” can be useful without exposing provider state to every reader of a dashboard.
Rehearse a restart on each side of checkpoint persistence
Pick a very small fake provider. Page one gives the worker two rows and a token. The worker stages the rows and, in the same durable transaction where available, persists the continuation or marks that step complete. Now interrupt the local test after retrieving the page but before staging it. On replay, the system asks for the prior step again. Interruption here may yield a different body if the provider is mutable; do not pretend exactly-once read semantics without evidence. The local invariant is that the scanner never jumps past data it did not durably accept. Report mutable-read limitations rather than burying them under a stronger claim.
Next interrupt after the rows were staged but before the supervisor received confirmation. If both stage contents and progress committed together, a restart can inspect durable state and determine the next step. If they are implemented in different components, deliberately stage the page twice in a test and show that second processing does not inflate the accepted set. With an append-only log, the duplicate can be stored as an observation event and ignored when assembling the current candidate view. With a strict upsert into a synthetic candidate table, it might update the same row without adding another. Those choices have different evidence and storage tradeoffs; they should not be accidentally selected through whichever operator happens to rerun the job.
Test the competing bad design explicitly. Let the fixture update its checkpoint to page two before page one's rows are durable. Interrupt at that moment and resume from page two. The final scan can look like it reached the documented end yet lack two first-page IDs. A last token of null would pass a naïve completion test. A receipt with a count of durably accepted page records and a known fixture expected identity set can catch the failure. This is a local integration check, not a claim that any arbitrary live collection has a knowable expected size. In the real job, stronger evidence may require a provider snapshot, periodic reconciliation, or a separate authoritative count interpreted under the provider's rules.
Pay attention to when the decision is visible. The candidate generation remains in staging until the writer observes durable completion and application validity. While the worker is restarting, the public reader should still see either the prior verified snapshot or an explicit initial no-result state, not the half-populated staging table. Write a reader fixture which runs before the interruption, after the interruption, and after the successful promote. The pointer should move once. Under concurrent test runners, an earlier generation that finishes second must not overwrite a newer checked decision. The point is not to build a universal distributed transaction. It's to choose and test a stop rule the operator can actually trust.
Notice repeat and movement without diagnosing from an ID count
Under a mutable provider listing, the same identifier can be observed at two page boundaries because ordering or content changed. Duplicate IDs do not always indicate a malformed API or a faulty scanner. Treat them as information about the attempt and apply a documented merge or review policy. In the fictional tool room, Pearl clamp's record may first arrive as eligible and later as held for maintenance. Simple last-write-wins processing will pick whichever page was processed later, not necessarily whichever event is most recent according to provider semantics. If the provider supplies a usable revision indicator, compare it under the documented rules. If not, make the ambiguity visible rather than pretending staging order is the authoritative effective date.
Construct a synthetic conflict. On the first page the clamp has revision four and an eligible state. Near the end of the same attempt, a repeated row gives revision three and says not eligible. If revision integers are documented as comparable per item in this fixture, preserve four and record that an older repeat was observed. Now construct a second case with two different payloads marked revision four. That isn't settled by comparing version order. Consider marking the row conflicting and holding it or choosing a documented provider resolution route. Record the effect on acceptance separately from walk completion. The catalog does not have to stop every time a repeat appears, but it should not silently lower quality to get a green counter.
Test movement across collection scopes intentionally. Imagine a tool transfers from the east room to the west room after the east listing passed the relevant ordering boundary, but before the west listing arrives there. Depending on provider consistency and timing, an item may appear in both snapshots or neither staging view. A full traversal of each room only says each traversal reached its own end under its own semantics. It cannot establish a simultaneous global inventory unless both scopes share a documented consistent snapshot. Use known transfer records if the product has them, or present both independently dated snapshots and explicitly reconcile uncertain moves. Do not turn absence from a single room into erasure of the underlying item.
If the provider reports partial search, absence is weaker still. Store that warning beside the staged receipt rather than burying it in a raw response dump. A fixture can make a listing walk reach a terminal token with a partial indicator. Depending on the product contract, reject promotion, or promote only a labeled limited-view summary where that's explicitly appropriate. Keep the acceptance decision consistent across API output and dashboard. Nobody should see a large trustworthy Complete stamp simply because a query stopped returning tokens while the provider was reporting incomplete coverage.
Separate error recovery from the timing of user alerts
Define what the operator hears as the scan progresses. A transient read timeout with a documented retry window can justify a waiting status; a malformed continuation field can justify a distinct integration error; a completed walk that fails candidate validation needs a diagnostic label different from provider outage. Use fixtures to make each failure class observable without touching a real service. Simulate retryable and permanent read responses using a local adapter and an injected bounded timer. A stop condition should point to the exact stage that didn't complete, not summarize everything as catalog offline.
A loop guard is one particularly useful fixture. Suppose the fake provider gives the same continuation repeatedly, or sends a two-token cycle. Treat this as unresolved traversal, not as normal completion or an excuse to keep requesting indefinitely. Determine what can safely be compared from the token values inside protected execution state, save only aggregate evidence where users can see it, and stop with a reviewed reason. Add an independent time or page safety bound for faults the duplicate detector misses. Reaching that bound does not make the result complete. An operator can restore a prior valid comparison and troubleshoot the incompatible listing rather than performing uncontrolled retries.
Report an expired token differently from an identical repeated token. The documentation may prescribe a fresh generation to continue reliably. If the proposed continuation changes snapshot semantics, give the operator a decision to accept or abandon based on the catalog's requirement. For example a high-integrity count that promises a consistent snapshot should restart from its initial input rather than grafting new observations into a half-read earlier attempt. A looser review feed may be able to show clearly labeled progress from moving inputs. Neither is correct if the public receipt calls both “snapshot verified.” This qualification is more valuable than a retry counter alone.
An alert should pull evidence from the stage it describes. A morning notification that no current count is available may be legitimate when the old one is still visible and labeled. Don't notify that two hundred tools are missing while the scanner is on page three of eleven. If a prior attempt timed out and a later attempt is in progress, preserve both facts. A dashboard badge can show progress without rewriting the previous result as failed, and an alert can mention whether further observation is planned. Keep the job logic and notification schedule independent so that an alert doesn't launch another traversal simply because it observed a pending one.
Prove export consumers interpret the same final generation
Staging is only useful if readers obey the boundary too. The fictional co-op offers a dashboard and a CSV export. Both should read the current approved generation, not “whatever rows are newest” from their own encounters with staging. A fixture first promotes an approved generation of three IDs. It then begins a new candidate with only one staged page and launches an export. Check that the export contains the three approved IDs and labels their generation. After candidate validation and promotion, repeat the export and confirm the approved pointer and content moved together. This catches a consumer that uses the updated count from a snapshot but fetches row details from a staging query.
For a long export, consider how page-by-page reads behave once the active pointer changes. If every export page resolves current anew, a single download may combine rows from different generations even while each query appears healthy. Pass the selected generation or use an appropriate stable read contract throughout the report. Verify an artificial mid-export promotion with a fixture and make the result explicit. A single-file report should either finish consistently at its starting generation or stop and tell the reader that it must be regenerated. It should not silently assemble a mixture and keep the old filename. This is a separate problem from reading the upstream pages correctly; both boundaries matter to a usable catalog.
The output receipt doesn't need to expose protected connector information. Public or shared reports can include a scan observation date, logical scope label, verified counts, and particular limitations. Sensitive row data, source identity material, and exact upstream cursors stay behind the appropriate controls. Keep one internal correlation identifier so an authorized maintainer can trace an export to a generation when a discrepancy is reported. A shorthand count is more helpful when it carries just enough provenance to establish what an ordinary reader expected, without becoming an accidental provider debug dump.
Review the acceptance evidence with someone who did not run the scan
A simple review session uses the latest two approved generations and the incomplete candidate from the workbook. Give the reviewer only a synthetic scope description, three count categories, the source completeness note, and the reasons each generation did or did not become current. Ask which count should appear in the next report and whether anything can be marked missing. If the reviewer chooses the staged candidate because its date is newer, investigate the presentation before changing the data logic. A correct database pointer is not enough if the evidence pack makes incomplete rows look more trustworthy than approved ones.
Build a receipt specifically for changes in eligibility. One of the tool room's braces is fully present in the provider response but marked maintenance-only in its current approved revision. Another brace does not appear in any accepted page. The first has evidence for exclusion under the current policy. The second has only an absence of observations from this scoped scan. A removal report should not combine the two. An operator might move the observed brace to the correct internal review category while leaving the absent one unresolved. Test that the projected count, searchable catalog, and support export still distinguish the two after the next scan.
Document the exception policy as part of the tested query contract. If an editor permits an old status field to be missing from a legacy provider scope, that exception can be recorded as a specific rule with known effect on eligibility. Do not blanket-permit every malformed row so the scan can finish green. In a synthetic trial, allow one agreed missing optional label and separately reject a row without a stable identifier. The job can still traverse the provider listing in the second trial without producing an accepted candidate. The reviewer learns both where the data came from and why the candidate was fenced off.
Finally, include a small query result that a developer or auditor can reproduce independently. For the same approved generation, fetch the projected ID set in a controlled fixture and compare the answer to what the dashboard shows by count and scope. If they differ, it's useful to know whether the cause is an unstable result, stale projection, or a grouping rule in the report itself. Don't reach for a broad provider rescan when the existing immutable evidence is already enough to test a disagreement between two consumers.
Put destructive reconciliation behind a higher bar
Suppose the completed scan sees 1,200 of the 1,400 tools previously recorded. That is a reason to investigate. Maybe 200 were legitimately transferred. Maybe the owner whose account ran the job lost access to one room. Maybe the scan started with the wrong parent scope. Missing observations aren't proof of intentional deletion.
Choose the confirmation threshold based on the business cost of an error. Possible rules include requiring explicit tombstones from the source, comparing multiple complete generations with the same scope, checking container visibility, or sending large drops to manual review. A temporary “not observed in latest scan” can remain useful to staff without erasing the last known item. Unusual count changes should be attached to the receipt and handled under a reviewed policy, not just buried in a log.
Run the system in shadow mode first. Compare new receipts against existing counts during ordinary use and during a controlled synthetic gap. Test that an empty intermediate page, a timed-out provider, an expired cursor, and a new scope each produce recognizably different status messages. Make the acceptance gate repeatable enough that the next developer can reproduce a false-complete bug without calling the actual service.
The habit that matters is simple to say and worth making difficult to violate in code: “no more pages” is a fact about traversal; “these are the official records” is a decision with evidence requirements. Your catalog should not blur those together. When the screen shows 27 again, someone can look at the receipt and know whether 27 is today's verified count, a partial staging count, or the size of a narrow result requested on purpose.