ApiaryActiveLive
Try: pause · settings · learn · wipe
← Community / Reading Room
TA
craft · 24 min read

Test a Retry Where the Worker Can Really Lose Its Place

A fictional county library makes a daily inventory snapshot for its volunteer repair crew. A worker receives a job, adds a snapshot record, and acknowledges…

AI disclosure: Drafted with AI assistance for Apiary. The catalog job and test inputs below are fictional. The design does not establish that any deployed worker passed these scenarios.

A fictional county library makes a daily inventory snapshot for its volunteer repair crew. A worker receives a job, adds a snapshot record, and acknowledges completion. That looks straightforward on the happy path. An engineer writes a test that calls the handler twice with the same identifier and confirms that there is one row. It is a useful start, but the test did not model what the crew actually worries about: what happens when the worker disappears after adding a row, before the dispatcher sees success? What if the first failure occurs slightly earlier, when the row exists only in an uncommitted transaction? A test that runs two polite, complete calls makes those outcomes look the same, though they are not.

A stronger fixture asks what durable evidence exists at each interruption point. It puts a tiny real database on a temporary filesystem and introduces a failure on either side of its commit. The result is evidence about a narrow property: under the chosen job scope, identity, and durable storage rules, retries are not applying the modeled effect twice. It is not evidence that a payment service, a mail provider, or some other network destination has exactly-once behavior. You don't need real customers, external APIs, or irreversible operations to learn the relevant local lesson.

Specify the effect rather than counting calls

Before looking at infrastructure, name the outcome that must not duplicate. In the library example, the desired effect is “create one inventory snapshot record for logical job x.” Executing the worker twice is allowed. Logging two starts is allowed. Observing one failed acknowledgment and a second successful one is allowed. Creating two snapshot records for the same logical job is not. A test that asserts the handler was called once would miss the distinction. An operational dashboard that marks every extra attempt as data corruption would be just as misleading.

Then define sameness. The input might contain a library branch identifier, a fixture day, a report kind, and a job identifier assigned by the dispatcher. Variations in JSON field order shouldn't manufacture a new logical action. A caller reusing the same job identifier to ask for a genuinely different snapshot shouldn't get the first result as if nothing changed. Choose a scope for the key: two independent branches could theoretically assign the same identifier, so namespace by branch or allocate keys centrally. Persist a compact fingerprint of the normalized fields you actually compare, with a named version for the fingerprint procedure. Make sure an event identity from the dispatcher is not replaced by a fresh random value generated on each attempt.

Define the expected outcomes for three rows in your test sheet. First delivery with an unused key: one result is created. Repeated delivery with the same scope and equivalent request: the existing result is returned and nothing additional commits. Repeated delivery with the same scope but incompatible request: the attempt is rejected as a conflict, leaving the recorded result intact. If the business rules legitimately say that a later submission can amend a result, model that as a deliberate operation, with its own identifier and semantics. Don't allow the test to pass because a duplicate was transformed into a silent update.

Keep the receipt and modeled effect in one database transaction

A fixture starts with two small tables. One stores the durable job receipt with a unique key over branch and job identifier. It stores a request fingerprint and a result identifier. The second stores the toy snapshot, with a stable reference back to the logical job. Production systems may use a single table if the effect and receipt are inherently the same row. The two-table version is useful in a teaching exercise because it makes a dangerous split visible: a snapshot could survive without its receipt, or a receipt could survive without its snapshot.

The SQLite transaction documentation explains explicit transactions and their boundaries. The experiment should choose those boundaries knowingly and scope both writes to one transaction. The application should look up an existing receipt within that design, check request equality, and return a result only when the invariant is satisfied. If it has no receipt, it computes the fixture snapshot from local data, writes both records and commits. It then acknowledges or returns. If it does have a receipt for an equivalent request, it returns the earlier result. If there is a mismatch, no snapshot changes.

A uniqueness constraint is a second line of evidence, not the whole policy. SQLite UPSERT documentation explains how a conflict against a uniqueness constraint can invoke alternate handling. If two workers compete for a key, the losing branch still has to determine whether the committed row represents the same request. A conflict clause that discards the new insert will produce just one row in some cases, but it cannot by itself tell you whether both callers asked for the same thing. A clause that updates the earlier receipt overwrites the evidence you need. A fixture should check the fingerprint and old result, not just row count.

If implementing in Python, see the standard library documentation for sqlite3 transaction control. Verify which mode your harness and worker use and test the expected boundary under that mode. Do not assume the behavior of a context manager from the appearance of its indentation. For a subprocess fixture, in particular, create a separate process and a fresh database connection for the next delivery. Two calls sharing one connection can see the same uncommitted state and conceal the very distinction the exercise was built to demonstrate. Use a file in the fixture directory rather than pretending the next connection can inspect your first connection's private in-memory database.

Stop once on each side of the line

Design the subprocess with explicitly named test hooks. One hook records that both inserts were attempted but stops the worker before it commits. Another hook runs after commit returns but before the worker emits the success acknowledgment. A parent test harness launches the worker with one selected hook, provides the same synthetic input for each replay, observes the marker, and makes the worker exit without running ordinary graceful teardown for that interruption test. Arrange a dedicated fresh fixture per case and use an independent reader after the worker has exited. The marker identifies where you intended to interrupt; the independent read establishes what state actually survived.

On the before-commit case, there should be neither a durable receipt nor a durable snapshot after interruption. The dispatcher has no success. Replaying the same input should create both records once and return one result. On the after-commit case, the dispatcher still has no success, but the database should contain both records. Replaying should return the result already in the receipt, with the same result identifier, without creating a second snapshot. The outer symptoms are identical; only the database and the retry tell you which case happened. If the first connection is still alive or the parent reads too early, busy or visibility conditions should be reported as inconclusive rather than made to look like absence.

There's a weaker fixture that injects an exception, lets the surrounding framework do graceful cleanup, and logs a simulated interruption. It can be good for verifying a rollback handler, but it is not evidence of abrupt worker departure. Keep both tests if they answer distinct questions. Label which shutdown path took place. Likewise, forcibly stopping a local process after a successful commit is not the same experiment as unplugging a machine or losing storage. It does not prove durability across a host crash, a device cache problem, or corruption of the underlying volume. Scope the test to a process failure and an available fixture database; layer additional evidence separately if your requirements demand it.

Run one control case with neither hook, using the same fixture seed and assertions. It should produce the expected result and a successful acknowledgment. A runner that always terminates because it can't find the database path is not an effective failure injector. Another useful control is to set a marker but let the worker continue, then inspect the expected acknowledgment. If the test harness never truly drops an acknowledgment, both named failure cases may only be success tests painted different colors. Keep the assertions separate enough that you know you observed the interruption, not just a test timeout.

A local synthetic smoke test for this article checked the two boundaries with a worker subprocess and a separate connection for each read. It used a fixed result identifier of 17 and a new temporary database for each scenario. Before commit, abrupt exit left zero receipts and zero snapshots; replay returned 17, then both counts were one. After commit but before the acknowledgment, abrupt exit left one receipt and one snapshot; replay also returned 17, then both counts remained one. Neither interrupted subprocess printed a success acknowledgment. That smoke test did not exercise matching fingerprints, overlapping workers or a network queue. Keep the minimal trace separate from the fuller test plan so reviewing the evidence doesn't require guessing which design suggestions were executed.

Include one intentionally broken design

A good fixture makes a bad design fail for a recognizable reason. In a local throwaway comparison, split the snapshot write and receipt write into separate committed transactions. Interrupt the worker after the first commit but before the second. The independent connection now sees a snapshot and no receipt. When the dispatcher repeats the job, either a second snapshot appears or a uniqueness constraint triggers a failure that refuses to return the already-existing result. Either violates some part of the specified behavior. If the broken design still passes, your assertions might only count receipts or might read the database before the simulated crash point occurs. This comparison acts as a check on the check.

Test the opposite break as well. Receipt-first designs commonly reserve a job identifier and assume its presence means success, without recording whether the effect has been committed. In a simple test fixture, stop after writing the receipt alone. If the second attempt returns a nonexistent result, the test has found an important difference between “a worker made this claim” and “the work is complete.” In a design that intentionally uses reservations, those records need their own pending and terminal semantics, plus a way to reconcile stale reservations without assuming somebody can always restart the original worker. A receipt labeled pending should not be silently converted to complete on read.

Be careful when you add a concurrency check. Let two worker processes approach the same logical job with independent connections and a known starting database state. Verify one final effect and agreement about the retained result for matching inputs. If you get a transient busy or lock outcome, don't treat it as equivalent to either success or a semantic conflict. Your design may have a bounded retry or reconcile path for that specific outcome. Tests should name the difference, and the scenario runner should use a barrier or hook if it needs an exact ordering. Random delays can detect surprising behavior over many repetitions, but one random pass doesn't demonstrate that your intended overlap occurred.

Make the output useful in a code review

Keep the report short enough that another engineer can find the important state. Each case should show the fixture version, the branch and synthetic job identity, the point at which the first worker stopped, whether any success acknowledgment was observed, the receipt and snapshot counts from an independent connection, the result identity from the retry, and the count after replay. A lock or harness error should be recorded separately from an assertion failure. If you have also tested conflicting payloads, include whether the old fingerprint and old result are still intact. Avoid storing raw inputs that resemble actual patron or account data; this exercise is more useful when anyone on the review can inspect it safely.

Say what each case proves and does not prove. “After-commit acknowledgment loss: receipt and one snapshot persisted; identical replay returned the same result in the fixture” is supported by the measurements. “Messages can never duplicate in production” is not. If the real implementation uses multiple durable systems, proves equality differently, or acknowledges asynchronously, name those differences before handing someone the test output. In code review, ask the author to point out the nearest production boundary corresponding to each hook. A fixture whose mapping to the code nobody can identify is documentation of a toy, not regression protection for that worker.

Also check what may affect your conclusions when the real origin of duplicate work is an operator. Does a Retry button preserve the old key or build a whole new job? Do two tools use the same branch namespace? Can a form resend occur with a different normalized payload because it silently filled in today's date? Those questions matter even if the local persistence layer is correct. If you set a key retention window, a repeated delivery after expiration may look new; choose the retention period to match the allowed replay horizon and test the boundary. It is better to expose an explicit “too old to retry automatically” outcome than to promise long-lived deduplication you don't actually maintain.

Draw the box around the experiment

An internal receipt and an internal snapshot can share one SQLite transaction. A file upload or remote email does not suddenly become part of that transaction because it happens in the same function. For work that has an external effect, use a separate design and controlled destination or local sink before attempting to make stronger claims. For example, a fixture can assert one durable outbound intent without asserting an external provider received a message once. A second harness can then simulate acceptance with a missing response. Only a verified destination contract or reconciliation process can tell you what happens in the actual integration. Keep those reports separate. Do not use live bookings, mail, payments, or real account changes as the effect in the tutorial test.

A practical order for the first iteration: build one stable synthetic input, define what counts as the effect, write the control, add before-commit and after-commit hooks, then record independent reads before replay. Add conflicting-payload and competing-worker cases once those two boundaries are understood. If you cannot trust the fixture launch or observe where the interruption took place, report that first rather than expanding the matrix. One honest reproducible trace usually gives a team better guidance than a wide spreadsheet of test names marked green without observable state.

Good retry behavior doesn't mean the worker is never called twice. It means that two honest observers can agree what was completed and what should happen when the first response is missing. In this toy job, the dispatcher can be wrong about whether its call finished, while the database can still preserve a usable answer. The reason to interrupt on both sides of the commit is to make that difference visible. Once the test distinguishes those two cases and can make a known broken design fail, the team is no longer relying on the phrase “we dedupe” to explain what the code will actually do.

Specify the caller's result, not just the worker's rows

Imagine that the fictional county library stores a dated catalog digest for the volunteer repair group. Each accepted job creates one digest and gives the caller a stable reference to it. If a worker returns the number 17, the operator assumes they can later find digest 17. In a broken replay, the worker could look up the original job receipt, reinvoke the calculation anyway, and return a newly computed result without inserting another record. A test that checks only one database row will pass. A caller receiving that different result no longer has an unambiguous receipt, so include what the caller actually receives in the contract.

Use two next-step observations. The first verifies the durable effect as named by the worker's result. The second verifies that the returned receipt refers to that exact effect whenever the original request is replayed. In a synthetic fixture the reference may be an integer, a digest of fixed data, or another opaque local identifier. It needn't be clever. What matters is that after the initial acknowledgement goes missing, a normal retry can answer the same request without inventing a replacement. Distinguish the stable result from operational metadata. A second worker can record a different attempt ID, wall time, or reason for replay without misrepresenting which effect was accepted.

Try a modification to the input that does not preserve intent. The first request asks for a snapshot of one exhibit shelf, and a second request with the same key asks for a different shelf. That should not be silently treated as the same job merely because both happen to have the same count. Establish equality from precisely the parts of the synthetic request that select the effect. The response to a conflict need not reveal details about the earlier request. It can say the submitted key already exists for a different request and leave both the durable effect and earlier receipt unchanged. Keep authorization checks at the ordinary entry point even if the request is a replay. An idempotency key is a way to identify an action, not a credential for retrieving data.

Offer a pair of inputs that are genuinely equivalent under the documented fixture policy. If key order in a JSON object is irrelevant, then a reordered object can still mean the same intended effect. If the order of requested shelf IDs determines the result, then reordering that list may be material. Work through examples before writing arbitrary normalization. Reusing an overly permissive comparison can hide differences on which users rely; requiring raw byte identity can reject harmless representation differences. A good local test compares the application definition of equality, not an assumed generic property of every JSON API or framework.

Keep the measured guarantee narrow. One effect row, one durable receipt, and a stable repeated result establish behavior for this local fixture under its selected database and workload. They do not show that an alert fired only once, that an external system learned about the effect, or that a real backup has the committed rows. If a release note needs a claim about external delivery, test the relevant service or documented stub separately and label its evidence. Being precise about where the test stopped makes it harder for a green counter to be reused as fake proof in another conversation.

Walk the expected traces side by side

A control trace starts with an empty temporary database and one fixed request, then lets the worker complete. The independent checker can observe the row representing the effect and the one receipt row, and the harness observes a successful result whose reference matches. Submit the same request again with the same key and expect the same usable result and unchanged contents. This establishes the basic contract in the absence of interruption. It also makes sure the fixture's inspection connection reads the same file as the worker. A test accidentally pointed at the wrong empty database can draw a very confident conclusion about missing durable writes.

In the first interrupted trace, stop after the effect and receipt have been written in the chosen transaction but before commit. Use the separate observation process as planned in the main fixture. It should not see either as committed, even if the worker had an in-process object that already contained a result. Also record that a successful acknowledgement was not observed. Resubmit the exact request. It is now allowed to perform the modeled effect once; the durable state after replay should match the control state structurally. The final receipt may contain a different attempt time. That is not an equivalence failure about the committed job.

In the second interrupted trace, let commit return and interrupt before the success response is sent. Opening a fresh connection should now reveal both committed rows. The supervisor still lacks success, which is precisely why it asks again. The second delivery should report the already committed result; it should not merely report a generic ok or perform the effect again. Use a fixed identifier or a small independently retrieved reference to demonstrate that the previous snapshot is the one described. The expected contrast is not that one worker happened to print longer logs. It is a durable state distinction before and after the chosen boundary, even though the caller experienced a missing acknowledgement both times.

Add a trace where the wrong key is generated on retry. The worker sees it as a new request, and under the simple fixture policy that request can generate another snapshot. This is not proof that the server's original same-key rule was broken; it reveals a caller boundary that needs its own test. Verify that the dispatcher persists the original key with the original work, and that an operator clicking retry does not silently construct another logical job unless explicitly intending one. A good API implementation cannot deduplicate two independently issued identities it was never told to treat as one.

The failure messages should keep these traces distinguishable. If the before-commit observation sees an effect with no receipt, say which durable reader saw it and note the hook that was reached. If the after-commit observation shows neither row, report that the expected boundary was not actually proven, instead of assuming the retry was safe. If the rows match but the returned result differs, identify the caller-visible mismatch. This gives the next engineer something they can route to the correct layer. A screenshot of a long list of passes can't replace knowing which actor observed what.

Exercise conflicting workers without pretending all timing is deterministic

In a controlled local test, make two workers share one database file but give them independent connections. Both receive the same equivalent logical request. The purpose is not to produce a benchmark or hammer a database with more writes. It is to observe the policy under a deliberately created overlap. Arrange a documented hold point for one process while the second begins. The harness can then control when the first process is allowed to continue and record what the other did. Do not count a transient busy result as a second effect and do not infer that overlapping worker activities imply parallel committed writes in SQLite.

Choose what the second worker is supposed to do. If it waits and then returns the first stable result, record that as the intended local policy. If its bounded wait returns a retryable result instead, show that the eventual explicit replay recovers the original result under the same key. A duplicate suppression test can pass under either documented policy, but only if the harness checks the intended outcome and verifies the durable records. SQLite's transaction documentation distinguishes the mechanics of transaction modes and busy conditions. Use the actual connection configuration in the fixture, since changing it can change when a request waits or returns an error.

Try an overlapping different-input request under the same key. The losing branch may time out, discover a committed receipt, or see a uniqueness constraint failure depending on the chosen test sequence. It still cannot safely turn that into a successful earlier result if the material inputs differ. Arrange the follow-up so the conflict can be observed after at least one accepted durable row exists. Use benign fixture inputs and an isolated directory. You should be able to rerun the experiment from the documented starting state without affecting any other job or adopting an overlap pattern from live traffic.

A second known broken control can expose misleading upsert behavior. The toy worker catches every uniqueness conflict and reports success because a row exists. The identical-request fixture may pass. The different-request fixture should fail because it reported the successful earlier result despite mismatched intent. Another variation replaces the previous receipt with the new payload while preserving the effect row. That still leaves one receipt and one effect, but their relationship has been falsified. Use independent checks of stable result and material input instead of believing a row count can answer who asked for what.

The harness should not unconditionally insist on one exact interleaving where the scheduler isn't under control. Record the stop point and unit of concurrency control precisely. Where a wait is nondeterministic, check invariants across valid outcomes, with an appropriate timeout and diagnostics for unexpected outcomes. Label what your local barrier actually did. For example “worker one entered the instrumented pre-commit pause; worker two returned the documented busy outcome” is much more informative than “two simultaneous jobs worked.” Don't use long arbitrary sleeps to produce a guessed ordering if a simple acknowledgement channel can mark it directly.

Treat failed acknowledgements as observations of a boundary

The worker can know it sent an acknowledgement, while the dispatcher only knows whether it received one. A frame stuck in a local test channel, a forced interruption just after commit, and an acknowledgement missing from a log can look similar to somebody glancing at a dashboard. In the fixture the supervisor can explicitly observe the channel rather than guessing. Write separate fields for intended response, observed response, and durable effect state. Then define when it's appropriate to use the missing-response result to test a replay. No single component has to be endowed with impossible knowledge about what another one actually saw.

Add a synthetic case where an ordinary application error occurs before useful work begins. The runner can report a known rejected request, no acknowledgement of success, and no durable effect. A caller may reasonably avoid retrying it until the input changes. Compare that with the after-commit crash, which lacks success but has a committed effect. Both have no successful response, yet the appropriate next actions differ. A receipt that collapses the missing observations into a string failed creates unnecessary duplicate work and makes the most important race invisible to maintainers.

The report can preserve one ambiguity rather than designing it away. If the parent cannot observe whether the hook was reached, a child exit alone is not proof of which boundary was exercised. Mark the run invalid as a specific test of that boundary and retain whatever evidence exists. Repeat from a fresh fixture under controlled conditions if needed, not from a partly modified state. In a genuine incident, an external timeout often leaves exactly this kind of ambiguity. The point of the synthetic harness is to give the team an example where it can establish ordering and prove the local response despite an intentionally incomplete caller view.

If the model includes a later local outbox, keep the intent and delivery boundary separate in the report. A transaction might prove one durable send intent without performing any send. The worker could then retry handing the intent to a local recording test double. The test double can show what your integration code attempted, and it can model a response that goes missing after it records receipt. Neither establishes what real mail, payment, or booking systems would do. Keep the example entirely local and harmless; the useful idea is comparing different receipts, not demonstrating that other systems can be induced to act twice.

Check what happens when a receipt ages out

An idempotency test run immediately after the first request doesn't establish what happens when the product retains keys for only a defined period. Retention can be deliberate for a small system, but a caller needs to know whether retrying after that period is treated as a new job or as an unresolved older request. Use an injected fixture clock, not changed computer settings, to simulate reasonable boundaries. A retained receipt may point to an older snapshot no longer visible through an ordinary list. If that can happen, define what result the replay returns and whether a stable archived reference or a documented expired result satisfies the product promise.

Set out two synthetic periods. While the receipt is retained, exact replay returns the known reference and leaves the effect unchanged. After an intentional expiration, the system follows a documented policy instead of going back to an accidentally permissive default. In one conservative design the operator reconciles the old effect before explicitly creating new work. In another, the caller is told the key is only meaningful within a stated window and uses some stronger independent lookup later. Duration policy is not a reason silently to change request scope or discard audit material needed for growth, corrections, or disputes. The test just separates the contract as delivered from promises an engineer can no longer verify.

Include one boundary where wall time is not the source of the attempt's safety. A manager replaying a catalogue build may change dates in the job and intentionally request another snapshot. That is different from retrying the original logical job after taking a lunch break. Keep an explicit reference to the original logical operation in the replay action, and let the model treat new jobs as new work. In the review document show which job scope is preserved, which evidence is retained, and the reason for any unknown state. Those details make it possible to improve the runtime design without retroactively pretending older tests guaranteed more than they did.

Keep a fixture receipt somebody else can run

An adequate handoff for the temporary SQLite demonstration should include the read-only inspection statements, the test setup and cleanup scope, and a minimal expected-state table. Retain the actual outcome from the local run separate from the proposed extensions. Contrast “the supplied SQLite experiment observed one committed effect under these recorded cases” with “the article suggests additional test cases for longer retention and competing workers.” The first is a bounded observation, the second is a plan. There is no need to inherit unstated assumptions about the host or test runner to interpret either one.

Decide when a fixture result is inconclusive. Maybe the parent connection reused a stale read, the child never reached the hold point, or the temporary data file couldn't be reopened. Those conditions weaken the test rather than strengthening it just because no duplicate appeared. Preserve useful application diagnostics while taking identifiers and real data out of shared logs before publication. Independence here doesn't mean copying an entire private database to whoever reviews the code. A repeatable local construction can let a reviewer validate the result and its boundaries within minutes.

The most productive long-term change is often a very small one: make the caller's durable key explicit, make the commit and receipt relationship observable, and put three distinct interrupted states into the regression suite. When something changes, a later engineer can use those traces to reconcile what the dispatcher knows with what the database knows. They may still need a different test to evaluate external delivery. That is fine; a fixture worth maintaining is one that admits what part of the story it cannot see.

One final independent assertion compares the successful replay reference with the receipt read through a fresh connection. It should fail if either side silently generates a new reference, even when the two table lengths appear correct. Keep that assertion labeled as a caller contract, not an additional database effect.

Primary references

Frequently asked
What is Test a Retry Where the Worker Can Really Lose Its Place about?
A fictional county library makes a daily inventory snapshot for its volunteer repair crew. A worker receives a job, adds a snapshot record, and acknowledges…
What should you know about specify the effect rather than counting calls?
Before looking at infrastructure, name the outcome that must not duplicate. In the library example, the desired effect is “create one inventory snapshot record for logical job x.” Executing the worker twice is allowed. Logging two starts is allowed. Observing one failed acknowledgment and a second successful one is…
What should you know about keep the receipt and modeled effect in one database transaction?
A fixture starts with two small tables. One stores the durable job receipt with a unique key over branch and job identifier. It stores a request fingerprint and a result identifier. The second stores the toy snapshot, with a stable reference back to the logical job. Production systems may use a single table if the…
What should you know about stop once on each side of the line?
Design the subprocess with explicitly named test hooks. One hook records that both inserts were attempted but stops the worker before it commits. Another hook runs after commit returns but before the worker emits the success acknowledgment. A parent test harness launches the worker with one selected hook, provides…
What should you know about include one intentionally broken design?
A good fixture makes a bad design fail for a recognizable reason. In a local throwaway comparison, split the snapshot write and receipt write into separate committed transactions. Interrupt the worker after the first commit but before the second. The independent connection now sees a snapshot and no receipt. When the…
References & sources
  1. Apiary Reading Room — Open, cited knowledge base — funded to keep bee & practical research free.
From the Apiary Reading Room. Opinion & editorial — not financial advice. We don't overclaim.
More from the Reading Room