AI disclosure: Drafted with AI assistance for Apiary. This guide uses fictional release records and synthetic byte strings; it does not describe any customer's deployment.
A client says the blue button is still there. Your teammate points at Git, where the button turned orange yesterday. The hosting dashboard shows a successful deployment and a preview URL that looks right. The client opens the custom domain and still gets blue. The strongest explanation in the room is often “cache.” Sometimes it is cache. Sometimes the custom domain still points at an earlier deployment, or the preview uses a different build command, or a service worker kept the old script even after the network returned the new document.
You cannot settle this with a screenshot, a branch name, or the last row in CI. You can settle much of it with three independently captured facts: the exact inputs that went into the build, the identity of the uploaded output, and the response a visitor obtained from a specific URL. This article sketches a receipt for a small site that needs to reconcile those facts without standing up a new operations platform.
Separate source identity from output identity
A commit names a version of the tracked source. A branch is a moving reference to a commit. The Git revision documentation explains this distinction and the forms that refer to commits. At build time, record the full resolved commit rather than writing down “main” and trying to resolve it during an incident. If the automation builds a merge preview, record the actual checked-out commit, not just the pull request head somebody expected to build.
Do not let a dirty working tree into a reproducibility claim. A generated logo file, changed package script, or locally untracked asset could affect a build without affecting the commit SHA. The easiest release input to describe is an isolated checkout with a pinned dependency lock, pinned build environment and versioned build command. A local build can still be useful, but label it honestly and record why it is not a clean reproduction.
Even a clean checkout can yield different outputs. Compiler version, build-time environment, minification behavior, timestamps, or external content pulled during compilation can change emitted bytes. Record a digest of the generated artifact or a manifest of generated files. Build A and build B may share the same commit and legitimately have different output digests. Decide which variables belong in the receipt; secret values do not belong in a public manifest or log. For a nonsecret environment distinction, a label like preview or production is often enough.
Use a small manifest you can regenerate from real files
Suppose a fictional art-supply co-op publishes a site with three representative outputs: the HTML document, a script, and an image that anchors the main design. At build time, write a machine-readable map of every file you plan to upload, not just those three. A minimal example for two synthetic files:
from hashlib import sha256
import json
def artifact_manifest(files):
entries = [
{"path": name, "bytes": len(data), "sha256": sha256(data).hexdigest()}
for name, data in sorted(files.items())
]
wire = json.dumps(entries, sort_keys=True, separators=(",", ":")).encode()
return {"entries": entries, "manifestSha256": sha256(wire).hexdigest()}
old = artifact_manifest({"index.html": b'<button>Blue</button>',
"app.js": b'console.log("site");'})
new = artifact_manifest({"index.html": b'<button>Orange</button>',
"app.js": b'console.log("site");'})
assert old["manifestSha256"] != new["manifestSha256"]
assert old["entries"][0] == new["entries"][0] # app.js unchanged
assert old["entries"][1] != new["entries"][1] # HTML changed
Here the digest of each file identifies the exact bytes the builder generated. The manifest digest identifies a deterministic serialization of the sorted list. Treat the serialization as part of the format and version it. Two teams using subtly different JSON serialization should not expect the same manifest digest. Don't include the generated manifest itself recursively. If you publish a public deployment marker, define whether it is intentionally outside the artifact manifest or written into the artifact before the manifest is computed. Otherwise your plan can ask a manifest to include its own hash, an impossible fixed-point expectation for ordinary tooling.
If you package a ZIP, distinguish its digest from the logical content manifest. ZIP metadata or ordering can change while the unpacked files match. An archive digest is useful when the host consumes that exact archive. The content map is useful when the hosting platform uploads individual files or the importer may transform the package. Keep both if they answer separate operational questions. When verifying something that changes bytes by design, like CDN-generated image sizes, compare a stable upstream representation or use an expected transformation contract instead of comparing unrelated byte streams.
Record a deployment receipt, not a celebratory log line
The upload should return a deployment ID or another immutable platform reference. Save it alongside the manifest digest, repository commit, environment, target project, and the time of the deployment. If the host gives you a unique preview URL per version, record it separately from the custom domain. The mapping from a hostname to a deployment may change through a separate promotion operation; capture it as its own event when the host provides an authoritative identifier.
A release receipt can be split into records:
| Record | Its identity | What it establishes |
|---|---|---|
| Build | Commit, build run ID, toolchain, manifest digest | A stated input produced stated outputs |
| Upload | Deployment ID, project, accepted artifact digest when available | The provider accepted or created a deployment |
| Promotion | Custom hostname, target deployment, time, source | Control plane says the hostname was assigned |
| Observation | Requested URL, final URL, time, status, selected response checks | One client saw a particular response |
Do not claim the host verified a digest if you only calculated it locally. Put the evidence source on each row. Those rows can look redundant when everything works. They matter during the exact kind of disagreement that originally motivated the system.
For public software distributed to others, formal provenance may be useful: the SLSA provenance specification describes a build attestation model; GitHub's artifact attestation documentation gives an implementation option. Such an attestation concerns the artifact and its build process. It does not establish that a custom domain currently points at the artifact. Likewise, a marker served with a site can be copied or stale, and is not a substitute for a signed provenance control where one is needed.
Observe what the customer actually sees
Fetch the custom hostname exactly as a customer would, without seeding a logged-in dashboard session. Record whether an HTTP redirect took you to a different host or path. Verify the document and the asset that implements the reported change. If the button's style is in CSS, just checking an HTML build marker misses the part that matters. If the site changes appearance only after JavaScript hydrates, use both the network receipt and a browser check.
At the start, keep observations small:
- Request the reported URL with the same desktop or phone viewport as the report when practical.
- Record the first response status and final URL after redirects; identify whether the document names the expected assets.
- Fetch at least one named asset known to differ between the two releases and compare its digest or expected marker.
- Verify the changed interaction or visual on the page; a fetched script is not evidence that it executed.
- Record whether this is a fresh profile, an existing client session, or both.
A cache-busting query string can be a useful diagnostic, but it is not the same request the customer made. Caches may treat query parameters differently, ignore some of them, or route them separately. Preserve the original observation and report the cache-busted request as a controlled second case. Use headers such as Age, Cache-Control, or provider-specific indicators as context, not as replacement evidence for response content. RFC 9111 defines the standard caching semantics; provider headers are interpreted under the provider's own documentation.
Think about what was hashed. Hashing a compressed transfer stream gives a different result from hashing the decompressed content. If your build digest represents emitted uncompressed files, request or capture the decoded content for a fair comparison. If a CDN transforms content even after decoding, choose an endpoint known to preserve bytes, or narrow your claim to verified content identity by another mechanism. One matching CSS query string identifies the reference, not necessarily the bytes behind the reference. A match on just one asset is not a proof that the whole release matches.
A worked incident, five facts instead of one theory
The co-op has a preview showing the orange button. The approved build receipt records commit C17, manifest M17, and deployment D17. The preview host's HTML and style digest both match M17. The custom domain initially returns the older HTML referring to the older style filename. The custom domain and preview are clearly serving different observed artifacts, regardless of whether a dashboard labels both “latest.”
The first question is where they diverged. The hosting control plane reports the custom domain mapping was never promoted from D16. In that scenario, purging cache is not the fix. The expected release was uploaded and previewed but not promoted. Affected owners can use the saved receipts to approve or execute the actual promotion instead of rebuilding the same commit and hoping it reaches the right place.
Consider a second case: the control plane says D17 is promoted, the custom hostname returns matching D17 HTML and CSS in a fresh profile, but the client's existing tab shows blue. Now local caching or service-worker state is more plausible. Compare that client's network entries and browser cache behavior. The Service Workers specification describes how a service worker may intercept fetches and provide responses. Do not tell every visitor to clear site data blindly; that can affect sessions or local-only work. A test profile and a deliberate update strategy are better ways to investigate.
A third case: a fresh session gets the new HTML and old bytes from the same unversioned style URL. The mixed release is visible. Fingerprinted file paths can help deploy assets without reusing a stale key. Retain old immutable assets until old documents can no longer request them. Deploying new HTML that references absent assets, or deleting last week's assets before cached HTML expires, can trade a stale color for a broken page.
Make precision available without publishing internals
An internal release screen can show commit, build run, manifest digest, deployment ID and promotion time. A public site may need only an opaque build ID. Publish that ID only if it helps diagnosis and doesn't leak private configuration, source paths, unpublished document names, or internal references. It need not reveal your private Git repository. A build ID should remain stable for a given output and be easy for support to map to the internal receipt.
Define vocabulary for team handoffs. “Committed” means the change exists in Git. “Built” means an artifact has a recorded identity. “Uploaded” means the host accepted the artifact. “Promoted” means the specified domain was assigned to the deployment according to a reliable control-plane record. “Verified on the domain” means a direct observation checked the expected output and behavior and names when that was done. These states can appear on one line, but do not combine them into an unqualified “done.”
Keep the inspection log append-only when practical. A second verification from another device can be added, not used to overwrite the first. If the checks disagree across regions, time windows, or client types, preserve the disagreement. A single receipt is a sample, not a claim about every visitor. More observations should narrow the uncertainty, not conceal it.
Build the receipt to admit uncertainty
Treat a partially observed release like a partially reconciled invoice. If the platform does not return an accepted digest, keep that field empty with an explicit reason; do not copy the local artifact digest into a field labeled “host verified.” If you cannot read a protected preview, a note from its builder may be useful context, but it is not independent browser evidence. If an observation reaches one host and then follows a redirect to another, store both URLs. A screenshot of the final site cannot retroactively verify the host that was supposed to serve it.
Here's what a sharply scoped status note looks like: “D17 was accepted on the preview project. At 10:12, a signed-out visit to the custom domain returned the D16 style path. The current promotion record is unavailable, so we haven't distinguished an old domain assignment from a cached response.” Another teammate knows exactly what to check without having to interrogate the person who posted it. The note is also honest if the failure resolves during investigation.
An overlooked failure mode is a release marker that is always “current” because it is loaded dynamically from the dashboard. If the old page includes a script that requests the newest ID from a shared endpoint, the old page will happily claim it is new. Bind the marker to the bytes it describes, and serve the inspection endpoint under the same release lifecycle or document its separate role. During the isolated mismatch rehearsal, verify the marker stays old alongside the old page.
When a provider rolls back to the previous artifact, keep its old asset hashes but add a new promotion event. The bytes are familiar; the action is new. This distinction makes it possible to explain why last week's artifact is being served today without pretending it was rebuilt today.
Make the expected output unambiguous before investigating
Start the incident with a written expectation. In the fictional co-op, the orange button belongs to the holiday-hours page for ordinary signed-out visitors, not to a retained alternate menu, a login preview, or a staff mode. The ticket should name the exact pathname, link used to reach it, display or behavior expected, and what release is meant to contain that change. When the report comes only with an image, ask just enough to identify where on the site it happened. An address and rough local time are generally more valuable than a detailed discussion of what the image looks like. Keep private browsing data outside the ordinary evidence handoff.
Some differences are approved rather than evidence of failure. A rendering might show one theme for dark mode and another for light mode, a static feature toggle might intentionally enable the new button only on a second route, or an alternate locale might choose a different stylesheet. In those cases publish the expected variants or at least save them in the issue. A single assertion that a whole domain is entirely orange hides the business rule. Scope the check so it does not accidentally make people “fix” the old button in a visitor mode where it is still intentional.
Separate visitor identity and representation from deployment identity. A public URL can legitimately vary the payload by language, version of the client, or a documented experimental feature. A check that validates one sample does not explain every variant. Record the relevant public request shape and the route decision for controlled fixtures rather than accumulating unnecessary headers from every visitor. If an existing product has intentional experimentation, ask which cohort or fixture is supposed to receive the changed behavior before interpreting a discrepancy as an old build. A deploy receipt remains useful even when two legitimate representations exist.
For the co-op, build a tiny expected-output sheet. The approvals lane records C17, M17, and D17. The primary verification lane identifies the holiday-hours route, an approved orange button DOM marker, and the named CSS asset expected to contain a selected rule. A second lane identifies the older design that is still explicitly accepted on the print-only page. A third lane sets aside logged-in preview because its intended state differs and the report involved a signed-out user. This sheet is not more paperwork than an incident already creates. It replaces a series of unrecorded guesses about what is supposed to happen.
In reviews it is easy to confuse names and proofs. A deployment called latest-production may have a reassuring name and no immutable identity. A file called app-final.css may be reused. A release note saying the holidays site is ready may mean somebody verified a preview. Translate each label into its actual evidence before gathering more data. If the expected deployment is still unknown, you can record the observed response identity without making a claim that it is outdated. That limit is useful. It tells the maintainer to recover the promotion decision rather than to repeat the public GET from another machine.
Follow a single incident through four independent receipts
Imagine you take over the co-op ticket at 10:20. The signed-out reported page returned blue at 10:12. The checkout wasn't changed afterward. First collect the approved build receipt. It says the output manifest includes the holiday-hours document and two styles, of which one is the asset associated with the button. Record whether the manifest includes root path and nested path outputs separately. If it covers only the homepage, ask for the receipt or content hash that describes the actual reported route. This is a valid clarification even if the branch name and release ID are known. A matching homepage can coexist with a stale nested page.
Next collect the upload receipt and check project identity. It is possible to upload a valid site to a similarly named preview or testing project and then promote a different deployment on the public project. The question is not only whether D17 exists, but whether it exists within the project and environment that can serve this route. A provider identifier should be carried alongside the environment name used by humans. A reproducible case can compare two fictional projects with the same nickname and intentionally different output files. The expected result is a project-mismatch gate, not an attempt to reconcile unrelated timestamps.
Then review the recorded promotion. A public domain pointing to a project does not by itself identify which version is active. Some hosts support a single selected version; some intentionally distribute traffic among versions; some route specific paths separately. For the controlled example, choose one public assignment and no experiment. If your real system uses deliberate split traffic, record the expected selection rule and version set. Two released manifests may both be valid during a staged rollout. Don't try to convert a split delivery into “fully promoted” until the verified control plane and sample observations support that statement.
Finally collect a fresh public observation from the actual path with its ordinary request. Record the served title or small content markers and the linked asset identity, plus when it was seen. Give this observation its own row, not a note inside the upload receipt. Then use the observed response to pick the narrowest next check. If the observed HTML and style both match M16, inspect the promotion assignment if it's unavailable. If the HTML matches M17 but the named style bytes match M16, investigate how assets were referenced and retained. If they both match M17 and the visual still appears blue, move up to rendering: a selector override, disabled stylesheet, or unrelated user theme is now a stronger hypothesis than rerunning the build.
The worked example needs one case where the answer is still unknown. Suppose CI has lost the manifest file. The host shows D17 accepted and the public button is orange. The confidence statement can say the visitor-facing visual matches the expected change and the observed styles were recorded, while cryptographic output mapping remains unverified. An engineer might rebuild C17 to attempt to recover a comparable manifest. Until reproducibility is established, label that as a reconstruction. A rebuild is not the original uploaded artifact just because it arrived from the same source revision.
Specify what a byte comparison can and cannot prove
A useful manifest makes the comparison operation explicit. The digest calculation should name which object and what bytes entered it. For a file produced by a local renderer, it can be the raw file bytes on disk. For an HTTP download, it can be the decoded representation after transport content coding, still before a browser applies it. For an image transformation, matching the original build image byte-for-byte may be deliberately impossible at the served URL. A receipt should not initialize every byte mismatch as failure unless bytes are actually expected to remain invariant. Name important stable representations rather than making the matrix less meaningful with a catch-all normalization method.
Consider a synthetic CSS response. Its logical output is a stylesheet with a line setting the button color. The build writes that file unchanged into M17. The public host may transmit compressed bytes while the test client decodes the body. Save which of these was hashed. If another person captures the compression layer rather than the decoded bytes, their different hash doesn't necessarily mean a different style reached the client. An accepted outcome could instead use an externally documented standard or known verifier. Do not put a decoded-file hash and a compressed network hash into the same column and ask a later script to compare them blindly.
Generated markup can be harder. Some services stamp a document with a request ID, a freshness token, a changing list, or personalized navigation. If the public HTML is dynamic, an exact digest may report a difference for each visitor even when they all received the same version of the actual app. Prefer invariants with an honest scope: a deployment-bound marker and a named immutable asset under the same navigation, an expected structure confirmed in the intended route, or a specific build output observable under a controlled preview. Save an exact capture for comparison when its public content is appropriate, but don't claim seed-dependent HTML proves an entire release map.
Choose an intentionally narrow comparison rule for each observable. Full file SHA-256 can show that a target asset equals the expected artifact bytes. A normalized screenshot can show the button rendered in the expected spot under an agreed environment. Parsing a style declaration can show that the document refers to an expected file and contains the selected property. Those checks are complementary. A hash cannot establish that a browser executed a script; a screenshot cannot certify exact bytes. Do not mix their evidence scores into a single percentage. A release report can say which checks passed and where observations differ.
If you compare directory inventories, also decide path rules before serializing. An unordered archive can contain duplicate entry names or questionable path spellings. A manifest should list a unique canonical path according to the build format and fail when it cannot identify which file would be uploaded. Exclude generated review files unless they are intended output. Write the manifest outside the gathered file set or emit it by a tested procedure that doesn't accidentally hash itself. Don't copy private source repo inventory into the public directory. These rules improve stability, but the number that matters is still the manifest of output actually intended for the host, not a digest of an unrelated checkout.
Test a controlled mixed deployment without publishing it
A small, isolated rehearsal can teach the whole comparison. Generate two versions of a static fictional page. Version blue references an older style filename and contains an old marker. Version orange references a new name and a changed marker. A toy preview setup can deliberately serve the blue document and assets, then orange document plus the matching style, then an orange document whose referenced style bytes are intentionally wrong. In each case hold names and content statements clear enough that a reviewer can recreate what happened without reaching a deployed host. Don't simulate by altering somebody else's asset or intercepting real users.
The ideal failure messages differ. For blue throughout, the probe says the document and style are an older coherent output. For orange with blue CSS, it says artifact verification failed only for the named style, while the document matched. For orange HTML declaring a new style that returns a 404 or an HTML fallback, it says the relevant response is absent or of the wrong type. That isn't old CSS, and a different team might have to address it. Design asserts against status, media type, and child bytes separately so one confusing kind of mismatch does not mask another.
Try a fourth fixture in which both local files match orange but a rule linked later changes the display. In a normal browser, inspect the rendered result and if appropriate the computed style for that one fixture button. This tests the report's difference between bytes delivered and function demonstrated. Give this case its own failure class rather than reassigning it to the deployment bucket. It also catches a simplistic check that just searches response text for the word orange. That test can pass if orange is present in a comment, an overridden declaration, or another element.
Keep a timeline. Fetch the direct HTML first, then whichever asset the returned document actually references. If you instead request the version you hoped the document would name, you learn that the desired asset is online, not that this page used it. If a deployment changes between those observations, the raw results are still evidence of two requests, but they may no longer describe a single consistent visit. A browser network recording from a controlled fixture can help tie the two together. Mark time precisely enough to note a concurrent version change instead of using an inconsistent trace as proof of broken asset routing.
A minimal handoff for the rehearsal includes the input marker mapping, manifest hashes, request mode, observed outputs, and expected classification for each case. When the check catches the deliberate mismatch, it gives evidence that the test detects that category of failure in the lab. It is not proof that the host will never serve a mixed version after an unrelated change. Run it as a small release gate and retain a real signed-out sample with the same field names, so a future operator knows which inferences are supported.
Report contradictory edge and client observations cautiously
In an actual incident a human may observe success and another may still see the wrong UI. An evidence record can hold both without forcing either person to prove the other wrong. Observe the same route at two known times. Label fresh context, warm context, and any documented variant selector. If one observer follows redirect to another hostname, note it explicitly rather than attributing all differences to region. On a platform that intentionally caches each region differently, sample only the authorized relevant regions where feasible. A single passing sample is enough to report what it saw, not enough to promise global coherence.
There is also a cost to treating cache controls as magic. A negative or old response might be cached by different layers independently, and a public request with altered headers may go down a different path. To understand a single user's report, start with their ordinary path. Then add one controlled less-cached sample if the product and read authorization permit. When a fresh tab behaves differently from a long-open app, inspect the page lifecycle or service-worker evidence using a test profile. Logging somebody out or resetting all their stored state is not an investigative prerequisite. A failing checked reproduction gives the team more information than a string of blanket reset instructions.
Separate temporal ordering from identity. A record reports that promotion was accepted at 10:05 and a customer observed blue at 10:12. That order alone does not tell you why. The custom route might point at a fallback version, an old cached object might remain eligible, or another independent release might have happened between. Query the administrative deployment timeline where authorized and correlate its states with the public observation. If historic assignment can't be queried, preserve that limitation and avoid looking for a culprit in who opened a browser last. Traceable identity beats a list of time coincidences.
The co-op can close the incident when its agreed scope is restored and the evidence supports closure. It might say: “At 10:39 two fresh signed-out visits to the holiday-hours page returned the expected build marker, requested the expected immutable style, and showed the intended button; one earlier warm profile remains unverified.” This lets a human decide whether to do another targeted check without dismissing the original report. It also prevents the word fixed from being mistaken for a claim about every edge, customer, or session. Store the previous conflicting snapshot with the closure, since that is often the fastest starting point if a similar discrepancy returns.
Keep the rollback version and the evidence version separate
If an authorized operator rolls back, their job is to restore a specifically identified deployment, not to erase the incident. Add a rollback event to the same log and run the exact visitor-facing checks that made the original incident visible. A receipt that says D16 was selected does not prove D16's assets are still accessible if storage or routing changed independently. Before promoting D17, rehearse the supported return to the prior version in a controlled preview and note any external dependencies the platform will not rewind. You do not need to perform a production rollback solely for the sake of the rehearsal. The test is a statement about the available recovery path, not proof that it was used.
Someone reviewing an old screenshot should be able to tell whether its timestamp falls before or after the recovery event, while knowing clock time alone doesn't prove which version served it. Keep the bytes-matched artifact field and the reason for the promotion as separate facts. A later event can select the same digest again without creating a new build. Conversely creating a new D18 from the same source to restore a missing style creates a different output claim to verify. Distinguishing those possibilities keeps a stable issue history when an intervention succeeds and engineers are tempted to rewrite the report as if the original problem never happened.
Set an explicit recovery acceptance criterion for the affected route, a small number of critical assets, and any preserved audience variant. If the goal was the old coherent page while the orange bug is repaired, don't report the orange feature as released. State that the prior presentation was verified as recovery and the intended feature is pending. A trustworthy receipt keeps those two outcomes legible even when they share one ticket.
Apply it to the next ordinary release
Choose one project with a public custom domain and a separate preview or hosting deployment reference. Make a preflight manifest in the build workflow. For the first run, store it in CI artifacts and do not change public response headers. Save a one-page receipt from the next deployment. Ask a teammate to look up the domain independently using the receipt rather than asking the builder what they clicked.
Next, rehearse a known mismatch in an isolated preview: build the orange version, intentionally leave the domain or staging alias on blue, and confirm your receipt shows where the versions diverge. Then rehearse the opposite: serving the expected HTML while a critical asset remains old. You want separate failures with distinct messages. If both become “deployment failed,” the tool will probably drive the same bad troubleshooting loop under pressure.
Decide a modest retention period so support can compare a report with the preceding output. Keep the manifest and necessary receipts; whether to retain all binaries is a storage and policy decision. Make sure you retain or can regenerate a rollback artifact according to your own release rules. Copying the whole project to a developer's laptop is unnecessary if an approved remote artifact store already does that job.
When your next change goes out, the earned result is a sentence you can defend: “I observed the orange release on this URL at this time; the document and style matched this manifest, and the button worked in this client.” That sentence is more useful than “Git says it shipped.” It also states the limits, so the next person knows where to look if another screen still shows blue.