Evidence checked October 31, 2026. The engineering papers cited are dated research, not current price quotations. All monetary figures and workload calculations below are illustrative. This article offers a measurement framework, not financial advice or a recommendation to buy particular hardware.
The cheap answer that required three attempts
A service charges very little for one generated answer. That sounds like a simple economic proposition until the answer is incomplete, the request is retried, and an employee spends several minutes correcting the result. Another service charges more for the initial request but completes the job reliably. Which is cheaper depends on the unit being purchased: generated tokens, attempted requests, accepted answers, or completed work.
The technical infrastructure has a similar ambiguity. A server can generate many tokens per second while making individual users wait too long. A model can fit into less memory while running more slowly on an unsuitable kernel. A cache can reduce repeated computation but provide little benefit for a workload whose prompts rarely repeat. These are not contradictions. They are different measurements of different boundaries.
Research on inference serving makes those boundaries visible. The DistServe paper, submitted in January 2024 and revised in June, distinguishes the processing of an input prompt from the subsequent generation phase and studies performance under separate latency requirements. Its results belong to the tested configurations, but its framing highlights why one throughput number cannot describe every user's experience. DistServe paper.
For a public institution or small organization considering AI, the useful question is not simply the advertised cost per million tokens. It is the cost of completing a specified task to an acceptable standard within an acceptable time. That requires a workload description, a system boundary, a quality criterion, and a record of failures. Without those, a lower price can be real while the claimed savings remain unproven.
Begin with the unit of useful work
The first accounting decision is what counts as one completed task. For a summarization service, it might be an accepted summary of a document within a defined length and factual standard. For a coding assistant, it might be a change that passes specified tests and human review. For a classification system, it might be a correctly processed record with uncertainty handled according to policy. The definition should be written before comparing costs.
A token is useful for measuring text processed by a model, but it is not a universal unit of value. Different tokenizers can split the same text differently. Different languages and formats can produce different token counts for comparable meaning. One model may answer concisely while another produces a long explanation. A cost comparison based only on output tokens can reward verbosity or conceal a model's need for more attempts.
The article's proposed denominator is accepted tasks, accompanied by attempted tasks and the criteria for acceptance. This does not imply that every application needs human review of every answer. Acceptance can be established through appropriate automated tests, expert sampling, or a combination, depending on the task. What matters is that the denominator reflects the result the organization actually needs.
Keep rejected and abandoned requests in the numerator. If a request consumes resources but produces no usable result, those resources still belong to the service's cost. Excluding failures makes an unreliable system look artificially efficient. Likewise, record work transferred to users. A service that is cheap only because employees perform unpaid cleanup has not eliminated that work; it has moved it outside the visible invoice.
Separate the bill from the underlying resource cost
An API invoice and the provider's resource consumption are related but different quantities. A provider may charge different rates for input, output, cached input, priority service, or batch processing. Those rates reflect a commercial offering, not a direct meter of every hardware operation. A customer can measure what it pays without claiming to know the provider's exact internal cost.
For a self-hosted system, the boundary changes. The organization may pay for rented accelerators, storage, networking, software support, engineering time, and idle capacity. If it owns equipment, an allocation method is needed for acquisition and maintenance. A comparison between an API and self-hosting should not count only the API invoice on one side and only electricity on the other. That would compare different portions of the service.
This article deliberately avoids current vendor price tables because rates and product terms change. The durable method is to record the actual tariff or rental agreement used in the calculation, its date, applicable discounts, and any minimum commitments. A result should state whether it represents marginal cost for an additional task or average cost including the capacity kept available to serve it.
A hypothetical server costing ten units per hour and processing one thousand accepted tasks has an allocated server cost of one hundredth of a unit per task for that hour. If it processes only one hundred, the allocated cost becomes one tenth. The hardware did not become ten times more expensive. Utilization changed. That arithmetic explains why a benchmark run at sustained saturation can look unlike a real service with quiet nights and bursty daytime demand.
Follow the request through two different phases
An autoregressive language-model request typically includes input processing followed by successive output generation. The input phase can exploit parallel work across the prompt. Generation must account for the sequence already produced as it advances. The exact bottlenecks depend on architecture, batch size, hardware, and implementation, so labels such as compute-bound and memory-bound should be treated as workload descriptions rather than permanent laws.
Sarathi-Serve, introduced in March 2024, studies the throughput-latency trade-off using chunked prompt processing and scheduling that seeks to avoid stalls in ongoing generation. Its paper demonstrates that how requests are interleaved can materially affect serving performance. The reported gains are specific experimental results, not a multiplier that can be applied to every installation. Sarathi-Serve paper.
A hypothetical document-summary workload may have long inputs and short outputs. A conversational drafting workload may have modest inputs and long outputs. Even if their total token counts are equal, their resource patterns can differ. A pricing model that treats all tokens alike may therefore obscure the engineering reasons one workload is harder to serve.
The evaluation should preserve the joint distribution of input and output lengths, not merely their separate averages. Long inputs may be associated with long outputs, and those combinations can stress memory and scheduling differently. A synthetic workload that pairs every long input with a short answer may miss the difficult cases found in production.
For reporting, distinguish time until the first visible token, the rate of subsequent generation, and time until a usable final result. These measures answer different questions. A quick opening phrase can improve perceived responsiveness while the actual requested answer remains far away. A user waiting for a completed report may care much more about the final completion time than the first word.
Memory capacity changes the number of simultaneous requests
The PagedAttention paper, submitted in September 2023, addresses memory management for the key-value cache used during language-model serving. It describes how fragmentation and duplication can waste memory and limit batching, then proposes a paging-inspired approach implemented in vLLM. The paper's performance comparisons concern its tested systems and workloads. PagedAttention paper.
The accounting implication is that memory is not only a question of whether model weights fit. Requests also require state, and the amount can grow with sequence length and concurrency. A model that loads successfully on a device may still leave too little room for the desired number of simultaneous users. A demonstration with one short request therefore cannot establish production capacity.
Consider a hypothetical service with many long conversations. Each active request may retain state while waiting for further generation. If the service must reduce concurrency to stay within memory limits, throughput and queueing can change. If it moves state elsewhere, transfer overhead may appear. If it recomputes state, additional work enters the cost. The choices should be measured in the deployed configuration rather than hidden behind the statement that the model “runs locally.”
A useful test records peak memory, steady-state memory, maximum admitted concurrency, and behavior near the limit. Does the system reject requests, queue them, shorten context, or fail? Those outcomes affect both cost and user experience. A graceful rejection may be preferable to an unpredictable crash, but it still reduces completed work.
Memory efficiency can increase the amount of work served on the same hardware. It does not automatically reduce every task's latency, and it does not remove the need to evaluate quality. The relevant claim is the measured effect under a specified workload, including the conditions where the optimization provides little benefit.
Moving bytes can matter as much as counting arithmetic
FlashAttention, first submitted in May 2022, studies attention computation with explicit attention to movement between levels of GPU memory. Its exact-attention approach uses tiling to reduce memory traffic. The original paper includes training experiments; those results should not be casually repackaged as a current inference speedup for an unrelated service. FlashAttention paper.
The broader engineering lesson is that an operation count does not fully determine elapsed time. Two implementations can perform mathematically equivalent work while moving different amounts of data or using hardware differently. A theoretical reduction in arithmetic may fail to improve wall-clock performance if it introduces expensive transfers, synchronization, or poorly supported operations.
For a cost comparison, record the actual software path. Model architecture, numerical format, kernel implementation, compiler settings, and hardware generation can interact. A low-bit model using an inefficient fallback may perform differently from the same nominal format using a specialized kernel. The label on the model file is insufficient to establish the realized speed.
A reproducible report should include software versions and relevant configuration, but it should also measure end-to-end behavior. A kernel microbenchmark can explain a mechanism while excluding tokenization, network handling, scheduling, and output processing. Those exclusions are acceptable if stated. They become misleading when the microbenchmark is presented as the cost of a complete user request.
The article's recommendation is to retain both levels: component measurements for diagnosis and whole-request measurements for purchasing decisions. If an optimization improves one kernel but barely changes accepted-task cost, the result is still scientifically interesting. It simply answers a narrower question. Good engineering does not require every improvement to become a dramatic claim about the entire service.
Batching trades waiting against shared work
Batching groups requests so that hardware can process work more efficiently. The benefit depends on the request mix and the scheduling policy. A service can wait for more requests to form a larger batch, but that waiting contributes to latency. Alternatively, it can launch small batches quickly and accept lower utilization. Neither choice is universally correct.
A hypothetical overnight document-processing job may tolerate minutes of delay in exchange for lower unit cost. A live captioning or interactive support task may not. Comparing those workloads under one throughput metric hides the service requirement that drives the scheduling decision. The evaluator should state whether requests arrive all at once, at a fixed rate, or according to a recorded traffic pattern.
Output lengths can also create imbalance. If one request generates a very long answer while others finish quickly, the scheduler's handling of departing and arriving requests affects efficiency. A fixed batch of identical lengths can flatter a system that struggles with realistic variation. Conversely, a carefully chosen adversarial mix can make an otherwise suitable system look poor. The workload should be justified by the intended use.
Report performance across load levels rather than only at maximum throughput. At low load, waiting to batch may dominate. Near capacity, queueing may grow sharply. Beyond capacity, failures or deadline misses may increase. The point at which a system stops meeting its service objective is more useful for planning than the largest token rate it can briefly produce.
Cost should be calculated at the operating point the organization can actually use. A saturated server generating impressive throughput while missing most deadlines does not deliver the same service as a less busy server meeting them. This is why raw throughput needs a quality-of-service companion rather than being treated as the final answer.
Goodput asks whether the result arrived in time
Throughput counts completed work per unit time under a stated definition. Goodput adds a constraint: the work must satisfy relevant service requirements. In language-model serving, those requirements may include time to first token and subsequent generation speed. DistServe uses this framing to evaluate serving performance under latency constraints. The article's own extension is to include task acceptance as another explicitly reported condition, not to imply that the paper measured every form of application quality.
Imagine a hypothetical system completing one hundred requests per minute, but only sixty arrive before a deadline. Another completes eighty, with seventy-five on time. If the deadline is essential, the second system may provide more usable capacity despite lower raw throughput. The comparison changes again if many of the timely answers are incorrect. The metric must match the service promise.
Avoid using averages alone. A mean response time can look acceptable while a minority of users experience very long waits. Report appropriate percentiles and the fraction of requests meeting the defined objective. Also identify whether failures and timeouts are included. Dropping unfinished requests from the latency calculation can make a struggling service appear faster.
The arrival process matters. Bursts can create queues even when average demand is below average capacity. An evaluation that submits the next request only after the previous one completes may conceal this behavior. A separate load generator can reveal what happens when demand arrives independently of service completion, provided the test remains within authorized infrastructure limits.
The economic interpretation is concrete: capacity purchased to satisfy a deadline differs from capacity purchased for unconstrained background work. A cost estimate should state which one it measures. Otherwise, an organization may budget for a cheap batch service and expect the responsiveness of a much more heavily provisioned interactive system.
Cache savings require a repeatable opportunity
Caching can avoid some repeated work when requests share reusable content or state. But a cache's benefit depends on what is reused, how long it remains available, how requests are routed, and whether the repeated material is identical in the relevant representation. A headline cache-hit rate without a definition can combine very different mechanisms.
For this article's proposed accounting, distinguish a cached final answer from reused model-computation state. A final-answer cache may bypass generation entirely for a repeated query, but it raises questions about freshness and whether the user's context is truly equivalent. Reusing a common prompt prefix still requires processing the new portion and generating an answer. These are different savings and should not be pooled without explanation.
A hypothetical service might have a long common policy instruction followed by a short user request. Reuse could be substantial if the common prefix remains stable and the serving system supports it. Another service may process unrelated documents every time, yielding little reusable work. Applying the first workload's discount to the second would overstate savings.
Test cold and warm conditions separately. A benchmark that begins with a fully populated cache may represent steady operation but not startup, failover, or infrequent usage. Conversely, a cold-only test can understate the benefit for a stable high-volume workload. Report both and explain the expected mixture in production.
Cache invalidation belongs in the cost model. When a policy document changes, old cached results may no longer be appropriate. Refreshing or discarding state can consume resources and temporarily reduce performance. That is not an argument against caching. It is a reason to measure the complete lifecycle rather than treating a successful warm-cache demonstration as a permanent property of every future request.
Quantization changes a configuration, not merely a file size
The AWQ paper, first submitted in June 2023 and showing an April 2026 revision when checked, presents activation-aware weight quantization and an associated inference framework. It studies low-bit weight representation and hardware-aware implementation. Its reported quality and speed results are evidence for the evaluated conditions, not a guarantee that all tasks preserve accuracy under every quantized format. AWQ paper.
Quantization can reduce storage and memory traffic, but the full configuration includes more than the bit width. Weights, activations, and cached state may use different formats. Scaling metadata and conversion operations add overhead. Hardware support and kernel quality determine whether the smaller representation translates into a faster service. A claim that a model is “four bit” leaves many of these details unresolved.
The relevant comparison pairs cost with task quality. A hypothetical configuration that is cheaper per request but fails more often can require retries or human correction. A small average accuracy change may also conceal a larger decline on a critical subset. The evaluation should examine the actual task distribution and any constraints that matter for use.
Calibration data should be documented where the method uses it. If it closely resembles the evaluation set, the comparison may not represent broader operation. If it differs sharply from deployment, quality may shift. This is a measurement question, not a reason to reject quantization categorically.
A useful report identifies the exact model artifact, numerical formats, calibration procedure, runtime, hardware, and acceptance results. It then states the measured cost at the required service level. That allows a reader to distinguish a storage achievement, a kernel achievement, and an application-level saving. All three can be valuable, but they are not interchangeable claims.
Speculation can reduce delay while adding work
Leviathan and colleagues' speculative-decoding paper, submitted in November 2022 and revised in May 2023, describes using a more efficient approximation model to propose tokens that a larger model verifies through a sampling procedure designed to preserve the target distribution. The reported acceleration is specific to its experiments. Speculative decoding paper.
The accounting subtlety is that faster completion need not mean fewer total operations across all components. A draft model performs work, some proposed tokens may be rejected, and verification has a cost. The speed benefit depends on how much useful progress is accepted and how the hardware handles the additional computation. Measuring only the target model's visible generation steps would miss part of the system.
For a hypothetical workload with highly predictable continuations, many draft tokens might be accepted. For a different workload requiring unusual terminology or complex decisions, acceptance could be lower. A benchmark should therefore report the task mix and relevant acceptance behavior, not assume one speedup transfers unchanged to every domain.
If the method is intended to preserve an output distribution, the implementation still needs validation. Numerical differences, altered sampling settings, or additional heuristics can change behavior. A product described as using speculation may not implement precisely the algorithm studied in a paper. The evaluator should distinguish the research method from the deployed configuration.
The economic question is whether the complete system delivers accepted tasks more efficiently under the relevant latency requirement. Extra work can be worthwhile if it reduces expensive waiting or allows better use of hardware. Conversely, a modest latency improvement may not justify added complexity for a background job. The decision follows from measured service needs rather than from the appealing word “faster” in isolation.
Long context can shift work instead of eliminating it
A large context window can simplify an application's design by allowing more material to be included directly. But the ability to accept a long prompt does not establish that using the maximum length is economical or improves the answer. The service may spend resources processing irrelevant material, and the resulting output still needs evaluation for completeness and correctness.
A hypothetical research assistant could receive an entire archive for every question, or it could retrieve a smaller set of relevant passages. The first approach avoids some retrieval engineering but increases input work. The second adds indexing, search, and selection costs while risking omission of important evidence. A fair comparison counts both pipelines and measures answer quality, rather than treating retrieval as free or long-context processing as universally superior.
Repeated conversations can also accumulate history. If every turn resends the entire exchange, the input volume may grow even when each new user message is short. Some systems reuse state or summarize history; those choices alter cost and potentially information quality. The experiment should record the actual context passed at each step, not estimate input from the newest message alone.
Truncation is another hidden boundary. A cheaper configuration may silently omit older material or shorten documents. That can reduce cost while changing the task. If the comparison does not record truncation, an apparent efficiency improvement may simply reflect doing less work. The acceptance test should include cases where omitted information matters.
The practical recommendation is to optimize context against the task, then measure. Report the distribution of supplied context, retained context, retrieved passages, and output length. A maximum window size is a capability limit; it is not a workload description or a promise of economical performance at that limit.
Retries create a branching bill
A user-visible task may involve several model calls: classification, retrieval planning, drafting, checking, and revision. Some calls may run in parallel; others depend on previous results. A cost model that counts only the final answer omits the workflow that produced it. The correct unit is the complete task graph, including failed branches that consumed resources.
Consider a hypothetical system that generates three candidate answers, evaluates them, and presents one. The user sees one response, but the service has paid for multiple generations and selection work. That may improve quality enough to be worthwhile. The point is to count the full process rather than compare its final token count with a single-pass baseline.
Automatic retries deserve separate labels. A retry caused by a temporary service error differs from a retry caused by an incorrect answer or an invalid output format. Both cost resources, but they suggest different improvements. One may require infrastructure reliability; the other may require model or prompt changes. Grouping them under “overhead” can conceal the actual failure mechanism.
The proposed measurement record assigns a task identifier to all related calls and records why each additional call occurred. It should also track termination conditions. An open-ended repair loop can create unpredictable cost, especially when the system repeatedly fails the same check. A bounded policy makes the trade-off observable: how often does the system succeed within the allowed budget, and what happens when it does not?
Human escalation is part of that graph too. If a difficult task is handed to a person after several failed attempts, the cost includes the attempts and the handoff. Reporting only the machine portion can be useful for engineering, but it should not be described as the total cost of completing the user's work.
A worked example with explicit boundaries
Take a purely hypothetical monthly workload of ten thousand document tasks. Service A charges an average of two hundredths of a monetary unit per attempted task and needs an average of one and a half attempts for each task submitted. Its model-call charge is therefore three hundred units: ten thousand multiplied by one and a half, multiplied by 0.02. This calculation does not yet establish how many tasks were accepted.
Suppose nine thousand tasks are accepted automatically and one thousand require manual completion. If manual handling costs one unit per affected task, the combined illustrative total is thirteen hundred units before any other overhead. Dividing by ten thousand completed tasks gives 0.13 units per task under these assumptions. Dividing only the model-call charge by accepted automated tasks would answer a different question.
Service B charges four hundredths per attempted task and averages one attempt. Its model-call charge is four hundred units. If only two hundred tasks require manual completion at the same assumed unit cost, the combined total is six hundred units, or 0.06 per completed task. In this constructed example, the higher per-attempt price produces the lower completion cost. No real provider or measured success rate is implied.
The calculation is deliberately simple, but it exposes the missing variables: attempts, acceptance, manual work, and the denominator. A real analysis should add relevant infrastructure, integration, monitoring, and support costs without double-counting items already included in the tariff. It should also test whether the manual-cost assumption is realistic.
Sensitivity analysis is essential. If manual handling is nearly free or both services have similar acceptance rates, the conclusion can change. A responsible report shows the assumptions that drive the result instead of presenting one illustrative total as an inevitable economic law.
Tail behavior can dominate a practical budget
Average request cost is useful, but a small number of unusually large tasks can consume a substantial share of resources. Long documents, repeated repairs, oversized outputs, and slow tool calls may create a heavy tail. An organization should inspect the distribution rather than assuming the typical request represents the whole month.
For a hypothetical service, ninety percent of tasks might be short and inexpensive, while the remaining ten percent require extensive processing. A benchmark made entirely of typical short tasks would understate capacity needs. A benchmark made entirely of the hardest tasks would overstate average cost. The workload should preserve the intended mixture and report the contribution of each group.
Limits can control exposure, but they change service behavior. A maximum output length may prevent runaway generation while cutting off a valid long answer. A maximum number of retries may stabilize spending while increasing escalation. These policies should be evaluated as part of the product, not treated as invisible accounting safeguards.
The budget should also distinguish predictable peaks from exceptional events. A monthly reporting deadline may create a recurring burst that requires extra capacity. A one-time migration may justify temporary batch resources. Averaging both into a single annual number can obscure why the system needs its chosen architecture.
The article's proposed report includes median cost, relevant upper percentiles, total cost, and the share attributable to failures or exceptional cases. It also includes the proportion of tasks rejected or shortened by limits. This makes efficiency claims auditable: readers can see whether savings came from better processing, a different task mix, or a policy that simply declines expensive work.
Standardized benchmarks help when their rules travel with them
MLCommons publishes MLPerf Inference datacenter benchmarks with defined workloads and reporting structures. The project provides a basis for comparing systems under specified conditions rather than relying only on vendor-selected demonstrations. The live page checked for this article is a changing resource; no particular current leaderboard result is used here as a purchasing recommendation. MLPerf Inference datacenter.
A standardized benchmark is most informative when the reader preserves its scope. A result on a specified model, dataset, scenario, and accuracy target does not automatically predict the cost of a different application. The benchmark can reveal system capability and support controlled comparison while still requiring a local workload test.
The distinction between an offline scenario and a server-like scenario is especially important conceptually. Work delivered in a large batch can be scheduled differently from requests arriving over time with latency requirements. A high offline throughput result should not be described as the number of simultaneous interactive users a service can support without additional evidence.
For procurement, request the full result reference and relevant configuration rather than a screenshot of one favorable number. Identify accelerator count, model version, numerical precision, software stack, and applicable constraints. If a supplier extrapolates from a benchmark to the customer's workload, ask for the assumptions and a validation run.
A local acceptance test can then complement the standardized result. It should use representative inputs, quality criteria, and realistic arrival patterns. The combination is stronger than either alone: standardized testing provides a common reference, while local testing checks whether the system meets the actual service requirement. Neither should be stretched into a universal measure of intelligence, economy, or user satisfaction.
Reproduce the cost experiment without exposing private data
A useful artifact package contains the workload generator or a representative public substitute, input and output length distributions, model and runtime identifiers, hardware configuration, request settings, and scoring procedure. It should also preserve the tariff or allocation method used to turn resource measurements into cost. Another evaluator should be able to see where a different assumption would change the result.
Private data need not be published to make the method credible. Synthetic or appropriately de-identified workloads can reproduce important structural properties, provided their limitations are stated. A substitute that preserves only average length may fail to preserve complexity, cache reuse, or output behavior. The report should explain which properties were matched and which were not.
Timing instrumentation needs clear boundaries. Does the clock start when the client submits the request, when the server accepts it, or when computation begins? Does it stop at the last generated token, the completed network transfer, or the finished application artifact? Each choice can be valid, but comparisons require consistent definitions. Client-side and server-side measurements can help locate delays rather than arguing over one supposedly correct clock.
Repeat runs under documented conditions. Warmup, background load, cache state, and network variability can affect results. Report the spread rather than selecting the fastest run. If a failure occurs, retain it with an explanation instead of silently removing it from the dataset.
The result should be a reproducible claim about a specified system and workload. It need not pretend to reveal every internal mechanism. The essential discipline is that the headline saving can be traced to observable work, an explicit boundary, and assumptions a reader can challenge.
Allocation rules should remain stable across alternatives
Shared infrastructure creates an accounting choice. A monitoring service, storage system, or engineering team may support several applications. Assigning every shared expense to one candidate and none to another would distort the comparison. The allocation rule should be stated before examining which alternative wins, and the report should show how sensitive the conclusion is to that rule.
One possible hypothetical approach assigns shared operating expense in proportion to measured usage. Another assigns a fixed share to each supported application. Neither is universally correct. Usage-based allocation can fit variable resources while understating the cost of maintaining a low-volume but demanding service. Equal allocation can be simple while overstating the burden of a small application. The organization should explain why its chosen method fits the decision.
Separate sunk costs from prospective costs. Equipment already owned may have little immediate acquisition cost for an experiment, but using it can consume capacity needed elsewhere. Conversely, including an entire historical development budget in every future request can overstate the marginal cost of expanding an established service. A decision about the next hundred tasks differs from a decision about building a new service from scratch.
Report both perspectives when they matter. An experimental pilot may be attractive at marginal cost yet unattractive as a standalone operation. A larger deployment may spread fixed costs across more work but require new reliability staffing. These distinctions make the budget more realistic without requiring false precision. The objective is consistent reasoning across alternatives, not a single accounting convention presented as the only possible truth.
The lowest price is not the final metric
A sensible inference-cost comparison ends with several numbers, not one. The per-attempt charge describes the tariff. Resource utilization describes how efficiently infrastructure is used. Latency describes when users receive output. Acceptance describes whether the output meets the task. Completion cost combines the relevant spending with the work that actually gets done.
These quantities can move in different directions. Batching may improve utilization while increasing waiting. Quantization may reduce memory use while requiring a careful quality check. Caching may reduce repeated input work while adding freshness management. Speculative decoding may improve latency while adding another model's computation. A good report makes those trade-offs visible rather than assuming that every optimization improves every outcome.
For an organization, the first practical step is to choose a small set of representative tasks and define acceptance. Then measure the complete workflow at realistic load, including retries and escalation. Use engineering benchmarks to understand bottlenecks, but calculate the economic result at the service boundary that matters to the organization.
The final question is modest and demanding: what did it cost to complete this kind of task, to this standard, within this time, under these conditions? A clear answer can support a decision even if it lacks a dramatic universal claim. An unclear answer can make an inexpensive token rate look like a saving before the work has actually been counted.