Optimal Experimental Design: How Statisticians Choose the Best Experiment

Every experiment costs something. A chemical plant trial burns raw material and production time. A clinical pharmacology study draws blood from volunteers. A wind-tunnel test books an expensive facility by the hour. In each case the person running the experiment has to decide which conditions to test and how many times to test each one. Optimal experimental design is the branch of statistics that turns that decision into a mathematical problem. It asks: given a model of how the response depends on the factors you control, and a fixed budget of runs, where should you put those runs so that the data tell you as much as possible?

The answer depends on what "as much as possible" means. Do you want the most precise estimates of all the model's parameters at once? The most precise estimate of one particular quantity? The most reliable predictions across the whole region you care about? Each goal leads to a different optimality criterion, and the criteria have one-letter names: D, A, E, G, I, V, C, T and others. This article explains where those criteria came from, what they measure, how optimal designs are actually computed, what they are good for, and where they can lead you astray.

The article assumes only a little statistics: the idea of fitting a model by least squares and the idea that an estimate has a variance. Where matrix notation appears, the plain-English meaning is given alongside.

The Core Idea in One Paragraph

Suppose the response y you measure depends on settings x of the factors you control, through a model with unknown parameters. With ordinary linear regression, the model can be written y = Xβ + ε, where each row of the matrix X describes one experimental run (for example 1, x, x² for a quadratic in one factor), β is the vector of unknown coefficients, and ε is random error with variance σ². The least-squares estimate of β has variance–covariance matrix σ²(X′X)⁻¹. That matrix tells you how uncertain every coefficient is and how the uncertainties are correlated. Crucially, X depends only on where you chose to run the experiment, not on the responses you will observe. So before you collect a single data point, you can calculate how precise your estimates will be for any proposed design. Optimal design chooses the rows of X to make (X′X)⁻¹ "small", or equivalently to make the information matrix X′X "large", in some well-defined sense.

That last phrase is where all the subtlety lives. A matrix does not have a single natural size. Its determinant, its trace, its smallest eigenvalue and its largest diagonal entry are all reasonable measures, and each one gives a different optimality criterion.

A Short History

Early precursors

The Wikipedia survey of the field traces the first published work on optimal designs for polynomial regression to the French mathematician Joseph Diez Gergonne in 1815. Later in the nineteenth century the American philosopher and scientist Charles Sanders Peirce proposed an economic theory of experimentation that aimed to get the most precision for the resources spent; in 1876 he wrote on the economy of research, and in an 1882 lecture at Johns Hopkins University he introduced experimental design to his audience. These were isolated contributions rather than the start of a continuous research tradition.

Kirstine Smith, 1918

The work usually credited with founding the field is a long paper by the Danish statistician Kirstine Smith (1878–1939), published in Biometrika in 1918 (volume 12, pages 1–85) under the title "On the standard deviations of adjusted and interpolated values of an observed polynomial function and its constants and the guidance they give towards a proper choice of the distribution of observations." Smith had studied mathematics and physics at the University of Copenhagen, worked as secretary to the astronomer and statistician Thorvald Thiele, and went to London in 1916 to train under Karl Pearson.

Her question was concrete: if you are going to fit a polynomial to observations taken over an interval, where should the observations go so that the fitted curve is as reliable as possible everywhere on that interval? She chose, for each polynomial degree, the design that minimizes the largest standard deviation of the predicted value across the range. In modern language she computed G-optimal designs, and she did so for polynomials up to degree six. Among her results: the optimal design for a polynomial of degree k uses exactly k + 1 distinct values of x, and for a straight line, the best plan is to put half the observations at each end of the interval. She also warned that these designs are only optimal if the true function really has the assumed polynomial form, a caveat that remains the central weakness of the whole approach more than a century later.

The statistician R. A. Bailey, in a history of experimental design papers in Biometrika, describes Smith's paper as remarkable and decades ahead of its time. Its importance was recognized properly only after Jack Kiefer credited it in the late 1950s.

Wald, Elfving and the decision-theoretic school

The modern theory grew out of the decision theory developed by Abraham Wald in the 1940s, which treated statistical procedures as decisions to be chosen to minimize expected loss. Wald also pioneered sequential analysis, published in 1945. The Finnish statistician Gustav Elfving was another early developer of optimal design; his best-known result is a geometric characterization of designs that are optimal for estimating a single linear combination of parameters (what is now called C-optimality).

Kiefer and Wolfowitz

Jack Kiefer turned optimal design into a coherent mathematical subject. His 1959 paper "Optimum Experimental Designs", published in the Journal of the Royal Statistical Society (Series B), set out a general framework. In 1960 Kiefer and Jacob Wolfowitz published "The equivalence of two extremum problems" in the Canadian Journal of Mathematics. This paper proved what is now called the Kiefer–Wolfowitz equivalence theorem, which links D-optimality to G-optimality and provides a practical test for whether a design is truly optimal. Kiefer also developed a theory of "universal optimality" for designs that are optimal under a whole family of criteria at once.

Fedorov, Box and the computer era

In 1972 the Soviet statistician Valerii Fedorov published Theory of Optimal Experiments, which presented the theory together with iterative algorithms for computing optimal designs. The same year Herman Chernoff published an overview of sequential design. Fedorov's exchange algorithm, which swaps design points one at a time with points from a candidate list whenever the swap improves the criterion, became one of the most influential ways of building designs by computer.

A parallel tradition, associated above all with George E. P. Box, developed classical response-surface designs. Box and K. B. Wilson's 1951 paper "On the experimental attainment of optimum conditions" (Journal of the Royal Statistical Society, Series B, volume 13, pages 1–45) introduced the central composite design and the sequential strategy now known as response surface methodology. Box and Donald Behnken's 1960 paper in Technometrics (volume 2, pages 455–475) introduced the three-level designs that bear their names. These designs were built for practical virtues such as rotatability, a manageable number of levels and the ability to run in blocks, rather than for optimality under a mathematical criterion. The Wikipedia survey notes that they have few optimality properties as such; it points out that Box–Behnken designs need more runs than necessary once there are more than three variables, and that central composite designs need more runs than the optimal designs computed by Kôno.

The tension between the "classical designs" school and the "optimal design" school was real for decades. In practice the two now coexist: classical designs are the default when they fit the problem, and computer-generated optimal designs are used when they do not.

Timeline

Year	Event
1815	Gergonne publishes on designs for polynomial regression
1876, 1882	C. S. Peirce writes on the economy of research and lectures on experimental design
1918	Kirstine Smith's Biometrika paper on G-optimal polynomial designs
1945	Wald's sequential analysis
1951	Box and Wilson introduce response surface methodology
1959	Kiefer, "Optimum Experimental Designs", JRSS B
1960	Kiefer–Wolfowitz equivalence theorem; Box–Behnken designs
1972	Fedorov, Theory of Optimal Experiments; Chernoff's overview of sequential design
1975	Atkinson and Fedorov introduce T-optimality for discriminating between models
1995	Meyer and Nachtsheim publish the coordinate-exchange algorithm

The Building Blocks

Models and design points

Optimal design starts with a model. For a single factor x scaled to run from −1 to +1, three common models are:

Straight line: y = β₀ + β₁x + ε, so each row of X is (1, x).
Quadratic: y = β₀ + β₁x + β₂x² + ε, so each row is (1, x, x²).
Multi-factor models with main effects, two-factor interactions and squared terms, used in response surface work.

A design is a list of the settings at which runs will be made. The set of all allowable settings is the design region. It may be a simple box (each factor between its low and high limits), or it may be irregular because some combinations are unsafe, impossible or meaningless. A mixture experiment, for example, requires the component proportions to add up to one.

Exact designs and approximate designs

There are two ways to describe a design mathematically.

An exact design specifies a whole number of runs at each point: for example, three runs at x = −1, four at x = 0, and three at x = +1, for ten runs in total. This is what you actually carry out.

An approximate design, also called a continuous design or design measure, specifies only the proportion of runs at each point: one third at −1, one third at 0, one third at +1. Kiefer's theory works mainly with approximate designs because the set of design measures is convex, which makes the optimization problem far better behaved. The price is that an approximate design must be rounded to an exact one with whole numbers of runs, and the rounded design may be slightly worse. The Wikipedia survey notes this as a practical limitation: probability-measure designs ignore the fact that you cannot run a third of an experiment.

For an approximate design ξ that puts weight wᵢ on point xᵢ, the normalized information matrix is M(ξ) = Σ wᵢ f(xᵢ) f(xᵢ)′, where f(x) is the row of the model matrix at x. For an exact design with n runs, X′X = n·M.

Why the information matrix matters

The information matrix is the inverse of the variance matrix of the estimates (up to the factor σ²/n). Big information means small variance. Everything else in the theory consists of different ways of summarizing "big".

The Alphabet of Optimality Criteria

The following table lists the main criteria, what each one optimizes, and the question it answers.

Criterion	Mathematical target	Plain-English goal
D-optimality	Maximize det(X′X), equivalently minimize det((X′X)⁻¹)	Shrink the joint confidence region for all parameters
A-optimality	Minimize trace((X′X)⁻¹)	Minimize the average variance of the coefficient estimates
E-optimality	Maximize the smallest eigenvalue of X′X	Protect against the worst-estimated direction in parameter space
C-optimality	Minimize the variance of c′β̂ for a chosen vector c	Estimate one specific combination of parameters as precisely as possible
G-optimality	Minimize the maximum over the region of the prediction variance f(x)′(X′X)⁻¹f(x)	Make the worst prediction in the region as good as possible
I-optimality (also IV)	Minimize the prediction variance averaged over the design region	Make predictions good on average across the region
V-optimality	Minimize the average prediction variance over a specified set of points	Predict well at the particular conditions you care about
T-optimality	Maximize the discrepancy between rival models at the design points	Tell competing models apart
S-optimality	Maximize a combination of column orthogonality and the determinant	Keep estimates nearly uncorrelated while retaining information

D-optimality

D stands for determinant. If the errors are normally distributed, the joint confidence region for the parameters is an ellipsoid whose volume is proportional to the square root of det((X′X)⁻¹). Maximizing det(X′X) therefore makes that ellipsoid as small as possible. D-optimality is by far the most widely used criterion in software, partly because it has an attractive invariance property: rescaling or linearly reparameterizing the factors does not change which design is D-optimal.

The NIST/SEMATECH e-Handbook of Statistical Methods, a standard free reference, describes D-optimal designs as one form of computer-generated design, generated by maximizing |X′X|, the determinant of the information matrix, which minimizes the generalized variance of the parameter estimates. The handbook recommends them in two situations: when a standard factorial or fractional factorial would need too many runs for the resources available, and when the design space is constrained so that some factor combinations cannot be run.

A-optimality

A stands for average. The diagonal entries of (X′X)⁻¹ are the variances of the individual coefficient estimates (divided by σ²). Their sum, the trace, is a natural measure of overall imprecision. Unlike D-optimality, A-optimality depends on how the factors are scaled, because changing the units of one coefficient changes how much its variance contributes to the sum.

E-optimality

E stands for eigenvalue. The smallest eigenvalue of the information matrix corresponds to the direction in parameter space about which the experiment tells you least. Maximizing it guards against having any linear combination of parameters badly estimated. It is a minimax criterion: it focuses on the worst case rather than the average.

C-optimality

Sometimes you do not care about all the parameters. You might want the slope of a dose–response curve at a particular dose, or the difference between two treatments. If the quantity of interest is the linear combination c′β, its estimate has variance σ²c′(X′X)⁻¹c, and a C-optimal design minimizes that. C-optimal designs can be singular: they may not allow all parameters to be estimated, because they spend no effort on things you have said you do not need. Elfving's geometric theorem characterizes these designs.

G-, I- and V-optimality: criteria about prediction

If the purpose of the experiment is prediction rather than parameter estimation, the relevant quantity is the variance of the predicted response at a point x, which is proportional to d(x) = f(x)′M⁻¹f(x), sometimes called the standardized prediction variance. G-optimality (G for global) minimizes the maximum of d(x) over the region, which is what Kirstine Smith did in 1918. I-optimality (I for integrated) minimizes the average of d(x) over the region. V-optimality minimizes its average over a specified finite set of points.

Response surface practitioners often prefer I-optimal designs, since the usual goal is to map the response well everywhere, not to estimate individual coefficients.

T-optimality and model discrimination

All the criteria above assume you already know the right model. T-optimality, introduced by A. C. Atkinson and V. V. Fedorov in 1975, addresses the earlier question of which of two rival models is correct. A T-optimal design places runs where the competing models disagree most, so that the data can discriminate between them. Experiments designed to test rival models are especially important in pharmacokinetics and pharmacodynamics, where competing mechanistic models are common.

Compound and custom criteria

Because the standard criteria are convex (or concave) functions of the information matrix, weighted combinations of them are also convex. The Wikipedia survey notes that this allows practitioners to build custom criteria, for example a weighted blend of D-efficiency for parameter estimation and T-efficiency for model checking, and still use the same theoretical tools.

The Equivalence Theorem: How to Know a Design Is Optimal

Finding an optimal design is an optimization problem. Proving that a candidate design is globally optimal, and not just a local improvement, is in general hard. The Kiefer–Wolfowitz equivalence theorem makes it easy for the most important case.

For approximate designs and a model with p parameters, the theorem states that three conditions are equivalent:

The design ξ* maximizes det M(ξ), so it is D-optimal.
The design ξ* minimizes the maximum over the region of d(x, ξ) = f(x)′M(ξ)⁻¹f(x), so it is G-optimal.
The maximum over the region of d(x, ξ*) equals p, the number of parameters.

Two consequences follow. First, for approximate designs D-optimality and G-optimality coincide, so the design that pins down the parameters best is also the design that makes the worst-case prediction best. Second, condition 3 gives a checkable certificate. Compute d(x) over the design region; if its maximum equals p, the design is optimal, and the maximum is attained at the support points of the design. If somewhere d(x) exceeds p, the design is not optimal, and adding weight near that point will improve it. That second observation is the basis of the first-order algorithms of Fedorov and Henry Wynn, which repeatedly add weight at the point of maximum prediction variance.

The equivalence between D- and G-optimality holds exactly for approximate designs. For exact designs with a fixed number of runs, the two criteria can pick slightly different designs.

Equivalence theorems also exist for other convex criteria, and the Wikipedia survey notes the connection between these theorems and Legendre–Fenchel conjugacy in convex analysis. When a criterion is not convex, there is no such guarantee, and both finding and verifying the global optimum become difficult.

Worked Examples

The examples below are textbook-style illustrations with hypothetical numbers. The arithmetic has been checked, and you can reproduce it with a calculator.

Example 1: a straight line on [−1, 1]

Model: y = β₀ + β₁x + ε, with x allowed anywhere from −1 to +1.

Design A puts half the runs at x = −1 and half at x = +1. Each run contributes f(x)f(x)′ with f(x) = (1, x)′. Averaging over the design:

The (1,1) entry of M is the average of 1, which is 1.
The off-diagonal entry is the average of x, which is 0 because the design is symmetric.
The (2,2) entry is the average of x², which is 1.

So M = diag(1, 1) and det M = 1.

Design B spreads runs equally over x = −1, 0 and +1. Now the average of x² is (1 + 0 + 1)/3 = 2/3, so M = diag(1, 2/3) and det M = 2/3.

To compare designs fairly, statisticians use D-efficiency: the ratio of determinants raised to the power 1/p, where p is the number of parameters. Here p = 2, so the D-efficiency of Design B relative to Design A is (2/3)^(1/2) ≈ 0.816. In practical terms, Design B needs about 1/0.816 ≈ 1.22 times as many runs to match the precision of Design A, roughly 22 percent more.

Check with the equivalence theorem. For Design A, d(x) = f(x)′M⁻¹f(x) = 1 + x². Its maximum on [−1, 1] is 2, attained at x = ±1, and 2 equals p. So Design A is D-optimal and G-optimal, which is exactly Kirstine Smith's 1918 result for a straight line.

The same conclusion is visible in a single number. With n runs, the variance of the estimated slope is σ² divided by Σ(xᵢ − x̄)². With ten runs, putting five at each end gives Σx² = 10. Spreading ten runs evenly from −1 to +1 in steps of 2/9 gives Σx² ≈ 4.07. The evenly spread design therefore has a slope variance about 2.45 times larger.

Example 2: a quadratic on [−1, 1]

Model: y = β₀ + β₁x + β₂x² + ε, so p = 3.

Consider the design that puts one third of the runs at each of −1, 0 and +1. The averages are: E[1] = 1, E[x] = 0, E[x²] = 2/3, E[x³] = 0, E[x⁴] = 2/3. The information matrix is therefore

	1	x	x²
1	1	0	2/3
x	0	2/3	0
x²	2/3	0	2/3

Its determinant is (2/3) × [1 × 2/3 − (2/3)²] = (2/3) × (2/9) = 4/27 ≈ 0.148.

Working out the inverse and multiplying through gives the standardized prediction variance

d(x) = 3 − 4.5x² + 4.5x⁴.

At x = 0 this equals 3. At x = ±1 it equals 3 − 4.5 + 4.5 = 3. In between it dips; at x² = 1/2 it equals 3 − 2.25 + 1.125 = 1.875. The maximum over the region is 3, which equals p. By the equivalence theorem, the equal-weight design on {−1, 0, +1} is D-optimal for the quadratic. This is consistent with Smith's finding that a degree-k polynomial needs exactly k + 1 support points.

How much worse are "common sense" alternatives?

Design (equal weights)	det M	D-efficiency relative to optimum
−1, 0, +1	0.1481	1.000
−1, −1/3, +1/3, +1	0.1097	0.905
−1, −0.5, 0, +0.5, +1	0.0875	0.839

The five-point equally spaced design, which many people would choose instinctively because it "covers the range", is about 84 percent efficient. To match the precision of the optimal design it needs roughly 1/0.839 ≈ 1.19 times as many runs. That is the practical meaning of efficiency: a 16 percent loss in efficiency costs about 19 percent more experimentation for the same information.

The trade-off is not all one way. The five-point design gives you observations in the gaps, which lets you check whether a quadratic really fits. The three-point optimal design gives no information at all about lack of fit, because a quadratic can always be fitted exactly through the three group means. This is the model-dependence problem in miniature, and it is discussed further below.

Example 3: from approximate to exact

Suppose your budget is ten runs. You cannot put exactly one third of ten at each point. The candidate exact designs and their D-efficiencies relative to the approximate optimum are:

Runs at (−1, 0, +1)	det(X′X/n)	D-efficiency
3, 4, 3	0.144	0.991
4, 3, 3	0.144	0.991
4, 2, 4	0.128	0.952

With nine runs, three at each point would give an efficiency of exactly 1. The tenth run adds information in absolute terms, but its placement cannot keep the proportions perfect. This is why the Wikipedia survey says that fixing the run count in advance is often impractical, and that statisticians instead compare optimal designs at several different sample sizes before deciding how many runs to make.

Example 4: D versus A on the same problem

For the quadratic model, restrict attention to symmetric designs that put weight w at each of −1 and +1 and weight 1 − 2w at 0.

The determinant of M works out to 4w²(1 − 2w). Setting its derivative to zero gives w = 1/3, confirming the D-optimal design above.
The trace of M⁻¹ (the A-criterion) is minimized at w = 1/4, giving the design with one quarter of the runs at each end and half at the centre. Its trace is 8, while the D-optimal design's trace is 9.

Judged by the A-criterion, the D-optimal design therefore has an A-efficiency of 8/9 ≈ 0.89. Judged by the D-criterion, the A-optimal design has determinant 4 × 0.0625 × 0.5 = 0.125, against 0.148 for the optimum, which is a D-efficiency of (0.125/0.148)^(1/3) ≈ 0.945. Both designs are good under both criteria. This illustrates a general observation in the Wikipedia survey: designs that are optimal for one traditional criterion are usually near-optimal for the others, so the choice of criterion matters less than the choice of model.

How Optimal Designs Are Built in Practice

Step 1: state the model and the region

The experimenter must specify the model before generating the design, because optimality is defined relative to it. The NIST handbook stresses this point: a D-optimal design generated for a first-order model is generally different from one generated for a quadratic model. The region includes the factor ranges and any constraints, such as combinations that are unsafe or impossible.

Step 2: choose a criterion

D-optimality is the common default for screening and estimation. I-optimality is common when the purpose is prediction or optimization of a response surface. C-optimality suits a single quantity of interest, and T-optimality or a compound criterion suits model discrimination.

Step 3: generate a candidate set or work on the continuous region

Classical exchange algorithms start from a candidate set, usually all combinations of a few levels of each factor. The NIST worked example uses three factors, with X₁ at five levels (−1, −0.5, 0, 0.5, 1) and X₂ and X₃ at two levels (−1, +1). The full factorial candidate set therefore has 5 × 2 × 2 = 20 points. The model is Y = β₀ + β₁X₁ + β₂X₂ + β₃X₃ + β₁₁X₁² + ε, and the run budget is twelve. The software selects twelve of the twenty candidates. The resulting design uses only the levels −1, 0 and +1 of X₁, crossed with all four combinations of X₂ and X₃, so it is a 3 × 2 × 2 full factorial. The handbook reports a D-efficiency figure of about 0.68 for this design, and notes that a fractional factorial can only reach 100 percent D-efficiency when there are no pure quadratic terms such as X₁² in the model. The handbook adds that other efficiency measures (A, G and I) help choose between designs whose D-efficiencies are similar.

Step 4: search

Several families of algorithms are in use.

Exchange algorithms. Fedorov's 1972 algorithm considers swapping a point in the current design for a candidate point and makes the swap that most improves the criterion. Each iteration exchanges at most one point. Cook and Nachtsheim's 1980 modification considers each design point in turn and makes any beneficial exchange as soon as it is found, which is faster.
Sequential or first-order algorithms. Wynn (1970) and Fedorov proposed adding weight at the point where the prediction variance d(x) is largest, guided by the equivalence theorem. These are natural for approximate designs.
Coordinate exchange. Meyer and Nachtsheim's 1995 coordinate-exchange algorithm improves one coordinate of one run at a time and does not need a candidate set. JMP's documentation describes the typical use: start from random factor values within the region, improve them coordinate by coordinate, and repeat from many random starts because each run finds only a local optimum.
Convex optimization. Because the approximate-design problem is convex for the main criteria, modern solvers can find approximate optimal designs directly, and these can then be rounded.

Step 5: check and randomize

The NIST handbook warns that there is no guarantee that a computer-generated design is actually D-optimal, and that different software packages may produce different designs for the same problem. MathWorks' documentation similarly notes that its algorithms may return designs that are locally but not globally optimal. In practice, users run the search from several starting points, compare the efficiencies, and inspect the result. Finally, the run order should be randomized before the experiment is carried out, just as with any designed experiment.

Software

The Wikipedia survey notes that major statistical systems, including SAS and R, have procedures for constructing optimal designs, and that catalogues of optimal designs are published in books and software libraries. Commercial tools such as JMP and MATLAB's Statistics and Machine Learning Toolbox include D-optimal and other custom design generators.

Nonlinear Models and the Chicken-and-Egg Problem

For linear models, the information matrix depends only on the design. For nonlinear models, it also depends on the unknown parameters. Consider an exponential decay y = θ₁ exp(−θ₂t) + ε, common in pharmacokinetics. The best times to sample depend on how fast the decay is, which is precisely what the experiment is meant to estimate.

Three main responses to this problem exist.

Locally optimal designs

The experimenter plugs in a best guess for the parameters and computes the design that would be optimal if that guess were correct. This is simple, and often works well if the guess is reasonable, but the design can perform poorly if the guess is far off.

Sequential designs

Run part of the experiment, estimate the parameters, redesign the rest using the new estimates, and repeat. This follows naturally from Wald's sequential ideas and Chernoff's work on sequential design. It is efficient but requires that the experiment can be run in stages.

Bayesian and "on-average" designs

Instead of a single guess, place a probability distribution on the parameters (or on a set of plausible models) and choose the design that maximizes the expected value of the criterion. The Wikipedia survey notes that this approach is used especially for generalized linear models, such as logistic regression for binary outcomes, and that the subsequent analysis need not be Bayesian. Because some researchers dislike the Bayesian label in this context, the same idea is also called "on-average" or "population" optimality.

Applications

Industrial experimentation and response surfaces

Chemical, pharmaceutical and manufacturing engineers use optimal designs when classical designs do not fit: when some factor combinations are infeasible, when the number of runs must be unusual, when some factors have more levels than others, or when a previous experiment must be augmented with additional runs. The NIST handbook's guidance on when to use D-optimal designs reflects this industrial setting.

Mixture experiments

When the factors are proportions of ingredients that must sum to one, the design region is a simplex rather than a box, often with further limits on each ingredient. Classical factorial designs do not apply, and computer-generated optimal designs are a standard tool.

Pharmacokinetics and dose–response

Choosing blood-sampling times and dose levels is a classic nonlinear design problem. The Wikipedia survey highlights discrimination experiments in pharmacokinetics and pharmacodynamics, following work by David Cox and Anthony Atkinson, as an especially important application.

Polynomial regression and calibration

Smith's original problem, fitting a curve over an interval, recurs in instrument calibration and in any setting where a smooth relationship must be estimated from a limited number of measurements.

Computer experiments and machine learning

Related ideas appear in the design of computer experiments and in what machine-learning practitioners call active learning, where the algorithm chooses which new data points to label. Those fields often use space-filling designs or information-based criteria that differ from the classical alphabet, but the underlying question of where to sample to learn the most is the same.

Limitations and Criticisms

Model dependence

This is the most important limitation, and Smith already saw it in 1918. An optimal design is optimal only for the assumed model. The Wikipedia survey states plainly that an optimal design can perform worse than a non-optimal design if the true model is different, and recommends benchmarking candidate designs under alternative models. The quadratic example above shows the risk: the three-point optimal design cannot reveal curvature beyond a quadratic, while a five-point design can.

Practical responses include:

adding extra points (center points, or points at intermediate levels) to allow a lack-of-fit test;
designing for a larger model than you expect to need;
using compound criteria that blend estimation efficiency with model-checking ability;
using Bayesian or on-average designs over a set of plausible models.

Robustness to the criterion versus robustness to the model

The Wikipedia survey observes that designs optimal for one traditional criterion are usually near-optimal for others, so robustness to the choice of criterion is generally greater than robustness to the choice of model. In practice, choosing the model carefully matters much more than agonizing over D versus A versus I.

Cost and the number of runs

Approximate design theory treats runs as freely divisible and costless to place. Real experiments have unequal costs: changing a furnace temperature may be expensive while changing a flow rate is cheap, which leads to split-plot structures that standard optimal designs ignore. The theory has been extended to handle blocking, split plots and cost constraints, but the simplest versions do not account for them.

Correlated estimates

The NIST handbook notes that D-optimal design matrices are usually not orthogonal, so effect estimates are correlated. Classical fractional factorials, by contrast, give uncorrelated estimates of main effects. Correlated estimates are harder to interpret, which is one reason the S-optimality criterion (which rewards orthogonality) and the use of classical designs where they apply remain popular.

Local optima and software differences

Exchange algorithms can stop at a local optimum, and different programs can produce different designs for the same specification. Running the search repeatedly, checking efficiency against theoretical bounds, and using the equivalence theorem where possible are standard safeguards.

Common Misconceptions

"An optimal design is the best design, full stop."

It is the best design for a stated model, a stated criterion and a stated region. Change any of the three and a different design may be better.

"Spreading points evenly is always safest."

Even spacing is sensible when you are unsure of the model, but if the model is known, it can waste a large share of the budget. In the straight-line example, ten evenly spaced runs give a slope variance about 2.45 times larger than five runs at each end.

"D-optimal designs are the same as fractional factorials."

When the region is a cube and the model has only main effects and interactions, a two-level fractional factorial can be D-optimal. When the model includes squared terms, the region is constrained, or the run count is unusual, the D-optimal design is something else.

"If the software says D-optimal, it is D-optimal."

As both NIST and MathWorks warn, the algorithms can return local optima. The label describes what the algorithm tried to do, not a guarantee.

"Optimal design is only about linear models."

The cleanest theory is for linear models, but much of the modern work, including Bayesian design and pharmacokinetic sampling design, concerns nonlinear models and generalized linear models.

Frequently Asked Questions

What is the difference between optimal design and classical design of experiments?

Classical designs, such as full and fractional factorials, central composite designs and Box–Behnken designs, are fixed templates with known good properties such as orthogonality and rotatability. Optimal designs are computed for a particular model, region and run budget by optimizing a criterion. Classical designs are often optimal or nearly so in the situations they were built for; optimal designs are most valuable when the situation does not match a template.

Which criterion should I use?

For estimating model parameters, D-optimality is the common default. For prediction and response surface work, I-optimality is often preferred. If one quantity matters most, use C-optimality. If you need to choose between models, use T-optimality or a compound criterion. Because designs that are optimal for one criterion are usually efficient for the others, the choice is rarely critical; the choice of model is.

What does D-efficiency mean?

D-efficiency compares a design's determinant with that of a reference design, raised to the power 1/p so that it scales like a number of runs. A D-efficiency of 0.84 means the design needs about 1/0.84 ≈ 1.19 times as many runs to achieve the same generalized precision as the reference.

Why do optimal designs often use only the extremes and the middle of the range?

For polynomial models, the information about slopes and curvature is greatest at the edges of the region, and the centre helps pin down the intercept and curvature. Smith showed that a polynomial of degree k needs only k + 1 support points. This is efficient but leaves no spare points to test whether the model is right.

What is the equivalence theorem used for in practice?

It provides a certificate of optimality for approximate designs: compute the standardized prediction variance d(x) over the region; if its maximum equals the number of parameters, the design is D-optimal (and G-optimal). If not, the location where d(x) is largest points to where the design should be improved.

Can optimal design handle constraints and existing data?

Yes. Computer-generated designs can respect arbitrary constraints on the region, and they can augment an existing set of runs by choosing additional runs that maximize the criterion for the combined data. These are among the most common reasons practitioners turn to optimal design software.

Who should I read to learn more?

The standard references include Kiefer's collected papers, Fedorov's Theory of Optimal Experiments (1972), Friedrich Pukelsheim's Optimal Design of Experiments, and Atkinson, Donev and Tobias's Optimum Experimental Designs, with SAS. The NIST/SEMATECH e-Handbook is a free practical introduction to computer-generated designs.

Summary

Optimal experimental design turns the question "where should I run my experiment?" into an optimization problem over the information matrix. Kirstine Smith posed and solved the first such problem in 1918; Jack Kiefer, Jacob Wolfowitz, Gustav Elfving, Valerii Fedorov and many others built the general theory in the decades after. The criteria (D, A, E, C, G, I, V, T and more) encode different goals, and the Kiefer–Wolfowitz equivalence theorem provides a powerful way to verify optimality. Worked examples show that optimal designs can save a fifth or more of the experimental budget compared with intuitive alternatives. The approach's great strength, tailoring the design to the model, is also its great weakness: if the model is wrong, the optimal design can be misleading. Sensible practice uses optimal design as a tool alongside classical designs, with extra runs for model checking and a clear statement of what the design is optimal for.