Methods and guidelines for effective model calibration

Prediction Uncertainty
The uncertainty with which predictions are simulated can be approximated using confi-
dence and prediction intervals. Confidence intervals are for the true, unknown predictions, which 
are not random variables, and result from the uncertainty with which the parameters are estimated, 
as represented by the variance-covariance matrix on the parameters (eq. 26). Prediction intervals 
also account for random measurement error in the quantity for which the interval is constructed, 
and are needed to construct an interval for an anticipated measurement of the prediction. Confi-
dence and prediction intervals are discussed in many texts, such as Seber and Wild (1989) and 
Helsel and Hirsch (1992), as well as by Cooley and Naff (1990) and Hill (1994). 
Linear Confidence and Prediction Intervals
Approximate linear confidence and prediction intervals for predictions can be calculated 
using output files produced by MODFLOWP and computer program YCINT of Hill (1994), or by 
setting the input variables appropriately for UCODE, in which YCINT has been converted to a sub-
routine. Linear confidence intervals are calculated as:
(32)
where z
’
l
is the lth simulated value;
t
s 
(n, 1.0-
α
/2) is the critical value, and equals the value for which there is an 
α
/2 probability that a 
student-t distributed random variable would be larger;
n is the degrees of freedom, here equal to ND+MPR-NP;
α
is the significance level and is commonly 0.05 or 0.10 (5 and 10 percent), and
is the standard deviation of the prediction, calculated as
.
(33)
The calculated confidence interval is said to have a (1-
α
) probability of including the true value of 
the predicted quantity. Corresponding to the values noted above, 90- and 95-percent confidence in-
tervals are the most common.
Approximate linear prediction intervals are calculated as:
z’
l
t
s
n 1.0
α
2
---
–
,




s
z’
l
±
s
z’
l
s
z’
l
b
j
∂
∂
z’
l
V b’
( )
b
i
∂
∂
z’
l
j
1
=
NP
∑
i
1
=
NP
∑
1
2
---
=

---

30
(34)
where 
is the standard error of the regression adjusted for the expected measurement error of the 
prediction (see Hill, 1994, p. 32; Miller, 1981).
Individual confidence intervals calculated using equation 32 are exact for linear models 
with normally distributed residuals, assuming that the model is correct. As these conditions are vi-
olated to a greater degree, the calculated intervals become progressively less accurate, so that the 
actual significance level of the interval can be substantially different than intended. This is of seri-
ous concern for the nonlinear problems considered in this work, as discussed by Donaldson and 
Schnabel (1987). Recent publications in the ground-water literature, however, indicate that in 
many ground-water flow problems linear intervals are accurate enough to be useful (Christensen 
and Cooley, in press). Other types of ground-water problems have not been evaluated. 
The calculation of confidence and prediction intervals can (and often needs to) include 
more parameters than were included in the regressions performed for model calibration, as dis-
cussed above in the section ‘Prediction Scaled Sensitivities’ and under guideline 13.
The individual intervals defined above apply when the uncertainty of only one quantitiy is 
of interest. When more than one quantity is of interest, different intervals are needed, and these are 
called simultaneous intervals. Simultaneous intervals calculated using linear theory are always of 
equal size or larger than equivalent linear individual intervals, reflecting the greater uncertainty in-
volved in trying to define intervals which are likely to include the true value of two or more pre-
dictions at the same time. As more intervals are considered, the intervals tend to become wider. 
The largest intervals are calculated when the number of predictions equals the number of parame-
ters included in the uncertainty analysis. Additional predictions do not increase the size of the si-
multaneous intervals.
Simultaneous intervals are difficult to calculate exactly, but can be approximated using the 
equations listed in Hill (1994), as discussed by Miller (1981). The equations for simultaneous con-
fidence and prediction intervals are of the same form as equations 32 and 34, respectively, and dif-
fer only in the critical values used. If the number of intervals considered is represented by k, the 
interval limits can be calculated using critical values from a Bonferroni-t distribution or from an F 
distribution. The Bonferroni critical value is 
t
B
(n,1.0-
α
/2k). (35a)
The F distribution critical value is 
[d x F
α
(d,n)]
1/2
, 
(35b)
where d equals k or the number of parameters (NP), whichever is less. Intervals calculated with the 
z'
l
t n 1.0
α
2
---
–
,




s
z’
l
s
a
+




±
s
a

---

31
F distribution critical value are called Scheffe intervals. Scheffe intervals are labeled either as 
Scheffe d=k or Scheffe d=NP.
Both Bonferroni and Scheffe intervals are approximate, and tend to be large. Thus, for any 
finite value of k, the smaller interval should be used. 
In some cases k is not finite. For example, if a prediction of interest is the largest simulated 
value over a defined area, the predicted quantity can not be specified exactly before the simulation, 
and the number of predictions considered simultaneously needs to be thought of as infinite. In this 
circumstance, the only applicable approximate simultaneous interval is the Scheffe d=NP.
As discussed in the section “Prediction Scaled Sensitivities”, in some circumstances the 
prediction on interest is the difference between two simulations. Both UCODE and MODFLOWP 
can calculate linear confidence and prediction intervals on differences, as discussed by Hill (1994).
Calculation of linear confidence intervals requires only the sensitivities calculated for the 
optimized parameter values and, therefore, takes very little computer execution time.