Error propagation
Standard deviation, standard error and a rejection decision on repeated distances
Ten EDM observations of one baseline, reduced to a residuals table, a standard deviation and a standard error of the mean — and one suspect observation tested against a criterion that can actually reject it.
Given
- A baseline between two pillars was observed ten times with the same instrument, on the same afternoon, by the same operator, with the prism re-set between observations.
- Observed lengths, in US survey feet: 2416.38, 2416.31, 2416.44, 2416.29, 2416.36, 2416.33, 2416.41, 2416.30, 2416.57, 2416.35.
- The ninth observation, 2416.57, was flagged in the field book with a note that a vehicle passed between the instrument and the prism.
- All observations were reduced to horizontal and to the same temperature and pressure, so the set differs only by random error and, possibly, one blunder.
Required
- The mean, the residuals, the sample standard deviation and the standard deviation of the mean.
- A defensible decision on whether to reject the ninth observation.
- The mean and its standard error after that decision is acted on.
Work it through yourself before reading on — the solution below shows every step, so there is no way to skim it without giving the answer away.
Worked solution
Mean and residuals
The mean of the ten observations is the most probable value of the length given the set. Subtract it from each observation to get the residuals, which are what every statistic below is built from.
The residuals must sum to zero, exactly and by construction — that is a property of the arithmetic mean, not a result. Use it as a check on the subtraction before doing anything else with the column; a residual column that does not close means an observation was mis-keyed into the mean or the mean was mis-keyed into the subtraction.
The ninth residual, +0.196 ft, is more than twice the next largest. That is a signal, not yet a verdict.
| No. | Observed (ft) | Residual v (ft) | v² (ft²) |
|---|---|---|---|
| 1 | 2416.38 | +0.006 | 0.000036 |
| 2 | 2416.31 | −0.064 | 0.004096 |
| 3 | 2416.44 | +0.066 | 0.004356 |
| 4 | 2416.29 | −0.084 | 0.007056 |
| 5 | 2416.36 | −0.014 | 0.000196 |
| 6 | 2416.33 | −0.044 | 0.001936 |
| 7 | 2416.41 | +0.036 | 0.001296 |
| 8 | 2416.30 | −0.074 | 0.005476 |
| 9 | 2416.57 | +0.196 | 0.038416 |
| 10 | 2416.35 | −0.024 | 0.000576 |
| Σ | 24,163.74 | 0.000 | 0.063440 |
Standard deviation and standard error of the mean
The sample standard deviation divides the sum of squared residuals by n − 1, not by n. With ten observations of one unknown there are nine degrees of freedom: one was consumed computing the mean the residuals are taken about. Dividing by n understates the spread, and understates it worst on the short sets a crew actually collects.
The standard deviation describes a single observation. The standard deviation of the mean — the standard error — describes the mean of the whole set, and is smaller by a factor of the square root of the number of observations. This is the return on repeating a measurement, and it is a poor return: quartering the standard error of the mean costs sixteen times the observations.
Note also what a single blunder does to these numbers. The ninth observation contributes 0.038416 of the 0.063440 total — sixty per cent of the sum of squares comes from one of ten observations.
mean = 24,163.74 / 10 = 2416.374 ft
σ = √( Σv² / (n − 1) ) = √( 0.063440 / 9 ) = √0.00704889 = 0.084 ft
σ_mean = σ / √n = 0.084 / √10 = 0.0840 / 3.1623 = 0.027 ft
E95 = 1.96 σ = 1.96 × 0.084 = 0.165 ft
range = 2416.57 − 2416.29 = 0.280 ftWhy the three-sigma test cannot decide this
The instinct is to compare the largest residual with three standard deviations. Here that gives 0.196 against 3 × 0.084 = 0.252, so the observation survives. But the test has not actually been applied — it cannot be, at this sample size.
With n observations, the largest possible value of |v| / σ is (n − 1)/√n. For n = 10 that ceiling is 2.846, which is below 3. A three-sigma test on a set of ten will never reject anything, no matter how gross the blunder, because the blunder inflates the very standard deviation it is being compared against. This masking is not a subtlety; it is the arithmetic.
The ceiling rises slowly: a three-sigma test only becomes capable of rejecting at n = 11 and does not become a meaningful screen until the set is much larger. For sets of the size a field crew collects, a criterion that adapts to n is needed.
largest residual |v|max = 0.196 ft
|v|max / σ = 0.196 / 0.084 = 2.335
ceiling for n = 10: (n − 1) / √n = 9 / 3.1623 = 2.846
3σ = 0.252 ft > 0.196 ft — but 3σ was never reachable, so this is not evidence of anythingChauvenet's criterion, which does adapt to n
Chauvenet's criterion asks a question the three-sigma rule does not: how many observations this deviant would a clean set of this size be expected to contain? If the expected number is below one half, the observation is rejected.
That gives a critical probability of 1/(2n) in the two tails together. For n = 10 the tail probability is 0.05, and the standard normal value that cuts off 5 per cent in two tails is 1.96. The observed ratio of 2.335 exceeds it, so the ninth observation is rejected — which is the conclusion the field note independently supports.
Reject once and once only. After removing an observation, recompute the statistics and re-test, but do not keep peeling observations off a set until it looks tidy: repeated rejection on a small sample manufactures a precision that the work never had. If a second observation also fails, the set has a procedural problem, not an outlier.
critical two-tail probability = 1 / (2n) = 1 / 20 = 0.05
critical ratio for n = 10: z = 1.96
observed ratio 2.335 > 1.96 → reject observation 9 (2416.57)
the field note recording a vehicle in the line is independent corroborationRecompute on the accepted nine
With the blunder removed the standard deviation falls from 0.084 ft to 0.051 ft and the standard error of the mean from 0.027 ft to 0.017 ft. The mean itself moves 0.022 ft — small in absolute terms, but the whole of that movement was one bad observation dragging nine good ones.
Re-test the new set. The largest residual is now 0.0878 ft against a standard deviation of 0.0509 ft, a ratio of 1.72. Chauvenet's critical ratio for n = 9 is 1.92, so nothing further is rejected and the set is clean.
Report the mean with its standard error, not with its standard deviation: the question the length is being reported for is 'how well is this baseline known', and that is the standard error.
| No. | Observed (ft) | Residual v (ft) | v² (ft²) |
|---|---|---|---|
| 1 | 2416.38 | +0.0278 | 0.000773 |
| 2 | 2416.31 | −0.0422 | 0.001781 |
| 3 | 2416.44 | +0.0878 | 0.007709 |
| 4 | 2416.29 | −0.0622 | 0.003869 |
| 5 | 2416.36 | +0.0078 | 0.000061 |
| 6 | 2416.33 | −0.0222 | 0.000493 |
| 7 | 2416.41 | +0.0578 | 0.003341 |
| 8 | 2416.30 | −0.0522 | 0.002725 |
| 10 | 2416.35 | −0.0022 | 0.000005 |
| Σ | 21,747.17 | 0.0000 | 0.020756 |
Answer
- All ten observations: mean 2416.374 ft, σ = 0.084 ft, σ of the mean = 0.027 ft.
- Observation 9 (2416.57 ft) is rejected. Its residual ratio is 2.335, against a Chauvenet critical ratio of 1.96 for n = 10. A three-sigma test is powerless here because the largest ratio attainable at n = 10 is 2.846.
- Accepted nine observations: mean = 2416.352 ft, σ = 0.051 ft, σ of the mean = 0.017 ft.
- The 95 per cent error of a single observation is 1.96 × 0.051 = 0.100 ft; of the mean, 1.96 × 0.017 = 0.033 ft.
- Reported length: 2416.352 ± 0.017 ft, a relative precision of the mean of about 1 : 142,000.
- No further observation is rejected: the largest remaining ratio is 1.72 against a critical 1.92 for n = 9.
Check
Shift the data and recompute. Subtract 2416.00 ft from every observation, giving 0.38, 0.31, 0.44, 0.29, 0.36, 0.33, 0.41, 0.30, 0.35 for the accepted nine. Their sum is 3.17, so the mean is 0.352222, and the full mean is 2416.352 — matching. Residuals about a shifted mean are identical to residuals about the original, so σ is unchanged. The check exercises the same arithmetic on two-digit numbers.
Range check against the standard deviation. For a normal sample the expected ratio of range to standard deviation is about 2.97 at n = 9 and 3.08 at n = 10. The accepted nine give a range of 2416.44 − 2416.29 = 0.15 ft against σ = 0.0509 ft, a ratio of 2.95 — right on expectation, which is what a clean set looks like.
Run the same check on the unrejected ten: range 0.280 ft against σ = 0.084 ft is a ratio of 3.34, against an expected 3.08. The set is stretched relative to its own spread, which is the fingerprint of a single outlier and confirms the rejection independently of Chauvenet.
Σv = 0.000 in both residual tables, so the means were subtracted correctly.
More error propagation
All in this category- Error in a taped distance measured in n tape lengths, against a single EDM shotThe same 648 ft line measured two ways. Series propagation over seven tape settings against one EDM observation with a constant-plus-ppm specification, and what the comparison says about when taping is still adequate.
- Error propagated into a computed area from errors in two sidesA rectangular parcel measured on two sides, and the uncertainty those two measurements put into the acreage — with the fractional form of the same result as a check and the worst case as a bound.
- Error propagated into latitude and departure from errors in an angle and a distanceOne course, one distance uncertainty and one azimuth uncertainty, carried through the general propagation formula with partial derivatives — and a rotation argument that checks the answer without repeating it.