# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: On potential error introduced by rounded values**

**From:**Peter Fogg

**Date:**2005 Jan 10, 16:06 +1100

George Huxtable wrote:

"If we take 6 such roundings, then

in theory the maximum amount they could possibly (but most unlikely)

contribute to an error in position is ±3 miles..."

And the chance of that most unlikely event is one in six to the power
of six, or 0.00002, or 0.002%? If this is correct, the results from models
similar to the Excel sample are unlikely to show an error result of more than 2
minutes of arc in a significant number of cases.

"The question then arises: what does a prudent navigator take as
the error

introduced by the roundings in this calculation process, to combine
with

his estimated errors of observation? How much skirting distance should
he

allow around a charted but unmarked rock in mid-ocean, relying only on

astro observations to avoid it? How narrow a passage between two such

hazards would he tackle? A careful family-man might take the view that
if

there's a possibility (even a tiny one) of the calculation being 3
miles

out either way, that's what he should allow for. A gambler might assess

that if there's no more than (say) a 1 in 50 chance of the calculation

error exceeding 1.5 miles, he will accept the risk and plot his course
in

that basis. After all, we're all risk-takers to some extent, or we
wouldn't

be out there on small boats in big waters."

As acknowledged a little later in George's
posting, trying to wring great precision from the deck of a small boat under
way may be a quixotic endeavour. One rule of thumb is that if a fix derived
UNDER AVERAGE CONDITIONS is within 10nm of the real position then that is a
good fix, and if within 5nm then that is an excellent result. I know more
accurate results are possible, as the actual conditions of sea, etc, can vary
greatly. When a fix is expressed to whole minutes of arc it doesn't mean the
boat is at that intersection (it is almost certainly not, even if the fix is
entirely accurate). It means that the position is somewhere within a rectangle (a
square at the equator) bounded by the halfway points to the next intersection
of minutes of arc of lat/long. This is why expressing a fix to a precision of
tenths of a minute of arc is essentially meaningless, just futile extra effort
- since the actual position is still five to ten or more miles away, depending
on conditions. This is the difference between accuracy and precision. Improving
the accuracy is desirable, and comparing the sights against the slope of the
body's apparent rise or fall is one way to do that. Increasing the precision
gives only a false sense of greater confidence in the calculated position.

"Peter offers two numerical models. One draws a parallel with the
numbers of

boys and girls in a family. I suggest that this is a very
over-simplified

approach to the problem, presuming that errors would always be the same
in

amount with just the sign being random, and his numerical results are

therefore not very relevant."

I think the boy/girl model is both a good and a bad example. Bad
because it doesn't just involve probability but also biology, a fascinating but
messy business compared to 'clean' numbers. Good because it illustrates the
concept in a way most people can understand and relate to. Yes, its an
either/or situation (we won't go into the exceptions) but I think the results
are still relevant, certainly as a first step in explaining the issue. The average
difference between rounded/unrounded is 0.25, itself either/or (plus or minus),
just like boy/girl.

"Finally, after all that accord, and if only to tweak Peter Fogg's
tail a

little, I can't resist pointing out that earlier criticisms of the
Bennett

tables revolved around the major errors in azimuth, of many degrees,
that

can occur when using Bennett's azimuth tables for directions anywhere
near

to due East or West: not the errors of a minute or two in intercept
that

are being considered at present."

George Bennett has made a preliminary
examination of the probability of error using these azimuth tables. The
following is taken from that work.

**Examples of Rounding-off errors in Sight Reduction Tables and Azimuth
Tables**

In this problem of finding the behavior of rounding-off errors it will
be necessary to construct a mathematical model of what is being performed,
defining the values and limits of the arguments and respondents and the number
of significant figures used in the tables. I wrote some QB programs that simulated
a human operator rounding-off and extracting values in accordance with certain
conditions.

A. An example of my analysis of the sight reduction tables in ‘The
Complete On-Board Celestial Navigator’ is to be found on my website at http://www.netspace.net.au/~gbennett/

You will see the details of the range of variables and other conditions
on the page ‘Detailed Description’.

B. Example of my analysis of the Azimuth Tables is as follows,

Sample Size: 14956717 total

Errors (degrees) |
Number of samples |
Percentage |

0
to 1 |
13999457 |
93.5998 |

1
to 2 |
767609 |
5.1322 |

2
to 3 |
116026 |
0.7757 |

3
to 4 |
37862 |
0.2531 |

4
to 5 |
17026 |
0.1138 |

5
to 6 |
8480 |
0.0567 |

6
to 7 |
4438 |
0.0297 |

7
to 8 |
2617 |
0.0175 |

8
to 9 |
1448 |
0.0097 |

9 to
10 |
798 |
0.0053 |

10 to
11 |
481 |
0.0032 |

11 to
12 |
245 |
0.0016 |

12 to
13 |
134 |
0.0009 |

13 to 14 |
68 |
0.0005 |

14 to
15 |
26 |
0.0002 |

15 to
16 |
2 |
0.0000 |

16 to
17 |
0 |
0.0000 |

The range of variables (degrees) was LHA 0.6 -179.6, Declination N89.4
- S89.6 and Altitude 0.6 – 70.6, rounding-off in the program to the
nearest degree. The step size was close to 0.5 degrees making sure that no
input value was a whole degree. So you can see that this data set and
errors should be a reasonable assessment of the quality of azimuths derived
from the tables.(If rounding off is not performed, azimuth errors will not
exceed 2 degrees in all cases )

Further analysis showed that as a general rule, errors greater than 2
degrees will only occur within about 15 degrees of the prime vertical (Azimuth
90 or 270 degrees).

An example was proposed that gave an azimuth error of about 15 degrees
based on variables which were within a minute of arc of half a degree. The
probability of this occurring is about 2/14956717. See above table.

I have added a footnote in the new edition which warns the user to take
care when observations have been made near the PV and substantial rounding off
must be made. When in doubt, interpolate.

(Peter Fogg’s comment): If this model
is accurate, about 94% of 15 million random samples of derivations of azimuth
using his tables show an error of less than one degree, 5% show an error of
less than 2 degrees, and greater error than that occurs in less than 1% of
cases. The highly selective example showing a 15 degree error is too remote to
be of practical concern.

It seems that altogether too much could be
made of the supposed shortcomings of the tables, by selecting 3 variables carefully
chosen to show the maximum amount of error possible in a juxtaposition of
exactly halfway values. Perhaps our careful family-man could more usefully direct
his capacities for worry towards the possibility of being taken out by an
errant asteroid.