NavList:

George Huxtable wrote:

 

"If we take 6 such roundings, then

in theory the maximum amount they could possibly (but most unlikely)

contribute to an error in position is ±3 miles..."

 

And the chance of that most unlikely event is one in six to the power of six, or 0.00002, or 0.002%? If this is correct, the results from models similar to the Excel sample are unlikely to show an error result of more than 2 minutes of arc in a significant number of cases.

 

 

"The question then arises: what does a prudent navigator take as the error

introduced by the roundings in this calculation process, to combine with

his estimated errors of observation? How much skirting distance should he

allow around a charted but unmarked rock in mid-ocean, relying only on

astro observations to avoid it? How narrow a passage between two such

hazards would he tackle? A careful family-man might take the view that if

there's a possibility (even a tiny one) of the calculation being 3 miles

out either way, that's what he should allow for. A gambler might assess

that if there's no more than (say) a 1 in 50 chance of the calculation

error exceeding 1.5 miles, he will accept the risk and plot his course in

that basis. After all, we're all risk-takers to some extent, or we wouldn't

be out there on small boats in big waters."

 

As acknowledged a little later in George's posting, trying to wring great precision from the deck of a small boat under way may be a quixotic endeavour. One rule of thumb is that if a fix derived UNDER AVERAGE CONDITIONS is within 10nm of the real position then that is a good fix, and if within 5nm then that is an excellent result. I know more accurate results are possible, as the actual conditions of sea, etc, can vary greatly. When a fix is expressed to whole minutes of arc it doesn't mean the boat is at that intersection (it is almost certainly not, even if the fix is entirely accurate). It means that the position is somewhere within a rectangle (a square at the equator) bounded by the halfway points to the next intersection of minutes of arc of lat/long. This is why expressing a fix to a precision of tenths of a minute of arc is essentially meaningless, just futile extra effort - since the actual position is still five to ten or more miles away, depending on conditions. This is the difference between accuracy and precision. Improving the accuracy is desirable, and comparing the sights against the slope of the body's apparent rise or fall is one way to do that. Increasing the precision gives only a false sense of greater confidence in the calculated position.   

 

 

"Peter offers two numerical models. One draws a parallel with the numbers of

boys and girls in a family. I suggest that this is a very over-simplified

approach to the problem, presuming that errors would always be the same in

amount with just the sign being random, and his numerical results are

therefore not very relevant."

 

I think the boy/girl model is both a good and a bad example. Bad because it doesn't just involve probability but also biology, a fascinating but messy business compared to 'clean' numbers. Good because it illustrates the concept in a way most people can understand and relate to. Yes, its an either/or situation (we won't go into the exceptions) but I think the results are still relevant, certainly as a first step in explaining the issue. The average difference between rounded/unrounded is 0.25, itself either/or (plus or minus), just like boy/girl.

 

 

"Finally, after all that accord, and if only to tweak Peter Fogg's tail a

little, I can't resist pointing out that earlier criticisms of the Bennett

tables revolved around the major errors in azimuth, of many degrees, that

can occur when using Bennett's azimuth tables for directions anywhere near

to due East or West: not the errors of a minute or two in intercept that

are being considered at present."

 

George Bennett has made a preliminary examination of the probability of error using these azimuth tables. The following is taken from that work.

 

Examples of Rounding-off errors in Sight Reduction Tables and Azimuth Tables

 

In this problem of finding the behavior of rounding-off errors it will be necessary to construct a mathematical model of what is being performed, defining the values and limits of the arguments and respondents and the number of significant figures used in the tables. I wrote some QB programs that simulated a human operator rounding-off and extracting values in accordance with certain conditions.

A. An example of my analysis of the sight reduction tables in ‘The Complete On-Board Celestial Navigator’ is to be found on my website at http://www.netspace.net.au/~gbennett/

You will see the details of the range of variables and other conditions on the page ‘Detailed Description’.

B. Example of my analysis of the Azimuth Tables is as follows,

 

                              Sample Size: 14956717 total

Range of Azimuth Errors (degrees)	Number of samples	Percentage
0   to   1	13999457	93.5998
1   to   2	767609	5.1322
2   to   3	116026	0.7757
3   to   4	37862	0.2531
4   to   5	17026	0.1138
5   to   6	8480	0.0567
6   to   7	4438	0.0297
7   to   8	2617	0.0175
8   to   9	1448	0.0097
9  to   10	798	0.0053
10   to  11	481	0.0032
11  to   12	245	0.0016
12  to   13	134	0.0009
13 to   14	68	0.0005
14   to  15	26	0.0002
15  to   16	2	0.0000
16  to   17	0	0.0000

 

The range of variables (degrees) was LHA 0.6 -179.6, Declination N89.4 - S89.6 and Altitude 0.6 – 70.6, rounding-off in the program to the nearest degree. The step size was close to 0.5 degrees making sure that no input value was a whole degree.  So you can see that this data set and errors should be a reasonable assessment of the quality of azimuths derived from the tables.(If rounding off is not performed, azimuth errors will not exceed 2 degrees in all cases )

Further analysis showed that as a general rule, errors greater than 2 degrees will only occur within about 15 degrees of the prime vertical (Azimuth 90 or 270 degrees).

An example was proposed that gave an azimuth error of about 15 degrees based on variables which were within a minute of arc of half a degree. The probability of this occurring is about 2/14956717. See above table.

I have added a footnote in the new edition which warns the user to take care when observations have been made near the PV and substantial rounding off must be made. When in doubt, interpolate.

 

(Peter Fogg’s comment): If this model is accurate, about 94% of 15 million random samples of derivations of azimuth using his tables show an error of less than one degree, 5% show an error of less than 2 degrees, and greater error than that occurs in less than 1% of cases. The highly selective example showing a 15 degree error is too remote to be of practical concern.

 

It seems that altogether too much could be made of the supposed shortcomings of the tables, by selecting 3 variables carefully chosen to show the maximum amount of error possible in a juxtaposition of exactly halfway values. Perhaps our careful family-man could more usefully direct his capacities for worry towards the possibility of being taken out by an errant asteroid.