# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**HO 211 (Ageton) sight reduction accuracy**

**From:**Paul Hirose

**Date:**2016 Jun 11, 21:42 -0700

Previously I've mentioned my Monte Carlo simulation of celestial navigation sight reduction with the HO 211 ("Ageton method") table. Recently the software has been improved to more closely replicate a human using a printed table, including linear interpolation between entries. It can also test HO 211 variants such as the Bayless table. More statistics are collected now too. Random observing sites and celestial bodies are generated with approximately uniform density on the Earth and celestial sphere. Latitude and altitude can be restricted to specified ranges. In the following tests latitudes were 0 to 70°, and altitudes 5° to 80°. In each case the statistics were accumulated from one million random test problems. The basic simulation chooses the closest table entry in every operation and does not interpolate. Another option is to interpolate as described in one of my previous messages: > "The interpolation of B(R) from A(R) will alone reduce the maximum error > to about two miles for K = 90°, 0.8 mile for K = 80°, and thus bring the > error within practical limits for most navigation problems." > > Samuel Herrick, THE ACCURACY OF AGETON'S METHOD IN CELESTIAL NAVIGATION, > Publications of the Astronomical Society of the Pacific, Vol. 56, No. > 331, p.149 (August 1944). > > http://adsabs.harvard.edu/abs/1944PASP...56..149H Now for my results. With no interpolation, HO 211 root mean square altitude error was 1.23', 14.6% exceeded half a minute, and worst was 31.3'. Azimuth RMS error was 2.76', 0.184% exceeded 0.5°, worst was 2.0°. (Note the mix of degrees and minutes in the azimuth statistics.) When every computation of B(R) from A(R) was interpolated, as suggested in the quote from Herrick, altitude RMS error improved greatly from 1.23' to .55', and errors more than half a minute decreased from 14.6% to 8.8%. But, contrary to Herrick's "two miles" claim, the worst altitude error was 25.0'. In his version of Ageton's table, Bayless recommends interpolating B(R) from A(R) if angle K (one of the intermediate results of the computation) is within 8° of 90°. That's much easier, since only 9% of sights are in that range. If we narrow the interpolation window further, to K = 90° plus or minus 5°, only 5% of sights fall in that range, yet accuracy is still much better than the non-interpolated solution. Still more work is saved if you interpolate when t (meridian angle), not K, is near 90°. That's because t is one of the inputs of the sight reduction, whereas K must be calculated, then recalculated if you decide to interpolate B(R). Whether you use K or t, the error statistics are almost identical. In other words, interpolating only the transformation of A(R) to B(R), when t is within 5° of 90, improves altitude accuracy significantly with only a small workload increase. Only 4.4% of the sights required interpolation. RMS altitude error was 0.59', 13.5% exceeded 0.5', worst altitude was 24.4' off. Azimuth RMS error was 2.65', .171% exceeded 0.5°, worst error was 2.0°. Whether or not you interpolate, the error distribution is not Gaussian. If it were, about 1/3 of the results would be more than one sigma (= the RMS value) from the true values. But if you interpolate per the preceding paragraph, only 9% of the altitudes and 3% of the azimuths exceed one sigma from the truth. With no interpolation, those figures become 3% and 4%. Compared to a Gaussian distribution, many fewer points are outside the 1-sigma zone, but they have a wider scatter. Speaking of the Bayless table, it's tabulated every minute instead of half minute. Another difference is that the precision of the A function increases from whole numbers to tenths at 85° instead of 84°. With no interpolation, altitude RMS error = 2.02', 43.0% exceed 0.5° error, worst error = 43.4'. Azimuth RMS error = 4.56', 0.503% exceeded 0.5' error, worst was 2.4°. If you interpolate B(R) from A(R) when t is within 5° of 90, altitude RMS error = 0.82', 41.8% exceed 0.5' error, worst error = 21.4' As expected, Bayless is generally less accurate than Ageton. However, in cases when B(R) is interpolated from A(R), the worst altitude error is consistently a little better with Bayless. The reason for this paradox is a mystery. Both tables increase the precision of function A from whole numbers to tenths when the angle is near 90°. Often I've wondered if that's worthwhile. In the case of standard Ageton, with no interpolation, RMS altitude error increases from 1.18' to 1.25' if all tabular values are whole numbers. The other error statistics have similar small increases. If B(R) is interpolated from A(R) when t is within 5° of 90, RMS altitude error increases from .59' to .68'. The other error statistics degrade too, though not so much. In my opinion, extracting A and B values to tenths of a unit from the standard Ageton table is not worth the extra work. Results are summarized below. The criterion for interpolation of B(R) from A(R) is shown on the right. For example, "K<8" means the program interpolated when K was within 8° of 90°. In some cases I also give the percentage of sights that meet the criterion. In a later message I'll explore some theoretical HO 211 variants, such as a 0.2' tabulation interval. Also, I plan to test the Sadler technique described in the Bayless book. alt alt alt az az az RMS >.5' max RMS >.5° max 1.23' 14.6% 31.3 2.76' .184% 2.0° Ageton no interp 1.26' 14.9% 31.5 2.87' .184% 2.2° no interp, whole numbers .55' 8.8% 25.0' 2.25' .111% 2.1° interp all .57' 12.8% 25.6' 2.65 .170% 1.9° interp K<8 8.5% .59' 13.6% 25.4' 2.65 .168% 2.1° interp K<5 5.3% .58' 12.8% 24.6' 2.68' .174% 2.1° interp t<8 7.3% .59' 13.5% 24.4' 2.65' .171% 2.0° interp t<5 4.4% .68' 14.0% 24.9' 2.82' .195% 2.3° interp t<5, whole numbers 2.02' 43.0% 43.4' 4.56' .503% 2.4° Bayless no interp .71' 34.2% 21.1' 3.56' .311% 2.4° interp all .77' 40.9% 20.7' 4.34' .463% 2.5° interp B(R) k<8 .80' 41.8% 22.0' 4.33' .460% 2.6° interp B(R) k<5 .78' 40.9% 21.1' 4.36' .460% 2.2° interp B(R) t<8 .82' 41.8% 21.4' 4.32' .459% 2.5° interp B(R) t<5