# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Accuracy of position**

**From:**George Huxtable

**Date:**1999 Oct 20, 6:11 PM

I think some points have been missed in the discussion of Dr Kolbe's question about errors in sextant altitudes at sea. If altitudes at sea could be measured with respect to the observer's vertical, the point directly over his head, which on land might be determined with an accurate plumb-bob or spirit-level, high accuracy could be obtained. Unfortunately, at sea the observer in a moving vessel has to measure altitudes with respect to the horizon. Such measurements are subject to the following inaccuracies- 1) In rough weather, the image of the observed object and of the horizon are leaping about in the field of view of the sextant, particularly when seen from a small vessel. It's a great skill to get these images well aligned, and the sextant vertical, in such conditions, a skill that I don't claim to possess. 2) From a small vessel, in rough weather, the horizon is by no means the smooth horizontal line that's found in the textbooks. The horizon consists of the peaks of waves, and the observer's vessel is also going up and down in the waves, from peaks to troughs. The best that can be done is to measure altitudes with respect to local wave-peaks, at a time when the vessel too is poised on a wave-peak; not an easy matter. From a large vessel with a high bridge to observe from, the horizon is much further away, and so it looks much flatter. These two contributions above are important only under rough conditions, and it's obvious to the observer when his measurements are being seriously affected by a rough sea. And perhaps Dr Kolbe's original question was really asking what accuracy is obtainable in calm weather. In which case, we can discount the two contributions above and consider a third, more insidious, source of error, as follows- 3) Abnormal refraction. If we can confine our observations to bodies with an altitude over, say, 30 degrees, the refraction correction to the light path from the body is very small, and any divergence from the standard value caused by local meteorolical influence is quite negligible. However, this can't be said of the light path from the horizon. And remember, we're measuring the angle between those two light paths. A correction has to be made to all sextant observations because of the dip of the horizon. Most of the dip correction is a purely geometrical effect resulting from the curvature of the Earth and the height of the observer. But part of the dip correction (about a fifth of it) is due to the bending of light by refraction in the air, in its path from just skimming the horizon to the eye of the observer. It's interesting to note that if the density of our atmosphere was about five times greater than it presently is, a light beam would curve so as to just follow the horizon, and we could see vessels "around" the horizon, as far away as we liked, if the air was clear enough. "Normal" refraction operates to reduce the correction for dip, and is taken into account when the tables for dip are calculated. For example, from a height-of-eye of 10 feet, the effective dip is 3.1 minutes, made up of a geometrical dip of 3.7 minutes, and a normal-refraction contribution of -0.6 minutes. The big problem is that this refraction is frequently far-from-normal. The light is skimming along within a few feet of the sea surface, and significant temperature gradients can occur in that air layer, particularly in calm conditions. We have all observed, on certain days, the strange sight of distant vessels on the horizon appearing to be stretched vertically, or even to float right above the horizon. These are days of abnormal refraction, and on such days you should not trust your sextant sights to a high accuracy. The problem arises when there aren't any other ships around to demonstrate these conditions. The horizon itself looks no different from the way it looks any other day. Your sights may suffer from abnormal refraction, and there's no way to know it. So how big might the error due to abnormal refraction become? If there has been a scientific survey of this matter, I'm unaware of it. But I understand that discrepancies of 1 minute from the normal value of horizon refraction are by no means uncommon. What's the maximum discrepacy that has been observed? I have no idea, but would welcome further information from anyone that does. This variability of horizon refraction sets a limit on the accuracy of all sextant altitude observations made up from the horizon. It doesn't matter how precise your sextant is, or how carefully you made the observation, or how often you repeated it, or how accurately you did the sight reduction, variable horizon refraction may affect the result. And you won't know. One way to alleviate this error when position-finding is to make a range of measurements at different azimuths, particularly if you can make pairs of measurements at opposite directions in the sky. Then this horizon error will enlarge your resulting cocked hat, but won't displace its centre from the true position. During much of the 19th century,one of the goals of accurate sextant measurement was to find the longitude from a lunar distance, the angle-in-the-sky between the Moon and another body, the Sun or a star. High accuracy was prized because the error in the resulting longitude was about 30 times the error in the lunar distance measurement. Because the horizon did not play a significant part in a lunar distace obsevation, the full accuracy of the sextant was available, unperturbed by the horizon effects referred to above. Those were the observations for which sextants with very high calibration accuracy were developed, accuracy that became greater than necessary once lunars were dropped in favour of chronometers and radio signals. Note that the effect of horizon refraction doesn't affect altitude measurements that are made on land from a theodolite, nor measurements such as Dr Kolbe's, made from land with a sextant and an artificial horizon. One of his questions concerned the ultimate accuracy of sextant observations made at sea, and this has been my shot at answering it. George Huxtable. ------------------------------ george@huxtable.u-net.com George Huxtable, 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK. Tel, or fax, to 01865 820222 or (int.) +44 1865 820222. ------------------------------