# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Sextant Observation Errors**

**From:**Dan Hogan

**Date:**1995 May 22, 16:18 PDT

The following courtesy of: The Navigation Foundation (Tel. 301-622-8448) P.O. Box 1126, Rockville MD 20850 The Navigator's Newsletter, IssueTwenty-Four, Spring 1989 Dan Hogan dhhogan@earthlink.net dhhogan@delphi.com ************* The Personal Equation, Variation in the Dip Correction, Random Error in Sextant Observations by John M. Luykx The two major sources of error in celestial observations are a) the observer's personal error in actually measuring the altitude of the celestial body with sextant and b) the variations in normal dip caused by changes in refraction due to sea/air surface temperature. Personal Error It is known that the technique, quality and accuracy of observation will vary from observer to observer. Each individual observer, however, can determine the quality of his observations by conducting a comprehensive series of tests over, generally a long period of time and with several celestial bodies. Such tests will provide siginificant results when the observer is satisfied that: a. the sextanr I.C. is correctly computed b. the sextant was in correct adjustment c. the observer's height of eye is accurately known d. the sextant was not "canted" during observation e. a "good" observation was made f. the most accurate sight reduction method was employed g. no computational errors were made A series of tests under the above conditions should be conducted by the observer: a. with different celestial bodies b. at various altitudes c. under various conditions of weather and sea d. at different places The altitudes obtianed during the tests should then be compared with computed values obtained by computer, sight reduction tables or mathematical computation. The difference between the two (observed and computed values) may be considered the personal error and applied with sign reversed to future observations as a personal correction. Dip Error The dip correction (height of eye) tabulated on the inside front cover of the Nautical Almanac is based on the following formula which incorporates a standard value for atmospheric refraction; D = 0.97 x SQRT h Where D = minutes of arc h = hight of eye in feet Significant variations in dip error caused by non-standard refraction may be experienced when large differences between sea and air temperatures at the surface occur. Because non-standard refraction is probably the single most important source of error in sextant observations, a considerable amount of research has been conducted to determine the mean values of refraction (used in the Dip tables), the conditions under which values differ from the mean and the value of such differences. The results of this research indicate generally that: a. When surface air temperature is less that the suface water temperature, the value of dip is greater than normal. b. When surface air temperature is greater than the surface water temperature, the value of dip is less than normal. When significant variation in sea/air surface temperature exists, corrections to the standard value of dip in the Nautical Almanac may be applied as follows (h. of e. 10'-30'): 1) For every ten degree decrease of thr surface air temperature below the value of the surface water temperature, the value of the tabulated dip is to be increased by 1 minute of arc. 2) For every ten degree increase in the surface air temperature above the value of the surface water temperature, the value of the dip is to be decreased by 1 minute of arc. Dip meters of varing types have been developed during the past 100 years to measure the actual value of dip for normal heights of eye and for any instant of time. These instruments are optical instruments which measure the vertical angle from a point on the horizion through the observers's zenith to the opposite point of the horizion. The angle subtracted from 180 and divided by 2 is equal to the actual value of dip at the time of observation. An instrument of this type developed by Admiral Davies, President of the Foundation, is described on pages 7-8 of issue 18 of the Newsletter (Fall. 1987). Random Error Individual observations by sextant taken on board a rolling, yawing, pitching vessel often contain large values of random observational error. To improve the reliability of position lines from such sextant observations, it is often recommended that a series of observations be taken and the results averaged to reduce random error. The mean time of observation and the mean value of altitude for the series of observations are then computed and used in sight reduction. Individual observations of the series may be plotted on cross section paper (altitude verses time) and a curve "faired" through a series of plotted points of observation. Unless the observations were made at or near the time of meridian passage, the faired curve would be nearly a straight line. Any point along the line may be used as an observation.The slope of the curve of observations when compared with the slope of a curve of altitude change, computed for the time of observation will further indicate the accuracy of individual observations in the series. The average of a series of observations was standard practice for celestial observations taken from aircraft. Accurate and reliable mechanical averaging devices for aircraft sextants [were manufactured] especially during World War II by C. Plath in germany, Henry Hughes in England and by Bendix and Kollsman in the United States. During World War II the Japanese sextant manufacturer, Tamaya, developed a mechanical averger whic was installed as an intergal part (not removeable) of a marine sextant of the Gago-Coutinho type.