# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Accuracy of position (sextant error simulation)**

**From:**Jim Manzari

**Date:**1999 Oct 21, 6:30 AM

Several years ago a similar question was discussed between myself and an acquaintance. He peaked my interest and I decided to attempt modeling sextant sight errors in a somewhat formal way. I focused on only these errors that involve the actual taking of a sextant altitude, the simulation ignores time errors, plotting errors (horizontal dilution of precision), or other errors such as table errors, blunders by the navigator, or unusual atmospheric conditions. The following is the short paper that resulted from this attempt to model sextant sight errors. I don't claim that the method I've used is perfect (or even correct!), it is offered it as food for thought. The simulation program is written in ANSI C and is available to anyone who wishes to play with it. To view the results in a browser, you will probably have to turn off "wrap long lines". Also be sure to look at the list of descriptive statistics symbols at the very bottom of the report in order to understand the meaning of these symbols in the results table. Unfortunately I'm must attend a funeral tomorrow, so if I don't respond quickly to any questions that arise, please be patient. Regards, Jim Manzari ---- Excerpted from previous email ---- It is important that you understand what the following simulation does not consider, such as errors introduced by inaccurate time or errors introduced when transferring the reduced sights to a plotting sheet. This simulation addresses only those errors arising in the use of the sextant. The conclusion is interesting: Given careful use of the sextant, accurate time, good observing weather, and no blunders on the part of the navigator, you should be able to fix your position to within plus or minus 2 nautical miles of the actual position. Misalignment of the sextant with the true vertical is THE major error to guard against! METHOD: ------- In order to estimate the scale of any possible errors I wrote a short program to simulate observations of 1000 randomly generated objects uniformly distributed between 15-60 degrees altitude (elevation angle). Next I generated a series of random values (of various statistical distributions) to simulate errors that might effect the accuracy of a position line. These simulated errors are: 1) Misalignment of the sextant's vertical axis with the local vertical, labeled Ve (vertical error) in the attached listing. hs = hs - ( hs * Cos( phi ) ) where hs = altitude of the object as measured by sextant, adjusted for error introduced by misalignment of sextant vertical axis with respect to the true local vertical, this angle called "phi". This is, by far, the largest single error and completely dominates all other errors, barring a blunder by the navigator!! This error varies directly by the height (elevation angle) of the object and the cosine of the misalignment angle. Therefore, low altitude stars will produce a smaller error for the same misalignment angle. Of course, a small misalignment error and a low object will produce the best results. In the simulation this error varied from 0 to -34 arc-minutes, with an average error of +1.8 arc-minutes. This would have the effect of placing the position line away from the actual position by 1.8 nautical miles in the direction of the object. See Dutton's for details regarding this type of error. 2) The sextant's index error, caused by sextant mis-calibration or changes in the sextant's mechanical structure due to temperature instabilities (Ie). Generally, a very small error, but may be a major contributor to overall error in some cheap plastic sextants. I don't know enough about plastic sextants to model this error realistically. In the simulation this error varied from -2.1 to +2.4 arc-minutes with an average of 0 arc-minutes. I assumed this error to be a normal/Gaussian distribution with a mean of 0 and a variance of 0.5 arc-minutes. This assumes no fixed bias in the sextant, which in fact may exist, but can be checked prior to each observation and removed from the solution. 3) The inability of the navigator to estimate the distance to the horizon or rather the exact line of the horizon. This may be caused by either wrongly estimated height of eye (dip) or changes in the height of eye due to wave action (dD). This may be a rather large error for all, but experienced navigators, and varies directly with the wave heights at the time of observation. I have assumed average wave height of 10 feet with variance 1/5 of this height for this simulation. dip = -60.0 * 0.0293 * sqrt( height-of-eye / 3.2808 ) where dip = correction for height of eye above LWL, in arc-minutes. Height-of-eye given in feet and converted to meters by 3.2... factor. The -60.0 factor produces the result with the correct sign and in arc-minutes. 4) The error in the refraction factor (f) caused by a difference between actual temperature and pressure at the time of the observation from the standard temperature of 10C and standard pressure of 1010 millibars. The error will effect the correction applied to refraction correction (r) to determine correct refraction correction (ro). For the simulation I have assumed a more reasonable temperature of 25C (77F) with a variance of 2 degrees around this mean value. For pressure I have assumed 1020 millibars with a variance of 25 millibars. These values, I believe are reasonable for someone cruising between 25N and 25S latitudes. In any case, these differences from the standard temperature and pressure produced only about 0.1 arc-minutes error. They can be disregarded for all practical purposes, except in extreme or unusual atmospheric conditions, when any sextant observations should be suspect anyway. Ro = -60.0 * ( 0.0167 / tan( Hs + 7.31 / ( Hs + 4.4 ) ) ) where Ro = refraction correction in arc-minutes. Hs = sextant height previously corrected for dip and index errors. R = f * Ro where R = refraction corrected for temperature and pressure differences from standard temperature and pressure. f = 0.28 * ( pressure + dP ) / ( temperature * dT + 273.0 ) where f = dimensionless factor used to adjust Ro. dP = error in estimated actual pressure at time of observation. dT = error in estimated temperature. CONCLUSION: ----------- Armed with all these corrections and their errors, I then calculated the corrected sextant altitude. This was then compared with the altitude with all the same corrections with no errors introduced. ho = hs + Ve + Ie + dip + dD + ro w/error Ho = hs + Dip + Ie + R w/o error dH = Ho - ho where dH = estimated error in sextant height, in arc-minutes. In this simulation the maximum error was -32.7 arc-minutes. 98% of this error was contributed by an extreme error in vertical alignment of the sextant. The mean error is -1.81 arc-minutes plus or minus 3.37 arc-minutes. TABLE OF RESULTS (see key to symbols and column label descriptions): -------------------------------------------------------------------- hs Ve Ie dip dD r f ro ho Dip R F Ro Ho dH -------------------------------------------------------------------------------------------------------- 15.1010v 0.0v -2.1v -3.1v -4.2v -3.6v 0.9441v -3.5v 14.9649v -3.1v -3.6v 0.9584v -3.5v 14.9918v -32.75v 59.8882^ 33.7^ 2.4^ -3.1^ 5.1^ -0.6^ 0.9773^ -0.6^ 60.0118^ -3.1^ -0.6^ 0.9584^ -0.6^ 59.8278^ 3.98^ 37.7077~ 1.8~ -0.0~ -3.1~ 0.1~ -1.5~ 0.9583~ -1.4~ 37.6627~ -3.1~ -1.5~ 0.9584~ -1.4~ 37.6324~ -1.81~ 37.6810M 0.7M 0.0M -3.1M 0.0M -1.3M 0.9582M -1.2M 37.6088M -3.1M -1.3M 0.9584M -1.2M 37.6093M -1.19M 59.5603! 0.0! -1.7! -3.1! -3.4! -3.6! 0.9488! -3.4! 33.9232! -3.1! -3.6! 0.9584! -3.5! 59.4998! -2.62! 12.9235D 3.0D 0.7D 0.0D 1.4D 0.8D 0.0051D 0.7D 12.9464D 0.0D 0.8D 0.0000D 0.7D 12.9353D 3.37D **** COLUMN LABEL DESCRIPTIONS **** hs = Sextant altitude of simulated object. Ve = Sextant vertical alignment error, arc-minutes. Ie = Sextant index error, arc-minutes. dip = Dip correction, arc-minutes. dD = Dip correction error, arc-minutes. r = Refraction correction, arc-minutes. f = Refraction factor, arc-minutes. ro = Total refraction correction, arc-minutes. ho = Height observed after corrections. Dip = Dip correction w/o random errors, arc-minutes. R = Refraction correction, arc-minutes. F = Refraction factor w/o random errors, arc-minutes. Ro = Refraction correction, arc-minutes. Ho = Height observed w/o random errors, degrees. dH = Difference with or without random errors, arc-minutes. **** KEY TO SYMBOLS **** * = sum ~ = mean ^ = max v = min # = count {at} = range ! = mode M = median V = variance D = stddev E = stderr S = skew K = kurtosis