# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Accuracy of position (sextant error simulation)**

**From:**Jim Manzari

**Date:**1999 Oct 25, 8:55 AM

George --please read my remarks interspersed below... George Huxtable wrote: > > This is a response too a posting made earlier today by Jim Manzari. > > His analysis illustrates the dangers in using statistics to draw > conclusions from unrealistic and incorrect assumptions. Before this thread gets too far off course due to your pejorative remarks and in the interests of education, which is the goal of this mailing list, let's make clear that my original post in this thread was intended to answer Geoffrey Kolbe's specific question: ...you say, "Right, that is what it is." Or, as an old sea salt do you say, "That's all very well, but in my experience, the actual altitude may be up to x' either side of that, so I will just keep that in mind when plotting my position." The question to the old sea salts is, how big is x? It is common practice, when faced with such a question, to model the errors in a systematic way to develop an overall error budget. In my view, the principle non-systematic sextant-related errors are... - Failure to align the sextant with the local vertical - Random or non-systematic errors in the sextant itself - Failure to find the correct horizon or height of eye - Failure to apply the correct refraction correction There are other sextant-related errors depending on the celestial object observed, but I don't know how to model them. The sun, moon, and planets suffer from several optical phenomenon that effect accuracy, such as irradiation, phase correction, glare, and limb darkening. These type of errors have much to do with the eye-sight or physiological capability of the observer, therefore I did not attempt to model these kinds of errors. Before going further it should be emphasised that sextant-related, non-systematic errors are truly _random_ in nature and can only be described in a statistical way. You may have a personal prejudice against the use of statistics to evaluate the scope of these errors, but your personal feelings won't change the fact that the errors are probabilistic. Indeed, Dutton's Navigation and Piloting has this to say... "The novice observer will find that his sights do not yield satisfactory lines of position...These sights should then be plotted on a large sheet of plotting paper...A "line of best fit" is then drawn through the string (of observations)...The divergence of the individual sights from this line will tend to indicate the magnitude of the observers random errors...The random error of a single sight is the greatest hazard to the accuracy of celestial navigation." > Let's deal first which what, according to Jim, is the "largest single error, > which dominates all others", and can give rise to an altitude error, in one > case, of -34 arc minutes. Perhaps I should have said..."the largest potential for error". Of the four types of errors enumerated in the analysis, this is the one with the greatest span -- from zero (0) to -34 arc-minutes. The other three types of errors have, at most, a span of only plus or minus 5 arc-minutes maximum. It is important to point out that the -34 arc-minute error occurred only once out of the total 1000 simulated observations. To model this error, I assumed a normal or Gaussian distribution for the mis-alignment angle with a mean or average value of zero and a variance of 5-degrees. > This depends entirely on what Jim has arbitrarily chosen as a reasonable > scatter for his angle phi, the tilt of the sextant from the vertical. In > that worst case of -34 minutes, my estimate of the tilt that produced it is > about 8 degrees from the vertical (it depends slightly on what the actual > altitude was for that measurement). No wonder tilt dominates all other > errors! For some reason you have focused on the single most extreme occurrence of vertical alignment error and have ignored the average error and its standard deviation. The average for this type of error was 1.8 arc-minutes with a standard deviation of plus or minus 3 arc-minutes. The average value should not give much cause for concern, but I'm surprised that you did not find the spread or standard deviation of this error "dominating". The following table shows the distribution of the vertical mis-alignment error (binned in 5 arc-minute bins)... Ve # of (arc-min) Occurrences ----------------------- 0.00 780 5.00 174 10.00 29 15.00 11 20.00 4 25.00 1 35.00 1 ----------------------- 1000 > What navigator would get a sighted body to kiss the horizon, and > then end up with an altitude corresponding to the value he would have got > with 8 degrees tilt, with enough error to put the upper limb of the Sun > where the lower limb ought to be? The notion is absurd. I ask Jim to state > how he has defined the scatter of sextant tilt, and on what basis he has > chosen that scatter. Is it realistic? You would know, if you had any real offshore experience, a navigator must frequently grab a quick sight of a star when it pops out from behind a cloud. In the tropics, for example, twilight lasts about 15-minutes and low, fast moving clouds are the norm. The navigator must work quickly. There is frequently no time to swing the sextant in the careful manner that is possible when working in your backyard or on a shoreline. When using an artifical horizon for practice, this problem of vertical alignment never happens. The simulation was intended to model the "randomness" of a real navigator at sea. Indeed, I think the simulation may have been overly optimistic. According to the simulation less than 50 of the observations would have an error of 10 arc-minutes or more. How many of us "old-salts" can say that we are consistently this good? The reason for multiple-star LOPs is to remove some of this "randomness". I normally shoot 6 stars and retain of the best 3 or 4 lines. In rough seas and cloudy weather I often see lines with errors of 10, 20, 30 or more miles due to the random nature of the sight-taking process. > > Another factor Jim has considered is sextant index error. This plays no > part whatsoever in the altitude error as long as the index error has been > checked and allowed-for (the easiest thing in the world to do). I > interleave index checks between each sextant sight, for my plastic sextant, > though it's not necessary to go that far. The index-error contribution to > altitude error is ZERO. It is wrong to assume that all the index-type error has been removed. My guess is that you can not see this error in your plastic sextant, because it lacks the necessary precision. The Plath sextant that I use has done good service for more than 25 years and when last checked by the manufacturer is accurate to within 12 arc-seconds (0.2 arc-minute) _anywhere_ on the arc. Also I've noticed that if I want 0.1 arc-minute accuracy when checking for index error, it depends on which direction I approach the reading when turning the adjustment knob. In addition, and there is always some random error in interpolating the graduations of the vernier. A simulation of 1000 sights shows that this error averages to be zero, but could be as large as plus or minus 0.7 arc-minutes in 33% of the observations (normal distribution with mean of 0 and variance of 0.5 arc-minutes). It is a small error, but it does add into the total error budget of the sextant and, in my view, must be included in any meaningful study of observational errors. > Jim has considered refraction of the light from the body, and correctly > found that at angles above 30 degrees, variations are negligible. This conclusion, George, is completely incorrect. The simulation attempts to model the error in the _refraction factor_ (f) caused by a difference between actual temperature and pressure at the time of the observation from the standard temperature of 10C and standard pressure of 1010 milli-bars. It has nothing to do with refraction as a function of elevation angle. George, I suggest you open a Nautical Almanac to the first few pages. You will see a special table designed to adjust refraction for non-standard air temperature and pressure. Standard refraction corrections assume a temperature of 10C (50F) and a barometric pressure of 1010mb (29.83 inches). What is the error introduced by ignoring this correction? That was the purpose of the simulation, not as you say to determine that refraction varies with elevation angle. > But he has ignored the unpredictable changes in the dip caused by refraction > in the light path from the horizon, an important factor if not THE most > important factor. See my mailings on this topic earlier today. Yes, I have ignored mirages, etc, just as I said I would in the introduction to my original post and in the paper itself. And for good reasons...(1) Bubbles (mirages) in the air mass close to the horizon are relatively rare, except when the sea and air temperature are markedly different, or when sights are taken over land, or in a few locations in the world such as the Persian Gulf; (2)I don't know how to model this kind of rare event. George, if you had any real experience working as a navigator you would probably know that conditions giving rise to air mass non-uniformity near the horizon are relatively easy to detect. Experienced navigators are sensitive to this problem whenever the air/sea temperature differential is large, whenever the sun or the moon appear distorted when rising or setting, or whenever a sight is taken across a land mass (island or peninsula). I suggest you spend some time studying a copy of the old fat Bowditch or Duttons. Both these books have a long discussion of this subject. In my view, this type of error is too rare (and too easily detected) to consider in an general study of sight errors. > I could go on, but I think I've said enough. Well, George, perhaps it is good you did not go on. Otherwise, I would feel compelled to continue answering your mistaken conclusions lest they be accepted by default. Regards, Jim Manzari