# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Resolution of systematic error**

**From:**Gary LaPook

**Date:**2007 Mar 06, 01:33 -0800

Gary LaPook wrote: We may be talking past each other. Let's make sure we are discussing the same thing. We are only talking about a position determined by celestial observations involving only random error and that any possible systematic error has been eliminated. The "fix" is the spot marked on the chart derived from analysis if the celestial observations. The "position of the observer" is the location where the observer was located when he took the sights and which can also be plotted on the chart. "Triangle" and "cocked hat" are names for the figure formed by plotting three lines of position on the chart that do not pass through the exact same point. Let's take, to start with, the simpler situation involving only two lines of position and let's have them cross at a 90� angle. In this situation we take their intersection as the fix since there is no triangle to be inside or outside of. However, no one actually believes that the position of the observer is actually located at this intersection given the normal random errors of observation and computation. In fact, we can draw circles around the fix at various radii to depict the various probabilities that the position of the observer is located within the various circles. This can also described as various levels of confidence. If the two lines do not cross at a right angle then you can draw ellipses for the same purpose. This is covered in great detail in appendix Q of the the 1977 edition of Bowditch. The size of the circles and ellipses is determined by the uncertainty in the sights expressed in terms of "standard deviation" or "sigma." I have spent a great deal of time lately analyzing the standard deviation of sights taken with bubble sextants and is appears to be about two minutes of arc or a two nautical mile sigma.. This also comports with the requirements of Federal Aviation Regulation part 63 (14 CFR 63, et. seq.) which sets the accuracy requirement for celestial fixes obtained in flight. Sights taken with a marine sextant should have a much smaller sigma but will be larger than just the accuracy of the instrument taking into account an indistinct horizon, waves on the horizon, observer bouncing up and down in heavy seas changing the dip, etc. For the sake of this discussion let's assume the sigma of sights taken with the marine sextant have a sigma of one nautical mile. Based on the analysis of this situation the position of the observer will be somewhere within the one mile circle (one sigma) centered on the plotted fix 39.3% of the time and within a circle of 1.177 sigma (1.177 NM) 50% of the time. This circle is also known as CEP or circular probable error. Continuing drawing circles, 66% within 1.48 NM, 75% within 1.67 NM, 90% within 2.15 NM, 99% within 3 nm and 99.9% of the time within 3.72 NM. At the other end of the distribution there is only a 10% chance that the position of the observer will be within .48 NM of the fix. ( See table Q6c on page 1221 of vol. 1 of Bowditch, 1977 ed.) So, what does this tell us. There is about a 30% chance that the position of the observer will be more than .48 NM but less than 1.0 NM and about a 61% chance that the position of the observer will be more than 1.0 NM from the plotted fix. So what do we do with this knowledge? We use the plotted fix at the intersection of the two LOPs for our navigational purposes such as measuring our progress and planning the next leg. We also use this fix to deal with the proximity of danger keeping always in mind that the vessel may actually be almost 4 NM from the fix in any direction. Why do we use the intersection as the fix, because there is no better one available since this spot marks the center of possible positions of the observer. No other spot would be as useful for planning purposes or avoiding danger. Also, what methodology would you use in determining another spot to mark the fix? Now moving onto the three line fix derived from three observations resulting in a triangle. The same analysis holds with the same circles of uncertainty since the boat doesn't know that you took three sights this time instead of just two. The only question left is where should we plot the fix to mark the center of these circles? For very practical reasons we take the center of the triangle as the fix. The size of the triangle is limited. If you wanted to plot the fix somewhere outside the triangle how would you decide where to place it, the choices are unlimited with no way to chose between. Again, no one is suggesting that the position of the observer is at the center of the triangle but this represents the center of possible positions of the observer. In fact, the position of the observer will be outside of the triangle often but I don't agree with the three out of four allegation. Counter intuitively, the smaller the triangle the more likely that the position of the observer is outside the triangle! If you think about it, this should be obvious. Using reducio ad absurdum, think about a triangle only one inch in size, it would be impossible for the observer to be within the triangle. At the other extreme, a very large triangle with all of the displacements of the LOPs from the center of the triangle equal to 3.3 NM (3.3 sigma's, linear sigma's are slightly different than circular sigma's, see Bowditch), the only place that the position of the observer could be is at the fix in the center of the triangle! On Mar 2, 5:56 am, Geoffrey Kolbewrote: > I agree you must make practical assumptions. And the practical assumption > that the position is inside the cocked hat is commonly held - but it is not > correct. In fact, where the size of the cocked hat is only determined by > random errors, the chances of the actual position being inside the cocked > hat is just one in four. So, the practical assumption you should made is > that the actual position is NOT inside the cocked hat. > > See the previous discussion under the subject of "Cocked Hats" back in > December last, and in particular my posting (NavList 1908) on the 16th of > December for more detail. > > Geoffrey Kolbe > > At 11:40 02/03/2007, you wrote: > > >Well, maybe it wouldn't be, in fact it might be many thousands of miles > >away but that is very unlikely (though with a probability greater than zero.) > > >But, the basic assumption in plotting the fix, after you have eliminated > >the systematic or constant error leaving only random errors, is that you > >are most likely located at the point where you are equal distant from the > >position lines and this is generally taken to be in the center of the > >triangle. "If a fix is obtained from three or more lines of position, and > >the error of each line is normal and equal to that of the others, the most > >probable position is the center of the figure. By 'center' is meant that > >point within the figure which is equidistant from the sides." Bowditch, > >volume 2, chapter III article 308, 1977 ed.; and see volume 1, chapter > >XVII, article 1708 , 1977 ed. Although it is "possible" to be way outside > >of the triangle you must make some practical assumptions otherwise > >everything becomes Jell-O. > > >What is your argument that after eliminating the systematic error that the > >fix is more likely to be outside the triangle than inside? > > >Gary LaPook --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to NavList@fer3.com To unsubscribe, send email to NavList-unsubscribe@fer3.com -~----------~----~----~----~------~----~------~--~---