# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Accuracy of sextant observations at sea**

**From:**Gary LaPook

**Date:**2010 Nov 20, 03:00 -0800

For those who don't want to do the math themselves, I have attached an extended table Q7d showing the relationship of standard deviation, sigma, and Circular Probable Error, CEP, to the percentage of fixes contained in circles of such radius. gl On 11/19/2010 2:40 PM, Gary LaPook wrote: > > Jeremy, I am trying to make sense out of your fix and LOP statistics. > > There is an inconsistency between your LOP sigma of .800225 NM and the > statistics for the fix accuracy. One sigma will only contain 39.35% of > the fixes but you show 60% of the fixes within only 0.5 NM, which is > .62 sigma, which should contain only 17.7% of the fixes. It should > take 1.354 sigma to contain 60% of the fixes so should be 1.08 NM > instead of the the 0.5 NM that you state. > > Similarly, you show 1.0 NM, which is 1.25 sigma, containing 88.6% of > the fixes but 1.25 sigma should only contain 54.2%. It takes a circle > of 2.084 sigma to contain 88.6% of the fixes so should be 1.67 NM > instead of the 1.0 NM that you state. > > Looking at the discrepancy from the other direction, if 88.6% of the > fixes were contained within just 1.0 NM (which means that 1.0 NM > equals 2.084 sigma) then sigma for the LOPs must have been only .481 > NM so there must be an error in the .800225 NM value that you stated. > > So either my math or your math (maths for our English friends) must be > in error (and I am not ruling out that it might be mine.) I am > attaching an excerpt from Bowditch that explains the statistics. Note, > to convert table Q7d to sigma from CEP just multiply each value by > 1.177 which is the ratio between these two measures. You can use the > given formulas to compute different probabilities and sigmas. > > gl > > > ------------------------------------------------------------ > > Jeremy wrote: > > 'll quote myself here [NavList 9887]: > > On Sep 23, 2009 3:40 pm, Anabasi...---com wrote: > > I shot 70 star fixes on my trip from Japan to the USA and here are a > few > > interesting statistics. > > > > 1) average fix error from GPS: 0.566 nm (low of 0.0 and high of 2.7 nm > > error) > > > > 2) 88.6% of the fixes were under 1.0 nm fix error. > > > > 3) 60.0% of the fixes were under 0.5 nm fix error > > > > 4) two fixes had errors under 0.05 nm (under 100 meters). > > > > My next project is to determine the average intercept based on the > GPS fix > > as the DR but that is over 700 data points so might be awhile. > > > > Jeremy > > > As you can see, these are position errors, not LOP errors. I also ran > standard Dev for my fixes and it was 0.5813 nm. I will add that my > worst fix was 2.7 nm off under awful conditions. > > The trouble I have with LOP statistics is that there are so many > variables at play as render them only interesting but not very useful. > With a good sky and horizon I can pinwheel every time (intercepts well > under 1 nm), if the horizon and/or sky is terrible, I can have errors > of several miles. Also, can someone expect the aforementioned > statistics in a bouncing sailboat as opposed to a large ship? Probably > not. > > I just ran the statistics for the individual LOPs, 551 in all. These > were shot in a wide variety of conditions including the haze of the > East China Sea, near Cape Hope, and in the tropics. There is a wide > variation of error from day to day and twilight to twilight depending > on the environment. > > I only ran them on the computer reduced sights as these use GPS fixes > for the AP as opposed to my tabular reductions. (planets, the moon, > and stars) I didn't use the computer for most of my sunlines, but > there are a few included in there. > > Standard deviation for all 551 sights was .800225. > > I did shoot 1 moonlight fix. The position error was about 2.5 nm which > would be good enough in a pinch, but well below my standard. > > Jeremy > ---------------------------------------------------------------- > NavList message boards and member settings: www.fer3.com/NavList > Members may optionally receive posts by email. > To cancel email delivery, send a message to NoMail[at]fer3.com > ---------------------------------------------------------------- > > >