Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Accuracy of sextant observations at sea
    From: Gary LaPook
    Date: 2010 Nov 20, 03:00 -0800

    For those who don't want to do the math themselves, I have attached an
    extended table Q7d showing the relationship of standard deviation,
    sigma, and Circular Probable Error, CEP, to the percentage of fixes
    contained in circles of such radius.
    On 11/19/2010 2:40 PM, Gary LaPook wrote:
    > Jeremy, I am trying to make sense out of your fix and LOP statistics.
    > There is an inconsistency between your LOP sigma of .800225 NM and the
    > statistics for the fix accuracy. One sigma will only contain 39.35% of
    > the fixes but you show 60% of the fixes within only 0.5 NM, which is
    > .62 sigma, which should contain only 17.7% of the fixes. It should
    > take 1.354 sigma to contain 60% of the fixes so should be 1.08 NM
    > instead of the the 0.5 NM that you state.
    > Similarly, you show 1.0 NM, which is 1.25 sigma, containing 88.6% of
    > the fixes but 1.25 sigma should only contain 54.2%. It takes a circle
    > of 2.084 sigma to contain 88.6% of the fixes so should be 1.67 NM
    > instead of the 1.0 NM that you state.
    > Looking at the discrepancy from the other direction, if 88.6% of the
    > fixes were contained within just 1.0 NM (which means that 1.0 NM
    > equals 2.084 sigma) then sigma for the LOPs must have been only .481
    > NM so there must be an error in the .800225 NM value that you stated.
    > So either my math or your math (maths for our English friends) must be
    > in error (and I am not ruling out that it might be mine.) I am
    > attaching an excerpt from Bowditch that explains the statistics. Note,
    > to convert table Q7d to sigma from CEP just multiply each value by
    > 1.177 which is the ratio between these two measures. You can use the
    > given formulas to compute different probabilities and sigmas.
    > gl
    > ------------------------------------------------------------
    > Jeremy wrote:
    > 'll quote myself here [NavList 9887]:
    > On Sep 23, 2009 3:40 pm, Anabasi...---com wrote:
    > > I shot 70 star fixes on my trip from Japan to the USA and here are a
    > few
    > > interesting statistics.
    > >
    > > 1) average fix error from GPS: 0.566 nm (low of 0.0 and high of 2.7 nm
    > > error)
    > >
    > > 2) 88.6% of the fixes were under 1.0 nm fix error.
    > >
    > > 3) 60.0% of the fixes were under 0.5 nm fix error
    > >
    > > 4) two fixes had errors under 0.05 nm (under 100 meters).
    > >
    > > My next project is to determine the average intercept based on the
    > GPS fix
    > > as the DR but that is over 700 data points so might be awhile.
    > >
    > > Jeremy
    > As you can see, these are position errors, not LOP errors. I also ran
    > standard Dev for my fixes and it was 0.5813 nm. I will add that my
    > worst fix was 2.7 nm off under awful conditions.
    > The trouble I have with LOP statistics is that there are so many
    > variables at play as render them only interesting but not very useful.
    > With a good sky and horizon I can pinwheel every time (intercepts well
    > under 1 nm), if the horizon and/or sky is terrible, I can have errors
    > of several miles. Also, can someone expect the aforementioned
    > statistics in a bouncing sailboat as opposed to a large ship? Probably
    > not.
    > I just ran the statistics for the individual LOPs, 551 in all. These
    > were shot in a wide variety of conditions including the haze of the
    > East China Sea, near Cape Hope, and in the tropics. There is a wide
    > variation of error from day to day and twilight to twilight depending
    > on the environment.
    > I only ran them on the computer reduced sights as these use GPS fixes
    > for the AP as opposed to my tabular reductions. (planets, the moon,
    > and stars) I didn't use the computer for most of my sunlines, but
    > there are a few included in there.
    > Standard deviation for all 551 sights was .800225.
    > I did shoot 1 moonlight fix. The position error was about 2.5 nm which
    > would be good enough in a pinch, but well below my standard.
    > Jeremy
    > ----------------------------------------------------------------
    > NavList message boards and member settings: www.fer3.com/NavList
    > Members may optionally receive posts by email.
    > To cancel email delivery, send a message to NoMail[at]fer3.com
    > ----------------------------------------------------------------



    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site