Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Accuracy of position (sextant error simulation)
    From: Jim Manzari
    Date: 1999 Oct 21, 6:30 AM

    Several years ago a similar question was discussed between myself and an
    acquaintance.  He peaked my interest and I decided to attempt modeling sextant
    sight errors in a somewhat formal way.  I focused on only these errors that
    involve the actual taking of a sextant altitude, the simulation ignores time
    errors, plotting errors (horizontal dilution of precision), or other errors
    such as table errors, blunders by the navigator, or unusual atmospheric
    The following is the short paper that resulted from this attempt to model
    sextant sight errors.  I don't claim that the method I've used is perfect (or
    even correct!), it is offered it as food for thought.  The simulation program
    is written in ANSI C and is available to anyone who wishes to play with it.
    To view the results in a browser, you will probably have to turn off "wrap
    long lines".  Also be sure to look at the list of descriptive statistics
    symbols at the very bottom of the report in order to understand the meaning of
    these symbols in the results table.
    Unfortunately I'm must attend a funeral tomorrow, so if I don't respond
    quickly to any questions that arise, please be patient.
    Jim Manzari
    ---- Excerpted from previous email ----
    It is important that you understand what the following simulation does not
    consider, such as errors introduced by inaccurate time or errors introduced
    when transferring the reduced sights to a plotting sheet.  This simulation
    addresses only those errors arising in the use of the sextant.
    The conclusion is interesting: Given careful use of the sextant, accurate
    time, good observing weather, and no blunders on the part of the navigator,
    you should be able to fix your position to within plus or minus 2 nautical
    miles of the actual position.
    Misalignment of the sextant with the true vertical is THE major error to guard
    In order to estimate the scale of any possible errors I wrote a short program
    to simulate observations of 1000 randomly generated objects uniformly
    distributed between 15-60 degrees altitude (elevation angle).
    Next I generated a series of random values (of various statistical
    distributions) to simulate errors that might effect the accuracy of a position
    line.  These simulated errors are:
    1) Misalignment of the sextant's vertical axis with the local vertical,
    labeled Ve (vertical error) in the attached listing.
            hs = hs - ( hs * Cos( phi ) )
    where hs = altitude of the object as measured by sextant, adjusted for error
    introduced by misalignment of sextant vertical axis with respect to the true
    local vertical, this angle called "phi".
    This is, by far, the largest single error and completely dominates all other
    errors, barring a blunder by the navigator!!
    This error varies directly by the height (elevation angle) of the object and
    the cosine of the misalignment angle.  Therefore, low altitude stars will
    produce a smaller error for the same misalignment angle.  Of course, a small
    misalignment error and a low object will produce the best results.
    In the simulation this error varied from 0 to -34 arc-minutes, with an average
    error of +1.8 arc-minutes.  This would have the effect of placing the position
    line away from the actual position by 1.8 nautical miles in the direction of
    the object.
    See Dutton's for details regarding this type of error.
    2) The sextant's index error, caused by sextant mis-calibration or changes in
    the sextant's mechanical structure due to temperature instabilities (Ie).
    Generally, a very small error, but may be a major contributor to overall error
    in some cheap plastic sextants.  I don't know enough about plastic sextants to
    model this error realistically.
    In the simulation this error varied from -2.1 to +2.4 arc-minutes with an
    average of 0 arc-minutes.  I assumed this error to be a normal/Gaussian
    distribution with a mean of 0 and a variance of 0.5 arc-minutes.  This assumes
    no fixed bias in the sextant, which in fact may exist, but can be checked
    prior to each observation and removed from the solution.
    3) The inability of the navigator to estimate the distance to the horizon or
    rather the exact line of the horizon.  This may be caused by either wrongly
    estimated height of eye (dip) or changes in the height of eye due to wave
    action (dD).  This may be a rather large error for all, but experienced
    navigators, and varies directly with the wave heights at the time of
    observation.  I have assumed average wave height of 10 feet with variance 1/5
    of this height for this simulation.
            dip = -60.0 * 0.0293 * sqrt( height-of-eye / 3.2808 )
    where dip = correction for height of eye above LWL, in arc-minutes.
    Height-of-eye given in feet and converted to meters by 3.2... factor.  The
    -60.0 factor produces the result with the correct sign and in arc-minutes.
    4) The error in the refraction factor (f) caused by a difference between
    actual temperature and pressure at the time of the observation from the
    standard temperature of 10C and standard pressure of 1010 millibars.  The
    error will effect the correction applied to refraction correction (r) to
    determine correct refraction correction (ro).
    For the simulation I have assumed a more reasonable temperature of 25C (77F)
    with a variance of 2 degrees around this mean value.  For pressure I have
    assumed 1020 millibars with a variance of 25 millibars.  These values, I
    believe are reasonable for someone cruising between 25N and 25S latitudes.
    In any case, these differences from the standard temperature and pressure
    produced only about 0.1 arc-minutes error.  They can be disregarded for all
    practical purposes, except in extreme or unusual atmospheric conditions, when
    any sextant observations should be suspect anyway.
            Ro = -60.0 * ( 0.0167 / tan( Hs + 7.31 / ( Hs + 4.4 ) ) )
    where Ro = refraction correction in arc-minutes.  Hs = sextant height
    previously corrected for dip and index errors.
            R = f * Ro
    where R = refraction corrected for temperature and pressure differences from
    standard temperature and pressure.
            f = 0.28 * ( pressure + dP ) / ( temperature * dT + 273.0 )
    where f = dimensionless factor used to adjust Ro.  dP = error in estimated
    actual pressure at time of observation.  dT = error in estimated temperature.
    Armed with all these corrections and their errors, I then calculated the
    corrected sextant altitude.  This was then compared with the altitude with all
    the same corrections with no errors introduced.
            ho = hs + Ve + Ie + dip + dD + ro     w/error
            Ho = hs + Dip + Ie + R                w/o error
            dH = Ho - ho
    where dH = estimated error in sextant height, in arc-minutes.  In this
    simulation the maximum error was -32.7 arc-minutes.  98% of this error was
    contributed by an extreme error in vertical alignment of the sextant.
    The mean error is -1.81 arc-minutes plus or minus 3.37 arc-minutes.
    TABLE OF RESULTS (see key to symbols and column label descriptions):
         hs    Ve    Ie   dip    dD     r       f    ro       ho   Dip     R       F    Ro       Ho      dH
    15.1010v  0.0v -2.1v -3.1v -4.2v -3.6v 0.9441v -3.5v 14.9649v -3.1v -3.6v 0.9584v -3.5v 14.9918v -32.75v
    59.8882^ 33.7^  2.4^ -3.1^  5.1^ -0.6^ 0.9773^ -0.6^ 60.0118^ -3.1^ -0.6^ 0.9584^ -0.6^ 59.8278^   3.98^
    37.7077~  1.8~ -0.0~ -3.1~  0.1~ -1.5~ 0.9583~ -1.4~ 37.6627~ -3.1~ -1.5~ 0.9584~ -1.4~ 37.6324~  -1.81~
    37.6810M  0.7M  0.0M -3.1M  0.0M -1.3M 0.9582M -1.2M 37.6088M -3.1M -1.3M 0.9584M -1.2M 37.6093M  -1.19M
    59.5603!  0.0! -1.7! -3.1! -3.4! -3.6! 0.9488! -3.4! 33.9232! -3.1! -3.6! 0.9584! -3.5! 59.4998!  -2.62!
    12.9235D  3.0D  0.7D  0.0D  1.4D  0.8D 0.0051D  0.7D 12.9464D  0.0D  0.8D 0.0000D  0.7D 12.9353D   3.37D
    hs  = Sextant altitude of simulated object.
    Ve  = Sextant vertical alignment error, arc-minutes.
    Ie  = Sextant index error, arc-minutes.
    dip = Dip correction, arc-minutes.
    dD  = Dip correction error, arc-minutes.
    r   = Refraction correction, arc-minutes.
    f   = Refraction factor, arc-minutes.
    ro  = Total refraction correction, arc-minutes.
    ho  = Height observed after corrections.
    Dip = Dip correction w/o random errors, arc-minutes.
    R   = Refraction correction, arc-minutes.
    F   = Refraction factor w/o random errors, arc-minutes.
    Ro  = Refraction correction, arc-minutes.
    Ho  = Height observed w/o random errors, degrees.
    dH  = Difference with or without random errors, arc-minutes.
    **** KEY TO SYMBOLS ****
    * = sum      ~ = mean     ^ = max      v = min        # = count
    {at} = range    ! = mode     M = median   V = variance   D = stddev
    E = stderr   S = skew     K = kurtosis

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site