Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Rejecting outliers
    From: Gary LaPook
    Date: 2011 Jan 8, 02:58 -0800

    (Since I never know how the formatting will work out I am attaching a PDF also)

    We have had some interesting discussions lately about improving the accuracy of our celestial
    navigation. First we reexamined the old cocked hat and discovered that there are many new ways
    to choose a center. We learned new words such as incenter, centroid, steiner point and
    symmedian point. Then we moved on to trying to improve the accuracy of the individual LOPs
    making up the cocked hat with some suggestions of averaging many sights, linear regression,
    least squares, and the use of a predicted slope to identify �outliers� that should be eliminated
    prior to averaging. My own opinion is that one sight each on two well spaced stars provides an
    accurate enough fix for normal off shore navigation. Shoot a third star to eliminate certain types
    of errors and you then have a fix that you can rely on in laying out the course for the next day.
    But I can envisage some situations where it would make sense to try to improve the accuracy of
    the fix such as when approaching land or when passing a mid-ocean reef. In these situations I
    wouldn�t use the possibly more accurate fix to cut it closer to the danger but would use the
    increased accuracy to provide more peace of mind.

    Peter Fogg described a method of graphing the sights and comparing them with a line drawn on
    the graph at the slope that the sights are expected to follow during the shooting period.. This
    method had also been mentioned in Dutton�s. This is different than the other methods discussed
    as this is to be the first step that could be used to identify data that should be eliminated prior to
    applying the other techniques. I think that everyone will agree that clearly erroneous data should
    be discarded but the question is how do you decide when data is so bad that it should be
    eliminated. If you detect some obvious problem, such as realizing that your watch had stopped,
    then the answer is easy. But when it is not so obvious then Fogg�s technique might make sense.
    So how do we choose which points are far enough away from the expected sloped line to be
    eliminated? I think a way to make this decision is to consider the normal Gaussian distribution. A
    navigator should have a good idea, based on his experience, what his sigma should be. He might
    have several, based on different observing conditions that he can apply in deciding which data
    points should be rejected. If you take a thousand sights you should expect 50 to exceed 2 sigmas
    and since this would be part of a normal distribution you should not exclude these points in
    calculating an average. But if you only take five shots you should not expect any to exceed two
    sigmas (though it is possible but highly unlikely) if they are part of a normal distribution. So my
    take on this is that for a small number of shots you should exclude any that exceed two sigmas
    since it is more likely that this is a result of some anomalous event and not part of a normal
    distribution that should be included in the average.

    I decided to try this with the data I had supplied for a series of sights of Venus taken on an
    Atlantic crossing in 2009. Although I provided fifteen sights I will concentrate on only six as
    they were taken in a six minute period while the others were taken over a much longer period of
    time. The first photo attached shows the six sights plotted on a graph, each grid line on the X axis
    is one minute of altitude and the time intervals are three seconds. (The data for these shots is
    attached at the bottom.) That was the easy part.

    Now we have to decide how to draw the slope of the expected change in altitude in the six
    minute period. There are two ways to compute this slope. You can pick an assumed position
    nearby and calculate the computed altitude at the beginning and at the end (or near the end) of the
    period, plot those two points and then draw a line between them. Off course a graph of computed
    altitudes will not coincide with the plotted observed altitudes but the slope should be the same,
    possibly being different my one tenth of a minute if there was a great enough change in the
    altitudes to cause a slight change in the refraction corrections. If the ship was moving during the
    shooting period then the slope will need to be adjusted to account for that also, either using a
    second A.P. for calculating the second altitude or by using the H.O. 249 Motion Of the Observer
    (MOO) table or the equivilent table that I computed and posted before.


    The other way is to determine the slope is by using the Motion Of the Body (MOB) correction
    table in H.O. 249. This is easier and can be done without looking at the Almanac as the only
    information needed is the DR latitude and the azimuth of the body, which can be measured
    accurately enough for this purpose. You can also calculate this with the formula: Delta H = 15 x
    cos latitude x sin azimuth, where Delta H = rate of change in arc minutes per minute of time.
    To account for the movement of the vessel you can also use the MOO table from H.O. 249 or the
    table I have previously posted that accomplishes the same thing.


    Using my data for this, my DR latitude was 14̊ 25' north and the azimuth of Venus was 103̊
    true. We look at the H.O 249 MOB table (attached) and we find (by visual interpolation) the
    altitude of the body is increasing 14.2' for every minute of time. For a five minute period this
    would result in 71.0' increase in altitude. To allow for the movement of the ship, which was
    sailing at 11 knots on a course of 251̊ true which placed Venus at a relative azimuth of 148̊, we
    can look at the table I created for the Motion of the Observer for 11 knots and visually
    interpolate between 140̊ and 160̊ relative azimuth and take out the correction of -0.8' for a five
    minute correction. If you want to use the H.O. 249 MOO table (attached) take out the value for
    550 knots and a relative azimuth of 148̊ which is -7.8' per minute of time. Divide by 100 to find
    the correction for 5.5 knots which is - 0.078' and then multiply by two to find the correction for
    11 knots resulting in -0.156' per minute. Multiply by 5 to find the total five minute Motion Of the
    Observer correction which is -0.78' rounded to -0.8', the same as my table. Combining the
    Motion Of the Observer and the Motion Of the Body correction and we find the slope to be 70.2'
    for five minutes.

    The second photo shows where I drew in the expected altitude slope. I just placed a dot at a
    convienent point for the beginning of the line and then moved five minutes to the right and up
    70.2', placed a dot and then drew a straight line between them. Then using a dividers (third
    photo) I measured the distance of each point from the slope and got 3.8', 2.5', 4.1', 2.6', 1.9', and
    1.1'. The average was 2.7' and I could move the sloped line up by this amount to draw the sloped
    line through the sextant altitude points but this is not really necessary since we are looking to find
    the displacement of the points from the average line. We can do this by simply subtracting 2.7'
    from the already measured displacements resulting in residual displacements of 1.1', -0.2', 1.4',
    -0.1', -0.8', and -1.6'.

    I had found that the historical standard deviation of my sights is 1.433' so applying my two sigma
    test to this data I find that none of the sights were displaced from the average slope line
    exceeding 2.866', two sigmas, the largest being only -1.6'. Based on this rule I eliminated none of
    the observations since there were no �outliers� in this data set. So using all the observation, I
    computed an average sextant altitude of 10̊ 35.7' and an average time of 9:16:36 Z. The third
    photo shows this average point plotted. The forth photo shows the slope line moved up by 2.7' to
    go through the data points.

    The purpose for averaging and the other techniques is to improve the accuracy of the LOP. Since
    I have gps positions for each of the sights, and for the average, we can see if averaging improved
    the accuracy of the resulting LOP. The six intercepts are 0.7 T; 0.2 T; 1.3 T; 0.2 T; 0.5 A; and 0.8
    A resulting in a standard deviation of 0.77 nm. The intercept for the average sight is only 0.1 T so
    it appears that the averaging did improve the accuracy of the resulting LOP. Peter Hakel analysed
    my data and, using his methodology, came up with an average observed altitude of 10̊ 27.7' at
    9:16:54 Z. Computing an Hc for the gps coordinates at that time produces 10̊ 27.9' making the
    intercept 0.2 A so his method also improved on the individual sights but not as well as the
    simple average which was only 0.1 T.

    Then there is the question of how many sights need to be averaged? Weems states that the error
    in the average of a series of sights is determined by the square root of the number of sights. So to
    halve the error you need to take 4 sights; to cut it to one-third requires 9 sights; one-fourth, 16
    sights; one-fifth, 25; and one-tenth, 100 sights.



    Some of you have requested real data to use to experiment with various data massaging
    techniques to improve the accuracy of observed altitudes. This data was obtained onboard the
    Royal Clipper on a transatlantic crossing in 2009. I am providing the raw data for six
    observations of Venus taken in a six minute period for you to number-crunch.

    Greenwich date: November 6, 2009
    All times are GMT with no watch error.
    Height of eye: 33 feet.
    Index error: + 1.3'.
    Course: 251̊ true.
    Speed: 11 knots.
    Temp. 70 F
    Pres. 1013 mb
    Body: Venus.
    Measured azimuth for the first shot was 102̊ true and for all the rest was 103̊ true.

    Approximate location at middle of series was 14̊ 25' north, 55̊ 00' west.

    I have the GPS coordinates for each shot and will supply them later so we can see how well the
    number-crunching did.

    Number Time Sextant altitude GPS

    1 9:13:38 9̊ 54.8' 14̊ 25.9' N, 54̊ 59.7'W

    2 9:15:06 10̊ 14.6' 14̊ 25.8' N, 55̊ 00.0'W

    3 9:16:13 10̊ 31.2' 14̊ 25.7' N, 55̊ 00.2'W

    4 9:17:06 10̊ 42.6' 14̊ 25.7' N, 55̊ 00.3'W

    5 9:18:15 10̊ 57.9' 14̊ 25.6' N, 55̊ 00.5'W

    6 9:19:22 11̊ 13.1' 14̊ 25.5' N, 55̊ 00.7'W


    My average 9:18:36 10̊ 35.7' 14̊ 25.7' N, 55̊ 00.2'W

    Hakel�s Ho 9:16:54 10̊ 27.7 14̊ 25.7' N, 55̊ 00.3'W

    I have another 6 shots of Venus taken a little bit earlier the same day but over a period of 21
    minutes which you may want to combine with the other six shots, or not, due to the long time
    span. I have 3 more shots of Venus taken slightly later that you can also combine, or not, due to
    the long time period. In total, there are 15 shots taken over a 39 minute period. All the rest of the
    conditions remain the same except for the measured azimuths for the earlier shots.

    Earlier series.

    The measured azimuth to the first two shots was 101̊ true and for the last four shots the azimuth
    was 102̊ true.

    Number Time Sextant altitude GPS

    A1 8:49:09 4̊ 14.8' 14̊ 27.4' N, 54̊ 55.4'W

    A2 8:51:04 4̊ 44.1' 14̊ 27.3' N, 54̊ 55.7'W

    A3 9:06:52 8̊ 20.6' 14̊ 26.3' N, 54̊ 58.5'W

    A4 9:08:18 8̊ 40.3' 14̊ 26.2' N, 54̊ 58.8'W

    A5 9:09:28 8̊ 55.9' 14̊ 26.2' N, 54̊ 59.0'W

    A6 9:10:32 9̊ 10.5' 14̊ 26.6' N, 54̊ 59.2'W


    The third series, all the measured azimuths were 103̊ true.

    B1 9:22:53 12̊ 01.8' 14̊ 25.3' N, 55̊ 01.3'W

    B2 9:24:06 12̊ 19.1' 14̊ 25.3' N, 55̊ 01.6'W

    B3 9:27:43 13̊ 09.5' 14̊ 25.1' N, 55̊ 02.2'W

    NavList message boards and member settings: www.fer3.com/NavList
    Members may optionally receive posts by email.
    To cancel email delivery, send a message to NoMail[at]fer3.com







    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site