Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Averaging
    From: Herbert Prinz
    Date: 2004 Oct 19, 14:42 -0400

    Alexandre, Peter, Jim, and all the others,
    While I was in the middle of compiling my answer to Alexandre's last two
    messages, the discussion has gone into the very wrong direction that I was
    afraid it would. We are proposing methods to average before having justified
    that averaging is appropriate at all. It seems that I have not made sufficiently
    clear why I said from the outset that it is wrong to average the observations
    before reducing them. Let me therefore address this point once more.
    Peter Fog wrote
    > > Getting back to nav, the process for averaging sights is simple
    > > and effective. As many sights as possible
    > > taken over about 5 minutes of time are plotted. Time is the
    > > horizontal axis, observed altitude on the
    > > vertical. The slope of this group of sights either rises; obs to
    > > the east, or descends; to the west. This slope
    > > is then compared to a calculated line, which is then best fitted
    > > to the slope of sights. Any extreme outliers
    > > are disregarded (probably best, although it goes against ideal
    > > statistical practice).
    > > Simple and effective. I'm always vaguely surprised its a
    > > technique not more widely known.
    To which Jim Thompson replied:
    > Microsoft Excel makes it drop-dead easy to draw such a graph and, with a
    > mouse click, to generate the equation for the line of best fit.  If a laptop
    > is at hand.
    Peter and Jim are shooting with guns at birds, and at the wrong birds at that.
    If Peter has in mind a simple visual approach with paper and pencil when he
    speaks of "best fit", I can accept that. But bringing Microsoft into this stage
    of the sight reduction game is a severe faux pas. To see where all this is
    coming from, let's look into what some would consider the semi-official current
    practice of pleasure boaters, namely the doctrine of the USPS.
    The US Power Squadron recommends (in fact demands for the sight folder to be
    submitted for graduation from the JN course) that every altitude sight be
    repeated at least three times and be checked for 'consistency'. Such a group of
    sights the call a 'run'. (Junior Navigation, 99/01, p. 2-11 and Appendix G)
    They specify what 'consistency' means: A rising body must show a steady growth
    in altitude, a setting body a steady decrease. The consistency rule is waved for
    sights near the meridian. (N.B. Alexandre: Altitudes above 75 deg are
    discouraged, but admitted!).
    In the USPS course, averaging the sights within a run is an option. One is
    supposed to record all sights in a log (Form ED-SL (98)) and enumerate them. The
    instructions on the back of the form say that for the reduction you can either
    pick one sight from a run, or average several ones. The only guidance given is
    to dismiss obviously 'inconsistent' sights.
    Now, for the purpose of checking consistency, some people will find it easier to
    sketch out the altitude versus time diagram in the manner Peter describes,
    rather than to inspect the raw numbers. Nothing wrong with that. But then
    inevitably the idea crops up that a best fitted line would give us the perfect
    answer to what sextant altitude to use for a chosen time. This is not so.
    First, the altitude grows in a linear fashion near the prime vertical. There,
    you would use linear regression. Near the meridian you would have to use a
    parabolic fit. That's also easy. But what kind of a fit do use in-between? The
    correct answer is obvious: You fit the altitudes to the cosine formula that you
    always use to obtain the altitudes from dec, lat and lha. But if you do this you
    end up implementing the core of  the algorithm in the N.A. right then and there
    at a place which is supposed to be only a preliminary step to remove
    'inconsistencies'. So, what kind of fit is simple enough and still adequate?
    Second, let's assume you observed two bodies, so you have two runs, and you have
    figured out the answer to the above. The idea of a best fit is that you allow
    for a random unknown observation error which varies slightly for each individual
    observation. In the absence of any knowledge whatsoever about this error you
    HOPE that you did the best, i.e the error is minimal. Hence the minimum least
    squares. When you fit a line to each run, you treat the observational errors in
    each run individually. Instead of minimizing the square sum of all observations
    together, you minimize within each run. What is the justification for this? (I
    am not saying it is impossible to argue for it, but I would like to see an
    explicit statement of the underlying reasons. Otherwise we might be unaware of
    what we are doing and why.) [*see footnote at the end]
    Third, let's assume we have sorted out the above, but this time we have three or
    more runs. Now we are in the awkward position that we have just done three or
    more LSQ fits and we still have an overdetermined fix, to which the only correct
    answer is - you guessed it - another LSQ fit. I leave it to you to work out the
    details. It's not going to be easy to show that there might in fact be rare
    situations where the fix obtained in such a round about manner is almost as
    accurate as the one that would have been obtained if a least square solution as
    proposed in the Nautical Almanac would have been used properly in the first
    Forth. Most people will shy away from solving the overdetermined system of three
    or more linear equations and will therefore resort to some simpler method. I can
    already see the writing on the wall: The most common mistake is to draw the fix
    at the intersection of the angle bisectors of the cocked hat, instead of where
    it really belongs. Therefore, some of our best fitting navigators will go
    through the labour of three or more least square fits only to throw away the
    fruits of their labour by marking an arbitrary spot in the cocked hat and claim
    "This is where I am, with utmost mathematical precision!"
    In short: If you want to be practical, follow Power Squadron procedures all the
    way through. If you want to use the modern statistical apparatus, follow the
    procedures in the N.A. p. 277-283. This is still very practical, but only if you
    have a programmable pocket calculator.
    (We have had many previous discussions on the subject of why the N.A. procedure
    is the only rigorously correct solution to the overdetermined fix and also about
    the location of the most probable position within the cocked hat. Therefore, I
    won't repeat the arguments.)
    Herbert Prinz
    *) My main argument against splitting observations in this manner is a practical
    one. If you have two sights of one body and another one or two sights of another
    body nearby (< 15 degrees), you cannot check for inconsistencies individually,
    but taking them together you can.
    After sunset, when the first bright star comes out in an easterly direction I
    may spend the time to take a run of 3 or 4 sights, since there may be nothing
    else to do. A little later, when many dimmer stars come out all around me and
    all at once, I rather go round and round trying to get a good spread in azimuth
    before I loose the horizon, instead of getting hung up with a prolonged run of
    one star. When I am done, it may turn out that I end up with several
    observations of the same stars. I treat them just the same as if they were
    observations of different stars. That is, when I have my HP-48 with me.
    If I were to reduce manually, I would carefully select TWO bodies with a good
    azimuth angle between them  (e.g. sun and half-moon), or at the most three stars
    separated 120 deg, and aim for ONE quality observation of each body so that I
    feel really good about it. One mile accuracy is what I am aiming for, three to
    five is what I often get.

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site