Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Averaging
    From: Herbert Prinz
    Date: 2004 Oct 9, 19:18 -0400

    From a theoretical viewpoint, it is wrong to average the observations
    before reducing them.
    Alexandre is basically right that the motion of the observer alone on a
    small sailing vessel during a relatively short interval, say 5 min, has
    little impact on the fix. But his analysis addresses the wrong problem.
    It is the motion of the observed object that really matters. If this
    motion is noticeably non-linear - and for an altitude observation it
    almost always is - averaging may lead to errors that exceed the expected
    random errors of measurement (see one exception, below). The motion of
    the observer aggravates the error further. In other words: The costs may
    be higher than the benefit. A position error of  1.5 nm due to naive
    averaging of altitudes over 5 minutes would not be atypical.
    If one takes more observations than are required to obtain a fix, the
    latter becomes overdetermined. The only rigorous solution to this
    problem is to minimize the sum of the squares of the individual unknown
    observation errors. The correct procedure is given in Chauvenet's
    "Manual" (in the appendix), in the Nautical Almanac (pp. 277-283), in
    the HMNAO publication "AstroNavPC and Compact Data". Discussions of
    various algorithms can be found in several articles in the Journal of
    In the few special cases where the observandum is a linear function of
    time, averaging is appropriate. The main candidate for this is the time
    sight, when the sun (or other object) is in the prime vertical. The
    other application is the lunar distance, provided the geometry of moon
    and second object is favourable. Theoretically, meridian transits also
    fall into this category when, for a brief time span, the altitude of the
    object changes less than what can be nominally resolved on the
    instrument. This is only relevant for land based latitude determination,
    since the observations have to be confined to the precomputed time
    For the said three applications, which were important at one time, Borda
    and others constructed repetition circles, a type of instrument that
    implements averaging in hardware.  Different kinds were used
    successfully by French navigators and, more so, by land surveyors. Their
    characteristic advantage is that they accumulate the measured angle, so
    one does not have to read the instrument after each individual
    observation. Only the time of each observation is recorded, and finally
    one reads the sum of the angles.
    Yet an other mechanical averaging device can be found on bubble
    sextants. As these were used for general altitude observations, they
    seem to be in conflict with the stated rule. But the comparatively large
    random error (which may actually be periodic when used on an aircraft)
    justifies the integration (= averaging) of the observed altitude during
    a brief interval, typically 2 minutes. The shorter interval makes for a
    better linear behaviour of the change in altitude during the
    Since both methods, the time sight as well as the lunar distance have
    fallen into total oblivion, manual averaging is a thing of the past.
    Before I address the idea of rejecting spurious data, let me clearify
    the terminology.
    What Alexandre refers to as "human error", I call a "blunder". This is
    not to be confused with "personal error", which is systematic and can be
    detected in the long run. A blunder, on the other hand, is an isolated
    occurrence of a gross error that can be detected within one individual
    data set on account of its size. Most common sources are procedural
    error, the introduction of a wrong number into any of the elements of
    the computation other than the observandum, a wrong reading of the
    latter, or improper use of the instrument. (E.g. picking a wrong date in
    the almanac, wrong sign, reading the wrong number of degrees in
    conjunction with a Vernier or micrometer reading, or the micrometer
    screw did not snap in properly after moving the index arm.)
    Of course, we would like to eliminate blunders. The problem is to
    identify them correctly. Not every spurious data point is a blunder. It
    is statistically not sound to remove out-liers as a matter of principle.
    For the typically small data sets that appear in celestial navigation,
    what are the sound, applicable criteria that tell us which observation
    to discard and which to keep? I do not think they exist. Therefore I do
    not feel good about removing out-liers if they cannot clearly be traced
    back to procedural errors.
    By the way, the repeating circle leaves no possibility whatsoever to
    remove out-liers due to bad measurement, but inherently eliminates
    blunders stemming from misreading of the individual observation.
    Much that has been written on the subject of error elimination is geared
    towards the needs of the astronomer or land surveyor and not applicable
    to celestial navigation. To give but one drastic example of how
    different the problems are: Finding latitude by meridian transit of a
    star. The navigator has a few minutes to do this, the astronomer a full
    year. The navigator seeks to find the extrem value during one transit
    (averaging would be pervers!), whereas the astronomer can apply the
    whole statistical apparatus to many such observations (one per day) of
    upper and (half a year later) lower transits of the same circumpolar
    Many navigators develop a "feeling" for the quality of an observation
    and decide right then and there, at the moment they take it, whether to
    keep it. But if one is not sure about an observation, why not include it
    in the set, reduce it with the other ones and then see how the minimized
    error residual for that particular observation fits in the overall
    picture? What is the point of spending the extra effort of plotting it
    Herbert Prinz
    Alexandre Eremenko wrote:
    > Many manuals advise to take several sights
    > in short sequence and then to average the result.
    > and to reduce the average as one sight.
    > (Chauvenet recommends at most 6, Russian manuals 3-5).
    > The purpose is to increase accuracy,
    > and, probably more importantly, to reject the sights
    > with an evident "human error".
    > [...]

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site