# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Jim Thompson

**Date:**2004 Oct 22, 07:46 -0300

So, is this a fair summary? Sextant sights are subject to a variety of errors, leading to imprecision and inaccuracy. One way to deal with random observational error is to average a set of several sights, and then reduce and plot the average time and altitude. Always apply basic principles to a run of sights: 1. Use only sights taken within a few minutes, a minute or less between sights. 2. Use either the raw sextant observations, or reduce each observation and use the reduced set. With modern programmable caculators, handheld computers and laptops, it is very easy for navigators to reduce every sight individually, and then use the reduced sights, rather than average the raw observations. This elimates variation owing to the way the corrections themselves might vary, but it still does not make the run of sights linear. 3. Inspect the set of sights for consistency and discard obvious outliers, or sights that fail to increase and decrease in time or altitude reasonably smoothly. It helps to plot the sights on a graph of altitude by time, or build a table and use arithmetic to calculate the change in time and altitude between sights. 4. Average time by adding up the minutes and seconds and dividing by the number of sights, then adding that average to the whole hour. 5. Average altitude by adding up the minutes and seconds and dividing by the number of sights, then adding that average to the whole degrees. 6. Work out the DR position of the average time based on vessel speed and course, and use the latitude and longitude of that position to reduce the average (if not done prior to averaging) and plot the resulting azimuth and intercept. Simple arithmetic averaging method improves precision in most cases, but beware some pitfalls: 1. The body is very likely to be changing altitude in a nonlinear fashion. Altitude changes parabolically at the meridian, and most linearly when the body is on the prime vertical (due east or west). Over a period of a few minutes this is unlikely to be an issue for most common sights, but can lead to significant error in some situations. For runs of sights of typcial bodies taken over five minutes and under, arithmetic averages are subject to a systematic error of about 0.1 to 0.5 nautical miles, owing to nonlinearity in body motion. (IS THIS CORRECT?:) In the very worst case scenario, 2 minutes before and after a body goes through the zenith, the error introduced by averaging might be up to 30 nautical miles, but this is a very rare situation. The errors are worse in some situations, and averaging should not be used in certain circumstances. The first two of the following rules were suggested by Alex on the Nav-L list: 1a. Use arithmetic averaging if the body is lower than 60?. 1b. If the body is over 60?, then use averaging if the azimuth is > 20? from the meridian. 1c. Use arithmetic averaging only if all sights are obtained within about about 5 minutes. 2. High vessel speed can also lead to non-linearity in the data. This is unlikely to be a problem over a few minutes at typical sailboat cruising speeds, but might need to be considered in very fast ships. Jim Thompson jim2@jimthompson.net www.jimthompson.net Outgoing mail scanned by Norton Antivirus ----------------------------------------- > -----Original Message----- > From: Navigation Mailing List > [mailto:NAVIGATION-L@LISTSERV.WEBKAHUNA.COM]On Behalf Of Alexandre > Eremenko > Sent: Thursday, October 21, 2004 8:46 PM > To: NAVIGATION-L@LISTSERV.WEBKAHUNA.COM > Subject: Re: Averaging > > > I would like to re-iterate one point for those NOT > mathematically inclined. (Mathematicians frequently use > the words "and" and "or" in a precise technical sense, > which may be different from their daily use by other people). > > So I restate the rule in the form of an algorithm > (or in the style of tax form instructions). > > 1. Look at the altitude. If less than 60d you can average safely. > 2. If the altitude is more than 60 degrees look at the asymuth. > If the body is 20 degrees away from S and 20 degrees away from N, > you can average safely. > > So the only "prohibited zone" consists of the > two sectors with vertices > at zenith, 30 degrees long and 40 degrees wide, whose > bisector is the meridian. > > Alex. > >