# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Jim Thompson

**Date:**2004 Oct 23, 08:03 -0300

I just spent the last couple of hours reading the entire set of messages in this thread, at the Nav-L archives. Thanks to that, and to the message Alex just sent, I have revised my summary. How does this look now? Jim Thompson jim2{at}jimthompson.net www.jimthompson.net Outgoing mail scanned by Norton Antivirus ----------------------------------------- Averaging Sights Sextant sights are subject to a variety of errors that lead to imprecision and inaccuracy. This is a serious concern for a navigator who is relying on celestial navigation to find his or her position at sea, making comfortable sleep difficult. One way to deal with random observational error is to average a set of several sights, and then to plot the average time and altitude. See also Bowditch article 1609, and the Nav-L thread "averaging" in October 2004. Always apply basic principles to a run of sights: 1. Use 3-5 sights taken within a few minutes, a minute or less between sights. 2. Use raw sextant observations, before applying corrections. 3. Inspect the set of sights for consistency and discard obvious outliers, or sights that fail to increase and decrease in time or altitude reasonably smoothly. It helps to plot the sights on a graph of altitude over time; or build a table, use arithmetic to calculate the change in time and altitude between sights, and then inspect the table for outliers. And then, to average a run of acceptable sights: 4. Average time by adding up the minutes and seconds and dividing by the number of sights, then adding that average to the whole hour. 5. Average altitude by adding up the minutes and seconds and dividing by the number of sights, then adding that average to the whole degrees. 6. Work out the DR position of the average time based on vessel speed and course, and use the latitude and longitude of that position to reduce the average and plot the resulting azimuth and intercept. Beware: Simple arithmetic averaging improves precision in most cases by taking out random error, but beware some pitfalls: 1. Averaging does not take out bias errors caused by problems in the sextant or observer. 2. A celestial body's altitude changes in a non-linear fashion. But, fortunately for navigators, altitude non-linearity in most of the celestial window is smaller than our ability to detect it with an observational device like a hand sextant, which in the hands of most of us allows a precision of about than 0.5' of arc. The change in altitude over short time durations of 5-10 minutes is so nearly linear that for our purposes we can assume it is linear. The important exception to this is when the body passes through the observer's meridian at high altitudes, over about 60?-75?. The change in altitude is significantly non-linear in that case. For runs of sights of typical bodies taken up to five minutes, nonlinearity introduces a systematic error of only 0.1 to 1.0 nautical miles. The error is in the lower half of that range under 60? of altitude, and greater than 20? from the meridian. In the very worst case scenario, 2 minutes before and after a body goes through the zenith, the error introduced by averaging is up to 30 nautical miles, but this is a very rare situation. In the usual sight-taking altitude-azimuth window, navigators can average their sights, comfortable that they are taking out random error without introducing a signficant error caused by non-linearity in the way the altitude changes during sight-taking. Averaging should not be used in certain circumstances. I obtained these rules from a long thread about averaging on the Nav-L list in October 2004. If these rules are followed, then the systematic error owing to non-linearity in the change of altitude over 5 minutes will be less than 1' of arc, or 1 nautical mile (0.5' arc and 0.5 NM if the navigator uses the 60? and 20? limits): 1. Use arithmetic averaging if the body is lower than 60?-75?. 2. If the body is over 60?-75?, then use averaging only if the azimuth is >10?-20? from the meridian, and do not use simple arithmetic averaging if the body is closer to the meridian at those high altitudes. 3. Use arithmetic averaging only if all sights are obtained within about about 5-10 minutes. Using Computers Instead of Averaging With modern programmable caculators, handheld computers and laptops, it is very easy for navigators to reduce every sight individually and then plot all the reduced sights, rather than average a run of raw observations and plot just the average sight. The navigator can then plot the 3-5 acceptable (consistent) reduced sights for each body and graphically find the best single intercept between them. Following this method, the navigator does not have to take sights on the same body before moving to another body, but can shoot bodies as the opportunity arrives, coming back to earlier bodies to take additional sights if the horizon is still good. And as Gary LaPook wrote on Nav-L during the averaging thread, "As long as we are talking about the St. Hilaire method (computing an azimuth and intercept from an assumed position) we should remember that it was developed as an easy method of laying down the "Sumner Line," now called an LOP, requiring only one computation. The original Sumner method required computing two time sights, twice as much work. With programmable calculators it is now just as easy to do the two computations and lay down the LOP without measuring an azimuth or intercept or using an assumed position. You simply choose two longitudes, one east and one west of your DR, and the calculator calculates the latitude where the LOP crosses those longitudes. You prick those latitudes on the chart and draw a straight line between them." The disadvantage of the computer methods is the risk of "blunder", of entering a wrong data element into the computer, so that the reduced sight ends up being wrong, even though the raw observation might have been excellent. The risk of such blunders increases with the number of sights reduced, especially for a fatigued navigator in a sailboat with a small crew. But there are many similar ways to think outside the box when so much computational power is available to modern celestial navigators.