# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Fred Hebard

**Date:**2004 Oct 19, 17:56 -0400

On Oct 19, 2004, at 2:11 PM, Chuck Taylor wrote: > George Huxtable wrote, concerning the concept of > averaging observed sextant altitudes over time: > > "But I don't see how you would apply that > technique to a quantity that was changing > systematically, in the way that observed altitudes > change rather steadily with time (either > increasing or decreasing), with a bit of random > scatter superimposed." > > Alex Eremenko commented: > > "To reduce random independent errors in a > measurement of a quantity that changes linearly > (or does not change at all), average is the proper > thing to compute." > > The question is, "Does it make sense to average > a quantity that is varying systematically over > time, such as observed sextant altitudes?" > > We are talking about two components of variation, > one systematic and one random. As Alex pointed > out, averaging is certainly useful in eliminating > random variation when there is no systematic > variation. I would argue that it also makes sense > when the magnitude of the random variation > overwhelms the magnitude of the systematic > variation, such as might occur at sea in rough > weather on a relatively small vessel. > > Peter Fogg proposed a method that in effects > allows one to average observations with variation > taken into account: > > "...the process for averaging sights is simple and > effective. As many sights as possible taken over > about 5 minutes of time are plotted. Time is the > horizontal axis, observed altitude on the > vertical. The slope of this group of sights either > rises; obs to the east, or descends; to the west. > This slope is then compared to a calculated line, > which is then best fitted to the slope of sights. > Any extreme outliers are disregarded (probably > best, although it goes against ideal statistical > practice). Simple and effective." > > This gives you the best of both worlds, with the > averaging done visually. As Jim Thompson pointed > out, this is quite easy to do with a computer > spreadsheet. The only issue is converting > altitudes and times to decimal fractions. If you simply average both the altitudes and the time, and use the average time to reduce the average sight, you will have eliminated most of the systematic variation, except around meridian passage. This procedure is more-or-less a numerical equivalent to graphical methods, especially when fitting a straight line to the data by some means. With the numerical method and a straight line graphical method, an implicit assumption is that the altitude is varying linearly with time, which holds over most small intervals, again, except around meridian passage. In fact, at least on land with an artificial horizon, plots of altitude by time (away from meridian passage) oftentimes show so little variation around a straight line that you exceed the resolution of the graphs in detecting errors, which, however, are clearly apparent if you know where you are and look at the deviation from the expected altitude. Here's an illustration of this, plotting Ho versus time of observation, where Ho is the observed altitude. The differences in arcminutes between Ho and Hc (the altitudes computed from the known position using almanac data) for these shots were -0.70, -0.81, -0.43, -0.39, -0.40, -0.41, -0.44. The first two shots were markedly at variance from the last four, but that's not apparent in the graph. The resolution limits of this sort of graph also should be evident when you consider that the x-axis is subdivided into 12 second increments; it would be very difficult to plot these sights more accurately than to about 4 seconds of time, which puts you out by about 0.7' here. With lunars by the way, graphs like these, plots of distance versus time, oftentimes are much more informative.