# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Herbert Prinz

**Date:**2004 Oct 21, 16:17 -0400

Bill has brought to my attention that the table that I posted earlier may be misleading in so far as it does not directly answer the question he asked, namely, "what is the maximum error resulting from linear averaging of a given data set of altitude observations?" I had tabulated the difference between the arithmetic mean of two altitudes and the actual altitude at the middle of the interval. In other words, the maximum error due to linear interpolation. My purpose was to demonstrate the non linearity of the curve. To find the answer to Bill's question, you have to apply the table in the data to your actual sample. For instance, if you take three sights, each 2 minutes apart, you have 2 sights that each contribute an error of 14" and one that contributes 0". So the total error is 66% of what you find in column 4 m. For this purpose, the table was incomplete as it lacked the column for 2 minutes separation; so I add it here (and remove the one for 5). Alt 2m 4m 6m 60 deg 0'03" 0'14" 0'31" 75 deg 0'07" 0'30" 1'06" 80 deg 0'11" 0'45" 1'40" 85 deg 0'22" 1'30" 3'21" Now, if you take five sights in 1 minute intervals, you add the entries in the appropriate line from columns 2m and 4m and multiply with 0.4 to find the error. For seven sights you add the entries from all three columns and multiply with 2/7 or 0.3. But remember, this is the MAXIMUM error. How bad it really is, say 30 deg of the meridian, I addressed in my previous message. Herbert Prinz