# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Rejecting outliers: was: Kurtosis.**

**From:**Lars Bergman

**Date:**2011 Jan 2, 14:40 -0800

I agree with George's analysis of Peter Fogg's data from 2007. Regarding the slope, which George questioned, I have come to the conclusion that 32' in 5 minutes of time seems correct. With the given altitude and latitude and knowing the declination of Canopus to be around 53 degrees south, you'll find the LHA to be around 339d with an azimuth of around 149d. Increasing LHA with 1d15.2' for 5 minutes change of GHA Aries gives an altitude that is 32' greater. I cannot see that Peter's mistake in plotting the slope as 34' instead of 32' makes any difference when it comes to "defining" observation 1 and 3 as outliers. With either slope those observations have the largest distance from the slope line.

What strikes me is the large "spread" of the observations. If all observations are transposed (don't know if this is the correct term) to a common instant of time, by moving each of them along a line of 32'/5m slope, it is found to be some 13', with #1 the largest and #6 the smallest. This is of course reflected in the standard deviation value as well. It must have been rough conditions (provided the instrument didn't have any loose screws). Thus I would be more suspicious about the "low" altitudes (# 5,6,7,8). Large seas or swells may very well obscure the horizon, making the sights too small, rather than the opposite. So maybe #1 and 3 are the best sights ... But this is of course pure speculations from my side. Anyway, I cannot see that #5 is excellent, 1 and 3 poor and the rest mediocre. I would say that the average value of all sights is the best choise possible, and I would have kept in mind the large spread of the observations when evaluationg the resulting LOP. You don't need to calculate the spread, just look at the numbers: #2 is expected to be some 3' larger than #1, actually is it 4' less; # 3,4,5,6 are nearly flat where you would expect an increase of some 10' between #3 and 6.

Lars 59N, 18E

----------------------------------------------------------------

NavList message boards and member settings: www.fer3.com/NavList

Members may optionally receive posts by email.

To cancel email delivery, send a message to NoMail[at]fer3.com

----------------------------------------------------------------