NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Re: Rejecting outliers: was: Kurtosis.
From: Peter Fogg
Date: 2011 Jan 1, 10:50 +1100
From: Peter Fogg
Date: 2011 Jan 1, 10:50 +1100
Geoffrey Kolbe wrote:
This puzzles me, Geoffrey, given the boundaries of the particular context we are discussing. You do understand that the calculated slope can be assumed to be a fact? (ie; apart from being an approximation of an arc, and assuming the DR is reasonable).
Therefore if you end up with a pattern of sights that more or less follows that slope, but one (or more) apparent outlier that obviously does not, what possible conclusion can be reached? Either the pattern is generally correct and the outlier an obvious indication of error, or the outlier is correct and the apparent pattern then must be entirely composed of erroneous data sets. It seems to me to be a common-sense choice between these alternatives.
There is a third way, that of averaging. This accords weight to the apparent outlier(s) in proportion to population extent. If there are 2 outliers, both on the same side of the line, and a restricted population (which is pretty-much a given) then significant error can result. Error that is easily avoided by use of the slope.
I agree. Use of slope allows for and encourages this consideration. Averaging does not.
You can't be serious. Firstly; remember that one of the most significant drawbacks to the use of celestial navigation in practice is the weather. Another is the limited extent of dawns and dusks available (only 2 per day!) which offer the great advantage of a multiple-body fix at much the same time, without introducing error through running forward or back a position.
I suggest that the navigator's time would be better spent in analysing the sights he/she has, and applying this simple technique in order to reduce random error.
Once the slope has been calculated and the pattern of sights compared with it, it is up to the individual navigator to make the decision about the best place to place the slope amongst the sights. As much or as little weight can be accorded to any apparent outliers as you like.
Frank seems to think that some pre-determined numerical quantity can be applied to assist that decision. Goodo. Feel free to do this if anyone wants to. I doubt very much whether much of this will ever happen in practice; one of the big advantages of slope is its simplicity and relative ease of use.
The other very obvious point is that without slope how would you be even aware of apparent outliers? Not though blind averaging, that's for sure. If you only take one sight and then reduce that then you have no idea of how good or bad that individual sight might be. Could be excellent. Could be an outlier. Could be anything at all.
Half right. The right part is that one should never assume that any fix is entirely free of error. However, remember that use of slope is only one-half of a two-pronged approach; the half of dealing with random error as best as can be practically done (until someone comes up with a better method, which is somewhat different to going to outlandish lengths to try to poke holes in this one - eg; assuming a vastly wrong DR).
The other half of the two-pronged approach is to assume the resulting position lines to be free of random error, leaving non-random or systematic error to be dealt with. This can be simply and effectively done by bisecting the angles of intersecting position lines, leading to a fix position where these bisecting lines meet at a point. It ain't perfect, but it can be reasonably expected to be a better fix, with reduced extents of both random and non-random error.
Nothing magic after all, George, Geoffrey et al. Sorry. Its really only common sense powered via some simple drafting.
I've just given up resisting the temptation to add this (you can think of this personal weakness as a kind of reverse New Year's resolution):
Pourquoi faire simple lorsque, avec tellement peu plus d'effort, l'on peut faire compliquer...
I have to say that I share George's disquiet about the notion of rejecting outliers simply because they do not seem to fit with the other data.
Perhaps it is that, like George, I have a background as an experimental physicist, and that the notion of rejecting some data simply because it does not sit neatly with the rest of the data is an anathema.
This puzzles me, Geoffrey, given the boundaries of the particular context we are discussing. You do understand that the calculated slope can be assumed to be a fact? (ie; apart from being an approximation of an arc, and assuming the DR is reasonable).
Therefore if you end up with a pattern of sights that more or less follows that slope, but one (or more) apparent outlier that obviously does not, what possible conclusion can be reached? Either the pattern is generally correct and the outlier an obvious indication of error, or the outlier is correct and the apparent pattern then must be entirely composed of erroneous data sets. It seems to me to be a common-sense choice between these alternatives.
There is a third way, that of averaging. This accords weight to the apparent outlier(s) in proportion to population extent. If there are 2 outliers, both on the same side of the line, and a restricted population (which is pretty-much a given) then significant error can result. Error that is easily avoided by use of the slope.
Experimental data is usually messy and experience shows that a lot can be learned from consideration of the possible causes of outliers.
I agree. Use of slope allows for and encourages this consideration. Averaging does not.
The navigator's time would be better spent taking another round of sights to force better precision on the mean than applying a statistical eraser to doubtful data.
You can't be serious. Firstly; remember that one of the most significant drawbacks to the use of celestial navigation in practice is the weather. Another is the limited extent of dawns and dusks available (only 2 per day!) which offer the great advantage of a multiple-body fix at much the same time, without introducing error through running forward or back a position.
I suggest that the navigator's time would be better spent in analysing the sights he/she has, and applying this simple technique in order to reduce random error.
Even if taking more sights is not practical, outliers should not be discarded unless a good reason presents itself as to why they should be discarded.
Once the slope has been calculated and the pattern of sights compared with it, it is up to the individual navigator to make the decision about the best place to place the slope amongst the sights. As much or as little weight can be accorded to any apparent outliers as you like.
Frank seems to think that some pre-determined numerical quantity can be applied to assist that decision. Goodo. Feel free to do this if anyone wants to. I doubt very much whether much of this will ever happen in practice; one of the big advantages of slope is its simplicity and relative ease of use.
The other very obvious point is that without slope how would you be even aware of apparent outliers? Not though blind averaging, that's for sure. If you only take one sight and then reduce that then you have no idea of how good or bad that individual sight might be. Could be excellent. Could be an outlier. Could be anything at all.
The consequence may be a rather more open cocked hat or a fix of somewhat looser precision than one would like. But better that than discarding "bad data" and risk a false sense of security from the resulting tight fix.
Half right. The right part is that one should never assume that any fix is entirely free of error. However, remember that use of slope is only one-half of a two-pronged approach; the half of dealing with random error as best as can be practically done (until someone comes up with a better method, which is somewhat different to going to outlandish lengths to try to poke holes in this one - eg; assuming a vastly wrong DR).
The other half of the two-pronged approach is to assume the resulting position lines to be free of random error, leaving non-random or systematic error to be dealt with. This can be simply and effectively done by bisecting the angles of intersecting position lines, leading to a fix position where these bisecting lines meet at a point. It ain't perfect, but it can be reasonably expected to be a better fix, with reduced extents of both random and non-random error.
Nothing magic after all, George, Geoffrey et al. Sorry. Its really only common sense powered via some simple drafting.
I've just given up resisting the temptation to add this (you can think of this personal weakness as a kind of reverse New Year's resolution):
Pourquoi faire simple lorsque, avec tellement peu plus d'effort, l'on peut faire compliquer...
