NavList:

Geoffrey Kolbe, you wrote:
"2) The fact that if the estimated position is not the actual position, it follows that the calculate slope from the estimated position cannot be the "best" fit.
It would seem that in general it is better to use a least squares fit rather the calculated slope from the estimated position. Is that logical?"

It's "logical", sure. But that doesn't mean it's the whole story, or right.

I think it's easier to understand why the calculated slope works if you start with the navigator's "normal" method for throwing out outliers, namely working up the sights and plotting them.

Suppose I have five sights of different bodies taken in twilight. I clear them and find that four of them cross in a fairly small area making a little quadrilateral on my chart maybe a mile across. Then I plot the fifth sight and it's nowhere near the crossing area of the other four. Instead it's fifteen miles away. What do I do with that sight? One option is to treat it as fully equal to the others and put my fix on the chart in the direction of that fifth LOP three miles away from the nice little quadrilateral made by the other four. That might be a reasonable choice. But many navigators would consider that sight suspect and ignore it. They would confidently put their fix in the small quadrilateral and perhaps draw a little 2.5 mile circle around it to indicate their degree of confidence in that position. In other words, they take the evidence of the long intercept of the fifth sight away from the others that are tightly grouped as reasonable evidence that the fifth sight was flawed somehow. None of this should be controversial. Of course, one needs a standard of some sort. If the fifth LOP is maybe six miles away, we should be much more careful about treating it as an outlier. It could be that the tight grouping of the first four is just an illusion of random chance. If the fifth LOP is instead seventy-five miles away, then we should have no trouble at all tossing it out.

Now let's apply these same rules to a set of sights of one single body taken over a short period of time. Perhaps I have eight altitudes of the Sun taken between 10:17 and 10:25 in the morning. One rather laborious option would be to take all those sights, work them up separately, and then plot individual LOPs on the chart. Since the time interval was short, the azimuths will be nearly identical and we will get a series of nearly parallel lines. Most of those lines should fall nearly on top of each other. We could even visually average them. But one LOP might fall well away from the others, just like the case of the five twilight sights above. By its significantly different intercept, we might have cause to treat it as an outlier and ignore it, just as we did with the twilight set.

For this set of sights of a single body, working up each separately and plotting the LOPs would be a lot of work. So instead, let's calculate the Hc of the Sun for 10:15 and also for 10:30. We plot those on a graph of altitude versus time. Then we draw a straight line through those two points. We have, with one stroke of the pencil, now calculated a very good estimate for Hc for any intermediate time. Now when we plot the eight values of Ho on that same graph, we can read off the values of the intercepts Ho-Hc just by looking at the difference between the line and the points for the observed altitudes. There's no change in methodology here, just a faster way of calculating the Hc values.

Next if we adopt Peter Fogg's approach, rather than calculating the value of Hc at two bracketing times, 10:15 and 10:30, we can calculate the slope of that line from that simple equation and for the vertical offset of the line (equivalent to the initial Hc at 10:15) we can pick any value that brings the sloping line close to our observed data points. This freedom to pick a convenient value for the base Hc is equivalent to the freedom of picking an AP for a standard LOP plot so long as it does not yield intercepts that are too long (by some standard). Now again, we can look at this graph and the differences between the line at any point and the Ho values are simply the standard intercepts. There's nothing wrong with using these directly off the graph. It's very nearly the same result that we would get from a standard LOP plot.

Clearly then the line with the correct slope is an excellent choice for analyzing the altitudes and spotting outliers. By contrast a least squares line can be severely skewed by the presence of outliers: it will skew towards them and make them seem less aberrant, less obvious. And in addition a least squares line does not bear this very close relationship to the standard methodology of intercept-based celestial navigation. The distances of the Ho points above or below the line sloping with the correct slope for that location are no more and no less than excellent estimates of the intercepts for each of the sights.

The question of throwing out outliers, and whether it is a legitimate one, has a long history. If you need a method for doing so that has the legitimacy of history, you might want to consider "Chauvenet's criterion" or "Peirce's criterion". There are Wikipedia pages for both of these. If anyone believes that it's always wrong to throw out outliers, you may want to hold a seance and have a stern talk with the ghost of William Chauvenet. :)

-FER

----------------------------------------------------------------
NavList message boards and member settings: www.fer3.com/NavList
Members may optionally receive posts by email.
To cancel email delivery, send a message to NoMail[at]fer3.com
----------------------------------------------------------------