NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Re: Averaging
From: Fred Hebard
Date: 2004 Oct 20, 15:08 -0400
From: Fred Hebard
Date: 2004 Oct 20, 15:08 -0400
On Oct 20, 2004, at 12:37 PM, Chuck Taylor wrote: > Certainly a straight-line fit, whether by eyeball or > by linear regression, is not rigorously correct in > this situation. Still, within appropriate limits, it > is useful for highlighting possible outliers. As > Kempthorne pointed out, a good statistician does not > necessarily *believe* his or her model. > > Standard procedure for plotting a line of position > using the St. Hilaire method calls for plotting a > straight line, when we know that what we "should" be > plotting is the arc of a great circle. Still, the > straight line is useful. > I think one can make a stronger statement than that a straight line is useful; it _is_ the appropriate model to be fitting in these circumstances. In that sense, it is rigorously correct. Within the limits of accuracy of graphical techniques or measurement by any instrument, a plot of altitude versus time is a straight line with the following two caveats: over a short interval of time and away from the meridian. As Herbert Prinz quite rightfully pointed out, there are exceptions where one indeed could introduce systematic error in excess of the random error that is removed, so one has to be cautious. These exceptions are not at meridian passage and over a short time interval, such as 5-10 minutes. Even at high altitudes and about 10 minutes after meridian passage, the straight line fits all the variation, as shown in the second graph below. Here in the first graph are some data illustrating this, among the best shots I've taken. Note that the R-squared is almost 1: there is no residual variation that could be usefully fit to another term. The date was 10/1/04, the object the sun. Time is UT1. The residuals (Hc-Ho) were -.08, .10, .01, -.06, .18 and .02 arc minutes. When fitting Hc by time for the same data, the R-squared was 0.999999 and the F value 2911325. Even 10 minutes after meridian passage close to the zenith, Hc versus time fits a straight line. The following data are for the sun on 10/1/04 at 4*48.6'S, 129*50.7'W. Time is UT1.