# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Least squares / was: Jupiter's BIG.**

**From:**Kieran Kelly

**Date:**2003 Dec 25, 11:40 +1100

Herbert, I have been following the discussion in relation to applications of line-of-best-fit to a series of lunar distance observations. Mention has been made of the analysis I performed on lunar distance observations undertaken by the Australian explorer/surveyor Augustus Gregory in 1856 (See www.ld-DEADLINK-com for the paper written this subject). I am not a mathematician and hesitate to tread such dangerous ground but there may be some confusion between consistency, accuracy and precision in this ongoing discussion. When I applied the line of best fit technique to Gregory's analysis I was trying to measure his consistency i.e. the deviation away from a line of best fit. This says nothing about accuracy. To determine that I had to find the actual location where the sights were performed and compare this with his computed location. Alternately, as I now know his exact latitude and longitude I could have calculated the actual lunar distance at the time of the observations and plotted these. This would have given a line which could be compared to the line of best fit from the observations. Error would have shown up in a different slope to both lines. To me these lines tell very different stories. A line of best fit applied to a series of observations may be very consistent with a small standard deviation and may be also very inaccurate not taking account of systematic error such as shade error, incorrect calculation of index error, observer bias, or failure to account for temperature and pressure. Thus a series of observations may be consistent and also consistently wrong leading to a wrong calculation of either GMT from a lunar or position from an astronomic fix. In Gregory's case the observations were both consistent and accurate but I could not tell this just from a line of best fit of his observations. The best explanation for the layman of laying an accurate line of best fit to a series of observations and the use of altitude/azimuth diagrams in this regard is in George Bennett's book, "The Complete On Board Celestial Navigator". Possibly some of the members more familiar with maths and statistics can expand on the differences between accuracy, precision and consistency better than myself. Merry Christmas from the land down under where it is never a White Christmas. Kieran Kelly Sydney Australia -----Original Message-----List [mailto:NAVIGATION-L{at}LISTSERV.WEBKAHUNA.COM]On Behalf Of Herbert Prinz Sent: Thursday, 25 December 2003 1:45 AM To: NAVIGATION-L{at}LISTSERV.WEBKAHUNA.COM Subject: Least squares / was: Jupiter's BIG. Fred, In your original post on this subject, you spoke of using linear regression for choosing the "best" single observation from a set. I assume the "best" would be the one with the smallest distance from the regression line. Yes? You also said, "The data look very nice; not as good as the lunar of that Australian explorer Kieran Kelly discussed a few months ago, but very good." You did not elaborate on what your criteria are to make data look "very good". But you are probably referring to the correlation between distance and time. Yes? Of course, you can use linear regression in this way only if there is a linear dependence between distance and time. Given that Frank's seven, or so, observations were taken within twenty minutes and Jupiter was well positioned, that's reasonable. We trust Frank that he checked this before he shot, but you can't really take it always for granted. For small distances or long time intervals, you might consider using a second order polynom for your regression. In your most recent post, you suggest to use regression for establishing chronometer rate. That's an interesting idea. Assuming that chronometer rate is constant over a long time, but fluctuates daily with temperature, the time shown by chronometer would in essence be linearly dependent on GMT. This dependence could be found via a linear regression of observations for time of various kinds spread out over a long interval. Used in the above way, and used correctly, linear regression could be considered a refined way of interpolation of observational data (hence my remark about guns and sparrows). However, when I picked up on the term "least squares fit" that you used, which is of course at the base of the linear regression method, I was not speaking of linear regression at all. I was thinking of using a least square fit for the _reduction_ of the sight, not for data preparation. I would not know how to reduce a single set of distance observations by linear (or any other type of) regression. I conceded that it is possible to use a least square sum method to reduce such a single set of distance observations. I am unable to provide at this moment the numerical example that you requested, but I will outline in a little more detail what I meant by saying in my previous message "solve for the watch error for which the sum of the square of the distance residuals becomes a minimum". The method I have in mind is nothing special; it is just a common technique applied to the particular problem. Say, you have a function that gives you the apparent topocentric distance between the moon and an other given object as a function of GMT. Such a function can be constructed by applying the inverse of any standard distance clearing method to the geocentric distances, which in turn may be computed, or directly obtained, from an almanac or from an ephemeris program. d = D(t_gmt, lat, lon) You also have a chronometer (watch), showing watch time t with unknown, but constant watch error -w. A number of n observations (t_k, d_k), k = 1..n, are all taken with this watch. t_gmt_k = t_k + w Now, if you have exactly one distance observation (d_1, t_1) you can solve heuristically the equation d_1 = D(t_1 + w, lat, lon) for w, and you are done. (You told us that you have a good computer. Any crude method such as bisection will do.) But if you have two or more observations (d_k, t_k) you get a system of equations which is overdetermined: d_k = D(t_k + w, lat, lon) k = 1..n Therefore, introduce a random observation error into your distance measurements (I am deliberately skipping a discussion of systematic error here. If you know that you have one, eliminate it! If you don't know about it, it's not systematic.) d_k + e_k = D(t_k + w, lat, lon) k = 1..n This system is underdetermined, because you have n equations and n + 1 variables. But hopeful wishing makes us search for that value of w that introduces the minimum observation errors: E(w) = Sum(Sqr(e_k), k = 1..n) ..... min where e_k = D(t_k + w, lat, lon) - d_k k = 1..n In words, E(w) is the sum of the squares of the individual observation errors (or distance residuals, as I called it in the earlier message). Obviously, E(w) is a function of w. Look for the minimum of this function (again heuristically, by iteration) and you have a rigorous solution for the most probable watch error belonging to 2 or more observations. As you can see, the individual observations need not have anything to do with each other. They can be taken at arbitrary times, from arbitrary objects. Their only connection is that the observation error is likely to be the same for all. This assumption may not always apply, but it is one that is often made, for the lack of any better knowledge of it. If you wish, you can relax the restriction of constant watch error and feed the known watch rate into the formula. As long as you need to iterate anyway, you can also in the first formula replace the longitude parameter by local apparent time. Or you can even replace the latter together with the latitude by two altitude observations. These need not be of the distance bodies, any two will do. They also need not be measured at the same time as the distances, as long as they are timed with the same watch. But these minor improvements and others I leave to your imagination. Herbert Prinz