# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Herbert Prinz

**Date:**2004 Oct 19, 14:42 -0400

Alexandre, Peter, Jim, and all the others, While I was in the middle of compiling my answer to Alexandre's last two messages, the discussion has gone into the very wrong direction that I was afraid it would. We are proposing methods to average before having justified that averaging is appropriate at all. It seems that I have not made sufficiently clear why I said from the outset that it is wrong to average the observations before reducing them. Let me therefore address this point once more. Peter Fog wrote > > Getting back to nav, the process for averaging sights is simple > > and effective. As many sights as possible > > taken over about 5 minutes of time are plotted. Time is the > > horizontal axis, observed altitude on the > > vertical. The slope of this group of sights either rises; obs to > > the east, or descends; to the west. This slope > > is then compared to a calculated line, which is then best fitted > > to the slope of sights. Any extreme outliers > > are disregarded (probably best, although it goes against ideal > > statistical practice). > > Simple and effective. I'm always vaguely surprised its a > > technique not more widely known. To which Jim Thompson replied: > Microsoft Excel makes it drop-dead easy to draw such a graph and, with a > mouse click, to generate the equation for the line of best fit. If a laptop > is at hand. Peter and Jim are shooting with guns at birds, and at the wrong birds at that. If Peter has in mind a simple visual approach with paper and pencil when he speaks of "best fit", I can accept that. But bringing Microsoft into this stage of the sight reduction game is a severe faux pas. To see where all this is coming from, let's look into what some would consider the semi-official current practice of pleasure boaters, namely the doctrine of the USPS. The US Power Squadron recommends (in fact demands for the sight folder to be submitted for graduation from the JN course) that every altitude sight be repeated at least three times and be checked for 'consistency'. Such a group of sights the call a 'run'. (Junior Navigation, 99/01, p. 2-11 and Appendix G) They specify what 'consistency' means: A rising body must show a steady growth in altitude, a setting body a steady decrease. The consistency rule is waved for sights near the meridian. (N.B. Alexandre: Altitudes above 75 deg are discouraged, but admitted!). In the USPS course, averaging the sights within a run is an option. One is supposed to record all sights in a log (Form ED-SL (98)) and enumerate them. The instructions on the back of the form say that for the reduction you can either pick one sight from a run, or average several ones. The only guidance given is to dismiss obviously 'inconsistent' sights. Now, for the purpose of checking consistency, some people will find it easier to sketch out the altitude versus time diagram in the manner Peter describes, rather than to inspect the raw numbers. Nothing wrong with that. But then inevitably the idea crops up that a best fitted line would give us the perfect answer to what sextant altitude to use for a chosen time. This is not so. First, the altitude grows in a linear fashion near the prime vertical. There, you would use linear regression. Near the meridian you would have to use a parabolic fit. That's also easy. But what kind of a fit do use in-between? The correct answer is obvious: You fit the altitudes to the cosine formula that you always use to obtain the altitudes from dec, lat and lha. But if you do this you end up implementing the core of the algorithm in the N.A. right then and there at a place which is supposed to be only a preliminary step to remove 'inconsistencies'. So, what kind of fit is simple enough and still adequate? Second, let's assume you observed two bodies, so you have two runs, and you have figured out the answer to the above. The idea of a best fit is that you allow for a random unknown observation error which varies slightly for each individual observation. In the absence of any knowledge whatsoever about this error you HOPE that you did the best, i.e the error is minimal. Hence the minimum least squares. When you fit a line to each run, you treat the observational errors in each run individually. Instead of minimizing the square sum of all observations together, you minimize within each run. What is the justification for this? (I am not saying it is impossible to argue for it, but I would like to see an explicit statement of the underlying reasons. Otherwise we might be unaware of what we are doing and why.) [*see footnote at the end] Third, let's assume we have sorted out the above, but this time we have three or more runs. Now we are in the awkward position that we have just done three or more LSQ fits and we still have an overdetermined fix, to which the only correct answer is - you guessed it - another LSQ fit. I leave it to you to work out the details. It's not going to be easy to show that there might in fact be rare situations where the fix obtained in such a round about manner is almost as accurate as the one that would have been obtained if a least square solution as proposed in the Nautical Almanac would have been used properly in the first place. Forth. Most people will shy away from solving the overdetermined system of three or more linear equations and will therefore resort to some simpler method. I can already see the writing on the wall: The most common mistake is to draw the fix at the intersection of the angle bisectors of the cocked hat, instead of where it really belongs. Therefore, some of our best fitting navigators will go through the labour of three or more least square fits only to throw away the fruits of their labour by marking an arbitrary spot in the cocked hat and claim "This is where I am, with utmost mathematical precision!" In short: If you want to be practical, follow Power Squadron procedures all the way through. If you want to use the modern statistical apparatus, follow the procedures in the N.A. p. 277-283. This is still very practical, but only if you have a programmable pocket calculator. (We have had many previous discussions on the subject of why the N.A. procedure is the only rigorously correct solution to the overdetermined fix and also about the location of the most probable position within the cocked hat. Therefore, I won't repeat the arguments.) Herbert Prinz *) My main argument against splitting observations in this manner is a practical one. If you have two sights of one body and another one or two sights of another body nearby (< 15 degrees), you cannot check for inconsistencies individually, but taking them together you can. After sunset, when the first bright star comes out in an easterly direction I may spend the time to take a run of 3 or 4 sights, since there may be nothing else to do. A little later, when many dimmer stars come out all around me and all at once, I rather go round and round trying to get a good spread in azimuth before I loose the horizon, instead of getting hung up with a prolonged run of one star. When I am done, it may turn out that I end up with several observations of the same stars. I treat them just the same as if they were observations of different stars. That is, when I have my HP-48 with me. If I were to reduce manually, I would carefully select TWO bodies with a good azimuth angle between them (e.g. sun and half-moon), or at the most three stars separated 120 deg, and aim for ONE quality observation of each body so that I feel really good about it. One mile accuracy is what I am aiming for, three to five is what I often get.