# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Averaging**

**From:**Herbert Prinz

**Date:**2004 Oct 21, 20:52 -0400

Chuck Taylor wrote: > --- Herbert Prinzwrote: > > > If Peter has in mind a simple visual approach with > > paper and pencil when he > > speaks of "best fit", I can accept that. But > > bringing Microsoft into this stage > > of the sight reduction game is a severe faux pas. > > Microsoft Excel is simply another tool, like pencil > and paper. Some people find it easier to make a > simple plot with a spreadsheet than with pencil and > paper, and I think that is what Peter and Jim were > suggesting. But my point was that once you have such a powerful tool, you are better off using it the right way (i.e. for the reduction algorithm) instead of the wrong way (i.e. data filtering). > Herbert goes on to say: > > > The US Power Squadron recommends (in fact demands > > for the sight folder to be > > submitted for graduation from the JN course) that > > every altitude sight be > > repeated at least three times and be checked for > > 'consistency'. Such a group of > > sights the call a 'run'. (Junior Navigation, 99/01, > > p. 2-11 and Appendix G) > > They specify what 'consistency' means: A rising body > > must show a steady growth > > in altitude, a setting body a steady decrease. The > > consistency rule is waved for > > sights near the meridian. (N.B. Alexandre: Altitudes > > above 75 deg are > > discouraged, but admitted!). > > I am a member of the national USPS (United States > Power Squadrons) committee that oversees the Junior > Navigation and Navigation courses, and have been > teaching both courses and grading USPS sight folders > for I-don't-remember-how-many years. High-altitude > sights are admitted for the Navigation course, but not > for the Junior Navigation course. I did not dream this it up. I gave the references already in my earlier message. Now I copy verbatim from my own manual, Junior Navigation 99/01, Appendix G : 2. Limits for Sights A. The observed altitude (Ho) may not be less than 15 deg 0.0'. Acceptable sights may have any altitude greater than this. High-altitude sights greater than about 75 deg, while acceptable, can be difficult to take accurately. [End of quote.] It could easily be that you were able to apply your own interpretation of the rules all these many years because nobody ever tried to submit an altitude higher than 75 deg. But I would have been VERY upset, if I had submitted such a sight, based on what the manual told me, and you would have rejected the sight folder for that reason. > > In the USPS course, averaging the sights within a > > run is an option. One is > > supposed to record all sights in a log (Form ED-SL > > (98)) and enumerate them. The > > instructions on the back of the form say that for > > the reduction you can either > > pick one sight from a run, or average several ones. > > The only guidance given is > > to dismiss obviously 'inconsistent' sights. > > This is not so. One run of 5 averaged sights is > required for the Navigation course, but averaged > sights are not otherwise admitted. Nowhere in the manual do I find such a rule. Please give me the reference with page number. My manual demands 3 sights in a run, but no requirement for averaging. I don't even understand what you are saying here. Besides a run for the sun, the moon, a planet and a star; a two body fix, must be submitted. For this one needs two runs. You are saying that the sights of only one body out of these two can be averaged? This does not make sense to me. > Herbert went on to discuss the appropriateness of > fitting observed altitudes with linear models. > > > First, the altitude grows in a linear fashion near > > the prime vertical. There, > > you would use linear regression. Near the meridian > > you would have to use a > > parabolic fit. That's also easy. But what kind of a > > fit do use in-between? ... > > I am reminded of the words of a famous statistician by > the name of Oscar Kempthorne, who taught at Iowa State > University: "All models are wrong; some are useful." > > Certainly a straight-line fit, whether by eyeball or > by linear regression, is not rigorously correct in > this situation. Still, within appropriate limits, it > is useful for highlighting possible outliers. As > Kempthorne pointed out, a good statistician does not > necessarily *believe* his or her model. I never denied that a straight-line fit by eyeball may be helpful for highlighting outliers (What you actually do with them after you have highlighted them, is another question.) But I insist that a straight-line fit by linear regression makes no sense, because with the identical effort you can get a rigorous reduction of all these points. Afterwards, you can always hunt outliers, if you must. Just look at LOPs that don't fit in. As I explained in my previous message, I do not, as you say, reject data fitting with a particular model on the ground that the particular model might not be perfect. I reject it on the grounds that the perfect data fitting is carried out anyway in the correct reduction process. This is what the N.A. procedure is all about: Data fitting. It uses the correct model without us having to think about it. It uses the cosine formula for the altitude to fit the observations, RIGOROUSLY. Everything else is nonsense. You don't need a computer to do a half baked job. For this, the pencil is sufficient. Why in the world would anybody want to do 3 least square fits in the wrong place when 1 in the right place is enough? If someone could explain this to me, I will keep quiet. This must be an idea that originated from long winter evenings at the fireplace, not from a real world nav-station. > Standard procedure for plotting a line of position > using the St. Hilaire method calls for plotting a > straight line, when we know that what we "should" be > plotting is the arc of a great circle. Still, the > straight line is useful. Funny that you should mention this. I had considered referring to this as an example for how one has to be careful about the limits within certain assumptions are valid. But I left it out because it strays too far from our topic. You will certainly know that the St. Hilaire method is inherently iterative. This is exactly because linearity breaks down, when the intercept is too big (i.e. fix is too far from assumed position). In this case one has to reiterate using the fix as new AP. To my utmost surprise, many popular texts on navigation sweep this problem under the rug. So, by the way, does the Power Squadron in their Junior Navigator's course. There are table based methods that prevent the iteration from an improved starting position. For instance, HO229 only works with integral Dec and LHA. In the extreme case this can force you to choose your AP 45 nm away from the fix. That's why HO229 contains correction tables that trim the straight LOPs into circles. These circles are actually small circles, and sometimes they are very small! This means that the altitude of the body becomes, after the intercept, the second contributing factor to non-linearity. The critical value is somewhere near 60 to 75 degrees. It might be worth mentioning that the error induced by high altitudes in this manner is in the same direction as the one stemming from averaging: The fix is pushed away from the observed body. Many small errors can add up. Maybe, the reason why the experienced navigator of old without electronic resources shied away from high altitudes is to be sought in the complexity of their correct mathematical treatment, rather than in the alleged difficulty of observation? Herbert Prinz