# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Consistency v Accuracy in celestial navigation**

**From:**Chuck Taylor

**Date:**2003 Dec 29, 21:32 -0800

These comments are in response to Kieran Kelly's query about consistency vs. accuracy vs. precision in the context of celestial navigation. Since I don't possess a PhD in the subject, I don't claim to be a professional statistician, but I do rub elbows with professional statisticians on a daily basis while testing statistical software. And I did teach the subject for a few years a couple of decades ago at the Naval Postgraduate School in Monterey, California. It seems to me that Kieran has a very good grasp of the topic. One can certainly be consistently off in one direction or the other in taking measurements. Statisticians call this "bias". In shooting an altitude, for example, not holding the sextant quite vertical would introduce a bias in that the measured altitude would consistently be too high. A "line of best fit" (which I would call a linear regression line or a least-squares fit) is appropriate when the errors in measurement are known to be random (not biased) and symmetric. (More precisely, the assumption is that errors are distributed according to a Gaussian or "Normal" distribution, but random and symmetric is usually good enough.) The classic example of this situation in the context of celestial navigation is taking sights from a pitching deck in heavy weather. A given measurement is as likely to be high as low. There is an old story that the man with one watch always knows what time it is, but the man with two watches is never sure. The story can be extended a bit in the case of celestial sights. With only a single sight, you have no choice but to use what you have. Two sights always plot in a straight line, so there is little reason to choose one over the other. Averaging can help if you have reason to believe that any errors are random. Three sights can be very useful if they plot in a straight line. Then you at least can conclude that they are consistent. If they do not plot in a straight line, you are stuck. You can make a case that two of them are consistent and the third one is not, but you have no way of knowing *which* one is the odd sight out. With four sights, things start to get better. More than likely, at least three will plot more or less in a straight line. If so, then you can probably write off the fourth sight as being flawed. Statisticians would call this an "outlier". Of course if all four sights plot in a straight line, so much the better. With five or more sights, the story is much the same as with four. The lesson to be learned here is that it is better to plot your sights and visually evaluate their quality before including them in any sort of averaging or line of best fit. Throw out any obvious outliers, *then* do your averaging or least squares line. All of the above is in the name of consistency. It says nothing about accuracy, or lack of bias. Besides a systematic error in measurement, other factors can introduce bias. One example is that actual celestial (or terrestial) refraction may differ (due to atmospheric conditions at your particular location and time) from that which is shown in the tables. (Terrestial refraction affects dip of the horizon.) Kieran mentioned some possible sources of bias in the text quoted below. As Kieran pointed out, the best way to judge accuracy is by comparing your computed results with known true positions. In real life, we often find out about accuracy only in hindsight. How close did we make our landfall? Another way is to compare positions determined by celestial means to positions determined by GPS or using bearings from known landmarks. If a particular navigator has a track record of accurate sights (evaluated in hindsight), then one has reason to expect that his future sights are likely to be similarly accurate. Kieran also mentioned precision, but didn't say much about it. Precision refers to how precise the measurements are. To how many decimal places do we record our measurements? For example, I have one sextant whose scales can be read to 2 minutes of arc, and another whose scales can be read to 0.2 minutes of arc. The latter allows you to take more precise measurements than the former. It may be, however, that a skilled navigator can take sights that are more accurate but less precise with the sextant with the 2-minute scale than a less-skilled navigator could achieve using the sextant with the 0.2-minute scale. By "more accurate" I mean he would end up with a computed position that is closer to the true position. In the case of sights taken from land, any errors may or may not be random. In the absence of bias, measurement errors generally are random. Judgement is required in deciding whether or not bias exists. If bias is present, averaging or using a line of best fit will not necessarily improve the accuracy of your result. By the way, I have found that computer spreadsheets such as Excel can be very useful in plotting sights to evaluate consistency. You first need to convert your times into hours and decimal fractions of an hour, and your altitudes into degrees and decimal fractions of a degree. Then let the computer do the plotting. I hope that this is the sort of discussion that Kieran was looking for. With best regards, Chuck Taylor 47d 55.2 N 122d 11.2 W --- Kieran Kellywrote: > ... there may be > some confusion between consistency, accuracy and > precision in this ongoing discussion. > > When I applied the line of best fit technique to > Gregory's analysis I was trying to measure his > consistency i.e. the deviation away from a line of > best fit. This says nothing about accuracy. ... > > ... A > line of best fit applied to > a series of observations may be very consistent with > a small standard > deviation and may be also very inaccurate not taking > account of systematic > error such as shade error, incorrect calculation of > index error, observer > bias, or failure to account for temperature and > pressure. Thus a series of > observations may be consistent and also consistently > wrong leading to a > wrong calculation of either GMT from a lunar or > position from an astronomic > fix. __________________________________ Do you Yahoo!? New Yahoo! Photos - easier uploading and sharing. http://photos.yahoo.com/