# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: On potential error introduced by rounded values**

**From:**George Huxtable

**Date:**2005 Jan 9, 16:45 +0000

In the past, when Peter Fogg and I have discussed Bennett's "The Complete On-Board Celestial Navigator", we have disagreed strongly: but not this time. I agree with the general thrust of his argument, with his numerical model, and his conclusions, though there might be a bit of quibbling about detail. Because, in Bennett's book, all the tabulations are only to the nearest whole minute, many "rounding" operations, to the nearest minute, are called for. Each such rounding introduces an error, at the most ? 0.5', but any value within that range is equally probable. Statistically, this is a "square" or "flat" error distribution, just 1' wide, differing markedly in shape from the familiar Gaussian, or "bell curve" error distribution. If there are two such roundings, they need to be combined by a "folding" operation, which results in a triangular error distribution, with a base-width of 2', and at the 50% level being 1' wide. Already, this is looking a bit more like the shape of a bell curve. Each successive rounding operation that's folded in brings the resulting error distribution closer to the shape of a bell curve, and widens it slightly. But the resulting width doesn't increase in proportion to the number of roundings, but to the square root of that number. Even so, the maximum conceivable error, if everything conspires in the same direction, is still ? 0.5 x the number of roundings. However, the more such roundings there are, the less likely it will be that they will all work, in the same direction, to their maximum extent: this is Peter's argument, and a valid one. Peter suggests- "There are 5 or 6 values (6 for stars, 5 for other bodies) entered during the sight reduction process used in George Bennett's book", and I won't argue with that assessment. If we take 6 such roundings, then in theory the maximum amount they could possibly (but most unlikely) contribute to an error in position is ?3 miles, though in practice the error is unlikely to exceed about ?1.5 miles, as Peter's numerical model shows. The question then arises: what does a prudent navigator take as the error introduced by the roundings in this calculation process, to combine with his estimated errors of observation? How much skirting distance should he allow around a charted but unmarked rock in mid-ocean, relying only on astro observations to avoid it? How narrow a passage between two such hazards would he tackle? A careful family-man might take the view that if there's a possibility (even a tiny one) of the calculation being 3 miles out either way, that's what he should allow for. A gambler might assess that if there's no more than (say) a 1 in 50 chance of the calculation error exceeding 1.5 miles, he will accept the risk and plot his course in that basis. After all, we're all risk-takers to some extent, or we wouldn't be out there on small boats in big waters. Of course, no navigator makes such a numerical hazard-assessment, or has the detailed information on which to base it: but still, those are the estimations that should be going through his mind, perhaps unconsciously. Peter offers two numerical models. One draws a parallel with the numbers of boys and girls in a family. I suggest that this is a very over-simplified approach to the problem, presuming that errors would always be the same in amount with just the sign being random, and his numerical results are therefore not very relevant. However, Peter's second model, using random numbers, is to my mind a good analogy with the real-life error situation. The results are meaningful, and the simulation could usefully be taken further. Peter's conclusion, that " rounding to whole numbers in a series does not lead to a great chance of the rounded amounts adding up to significant error." is somewhat woolly, but I can see what he's getting at. He states- "The result of this sight reduction is an intercept that is expressed to a whole minute of arc, leading to a fix consisting of latitude and longitude expressed as whole minutes of arc." Which is fair enough, as long as the user appreciates that a resulting intercept, given to an arc-minute, may be in error; that the true answer could very likely be the adjacent minute either way, and much less likely, yet another minute away than that. He ends- "I suspect there is also some wooly thinking involved in the assumption that a fix expressed to the nearest tenth of a minute of arc is better for the purpose of position finding from a small boat ..." Agreed. The Nav-l list frequently concerns itself with a quest for perfection, in observing to the nearest 0.1'. However, such ambitions crumble when faced with the reality of a small boat on an unquiet sea with an indefinite horizon. I accept that the standard of precision for intercepts, achieved in the Bennett tables (generally speaking, to an arc-minute or two) is perfectly appropriate for small craft, when viewed in the light of the limited precision achievable in such observations. The important proviso is that these limitations should be understood, and accepted, by the user. ================ Finally, after all that accord, and if only to tweak Peter Fogg's tail a little, I can't resist pointing out that earlier criticisms of the Bennett tables revolved around the major errors in azimuth, of many degrees, that can occur when using Bennett's azimuth tables for directions anywhere near to due East or West: not the errors of a minute or two in intercept that are being considered at present. George. ================================================================ contact George Huxtable by email at george@huxtable.u-net.com, by phone at 01865 820222 (from outside UK, +44 1865 820222), or by mail at 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK. ================================================================