# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Round-off**

**From:**Frank Reed

**Date:**2009 May 22, 10:29 -0700

Suppose the standard deviation of your observational error in each sight is known approximately from experience. It might be 1.0 minutes of arc... perhaps 2.0 minutes of arc. Let's call it s0 generally. Suppose also that you make a dozen round-offs in a navigational calculation dropping the "tenths" at each step. How much does it matter? Round-off error first: each individual rounding is equivalent to taking a random number from a uniformly distributed random variable with a range from -0.5 to +0.5. This distribution has a standard deviation of 0.29 (actually 1/sqrt(12)). When we do multiple roundings they combine like a "random walk" increasing the expected error at a rate equal to sqrt(N) where N is the number of steps in the random walk or the number of roundings in this case. In the case of twelve roundings, this gives the convenient result that the standard deviation of the error is 1.0 minutes of arc. Note that twelve roundings combined together takes us from a uniformly distributed random variable to a distribution that is nearly a bell curve or gaussian distribution and therefore it is nearly indistinguishable from any other source of error in its distribution (the tails of the round-off error distribution are cut off at the maximum possible error but this is not important in practice). Net observational error in celestial sights is approximately gaussian (a normal distribution, bell curve, etc.), though I think most people find that the tails of the distribution are "heavier" or "fatter" than a pure gaussian. The only significance to this aspect here is that large errors are much more likely to come from the observation process than round-off in most cases. When working a celestial sight, the final result does not depend on these errors separately (except in practice sights, see below). All you get is a final number around which you place some confidence limits. In terms of plotting, you get an LOP and you visualize a band of uncertainty along it with some standard deviation width. How wide are those confidence limits? To combine two independent sources of error and find the net standard deviation of the result, we take the square root of the sum of the squares of the separate standard deviations. This is the CRITICAL step in the error analysis that would help you decide whether you feel it's legitimate to round-off under your observational circumstances. The numerical results of rounding compared to observational error are surprising to many people. Rounding off the tenths does not matter much if the observational error is a bit larger than one minute of arc. If the observational error, s0, is 1.0, then (again assuming twelve roundings in the total calculation) the net error is 1.4 minutes of arc. Is that acceptable? That's up to the user -- I would say it's borderline. If the observational error is 2.0 minutes of arc, then the net error is 2.2 minutes of arc. At that level, I think most navigators would agree that the small increase in the net standard deviation is acceptable and, in fact, effectively indistinguishable. Your "confidence in confidence limits" would not distinguish 2.0 from 2.2. Whether you round off or not, you would hardly notice the difference. And finally, suppose that your observational error is 0.5 minutes of arc. In that case, the net error is 1.1 minutes of arc so rounding off would dominate the net error at that level of accuracy. Under the assumption that there are twelve round-offs of tenths in the clearing process, the difference between rounding and not rounding is probably negligible (it's less than 10%) if the observational error is 2.2 minutes of arc or higher, and it is not too significant (less than 33%) if the observational error is 1.14 minutes of arc or higher. Observational errors of this size would frequently be found in small boat conditions so dropping tenths would not be unreasonable in those circumstances. John Karl previously mentioned one good reason for not droppint tenths, and that is when the whole purpose of the sights you're taking is to assess your observational error on its own. If you're taking practice sights from a known location (and most of us do this), there would no reason to work the calculations with reduced precision. Of course, if the goal is really to assess observational error, I would skip hand calculations and work them with a calculating device which would eliminate the issue of round-off entirely. For the lunarians, divide everything by ten. We usually work to tenths and drop the hundredths of a minute of arc in lunars. Historically, they usually worked to seconds of arc, which is in between. If your standard deviation in lunar distance observations is 0.25 minutes of arc (which I find to be the case regularly) and you're working to tenths (dropping hundredths) with a dozen steps in the calculation, then the net standard deviation in the result is s=sqrt(0.25^2+0.1^2) or 0.27 minutes of arc, less than a ten percent difference and presumably indistinguishable from the observational error alone. Finally, consider what happens when you average four sights, either four altitudes or four lunar observations. Assuming systematic error has been eliminated, the error of observation is generally cut in half. If that's the case, and you average the observations before doing the clearing calculation, then you might need to work with greater precision in the calculations. Alternatively, you would work up each sight separately, working the calculation with the precision appropriate for individual sights, and average the results at the end. -FER --~--~---------~--~----~------------~-------~--~----~ Navigation List archive: www.fer3.com/arc To post, email NavList@fer3.com To unsubscribe, email NavList-unsubscribe@fer3.com -~----------~----~----~----~------~----~------~--~---