# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Round-off**

**From:**John Karl

**Date:**2009 May 18, 20:18 -0700

Sorry, I messed up my explanation of the expected round off error, even though I had the right answer. The standard deviation (RMS value) of a uniform random distribution of width L (center on zero) is L/sqrt(12). For calculations carried out to 0.1', L= 0.1'. For calculations carried out to 1.0', L = 1.0'. A series of N additions (or subtractions) of these numbers then produces a standard deviation sqrt(N) greater. That is, the standard deviation of the result is now L*sqrt(N/12). Frank gave this result also. This is an interesting result because most of our sight reductions (lunar maybe excepted) have N less than 12, so we maintain the precision of the original numbers. I.e., using all values precise to 0.1' gives a final result to nearly the same precision of 0.1' -- But remember, that's only one of many different probability estimates. The max error is still L*N/2. And as before, I'm just talking about round off error here. JK --~--~---------~--~----~------------~-------~--~----~ Navigation List archive: www.fer3.com/arc To post, email NavList@fer3.com To unsubscribe, email NavList-unsubscribe@fer3.com -~----------~----~----~----~------~----~------~--~---