NavList:

A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: Round-off
From: Frank Reed
Date: 2009 May 22, 10:29 -0700

```Suppose the standard deviation of your observational error in each sight is
known approximately from experience. It might be 1.0 minutes of arc...
perhaps 2.0 minutes of arc. Let's call it s0 generally. Suppose also that you
make a dozen round-offs in a navigational calculation dropping the "tenths"
at each step. How much does it matter?

Round-off error first: each individual rounding is equivalent to taking a
random number from a uniformly distributed random variable with a range from
-0.5 to +0.5. This distribution has a standard deviation of 0.29 (actually
1/sqrt(12)). When we do multiple roundings they combine like a "random walk"
increasing the expected error at a rate equal to sqrt(N) where N is the
number of steps in the random walk or the number of roundings in this case.
In the case of twelve roundings, this gives the convenient result that the
standard deviation of the error is 1.0 minutes of arc. Note that twelve
roundings combined together takes us from a uniformly distributed random
variable to a distribution that is nearly a bell curve or gaussian
distribution and therefore it is nearly indistinguishable from any other
source of error in its distribution (the tails of the round-off error
distribution are cut off at the maximum possible error but this is not
important in practice).

Net observational error in celestial sights is approximately gaussian (a
normal distribution, bell curve, etc.), though I think most people find that
the tails of the distribution are "heavier" or "fatter" than a pure gaussian.
The only significance to this aspect here is that large errors are much more
likely to come from the observation process than round-off in most cases.

When working a celestial sight, the final result does not depend on these
errors separately (except in practice sights, see below). All you get is a
final number around which you place some confidence limits. In terms of
plotting, you get an LOP and you visualize a band of uncertainty along it
with some standard deviation width. How wide are those confidence limits? To
combine two independent sources of error and find the net standard deviation
of the result, we take the square root of the sum of the squares of the
separate standard deviations. This is the CRITICAL step in the error analysis
that would help you decide whether you feel it's legitimate to round-off

The numerical results of rounding compared to observational error are
surprising to many people. Rounding off the tenths does not matter much if
the observational error is a bit larger than one minute of arc. If the
observational error, s0, is 1.0, then (again assuming twelve roundings in the
total calculation) the net error is 1.4 minutes of arc. Is that acceptable?
That's up to the user -- I would say it's borderline. If the observational
error is 2.0 minutes of arc, then the net error is 2.2 minutes of arc. At
that level, I think most navigators would agree that the small increase in
the net standard deviation is acceptable and, in fact, effectively
indistinguishable. Your "confidence in confidence limits" would not
distinguish 2.0 from 2.2. Whether you round off or not, you would hardly
notice the difference. And finally, suppose that your observational error is
0.5 minutes of arc. In that case, the net error is 1.1 minutes of arc so
rounding off would dominate the net error at that level of accuracy. Under
the assumption that there are twelve round-offs of tenths in the clearing
process, the difference between rounding and not rounding is probably
negligible (it's less than 10%) if the observational error is 2.2 minutes of
arc or higher, and it is not too significant (less than 33%) if the
observational error is 1.14 minutes of arc or higher. Observational errors of
this size would frequently be found in small boat conditions so dropping
tenths would not be unreasonable in those circumstances.

John Karl previously mentioned one good reason for not droppint tenths, and
that is when the whole purpose of the sights you're taking is to assess your
observational error on its own. If you're taking practice sights from a known
location (and most of us do this), there would no reason to work the
calculations with reduced precision. Of course, if the goal is really to
assess observational error, I would skip hand calculations and work them with
a calculating device which would eliminate the issue of round-off entirely.

For the lunarians, divide everything by ten. We usually work to tenths and
drop the hundredths of a minute of arc in lunars. Historically, they usually
worked to seconds of arc, which is in between. If your standard deviation in
lunar distance observations is 0.25 minutes of arc (which I find to be the
case regularly) and you're working to tenths (dropping hundredths) with a
dozen steps in the calculation, then the net standard deviation in the result
is s=sqrt(0.25^2+0.1^2) or 0.27 minutes of arc, less than a ten percent
difference and presumably indistinguishable from the observational error
alone.

Finally, consider what happens when you average four sights, either four
altitudes or four lunar observations. Assuming systematic error has been
eliminated, the error of observation is generally cut in half. If that's the
case, and you average the observations before doing the clearing calculation,
then you might need to work with greater precision in the calculations.
Alternatively, you would work up each sight separately, working the
calculation with the precision appropriate for individual sights, and average
the results at the end.

-FER

--~--~---------~--~----~------------~-------~--~----~
To post, email NavList@fer3.com
To unsubscribe, email NavList-unsubscribe@fer3.com
-~----------~----~----~----~------~----~------~--~---

```
Browse Files

Drop Files

Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

Email Settings

 Posting Code:

Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)