# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: Round-off
From: Frank Reed
Date: 2009 May 22, 12:33 -0700

```John, you wrote:
"But remember, that's only one of many different probability estimates.  The max error is still L*N/2."

You're right that there are many different ways of describing probability
limits and estimating the odds of extreme outcomes. But the key is that you
should always compare apples with apples. If we're looking at ONE standard
deviation error in one variable, then we should never be comparing that to a
TWO standard deviation measure of error, or yet some other measure of error
(like max error), in another variable.

So what about that maximum potential error in round-off? If we're dropping
tenths, each round-off could be as large as 0.5 minutes of arc, and if there
are a dozen round-offs, then that might lead to an error as large as +6.0 or
-6.0, equal to the L*N/2 you gave. But this just isn't relevant in practice
because we have no means of distinguishing round-off error from observational
error (except, of course, by the obvious approach of running the calculation
again without rounding off). To put it differently, we already should
consider the possibility, rather low in probability, that our observations
are off by +/-6.0 minutes of arc, and unlike round-off error, this is not an
upper limit. Then we should ask whether the probability of being that far off
is significantly increased by the possibility of round-off error. And that
all depends on how the standard deviation of observational error compares
with the standard deviation of round-off error. As noted in my post a little
earlier today, if the standard deviation of the observational error is just a
little above 1 minute of arc, the increase in the net standard deviation from
rounding off (dropping the tenths) is not that great, less than 33%. It is
nearly identical to assuming that your observational error is moderately
higher and the round-off error is zero.

-FER

--~--~---------~--~----~------------~-------~--~----~
To post, email NavList@fer3.com
To unsubscribe, email NavList-unsubscribe@fer3.com
-~----------~----~----~----~------~----~------~--~---

```
Browse Files

Drop Files

### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

### Email Settings

 Posting Code:

### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)