NavList:

A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: Round-off
From: George Huxtable
Date: 2009 May 16, 16:13 +0100

```In spirit, I'm very much with John Karl in pressing for arithmetical rigour
in data reduction, especially when that comes so readily with modern
technology.

But I am worried by his assessments of error.
He wrote-
"-- a St. Hilaire reduction.  I count 10 numbers used in such a reduction.
If every number is off by 0.05' in the unfavorable direction, the result is
0.5' in error due to round off."

Well, yes, and no. If it was, then it would. But what is the likelihood?

Take those 10 numbers. Presumably, each has an equal likelihood of being
rounded up or rounded down, when we round. For all 10 to be in the same
direction is like tossing 10 pennies and getting heads each time. Or 10
tails of course. So the chance of them all being rounded the same way is 2
in 2 to the power 10, or 1 in 512.

So lets consider, for example, that occasional 1 in 1024 case when all 10
have been rounded upwards, none downwards. How much will the result have
been shifted, by that rounding? All we have said about the original values,
to be rounded, is that in this rare case every one will have been somewhere
in the range x.500 and x.999, where x is any integer. We can only assume a
flat distribution over that range. So, even in this unusual case, although
the maximum shift that the rounding can cause can indeed be 0.5, as John
states, the average shift will only be 0.25 or so.

Then we have to sum that result in with other cases, with a more equal
balance of roundings up and down.

So, although John's stated maximum shift by rounding is theoretically
possible, in the real-world it's inconceivably unlikely, and he would do
better to base his arguments on a better estimate.

George.

contact George Huxtable, at  george@hux.me.uk
or at +44 1865 820222 (from UK, 01865 820222)
or at 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK.

--~--~---------~--~----~------------~-------~--~----~
To post, email NavList@fer3.com
To unsubscribe, email NavList-unsubscribe@fer3.com
-~----------~----~----~----~------~----~------~--~---

```
Browse Files

Drop Files

Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

Email Settings

 Posting Code:

Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)