NavList:

   

Reply

In spirit, I'm very much with John Karl in pressing for arithmetical rigour
in data reduction, especially when that comes so readily with modern
technology.

But I am worried by his assessments of error.
He wrote-
"-- a St. Hilaire reduction.  I count 10 numbers used in such a reduction.
If every number is off by 0.05' in the unfavorable direction, the result is
0.5' in error due to round off."

Well, yes, and no. If it was, then it would. But what is the likelihood?

Take those 10 numbers. Presumably, each has an equal likelihood of being
rounded up or rounded down, when we round. For all 10 to be in the same
direction is like tossing 10 pennies and getting heads each time. Or 10
tails of course. So the chance of them all being rounded the same way is 2
in 2 to the power 10, or 1 in 512.

So lets consider, for example, that occasional 1 in 1024 case when all 10
have been rounded upwards, none downwards. How much will the result have
been shifted, by that rounding? All we have said about the original values,
to be rounded, is that in this rare case every one will have been somewhere
in the range x.500 and x.999, where x is any integer. We can only assume a
flat distribution over that range. So, even in this unusual case, although
the maximum shift that the rounding can cause can indeed be 0.5, as John
states, the average shift will only be 0.25 or so.

Then we have to sum that result in with other cases, with a more equal
balance of roundings up and down.

So, although John's stated maximum shift by rounding is theoretically
possible, in the real-world it's inconceivably unlikely, and he would do
better to base his arguments on a better estimate.

George.

contact George Huxtable, at  george@hux.me.uk
or at +44 1865 820222 (from UK, 01865 820222)
or at 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK.


--~--~---------~--~----~------------~-------~--~----~
Navigation List archive: www.fer3.com/arc
To post, email NavList@fer3.com
To , email NavList-@fer3.com
-~----------~----~----~----~------~----~------~--~---

   

Reply