Welcome to the NavList Message Boards.

NavList:

A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Message:αβγ
Message:abc
Add Images & Files
    or...
       
    Reply
    Re: Round-off
    From: George Huxtable
    Date: 2009 May 16, 16:13 +0100

    In spirit, I'm very much with John Karl in pressing for arithmetical rigour
    in data reduction, especially when that comes so readily with modern
    technology.
    
    But I am worried by his assessments of error.
    He wrote-
    "-- a St. Hilaire reduction.  I count 10 numbers used in such a reduction.
    If every number is off by 0.05' in the unfavorable direction, the result is
    0.5' in error due to round off."
    
    Well, yes, and no. If it was, then it would. But what is the likelihood?
    
    Take those 10 numbers. Presumably, each has an equal likelihood of being
    rounded up or rounded down, when we round. For all 10 to be in the same
    direction is like tossing 10 pennies and getting heads each time. Or 10
    tails of course. So the chance of them all being rounded the same way is 2
    in 2 to the power 10, or 1 in 512.
    
    So lets consider, for example, that occasional 1 in 1024 case when all 10
    have been rounded upwards, none downwards. How much will the result have
    been shifted, by that rounding? All we have said about the original values,
    to be rounded, is that in this rare case every one will have been somewhere
    in the range x.500 and x.999, where x is any integer. We can only assume a
    flat distribution over that range. So, even in this unusual case, although
    the maximum shift that the rounding can cause can indeed be 0.5, as John
    states, the average shift will only be 0.25 or so.
    
    Then we have to sum that result in with other cases, with a more equal
    balance of roundings up and down.
    
    So, although John's stated maximum shift by rounding is theoretically
    possible, in the real-world it's inconceivably unlikely, and he would do
    better to base his arguments on a better estimate.
    
    George.
    
    contact George Huxtable, at  george{at}hux.me.uk
    or at +44 1865 820222 (from UK, 01865 820222)
    or at 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK.
    
    
    --~--~---------~--~----~------------~-------~--~----~
    Navigation List archive: www.fer3.com/arc
    To post, email NavList@fer3.com
    To unsubscribe, email NavList-unsubscribe@fer3.com
    -~----------~----~----~----~------~----~------~--~---
    
    

       
    Reply
    Browse Files

    Drop Files

    NavList

    What is NavList?

    Join NavList

    Name:
    (please, no nicknames or handles)
    Email:
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
    Email:

    Email Settings

    Posting Code:

    Custom Index

    Subject:
    Author:
    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site