Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Round-off
    From: Frank Reed
    Date: 2009 May 22, 10:29 -0700

    Suppose the standard deviation of your observational error in each sight is 
    known approximately from experience. It might be 1.0 minutes of arc... 
    perhaps 2.0 minutes of arc. Let's call it s0 generally. Suppose also that you 
    make a dozen round-offs in a navigational calculation dropping the "tenths" 
    at each step. How much does it matter?
    Round-off error first: each individual rounding is equivalent to taking a 
    random number from a uniformly distributed random variable with a range from 
    -0.5 to +0.5. This distribution has a standard deviation of 0.29 (actually 
    1/sqrt(12)). When we do multiple roundings they combine like a "random walk" 
    increasing the expected error at a rate equal to sqrt(N) where N is the 
    number of steps in the random walk or the number of roundings in this case. 
    In the case of twelve roundings, this gives the convenient result that the 
    standard deviation of the error is 1.0 minutes of arc. Note that twelve 
    roundings combined together takes us from a uniformly distributed random 
    variable to a distribution that is nearly a bell curve or gaussian 
    distribution and therefore it is nearly indistinguishable from any other 
    source of error in its distribution (the tails of the round-off error 
    distribution are cut off at the maximum possible error but this is not 
    important in practice).
    Net observational error in celestial sights is approximately gaussian (a 
    normal distribution, bell curve, etc.), though I think most people find that 
    the tails of the distribution are "heavier" or "fatter" than a pure gaussian. 
    The only significance to this aspect here is that large errors are much more 
    likely to come from the observation process than round-off in most cases.
    When working a celestial sight, the final result does not depend on these 
    errors separately (except in practice sights, see below). All you get is a 
    final number around which you place some confidence limits. In terms of 
    plotting, you get an LOP and you visualize a band of uncertainty along it 
    with some standard deviation width. How wide are those confidence limits? To 
    combine two independent sources of error and find the net standard deviation 
    of the result, we take the square root of the sum of the squares of the 
    separate standard deviations. This is the CRITICAL step in the error analysis 
    that would help you decide whether you feel it's legitimate to round-off 
    under your observational circumstances. 
    The numerical results of rounding compared to observational error are 
    surprising to many people. Rounding off the tenths does not matter much if 
    the observational error is a bit larger than one minute of arc. If the 
    observational error, s0, is 1.0, then (again assuming twelve roundings in the 
    total calculation) the net error is 1.4 minutes of arc. Is that acceptable? 
    That's up to the user -- I would say it's borderline. If the observational 
    error is 2.0 minutes of arc, then the net error is 2.2 minutes of arc. At 
    that level, I think most navigators would agree that the small increase in 
    the net standard deviation is acceptable and, in fact, effectively 
    indistinguishable. Your "confidence in confidence limits" would not 
    distinguish 2.0 from 2.2. Whether you round off or not, you would hardly 
    notice the difference. And finally, suppose that your observational error is 
    0.5 minutes of arc. In that case, the net error is 1.1 minutes of arc so 
    rounding off would dominate the net error at that level of accuracy. Under 
    the assumption that there are twelve round-offs of tenths in the clearing 
    process, the difference between rounding and not rounding is probably 
    negligible (it's less than 10%) if the observational error is 2.2 minutes of 
    arc or higher, and it is not too significant (less than 33%) if the 
    observational error is 1.14 minutes of arc or higher. Observational errors of 
    this size would frequently be found in small boat conditions so dropping 
    tenths would not be unreasonable in those circumstances.
    John Karl previously mentioned one good reason for not droppint tenths, and 
    that is when the whole purpose of the sights you're taking is to assess your 
    observational error on its own. If you're taking practice sights from a known 
    location (and most of us do this), there would no reason to work the 
    calculations with reduced precision. Of course, if the goal is really to 
    assess observational error, I would skip hand calculations and work them with 
    a calculating device which would eliminate the issue of round-off entirely.
    For the lunarians, divide everything by ten. We usually work to tenths and 
    drop the hundredths of a minute of arc in lunars. Historically, they usually 
    worked to seconds of arc, which is in between. If your standard deviation in 
    lunar distance observations is 0.25 minutes of arc (which I find to be the 
    case regularly) and you're working to tenths (dropping hundredths) with a 
    dozen steps in the calculation, then the net standard deviation in the result 
    is s=sqrt(0.25^2+0.1^2) or 0.27 minutes of arc, less than a ten percent 
    difference and presumably indistinguishable from the observational error 
    Finally, consider what happens when you average four sights, either four 
    altitudes or four lunar observations. Assuming systematic error has been 
    eliminated, the error of observation is generally cut in half. If that's the 
    case, and you average the observations before doing the clearing calculation, 
    then you might need to work with greater precision in the calculations. 
    Alternatively, you would work up each sight separately, working the 
    calculation with the precision appropriate for individual sights, and average 
    the results at the end.
    Navigation List archive: www.fer3.com/arc
    To post, email NavList@fer3.com
    To unsubscribe, email NavList-unsubscribe@fer3.com

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site