Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Moon total correction- round it up?
    From: Paul Hirose
    Date: 2014 Oct 11, 17:03 -0700

    On 2014-10-10 7:46, Samuel L wrote:
    > Gary,
    > I never said my sextant reads to seconds. Read the original post.
    You asked how to round off an Ho of 25°37′50.4″. Writing the angle with
    that much precision implies Ho is known to better than a second of arc.
    It's not surprising someone would ask how you'd accomplish such a
    Good practice is to omit digits that contribute nothing useful. For
    instance, suppose I find the circumference of a tree trunk is 25 1/2
    inches by tape measure. Then I divide by pi and get a diameter of
    8.1169020976866621242130719319982 inches.
    But it's a good guess that I rounded off the tape measure reading to the
    nearest half inch. Roundoff error is therefore up to 1/4 inch, or about
    1% of circumference. It follows that the diameter computation can be off
    by 1% too, or roughly .1 inch. Furthermore, consider the rough surface
    and non circular cross section of the trunk. To real world precision, we
    have an 8-inch tree trunk. I can calculate the diameter to far higher
    precision, but the additional digits are just noise.
    Similar considerations apply in an angle format conversion. The end
    result can be no better than the original data. Say a sextant reads
    57°29.9′. What is that in decimal degrees, to appropriate precision?
    First consider the precision of the original data. It's .1′, which is
    about .0017°. So .001° and .1′ are approximately equivalent precision,
    and 57°29.9′ can be expressed as 57.498°.
    Now convert 57.498° to DMS format. One second is about .0003°, so tenth
    seconds are not justified. Even single seconds are about three times
    more precise than the decimal angle. But that's the most reasonable
    match, so we round the conversion to one second precision: 57°29′53″.
    Note the original sextant reading would have been 57°29′54″ in that
    format, so the "round trip" missed its target by 1″. That's more than
    good enough, but it does show how small errors accumulate in the
    conversions from DDMM.m to DD.ddd to DDMMSS.
    Choice of an appropriate precision is often a judgement call. If you're
    testing a sextant with an indexing table and collimators, the .03′ max
    roundoff error that can occur at .001° precision may be an uncomfortable
    fraction of your error budget. In practical navigation the same error is
    > What I needed to know was when any readings or figures be rounded up.
    > I'm always using an AH so once IE is removed during the reduction and that 
    figure is divided by 2 the majority of figures end up with seconds of arc at 
    the end.
    > For example;
    > Hs= 43d 43.4min
    > IE   -.1
    > Result divided by 2=  21d 51min 39sec (that's what the calculator says)
    Sam, errors are subtracted and corrections added. The -.1′ index error
    ought to be subtracted (not added) to obtain 43°43.5′, which you then
    divide by 2 to obtain 21°51.8′.
    To avoid confusion between errors and corrections, remember the goal is
    to remove error from observations. Remove = subtract.

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site