Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Formula for dis. between two points on the Globe
    From: Mark Sienkiewicz
    Date: 1996 Oct 28, 23:14 EST

    One of the problems I have following the math on this mailing list
    (and in navigation in general) is that people often treat the equations
    as if the numbers are all unitless values.  Nearly all measurements
    of the real world have units associated with them, and if you ignore
    the units you run a real risk of not fully understanding the math you
    are using.
    I run into this all the time, and I used to think I was the only
    one having problems with it.  I thought everybody either used the
    formulas without understanding, or maybe you are all just assuming
    that everybody just knows the part you aren't saying.
    I find that I was wrong: I'm not the only one who is having
    For example,
    >>dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona))
    >        ??????
    >        ??????
    >The equation above is incorrrect.
    >The above equation should read for the Great Circle Distance between Points
    >A and B
    >located as;
    >        A at location (Lat_A, Lon_A) and B at  (Lat_B, Lon_B)
    >Distance A to B = 60*ArcCos[Sin(Lat_A)*Sin(Lat_B) + Cos(Lat_A)*Cos(Lat_B) *
    >Cos(Lon_A -Lon_B)]
    >The result is in nautical miles.
    These are both the same formula, but written in different units.
    Some math follows here, but it isn't very complex. [note 1]
    You probably have latitude and longitude in degrees, though radians
    make an equally valid measure.
    Sin() and Cos() are functions that accept arguments of degrees or
    radians.  Because they are ratios of the lengths of two sides of
    a triangle, the functions yield unitless numbers.
    ArcCos accepts a unitless number and yields a result in some
    measure of angles.  You probably use degrees, but radians are
    very common if you use a computer or calculator.  [note 2]
    In the second equation, the value 60 is really
            60 nautical miles / degree
    so the equation is
            60 nautical miles / degree * ArcCos [ ... ] degrees
    factor out the implicit constant of " 1 degree / degree " and
    you are left with
            60 nautical miles * ArcCos [ ... ]
    This equation yields a result in nautical miles.
    In the first equation, the number 6366.7 is another conversion
    factor. In fact, if you read the original post where it was given,
    you see
    >dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona))
    >Dist is in NM, 6366.7 is (180/pi)*60*1.852.  If your arccos function returns
    >degrees rather than radians, adjust accordingly.
    Here the author has explicitly defined ArcCos as yielding a result in
    radians.  Use the identity
            2 * pi radians = 360 degrees
    to get the conversion factor
            ( 360 / 2 * pi ) degrees / radian
    This makes it
            (360 / 2 * pi ) degrees / radian * ArcCos [ ... ] radians
    Now we have our formula in degrees.  Apply the conversion factor
            60 nautical miles / degree
    to get the equation
            ( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian *
                    ArcCos [ ... ] radians
    and again, we have a formula which yields nautical miles.  I am not
    sure what the 1.852 is in the original post, but something suggests
    to me that it is
            1.852 kilometers / nautical mile
    Let's assume I'm right, and write out just the various conversion
            ( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian *
                    1.852 kilometers / nm
    Compute it out and you get
            6366.70702 kilometers / radian.
    This means the formula was misrepresented by difference of 1 letter
    in the original message:
    >Dist is in NM, 6366.7 is (180/pi)*60*1.852.  If your arccos function returns
            should have said KM.  A natural enough mistake; probably a typo
    So now I see that both equations were the same.  One yields a distance
    in nautical miles and the other a distance in kilometers.  If the
    constants in the original equations were written with their units,
    it would have been obvious.
    Mark S.
    [note 1]
            I hope you are able to read through it all.  I often read
            my mail when I get home from work and I just don't want
            to see a lot of equations. :)
    [note 2]
            If you calculator has an ArcCos that shows the answer in
            degrees, it probably computed it in radians and then
            converted it.  It just happens that all kinds of math
            relating to angles gets a lot simpler if you use radians
            instead of degrees.  The radian is the "natural unit".

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site