 # NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: Formula for dis. between two points on the Globe
From: Mark Sienkiewicz
Date: 1996 Oct 28, 23:14 EST

```One of the problems I have following the math on this mailing list
(and in navigation in general) is that people often treat the equations
as if the numbers are all unitless values.  Nearly all measurements
of the real world have units associated with them, and if you ignore
the units you run a real risk of not fully understanding the math you
are using.
I run into this all the time, and I used to think I was the only
one having problems with it.  I thought everybody either used the
formulas without understanding, or maybe you are all just assuming
that everybody just knows the part you aren't saying.
I find that I was wrong: I'm not the only one who is having
problems.
For example,
>>dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona))
>        ??????
>        ??????
>The equation above is incorrrect.
>
>--------
>The above equation should read for the Great Circle Distance between Points
>A and B
>located as;
>        A at location (Lat_A, Lon_A) and B at  (Lat_B, Lon_B)
>
>Distance A to B = 60*ArcCos[Sin(Lat_A)*Sin(Lat_B) + Cos(Lat_A)*Cos(Lat_B) *
>Cos(Lon_A -Lon_B)]
>
>The result is in nautical miles.
These are both the same formula, but written in different units.
Some math follows here, but it isn't very complex. [note 1]
You probably have latitude and longitude in degrees, though radians
make an equally valid measure.
Sin() and Cos() are functions that accept arguments of degrees or
radians.  Because they are ratios of the lengths of two sides of
a triangle, the functions yield unitless numbers.
ArcCos accepts a unitless number and yields a result in some
measure of angles.  You probably use degrees, but radians are
very common if you use a computer or calculator.  [note 2]
In the second equation, the value 60 is really
60 nautical miles / degree
so the equation is
60 nautical miles / degree * ArcCos [ ... ] degrees
factor out the implicit constant of " 1 degree / degree " and
you are left with
60 nautical miles * ArcCos [ ... ]
This equation yields a result in nautical miles.
In the first equation, the number 6366.7 is another conversion
factor. In fact, if you read the original post where it was given,
you see
>dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona))
>
>Dist is in NM, 6366.7 is (180/pi)*60*1.852.  If your arccos function returns
Here the author has explicitly defined ArcCos as yielding a result in
2 * pi radians = 360 degrees
to get the conversion factor
( 360 / 2 * pi ) degrees / radian
This makes it
(360 / 2 * pi ) degrees / radian * ArcCos [ ... ] radians
Now we have our formula in degrees.  Apply the conversion factor
60 nautical miles / degree
to get the equation
( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian *
and again, we have a formula which yields nautical miles.  I am not
sure what the 1.852 is in the original post, but something suggests
to me that it is
1.852 kilometers / nautical mile
Let's assume I'm right, and write out just the various conversion
factors:
( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian *
1.852 kilometers / nm
Compute it out and you get
This means the formula was misrepresented by difference of 1 letter
in the original message:
>Dist is in NM, 6366.7 is (180/pi)*60*1.852.  If your arccos function returns
^^
should have said KM.  A natural enough mistake; probably a typo
So now I see that both equations were the same.  One yields a distance
in nautical miles and the other a distance in kilometers.  If the
constants in the original equations were written with their units,
it would have been obvious.
Mark S.
[note 1]
I hope you are able to read through it all.  I often read
my mail when I get home from work and I just don't want
to see a lot of equations. :)
[note 2]
If you calculator has an ArcCos that shows the answer in
degrees, it probably computed it in radians and then
converted it.  It just happens that all kinds of math
relating to angles gets a lot simpler if you use radians
``` Browse Files

Drop Files ### NavList ### What is NavList? ### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member. ### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email: ### Email Settings

 Posting Code: ### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)