# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Formula for dis. between two points on the Globe**

**From:**Mark Sienkiewicz

**Date:**1996 Oct 28, 23:14 EST

One of the problems I have following the math on this mailing list (and in navigation in general) is that people often treat the equations as if the numbers are all unitless values. Nearly all measurements of the real world have units associated with them, and if you ignore the units you run a real risk of not fully understanding the math you are using. I run into this all the time, and I used to think I was the only one having problems with it. I thought everybody either used the formulas without understanding, or maybe you are all just assuming that everybody just knows the part you aren't saying. I find that I was wrong: I'm not the only one who is having problems. For example, >>dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona)) > ?????? > ?????? >The equation above is incorrrect. > >-------- >The above equation should read for the Great Circle Distance between Points >A and B >located as; > A at location (Lat_A, Lon_A) and B at (Lat_B, Lon_B) > >Distance A to B = 60*ArcCos[Sin(Lat_A)*Sin(Lat_B) + Cos(Lat_A)*Cos(Lat_B) * >Cos(Lon_A -Lon_B)] > >The result is in nautical miles. These are both the same formula, but written in different units. Some math follows here, but it isn't very complex. [note 1] You probably have latitude and longitude in degrees, though radians make an equally valid measure. Sin() and Cos() are functions that accept arguments of degrees or radians. Because they are ratios of the lengths of two sides of a triangle, the functions yield unitless numbers. ArcCos accepts a unitless number and yields a result in some measure of angles. You probably use degrees, but radians are very common if you use a computer or calculator. [note 2] In the second equation, the value 60 is really 60 nautical miles / degree so the equation is 60 nautical miles / degree * ArcCos [ ... ] degrees factor out the implicit constant of " 1 degree / degree " and you are left with 60 nautical miles * ArcCos [ ... ] This equation yields a result in nautical miles. In the first equation, the number 6366.7 is another conversion factor. In fact, if you read the original post where it was given, you see >dist = 6366.7*Arccos(sin(lata)*sin(latb)+cos(lata)*cos(latb)*cos(lonb-lona)) > >Dist is in NM, 6366.7 is (180/pi)*60*1.852. If your arccos function returns >degrees rather than radians, adjust accordingly. Here the author has explicitly defined ArcCos as yielding a result in radians. Use the identity 2 * pi radians = 360 degrees to get the conversion factor ( 360 / 2 * pi ) degrees / radian This makes it (360 / 2 * pi ) degrees / radian * ArcCos [ ... ] radians Now we have our formula in degrees. Apply the conversion factor 60 nautical miles / degree to get the equation ( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian * ArcCos [ ... ] radians and again, we have a formula which yields nautical miles. I am not sure what the 1.852 is in the original post, but something suggests to me that it is 1.852 kilometers / nautical mile Let's assume I'm right, and write out just the various conversion factors: ( 60 nm / degree ) * (360 / 2 * pi ) degrees / radian * 1.852 kilometers / nm Compute it out and you get 6366.70702 kilometers / radian. This means the formula was misrepresented by difference of 1 letter in the original message: >Dist is in NM, 6366.7 is (180/pi)*60*1.852. If your arccos function returns ^^ should have said KM. A natural enough mistake; probably a typo So now I see that both equations were the same. One yields a distance in nautical miles and the other a distance in kilometers. If the constants in the original equations were written with their units, it would have been obvious. Mark S. [note 1] I hope you are able to read through it all. I often read my mail when I get home from work and I just don't want to see a lot of equations. :) [note 2] If you calculator has an ArcCos that shows the answer in degrees, it probably computed it in radians and then converted it. It just happens that all kinds of math relating to angles gets a lot simpler if you use radians instead of degrees. The radian is the "natural unit".