# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: distance with atan2**

**From:**Herbert Prinz

**Date:**2005 Nov 12, 13:59 -0500

I hope Paul Hirose does not mind if I step in here to explain the essential concept behind his method and while I am at it, correct a tiny inaccuracy. Paul Hirose wrote: > One way to calculate distance (separation angle) on a sphere is to use > vectors and the atan2 function or rectangular to polar conversion. Set x > to the dot product of the two vectors, and y to the magnitude of their > cross product. The separation angle is atan2(y, x), or you can get the > angle by converting x and y to polar form. [...] > An advantage of the vector method is its accuracy at all angles. A large > portion of Meeus' chapter on separation angle discusses ways to avoid > the inaccuracy of the traditional spherical trig formula near 0 or 180 > degrees. > The numerical advantage of the method is not due to it being a vector method, but rather lies in the avoidance of the arccos function. The goal is to replace arccos by arctan. If we didn't mind the arccos, the vector method would be rather simple. The angle between two unit vectors is DEFINED (!) as the arccos of their dot product. Therefore, if you want the angular separation of two such vectors representing positions on the celestial sphere or earth, compute their dot product (which is a cheap operation once you have the cartesian representation), take the arccos and you are done! Check this with the numbers in Paul's example. (Side note. As I have said before on this list, this simple method is good to at least 1 arc second on all programmable pocket calculators and any modern computer under a suitable programming language. In fact, on my PC, using 80 bit floating numbers and the C# math library, I can produce a maximum error of 3 milliarcseconds. The more refined methods in Meeus are for astronomers, not for boat navigators!) But let's go ahead and avoid the arccos at all cost. Since tan x = sin x / cos x, we can work with the arctan if we obtain the sine of the given angle in addition to the cosine, which we already have. That's where the X product comes in. Never mind its exact definition, the essential feature that we need here is that it is a vector the magnitude of which is equal to the area subtended by the two given vectors. Since these are unit vectors, it's easy to see that said area is equal to the sine of the angle. You can again verify this with the numbers in Paul's example. Now that we have the sine of the required angle (from the magnitude of the x product) and the cosine of the required angle (from the dot product), we may as well build the quotient (i.e. its tangent) and obtain the angle from it. The x product is, of course not a cheap operation on most calculators, and this is where the rub is. I hope that this comment is a useful starting point for those who want to explore the inner workings of the given recipe. Herbert Prinz