# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: The Great Circle Challenge**

**From:**Frank Reed

**Date:**2014 Dec 29, 15:59 -0800

David, when you started this challenge, you wrote:

"I’ll stick with the diagram method"

I don't recall which diagram method you were using, but I think one aspect of this that most people will find suprising is that a diagram plotted on something like a "universal plotting sheet" will produce very good results even over substantial distances in many cases, and over short distances it really can't be beat. The key feature that we get from a plotting sheet is the scaling of longitude minutes and degrees by latitude. For relatively short distances, that's all that we're really getting out of the full-blown cosine formula. Suppose I have two points at latitudes 40° and 42° with a longitude separation of 2°. If you plot those on a chart using a longitude scaling for the mean latitude (equal to 0.7547 which is the cosine of 41°, the mean latitude) then the distance that you will measure off with a straight edge between the plotted points, ignoring the issue of actual reading and plotting inaccuracies, will differ from the complete spherical cosine result by 0.01 nautical miles or less than one one-hundredth of one percent.

If you need a value for a great circle distance and a computing device is handy (or you're entertained by tables), and ** if you can remember it**, then use the cosine equation:

cos

*D*= cos

*pd*· cos

_{1 }*pd*+ sin

_{2}*pd*· sin

_{1}*pd*· cos

_{2}*dLon*

where the pd's are polar distances, or in terms of latitudes,

cos

*D*= sin

*L*· sin

_{1}*L*+ cos

_{2}*L*· cos

_{1}*L*· cos

_{2}*dLon.*

It's really the fundamental, universal equation of spherical trigonometry, so if you're trying to be a mathematical navigator, it's really something that should be memorized permanently (I still remember the week I spent repeating it in my head as "cosine equals cosine cosine plus sine sine cosine" back in 1979). But for short distances, less than 100 miles or so, and avoiding cases in arctic latitudes, you can use the much simpler, more intuitive Pythagorean distance:

*D*

^{2}= (

*L*-

_{2}*L*)

_{1}^{2}+ [cos

*L*· (

*Lon*-

_{2}*Lon*

_{1}*)]*

^{2},

where L is the mean latitude. The biggest advantage of this latter formula is that it's easier to teach to anyone who has learned some basic plane geometry. A second advantage is that it is equivalent to plotting points on a universal plotting sheet and reading off the distance with a straight edge. Indeed, our ability to use a universal plotting sheet is identical to the approximation of the spherical geometry of the full cosine equation by the plane geometry of the Pythagorean formula with longitude differences scaled by cos(latitude). Even in the case of the very large distance from Cook Strait to the Hawaiian Islands, this relatively crude Pythagorean distance is not far from the "exact" cosine distance, and within the limits of the problem (Cook Strait is not a point; the Earth is not actually a sphere, etc.), it's actually close enough.

You mentioned in a follow-up post thinking briefly that you might need to use a haversine formula. I would guess that this was a result of a little Googling. Many years ago, but within the range of time that the Internet has been accumulating information, some computer processors and associated software had difficulty with extreme cases using the cosine equation. This led to a recommendation of a different form of the equation (that "haversine") with better properties in extreme cases. This is no longer the case --the information is obsolte, and yet coders often still follow that recommendation since the Internet tells them to do so! I have just recently read an article where someone was blindly calculating local neighborhood distances (distances less than five miles!) from latitudes and longitudes and using the haversine formula to do it, quite oblivious to the fact that the plain cosine equation would work or, with even a bit less complexity, the simple Pythagorean distance equation. And what's more, either the haversine formula or the cosine equation in such a case are overkill and misleading in their implied accuracy (see PS).

Finally, I don't think anyone mentioned this yet: great circle distances also have relevance in certain celestial navigation problems. In lunars, and also in cases of measuring star-to-star distances for sextant calibration, we need the angular distance between bodies given their coordinates. This is exactly the same great circle problem projected onto the celestial "sphere". And since the celestial sphere doesn't really exist, it is a "sphere" by definition, and we don't have to worry about any datum issues or the ellipsoidal shape of the Earth.

Frank Reed

ReedNavigation.com

Conanicut Island USA

PS: A little aside on that problem of calculating distances in a neighborhood, it's an interesting case of pure math in opposition to practical math. If you want to know the quickest route across the middle of a city, calculating the great circle distance with its illusion of accuracy and exactness can be quite misleading. It won't help you deliver pizzas on time! Even if we don't have street navigation software enabled to determine the quickest route based on traffic, speed limits, and all that, the concept of distance is quite different on a city grid. The Pythagorean equation D^{2} = (x_{2 }- x_{1})^{2} + (y_{2 }- y_{1})^{2} should often be replaced by D = |x_{2 }- x_{1}| + |y_{2 }- y_{1}| (using the vertical bars, ||, here to represent 'absolute value'). If you live in a city where the streets are aligned on a square grid, like in many US cities, then distances based on the Pythagorean equation or its spherical extension, the cosine equation, both make diagonal distances look too short. If I am driving from 4th Avenue and 30th Street to 14th Avenue and 50th Street on a square grid, then taking the apparently shorter diagonal route saves no distance at all. Alternating a block north with a block east all the way from start to end for a total of 10 blocks to the north and 10 blocks to the east is the same distance as travelling first 10 blocks east and then after that 10 blocks north. The abstract formulation of the problem as geometry on a sheet of paper or even more abstractly on a spherical Earth ignores the practical geometry of the grid of streets. This disparity is actually very similar to the issues for ocean and air navigation. Shorter by geometry is not necessarily shorter by time or by cost. Real voyages today are highly efficient in part because they do not aim for great circle routes. The geometric distance is only a part of the equation of efficiency.