# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Calculators, cosines, and floating point computation**

**From:**Frank Reed

**Date:**2019 Jun 3, 07:50 -0700

Greg, you wrote:

"The free Hiper scientific calculator Google Play app has arc cos (cos x) angling down to 10 ^ -15 before displaying zero."

Yeah. Apps can do that. :) You just toss in an extended precision library. But a real calculator? I don't think you'll find one! One of the things I didn't mention in my original post is that it's possible to create software libraries that go beyond the hardware capabilities. This emphatically turns back the clock, from one perspective, since we're now ignoring most of the built-in FPU hardware. But from another perspective, it's thoroughly modern since it assumes that all computing devices are so incredibly fast that we don't need to worry about hardware acceleration.

For another example of a beautiful extended precision product, try Wolfram Mathematica, or its web-based cousin Wolfram Alpha:

https://www.wolframalpha.com/input/?i=acos(cos(0.00000000000000001)) ...

In fact, you can build a super-calculator this way as a smartphone app. Rather than doing the calculations in the app, the "calculator" can just query Wolfram Alpha for any result that it requires. Works great... on-network. :) Oh, but there is one little problem with this plan: products.wolframalpha.com/api/pricing/. A minimum annual fee of $1000? No thanks. Free for experimental use though...

I'm not a fan of calculator apps in general, except for training purposes for those few people who can't make it to an office supply store to buy a real calculator (or who can't afford to spend, or don't want to spend, $9 on a Casio fx-260 Solar). After all, if it's an app, then put it to work. Make an app that does complete celestial calculations, rather than something that requires virtual keypresses like a handheld calculator.

This should not be a quest for the Holy Grail: a device that can manage acos(cos(x)) down to the tiniest imaginable angle. That's a solution to a problem that doesn't exist. The answer to this problem, for navigation, is the same as it has been for decades: simply use the Pythagorean-like calculation for very small distances, as described in my previous post: Dist = sqrt(dLat² + dLon² **· **cos²Lat). If I know my boat's Lat/Lon, and I know a buoy's Lat/Lon, you can dig out the cosine law for the great circle calculation and pat yourself on the back for being "mathematically rigorous" (a fiction), or you can just work it as the distance between two points with x,y coordinates on a plane. The y coordinate is usually taken as Lat, so dLat is just the difference in latitude between my boat and the buoy, and the x coordinate is Lon but scaled by cos(Lat) since lines of longitude converge as we approach the poles, do I take the difference in longitude between boat and buoy and multiply by cos(Lat) --that's dx.

Frank Reed