# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Single precision sight reduction**

**From:**Paul Hirose

**Date:**2015 May 07, 15:07 -0700

Recently there has been discussion of sight reduction accuracy with single precision floating point values. On the principle that an ounce of experimental data is worth a pound of theory, I ran a Monte Carlo simulation. In my C# source code the formulas are: Hc = (float)Math.Asin(sinL * sind + cosL * cosd * cosLHA); Z = (float)Math.Atan((cosd * sinT) / (cosL * sind - sinL * cosd * cosT)); if (Z < 0.0) Z = (float) (Z + Math.PI); The (float) keyword forces a value into single precision. It's necessary because the arc sine and arc tangent functions return a double precision result. All variables are single precision. The test for negative Z is necessary because the Atan() function returns a result in the interval [+pi/2, -pi/2] radians, i.e., [+90, -90] degrees. So if it's negative, the angle is converted into the equivalent in the range 90 to 180 degrees. To generate a sight reduction problem, the program randomly selects a latitude, azimuth, and altitude. Latitude and altitude can be restricted to less than the full 0 and 90 degree range if desired, but I didn't do so in this test. With simple random selection of altitude, half the stars will be above 45°. But that's less than half the area of the visible hemisphere. So altitude selection is weighted to generate constant star density throughout the sky. From latitude, azimuth, and altitude the program computes LHA and declination. These data are passed to the sight solver routine. Up to this point all computations are in double precision. The solver computes altitude and azimuth with the desired reduction method. Its results are compared to the correct answers, and the errors recorded. For this test I used 100 000 simulated observations. Root mean square error in azimuth and altitude was .000667′ and .000296′ respectively. Max azimuth error was .0878′ at altitude 89°40.2′. Max altitude error was .0132′ at altitude 89°40.9′. In production code I wouldn't use the above formulas. The one for Hc loses accuracy the higher the body, and the one for Z fails if Z = 90° since the tangent of that angle is undefined. Trouble isn't likely with real world data, but a user experimenting with the program can create a degenerate case. You can blow up the Z formula with zero latitude and declination.