# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Refraction formula comparison**

**From:**Paul Hirose

**Date:**2000 Oct 01, 2:37 PM

The Explanatory Supplement to the Astronomical Almanac gives this low-precision formula for refraction below 15 degrees: R = (P / (T + 273)) * (.1594 + .0196 * H + .00002 * H^2) / (1 + .505 * H + .0845 * H^2) where H = apparent altitude, P = air pressure in millibars, and T = degrees Celsius. Result is in degrees. The Explanatory Supplement also gives first-order approximations to correct celestial coordinates for refraction. I will not give them here, but they are simple formulas. You could use them to determine the refraction-corrected separation angle between stars. That was some sharp thinking by George Huxtable, to spot the problem lurking in the Bennett refraction formula. I'd expect the exact location of the singularity to depend on your machine. With my HP-15C calculator, it's at 89.92249993. Any change in the last decimal place avoids it. George is quite correct that a programmer should not shrug off a potential bomb like this. I once crashed an astronomy program by innocently entering exactly 90 degrees for a parameter. Here is a comparison of the refraction formulas: H Bennett Supp. Keys Smart 0 34.5 34.1 35.0 - 1 24.3 24.1 25.4 151.1 2 18.2 18.1 18.8 2.0 5 9.9 9.8 9.9 9.5 10 5.4 5.3 5.3 5.3 12 4.5 4.4 4.5 4.5 15 3.6 3.6 3.6 3.6 30 1.7 1.8 1.7 1.7 Except for formula no. 2, all are equal at higher altitudes. The Explanatory Supplement formula is only supposed to be used below 15 degrees, and the Smart formula from 15 up, but both are good for navigation outside those ranges. I included 12 degrees because that's where the maximum error of the Bennett formula is supposed to occur.