NavList:

The Explanatory Supplement to the Astronomical Almanac gives the Saastamoinen Formula.  I thought that the "low-precision" refraction formula (which immediately follows in the text) was adequate for celestial navigation purposes.  Perhaps I was wrong, especially when it comes to lunar and other distances.

---- begin paraphrase ----
This formula, valid for zenith distances down to 70 degrees, was devised by Saastamoinen (1972), and it is equivalent to the refraction tables in The Star Almanac.  Given the observed zenith distance, z0, the temperature (Kelvin), pressure (mb), and partial pressure of water vapor (see below), (T0,P0, Pw0), the refraction for a wavelength of 574 nm, (i.e., visible light), and for an observer at sea level is

16".271 * Q * tan z0 * ( 1 + 0.0000394*Q*tan^2 z0 )  -  0".0000749 * P0 * ( tan z0  +  tan^3 z0 )

where Q = ( P0 - 0.156*Pw0 ) / T0,
and Pw0 = Rh * ( T0 / 247.1)^18.36, with Rh being the relative humidity.

Saastamoinen gives correction tables for other wavelengths and heights above sea level.

---- end paraphrase ----

I assume that this gives results very similar to the paper of Auer and Standish, to which Frank Reed kindly placed a link in one of his posts back in 2005.


Peter Hakel