Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Re: Refraction:Which formula for Heights below 15°
    From: David Iwancio
    Date: 2022 May 16, 20:13 -0700


    Both formulas are simply different attempts to make a curve that fits the results of far more complicated math.  The constants don't represent anything beyond what the author feels is the best fit for whichever refraction model they're trying to approximate.

    I'm pretty sure the model of "A * cot[Ha + B/(Ha+C)]" was the brainchild of the author G. G. Bennett in a paper, published by the Royal Institute of Navigation, titled "The Calculation of Astronomical Refraction in Marine Navigation."  It was one of two or three models he introduced, of various complexities, and analyzed them (and others) against the then-standard refraction model developed by B. Garfinkel, published in the Astronomical Journal.  Bennett was specifically looking for simple-ish formulas that could easily be calculated on an inexpensive handheld calculator.

    The form using 7.31 and 4.4 is from Bennett's original paper.  Since then the Nautical Almanac has moved to a new refraction model, and C. Y. Hohenkerk of the HMNAO tweaked the constants in Bennett's formula to 7.32 and 4.32 to better fit (two specific points in) the new tables, and that version is in the appendix of recent editions of the Nautical Almanac.  I'd guess the version with 6.82 and 4.024 is something Frank came up with himself.

    For anything over 15° above the horizon or so, any formula involving the contangent of the altitude is just as good as any other.  The usual one I've seen for just those altitudes is 0.95 * cot(Ha).  Some authors suggest using different formulas for different ranges of altitudes, to avoid things like having a non-zero refraction at altitude 90° (which is a sin committed by Bennett's model).

    Refraction much below 10° gets into "It's Complicated!" territory and you really shouldn't worry about fractions of an arcminute. 

    The Nautical Almanac and other sources almost always allow for modifications for local temperature and barometric pressure, and for most practical purposes those are the only modifications needed to get a usable refraction number down to maybe 5° or so.  But at lower altitudes other factors become signifigant, like humidity and even the local gravity (varying with latitude).  A 2004 paper I've looked at by A. T. Young in the Astronomical Journal suggested that you could get refraction numbers accurate to the nearest arcminute down to 2° altitude if you launched a weather balloon to sound air temperatures up to around 1000 meters above you.  Refraction at altitudes below 2° you're mostly just fooling yourself and the numbers you find are simply reasonable guesses; there are just too many unknowns above and/or beyond your visible horizon.

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site