# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Refraction near the Horizon – Observation vs. Calculation**

**From:**Frank Reed

**Date:**2013 Apr 5, 12:11 -0700

Bruce Pennino wrote:

"I was interested to see that you use the equation for dip 1.76 sq rt H meters, which is the same as 0.971 sq rt H feet. I gather that this is the universally accepted equation, which I've proven to myself is from basic trig knowing the average radius of the earth? Right?"

Not quite. The constant 0.97 has TWO sources. You can do trig, work a series expansion, and calculate. That will give you one constant (see below). But, as I always try to point out, and HAVE SAID many times in the past couple of months, that calculation is strictly a geometry result: it gives the wrong constant. Unfortunately, dip is not a geometry problem -- it's a physics problem. Light rays don't travel in straight lines. They are bent by refraction. It's a well-established physics problem, and the result under the simplest assumptions about the nature of the air turn out to be very simple: the result of the physics calculation is identical to the geometric calculation but with the actual radius of the Earth, R, replaced by an "effective" radius, R/(1-k), and this applies to any calculation involving terrestrial refraction --not just dip. I posted a diagram on this after I thought everyone here understood all of this (link below). NONE of this dip discussion will make sense to you until you understand this concept of refraction modifying the dip relationship. The average value of that quantity k, which is the downward rotation of a light ray per nautical mile is 0.16. But the reason there's variability is because k is fundamentally weather-driven. If you can precisely measure the properties of the atmosphere when you're making observations, you can calculate a more accurate value of dip. But in general, this is not possible. It's WEATHER.

The geometry for calculating dip is simple enough. I'll just quote the result for the basic triangle, which I think everyone can draw without explanation, before we narrow down from geometry to reality:

cos(dip) = R/(R+h)

where R is the radius of the Earth and h is the observer's height of eye. When you draw your triangle, you also find that the angle "dip" is ALSO equal to the angle at the center of the Earth between the observer's location and the point where a straight ray from the observer's eye just meets the Earth. That means that the dip is identical to the distance to the horizon, measured as an angle geocentrically. Or, in more practical terms, dip in minutes of arc equals distance to the horizon in nautical miles. Note carefully: this is the pure geometric solution. The equation does NOT match observations.

The next step in the pure geometric solution is to limit our altitude h to something less than a mile or two. We're not interested in the horizon as seen from space at this point in time. We're also not interested in horizon calculations from five or ten miles up since the horizon is always lost to "extinction" and simple atmospheric haze from altitudes above a mile or so (and frequently much lower). In practical terms, you can't see the sea horizon from a high-altitude airplane.

Since we are limiting h to be less than a mile, the quantity h/R is a very small number. Also, if we run an exact calculation of the angle, the dip angle is a very small number as long as h is less than a mile. That means that we can use series expansions on the equation above. The series expansions we need are as follows (for the cosine, this assumes the angle is "pure ratio" or --same thing-- the angle is "in radians"):

cos(x) = 1 - x^2/2

1/(1+x) = 1 - x

These are valid approximations for any x so long as x is much less than one. We take the original equation and re-write it (now using d for dip) first by dividing the numerator and denominator on the right by R:

cos(d) = 1/(1+h/R).

We use the cosine series on the left and the binomial series on the right:

1 - d^2/2 = 1 - h/R.

Cancel the 1, reverse the signs:

d^2/2 = h/R.

And finally:

d = sqrt(2*h/R).

This version of the equation applies for any units, and its result is an angle as a pure ratio. But if we want the angle in minutes of arc, we have to multiply by 3438:

d = 3438'*sqrt(2*h/R).

The value of the Earth's true radius is a fixed number. If h is in feet, then we should express the radius in feet. That's about 20,909,000 so now

d = 3438'*sqrt(2*h/20909000).

Pull the constants out of the square root and you get

d = 1.06'*sqrt(h), when h is in feet.

This geometric results was worked out centuries ago, but when dip was measured, the results were always smaller. The constant derived from this simple geometric analysis is too big by about 10%. And the distance to the horizon is correspondingly too small by about 10% (they no longer match). The "square root of height" variation in altitude matched observations and there's no significant debate on that, but only if 1.06' is replaced, on average by 0.97' as the constant of proportionality. It took no great science to realize that this was due to refraction, but there was some "academic" disagreement 200 years ago on the exact value of the refracted constant. Should it be 0.97 or 0.98 or even 0.90? There were cases of editors of navigation manuals, including Bowditch, claiiming that their dip table was better because they used a more "scientific" value for this constant. Those claims were hot air. Today, it's much easier to recognize that the constant just isn't a constant. It's relatively stable most of the time, and the value 0.97 is an excellent choice for an average, but there can be substantial variations. In simple modelled cases, like linear lapse rate variations, different "constants" can be calculated for different weather conditions. If we change the lapse rate of the lower atmosphere (and we know from observations that it is quite variable), we get a different constant. Unfortunately, this simple variation is only a piece of the puzzle. Developing a modified formula for dip that would incorporate real variations in conditions that could be used in the real world is as complicated as predicting fine details of the weather.

Here's the message with the dip "cartoon":

http://fer3.com/arc/m2.aspx/Refraction-dip-diagram-FrankReed-mar-2013-g22963

-FER

PS: Bruce, consider using "sqrt(h)" for square root of h rather than "sq rt h". The former is standard.

----------------------------------------------------------------

NavList message boards and member settings: www.fer3.com/NavList

Members may optionally receive posts by email.

To cancel email delivery, send a message to NoMail[at]fer3.com

----------------------------------------------------------------