# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Refraction, Cassini model**

**From:**Paul Hirose

**Date:**2018 Nov 27, 14:04 -0800

I added another version of the Cassini model so now there are three to evaluate. In addition, my refraction tables have been expanded to 1-degree steps from 10° to the horizon. To save readers the bother of referring to my original message, I'll restate the formulas here and include some new material. CORRECTIONS FOR TEMPERATURE AND PRESSURE For standard sea level conditions I assume 50% humidity, 10° Celsius (50 F), and 1010 millibars (29.83 inches Hg). For different temperature or pressure, compute correction factor f. If P is pressure (millibars) and T is temperature (Celsius), f = .280 * P / (273 + T) For inches Hg and Fahrenheit, f = 17.1 * P / (460 + T) In those formulas P is the actual air pressure at the observer. It can be calculated from altimeter setting A for an observer h meters above sea level: P = A * (1 - h / 44 300) ^ 5.26 For h in feet, P = A * (1 - h / 145 000) ^ 5.26 In both formulas A can be millibars or inches of mercury. The result will have the same unit of measure. For example, at 5000 feet and altimeter setting 29.8 inches, P = 24.78 inches. A precise calculation says 24.80. I use the term "altimeter setting" instead of "barometric pressure" because when the latter is publicly distributed (in the US at least) it's really the former. Unless best accuracy is important, it's enough to use a fixed nominal altimeter setting since the normal range is small. A 2% change, say from 29.9 to 30.5 inches, is unusual. A corresponding change in absolute temperature, say from 60 to 70 F, is easily exceeded every day. REFRACTION MODELS "Refco" is the routine from the International Astronomical Union SOFA (Standards of Fundamental Astronomy) library. Its documentation says its accuracy is better than 1″ at 10° altitude, compared to a ray trace through a model atmosphere. Performance below 10° is not stated. http://www.iausofa.org/2018_0130_C/sofa/refco.html "Low alt" is the formula in the Astronomical Almanac section B for altitude less than 15°. I have factored its polynomials into a format convenient for a computer program: refraction = P * (.1594 + (.0196 + 2e-5 * a) * a) / (273 + T) / (1.0 + (.505 + .0845 * a) * a)) where "a" is altitude (degrees) and the result is degrees. At the horizon in standard conditions it gives a reasonable 34.1′. Cassini's refraction model is explained by astronomer Bruce Young: https://aty.sdsu.edu/explain/atmos_refr/understanding.html In standard conditions: R = Earth radius = 6,371,000 meters h = height of homogeneous atmosphere = 8307 m n = air index of refraction = 1.0002816 Z = zenith distance = 90° - altitude First compute a quantity I call "sinZ1": sinZ1 = R / (R + h) * sin Z sinZ1 = .9987 * sin Z Then compute refraction from sinZ1 and the index of refraction: refraction = arcsin (n * sinZ1) - arcsin (sinZ1) To be strictly correct, R is the distance from the observer to Earth center and therefore should include height above sea level. However, at .01′ accuracy and 2000 meters (6600 feet) above sea level the small adjustment has no noticeable effect. In the tables I evaluate three Cassini versions. "Cas1" is refraction from the basic formula, multiplied by correction factor f. In other words, Cas1 assumes refraction is directly proportional to air density. "Cas2" applies separate corrections to homogeneous atmosphere height h and refractive index n. It assumes refractivity (the fractional part of index of refraction) is proportional to air density: h = 8.22 * P / f (P is millibars) h = 279 * P / f (P is inches Hg) n = 1.0 + f * .0002816 "Cas3" is a formally correct implementation of the Cassini model. Height of observer is applied to R, h is computed as in Cas2, and refractive index n is computed from temperature, pressure, and humidity with the Ciddor formulas, which are too long to quote. See this NIST page: https://emtoolbox.nist.gov/Wavelength/Documentation.asp "Ben1" is the basic Bennett formula in the Nautical Almanac: R = 1 / tan (H + 7.32 / (H + 4.32)) where H is refracted altitude (degrees) and R is refraction (minutes). "Ben2" adds the correction from "Astronomical Algorithms" by Meeus: -0.06′ * sin (14.7 * R + 13) where R is the refraction (minutes) from the basic Bennett formula but — CAUTION — the result of the expression in parentheses is degrees. The mixture of degrees, minutes, and radians requires careful coding on a computer. RESULTS At 10° and up, the SOFA Refco routine is claimed to be highly accurate. Therefore, it is the standard, and refraction values from the other models are the difference from Refco. Below 10° I use the Astronomical Almanac low altitude formula as the reference. The first table is at sea level standard conditions of 10 C (50 F), 1010 millibars (29.83 inches Hg) and 50% humidity. Altitude is the refracted altitude. Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2 45 0.97 0.22 0.00 0.00 0.00 0.03 0.00 40 1.15 0.18 0.00 0.00 0.00 0.03 0.00 35 1.38 0.15 0.00 0.00 0.00 0.04 0.01 30 1.67 0.11 0.00 0.00 0.00 0.05 0.01 25 2.06 0.07 0.00 0.00 0.00 0.06 0.01 20 2.63 0.03 0.00 0.00 0.00 0.07 0.02 15 3.55 0.00 0.00 0.00 0.00 0.08 0.03 10 5.29 -0.01 0.00 0.00 0.00 0.10 0.04 9 0.00 5.83 0.00 0.00 0.01 0.11 0.05 8 -0.02 6.51 0.00 0.00 0.00 0.11 0.05 7 -0.05 7.35 -0.02 -0.02 -0.01 0.10 0.05 6 -0.14 8.41 -0.05 -0.05 -0.05 0.09 0.05 5 -0.36 9.80 -0.12 -0.12 -0.11 0.07 0.06 4 -1.00 11.66 -0.28 -0.28 -0.28 0.05 0.07 3 -3.35 14.28 -0.70 -0.70 -0.69 0.02 0.10 2 -15.86 18.12 -1.87 -1.87 -1.86 0.00 0.14 1 -172.34 24.12 -5.24 -5.24 -5.22 -0.02 0.18 0 --- 34.13 -14.05 -14.05 -14.03 -0.33 0.30 Cassini is practically a perfect match to the reference models — better than either Bennett formula — down to 7°. (In standard atmosphere conditions the three Cassini versions should be identical, but tiny differences occur due to my rounded constants.) At very low altitude Bennett is remarkably close to the low altitude formula. However, we must remember the latter is only another model and not the absolute truth. Accuracy of a few hundredths of a minute in this part of the table is an illusion. In the second table air density is 116% of standard: pressure 1070 mb (31.60 inches) and temperature -15 C (5 F). Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2 45 1.12 0.26 0.00 0.00 0.00 0.03 0.00 40 1.34 0.21 0.00 0.00 0.00 0.04 0.00 35 1.60 0.17 0.00 0.00 0.00 0.05 0.01 30 1.94 0.12 0.00 0.00 0.00 0.05 0.01 25 2.40 0.08 -0.01 0.00 0.00 0.06 0.02 20 3.07 0.03 -0.01 -0.01 0.00 0.07 0.02 15 4.14 -0.01 -0.02 -0.01 0.00 0.09 0.03 10 6.18 -0.04 -0.03 -0.01 0.00 0.09 0.03 9 0.04 6.78 0.00 0.04 0.05 0.13 0.07 8 0.04 7.56 0.00 0.04 0.06 0.13 0.08 7 0.03 8.54 -0.02 0.05 0.06 0.12 0.08 6 -0.03 9.77 -0.06 0.04 0.06 0.11 0.09 5 -0.18 11.38 -0.14 0.02 0.04 0.09 0.09 4 -0.71 13.55 -0.33 -0.07 -0.04 0.06 0.11 3 -2.83 16.59 -0.81 -0.36 -0.32 0.03 0.13 2 -14.88 21.06 -2.18 -1.39 -1.33 0.00 0.13 1 -171.99 28.03 -6.09 -4.82 -4.75 -0.02 0.17 0 --- 39.66 -16.32 -14.77 -14.69 -0.38 0.42 Down to 10° Cas3 is a perfect match to Refco, Cas2 practically as good, and Cas1 only slightly less so. All three are better than Ben1 and Ben2, though any advantage over the latter is marginal. At 9° the change of reference model accounts for the discontinuity in the table. The simplest Cassini version, Cas1, begins with the lead, but soon loses to Cas2. The complex refractive index computation in Cas3 is no advantage here. In the third table air density is 84% of standard. Pressure is 896 mb (29.46 inches), equivalent to 1000 meters (3300 feet) above sea level and temperature 25 C (77 F). Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2 45 0.81 0.19 0.00 0.00 0.00 0.03 0.00 40 0.97 0.16 0.00 0.00 0.00 0.03 0.00 35 1.16 0.13 0.00 0.00 0.00 0.04 0.01 30 1.40 0.09 0.00 0.00 0.00 0.04 0.01 25 1.73 0.06 0.00 0.00 0.00 0.05 0.01 20 2.21 0.03 0.00 0.00 0.00 0.06 0.02 15 2.98 0.01 0.01 0.00 0.00 0.08 0.03 10 4.43 0.01 0.02 0.01 0.00 0.11 0.05 9 -0.03 4.91 0.00 -0.01 -0.02 0.09 0.03 8 -0.05 5.48 0.00 -0.03 -0.03 0.09 0.03 7 -0.10 6.19 -0.02 -0.05 -0.06 0.09 0.03 6 -0.19 7.08 -0.04 -0.09 -0.10 0.08 0.03 5 -0.43 8.25 -0.10 -0.18 -0.19 0.06 0.03 4 -1.07 9.82 -0.24 -0.36 -0.38 0.04 0.03 3 -3.34 12.03 -0.59 -0.79 -0.82 0.02 0.06 2 -15.07 15.26 -1.58 -1.92 -1.95 0.00 0.12 1 -158.61 20.32 -4.41 -4.95 -4.99 -0.01 0.21 0 --- 28.75 -11.83 -12.47 -12.53 -0.28 0.21 Cas1 is practically as good as its siblings down to 10°. Below 10° the simplest Cassini version is best and the most complex is worst! But all are within 2% of the low altitude formula down to 5°. Based on my results, I don't think the complexity of a precise refractive index calculation in the Cas3 model is worthwhile. The difference from the simple corrections of Cas1 and Cas2 are insignificant for navigation. It's difficult a choose the winner between the latter two, so I suppose it should be Cas1 due to its simplicity. Since it applies the air density correction to refraction itself, the expression R / (R + h) in the formula for sinZ1 is a constant. All three Cassini versions have superior accuracy to even the improved Bennett model except at very low altitude where no formula is reliable. However, the Cassini model in any variant requires evaluation of three trig functions, vs. none in Ben1 and one in Ben2. Though it's more than 300 years old, Cassini's refraction model with simple corrections for nonstandard conditions has amazing accuracy — more than enough for celestial navigation.