# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Sextant Positions versus Map Datums?**

**From:**Paul Hirose

**Date:**2002 Jan 21, 1:25 AM

What an interesting discussion. I've followed it closely and saved all the messages in a dedicated folder. For the first time I've had to really think out how our spherical Earth assumptions affect celestial navigation. It's my belief that the almanac and sight reduction tables are equally valid on both a sphere and an ellipsoid. Let's explore this with a thought experiment. Imagine the inside of the celestial sphere is marked with parallels and meridians, so an observer on Earth can look up and see them. For simplicity's sake, assume there's no Earth rotation. Initially, let's pretend Earth is a perfect sphere and you are at 30N 90W. At this location set up a surveyor's transit, level it, and adjust the horizontal circle to read zero when the telescope is sighted on the celestial pole. By doing this you have aligned the instrument's vertical axis to the zenith, which is the intersection of the 30N and 90W lines on the celestial sphere. You've also brought the horizontal circle's zero point into the plane of the 90W celestial meridian. If I name a specific parallel and meridian on the celestial sphere, can you compute the altitude and azimuth required to direct the transit's scope to that spot in the sky? Yes, of course. This is basically the same problem we solve during a sight reduction. Like our imaginary lines in the sky, the coordinates in the almanac are pure spherical coordinates and aren't based on any datum. If we assume Earth is spherical too, computing altitude and azimuth is straightforward. But what if the world is ellipsoidal? Doesn't that make the computation much more difficult? No, because of the way latitude is defined on an ellipsoid. Mapping and navigation use geodetic latitude, which is the angle between a normal to the ellipsoid (e.g., a plumb line) at your location and the equatorial plane. (Meeus calls this "geographical latitude".) So if you're at 30N 90W on an ellipsoid, the 30N and 90W lines on the celestial sphere intersect at your zenith, just as they did on the spherical Earth. Leveling your transit and aligning it to north gives it the same orientation relative to the celestial sphere that it had on the spherical world. To be sure, its location has changed. But since this movement is negligible compared to the enormous radius of the celestial sphere, the apparent positions of points on the sphere don't change. Thus on an ellipsoidal Earth you can accurately compute Hc and Zn with spherical trig. But we're not quite done yet. The actual shape of the Earth, as sensed by a transit or sextant, is the geoid. That's an imaginary sea level surface perpendicular to gravity at all points. Its shape is extremely complex, too hard to handle in routine computations. That's why datums assume Earth's cross section through the poles is a perfect ellipse. Unfortunately our instruments know nothing about this, and align themselves to gravity. Their celestial observations produce astronomic latitude and longitude, which are coordinates on the geoid and aren't in any datum at all. (I guess you could call it "God's datum".) But the geodesists who create a datum always try to make it a good overall fit to the geoid. Astronomic coordinates should match the geodetic coordinates quite well. Exactly what "quite well" means, depends on the angle between a normal to the ellipsoid and the local vertical at that spot. The angle is called "deflection of the vertical", and is separated into north-south and east-west components. By the standards of the celestial navigator, deflection of the vertical is very small, but it can be greater than some observational errors. http://www.ngs.noaa.gov/GEOID/DEFLEC99/ Click the interactive computations link and try inputting some coordinates. I tried that star atop the Texas capitol dome -- sum of N-S and E-W deflections came to 4.08 seconds, about 400 feet on the ground. That's how much a perfect celestial fix would differ from NAD 83. Bottom line: 1. The tabulated coordinates in the almanac are independent of datum. 2. The correction for Moon's parallax is slightly (.2') affected by the non-spherical shape of Earth. The almanac's text explains how to compute an accurate value. The Moon correction tables don't bother about this, though. All other bodies are so far away, Earth may be considered a sphere as far as parallax is concerned. 3. Sight reduction tables and formulas lose no accuracy on an ellipsoidal world. The plotting is slighly affected. But given the geometric liberties taken by the intercept method, the accuracy of sextant observations, and the small size of the intercepts, "one minute equals one mile" is close enough. Sight reduction programs which converge interatively on the solution, like the method described in the almanac, can finish with very short intercepts. Here non-spherocity should have virtually no effect. 4. The discrepancy between the astronomic coordinates obtained from celestial observations and the geodetic coordinates on charts is slight, at least in NAD 83. However, for a perfectionist applying all possible corrections it can be significant. -- paulhirose@earthlink.net (Paul Hirose)