A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Frank Reed
Date: 2018 Nov 6, 14:45 -0800
Bill L., you wrote:
"the ellipse is not a circle if the LOPs do not meet at 90 degrees"
Here's an amazing thing: if you dig back into the history of the old mailing lists that were the ancestors of NavList, the very first iage attachments to messages back in the late 1990s (-ish... I would have to check) were example of plots of error ellipses associated with various combinations of lines of positions.
Of course the width of the ellipse on the minor axis is going to be approximately the usual uncertainty in celestial sights generally. Call it one minute of arc. The length of the major axis, the long axis of the ellipse, is simply related to the angle between the lines of position. For the smaller angles that are most of interest, where the lines cross at an angle or 30 or 40° or less, the ratio of long axis to short axis is approximately equal to 10 divided by the number of compass points of separation . That is, if two lines of position are separated by one compass point, then the long axis is ten times longer than the short axis. If separated by two compass points, the ratio is five. And that's a useful bit of info! If I have two lines of position separated by about two points, 22.5°, then if the usual "fuzziness" of the ellipse along the good axis is about a mile, the "fuzziness" of the ellipse, in a real mathematical sense, is five miles in the perpendicular direction (which naturally is aligned right between the two lines of position. In more exact terms, the ratio of long axis to short axis is cot(dZ/2) where dZ is the difference in azimuth between the two lines or equivalently the difference in bearing between the two bodies. Note: this result follows from a simple calculation using the usual least squares analysis --nothing especially clever about it-- it's just a way of converting a mathematical result into a bit-sized morsel.
A serious problem with the history of the computation of error ellipses was a choice made by the editors at the Nautical Almanac office years ago. Around the same time that the least squares solution started to be published regularly in the Nautical Almanac itself (after 1989), a volume was published covering a variety of calculation methods including the calculation of error ellipses. Unfortunately they chose to calculate the standard deviation of the observations from the observed sights themselves. This is an option but it meant that error ellipses were undefined for a pair of lines of position, and it implied certain other oddities. For example, a larger triangle for three crossing lines automatically yielded a larger error ellipse. This apparently satisfied the preconceived notions of navigation lore, but it was a poor choice in mathematical terms. The standard deviation of observations should be estimated a priori. Then two sights have an error ellipse that means somethings. And also three sights have an error ellipse that does not misleadingly shrink when the lines of position cross in a tiny triangle.