# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: least squares method for determining a fix**

**From:**Paul Werner

**Date:**2003 Oct 14, 11:06 +0200

----- Original Message -----From:Federico RossiSent:Friday, October 10, 2003 1:39 PMSubject:least squares method for determining a fixHi,does anybody know the formulas to calculate the error ellipse around the fix obtained with the least squares method?I've got some formulas that I found on a Spanish site, but they don't explain what is(I suppose a variance) and most of all how to compute it.sigmaThe formulas are (a probability of 0.95 is generally considered):k=sqrt(-2*log(1-probability)theta=arctan(2*B/(A-C)a=sigma*k/sqrt(nObs/2+B/sin(theta))b=sigma*k/sqrt(nObs/2-B/sin(theta))where: a and b are the axis of the ellipsenObs is the number of sightsA, B and C are parameters calculated with the least squares method as found in the Nautical AlmanacThese formulas only give a and b, what about the orientation of the two axis?Thanks.Federico