# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: least squares method for determining a fix**

**From:**Andr?s Ruiz

**Date:**2003 Oct 14, 12:44 +0200

*Estimated Position Error*If three or more position lines are obtained an estimate of the error in position may be calculated. In general as the number of observation increases the error in the estimated position decreased.

The standard deviation of the estimated position, in nautical miles is:

`S = F-D*dB-E*dL*COS( BI );`

`sigma = 60.0*sqrt( S/(nObs-2.0) );`

The *Confidence Ellipse* of axes (a,b) is:

`Prob = 0.95;`

`k = sqrt(-2.0*log(1.0-Prob)); // scale factor`

`theta = ATAN( 2.0*B/(A-C) )/2.0;`

`a = sigma * k / sqrt( nObs/2.0 + B/SIN(2.0*theta) );`

`b = sigma * k / sqrt( nObs/2.0 - B/SIN(2.0*theta) );`

More
information:

http://www.geocities.com/CapeCanaveral/Runway/3568/index.html

http://www.geocities.com/andresruizgonzalez/

http://www.geocities.com/andresruizgonzalez/

-----Mensaje original-----De:Navigation Mailing List [mailto:NAVIGATION-L{at}LISTSERV.WEBKAHUNA.COM]En nombre dePaul WernerEnviado el:martes, 14 de octubre de 2003 11:06Para:NAVIGATION-L{at}LISTSERV.WEBKAHUNA.COMAsunto:Re: [NAV-L] least squares method for determining a fix----- Original Message -----From:Federico RossiSent:Friday, October 10, 2003 1:39 PMSubject:least squares method for determining a fixHi,does anybody know the formulas to calculate the error ellipse around the fix obtained with the least squares method?I've got some formulas that I found on a Spanish site, but they don't explain what is(I suppose a variance) and most of all how to compute it.sigmaThe formulas are (a probability of 0.95 is generally considered):k=sqrt(-2*log(1-probability)theta=arctan(2*B/(A-C)a=sigma*k/sqrt(nObs/2+B/sin(theta))b=sigma*k/sqrt(nObs/2-B/sin(theta))where: a and b are the axis of the ellipsenObs is the number of sightsA, B and C are parameters calculated with the least squares method as found in the Nautical AlmanacThese formulas only give a and b, what about the orientation of the two axis?Thanks.Federico