# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: least squares method for determining a fix
From: Paul Werner
Date: 2003 Oct 14, 11:06 +0200

----- Original Message -----
Sent: Friday, October 10, 2003 1:39 PM
Subject: least squares method for determining a fix

Hi,
does anybody know the formulas to calculate the error ellipse around the fix obtained with the least squares method?
I've got some formulas that I found on a Spanish site, but they don't explain what is sigma (I suppose a variance) and most of all how to compute it.
The formulas are (a probability of 0.95 is generally considered):

k=sqrt(-2*log(1-probability)

theta=arctan(2*B/(A-C)

a=sigma*k/sqrt(nObs/2+B/sin(theta))

b=sigma*k/sqrt(nObs/2-B/sin(theta))

where:    a and b are the axis of the ellipse
nObs is the number of sights
A, B and C are parameters calculated with the least squares method as found in the Nautical Almanac

These formulas only give a and b, what about the orientation of the two axis?

Thanks.
Federico
Browse Files

Drop Files

### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

### Email Settings

 Posting Code:

### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)