# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Re: Linear regression and other tools
From: Antoine Couëtte
Date: 2018 Oct 13, 22:54 -0700

RE : Linear-regression-other-tools-Cutting-oct-2018-g43062

Dear Bruce,

When you observe heights in a "noisy" environment - i.e. with quite high random observation errors (SDEV exceeding 1' or even 2') e.g. when observing close from the sea level with high waves or swell - I have experienced that the (x22nd order coefficient derived from the data set may be quite wrong, especially when you have only a limited number of observations at hand.

Hence the idea of replacing it by its theoretical value which can be derived from the Observer's "approximate environment" whenever feasible.

The Navigator generally knows his position to better than 1° (60 NM). Hence starting from the well known formula :

sin h = sin φ * sin D + cos φ * cos D * cos T , with :

h = Geocentric Height

φ = D.R. Latitude

D = Body Declination

T = Body Local Hour Angle computed from D.R. Longitude and Greenwich Hour Angle.

it is possible to compute 1/2 * d²h/dUT²  which is quite close to the theoretical value of this (x22nd order coefficient. Hint: in the course of the computation dT/dUT₁ should take in account your East-West speed especially on a fast boat.

When the effects of the second order terms have been removed, it is only necessary to perform 1st order linear regression on the remaining data.

Sometimes one iteration may improve your final observed fix. From your computed fix, compute again 1/2 * d²h/dUT²  and process your data again into a "refined" fix.

However, as you can guess, this is starting to become a bit complicated in terms of computation (do not mess with the various units and 1st/2nd derivatives coefficients !), much more complex actually than simply deriving this (x22nd order coefficient directly from your data set.

From personal experience, I can say that if the (x22nd order coefficient derived from the data set may and will at times spoil your end results - i.e. your computed fix - under extreme conditions, on the other hand using its theoretical value - even computed from some "approximate" environment - has always yielded more stable computed fixes equal or better than the fixes derived from the data set (x22nd order coefficient.

In other words, and from personal experience: the theoretical  (x22nd order coefficient never performs worse than its data derived counterpart, but it often performs better especially under a noisy data environment.

Best Regards,

Antoine

Browse Files

Drop Files

### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

### Email Settings

 Posting Code:

### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)