# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Accuracy of sextant observations at sea**

**From:**Antoine Couëtte

**Date:**2010 Nov 26, 08:13 -0800

Mon Cher Piterr,

Wonderful conclusion sentence in French, Merci Beaucoup !

We are in full agreement about the importance of removing "bad data" at some point of our computations.

Your use of the manual slope plotting line in order to get N/S or E/W azimuths is very clever : well done !

*******

In the middle of my former post, one sentence read : "However and for the same reasons as above - and thanks to M. GPS again - I generally keep avoiding discarting LOP's.".

This sentence should be corrected into : " However and for the same reasons as above - and thanks to Mr GPS again - I generally keep avoiding discarting AVERAGED LOP's ".

*******

You are asking :

" How does this averaging work in practice, given you're dealing with different times and altitudes? How do you go about averaging them? ".

Just an example here taken from one of my archives :

28 Oct 1984, TT-UT = 54.8s, FN Aircraft Carrier FOCH (now property of the Brasilian Navy), on an evening where I was not night flying (maybe because the Moon was out that night - since I also shot it on that evening ... - , so these "moonly" nights were devoted to "nuggests" training ... :-)).

Loran C Position at 16H30M00.0s UT : N34d30.3m/E019d59.1m, and

Course/speed made good 270d/14.0kts

Height of Eye 28 meters, P=1016 Mb, T=23�C

Vega observations are given under the following format : UT / Sextant value corrected for instrument error (cfie) :

16h28m24.0s / 69d31.9m

16h29m21.3s / 69d21.4m

16h30m17.3s / 69d11.2m

16h31m05.7s / 69d01.8m

16h31m47.0s / 68d55.1m

I am getting the following results (FAR too many digits given ....):

1 - Averaged observation time : 16h30m11.060s, and Averaged Observed geocentric height 69d02.558m, and

2 - Vega apparent geocentric coordinates at averaged observation time :

Apparent RA = 18h36m23.984s, Apparent Dec = 38d46m18.355s , GAST = 284d49m39.404s, and

3 - DR Position at the time of the averaged observation time : N34d30.300m/E019d59.048m, and

4 - Geocentric height and azimuth derived from DR at the averaged observation time : 69d04.213m, which by substraction from true observed height yields an intercept of 1.655 NM rounded to 1.7 NM

*******

"Softwarewise" I have chosen to correct each and every instrument height into its geocentric height counterpart BEFORE averaging the resulting UT/Heights values. From such paired averaged UT/geocentric height, I compute a standard LOP (Intercept and Azimuth). My records here show : 1.7 NM away/Azimuth 289.3 d.

Note : I personnally limit the maximum observation time span to 5 minutes between the first and the last of the shots to be averaged as this generates no appreciable systematic resulting error on the final results if you use a linear approximation to the data. I have chosen to average geocentric heights (instead of sextant heights cfie) simply because I have to correct heights anyway, and through so doing, I am getting rid of (very minor) refraction non linearities at low altitudes. However, it is (generally) NOT necessary to use averaged geocentric heights values vs averaged Instrument ones (cfie) just to avoid any refraction issues at low altitudes when you limit your observation time span to 5 minutes of time. A 5 minute time span also generates no additionnal error whatsoever under any circumstance if you simply perform linear data approximation (i.e. our traditional straight line curve).

Let us then rework our example through the more traditional method of averaging sextant heights first, then reducing such averaged instrument height into its geocentric height counterpart:

The averaged values fo the data written on top of this post are :

UT = 16h30m11.060s, (obviously it has to be the same value as above), and

H = 69d12.280m.

From this paired data we get the following :

Observed Geocentric height from Averaged Sextant heights : 69d02.558m. This is exactly the same value obtained through the first method of individually reducing each sextant height into its geocentric counterpart BEFORE averaging the whole set.

And again, if Math have not changed since the past 30 years or so, the average values of UT and Height belong to the traditional least square linear regression of these data. So, except when you need a specific point from this line, e.g. for N/S or E/W azimuts as you earlier mentioned, since all points of this regression line in the vicinity of the observation time contain exactly the SAME INFORMATION, just picking up in particular the (easily computed) averaged data values brings no appreciable consequence on the overall accuracy of the LOP computation.

A bit lenghty this time, and I hope that there are no typos, but I should have said it all ..

Thank you Piterr for your Kind Attention and

Very Best Regards

Antoine

Antoine M. "Kermit" Cou�tte

----------------------------------------------------------------

NavList message boards and member settings: www.fer3.com/NavList

Members may optionally receive posts by email.

To cancel email delivery, send a message to NoMail[at]fer3.com

----------------------------------------------------------------