NavList:

   

Reply

Mitch-

<1. Always round up, like a modern electronic calculator>

But they don't. "Modern" calculators for at least 20 years have had the option 
of round-up, round-off, or round-down available with sometimes varying 
results. And the option sometimes to simply truncate the digits, i.e. 
truncate at 2 places, 3 places, etc.

And there's an additional source of error I suppose for degrees+minutes 
figures as opposed to degrees+decimal. The latter is now the international 
engineering standard for lat/lon positioning apparently, minutes and seconds 
are another obsoletism we practice here.

AFAIK any sine conversions break down the minutes into decimal degrees first 
and then compute, but one could certainly prepare a look-up table with 60 
values in it (rounded properly or not) and then add them. Another table, 
another opportunity for a typo?

In all cases, I've seen some LONG decimals come out of simple calculations, 
like converting 7 minutes (from your 32d07m example) into decimals. Long 
enough to exceed capacity and cause rounding routines to kick in on many 
little creatures.

Somewhere, something, I'll bet is running the wrong roundings--but the 
practical difference in position? Have you tried reducing a sextant sight yet 
to see what difference it makes?

   

Reply