A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Gary LaPook
Date: 2017 Apr 3, 14:18 -0700
You say that the readiout shows 6 digits behind the decimal point. ddd.dddddd° format. And then you say that the last two digits are not reliable which means that 4 digits, at least, are relliable so it is accurate, at least, to 1/10,000 th of a degree. A degree is 60 nautical miles and a nautical mile is 6,076 feet so a degree is 364,560 feet. So to find the accuracy of your GPS divide that by 10,000 and you find the accuracy is at least 35.456 feet. Since you can't read your sextant to a greater precision than 0.1' which is 607.6 feet your concern is unfounded. Can you plot your AP to a precision of 35 feet on your plotting sheet?
As to the altitude accuracy, that is the least accurate dimension for GPS navigation systems. You are expecting way to much accuracy in the altitude readout.
"Generally, Altitude error is specified to be 1.5 x Horizontal error specification. This means that the user of standard consumer GPS receivers should consider +/-23meters (75ft) with a DOP of 1 for 95% confidence. Altitude error is always considerably worse than the horizontal (position error)."
So your horizontal accuracy is well within specifications as is the vertical accuracy.