# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Moon total correction- round it up?**

**From:**Paul Hirose

**Date:**2014 Oct 11, 17:03 -0700

On 2014-10-10 7:46, Samuel L wrote: > Gary, > I never said my sextant reads to seconds. Read the original post. You asked how to round off an Ho of 25°37′50.4″. Writing the angle with that much precision implies Ho is known to better than a second of arc. It's not surprising someone would ask how you'd accomplish such a measurement. Good practice is to omit digits that contribute nothing useful. For instance, suppose I find the circumference of a tree trunk is 25 1/2 inches by tape measure. Then I divide by pi and get a diameter of 8.1169020976866621242130719319982 inches. But it's a good guess that I rounded off the tape measure reading to the nearest half inch. Roundoff error is therefore up to 1/4 inch, or about 1% of circumference. It follows that the diameter computation can be off by 1% too, or roughly .1 inch. Furthermore, consider the rough surface and non circular cross section of the trunk. To real world precision, we have an 8-inch tree trunk. I can calculate the diameter to far higher precision, but the additional digits are just noise. Similar considerations apply in an angle format conversion. The end result can be no better than the original data. Say a sextant reads 57°29.9′. What is that in decimal degrees, to appropriate precision? First consider the precision of the original data. It's .1′, which is about .0017°. So .001° and .1′ are approximately equivalent precision, and 57°29.9′ can be expressed as 57.498°. Now convert 57.498° to DMS format. One second is about .0003°, so tenth seconds are not justified. Even single seconds are about three times more precise than the decimal angle. But that's the most reasonable match, so we round the conversion to one second precision: 57°29′53″. Note the original sextant reading would have been 57°29′54″ in that format, so the "round trip" missed its target by 1″. That's more than good enough, but it does show how small errors accumulate in the conversions from DDMM.m to DD.ddd to DDMMSS. Choice of an appropriate precision is often a judgement call. If you're testing a sextant with an indexing table and collimators, the .03′ max roundoff error that can occur at .001° precision may be an uncomfortable fraction of your error budget. In practical navigation the same error is negligible. > What I needed to know was when any readings or figures be rounded up. > I'm always using an AH so once IE is removed during the reduction and that figure is divided by 2 the majority of figures end up with seconds of arc at the end. > For example; > Hs= 43d 43.4min > IE -.1 > Result divided by 2= 21d 51min 39sec (that's what the calculator says) Sam, errors are subtracted and corrections added. The -.1′ index error ought to be subtracted (not added) to obtain 43°43.5′, which you then divide by 2 to obtain 21°51.8′. To avoid confusion between errors and corrections, remember the goal is to remove error from observations. Remove = subtract.