# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
Rounding decimal fractions
From: Paul Hirose
Date: 2013 Jul 18, 12:28 -0700

```Stan K wrote:
> A couple of years ago a friend of mine and I were discussing how to
implement Increments and Corrections in my Celestial Tools program and
of his spreadsheet, we noticed that the Almanac values apparently did
not use what we call "standard" rounding, in this case, for instance,
0.00 through 0.04 would round down to 0.0 and 0.05 through 0.09 would
round up to 0.1.

I thought the convention, when a decimal can be rounded up or down with
equal accuracy, is "round to even." E.g., 1.05 rounded to the nearest
tenth is 1.0, but 1.15 is 1.2. But I see that my HP 49G calculator
rounds up in these borderline cases. With 2 decimal point precision
selected, 1/8 = .13, 3/8 = .38, 5/8 = .63, 7/8 = .88.

The composite formatting feature in Microsoft's .NET Framework does the
same thing as the calculator. But I discovered the .NET Math.Round class
gives the programmer a "round to even" option. Results from a test program:

0.125 0.13 0.12
0.375 0.38 0.38
0.625 0.63 0.62
0.875 0.88 0.88

The first column is the fraction to full accuracy. In the second and
third columns I used composite formatting with precision of two decimal
places. The difference is that in the third column the value was first
rounded with Math.Round.

The sum of the second column is a little high (1/8 + 3/8 + 5/8 + 7/8 = 2
exactly) because rounding all borderline cases up introduces a small
systematic error.

I have to admit I never pay attention to these fine points in my own
software. Since Tinyac and Lunar3 both have selectable precision, I
assume the user will use, say, .01 precision if "tenths" are critical.

--
I filter out messages with attachments or HTML.
```
Browse Files

Drop Files

### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

### Email Settings

 Posting Code:

### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)