NavList:

This issue came up on this List, a little while ago, in a discussion of the book ‘The Complete On-Board Celestial Navigator’ that uses values rounded to the nearest minute of arc for the sight reduction process. Someone outside the list brought up the same objection recently, which started me thinking about the matter again.

With minutes of arc of, for example, anywhere between 23.0 and 23.5, 23 is adopted. Between 23.5 and 24.0, 24 is adopted. If a value rounded up is followed by a value rounded down, a canceling out effect occurs. On the other hand, if a rounded up value is consistently followed by rounded up values (or down by down) then the rounded amounts can add up to error. If 15.4 is followed by 57.3 then followed by 19.4, all rounded down, then an error of a little over one minute of arc has been introduced. If 15.4 is followed by 57.8 then followed by 19.9, then the rounded amounts have tended to cancel each other out.

The objection raised was that the errors introduced by this rounding process are potentially accumulative when a series of rounded values are used; as during the sight reduction process.

I propose to use an example to show why this is not so, why the rounded values tend to cancel each other out and do not tend to accumulate. Let’s call the values rounded up ‘boys’ and the values rounded down ‘girls’ – for want of better names …

Couples setting out to make a family have a 50/50 chance of having a boy or girl (actually this is not strictly correct, for biological reasons which need not concern us here).  Thus a probability of 0.5 of either, expressed as a decimal. The chance of having a girl as a second child, GIVEN THAT the first was a girl, is 0.5 x 0.5 = 0.25. Thus the chance of NOT having two girls in a row is 75%. By extension, the chance of not having three girls in a row (or three boys) is 87.5%.

There are 5 or 6 values (6 for stars, 5 for other bodies) entered during the sight reduction process used in George Bennett’s book. So the chance of all of them conspiring towards error (all boys or all girls) is about 1.6%. The chance of this NOT happening, of the amounts rounded tending to cancel each other out, is about 98.4%.

To test this idea, I used Excel to randomly generate 100 series of 6 values between zero and 60, the same random numbers repeated in the adjoining column. One was expressed to one decimal place; the other was rounded to the nearest whole number by Excel. Then the software added the values in each column.

Here is a sample:

And here are the summed amounts:

 1)  106.4, 106   254.8, 254   164.5, 165   252.2, 252   222.5, 222   265.5, 266

 2)  184.6, 186   228.2, 228   130.9, 132   191.4, 192   154.4, 155   195.5, 197

 3)   26.4,   26   198.0, 198   168.7, 169   174.7, 174   187.1, 188    222.0, 223

 4)  198.0, 198   207.3, 208   166.0, 167   214.5, 215    242.6, 241  184.1, 184

 5)  206.8, 207   164.8,165    151.3, 153   239.9, 240    237.9, 238   181.3, 180

 6)  152.6, 153   207.0, 207   178.6, 179   155.9, 155   228.2, 229    131.0, 131

 7)  191.8, 192   128.8, 129   162.7, 163   221.6, 222   239.0, 239   170.1, 170

 8)  177.2, 177   238.4, 239   206.4, 206   216.5, 217   204.7, 205   146.0, 146

 9)  247.9, 248   163.8, 164   192.3, 194   191.8, 193    180.8, 181   272.0, 272

10)  194.1, 194   178.8, 179   142.6, 144   136.0, 137   228.0, 229     25.1,   25

11)  159.3, 159   255.3, 256   145.4, 146   200.8, 200   163.1, 163   144.8, 145

12)  206.4, 207   159.5, 160   194.8, 194   137.3, 138   140.6, 140     57.0,   57

13)  211.9, 212   160.6, 161   106.2, 106   211.9, 212   267.5, 268   201.5, 202

14) 180.1, 180   168.1, 169   187.7, 187   177.6, 178   173.5, 175    98.6, 100

15) 145.5, 145   179.7, 180   215.5, 217   132.6, 134   264.6, 265   175.8, 176

16) 147.0, 146   178.7, 180   147.0, 146   178.7, 180   212.7, 213   129.8, 129

17) 131.5, 132   236.4, 235   191.5, 192   204.3, 205  

Out of these 100 samples, in 86 cases the sum of the whole numbers is within one whole number of the sum of the numbers expressed to one decimal point, and in 14 cases it is within 2 whole numbers. I was expecting a few cases of larger differences, but guess that this non-occurrence is due to the admittedly small sample of only one hundred.

The conclusion is that rounding to whole numbers in a series does not lead to a great chance of the rounded amounts adding up to significant error.

A potential objection is that all of us know of families that do, in fact, have all girls or boys - I have an old friend from high school days who has had at least 6 children with a number of women, and each one was a girl. Firstly, there are an awfully large number of families. Secondly, and I suspect more importantly, there are biological factors involved in real families beyond simple probability – which is all this issue is about.

The advantages of using rounded values are twofold: simplification and a great saving in the amount of data that needs to be presented and then manipulated (about one hundredth, given two dimensional tables). These savings, in the case of the book cited, mean that all the data necessary for celestial navigation, including 4 years of almanac data, can be included in the one slim volume.

The result of this sight reduction is an intercept that is expressed to a whole minute of arc, leading to a fix consisting of latitude and longitude expressed as whole minutes of arc. I suspect there is also some wooly thinking involved in the assumption that a fix expressed to the nearest tenth of a minute of arc is better for the purpose of position finding from a small boat, and that this idea is related to confusion between the concepts of accuracy and precision. But let’s leave that issue for another day …