NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Peter Fogg
Date: 2005 Jan 8, 20:49 +1100
This issue came up on this List, a little while ago, in a discussion of
the book ‘The Complete On-Board Celestial Navigator’ that uses
values rounded to the nearest minute of arc for the sight reduction process. Someone
outside the list brought up the same objection recently, which started me
thinking about the matter again.
With minutes of arc of, for example, anywhere between 23.0 and 23.5, 23
is adopted. Between 23.5 and 24.0, 24 is adopted. If a value rounded up is
followed by a value rounded down, a canceling out effect occurs. On the other
hand, if a rounded up value is consistently followed by rounded up values (or
down by down) then the rounded amounts can add up to error. If 15.4 is followed
by 57.3 then followed by 19.4, all rounded down, then an error of a little over
one minute of arc has been introduced. If 15.4 is followed by 57.8 then
followed by 19.9, then the rounded amounts have tended to cancel each other
out.
The objection raised was that the errors introduced by this rounding
process are potentially accumulative when a series of rounded values are used;
as during the sight reduction process.
I propose to use an example to show why this is not so, why the rounded
values tend to cancel each other out and do not tend to accumulate. Let’s
call the values rounded up ‘boys’ and the values rounded down
‘girls’ – for want of better names …
Couples setting out to make a family have a 50/50 chance of having a
boy or girl (actually this is not strictly correct, for biological reasons
which need not concern us here). Thus a probability of 0.5 of either,
expressed as a decimal. The chance of having a girl as a second child, GIVEN
THAT the first was a girl, is 0.5 x 0.5 = 0.25. Thus the chance of NOT having
two girls in a row is 75%. By extension, the chance of not having three girls
in a row (or three boys) is 87.5%.
There are 5 or 6 values (6 for stars, 5 for other bodies) entered
during the sight reduction process used in George Bennett’s book. So the
chance of all of them conspiring towards error (all boys or all girls) is about
1.6%. The chance of this NOT happening, of the amounts rounded tending to
cancel each other out, is about 98.4%.
To test this idea, I used Excel to randomly generate 100 series of 6
values between zero and 60, the same random numbers repeated in the adjoining
column. One was expressed to one decimal place; the other was rounded to the
nearest whole number by Excel. Then the software added the values in each
column.
Here is a sample:
55.2 |
55 |
43.1 |
43 |
22.9 |
23 |
19.9 |
20 |
34.3 |
34 |
3.8 |
4 |
179.2 |
179 |
And here are the summed amounts:
1) 106.4, 106 254.8, 254 164.5,
165 252.2, 252 222.5, 222 265.5, 266
2) 184.6, 186 228.2, 228 130.9, 132
191.4, 192 154.4, 155 195.5, 197
3) 26.4, 26 198.0,
198 168.7, 169 174.7, 174 187.1, 188 222.0,
223
4) 198.0, 198 207.3, 208 166.0,
167 214.5, 215 242.6, 241 184.1, 184
5) 206.8, 207 164.8,165
151.3, 153 239.9, 240 237.9, 238 181.3,
180
6) 152.6, 153 207.0, 207 178.6,
179 155.9, 155 228.2, 229 131.0, 131
7) 191.8, 192 128.8, 129 162.7,
163 221.6, 222 239.0, 239 170.1, 170
8) 177.2, 177 238.4, 239 206.4,
206 216.5, 217 204.7, 205 146.0, 146
9) 247.9, 248 163.8, 164 192.3, 194
191.8, 193 180.8, 181 272.0, 272
10) 194.1, 194 178.8, 179 142.6,
144 136.0, 137 228.0, 229
25.1, 25
11) 159.3, 159 255.3, 256 145.4,
146 200.8, 200 163.1, 163 144.8, 145
12) 206.4, 207 159.5, 160 194.8,
194 137.3, 138 140.6, 140 57.0,
57
13) 211.9, 212 160.6, 161 106.2,
106 211.9, 212 267.5, 268 201.5, 202
14) 180.1, 180 168.1, 169 187.7,
187 177.6, 178 173.5, 175 98.6, 100
15) 145.5, 145 179.7, 180 215.5,
217 132.6, 134 264.6, 265 175.8, 176
16) 147.0, 146 178.7, 180 147.0,
146 178.7, 180 212.7, 213 129.8, 129
17) 131.5, 132 236.4, 235 191.5,
192 204.3, 205
Out of these 100 samples, in 86 cases the sum of the whole numbers is
within one whole number of the sum of the numbers expressed to one decimal point,
and in 14 cases it is within 2 whole numbers. I was expecting a few cases of
larger differences, but guess that this non-occurrence is due to the admittedly
small sample of only one hundred.
The conclusion is that rounding to whole numbers in a series does not
lead to a great chance of the rounded amounts adding up to significant error.
A potential objection is that all of us know of families that do, in
fact, have all girls or boys - I have an old friend from high school days who
has had at least 6 children with a number of women, and each one was a girl.
Firstly, there are an awfully large number of families. Secondly, and I suspect
more importantly, there are biological factors involved in real families beyond
simple probability – which is all this issue is about.
The advantages of using rounded values are twofold: simplification and
a great saving in the amount of data that needs to be presented and then
manipulated (about one hundredth, given two dimensional tables). These savings,
in the case of the book cited, mean that all the data necessary for celestial
navigation, including 4 years of almanac data, can be included in the one slim
volume.
The result of this sight reduction is an intercept that is expressed to
a whole minute of arc, leading to a fix consisting of latitude and longitude
expressed as whole minutes of arc. I suspect there is also some wooly thinking
involved in the assumption that a fix expressed to the nearest tenth of a
minute of arc is better for the purpose of position finding from a small boat,
and that this idea is related to confusion between the concepts of accuracy and
precision. But let’s leave that issue for another day …