# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Azimuth Tables and Refraction**

**From:**George Huxtable

**Date:**2003 May 5, 15:16 +0100

Thanks to George Bennett for a full reply to my posting with thread-subject "The Complete On-Board Celestial Navigator Second Edition." Referring to the two examples I quoted, showing large errors in the azimuth from the tables, he said- >(1) If the LHA is 54�, the azimuth is found from the opposite side of >the table, see >explanation p19, Step 2. George Bennett is quite right, and I apologise for "muddying the waters" in that way, stating near-Easterly azimuths when they clearly should have been near-Westerly. The instruction in the table is quite clear, and I was indeed aware of it, but carelessly failed to follow it. Sorry about that. To clarify matters, I will restate the examples, below. However, this won't alter the size of the azimuth errors that arise from using the table. >(2) The data in the examples is incomplete. Yes, I agree. I didn't include an observed altitude, which is needed to resolve the discrepancy between azimuths close to (but either side of) the East or West directions. That was because I accepted without question the procedure for resolving the ambiguity. My concern was with the angular discrepancy, not with the ambiguity. >To resolve the azimuth quadrant ambiguity, the procedure via the Prime >Vertical Altitude should be followed. Yes. To remove this question of ambiguity from the argument about angular discrepancies, I will include a value for the altitude in the restated examples below. >The tangent formula, heeding the signs of the numerator and the >denominator, does not >have this disadvantage. Bennett then provides the formula for calculating az, using its tan. Unfortunately, in translating the plain-text version, the layout became rather garbled, (perhaps because Bennett's email program and mine have been set to interpret tab characters rather differently?). I hope he won't mind if I recast his tan formula into a layout that other emailing programs should get right in plain-text, as follows- Az = arctan ((-Sin LHA) / (Cos Lat * Tan Dec - Sin Lat * Cos LHA)) With this formula, there are no regions of low precision, as there would be if calculating az from its sin or cos. However, the ordinary arc-tan function still has two possible values, so it's not yet quite clear of ambiguity. Some calculators and computer programs assist by examining the sign of numerator and denominator separately. Many Casio calculators provide the POL (polar) function, with two inputs, and when you apply X = POL (( Cos Lat * Tan Dec - Sin Lat * Cos LHA) , (- Sin LHA)) then what appears in the Y variable is a quantity between -180 and +180, unambiguously defining the azimuth. (Negative azimuths can have 360 added to put them into familiar form.) Some computers provide a similar function which is often labelled ATAN2, to distinguish it from the normal ATAN inverse tangent. The signs of the two inputs must be carefully chosen, however (as they were for the POL example above). Rectangular to polar conversions are conventionally intended to produce an angle which increases anticlockwise from the x-axis, whereas we require an angle that increases clockwise from the y axis, so a bit of thoughtful tinkering is called for. When using these methods you have to be careful with the signs: as Bennett puts it, "heeding the signs of the numerator and denominator". In the formula for Az, using arctan, above, don't try to cancel out the negative sign in the numerator by reversing the order of the subtracted terms in the denominator! I think the method of deducing az from its tan is much superior to Bennett's derivation using its sine, but I doubt whether it would lend itself to such a simple tabular solution. George Bennett added- >In the two examples chosen to highlight the shortcomings of the Azimuth >Table all three variables >are in error by 0.5� (� 1�) Well, not "in error" really, just happening to diverge from the rounded whole-number values by that amount. It could easily happen in real-life, though perhaps you wouldn't expect them all to be that far out simultaneously. To that extent, my examples are extreme cases. >and the circumstances are in the vicinity of the Prime Vertical. The ACTUAL azimuth of the body, in the examples, was all of 15deg away from the prime vertical, so in no way an "extreme" situation. >In these extreme, but possible, situations the azimuth derived from its >sine is somewhat uncertain >as will be seen from an inspection of the >Table. This was the main point, and the two Georges now seem to agree about it. >If, however, the Tables are interpolated (X=460) the azimuth is found to >be 255� or 285� (not 075� >or 105�) which compares favourably with the >results from direct calculation of 255.3� and 254.8�. I concur: but the instructions state specifically "No interpolation is required, and it is one of the simplest techniques for finding azimuth with an accuracy of one or two degrees". >The user of the book is not informed that this situation can arise. In the >examples given in the >book it is implied that all values are rounded off >to the nearest degree. I have used the tables on >innumerable occasions, >checking the results by calculator, without this problem occurring. >>Nevertheless, I accept that a note to this effect should be included. I >thank George Huxtable for >drawing my attention to this situation. Good. We agree completely. ===================== EXAMPLES RECONSIDERED.. Here are the two examples I provided to try to convince Peter Fogg of the possible magnitude of azimuth errors, but reconsidered in the light of the points made by George Bennett. EXAMPLE 1. dec = N55deg 29', to be rounded to 55. LHA = 54deg 31', to be rounded to 55. alt = 61deg 31', to be rounded to 62. From the azimuth table we get a value for x of 469, and a resulting azimuth of (268deg to 270deg inclusive) or alternatively (270 to 272 inclusive), depending on which is the correct quadrant. Within each of those brackets, there's just no way of choosing between those three possible values. To select the correct quadrant, and resolve that ambiguity, we need to recall the presumed latitude of the observer, which we used in the sight reduction tables to obtain altitude. It was, in this case, N60deg 26'. We can now go back into the azimuth table, as Bennett describes in step 4 of 7.2. This allows us to obtain a value for Y of 818, and an altitude, on the prime vertical (i.e, when due West), of 71deg. But we know that the altitude was actually 61deg 31', so, as the body was descending in the West, it must have already passed the Westerly direction. Because the observer is in the Northern hemisphere, this means that the body must now be rather North of West. These tests are summarised in the short tables that follow step 4. So we can eliminate any azimuth that's South of due West, which leaves us with a possible range of azimuths, from the table, of 270deg to 272deg inclusive. Now we have to compare that result with the azimuth, accurately calculated from the same data but without any rounding, and not by using Bennett's azimuth table. We can use the following formula (which I have corrected here, as I got it wrong in the last posting, sorry to say). az = arcsin (-cos dec sin LHA / cos alt) Or there's a better method using arctan, as described above, with no ambiguity. Taking the minutes into account these give an ACTUAL azimuth of 284deg 36'. It's noteworthy that although the body is nearly 15deg away from the prime vertical, the error introduced by using the Bennett azimuth table is as much as 14deg 24' to 12deg 24' (depending on which of the equal-valued rows is chosen to read azimuth out from). EXAMPLE 2. dec = N55deg 31', to be rounded as 56. LHA = 54deg 29', to be rounded as 54. alt = 61deg 29', to be rounded as 61. You will notice these values are almost the same as above, but round off quite differently into whole degrees. From the azimuth table, this gives a value for x of 452, and a resulting azimuth of 249deg. or 291deg. As before, we need the presumed latitude of the observer, which was 60deg 10', to help resolve the ambiguity. Then in the table a Y value of 828 results, and a prime-vertical altitude of 73deg. Once again, the azimuth to the South of due West is eliminated, by the same reasoning as before. This leaves the resulting azimuth, from the table, as 291 deg. If we calculate the ACTUAL azimuth as in example 1, we find it to be 285deg 06'. In this case the prediction from the azimuth table is in error by nearly 6 degrees. Examples 1 and 2 were chosen so that the input values were almost the same, differing only by a couple of arcminutes, and their resulting actual azimuths differed by only 30 arcminutes. Yet the azimuths obtained from the tables differed by 20 degrees! ==================== Refraction. Bennett added- >The formula that is used in the Nautical Almanac is from my 1982 paper in >the British Journal of >Navigation (Vol 35, No2) and quoted previously, is >the dominant first term of an accurate >representation of refraction, >which accords with Garfinkel's algorithm with a maximum error of >0.015� >in the altitude range of 0� to 90�. An error of less than a second of arc >should satisfy most >needs. > RM = R�M - 0.06sin (14.7 R�M + 13) > >Where R�M = cot ( h + 7.31/(h +4.4)) ============== Thank you for providing that extra term, which should satisfy even the most extreme purist! George Huxtable. ================================================================ contact George Huxtable by email at george@huxtable.u-net.com, by phone at 01865 820222 (from outside UK, +44 1865 820222), or by mail at 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK. ================================================================