# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Sensitivity of Mercator sailing to round off error in course**

**From:**Sam Chan

**Date:**2001 Apr 30, 5:47 PM

George is correct in pointing out my typo. The equation for distance should be as George pointed out D = DLat/Cos C. My observation regarding the sensitivity of the Mercator sailing was only directed towards non- East/West courses. When we are sailing 090 or 270 or there about, Parallel sailing must be used. Also, rounding of the course is not an issue with calculator programs or computer programs. My comments are more directed at those of us who want to use tables and basic calculators to do the calculations. For parallel sailing, the traditional equation for departure is p = DLo Cos Lat. The way I remember whether it is Sin or Cos is that as lat increases, the departure for a given DLo gets smaller. The Cos function starts at 1 for Lat equal 0 and decreases to 0 at lat equal 90. Now to account for the eccentricity of the earth. The equation for p can be modified to p = DLo k Cos Lat where k is a factor dependent on Lat and eccentricity of the earth (radius of the earth as a function of Lat). k = sqrt((1-e^2)/(1-e^2 (cos Lat)^2) ). The factor k is very close to 1 so the impact on p is negligible (less then 0.4% max). Sam Chan ----- Original Message ----- From "George Huxtable"To: Sent: Monday, April 30, 2001 3:06 PM Subject: Re: Sensitivity of Mercator sailing to round off error in course > Sam Chan wrote, on 27 April- > > > >I was looking at the distance calculations from various list members > >responding to Silicon Sea Leg 70 and was puzzled by the differences in the > >same calculations. I looked into the equation used for calculating > >distance and determined that it is very sensitive to round off errors in > >the course. > > > >Given a departure and destination point, the equations used to determine > >course and distance for Mercator sailing is: > > > >C = arctan (Dlo/DMP), where C is course, Dlo is difference in Longitude in > >minutes, and DMP is difference in meridional parts. > > > >D = Dlat cos C, where Dlat is difference in latitude in minutes. > > > >Typically in writing the course C, it is rounded to 1 decimal place. If > >this rounded value of C is used in the calculation D, the error can be > >significant if C is near east or west. The rounding is not an issue if the > >calculation is being done by a computer program where the full precision > >of C is maintained. > > > >The differential of D is dD = D tan C dC. For typical rounding of C to 1 > >decimal place, the error dC is ~+/- 0.05. The resulting error in D, dD, is > >highly dependent on C. If C is near east or west, tan C is very large. > >Thus explains the differences in the distances from the various list > >members for SS 70. > > > >As an alternate approach for those of us who like to use pencil, paper, > >and tables to do the calculations, we can first calculate the departure > >before we calculate D. But we need a calculation of departure not > >explicitly dependent on course. From Roy Williams, "Geometry of > >Navigation", we have p = Dlo (Dlat/DMP) where p is the departure. Then > >distance is then: > > > >D = sqrt(Dlat^2 + p^2) > > > >The differential error in this format is dD = (Dlat d(Dlat) + p dp)/D. The > >error is significantly insensitive to errors in Dlat and p. > > > >Sam Chan > ===================== > > I have some comments to make. > > First, there seems to be a slip, when Sam writes "D = Dlat cos C , where > Dlat is difference in latitude in minutes.". > Shouldn't this be D = Dlat / cos C ? > > Second, Sam hasn't entirely resolved the problems that occur when the > course is exactly 90 or 270 degrees, that is, sailing due East or West > along a rhumb line which is a parallel of latitude. > > In that case the difference in lat between the starting point (Lat 0, Long > 0) and the end point (Lat 1, Long 1) is zero because Lat 1 and Lat 0 are > the same.And so the meridional parts of the two positions will also be the > same. Therefore DMP becomes zero, and an infinity arises when dividing dlo > by DMP, to get the course using arctan. The computer should be prevented > from attempting this division. Instead one can trap the computer when DMP = > 0 by putting course = 90 or 270 degrees depending on the sign of dlo. > > Sam then suggests we get the departure p from- > p = Dlo (Dlat/DMP) (Dlo and Dlat have to be expressed in mimutes) > > Trouble with this is when sailing along a parallel then both Dlat and DMP > are zero, so dividing one by the other gives an indeterminate answer for p, > which can produce crazy results. > > Instead, when the trap for DMP=0 has been activated, the departure, which > in these circumstances is also the distance between the start and finish > points, is simply Dlo * Sin lat in miles (again assuming Dlo is in > minutes). This is the value that a spherical Earth would give, anyway. I'm > not competent to compute the correction for a standard Earth ellipsoid, but > perhaps another reader is. > > George Huxtable. > > ------------------------------ > > george@huxtable.u-net.com > George Huxtable, 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK. > Tel. 01865 820222 or (int.) +44 1865 820222. > ------------------------------ >