NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
GreatCircle Calculations - precision
From: Rino
Date: 2002 Sep 30, 20:57 -0400
From: Rino
Date: 2002 Sep 30, 20:57 -0400
This is my first post on this list. I happened upon it when I did a search for 'haversines' (an alien concept to me). I thought some of the posts were very insightful and entertaining, and I hope I can get an answer to some of my questions: Last weekend I decided to convert an old navigation spreadsheet application I had built in my seafaring days to a VB.NET solution. It basically became a full rewrite; first because the spreadsheet used old Excel macro's and not VBA, and second because I hadn't given a thought to spherical triangles, Mr. Neper's rules and various other laws, in over ten years. In rewriting the whole thing in .NET, I used 'double' (64-bit floating point) precision data type for all variables (Latitude, Longitude, course, etc). I did this because I use the value PI, which is an .NET constant of type 'Double', and I wanted to preserve that precision in my calculations. When testing my code, I found that my results varied slightly from the answers given in one of my old textbooks, from results returned by my old casio programmable calculator, and also from my spreadsheet. I am 99% certain that my code is accurate, I checked all intermediate results, and they all checked out. So, my question to those of you with a better grasp of math (and programming perhaps), is: why do I see these differences? Is the textbook correct, or my code? The differences are negligible in outcomes for course and distance, but they have a more significant impact on the Vertex (probably because the slight difference in course propagated through the formulas for the vertex). I will include an example here: Lat1: 34 40.0N Lon1:139 55.0E Lat2: 37 45.0N Lon2:122 50.0W GreatCircle distance: TextBook: 4473.8 VB.NET: 4473.8 GreatCircle course: TextBook: 54.5 VB.NET 54.3 Vertex: TextBook: 47 59.4N 168 36.8W VB.NEt: 48 06.0N 168 26.2W I would appreciate any insights from the 'brains' on this list. Thanks. Rino