# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Message:αβγ
Message:abc
 Add Images & Files Posting Code: Name: Email:
GreatCircle Calculations - precision
From: Rino
Date: 2002 Sep 30, 20:57 -0400

```This is my first post on this list.  I happened upon it when I did a search
for 'haversines' (an alien concept to me).  I thought some of the posts were
very insightful and entertaining, and I hope I can get an answer to some of
my questions:

I had built in my seafaring days to a VB.NET solution.
It basically became a full rewrite; first because the spreadsheet used old
Excel macro's and not VBA, and second because I hadn't given a thought to
spherical triangles, Mr. Neper's rules and various other laws, in over ten
years.

In rewriting the whole thing in .NET, I used 'double' (64-bit floating
point) precision data type for all variables (Latitude, Longitude, course,
etc).  I did this because I use the value PI, which is an .NET constant of
type 'Double', and I wanted to preserve that precision in my calculations.

When testing my code, I found that my results varied slightly from the
answers given in one of my old textbooks, from results returned by my old
casio programmable calculator, and also from my spreadsheet.  I am 99%
certain that my code is accurate, I checked all intermediate results, and
they all checked out.

So, my question to those of you with a better grasp of math (and programming
perhaps), is:  why do I see these differences? Is the textbook correct, or
my code?  The differences are negligible in outcomes for course and
distance, but they have a more significant impact on the Vertex (probably
because the slight difference in course propagated through the formulas for
the vertex).

I will include an example here:

Lat1: 34 40.0N  Lon1:139 55.0E
Lat2: 37 45.0N  Lon2:122 50.0W

GreatCircle distance:
TextBook:       4473.8
VB.NET: 4473.8

GreatCircle course:
TextBook:       54.5
VB.NET  54.3

Vertex:
TextBook:       47 59.4N        168 36.8W
VB.NEt: 48 06.0N        168 26.2W

I would appreciate any insights from the 'brains' on this list.

Thanks.

Rino

```
Browse Files

Drop Files

### Join NavList

 Name: (please, no nicknames or handles) Email:
 Do you want to receive all group messages by email? Yes No
You can also join by posting. Your first on-topic post automatically makes you a member.

### Posting Code

Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.
 Email:

### Email Settings

 Posting Code:

### Custom Index

 Subject: Author: Start date: (yyyymm dd) End date: (yyyymm dd)