Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    GreatCircle Calculations - precision
    From: Rino
    Date: 2002 Sep 30, 20:57 -0400

    This is my first post on this list.  I happened upon it when I did a search
    for 'haversines' (an alien concept to me).  I thought some of the posts were
    very insightful and entertaining, and I hope I can get an answer to some of
    my questions:
    Last weekend I decided to convert an old navigation spreadsheet application
    I had built in my seafaring days to a VB.NET solution.
    It basically became a full rewrite; first because the spreadsheet used old
    Excel macro's and not VBA, and second because I hadn't given a thought to
    spherical triangles, Mr. Neper's rules and various other laws, in over ten
    In rewriting the whole thing in .NET, I used 'double' (64-bit floating
    point) precision data type for all variables (Latitude, Longitude, course,
    etc).  I did this because I use the value PI, which is an .NET constant of
    type 'Double', and I wanted to preserve that precision in my calculations.
    When testing my code, I found that my results varied slightly from the
    answers given in one of my old textbooks, from results returned by my old
    casio programmable calculator, and also from my spreadsheet.  I am 99%
    certain that my code is accurate, I checked all intermediate results, and
    they all checked out.
    So, my question to those of you with a better grasp of math (and programming
    perhaps), is:  why do I see these differences? Is the textbook correct, or
    my code?  The differences are negligible in outcomes for course and
    distance, but they have a more significant impact on the Vertex (probably
    because the slight difference in course propagated through the formulas for
    the vertex).
    I will include an example here:
    Lat1: 34 40.0N  Lon1:139 55.0E
    Lat2: 37 45.0N  Lon2:122 50.0W
    GreatCircle distance:
    TextBook:       4473.8
    VB.NET: 4473.8
    GreatCircle course:
    TextBook:       54.5
    VB.NET  54.3
    TextBook:       47 59.4N        168 36.8W
    VB.NEt: 48 06.0N        168 26.2W
    I would appreciate any insights from the 'brains' on this list.

    Browse Files

    Drop Files


    What is NavList?

    Join NavList

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    You can also join by posting. Your first on-topic post automatically makes you a member.

    Posting Code

    Enter the email address associated with your NavList messages. Your posting code will be emailed to you immediately.

    Email Settings

    Posting Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site