[Octave-bug-tracker] [bug #35827] [octave forge] (statistics) hygepdf and hygecdf functions are numerically unstable for large populations

[Nicholas Jankowski]

Update of bug #35827 (project octave):

                  Status:               Confirmed => Patch Submitted        

    _______________________________________________________

Follow-up Comment #7:

following the comment #2 and #3 recommendations, attached is a patch to
hygepdf that replaces the pdf(k) lines with the direct calls to gammaln vs
calling bincoeff.  as hygecdf just calls hygepdf, i think this takes care of
both?

modification passes all tests after I added eps tolerances to the BISTs.  also
noticed it's 3-5x faster.

I just used the comment #2 formula directly (and modified for the array case).
Didn't check to see if there are any algebraic simplifications available
there, but none were obvious.

I didn't see any other corner cases in the comments to test, so those with
more intimate knowledge of the function may want to ensure this entirely fixes
the bug.

(file #52393)
    _______________________________________________________

Additional Item Attachment:

File name: octave-statistics-hygepdf-bug35827.diff Size:3 KB
   
<https://file.savannah.gnu.org/file/octave-statistics-hygepdf-bug35827.diff?file_id=52393>



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?35827>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/