|September 30, 1999 No. 91|
NPL Division Bug
The Division Error, and the Affected RunTime Versions
A problem has recently been detected in NPL regarding division operations. Specifically, when dividing two numbers NPL may, under certain circumstances, compute an incorrect result. Niakwa has tested and confirmed the error in the following versions of the RunTime:
32-Bit Windows Revision 5.00.09 (8/98)
SCO UNIX Revision 5.01.03 (6/99)
AIX UNIX All Revisions
HP All Revisions
Sun All Revisions
VMS Revision 3.2 and later
The 16-bit DOS/Windows RunTimes were tested and found exempt from the bug. The bug is also absent from the Release III and Release IV SCO UNIX RunTimes. The error may potentially occur in additional, untested versions of the RunTime. The following program can be run to test for the presence of the division bug:
: PRINT X/65537
: PRINT "Correct Answer Should Be 65535.000015259"
If the bug is present, the RunTime will return an incorrect answer.
The division error also may affect the results of some NPL functions that use division internally. For example, MOD() uses division to determine the remainder, SQR(), LOG(), LGT(), and TAN() use division in their computations, and MAT INV uses division for some matrix reduction operations.
The problem is corrected in NPL Revision 5.01.07 for Windows and Revision 5.01.11 for SCO UNIX. A maintenance release is available for download from our web site (at www.niakwa.com, select NPL Downloads). The release can also be obtained by contacting Niakwas Technical Support department at (847) 816-7400, or via e-mail at email@example.com. Revisions for AIX, Sun and HP are pending.
Back to Top
When and Why the Error Occurs
The source of the problem is NPLs optimization section for division, which speeds up the division process when a numerator divided by a denominator yields an exact result.
Internally, NPL represents numeric data as floating point numbers. Floating point numbers are stored as two values: a magnitude part called the mantissa and a scale part called the exponent. NPL combines the mantissa and exponent to represent a number using the formula: number = mantissa * 10^exponent. Both mantissa and exponent are always stored as integers by NPL.
The division error may occur when dividing X/Y when all four of the following conditions exist:
- The mantissa of X is 2^32 or greater.
- The mantissa of Y is 65536 or greater.
- The remainder R = MOD(INT(mant(X)/65536), mant(Y)) is 65536 or greater.
- The division of K = MOD(R,65536) * 65536 + MOD(mant(X),65536) by mant(Y) is exact.
The following examples illustrate the division error, and the conditions under which it could occur:
Example 1: X=4294967296 Y=65537 (R=65536, K=0)
Incorrect result: X/Y=0
Example 2: X=.4294967296 Y=.65537 (R=65536, K=0)
Incorrect result: X/Y=0
Since NPL stores numbers as a 48-bit integer mantissa and 16-bit power of 10, the internal form of an NPL number is not necessarily unique. The exponent of a number may affect how NPL stores the number internally, and the mantissa of two equal values may be different, as in the following example:
Both X1 and X2 equal 4294967296, but NPL internally represents the numbers as follows:
Substituting X1 for X in Example 1 above results in an inaccurate calculation. However, NPL computes the operation correctly when substituting X2 for X in Example 1.