Niakwa, Inc.



What's News
NPL Products
NPL Support
NPL Downloads
System Integration
swiggle_small.gif (991 bytes)

swiggle_small.gif (991 bytes)

September 30, 1999 No. 91

NPL Division Bug

The Division Error, and the Affected RunTime Versions

A problem has recently been detected in NPL regarding division operations. Specifically, when dividing two numbers NPL may, under certain circumstances, compute an incorrect result. Niakwa has tested and confirmed the error in the following versions of the RunTime:

32-Bit Windows – Revision 5.00.09 (8/98)
SCO UNIX – Revision 5.01.03 (6/99)
AIX UNIX – All Revisions
HP – All Revisions
Sun – All Revisions
VMS – Revision 3.2 and later

The 16-bit DOS/Windows RunTimes were tested and found exempt from the bug. The bug is also absent from the Release III and Release IV SCO UNIX RunTimes. The error may potentially occur in additional, untested versions of the RunTime. The following program can be run to test for the presence of the division bug:

:10 X=4294967296
       : PRINT X/65537
       : PRINT "Correct Answer Should Be 65535.000015259"

If the bug is present, the RunTime will return an incorrect answer.

The division error also may affect the results of some NPL functions that use division internally. For example, MOD() uses division to determine the remainder, SQR(), LOG(), LGT(), and TAN() use division in their computations, and MAT INV uses division for some matrix reduction operations.

The problem is corrected in NPL Revision 5.01.07 for Windows and Revision 5.01.11 for SCO UNIX. A maintenance release is available for download from our web site (at, select NPL Downloads). The release can also be obtained by contacting Niakwa’s Technical Support department at (847) 816-7400, or via e-mail at Revisions for AIX, Sun and HP are pending.

Back to Top

When and Why the Error Occurs

The source of the problem is NPL’s optimization section for division, which speeds up the division process when a numerator divided by a denominator yields an exact result.

Internally, NPL represents numeric data as floating point numbers. Floating point numbers are stored as two values: a magnitude part called the mantissa and a scale part called the exponent. NPL combines the mantissa and exponent to represent a number using the formula: number = mantissa * 10^exponent. Both mantissa and exponent are always stored as integers by NPL.

The division error may occur when dividing X/Y when all four of the following conditions exist:

  1. The mantissa of X is 2^32 or greater.
  2. The mantissa of Y is 65536 or greater.
  3. The remainder R = MOD(INT(mant(X)/65536), mant(Y)) is 65536 or greater.
  4. The division of K = MOD(R,65536) * 65536 + MOD(mant(X),65536) by mant(Y) is exact.

The following examples illustrate the division error, and the conditions under which it could occur:

Example 1: X=4294967296 Y=65537 (R=65536, K=0)
Incorrect result:

Example 2: X=.4294967296 Y=.65537 (R=65536, K=0)
Incorrect result:

Since NPL stores numbers as a 48-bit integer mantissa and 16-bit power of 10, the internal form of an NPL number is not necessarily unique. The exponent of a number may affect how NPL stores the number internally, and the mantissa of two equal values may be different, as in the following example:


Both X1 and X2 equal 4294967296, but NPL internally represents the numbers as follows:


Substituting X1 for X in Example 1 above results in an inaccurate calculation. However, NPL computes the operation correctly when substituting X2 for X in Example 1.


blank.gif (841 bytes)

Niakwa, Basic-2C and NPL are trademarks of Niakwa, Inc. All other products mentioned are registered trademarks or trademarks of their respective companies.

Questions or problems regarding this web site should be directed to
Copyright 1996-2010 Niakwa, Inc. All rights reserved.
Last updated: Thursday January 07, 2010.