VB Forum / General 2 / May 2004

Evaluate (2.55 * 10 + .5 ) - 26 in your debugger to see the reason why
it returns 25 instead of 26.

However
   x = (2.55 * 10 + .5)
   debug.print int(x)

produces 26.

Floating point can be such fun. :)

Yes.

The answer to the mystery is the "floating point representation"
problem. the decimal value 2.55 cannot exactly fit into a binary number.
As it turns out, its binary representation is ever so slightly less then
2.55 . We can see this by doing the following in order to unmask the
"guard" digit that is masking this fact (the guard digit is that 16
decimal place of a Double that VB doesn't display; it is used to force
rounding of values like 2.55 instead of converting to the nearest
16-decimal place representation and displaying it). In the Immediate
window, perform this calculation

    Print 2.55 - 2.5

The answer turns out to be 4.99999999999998E-02 and not 0.05. So, the
decimal representation of 2.55 is something 0.00499999999999998 with the
16-decimal digit, the "guard" digit, hidden from us. So, the Int
function is returning the correct value because to it, the numbers you
gave it did not calculate to 26; but, rather, to something ever so
slightly less that that.

While not specifically written for VB, the following summary from the
MSDN help files (it was in a section for C++ and Fortran) is every bit
applicable.

Summary
=========
There are many situations in which precision, rounding,
and accuracy in floating-point calculations can work to
generate results that are surprising to the programmer.
There are four general rules that should be followed:

1) In a calculation involving both single and double
   precision, the result will not usually be any more
   accurate than single precision. If double precision
   is required, be certain all terms in the calculation,
   including constants, are specified in double precision.

2) Never assume that a simple numeric value is
   accurately represented in the computer. Most
   floating-point values can't be precisely represented
   as a finite binary value. For example .1 is
   .0001100110011... in binary (it repeats forever), so
   it can't be represented with complete accuracy on
   a computer using binary arithmetic, which includes
   all PCs.

3) Never assume that the result is accurate to the
   last decimal place. There are always small
   differences between the "true" answer and what
   can be calculated with the finite precision of any
   floating point processing unit.

4) Never compare two floating-point values to see if
   they are equal or not- equal. This is a corollary to
   rule 3. There are almost always going to be small
   differences between numbers that "should" be
   equal. Instead, always check to see if the numbers
   are nearly equal. In other words, check to see if the
   difference between them is very small or insignificant.

Rick - MVP