2 Replies Latest reply on Jul 18, 2008 10:57 PM by sgratton

    Negating doubles: possible error in lushaders.h

    sgratton

      Hi there,

      I've been playing with the double precision facilities in IL and testing out how the _neg() operator affects them.

      The IL doc suggests _neg() is different for ints and floats but doesn't say what it does on doubles. Usage in lushaders.h (e.g. dmad r1.zw, r0.zw_neg(xyzw), r1.xy, r1.zw) seems to indicate that _neg(xyzw) will negate the double in xy and the one in zw. However, if _neg() acts on a register containing two doubles just as it might on one containing four floats you think you might only need _neg(yw) to flip the sign bits.

      On testing it seems that either work for dadd but not for dmul or dmad: only _neg(yw) consistently works. If you use _neg(xyzw) for dmul or dmad you will get almost the right answer but not quite: one of the bits of the mantissa, corresponding to the high bit of x or z, of the operand will be wrong.

      So it seems LUShaders and code that similarly negates doubles will suffer from a loss of precision.

      Will _neg(yw) remain the correct way to negate doubles or should _neg(xyzw) work for dmul and dmad etc.?

      Thanks,
      Steven.
        • Negating doubles: possible error in lushaders.h
          MicahVillmow
          Steven,
          Your testing is correct in that doing _neg on a double produces incorrect results for dmul and dmad but not dadd. As you have found out, the only correct way to negate is by negating the y and w components. Negating all components of a double should produce incorrect results because you are flipping a bit in the mantissa. Lu_decomposition is being removed from the SDK for now until we can clean up its code and produce better documentation for it. I'll make sure that this gets corrected before lu_decomposition is replaced in the SDK.