sgratton

Negating doubles: possible error in lushaders.h

Discussion created by sgratton on Jul 18, 2008
Latest reply on Jul 18, 2008 by sgratton

Hi there,

I've been playing with the double precision facilities in IL and testing out how the _neg() operator affects them.

The IL doc suggests _neg() is different for ints and floats but doesn't say what it does on doubles. Usage in lushaders.h (e.g. dmad r1.zw, r0.zw_neg(xyzw), r1.xy, r1.zw) seems to indicate that _neg(xyzw) will negate the double in xy and the one in zw. However, if _neg() acts on a register containing two doubles just as it might on one containing four floats you think you might only need _neg(yw) to flip the sign bits.

On testing it seems that either work for dadd but not for dmul or dmad: only _neg(yw) consistently works. If you use _neg(xyzw) for dmul or dmad you will get almost the right answer but not quite: one of the bits of the mantissa, corresponding to the high bit of x or z, of the operand will be wrong.

So it seems LUShaders and code that similarly negates doubles will suffer from a loss of precision.

Will _neg(yw) remain the correct way to negate doubles or should _neg(xyzw) work for dmul and dmad etc.?

Thanks,
Steven.

Outcomes