Messages in this thread | | | From | Andy Shevchenko <> | Date | Sun, 9 Jan 2022 15:02:01 +0200 | Subject | Re: [PATCH v12 09/16] iio: afe: rescale: fix accuracy for small fractional scales |
| |
On Sat, Jan 8, 2022 at 10:53 PM Liam Beguin <liambeguin@gmail.com> wrote: > > The approximation caused by integer divisions can be costly on smaller > scale values since the decimal part is significant compared to the > integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such > cases to maintain accuracy.
...
> + tmp = 1 << *val2;
Unfortunately, potential UB is still here. I think you missed a subtle detail in the implementation of BIT()/BIT_ULL(). Let's put it here:
#define BIT(nr) (UL(1) << (nr))
where
#define UL(x) (_UL(x)) #define _UL(x) (_AC(x, UL))
For non-assembler case
#define __AC(X,Y) (X##Y) #define _AC(X,Y) __AC(X,Y)
Now you may easily see the difference.
...
> + rem2 = *val % (int)tmp; > + *val = *val / (int)tmp; > + > + *val2 = rem / (int)tmp;
Hmm... You divide s64 by 10^9, which means that the maximum value can be ~10^10 / 2 (because 2^64-1 ~= 10^19), but this _might_ be bigger than 'int' can hold. Can you confirm that tmp can't be so big?
-- With Best Regards, Andy Shevchenko
| |