lkml.org 
[lkml]   [2022]   [Jan]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [PATCH v12 09/16] iio: afe: rescale: fix accuracy for small fractional scales
    On Sat, Jan 8, 2022 at 10:53 PM Liam Beguin <liambeguin@gmail.com> wrote:
    >
    > The approximation caused by integer divisions can be costly on smaller
    > scale values since the decimal part is significant compared to the
    > integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such
    > cases to maintain accuracy.

    ...

    > + tmp = 1 << *val2;

    Unfortunately, potential UB is still here. I think you missed a subtle
    detail in the implementation of BIT()/BIT_ULL(). Let's put it here:

    #define BIT(nr) (UL(1) << (nr))

    where

    #define UL(x) (_UL(x))
    #define _UL(x) (_AC(x, UL))

    For non-assembler case

    #define __AC(X,Y) (X##Y)
    #define _AC(X,Y) __AC(X,Y)

    Now you may easily see the difference.

    ...

    > + rem2 = *val % (int)tmp;
    > + *val = *val / (int)tmp;
    > +
    > + *val2 = rem / (int)tmp;

    Hmm... You divide s64 by 10^9, which means that the maximum value can
    be ~10^10 / 2 (because 2^64-1 ~= 10^19), but this _might_ be bigger
    than 'int' can hold. Can you confirm that tmp can't be so big?

    --
    With Best Regards,
    Andy Shevchenko

    \
     
     \ /
      Last update: 2022-01-09 14:02    [W:4.233 / U:0.152 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site