lkml.org 
[lkml]   [2023]   [Jan]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH 8/9] net/mlx5e: Improve remote NUMA preferences used for the IRQ affinity hints
    Date
    From: Tariq Toukan <tariqt@nvidia.com>

    In the IRQ affinity hints, replace the binary NUMA preference (local /
    remote) with the improved for_each_numa_hop_cpu() API that minds the
    actual distances, so that remote NUMAs with short distance are preferred
    over farther ones.

    This has significant performance implications when using NUMA-aware
    allocated memory (follow [1] and derivatives for example).

    [1]
    drivers/net/ethernet/mellanox/mlx5/core/en_main.c :: mlx5e_open_channel()
    int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));

    Performance tests:

    TCP multi-stream, using 16 iperf3 instances pinned to 16 cores (with aRFS on).
    Active cores: 64,65,72,73,80,81,88,89,96,97,104,105,112,113,120,121

    +-------------------------+-----------+------------------+------------------+
    | | BW (Gbps) | TX side CPU util | RX side CPU util |
    +-------------------------+-----------+------------------+------------------+
    | Baseline | 52.3 | 6.4 % | 17.9 % |
    +-------------------------+-----------+------------------+------------------+
    | Applied on TX side only | 52.6 | 5.2 % | 18.5 % |
    +-------------------------+-----------+------------------+------------------+
    | Applied on RX side only | 94.9 | 11.9 % | 27.2 % |
    +-------------------------+-----------+------------------+------------------+
    | Applied on both sides | 95.1 | 8.4 % | 27.3 % |
    +-------------------------+-----------+------------------+------------------+

    Bottleneck in RX side is released, reached linerate (~1.8x speedup).
    ~30% less cpu util on TX.

    * CPU util on active cores only.

    Setups details (similar for both sides):

    NIC: ConnectX6-DX dual port, 100 Gbps each.
    Single port used in the tests.

    $ lscpu
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 256
    On-line CPU(s) list: 0-255
    Thread(s) per core: 2
    Core(s) per socket: 64
    Socket(s): 2
    NUMA node(s): 16
    Vendor ID: AuthenticAMD
    CPU family: 25
    Model: 1
    Model name: AMD EPYC 7763 64-Core Processor
    Stepping: 1
    CPU MHz: 2594.804
    BogoMIPS: 4890.73
    Virtualization: AMD-V
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 512K
    L3 cache: 32768K
    NUMA node0 CPU(s): 0-7,128-135
    NUMA node1 CPU(s): 8-15,136-143
    NUMA node2 CPU(s): 16-23,144-151
    NUMA node3 CPU(s): 24-31,152-159
    NUMA node4 CPU(s): 32-39,160-167
    NUMA node5 CPU(s): 40-47,168-175
    NUMA node6 CPU(s): 48-55,176-183
    NUMA node7 CPU(s): 56-63,184-191
    NUMA node8 CPU(s): 64-71,192-199
    NUMA node9 CPU(s): 72-79,200-207
    NUMA node10 CPU(s): 80-87,208-215
    NUMA node11 CPU(s): 88-95,216-223
    NUMA node12 CPU(s): 96-103,224-231
    NUMA node13 CPU(s): 104-111,232-239
    NUMA node14 CPU(s): 112-119,240-247
    NUMA node15 CPU(s): 120-127,248-255
    ..

    $ numactl -H
    ..
    node distances:
    node 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
    0: 10 11 11 11 12 12 12 12 32 32 32 32 32 32 32 32
    1: 11 10 11 11 12 12 12 12 32 32 32 32 32 32 32 32
    2: 11 11 10 11 12 12 12 12 32 32 32 32 32 32 32 32
    3: 11 11 11 10 12 12 12 12 32 32 32 32 32 32 32 32
    4: 12 12 12 12 10 11 11 11 32 32 32 32 32 32 32 32
    5: 12 12 12 12 11 10 11 11 32 32 32 32 32 32 32 32
    6: 12 12 12 12 11 11 10 11 32 32 32 32 32 32 32 32
    7: 12 12 12 12 11 11 11 10 32 32 32 32 32 32 32 32
    8: 32 32 32 32 32 32 32 32 10 11 11 11 12 12 12 12
    9: 32 32 32 32 32 32 32 32 11 10 11 11 12 12 12 12
    10: 32 32 32 32 32 32 32 32 11 11 10 11 12 12 12 12
    11: 32 32 32 32 32 32 32 32 11 11 11 10 12 12 12 12
    12: 32 32 32 32 32 32 32 32 12 12 12 12 10 11 11 11
    13: 32 32 32 32 32 32 32 32 12 12 12 12 11 10 11 11
    14: 32 32 32 32 32 32 32 32 12 12 12 12 11 11 10 11
    15: 32 32 32 32 32 32 32 32 12 12 12 12 11 11 11 10

    $ cat /sys/class/net/ens5f0/device/numa_node
    14

    Affinity hints (127 IRQs):
    Before:
    331: 00000000,00000000,00000000,00000000,00010000,00000000,00000000,00000000
    332: 00000000,00000000,00000000,00000000,00020000,00000000,00000000,00000000
    333: 00000000,00000000,00000000,00000000,00040000,00000000,00000000,00000000
    334: 00000000,00000000,00000000,00000000,00080000,00000000,00000000,00000000
    335: 00000000,00000000,00000000,00000000,00100000,00000000,00000000,00000000
    336: 00000000,00000000,00000000,00000000,00200000,00000000,00000000,00000000
    337: 00000000,00000000,00000000,00000000,00400000,00000000,00000000,00000000
    338: 00000000,00000000,00000000,00000000,00800000,00000000,00000000,00000000
    339: 00010000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    340: 00020000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    341: 00040000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    342: 00080000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    343: 00100000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    344: 00200000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    345: 00400000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    346: 00800000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    347: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
    348: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
    349: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
    350: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008
    351: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000010
    352: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000020
    353: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000040
    354: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000080
    355: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000100
    356: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000200
    357: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000400
    358: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000800
    359: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00001000
    360: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00002000
    361: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00004000
    362: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00008000
    363: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00010000
    364: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00020000
    365: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00040000
    366: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00080000
    367: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00100000
    368: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00200000
    369: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00400000
    370: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00800000
    371: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,01000000
    372: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,02000000
    373: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,04000000
    374: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,08000000
    375: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,10000000
    376: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,20000000
    377: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,40000000
    378: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,80000000
    379: 00000000,00000000,00000000,00000000,00000000,00000000,00000001,00000000
    380: 00000000,00000000,00000000,00000000,00000000,00000000,00000002,00000000
    381: 00000000,00000000,00000000,00000000,00000000,00000000,00000004,00000000
    382: 00000000,00000000,00000000,00000000,00000000,00000000,00000008,00000000
    383: 00000000,00000000,00000000,00000000,00000000,00000000,00000010,00000000
    384: 00000000,00000000,00000000,00000000,00000000,00000000,00000020,00000000
    385: 00000000,00000000,00000000,00000000,00000000,00000000,00000040,00000000
    386: 00000000,00000000,00000000,00000000,00000000,00000000,00000080,00000000
    387: 00000000,00000000,00000000,00000000,00000000,00000000,00000100,00000000
    388: 00000000,00000000,00000000,00000000,00000000,00000000,00000200,00000000
    389: 00000000,00000000,00000000,00000000,00000000,00000000,00000400,00000000
    390: 00000000,00000000,00000000,00000000,00000000,00000000,00000800,00000000
    391: 00000000,00000000,00000000,00000000,00000000,00000000,00001000,00000000
    392: 00000000,00000000,00000000,00000000,00000000,00000000,00002000,00000000
    393: 00000000,00000000,00000000,00000000,00000000,00000000,00004000,00000000
    394: 00000000,00000000,00000000,00000000,00000000,00000000,00008000,00000000
    395: 00000000,00000000,00000000,00000000,00000000,00000000,00010000,00000000
    396: 00000000,00000000,00000000,00000000,00000000,00000000,00020000,00000000
    397: 00000000,00000000,00000000,00000000,00000000,00000000,00040000,00000000
    398: 00000000,00000000,00000000,00000000,00000000,00000000,00080000,00000000
    399: 00000000,00000000,00000000,00000000,00000000,00000000,00100000,00000000
    400: 00000000,00000000,00000000,00000000,00000000,00000000,00200000,00000000
    401: 00000000,00000000,00000000,00000000,00000000,00000000,00400000,00000000
    402: 00000000,00000000,00000000,00000000,00000000,00000000,00800000,00000000
    403: 00000000,00000000,00000000,00000000,00000000,00000000,01000000,00000000
    404: 00000000,00000000,00000000,00000000,00000000,00000000,02000000,00000000
    405: 00000000,00000000,00000000,00000000,00000000,00000000,04000000,00000000
    406: 00000000,00000000,00000000,00000000,00000000,00000000,08000000,00000000
    407: 00000000,00000000,00000000,00000000,00000000,00000000,10000000,00000000
    408: 00000000,00000000,00000000,00000000,00000000,00000000,20000000,00000000
    409: 00000000,00000000,00000000,00000000,00000000,00000000,40000000,00000000
    410: 00000000,00000000,00000000,00000000,00000000,00000000,80000000,00000000
    411: 00000000,00000000,00000000,00000000,00000000,00000001,00000000,00000000
    412: 00000000,00000000,00000000,00000000,00000000,00000002,00000000,00000000
    413: 00000000,00000000,00000000,00000000,00000000,00000004,00000000,00000000
    414: 00000000,00000000,00000000,00000000,00000000,00000008,00000000,00000000
    415: 00000000,00000000,00000000,00000000,00000000,00000010,00000000,00000000
    416: 00000000,00000000,00000000,00000000,00000000,00000020,00000000,00000000
    417: 00000000,00000000,00000000,00000000,00000000,00000040,00000000,00000000
    418: 00000000,00000000,00000000,00000000,00000000,00000080,00000000,00000000
    419: 00000000,00000000,00000000,00000000,00000000,00000100,00000000,00000000
    420: 00000000,00000000,00000000,00000000,00000000,00000200,00000000,00000000
    421: 00000000,00000000,00000000,00000000,00000000,00000400,00000000,00000000
    422: 00000000,00000000,00000000,00000000,00000000,00000800,00000000,00000000
    423: 00000000,00000000,00000000,00000000,00000000,00001000,00000000,00000000
    424: 00000000,00000000,00000000,00000000,00000000,00002000,00000000,00000000
    425: 00000000,00000000,00000000,00000000,00000000,00004000,00000000,00000000
    426: 00000000,00000000,00000000,00000000,00000000,00008000,00000000,00000000
    427: 00000000,00000000,00000000,00000000,00000000,00010000,00000000,00000000
    428: 00000000,00000000,00000000,00000000,00000000,00020000,00000000,00000000
    429: 00000000,00000000,00000000,00000000,00000000,00040000,00000000,00000000
    430: 00000000,00000000,00000000,00000000,00000000,00080000,00000000,00000000
    431: 00000000,00000000,00000000,00000000,00000000,00100000,00000000,00000000
    432: 00000000,00000000,00000000,00000000,00000000,00200000,00000000,00000000
    433: 00000000,00000000,00000000,00000000,00000000,00400000,00000000,00000000
    434: 00000000,00000000,00000000,00000000,00000000,00800000,00000000,00000000
    435: 00000000,00000000,00000000,00000000,00000000,01000000,00000000,00000000
    436: 00000000,00000000,00000000,00000000,00000000,02000000,00000000,00000000
    437: 00000000,00000000,00000000,00000000,00000000,04000000,00000000,00000000
    438: 00000000,00000000,00000000,00000000,00000000,08000000,00000000,00000000
    439: 00000000,00000000,00000000,00000000,00000000,10000000,00000000,00000000
    440: 00000000,00000000,00000000,00000000,00000000,20000000,00000000,00000000
    441: 00000000,00000000,00000000,00000000,00000000,40000000,00000000,00000000
    442: 00000000,00000000,00000000,00000000,00000000,80000000,00000000,00000000
    443: 00000000,00000000,00000000,00000000,00000001,00000000,00000000,00000000
    444: 00000000,00000000,00000000,00000000,00000002,00000000,00000000,00000000
    445: 00000000,00000000,00000000,00000000,00000004,00000000,00000000,00000000
    446: 00000000,00000000,00000000,00000000,00000008,00000000,00000000,00000000
    447: 00000000,00000000,00000000,00000000,00000010,00000000,00000000,00000000
    448: 00000000,00000000,00000000,00000000,00000020,00000000,00000000,00000000
    449: 00000000,00000000,00000000,00000000,00000040,00000000,00000000,00000000
    450: 00000000,00000000,00000000,00000000,00000080,00000000,00000000,00000000
    451: 00000000,00000000,00000000,00000000,00000100,00000000,00000000,00000000
    452: 00000000,00000000,00000000,00000000,00000200,00000000,00000000,00000000
    453: 00000000,00000000,00000000,00000000,00000400,00000000,00000000,00000000
    454: 00000000,00000000,00000000,00000000,00000800,00000000,00000000,00000000
    455: 00000000,00000000,00000000,00000000,00001000,00000000,00000000,00000000
    456: 00000000,00000000,00000000,00000000,00002000,00000000,00000000,00000000
    457: 00000000,00000000,00000000,00000000,00004000,00000000,00000000,00000000

    After:
    331: 00000000,00000000,00000000,00000000,00010000,00000000,00000000,00000000
    332: 00000000,00000000,00000000,00000000,00020000,00000000,00000000,00000000
    333: 00000000,00000000,00000000,00000000,00040000,00000000,00000000,00000000
    334: 00000000,00000000,00000000,00000000,00080000,00000000,00000000,00000000
    335: 00000000,00000000,00000000,00000000,00100000,00000000,00000000,00000000
    336: 00000000,00000000,00000000,00000000,00200000,00000000,00000000,00000000
    337: 00000000,00000000,00000000,00000000,00400000,00000000,00000000,00000000
    338: 00000000,00000000,00000000,00000000,00800000,00000000,00000000,00000000
    339: 00010000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    340: 00020000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    341: 00040000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    342: 00080000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    343: 00100000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    344: 00200000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    345: 00400000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    346: 00800000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    347: 00000000,00000000,00000000,00000000,00000001,00000000,00000000,00000000
    348: 00000000,00000000,00000000,00000000,00000002,00000000,00000000,00000000
    349: 00000000,00000000,00000000,00000000,00000004,00000000,00000000,00000000
    350: 00000000,00000000,00000000,00000000,00000008,00000000,00000000,00000000
    351: 00000000,00000000,00000000,00000000,00000010,00000000,00000000,00000000
    352: 00000000,00000000,00000000,00000000,00000020,00000000,00000000,00000000
    353: 00000000,00000000,00000000,00000000,00000040,00000000,00000000,00000000
    354: 00000000,00000000,00000000,00000000,00000080,00000000,00000000,00000000
    355: 00000000,00000000,00000000,00000000,00000100,00000000,00000000,00000000
    356: 00000000,00000000,00000000,00000000,00000200,00000000,00000000,00000000
    357: 00000000,00000000,00000000,00000000,00000400,00000000,00000000,00000000
    358: 00000000,00000000,00000000,00000000,00000800,00000000,00000000,00000000
    359: 00000000,00000000,00000000,00000000,00001000,00000000,00000000,00000000
    360: 00000000,00000000,00000000,00000000,00002000,00000000,00000000,00000000
    361: 00000000,00000000,00000000,00000000,00004000,00000000,00000000,00000000
    362: 00000000,00000000,00000000,00000000,00008000,00000000,00000000,00000000
    363: 00000000,00000000,00000000,00000000,01000000,00000000,00000000,00000000
    364: 00000000,00000000,00000000,00000000,02000000,00000000,00000000,00000000
    365: 00000000,00000000,00000000,00000000,04000000,00000000,00000000,00000000
    366: 00000000,00000000,00000000,00000000,08000000,00000000,00000000,00000000
    367: 00000000,00000000,00000000,00000000,10000000,00000000,00000000,00000000
    368: 00000000,00000000,00000000,00000000,20000000,00000000,00000000,00000000
    369: 00000000,00000000,00000000,00000000,40000000,00000000,00000000,00000000
    370: 00000000,00000000,00000000,00000000,80000000,00000000,00000000,00000000
    371: 00000001,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    372: 00000002,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    373: 00000004,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    374: 00000008,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    375: 00000010,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    376: 00000020,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    377: 00000040,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    378: 00000080,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    379: 00000100,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    380: 00000200,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    381: 00000400,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    382: 00000800,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    383: 00001000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    384: 00002000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    385: 00004000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    386: 00008000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    387: 01000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    388: 02000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    389: 04000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    390: 08000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    391: 10000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    392: 20000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    393: 40000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    394: 80000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
    395: 00000000,00000000,00000000,00000000,00000000,00000001,00000000,00000000
    396: 00000000,00000000,00000000,00000000,00000000,00000002,00000000,00000000
    397: 00000000,00000000,00000000,00000000,00000000,00000004,00000000,00000000
    398: 00000000,00000000,00000000,00000000,00000000,00000008,00000000,00000000
    399: 00000000,00000000,00000000,00000000,00000000,00000010,00000000,00000000
    400: 00000000,00000000,00000000,00000000,00000000,00000020,00000000,00000000
    401: 00000000,00000000,00000000,00000000,00000000,00000040,00000000,00000000
    402: 00000000,00000000,00000000,00000000,00000000,00000080,00000000,00000000
    403: 00000000,00000000,00000000,00000000,00000000,00000100,00000000,00000000
    404: 00000000,00000000,00000000,00000000,00000000,00000200,00000000,00000000
    405: 00000000,00000000,00000000,00000000,00000000,00000400,00000000,00000000
    406: 00000000,00000000,00000000,00000000,00000000,00000800,00000000,00000000
    407: 00000000,00000000,00000000,00000000,00000000,00001000,00000000,00000000
    408: 00000000,00000000,00000000,00000000,00000000,00002000,00000000,00000000
    409: 00000000,00000000,00000000,00000000,00000000,00004000,00000000,00000000
    410: 00000000,00000000,00000000,00000000,00000000,00008000,00000000,00000000
    411: 00000000,00000000,00000000,00000000,00000000,00010000,00000000,00000000
    412: 00000000,00000000,00000000,00000000,00000000,00020000,00000000,00000000
    413: 00000000,00000000,00000000,00000000,00000000,00040000,00000000,00000000
    414: 00000000,00000000,00000000,00000000,00000000,00080000,00000000,00000000
    415: 00000000,00000000,00000000,00000000,00000000,00100000,00000000,00000000
    416: 00000000,00000000,00000000,00000000,00000000,00200000,00000000,00000000
    417: 00000000,00000000,00000000,00000000,00000000,00400000,00000000,00000000
    418: 00000000,00000000,00000000,00000000,00000000,00800000,00000000,00000000
    419: 00000000,00000000,00000000,00000000,00000000,01000000,00000000,00000000
    420: 00000000,00000000,00000000,00000000,00000000,02000000,00000000,00000000
    421: 00000000,00000000,00000000,00000000,00000000,04000000,00000000,00000000
    422: 00000000,00000000,00000000,00000000,00000000,08000000,00000000,00000000
    423: 00000000,00000000,00000000,00000000,00000000,10000000,00000000,00000000
    424: 00000000,00000000,00000000,00000000,00000000,20000000,00000000,00000000
    425: 00000000,00000000,00000000,00000000,00000000,40000000,00000000,00000000
    426: 00000000,00000000,00000000,00000000,00000000,80000000,00000000,00000000
    427: 00000000,00000001,00000000,00000000,00000000,00000000,00000000,00000000
    428: 00000000,00000002,00000000,00000000,00000000,00000000,00000000,00000000
    429: 00000000,00000004,00000000,00000000,00000000,00000000,00000000,00000000
    430: 00000000,00000008,00000000,00000000,00000000,00000000,00000000,00000000
    431: 00000000,00000010,00000000,00000000,00000000,00000000,00000000,00000000
    432: 00000000,00000020,00000000,00000000,00000000,00000000,00000000,00000000
    433: 00000000,00000040,00000000,00000000,00000000,00000000,00000000,00000000
    434: 00000000,00000080,00000000,00000000,00000000,00000000,00000000,00000000
    435: 00000000,00000100,00000000,00000000,00000000,00000000,00000000,00000000
    436: 00000000,00000200,00000000,00000000,00000000,00000000,00000000,00000000
    437: 00000000,00000400,00000000,00000000,00000000,00000000,00000000,00000000
    438: 00000000,00000800,00000000,00000000,00000000,00000000,00000000,00000000
    439: 00000000,00001000,00000000,00000000,00000000,00000000,00000000,00000000
    440: 00000000,00002000,00000000,00000000,00000000,00000000,00000000,00000000
    441: 00000000,00004000,00000000,00000000,00000000,00000000,00000000,00000000
    442: 00000000,00008000,00000000,00000000,00000000,00000000,00000000,00000000
    443: 00000000,00010000,00000000,00000000,00000000,00000000,00000000,00000000
    444: 00000000,00020000,00000000,00000000,00000000,00000000,00000000,00000000
    445: 00000000,00040000,00000000,00000000,00000000,00000000,00000000,00000000
    446: 00000000,00080000,00000000,00000000,00000000,00000000,00000000,00000000
    447: 00000000,00100000,00000000,00000000,00000000,00000000,00000000,00000000
    448: 00000000,00200000,00000000,00000000,00000000,00000000,00000000,00000000
    449: 00000000,00400000,00000000,00000000,00000000,00000000,00000000,00000000
    450: 00000000,00800000,00000000,00000000,00000000,00000000,00000000,00000000
    451: 00000000,01000000,00000000,00000000,00000000,00000000,00000000,00000000
    452: 00000000,02000000,00000000,00000000,00000000,00000000,00000000,00000000
    453: 00000000,04000000,00000000,00000000,00000000,00000000,00000000,00000000
    454: 00000000,08000000,00000000,00000000,00000000,00000000,00000000,00000000
    455: 00000000,10000000,00000000,00000000,00000000,00000000,00000000,00000000
    456: 00000000,20000000,00000000,00000000,00000000,00000000,00000000,00000000
    457: 00000000,40000000,00000000,00000000,00000000,00000000,00000000,00000000

    Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
    [Tweaked API use]
    Suggested-by: Yury Norov <yury.norov@gmail.com>
    Signed-off-by: Valentin Schneider <vschneid@redhat.com>
    Reviewed-by: Yury Norov <yury.norov@gmail.com>
    Signed-off-by: Yury Norov <yury.norov@gmail.com>
    ---
    drivers/net/ethernet/mellanox/mlx5/core/eq.c | 18 ++++++++++++++++--
    1 file changed, 16 insertions(+), 2 deletions(-)

    diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
    index 8f7580fec193..c111e8657d54 100644
    --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
    +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
    @@ -817,9 +817,12 @@ static void comp_irqs_release(struct mlx5_core_dev *dev)
    static int comp_irqs_request(struct mlx5_core_dev *dev)
    {
    struct mlx5_eq_table *table = dev->priv.eq_table;
    + const struct cpumask *prev = cpu_none_mask;
    + const struct cpumask *mask;
    int ncomp_eqs = table->num_comp_eqs;
    u16 *cpus;
    int ret;
    + int cpu;
    int i;

    ncomp_eqs = table->num_comp_eqs;
    @@ -838,8 +841,19 @@ static int comp_irqs_request(struct mlx5_core_dev *dev)
    ret = -ENOMEM;
    goto free_irqs;
    }
    - for (i = 0; i < ncomp_eqs; i++)
    - cpus[i] = cpumask_local_spread(i, dev->priv.numa_node);
    +
    + i = 0;
    + rcu_read_lock();
    + for_each_numa_hop_mask(mask, dev->priv.numa_node) {
    + for_each_cpu_andnot(cpu, mask, prev) {
    + cpus[i] = cpu;
    + if (++i == ncomp_eqs)
    + goto spread_done;
    + }
    + prev = mask;
    + }
    +spread_done:
    + rcu_read_unlock();
    ret = mlx5_irqs_request_vectors(dev, cpus, ncomp_eqs, table->comp_irqs);
    kfree(cpus);
    if (ret < 0)
    --
    2.34.1
    \
     
     \ /
      Last update: 2023-03-26 23:51    [W:4.362 / U:0.380 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site