lkml.org 
[lkml]   [2021]   [Sep]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH AUTOSEL 5.4 29/37] RDMA/core/sa_query: Retry SA queries
    Date
    From: Håkon Bugge <haakon.bugge@oracle.com>

    [ Upstream commit 5f5a650999d5718af766fc70a120230b04235a6f ]

    A MAD packet is sent as an unreliable datagram (UD). SA requests are sent
    as MAD packets. As such, SA requests or responses may be silently dropped.

    IB Core's MAD layer has a timeout and retry mechanism, which amongst
    other, is used by RDMA CM. But it is not used by SA queries. The lack of
    retries of SA queries leads to long specified timeout, and error being
    returned in case of packet loss. The ULP or user-land process has to
    perform the retry.

    Fix this by taking advantage of the MAD layer's retry mechanism.

    First, a check against a zero timeout is added in rdma_resolve_route(). In
    send_mad(), we set the MAD layer timeout to one tenth of the specified
    timeout and the number of retries to 10. The special case when timeout is
    less than 10 is handled.

    With this fix:

    # ucmatose -c 1000 -S 1024 -C 1

    runs stable on an Infiniband fabric. Without this fix, we see an
    intermittent behavior and it errors out with:

    cmatose: event: RDMA_CM_EVENT_ROUTE_ERROR, error: -110

    (110 is ETIMEDOUT)

    Link: https://lore.kernel.org/r/1628784755-28316-1-git-send-email-haakon.bugge@oracle.com
    Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
    Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
    Signed-off-by: Sasha Levin <sashal@kernel.org>
    ---
    drivers/infiniband/core/cma.c | 3 +++
    drivers/infiniband/core/sa_query.c | 9 ++++++++-
    2 files changed, 11 insertions(+), 1 deletion(-)

    diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
    index ec9e9598894f..261d284e5ff3 100644
    --- a/drivers/infiniband/core/cma.c
    +++ b/drivers/infiniband/core/cma.c
    @@ -2937,6 +2937,9 @@ int rdma_resolve_route(struct rdma_cm_id *id, unsigned long timeout_ms)
    struct rdma_id_private *id_priv;
    int ret;

    + if (!timeout_ms)
    + return -EINVAL;
    +
    id_priv = container_of(id, struct rdma_id_private, id);
    if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_RESOLVED, RDMA_CM_ROUTE_QUERY))
    return -EINVAL;
    diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
    index d2d70c89193f..8c0ff50cbcfc 100644
    --- a/drivers/infiniband/core/sa_query.c
    +++ b/drivers/infiniband/core/sa_query.c
    @@ -1360,6 +1360,7 @@ static int send_mad(struct ib_sa_query *query, unsigned long timeout_ms,
    {
    unsigned long flags;
    int ret, id;
    + const int nmbr_sa_query_retries = 10;

    xa_lock_irqsave(&queries, flags);
    ret = __xa_alloc(&queries, &id, query, xa_limit_32b, gfp_mask);
    @@ -1367,7 +1368,13 @@ static int send_mad(struct ib_sa_query *query, unsigned long timeout_ms,
    if (ret < 0)
    return ret;

    - query->mad_buf->timeout_ms = timeout_ms;
    + query->mad_buf->timeout_ms = timeout_ms / nmbr_sa_query_retries;
    + query->mad_buf->retries = nmbr_sa_query_retries;
    + if (!query->mad_buf->timeout_ms) {
    + /* Special case, very small timeout_ms */
    + query->mad_buf->timeout_ms = 1;
    + query->mad_buf->retries = timeout_ms;
    + }
    query->mad_buf->context[0] = query;
    query->id = id;

    --
    2.30.2
    \
     
     \ /
      Last update: 2021-09-10 03:07    [W:6.292 / U:0.020 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site