kernel/linux.git/net/sunrpc, branch v6.1.178

xprtrdma: Return sendctx slot after Send preparation failure

2026-07-24T13:54:47+00:00

[ Upstream commit 60e7870052f417d83965db144f70ae21fcfcf37f ] rpcrdma_prepare_send_sges() gets a sendctx before it maps the SGEs for the Send WR. If one of the mapping helpers fails, no Send WR is posted, so no Send completion is guaranteed to advance rb_sc_tail. Current cleanup clears sc_req so a later completion can sweep over that slot, but a consecutive run of preparation failures can still advance rb_sc_head until the ring appears full. At that point rpcrdma_sendctx_get_locked() returns NULL and no Send can be posted to produce the completion needed to recover the ring. The trigger requires CONFIG_SUNRPC_XPRT_RDMA and an NFS/RDMA mount. Mount setup and reliable DMA-map fault injection require local admin authority. Unprivileged I/O on an existing mount can exercise the send path, but a remote peer alone cannot force this local DMA-map failure. Add rpcrdma_sendctx_unget_locked() for the single-consumer send path to rewind rb_sc_head when the just-acquired sendctx is canceled before ib_post_send(). Wake waiters after making the slot available again. After the rewind, every slot the completion sweep visits belongs to a posted Send, so rpcrdma_sendctx_put_locked() no longer needs to test sc_req before unmapping. Fixes: ae72950abf99 ("xprtrdma: Add data structure to manage RDMA Send arguments") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Repost Receive buffers for malformed replies

2026-07-24T13:54:47+00:00

[ Upstream commit abc011ddaf1617e3e82d8a1e87daa7ddbfb9bac5 ] rpcrdma_wc_receive() decrements the transport's Receive count for every completion before it dispatches a successful Receive to rpcrdma_reply_handler(). The handler must post a replacement Receive WR before returning unless ownership of the rep has moved elsewhere, as on the backchannel path. Commit 2ae50ad68cd7 ("xprtrdma: Close window between waking RPC senders and posting Receives") moved the Receive refill out of rpcrdma_wc_receive(), where it had run ahead of every reply, into rpcrdma_reply_handler() so that the responder's credit grant could be parsed before reposting. The bad-version and short-reply exits never reach that refill: they recycle the rep and return without calling rpcrdma_post_recvs(). A remote peer can therefore drain the client's posted Receive queue by sending a sustained stream of replies that are shorter than the fixed transport header or that carry an unrecognized RPC/RDMA version. Each such reply consumes one posted Receive without replacing it. Once the queue empties, the peer's next Send finds no posted Receive and the transport stalls until reconnect. Route both malformed-reply exits through the shared repost tail after recycling the rep, refilling against buf->rb_credits, the most recent accepted credit grant. Neither exit updates the congestion window, so RPCs admitted under the previous grant remain in flight awaiting replies. A smaller refill target would let a stream of malformed replies ratchet the posted Receive count down to the batch floor while the congestion window still admits rb_credits RPCs; a burst of valid replies to those RPCs could then overrun the posted Receives, and because the client connects with rnr_retry_count of zero, a single RNR NAK terminates the connection. Refilling against rb_credits also restores the target that applied to malformed replies before commit 2ae50ad68cd7 ("xprtrdma: Close window between waking RPC senders and posting Receives") when rpcrdma_post_recvs() computed it from rb_credits internally. rb_credits is at least one from connection establishment onward, so the repost path always keeps Receives posted. Fixes: 2ae50ad68cd7 ("xprtrdma: Close window between waking RPC senders and posting Receives") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Sanitize the reply credit grant after parsing

2026-07-24T13:54:47+00:00

[ Upstream commit c3a628aab2dc8f5fd7bff86ceaeae64de590e60a ] The out_norqst exit in rpcrdma_reply_handler() branches away before the credit clamp, so a reply that matches no pending request reaches out_post carrying the raw credit value parsed from the wire. rpcrdma_post_recvs() does not bound its @needed argument: the refill loop allocates and chains Receive WRs until the count is satisfied or allocation fails. A peer that sends a well-formed reply carrying an unknown XID and an inflated credit grant therefore drives rep allocation and Receive posting past re_max_requests on every such reply. Move the clamp to immediately after the credit field is parsed, ahead of the first branch that can reach out_post, so every later consumer sees a sanitized value. The cwnd update stays on the matched-request path. Fixes: 704f3f640f72 ("xprtrdma: Post receive buffers after RPC completion") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Fix bcall rep leak and unbounded peek

2026-07-24T13:54:47+00:00

[ Upstream commit c7653d5cebc8492c77ec0415b5e9c0fb3e644bc6 ] rpcrdma_is_bcall() decodes a reply's first words to decide whether the frame is a backchannel call. Two issues in that decode path let a short or malformed reply leak the receive buffer and drain the Receive queue. First, the speculative peek p = xdr_inline_decode(xdr, 0); /* five p++ reads follow */ asks xdr_inline_decode() for zero bytes, which returns xdr->p without consulting xdr->end. The five subsequent __be32 reads can then walk up to 20 bytes past the wire payload into stale regbuf contents and misclassify the reply as a backchannel call. Second, after the post-peek p = xdr_inline_decode(xdr, 3 * sizeof(*p)); if (unlikely(!p)) return true; the short-header arm returns true without calling rpcrdma_bc_receive_call(). The contract with the caller is that a true return transfers ownership of rep to the backchannel path: rpcrdma_reply_handler() if (rpcrdma_is_bcall(r_xprt, rep)) return; /* bare return, skips out_post */ ... out_post: rpcrdma_post_recvs(r_xprt, credits + ...); Because rpcrdma_bc_receive_call() never ran, no one took rep, but rpcrdma_reply_handler still bare-returns past rpcrdma_rep_put() and rpcrdma_post_recvs(). The rep, with its persistently DMA-mapped receive buffer, is orphaned on rb_all_reps and freed only at transport teardown. This completion reposts nothing, so its slot is reclaimed only when a later forward-channel reply reaches out_post and rpcrdma_post_recvs() allocates a fresh rep to backfill; absent that traffic the Receive queue drains and the peer's Sends draw RNR NAKs. Fix by consulting xdr->end after the zero-length peek so the five __be32 reads cannot run unless 20 bytes of wire payload remain. A byte-precise comparison against xdr->end is required because a non-4-aligned receive rounds the stream's word count up past the true payload. Also return false from the short-header arm so the reply falls through the normal out_norqst cleanup chain (rpcrdma_rep_put() plus rpcrdma_post_recvs()). Fixes: 41c8f70f5a3d ("xprtrdma: Harden backchannel call decoding") Assisted-by: kres:claude-opus-4-7 Signed-off-by: Chris Mason Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Resize reply buffers before reposting receives

2026-07-24T13:54:47+00:00

[ Upstream commit 234c0ff695ef3ffb656931000e6b823d0c2f30fd ] Commit 0e13dd9ea8be ("xprtrdma: Remove temp allocation of rpcrdma_rep objects") made rpcrdma_rep objects survive disconnects. That is normally fine, but it also means their receive regbufs keep the size they had when they were first allocated. Each rep's receive buffer is sized to ep->re_inline_recv when the rep is created. rpcrdma_ep_create() resets that threshold to the rdma_max_inline_read ceiling for every new endpoint, and the connect handshake then shrinks it to the peer's advertised inline send size. A rep allocated under a smaller negotiated threshold keeps that size: on disconnect, rpcrdma_xprt_disconnect() drains and DMA-unmaps the surviving reps but does not free or resize them. The threshold can come back larger on the next connection. The first peer may supply no RPC-over-RDMA CM private data, defaulting its send size to 1024, while the reconnect target is an ordinary server offering 4096; or, with rdma_max_inline_read raised above its default, the reconnect target may advertise a larger svcrdma_max_req_size than the first. rpcrdma_post_recvs() then reposts a surviving rep whose SGE length is still the old, smaller value, and a larger inline Reply hits a receive length error and forces another disconnect. The undersized rep returns to the free list when its failed Receive flushes, so the following reconnect reposts the same rep and fails the same way. The transport flaps without making forward progress for as long as the peer keeps advertising the larger inline size. This is local/admin-triggerable rather than remote-triggerable: a local administrator must create and maintain the NFS/RDMA mount, while the server or reconnect target has to advertise a larger inline send size and return a reply that uses it. Fix this by checking each rep before it is reposted. If the receive regbuf is smaller than the current endpoint's inline receive size, reallocate it on the current RDMA device's NUMA node and reinitialize the rep's xdr_buf before DMA-mapping and posting the Receive WR. Fixes: 0e13dd9ea8be ("xprtrdma: Remove temp allocation of rpcrdma_rep objects") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Document and assert reply-handler invariants

2026-07-24T13:54:47+00:00

[ Upstream commit 797943e8bd1ffcc63bfe79d24faad9a77054ec40 ] The xprtrdma reply path has been the subject of recurring LLM-driven review claims that 'an RPC can complete while receive buffers are still DMA-mapped' or that 'the req can be freed while the HCA still owns the send buffer.' No runtime reproducer has surfaced, but the absence of a written-down invariant set lets each pass of automated review reach the same hypothetical conclusion. Subsequent fixes against ce2f9a4d9ccc ('xprtrdma: Decouple req recycling from RPC completion') closed the underlying races but did not document the closure where future readers will look for it. State the invariants explicitly in a comment above rpcrdma_reply_handler() and back four of them with WARN_ON_ONCE() probes positioned where each invariant is locally checkable on the previous patch's cleaned-up ownership state: - I1 (Receive WR ownership): WARN at rpcrdma_post_recvs() that a rep pulled from rb_free_reps carries rr_rqst == NULL. - I2 (rep attachment): WARN at rpcrdma_reply_put() that req->rl_reply was NULLed before the matching rep_put. - I3 (Registered-MR fence): WARN at rpcrdma_complete_rqst() that req->rl_registered is empty. Strong send-queue ordering of the LocalInv WR chain makes the last completion observe the ib_dma_unmap_sg() of every earlier MR, so 'list empty' implies 'all MRs unmapped'. - I4 (Send-buffer release): WARN at rpcrdma_req_release() that req->rl_sendctx is NULL. Reaching the kref release callback requires both the RPC-layer and Send-side references to have dropped; the Send-side drop runs in rpcrdma_sendctx_unmap(), which clears rl_sendctx (previous patch). A non-NULL rl_sendctx here would mean the Send-side owner had not run -- a contradiction. The XXX comment in xprt_rdma_free() about signal-driven release racing the Send completion described the pre-decouple state. Replace it with a one-line note pointing at the invariant set, since the kref scheme now holds the req across the in-flight Send regardless of which path released the rpc_task. I5 (req lifecycle) is stated in the comment but not probed: making it locally assertible would require moving kref_init out of rpcrdma_req_release(), which in turn requires adding kref_init to the bc_pa_list and backlog-wake reuse paths. That restructuring is deferred -- the invariant is unchanged either way. Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Stable-dep-of: 234c0ff695ef ("xprtrdma: Resize reply buffers before reposting receives") Signed-off-by: Sasha Levin

xprtrdma: Check frwr_wp_create() during connect

2026-07-24T13:54:47+00:00

[ Upstream commit 0f13fc7c7d2e0427517e63c739277a4cd338b0c5 ] frwr_wp_create() creates the singleton Memory Region used to encode padding for Write chunks whose payload length is not XDR-aligned. Its failure paths return a negative errno and leave ep->re_write_pad_mr set to NULL. rpcrdma_xprt_connect() currently ignores that return value. If frwr_wp_create() fails after the rest of the connection setup succeeds, xprt_rdma_connect_worker() treats the connection attempt as successful and sets XPRT_CONNECTED. A later NFS/RDMA read with a non-4-byte-aligned receive page length reaches rpcrdma_encode_write_list(), passes the NULL write-pad MR to encode_rdma_segment(), and dereferences it. This is locally triggerable on an NFS/RDMA client after a connect or reconnect hits a local MR allocation, DMA-map, MR-map, or post-send failure; a remote peer alone cannot force the local MR setup failure. Check the return value and fail the connect as -ENOTCONN, matching the adjacent setup failures. This keeps XPRT_CONNECTED clear and lets the normal reconnect path retry. Fixes: 21037b8c2258 ("xprtrdma: Provide a buffer to pad Write chunks of unaligned length") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Decouple req recycling from RPC completion

2026-07-24T13:54:46+00:00

[ Upstream commit e786233d2e0bbff9a82e43f02ae3a46ab4b08ec3 ] rl_kref formerly served two distinct lifetimes through a single refcount: it gated when a Reply could wake its RPC task, and it gated when an rpcrdma_req could return to its free pool. The marshal path took the Send-side reference only when SGEs needed DMA-unmap (sc_unmap_count > 0), which made a Send carrying only pre-registered buffers an exception: the Reply handler dropped rl_kref from 1 to 0 and freed the req while the HCA might still be DMA-reading from its send buffer. Give rl_kref a narrower job. The RPC layer takes one reference when slot allocation hands a req out. rpcrdma_prepare_send_sges() takes a Send-side reference unconditionally after WR preparation succeeds. xprt_rdma_free_slot() and xprt_rdma_bc_free_rqst() drop the RPC-layer reference; rpcrdma_sendctx_unmap() drops the Send-side reference. The req returns to its free pool only after both owners have signed off. The existing kref_init(&req->rl_kref) call in rpcrdma_prepare_send_sges() is removed. Initialization moves to the slot-allocation paths (xprt_rdma_alloc_slot and rpcrdma_bc_rqst_get), and the release callback re-arms rl_kref before the req returns to a free pool. A re-init in the marshal path would discard the RPC-layer reference that already exists on entry. Three invariants follow: - Any rpcrdma_req held by an rpc_rqst has rl_kref >= 1. xprt_rdma_alloc_slot(), rpcrdma_bc_rqst_get(), and the backlog-wake branch in xprt_rdma_alloc_slot() each kref_init rl_kref before publishing the req. Without this invariant, an RPC task that aborts between slot allocation and marshal (gss_refresh failure or signal during call_connect, for example) would drive xprt_release() -> xprt_rdma_free_slot() -> kref_put against a refcount of zero, saturating refcount_t and stranding the slot. - The Send-side reference is taken only after WR prep succeeds. A mapping failure in rpcrdma_prepare_send_sges() runs rpcrdma_sendctx_cancel(), which DMA-unmaps the sendctx and clears sc_req without touching rl_kref. The sendctx ring walks in rpcrdma_sendctx_put_locked() and rpcrdma_sendctxs_destroy() skip entries with sc_req == NULL, so a burst of -EIO marshal failures cannot hold reqs off rb_send_bufs. - The release callback re-arms rl_kref so the next consumer enters with the invariant satisfied. Replies now complete the RPC directly. rpcrdma_reply_handler() calls rpcrdma_complete_rqst() in place of kref_put on the non-LocalInv branch. The LocalInv branch already completes the RPC from frwr_unmap_async() and is unaffected. Because Send-side references can now outlive RPC completion, connection teardown drains sendctx entries whose unsignaled Sends never had a later signaled completion to walk the ring. rpcrdma_sendctxs_destroy() walks the active range and runs rpcrdma_sendctx_unmap() on each entry with a non-NULL sc_req before the request buffers are reset, and is moved ahead of rpcrdma_reqs_reset() in rpcrdma_xprt_disconnect() so the reqs are still in their pre-reset state when the Send-side refs are released. The drain creates a teardown-ordering hazard on the backchannel path. With the new lifetime, releasing a bc_prealloc req from rpcrdma_req_release() re-adds it to bc_pa_list. The disconnect in xprt_rdma_destroy() runs after xprt_destroy_backchannel() has already emptied bc_pa_list, so the drained reqs would otherwise leak. xprt_rdma_destroy() now runs xprt_rdma_bc_destroy(xprt, 0) a second time after the disconnect to reclaim them. Fixes: 0ab115237025 ("xprtrdma: Wake RPCs directly in rpcrdma_wc_send path") Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Signed-off-by: Sasha Levin

xprtrdma: Use sendctx DMA state for Send signaling

2026-07-24T13:54:46+00:00

[ Upstream commit 2797ae7c929610fb2d2303a996a08173fa096730 ] Send signaling matters only when the prepared Send has page mappings to unmap. Today that test is expressed indirectly with rl_kref, because the Send-side reference is taken only for Sends with mapped SGEs. Split the SGE DMA unmap loop into its own helper and use sc_unmap_count directly for the signaling decision. This keeps the current behavior but removes one dependency on the old rl_kref semantics before the request lifetime rules are changed. Signed-off-by: Chuck Lever Signed-off-by: Anna Schumaker Stable-dep-of: e786233d2e0b ("xprtrdma: Decouple req recycling from RPC completion") Signed-off-by: Sasha Levin

xprtrdma: Post receive buffers after RPC completion

2026-07-24T13:54:46+00:00

[ Upstream commit 704f3f640f72db4d44ec5ce3db8d4e150c974bc7 ] rpcrdma_post_recvs() runs in CQ poll context and its cost falls on the latency-critical path between polling a Receive completion and waking the RPC consumer. Every cycle spent refilling the Receive Queue delays delivery of the reply to the NFS layer. Move the rpcrdma_post_recvs() call in rpcrdma_reply_handler() to after the RPC has been decoded and completed. The larger batch size from the preceding patch provides sufficient Receive Queue headroom to absorb the brief delay before buffers are replenished. Signed-off-by: Chuck Lever Signed-off-by: Trond Myklebust Stable-dep-of: e786233d2e0b ("xprtrdma: Decouple req recycling from RPC completion") Signed-off-by: Sasha Levin