kernel/linux.git/drivers/net/veth.c, branch v6.18.48

veth: fix queue index used to wake the peer txq in veth_poll

2026-08-23T12:27:08+00:00

[ Upstream commit 60db47f02bfa2aa688938aa199117ec4f8e31d23 ] veth_poll() derives the index of the peer TX queue to wake from rq->xdp_rxq.queue_index. That field is only initialized by xdp_rxq_info_reg() in veth_enable_xdp_range(), which runs only when an XDP program is attached. On the plain GRO/NAPI path (veth_napi_enable_range()) xdp_rxq_info_reg() is never called, so queue_index stays 0 for every queue, as priv->rq is zero-allocated. So in a multi-queue setup with GRO enabled and no XDP program attached, every NAPI instance looks at the peer's TX queue 0. If veth_xmit() stops peer TX queue 1 because the ptr_ring is full (NETDEV_TX_BUSY), nothing ever wakes it again: the poller draining queue 1 wakes queue 0 instead. veth implements no ndo_tx_timeout, so the netdev watchdog does not kick in either, and the queue stays stopped indefinitely. Derive the index from the position of the rq within priv->rq instead, which is correct regardless of whether XDP was ever enabled. Scripts to reproduce the stall are available at https://github.com/netoptimizer/veth-backpressure-performance-testing Fixes: dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops") Signed-off-by: Jonas Köppeler Tested-by: Jesper Dangaard Brouer Acked-by: Jesper Dangaard Brouer Link: https://patch.msgid.link/20260806-veth-fix-poll-queue-idx-v1-1-c5357fb7573d@tu-berlin.de Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin

veth: fix skb length accounting after XDP frag adjustment

2026-08-19T16:18:19+00:00

commit cb6379feaaff11c4e1e79c26c745ffa23182768a upstream. veth exposes non-linear skb fragments through an xdp_buff. If an XDP program adjusts the fragment area, veth_xdp_rcv_skb() copies xdp_frags_size back to skb->data_len but leaves skb->len containing the old fragment contribution. After a fragment shrink, this makes skb_headlen() larger than the actual linear area. In the reproduced UDP receive path, __skb_datagram_iter() copied 1024 bytes past the actual linear tail to userspace, starting at struct skb_shared_info. The copied bytes included the affected skb's nr_frags, xdp_frags_size, and a kernel pointer from skb_shinfo(skb)->frags[0]. Real packet data was displaced by the same amount and truncated at the end. Subtract the old data_len before replacing it and add the new data_len afterwards, keeping skb->len and skb->data_len synchronized. Additionally, bpf_xdp_pull_data() can advance data_end while leaving frags present. The skb is then still non-linear, so the old __skb_put(skb, off) triggers SKB_LINEAR_ASSERT(). Use skb_set_tail_pointer() and update skb->len explicitly instead, following bpf_prog_run_generic_xdp(). Unlike __skb_put(), skb_set_tail_pointer() does not require a linear skb. A 60000-byte UDP datagram on a veth pair with MTU 64000 was shortened by 1024 bytes from its fragment area. Before the fix, all 10 runs produced corrupted payloads. After the fix, all 10 runs matched the expected payload exactly. A forced-tailroom reproducer also exercises bpf_xdp_pull_data() with frags still present; the old code triggers SKB_LINEAR_ASSERT(), while this fix passes 10/10 runs. Fixes: 718a18a0c8a6 ("veth: Rework veth_xdp_rcv_skb in order to accept non-linear skb") Cc: stable@vger.kernel.org Reported-by: Mohsin Bashir Link: https://lore.kernel.org/bpf/80687d9c-9c27-494c-b3f2-efd0230b1895@gmail.com/ Suggested-by: Lorenzo Bianconi Acked-by: Lorenzo Bianconi Signed-off-by: Sun Jian Link: https://patch.msgid.link/20260804054040.613675-3-sun.jian.kdev@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Greg Kroah-Hartman

veth: convert frag_list skbs before running XDP

2026-08-09T18:25:17+00:00

commit d0d6415963040c401e7a7e4e482a698ba52448cb upstream. A frag_list skb can reach veth with data_len set but nr_frags zero. veth_convert_skb_to_xdp_buff() only converts skbs that are shared, locked, have frags[], or do not have enough headroom. It later uses skb_is_nonlinear() to decide whether to set XDP_FLAGS_HAS_FRAGS and xdp_frags_size. That exposes frag_list data to XDP as if it were stored in frags[], but frags[] is empty. AF_XDP copy mode can then trust the bogus XDP fragment metadata, walk an empty fragment entry, and crash in memcpy() from __xsk_rcv(). Route non-linear skbs through skb_pp_cow_data() before exposing them to XDP, and only advertise XDP frags when the resulting skb has frags[]. skb_copy_bits() already handles frag_list input, and skb_pp_cow_data() builds frags[] output with skb_add_rx_frag(), which is the representation XDP multi-buffer expects. Fixes: 718a18a0c8a6 ("veth: Rework veth_xdp_rcv_skb in order to accept non-linear skb") Cc: stable@vger.kernel.org Signed-off-by: Matt Fleming Reviewed-by: Toke Høiland-Jørgensen Acked-by: Lorenzo Bianconi Link: https://patch.msgid.link/20260722191925.2192070-1-matt@readmodwrite.com Signed-off-by: Jakub Kicinski Signed-off-by: Greg Kroah-Hartman

veth: fix NAPI leak in XDP enable error path

2026-07-24T14:16:38+00:00

[ Upstream commit 6739027cb72da26890edd424c77080d187b2a92e ] During XDP enablement in veth, if xdp_rxq_info_reg() or xdp_rxq_info_reg_mem_model() fails, the driver rolls back the changes. However, the rollback loop: for (i--; i >= start; i--) { decrements the loop index 'i' before the first iteration. This correctly skips unregistering the rxq for the failed index 'i' (as registration failed or was already cleaned up), but it also erroneously skips calling netif_napi_deli() for rq[i].xdp_napi. Since netif_napi_add() was already called for index 'i', this leaves a dangling napi_struct in the device's napi_list. When the veth device is later destroyed, the freed queue memory (which contains the leaked NAPI structure) can be reused. The subsequent device teardown iterates the NAPI list and corrupts the reallocated memory, leading to UAF. Fix this by explicitly deleting the NAPI association for the failed index 'i' before rolling back the successfully configured queues. Fixes: b02e5a0ebb17 ("xsk: Propagate napi_id to XDP socket Rx path") Reported-by: Guenter Roeck Signed-off-by: Eric Dumazet Cc: Björn Töpel Cc: Daniel Borkmann Cc: Ilias Apalodimas Cc: Michael S. Tsirkin Cc: Tariq Toukan Reviewed-by: Pavan Chebbi Link: https://patch.msgid.link/20260622111825.88337-1-edumazet@google.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin

veth: fix data race in veth_get_ethtool_stats

2026-01-30T09:32:12+00:00

[ Upstream commit b47adaab8b3d443868096bac08fdbb3d403194ba ] In veth_get_ethtool_stats(), some statistics protected by u64_stats_sync, are read and accumulated in ignorance of possible u64_stats_fetch_retry() events. These statistics, peer_tq_xdp_xmit and peer_tq_xdp_xmit_err, are already accumulated by veth_xdp_xmit(). Fix this by reading them into a temporary buffer first. Fixes: 5fe6e56776ba ("veth: rely on peer veth_rq for ndo_xdp_xmit accounting") Signed-off-by: David Yang Link: https://patch.msgid.link/20260114122450.227982-1-mmyangfl@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin

veth: reduce XDP no_direct return section to fix race

2025-11-21T02:44:32+00:00

As explain in commit fa349e396e48 ("veth: Fix race with AF_XDP exposing old or uninitialized descriptors") for veth there is a chance after napi_complete_done() that another CPU can manage start another NAPI instance running veth_pool(). For NAPI this is correctly handled as the napi_schedule_prep() check will prevent multiple instances from getting scheduled, but for the remaining code in veth_pool() this can run concurrent with the newly started NAPI instance. The problem/race is that xdp_clear_return_frame_no_direct() isn't designed to be nested. Prior to commit 401cb7dae813 ("net: Reference bpf_redirect_info via task_struct on PREEMPT_RT.") the temporary BPF net context bpf_redirect_info was stored per CPU, where this wasn't an issue. Since this commit the BPF context is stored in 'current' task_struct. When running veth in threaded-NAPI mode, then the kthread becomes the storage area. Now a race exists between two concurrent veth_pool() function calls one exiting NAPI and one running new NAPI, both using the same BPF net context. Race is when another CPU gets within the xdp_set_return_frame_no_direct() section before exiting veth_pool() calls the clear-function xdp_clear_return_frame_no_direct(). Fixes: 401cb7dae8130 ("net: Reference bpf_redirect_info via task_struct on PREEMPT_RT.") Signed-off-by: Jesper Dangaard Brouer Link: https://patch.msgid.link/176356963888.337072.4805242001928705046.stgit@firesoul Signed-off-by: Jakub Kicinski

veth: more robust handing of race to avoid txq getting stuck

2025-11-15T02:16:53+00:00

Commit dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops") introduced a race condition that can lead to a permanently stalled TXQ. This was observed in production on ARM64 systems (Ampere Altra Max). The race occurs in veth_xmit(). The producer observes a full ptr_ring and stops the queue (netif_tx_stop_queue()). The subsequent conditional logic, intended to re-wake the queue if the consumer had just emptied it (if (__ptr_ring_empty(...)) netif_tx_wake_queue()), can fail. This leads to a "lost wakeup" where the TXQ remains stopped (QUEUE_STATE_DRV_XOFF) and traffic halts. This failure is caused by an incorrect use of the __ptr_ring_empty() API from the producer side. As noted in kernel comments, this check is not guaranteed to be correct if a consumer is operating on another CPU. The empty test is based on ptr_ring->consumer_head, making it reliable only for the consumer. Using this check from the producer side is fundamentally racy. This patch fixes the race by adopting the more robust logic from an earlier version V4 of the patchset, which always flushed the peer: (1) In veth_xmit(), the racy conditional wake-up logic and its memory barrier are removed. Instead, after stopping the queue, we unconditionally call __veth_xdp_flush(rq). This guarantees that the NAPI consumer is scheduled, making it solely responsible for re-waking the TXQ. This handles the race where veth_poll() consumes all packets and completes NAPI *before* veth_xmit() on the producer side has called netif_tx_stop_queue. The __veth_xdp_flush(rq) will observe rx_notify_masked is false and schedule NAPI. (2) On the consumer side, the logic for waking the peer TXQ is moved out of veth_xdp_rcv() and placed at the end of the veth_poll() function. This placement is part of fixing the race, as the netif_tx_queue_stopped() check must occur after rx_notify_masked is potentially set to false during NAPI completion. This handles the race where veth_poll() consumes all packets, but haven't finished (rx_notify_masked is still true). The producer veth_xmit() stops the TXQ and __veth_xdp_flush(rq) will observe rx_notify_masked is true, meaning not starting NAPI. Then veth_poll() change rx_notify_masked to false and stops NAPI. Before exiting veth_poll() will observe TXQ is stopped and wake it up. Fixes: dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops") Reviewed-by: Toshiaki Makita Signed-off-by: Jesper Dangaard Brouer Link: https://patch.msgid.link/176295323282.307447.14790015927673763094.stgit@firesoul Signed-off-by: Jakub Kicinski

veth: prevent NULL pointer dereference in veth_xdp_rcv

2025-06-12T15:08:32+00:00

The veth peer device is RCU protected, but when the peer device gets deleted (veth_dellink) then the pointer is assigned NULL (via RCU_INIT_POINTER). This patch adds a necessary NULL check in veth_xdp_rcv when accessing the veth peer net_device. This fixes a bug introduced in commit dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops"). The bug is a race and only triggers when having inflight packets on a veth that is being deleted. Reported-by: Ihor Solodrai Closes: https://lore.kernel.org/all/fecfcad0-7a16-42b8-bff2-66ee83a6e5c4@linux.dev/ Reported-by: syzbot+c4c7bf27f6b0c4bd97fe@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/683da55e.a00a0220.d8eae.0052.GAE@google.com/ Fixes: dc82a33297fc ("veth: apply qdisc backpressure on full ptr_ring to reduce TX drops") Signed-off-by: Jesper Dangaard Brouer Acked-by: Ihor Solodrai Link: https://patch.msgid.link/174964557873.519608.10855046105237280978.stgit@firesoul Signed-off-by: Jakub Kicinski

veth: apply qdisc backpressure on full ptr_ring to reduce TX drops

2025-04-28T21:06:58+00:00

In production, we're seeing TX drops on veth devices when the ptr_ring fills up. This can occur when NAPI mode is enabled, though it's relatively rare. However, with threaded NAPI - which we use in production - the drops become significantly more frequent. The underlying issue is that with threaded NAPI, the consumer often runs on a different CPU than the producer. This increases the likelihood of the ring filling up before the consumer gets scheduled, especially under load, leading to drops in veth_xmit() (ndo_start_xmit()). This patch introduces backpressure by returning NETDEV_TX_BUSY when the ring is full, signaling the qdisc layer to requeue the packet. The txq (netdev queue) is stopped in this condition and restarted once veth_poll() drains entries from the ring, ensuring coordination between NAPI and qdisc. Backpressure is only enabled when a qdisc is attached. Without a qdisc, the driver retains its original behavior - dropping packets immediately when the ring is full. This avoids unexpected behavior changes in setups without a configured qdisc. With a qdisc in place (e.g. fq, sfq) this allows Active Queue Management (AQM) to fairly schedule packets across flows and reduce collateral damage from elephant flows. A known limitation of this approach is that the full ring sits in front of the qdisc layer, effectively forming a FIFO buffer that introduces base latency. While AQM still improves fairness and mitigates flow dominance, the latency impact is measurable. In hardware drivers, this issue is typically addressed using BQL (Byte Queue Limits), which tracks in-flight bytes needed based on physical link rate. However, for virtual drivers like veth, there is no fixed bandwidth constraint - the bottleneck is CPU availability and the scheduler's ability to run the NAPI thread. It is unclear how effective BQL would be in this context. This patch serves as a first step toward addressing TX drops. Future work may explore adapting a BQL-like mechanism to better suit virtual devices like veth. Reported-by: Yan Zhai Signed-off-by: Jesper Dangaard Brouer Reviewed-by: Toshiaki Makita Link: https://patch.msgid.link/174559294022.827981.1282809941662942189.stgit@firesoul Signed-off-by: Jakub Kicinski

net: move misc netdev_lock flavors to a separate header

2025-03-08T17:06:50+00:00

Move the more esoteric helpers for netdev instance lock to a dedicated header. This avoids growing netdevice.h to infinity and makes rebuilding the kernel much faster (after touching the header with the helpers). The main netdev_lock() / netdev_unlock() functions are used in static inlines in netdevice.h and will probably be used most commonly, so keep them in netdevice.h. Acked-by: Stanislav Fomichev Link: https://patch.msgid.link/20250307183006.2312761-1-kuba@kernel.org Signed-off-by: Jakub Kicinski