kernel/linux.git/fs/eventpoll.c, branch v6.1.178

eventpoll: fix ep_remove struct eventpoll / struct file UAF

2026-07-24T13:54:18+00:00

[ Upstream commit a6dc643c69311677c574a0f17a3f4d66a5f3744b ] ep_remove() (via ep_remove_file()) cleared file->f_ep under file->f_lock but then kept using @file inside the critical section (is_file_epoll(), hlist_del_rcu() through the head, spin_unlock). A concurrent __fput() taking the eventpoll_release() fastpath in that window observed the transient NULL, skipped eventpoll_release_file() and ran to f_op->release / file_free(). For the epoll-watches-epoll case, f_op->release is ep_eventpoll_release() -> ep_clear_and_put() -> ep_free(), which kfree()s the watched struct eventpoll. Its embedded ->refs hlist_head is exactly where epi->fllink.pprev points, so the subsequent hlist_del_rcu()'s "*pprev = next" scribbles into freed kmalloc-192 memory. In addition, struct file is SLAB_TYPESAFE_BY_RCU, so the slot backing @file could be recycled by alloc_empty_file() -- reinitializing f_lock and f_ep -- while ep_remove() is still nominally inside that lock. The upshot is an attacker-controllable kmem_cache_free() against the wrong slab cache. Pin @file via epi_fget() at the top of ep_remove() and gate the critical section on the pin succeeding. With the pin held @file cannot reach refcount zero, which holds __fput() off and transitively keeps the watched struct eventpoll alive across the hlist_del_rcu() and the f_lock use, closing both UAFs. If the pin fails @file has already reached refcount zero and its __fput() is in flight. Because we bailed before clearing f_ep, that path takes the eventpoll_release() slow path into eventpoll_release_file() and blocks on ep->mtx until the waiter side's ep_clear_and_put() drops it. The bailed epi's share of ep->refcount stays intact, so the trailing ep_refcount_dec_and_test() in ep_clear_and_put() cannot free the eventpoll out from under eventpoll_release_file(); the orphaned epi is then cleaned up there. A successful pin also proves we are not racing eventpoll_release_file() on this epi, so drop the now-redundant re-check of epi->dying under f_lock. The cheap lockless READ_ONCE(epi->dying) fast-path bailout stays. Fixes: 58c9b016e128 ("epoll: use refcount to reduce ep_mutex contention") Reported-by: Jaeyoung Chung Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-6-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) (cherry picked from commit a6dc643c69311677c574a0f17a3f4d66a5f3744b) Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: move epi_fget() up

2026-07-24T13:54:18+00:00

[ Upstream commit 86e87059e6d1fd5115a31949726450ed03c1073b ] We'll need it when removing files so move it up. No functional change. Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-5-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") [file_ref_get(&file->f_ref) from original commit left as atomic_long_inc_not_zero(&file->f_count) due to v6.12.y missing commit 90ee6ed776c0 ("fs: port files to file_ref") and its dependent commit 08ef26ea9ab3 ("fs: add file_ref")] Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: rename ep_remove_safe() back to ep_remove()

2026-07-24T13:54:18+00:00

[ Upstream commit 0bade234723e40e4937be912e105785d6a51464e ] The current name is just confusing and doesn't clarify anything. Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-4-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: drop vestigial __ prefix from ep_remove_{file,epi}()

2026-07-24T13:54:18+00:00

[ Upstream commit 0feaf644f7180c4a91b6b405a881afbfd958f1cf ] With __ep_remove() gone, the double-underscore on __ep_remove_file() and __ep_remove_epi() no longer contrasts with a __-less parent and just reads as noise. Rename both to ep_remove_file() and ep_remove_epi(). No functional change. Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: kill __ep_remove()

2026-07-24T13:54:18+00:00

[ Upstream commit e9e5cd40d7c403e19f21d0f7b8b8ba3a76b58330 ] Remove the boolean conditional in __ep_remove() and restructure the code so the check for racing with eventpoll_release_file() are only done in the ep_remove_safe() path where they belong. Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-3-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: split __ep_remove()

2026-07-24T13:54:18+00:00

[ Upstream commit 0f7bdfd413000985de09fc39eb9efa1e091a3ce0 ] Split __ep_remove() to delineate file removal from epoll item removal. Suggested-by: Linus Torvalds Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-2-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: use hlist_is_singular_node() in __ep_remove()

2026-07-24T13:54:18+00:00

[ Upstream commit 3d9fd0abc94d8cd430cc7cd7d37ce5e5aae2cd2b ] Replace the open-coded "epi is the only entry in file->f_ep" check with hlist_is_singular_node(). Same semantics, and the helper avoids the head-cacheline access in the common false case. Link: https://patch.msgid.link/20260423-work-epoll-uaf-v1-1-2470f9eec0f5@kernel.org Signed-off-by: Christian Brauner (Amutable) Stable-dep-of: a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") Signed-off-by: Quentin Schulz Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: don't decrement ep refcount while still holding the ep mutex

2026-07-24T13:54:17+00:00

commit 8c2e52ebbe885c7eeaabd3b7ddcdc1246fc400d2 upstream. Jann Horn points out that epoll is decrementing the ep refcount and then doing a mutex_unlock(&ep->mtx); afterwards. That's very wrong, because it can lead to a use-after-free. That pattern is actually fine for the very last reference, because the code in question will delay the actual call to "ep_free(ep)" until after it has unlocked the mutex. But it's wrong for the much subtler "next to last" case when somebody *else* may also be dropping their reference and free the ep while we're still using the mutex. Note that this is true even if that other user is also using the same ep mutex: mutexes, unlike spinlocks, can not be used for object ownership, even if they guarantee mutual exclusion. A mutex "unlock" operation is not atomic, and as one user is still accessing the mutex as part of unlocking it, another user can come in and get the now released mutex and free the data structure while the first user is still cleaning up. See our mutex documentation in Documentation/locking/mutex-design.rst, in particular the section [1] about semantics: "mutex_unlock() may access the mutex structure even after it has internally released the lock already - so it's not safe for another context to acquire the mutex and assume that the mutex_unlock() context is not using the structure anymore" So if we drop our ep ref before the mutex unlock, but we weren't the last one, we may then unlock the mutex, another user comes in, drops _their_ reference and releases the 'ep' as it now has no users - all while the mutex_unlock() is still accessing it. Fix this by simply moving the ep refcount dropping to outside the mutex: the refcount itself is atomic, and doesn't need mutex protection (that's the whole _point_ of refcounts: unlike mutexes, they are inherently about object lifetimes). Reported-by: Jann Horn Link: https://docs.kernel.org/locking/mutex-design.html#semantics [1] Cc: Alexander Viro Cc: Christian Brauner Cc: Jan Kara Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

eventpoll: defer struct eventpoll free to RCU grace period

2026-06-01T15:38:34+00:00

[ Upstream commit 07712db80857d5d09ae08f3df85a708ecfc3b61f ] In certain situations, ep_free() in eventpoll.c will kfree the epi->ep eventpoll struct while it still being used by another concurrent thread. Defer the kfree() to an RCU callback to prevent UAF. Fixes: f2e467a48287 ("eventpoll: Fix semi-unbounded recursion") Signed-off-by: Nicholas Carlini Signed-off-by: Christian Brauner Signed-off-by: Sasha Levin

epoll: use refcount to reduce ep_mutex contention

2026-06-01T15:38:33+00:00

[ Upstream commit 58c9b016e12855286370dfb704c08498edbc857a ] We are observing huge contention on the epmutex during an http connection/rate test: 83.17% 0.25% nginx [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe [...] |--66.96%--__fput |--60.04%--eventpoll_release_file |--58.41%--__mutex_lock.isra.6 |--56.56%--osq_lock The application is multi-threaded, creates a new epoll entry for each incoming connection, and does not delete it before the connection shutdown - that is, before the connection's fd close(). Many different threads compete frequently for the epmutex lock, affecting the overall performance. To reduce the contention this patch introduces explicit reference counting for the eventpoll struct. Each registered event acquires a reference, and references are released at ep_remove() time. The eventpoll struct is released by whoever - among EP file close() and and the monitored file close() drops its last reference. Additionally, this introduces a new 'dying' flag to prevent races between the EP file close() and the monitored file close(). ep_eventpoll_release() marks, under f_lock spinlock, each epitem as dying before removing it, while EP file close() does not touch dying epitems. The above is needed as both close operations could run concurrently and drop the EP reference acquired via the epitem entry. Without the above flag, the monitored file close() could reach the EP struct via the epitem list while the epitem is still listed and then try to put it after its disposal. An alternative could be avoiding touching the references acquired via the epitems at EP file close() time, but that could leave the EP struct alive for potentially unlimited time after EP file close(), with nasty side effects. With all the above in place, we can drop the epmutex usage at disposal time. Overall this produces a significant performance improvement in the mentioned connection/rate scenario: the mutex operations disappear from the topmost offenders in the perf report, and the measured connections/rate grows by ~60%. To make the change more readable this additionally renames ep_free() to ep_clear_and_put(), and moves the actual memory cleanup in a separate ep_free() helper. Link: https://lkml.kernel.org/r/4a57788dcaf28f5eb4f8dfddcc3a8b172a7357bb.1679504153.git.pabeni@redhat.com Signed-off-by: Paolo Abeni Co-developed-by: Eric Dumazet Signed-off-by: Eric Dumazet Tested-by: Xiumei Mu Acked-by: Soheil Hassas Yeganeh Reviewed-by: Davidlohr Bueso Cc: Alexander Viro Cc: Carlos Maiolino Cc: Christian Brauner Cc: Eric Biggers Cc: Jacob Keller Cc: Jens Axboe Signed-off-by: Andrew Morton Stable-dep-of: 07712db80857 ("eventpoll: defer struct eventpoll free to RCU grace period") Signed-off-by: Sasha Levin