kernel/linux.git/io_uring/io_uring.h, branch v6.6.132

io_uring: include dying ring in task_work "should cancel" state

2025-09-25T09:00:09+00:00

Commit 3539b1467e94336d5854ebf976d9627bfb65d6c3 upstream. When running task_work for an exiting task, rather than perform the issue retry attempt, the task_work is canceled. However, this isn't done for a ring that has been closed. This can lead to requests being successfully completed post the ring being closed, which is somewhat confusing and surprising to an application. Rather than just check the task exit state, also include the ring ref state in deciding whether or not to terminate a given request when run from task_work. Cc: stable@vger.kernel.org # 6.1+ Link: https://github.com/axboe/liburing/discussions/1459 Reported-by: Benedek Thaler Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: backport io_should_terminate_tw()

2025-09-25T09:00:09+00:00

Parts of commit b6f58a3f4aa8dba424356c7a69388a81f4459300 upstream. Backport io_should_terminate_tw() helper to judge whether task_work should be run or terminated. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: add io_file_can_poll() helper

2025-06-19T13:28:44+00:00

Commit 95041b93e90a06bb613ec4bef9cd4d61570f68e4 upstream. This adds a flag to avoid dipping dereferencing file and then f_op to figure out if the file has a poll handler defined or not. We generally call this at least twice for networked workloads, and if using ring provided buffers, we do it on every buffer selection. Particularly the latter is troublesome, as it's otherwise a very fast operation. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring/kbuf: use vm_insert_pages() for mmap'ed pbuf ring

2025-03-22T19:50:45+00:00

Commit 87585b05757dc70545efb434669708d276125559 upstream. Rather than use remap_pfn_range() for this and manually free later, switch to using vm_insert_page() and have it Just Work. This requires a bit of effort on the mmap lookup side, as the ctx uring_lock isn't held, which otherwise protects buffer_lists from being torn down, and it's not safe to grab from mmap context that would introduce an ABBA deadlock between the mmap lock and the ctx uring_lock. Instead, lookup the buffer_list under RCU, as the the list is RCU freed already. Use the existing reference count to determine whether it's possible to safely grab a reference to it (eg if it's not zero already), and drop that reference when done with the mapping. If the mmap reference is the last one, the buffer_list and the associated memory can go away, since the vma insertion has references to the inserted pages at that point. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: get rid of remap_pfn_range() for mapping rings/sqes

2025-03-22T19:50:44+00:00

Commit 3ab1db3c6039e02a9deb9d5091d28d559917a645 upstream. Rather than use remap_pfn_range() for this and manually free later, switch to using vm_insert_pages() and have it Just Work. If possible, allocate a single compound page that covers the range that is needed. If that works, then we can just use page_address() on that page. If we fail to get a compound page, allocate single pages and use vmap() to map them into the kernel virtual address space. This just covers the rings/sqes, the other remaining user of the mmap remap_pfn_range() user will be converted separately. Once that is done, we can kill the old alloc/free code. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring/rw: avoid punting to io-wq directly

2024-12-27T12:58:57+00:00

Commit 6e6b8c62120a22acd8cb759304e4cd2e3215d488 upstream. kiocb_done() should care to specifically redirecting requests to io-wq. Remove the hopping to tw to then queue an io-wq, return -EAGAIN and let the core code io_uring handle offloading. Signed-off-by: Pavel Begunkov Tested-by: Ming Lei Link: https://lore.kernel.org/r/413564e550fe23744a970e1783dfa566291b0e6f.1710799188.git.asml.silence@gmail.com Signed-off-by: Jens Axboe (cherry picked from commit 6e6b8c62120a22acd8cb759304e4cd2e3215d488) Signed-off-by: Greg Kroah-Hartman

io_uring/sqpoll: close race on waiting for sqring entries

2024-10-22T13:46:27+00:00

commit 28aabffae6be54284869a91cd8bccd3720041129 upstream. When an application uses SQPOLL, it must wait for the SQPOLL thread to consume SQE entries, if it fails to get an sqe when calling io_uring_get_sqe(). It can do so by calling io_uring_enter(2) with the flag value of IORING_ENTER_SQ_WAIT. In liburing, this is generally done with io_uring_sqring_wait(). There's a natural expectation that once this call returns, a new SQE entry can be retrieved, filled out, and submitted. However, the kernel uses the cached sq head to determine if the SQRING is full or not. If the SQPOLL thread is currently in the process of submitting SQE entries, it may have updated the cached sq head, but not yet committed it to the SQ ring. Hence the kernel may find that there are SQE entries ready to be consumed, and return successfully to the application. If the SQPOLL thread hasn't yet committed the SQ ring entries by the time the application returns to userspace and attempts to get a new SQE, it will fail getting a new SQE. Fix this by having io_sqring_full() always use the user visible SQ ring head entry, rather than the internally cached one. Cc: stable@vger.kernel.org # 5.10+ Link: https://github.com/axboe/liburing/discussions/1267 Reported-by: Benedek Thaler Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: use the right type for work_llist empty check

2024-06-12T09:11:32+00:00

[ Upstream commit 22537c9f79417fed70b352d54d01d2586fee9521 ] io_task_work_pending() uses wq_list_empty() on ctx->work_llist, but it's not an io_wq_work_list, it's a struct llist_head. They both have ->first as head-of-list, and it turns out the checks are identical. But be proper and use the right helper. Fixes: dac6a0eae793 ("io_uring: ensure iopoll runs local task work as well") Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/unix: drop usage of io_uring socket

2024-03-26T22:19:09+00:00

Commit a4104821ad651d8a0b374f0b2474c345bbb42f82 upstream. Since we no longer allow sending io_uring fds over SCM_RIGHTS, move to using io_is_uring_fops() to detect whether this is a io_uring fd or not. With that done, kill off io_uring_get_socket() as nobody calls it anymore. This is in preparation to yanking out the rest of the core related to unix gc with io_uring. Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/poll: add requeue return code from poll multishot handling

2024-02-16T18:10:56+00:00

Commit 704ea888d646cb9d715662944cf389c823252ee0 upstream. Since our poll handling is edge triggered, multishot handlers retry internally until they know that no more data is available. In preparation for limiting these retries, add an internal return code, IOU_REQUEUE, which can be used to inform the poll backend about the handler wanting to retry, but that this should happen through a normal task_work requeue rather than keep hammering on the issue side for this one request. No functional changes in this patch, nobody is using this return code just yet. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman