kernel/linux.git/io_uring/io_uring.h, branch v6.1.175

io_uring: include dying ring in task_work "should cancel" state

2025-09-25T08:58:53+00:00

Commit 3539b1467e94336d5854ebf976d9627bfb65d6c3 upstream. When running task_work for an exiting task, rather than perform the issue retry attempt, the task_work is canceled. However, this isn't done for a ring that has been closed. This can lead to requests being successfully completed post the ring being closed, which is somewhat confusing and surprising to an application. Rather than just check the task exit state, also include the ring ref state in deciding whether or not to terminate a given request when run from task_work. Cc: stable@vger.kernel.org # 6.1+ Link: https://github.com/axboe/liburing/discussions/1459 Reported-by: Benedek Thaler Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: backport io_should_terminate_tw()

2025-09-25T08:58:53+00:00

Parts of commit b6f58a3f4aa8dba424356c7a69388a81f4459300 upstream. Backport io_should_terminate_tw() helper to judge whether task_work should be run or terminated. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: get rid of remap_pfn_range() for mapping rings/sqes

2025-03-28T20:58:53+00:00

Commit 3ab1db3c6039e02a9deb9d5091d28d559917a645 upstream. Rather than use remap_pfn_range() for this and manually free later, switch to using vm_insert_pages() and have it Just Work. If possible, allocate a single compound page that covers the range that is needed. If that works, then we can just use page_address() on that page. If we fail to get a compound page, allocate single pages and use vmap() to map them into the kernel virtual address space. This just covers the rings/sqes, the other remaining user of the mmap remap_pfn_range() user will be converted separately. Once that is done, we can kill the old alloc/free code. Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring/rw: avoid punting to io-wq directly

2024-12-27T12:53:02+00:00

Commit 6e6b8c62120a22acd8cb759304e4cd2e3215d488 upstream. kiocb_done() should care to specifically redirecting requests to io-wq. Remove the hopping to tw to then queue an io-wq, return -EAGAIN and let the core code io_uring handle offloading. Signed-off-by: Pavel Begunkov Tested-by: Ming Lei Link: https://lore.kernel.org/r/413564e550fe23744a970e1783dfa566291b0e6f.1710799188.git.asml.silence@gmail.com Signed-off-by: Jens Axboe (cherry picked from commit 6e6b8c62120a22acd8cb759304e4cd2e3215d488) Signed-off-by: Greg Kroah-Hartman

io_uring/sqpoll: close race on waiting for sqring entries

2024-10-22T13:56:45+00:00

commit 28aabffae6be54284869a91cd8bccd3720041129 upstream. When an application uses SQPOLL, it must wait for the SQPOLL thread to consume SQE entries, if it fails to get an sqe when calling io_uring_get_sqe(). It can do so by calling io_uring_enter(2) with the flag value of IORING_ENTER_SQ_WAIT. In liburing, this is generally done with io_uring_sqring_wait(). There's a natural expectation that once this call returns, a new SQE entry can be retrieved, filled out, and submitted. However, the kernel uses the cached sq head to determine if the SQRING is full or not. If the SQPOLL thread is currently in the process of submitting SQE entries, it may have updated the cached sq head, but not yet committed it to the SQ ring. Hence the kernel may find that there are SQE entries ready to be consumed, and return successfully to the application. If the SQPOLL thread hasn't yet committed the SQ ring entries by the time the application returns to userspace and attempts to get a new SQE, it will fail getting a new SQE. Fix this by having io_sqring_full() always use the user visible SQ ring head entry, rather than the internally cached one. Cc: stable@vger.kernel.org # 5.10+ Link: https://github.com/axboe/liburing/discussions/1267 Reported-by: Benedek Thaler Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: use the right type for work_llist empty check

2024-06-12T09:03:05+00:00

[ Upstream commit 22537c9f79417fed70b352d54d01d2586fee9521 ] io_task_work_pending() uses wq_list_empty() on ctx->work_llist, but it's not an io_wq_work_list, it's a struct llist_head. They both have ->first as head-of-list, and it turns out the checks are identical. But be proper and use the right helper. Fixes: dac6a0eae793 ("io_uring: ensure iopoll runs local task work as well") Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring: don't use TIF_NOTIFY_SIGNAL to test for availability of task_work

2024-06-12T09:03:05+00:00

[ Upstream commit 6434ec0186b80c734aa7a2acf95f75f5c6dd943b ] Use task_work_pending() as a better test for whether we have task_work or not, TIF_NOTIFY_SIGNAL is only valid if the any of the task_work items had been queued with TWA_SIGNAL as the notification mechanism. Hence task_work_pending() is a more reliable check. Signed-off-by: Jens Axboe Stable-dep-of: 22537c9f7941 ("io_uring: use the right type for work_llist empty check") Signed-off-by: Sasha Levin

io_uring/unix: drop usage of io_uring socket

2024-03-26T22:20:22+00:00

Commit a4104821ad651d8a0b374f0b2474c345bbb42f82 upstream. Since we no longer allow sending io_uring fds over SCM_RIGHTS, move to using io_is_uring_fops() to detect whether this is a io_uring fd or not. With that done, kill off io_uring_get_socket() as nobody calls it anymore. This is in preparation to yanking out the rest of the core related to unix gc with io_uring. Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring: mark task TASK_RUNNING before handling resume/task work

2023-03-10T08:34:07+00:00

commit 2f2bb1ffc9983e227424d0787289da5483b0c74f upstream. Just like for task_work, set the task mode to TASK_RUNNING before doing any potential resume work. We're not holding any locks at this point, but we may have already set the task state to TASK_INTERRUPTIBLE in preparation for going to sleep waiting for events. Ensure that we set it back to TASK_RUNNING if we have work to process, to avoid warnings on calling blocking operations with !TASK_RUNNING. Fixes: b5d3ae202fbf ("io_uring: handle TIF_NOTIFY_RESUME when checking for task_work") Reported-by: kernel test robot Link: https://lore.kernel.org/oe-lkp/202302062208.24d3e563-oliver.sang@intel.com Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring: handle TIF_NOTIFY_RESUME when checking for task_work

2023-03-10T08:34:03+00:00

commit b5d3ae202fbfe055aa2a8ae8524531ee1dcab717 upstream. If TIF_NOTIFY_RESUME is set, then we need to call resume_user_mode_work() for PF_IO_WORKER threads. They never return to usermode, hence never get a chance to process any items that are marked by this flag. Most notably this includes the final put of files, but also any throttling markers set by block cgroups. Cc: stable@vger.kernel.org # 5.10+ Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman