summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2022-07-25io_uring: better caching for ctx timeout fieldsPavel Begunkov1-6/+9
Following timeout fields access patterns, move all of them into a separate cache line inside ctx, so they don't intervene with normal completion caching, especially since timeout removals and completion are separated and the later is done via tw. It also sheds some bytes from io_ring_ctx, 1216B -> 1152B Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/4b163793072840de53b3cb66e0c2995e7226ff78.1655310733.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move defer_list to slow dataPavel Begunkov1-1/+4
draining is slow path, move defer_list to the end where slow data lives inside the context. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/e16379391ca72b490afdd24e8944baab849b4a7b.1655310733.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: make reg buf init consistentPavel Begunkov1-6/+3
The default (i.e. empty) state of register buffer is dummy_ubuf, so set it to dummy on init instead of NULL. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/c5456aecf03d9627fbd6e65e100e2b5293a6151e.1655310733.git.asml.silence@gmail.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: deprecate epoll_ctl supportJens Axboe1-0/+4
As far as we know, nobody ever adopted the epoll_ctl management via io_uring. Deprecate it now with a warning, and plan on removing it in a later kernel version. When we do remove it, we can revert the following commits as well: 39220e8d4a2a ("eventpoll: support non-blocking do_epoll_ctl() calls") 58e41a44c488 ("eventpoll: abstract out epoll_ctl() handler") Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> Link: https://lore.kernel.org/io-uring/CAHk-=wiTyisXBgKnVHAGYCNvkmjk=50agS2Uk6nr+n3ssLZg2w@mail.gmail.com/ Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: add support for level triggered pollJens Axboe2-5/+13
By default, the POLL_ADD command does edge triggered poll - if we get a non-zero mask on the initial poll attempt, we complete the request successfully. Support level triggered by always waiting for a notification, regardless of whether or not the initial mask matches the file state. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move opcode table to opdef.cJens Axboe5-469/+501
We already have the declarations in opdef.h, move the rest into its own file rather than in the main io_uring.c file. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move read/write related opcodes to its own fileJens Axboe5-1231/+1263
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move remaining file table manipulation to filetable.cJens Axboe4-88/+90
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move rsrc related data, core, and commandsJens Axboe6-1418/+1480
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split provided buffers handling into its own fileJens Axboe7-636/+672
Move both the opcodes related to it, and the internals code dealing with it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move cancelation into its own fileJens Axboe6-178/+204
This also helps cleanup the io_uring.h cancel parts, as we can make things static in the cancel.c file, mostly. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move poll handling into its own fileJens Axboe6-827/+879
Add a io_poll_issue() rather than export the general task_work locking and io_issue_sqe(), and put the io_op_defs definition and structure into a separate header file so that poll can use it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: add opcode name to io_op_defsJens Axboe1-98/+52
This kills the last per-op switch. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: include and forward-declaration sanitationJens Axboe1-12/+5
Remove some dead headers we no longer need, and get rid of the io_ring_ctx and io_uring_fops forward declarations. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move io_uring_task (tctx) helpers into its own fileJens Axboe5-365/+396
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move fdinfo helpers to its own fileJens Axboe6-208/+230
This also means moving a bit more of the fixed file handling to the filetable side, which makes sense separately too. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: use io_is_uring_fops() consistentlyJens Axboe1-8/+8
Convert the last spots that check for io_uring_fops to use the provided helper instead. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move SQPOLL related handling into its own fileJens Axboe5-462/+497
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move timeout opcodes and handling into its own fileJens Axboe6-660/+701
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move our reference counting into a headerJens Axboe2-42/+49
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move msg_ring into its own fileJens Axboe4-55/+71
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split network related opcodes into its own fileJens Axboe5-835/+884
While at it, convert the handlers to just use io_eopnotsupp_prep() if CONFIG_NET isn't set. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move statx handling to its own fileJens Axboe4-62/+82
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move epoll handler to its own fileJens Axboe4-50/+70
Would be nice to sort out Kconfig for this and don't even compile epoll.c if we don't have epoll configured. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: add a dummy -EOPNOTSUPP prep handlerJens Axboe1-9/+14
Add it and use it for the epoll handling, if epoll isn't configured. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move uring_cmd handling to its own fileJens Axboe5-124/+142
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split out open/close operationsJens Axboe5-298/+345
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: separate out file table handling codeJens Axboe5-93/+117
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split out fadvise/madvise operationsJens Axboe4-85/+109
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split out fs related sync/fallocate functionsJens Axboe4-97/+124
This splits out sync_file_range, fsync, and fallocate. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split out splice related operationsJens Axboe5-130/+154
This splits out splice and tee support. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: split out filesystem related operationsJens Axboe4-283/+316
This splits out renameat, unlinkat, mkdirat, symlinkat, and linkat. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move nop into its own fileJens Axboe4-15/+32
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: move xattr related opcodes to its own fileJens Axboe4-247/+277
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: handle completions in the coreJens Axboe2-134/+142
Normally request handlers complete requests themselves, if they don't return an error. For the latter case, the core will complete it for them. This is unhandy for pushing opcode handlers further out, as we don't want a bunch of inline completion code and we don't want to make the completion path slower than it is now. Let the core handle any completion, unless the handler explicitly asks us not to. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: set completion results upfrontJens Axboe2-12/+22
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: add io_uring_types.hJens Axboe2-488/+499
This adds definitions of structs that both the core and the various opcode handlers need to know about. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: define a request type cleanup handlerJens Axboe1-69/+86
This can move request type specific cleanup into a private handler, removing the need for the core io_uring parts to know what types they are dealing with. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: unify struct io_symlink and io_hardlinkJens Axboe1-20/+7
They are really just a subset of each other, just use the one type. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert iouring_cmd to io_cmd_typeJens Axboe1-8/+10
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert xattr to use io_cmd_typeJens Axboe1-10/+9
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert rsrc_update to io_cmd_typeJens Axboe1-16/+19
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert msg and nop to io_cmd_typeJens Axboe1-9/+9
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert splice to use io_cmd_typeJens Axboe1-5/+4
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert epoll to io_cmd_typeJens Axboe1-7/+8
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert file system request types to use io_cmd_typeJens Axboe1-45/+57
This converts statx, rename, unlink, mkdir, symlink, and hardlink to use io_cmd_type. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert madvise/fadvise to use io_cmd_typeJens Axboe1-10/+12
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert open/close path to use io_cmd_typeJens Axboe1-32/+40
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert timeout path to use io_cmd_typeJens Axboe1-49/+68
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-07-25io_uring: convert cancel path to use io_cmd_typeJens Axboe1-11/+13
Signed-off-by: Jens Axboe <axboe@kernel.dk>
generated by cgit v1.2.3 (git 2.39.0) at 2025-09-30 08:59:36 +0300