Pavel Begunkov 475eb39b00 io_uring/zcrx: add sync refill queue flushing ... Add an zcrx interface via IORING_REGISTER_ZCRX_CTRL that forces the kernel to flush / consume entries from the refill queue. Just as with the IORING_REGISTER_ZCRX_REFILL attempt, the motivation is to address cases where the refill queue becomes full, and the user can't return buffers and needs to stash them. It's still a slow path, and the user should size refill queue appropriately, but it should be helpful for handling temporary traffic spikes and other unpredictable conditions. The interface is simpler comparing to ZCRX_REFILL as it doesn't need temporary refill entry arrays and gives natural batching, whereas ZCRX_REFILL requires even more user logic to be somewhat efficient. Also, add a structure for the operation. It's not currently used but can serve for future improvements like limiting the number of buffers to process, etc. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2025-11-13 11:19:37 -07:00
..
advise.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
advise.h
alloc_cache.c	io_uring: add alloc_cache.c	2025-01-28 15:10:40 -07:00
alloc_cache.h	io_uring/net: convert to struct iou_vec	2025-03-07 13:41:08 -07:00
cancel.c	io_uring/cancel: move cancelation code from io_uring.c to cancel.c	2025-11-04 09:32:09 -07:00
cancel.h	io_uring/cancel: move cancelation code from io_uring.c to cancel.c	2025-11-04 09:32:09 -07:00
cmd_net.c	io_uring/uring_cmd: add support for IORING_SETUP_CQE_MIXED	2025-08-27 11:24:15 -06:00
epoll.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
epoll.h	io_uring/epoll: add support for IORING_OP_EPOLL_WAIT	2025-02-20 07:59:56 -07:00
eventfd.c	io_uring/eventfd: open code io_eventfd_grab()	2025-04-24 08:33:54 -06:00
eventfd.h	io_uring/eventfd: dedup signalling helpers	2025-04-24 08:33:54 -06:00
fdinfo.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
fdinfo.h
filetable.c	io_uring: correct __must_hold annotation in io_install_fixed_file	2025-10-23 07:25:07 -06:00
filetable.h	io_uring/rsrc: pass 'struct io_ring_ctx' reference to rsrc helpers	2024-11-07 15:24:33 -07:00
fs.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
fs.h
futex.c	io_uring/futex: move futexv owned status to struct io_futexv_data	2025-11-05 12:55:07 -07:00
futex.h	io_uring: move cancelations to be io_uring_task based	2024-11-06 13:55:38 -07:00
io-wq.c	io_uring/io-wq: fix max_workers breakage and nr_workers underflow	2025-09-15 10:46:13 -06:00
io-wq.h	io_uring/wq: avoid indirect do_work/free_work calls	2025-04-21 05:06:58 -06:00
io_uring.c	io_uring: move cq/sq user offset init around	2025-11-13 07:27:35 -07:00
io_uring.h	io_uring: pre-calculate scq layout	2025-11-13 07:27:34 -07:00
kbuf.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
kbuf.h	io-uring: move struct io_br_sel into io_uring_types.h	2025-08-24 11:41:12 -06:00
Kconfig	io_uring: make zcrx depend on CONFIG_IO_URING	2025-03-31 07:07:44 -06:00
Makefile	io_uring: introduce io_uring querying	2025-09-08 08:06:37 -06:00
memmap.c	io_uring: add helper calculating region byte size	2025-11-11 07:53:33 -07:00
memmap.h	io_uring: add helper calculating region byte size	2025-11-11 07:53:33 -07:00
mock_file.c	io_uring/mock: add trivial poll handler	2025-07-02 08:10:26 -06:00
msg_ring.c	io_uring: add wrapper type for io_req_tw_func_t arg	2025-11-03 08:31:26 -07:00
msg_ring.h	io_uring/msg_ring: Drop custom destructor	2024-12-27 10:08:21 -07:00
napi.c	net: use napi_id_valid helper	2025-02-17 16:43:04 -08:00
napi.h	io_uring/napi: add static napi tracking strategy	2024-11-06 13:55:38 -07:00
net.c	io_uring/zcrx: elide passing msg flags	2025-11-13 11:19:37 -07:00
net.h	io_uring/net: convert to struct iou_vec	2025-03-07 13:41:08 -07:00
nop.c	io_uring/nop: add support for IORING_SETUP_CQE_MIXED	2025-08-27 11:24:15 -06:00
nop.h
notif.c	io_uring: fix typos and comment wording	2025-11-05 08:21:43 -07:00
notif.h	io_uring/notif: implement notification stacking	2024-04-22 19:31:18 -06:00
opdef.c	io_uring: add support for IORING_SETUP_SQE_MIXED	2025-10-22 07:34:57 -06:00
opdef.h	io_uring: add support for IORING_SETUP_SQE_MIXED	2025-10-22 07:34:57 -06:00
openclose.c	io_uring: don't include filetable.h in io_uring.h	2025-09-08 13:20:46 -06:00
openclose.h	io_uring: add support for IORING_OP_PIPE	2025-04-21 05:06:58 -06:00
poll.c	io_uring: add wrapper type for io_req_tw_func_t arg	2025-11-03 08:31:26 -07:00
poll.h	io_uring: add wrapper type for io_req_tw_func_t arg	2025-11-03 08:31:26 -07:00
query.c	io_uring/query: introduce rings info query	2025-11-13 11:17:36 -07:00
query.h	io_uring: introduce io_uring querying	2025-09-08 08:06:37 -06:00
refs.h	io_uring: always do atomic put from iowq	2025-04-03 08:31:57 -06:00
register.c	io_uring/zcrx: introduce IORING_REGISTER_ZCRX_CTRL	2025-11-13 11:19:37 -07:00
register.h	io_uring: temporarily disable registered waits	2024-11-15 09:58:34 -07:00
rsrc.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
rsrc.h	io_uring/rsrc: refactor io_{un}account_mem() to take {user,mm}_struct param	2025-11-06 16:23:21 -07:00
rw.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
rw.h	io_uring: add wrapper type for io_req_tw_func_t arg	2025-11-03 08:31:26 -07:00
slist.h	io_uring/slist: remove unused wq list splice helpers	2025-11-03 10:51:39 -07:00
splice.c	io_uring: don't include filetable.h in io_uring.h	2025-09-08 13:20:46 -06:00
splice.h	io_uring/splice: open code 2nd direct file assignment	2024-10-29 13:43:28 -06:00
sqpoll.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
sqpoll.h	io_uring/sqpoll: switch away from getrusage() for CPU accounting	2025-10-22 10:51:20 -06:00
statx.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
statx.h
sync.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
sync.h
tctx.c	io_uring/wq: avoid indirect do_work/free_work calls	2025-04-21 05:06:58 -06:00
tctx.h	io_uring: simplify __io_uring_add_tctx_node	2022-10-07 12:25:30 -06:00
timeout.c	io_uring: add wrapper type for io_req_tw_func_t arg	2025-11-03 08:31:26 -07:00
timeout.h	io_uring/timeout: don't export link t-out disarm helper	2025-05-06 10:11:23 -06:00
truncate.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
truncate.h	io_uring: add support for ftruncate	2024-02-09 09:04:39 -07:00
uring_cmd.c	io_uring/uring_cmd: avoid double indirect call in task work dispatch	2025-11-03 08:31:26 -07:00
uring_cmd.h	io_uring/cmd: remove struct io_uring_cmd_data	2025-07-18 12:34:56 -06:00
waitid.c	Merge branch 'io_uring-6.18' into for-6.19/io_uring	2025-11-13 07:26:37 -07:00
waitid.h	io_uring: move cancelations to be io_uring_task based	2024-11-06 13:55:38 -07:00
xattr.c	io_uring: finish IOU_OK -> IOU_COMPLETE transition	2025-05-21 08:41:16 -06:00
xattr.h
zcrx.c	io_uring/zcrx: add sync refill queue flushing	2025-11-13 11:19:37 -07:00
zcrx.h	io_uring/zcrx: introduce IORING_REGISTER_ZCRX_CTRL	2025-11-13 11:19:37 -07:00