1.\" $NetBSD: percpu.9,v 1.16 2023/08/02 06:24:46 riastradh Exp $ 2.\" 3.\" Copyright (c) 2010 The NetBSD Foundation, Inc. 4.\" All rights reserved. 5.\" 6.\" This code is derived from software contributed to The NetBSD Foundation 7.\" by David Young. 8.\" 9.\" Redistribution and use in source and binary forms, with or without 10.\" modification, are permitted provided that the following conditions 11.\" are met: 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 18.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS 19.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 20.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 21.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS 22.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 23.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 24.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 25.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 26.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 27.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 28.\" POSSIBILITY OF SUCH DAMAGE. 29.\" 30.Dd February 7, 2020 31.Dt PERCPU 9 32.Os 33.Sh NAME 34.Nm percpu , 35.Nm percpu_alloc , 36.Nm percpu_create , 37.Nm percpu_free , 38.Nm percpu_getref , 39.Nm percpu_putref , 40.Nm percpu_foreach , 41.Nm percpu_foreach_xcall 42.Nd per-CPU storage allocator 43.Sh SYNOPSIS 44.In sys/percpu.h 45.Vt typedef void (*percpu_callback_t)(void *, void *, struct cpu_info *); 46.Ft percpu_t * 47.Fn percpu_alloc "size_t size" 48.Ft percpu_t * 49.Fn percpu_create "size_t size" "percpu_callback_t ctor" "percpu_callback_t dtor" "void *arg" 50.Ft void 51.Fn percpu_free "percpu_t *pc" "size_t size" 52.Ft void * 53.Fn percpu_getref "percpu_t *pc" 54.Ft void 55.Fn percpu_putref "percpu_t *pc" 56.Ft void 57.Fn percpu_foreach "percpu_t *pc" "percpu_callback_t cb" "void *arg" 58.Ft void 59.Fn percpu_foreach_xcall "percpu_t *pc" "u_int xcflags" "percpu_callback_t cb" "void *arg" 60.Sh DESCRIPTION 61The machine-independent 62.Nm 63interface provides per-CPU, CPU-local memory reservations to kernel 64subsystems. 65.Fo percpu_alloc 66.Fa size 67.Fc 68reserves on each CPU an independent memory region of 69.Fa size 70bytes that is local to that CPU, returning a handle 71.Po 72.Vt percpu_t 73.Pc 74to those regions. 75A thread may subsequently ask for a pointer, 76.Fa p , 77to the region held by the 78.Vt percpu_t 79on the thread's current CPU. 80Until the thread relinquishes the pointer, or voluntarily sleeps, 81the thread may read or write the region at 82.Fa p 83without causing interprocessor memory synchronization. 84.Sh FUNCTIONS 85.Bl -tag -width compact 86.It Fn percpu_alloc "size" 87Call this in thread context to allocate 88.Fa size 89bytes of local storage on each CPU. 90The storage is initialized with zeroes. 91Treat this as an expensive operation. 92.Fn percpu_alloc 93returns a handle for the per-CPU storage. 94.It Fn percpu_create "size" "ctor" "dtor" "arg" 95Like 96.Fn percpu_alloc , 97but before any access to the storage on any CPU, arrange to call 98.Fn "(*ctor)" p arg ci , 99where 100.Fa p 101is the pointer to that CPU's storage and 102.Fa ci 103is the 104.Vt "struct cpu_info *" 105for that CPU. 106This may happen synchronously in 107.Fn percpu_create 108or when a CPU is attached later. 109Further, arrange that 110.Fn percpu_free 111will do the same with 112.Fn "(*dtor)" p arg ci . 113.Pp 114.Fa ctor 115and 116.Fa dtor 117.Em MAY 118sleep, e.g. to allocate memory or to wait for users to drain before 119deallocating memory. 120Do not rely on any particular order of iteration over the CPUs. 121.It Fn percpu_free "pc" "size" 122Call this in thread context to 123return to the system the per-CPU storage held by 124.Fa pc . 125.Fa size 126should match the 127.Fa size 128passed to 129.Fn percpu_alloc 130or 131.Fn percpu_create . 132When 133.Fn percpu_free 134returns, 135.Fa pc 136is undefined. 137Treat this as an expensive operation. 138.It Fn percpu_getref "pc" 139Disable preemption and return a pointer to the storage held by 140.Fa pc 141on the local CPU. 142Use 143.Fn percpu_getref 144in either thread or interrupt context. 145Follow each 146.Fn percpu_getref 147call with a matching call to 148.Fn percpu_putref . 149.Pp 150Caller 151.Em MUST NOT 152sleep after 153.Fn percpu_getref , 154not even on an adaptive lock, before 155.Fn percpu_putref . 156.It Fn percpu_putref "pc" 157Indicate that the thread is finished 158with the pointer returned by the matching 159call to 160.Fn percpu_getref . 161Re-enables preemption. 162.It Fn percpu_foreach "pc" "cb" "arg" 163For each CPU, with 164.Fa ci 165being the corresponding 166.Vt "struct cpu_info *" 167and 168.Fa "p" 169the CPU-local storage held by 170.Fa pc , 171run 172.Fo "(*cb)" 173.Fa "p" 174.Fa "arg" 175.Fa "ci" 176.Fc . 177The call to 178.Fa cb 179runs in the current thread; use 180.Fn percpu_foreach_xcall 181to execute the call to 182.Fa cb 183on the remote CPUs. 184.Pp 185Must be used in thread context. 186.Fa cb 187.Em MUST NOT 188sleep except on adaptive locks, and should be fast. 189Do not rely on any particular order of iteration over the CPUs. 190.It Fn percpu_foreach_xcall "pc" "xcflags" "cb" "arg" 191Like 192.Fn percpu_foreach , 193except the call to 194.Fa cb 195executes on the remote CPUs. 196The 197.Fa xcflags 198argument specifies how the cross calls are invoked; see 199.Xr xc_broadcast 9 . 200The same caveats and restrictions that apply to 201.Fn percpu_foreach 202also apply to 203.Fn percpu_foreach_xcall . 204.El 205.Sh CODE REFERENCES 206The 207.Nm 208interface is implemented within the file 209.Pa sys/kern/subr_percpu.c . 210.\" .Sh EXAMPLES 211.Sh SEE ALSO 212.Xr atomic_ops 3 , 213.Xr kmem 9 , 214.Xr pcq 9 , 215.Xr pool_cache 9 , 216.Xr xcall 9 217.Sh HISTORY 218The 219.Nm 220interface first appeared in 221.Nx 6.0 . 222.Sh AUTHORS 223.An YAMAMOTO Takashi Aq Mt yamt@NetBSD.org 224.\" .Sh CAVEATS 225.\" .Sh BUGS 226.\" .Sh SECURITY CONSIDERATIONS 227