1.\" $NetBSD: locking.9,v 1.8 2017/08/27 20:44:42 wiz Exp $ 2.\" 3.\" Copyright (c) 2015 The NetBSD Foundation, Inc. 4.\" All rights reserved. 5.\" 6.\" This code is derived from software contributed to The NetBSD Foundation 7.\" by Kamil Rytarowski. 8.\" 9.\" Redistribution and use in source and binary forms, with or without 10.\" modification, are permitted provided that the following conditions 11.\" are met: 12.\" 1. Redistributions of source code must retain the above copyright 13.\" notice, this list of conditions and the following disclaimer. 14.\" 2. Redistributions in binary form must reproduce the above copyright 15.\" notice, this list of conditions and the following disclaimer in the 16.\" documentation and/or other materials provided with the distribution. 17.\" 18.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS 19.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 20.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 21.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS 22.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 23.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 24.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 25.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 26.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 27.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 28.\" POSSIBILITY OF SUCH DAMAGE. 29.\" 30.Dd August 23, 2017 31.Dt LOCKING 9 32.Os 33.Sh NAME 34.Nm locking 35.Nd introduction to kernel synchronization and interrupt control 36.Sh DESCRIPTION 37The 38.Nx 39kernel provides several synchronization and interrupt control primitives. 40This man page aims to give an overview of these interfaces and their proper 41application. 42Also included are basic kernel thread control primitives and a rough 43overview of the 44.Nx 45kernel design. 46.Sh KERNEL OVERVIEW 47The aim of synchronization, threads and interrupt control in the kernel is: 48.Bl -bullet -offset indent 49.It 50To control concurrent access to shared resources (critical sections). 51.It 52Spawn tasks from an interrupt in the thread context. 53.It 54Mask interrupts from threads. 55.It 56Scale on multiple CPU system. 57.El 58.Pp 59There are three types of contexts in the 60.Nx 61kernel: 62.Bl -bullet -offset indent 63.It 64.Em Thread context 65- running processes (represented by 66.Dv struct proc ) 67and light-weight processes (represented by 68.Dv struct lwp , 69also known as kernel threads). 70Code in this context can sleep, block resources and own address-space. 71.It 72.Em Software interrupt context 73- limited by thread context. 74Code in this context must be processed shortly. 75These interrupts don't own any address space context. 76Software interrupts are a way of deferring hardware interrupts to do more 77expensive processing at a lower interrupt priority. 78.It 79.Em Hard interrupt context 80- Code in this context must be processed as quickly as possible. 81It is forbidden for a piece of code to sleep or access long-awaited resources here. 82.El 83.Pp 84The main differences between processes and kernel threads are: 85.Bl -bullet -offset indent 86.It 87A single process can own multiple kernel threads (LWPs). 88.It 89A process owns address space context to map userland address space. 90.It 91Processes are designed for userland executables and kernel threads for 92in-kernel tasks. 93The only process running in the kernel-space is 94.Dv proc0 95(called swapper). 96.El 97.Sh INTERFACES 98.Ss Atomic memory operations 99The 100.Nm atomic_ops 101family of functions provide atomic memory operations. 102There are 7 classes of atomic memory operations available: 103addition, logical 104.Dq and , 105compare-and-swap, decrement, increment, logical 106.Dq or , 107swap. 108.Pp 109See 110.Xr atomic_ops 3 . 111.Ss Condition variables 112Condition variables (CVs) are used in the kernel to synchronize access to 113resources that are limited (for example, memory) and to wait for pending I/O 114operations to complete. 115.Pp 116See 117.Xr condvar 9 . 118.Ss Memory access barrier operations 119The 120.Nm membar_ops 121family of functions provide memory access barrier operations necessary for 122synchronization in multiprocessor execution environments that have relaxed load 123and store order. 124.Pp 125See 126.Xr membar_ops 3 . 127.Ss Memory barriers 128The memory barriers can be used to control the order in which memory accesses 129occur, and thus the order in which those accesses become visible to other 130processors. 131They can be used to implement 132.Dq lockless 133access to data structures where the necessary barrier conditions are well 134understood. 135.Ss Mutual exclusion primitives 136Thread-base adaptive mutexes. 137These are lightweight, 138exclusive locks that use threads as the focus of synchronization activity. 139Adaptive mutexes typically behave like spinlocks, 140but under specific conditions an attempt to acquire an already held adaptive 141mutex may cause the acquiring thread to sleep. 142Sleep activity occurs rarely. 143Busy-waiting is typically more efficient because mutex hold times are most 144often short. 145In contrast to pure spinlocks, 146a thread holding an adaptive mutex may be pre-empted in the kernel, 147which can allow for reduced latency where soft real-time application are in use 148on the system. 149.Pp 150See 151.Xr mutex 9 . 152.Ss Restartable atomic sequences 153Restartable atomic sequences are user code only sequences which are guaranteed 154to execute without preemption. 155This property is assured by checking the set of restartable atomic sequences 156registered for a process during 157.Xr cpu_switchto 9 . 158If a process is found to have been preempted during a restartable sequence, 159then its execution is rolled-back to the start of the sequence by resetting its 160program counter which is saved in its process control block (PCB). 161.Pp 162See 163.Xr ras 9 . 164.Ss Reader / writer lock primitives 165Reader / writer locks (RW locks) are used in the kernel to synchronize access 166to an object among LWPs (lightweight processes) and soft interrupt handlers. 167In addition to the capabilities provided by mutexes, 168RW locks distinguish between read (shared) and write (exclusive) access. 169.Pp 170See 171.Xr rwlock 9 . 172.Ss Functions to modify system interrupt priority level 173These functions raise and lower the interrupt priority level. 174They are used by kernel code to block interrupts in critical sections, 175in order to protect data structures. 176.Pp 177See 178.Xr spl 9 . 179.Ss Machine-independent software interrupt framework 180The software interrupt framework is designed to provide a generic software 181interrupt mechanism which can be used any time a low-priority callback is 182required. 183It allows dynamic registration of software interrupts for loadable drivers, 184protocol stacks, software interrupt prioritization, software interrupt fair 185queuing and allows machine-dependent optimizations to reduce cost. 186.Pp 187See 188.Xr softint 9 . 189.Ss Functions to raise the system priority level 190The 191.Nm splraiseipl 192function raises the system priority level to the level specified by 193.Dv icookie , 194which should be a value returned by 195.Xr makeiplcookie 9 . 196In general, device drivers should not make use of this interface. 197To ensure correct synchronization, 198device drivers should use the 199.Xr condvar 9 , 200.Xr mutex 9 , 201and 202.Xr rwlock 9 203interfaces. 204.Pp 205See 206.Xr splraiseipl 9 . 207.Ss Passive serialization mechanism 208Passive serialization is a reader / writer synchronization mechanism designed 209for lock-less read operations. 210The read operations may happen from software interrupt at 211.Dv IPL_SOFTCLOCK . 212.Pp 213See 214.Xr pserialize 9 . 215.Ss Passive reference mechanism 216Passive references allow CPUs to cheaply acquire and release passive 217references to a resource, which guarantee the resource will not be 218destroyed until the reference is released. 219Acquiring and releasing passive references requires no interprocessor 220synchronization, except when the resource is pending destruction. 221.Pp 222See 223.Xr psref 9 . 224.Ss Localcount mechanism 225Localcounts are used in the kernel to implement a medium-weight reference 226counting mechanism. 227During normal operations, localcounts do not need 228the interprocessor synchronization associated with 229.Xr atomic_ops 3 230atomic memory operations, and (unlike 231.Xr psref 9 ) 232localcount references can be held across sleeps and can migrate between 233CPUs. 234Draining a localcount requires more expensive interprocessor 235synchronization than 236.Xr atomic_ops 3 237(similar to 238.Xr psref 9 ) . 239And localcount references require eight bytes of memory per object per-CPU, 240significantly more than 241.Xr atomic_ops 3 242and almost always more than 243.Xr psref 9 . 244.Pp 245See 246.Xr localcount 9 . 247.Ss Simple do-it-in-thread-context framework 248The workqueue utility routines are provided to defer work which is needed to be 249processed in a thread context. 250.Pp 251See 252.Xr workqueue 9 . 253.Sh USAGE 254The following table describes in which contexts the use of the 255.Nx 256kernel interfaces are valid. 257Synchronization primitives which are available in more than one context 258can be used to protect shared resources between the contexts they overlap. 259.Bl -column -offset indent \ 260"xxxxxxxxxxxx " "xxxxxxx " "xxxxxxx " "xxxxxxx " 261.It Sy interface Ta Sy thread Ta Sy softirq Ta Sy hardirq 262.It Xr atomic_ops 3 Ta yes Ta yes Ta yes 263.It Xr condvar 9 Ta yes Ta partly Ta no 264.It Xr membar_ops 3 Ta yes Ta yes Ta yes 265.It Xr mutex 9 Ta yes Ta depends Ta depends 266.It Xr rwlock 9 Ta yes Ta yes Ta no 267.It Xr softint 9 Ta yes Ta yes Ta yes 268.It Xr spl 9 Ta yes Ta no Ta no 269.It Xr splraiseipl 9 Ta yes Ta no Ta no 270.It Xr pserialize 9 Ta yes Ta yes Ta no 271.It Xr psref 9 Ta yes Ta yes Ta no 272.It Xr localcount 9 Ta yes Ta yes Ta no 273.It Xr workqueue 9 Ta yes Ta yes Ta yes 274.El 275.Sh SEE ALSO 276.Xr atomic_ops 3 , 277.Xr membar_ops 3 , 278.Xr condvar 9 , 279.Xr mutex 9 , 280.Xr ras 9 , 281.Xr rwlock 9 , 282.Xr softint 9 , 283.Xr spl 9 , 284.Xr splraiseipl 9 , 285.Xr workqueue 9 286.Sh HISTORY 287Initial SMP support was introduced in 288.Nx 2.0 289and was designed with a giant kernel lock. 290Through 291.Nx 4.0 , 292the kernel used spinlocks and a per-CPU interrupt priority level (the 293.Xr spl 9 294system). 295These mechanisms did not lend themselves well to a multiprocessor environment 296supporting kernel preemption. 297The use of thread based (lock) synchronization was limited and the available 298synchronization primitive (lockmgr) was inefficient and slow to execute. 299.Nx 5.0 300introduced massive performance improvements on multicore hardware 301by Andrew Doran. 302This work was sponsored by The 303.Nx 304Foundation. 305.Pp 306A 307.Nm 308manual first appeared in 309.Nx 8.0 310and was inspired by the corresponding 311.Nm 312manuals in 313.Fx 314and 315.Dx . 316.Sh AUTHORS 317.An Kamil Rytarowski Aq Mt kamil@NetBSD.org . 318