1.\" $OpenBSD: bulk.8,v 1.10 2024/05/16 10:31:55 sthen Exp $ 2.\" 3.\" Copyright (c) 2016 Marc Espie <espie@openbsd.org> 4.\" 5.\" 6.\" Permission to use, copy, modify, and distribute this software for any 7.\" purpose with or without fee is hereby granted, provided that the above 8.\" copyright notice and this permission notice appear in all copies. 9.\" 10.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17.Dd $Mdocdate: May 16 2024 $ 18.Dt BULK 8 19.Os 20.Sh NAME 21.Nm bulk 22.Nd building OpenBSD packages in bulk 23.Sh DESCRIPTION 24There are quite a few steps necessary to build packages on a cluster. 25They are: 26.Pp 27.Bl -enum -compact -offset indent 28.It 29Choose master machine setup and create partitions. 30.It 31Setup chrooted builds on the master. 32.It 33Add slaves and do a full bulk. 34.It 35Clean up and do subsequent bulks. 36.It 37Perform additional maintenance. 38.El 39.Ss 1. Choose master machine setup and create partitions 40Setup a master machine with enough room for a chroot, say 41.Pa /build . 42Assuming you are using a cluster of machines, 43this chroot should contain NFS exportable partitions for distfiles, 44plists, and packages (one single partition can be used for simplicity). 45A full setup currently requires in the order of 120GB for distfiles 46and 70GB for packages. 47Expect these numbers to grow. 48150GB for each should last a few years. 49.Pp 50It is possible to build packages without a chroot, but the space 51requirement difference is negligible (a full 52.Ox 53install is less than 2GB), and having everything chrooted means you may 54install useful tools to help with the process outside of the chroot 55.Po 56for instance 57.Xr rsync 1 58.Pc . 59.Pp 60Reserve one "scratch" partition under the chroot for WRKOBJDIR 61(for instance, mfs, async, or SSD). 62This can often double as 63.Pa /tmp 64under the chroot. 65The largest ports can take in excess of 20GB each (more for a debug 66build) and you may have several of these built at the same time. 6750GB is probably a reasonable minimum on an architecture which can build 68chromium, though 100GB would not be overkill, especially if you have 69many cores. 70.Pp 71Alternately, you can setup your whole chroot as a scratch partition, 72and reserve one more permanent space under it for distfiles, 73packages, and plists. 74.Pp 75Choose a strategy for the ports tree itself. 76There must be a copy under 77.Pa /build . 78You can either copy it from outside the chroot, have it in an NFS 79partition, or manually make sure all machines on the cluster have the 80same ports tree (cvs checkout, rsync ...). 81.Pp 82Note that logs are only produced on the master, and thus do not 83need to be nfs exportable, nor even inside the chroot. 84.Pp 85.Ox 86now comes with default users for package builds, namely _pbuild and _pfetch. 87.Pp 88The default 89.Xr login.conf 5 90is appropriate for most setups, but _pbuild's datasize-cur may need 91to be bumped for a few ports. 92Likewise, maxproc-cur is too small for machines with more than 4-6 cpus. 93.Pp 94Note that _pbuild does not need network access, and is now blocked by default 95in 96.Xr pf 4 . 97.Pp 98Recent 99.Ox 100systems do not need any kind of 101.Xr doas 1 102setup for bulk ports builds, as 103.Xr dpb 1 104is run as root and drops permissions appropriately. 105.Pp 106However, you may still want to setup 107.Xr doas 1 108for root, if you want to manually fix ports, as 109.Ar PORTS_PRIVSEP 110relies on it. 111.Ss 2. Setup chrooted builds on the master 112Populate the initial chroot with 113.Xr proot 1 . 114Point DISTDIR, PACKAGE_REPOSITORY, PLIST_REPOSITORY, WRKOBJDIR 115to appropriate locations. 116.Pp 117Pay attention to nodev and wxallowed warnings. 118A chroot setup that can't have devices won't work at all. 119A setup without wxallowed in /usr/local and WRKOBJDIR won't 120build a lot of things. 121.Pp 122Check that this setup can build ports by running 123.Li dpb -B /build 124as root. 125Fix any issues related to file ownership at this point. 126See 127.Xr dpb 1 128for details. 129.Pp 130If your WRKOBJDIR is a temporary partition, make sure it 131belongs to _pbuild:_pbuild after a reboot. 132.Ss 3. Add slaves and do a full bulk 133Create identical slave machines with the same release material. 134Have them import the NFS partitions from the master, they 135don't need root access to the partitions. 136Set up 137.Xr ssh 1 138so that the master can connect to the slaves, using ssh protocol 2, 139as root, preferably without a password or passphrase (however, 140.Xr dpb 1 141uses a master connection, so a password would be required just once per host). 142.Pp 143Note that code on slave machines will only run as _pbuild 144(during builds) or root (during dependency installation). 145Slave machines only require highly restricted network access. 146They just need to act as nfs clients to the master and to be reachable 147through ssh from the master. 148.Pp 149Use a similar 150.Xr proot 1 151config to populate each slave. 152.Pp 153You should now be able to build ports on the slaves. 154A simple config will just have 155.Bd -literal -offset indent 156DEFAULT chroot=/build 157localhost 158host1 159\&... 160.Ed 161.Pp 162Check that the full config can still build ports. 163.Pp 164You're now ready for a full bulk. 165Beware that even fast configs (3 amd64 with 8 cores each) may take over 24 hours 166to finish. 167It's generally appropriate to run 168.Xr dpb 1 169under 170.Xr tmux 1 . 171.Ss 4. Clean up and do subsequent bulks 172Before running the next bulk, you should set up rotating logs and move the 173PACKAGE_REPOSITORY away. 174Save the PLIST_REPOSITORY and DISTDIR though. 175PLIST_REPOSITORY will catch problems in packing-lists. 176.Pa ${PLIST_REPOSITORY}/${ARCH}/history 177is also used to store 178.Xr sha256 1 179history, necessary to reorder files inside packages to speed updates up. 180.Pp 181The DISTDIR contains history information as well as DISTDIR/build-stats 182to speed further runs up. 183.Pp 184How you wipe things out depends on your setup. 185If you run 186.Xr proot 1 187again each time, most things will get cleaned up automatically 188.Po 189.Pa /build/usr/local , /build/var/db/pkg ... 190.Pc . 191Note that known directories such as WRKOBJDIR do not get cleaned up 192automatically, so you may want to set up a STARTUP_SCRIPT in 193.Xr dpb 1 . 194.Ss 5. Perform additional maintenance 195.Xr clean-old-distfiles 1 196should be run occasionally since the DISTDIR will continue growing. 197.Pp 198.Xr pkg_check-problems 1 199should be run occasionally to find out conflicts and dependency issues. 200.Sh SEE ALSO 201.Xr clean-old-distfiles 1 , 202.Xr dpb 1 , 203.Xr pkg_check-problems 1 , 204.Xr proot 1 , 205.Xr register-plist 1 , 206.Xr tmux 1 , 207.Xr bsd.port.mk 5 , 208.Xr release 8 209