1.HTML "Bootstrapping Plan 9 on PCs 2.de Os\" overstrike argument 3\\$1\l'|0–' 4.. 5. 6.TL 7Bootstrapping Plan 9 on PCs 8.AU 9Geoff Collyer 10.br 11.CW geoff@plan9.bell-labs.com 12.AI 13.MH 14.AB 15What's interesting or tricky about bootstrapping Plan 9 on PCs? 16.AE 17. 18.SH 19Introduction 20.LP 21Plan 9 has new PC bootstraps, 22.I 9boot 23and 24.I 9load, 25replacing the decade-old 26.I 9pxeload 27and 28.I 9load 29programs. 30What did we learn while writing them? 31.SH 32PC Constraints 33.LP 34The IBM PC imposes quite a few constraints on bootstrap programs 35(programs that load operating system kernels). 36A PC starts executing in 16-bit `real' (Intel 8086) mode 37and has no boot monitor, as other machines do, 38just a primitive BIOS that will perform a power-on self-test (POST) 39and attempt to read boot sectors 40from disks or load a modest payload from the network via TFTP. 41(Actually some new machines have slightly less primitive 42boot loaders called (U)EFI, but we don't deal with EFI.) 43The boot sectors must load further bootstrap programs 44that resemble the TFTP payload. 45These bootstrap programs can only address the first megabyte of memory 46until they get out of real mode, 47and even then the upper 384KB of the initial megabyte is reserved 48for device ROMs. 49.LP 50BIOS calls (via the 51.CW INT 52instruction) 53only work in real mode, 54so the bootstraps execute BIOS calls to learn the machine's 55memory map and power management configuration, 56and stash the results in the first megabyte 57for later retrieval by the loaded kernel. 58Empirically, some BIOSes enable interrupts (with 59.CW STI 60instructions) 61during BIOS calls, 62so the bootstraps disable them again after each call; 63failure to do so often results in an interrupt, 64perhaps from the clock, resetting the machine. 65.CW 9loadusb 66returns briefly to real mode 67to read USB devices and has mixed results with that. 68.LP 69Getting into 32-bit protected mode permits addressing the first 4GB of memory, 70but first it is necessary to enable the A20 address line (the 71.CW 1<<20 72bit). 73For (extreme) backward compatibility, this bit is normally held to zero 74until software requests that it be released, and holding it to zero will cause 75references to the second megabyte to be mapped to the first, etc., 76causing bizarre-seeming memory corruption. 77The old technique was to ask the keyboard controller to release it, 78but some systems now have no keyboard controller (they are servers 79or have USB keyboards). 80We have found it necessary to keep trying different methods until one succeeds 81and is verified to have succeeded. 82The new bootstraps also try an 83.CW INT 84.CW 15 85BIOS call and 86manipulation of port 87.CW 0x92 88(`system control' on some systems). 89.LP 90Even in protected mode with A20 enabled, some systems 91force a gap in the physical address space between 15MB and 16MB, 92which must be avoided. 93. 94.SH 95Plan 9 Requirements 96.IP • 3 97The new bootstraps must be able to load 64-bit 98.CW amd64 99kernels as well as 100.CW 386 101ones. 102In addition to Plan 9 boot image format, 103the new bootstraps understand ELF and ELF64 formats. 104.IP • 105Plan 9 kernels need to be started in 32-bit protected mode and 106implicitly assume that A20 is enabled. 107.IP • 108They expect a parsed 109.CW /cfg/pxe/\fIether 110or 111.CW plan9.ini 112file to be present at physical address 113.CW 0x1200 114and that 115.I 9load 116will have added entries to it describing 117any disk partitions found. 118(\c 119.I 9boot 120does not do this and so kernels loaded by it 121that care about disk partitions will need 122.CW readparts= 123in 124.CW plan9.ini .) 125.IP • 126They expect automatic power management information obtained from the 127BIOS to be present in the first megabyte. 128.IP • 129Our 130.CW amd64 131kernels (9k, Nix, etc.) 132also expect a Gnu 133.I multiboot 134header containing any arguments and a memory map. 135. 136.SH 137Non-Requirements 138.IP • 3 139The bootstraps should ignore secondary processors, leaving them in reset. 140.IP • 141The bootstraps need not do anything with floating point. 142. 143.SH 144Techniques and Tricks 145.LP 146Our new bootstraps are stripped-down Plan 9 PC kernels 147without system calls and user mode processes. 148They share the vast majority of their code with the ordinary PC kernels; 149about 10,000 lines of C are new or different in the bootstraps. 150In particular, they use the ordinary PC kernel's device drivers, 151unmodified. 152.I 9boot 153loads kernels via PXE; 154.I 9load 155loads kernels from disk. 156This is more specialised than the old all-in-one 157.I 9load . 158.LP 159From protected mode, 160the bootstraps initially enable paging for their own use. 161Before jumping to the loaded kernel, they revert 162to 32-bit protected mode, providing a known initial CPU state 163for the kernels. 164.LP 165Self-decompression of the bootstraps 166can help to relieve the 512KB/640KB payload limit. 167Russ Cox's decompressing header* code is about 9K all told, 168.FS 169* see 170.CW http://plan9.bell-labs.com/wiki/plan9/Replacing_9load 171.FE 172including BIOS calls to get APM and E820 memory map info. 173We only bother with this currently for 174.I 9boot . 175The bootstraps also will decompress 176.I gzip -ped 177kernels loaded from disk, 178mainly for CD or floppy booting, where they are limits on kernel size. 179.LP 180Figure 1 shows the memory map in effect while the bootstraps run. 181.KF 182.TS 183center ; 184cb s , 185n cw(3i) . 186Figure 1: Layout of physical memory during bootstrapping 187.sp 0.3v 1880 misc., including bios data area 189_ 19031K T{ 191start of pxe decomp + compressed 9boot. 192decompresses to 9MB. 193T} 194_ 19564K T{ 196start of pbs 197T} 198_ 199512K pxe loader from ROM 200_ 201640K UMB; device ROMs 202_ 2031M kernel 204_ 2059M T{ 2069boot after decomp. 207(decompresses kernel.gz at 13M.) 208loads kernel at 1M. 209T} 210_ 21113M (kernel.gz) 212_ 21315M no-man's land 214_ 21516M malloc arena for 9boot 216\&... 217.TE 218.KE 219.LP 220Our USB stack (at least 3 HCI drivers plus user-mode drivers, 221implying system calls and user-mode support) is too big 222to fit in the first 640KB, 223so the bootstraps try to get BIOSes to read from USB devices 224and some of them do. 225.LP 226We strongly prefer PXE booting; disk booting is a poor second. 227PXE booting minimises the number of copies of kernels that must 228be updated and ensures that machines boot the latest kernels. 229Reading via 9P (as user 230.I none ) 231would be even better: just read 232.CW /cfg/pxe/$ether 233and 234.CW /386/9pccpu . 235This would probably require adding 236.CW devmnt 237back into the bootstrap kernels. 238. 239.br 240.ne 6 241.SH 242Future 243.Os Horrors 244Directions 245.LP 246We haven't dealt at all with (U)EFI, `secure boot', GPTs nor GUIDs. 247We can use Plan 9 partition tables instead of GPTs to address disks larger 248than 2 TB. 249. 250.SH 251Lessons Learned 252.LP 253A disabled A20 line can masquerade as all sorts of baffling problems. 254It is well worth ensuring that it is truly enabled. 255.LP 256Virtual-machine hypervisors can be good test-beds and provide 257better crash diagnostics than the 258blank screen you get on real hardware, 259but they can also mislead 260(e.g., 261.CW amd64 262kernels on Virtualbox, 263Vmware 7 on Ubuntu 12.04). 264.LP 265All of these bootstrap programs and the BIOS (and POST) can be avoided, 266once Plan 9 is running, 267by using 268.CW /dev/reboot 269as packaged up in 270.I fshalt (8), 271which is much faster. 272