1$NetBSD: UPDATING,v 1.224 2011/04/10 22:46:09 dholland Exp $ 2 3This file (UPDATING) is intended to be a brief reference to recent 4changes that might cause problems in the build process, and a guide for 5what to do if something doesn't work. 6 7For a more detailed description of the recommended way to build NetBSD 8using build.sh, see the BUILDING file. 9 10Note that much of the advice in this UPDATING file was written before 11build.sh existed. Nevertheless, the advice here may be useful for 12working around specific problems with build.sh. 13 14See also: BUILDING, build.sh, Makefile. 15 16Recent changes: 17^^^^^^^^^^^^^^^ 1820110410: 19 The configuration of src/tools/gcc has changed. To do an 20 update build you have to clean both tools/binutils and 21 tools/gcc by hand. 22 2320110328: 24 Building the Xorg binary was moved into a subdirectory to fix 25 ordering issues with "make all". It may be necessary to remove 26 the OBJDIR for external/mit/xorg/server/xorg-server/hw/xfree86 27 if your update build fails, as the "Xorg" entry there is now a 28 directory. 29 3020110121: 31 Assembler files no longer use -traditional-cpp. This can break 32 the build of individual parts of the tree. This is handled 33 correctly by build.sh. Manual builds have to update /usr/share/mk 34 and re-run config(1) for any kernel configurations as needed. 35 3620101217: 37 The tcpdump(8) program was changed to drop privileges and chroot(2) 38 by default. It may be necessary to manually update passwd(5) and 39 group(5) in order to make the program work with existing setups. 40 4120101125: 42 The latest changes to setenv(3) dissallow setting environment 43 variables with names that contain '='. Revision 1.18 of env.c 44 assumed that this was allowed. Installing a new libc with an 45 old copy of /usr/bin/env causes env x=1 printenv | grep x= to 46 break which affects the autoconf tests for dependency finding, 47 so building gcc will end up printing: 48 checking dependency style of gcc... none 49 configure: error: no usable dependency style found 50 Fix it by rebuilding and re-installing env. 51 5220101119: 53 Recent Xorg updates in xsrc/external/mit/ may cause various build 54 or run-time problems. Delete your entire DESTDIR and OBJDIR if you 55 have any build problems with xsrc, or problems with mismatched 56 versions between xorg-server and drivers. 57 5820100604: 59 The update of ATF to 0.9 causes old tests written in shell to fail 60 unless they are rebuilt. If you are building with MKUPDATE=yes, 61 you need to clean the src/external/bsd/atf/tests/ and the src/tests/ 62 trees by hand. 63 6420100522: 65 Recent Xorg updates in xsrc/external/mit/ will cause various build 66 problems. Delete your entire DESTDIR and OBJDIR if you have any 67 build problems with xsrc. 68 6920100522: 70 private section of <ctype.h> was splitted, and now mklocale(1) 71 include ctype_local.h, so you have to make cleandir in tools/mklocale. 72 7320100520: 74 The location of the xkb compiled descriptions has changed. Please 75 remove usr/X11R7/lib/X11/xkb/compiled from your $DESTDIR. 76 7720100222: 78 The shared objects file extension has been changed from .so to 79 .pico, in order to avoid conflicts with shared libraries names 80 libXX.so. All now stale regular .so files can be removed from 81 your object directories. 82 8320100204: 84 The termcap database has been removed from the sources, 85 but has not been marked obsolete so it is not removed 86 from the system when upgrading. 87 As such, you will need to remove them from your object 88 and destination directories. 89 9020091101: 91 After updating, it may be necessary to make the 'cleandir' 92 target in src/tools/yacc/ and in src/usr.bin/yacc/ before a 93 'build.sh -u tools' or 'build.sh -u distribution'. Ditto 94 src/tools/lex/ and src/usr.bin/lex/. 95 9620091001: 97 On amd64 you must rebuild tools (to get the new binutils) 98 before building a kernel, or the build fails on cpufunc.S. 99 10020091001: 101 An error will create a ./usr/X11R7/lib/X11/xkb/compiled/xkb 102 symlink, failing the build. Delete the link, and the subdir 103 it is in, and retry your build. 104 10520090718: 106 libc build changed so that strchr() provides the extra entry 107 point for index(). Update build of libc.a (and libc_pic.a) 108 may fail because the archive contains the unwanted index.o. 109 (Similarly for strrchr() and rindex().) 110 11120090709: 112 Native Xorg was upgraded again. Builds will probably fail again 113 without a clean objdir, at least for src/external/mit/xorg. 114 11520090616: 116 Native Xorg was upgraded. Builds will need a clean objdir for 117 src/external/mit/xorg. Upgrading a system from sets will not 118 work properly yet as the /usr/X11R7/lib/X11/xkb/symbols/pc 119 subdirectory has been changed into a file, and this needs to 120 be manually "rm -r"'ed before installing xbase.tgz. 121 12220090501: 123 Several new functions were added to string.h/libc, and this 124 can cause autoconf problems during the tool build for people 125 who fail to clean out their tools objects properly. If you 126 note messages about stpcpy, stpncpy or strnlen accompanying a 127 failure during the tool build, clean out all your tools 128 objects and start again. 129 13020090325: 131 The i386 port was switched to i486 default toolchain. This requires 132 cleaning your src/tools directory and $TOOLDIR and rebuilding them. 133 13420090126: 135 The __posix_fadvise50 system call changed assembly stub type. You 136 need to 'rm -f __posix_fadvise50.*' in the libc build directory 137 to avoid using the old assembly stub. 138 13920090202: 140 pkg_install now depends on the pkgdb cache for automatic conflict 141 detection. It is recommented to rebuild the cache with 142 ``pkg_admin rebuild''. 143 14420090110: 145 time_t and dev_t have been bumped to 64 bit quantities. To upgrade: 146 1. Make sure your kernel has COMPAT_50 in it. Build and install. 147 This is needed even in the MODULAR kernel because there is 148 conditionally compiled code in rtsock.c. 149 2. make sure build.sh completes and the binaries in a chroot work 150 before installing. 151 3. If you don't use build.sh and you build directly to root, and 152 your build breaks in the middle, don't despair. Make sure headers 153 are installed properly, and start building libraries first libc 154 and libutil, install them and then continue building all the 155 libraries in src/lib and src/gnu/lib and install them. Once 156 the new libraries are installed, you can restart the build. 157 4. If you compile packages and you notice link time warnings, 158 rebuild the required packages to update their shared libraries. 159 Any package you rebuild will require rebuilding all the packages 160 that depend on it. 161 5. Next time you run pwd_mkdb with the new binary, the file 162 will be upgraded and it will not be backwards compatible. 163 6. The utmpx/wtmpx files (/var/run/utmpx and /var/log/wtmpx, see 164 lastlogx(5)) have been versioned, and there is a heuristic 165 for utmp. You are better off removing the old files after 166 upgrading. The automated clearing of /var/run during 167 boot, and the automated rotating of files in /var/log by 168 newsyslog(8), may mean that you do not have to remove the 169 files manually. 170 7. The optional accounting file (/var/account/acct, see 171 accton(8)) has not been versioned, and will need to be 172 removed. The automatic rotation of the accounting file by 173 /etc/daily limits the bad consequences of failure to remove 174 the file. 175 8. Application software that writes time_t to binary files on 176 disk will break or need attention. Most notably: if you are 177 using PostgreSQL < 8.4, you need to dump your databases, 178 rebuild PostgreSQL with the new time_t, then restore. 179 18020081219: 181 config(1) has been updated, and one of the files it creates - 182 swapnetbsd.o - has changed format. You need to rebuild config 183 (done automatically by build.sh) and then you need to rerun 184 config on all kernel configuration files before rebuilding those 185 kernels. 186 18720081205: 188 If you build with MKX11=no, you should remove /etc/rc.d/xdm and 189 /etc/rc.d/xfs from DESTDIR because those files were moved to the xetc 190 set and will appear as extra files for MKX11=no update builds. 191 19220081122: 193 On i386, various kernel options(4) in GENERIC including 194 file systems have been disabled and moved into kernel modules. 195 Before trying a new GENERIC kernel, you have to prepare the 196 following files as well as a new GENERIC kernel: 197 198 - build and install kernel modules from src/sys/modules 199 200 - install the latest bootloader, which will load a module 201 for the file system from which the kernel is loaded automatically 202 203 If you have to load your kernel from a file system which is not of 204 the same type as the root file system, you have to load the necessary 205 file system module manually on the boot prompt or in the boot.cfg file. 206 20720080827: 208 If you built and installed a libc from sources between 209 2008/08/20 and 2008/08/26 you got a broken strtouq(3) 210 which results in false errors reported by lint(1). 211 Since this breaks the libc build itself, manual help is 212 needed -- lint must be disabled temporarily, e.g.: 213 $ (cd lib/libc && make MKLINT=no dependall install) 214 21520080813: 216 MKDEBUG build was broken because the .depend files did not know 217 about .go files. You need to remove all .depend files and rebuild. 218 21920080802: 220 A regression in binary compatibility for pthread_mutex_t has 221 been fixed. Unfortunately, the price is breaking compatibility 222 for -current. 223 224 Threaded programs (using libpthread) and C++ programs (using 225 libstdc++) compiled after 20070907 and before 20080802 need to 226 be recompiled. 227 228 One way to find affected pkgsrc packages: 229 230 $ grep REQUIRES=/usr/lib/libpthread /var/db/pkg/*/+BUILD_INFO 231 $ grep REQUIRES=/usr/lib/libstdc++ /var/db/pkg/*/+BUILD_INFO 232 23320080731: 234 WAPBL (metadata journaling support) has been added, but at this 235 time isn't backwards compatible with pre-WAPBL aware kernels 236 and userland (fsck_ffs in particular). Please make sure you 237 don't use a journaled filesystem with an older kernel/userland, 238 especially an uncleanly mounted journaled filesystem. WAPBL 239 also requires the super block to be in the UFS2 format. You 240 can use fsck_ffs -c 4 to update the superblock format. 241 24220080721: 243 Assembler warnings are now fatal if $WARNS>0 and $NOGCCERROR 244 isn't defined. 245 24620080531: 247 The ioctl number of DRVSUSPENDDEV command on /dev/drvctl changed 248 from 125 (conflicted with DRVCTLCOMMAND) to 129. The drvctl(8) 249 utility needs to be rebuilt and reinstalled as older binaries 250 won't work correctly. The following sequence of commands: 251 252 $ (cd sys/sys/ && nbmake-$arch includes) 253 $ (cd sbin/drvctl/ && nbmake-$arch clean) 254 $ (cd sbin/drvctl/ && nbmake-$arch all) 255 256 leaves new drvctl utility in sbin/drvctl build directory. 257 25820080503: 259 The <bsd.lib.mk> variable MKPRIVATELIB was renamed to LIBISPRIVATE. 260 26120080521: 262 For a while, unprivileged UPDATE builds would fail to 263 succeed at the checkflist stage, complaining that 264 ${DESTDIR}/stand/<arch>/ did not exist. A fix for this 265 problem was committed to share/mk/bsd.kmodule.mk, revision 1.9. 266 If you already hit this problem, update the .mk file, 267 remove ${DESTDIR}/stand/<arch>, and re-run the build. 268 26920080303: 270 Linker warnings are now fatal if $WARNS>0. 271 27220080126: 273 The posix_fadvise system call has been changed from an assembly 274 stub, to a c file that calls an assembly stub. You need to 275 'rm -f posix_fadvise.* .depend' in the libc build directory to 276 avoid using the old assembly stub. 277 27820071209: 279 The acpiec(4) driver has been split into two attachments. If you 280 get ACPI errors before the attachment, please update your kernel 281 configuration file appropriately or see GENERIC for more details. 282 28320071115: 284 The it(4) driver has been renamed to itesio(4) and the old port 285 argument specified in the kernel configuration file is not valid 286 anymore. The itesio(4) driver now uses the Super I/O address port 287 rather than the EC address port. Please update your kernel 288 configuration file appropriately or see GENERIC for more details. 289 29020071028: 291 The pccons(4) driver has been removed from the NetBSD/shark port. 292 You need to update any custom kernel configuration file you have 293 to remove any references to pccons (which includes removing the 294 now useless XSERVER option) and replace them with the correct 295 entries for the wscons driver. See the GENERIC configuration file 296 for more details. 297 29820070913: 299 A latent bug in dhclient/dhcpd that caused it to be unable to 300 enumerate interfaces was fixed. The bug began to cause 301 problems after 20070911 when the kernel's SIOCGIFCONF 302 implementation was repaired. From 20070529 to 20070911 racoon 303 could not enumerate interfaces. (These are noted because 304 normal kernel/userspace version matching hygiene is not 305 sufficient to avoid this problem.) Ensure that both kernel 306 and userland are from after 20070913. 307 30820070703: 309 nbinstall has been renamed ${MACHINE_GNU_ARCH}-install. It 310 calls the target-specific strip program, and the logic to pass 311 down STRIP from make has been removed. This requires a 312 re-installation of tools. 313 31420070422: 315 The way OS emulations lookup filenames inside the emulation root 316 has been changed. Rather than modify the pathname (and copy back 317 to userspace) namei() and lookup() directly check the emulation 318 root. One side effect is that absolute symlinks inside the emulated 319 root file system will be relative to that file system - unless they 320 start /../ this is useful when the emulated root is a real install 321 that has such links. 322 This might affect symlinks that have been added to reference outside 323 the emulated root. 324 32520070412: 326 The pckbc driver on sgimips IP32 has been removed. Use macekbc 327 instead. See the GENERIC32_IP3x kernel configuration for an 328 example. 329 33020070319: 331 src/lib/libc/Makefile revision 1.129 broke libc and ld.elf_so 332 on many platforms due to incorrect flags settings. If you 333 updated and built after about 20070315, do "nbmake-$arch 334 cleandir" in src/lib/libc and src/libexec/ld.elf_so to force a 335 rebuild of object files that might have been built 336 incorrectly, and ensure that you have at least 337 src/lib/libc/Makefile 1.130. 338 33920070210: 340 src/sys/sys/{sa.h,savar.h} were removed. 341 find ${OBJDIR} \( -name .depend -o -name '*.d' \) -print \ 342 | xargs egrep -l '/sa.h|/savar.h' | xargs rm 343 will allow dependencies on those files to get get rebuilt 344 34520070209: 346 The threading model was changed when the newlock2 branch 347 was merged to NetBSD-current. If you boot with a new 348 kernel (version 4.99.10), then you also need a new pthread 349 library (/usr/lib/libpthread.so.0.7). If you boot with 350 an old kernel, then you need the old pthread library 351 (/usr/lib/libpthread.so.0.6). Provided you keep the kernel and 352 the pthread library in sync, old threaded applications should 353 continue to work with an old or new kernel. Note that named(8) 354 is the only threaded application in the base system. 355 35620061214: 357 Following the move of string_to_flags() and flags_to_string() 358 from the bin/ls/ sources to libutil, users doing UPDATE builds 359 will need to do a "make cleandir" in 360 tools/mtree/, tools/makefs/, tools/binstall/, tools/pax/, 361 bin/pax/, bin/ls/, usr.sbin/mtree/, usr.sbin/makefs/, 362 usr.bin/xinstall/, libexec/ftpd/, rescue/, as well 363 as the installation images in distrib/ 364 in order to excise stale references to the old stat_flags.h header 365 file in the ls sources -- stat_flags.h has been removed. 366 36720061108: 368 The configure script used in the src/tools/gcc compiler has been 369 changed to indicate that our libc has ssp support built-in and 370 does not depend on -lssp and -lssp-nonshared. You'll need to 371 make clean in src/tools/gcc first to rebuild the compiler. 372 37320061009: 374 The sysctl variables net.inet{,6}.tcp{,6}.newreno are no longer 375 available. Use net.inet{,6}.tcp{,6}.congctl.selected instead. 376 37720060814: 378 The vt, vidcconsole, kbd, and rpckbd drivers on acorn32 have been 379 withdrawn. Use vidcvideo and pckbd instead. See the GENERIC 380 kernel configuration for an example. X servers from the last 381 few years should cope. 382 38320060703: 384 MPACPI is no more. We always configure PCI interrupts using ACPI 385 if we have an ACPI kernel. The option MPACPI_SCANPCI has been renamed 386 to ACPI_SCANPCI. Thanks to work from fvdl. 387 38820060627: 389 socket(2) has changed, and its system call has been versioned. 390 For userlands with the old version of socket(2), make sure that 391 your kernel has 'options COMPAT_30' set, or else 'bad system call' 392 errors will result. 393 394Hints for a more successful build: 395^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 396 Use build.sh, but do not use its "expert mode": 397 This will automatically build the tools in the 398 correct order, and it will keep the tools and the 399 new build products from interfering with the running 400 system. This will allow you to ignore most of the 401 other advice in this file. 402 Build a new kernel first: 403 This makes sure that any new system calls or features 404 expected by the new userland will be present. This 405 helps to avoid critical errors when upgrading. 406 Use object directories: 407 This helps to keep stale object 408 files from polluting the build if a Makefile "forgets" 409 about one. It also makes it easier to clean up after 410 a build. It's also necessary if you want to use the 411 same source tree for multiple machines. 412 To use object directories with build.sh: 413 a) invoke build.sh with the "-M" or "-O" options. 414 To use object directories without using build.sh: 415 a) cd /usr/src ; make cleandir 416 b) Add "OBJMACHINE=yes" to /etc/mk.conf 417 c) Add "MKOBJDIRS=yes" to /etc/mk.conf 418 d) cd /usr/src ; make build 419 Note that running "make obj" in a directory will create 420 in obj.$MACHINE directory. 421 Build to a DESTDIR: 422 This helps to keep old installed files (especially libraries) 423 from interfering with the new build. 424 To build to a DESTDIR with build.sh, use the "-D" option. 425 To build to a DESTDIR without using build.sh, set the DESTDIR 426 environment variable before running make build. It should be 427 set to the pathname of an initially empty directory. 428 Problems: if you do not use build.sh, you might need to 429 update critical utilities without using DESTDIR since 430 nothing is executed from what is installed in DESTDIR. 431 (See critical utils, below.) 432 Build often: 433 This keeps critical utilities current enough to not choke 434 on any other part of the source tree that depends on up to 435 date functionality. If you use build.sh, you should not have 436 this problem. 437 438What to do if things don't work: 439^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 440When things don't work there is usually a few things that commonly 441should be done. 442 1) make includes 443 This should be done automatically by make build. 444 2) cd share/mk && make install 445 Again, automatically done by make build. 446 447Failsafe rebuild of a small part of the tree: 448^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 449To make sure you rebuild something correctly you want to do 450something like the following: 451 1) Make sure the includes and .mk files are up to date. 452 2) Make sure any program used to build the particular 453 utility is up to date. (yacc, lex, etc...) 454 3) cd ...path/to/util... 455 make cleandir 456 rm ...all obj directories... 457 make cleandir # yes, again 458 make obj 459 make depend && make 460 461Failsafe rebuild of the entire tree: 462^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 463If you really want to make sure the source tree is clean and 464ready for a build try the following. Note that sourcing /etc/mk.conf 465(a make(1) Makefile) in this manner is not right, and will not work 466for anyone who uses any make(1) features in /etc/mk.conf. 467 468---cut here--- 469#!/bin/sh 470. /etc/mk.conf 471 472if [ -z $NETBSDSRCDIR ] ; then 473 NETBSDSRCDIR=/usr/src 474fi 475if [ \! -d $NETBSDSRCDIR ] ; then 476 echo Unable to find sources 477 exit 1 478fi 479find $NETBSDSRCDIR -name \*.o -o -name obj.\* -o -name obj -exec rm \{\} \; 480 481if [ -z $BSDOBJDIR ] ; then 482 BSDOBJDIR=/usr/obj 483fi 484if [ -d $BSDOBJDIR ] ; then 485 rm -rf $BSDOBJDIR 486fi 487 488cd $NETBSDSRCDIR && make cleandir 489 490---cut here--- 491 492Critical utilities: 493^^^^^^^^^^^^^^^^^^^ 494 usr.bin/compile_et 495 usr.bin/make 496 usr.bin/yacc 497 usr.bin/lex 498 usr.bin/xlint 499 usr.bin/config 500 501Other problems and possible solutions: 502^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 503Symptom:Complaints involving a Makefile. 504Fix: Rebuild usr.bin/make: 505 cd usr.bin/make && make && make install 506 Or, a failsafe method if that doesn't work: 507 cd usr.bin/make && cc *.c */*.c -I . -o make && mv make /usr/bin 508 509Fix: Make sure .mk files are up to date. 510 cd share/mk && make install 511 512Symptom:Kernel `config' fails to configure any kernel, including GENERIC. 513Fix: Rebuild usr.bin/config 514 515Symptom: 516Fix: Rebuild usr.bin/yacc 517 518Symptom: 519Fix: Rebuild usr.bin/lex 520 521Symptom: 522Fix: rm /usr/lib/libbfd.a 523 524Symptom:Obsolete intermediate files are used during compilation 525Fix: Try the following sequence of commands in the directory in question. 526 make cleandir; rm `make print-objdir`; make cleandir; make obj 527 (If you built the tree without "make obj" in the past, obsolete files 528 may remain. The command tries to clean everything up) 529 530Symptom:.../sysinst/run.c:xx: warning: initialization from incompatible 531 pointer type 532Fix: Rebuild and install usr.bin/menuc 533 534Symptom:mklocale not found during build in share/locale/ctype 535Fix: Build and install usr.bin/mklocale 536 537Symptom:undefined reference to `__assert13' or `__unsetenv13' 538Fix: Rebuild and install lib/libc 539 540Symptom:usr.bin/config fails to build. 541Fix: Try building with -DMAKE_BOOTSTRAP added to CFLAGS in Makefile. 542 543Symptom:undefined reference to `getprogname' or `setprogname' 544Fix: Rebuild and install lib/libc 545 546Symptom:lint does not understand the '-X' option 547Fix: May need to build & install libs with NOLINT=1 before rebuilding lint 548 549Symptom:Update build fails in src/tools/gcc complaining that a variable 550 (e.g. CPPFLAGS) has changed since the previous run. 551Fix: Run "nbmake-${ARCH} clean" in src/tools/gcc or do a clean build. 552