1.\" $NetBSD: file.1,v 1.25 2020/06/15 00:37:24 christos Exp $ 2.\" 3.\" $File: file.man,v 1.140 2020/06/07 17:41:07 christos Exp $ 4.Dd June 7, 2020 5.Dt FILE 1 6.Os 7.Sh NAME 8.Nm file 9.Nd determine file type 10.Sh SYNOPSIS 11.Nm 12.Bk -words 13.Op Fl bcdEhiklLNnprsSvzZ0 14.Op Fl Fl apple 15.Op Fl Fl exclude-quiet 16.Op Fl Fl extension 17.Op Fl Fl mime-encoding 18.Op Fl Fl mime-type 19.Op Fl e Ar testname 20.Op Fl F Ar separator 21.Op Fl f Ar namefile 22.Op Fl m Ar magicfiles 23.Op Fl P Ar name=value 24.Ar 25.Ek 26.Nm 27.Fl C 28.Op Fl m Ar magicfiles 29.Nm 30.Op Fl Fl help 31.Sh DESCRIPTION 32This manual page documents version 5.39 of the 33.Nm 34command. 35.Pp 36.Nm 37tests each argument in an attempt to classify it. 38There are three sets of tests, performed in this order: 39filesystem tests, magic tests, and language tests. 40The 41.Em first 42test that succeeds causes the file type to be printed. 43.Pp 44The type printed will usually contain one of the words 45.Em text 46(the file contains only 47printing characters and a few common control 48characters and is probably safe to read on an 49.Dv ASCII 50terminal), 51.Em executable 52(the file contains the result of compiling a program 53in a form understandable to some 54.Tn UNIX 55kernel or another), 56or 57.Em data 58meaning anything else (data is usually 59.Dq binary 60or non-printable). 61Exceptions are well-known file formats (core files, tar archives) 62that are known to contain binary data. 63When modifying magic files or the program itself, make sure to 64.Em "preserve these keywords" . 65Users depend on knowing that all the readable files in a directory 66have the word 67.Dq text 68printed. 69Don't do as Berkeley did and change 70.Dq shell commands text 71to 72.Dq shell script . 73.Pp 74The filesystem tests are based on examining the return from a 75.Xr stat 2 76system call. 77The program checks to see if the file is empty, 78or if it's some sort of special file. 79Any known file types appropriate to the system you are running on 80(sockets, symbolic links, or named pipes (FIFOs) on those systems that 81implement them) 82are intuited if they are defined in the system header file 83.In sys/stat.h . 84.Pp 85The magic tests are used to check for files with data in 86particular fixed formats. 87The canonical example of this is a binary executable (compiled program) 88.Dv a.out 89file, whose format is defined in 90.In elf.h , 91.In a.out.h 92and possibly 93.In exec.h 94in the standard include directory. 95These files have a 96.Dq "magic number" 97stored in a particular place 98near the beginning of the file that tells the 99.Tn UNIX 100operating system 101that the file is a binary executable, and which of several types thereof. 102The concept of a 103.Dq "magic" 104has been applied by extension to data files. 105Any file with some invariant identifier at a small fixed 106offset into the file can usually be described in this way. 107The information identifying these files is read from the compiled 108magic file 109.Pa /usr/share/misc/magic.mgc , 110or the files in the directory 111.Pa /usr/share/misc/magic 112if the compiled file does not exist. 113In addition, if 114.Pa $HOME/.magic.mgc 115or 116.Pa $HOME/.magic 117exists, it will be used in preference to the system magic files. 118.Pp 119If a file does not match any of the entries in the magic file, 120it is examined to see if it seems to be a text file. 121ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets 122(such as those used on Macintosh and IBM PC systems), 123UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC 124character sets can be distinguished by the different 125ranges and sequences of bytes that constitute printable text 126in each set. 127If a file passes any of these tests, its character set is reported. 128ASCII, ISO-8859-x, UTF-8, and extended-ASCII files are identified 129as 130.Dq text 131because they will be mostly readable on nearly any terminal; 132UTF-16 and EBCDIC are only 133.Dq character data 134because, while 135they contain text, it is text that will require translation 136before it can be read. 137In addition, 138.Nm 139will attempt to determine other characteristics of text-type files. 140If the lines of a file are terminated by CR, CRLF, or NEL, instead 141of the Unix-standard LF, this will be reported. 142Files that contain embedded escape sequences or overstriking 143will also be identified. 144.Pp 145Once 146.Nm 147has determined the character set used in a text-type file, 148it will 149attempt to determine in what language the file is written. 150The language tests look for particular strings (cf. 151.In names.h ) 152that can appear anywhere in the first few blocks of a file. 153For example, the keyword 154.Em .br 155indicates that the file is most likely a 156.Xr troff 1 157input file, just as the keyword 158.Em struct 159indicates a C program. 160These tests are less reliable than the previous 161two groups, so they are performed last. 162The language test routines also test for some miscellany 163(such as 164.Xr tar 1 165archives, JSON files). 166.Pp 167Any file that cannot be identified as having been written 168in any of the character sets listed above is simply said to be 169.Dq data . 170.Sh OPTIONS 171.Bl -tag -width indent 172.It Fl Fl apple 173Causes the file command to output the file type and creator code as 174used by older MacOS versions. 175The code consists of eight letters, 176the first describing the file type, the latter the creator. 177This option works properly only for file formats that have the 178apple-style output defined. 179.It Fl b , Fl Fl brief 180Do not prepend filenames to output lines (brief mode). 181.It Fl C , Fl Fl compile 182Write a 183.Pa magic.mgc 184output file that contains a pre-parsed version of the magic file or directory. 185.It Fl c , Fl Fl checking-printout 186Cause a checking printout of the parsed form of the magic file. 187This is usually used in conjunction with the 188.Fl m 189flag to debug a new magic file before installing it. 190.It Fl d 191Prints internal debugging information to stderr. 192.It Fl E 193On filesystem errors (file not found etc), instead of handling the error 194as regular output as POSIX mandates and keep going, issue an error message 195and exit. 196.It Fl e , Fl Fl exclude Ar testname 197Exclude the test named in 198.Ar testname 199from the list of tests made to determine the file type. 200Valid test names are: 201.Bl -tag -width compress 202.It apptype 203.Dv EMX 204application type (only on EMX). 205.It ascii 206Various types of text files (this test will try to guess the text 207encoding, irrespective of the setting of the 208.Sq encoding 209option). 210.It encoding 211Different text encodings for soft magic tests. 212.It tokens 213Ignored for backwards compatibility. 214.It cdf 215Prints details of Compound Document Files. 216.It compress 217Checks for, and looks inside, compressed files. 218.It csv 219Checks Comma Separated Value files. 220.It elf 221Prints ELF file details, provided soft magic tests are enabled and the 222elf magic is found. 223.It json 224Examines JSON (RFC-7159) files by parsing them for compliance. 225.It soft 226Consults magic files. 227.It tar 228Examines tar files by verifying the checksum of the 512 byte tar header. 229Excluding this test can provide more detailed content description by using 230the soft magic method. 231.It text 232A synonym for 233.Sq ascii . 234.El 235.It Fl Fl exclude-quiet 236Like 237.Fl Fl exclude 238but ignore tests that 239.Nm 240does not know about. 241This is intended for compatilibity with older versions of 242.Nm . 243.It Fl Fl extension 244Print a slash-separated list of valid extensions for the file type found. 245.It Fl F , Fl Fl separator Ar separator 246Use the specified string as the separator between the filename and the 247file result returned. 248Defaults to 249.Sq \&: . 250.It Fl f , Fl Fl files-from Ar namefile 251Read the names of the files to be examined from 252.Ar namefile 253(one per line) 254before the argument list. 255Either 256.Ar namefile 257or at least one filename argument must be present; 258to test the standard input, use 259.Sq - 260as a filename argument. 261Please note that 262.Ar namefile 263is unwrapped and the enclosed filenames are processed when this option is 264encountered and before any further options processing is done. 265This allows one to process multiple lists of files with different command line 266arguments on the same 267.Nm 268invocation. 269Thus if you want to set the delimiter, you need to do it before you specify 270the list of files, like: 271.Dq Fl F Ar @ Fl f Ar namefile , 272instead of: 273.Dq Fl f Ar namefile Fl F Ar @ . 274.It Fl h , Fl Fl no-dereference 275option causes symlinks not to be followed 276(on systems that support symbolic links). 277This is the default if the environment variable 278.Dv POSIXLY_CORRECT 279is not defined. 280.It Fl i , Fl Fl mime 281Causes the file command to output mime type strings rather than the more 282traditional human readable ones. 283Thus it may say 284.Sq text/plain; charset=us-ascii 285rather than 286.Dq ASCII text . 287.It Fl Fl mime-type , Fl Fl mime-encoding 288Like 289.Fl i , 290but print only the specified element(s). 291.It Fl k , Fl Fl keep-going 292Don't stop at the first match, keep going. 293Subsequent matches will be 294have the string 295.Sq "\[rs]012\- " 296prepended. 297(If you want a newline, see the 298.Fl r 299option.) 300The magic pattern with the highest strength (see the 301.Fl l 302option) comes first. 303.It Fl l , Fl Fl list 304Shows a list of patterns and their strength sorted descending by 305.Xr magic 5 306strength 307which is used for the matching (see also the 308.Fl k 309option). 310.It Fl L , Fl Fl dereference 311option causes symlinks to be followed, as the like-named option in 312.Xr ls 1 313(on systems that support symbolic links). 314This is the default if the environment variable 315.Ev POSIXLY_CORRECT 316is defined. 317.It Fl m , Fl Fl magic-file Ar magicfiles 318Specify an alternate list of files and directories containing magic. 319This can be a single item, or a colon-separated list. 320If a compiled magic file is found alongside a file or directory, 321it will be used instead. 322.It Fl N , Fl Fl no-pad 323Don't pad filenames so that they align in the output. 324.It Fl n , Fl Fl no-buffer 325Force stdout to be flushed after checking each file. 326This is only useful if checking a list of files. 327It is intended to be used by programs that want filetype output from a pipe. 328.It Fl p , Fl Fl preserve-date 329On systems that support 330.Xr utime 3 331or 332.Xr utimes 2 , 333attempt to preserve the access time of files analyzed, to pretend that 334.Nm 335never read them. 336.It Fl P , Fl Fl parameter Ar name=value 337Set various parameter limits. 338.Bl -column "elf_phnum" "Default" "XXXXXXXXXXXXXXXXXXXXXXXXXXX" -offset indent 339.It Sy "Name" Ta Sy "Default" Ta Sy "Explanation" 340.It Li bytes Ta 1048576 Ta max number of bytes to read from file 341.It Li elf_notes Ta 256 Ta max ELF notes processed 342.It Li elf_phnum Ta 2048 Ta max ELF program sections processed 343.It Li elf_shnum Ta 32768 Ta max ELF sections processed 344.It Li indir Ta 50 Ta recursion limit for indirect magic 345.It Li name Ta 50 Ta use count limit for name/use magic 346.It Li regex Ta 8192 Ta length limit for regex searches 347.El 348.It Fl r , Fl Fl raw 349Don't translate unprintable characters to \eooo. 350Normally 351.Nm 352translates unprintable characters to their octal representation. 353.It Fl s , Fl Fl special-files 354Normally, 355.Nm 356only attempts to read and determine the type of argument files which 357.Xr stat 2 358reports are ordinary files. 359This prevents problems, because reading special files may have peculiar 360consequences. 361Specifying the 362.Fl s 363option causes 364.Nm 365to also read argument files which are block or character special files. 366This is useful for determining the filesystem types of the data in raw 367disk partitions, which are block special files. 368This option also causes 369.Nm 370to disregard the file size as reported by 371.Xr stat 2 372since on some systems it reports a zero size for raw disk partitions. 373.It Fl S , Fl Fl no-sandbox 374On systems where libseccomp 375.Pa ( https://github.com/seccomp/libseccomp ) 376is available, the 377.Fl S 378flag disables sandboxing which is enabled by default. 379This option is needed for file to execute external decompressing programs, 380i.e. when the 381.Fl z 382flag is specified and the built-in decompressors are not available. 383On systems where sandboxing is not available, this option has no effect. 384.It Fl v , Fl Fl version 385Print the version of the program and exit. 386.It Fl z , Fl Fl uncompress 387Try to look inside compressed files. 388.It Fl Z , Fl Fl uncompress-noreport 389Try to look inside compressed files, but report information about the contents 390only not the compression. 391.It Fl 0 , Fl Fl print0 392Output a null character 393.Sq \e0 394after the end of the filename. 395Nice to 396.Xr cut 1 397the output. 398This does not affect the separator, which is still printed. 399.Pp 400If this option is repeated more than once, then 401.Nm 402prints just the filename followed by a NUL followed by the description 403(or ERROR: text) followed by a second NUL for each entry. 404.It Fl -help 405Print a help message and exit. 406.El 407.Sh ENVIRONMENT 408The environment variable 409.Ev MAGIC 410can be used to set the default magic file name. 411If that variable is set, then 412.Nm 413will not attempt to open 414.Pa $HOME/.magic . 415.Nm 416adds 417.Dq Pa .mgc 418to the value of this variable as appropriate. 419The environment variable 420.Ev POSIXLY_CORRECT 421controls (on systems that support symbolic links), whether 422.Nm 423will attempt to follow symlinks or not. 424If set, then 425.Nm 426follows symlink, otherwise it does not. 427This is also controlled by the 428.Fl L 429and 430.Fl h 431options. 432.Sh FILES 433.Bl -tag -width /usr/share/misc/magic.mgc -compact 434.It Pa /usr/share/misc/magic.mgc 435Default compiled list of magic. 436.It Pa /usr/share/misc/magic 437Directory containing default magic files. 438.El 439.Sh EXIT STATUS 440.Nm 441will exit with 442.Dv 0 443if the operation was successful or 444.Dv >0 445if an error was encountered. 446The following errors cause diagnostic messages, but don't affect the program 447exit code (as POSIX requires), unless 448.Fl E 449is specified: 450.Bl -bullet -compact -offset indent 451.It 452A file cannot be found 453.It 454There is no permission to read a file 455.It 456The file type cannot be determined 457.El 458.Sh EXAMPLES 459.Bd -literal -offset indent 460$ file file.c file /dev/{wd0a,hda} 461file.c: C program text 462file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 463 dynamically linked (uses shared libs), stripped 464/dev/wd0a: block special (0/0) 465/dev/hda: block special (3/0) 466 467$ file -s /dev/wd0{b,d} 468/dev/wd0b: data 469/dev/wd0d: x86 boot sector 470 471$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} 472/dev/hda: x86 boot sector 473/dev/hda1: Linux/i386 ext2 filesystem 474/dev/hda2: x86 boot sector 475/dev/hda3: x86 boot sector, extended partition table 476/dev/hda4: Linux/i386 ext2 filesystem 477/dev/hda5: Linux/i386 swap file 478/dev/hda6: Linux/i386 swap file 479/dev/hda7: Linux/i386 swap file 480/dev/hda8: Linux/i386 swap file 481/dev/hda9: empty 482/dev/hda10: empty 483 484$ file -i file.c file /dev/{wd0a,hda} 485file.c: text/x-c 486file: application/x-executable 487/dev/hda: application/x-not-regular-file 488/dev/wd0a: application/x-not-regular-file 489 490.Ed 491.Sh SEE ALSO 492.Xr hexdump 1 , 493.Xr od 1 , 494.Xr strings 1 , 495.Xr magic 5 496.Sh STANDARDS CONFORMANCE 497This program is believed to exceed the System V Interface Definition 498of FILE(CMD), as near as one can determine from the vague language 499contained therein. 500Its behavior is mostly compatible with the System V program of the same name. 501This version knows more magic, however, so it will produce 502different (albeit more accurate) output in many cases. 503.\" URL: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html 504.Pp 505The one significant difference 506between this version and System V 507is that this version treats any white space 508as a delimiter, so that spaces in pattern strings must be escaped. 509For example, 510.Bd -literal -offset indent 511\*[Gt]10 string language impress\ (imPRESS data) 512.Ed 513.Pp 514in an existing magic file would have to be changed to 515.Bd -literal -offset indent 516\*[Gt]10 string language\e impress (imPRESS data) 517.Ed 518.Pp 519In addition, in this version, if a pattern string contains a backslash, 520it must be escaped. 521For example 522.Bd -literal -offset indent 5230 string \ebegindata Andrew Toolkit document 524.Ed 525.Pp 526in an existing magic file would have to be changed to 527.Bd -literal -offset indent 5280 string \e\ebegindata Andrew Toolkit document 529.Ed 530.Pp 531SunOS releases 3.2 and later from Sun Microsystems include a 532.Nm 533command derived from the System V one, but with some extensions. 534This version differs from Sun's only in minor ways. 535It includes the extension of the 536.Sq \*[Am] 537operator, used as, 538for example, 539.Bd -literal -offset indent 540\*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped 541.Ed 542.Sh SECURITY 543On systems where libseccomp 544.Pa ( https://github.com/seccomp/libseccomp ) 545is available, 546.Nm 547is enforces limiting system calls to only the ones necessary for the 548operation of the program. 549This enforcement does not provide any security benefit when 550.Nm 551is asked to decompress input files running external programs with 552the 553.Fl z 554option. 555To enable execution of external decompressors, one needs to disable 556sandboxing using the 557.Fl S 558flag. 559.Sh MAGIC DIRECTORY 560The magic file entries have been collected from various sources, 561mainly USENET, and contributed by various authors. 562Christos Zoulas (address below) will collect additional 563or corrected magic file entries. 564A consolidation of magic file entries 565will be distributed periodically. 566.Pp 567The order of entries in the magic file is significant. 568Depending on what system you are using, the order that 569they are put together may be incorrect. 570If your old 571.Nm 572command uses a magic file, 573keep the old magic file around for comparison purposes 574(rename it to 575.Pa /usr/share/misc/magic.orig ) . 576.Sh HISTORY 577There has been a 578.Nm 579command in every 580.Dv UNIX since at least Research Version 4 581(man page dated November, 1973). 582The System V version introduced one significant major change: 583the external list of magic types. 584This slowed the program down slightly but made it a lot more flexible. 585.Pp 586This program, based on the System V version, 587was written by Ian Darwin 588.Aq ian@darwinsys.com 589without looking at anybody else's source code. 590.Pp 591John Gilmore revised the code extensively, making it better than 592the first version. 593Geoff Collyer found several inadequacies 594and provided some magic file entries. 595Contributions of the 596.Sq \*[Am] 597operator by Rob McMahon, 598.Aq cudcv@warwick.ac.uk , 5991989. 600.Pp 601Guy Harris, 602.Aq guy@netapp.com , 603made many changes from 1993 to the present. 604.Pp 605Primary development and maintenance from 1990 to the present by 606Christos Zoulas 607.Aq christos@astron.com . 608.Pp 609Altered by Chris Lowth 610.Aq chris@lowth.com , 6112000: handle the 612.Fl i 613option to output mime type strings, using an alternative 614magic file and internal logic. 615.Pp 616Altered by Eric Fischer 617.Aq enf@pobox.com , 618July, 2000, 619to identify character codes and attempt to identify the languages 620of non-ASCII files. 621.Pp 622Altered by Reuben Thomas 623.Aq rrt@sc3d.org , 6242007-2011, to improve MIME support, merge MIME and non-MIME magic, 625support directories as well as files of magic, apply many bug fixes, 626update and fix a lot of magic, improve the build system, improve the 627documentation, and rewrite the Python bindings in pure Python. 628.Pp 629The list of contributors to the 630.Sq magic 631directory (magic files) 632is too long to include here. 633You know who you are; thank you. 634Many contributors are listed in the source files. 635.Sh LEGAL NOTICE 636Copyright (c) Ian F. Darwin, Toronto, Canada, 1986-1999. 637Covered by the standard Berkeley Software Distribution copyright; see the file 638COPYING in the source distribution. 639.Pp 640The files 641.Pa tar.h 642and 643.Pa is_tar.c 644were written by John Gilmore from his public-domain 645.Xr tar 1 646program, and are not covered by the above license. 647.Sh BUGS 648Please report bugs and send patches to the bug tracker at 649.Pa https://bugs.astron.com/ 650or the mailing list at 651.Aq file@astron.com 652(visit 653.Pa https://mailman.astron.com/mailman/listinfo/file 654first to subscribe). 655.Sh TODO 656Fix output so that tests for MIME and APPLE flags are not needed all 657over the place, and actual output is only done in one place. 658This needs a design. 659Suggestion: push possible outputs on to a list, then pick the 660last-pushed (most specific, one hopes) value at the end, or 661use a default if the list is empty. 662This should not slow down evaluation. 663.Pp 664The handling of 665.Dv MAGIC_CONTINUE 666and printing \e012- between entries is clumsy and complicated; refactor 667and centralize. 668.Pp 669Some of the encoding logic is hard-coded in encoding.c and can be moved 670to the magic files if we had a !:charset annotation 671.Pp 672Continue to squash all magic bugs. 673See Debian BTS for a good source. 674.Pp 675Store arbitrarily long strings, for example for %s patterns, so that 676they can be printed out. 677Fixes Debian bug #271672. 678This can be done by allocating strings in a string pool, storing the 679string pool at the end of the magic file and converting all the string 680pointers to relative offsets from the string pool. 681.Pp 682Add syntax for relative offsets after current level (Debian bug #466037). 683.Pp 684Make file -ki work, i.e. give multiple MIME types. 685.Pp 686Add a zip library so we can peek inside Office2007 documents to 687print more details about their contents. 688.Pp 689Add an option to print URLs for the sources of the file descriptions. 690.Pp 691Combine script searches and add a way to map executable names to MIME 692types (e.g. have a magic value for !:mime which causes the resulting 693string to be looked up in a table). 694This would avoid adding the same magic repeatedly for each new 695hash-bang interpreter. 696.Pp 697When a file descriptor is available, we can skip and adjust the buffer 698instead of the hacky buffer management we do now. 699.Pp 700Fix 701.Dq name 702and 703.Dq use 704to check for consistency at compile time (duplicate 705.Dq name , 706.Dq use 707pointing to undefined 708.Dq name 709). 710Make 711.Dq name 712/ 713.Dq use 714more efficient by keeping a sorted list of names. 715Special-case ^ to flip endianness in the parser so that it does not 716have to be escaped, and document it. 717.Pp 718If the offsets specified internally in the file exceed the buffer size 719( 720.Dv HOWMANY 721variable in file.h), then we don't seek to that offset, but we give up. 722It would be better if buffer managements was done when the file descriptor 723is available so move around the file. 724One must be careful though because this has performance (and thus security 725considerations). 726.Sh AVAILABILITY 727You can obtain the original author's latest version by anonymous FTP 728on 729.Pa ftp.astron.com 730in the directory 731.Pa /pub/file/file-X.YZ.tar.gz . 732