1.\" $NetBSD: file.1,v 1.21 2018/04/15 19:45:32 christos Exp $ 2.\" 3.\" $File: file.man,v 1.129 2018/03/02 16:17:54 christos Exp $ 4.Dd March 2, 2018 5.Dt FILE 1 6.Os 7.Sh NAME 8.Nm file 9.Nd determine file type 10.Sh SYNOPSIS 11.Nm 12.Bk -words 13.Op Fl bcdEhiklLNnprsSvzZ0 14.Op Fl Fl apple 15.Op Fl Fl extension 16.Op Fl Fl mime-encoding 17.Op Fl Fl mime-type 18.Op Fl e Ar testname 19.Op Fl F Ar separator 20.Op Fl f Ar namefile 21.Op Fl m Ar magicfiles 22.Op Fl P Ar name=value 23.Ar 24.Ek 25.Nm 26.Fl C 27.Op Fl m Ar magicfiles 28.Nm 29.Op Fl Fl help 30.Sh DESCRIPTION 31This manual page documents version 5.33 of the 32.Nm 33command. 34.Pp 35.Nm 36tests each argument in an attempt to classify it. 37There are three sets of tests, performed in this order: 38filesystem tests, magic tests, and language tests. 39The 40.Em first 41test that succeeds causes the file type to be printed. 42.Pp 43The type printed will usually contain one of the words 44.Em text 45(the file contains only 46printing characters and a few common control 47characters and is probably safe to read on an 48.Dv ASCII 49terminal), 50.Em executable 51(the file contains the result of compiling a program 52in a form understandable to some 53.Tn UNIX 54kernel or another), 55or 56.Em data 57meaning anything else (data is usually 58.Dq binary 59or non-printable). 60Exceptions are well-known file formats (core files, tar archives) 61that are known to contain binary data. 62When modifying magic files or the program itself, make sure to 63.Em "preserve these keywords" . 64Users depend on knowing that all the readable files in a directory 65have the word 66.Dq text 67printed. 68Don't do as Berkeley did and change 69.Dq shell commands text 70to 71.Dq shell script . 72.Pp 73The filesystem tests are based on examining the return from a 74.Xr stat 2 75system call. 76The program checks to see if the file is empty, 77or if it's some sort of special file. 78Any known file types appropriate to the system you are running on 79(sockets, symbolic links, or named pipes (FIFOs) on those systems that 80implement them) 81are intuited if they are defined in the system header file 82.In sys/stat.h . 83.Pp 84The magic tests are used to check for files with data in 85particular fixed formats. 86The canonical example of this is a binary executable (compiled program) 87.Dv a.out 88file, whose format is defined in 89.In elf.h , 90.In a.out.h 91and possibly 92.In exec.h 93in the standard include directory. 94These files have a 95.Dq "magic number" 96stored in a particular place 97near the beginning of the file that tells the 98.Tn UNIX 99operating system 100that the file is a binary executable, and which of several types thereof. 101The concept of a 102.Dq "magic" 103has been applied by extension to data files. 104Any file with some invariant identifier at a small fixed 105offset into the file can usually be described in this way. 106The information identifying these files is read from the compiled 107magic file 108.Pa /usr/share/misc/magic.mgc , 109or the files in the directory 110.Pa /usr/share/misc/magic 111if the compiled file does not exist. 112In addition, if 113.Pa $HOME/.magic.mgc 114or 115.Pa $HOME/.magic 116exists, it will be used in preference to the system magic files. 117.Pp 118If a file does not match any of the entries in the magic file, 119it is examined to see if it seems to be a text file. 120ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets 121(such as those used on Macintosh and IBM PC systems), 122UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC 123character sets can be distinguished by the different 124ranges and sequences of bytes that constitute printable text 125in each set. 126If a file passes any of these tests, its character set is reported. 127ASCII, ISO-8859-x, UTF-8, and extended-ASCII files are identified 128as 129.Dq text 130because they will be mostly readable on nearly any terminal; 131UTF-16 and EBCDIC are only 132.Dq character data 133because, while 134they contain text, it is text that will require translation 135before it can be read. 136In addition, 137.Nm 138will attempt to determine other characteristics of text-type files. 139If the lines of a file are terminated by CR, CRLF, or NEL, instead 140of the Unix-standard LF, this will be reported. 141Files that contain embedded escape sequences or overstriking 142will also be identified. 143.Pp 144Once 145.Nm 146has determined the character set used in a text-type file, 147it will 148attempt to determine in what language the file is written. 149The language tests look for particular strings (cf. 150.In names.h ) 151that can appear anywhere in the first few blocks of a file. 152For example, the keyword 153.Em .br 154indicates that the file is most likely a 155.Xr troff 1 156input file, just as the keyword 157.Em struct 158indicates a C program. 159These tests are less reliable than the previous 160two groups, so they are performed last. 161The language test routines also test for some miscellany 162(such as 163.Xr tar 1 164archives). 165.Pp 166Any file that cannot be identified as having been written 167in any of the character sets listed above is simply said to be 168.Dq data . 169.Sh OPTIONS 170.Bl -tag -width indent 171.It Fl Fl apple 172Causes the file command to output the file type and creator code as 173used by older MacOS versions. 174The code consists of eight letters, 175the first describing the file type, the latter the creator. 176.It Fl b , Fl Fl brief 177Do not prepend filenames to output lines (brief mode). 178.It Fl C , Fl Fl compile 179Write a 180.Pa magic.mgc 181output file that contains a pre-parsed version of the magic file or directory. 182.It Fl c , Fl Fl checking-printout 183Cause a checking printout of the parsed form of the magic file. 184This is usually used in conjunction with the 185.Fl m 186flag to debug a new magic file before installing it. 187.It Fl d 188Prints internal debugging information to stderr. 189.It Fl E 190On filesystem errors (file not found etc), instead of handling the error 191as regular output as POSIX mandates and keep going, issue an error message 192and exit. 193.It Fl e , Fl Fl exclude Ar testname 194Exclude the test named in 195.Ar testname 196from the list of tests made to determine the file type. 197Valid test names are: 198.Bl -tag -width compress 199.It apptype 200.Dv EMX 201application type (only on EMX). 202.It ascii 203Various types of text files (this test will try to guess the text 204encoding, irrespective of the setting of the 205.Sq encoding 206option). 207.It encoding 208Different text encodings for soft magic tests. 209.It tokens 210Ignored for backwards compatibility. 211.It cdf 212Prints details of Compound Document Files. 213.It compress 214Checks for, and looks inside, compressed files. 215.It elf 216Prints ELF file details, provided soft magic tests are enabled and the 217elf magic is found. 218.It soft 219Consults magic files. 220.It tar 221Examines tar files by verifying the checksum of the 512 byte tar header. 222Excluding this test can provide more detailed content description by using 223the soft magic method. 224.It text 225A synonym for 226.Sq ascii . 227.El 228.It Fl Fl extension 229Print a slash-separated list of valid extensions for the file type found. 230.It Fl F , Fl Fl separator Ar separator 231Use the specified string as the separator between the filename and the 232file result returned. 233Defaults to 234.Sq \&: . 235.It Fl f , Fl Fl files-from Ar namefile 236Read the names of the files to be examined from 237.Ar namefile 238(one per line) 239before the argument list. 240Either 241.Ar namefile 242or at least one filename argument must be present; 243to test the standard input, use 244.Sq - 245as a filename argument. 246Please note that 247.Ar namefile 248is unwrapped and the enclosed filenames are processed when this option is 249encountered and before any further options processing is done. 250This allows one to process multiple lists of files with different command line 251arguments on the same 252.Nm 253invocation. 254Thus if you want to set the delimiter, you need to do it before you specify 255the list of files, like: 256.Dq Fl F Ar @ Fl f Ar namefile , 257instead of: 258.Dq Fl f Ar namefile Fl F Ar @ . 259.It Fl h , Fl Fl no-dereference 260option causes symlinks not to be followed 261(on systems that support symbolic links). 262This is the default if the environment variable 263.Dv POSIXLY_CORRECT 264is not defined. 265.It Fl i , Fl Fl mime 266Causes the file command to output mime type strings rather than the more 267traditional human readable ones. 268Thus it may say 269.Sq text/plain; charset=us-ascii 270rather than 271.Dq ASCII text . 272.It Fl Fl mime-type , Fl Fl mime-encoding 273Like 274.Fl i , 275but print only the specified element(s). 276.It Fl k , Fl Fl keep-going 277Don't stop at the first match, keep going. 278Subsequent matches will be 279have the string 280.Sq "\[rs]012\- " 281prepended. 282(If you want a newline, see the 283.Fl r 284option.) 285The magic pattern with the highest strength (see the 286.Fl l 287option) comes first. 288.It Fl l , Fl Fl list 289Shows a list of patterns and their strength sorted descending by 290.Xr magic 4 291strength 292which is used for the matching (see also the 293.Fl k 294option). 295.It Fl L , Fl Fl dereference 296option causes symlinks to be followed, as the like-named option in 297.Xr ls 1 298(on systems that support symbolic links). 299This is the default if the environment variable 300.Ev POSIXLY_CORRECT 301is defined. 302.It Fl m , Fl Fl magic-file Ar magicfiles 303Specify an alternate list of files and directories containing magic. 304This can be a single item, or a colon-separated list. 305If a compiled magic file is found alongside a file or directory, 306it will be used instead. 307.It Fl N , Fl Fl no-pad 308Don't pad filenames so that they align in the output. 309.It Fl n , Fl Fl no-buffer 310Force stdout to be flushed after checking each file. 311This is only useful if checking a list of files. 312It is intended to be used by programs that want filetype output from a pipe. 313.It Fl p , Fl Fl preserve-date 314On systems that support 315.Xr utime 3 316or 317.Xr utimes 2 , 318attempt to preserve the access time of files analyzed, to pretend that 319.Nm 320never read them. 321.It Fl P , Fl Fl parameter Ar name=value 322Set various parameter limits. 323.Bl -column "elf_phnum" "Default" "XXXXXXXXXXXXXXXXXXXXXXXXXXX" -offset indent 324.It Sy "Name" Ta Sy "Default" Ta Sy "Explanation" 325.It Li indir Ta 15 Ta recursion limit for indirect magic 326.It Li name Ta 30 Ta use count limit for name/use magic 327.It Li elf_notes Ta 256 Ta max ELF notes processed 328.It Li elf_phnum Ta 128 Ta max ELF program sections processed 329.It Li elf_shnum Ta 32768 Ta max ELF sections processed 330.It Li regex Ta 8192 Ta length limit for regex searches 331.It Li bytes Ta 1048576 Ta max number of bytes to read from file 332.El 333.It Fl r , Fl Fl raw 334Don't translate unprintable characters to \eooo. 335Normally 336.Nm 337translates unprintable characters to their octal representation. 338.It Fl s , Fl Fl special-files 339Normally, 340.Nm 341only attempts to read and determine the type of argument files which 342.Xr stat 2 343reports are ordinary files. 344This prevents problems, because reading special files may have peculiar 345consequences. 346Specifying the 347.Fl s 348option causes 349.Nm 350to also read argument files which are block or character special files. 351This is useful for determining the filesystem types of the data in raw 352disk partitions, which are block special files. 353This option also causes 354.Nm 355to disregard the file size as reported by 356.Xr stat 2 357since on some systems it reports a zero size for raw disk partitions. 358.If Fl S , Fl Fl no-sandbox 359On systems where libseccomp 360.Pa ( https://github.com/seccomp/libseccomp ) 361is available, the 362.Fl S 363flag disables sandboxing which is enabled by default. 364This option is needed for file to execute external descompressing programs, 365i.e. when the 366.Fl z 367flag is specified and the built-in decompressors are not available. 368.It Fl v , Fl Fl version 369Print the version of the program and exit. 370.It Fl z , Fl Fl uncompress 371Try to look inside compressed files. 372.It Fl Z , Fl Fl uncompress-noreport 373Try to look inside compressed files, but report information about the contents 374only not the compression. 375.It Fl 0 , Fl Fl print0 376Output a null character 377.Sq \e0 378after the end of the filename. 379Nice to 380.Xr cut 1 381the output. 382This does not affect the separator, which is still printed. 383.Pp 384If this option is repeated more than once, then 385.Nm 386prints just the filename followed by a NUL followed by the description 387(or ERROR: text) followed by a second NUL for each entry. 388.It Fl -help 389Print a help message and exit. 390.El 391.Sh ENVIRONMENT 392The environment variable 393.Ev MAGIC 394can be used to set the default magic file name. 395If that variable is set, then 396.Nm 397will not attempt to open 398.Pa $HOME/.magic . 399.Nm 400adds 401.Dq Pa .mgc 402to the value of this variable as appropriate. 403The environment variable 404.Ev POSIXLY_CORRECT 405controls (on systems that support symbolic links), whether 406.Nm 407will attempt to follow symlinks or not. 408If set, then 409.Nm 410follows symlink, otherwise it does not. 411This is also controlled by the 412.Fl L 413and 414.Fl h 415options. 416.Sh FILES 417.Bl -tag -width /usr/share/misc/magic.mgc -compact 418.It Pa /usr/share/misc/magic.mgc 419Default compiled list of magic. 420.It Pa /usr/share/misc/magic 421Directory containing default magic files. 422.El 423.Sh EXIT STATUS 424.Nm 425will exit with 426.Dv 0 427if the operation was successful or 428.Dv >0 429if an error was encountered. 430The following errors cause diagnostic messages, but don't affect the program 431exit code (as POSIX requires), unless 432.Fl E 433is specified: 434.Bl -bullet -compact -offset indent 435.It 436A file cannot be found 437.It 438There is no permission to read a file 439.It 440The file type cannot be determined 441.El 442.Sh EXAMPLES 443.Bd -literal -offset indent 444$ file file.c file /dev/{wd0a,hda} 445file.c: C program text 446file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 447 dynamically linked (uses shared libs), stripped 448/dev/wd0a: block special (0/0) 449/dev/hda: block special (3/0) 450 451$ file -s /dev/wd0{b,d} 452/dev/wd0b: data 453/dev/wd0d: x86 boot sector 454 455$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} 456/dev/hda: x86 boot sector 457/dev/hda1: Linux/i386 ext2 filesystem 458/dev/hda2: x86 boot sector 459/dev/hda3: x86 boot sector, extended partition table 460/dev/hda4: Linux/i386 ext2 filesystem 461/dev/hda5: Linux/i386 swap file 462/dev/hda6: Linux/i386 swap file 463/dev/hda7: Linux/i386 swap file 464/dev/hda8: Linux/i386 swap file 465/dev/hda9: empty 466/dev/hda10: empty 467 468$ file -i file.c file /dev/{wd0a,hda} 469file.c: text/x-c 470file: application/x-executable 471/dev/hda: application/x-not-regular-file 472/dev/wd0a: application/x-not-regular-file 473 474.Ed 475.Sh SEE ALSO 476.Xr hexdump 1 , 477.Xr od 1 , 478.Xr strings 1 , 479.Xr magic 5 480.Sh STANDARDS CONFORMANCE 481This program is believed to exceed the System V Interface Definition 482of FILE(CMD), as near as one can determine from the vague language 483contained therein. 484Its behavior is mostly compatible with the System V program of the same name. 485This version knows more magic, however, so it will produce 486different (albeit more accurate) output in many cases. 487.\" URL: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html 488.Pp 489The one significant difference 490between this version and System V 491is that this version treats any white space 492as a delimiter, so that spaces in pattern strings must be escaped. 493For example, 494.Bd -literal -offset indent 495\*[Gt]10 string language impress\ (imPRESS data) 496.Ed 497.Pp 498in an existing magic file would have to be changed to 499.Bd -literal -offset indent 500\*[Gt]10 string language\e impress (imPRESS data) 501.Ed 502.Pp 503In addition, in this version, if a pattern string contains a backslash, 504it must be escaped. 505For example 506.Bd -literal -offset indent 5070 string \ebegindata Andrew Toolkit document 508.Ed 509.Pp 510in an existing magic file would have to be changed to 511.Bd -literal -offset indent 5120 string \e\ebegindata Andrew Toolkit document 513.Ed 514.Pp 515SunOS releases 3.2 and later from Sun Microsystems include a 516.Nm 517command derived from the System V one, but with some extensions. 518This version differs from Sun's only in minor ways. 519It includes the extension of the 520.Sq \*[Am] 521operator, used as, 522for example, 523.Bd -literal -offset indent 524\*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped 525.Ed 526.Sh SECURITY 527On systems where libseccomp 528.Pa ( https://github.com/seccomp/libseccomp ) 529is available, 530.Nm 531is enforces limiting system calls to only the ones necessary for the 532operation of the program. 533This enforcement does not provide any security benefit when 534.Nm 535is asked to decompress input files running external programs with 536the 537.Fl z 538option. 539To enable execution of external decompressors, one needs to disable 540sandboxing using the 541.Fl S 542flag. 543.Sh MAGIC DIRECTORY 544The magic file entries have been collected from various sources, 545mainly USENET, and contributed by various authors. 546Christos Zoulas (address below) will collect additional 547or corrected magic file entries. 548A consolidation of magic file entries 549will be distributed periodically. 550.Pp 551The order of entries in the magic file is significant. 552Depending on what system you are using, the order that 553they are put together may be incorrect. 554If your old 555.Nm 556command uses a magic file, 557keep the old magic file around for comparison purposes 558(rename it to 559.Pa /usr/share/misc/magic.orig ) . 560.Sh HISTORY 561There has been a 562.Nm 563command in every 564.Dv UNIX since at least Research Version 4 565(man page dated November, 1973). 566The System V version introduced one significant major change: 567the external list of magic types. 568This slowed the program down slightly but made it a lot more flexible. 569.Pp 570This program, based on the System V version, 571was written by Ian Darwin 572.Aq ian@darwinsys.com 573without looking at anybody else's source code. 574.Pp 575John Gilmore revised the code extensively, making it better than 576the first version. 577Geoff Collyer found several inadequacies 578and provided some magic file entries. 579Contributions of the 580.Sq \*[Am] 581operator by Rob McMahon, 582.Aq cudcv@warwick.ac.uk , 5831989. 584.Pp 585Guy Harris, 586.Aq guy@netapp.com , 587made many changes from 1993 to the present. 588.Pp 589Primary development and maintenance from 1990 to the present by 590Christos Zoulas 591.Aq christos@astron.com . 592.Pp 593Altered by Chris Lowth 594.Aq chris@lowth.com , 5952000: handle the 596.Fl i 597option to output mime type strings, using an alternative 598magic file and internal logic. 599.Pp 600Altered by Eric Fischer 601.Aq enf@pobox.com , 602July, 2000, 603to identify character codes and attempt to identify the languages 604of non-ASCII files. 605.Pp 606Altered by Reuben Thomas 607.Aq rrt@sc3d.org , 6082007-2011, to improve MIME support, merge MIME and non-MIME magic, 609support directories as well as files of magic, apply many bug fixes, 610update and fix a lot of magic, improve the build system, improve the 611documentation, and rewrite the Python bindings in pure Python. 612.Pp 613The list of contributors to the 614.Sq magic 615directory (magic files) 616is too long to include here. 617You know who you are; thank you. 618Many contributors are listed in the source files. 619.Sh LEGAL NOTICE 620Copyright (c) Ian F. Darwin, Toronto, Canada, 1986-1999. 621Covered by the standard Berkeley Software Distribution copyright; see the file 622COPYING in the source distribution. 623.Pp 624The files 625.Pa tar.h 626and 627.Pa is_tar.c 628were written by John Gilmore from his public-domain 629.Xr tar 1 630program, and are not covered by the above license. 631.Sh BUGS 632Please report bugs and send patches to the bug tracker at 633.Pa http://bugs.gw.com/ 634or the mailing list at 635.Aq file@mx.gw.com 636(visit 637.Pa http://mx.gw.com/mailman/listinfo/file 638first to subscribe). 639.Sh TODO 640Fix output so that tests for MIME and APPLE flags are not needed all 641over the place, and actual output is only done in one place. 642This needs a design. 643Suggestion: push possible outputs on to a list, then pick the 644last-pushed (most specific, one hopes) value at the end, or 645use a default if the list is empty. 646This should not slow down evaluation. 647.Pp 648The handling of 649.Dv MAGIC_CONTINUE 650and printing \e012- between entries is clumsy and complicated; refactor 651and centralize. 652.Pp 653Some of the encoding logic is hard-coded in encoding.c and can be moved 654to the magic files if we had a !:charset annotation 655.Pp 656Continue to squash all magic bugs. 657See Debian BTS for a good source. 658.Pp 659Store arbitrarily long strings, for example for %s patterns, so that 660they can be printed out. 661Fixes Debian bug #271672. 662This can be done by allocating strings in a string pool, storing the 663string pool at the end of the magic file and converting all the string 664pointers to relative offsets from the string pool. 665.Pp 666Add syntax for relative offsets after current level (Debian bug #466037). 667.Pp 668Make file -ki work, i.e. give multiple MIME types. 669.Pp 670Add a zip library so we can peek inside Office2007 documents to 671print more details about their contents. 672.Pp 673Add an option to print URLs for the sources of the file descriptions. 674.Pp 675Combine script searches and add a way to map executable names to MIME 676types (e.g. have a magic value for !:mime which causes the resulting 677string to be looked up in a table). 678This would avoid adding the same magic repeatedly for each new 679hash-bang interpreter. 680.Pp 681When a file descriptor is available, we can skip and adjust the buffer 682instead of the hacky buffer management we do now. 683.Pp 684Fix 685.Dq name 686and 687.Dq use 688to check for consistency at compile time (duplicate 689.Dq name , 690.Dq use 691pointing to undefined 692.Dq name 693). 694Make 695.Dq name 696/ 697.Dq use 698more efficient by keeping a sorted list of names. 699Special-case ^ to flip endianness in the parser so that it does not 700have to be escaped, and document it. 701.Pp 702If the offsets specified internally in the file exceed the buffer size 703( 704.Dv HOWMANY 705variable in file.h), then we don't seek to that offset, but we give up. 706It would be better if buffer managements was done when the file descriptor 707is available so move around the file. 708One must be careful though because this has performance (and thus security 709considerations). 710.Sh AVAILABILITY 711You can obtain the original author's latest version by anonymous FTP 712on 713.Pa ftp.astron.com 714in the directory 715.Pa /pub/file/file-X.YZ.tar.gz . 716