1.\" $NetBSD: file.1,v 1.27 2022/09/24 20:21:45 christos Exp $ 2.\" 3.\" $File: file.man,v 1.144 2021/02/05 22:08:31 christos Exp $ 4.Dd February 5, 2021 5.Dt FILE 1 6.Os 7.Sh NAME 8.Nm file 9.Nd determine file type 10.Sh SYNOPSIS 11.Nm 12.Bk -words 13.Op Fl bcdEhiklLNnprsSvzZ0 14.Op Fl Fl apple 15.Op Fl Fl exclude-quiet 16.Op Fl Fl extension 17.Op Fl Fl mime-encoding 18.Op Fl Fl mime-type 19.Op Fl e Ar testname 20.Op Fl F Ar separator 21.Op Fl f Ar namefile 22.Op Fl m Ar magicfiles 23.Op Fl P Ar name=value 24.Ar 25.Ek 26.Nm 27.Fl C 28.Op Fl m Ar magicfiles 29.Nm 30.Op Fl Fl help 31.Sh DESCRIPTION 32This manual page documents version 5.43 of the 33.Nm 34command. 35.Pp 36.Nm 37tests each argument in an attempt to classify it. 38There are three sets of tests, performed in this order: 39filesystem tests, magic tests, and language tests. 40The 41.Em first 42test that succeeds causes the file type to be printed. 43.Pp 44The type printed will usually contain one of the words 45.Em text 46(the file contains only 47printing characters and a few common control 48characters and is probably safe to read on an 49.Dv ASCII 50terminal), 51.Em executable 52(the file contains the result of compiling a program 53in a form understandable to some 54.Tn UNIX 55kernel or another), 56or 57.Em data 58meaning anything else (data is usually 59.Dq binary 60or non-printable). 61Exceptions are well-known file formats (core files, tar archives) 62that are known to contain binary data. 63When modifying magic files or the program itself, make sure to 64.Em preserve these keywords . 65Users depend on knowing that all the readable files in a directory 66have the word 67.Dq text 68printed. 69Don't do as Berkeley did and change 70.Dq shell commands text 71to 72.Dq shell script . 73.Pp 74The filesystem tests are based on examining the return from a 75.Xr stat 2 76system call. 77The program checks to see if the file is empty, 78or if it's some sort of special file. 79Any known file types appropriate to the system you are running on 80(sockets, symbolic links, or named pipes (FIFOs) on those systems that 81implement them) 82are intuited if they are defined in the system header file 83.In sys/stat.h . 84.Pp 85The magic tests are used to check for files with data in 86particular fixed formats. 87The canonical example of this is a binary executable (compiled program) 88.Dv a.out 89file, whose format is defined in 90.In elf.h , 91.In a.out.h 92and possibly 93.In exec.h 94in the standard include directory. 95These files have a 96.Dq magic number 97stored in a particular place 98near the beginning of the file that tells the 99.Tn UNIX 100operating system 101that the file is a binary executable, and which of several types thereof. 102The concept of a 103.Dq magic number 104has been applied by extension to data files. 105Any file with some invariant identifier at a small fixed 106offset into the file can usually be described in this way. 107The information identifying these files is read from the compiled 108magic file 109.Pa /usr/share/misc/magic.mgc , 110or the files in the directory 111.Pa /usr/share/misc/magic 112if the compiled file does not exist. 113In addition, if 114.Pa $HOME/.magic.mgc 115or 116.Pa $HOME/.magic 117exists, it will be used in preference to the system magic files. 118.Pp 119If a file does not match any of the entries in the magic file, 120it is examined to see if it seems to be a text file. 121ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets 122(such as those used on Macintosh and IBM PC systems), 123UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC 124character sets can be distinguished by the different 125ranges and sequences of bytes that constitute printable text 126in each set. 127If a file passes any of these tests, its character set is reported. 128ASCII, ISO-8859-x, UTF-8, and extended-ASCII files are identified 129as 130.Dq text 131because they will be mostly readable on nearly any terminal; 132UTF-16 and EBCDIC are only 133.Dq character data 134because, while 135they contain text, it is text that will require translation 136before it can be read. 137In addition, 138.Nm 139will attempt to determine other characteristics of text-type files. 140If the lines of a file are terminated by CR, CRLF, or NEL, instead 141of the Unix-standard LF, this will be reported. 142Files that contain embedded escape sequences or overstriking 143will also be identified. 144.Pp 145Once 146.Nm 147has determined the character set used in a text-type file, 148it will 149attempt to determine in what language the file is written. 150The language tests look for particular strings (cf. 151.In names.h ) 152that can appear anywhere in the first few blocks of a file. 153For example, the keyword 154.Em .br 155indicates that the file is most likely a 156.Xr troff 1 157input file, just as the keyword 158.Em struct 159indicates a C program. 160These tests are less reliable than the previous 161two groups, so they are performed last. 162The language test routines also test for some miscellany 163(such as 164.Xr tar 1 165archives, JSON files). 166.Pp 167Any file that cannot be identified as having been written 168in any of the character sets listed above is simply said to be 169.Dq data . 170.Sh OPTIONS 171.Bl -tag -width indent 172.It Fl Fl apple 173Causes the 174.Nm 175command to output the file type and creator code as 176used by older MacOS versions. 177The code consists of eight letters, 178the first describing the file type, the latter the creator. 179This option works properly only for file formats that have the 180apple-style output defined. 181.It Fl b , Fl Fl brief 182Do not prepend filenames to output lines (brief mode). 183.It Fl C , Fl Fl compile 184Write a 185.Pa magic.mgc 186output file that contains a pre-parsed version of the magic file or directory. 187.It Fl c , Fl Fl checking-printout 188Cause a checking printout of the parsed form of the magic file. 189This is usually used in conjunction with the 190.Fl m 191option to debug a new magic file before installing it. 192.It Fl d 193Prints internal debugging information to stderr. 194.It Fl E 195On filesystem errors (file not found etc), instead of handling the error 196as regular output as POSIX mandates and keep going, issue an error message 197and exit. 198.It Fl e , Fl Fl exclude Ar testname 199Exclude the test named in 200.Ar testname 201from the list of tests made to determine the file type. 202Valid test names are: 203.Bl -tag -width compress 204.It apptype 205.Dv EMX 206application type (only on EMX). 207.It ascii 208Various types of text files (this test will try to guess the text 209encoding, irrespective of the setting of the 210.Sq encoding 211option). 212.It encoding 213Different text encodings for soft magic tests. 214.It tokens 215Ignored for backwards compatibility. 216.It cdf 217Prints details of Compound Document Files. 218.It compress 219Checks for, and looks inside, compressed files. 220.It csv 221Checks Comma Separated Value files. 222.It elf 223Prints ELF file details, provided soft magic tests are enabled and the 224elf magic is found. 225.It json 226Examines JSON (RFC-7159) files by parsing them for compliance. 227.It soft 228Consults magic files. 229.It tar 230Examines tar files by verifying the checksum of the 512 byte tar header. 231Excluding this test can provide more detailed content description by using 232the soft magic method. 233.It text 234A synonym for 235.Sq ascii . 236.El 237.It Fl Fl exclude-quiet 238Like 239.Fl Fl exclude 240but ignore tests that 241.Nm 242does not know about. 243This is intended for compatibility with older versions of 244.Nm . 245.It Fl Fl extension 246Print a slash-separated list of valid extensions for the file type found. 247.It Fl F , Fl Fl separator Ar separator 248Use the specified string as the separator between the filename and the 249file result returned. 250Defaults to 251.Sq \&: . 252.It Fl f , Fl Fl files-from Ar namefile 253Read the names of the files to be examined from 254.Ar namefile 255(one per line) 256before the argument list. 257Either 258.Ar namefile 259or at least one filename argument must be present; 260to test the standard input, use 261.Sq - 262as a filename argument. 263Please note that 264.Ar namefile 265is unwrapped and the enclosed filenames are processed when this option is 266encountered and before any further options processing is done. 267This allows one to process multiple lists of files with different command line 268arguments on the same 269.Nm 270invocation. 271Thus if you want to set the delimiter, you need to do it before you specify 272the list of files, like: 273.Dq Fl F Ar @ Fl f Ar namefile , 274instead of: 275.Dq Fl f Ar namefile Fl F Ar @ . 276.It Fl h , Fl Fl no-dereference 277This option causes symlinks not to be followed 278(on systems that support symbolic links). 279This is the default if the environment variable 280.Dv POSIXLY_CORRECT 281is not defined. 282.It Fl i , Fl Fl mime 283Causes the 284.Nm 285command to output mime type strings rather than the more 286traditional human readable ones. 287Thus it may say 288.Sq text/plain; charset=us-ascii 289rather than 290.Dq ASCII text . 291.It Fl Fl mime-type , Fl Fl mime-encoding 292Like 293.Fl i , 294but print only the specified element(s). 295.It Fl k , Fl Fl keep-going 296Don't stop at the first match, keep going. 297Subsequent matches will be 298have the string 299.Sq "\[rs]012\- " 300prepended. 301(If you want a newline, see the 302.Fl r 303option.) 304The magic pattern with the highest strength (see the 305.Fl l 306option) comes first. 307.It Fl l , Fl Fl list 308Shows a list of patterns and their strength sorted descending by 309.Xr magic 5 310strength 311which is used for the matching (see also the 312.Fl k 313option). 314.It Fl L , Fl Fl dereference 315This option causes symlinks to be followed, as the like-named option in 316.Xr ls 1 317(on systems that support symbolic links). 318This is the default if the environment variable 319.Ev POSIXLY_CORRECT 320is defined. 321.It Fl m , Fl Fl magic-file Ar magicfiles 322Specify an alternate list of files and directories containing magic. 323This can be a single item, or a colon-separated list. 324If a compiled magic file is found alongside a file or directory, 325it will be used instead. 326.It Fl N , Fl Fl no-pad 327Don't pad filenames so that they align in the output. 328.It Fl n , Fl Fl no-buffer 329Force stdout to be flushed after checking each file. 330This is only useful if checking a list of files. 331It is intended to be used by programs that want filetype output from a pipe. 332.It Fl p , Fl Fl preserve-date 333On systems that support 334.Xr utime 3 335or 336.Xr utimes 2 , 337attempt to preserve the access time of files analyzed, to pretend that 338.Nm 339never read them. 340.It Fl P , Fl Fl parameter Ar name=value 341Set various parameter limits. 342.Bl -column "elf_phnum" "Default" "XXXXXXXXXXXXXXXXXXXXXXXXXXX" -offset indent 343.It Sy "Name" Ta Sy "Default" Ta Sy "Explanation" 344.It Li bytes Ta 1048576 Ta max number of bytes to read from file 345.It Li elf_notes Ta 256 Ta max ELF notes processed 346.It Li elf_phnum Ta 2048 Ta max ELF program sections processed 347.It Li elf_shnum Ta 32768 Ta max ELF sections processed 348.It Li encoding Ta 65536 Ta max number of bytes to scan for encoding evaluation 349.It Li indir Ta 50 Ta recursion limit for indirect magic 350.It Li name Ta 50 Ta use count limit for name/use magic 351.It Li regex Ta 8192 Ta length limit for regex searches 352.El 353.It Fl r , Fl Fl raw 354Don't translate unprintable characters to \eooo. 355Normally 356.Nm 357translates unprintable characters to their octal representation. 358.It Fl s , Fl Fl special-files 359Normally, 360.Nm 361only attempts to read and determine the type of argument files which 362.Xr stat 2 363reports are ordinary files. 364This prevents problems, because reading special files may have peculiar 365consequences. 366Specifying the 367.Fl s 368option causes 369.Nm 370to also read argument files which are block or character special files. 371This is useful for determining the filesystem types of the data in raw 372disk partitions, which are block special files. 373This option also causes 374.Nm 375to disregard the file size as reported by 376.Xr stat 2 377since on some systems it reports a zero size for raw disk partitions. 378.It Fl S , Fl Fl no-sandbox 379On systems where libseccomp 380.Pa ( https://github.com/seccomp/libseccomp ) 381is available, the 382.Fl S 383option disables sandboxing which is enabled by default. 384This option is needed for 385.Nm 386to execute external decompressing programs, 387i.e. when the 388.Fl z 389option is specified and the built-in decompressors are not available. 390On systems where sandboxing is not available, this option has no effect. 391.It Fl v , Fl Fl version 392Print the version of the program and exit. 393.It Fl z , Fl Fl uncompress 394Try to look inside compressed files. 395.It Fl Z , Fl Fl uncompress-noreport 396Try to look inside compressed files, but report information about the contents 397only not the compression. 398.It Fl 0 , Fl Fl print0 399Output a null character 400.Sq \e0 401after the end of the filename. 402Nice to 403.Xr cut 1 404the output. 405This does not affect the separator, which is still printed. 406.Pp 407If this option is repeated more than once, then 408.Nm 409prints just the filename followed by a NUL followed by the description 410(or ERROR: text) followed by a second NUL for each entry. 411.It Fl -help 412Print a help message and exit. 413.El 414.Sh ENVIRONMENT 415The environment variable 416.Ev MAGIC 417can be used to set the default magic file name. 418If that variable is set, then 419.Nm 420will not attempt to open 421.Pa $HOME/.magic . 422.Nm 423adds 424.Dq Pa .mgc 425to the value of this variable as appropriate. 426The environment variable 427.Ev POSIXLY_CORRECT 428controls (on systems that support symbolic links), whether 429.Nm 430will attempt to follow symlinks or not. 431If set, then 432.Nm 433follows symlink, otherwise it does not. 434This is also controlled by the 435.Fl L 436and 437.Fl h 438options. 439.Sh FILES 440.Bl -tag -width /usr/share/misc/magic.mgc -compact 441.It Pa /usr/share/misc/magic.mgc 442Default compiled list of magic. 443.It Pa /usr/share/misc/magic 444Directory containing default magic files. 445.El 446.Sh EXIT STATUS 447.Nm 448will exit with 449.Dv 0 450if the operation was successful or 451.Dv >0 452if an error was encountered. 453The following errors cause diagnostic messages, but don't affect the program 454exit code (as POSIX requires), unless 455.Fl E 456is specified: 457.Bl -bullet -compact -offset indent 458.It 459A file cannot be found 460.It 461There is no permission to read a file 462.It 463The file type cannot be determined 464.El 465.Sh EXAMPLES 466.Bd -literal -offset indent 467$ file file.c file /dev/{wd0a,hda} 468file.c: C program text 469file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 470 dynamically linked (uses shared libs), stripped 471/dev/wd0a: block special (0/0) 472/dev/hda: block special (3/0) 473 474$ file -s /dev/wd0{b,d} 475/dev/wd0b: data 476/dev/wd0d: x86 boot sector 477 478$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} 479/dev/hda: x86 boot sector 480/dev/hda1: Linux/i386 ext2 filesystem 481/dev/hda2: x86 boot sector 482/dev/hda3: x86 boot sector, extended partition table 483/dev/hda4: Linux/i386 ext2 filesystem 484/dev/hda5: Linux/i386 swap file 485/dev/hda6: Linux/i386 swap file 486/dev/hda7: Linux/i386 swap file 487/dev/hda8: Linux/i386 swap file 488/dev/hda9: empty 489/dev/hda10: empty 490 491$ file -i file.c file /dev/{wd0a,hda} 492file.c: text/x-c 493file: application/x-executable 494/dev/hda: application/x-not-regular-file 495/dev/wd0a: application/x-not-regular-file 496 497.Ed 498.Sh SEE ALSO 499.Xr hexdump 1 , 500.Xr od 1 , 501.Xr strings 1 , 502.Xr magic 5 503.Sh STANDARDS CONFORMANCE 504This program is believed to exceed the System V Interface Definition 505of FILE(CMD), as near as one can determine from the vague language 506contained therein. 507Its behavior is mostly compatible with the System V program of the same name. 508This version knows more magic, however, so it will produce 509different (albeit more accurate) output in many cases. 510.\" URL: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html 511.Pp 512The one significant difference 513between this version and System V 514is that this version treats any white space 515as a delimiter, so that spaces in pattern strings must be escaped. 516For example, 517.Bd -literal -offset indent 518\*[Gt]10 string language impress\ (imPRESS data) 519.Ed 520.Pp 521in an existing magic file would have to be changed to 522.Bd -literal -offset indent 523\*[Gt]10 string language\e impress (imPRESS data) 524.Ed 525.Pp 526In addition, in this version, if a pattern string contains a backslash, 527it must be escaped. 528For example 529.Bd -literal -offset indent 5300 string \ebegindata Andrew Toolkit document 531.Ed 532.Pp 533in an existing magic file would have to be changed to 534.Bd -literal -offset indent 5350 string \e\ebegindata Andrew Toolkit document 536.Ed 537.Pp 538SunOS releases 3.2 and later from Sun Microsystems include a 539.Nm 540command derived from the System V one, but with some extensions. 541This version differs from Sun's only in minor ways. 542It includes the extension of the 543.Sq \*[Am] 544operator, used as, 545for example, 546.Bd -literal -offset indent 547\*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped 548.Ed 549.Sh SECURITY 550On systems where libseccomp 551.Pa ( https://github.com/seccomp/libseccomp ) 552is available, 553.Nm 554is enforces limiting system calls to only the ones necessary for the 555operation of the program. 556This enforcement does not provide any security benefit when 557.Nm 558is asked to decompress input files running external programs with 559the 560.Fl z 561option. 562To enable execution of external decompressors, one needs to disable 563sandboxing using the 564.Fl S 565option. 566.Sh MAGIC DIRECTORY 567The magic file entries have been collected from various sources, 568mainly USENET, and contributed by various authors. 569Christos Zoulas (address below) will collect additional 570or corrected magic file entries. 571A consolidation of magic file entries 572will be distributed periodically. 573.Pp 574The order of entries in the magic file is significant. 575Depending on what system you are using, the order that 576they are put together may be incorrect. 577If your old 578.Nm 579command uses a magic file, 580keep the old magic file around for comparison purposes 581(rename it to 582.Pa /usr/share/misc/magic.orig ) . 583.Sh HISTORY 584There has been a 585.Nm 586command in every 587.Dv UNIX since at least Research Version 4 588(man page dated November, 1973). 589The System V version introduced one significant major change: 590the external list of magic types. 591This slowed the program down slightly but made it a lot more flexible. 592.Pp 593This program, based on the System V version, 594was written by Ian Darwin 595.Aq ian@darwinsys.com 596without looking at anybody else's source code. 597.Pp 598John Gilmore revised the code extensively, making it better than 599the first version. 600Geoff Collyer found several inadequacies 601and provided some magic file entries. 602Contributions of the 603.Sq \*[Am] 604operator by Rob McMahon, 605.Aq cudcv@warwick.ac.uk , 6061989. 607.Pp 608Guy Harris, 609.Aq guy@netapp.com , 610made many changes from 1993 to the present. 611.Pp 612Primary development and maintenance from 1990 to the present by 613Christos Zoulas 614.Aq christos@astron.com . 615.Pp 616Altered by Chris Lowth 617.Aq chris@lowth.com , 6182000: handle the 619.Fl i 620option to output mime type strings, using an alternative 621magic file and internal logic. 622.Pp 623Altered by Eric Fischer 624.Aq enf@pobox.com , 625July, 2000, 626to identify character codes and attempt to identify the languages 627of non-ASCII files. 628.Pp 629Altered by Reuben Thomas 630.Aq rrt@sc3d.org , 6312007-2011, to improve MIME support, merge MIME and non-MIME magic, 632support directories as well as files of magic, apply many bug fixes, 633update and fix a lot of magic, improve the build system, improve the 634documentation, and rewrite the Python bindings in pure Python. 635.Pp 636The list of contributors to the 637.Sq magic 638directory (magic files) 639is too long to include here. 640You know who you are; thank you. 641Many contributors are listed in the source files. 642.Sh LEGAL NOTICE 643Copyright (c) Ian F. Darwin, Toronto, Canada, 1986-1999. 644Covered by the standard Berkeley Software Distribution copyright; see the file 645COPYING in the source distribution. 646.Pp 647The files 648.Pa tar.h 649and 650.Pa is_tar.c 651were written by John Gilmore from his public-domain 652.Xr tar 1 653program, and are not covered by the above license. 654.Sh BUGS 655Please report bugs and send patches to the bug tracker at 656.Pa https://bugs.astron.com/ 657or the mailing list at 658.Aq file@astron.com 659(visit 660.Pa https://mailman.astron.com/mailman/listinfo/file 661first to subscribe). 662.Sh TODO 663Fix output so that tests for MIME and APPLE flags are not needed all 664over the place, and actual output is only done in one place. 665This needs a design. 666Suggestion: push possible outputs on to a list, then pick the 667last-pushed (most specific, one hopes) value at the end, or 668use a default if the list is empty. 669This should not slow down evaluation. 670.Pp 671The handling of 672.Dv MAGIC_CONTINUE 673and printing \e012- between entries is clumsy and complicated; refactor 674and centralize. 675.Pp 676Some of the encoding logic is hard-coded in encoding.c and can be moved 677to the magic files if we had a !:charset annotation. 678.Pp 679Continue to squash all magic bugs. 680See Debian BTS for a good source. 681.Pp 682Store arbitrarily long strings, for example for %s patterns, so that 683they can be printed out. 684Fixes Debian bug #271672. 685This can be done by allocating strings in a string pool, storing the 686string pool at the end of the magic file and converting all the string 687pointers to relative offsets from the string pool. 688.Pp 689Add syntax for relative offsets after current level (Debian bug #466037). 690.Pp 691Make file -ki work, i.e. give multiple MIME types. 692.Pp 693Add a zip library so we can peek inside Office2007 documents to 694print more details about their contents. 695.Pp 696Add an option to print URLs for the sources of the file descriptions. 697.Pp 698Combine script searches and add a way to map executable names to MIME 699types (e.g. have a magic value for !:mime which causes the resulting 700string to be looked up in a table). 701This would avoid adding the same magic repeatedly for each new 702hash-bang interpreter. 703.Pp 704When a file descriptor is available, we can skip and adjust the buffer 705instead of the hacky buffer management we do now. 706.Pp 707Fix 708.Dq name 709and 710.Dq use 711to check for consistency at compile time (duplicate 712.Dq name , 713.Dq use 714pointing to undefined 715.Dq name 716). 717Make 718.Dq name 719/ 720.Dq use 721more efficient by keeping a sorted list of names. 722Special-case ^ to flip endianness in the parser so that it does not 723have to be escaped, and document it. 724.Pp 725If the offsets specified internally in the file exceed the buffer size 726( 727.Dv HOWMANY 728variable in file.h), then we don't seek to that offset, but we give up. 729It would be better if buffer managements was done when the file descriptor 730is available so we can seek around the file. 731One must be careful though because this has performance and thus security 732considerations, because one can slow down things by repeateadly seeking. 733.Pp 734There is support now for keeping separate buffers and having offsets from 735the end of the file, but the internal buffer management still needs an 736overhaul. 737.Sh AVAILABILITY 738You can obtain the original author's latest version by anonymous FTP 739on 740.Pa ftp.astron.com 741in the directory 742.Pa /pub/file/file-X.YZ.tar.gz . 743