1.\" $NetBSD: magic.5,v 1.20 2019/05/22 17:26:05 christos Exp $ 2.\" 3.\" $File: magic.man,v 1.96 2019/01/21 14:56:53 christos Exp $ 4.Dd January 21, 2019 5.Dt MAGIC 5 6.Os 7.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 8.Sh NAME 9.Nm magic 10.Nd file command's magic pattern file 11.Sh DESCRIPTION 12This manual page documents the format of magic files as 13used by the 14.Xr file 1 15command, version 5.37. 16The 17.Xr file 1 18command identifies the type of a file using, 19among other tests, 20a test for whether the file contains certain 21.Dq "magic patterns" . 22The database of these 23.Dq "magic patterns" 24is usually located in a binary file in 25.Pa /usr/share/misc/magic.mgc 26or a directory of source text magic pattern fragment files in 27.Pa /usr/share/misc/magic . 28The database specifies what patterns are to be tested for, what message or 29MIME type to print if a particular pattern is found, 30and additional information to extract from the file. 31.Pp 32The format of the source fragment files that are used to build this database 33is as follows: 34Each line of a fragment file specifies a test to be performed. 35A test compares the data starting at a particular offset 36in the file with a byte value, a string or a numeric value. 37If the test succeeds, a message is printed. 38The line consists of the following fields: 39.Bl -tag -width ".Dv message" 40.It Dv offset 41A number specifying the offset (in bytes) into the file of the data 42which is to be tested. 43This offset can be a negative number if it is: 44.Bl -bullet -compact 45.It 46The first direct offset of the magic entry (at continuation level 0), 47in which case it is interpreted an offset from end end of the file 48going backwards. 49This works only when a file descriptor to the file is a available and it 50is a regular file. 51.It 52A continuation offset relative to the end of the last up-level field 53.Dv ( \*[Am] ) . 54.El 55.It Dv type 56The type of the data to be tested. 57The possible values are: 58.Bl -tag -width ".Dv lestring16" 59.It Dv byte 60A one-byte value. 61.It Dv short 62A two-byte value in this machine's native byte order. 63.It Dv long 64A four-byte value in this machine's native byte order. 65.It Dv quad 66An eight-byte value in this machine's native byte order. 67.It Dv float 68A 32-bit single precision IEEE floating point number in this machine's native byte order. 69.It Dv double 70A 64-bit double precision IEEE floating point number in this machine's native byte order. 71.It Dv string 72A string of bytes. 73The string type specification can be optionally followed 74by /[WwcCtbT]*. 75The 76.Dq W 77flag compacts whitespace in the target, which must 78contain at least one whitespace character. 79If the magic has 80.Dv n 81consecutive blanks, the target needs at least 82.Dv n 83consecutive blanks to match. 84The 85.Dq w 86flag treats every blank in the magic as an optional blank. 87The 88.Dq c 89flag specifies case insensitive matching: lower case 90characters in the magic match both lower and upper case characters in the 91target, whereas upper case characters in the magic only match upper case 92characters in the target. 93The 94.Dq C 95flag specifies case insensitive matching: upper case 96characters in the magic match both lower and upper case characters in the 97target, whereas lower case characters in the magic only match upper case 98characters in the target. 99To do a complete case insensitive match, specify both 100.Dq c 101and 102.Dq C . 103The 104.Dq t 105flag forces the test to be done for text files, while the 106.Dq b 107flag forces the test to be done for binary files. 108The 109.Dq T 110flag causes the string to be trimmed, i.e. leading and trailing whitespace 111is deleted before the string is printed. 112.It Dv pstring 113A Pascal-style string where the first byte/short/int is interpreted as the 114unsigned length. 115The length defaults to byte and can be specified as a modifier. 116The following modifiers are supported: 117.Bl -tag -compact -width B 118.It B 119A byte length (default). 120.It H 121A 2 byte big endian length. 122.It h 123A 2 byte little endian length. 124.It L 125A 4 byte big endian length. 126.It l 127A 4 byte little endian length. 128.It J 129The length includes itself in its count. 130.El 131The string is not NUL terminated. 132.Dq J 133is used rather than the more 134valuable 135.Dq I 136because this type of length is a feature of the JPEG 137format. 138.It Dv date 139A four-byte value interpreted as a UNIX date. 140.It Dv qdate 141A eight-byte value interpreted as a UNIX date. 142.It Dv ldate 143A four-byte value interpreted as a UNIX-style date, but interpreted as 144local time rather than UTC. 145.It Dv qldate 146An eight-byte value interpreted as a UNIX-style date, but interpreted as 147local time rather than UTC. 148.It Dv qwdate 149An eight-byte value interpreted as a Windows-style date. 150.It Dv beid3 151A 32-bit ID3 length in big-endian byte order. 152.It Dv beshort 153A two-byte value in big-endian byte order. 154.It Dv belong 155A four-byte value in big-endian byte order. 156.It Dv bequad 157An eight-byte value in big-endian byte order. 158.It Dv befloat 159A 32-bit single precision IEEE floating point number in big-endian byte order. 160.It Dv bedouble 161A 64-bit double precision IEEE floating point number in big-endian byte order. 162.It Dv bedate 163A four-byte value in big-endian byte order, 164interpreted as a Unix date. 165.It Dv beqdate 166An eight-byte value in big-endian byte order, 167interpreted as a Unix date. 168.It Dv beldate 169A four-byte value in big-endian byte order, 170interpreted as a UNIX-style date, but interpreted as local time rather 171than UTC. 172.It Dv beqldate 173An eight-byte value in big-endian byte order, 174interpreted as a UNIX-style date, but interpreted as local time rather 175than UTC. 176.It Dv beqwdate 177An eight-byte value in big-endian byte order, 178interpreted as a Windows-style date. 179.It Dv bestring16 180A two-byte unicode (UCS16) string in big-endian byte order. 181.It Dv leid3 182A 32-bit ID3 length in little-endian byte order. 183.It Dv leshort 184A two-byte value in little-endian byte order. 185.It Dv lelong 186A four-byte value in little-endian byte order. 187.It Dv lequad 188An eight-byte value in little-endian byte order. 189.It Dv lefloat 190A 32-bit single precision IEEE floating point number in little-endian byte order. 191.It Dv ledouble 192A 64-bit double precision IEEE floating point number in little-endian byte order. 193.It Dv ledate 194A four-byte value in little-endian byte order, 195interpreted as a UNIX date. 196.It Dv leqdate 197An eight-byte value in little-endian byte order, 198interpreted as a UNIX date. 199.It Dv leldate 200A four-byte value in little-endian byte order, 201interpreted as a UNIX-style date, but interpreted as local time rather 202than UTC. 203.It Dv leqldate 204An eight-byte value in little-endian byte order, 205interpreted as a UNIX-style date, but interpreted as local time rather 206than UTC. 207.It Dv leqwdate 208An eight-byte value in little-endian byte order, 209interpreted as a Windows-style date. 210.It Dv lestring16 211A two-byte unicode (UCS16) string in little-endian byte order. 212.It Dv melong 213A four-byte value in middle-endian (PDP-11) byte order. 214.It Dv medate 215A four-byte value in middle-endian (PDP-11) byte order, 216interpreted as a UNIX date. 217.It Dv meldate 218A four-byte value in middle-endian (PDP-11) byte order, 219interpreted as a UNIX-style date, but interpreted as local time rather 220than UTC. 221.It Dv indirect 222Starting at the given offset, consult the magic database again. 223The offset of the 224.Dv indirect 225magic is by default absolute in the file, but one can specify 226.Dv /r 227to indicate that the offset is relative from the beginning of the entry. 228.It Dv name 229Define a 230.Dq named 231magic instance that can be called from another 232.Dv use 233magic entry, like a subroutine call. 234Named instance direct magic offsets are relative to the offset of the 235previous matched entry, but indirect offsets are relative to the beginning 236of the file as usual. 237Named magic entries always match. 238.It Dv use 239Recursively call the named magic starting from the current offset. 240If the name of the referenced begins with a 241.Dv ^ 242then the endianness of the magic is switched; if the magic mentioned 243.Dv leshort 244for example, 245it is treated as 246.Dv beshort 247and vice versa. 248This is useful to avoid duplicating the rules for different endianness. 249.It Dv regex 250A regular expression match in extended POSIX regular expression syntax 251(like egrep). 252Regular expressions can take exponential time to process, and their 253performance is hard to predict, so their use is discouraged. 254When used in production environments, their performance 255should be carefully checked. 256The size of the string to search should also be limited by specifying 257.Dv /<length> , 258to avoid performance issues scanning long files. 259The type specification can also be optionally followed by 260.Dv /[c][s][l] . 261The 262.Dq c 263flag makes the match case insensitive, while the 264.Dq s 265flag update the offset to the start offset of the match, rather than the end. 266The 267.Dq l 268modifier, changes the limit of length to mean number of lines instead of a 269byte count. 270Lines are delimited by the platforms native line delimiter. 271When a line count is specified, an implicit byte count also computed assuming 272each line is 80 characters long. 273If neither a byte or line count is specified, the search is limited automatically 274to 8KiB. 275.Dv ^ 276and 277.Dv $ 278match the beginning and end of individual lines, respectively, 279not beginning and end of file. 280.It Dv search 281A literal string search starting at the given offset. 282The same modifier flags can be used as for string patterns. 283The search expression must contain the range in the form 284.Dv /number, 285that is the number of positions at which the match will be 286attempted, starting from the start offset. 287This is suitable for 288searching larger binary expressions with variable offsets, using 289.Dv \e 290escapes for special characters. 291The order of modifier and number is not relevant. 292.It Dv default 293This is intended to be used with the test 294.Em x 295(which is always true) and it has no type. 296It matches when no other test at that continuation level has matched before. 297Clearing that matched tests for a continuation level, can be done using the 298.Dv clear 299test. 300.It Dv clear 301This test is always true and clears the match flag for that continuation level. 302It is intended to be used with the 303.Dv default 304test. 305.El 306.Pp 307For compatibility with the Single 308.Ux 309Standard, the type specifiers 310.Dv dC 311and 312.Dv d1 313are equivalent to 314.Dv byte , 315the type specifiers 316.Dv uC 317and 318.Dv u1 319are equivalent to 320.Dv ubyte , 321the type specifiers 322.Dv dS 323and 324.Dv d2 325are equivalent to 326.Dv short , 327the type specifiers 328.Dv uS 329and 330.Dv u2 331are equivalent to 332.Dv ushort , 333the type specifiers 334.Dv dI , 335.Dv dL , 336and 337.Dv d4 338are equivalent to 339.Dv long , 340the type specifiers 341.Dv uI , 342.Dv uL , 343and 344.Dv u4 345are equivalent to 346.Dv ulong , 347the type specifier 348.Dv d8 349is equivalent to 350.Dv quad , 351the type specifier 352.Dv u8 353is equivalent to 354.Dv uquad , 355and the type specifier 356.Dv s 357is equivalent to 358.Dv string . 359In addition, the type specifier 360.Dv dQ 361is equivalent to 362.Dv quad 363and the type specifier 364.Dv uQ 365is equivalent to 366.Dv uquad . 367.Pp 368Each top-level magic pattern (see below for an explanation of levels) 369is classified as text or binary according to the types used. 370Types 371.Dq regex 372and 373.Dq search 374are classified as text tests, unless non-printable characters are used 375in the pattern. 376All other tests are classified as binary. 377A top-level 378pattern is considered to be a test text when all its patterns are text 379patterns; otherwise, it is considered to be a binary pattern. 380When 381matching a file, binary patterns are tried first; if no match is 382found, and the file looks like text, then its encoding is determined 383and the text patterns are tried. 384.Pp 385The numeric types may optionally be followed by 386.Dv \*[Am] 387and a numeric value, 388to specify that the value is to be AND'ed with the 389numeric value before any comparisons are done. 390Prepending a 391.Dv u 392to the type indicates that ordered comparisons should be unsigned. 393.It Dv test 394The value to be compared with the value from the file. 395If the type is 396numeric, this value 397is specified in C form; if it is a string, it is specified as a C string 398with the usual escapes permitted (e.g. \en for new-line). 399.Pp 400Numeric values 401may be preceded by a character indicating the operation to be performed. 402It may be 403.Dv = , 404to specify that the value from the file must equal the specified value, 405.Dv \*[Lt] , 406to specify that the value from the file must be less than the specified 407value, 408.Dv \*[Gt] , 409to specify that the value from the file must be greater than the specified 410value, 411.Dv \*[Am] , 412to specify that the value from the file must have set all of the bits 413that are set in the specified value, 414.Dv ^ , 415to specify that the value from the file must have clear any of the bits 416that are set in the specified value, or 417.Dv ~ , 418the value specified after is negated before tested. 419.Dv x , 420to specify that any value will match. 421If the character is omitted, it is assumed to be 422.Dv = . 423Operators 424.Dv \*[Am] , 425.Dv ^ , 426and 427.Dv ~ 428don't work with floats and doubles. 429The operator 430.Dv !\& 431specifies that the line matches if the test does 432.Em not 433succeed. 434.Pp 435Numeric values are specified in C form; e.g. 436.Dv 13 437is decimal, 438.Dv 013 439is octal, and 440.Dv 0x13 441is hexadecimal. 442.Pp 443Numeric operations are not performed on date types, instead the numeric 444value is interpreted as an offset. 445.Pp 446For string values, the string from the 447file must match the specified string. 448The operators 449.Dv = , 450.Dv \*[Lt] 451and 452.Dv \*[Gt] 453(but not 454.Dv \*[Am] ) 455can be applied to strings. 456The length used for matching is that of the string argument 457in the magic file. 458This means that a line can match any non-empty string (usually used to 459then print the string), with 460.Em \*[Gt]\e0 461(because all non-empty strings are greater than the empty string). 462.Pp 463Dates are treated as numerical values in the respective internal 464representation. 465.Pp 466The special test 467.Em x 468always evaluates to true. 469.It Dv message 470The message to be printed if the comparison succeeds. 471If the string contains a 472.Xr printf 3 473format specification, the value from the file (with any specified masking 474performed) is printed using the message as the format string. 475If the string begins with 476.Dq \eb , 477the message printed is the remainder of the string with no whitespace 478added before it: multiple matches are normally separated by a single 479space. 480.El 481.Pp 482An APPLE 4+4 character APPLE creator and type can be specified as: 483.Bd -literal -offset indent 484!:apple CREATYPE 485.Ed 486.Pp 487A MIME type is given on a separate line, which must be the next 488non-blank or comment line after the magic line that identifies the 489file type, and has the following format: 490.Bd -literal -offset indent 491!:mime MIMETYPE 492.Ed 493.Pp 494i.e. the literal string 495.Dq !:mime 496followed by the MIME type. 497.Pp 498An optional strength can be supplied on a separate line which refers to 499the current magic description using the following format: 500.Bd -literal -offset indent 501!:strength OP VALUE 502.Ed 503.Pp 504The operand 505.Dv OP 506can be: 507.Dv + , 508.Dv - , 509.Dv * , 510or 511.Dv / 512and 513.Dv VALUE 514is a constant between 0 and 255. 515This constant is applied using the specified operand 516to the currently computed default magic strength. 517.Pp 518Some file formats contain additional information which is to be printed 519along with the file type or need additional tests to determine the true 520file type. 521These additional tests are introduced by one or more 522.Em \*[Gt] 523characters preceding the offset. 524The number of 525.Em \*[Gt] 526on the line indicates the level of the test; a line with no 527.Em \*[Gt] 528at the beginning is considered to be at level 0. 529Tests are arranged in a tree-like hierarchy: 530if the test on a line at level 531.Em n 532succeeds, all following tests at level 533.Em n+1 534are performed, and the messages printed if the tests succeed, until a line 535with level 536.Em n 537(or less) appears. 538For more complex files, one can use empty messages to get just the 539"if/then" effect, in the following way: 540.Bd -literal -offset indent 5410 string MZ 542\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 543\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 544.Ed 545.Pp 546Offsets do not need to be constant, but can also be read from the file 547being examined. 548If the first character following the last 549.Em \*[Gt] 550is a 551.Em \&( 552then the string after the parenthesis is interpreted as an indirect offset. 553That means that the number after the parenthesis is used as an offset in 554the file. 555The value at that offset is read, and is used again as an offset 556in the file. 557Indirect offsets are of the form: 558.Em (( x [[.,][bBcCeEfFgGhHiIlmsSqQ]][+\-][ y ]) . 559The value of 560.Em x 561is used as an offset in the file. 562A byte, id3 length, short or long is read at that offset depending on the 563.Em [bBcCeEfFgGhHiIlmsSqQ] 564type specifier. 565The value is treated as signed if 566.Dq , 567is specified or unsigned if 568.Dq . 569is specified. 570The capitalized types interpret the number as a big endian 571value, whereas the small letter versions interpret the number as a little 572endian value; 573the 574.Em m 575type interprets the number as a middle endian (PDP-11) value. 576To that number the value of 577.Em y 578is added and the result is used as an offset in the file. 579The default type if one is not specified is long. 580The following types are recognized: 581.Bl -column -offset indent "Type" "Half/Short" "Little" "Size" 582.It Sy Type Sy Mnemonic Sy Endian Sy Size 583.It bcBc Byte/Char N/A 1 584.It efg Double Little 8 585.It EFG Double Big 8 586.It hs Half/Short Little 2 587.It HS Half/Short Big 2 588.It i ID3 Little 4 589.It I ID3 Big 4 590.It m Middle Middle 4 591.It q Quad Little 8 592.It Q Quad Big 8 593.El 594.Pp 595That way variable length structures can be examined: 596.Bd -literal -offset indent 597# MS Windows executables are also valid MS-DOS executables 5980 string MZ 599\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 600# skip the whole block below if it is not an extended executable 601\*[Gt]0x18 leshort \*[Gt]0x3f 602\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 603\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 604.Ed 605.Pp 606This strategy of examining has a drawback: you must make sure that you 607eventually print something, or users may get empty output (such as when 608there is neither PE\e0\e0 nor LE\e0\e0 in the above example). 609.Pp 610If this indirect offset cannot be used directly, simple calculations are 611possible: appending 612.Em [+-*/%\*[Am]|^]number 613inside parentheses allows one to modify 614the value read from the file before it is used as an offset: 615.Bd -literal -offset indent 616# MS Windows executables are also valid MS-DOS executables 6170 string MZ 618# sometimes, the value at 0x18 is less that 0x40 but there's still an 619# extended executable, simply appended to the file 620\*[Gt]0x18 leshort \*[Lt]0x40 621\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 622\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 623.Ed 624.Pp 625Sometimes you do not know the exact offset as this depends on the length or 626position (when indirection was used before) of preceding fields. 627You can specify an offset relative to the end of the last up-level 628field using 629.Sq \*[Am] 630as a prefix to the offset: 631.Bd -literal -offset indent 6320 string MZ 633\*[Gt]0x18 leshort \*[Gt]0x3f 634\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 635# immediately following the PE signature is the CPU type 636\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 637\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 638.Ed 639.Pp 640Indirect and relative offsets can be combined: 641.Bd -literal -offset indent 6420 string MZ 643\*[Gt]0x18 leshort \*[Lt]0x40 644\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 645# if it's not COFF, go back 512 bytes and add the offset taken 646# from byte 2/3, which is yet another way of finding the start 647# of the extended executable 648\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 649.Ed 650.Pp 651Or the other way around: 652.Bd -literal -offset indent 6530 string MZ 654\*[Gt]0x18 leshort \*[Gt]0x3f 655\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 656# at offset 0x80 (-4, since relative offsets start at the end 657# of the up-level match) inside the LE header, we find the absolute 658# offset to the code area, where we look for a specific signature 659\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 660.Ed 661.Pp 662Or even both! 663.Bd -literal -offset indent 6640 string MZ 665\*[Gt]0x18 leshort \*[Gt]0x3f 666\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 667# at offset 0x58 inside the LE header, we find the relative offset 668# to a data area where we look for a specific signature 669\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 670.Ed 671.Pp 672If you have to deal with offset/length pairs in your file, even the 673second value in a parenthesized expression can be taken from the file itself, 674using another set of parentheses. 675Note that this additional indirect offset is always relative to the 676start of the main indirect offset. 677.Bd -literal -offset indent 6780 string MZ 679\*[Gt]0x18 leshort \*[Gt]0x3f 680\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 681# search for the PE section called ".idata"... 682\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 683# ...and go to the end of it, calculated from start+length; 684# these are located 14 and 10 bytes after the section name 685\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 686.Ed 687.Pp 688If you have a list of known values at a particular continuation level, 689and you want to provide a switch-like default case: 690.Bd -literal -offset indent 691# clear that continuation level match 692\*[Gt]18 clear 693\*[Gt]18 lelong 1 one 694\*[Gt]18 lelong 2 two 695\*[Gt]18 default x 696# print default match 697\*[Gt]\*[Gt]18 lelong x unmatched 0x%x 698.Ed 699.Sh SEE ALSO 700.Xr file 1 701\- the command that reads this file. 702.Sh BUGS 703The formats 704.Dv long , 705.Dv belong , 706.Dv lelong , 707.Dv melong , 708.Dv short , 709.Dv beshort , 710and 711.Dv leshort 712do not depend on the length of the C data types 713.Dv short 714and 715.Dv long 716on the platform, even though the Single 717.Ux 718Specification implies that they do. However, as OS X Mountain Lion has 719passed the Single 720.Ux 721Specification validation suite, and supplies a version of 722.Xr file 1 723in which they do not depend on the sizes of the C data types and that is 724built for a 64-bit environment in which 725.Dv long 726is 8 bytes rather than 4 bytes, presumably the validation suite does not 727test whether, for example 728.Dv long 729refers to an item with the same size as the C data type 730.Dv long . 731There should probably be 732.Dv type 733names 734.Dv int8 , 735.Dv uint8 , 736.Dv int16 , 737.Dv uint16 , 738.Dv int32 , 739.Dv uint32 , 740.Dv int64 , 741and 742.Dv uint64 , 743and specified-byte-order variants of them, 744to make it clearer that those types have specified widths. 745.\" 746.\" From: guy@sun.uucp (Guy Harris) 747.\" Newsgroups: net.bugs.usg 748.\" Subject: /etc/magic's format isn't well documented 749.\" Message-ID: <2752@sun.uucp> 750.\" Date: 3 Sep 85 08:19:07 GMT 751.\" Organization: Sun Microsystems, Inc. 752.\" Lines: 136 753.\" 754.\" Here's a manual page for the format accepted by the "file" made by adding 755.\" the changes I posted to the S5R2 version. 756.\" 757.\" Modified for Ian Darwin's version of the file command. 758