1.\" $File: magic.man,v 1.66 2011/01/06 23:54:41 rrt Exp $ 2.Dd August 30, 2008 3.Dt MAGIC __FSECTION__ 4.Os 5.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 6.Sh NAME 7.Nm magic 8.Nd file command's magic pattern file 9.Sh DESCRIPTION 10This manual page documents the format of the magic file as 11used by the 12.Xr file __CSECTION__ 13command, version __VERSION__. 14The 15.Xr file __CSECTION__ 16command identifies the type of a file using, 17among other tests, 18a test for whether the file contains certain 19.Dq "magic patterns" . 20The file 21.Pa __MAGIC__ 22specifies what patterns are to be tested for, what message or 23MIME type to print if a particular pattern is found, 24and additional information to extract from the file. 25.Pp 26Each line of the file specifies a test to be performed. 27A test compares the data starting at a particular offset 28in the file with a byte value, a string or a numeric value. 29If the test succeeds, a message is printed. 30The line consists of the following fields: 31.Bl -tag -width ".Dv message" 32.It Dv offset 33A number specifying the offset, in bytes, into the file of the data 34which is to be tested. 35.It Dv type 36The type of the data to be tested. 37The possible values are: 38.Bl -tag -width ".Dv lestring16" 39.It Dv byte 40A one-byte value. 41.It Dv short 42A two-byte value in this machine's native byte order. 43.It Dv long 44A four-byte value in this machine's native byte order. 45.It Dv quad 46An eight-byte value in this machine's native byte order. 47.It Dv float 48A 32-bit single precision IEEE floating point number in this machine's native byte order. 49.It Dv double 50A 64-bit double precision IEEE floating point number in this machine's native byte order. 51.It Dv string 52A string of bytes. 53The string type specification can be optionally followed 54by /[Bbc]*. 55The 56.Dq B 57flag compacts whitespace in the target, which must 58contain at least one whitespace character. 59If the magic has 60.Dv n 61consecutive blanks, the target needs at least 62.Dv n 63consecutive blanks to match. 64The 65.Dq b 66flag treats every blank in the target as an optional blank. 67Finally the 68.Dq c 69flag, specifies case insensitive matching: lowercase 70characters in the magic match both lower and upper case characters in the 71target, whereas upper case characters in the magic only match uppercase 72characters in the target. 73.It Dv pstring 74A Pascal-style string where the first byte/short/int is interpreted as the an 75unsigned length. 76The length defaults to byte and can be specified as a modifier. 77The following modifiers are supported: 78.Bl -tag -compact -width B 79.It B 80A byte length (default). 81.It H 82A 2 byte big endian length. 83.It h 84A 2 byte big little length. 85.It L 86A 4 byte big endian length. 87.It l 88A 4 byte big little length. 89.It J 90The length includes itself in its count. 91.El 92The string is not NUL terminated. 93.Dq J 94is used rather than the more 95valuable 96.Dq I 97because this type of length is a feature of the JPEG 98format. 99.It Dv date 100A four-byte value interpreted as a UNIX date. 101.It Dv qdate 102A eight-byte value interpreted as a UNIX date. 103.It Dv ldate 104A four-byte value interpreted as a UNIX-style date, but interpreted as 105local time rather than UTC. 106.It Dv qldate 107An eight-byte value interpreted as a UNIX-style date, but interpreted as 108local time rather than UTC. 109.It Dv beid3 110A 32-bit ID3 length in big-endian byte order. 111.It Dv beshort 112A two-byte value in big-endian byte order. 113.It Dv belong 114A four-byte value in big-endian byte order. 115.It Dv bequad 116An eight-byte value in big-endian byte order. 117.It Dv befloat 118A 32-bit single precision IEEE floating point number in big-endian byte order. 119.It Dv bedouble 120A 64-bit double precision IEEE floating point number in big-endian byte order. 121.It Dv bedate 122A four-byte value in big-endian byte order, 123interpreted as a Unix date. 124.It Dv beqdate 125An eight-byte value in big-endian byte order, 126interpreted as a Unix date. 127.It Dv beldate 128A four-byte value in big-endian byte order, 129interpreted as a UNIX-style date, but interpreted as local time rather 130than UTC. 131.It Dv beqldate 132An eight-byte value in big-endian byte order, 133interpreted as a UNIX-style date, but interpreted as local time rather 134than UTC. 135.It Dv bestring16 136A two-byte unicode (UCS16) string in big-endian byte order. 137.It Dv leid3 138A 32-bit ID3 length in little-endian byte order. 139.It Dv leshort 140A two-byte value in little-endian byte order. 141.It Dv lelong 142A four-byte value in little-endian byte order. 143.It Dv lequad 144An eight-byte value in little-endian byte order. 145.It Dv lefloat 146A 32-bit single precision IEEE floating point number in little-endian byte order. 147.It Dv ledouble 148A 64-bit double precision IEEE floating point number in little-endian byte order. 149.It Dv ledate 150A four-byte value in little-endian byte order, 151interpreted as a UNIX date. 152.It Dv leqdate 153An eight-byte value in little-endian byte order, 154interpreted as a UNIX date. 155.It Dv leldate 156A four-byte value in little-endian byte order, 157interpreted as a UNIX-style date, but interpreted as local time rather 158than UTC. 159.It Dv leqldate 160An eight-byte value in little-endian byte order, 161interpreted as a UNIX-style date, but interpreted as local time rather 162than UTC. 163.It Dv lestring16 164A two-byte unicode (UCS16) string in little-endian byte order. 165.It Dv melong 166A four-byte value in middle-endian (PDP-11) byte order. 167.It Dv medate 168A four-byte value in middle-endian (PDP-11) byte order, 169interpreted as a UNIX date. 170.It Dv meldate 171A four-byte value in middle-endian (PDP-11) byte order, 172interpreted as a UNIX-style date, but interpreted as local time rather 173than UTC. 174.It Dv indirect 175Starting at the given offset, consult the magic database again. 176.It Dv regex 177A regular expression match in extended POSIX regular expression syntax 178(like egrep). Regular expressions can take exponential time to 179process, and their performance is hard to predict, so their use is 180discouraged. When used in production environments, their performance 181should be carefully checked. The type specification can be optionally 182followed by 183.Dv /[c][s] . 184The 185.Dq c 186flag makes the match case insensitive, while the 187.Dq s 188flag update the offset to the start offset of the match, rather than the end. 189The regular expression is tested against line 190.Dv N + 1 191onwards, where 192.Dv N 193is the given offset. 194Line endings are assumed to be in the machine's native format. 195.Dv ^ 196and 197.Dv $ 198match the beginning and end of individual lines, respectively, 199not beginning and end of file. 200.It Dv search 201A literal string search starting at the given offset. The same 202modifier flags can be used as for string patterns. The modifier flags 203(if any) must be followed by 204.Dv /number 205the range, that is, the number of positions at which the match will be 206attempted, starting from the start offset. This is suitable for 207searching larger binary expressions with variable offsets, using 208.Dv \e 209escapes for special characters. The offset works as for regex. 210.It Dv default 211This is intended to be used with the test 212.Em x 213(which is always true) and a message that is to be used if there are 214no other matches. 215.El 216.Pp 217Each top-level magic pattern (see below for an explanation of levels) 218is classified as text or binary according to the types used. Types 219.Dq regex 220and 221.Dq search 222are classified as text tests, unless non-printable characters are used 223in the pattern. All other tests are classified as binary. A top-level 224pattern is considered to be a test text when all its patterns are text 225patterns; otherwise, it is considered to be a binary pattern. When 226matching a file, binary patterns are tried first; if no match is 227found, and the file looks like text, then its encoding is determined 228and the text patterns are tried. 229.Pp 230The numeric types may optionally be followed by 231.Dv \*[Am] 232and a numeric value, 233to specify that the value is to be AND'ed with the 234numeric value before any comparisons are done. 235Prepending a 236.Dv u 237to the type indicates that ordered comparisons should be unsigned. 238.It Dv test 239The value to be compared with the value from the file. 240If the type is 241numeric, this value 242is specified in C form; if it is a string, it is specified as a C string 243with the usual escapes permitted (e.g. \en for new-line). 244.Pp 245Numeric values 246may be preceded by a character indicating the operation to be performed. 247It may be 248.Dv = , 249to specify that the value from the file must equal the specified value, 250.Dv \*[Lt] , 251to specify that the value from the file must be less than the specified 252value, 253.Dv \*[Gt] , 254to specify that the value from the file must be greater than the specified 255value, 256.Dv \*[Am] , 257to specify that the value from the file must have set all of the bits 258that are set in the specified value, 259.Dv ^ , 260to specify that the value from the file must have clear any of the bits 261that are set in the specified value, or 262.Dv ~ , 263the value specified after is negated before tested. 264.Dv x , 265to specify that any value will match. 266If the character is omitted, it is assumed to be 267.Dv = . 268Operators 269.Dv \*[Am] , 270.Dv ^ , 271and 272.Dv ~ 273don't work with floats and doubles. 274The operator 275.Dv !\& 276specifies that the line matches if the test does 277.Em not 278succeed. 279.Pp 280Numeric values are specified in C form; e.g. 281.Dv 13 282is decimal, 283.Dv 013 284is octal, and 285.Dv 0x13 286is hexadecimal. 287.Pp 288For string values, the string from the 289file must match the specified string. 290The operators 291.Dv = , 292.Dv \*[Lt] 293and 294.Dv \*[Gt] 295(but not 296.Dv \*[Am] ) 297can be applied to strings. 298The length used for matching is that of the string argument 299in the magic file. 300This means that a line can match any non-empty string (usually used to 301then print the string), with 302.Em \*[Gt]\e0 303(because all non-empty strings are greater than the empty string). 304.Pp 305The special test 306.Em x 307always evaluates to true. 308.It Dv message 309The message to be printed if the comparison succeeds. 310If the string contains a 311.Xr printf 3 312format specification, the value from the file (with any specified masking 313performed) is printed using the message as the format string. 314If the string begins with 315.Dq \eb , 316the message printed is the remainder of the string with no whitespace 317added before it: multiple matches are normally separated by a single 318space. 319.El 320.Pp 321An APPLE 4+4 character APPLE creator and type can be specified as: 322.Bd -literal -offset indent 323!:apple CREATYPE 324.Ed 325.Pp 326A MIME type is given on a separate line, which must be the next 327non-blank or comment line after the magic line that identifies the 328file type, and has the following format: 329.Bd -literal -offset indent 330!:mime MIMETYPE 331.Ed 332.Pp 333i.e. the literal string 334.Dq !:mime 335followed by the MIME type. 336.Pp 337An optional strength can be supplied on a separate line which refers to 338the current magic description using the following format: 339.Bd -literal -offset indent 340!:strength OP VALUE 341.Ed 342.Pp 343The operand 344.Dv OP 345can be: 346.Dv + , 347.Dv - , 348.Dv * , 349or 350.Dv / 351and 352.Dv VALUE 353is a constant between 0 and 255. 354This constant is applied using the specified operand 355to the currently computed default magic strength. 356.Pp 357Some file formats contain additional information which is to be printed 358along with the file type or need additional tests to determine the true 359file type. 360These additional tests are introduced by one or more 361.Em \*[Gt] 362characters preceding the offset. 363The number of 364.Em \*[Gt] 365on the line indicates the level of the test; a line with no 366.Em \*[Gt] 367at the beginning is considered to be at level 0. 368Tests are arranged in a tree-like hierarchy: 369if the test on a line at level 370.Em n 371succeeds, all following tests at level 372.Em n+1 373are performed, and the messages printed if the tests succeed, until a line 374with level 375.Em n 376(or less) appears. 377For more complex files, one can use empty messages to get just the 378"if/then" effect, in the following way: 379.Bd -literal -offset indent 3800 string MZ 381\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 382\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 383.Ed 384.Pp 385Offsets do not need to be constant, but can also be read from the file 386being examined. 387If the first character following the last 388.Em \*[Gt] 389is a 390.Em \&( 391then the string after the parenthesis is interpreted as an indirect offset. 392That means that the number after the parenthesis is used as an offset in 393the file. 394The value at that offset is read, and is used again as an offset 395in the file. 396Indirect offsets are of the form: 397.Em (( x [.[bislBISL]][+\-][ y ]) . 398The value of 399.Em x 400is used as an offset in the file. 401A byte, id3 length, short or long is read at that offset depending on the 402.Em [bislBISLm] 403type specifier. 404The capitalized types interpret the number as a big endian 405value, whereas the small letter versions interpret the number as a little 406endian value; 407the 408.Em m 409type interprets the number as a middle endian (PDP-11) value. 410To that number the value of 411.Em y 412is added and the result is used as an offset in the file. 413The default type if one is not specified is long. 414.Pp 415That way variable length structures can be examined: 416.Bd -literal -offset indent 417# MS Windows executables are also valid MS-DOS executables 4180 string MZ 419\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 420# skip the whole block below if it is not an extended executable 421\*[Gt]0x18 leshort \*[Gt]0x3f 422\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 423\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 424.Ed 425.Pp 426This strategy of examining has a drawback: You must make sure that 427you eventually print something, or users may get empty output (like, when 428there is neither PE\e0\e0 nor LE\e0\e0 in the above example) 429.Pp 430If this indirect offset cannot be used directly, simple calculations are 431possible: appending 432.Em [+-*/%\*[Am]|^]number 433inside parentheses allows one to modify 434the value read from the file before it is used as an offset: 435.Bd -literal -offset indent 436# MS Windows executables are also valid MS-DOS executables 4370 string MZ 438# sometimes, the value at 0x18 is less that 0x40 but there's still an 439# extended executable, simply appended to the file 440\*[Gt]0x18 leshort \*[Lt]0x40 441\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 442\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 443.Ed 444.Pp 445Sometimes you do not know the exact offset as this depends on the length or 446position (when indirection was used before) of preceding fields. 447You can specify an offset relative to the end of the last up-level 448field using 449.Sq \*[Am] 450as a prefix to the offset: 451.Bd -literal -offset indent 4520 string MZ 453\*[Gt]0x18 leshort \*[Gt]0x3f 454\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 455# immediately following the PE signature is the CPU type 456\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 457\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 458.Ed 459.Pp 460Indirect and relative offsets can be combined: 461.Bd -literal -offset indent 4620 string MZ 463\*[Gt]0x18 leshort \*[Lt]0x40 464\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 465# if it's not COFF, go back 512 bytes and add the offset taken 466# from byte 2/3, which is yet another way of finding the start 467# of the extended executable 468\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 469.Ed 470.Pp 471Or the other way around: 472.Bd -literal -offset indent 4730 string MZ 474\*[Gt]0x18 leshort \*[Gt]0x3f 475\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 476# at offset 0x80 (-4, since relative offsets start at the end 477# of the up-level match) inside the LE header, we find the absolute 478# offset to the code area, where we look for a specific signature 479\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 480.Ed 481.Pp 482Or even both! 483.Bd -literal -offset indent 4840 string MZ 485\*[Gt]0x18 leshort \*[Gt]0x3f 486\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 487# at offset 0x58 inside the LE header, we find the relative offset 488# to a data area where we look for a specific signature 489\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 490.Ed 491.Pp 492Finally, if you have to deal with offset/length pairs in your file, even the 493second value in a parenthesized expression can be taken from the file itself, 494using another set of parentheses. 495Note that this additional indirect offset is always relative to the 496start of the main indirect offset. 497.Bd -literal -offset indent 4980 string MZ 499\*[Gt]0x18 leshort \*[Gt]0x3f 500\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 501# search for the PE section called ".idata"... 502\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 503# ...and go to the end of it, calculated from start+length; 504# these are located 14 and 10 bytes after the section name 505\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 506.Ed 507.Sh SEE ALSO 508.Xr file __CSECTION__ 509\- the command that reads this file. 510.Sh BUGS 511The formats 512.Dv long , 513.Dv belong , 514.Dv lelong , 515.Dv melong , 516.Dv short , 517.Dv beshort , 518.Dv leshort , 519.Dv date , 520.Dv bedate , 521.Dv medate , 522.Dv ledate , 523.Dv beldate , 524.Dv leldate , 525and 526.Dv meldate 527are system-dependent; perhaps they should be specified as a number 528of bytes (2B, 4B, etc), 529since the files being recognized typically come from 530a system on which the lengths are invariant. 531.\" 532.\" From: guy@sun.uucp (Guy Harris) 533.\" Newsgroups: net.bugs.usg 534.\" Subject: /etc/magic's format isn't well documented 535.\" Message-ID: <2752@sun.uucp> 536.\" Date: 3 Sep 85 08:19:07 GMT 537.\" Organization: Sun Microsystems, Inc. 538.\" Lines: 136 539.\" 540.\" Here's a manual page for the format accepted by the "file" made by adding 541.\" the changes I posted to the S5R2 version. 542.\" 543.\" Modified for Ian Darwin's version of the file command. 544