1.\" $NetBSD: magic.5,v 1.9 2013/01/04 15:37:53 christos Exp $ 2.\" 3.\" $File: magic.man,v 1.76 2012/11/07 20:29:27 christos Exp $ 4.Dd November 7, 2012 5.Dt MAGIC 5 6.Os 7.\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. 8.Sh NAME 9.Nm magic 10.Nd file command's magic pattern file 11.Sh DESCRIPTION 12This manual page documents the format of the magic file as 13used by the 14.Xr file 1 15command, version 5.12. 16The 17.Xr file 1 18command identifies the type of a file using, 19among other tests, 20a test for whether the file contains certain 21.Dq "magic patterns" . 22The file 23.Pa /usr/share/misc/magic 24specifies what patterns are to be tested for, what message or 25MIME type to print if a particular pattern is found, 26and additional information to extract from the file. 27.Pp 28Each line of the file specifies a test to be performed. 29A test compares the data starting at a particular offset 30in the file with a byte value, a string or a numeric value. 31If the test succeeds, a message is printed. 32The line consists of the following fields: 33.Bl -tag -width ".Dv message" 34.It Dv offset 35A number specifying the offset, in bytes, into the file of the data 36which is to be tested. 37.It Dv type 38The type of the data to be tested. 39The possible values are: 40.Bl -tag -width ".Dv lestring16" 41.It Dv byte 42A one-byte value. 43.It Dv short 44A two-byte value in this machine's native byte order. 45.It Dv long 46A four-byte value in this machine's native byte order. 47.It Dv quad 48An eight-byte value in this machine's native byte order. 49.It Dv float 50A 32-bit single precision IEEE floating point number in this machine's native byte order. 51.It Dv double 52A 64-bit double precision IEEE floating point number in this machine's native byte order. 53.It Dv string 54A string of bytes. 55The string type specification can be optionally followed 56by /[WwcCtbT]*. 57The 58.Dq W 59flag compacts whitespace in the target, which must 60contain at least one whitespace character. 61If the magic has 62.Dv n 63consecutive blanks, the target needs at least 64.Dv n 65consecutive blanks to match. 66The 67.Dq w 68flag treats every blank in the magic as an optional blank. 69The 70.Dq c 71flag specifies case insensitive matching: lower case 72characters in the magic match both lower and upper case characters in the 73target, whereas upper case characters in the magic only match upper case 74characters in the target. 75The 76.Dq C 77flag specifies case insensitive matching: upper case 78characters in the magic match both lower and upper case characters in the 79target, whereas lower case characters in the magic only match upper case 80characters in the target. 81To do a complete case insensitive match, specify both 82.Dq c 83and 84.Dq C . 85The 86.Dq t 87flag forces the test to be done for text files, while the 88.Dq b 89flag forces the test to be done for binary files. 90The 91.Dq T 92flag causes the string to be trimmed, i.e. leading and trailing whitespace 93is deleted before the string is printed. 94.It Dv pstring 95A Pascal-style string where the first byte/short/int is interpreted as the 96unsigned length. 97The length defaults to byte and can be specified as a modifier. 98The following modifiers are supported: 99.Bl -tag -compact -width B 100.It B 101A byte length (default). 102.It H 103A 2 byte big endian length. 104.It h 105A 2 byte big little length. 106.It L 107A 4 byte big endian length. 108.It l 109A 4 byte big little length. 110.It J 111The length includes itself in its count. 112.El 113The string is not NUL terminated. 114.Dq J 115is used rather than the more 116valuable 117.Dq I 118because this type of length is a feature of the JPEG 119format. 120.It Dv date 121A four-byte value interpreted as a UNIX date. 122.It Dv qdate 123A eight-byte value interpreted as a UNIX date. 124.It Dv ldate 125A four-byte value interpreted as a UNIX-style date, but interpreted as 126local time rather than UTC. 127.It Dv qldate 128An eight-byte value interpreted as a UNIX-style date, but interpreted as 129local time rather than UTC. 130.It Dv qwdate 131An eight-byte value interpreted as a Windows-style date. 132.It Dv beid3 133A 32-bit ID3 length in big-endian byte order. 134.It Dv beshort 135A two-byte value in big-endian byte order. 136.It Dv belong 137A four-byte value in big-endian byte order. 138.It Dv bequad 139An eight-byte value in big-endian byte order. 140.It Dv befloat 141A 32-bit single precision IEEE floating point number in big-endian byte order. 142.It Dv bedouble 143A 64-bit double precision IEEE floating point number in big-endian byte order. 144.It Dv bedate 145A four-byte value in big-endian byte order, 146interpreted as a Unix date. 147.It Dv beqdate 148An eight-byte value in big-endian byte order, 149interpreted as a Unix date. 150.It Dv beldate 151A four-byte value in big-endian byte order, 152interpreted as a UNIX-style date, but interpreted as local time rather 153than UTC. 154.It Dv beqldate 155An eight-byte value in big-endian byte order, 156interpreted as a UNIX-style date, but interpreted as local time rather 157than UTC. 158.It Dv beqwdate 159An eight-byte value in big-endian byte order, 160interpreted as a Windows-style date. 161.It Dv bestring16 162A two-byte unicode (UCS16) string in big-endian byte order. 163.It Dv leid3 164A 32-bit ID3 length in little-endian byte order. 165.It Dv leshort 166A two-byte value in little-endian byte order. 167.It Dv lelong 168A four-byte value in little-endian byte order. 169.It Dv lequad 170An eight-byte value in little-endian byte order. 171.It Dv lefloat 172A 32-bit single precision IEEE floating point number in little-endian byte order. 173.It Dv ledouble 174A 64-bit double precision IEEE floating point number in little-endian byte order. 175.It Dv ledate 176A four-byte value in little-endian byte order, 177interpreted as a UNIX date. 178.It Dv leqdate 179An eight-byte value in little-endian byte order, 180interpreted as a UNIX date. 181.It Dv leldate 182A four-byte value in little-endian byte order, 183interpreted as a UNIX-style date, but interpreted as local time rather 184than UTC. 185.It Dv leqldate 186An eight-byte value in little-endian byte order, 187interpreted as a UNIX-style date, but interpreted as local time rather 188than UTC. 189.It Dv leqwdate 190An eight-byte value in little-endian byte order, 191interpreted as a Windows-style date. 192.It Dv lestring16 193A two-byte unicode (UCS16) string in little-endian byte order. 194.It Dv melong 195A four-byte value in middle-endian (PDP-11) byte order. 196.It Dv medate 197A four-byte value in middle-endian (PDP-11) byte order, 198interpreted as a UNIX date. 199.It Dv meldate 200A four-byte value in middle-endian (PDP-11) byte order, 201interpreted as a UNIX-style date, but interpreted as local time rather 202than UTC. 203.It Dv indirect 204Starting at the given offset, consult the magic database again. 205.It Dv name 206Define a 207.Dq named 208magic instance that can be called from another 209.Dv use 210magic entry, like a subroutine call. 211Named instance direct magic offsets are relative to the offset of the 212previous matched entry, but indirect offsets are relative to the beginning 213of the file as usual. 214Named magic entries always match. 215.It Dv use 216Recursively call the named magic starting from the current offset. 217If the name of the referenced begins with a 218.Dv ^ 219then the endianness of the magic is switched; if the magic mentioned 220.Dv leshort 221for example, 222it is treated as 223.Dv beshort 224and vice versa. 225This is useful to avoid duplicating the rules for different endianness. 226.It Dv regex 227A regular expression match in extended POSIX regular expression syntax 228(like egrep). 229Regular expressions can take exponential time to process, and their 230performance is hard to predict, so their use is discouraged. 231When used in production environments, their performance 232should be carefully checked. 233The type specification can be optionally followed by 234.Dv /[c][s] . 235The 236.Dq c 237flag makes the match case insensitive, while the 238.Dq s 239flag update the offset to the start offset of the match, rather than the end. 240The regular expression is tested against line 241.Dv N + 1 242onwards, where 243.Dv N 244is the given offset. 245Line endings are assumed to be in the machine's native format. 246.Dv ^ 247and 248.Dv $ 249match the beginning and end of individual lines, respectively, 250not beginning and end of file. 251.It Dv search 252A literal string search starting at the given offset. 253The same modifier flags can be used as for string patterns. 254The modifier flags (if any) must be followed by 255.Dv /number 256the range, that is, the number of positions at which the match will be 257attempted, starting from the start offset. 258This is suitable for 259searching larger binary expressions with variable offsets, using 260.Dv \e 261escapes for special characters. 262The offset works as for regex. 263.It Dv default 264This is intended to be used with the test 265.Em x 266(which is always true) and a message that is to be used if there are 267no other matches. 268.El 269.Pp 270Each top-level magic pattern (see below for an explanation of levels) 271is classified as text or binary according to the types used. 272Types 273.Dq regex 274and 275.Dq search 276are classified as text tests, unless non-printable characters are used 277in the pattern. 278All other tests are classified as binary. 279A top-level 280pattern is considered to be a test text when all its patterns are text 281patterns; otherwise, it is considered to be a binary pattern. 282When 283matching a file, binary patterns are tried first; if no match is 284found, and the file looks like text, then its encoding is determined 285and the text patterns are tried. 286.Pp 287The numeric types may optionally be followed by 288.Dv \*[Am] 289and a numeric value, 290to specify that the value is to be AND'ed with the 291numeric value before any comparisons are done. 292Prepending a 293.Dv u 294to the type indicates that ordered comparisons should be unsigned. 295.It Dv test 296The value to be compared with the value from the file. 297If the type is 298numeric, this value 299is specified in C form; if it is a string, it is specified as a C string 300with the usual escapes permitted (e.g. \en for new-line). 301.Pp 302Numeric values 303may be preceded by a character indicating the operation to be performed. 304It may be 305.Dv = , 306to specify that the value from the file must equal the specified value, 307.Dv \*[Lt] , 308to specify that the value from the file must be less than the specified 309value, 310.Dv \*[Gt] , 311to specify that the value from the file must be greater than the specified 312value, 313.Dv \*[Am] , 314to specify that the value from the file must have set all of the bits 315that are set in the specified value, 316.Dv ^ , 317to specify that the value from the file must have clear any of the bits 318that are set in the specified value, or 319.Dv ~ , 320the value specified after is negated before tested. 321.Dv x , 322to specify that any value will match. 323If the character is omitted, it is assumed to be 324.Dv = . 325Operators 326.Dv \*[Am] , 327.Dv ^ , 328and 329.Dv ~ 330don't work with floats and doubles. 331The operator 332.Dv !\& 333specifies that the line matches if the test does 334.Em not 335succeed. 336.Pp 337Numeric values are specified in C form; e.g. 338.Dv 13 339is decimal, 340.Dv 013 341is octal, and 342.Dv 0x13 343is hexadecimal. 344.Pp 345For string values, the string from the 346file must match the specified string. 347The operators 348.Dv = , 349.Dv \*[Lt] 350and 351.Dv \*[Gt] 352(but not 353.Dv \*[Am] ) 354can be applied to strings. 355The length used for matching is that of the string argument 356in the magic file. 357This means that a line can match any non-empty string (usually used to 358then print the string), with 359.Em \*[Gt]\e0 360(because all non-empty strings are greater than the empty string). 361.Pp 362The special test 363.Em x 364always evaluates to true. 365.It Dv message 366The message to be printed if the comparison succeeds. 367If the string contains a 368.Xr printf 3 369format specification, the value from the file (with any specified masking 370performed) is printed using the message as the format string. 371If the string begins with 372.Dq \eb , 373the message printed is the remainder of the string with no whitespace 374added before it: multiple matches are normally separated by a single 375space. 376.El 377.Pp 378An APPLE 4+4 character APPLE creator and type can be specified as: 379.Bd -literal -offset indent 380!:apple CREATYPE 381.Ed 382.Pp 383A MIME type is given on a separate line, which must be the next 384non-blank or comment line after the magic line that identifies the 385file type, and has the following format: 386.Bd -literal -offset indent 387!:mime MIMETYPE 388.Ed 389.Pp 390i.e. the literal string 391.Dq !:mime 392followed by the MIME type. 393.Pp 394An optional strength can be supplied on a separate line which refers to 395the current magic description using the following format: 396.Bd -literal -offset indent 397!:strength OP VALUE 398.Ed 399.Pp 400The operand 401.Dv OP 402can be: 403.Dv + , 404.Dv - , 405.Dv * , 406or 407.Dv / 408and 409.Dv VALUE 410is a constant between 0 and 255. 411This constant is applied using the specified operand 412to the currently computed default magic strength. 413.Pp 414Some file formats contain additional information which is to be printed 415along with the file type or need additional tests to determine the true 416file type. 417These additional tests are introduced by one or more 418.Em \*[Gt] 419characters preceding the offset. 420The number of 421.Em \*[Gt] 422on the line indicates the level of the test; a line with no 423.Em \*[Gt] 424at the beginning is considered to be at level 0. 425Tests are arranged in a tree-like hierarchy: 426if the test on a line at level 427.Em n 428succeeds, all following tests at level 429.Em n+1 430are performed, and the messages printed if the tests succeed, until a line 431with level 432.Em n 433(or less) appears. 434For more complex files, one can use empty messages to get just the 435"if/then" effect, in the following way: 436.Bd -literal -offset indent 4370 string MZ 438\*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable 439\*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows) 440.Ed 441.Pp 442Offsets do not need to be constant, but can also be read from the file 443being examined. 444If the first character following the last 445.Em \*[Gt] 446is a 447.Em \&( 448then the string after the parenthesis is interpreted as an indirect offset. 449That means that the number after the parenthesis is used as an offset in 450the file. 451The value at that offset is read, and is used again as an offset 452in the file. 453Indirect offsets are of the form: 454.Em (( x [.[bislBISL]][+\-][ y ]) . 455The value of 456.Em x 457is used as an offset in the file. 458A byte, id3 length, short or long is read at that offset depending on the 459.Em [bislBISLm] 460type specifier. 461The capitalized types interpret the number as a big endian 462value, whereas the small letter versions interpret the number as a little 463endian value; 464the 465.Em m 466type interprets the number as a middle endian (PDP-11) value. 467To that number the value of 468.Em y 469is added and the result is used as an offset in the file. 470The default type if one is not specified is long. 471.Pp 472That way variable length structures can be examined: 473.Bd -literal -offset indent 474# MS Windows executables are also valid MS-DOS executables 4750 string MZ 476\*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS) 477# skip the whole block below if it is not an extended executable 478\*[Gt]0x18 leshort \*[Gt]0x3f 479\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 480\*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2) 481.Ed 482.Pp 483This strategy of examining has a drawback: You must make sure that 484you eventually print something, or users may get empty output (like, when 485there is neither PE\e0\e0 nor LE\e0\e0 in the above example) 486.Pp 487If this indirect offset cannot be used directly, simple calculations are 488possible: appending 489.Em [+-*/%\*[Am]|^]number 490inside parentheses allows one to modify 491the value read from the file before it is used as an offset: 492.Bd -literal -offset indent 493# MS Windows executables are also valid MS-DOS executables 4940 string MZ 495# sometimes, the value at 0x18 is less that 0x40 but there's still an 496# extended executable, simply appended to the file 497\*[Gt]0x18 leshort \*[Lt]0x40 498\*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP) 499\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 500.Ed 501.Pp 502Sometimes you do not know the exact offset as this depends on the length or 503position (when indirection was used before) of preceding fields. 504You can specify an offset relative to the end of the last up-level 505field using 506.Sq \*[Am] 507as a prefix to the offset: 508.Bd -literal -offset indent 5090 string MZ 510\*[Gt]0x18 leshort \*[Gt]0x3f 511\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 512# immediately following the PE signature is the CPU type 513\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386 514\*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha 515.Ed 516.Pp 517Indirect and relative offsets can be combined: 518.Bd -literal -offset indent 5190 string MZ 520\*[Gt]0x18 leshort \*[Lt]0x40 521\*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS) 522# if it's not COFF, go back 512 bytes and add the offset taken 523# from byte 2/3, which is yet another way of finding the start 524# of the extended executable 525\*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver) 526.Ed 527.Pp 528Or the other way around: 529.Bd -literal -offset indent 5300 string MZ 531\*[Gt]0x18 leshort \*[Gt]0x3f 532\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 533# at offset 0x80 (-4, since relative offsets start at the end 534# of the up-level match) inside the LE header, we find the absolute 535# offset to the code area, where we look for a specific signature 536\*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed 537.Ed 538.Pp 539Or even both! 540.Bd -literal -offset indent 5410 string MZ 542\*[Gt]0x18 leshort \*[Gt]0x3f 543\*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows) 544# at offset 0x58 inside the LE header, we find the relative offset 545# to a data area where we look for a specific signature 546\*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive 547.Ed 548.Pp 549Finally, if you have to deal with offset/length pairs in your file, even the 550second value in a parenthesized expression can be taken from the file itself, 551using another set of parentheses. 552Note that this additional indirect offset is always relative to the 553start of the main indirect offset. 554.Bd -literal -offset indent 5550 string MZ 556\*[Gt]0x18 leshort \*[Gt]0x3f 557\*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows) 558# search for the PE section called ".idata"... 559\*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata 560# ...and go to the end of it, calculated from start+length; 561# these are located 14 and 10 bytes after the section name 562\*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive 563.Ed 564.Sh SEE ALSO 565.Xr file 1 566\- the command that reads this file. 567.Sh BUGS 568The formats 569.Dv long , 570.Dv belong , 571.Dv lelong , 572.Dv melong , 573.Dv short , 574.Dv beshort , 575.Dv leshort , 576.Dv date , 577.Dv bedate , 578.Dv medate , 579.Dv ledate , 580.Dv beldate , 581.Dv leldate , 582and 583.Dv meldate 584are system-dependent; perhaps they should be specified as a number 585of bytes (2B, 4B, etc), 586since the files being recognized typically come from 587a system on which the lengths are invariant. 588.\" 589.\" From: guy@sun.uucp (Guy Harris) 590.\" Newsgroups: net.bugs.usg 591.\" Subject: /etc/magic's format isn't well documented 592.\" Message-ID: <2752@sun.uucp> 593.\" Date: 3 Sep 85 08:19:07 GMT 594.\" Organization: Sun Microsystems, Inc. 595.\" Lines: 136 596.\" 597.\" Here's a manual page for the format accepted by the "file" made by adding 598.\" the changes I posted to the S5R2 version. 599.\" 600.\" Modified for Ian Darwin's version of the file command. 601