xref: /openbsd-src/gnu/usr.bin/perl/pod/perldebguts.pod (revision e068048151d29f2562a32185e21a8ba885482260)
1=head1 NAME
2
3perldebguts - Guts of Perl debugging
4
5=head1 DESCRIPTION
6
7This is not L<perldebug>, which tells you how to use
8the debugger.  This manpage describes low-level details concerning
9the debugger's internals, which range from difficult to impossible
10to understand for anyone who isn't incredibly intimate with Perl's guts.
11Caveat lector.
12
13=head1 Debugger Internals
14
15Perl has special debugging hooks at compile-time and run-time used
16to create debugging environments.  These hooks are not to be confused
17with the I<perl -Dxxx> command described in L<perlrun|perlrun/-Dletters>,
18which is usable only if a special Perl is built per the instructions in
19the F<INSTALL> file in the Perl source tree.
20
21For example, whenever you call Perl's built-in C<caller> function
22from the package C<DB>, the arguments that the corresponding stack
23frame was called with are copied to the C<@DB::args> array.  These
24mechanisms are enabled by calling Perl with the B<-d> switch.
25Specifically, the following additional features are enabled
26(cf. L<perlvar/$^P>):
27
28=over 4
29
30=item *
31
32Perl inserts the contents of C<$ENV{PERL5DB}> (or C<BEGIN {require
33'perl5db.pl'}> if not present) before the first line of your program.
34
35=item *
36
37Each array C<@{"_<$filename"}> holds the lines of $filename for a
38file compiled by Perl.  The same is also true for C<eval>ed strings
39that contain subroutines, or which are currently being executed.
40The $filename for C<eval>ed strings looks like C<(eval 34)>.
41
42Values in this array are magical in numeric context: they compare
43equal to zero only if the line is not breakable.
44
45=item *
46
47Each hash C<%{"_<$filename"}> contains breakpoints and actions keyed
48by line number.  Individual entries (as opposed to the whole hash)
49are settable.  Perl only cares about Boolean true here, although
50the values used by F<perl5db.pl> have the form
51C<"$break_condition\0$action">.
52
53The same holds for evaluated strings that contain subroutines, or
54which are currently being executed.  The $filename for C<eval>ed strings
55looks like C<(eval 34)>.
56
57=item *
58
59Each scalar C<${"_<$filename"}> contains C<$filename>.  This is
60also the case for evaluated strings that contain subroutines, or
61which are currently being executed.  The C<$filename> for C<eval>ed
62strings looks like C<(eval 34)>.
63
64=item *
65
66After each C<require>d file is compiled, but before it is executed,
67C<DB::postponed(*{"_<$filename"})> is called if the subroutine
68C<DB::postponed> exists.  Here, the $filename is the expanded name of
69the C<require>d file, as found in the values of %INC.
70
71=item *
72
73After each subroutine C<subname> is compiled, the existence of
74C<$DB::postponed{subname}> is checked.  If this key exists,
75C<DB::postponed(subname)> is called if the C<DB::postponed> subroutine
76also exists.
77
78=item *
79
80A hash C<%DB::sub> is maintained, whose keys are subroutine names
81and whose values have the form C<filename:startline-endline>.
82C<filename> has the form C<(eval 34)> for subroutines defined inside
83C<eval>s.
84
85=item *
86
87When the execution of your program reaches a point that can hold a
88breakpoint, the C<DB::DB()> subroutine is called if any of the variables
89C<$DB::trace>, C<$DB::single>, or C<$DB::signal> is true.  These variables
90are not C<local>izable.  This feature is disabled when executing
91inside C<DB::DB()>, including functions called from it
92unless C<< $^D & (1<<30) >> is true.
93
94=item *
95
96When execution of the program reaches a subroutine call, a call to
97C<&DB::sub>(I<args>) is made instead, with C<$DB::sub> set to identify
98the called subroutine.  (This doesn't happen if the calling subroutine
99was compiled in the C<DB> package.)  C<$DB::sub> normally holds the name
100of the called subroutine, if it has a name by which it can be looked up.
101Failing that, C<$DB::sub> will hold a reference to the called subroutine.
102Either way, the C<&DB::sub> subroutine can use C<$DB::sub> as a reference
103by which to call the called subroutine, which it will normally want to do.
104
105X<&DB::lsub>If the call is to an lvalue subroutine, and C<&DB::lsub>
106is defined C<&DB::lsub>(I<args>) is called instead, otherwise falling
107back to C<&DB::sub>(I<args>).
108
109=item *
110
111When execution of the program uses C<goto> to enter a non-XS subroutine
112and the 0x80 bit is set in C<$^P>, a call to C<&DB::goto> is made, with
113C<$DB::sub> set to identify the subroutine being entered.  The call to
114C<&DB::goto> does not replace the C<goto>; the requested subroutine will
115still be entered once C<&DB::goto> has returned.  C<$DB::sub> normally
116holds the name of the subroutine being entered, if it has one.  Failing
117that, C<$DB::sub> will hold a reference to the subroutine being entered.
118Unlike when C<&DB::sub> is called, it is not guaranteed that C<$DB::sub>
119can be used as a reference to operate on the subroutine being entered.
120
121=back
122
123Note that if C<&DB::sub> needs external data for it to work, no
124subroutine call is possible without it. As an example, the standard
125debugger's C<&DB::sub> depends on the C<$DB::deep> variable
126(it defines how many levels of recursion deep into the debugger you can go
127before a mandatory break).  If C<$DB::deep> is not defined, subroutine
128calls are not possible, even though C<&DB::sub> exists.
129
130=head2 Writing Your Own Debugger
131
132=head3 Environment Variables
133
134The C<PERL5DB> environment variable can be used to define a debugger.
135For example, the minimal "working" debugger (it actually doesn't do anything)
136consists of one line:
137
138  sub DB::DB {}
139
140It can easily be defined like this:
141
142  $ PERL5DB="sub DB::DB {}" perl -d your-script
143
144Another brief debugger, slightly more useful, can be created
145with only the line:
146
147  sub DB::DB {print ++$i; scalar <STDIN>}
148
149This debugger prints a number which increments for each statement
150encountered and waits for you to hit a newline before continuing
151to the next statement.
152
153The following debugger is actually useful:
154
155  {
156    package DB;
157    sub DB  {}
158    sub sub {print ++$i, " $sub\n"; &$sub}
159  }
160
161It prints the sequence number of each subroutine call and the name of the
162called subroutine.  Note that C<&DB::sub> is being compiled into the
163package C<DB> through the use of the C<package> directive.
164
165When it starts, the debugger reads your rc file (F<./.perldb> or
166F<~/.perldb> under Unix), which can set important options.
167(A subroutine (C<&afterinit>) can be defined here as well; it is executed
168after the debugger completes its own initialization.)
169
170After the rc file is read, the debugger reads the PERLDB_OPTS
171environment variable and uses it to set debugger options. The
172contents of this variable are treated as if they were the argument
173of an C<o ...> debugger command (q.v. in L<perldebug/"Configurable Options">).
174
175=head3 Debugger Internal Variables
176
177In addition to the file and subroutine-related variables mentioned above,
178the debugger also maintains various magical internal variables.
179
180=over 4
181
182=item *
183
184C<@DB::dbline> is an alias for C<@{"::_<current_file"}>, which
185holds the lines of the currently-selected file (compiled by Perl), either
186explicitly chosen with the debugger's C<f> command, or implicitly by flow
187of execution.
188
189Values in this array are magical in numeric context: they compare
190equal to zero only if the line is not breakable.
191
192=item *
193
194C<%DB::dbline> is an alias for C<%{"::_<current_file"}>, which
195contains breakpoints and actions keyed by line number in
196the currently-selected file, either explicitly chosen with the
197debugger's C<f> command, or implicitly by flow of execution.
198
199As previously noted, individual entries (as opposed to the whole hash)
200are settable.  Perl only cares about Boolean true here, although
201the values used by F<perl5db.pl> have the form
202C<"$break_condition\0$action">.
203
204=back
205
206=head3 Debugger Customization Functions
207
208Some functions are provided to simplify customization.
209
210=over 4
211
212=item *
213
214See L<perldebug/"Configurable Options"> for a description of options parsed by
215C<DB::parse_options(string)>.
216
217=item *
218
219C<DB::dump_trace(skip[,count])> skips the specified number of frames
220and returns a list containing information about the calling frames (all
221of them, if C<count> is missing).  Each entry is reference to a hash
222with keys C<context> (either C<.>, C<$>, or C<@>), C<sub> (subroutine
223name, or info about C<eval>), C<args> (C<undef> or a reference to
224an array), C<file>, and C<line>.
225
226=item *
227
228C<DB::print_trace(FH, skip[, count[, short]])> prints
229formatted info about caller frames.  The last two functions may be
230convenient as arguments to C<< < >>, C<< << >> commands.
231
232=back
233
234Note that any variables and functions that are not documented in
235this manpages (or in L<perldebug>) are considered for internal
236use only, and as such are subject to change without notice.
237
238=head1 Frame Listing Output Examples
239
240The C<frame> option can be used to control the output of frame
241information.  For example, contrast this expression trace:
242
243 $ perl -de 42
244 Stack dump during die enabled outside of evals.
245
246 Loading DB routines from perl5db.pl patch level 0.94
247 Emacs support available.
248
249 Enter h or 'h h' for help.
250
251 main::(-e:1):   0
252   DB<1> sub foo { 14 }
253
254   DB<2> sub bar { 3 }
255
256   DB<3> t print foo() * bar()
257 main::((eval 172):3):   print foo() + bar();
258 main::foo((eval 168):2):
259 main::bar((eval 170):2):
260 42
261
262with this one, once the C<o>ption C<frame=2> has been set:
263
264   DB<4> o f=2
265                frame = '2'
266   DB<5> t print foo() * bar()
267 3:      foo() * bar()
268 entering main::foo
269  2:     sub foo { 14 };
270 exited main::foo
271 entering main::bar
272  2:     sub bar { 3 };
273 exited main::bar
274 42
275
276By way of demonstration, we present below a laborious listing
277resulting from setting your C<PERLDB_OPTS> environment variable to
278the value C<f=n N>, and running I<perl -d -V> from the command line.
279Examples using various values of C<n> are shown to give you a feel
280for the difference between settings.  Long though it may be, this
281is not a complete listing, but only excerpts.
282
283=over 4
284
285=item 1
286
287 entering main::BEGIN
288  entering Config::BEGIN
289   Package lib/Exporter.pm.
290   Package lib/Carp.pm.
291  Package lib/Config.pm.
292  entering Config::TIEHASH
293  entering Exporter::import
294   entering Exporter::export
295 entering Config::myconfig
296  entering Config::FETCH
297  entering Config::FETCH
298  entering Config::FETCH
299  entering Config::FETCH
300
301=item 2
302
303 entering main::BEGIN
304  entering Config::BEGIN
305   Package lib/Exporter.pm.
306   Package lib/Carp.pm.
307  exited Config::BEGIN
308  Package lib/Config.pm.
309  entering Config::TIEHASH
310  exited Config::TIEHASH
311  entering Exporter::import
312   entering Exporter::export
313   exited Exporter::export
314  exited Exporter::import
315 exited main::BEGIN
316 entering Config::myconfig
317  entering Config::FETCH
318  exited Config::FETCH
319  entering Config::FETCH
320  exited Config::FETCH
321  entering Config::FETCH
322
323=item 3
324
325 in  $=main::BEGIN() from /dev/null:0
326  in  $=Config::BEGIN() from lib/Config.pm:2
327   Package lib/Exporter.pm.
328   Package lib/Carp.pm.
329  Package lib/Config.pm.
330  in  $=Config::TIEHASH('Config') from lib/Config.pm:644
331  in  $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
332   in  $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from li
333 in  @=Config::myconfig() from /dev/null:0
334  in  $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
335  in  $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
336  in  $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
337  in  $=Config::FETCH(ref(Config), 'PERL_SUBVERSION') from lib/Config.pm:574
338  in  $=Config::FETCH(ref(Config), 'osname') from lib/Config.pm:574
339  in  $=Config::FETCH(ref(Config), 'osvers') from lib/Config.pm:574
340
341=item 4
342
343 in  $=main::BEGIN() from /dev/null:0
344  in  $=Config::BEGIN() from lib/Config.pm:2
345   Package lib/Exporter.pm.
346   Package lib/Carp.pm.
347  out $=Config::BEGIN() from lib/Config.pm:0
348  Package lib/Config.pm.
349  in  $=Config::TIEHASH('Config') from lib/Config.pm:644
350  out $=Config::TIEHASH('Config') from lib/Config.pm:644
351  in  $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
352   in  $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/
353   out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/
354  out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
355 out $=main::BEGIN() from /dev/null:0
356 in  @=Config::myconfig() from /dev/null:0
357  in  $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
358  out $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
359  in  $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
360  out $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
361  in  $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
362  out $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
363  in  $=Config::FETCH(ref(Config), 'PERL_SUBVERSION') from lib/Config.pm:574
364
365=item 5
366
367 in  $=main::BEGIN() from /dev/null:0
368  in  $=Config::BEGIN() from lib/Config.pm:2
369   Package lib/Exporter.pm.
370   Package lib/Carp.pm.
371  out $=Config::BEGIN() from lib/Config.pm:0
372  Package lib/Config.pm.
373  in  $=Config::TIEHASH('Config') from lib/Config.pm:644
374  out $=Config::TIEHASH('Config') from lib/Config.pm:644
375  in  $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
376   in  $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/E
377   out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/E
378  out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
379 out $=main::BEGIN() from /dev/null:0
380 in  @=Config::myconfig() from /dev/null:0
381  in  $=Config::FETCH('Config=HASH(0x1aa444)', 'package') from lib/Config.pm:574
382  out $=Config::FETCH('Config=HASH(0x1aa444)', 'package') from lib/Config.pm:574
383  in  $=Config::FETCH('Config=HASH(0x1aa444)', 'baserev') from lib/Config.pm:574
384  out $=Config::FETCH('Config=HASH(0x1aa444)', 'baserev') from lib/Config.pm:574
385
386=item 6
387
388 in  $=CODE(0x15eca4)() from /dev/null:0
389  in  $=CODE(0x182528)() from lib/Config.pm:2
390   Package lib/Exporter.pm.
391  out $=CODE(0x182528)() from lib/Config.pm:0
392  scalar context return from CODE(0x182528): undef
393  Package lib/Config.pm.
394  in  $=Config::TIEHASH('Config') from lib/Config.pm:628
395  out $=Config::TIEHASH('Config') from lib/Config.pm:628
396  scalar context return from Config::TIEHASH:   empty hash
397  in  $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
398   in  $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/Exporter.pm:171
399   out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/Exporter.pm:171
400   scalar context return from Exporter::export: ''
401  out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
402  scalar context return from Exporter::import: ''
403
404=back
405
406In all cases shown above, the line indentation shows the call tree.
407If bit 2 of C<frame> is set, a line is printed on exit from a
408subroutine as well.  If bit 4 is set, the arguments are printed
409along with the caller info.  If bit 8 is set, the arguments are
410printed even if they are tied or references.  If bit 16 is set, the
411return value is printed, too.
412
413When a package is compiled, a line like this
414
415    Package lib/Carp.pm.
416
417is printed with proper indentation.
418
419=head1 Debugging Regular Expressions
420
421There are two ways to enable debugging output for regular expressions.
422
423If your perl is compiled with C<-DDEBUGGING>, you may use the
424B<-Dr> flag on the command line, and C<-Drv> for more verbose
425information.
426
427Otherwise, one can C<use re 'debug'>, which has effects at both
428compile time and run time.  Since Perl 5.9.5, this pragma is lexically
429scoped.
430
431=head2 Compile-time Output
432
433The debugging output at compile time looks like this:
434
435  Compiling REx '[bc]d(ef*g)+h[ij]k$'
436  size 45 Got 364 bytes for offset annotations.
437  first at 1
438  rarest char g at 0
439  rarest char d at 0
440     1: ANYOF[bc](12)
441    12: EXACT <d>(14)
442    14: CURLYX[0] {1,32767}(28)
443    16:   OPEN1(18)
444    18:     EXACT <e>(20)
445    20:     STAR(23)
446    21:       EXACT <f>(0)
447    23:     EXACT <g>(25)
448    25:   CLOSE1(27)
449    27:   WHILEM[1/1](0)
450    28: NOTHING(29)
451    29: EXACT <h>(31)
452    31: ANYOF[ij](42)
453    42: EXACT <k>(44)
454    44: EOL(45)
455    45: END(0)
456  anchored 'de' at 1 floating 'gh' at 3..2147483647 (checking floating)
457        stclass 'ANYOF[bc]' minlen 7
458  Offsets: [45]
459  	1[4] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 5[1]
460  	0[0] 12[1] 0[0] 6[1] 0[0] 7[1] 0[0] 9[1] 8[1] 0[0] 10[1] 0[0]
461  	11[1] 0[0] 12[0] 12[0] 13[1] 0[0] 14[4] 0[0] 0[0] 0[0] 0[0]
462  	0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 18[1] 0[0] 19[1] 20[0]
463  Omitting $` $& $' support.
464
465The first line shows the pre-compiled form of the regex.  The second
466shows the size of the compiled form (in arbitrary units, usually
4674-byte words) and the total number of bytes allocated for the
468offset/length table, usually 4+C<size>*8.  The next line shows the
469label I<id> of the first node that does a match.
470
471The
472
473  anchored 'de' at 1 floating 'gh' at 3..2147483647 (checking floating)
474        stclass 'ANYOF[bc]' minlen 7
475
476line (split into two lines above) contains optimizer
477information.  In the example shown, the optimizer found that the match
478should contain a substring C<de> at offset 1, plus substring C<gh>
479at some offset between 3 and infinity.  Moreover, when checking for
480these substrings (to abandon impossible matches quickly), Perl will check
481for the substring C<gh> before checking for the substring C<de>.  The
482optimizer may also use the knowledge that the match starts (at the
483C<first> I<id>) with a character class, and no string
484shorter than 7 characters can possibly match.
485
486The fields of interest which may appear in this line are
487
488=over 4
489
490=item C<anchored> I<STRING> C<at> I<POS>
491
492=item C<floating> I<STRING> C<at> I<POS1..POS2>
493
494See above.
495
496=item C<matching floating/anchored>
497
498Which substring to check first.
499
500=item C<minlen>
501
502The minimal length of the match.
503
504=item C<stclass> I<TYPE>
505
506Type of first matching node.
507
508=item C<noscan>
509
510Don't scan for the found substrings.
511
512=item C<isall>
513
514Means that the optimizer information is all that the regular
515expression contains, and thus one does not need to enter the regex engine at
516all.
517
518=item C<GPOS>
519
520Set if the pattern contains C<\G>.
521
522=item C<plus>
523
524Set if the pattern starts with a repeated char (as in C<x+y>).
525
526=item C<implicit>
527
528Set if the pattern starts with C<.*>.
529
530=item C<with eval>
531
532Set if the pattern contain eval-groups, such as C<(?{ code })> and
533C<(??{ code })>.
534
535=item C<anchored(TYPE)>
536
537If the pattern may match only at a handful of places, with C<TYPE>
538being C<SBOL>, C<MBOL>, or C<GPOS>.  See the table below.
539
540=back
541
542If a substring is known to match at end-of-line only, it may be
543followed by C<$>, as in C<floating 'k'$>.
544
545The optimizer-specific information is used to avoid entering (a slow) regex
546engine on strings that will not definitely match.  If the C<isall> flag
547is set, a call to the regex engine may be avoided even when the optimizer
548found an appropriate place for the match.
549
550Above the optimizer section is the list of I<nodes> of the compiled
551form of the regex.  Each line has format
552
553C<   >I<id>: I<TYPE> I<OPTIONAL-INFO> (I<next-id>)
554
555=head2 Types of Nodes
556
557Here are the current possible types, with short descriptions:
558
559=for comment
560This table is generated by regen/regcomp.pl.  Any changes made here
561will be lost.
562
563=for regcomp.pl begin
564
565 # TYPE arg-description [regnode-struct-suffix] [longjump-len] DESCRIPTION
566
567 # Exit points
568
569 END              no         End of program.
570 SUCCEED          no         Return from a subroutine, basically.
571
572 # Line Start Anchors:
573 SBOL             no         Match "" at beginning of line: /^/, /\A/
574 MBOL             no         Same, assuming multiline: /^/m
575
576 # Line End Anchors:
577 SEOL             no         Match "" at end of line: /$/
578 MEOL             no         Same, assuming multiline: /$/m
579 EOS              no         Match "" at end of string: /\z/
580
581 # Match Start Anchors:
582 GPOS             no         Matches where last m//g left off.
583
584 # Word Boundary Opcodes:
585 BOUND            no         Like BOUNDA for non-utf8, otherwise like
586                             BOUNDU
587 BOUNDL           no         Like BOUND/BOUNDU, but \w and \W are
588                             defined by current locale
589 BOUNDU           no         Match "" at any boundary of a given type
590                             using /u rules.
591 BOUNDA           no         Match "" at any boundary between \w\W or
592                             \W\w, where \w is [_a-zA-Z0-9]
593 NBOUND           no         Like NBOUNDA for non-utf8, otherwise like
594                             BOUNDU
595 NBOUNDL          no         Like NBOUND/NBOUNDU, but \w and \W are
596                             defined by current locale
597 NBOUNDU          no         Match "" at any non-boundary of a given
598                             type using using /u rules.
599 NBOUNDA          no         Match "" betweeen any \w\w or \W\W, where
600                             \w is [_a-zA-Z0-9]
601
602 # [Special] alternatives:
603 REG_ANY          no         Match any one character (except newline).
604 SANY             no         Match any one character.
605 ANYOF            sv         Match character in (or not in) this class,
606                  charclass  single char match only
607 ANYOFD           sv         Like ANYOF, but /d is in effect
608                  charclass
609 ANYOFL           sv         Like ANYOF, but /l is in effect
610                  charclass
611 ANYOFPOSIXL      sv         Like ANYOFL, but matches [[:posix:]]
612                  charclass_ classes
613                  posixl
614
615 ANYOFH           sv 1       Like ANYOF, but only has "High" matches,
616                             none in the bitmap; the flags field
617                             contains the lowest matchable UTF-8 start
618                             byte
619 ANYOFHb          sv 1       Like ANYOFH, but all matches share the same
620                             UTF-8 start byte, given in the flags field
621 ANYOFHr          sv 1       Like ANYOFH, but the flags field contains
622                             packed bounds for all matchable UTF-8 start
623                             bytes.
624 ANYOFHs          sv:str 1   Like ANYOFHb, but has a string field that
625                             gives the leading matchable UTF-8 bytes;
626                             flags field is len
627 ANYOFR           packed 1   Matches any character in the range given by
628                             its packed args: upper 12 bits is the max
629                             delta from the base lower 20; the flags
630                             field contains the lowest matchable UTF-8
631                             start byte
632 ANYOFRb          packed 1   Like ANYOFR, but all matches share the same
633                             UTF-8 start byte, given in the flags field
634
635 ANYOFHbbm        none bbm   Like ANYOFHb, but only for 2-byte UTF-8
636                             characters; uses a bitmap to match the
637                             continuation byte
638
639 ANYOFM           byte 1     Like ANYOF, but matches an invariant byte
640                             as determined by the mask and arg
641 NANYOFM          byte 1     complement of ANYOFM
642
643 # POSIX Character Classes:
644 POSIXD           none       Some [[:class:]] under /d; the FLAGS field
645                             gives which one
646 POSIXL           none       Some [[:class:]] under /l; the FLAGS field
647                             gives which one
648 POSIXU           none       Some [[:class:]] under /u; the FLAGS field
649                             gives which one
650 POSIXA           none       Some [[:class:]] under /a; the FLAGS field
651                             gives which one
652 NPOSIXD          none       complement of POSIXD, [[:^class:]]
653 NPOSIXL          none       complement of POSIXL, [[:^class:]]
654 NPOSIXU          none       complement of POSIXU, [[:^class:]]
655 NPOSIXA          none       complement of POSIXA, [[:^class:]]
656
657 CLUMP            no         Match any extended grapheme cluster
658                             sequence
659
660 # Alternation
661
662 # BRANCH        The set of branches constituting a single choice are
663 #               hooked together with their "next" pointers, since
664 #               precedence prevents anything being concatenated to
665 #               any individual branch.  The "next" pointer of the last
666 #               BRANCH in a choice points to the thing following the
667 #               whole choice.  This is also where the final "next"
668 #               pointer of each individual branch points; each branch
669 #               starts with the operand node of a BRANCH node.
670 #
671 BRANCH           node 1     Match this alternative, or the next...
672
673 # Literals
674
675 EXACT            str        Match this string (flags field is the
676                             length).
677
678 # In a long string node, the U32 argument is the length, and is
679 # immediately followed by the string.
680 LEXACT           len:str 1  Match this long string (preceded by length;
681                             flags unused).
682 EXACTL           str        Like EXACT, but /l is in effect (used so
683                             locale-related warnings can be checked for)
684 EXACTF           str        Like EXACT, but match using /id rules;
685                             (string not UTF-8, ASCII folded; non-ASCII
686                             not)
687 EXACTFL          str        Like EXACT, but match using /il rules;
688                             (string not likely to be folded)
689 EXACTFU          str        Like EXACT, but match using /iu rules;
690                             (string folded)
691
692 EXACTFAA         str        Like EXACT, but match using /iaa rules;
693                             (string folded except MICRO in non-UTF8
694                             patterns; doesn't contain SHARP S unless
695                             UTF-8; folded length <= unfolded)
696 EXACTFAA_NO_TRIE str        Like EXACTFAA, (string not UTF-8, folded
697                             except: MICRO, SHARP S; folded length <=
698                             unfolded, not currently trie-able)
699
700 EXACTFUP         str        Like EXACT, but match using /iu rules;
701                             (string not UTF-8, folded except MICRO:
702                             hence Problematic)
703
704 EXACTFLU8        str        Like EXACTFU, but use /il, UTF-8, (string
705                             is folded, and everything in it is above
706                             255
707 EXACT_REQ8       str        Like EXACT, but only UTF-8 encoded targets
708                             can match
709 LEXACT_REQ8      len:str 1  Like LEXACT, but only UTF-8 encoded targets
710                             can match
711 EXACTFU_REQ8     str        Like EXACTFU, but only UTF-8 encoded
712                             targets can match
713
714 EXACTFU_S_EDGE   str        /di rules, but nothing in it precludes /ui,
715                             except begins and/or ends with [Ss];
716                             (string not UTF-8; compile-time only)
717
718 # New charclass like patterns
719 LNBREAK          none       generic newline pattern
720
721 # Trie Related
722
723 # Behave the same as A|LIST|OF|WORDS would. The '..C' variants
724 # have inline charclass data (ascii only), the 'C' store it in the
725 # structure.
726
727 TRIE             trie 1     Match many EXACT(F[ALU]?)? at once.
728                             flags==type
729 TRIEC            trie       Same as TRIE, but with embedded charclass
730                  charclass  data
731
732 AHOCORASICK      trie 1     Aho Corasick stclass. flags==type
733 AHOCORASICKC     trie       Same as AHOCORASICK, but with embedded
734                  charclass  charclass data
735
736 # Do nothing types
737
738 NOTHING          no         Match empty string.
739 # A variant of above which delimits a group, thus stops optimizations
740 TAIL             no         Match empty string. Can jump here from
741                             outside.
742
743 # Loops
744
745 # STAR,PLUS    '?', and complex '*' and '+', are implemented as
746 #               circular BRANCH structures.  Simple cases
747 #               (one character per match) are implemented with STAR
748 #               and PLUS for speed and to minimize recursive plunges.
749 #
750 STAR             node       Match this (simple) thing 0 or more times:
751                             /A{0,}B/ where A is width 1 char
752 PLUS             node       Match this (simple) thing 1 or more times:
753                             /A{1,}B/ where A is width 1 char
754
755 CURLY            sv 3       Match this (simple) thing {n,m} times:
756                             /A{m,n}B/ where A is width 1 char
757 CURLYN           no 3       Capture next-after-this simple thing:
758                             /(A){m,n}B/ where A is width 1 char
759 CURLYM           no 3       Capture this medium-complex thing {n,m}
760                             times: /(A){m,n}B/ where A is fixed-length
761 CURLYX           sv 3       Match/Capture this complex thing {n,m}
762                             times.
763
764 # This terminator creates a loop structure for CURLYX
765 WHILEM           no         Do curly processing and see if rest
766                             matches.
767
768 # Buffer related
769
770 # OPEN,CLOSE,GROUPP     ...are numbered at compile time.
771 OPEN             num 1      Mark this point in input as start of #n.
772 CLOSE            num 1      Close corresponding OPEN of #n.
773 SROPEN           none       Same as OPEN, but for script run
774 SRCLOSE          none       Close preceding SROPEN
775
776 REF              num 2      Match some already matched string
777 REFF             num 2      Match already matched string, using /di
778                             rules.
779 REFFL            num 2      Match already matched string, using /li
780                             rules.
781 REFFU            num 2      Match already matched string, usng /ui.
782 REFFA            num 2      Match already matched string, using /aai
783                             rules.
784
785 # Named references.  Code in regcomp.c assumes that these all are after
786 # the numbered references
787 REFN             no-sv 2    Match some already matched string
788 REFFN            no-sv 2    Match already matched string, using /di
789                             rules.
790 REFFLN           no-sv 2    Match already matched string, using /li
791                             rules.
792 REFFUN           num 2      Match already matched string, using /ui
793                             rules.
794 REFFAN           num 2      Match already matched string, using /aai
795                             rules.
796
797 # Support for long RE
798 LONGJMP          off 1 1    Jump far away.
799 BRANCHJ          off 2 1    BRANCH with long offset.
800
801 # Special Case Regops
802 IFMATCH          off 1 1    Succeeds if the following matches; non-zero
803                             flags "f", next_off "o" means lookbehind
804                             assertion starting "f..(f-o)" characters
805                             before current
806 UNLESSM          off 1 1    Fails if the following matches; non-zero
807                             flags "f", next_off "o" means lookbehind
808                             assertion starting "f..(f-o)" characters
809                             before current
810 SUSPEND          off 1 1    "Independent" sub-RE.
811 IFTHEN           off 1 1    Switch, should be preceded by switcher.
812 GROUPP           num 1      Whether the group matched.
813
814 # The heavy worker
815
816 EVAL             evl/flags  Execute some Perl code.
817                  2
818
819 # Modifiers
820
821 MINMOD           no         Next operator is not greedy.
822 LOGICAL          no         Next opcode should set the flag only.
823
824 # This is not used yet
825 RENUM            off 1 1    Group with independently numbered parens.
826
827 # Regex Subroutines
828 GOSUB            num/ofs 2  recurse to paren arg1 at (signed) ofs arg2
829
830 # Special conditionals
831 GROUPPN          no-sv 1    Whether the group matched.
832 INSUBP           num 1      Whether we are in a specific recurse.
833 DEFINEP          none 1     Never execute directly.
834
835 # Backtracking Verbs
836 ENDLIKE          none       Used only for the type field of verbs
837 OPFAIL           no-sv 1    Same as (?!), but with verb arg
838 ACCEPT           no-sv/num  Accepts the current matched string, with
839                  2          verbar
840
841 # Verbs With Arguments
842 VERB             no-sv 1    Used only for the type field of verbs
843 PRUNE            no-sv 1    Pattern fails at this startpoint if no-
844                             backtracking through this
845 MARKPOINT        no-sv 1    Push the current location for rollback by
846                             cut.
847 SKIP             no-sv 1    On failure skip forward (to the mark)
848                             before retrying
849 COMMIT           no-sv 1    Pattern fails outright if backtracking
850                             through this
851 CUTGROUP         no-sv 1    On failure go to the next alternation in
852                             the group
853
854 # Control what to keep in $&.
855 KEEPS            no         $& begins here.
856
857 # Validate that lookbehind IFMATCH and UNLESSM end at the right place
858 LOOKBEHIND_END   no         Return from lookbehind (IFMATCH/UNLESSM)
859                             and validate position
860
861 # SPECIAL  REGOPS
862
863 # This is not really a node, but an optimized away piece of a "long"
864 # node.  To simplify debugging output, we mark it as if it were a node
865 OPTIMIZED        off        Placeholder for dump.
866
867 # Special opcode with the property that no opcode in a compiled program
868 # will ever be of this type. Thus it can be used as a flag value that
869 # no other opcode has been seen. END is used similarly, in that an END
870 # node cant be optimized. So END implies "unoptimizable" and PSEUDO
871 # mean "not seen anything to optimize yet".
872 PSEUDO           off        Pseudo opcode for internal use.
873
874 REGEX_SET        depth p    Regex set, temporary node used in pre-
875                             optimization compilation
876
877=for regcomp.pl end
878
879=for unprinted-credits
880Next section M-J. Dominus (mjd-perl-patch+@plover.com) 20010421
881
882Following the optimizer information is a dump of the offset/length
883table, here split across several lines:
884
885  Offsets: [45]
886  	1[4] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 5[1]
887  	0[0] 12[1] 0[0] 6[1] 0[0] 7[1] 0[0] 9[1] 8[1] 0[0] 10[1] 0[0]
888  	11[1] 0[0] 12[0] 12[0] 13[1] 0[0] 14[4] 0[0] 0[0] 0[0] 0[0]
889  	0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 18[1] 0[0] 19[1] 20[0]
890
891The first line here indicates that the offset/length table contains 45
892entries.  Each entry is a pair of integers, denoted by C<offset[length]>.
893Entries are numbered starting with 1, so entry #1 here is C<1[4]> and
894entry #12 is C<5[1]>.  C<1[4]> indicates that the node labeled C<1:>
895(the C<1: ANYOF[bc]>) begins at character position 1 in the
896pre-compiled form of the regex, and has a length of 4 characters.
897C<5[1]> in position 12
898indicates that the node labeled C<12:>
899(the C<< 12: EXACT <d> >>) begins at character position 5 in the
900pre-compiled form of the regex, and has a length of 1 character.
901C<12[1]> in position 14
902indicates that the node labeled C<14:>
903(the C<< 14: CURLYX[0] {1,32767} >>) begins at character position 12 in the
904pre-compiled form of the regex, and has a length of 1 character---that
905is, it corresponds to the C<+> symbol in the precompiled regex.
906
907C<0[0]> items indicate that there is no corresponding node.
908
909=head2 Run-time Output
910
911First of all, when doing a match, one may get no run-time output even
912if debugging is enabled.  This means that the regex engine was never
913entered and that all of the job was therefore done by the optimizer.
914
915If the regex engine was entered, the output may look like this:
916
917  Matching '[bc]d(ef*g)+h[ij]k$' against 'abcdefg__gh__'
918    Setting an EVAL scope, savestack=3
919     2 <ab> <cdefg__gh_>    |  1: ANYOF
920     3 <abc> <defg__gh_>    | 11: EXACT <d>
921     4 <abcd> <efg__gh_>    | 13: CURLYX {1,32767}
922     4 <abcd> <efg__gh_>    | 26:   WHILEM
923				0 out of 1..32767  cc=effff31c
924     4 <abcd> <efg__gh_>    | 15:     OPEN1
925     4 <abcd> <efg__gh_>    | 17:     EXACT <e>
926     5 <abcde> <fg__gh_>    | 19:     STAR
927			     EXACT <f> can match 1 times out of 32767...
928    Setting an EVAL scope, savestack=3
929     6 <bcdef> <g__gh__>    | 22:       EXACT <g>
930     7 <bcdefg> <__gh__>    | 24:       CLOSE1
931     7 <bcdefg> <__gh__>    | 26:       WHILEM
932				    1 out of 1..32767  cc=effff31c
933    Setting an EVAL scope, savestack=12
934     7 <bcdefg> <__gh__>    | 15:         OPEN1
935     7 <bcdefg> <__gh__>    | 17:         EXACT <e>
936       restoring \1 to 4(4)..7
937				    failed, try continuation...
938     7 <bcdefg> <__gh__>    | 27:         NOTHING
939     7 <bcdefg> <__gh__>    | 28:         EXACT <h>
940				    failed...
941				failed...
942
943The most significant information in the output is about the particular I<node>
944of the compiled regex that is currently being tested against the target string.
945The format of these lines is
946
947C<    >I<STRING-OFFSET> <I<PRE-STRING>> <I<POST-STRING>>   |I<ID>:  I<TYPE>
948
949The I<TYPE> info is indented with respect to the backtracking level.
950Other incidental information appears interspersed within.
951
952=head1 Debugging Perl Memory Usage
953
954Perl is a profligate wastrel when it comes to memory use.  There
955is a saying that to estimate memory usage of Perl, assume a reasonable
956algorithm for memory allocation, multiply that estimate by 10, and
957while you still may miss the mark, at least you won't be quite so
958astonished.  This is not absolutely true, but may provide a good
959grasp of what happens.
960
961Assume that an integer cannot take less than 20 bytes of memory, a
962float cannot take less than 24 bytes, a string cannot take less
963than 32 bytes (all these examples assume 32-bit architectures, the
964result are quite a bit worse on 64-bit architectures).  If a variable
965is accessed in two of three different ways (which require an integer,
966a float, or a string), the memory footprint may increase yet another
96720 bytes.  A sloppy malloc(3) implementation can inflate these
968numbers dramatically.
969
970On the opposite end of the scale, a declaration like
971
972  sub foo;
973
974may take up to 500 bytes of memory, depending on which release of Perl
975you're running.
976
977Anecdotal estimates of source-to-compiled code bloat suggest an
978eightfold increase.  This means that the compiled form of reasonable
979(normally commented, properly indented etc.) code will take
980about eight times more space in memory than the code took
981on disk.
982
983The B<-DL> command-line switch is obsolete since circa Perl 5.6.0
984(it was available only if Perl was built with C<-DDEBUGGING>).
985The switch was used to track Perl's memory allocations and possible
986memory leaks.  These days the use of malloc debugging tools like
987F<Purify> or F<valgrind> is suggested instead.  See also
988L<perlhacktips/PERL_MEM_LOG>.
989
990One way to find out how much memory is being used by Perl data
991structures is to install the Devel::Size module from CPAN: it gives
992you the minimum number of bytes required to store a particular data
993structure.  Please be mindful of the difference between the size()
994and total_size().
995
996If Perl has been compiled using Perl's malloc you can analyze Perl
997memory usage by setting $ENV{PERL_DEBUG_MSTATS}.
998
999=head2 Using C<$ENV{PERL_DEBUG_MSTATS}>
1000
1001If your perl is using Perl's malloc() and was compiled with the
1002necessary switches (this is the default), then it will print memory
1003usage statistics after compiling your code when C<< $ENV{PERL_DEBUG_MSTATS}
1004> 1 >>, and before termination of the program when C<<
1005$ENV{PERL_DEBUG_MSTATS} >= 1 >>.  The report format is similar to
1006the following example:
1007
1008 $ PERL_DEBUG_MSTATS=2 perl -e "require Carp"
1009 Memory allocation statistics after compilation: (buckets 4(4)..8188(8192)
1010    14216 free:   130   117    28     7     9   0   2     2   1 0 0
1011		437    61    36     0     5
1012    60924 used:   125   137   161    55     7   8   6    16   2 0 1
1013		 74   109   304    84    20
1014 Total sbrk(): 77824/21:119. Odd ends: pad+heads+chain+tail: 0+636+0+2048.
1015 Memory allocation statistics after execution:   (buckets 4(4)..8188(8192)
1016    30888 free:   245    78    85    13     6   2   1     3   2 0 1
1017		315   162    39    42    11
1018   175816 used:   265   176  1112   111    26  22  11    27   2 1 1
1019		196   178  1066   798    39
1020 Total sbrk(): 215040/47:145. Odd ends: pad+heads+chain+tail: 0+2192+0+6144.
1021
1022It is possible to ask for such a statistic at arbitrary points in
1023your execution using the mstat() function out of the standard
1024Devel::Peek module.
1025
1026Here is some explanation of that format:
1027
1028=over 4
1029
1030=item C<buckets SMALLEST(APPROX)..GREATEST(APPROX)>
1031
1032Perl's malloc() uses bucketed allocations.  Every request is rounded
1033up to the closest bucket size available, and a bucket is taken from
1034the pool of buckets of that size.
1035
1036The line above describes the limits of buckets currently in use.
1037Each bucket has two sizes: memory footprint and the maximal size
1038of user data that can fit into this bucket.  Suppose in the above
1039example that the smallest bucket were size 4.  The biggest bucket
1040would have usable size 8188, and the memory footprint would be 8192.
1041
1042In a Perl built for debugging, some buckets may have negative usable
1043size.  This means that these buckets cannot (and will not) be used.
1044For larger buckets, the memory footprint may be one page greater
1045than a power of 2.  If so, the corresponding power of two is
1046printed in the C<APPROX> field above.
1047
1048=item Free/Used
1049
1050The 1 or 2 rows of numbers following that correspond to the number
1051of buckets of each size between C<SMALLEST> and C<GREATEST>.  In
1052the first row, the sizes (memory footprints) of buckets are powers
1053of two--or possibly one page greater.  In the second row, if present,
1054the memory footprints of the buckets are between the memory footprints
1055of two buckets "above".
1056
1057For example, suppose under the previous example, the memory footprints
1058were
1059
1060   free:    8     16    32    64    128  256 512 1024 2048 4096 8192
1061	   4     12    24    48    80
1062
1063With a non-C<DEBUGGING> perl, the buckets starting from C<128> have
1064a 4-byte overhead, and thus an 8192-long bucket may take up to
10658188-byte allocations.
1066
1067=item C<Total sbrk(): SBRKed/SBRKs:CONTINUOUS>
1068
1069The first two fields give the total amount of memory perl sbrk(2)ed
1070(ess-broken? :-) and number of sbrk(2)s used.  The third number is
1071what perl thinks about continuity of returned chunks.  So long as
1072this number is positive, malloc() will assume that it is probable
1073that sbrk(2) will provide continuous memory.
1074
1075Memory allocated by external libraries is not counted.
1076
1077=item C<pad: 0>
1078
1079The amount of sbrk(2)ed memory needed to keep buckets aligned.
1080
1081=item C<heads: 2192>
1082
1083Although memory overhead of bigger buckets is kept inside the bucket, for
1084smaller buckets, it is kept in separate areas.  This field gives the
1085total size of these areas.
1086
1087=item C<chain: 0>
1088
1089malloc() may want to subdivide a bigger bucket into smaller buckets.
1090If only a part of the deceased bucket is left unsubdivided, the rest
1091is kept as an element of a linked list.  This field gives the total
1092size of these chunks.
1093
1094=item C<tail: 6144>
1095
1096To minimize the number of sbrk(2)s, malloc() asks for more memory.  This
1097field gives the size of the yet unused part, which is sbrk(2)ed, but
1098never touched.
1099
1100=back
1101
1102=head1 SEE ALSO
1103
1104L<perldebug>,
1105L<perl5db.pl>,
1106L<perlguts>,
1107L<perlrun>,
1108L<re>,
1109and
1110L<Devel::DProf>.
1111