Lines Matching full:the
11 This document describes some of the more important APIs and internal design
12 decisions made in the Clang C front-end. The purpose of this document is to
13 both capture some of this high level information and also describe some of the
15 Clang, not for end-users. The description below is categorized by libraries,
16 and does not describe any of the clients of the libraries.
21 The LLVM ``libSupport`` library provides many underlying libraries and
26 The Clang "Basic" Library
29 This library certainly needs a better name. The "basic" library contains a
31 locations within the source buffers, diagnostics, tokens, target abstraction,
32 and information about the subset of the language being compiled for.
34 Part of this infrastructure is specific to C (such as the ``TargetInfo``
38 introduce a new library, move the general classes somewhere else, or introduce
41 We describe the roles of these classes in order of their dependencies.
43 The Diagnostics Subsystem
46 The Clang Diagnostics subsystem is an important part of how the compiler
47 communicates with the human. Diagnostics are the warnings and errors produced
48 when the code is incorrect or dubious. In Clang, each diagnostic produced has
49 (at the minimum) a unique ID, an English translation associated with it, a
50 :ref:`SourceLocation <SourceLocation>` to "put the caret", and a severity
52 arguments to the diagnostic (which fill in "%0"'s in the string) as well as a
53 number of source ranges that related to the diagnostic.
55 In this section, we'll be giving examples produced by the Clang command line
57 <DiagnosticConsumer>` depending on how the ``DiagnosticConsumer`` interface is
66 In this example, you can see the English translation, the severity (error), you
67 can see the source location (the caret ("``^``") and file/line/column info),
68 the source ranges "``~~~~``", arguments to the diagnostic ("``int*``" and
70 backing the diagnostic :).
76 The ``Diagnostic*Kinds.td`` files
79 Diagnostics are created by adding an entry to one of the
81 using it. From this file, :program:`tblgen` generates the unique ID of the
82 diagnostic, the severity of the diagnostic and the English translation + format
85 There is little sanity with the naming of the unique ID's right now. Some
86 start with ``err_``, ``warn_``, ``ext_`` to encode the severity into the name.
87 Since the enum is referenced in the C++ code that produces the diagnostic, it
90 The severity of the diagnostic comes from the set {``NOTE``, ``REMARK``,
92 ``EXTENSION``, ``EXTWARN``, ``ERROR``}. The ``ERROR`` severity is used for
93 diagnostics indicating the program is never acceptable under any circumstances.
94 When an error is emitted, the AST for the input code may not be fully built.
95 The ``EXTENSION`` and ``EXTWARN`` severities are used for extensions to the
97 represent them in the AST, but we produce diagnostics to tell the user their
98 code is non-portable. The difference is that the former are ignored by
99 default, and the later warn by default. The ``WARNING`` severity is used for
100 constructs that are valid in the currently selected source language but that
101 are dubious in some way. The ``REMARK`` severity provides generic information
102 about the compilation that is not necessarily related to any dubious code. The
105 These *severities* are mapped into a smaller set (the ``Diagnostic::Level``
108 *levels* by the diagnostics subsystem based on various configuration options.
110 you to map almost any diagnostic to the output level that you want. The only
111 diagnostics that cannot be mapped are ``NOTE``\ s, which always follow the
112 severity of the previously emitted diagnostic and ``ERROR``\ s, which can only
116 Diagnostic mappings are used in many ways. For example, if the user specifies
128 The wording used for a diagnostic is critical because it is the only way for a
129 user to know how to correct their code. Use the following suggestions when
143 * The wording should be succinct. If necessary, use a semicolon to combine
148 * The wording should be actionable and avoid using standards terms or grammar
152 * The wording should clearly explain what is wrong with the code rather than
153 restating what the code does. e.g., prefer wording like ``type %0 requires a
154 value in the range %1 to %2`` over wording like ``%0 is invalid``.
155 * The wording should have enough contextual information to help the user
156 identify the issue in a complex expression. e.g., prefer wording like
157 ``both sides of the %0 binary operator are identical`` over wording like
163 * Prefer diagnostic wording without contractions whenever possible. The single
168 The Format String
171 The format string for the diagnostic is very simple, but it has some power. It
172 takes the form of a string in English with markers that indicate where and how
173 arguments to the diagnostic are inserted and formatted. For example, here are
179 "format string contains '\\0' within the string body"
186 plain ASCII character in the diagnostic string except "``%``" without a
187 problem, but these are C strings, so you have to use and be aware of all the C
188 escape sequences (as in the second example). If you want to produce a "``%``"
189 in the output, use the "``%%``" escape sequence, like the third diagnostic.
190 Finally, Clang uses the "``%...[digit]``" sequences to specify where and how
191 arguments to the diagnostic are formatted.
193 Arguments to the diagnostic are numbered according to how they are specified by
194 the C++ code that :ref:`produces them <internals-producing-diag>`, and are
197 requirement that arguments to the diagnostic end up in the output in the same
199 that swaps them, for example. The text in between the percent and digit are
200 formatting instructions. If there are no instructions, the argument is just
203 Here are some "best practices" for writing the English format string:
205 * Keep the string short. It should ideally fit in the 80 column limit of the
206 ``DiagnosticKinds.td`` file. This avoids the diagnostic wrapping when
207 printed, and forces you to think about the important point you are conveying
208 with the diagnostic.
209 * Take advantage of location information. The user will be able to see the
210 line and location of the caret, so you don't need to tell them that the
211 problem is with the 4th argument to the function: just point to it.
212 * Do not capitalize the diagnostic string, and do not end it with a period.
213 * If you need to quote something in the diagnostic string, use single quotes.
218 :ref:`translating <internals-diag-translation>` the Clang diagnostics to other
220 localized diagnostic). The exceptions to this are C/C++ language keywords
222 Note that things like "pointer" and "reference" are not keywords. On the other
223 hand, you *can* include anything that comes from the user's source code,
224 including variable names, types, labels, etc. The "``select``" format can be
232 the class of the argument, it can be optionally formatted in different ways.
233 This gives the ``DiagnosticConsumer`` information about what the argument means
237 It is really easy to add format specifiers to the Clang diagnostics system, but
240 it up on the cfe-dev mailing list.
242 Here are the different diagnostic argument formats currently supported by
253 diagnostics. When the integer is 1, it prints as nothing. When the integer
255 be to be handled correctly, and eliminates the need to use gross things like
259 ``mice``. You can use the "plural" format specifier to handle such situations.
269 into one common one, without requiring the difference to be specified as an
270 English string argument. Instead of specifying the string, the diagnostic
271 gets an integer argument and the format string selects the numbered option.
272 In this case, the "``%0``" value must be an integer in the range [0..2]. If
275 substitute reasonable words (or entire phrases) based on the semantics of the
276 diagnostic instead of having to do things textually. The selected string
288 the format string given. In the above case, a namespace is generated named
289 ``FrobbleKind`` that has an unscoped enumeration with the enumerators
290 ``VarDecl`` and ``FuncDecl`` which correspond to the values 0 and 1. This
291 permits a clearer use of the ``Diag`` in source code, as the above could be
302 the requirements of languages with very complex plural forms, as many Baltic
303 languages have. The argument consists of a series of expression/form pairs,
304 separated by ":", where the first form whose expression evaluates to true is
305 the result of the modifier.
307 An expression can be empty, in which case it is always true. See the example
308 at the top. Otherwise, it is a series of one or more numeric conditions,
309 separated by ",". If any condition matches, the expression matches. Each
312 * number: A simple decimal number matches if the argument is the same as the
314 * range: A range in square brackets matches if the argument is within the
318 either a number or a range. The tests are the same as for plain numbers
319 and ranges, but the argument is taken modulo the number first. Example:
322 The parser is very unforgiving. A syntax error, even whitespace, will abort,
323 as will a failure to match the argument against any expression.
332 This is a formatter which represents the argument number as an ordinal: the
344 This is a formatter which represents the argument number in a human readable
345 format: the value ``123`` stays ``123``, ``12345`` becomes ``12.34k``,
355 This is a simple formatter that indicates the ``DeclarationName`` corresponds
356 to an Objective-C class method selector. As such, it prints the selector
366 This is a simple formatter that indicates the ``DeclarationName`` corresponds
367 to an Objective-C instance method selector. As such, it prints the selector
377 This formatter indicates that the fully-qualified name of the declaration
388 difference between the two. If tree printing is off, the text inside the
389 braces before the pipe is printed, with the formatted text replacing the $.
390 If tree printing is on, the text after the pipe is printed and a type tree is
391 printed after the diagnostic message.
396 Given the following record definition of type ``TextSubstitution``:
414 diagnostics. The argument to ``%sub`` must name a ``TextSubstitution`` tblgen
415 record. The substitution must specify all arguments used by the substitution,
416 and the modifier indexes in the substitution are re-numbered accordingly. The
421 Producing the Diagnostic
424 Now that you've created the diagnostic in the ``Diagnostic*Kinds.td`` file, you
425 need to write the code that detects the condition in question and emits the new
426 diagnostic. Various components of Clang (e.g., the preprocessor, ``Sema``,
428 accepts the arguments, ranges, and other information that goes along with it.
430 For example, the binary expression error comes from code like this:
439 This shows that use of the ``Diag`` method: it takes a location (a
441 (which matches the name from ``Diagnostic*Kinds.td``). If the diagnostic takes
442 arguments, they are specified with the ``<<`` operator: the first argument
443 becomes ``%0``, the second becomes ``%1``, etc. The diagnostic interface
447 ``QualType`` for types, etc. ``SourceRange``\ s are also specified with the
451 The hard part is deciding exactly what you need to say to help the user,
452 picking a suitable wording, and providing the information needed to format it
453 correctly. The good news is that the call site that issues a diagnostic should
454 be completely independent of how the diagnostic is formatted and in what
460 In some cases, the front end emits diagnostics when it is clear that some small
461 change to the source code would fix the problem. For example, a missing
462 semicolon at the end of a statement or a use of deprecated syntax that is
463 easily rewritten into a more modern form. Clang tries very hard to emit the
466 However, for these cases where the fix is obvious, the diagnostic can be
468 change the code referenced by the diagnostic to fix the problem. For example,
469 it might add the missing semicolon at the end of the statement or rewrite the
471 example from the C++ front end, where we warn about the right-shift operator
482 Here, the fix-it hint is suggesting that parentheses be added, and showing
483 exactly where those parentheses would be inserted into the source code. The
484 fix-it hints themselves describe what changes to make to the source code in an
485 abstract manner, which the text diagnostic printer renders as a line of
486 "insertions" below the caret line. :ref:`Other diagnostic clients
487 <DiagnosticConsumer>` might choose to render the code differently (e.g., as
488 markup inline) or even give the user the ability to automatically fix the
493 * Since they are automatically applied if ``-Xclang -fixit`` is passed to the
494 driver, they should only be used when it's very likely they match the user's
496 * Clang must recover from errors as if the fix-it had been applied.
497 * Fix-it hints on a warning must not change the meaning of the code.
498 However, a hint may clarify the meaning as intentional, for example by adding
499 parentheses when the precedence of operators isn't obvious.
501 If a fix-it can't obey these rules, put the fix-it on a note. Fix-its on notes
504 All fix-it hints are described by the ``FixItHint`` class, instances of which
505 should be attached to the diagnostic using the ``<<`` operator in the same way
506 that highlighted source ranges and arguments are passed to the diagnostic.
511 Specifies that the given ``Code`` (a string) should be inserted before the
516 Specifies that the code in the given source ``Range`` should be removed.
520 Specifies that the code in the given source ``Range`` should be removed,
521 and replaced with the given ``Code`` string.
525 The ``DiagnosticConsumer`` Interface
528 Once code generates a diagnostic with all of the arguments and the rest of the
530 mentioned, the diagnostic machinery goes through some filtering to map a
531 severity onto a diagnostic level, then (assuming the diagnostic is not mapped
532 to "``Ignore``") it invokes an object that implements the ``DiagnosticConsumer``
533 interface with the information.
536 example, the normal Clang ``DiagnosticConsumer`` (named
537 ``TextDiagnosticPrinter``) turns the arguments into strings (according to the
538 various formatting rules), prints out the file/line/column information and the
539 string, then prints out the line of code, the source ranges, and the caret.
542 Another implementation of the ``DiagnosticConsumer`` interface is the
544 mode. Instead of formatting and printing out the diagnostics, this
545 implementation just captures and remembers the diagnostics as they fly by.
546 Then ``-verify`` compares the list of produced diagnostics to the list of
548 documentation for the ``-verify`` mode can be found at
554 linkified to where they come from in the source. Another example is that a GUI
556 pass significantly more information about types through to the GUI than a
557 simple flat string. The interface allows this to happen.
564 Not possible yet! Diagnostic strings should be written in UTF-8, the client can
565 translate to the relevant code page if needed. Each translation completely
566 replaces the format string for the diagnostic.
571 The ``SourceLocation`` and ``SourceManager`` classes
574 Strangely enough, the ``SourceLocation`` class represents a location within the
575 source code of the program. Important design points include:
582 file. This includes in the middle of tokens, in whitespace, in trigraphs,
584 #. A ``SourceLocation`` must encode the current ``#include`` stack that was
585 active when the location was processed. For example, if the location
586 corresponds to a token, it should contain the set of ``#include``\ s active
587 when the token was lexed. This allows us to print the ``#include`` stack
590 the ultimate instantiation point and the source of the original character
593 In practice, the ``SourceLocation`` works together with the ``SourceManager``
595 location and its expansion location. For most tokens, these will be the
597 directive) these will describe the location of the characters corresponding to
598 the token and the location where the token was used (i.e., the macro
599 expansion point or the location of the ``_Pragma`` itself).
601 The Clang front-end inherently depends on the location of a token being tracked
602 correctly. If it is ever incorrect, the front-end may get confused and die.
603 The reason for this is that the notion of the "spelling" of a ``Token`` in
604 Clang depends on being able to find the original input characters for the
605 token. This concept maps directly to the "spelling location" for the token.
613 each point to the beginning of their respective tokens. For example consider
614 the ``SourceRange`` of the following statement:
621 To map from this representation to a character-based representation, the "last"
622 location needs to be adjusted to point to (or past) the end of that token with
624 the rare cases where character-level source ranges information is needed we use
625 the ``CharSourceRange`` class.
627 The Driver Library
630 The clang Driver and library are documented :doc:`here <DriverInternals>`.
636 serialized representation of Clang's internal data structures, encoded with the
639 The Frontend Library
642 The Frontend library contains functionality useful for building tools on top of
643 the Clang libraries, for example several methods for outputting diagnostics.
648 One of the classes provided by the Frontend library is ``CompilerInvocation``,
649 which holds information that describe current invocation of the Clang ``-cc1``
650 frontend. The information typically comes from the command line constructed by
651 the Clang driver or from clients performing custom initialization. The data
652 structure is split into logical units used by different parts of the compiler,
658 The command line interface of the Clang ``-cc1`` frontend is defined alongside
659 the driver options in ``clang/Driver/Options.td``. The information making up an
661 position of the option value, help text, aliases and more. Each option may
663 accepted by the ``-cc1`` frontend are marked with the ``CC1Option`` flag.
668 Option definitions are processed by the ``-gen-opt-parser-defs`` tablegen
669 backend during early stages of the build. Options are then used for querying an
670 instance ``llvm::opt::ArgList``, a wrapper around the command line arguments.
671 This is done in the Clang driver to construct individual jobs based on the
672 driver arguments and also in the ``CompilerInvocation::CreateFromArgs`` function
673 that parses the ``-cc1`` frontend arguments.
689 When adding a new command line option, the first place of interest is the header
690 file declaring the corresponding options class (e.g. ``CodeGenOptions.h`` for
691 command line option that affects the code generation). Create new member
692 variable for the option value:
703 Next, declare the command line interface of the option in the tablegen file
704 ``clang/include/clang/Driver/Options.td``. This is done by instantiating the
705 ``Option`` class (defined in ``llvm/include/llvm/Option/OptParser.td``). The
706 instance is typically created through one of the helper classes that encode the
707 acceptable ways to specify the option value on the command line:
709 * ``Flag`` - the option does not accept any value,
710 * ``Joined`` - the value must immediately follow the option name within the same
712 * ``Separate`` - the value must follow the option name in the next command line
714 * ``JoinedOrSeparate`` - the value can be specified either as ``Joined`` or
716 * ``CommaJoined`` - the values are comma-separated and must immediately follow
717 the option name within the same argument (see ``Wl,`` for an example).
719 The helper classes take a list of acceptable prefixes of the option (e.g.
720 ``"-"``, ``"--"`` or ``"/"``) and the option name:
730 * ``HelpText`` holds the text that will be printed besides the option name when
731 the user requests help (e.g. via ``clang --help``).
732 * ``Group`` specifies the "category" of options this option belongs to. This is
734 * ``Flags`` may contain "tags" associated with the option. These may affect how
735 the option is rendered, or if it's hidden in some contexts.
736 * ``Visibility`` should be used to specify the drivers in which a particular
738 * ``Alias`` denotes that the option is an alias of another option. This may be
739 combined with ``AliasArgs`` that holds the implied value.
749 New options are recognized by the ``clang`` driver mode if ``Visibility`` is
751 must be explicitly marked with the ``CC1Option`` flag. Flags that specify
755 Next, parse (or manufacture) the command line arguments in the Clang driver and
756 use them to construct the ``-cc1`` job:
770 The last step is implementing the ``-cc1`` command line argument
771 parsing/generation that initializes/serializes the option class (in our case
773 automatically by using the marshalling annotations on the option definition:
784 Inner workings of the system are introduced in the :ref:`marshalling
785 infrastructure <OptionMarshalling>` section and the available annotations are
788 In case the marshalling infrastructure does not support the desired semantics,
789 consider simplifying it to fit the existing model. This makes the command line
790 more uniform and reduces the amount of custom, manually written code. Remember
791 that the ``-cc1`` command line interface is intended only for Clang developers,
792 meaning it does not need to mirror the driver interface, maintain backward
795 If the option semantics cannot be encoded via marshalling annotations, you can
796 resort to parsing/serializing the command line arguments manually:
817 Finally, you can specify the argument on the command line:
818 ``clang -fpass-plugin=a -fpass-plugin=b`` and use the new member variable as
835 The option marshalling infrastructure automates the parsing of the Clang
837 generation from ``CompilerInvocation``. The system replaces lots of repetitive
839 the majority of the ``-cc1`` command line interface. This section provides an
840 overview of the system.
842 **Note:** The marshalling infrastructure is not intended for driver-only
843 options. Only options of the ``-cc1`` frontend need to be marshalled to/from
846 To read and modify contents of ``CompilerInvocation``, the marshalling system
848 for the ``CompilerInvocation`` member is created by inheriting from
859 The first argument to the parent class is the beginning of the key path that
860 references the ``CompilerInvocation`` member. This argument ends with ``->`` if
861 the member is a pointer type or with ``.`` if it's a value type. The child class
862 takes a single parameter ``field`` that is forwarded as the second argument to
863 the base class. The child class can then be used like so:
864 ``LangOpts<"IgnoreExceptions">``, constructing a key path to the field
865 ``LangOpts->IgnoreExceptions``. The third argument passed to the parent class is
866 a string that the tablegen backend uses as a prefix to the
867 ``OPTION_WITH_MARSHALLING`` macro. Using the key path as a mix-in on an
868 ``Option`` instance instructs the backend to generate the following code:
878 Such definition can be used used in the function for parsing and generating
923 The ``PARSE_OPTION_WITH_MARSHALLING`` and ``GENERATE_OPTION_WITH_MARSHALLING``
924 macros are defined in ``CompilerInvocation.cpp`` and they implement the generic
932 How does the tablegen backend know what to put in place of ``[...]`` in the
933 generated ``Options.inc``? This is specified by the ``Marshalling`` utilities
935 information required for parsing or generating the command line argument.
937 **Note:** The marshalling infrastructure is not intended for driver-only
938 options. Only options of the ``-cc1`` frontend need to be marshalled to/from
943 The key path defaults to ``false`` and is set to ``true`` when the flag is
954 The key path defaults to ``true`` and is set to ``false`` when the flag is
965 The key path defaults to the specified value (``false``, ``true`` or some
966 boolean value that's statically unknown in the tablegen file). Then, the key
967 path is set to the value associated with the flag that appears last on command
974 PosFlag<SetTrue, [], [], "Use the legacy pass manager in LLVM">,
975 NegFlag<SetFalse, [], [], "Use the new pass manager in LLVM">,
978 With most such pair of flags, the ``-cc1`` frontend accepts only the flag that
979 changes the default key path value. The Clang driver is responsible for
980 accepting both and either forwarding the changing flag or discarding the flag
981 that would just set the key path to its default.
983 The first argument to ``BoolOption`` is a prefix that is used to construct the
984 full names of both flags. The positive flag would then be named
985 ``flegacy-pass-manager`` and the negative ``fno-legacy-pass-manager``.
986 ``BoolOption`` also implies the ``-`` prefix for both flags. It's also possible
987 to use ``BoolFOption`` that implies the ``"f"`` prefix and ``Group<f_Group>``.
988 The ``PosFlag`` and ``NegFlag`` classes hold the associated boolean value,
989 arrays of elements passed to the ``Flag`` and ``Visibility`` classes and the
990 help text. The optional ``BothFlags`` class holds arrays of ``Flag`` and
991 ``Visibility`` elements that are common for both the positive and negative flag
996 The key path defaults to the specified string, or an empty one, if omitted. When
997 the option appears on the command line, the argument value is simply copied.
1007 The key path defaults to an empty ``std::vector<std::string>``. Values specified
1008 with each appearance of the option on the command line are appended to the
1019 The key path defaults to the specified integer value, or ``0`` if omitted. When
1020 the option appears on the command line, its value gets parsed by ``llvm::APInt``
1021 and the result is assigned to the key path on success.
1031 The key path defaults to the value specified in ``MarshallingInfoEnum`` prefixed
1032 by the contents of ``NormalizedValuesScope`` and ``::``. This ensures correct
1033 reference to an enum case is formed even if the enum resides in different
1034 namespace or is an enum class. If the value present on command line does not
1035 match any of the comma-separated values from ``Values``, an error diagnostics is
1036 issued. Otherwise, the corresponding element from ``NormalizedValues`` at the
1037 same index is assigned to the key path (also correctly scoped). The number of
1038 comma-separated string values and elements of the array within
1051 complexity to the marshalling infrastructure and might be removed.
1057 The key path defaults to the default value from the primary ``Marshalling``
1058 annotation. Then, if any of the elements of ``ImpliedByAnyOf`` evaluate to true,
1059 the key path value is changed to the specified value or ``true`` if missing.
1060 Finally, the command line is parsed according to the primary annotation.
1071 The option is parsed only if the expression in ``ShouldParseIf`` evaluates to
1081 The Lexer and Preprocessor Library
1084 The Lexer library contains several tightly-connected classes that are involved
1085 with the nasty process of lexing and preprocessing C source code. The main
1086 interface to this library for outside clients is the large ``Preprocessor``
1087 class. It contains the various pieces of state that are required to coherently
1090 The core interface to the ``Preprocessor`` object (once it is set up) is the
1091 ``Preprocessor::Lex`` method, which returns the next :ref:`Token <Token>` from
1092 the preprocessor stream. There are two types of token providers that the
1093 preprocessor is capable of reading from: a buffer lexer (provided by the
1094 :ref:`Lexer <Lexer>` class) and a buffered token stream (provided by the
1099 The Token class
1102 The ``Token`` class is used to represent a single lexed token. Tokens are
1103 intended to be used by the lexer/preprocess and parser libraries, but are not
1104 intended to live beyond them (for example, they should not live in the ASTs).
1106 Tokens most often live on the stack (or some other location that is efficient
1107 to access) as the parser is running, but occasionally do get buffered up. For
1108 example, macro definitions are stored as a series of tokens, and the C++
1110 various pieces of look-ahead. As such, the size of a ``Token`` matters. On a
1114 normal tokens. Normal tokens are those returned by the lexer, annotation
1115 tokens represent semantic information and are produced by the parser, replacing
1116 normal tokens in the token stream. Normal tokens contain the following
1119 * **A SourceLocation** --- This indicates the location of the start of the
1122 * **A length** --- This stores the length of the token as stored in the
1124 trigraphs and escaped newlines which are ignored by later phases of the
1125 compiler. By pointing into the original source buffer, it is always possible
1126 to get the original spelling of a token completely accurately.
1128 * **IdentifierInfo** --- If a token takes the form of an identifier, and if
1129 identifier lookup was enabled when the token was lexed (e.g., the lexer was
1130 not reading in "raw" mode) this contains a pointer to the unique hash value
1131 for the identifier. Because the lookup happens before keyword
1134 * **TokenKind** --- This indicates the kind of token as classified by the
1135 lexer. This includes things like ``tok::starequal`` (for the "``*=``"
1136 operator), ``tok::ampamp`` for the "``&&``" token, and keyword values (e.g.,
1139 "operator keywords", where things like "``and``" are treated exactly like the
1140 "``&&``" operator. In these cases, the kind value is set to ``tok::ampamp``,
1141 which is good for the parser, which doesn't have to consider both forms. For
1142 something that cares about which form is used (e.g., the preprocessor
1143 "stringize" operator) the spelling indicates the original form.
1145 * **Flags** --- There are currently four flags tracked by the
1148 #. **StartOfLine** --- This was the first token that occurred on its input
1151 the token or transitively before the token as it was expanded through a
1152 macro. The definition of this flag is very closely defined by the
1153 stringizing requirements of the preprocessor.
1154 #. **DisableExpand** --- This flag is used internally to the preprocessor to
1157 in the future.
1158 #. **NeedsCleaning** --- This flag is set if the original spelling for the
1163 don't contain any semantic information about the lexed value. For example, if
1164 the token was a pp-number token, we do not represent the value of the number
1166 Additionally, the lexer library has no notion of typedef names vs variable
1167 names: both are returned as identifiers, and the parser is left to decide
1169 requires scope information among other things). The parser can do this
1170 translation by replacing tokens returned by the preprocessor with "Annotation
1178 Annotation tokens are tokens that are synthesized by the parser and injected
1179 into the preprocessor's token stream (replacing existing tokens) to record
1180 semantic information found by the parser. For example, if "``foo``" is found
1181 to be a typedef, the "``foo``" ``tok::identifier`` token is replaced with an
1184 C++ as a single "token" in the parser. 2) if the parser backtracks, the
1188 Annotation tokens are created by the parser and reinjected into the parser's
1190 tokens that the preprocessor-proper is done with, it doesn't need to keep
1191 around flags like "start of line" that the preprocessor uses to do its job.
1193 (e.g., "``a::b::c``" is five preprocessor tokens). As such, the valid fields
1194 of an annotation token are different than the fields for a normal token (but
1195 they are multiplexed into the normal ``Token`` fields):
1197 * **SourceLocation "Location"** --- The ``SourceLocation`` for the annotation
1198 token indicates the first token replaced by the annotation token. In the
1199 example above, it would be the location of the "``a``" identifier.
1200 * **SourceLocation "AnnotationEndLoc"** --- This holds the location of the last
1201 token replaced with the annotation token. In the example above, it would be
1202 the location of the "``c``" identifier.
1203 * **void* "AnnotationValue"** --- This contains an opaque object that the
1204 parser gets from ``Sema``. The parser merely preserves the information for
1205 ``Sema`` to later interpret based on the annotation token kind.
1206 * **TokenKind "Kind"** --- This indicates the kind of Annotation token this is.
1207 See below for the different valid kinds.
1212 typename token that is potentially qualified. The ``AnnotationValue`` field
1213 contains the ``QualType`` returned by ``Sema::getTypeName()``, possibly with
1216 specifier, such as "``A::B::``". This corresponds to the grammar
1217 productions "*::*" and "*:: [opt] nested-name-specifier*". The
1218 ``AnnotationValue`` pointer is a ``NestedNameSpecifier *`` returned by the
1222 template-id such as "``foo<int, 4>``", where "``foo``" is the name of a
1223 template. The ``AnnotationValue`` pointer is a pointer to a ``malloc``'d
1224 ``TemplateIdAnnotation`` object. Depending on the context, a parsed
1226 all we care about is the named type, e.g., because it occurs in a type
1230 to a type can be "upgraded" to typename annotation tokens by the parser.
1232 As mentioned above, annotation tokens are not returned by the preprocessor,
1233 they are formed on demand by the parser. This means that the parser has to be
1235 This is somewhat similar to how the parser handles Translation Phase 6 of C99:
1236 String Concatenation (see C99 5.1.1.2). In the case of string concatenation,
1237 the preprocessor just returns distinct ``tok::string_literal`` and
1238 ``tok::wide_string_literal`` tokens and the parser eats a sequence of them
1239 wherever the grammar indicates that a string literal can occur.
1241 In order to do this, whenever the parser expects a ``tok::identifier`` or
1242 ``tok::coloncolon``, it should call the ``TryAnnotateTypeOrScopeToken`` or
1243 ``TryAnnotateCXXScopeToken`` methods to form the annotation token. These
1244 methods will maximally form the specified annotation tokens and replace the
1245 current token with them, if applicable. If the current tokens is not valid for
1250 The ``Lexer`` class
1253 The ``Lexer`` class provides the mechanics of lexing tokens out of a source
1254 buffer and deciding what they mean. The ``Lexer`` is complicated by the fact
1257 coding as well as standard performance techniques (for example, the comment
1260 The lexer has a couple of interesting modal features:
1262 * The lexer can operate in "raw" mode. This mode has several features that
1263 make it possible to quickly lex the file (e.g., it stops identifier lookup,
1266 * The lexer can capture and return comments as tokens. This is required to
1267 support the ``-C`` preprocessor mode, which passes comments through, and is
1268 used by the diagnostic checker to identifier expect-error annotations.
1269 * The lexer can be in ``ParsingFilename`` mode, which happens when
1270 preprocessing after reading a ``#include`` directive. This mode changes the
1272 for each thing within the filename.
1273 * When parsing a preprocessor directive (after "``#``") the
1274 ``ParsingPreprocessorDirective`` mode is entered. This changes the parser to
1276 * The ``Lexer`` uses a ``LangOptions`` object to know whether trigraphs are
1279 In addition to these modes, the lexer keeps track of a couple of other features
1280 that are local to a lexed buffer, which change as the buffer is lexed:
1282 * The ``Lexer`` uses ``BufferPtr`` to keep track of the current character being
1284 * The ``Lexer`` uses ``IsAtStartOfLine`` to keep track of whether the next
1286 * The ``Lexer`` keeps track of the current "``#if``" directives that are active
1288 * The ``Lexer`` keeps track of an :ref:`MultipleIncludeOpt
1289 <MultipleIncludeOpt>` object, which is used to detect whether the buffer uses
1290 the standard "``#ifndef XX`` / ``#define XX``" idiom to prevent multiple
1291 inclusion. If a buffer does, subsequent includes can be ignored if the
1296 The ``TokenLexer`` class
1299 The ``TokenLexer`` class is a token provider that returns tokens from a list of
1302 tokens from an arbitrary buffer of tokens. The later use is used by
1303 ``_Pragma`` and will most likely be used to handle unbounded look-ahead for the
1308 The ``MultipleIncludeOpt`` class
1311 The ``MultipleIncludeOpt`` class implements a really simple little state
1312 machine that is used to detect the standard "``#ifndef XX`` / ``#define XX``"
1314 buffer uses this idiom and is subsequently ``#include``'d, the preprocessor can
1315 simply check to see whether the guarding condition is defined or not. If so,
1316 the preprocessor can completely ignore the include of the header.
1320 The Parser Library
1323 This library contains a recursive-descent parser that polls tokens from the
1324 preprocessor and notifies a client of the parsing progress.
1326 Historically, the parser used to talk to an abstract ``Action`` interface that
1328 grew C++ support, the parser stopped supporting general ``Action`` clients --
1329 it now always talks to the :ref:`Sema library <Sema>`. However, the Parser
1331 ``StmtResult``. Only :ref:`Sema <Sema>` looks at the AST node contents of these
1336 The AST Library
1351 * Canonicalization of the "meaning" of nodes is possible as soon as the nodes
1355 function template declarations involving dependent expressions declare the
1357 * AST nodes can be reused when they have the same meaning. For example, we
1358 reuse ``Type`` nodes when representing the same type (but maintain separate
1362 * Serialization and deserialization of the AST to/from AST files is simpler:
1368 * The first declaration of a redeclarable entity maintains a pointer to the
1371 * Name lookup tables in declaration contexts change after the namespace
1375 the declaration versus the definition, so template instantiation often
1379 instantiations update the existing declaration.
1387 part of the complete AST, or where language semantics require after-the-fact
1393 The AST intends to provide a representation of the program that is faithful to
1394 the original source. We intend for it to be possible to write refactoring tools
1395 using only information stored in, or easily reconstructible from, the Clang AST.
1396 This means that the AST representation should either not desugar source-level
1398 or a clear engineering tradeoff -- should desugar minimally and wrap the result
1399 in a construct representing the original source form.
1401 For example, ``CXXForRangeStmt`` directly represents the syntactic form of a
1402 range-based for statement, but also holds a semantic representation of the
1410 with the same or similar semantics.
1414 The ``Type`` class and its subclasses
1417 The ``Type`` class (and its subclasses) are an important part of the AST.
1418 Types are accessed through the ``ASTContext`` class, which implicitly creates
1425 them. The issue is that we want to capture typedef information and represent it
1426 in the AST perfectly, but the semantics of operations need to "see through"
1441 The code above is illegal, and thus we expect there to be diagnostics emitted
1442 on the annotated lines. In this example, we expect to get:
1456 While this example is somewhat silly, it illustrates the point: we want to
1459 requires properly keeping typedef information (for example, the type of ``X``
1460 is "``foo``", not "``int``"), and requires properly propagating it through the
1461 various operators (for example, the type of ``*Y`` is "``foo``", not
1462 "``int``"). In order to retain this information, the type of these expressions
1463 is an instance of the ``TypedefType`` class, which indicates that the type of
1466 Representing types like this is great for diagnostics, because the
1468 with this: first, various semantic checks need to make judgements about the
1471 ignoring typedefs. The solution to both of these problems is the idea of
1479 Every instance of the ``Type`` class contains a canonical type pointer. For
1481 "``int**``"), the type just points to itself. For types that have a typedef
1483 "``bar``"), the canonical type pointer points to their structurally equivalent
1487 This design provides a constant time operation (dereferencing the canonical type
1488 pointer) that gives us access to the structure of types. For example, we can
1489 trivially tell that "``bar``" and "``foo*``" are the same type by dereferencing
1491 to the single "``int*``" type).
1494 carefully managed. Specifically, the ``isa``/``cast``/``dyn_cast`` operators
1495 generally shouldn't be used in code that is inspecting the AST. For example,
1496 when type checking the indirection operator (unary "``*``" on a pointer), the
1497 type checker must verify that the operand has a pointer type. It would not be
1499 this predicate would fail if the subexpression had a typedef type.
1501 The solution to this problem are a set of helper methods on ``Type``, used to
1503 "``SubExpr->getType()->isPointerType()``" to do the check. This predicate will
1504 return true if the *canonical type is a pointer*, which is true any time the
1505 type is structurally a pointer type. The only hard part here is remembering
1506 not to use the ``isa``/``cast``/``dyn_cast`` operations.
1508 The second problem we face is how to get access to the pointer type once we
1509 know it exists. To continue the example, the result type of the indirection
1510 operator is the pointee type of the subexpression. In order to determine the
1511 type, we need to get the instance of ``PointerType`` that best captures the
1512 typedef information in the program. If the type of the expression is literally
1513 a ``PointerType``, we can return that, otherwise we have to dig through the
1514 typedefs to find the pointer type. For example, if the subexpression had type
1515 "``foo*``", we could return that type as the result. If the subexpression had
1518 ``getAsPointerType()`` method that checks whether the type is structurally a
1519 ``PointerType`` and, if so, returns the best one. If not, it returns a null
1527 The ``QualType`` class
1530 The ``QualType`` class is designed as a trivial value class that is small,
1531 passed by-value and is efficient to query. The idea of ``QualType`` is that it
1532 stores the type qualifiers (``const``, ``volatile``, ``restrict``, plus some
1533 extended qualifiers required by language extensions) separately from the types
1534 themselves. ``QualType`` is conceptually a pair of "``Type*``" and the bits
1537 By storing the type qualifiers as bits in the conceptual pair, it is extremely
1538 efficient to get the set of qualifiers on a ``QualType`` (just return the field
1539 of the pair), add a type qualifier (which is a trivial constant-time operation
1541 ``QualType`` with the bitfield set to empty).
1543 Further, because the bits are stored outside of the type itself, we do not need
1546 const int``" both point to the same heap allocated "``int``" type). This
1547 reduces the heap size used to represent bits and also means we do not have to
1551 In practice, the two most common type qualifiers (``const`` and ``restrict``)
1552 are stored in the low bits of the pointer to the ``Type`` object, together with
1554 heap-allocated). This means that ``QualType`` is exactly the same size as a
1562 The ``DeclarationName`` class represents the name of a declaration in Clang.
1563 Declarations in the C family of languages can take several different forms.
1565 the function declaration ``f(int x)``. In C++, declaration names can also name
1569 declaration names can refer to the names of Objective-C methods, which involve
1570 the method name and the parameters, collectively called a *selector*, e.g.,
1578 the names are inside the ``DeclarationName`` class).
1582 The name is a simple identifier. Use ``N.getAsIdentifierInfo()`` to retrieve
1583 the corresponding ``IdentifierInfo*`` pointing to the actual identifier.
1587 The name is an Objective-C selector, which can be retrieved as a ``Selector``
1588 instance via ``N.getObjCSelector()``. The three possible name kinds for
1589 Objective-C reflect an optimization within the ``DeclarationName`` class:
1597 The name is a C++ constructor name. Use ``N.getCXXNameType()`` to retrieve
1598 the :ref:`type <QualType>` that this constructor is meant to construct. The
1599 type is always the canonical type, since all constructors for a given type
1600 have the same name.
1604 The name is a C++ destructor name. Use ``N.getCXXNameType()`` to retrieve
1605 the :ref:`type <QualType>` whose destructor is being named. This type is
1610 The name is a C++ conversion function. Conversion functions are named
1611 according to the type they convert to, e.g., "``operator void const *``".
1612 Use ``N.getCXXNameType()`` to retrieve the type that this conversion function
1617 The name is a C++ overloaded operator name. Overloaded operators are named
1619 Use ``N.getCXXOverloadedOperator()`` to retrieve the overloaded operator (a
1624 The name is a C++11 user defined literal operator. User defined
1625 Literal operators are named according to the suffix they define,
1627 ``N.getCXXLiteralIdentifier()`` to retrieve the corresponding
1628 ``IdentifierInfo*`` pointing to the identifier.
1632 The name is a C++ using directive. Using directives are not really
1633 NamedDecls, in that they all have the same name, but they are
1638 only a single pointer's worth of storage in the common cases (identifiers,
1640 for the other kinds of names. Two ``DeclarationName``\ s can be compared for
1647 what kind of name the instance will store. Normal identifiers
1651 from the ``DeclarationNameTable``, an instance of which is available as
1652 ``ASTContext::DeclarationNames``. The member functions
1655 return ``DeclarationName`` instances for the four kinds of C++ special function
1665 Clang are represented by the ``DeclContext`` class, from which the various
1667 ``RecordDecl``, ``FunctionDecl``, etc.) will derive. The ``DeclContext`` class
1672 ``DeclContext`` provides two views of the declarations stored within a
1673 declaration context. The source-centric view accurately represents the
1675 where present (see the section :ref:`Redeclarations and Overloads
1676 <Redeclarations>`), while the semantics-centric view represents the program
1677 semantics. The two views are kept synchronized by semantic analysis while
1678 the ASTs are being constructed.
1685 be stored within the ``DeclContext``, and one can iterate over the
1687 ``DeclContext::decls_end()``). This mechanism provides the source-centric
1688 view of declarations in the context.
1692 The ``DeclContext`` structure provides efficient name lookup for names within
1694 for the name ``N::f`` using ``DeclContext::lookup``. The lookup itself is
1697 declarations). The lookup operation provides the semantics-centric view of
1698 the declarations in the context.
1702 The ``DeclContext`` owns all of the declarations that were declared within
1703 its declaration context, and is responsible for the management of their
1707 information about the context in which each declaration lives. One can
1708 retrieve the ``DeclContext`` that contains a particular ``Decl`` using
1709 ``Decl::getDeclContext``. However, see the section
1728 The representation of "``f``" differs in the source-centric and
1729 semantics-centric views of a declaration context. In the source-centric view,
1730 all redeclarations will be present, in the order they occurred in the source
1731 code, making this view suitable for clients that wish to see the structure of
1732 the source code. In the semantics-centric view, only the most recent "``f``"
1733 will be found by the lookup, since it effectively replaces the first
1737 declaration, etc. it is possible that the declaration of ``f`` found by name
1738 lookup will not be the most recent one.)
1740 In the semantics-centric view, overloading of functions is represented
1749 the ``DeclContext::lookup`` operation will return a
1752 that is not concerned with the actual source code will primarily use this
1761 *lexical* context, which corresponds to the source-centric view of the
1762 declaration context, and a *semantic* context, which corresponds to the
1763 semantics-centric view. The lexical context is accessible via
1764 ``Decl::getLexicalDeclContext`` while the semantic context is accessible via
1766 most declarations, the two contexts are identical. For example:
1775 Here, the semantic and lexical contexts of ``X::f`` are the ``DeclContext``
1776 associated with the class ``X`` (itself stored as a ``RecordDecl`` AST node).
1783 This definition of "``f``" has different lexical and semantic contexts. The
1784 lexical context corresponds to the declaration context in which the actual
1785 declaration occurred in the source code, e.g., the translation unit containing
1786 ``X``. Thus, this declaration of ``X::f`` can be found by traversing the
1787 declarations provided by [``decls_begin()``, ``decls_end()``) in the
1790 The semantic context of ``X::f`` corresponds to the class ``X``, since this
1791 member function is (semantically) a member of ``X``. Lookup of the name ``f``
1792 into the ``DeclContext`` associated with ``X`` will then return the definition
1793 of ``X::f`` (including information about the default argument).
1799 declared inside another declaration will actually "leak" out into the enclosing
1800 scope from the perspective of name lookup. The most obvious instance of this
1812 the enumerators ``Red``, ``Green``, and ``Blue``. Thus, traversing the list of
1813 declarations contained in the enumeration ``Color`` will yield ``Red``,
1814 ``Green``, and ``Blue``. However, outside of the scope of ``Color`` one can
1815 name the enumerator ``Red`` without qualifying the name, e.g.,
1832 For source-level accuracy, we treat the linkage specification and enumeration
1835 declarations are visible outside of the scope of the declaration context.
1837 These language features (and several others, described below) have roughly the
1839 context, but the declarations are also found via name lookup in scopes
1840 enclosing the declaration itself. This feature is implemented via
1842 ``DeclContext::isTransparentContext()``), whose declarations are visible in the
1843 nearest enclosing non-transparent declaration context. This means that the
1844 lexical context of the declaration (e.g., an enumerator) will be the
1845 transparent ``DeclContext`` itself, as will the semantic context, but the
1846 declaration will be visible in every outer context up to and including the
1850 The transparent ``DeclContext``\ s are:
1886 LT.Vector = 0; // Okay: finds Vector inside the unnamed union
1904 C++ namespaces have the interesting property that
1905 the namespace can be defined multiple times, and the declarations provided by
1906 each namespace definition are effectively merged (from the semantic point of
1907 view). For example, the following two code snippets are semantically
1926 In Clang's representation, the source-centric view of declaration contexts will
1929 However, the semantics-centric view provided by name lookup into the namespace
1933 ``DeclContext`` manages multiply-defined declaration contexts internally. The
1934 function ``DeclContext::getPrimaryContext`` retrieves the "primary" context for
1935 a given ``DeclContext`` instance, which is the ``DeclContext`` responsible for
1936 maintaining the lookup table used for the semantics-centric view. Given a
1937 DeclContext, one can obtain the set of declaration contexts that are
1939 this context (which will be the only result, for non-namespace contexts) via
1941 internally within the lookup and insertion methods of the ``DeclContext``, so
1942 the vast majority of clients can ignore them.
1944 Because the same entity can be defined multiple times in different modules,
1946 a ``CXXRecordDecl``, all of which describe a definition of the same class.
1948 the definition of the class, and the others are treated as non-defining
1951 either by redeclaration chains (if the members are ``Redeclarable``)
1952 or by simply a pointer to the canonical declaration (if the declarations
1959 Clang produces an AST even when the code contains errors. Clang won't generate
1961 errors in the input. Clang-based tools also depend on such ASTs, and IDEs in
1964 In presence of errors, clang uses a few error-recovery strategies to present the
1965 broken code in the AST:
1967 - correcting errors: in cases where clang is confident about the fix, it
1968 provides a FixIt attaching to the error diagnostic and emits a corrected AST
1969 (reflecting the written code with FixIts applied). The advantage of that is to
1972 - representing invalid node: the invalid node is preserved in the AST in some
1973 form, e.g. when the "declaration" part of the declaration contains semantic
1974 errors, the Decl node is marked as invalid.
1980 consumers a rich AST reflecting the written source code as much as possible even
1986 The idea of Recovery AST is to use recovery nodes which act as a placeholder to
1987 maintain the rough structure of the parsing tree, preserve locations and
1990 For example, consider the following mismatched function call:
1999 Without Recovery AST, the invalid function call expression (and its child
2000 expressions) would be dropped in the AST:
2010 With Recovery AST, the AST looks like:
2023 An alternative is to use existing Exprs, e.g. CallExpr for the above example.
2025 it to be treated uniformly with valid CallExprs. However, jamming the data we
2027 wrong. This would introduce a huge burden on consumers of the AST to handle such
2031 ``RecoveryExpr`` is the only recovery node so far. In practice, broken decls
2032 need more detailed semantics preserved (the current ``Invalid`` flag works
2034 structure are rare (so dropping the statements is OK).
2039 ``RecoveryExpr`` is an ``Expr``, so it must have a type. In many cases the true
2040 type can't really be known until the code is corrected (e.g. a call to a
2044 To model this, we generalize the concept of dependence from C++ templates to
2045 mean dependence on a template parameter or how an error is repaired. The
2046 ``RecoveryExpr`` ``unknownFunction()`` has the totally unknown type
2047 ``DependentTy``, and this suppresses type-based analysis in the same way it
2050 In cases where we are confident about the concrete type (e.g. the return type
2051 for a broken non-overloaded function call), the ``RecoveryExpr`` will have this
2060 Whether or not the ``RecoveryExpr`` has a dependent type, it is always
2061 considered value-dependent, because its value isn't well-defined until the error
2069 Beyond the template dependence bits, we add a new “ContainsErrors” bit to
2082 // not type-dependent, as we know the type is std::string
2094 The ASTImporter
2097 The ``ASTImporter`` class imports nodes of an ``ASTContext`` into another
2098 ``ASTContext``. Please refer to the document :doc:`ASTImporter: Merging Clang
2099 ASTs <LibASTImporter>` for an introduction. And please read through the
2100 high-level `description of the import algorithm
2101 <LibASTImporter.html#algorithm-of-the-import>`_, this is essential for
2102 understanding further implementation details of the importer.
2109 Despite the name, the Clang AST is not a tree. It is a directed graph with
2110 cycles. One example of a cycle is the connection between a
2111 ``ClassTemplateDecl`` and its "templated" ``CXXRecordDecl``. The *templated*
2112 ``CXXRecordDecl`` represents all the fields and methods inside the class
2113 template, while the ``ClassTemplateDecl`` holds the information which is
2114 related to being a template, i.e. template arguments, etc. We can get the
2115 *templated* class (the ``CXXRecordDecl``) of a ``ClassTemplateDecl`` with
2116 ``ClassTemplateDecl::getTemplatedDecl()``. And we can get back a pointer of the
2117 "described" class template from the *templated* class:
2119 nodes: between the *templated* and the *described* node. There may be various
2120 other kinds of cycles in the AST especially in case of declarations.
2127 Importing one AST node copies that node into the destination ``ASTContext``. To
2128 copy one node means that we create a new node in the "to" context then we set
2129 its properties to be equal to the properties of the source node. Before the
2130 copy, we make sure that the source node is not *structurally equivalent* to any
2131 existing node in the destination context. If it happens to be equivalent then
2132 we skip the copy.
2134 The informal definition of structural equivalency is the following:
2137 - builtin types and refer to the same type, e.g. ``int`` and ``int`` are
2140 - record types and all their fields in order of their definition have the same
2142 - variable or function declarations and they have the same identifier name and
2146 a formal definition of *compatible types*, please refer to 6.2.7/1 in the C11
2147 standard. However, there is no definition for *compatible types* in the C++
2148 standard. Still, we extend the definition of structural equivalency to
2149 templates and their instantiations similarly: besides checking the previously
2153 The structural equivalent check can be and is used independently from the
2154 ASTImporter, e.g. the ``clang::Sema`` class uses it also.
2156 The equivalence of nodes may depend on the equivalency of other pairs of nodes.
2157 Thus, the check is implemented as a parallel graph traversal. We traverse
2158 through the nodes of both graphs at the same time. The actual implementation is
2159 similar to breadth-first-search. Let's say we start the traverse with the <A,B>
2160 pair of nodes. Whenever the traversal reaches a pair <X,Y> then the following
2163 - A and X are nodes from the same ASTContext.
2164 - B and Y are nodes from the same ASTContext.
2165 - A and B may or may not be from the same ASTContext.
2166 - if A == X and B == Y (pointer equivalency) then (there is a cycle during the
2171 - All dependent nodes on the path from <A,B> to <X,Y> are structurally
2177 have the same names. This is the way how we compare forward declarations with
2180 .. TODO Should we elaborate the actual implementation of the graph traversal,
2186 The early version of the ``ASTImporter``'s merge mechanism squashed the
2189 function prototype, but it imported a definition. To demonstrate the problem
2190 with this approach let's consider an empty "to" context and the following
2191 ``virtual`` function declarations of ``f`` in the "from" context:
2198 If we imported the definition with the "squashing" approach then we would
2200 returns ``false`` for it. The reason is that the definition is indeed not
2201 virtual, it is the property of the prototype!
2203 Consequently, we must either set the virtual flag for the definition (but then
2204 we create a malformed AST which the parser would never create), or we import
2205 the whole redeclaration chain of the function. The most recent version of the
2206 ``ASTImporter`` uses the latter mechanism. We do import all function
2207 declarations - regardless if they are definitions or prototypes - in the order
2208 as they appear in the "from" context.
2212 If we have an existing definition in the "to" context, then we cannot import
2213 another definition, we will use the existing definition. However, we can import
2214 prototype(s): we chain the newly imported prototype(s) to the existing
2216 be added to the end of the redeclaration chain. This may result in long
2218 translation units which include the same header with the prototype.
2222 To mitigate the problem of long redeclaration chains of free functions, we
2223 could compare prototypes to see if they have the same properties and if yes
2224 then we could merge these prototypes. The implementation of squashing of
2229 Chaining functions this way ensures that we do copy all information from the
2246 attach a new prototype to the existing in-class prototype. Consider the
2264 When we import the prototype and the definition of ``f`` from the "from"
2265 context, then the resulting redecl chain will look like this ``D0 -> D2'``,
2266 where ``D2'`` is the copy of ``D2`` in the "to" context.
2271 attach the newly imported declaration to the existing redeclaration chain (if
2272 there is structural equivalency). We do not import, however, the whole
2274 found any essential property of forward declarations which is similar to the
2275 case of the virtual flag in a member function prototype. In the future, this
2278 Traversal during the Import
2281 The node specific import mechanisms are implemented in
2284 call the constructor of that declaration node. Everything which can be set
2285 later is set after the node is created. For example, in case of a
2286 ``FunctionDecl`` we first import the declaration context in which the function
2287 is declared, then we create the ``FunctionDecl`` and only then we import the
2288 body of the function. This means there are implicit dependencies between AST
2289 nodes. These dependencies determine the order in which we visit nodes in the
2290 "from" context. As with the regular graph traversal algorithms like DFS, we
2293 add that to the ``ImportedDecls``. We must not start the import of any other
2294 declarations before we keep track of the newly created one. This is essential,
2299 already marked as imported then we just return its counterpart in the "to"
2303 Even with the use of ``GetImportedOrCreateDecl()`` there is still a
2305 each other in wrong way. Imagine that during the import of ``A``, the import of
2306 ``B`` is requested before we could create the node for ``A`` (the constructor
2307 needs a reference to ``B``). And the same could be true for the import of ``B``
2308 (``A`` is requested to be imported before we could create the node for ``B``).
2309 In case of the :ref:`templated-described swing <templated>` we take
2310 extra attention to break the cyclical dependency: we import and set the
2311 described template only after the ``CXXRecordDecl`` is created. As a best
2312 practice, before creating the node in the "to" context, avoid importing of
2313 other nodes which are not needed for the constructor of node ``A``.
2319 ``llvm::Expected<T>`` object. This enforces to check the return value of the
2321 that error. (Exception: when we import the members of a class, we collect the
2323 object.) We cache these errors in cases of declarations. During the next import
2324 call if there is an existing error we just return with that. So, clients of the
2329 the error to the caller, but the "to" context remains polluted with those nodes
2331 that time we did not know about the error, the error happened later. Since the
2332 AST is immutable (most of the cases we can't remove existing nodes) we choose
2335 We cache the errors associated with declarations in the "from" context in
2336 ``ASTImporter::ImportDeclErrors`` and the ones which are associated with the
2338 be several ASTImporter objects which import into the same "to" context but from
2339 different "from" contexts; in this case, they have to share the associated
2340 errors of the "to" context.
2342 When an error happens, that propagates through the call stack, through all the
2344 because we strive to mark the erroneous nodes so clients can act upon. In those
2345 cases, we have to keep track of the errors for those nodes which are
2348 An **import path** is the list of the AST nodes which we visit during an Import
2349 call. If node ``A`` depends on node ``B`` then the path contains an ``A->B``
2350 edge. From the call stack of the import functions, we can read the very same
2353 Now imagine the following AST, where the ``->`` represents dependency in terms
2354 of the import (all nodes are declarations).
2362 The import behaves like a DFS, so we will visit the nodes in this order: ABCDE.
2363 During the visitation we will have the following import paths:
2377 If during the visit of E there is an error then we set an error for E, then as
2378 the call stack shrinks for B, then for A:
2392 However, during the import we could import C and D without any error and they
2394 the end of the import we have an entry in ``ImportDeclErrors`` for A,B,E but
2397 Now, what happens if there is a cycle in the import path? Let's consider this
2405 During the visitation, we will have the below import paths and if during the
2422 up an error for C too. As the call stack reverses back we get to A and we must
2424 longer on the import path, it just had been previously. Such a situation can
2425 happen only if during the visitation we had a cycle. If we didn't have any
2426 cycle, then the normal way of passing an Error object through the call stack
2427 could handle the situation. This is why we must track cycles during the import
2433 When we import a declaration from the source context then we check whether we
2434 already have a structurally equivalent node with the same name in the "to"
2435 context. If the "from" node is a definition and the found one is also a
2436 definition, then we do not create a new node, instead, we mark the found node
2437 as the imported node. If the found definition and the one we want to import
2438 have the same name but they are structurally in-equivalent, then we have an ODR
2439 violation in case of C++. If the "from" node is not a definition then we add
2440 that to the redeclaration chain of the found node. This behaviour is essential
2441 when we merge ASTs from different translation units which include the same
2442 header file(s). For example, we want to have only one definition for the class
2446 To find a structurally equivalent node we can use the regular C/C++ lookup
2448 ``DeclContext::localUncachedLookup()``. These functions do respect the C/C++
2452 is a problem, because if we use the regular C/C++ lookup then we create
2453 redundant AST nodes during the merge! Also, having two instances of the same
2455 of other nodes which depend on the duplicated node. Because of these reasons,
2456 we created a lookup class which has the sole purpose to register all
2458 This is the ``ASTImporterLookupTable`` class. This lookup table should be
2459 shared amongst the different ``ASTImporter`` instances if they happen to import
2460 to the very same "to" context. This is why we can use the importer specific
2461 lookup only via the ``ASTImporterSharedState`` class.
2466 The ``ExternalASTSource`` is an abstract interface associated with the
2467 ``ASTContext`` class. It provides the ability to read the declarations stored
2470 on-demand. This means that the list of declarations (represented as a linked
2471 list, the head is ``DeclContext::FirstDecl``) could be empty. However, member
2475 when we load a class from a PCH then the members are loaded only if we do want
2476 to look up something in the class' context.
2478 In case of LLDB, an implementation of the ``ExternalASTSource`` interface is
2479 attached to the AST context which is related to the parsed expression. This
2480 implementation of the ``ExternalASTSource`` interface is realized with the help
2481 of the ``ASTImporter`` class. This way, LLDB can reuse Clang's parsing
2482 machinery while synthesizing the underlying AST from the debug data (e.g. from
2483 DWARF). From the view of the ``ASTImporter`` this means both the "to" and the
2485 a ``DeclContext`` in the "to" AST context has external lexical storage then we
2486 must take extra attention to work only with the already loaded declarations!
2488 if we used the regular ``DeclContext::lookup()`` to find the existing
2489 declarations in the "to" context then the ``lookup()`` call itself would
2490 initiate a new import while we are in the middle of importing a declaration!
2491 (By the time we initiate the lookup we haven't registered yet that we already
2492 started to import the node of the "from" context.) This is why we use
2498 Different translation units may have class template instantiations with the
2500 ``MethodDecls`` and ``FieldDecls``. Consider the following files:
2524 In ``foo.cpp`` we use the constructor with number ``(1)``, which explicitly
2525 initializes the member ``a`` to ``3``, thus the ``InitListExpr`` ``{0}`` is not
2526 used here and the AST node is not instantiated. However, in the case of
2527 ``bar.cpp`` we use the constructor with number ``(2)``, which does not
2528 explicitly initialize the ``a`` member, so the default ``InitListExpr`` is
2529 needed and thus instantiated. When we merge the AST of ``foo.cpp`` and
2530 ``bar.cpp`` we must create an AST node for the class template instantiation of
2531 ``X<char>`` which has all the required nodes. Therefore, when we find an
2532 existing ``ClassTemplateSpecializationDecl`` then we merge the fields of the
2533 ``ClassTemplateSpecializationDecl`` in the "from" context in a way that the
2534 ``InitListExpr`` is copied if not existent yet. The same merge mechanism should
2535 be done in the cases of instantiated default arguments and exception
2543 During import of a global variable with external visibility, the lookup will
2544 find variables (with the same name) but with static visibility (linkage).
2545 Clearly, we cannot put them into the same redeclaration chain. The same is true
2546 the in case of functions. Also, we have to take care of other kinds of
2548 Therefore, we filter the lookup results and consider only those which have the
2549 same visibility as the declaration we currently import.
2551 We consider two declarations in two anonymous namespaces to have the same
2552 visibility only if they are imported from the same AST context.
2557 During the import we lookup existing declarations with the same name. We filter
2558 the lookup results based on their :ref:`visibility <visibility>`. If any of the
2561 ``Error`` and we set up the ``Error`` object for the declaration. However, some
2562 clients of the ``ASTImporter`` may require a different, perhaps less
2565 E.g. static analysis clients may benefit if the node is created even if there
2566 is a name conflict. During the CTU analysis of certain projects, we recognized
2570 these collisions liberally then CTU analysis can find more results. Note, the
2576 The ``CFG`` class
2579 The ``CFG`` class is designed to represent a source-level control-flow graph
2582 can also be instantiated to represent the control-flow of any class that
2593 of ``Stmt*`` (each referring to statements in the AST). The ordering of
2595 statement to the next. :ref:`Conditional control-flow
2596 <ConditionalControlFlow>` is represented using edges between basic blocks. The
2597 statements within a given ``CFGBlock`` can be traversed using the
2600 A ``CFG`` object owns the instances of ``CFGBlock`` within the control-flow
2602 (accessible via ``CFGBlock::getBlockID()``). Currently the number is based on
2603 the ordering the blocks were created, but no assumptions should be made on how
2605 are numbered from 0..N-1 (where N is the number of basic blocks in the CFG).
2613 Neither block contains any statements, and they serve the role of providing a
2614 clear entrance and exit for a body of code such as a function body. The
2615 presence of these empty blocks greatly simplifies the implementation of many
2626 ``Stmt*`` that represents the *terminator* of the block. A terminator is
2627 simply the statement that caused the control-flow, and is used to identify the
2628 nature of the conditional control-flow between blocks. For example, in the
2629 case of an if-statement, the terminator refers to the ``IfStmt`` object in the
2630 AST that represented the given branch.
2632 To illustrate, consider the following code example:
2648 After invoking the parser+semantic analyzer on this code fragment, the AST of
2649 the body of ``foo`` is referenced by a single ``Stmt*``. We can then construct
2650 an instance of ``CFG`` representing the control-flow graph of this function
2658 Along with providing an interface to iterate over its ``CFGBlocks``, the
2660 visualizing CFGs. For example, the method ``CFG::dump()`` dumps a
2661 pretty-printed version of the CFG to standard error. This is especially useful
2662 when one is using a debugger such as gdb. For example, here is the output of
2698 For each block, the pretty-printed output displays for each block the number of
2699 *predecessor* blocks (blocks that have outgoing control-flow to the given
2701 control-flow from the given block). We can also clearly see the special entry
2702 and exit blocks at the beginning and end of the pretty-printed output. For the
2703 entry block (block B5), the number of predecessor blocks is 0, while for the
2704 exit block (block B0) the number of successor blocks is 0.
2706 The most interesting block here is B4, whose outgoing control-flow represents
2707 the branching caused by the sole if-statement in ``foo``. Of particular
2708 interest is the second statement in the block, ``(x > 2)``, and the terminator,
2709 printed as ``if [B4.2]``. The second statement represents the evaluation of
2710 the condition of the if-statement, which occurs before the actual branching of
2711 control-flow. Within the ``CFGBlock`` for B4, the ``Stmt*`` for the second
2712 statement refers to the actual expression in the AST for ``(x > 2)``. Thus
2713 pointers to subclasses of ``Expr`` can appear in the list of statements in a
2716 The terminator of block B4 is a pointer to the ``IfStmt`` object in the AST.
2717 The pretty-printer outputs ``if [B4.2]`` because the condition expression of
2718 the if-statement has an actual place in the basic block, and thus the
2719 terminator is essentially *referring* to the expression that is the second
2722 are hoisted into the actual basic block.
2727 .. A key design principle of the ``CFG`` class was to not require any
2728 .. transformations to the AST in order to represent control-flow. Thus the
2729 .. ``CFG`` does not perform any "lowering" of the statements in an AST: loops
2733 Constant Folding in the Clang AST
2737 the Clang front-end. First, in general, we prefer the AST to retain the source
2738 code as close to how the user wrote it as possible. This means that if they
2739 wrote "``5+4``", we want to keep the addition and two constants in the AST, we
2741 ways turns into a tree walk that needs to handle the various cases.
2744 folded. For example, the C standard defines what an "integer constant
2745 expression" (i-c-e) is with very precise and specific requirements. The
2746 language then requires i-c-e's in a lot of places (for example, the size of a
2747 bitfield, the value for a case statement, etc). For these, we have to be able
2748 to constant fold the constants, to do semantic checks (e.g., verify bitfield
2750 Clang to be very pedantic about this, diagnosing cases when the code does not
2751 use an i-c-e where one is required, but accepting the code unless running with
2757 this unfortunate accident of history (including, e.g., the glibc system
2759 an integer constant, which means that the definition of what it accepts changes
2763 Another issue are how constants interact with the extensions we support, such
2765 others. C99 obviously does not specify the semantics of any of these
2766 extensions, and the definition of i-c-e does not include them. However, these
2770 Finally, this is not just a problem for semantic analysis. The code generator
2774 "``foo() || 1``" always evaluates to ``true``, but we can't replace the
2781 (Note, at the time of this writing, not all of this has been implemented,
2785 fp, complex, or pointer) this method returns the following information:
2787 * Whether the expression is an integer constant expression, a general constant
2790 * If the expression was computable in any way, this method returns the
2791 ``APValue`` for the result of the expression.
2792 * If the expression is not evaluatable at all, this method returns information
2793 on one of the problems with the expression. This includes a
2794 ``SourceLocation`` for where the problem is, and a diagnostic ID that explains
2795 the problem. The diagnostic should have ``ERROR`` type.
2796 * If the expression is not an integer constant expression, this method returns
2797 information on one of the problems with the expression. This includes a
2798 ``SourceLocation`` for where the problem is, and a diagnostic ID that
2799 explains the problem. The diagnostic should have ``EXTENSION`` type.
2801 This information gives various clients the flexibility that they want, and we
2804 calls ``Evaluate`` on the expression. If the expression is not foldable, the
2805 error is emitted, and it would return ``true``. If the expression is not an
2806 i-c-e, the ``EXTENSION`` diagnostic is emitted. Finally it would return
2807 ``false`` to indicate that the AST is OK.
2809 Other clients can use the information in other ways, for example, codegen can
2815 This section describes how some of the various extensions Clang supports
2818 * ``__extension__``: The expression form of this extension causes any
2821 expression) if the operand evaluates to either a numeric value (that is, not
2823 complex type, or if it evaluates to the address of the first character of a
2825 ``__builtin_constant_p`` is the (potentially parenthesized) condition of a
2826 conditional operator expression ("``?:``"), only the true side of the
2829 * ``__builtin_choose_expr``: The condition is required to be an integer
2831 extension". This only evaluates one operand depending on which way the
2840 constant expressions if the argument is a string literal.
2844 The Sema Library
2847 This library is called by the :ref:`Parser library <Parser>` during parsing to
2848 do semantic analysis of the input. For valid programs, Sema builds an AST for
2853 The CodeGen Library
2865 allowing the programmer to pass semantic information along to the compiler for
2866 various uses. For example, attributes may be used to alter the code generation
2876 and then the semantic handling of the attribute.
2878 Parsing of the attribute is determined by the various syntactic forms attributes
2880 information provided by the table definition of the attribute. Ultimately, the
2883 to a declarator or declaration specifier. The parsing of attributes is handled
2885 keywords. When implementing a custom keyword attribute, the parsing of the
2886 keyword and creation of the ``ParsedAttr`` object must be done manually.
2889 a ``ParsedAttr``, at which point the parsed attribute can be transformed
2890 into a semantic attribute. The process by which a parsed attribute is converted
2891 into a semantic attribute depends on the attribute definition and semantic
2892 requirements of the attribute. The end result, however, is that the semantic
2893 attribute object is attached to the ``Decl`` object, and can be obtained by a
2898 The structure of the semantic attribute is also governed by the attribute
2900 functionality used for the implementation of the attribute, such as a class
2901 derived from ``clang::Attr``, information for the parser to use, automated
2907 The first step to adding a new attribute to Clang is to add its definition to
2910 This tablegen definition must derive from the ``Attr`` (tablegen, not
2911 semantic) type, or one of its derivatives. Most attributes will derive from the
2912 ``InheritableAttr`` type, which specifies that the attribute can be inherited by
2913 later redeclarations of the ``Decl`` it is associated with.
2914 ``InheritableParamAttr`` is similar to ``InheritableAttr``, except that the
2915 attribute is written on a parameter instead of a declaration. If the attribute
2916 applies to statements, it should inherit from ``StmtAttr``. If the attribute is
2919 (Note that this document does not cover the creation of type attributes.) An
2924 The definition will specify several key pieces of information, such as the
2925 semantic name of the attribute, the spellings the attribute supports, the
2926 arguments the attribute expects, and more. Most members of the ``Attr`` tablegen
2927 type do not require definitions in the derived definition as the default
2933 All attributes are required to specify a spelling list that denotes the ways in
2934 which the attribute can be spelled. For instance, a single semantic attribute
2937 are created implicitly. The following spellings are accepted:
2950 ``CustomKeyword`` The attribute is spelled as a keyword, and requires
2952 ``RegularKeyword`` The attribute is spelled as a keyword. It can be
2953 used in exactly the places that the standard
2955 exactly the same thing that a standard attribute
2956 would appertain to. Lexing and parsing of the keyword
2958 ``GCC`` Specifies two or three spellings: the first is a
2959 GNU-style spelling, the second is a C++-style spelling
2960 with the ``gnu`` namespace, and the third is an optional
2961 C-style spelling with the ``gnu`` namespace. Attributes
2964 ``Clang`` Specifies two or three spellings: the first is a
2965 GNU-style spelling, the second is a C++-style spelling
2966 with the ``clang`` namespace, and the third is an
2967 optional C-style spelling with the ``clang`` namespace.
2969 ``Pragma`` The attribute is spelled as a ``#pragma``, and requires
2970 custom processing within the preprocessor. If the
2972 set the namespace to ``"clang"``. Note that this
2976 The C++ standard specifies that “any [non-standard attribute] that is not
2977 recognized by the implementation is ignored” (``[dcl.attr.grammar]``).
2978 The rule for C is similar. This makes ``CXX11`` and ``C23`` spellings
2979 unsuitable for attributes that affect the type system, that change the
2980 binary interface of the code, or that have other similar semantic meaning.
2983 It reuses the production rules for standard attributes, but it applies them
2985 recognize the keyword are likely to report an error of some kind.
2987 For example, the ``ArmStreaming`` function type attribute affects
2988 both the type system and the binary interface of the function.
2991 the code. ``ArmStreaming`` is instead spelled ``__arm_streaming``, but it
2996 Attributes appertain to one or more subjects. If the attribute attempts to
2997 attach to a subject that is not in the subject list, a diagnostic is issued
2998 automatically. Whether the diagnostic is a warning or an error depends on how
2999 the attribute's ``SubjectList`` is defined, but the default behavior is to warn.
3000 The diagnostics displayed to the user are automatically determined based on the
3001 subjects in the list, but a custom diagnostic parameter can also be specified in
3002 the ``SubjectList``. The diagnostics generated for subject list violations are
3003 calculated automatically or specified by the subject list itself. If a
3004 previously unused Decl node is added to the ``SubjectList``, the logic used to
3005 automatically determine the diagnostic parameter in `utils/TableGen/ClangAttrEmitter.cpp
3009 By default, all subjects in the SubjectList must either be a Decl node defined
3014 called when determining whether an attribute appertains to the subject. For
3016 tests whether the given FieldDecl is a bit field. When a SubsetSubject is
3025 Documentation is table generated on the public web server by a server-side
3026 process that runs daily. Generally, the documentation for an attribute is a
3029 that is named after the attribute being documented.
3031 If the attribute is not for public consumption, or is an implicitly-created
3032 attribute that has no visible spelling, the documentation list can specify the
3033 ``InternalOnly`` object. Otherwise, the attribute should have its documentation
3036 Documentation derives from the ``Documentation`` tablegen type. All derived
3037 types must specify a documentation category and the actual documentation itself.
3038 Additionally, it can specify a custom heading for the attribute, though a
3046 Custom categories are good for providing overview information for the attributes
3047 grouped under it. For instance, the consumed annotation attributes define a
3054 After writing the documentation for the attribute, it should be locally tested
3055 to ensure that there are no issues generating the documentation on the server.
3056 Local testing requires a fresh build of clang-tblgen. To generate the attribute
3057 documentation, execute the following command::
3062 This file is generated by the server automatically, and any changes made to this
3067 Attributes may optionally specify a list of arguments that can be passed to the
3068 attribute. Attribute arguments specify both the parsed form and the semantic
3069 form of the attribute. For example, if ``Args`` is
3072 two arguments while parsing, and the Attr subclass' constructor for the
3075 All arguments have a name and a flag that specifies whether the argument is
3076 optional. The associated C++ type of the argument is determined by the argument
3077 definition type. If the existing argument types are insufficient, new types can
3080 to properly support the type.
3084 The ``Attr`` definition has other members which control the behavior of the
3085 attribute. Many of them are special-purpose and beyond the scope of this
3088 If the parsed form of the attribute is more complex, or differs from the
3089 semantic form, the ``HasCustomParsing`` bit can be set to ``1`` for the class,
3090 and the parsing code in `Parser::ParseGNUAttributeArgs()
3092 can be updated for the special case. Note that this only applies to arguments
3097 handling, requiring extra implementation efforts to ensure the attribute
3098 appertains to the appropriate subject, etc.
3100 If the attribute should not be propagated from a template declaration to an
3101 instantiation of the template, set the ``Clone`` member to 0. By default, all
3104 Attributes that do not require an AST node should set the ``ASTNode`` field to
3105 ``0`` to avoid polluting the AST. Note that anything inheriting from
3107 other attributes generate an AST node by default. The AST node is the semantic
3108 representation of the attribute.
3110 The ``LangOpts`` field specifies a list of language options required by the
3111 attribute. For instance, all of the CUDA-specific attributes specify ``[CUDA]``
3112 for the ``LangOpts`` field, and when the CUDA language option is not enabled, an
3115 should specify the spelling used by ``LangOptions`` class.
3117 Custom accessors can be generated for an attribute based on the spelling list
3121 These accessors will be generated on the semantic form of the attribute,
3124 Attributes that do not require custom semantic handling should set the
3131 is automatically provided, should set the ``SimpleHandler`` field to ``1``.
3134 different targets. For instance, the ARM and MSP430 targets both have an
3138 should be the same value between all arguments sharing a spelling, and
3139 corresponds to the parsed attribute's ``Kind`` enumerator. This allows
3141 attribute classes. For instance, ``ParsedAttr`` is the shared
3142 parsed attribute kind, but ARMInterruptAttr and MSP430InterruptAttr are the
3145 By default, attribute arguments are parsed in an evaluated context. If the
3147 the way the argument to a ``sizeof`` expression is parsed), set
3150 If additional functionality is desired for the semantic form of the attribute,
3151 the ``AdditionalMembers`` field specifies code to be copied verbatim into the
3154 If two or more attributes cannot be used in combination on the same declaration
3156 generate diagnostic code. This will disallow the attribute combinations
3158 the same attribute list, different attribute list, and redeclarations, as
3165 and generally starts in the ``ProcessDeclAttribute()`` function. If the
3166 attribute has the ``SimpleHandler`` field set to ``1`` then the function to
3167 process the attribute will be automatically generated, and nothing needs to be
3169 the switch statement. Please do not implement handling logic directly in the
3170 ``case`` for the attribute.
3172 Unless otherwise specified by the attribute definition, common semantic checking
3173 of the parsed attribute is handled automatically. This includes diagnosing
3174 parsed attributes that do not appertain to the given ``Decl`` or ``Stmt``,
3175 ensuring the correct minimum number of arguments are passed, etc.
3177 If the attribute adds additional warnings, define a ``DiagGroup`` in
3180 named after the attribute's ``Spelling`` with "_"s replaced by "-"s. If there
3191 Most attributes are implemented to have some effect on the compiler. For
3192 instance, to modify the way code is generated, or to add extra semantic checks
3193 for an analysis pass, etc. Having added the attribute definition and conversion
3194 to the semantic representation for the attribute, what remains is to implement
3195 the custom logic requiring use of the attribute.
3197 The ``clang::Decl`` object can be queried for the presence or absence of an
3198 attribute using ``hasAttr<T>()``. To obtain a pointer to the semantic
3199 representation of the attribute, ``getAttr<T>`` may be used.
3201 The ``clang::AttributedStmt`` object can be queried for the presence or absence
3202 of an attribute by calling ``getAttrs()`` and looping over the list of
3208 Expressions and statements are one of the most fundamental constructs within a
3209 compiler, because they interact with many different parts of the AST, semantic
3211 kind into Clang requires some care. The following list details the various
3213 with patterns to follow to ensure that the new expression or statement works
3214 well across all of the C languages. We focus on expressions, but statements
3217 #. Introduce parsing actions into the parser. Recursive-descent parsing is
3223 between source code and the AST.
3224 * Write tests for all of the "bad" parsing cases, to make sure your recovery
3231 directly from the parser, and a ``BuildXXX`` function that performs the
3232 actual semantic analysis and will (eventually!) build the AST node. It's
3233 fairly common for the ``ActOnXXX`` function to do very little (often just
3234 some minor translation from the parser's representation to ``Sema``'s
3235 representation of the same thing), but the separation is still important:
3236 C++ template instantiation, for example, should always call the ``BuildXXX``
3238 of the AST:
3241 Make sure to fully check that those types, and the types of those
3243 necessary to make sure that all of the types line up exactly the way you
3248 whether the type is "dependent" (``Type::isDependentType()``) or whether a
3253 use your expression within templates, but don't try to instantiate the
3258 (``Sema::DefaultLvalueConversions``) or the usual unary conversions
3259 (``Sema::UsualUnaryConversions``), for places where the subexpression is
3266 the node in ``include/Basic/StmtNodes.td`` and creating a new class for your
3267 expression in the appropriate ``include/AST/Expr*.h`` header. It's best to
3268 look at the class for a similar expression to get ideas, and there are some
3271 * If you need to allocate memory, use the ``ASTContext`` allocator to
3273 resources in an AST node, because the destructor of an AST node is never
3275 * Make sure that ``getSourceRange()`` covers the exact source range of your
3277 * Make sure that ``children()`` visits all of the subexpressions. This is
3282 * Add profiling support (``StmtProfile.cpp``) for your AST node, noting the
3296 that the object gets properly destructed. An easy way to test this is to
3298 flag an error here with the attempt to call the destructor.
3299 * Inspect the generated AST by printing it using ``clang -cc1 -ast-print``,
3300 to make sure you're capturing all of the important information about how
3301 the AST was written.
3302 * Inspect the generated AST under ``clang -cc1 -ast-dump`` to verify that
3303 all of the types in the generated AST line up the way you want them.
3304 Remember that clients of the AST should never have to "think" to
3306 show up explicitly in the AST.
3309 an argument? Can you use the ternary operator?
3311 #. Teach code generation to create IR to your AST node. This step is the first
3321 ``clang::QualType``) to LLVM types. Use the former for values, and the
3322 latter for memory locations: test with the C++ "``bool``" type to check
3323 this. If you find that you are having to use LLVM bitcasts to make the
3324 subexpressions of your expression have the type that your expression
3325 expects, STOP! Go fix semantic analysis and the AST so that you don't
3327 * The ``CodeGenFunction`` class has a number of helper functions to make
3331 because these functions take care of some of the tricky details for you
3333 * If your expression requires some special behavior in the event of an
3334 exception, look at the ``push*Cleanup`` functions in ``CodeGenFunction``
3340 generating the right IR.
3345 * Make sure that your expression's constructor properly computes the flags
3346 for type dependence (i.e., the type your expression produces can change
3347 from one instantiation to the next), value dependence (i.e., the constant
3348 value your expression produces can change from one instantiation to the
3352 just means combining the results from the various types and
3354 * Add ``TransformXXX`` and ``RebuildXXX`` functions to the ``TreeTransform``
3356 transform all of the subexpressions and types within your expression,
3357 using ``getDerived().TransformYYY``. If all of the subexpressions and
3358 types transform without error, it will then call the ``RebuildXXX``
3364 some of which type-check and some that don't, and test the error messages
3376 as syntax highlighting, cross-referencing, and so on. The
3382 the change in behavior.
3388 Clang ``-cc1`` supports the ``-verify`` command line option as a way to
3389 validate diagnostic behavior. This option will use special comments within the
3390 test file to verify that expected diagnostics appear in the correct source
3391 locations. If all of the expected diagnostics match the actual output of Clang,
3392 then the invocation will return normally. If there are discrepancies between
3393 the expected and actual output, Clang will emit detailed information about
3402 If the test is run and the expected error is emitted on the expected line, the
3403 diagnostic verifier will pass. However, if the expected error does not appear
3405 appear, the diagnostic verifier will fail and emit information as to why.
3407 The ``-verify`` command optionally accepts a comma-delimited list of one or
3413 ``RUN:`` lines in the same test file which result in differing diagnostic
3424 The verifier will recognize ``foo-error`` and ``bar-error`` as special comments
3425 but will not recognize ``expected-error`` as one because the ``-verify`` line
3427 because an unexpected diagnostic would appear on the declaration of ``E``.
3436 on the line that has the diagnostic, use
3438 warning, remark, or note (respectively), and place the expected text between
3439 ``{{`` and ``}}`` markers. The full text doesn't have to be included, only
3440 enough to ensure that the correct diagnostic was emitted. (Note: full text
3444 For a full description of the matching behavior, including more complex
3447 Here's an example of the most commonly used way to specify expected
3454 You can place as many diagnostics on one line as you wish. To make the code
3455 more readable, you can use slash-newline to separate out the diagnostics.
3457 Alternatively, it is possible to specify the line on which the diagnostic
3465 The line number may be absolute (as above), or relative to the current line by
3466 prefixing the number with either ``+`` or ``-``.
3468 If the diagnostic is generated in a separate file, for example in a shared
3469 header file, it may be beneficial to be able to declare the file in which the
3470 diagnostic will appear, rather than placing the ``expected-*`` directive in the
3471 actual file itself. This can be done using the following syntax:
3477 The path can be absolute or relative and the same search paths will be used as
3478 for ``#include`` directives. The line number in an external file may be
3480 the included file is, for example, a system header where the actual line number
3483 As an alternative to specifying a fixed line number, the location of a
3484 diagnostic can instead be indicated by a marker of the form ``#<marker>``.
3486 appending the marker to the diagnostic with ``@#<marker>``, as with:
3494 The name of a marker used in a directive must be unique within the compilation.
3496 The simple syntax above allows each specification to match exactly one
3497 diagnostic. You can use the extended syntax to customize this. The extended
3500 integer. This allows the diagnostic to appear as many times as specified. For
3507 Where the diagnostic is expected to occur a minimum number of times, this can
3508 be specified by appending a ``+`` to the number. For example:
3515 In the first example, the diagnostic becomes optional, i.e. it will be
3517 the second example, the diagnostic must occur at least once. As a short-hand,
3530 In this example, the diagnostic may appear only once, if at all.
3537 The default matching mode is simple string, which looks for the expected text
3538 that appears between the first `{{` and `}}` pair of the comment. The string is
3539 interpreted just as-is, with one exception: the sequence `\n` is converted to a
3540 single newline character. This mode matches the emitted diagnostic when the
3541 text appears as a substring at any position of the emitted message.
3543 To enable matching against desired strings that contain `}}` or `{{`, the
3552 The intent is to allow the delimeter to be wider than the longest `{` or `}`
3553 brace sequence in the content, so that if your expected text contains `{{{`
3556 Regex matching mode may be selected by appending ``-re`` to the diagnostic type
3557 and including regexes wrapped in double curly braces (`{{` and `}}`) in the
3583 The common theme among all the various feature tests is that they are a utility
3586 lingering bugs, may only work on some targets, etc. We use the following
3588 result value for the feature test):
3595 If the answer to any of these is "yes", the feature test macro should either
3596 not be defined or there should be very strong rationale for why the issues
3597 should not prevent defining it. Note, it is acceptable to define the feature
3601 claim support for the feature but it does useful things, users can still use it
3603 for a feature that has significant bugs, we've eliminated most of the utility
3607 The status reported by the feature test macro should always be reflected in the
3608 language support page for the corresponding feature (`C++
3611 more nuanced information to the user as well, such as claiming partial support