[clang-repl] Simplify the value printing logic to enable out-of-process. (#107737)

This patch improves the design of the IncrementalParser and Interpreter
classes. Now the incremental parser is onl

[clang-repl] Simplify the value printing logic to enable out-of-process. (#107737)

This patch improves the design of the IncrementalParser and Interpreter
classes. Now the incremental parser is only responsible for building the
partial translation unit declaration and the AST, while the Interpreter
fills in the lower level llvm::Module and other JIT-related
infrastructure. Finally the Interpreter class now orchestrates the AST
and the LLVM IR with the IncrementalParser and IncrementalExecutor
classes.

The design improvement allows us to rework some of the logic that
extracts an interpreter value into the clang::Value object. The new
implementation simplifies use-cases which are used for out-of-process
execution by allowing interpreter to be inherited or customized with an
clang::ASTConsumer.

This change will enable completing the pretty printing work which is in
llvm/llvm-project#84769

show more ...

[BPF] Fix linking issues in static map initializers (#91310)

When BPF object files are linked with bpftool, every symbol must be
accompanied by BTF info. Ensure that extern functions referenced by

[BPF] Fix linking issues in static map initializers (#91310)

When BPF object files are linked with bpftool, every symbol must be
accompanied by BTF info. Ensure that extern functions referenced by
global variable initializers are included in BTF.

The primary motivation is "static" initialization of PROG maps:

```c
extern int elsewhere(struct xdp_md *);

struct {
__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
__uint(max_entries, 1);
__type(key, int);
__type(value, int);
__array(values, int (struct xdp_md *));
} prog_map SEC(".maps") = { .values = { elsewhere } };
```

BPF backend needs debug info to produce BTF. Debug info is not
normally generated for external variables and functions. Previously, it
was solved differently for variables (collecting variable declarations
in ExternalDeclarations vector) and functions (logic invoked during
codegen in CGExpr.cpp).

This patch generalises ExternalDefclarations to include both function
and variable declarations. This change ensures that function references
are not missed no matter the context. Previously external functions
referenced in constant expressions lacked debug info.

show more ...

Fix clang reject valid C++ code after d999ce0302f06d250f6d496b56a5a5f (#94471)

The incremental processing mode doesn't seem to work well for C++, see
the
https://github.com/llvm/llvm-project/pull/

Fix clang reject valid C++ code after d999ce0302f06d250f6d496b56a5a5f (#94471)

The incremental processing mode doesn't seem to work well for C++, see
the
https://github.com/llvm/llvm-project/pull/89804#issuecomment-2149840711
for details.

show more ...

Reland "[clang-repl] Extend the C support. (#89804)"

Original commit message:"

[clang-repl] Extend the C support. (#89804)

The IdResolver chain is the main way for C to implement lookup rules. Ev

Reland "[clang-repl] Extend the C support. (#89804)"

Original commit message:"

[clang-repl] Extend the C support. (#89804)

The IdResolver chain is the main way for C to implement lookup rules. Every new
partial translation unit caused clang to exit the top-most scope which in turn
cleaned up the IdResolver chain. That was not an issue for C++ because its
lookup is implemented on the level of declaration contexts.

This patch keeps the IdResolver chain across partial translation units
maintaining proper C-style lookup infrastructure.
"

It was reverted in dfdf1c5fe45a82b9c578306f3d7627fd251d63f8 because it broke the
bots of lldb. This failure was subtle to debug but the current model does not
work well with ObjectiveC support in lldb. This patch does cleans up the
partial translation units in ObjectiveC. In future if we want to support
ObjectiveC we need to understand what exactly lldb is doing when recovering from
errors...

show more ...

Revert "[clang-repl] Extend the C support. (#89804)"

This reverts commit 253c28fa829cee0104c2fc59ed1a958980b5138c.

This commit is causing failures on the lldb CI bots, e.g.
https://ci.swift.org/vie

Revert "[clang-repl] Extend the C support. (#89804)"

This reverts commit 253c28fa829cee0104c2fc59ed1a958980b5138c.

This commit is causing failures on the lldb CI bots, e.g.
https://ci.swift.org/view/all/job/llvm.org/view/LLDB/job/as-lldb-cmake/4307/

On my local macOS desktop build,
```
bin/lldb-dotest -p TestImportBuiltinFileID.py
Assertion failed: (D->getLexicalDeclContext() == this && "Decl inserted into wrong lexical context"), function addHiddenDecl, file DeclBase.cpp, line 1692.

6 libsystem_c.dylib 0x0000000185f0b8d0 abort + 128
7 libsystem_c.dylib 0x0000000185f0abc8 err + 0
8 liblldb.19.0.0git.dylib 0x00000001311e5800 clang::DeclContext::addHiddenDecl(clang::Decl*) + 120
9 liblldb.19.0.0git.dylib 0x00000001311e5978 clang::DeclContext::addDecl(clang::Decl*) + 32
10 liblldb.19.0.0git.dylib 0x000000012f617b48 clang::Sema::ActOnStartTopLevelStmtDecl(clang::Scope*) + 64
11 liblldb.19.0.0git.dylib 0x000000012eaf76c8 clang::Parser::ParseTopLevelStmtDecl() + 208
12 liblldb.19.0.0git.dylib 0x000000012ec051fc clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) + 3412
13 liblldb.19.0.0git.dylib 0x000000012ec03274 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) + 2020
14 liblldb.19.0.0git.dylib 0x000000012eaca860 clang::ParseAST(clang::Sema&, bool, bool) + 604
15 liblldb.19.0.0git.dylib 0x000000012e8554c0 clang::ASTFrontendAction::ExecuteAction() + 308
16 liblldb.19.0.0git.dylib 0x000000012e854c78 clang::FrontendAction::Execute() + 124
17 liblldb.19.0.0git.dylib 0x000000012e76dcfc clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) + 984
18 liblldb.19.0.0git.dylib 0x000000012e784500 compileModuleImpl(clang::CompilerInstance&, clang::SourceLocation, llvm::StringRef, clang::FrontendInputFile, llvm::StringRef, llvm::StringRef, llvm::function_ref<void (clang::CompilerInstance&)>, llvm::function_ref<void (clang::CompilerInstance&)>)::$_1::operator()() const + 52
```
Reverting until Vassil has a chance to look int oit.

show more ...

[clang-repl] Extend the C support. (#89804)

The IdResolver chain is the main way for C to implement lookup rules.
Every new partial translation unit caused clang to exit the top-most
scope which i

[clang-repl] Extend the C support. (#89804)

The IdResolver chain is the main way for C to implement lookup rules.
Every new partial translation unit caused clang to exit the top-most
scope which in turn cleaned up the IdResolver chain. That was not an
issue for C++ because its lookup is implemented on the level of
declaration contexts.

This patch keeps the IdResolver chain across partial translation units
maintaining proper C-style lookup infrastructure.

show more ...

Reland "[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)"

Original commit message: "

Clang's CodeGen is designed to work with a single llvm::Module. In many c

Reland "[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)"

Original commit message: "

Clang's CodeGen is designed to work with a single llvm::Module. In many cases
for convenience various CodeGen parts have a reference to the llvm::Module
(TheModule or Module) which does not change when a new module is pushed.
However, the execution engine wants to take ownership of the module which does
not map well to CodeGen's design. To work this around we clone the module and
pass it down.

With some effort it is possible to teach CodeGen to ask the CodeGenModule for
its current module and that would have an overall positive impact on CodeGen
improving the encapsulation of various parts but that's not resilient to future
regression.

This patch takes a more conservative approach and keeps the first llvm::Module
empty intentionally and does not pass it to the Jit. That's also not bullet
proof because we have to guarantee that CodeGen does not write on the
blueprint. However, we have inserted some assertions to catch accidental
additions to that canary module.

This change will fixes a long-standing invalid memory access reported by
valgrind when we enable the TBAA optimization passes. It also unblock progress
on https://github.com/llvm/llvm-project/pull/84758.
"

This patch reverts adc4f6233df734fbe3793118ecc89d3584e0c90f and removes
the check of `named_metadata_empty` of the first llvm::Module because on darwin
clang inserts some harmless metadata which we can ignore.

show more ...

Revert "[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)"

This reverts commit ca090452d64e229b539a66379a3be891c4e8f3d8 and
1faf3148fdef34ce0d556ec6a4049e06cbde

Revert "[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)"

This reverts commit ca090452d64e229b539a66379a3be891c4e8f3d8 and
1faf3148fdef34ce0d556ec6a4049e06cbde71b3 because it broke a darwin bot.

show more ...

[Interpreter] Fix warnings

This patch fixes:

clang/lib/Interpreter/IncrementalParser.cpp:214:29: error: moving a
temporary object prevents copy elision [-Werror,-Wpessimizing-move]

clang/lib

[Interpreter] Fix warnings

This patch fixes:

clang/lib/Interpreter/IncrementalParser.cpp:214:29: error: moving a
temporary object prevents copy elision [-Werror,-Wpessimizing-move]

clang/lib/Interpreter/IncrementalParser.cpp:232:22: error: moving a
temporary object prevents copy elision [-Werror,-Wpessimizing-move]

show more ...

[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)

Clang's CodeGen is designed to work with a single llvm::Module. In many
cases
for convenience various CodeGe

[clang-repl] Keep the first llvm::Module empty to avoid invalid memory access. (#89031)

Clang's CodeGen is designed to work with a single llvm::Module. In many
cases
for convenience various CodeGen parts have a reference to the
llvm::Module
(TheModule or Module) which does not change when a new module is pushed.
However, the execution engine wants to take ownership of the module
which does
not map well to CodeGen's design. To work this around we clone the
module and
pass it down.

With some effort it is possible to teach CodeGen to ask the
CodeGenModule for
its current module and that would have an overall positive impact on
CodeGen
improving the encapsulation of various parts but that's not resilient to
future
regression.

This patch takes a more conservative approach and keeps the first
llvm::Module
empty intentionally and does not pass it to the Jit. That's also not
bullet
proof because we have to guarantee that CodeGen does not write on the
blueprint. However, we have inserted some assertions to catch accidental
additions to that canary module.

This change will fixes a long-standing invalid memory access reported by
valgrind when we enable the TBAA optimization passes. It also unblock
progress
on https://github.com/llvm/llvm-project/pull/84758.

show more ...

[clang-repl] Fix remove invalidates iterators in CleanUpPTU() (#85378)

Using remove() on DeclContext::lookup_result list invalidates iterators.

This assertion failure was one (fortunate) symptom:

[clang-repl] Fix remove invalidates iterators in CleanUpPTU() (#85378)

Using remove() on DeclContext::lookup_result list invalidates iterators.

This assertion failure was one (fortunate) symptom:
```
clang/include/clang/AST/DeclBase.h:1337: reference clang::DeclListNode::iterator::operator*() const: Assertion `Ptr && "dereferencing end() iterator"' failed.
```

show more ...

Reland "[clang-repl] support code completion at a REPL."

Original commit message:
"
This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a li

Reland "[clang-repl] support code completion at a REPL."

Original commit message:
"
This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a list completer for LineEditor, a
CompletionConsumer from SemaCodeCompletion, and the ASTUnit::codeComplete method.
The first component serves as the main entry point of handling interactive inputs.

Because a completion point for a compiler instance has to be unchanged once it
is set, an incremental compiler instance is created for each code
completion. Such a compiler instance carries over AST context source from the
main interpreter compiler in order to obtain declarations or bindings from
previous input in the same REPL session.

The most important API codeComplete in Interpreter/CodeCompletion is a thin
wrapper that calls with ASTUnit::codeComplete with necessary arguments, such as
a code completion point and a ReplCompletionConsumer, which communicates
completion results from SemaCodeCompletion back to the list completer for the
REPL.

In addition, PCC_TopLevelOrExpression and CCC_TopLevelOrExpression` top levels
were added so that SemaCodeCompletion can treat top level statements like
expression statements at the REPL. For example,

clang-repl> int foo = 42;
clang-repl> f<tab>

From a parser's persective, the cursor is at a top level. If we used code
completion without any changes, PCC_Namespace would be supplied to
Sema::CodeCompleteOrdinaryName, and thus the completion results would not
include foo.

Currently, the way we use PCC_TopLevelOrExpression and
CCC_TopLevelOrExpression is no different from the way we use PCC_Statement
and CCC_Statement respectively.

Differential revision: https://reviews.llvm.org/D154382
"

The new patch also fixes clangd and several memory issues that the bots reported
and upload the missing files.

show more ...

Revert "Reland "[clang-repl] support code completion at a REPL.""

This reverts commit 5ab25a42ba70c4b50214b0e78eaaccd30696fa09 due to forgotten
files.

Reland "[clang-repl] support code completion at a REPL."

Original commit message:
"
This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a li

Reland "[clang-repl] support code completion at a REPL."

Original commit message:
"
This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a list completer for LineEditor, a
CompletionConsumer from SemaCodeCompletion, and the ASTUnit::codeComplete method.
The first component serves as the main entry point of handling interactive inputs.

Because a completion point for a compiler instance has to be unchanged once it
is set, an incremental compiler instance is created for each code
completion. Such a compiler instance carries over AST context source from the
main interpreter compiler in order to obtain declarations or bindings from
previous input in the same REPL session.

The most important API codeComplete in Interpreter/CodeCompletion is a thin
wrapper that calls with ASTUnit::codeComplete with necessary arguments, such as
a code completion point and a ReplCompletionConsumer, which communicates
completion results from SemaCodeCompletion back to the list completer for the
REPL.

In addition, PCC_TopLevelOrExpression and CCC_TopLevelOrExpression` top levels
were added so that SemaCodeCompletion can treat top level statements like
expression statements at the REPL. For example,

clang-repl> int foo = 42;
clang-repl> f<tab>

From a parser's persective, the cursor is at a top level. If we used code
completion without any changes, PCC_Namespace would be supplied to
Sema::CodeCompleteOrdinaryName, and thus the completion results would not
include foo.

Currently, the way we use PCC_TopLevelOrExpression and
CCC_TopLevelOrExpression is no different from the way we use PCC_Statement
and CCC_Statement respectively.

Differential revision: https://reviews.llvm.org/D154382
"

The new patch also fixes clangd and several memory issues that the bots reported.

show more ...

Revert "[clang-repl] support code completion at a REPL."

This reverts commit eb0e6c3134ef6deafe0a4958e9e1a1214b3c2f14 due to failures in
clangd such as https://lab.llvm.org/buildbot/#/builders/57/bu

Revert "[clang-repl] support code completion at a REPL."

This reverts commit eb0e6c3134ef6deafe0a4958e9e1a1214b3c2f14 due to failures in
clangd such as https://lab.llvm.org/buildbot/#/builders/57/builds/29377

show more ...

[clang-repl] support code completion at a REPL.

This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a list completer for LineEditor, a
Compl

[clang-repl] support code completion at a REPL.

This patch enabled code completion for ClangREPL. The feature was built upon
three existing Clang components: a list completer for LineEditor, a
CompletionConsumer from SemaCodeCompletion, and the ASTUnit::codeComplete method.
The first component serves as the main entry point of handling interactive inputs.

Because a completion point for a compiler instance has to be unchanged once it
is set, an incremental compiler instance is created for each code
completion. Such a compiler instance carries over AST context source from the
main interpreter compiler in order to obtain declarations or bindings from
previous input in the same REPL session.

The most important API codeComplete in Interpreter/CodeCompletion is a thin
wrapper that calls with ASTUnit::codeComplete with necessary arguments, such as
a code completion point and a ReplCompletionConsumer, which communicates
completion results from SemaCodeCompletion back to the list completer for the
REPL.

In addition, PCC_TopLevelOrExpression and CCC_TopLevelOrExpression` top levels
were added so that SemaCodeCompletion can treat top level statements like
expression statements at the REPL. For example,

clang-repl> int foo = 42;
clang-repl> f<tab>

From a parser's persective, the cursor is at a top level. If we used code
completion without any changes, PCC_Namespace would be supplied to
Sema::CodeCompleteOrdinaryName, and thus the completion results would not
include foo.

Currently, the way we use PCC_TopLevelOrExpression and
CCC_TopLevelOrExpression is no different from the way we use PCC_Statement
and CCC_Statement respectively.

Differential revision: https://reviews.llvm.org/D154382

show more ...

[clang-repl][CUDA] Re-land: Initial interactive CUDA support for clang-repl

CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used

[clang-repl][CUDA] Re-land: Initial interactive CUDA support for clang-repl

CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used with all
__device__ functions.

Differential Revision: https://reviews.llvm.org/D146389

show more ...

Reland "Reland [clang-repl] Introduce Value to capture expression results"

This reverts commit 094ab4781262b6cb49d57b0ecdf84b047c879295.

Reland with changing `ParseAndExecute` to `Parse` in
`Interp

Reland "Reland [clang-repl] Introduce Value to capture expression results"

This reverts commit 094ab4781262b6cb49d57b0ecdf84b047c879295.

Reland with changing `ParseAndExecute` to `Parse` in
`Interpreter::create`. This avoid creating JIT instance everytime even
if we don't really need them.

This should fixes failures like https://lab.llvm.org/buildbot/#/builders/38/builds/11955

The original reverted patch also causes GN bot fails on M1. (https://lab.llvm.org/buildbot/#/builders/38/builds/11955)
However, we can't reproduce it so let's reland it and see what happens.
See discussions here: https://reviews.llvm.org/rGd71a4e02277a64a9dece591cdf2b34f15c3b19a0

show more ...

Revert "[clang-repl][CUDA] Initial interactive CUDA support for clang-repl"

This reverts commit 80e7eed6a610ab3c7289e6f9b7ec006bc7d7ae31.

[clang-repl][CUDA] Initial interactive CUDA support for clang-repl

CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used with all

[clang-repl][CUDA] Initial interactive CUDA support for clang-repl

CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used with all
__device__ functions.

Differential Revision: https://reviews.llvm.org/D146389

show more ...

Revert "Reland [clang-repl] Introduce Value to capture expression results"

This reverts commit d71a4e02277a64a9dece591cdf2b34f15c3b19a0.
See http://45.33.8.238/macm1/61024/step_7.txt

Reland [clang-repl] Introduce Value to capture expression results

This reverts commit 7158fd381a0bc0222195d6a07ebb42ea57957bda.
* Fixes endianness issue on big endian machines like PowerPC-bl
* Disa

Reland [clang-repl] Introduce Value to capture expression results

This reverts commit 7158fd381a0bc0222195d6a07ebb42ea57957bda.
* Fixes endianness issue on big endian machines like PowerPC-bl
* Disable tests on platforms that having trouble to support JIT

Signed-off-by: Jun Zhang <jun@junz.org>

show more ...

Revert "[clang-repl] Introduce Value to capture expression results"

This reverts commit a423b7f1d7ca8b263af85944f57a69aa08fc942c.
See https://lab.llvm.org/buildbot/#/changes/95083

[clang-repl] Introduce Value to capture expression results

This is the second part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

This patch impleme

[clang-repl] Introduce Value to capture expression results

This is the second part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

This patch implements a Value class that can be used to carry expression
results in clang-repl. In other words, when we see a top expression
without semi, it will be captured and stored to a Value object. You can
explicitly specify where you want to store the object, like:

```
Value V;
llvm::cantFail(Interp->ParseAndExecute("int x = 42;"));
llvm::cantFail(Interp->ParseAndExecute("x", &V));
```

`V` now stores some useful infomation about `x`, you can get its real
value (42), it's `clang::QualType` or anything interesting.

However, if you don't specify the optional argument, it will be captured
to a local variable, and automatically called `Value::dump`, which is
not implemented yet in this patch.

Signed-off-by: Jun Zhang <jun@junz.org>

show more ...

[clang] Add a new annotation token: annot_repl_input_end

This patch is the first part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

It adds an anno

[clang] Add a new annotation token: annot_repl_input_end

This patch is the first part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

It adds an annotation token which will replace the original EOF token
when we are in the incremental C++ mode. In addition, when we're
parsing an ExprStmt and there's a missing semicolon after the
expression, we set a marker in the annotation token and continue
parsing.

Eventually, we propogate this info in ParseTopLevelStmtDecl and are able
to mark this Decl as something we want to do value printing. Below is a
example:

clang-repl> int x = 42;
clang-repl> x
// `x` is a TopLevelStmtDecl and without a semicolon, we should set
// it's IsSemiMissing bit so we can do something interesting in
// ASTConsumer::HandleTopLevelDecl.

The idea about annotation toke is proposed by Richard Smith, thanks!

Signed-off-by: Jun Zhang <jun@junz.org>

Differential Revision: https://reviews.llvm.org/D148997

show more ...